JSOC/doc/whattodo_start_stop_lev1_0_sums.txt - annotate

Return to whattodo_start_stop_lev1_0_sums.txt CVS log

Up to [Development] / JSOC / doc

1 production 1.3 /home/production/cvs/JSOC/doc/whattodo_start_stop_lev1_0_sums.txt 28Jun2012
2 production 1.5 /home/production/cvs/JSOC/doc/whattodo_start_stop_lev1_0_sums.txt 25Oct2013
3 production 1.1 4
5 production 1.2 Do as user production or jsocprod as indicated. Do in this order:
6 production 1.1 7 NOTE: Now the drms/sums restart feature is working, there is no need 8 to do the "Stop lev1" and "Start lev0" sections below. 9 10 STOP: 11 ===== 12
13 production 1.2 On cl1n003: Stop lev1 processing for AIA as jsocprod 14 ----------------------------------------------------
15 production 1.1 > touch /usr/local/logs/lev1/build_mgr_stop_aia 16 17 Wait until you see on the screen running build_lev1_mgr on cl1n003: 18 19 Stop file /usr/local/logs/lev1/build_mgr_stop_aia seen. 20 Wait until all children are done and exit... 21 22 Of course, if you not local with the screen, use ps. 23 24 (note: ignore any core dump) 25
26 production 1.3
27 production 1.2 On cl1n002: Stop lev1 processing for HMI as jsocprod 28 ----------------------------------------------------
29 production 1.1 > touch /usr/local/logs/lev1/build_mgr_stop_hmi 30 31 Wait until you see on the screen running build_lev1_mgr on cl1n002: 32 33 Stop file /usr/local/logs/lev1/build_mgr_stop_hmi seen. 34 Wait until all children are done and exit... 35 36 Of course, if you not local with the screen, use ps. 37 38 (note: ignore any core dump) 39 40
41 production 1.2 On cl1n001: Stop lev0 processing as jsocprod 42 --------------------------------------------
43 production 1.1
44 production 1.2 > cd /home/jsocprod
45 production 1.1 > stop_lev0_HMI.pl 46 47 This may take a few minutes while the processing of ingest_lev0 for 48 VC02 and VC05 completes to a commit point.
49 production 1.3 If the ingest_lev0 process is hung, it may not see the stop file signal. 50 Then do a kill -9 of the ingest_lev0 processes.
51 production 1.4 Make sure ingest_lev0 is not running before you try to restart it.
52 production 1.3
53 production 1.1 Likewise for AIA VC01 and VC04: 54 55 > stop_lev0_AIA.pl 56
57 production 1.3
58 production 1.2 On j1: Stop SUMS as production
59 production 1.1 ------------------------------------- 60 61 > cd /home/production/cvs/JSOC/base/sums/scripts
62 production 1.2 > sum_stop_j1.MULTI -f (note: may need the -f(orce) flag)
63 production 1.1 64 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
65 production 1.3 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
66 production 1.1 67 START: 68 ===== 69
70 production 1.2 On j1: Start SUMS as production
71 production 1.1 ------------------------------------- 72
73 production 1.5 !!!NOTE: sums will be moved to k1. Use these: 74 sum_start_k1.MULTI sum_stop_k1.MULTI 75
76 production 1.1 > cd /home/production/cvs/JSOC/base/sums/scripts
77 production 1.2 > sum_start_j1.MULTI
78 production 1.1
79 production 1.3
80 production 1.2 On d02: Restart t950view as production 81 --------------------------------------
82 production 1.1 This is optional if you want to view the tape activity. 83 Restore the minimized t950view window and hit the Exit button (upper right), 84 or use ps and kill. 85 > /home/production/cvs/JSOC/base/sums/scripts/t950view -i jsoc_sums 86 (password: hmi4sdo) 87 88
89 production 1.2 On cl1n001: Start lev0 processing (if needed) as jsocprod 90 --------------------------------------------------------- 91 This should be done through 'screen' running on cl1n001. 92 For "How to Use 'screen'" see: 93 http://linux.about.com/od/Bash_Scripting_Solutions/a/How-To-Use-The-Gnu-Linux-Screen-Terminal-Window-Manager.htm 94 This will allow the lev1 processing to continue if your w/s (xim) crashes, 95 and the screens to be re-attached to, when xim reboots, or from another w/s. 96 97 xim:/home/jim> ssh jsocprod@j1 98 Password: 99 Last login: Fri Apr 27 12:14:41 2012 from xim.stanford.edu 100 Fri Apr 27 12:20:49 PDT 2012 101 j1:/home/jsocprod> ssh cl1n001 102 cl1n001:/home/jsocprod> screen 103 Directory: /home/jsocprod 104 csh> cd
105 production 1.3 cl1n001:/home/jsocprod> Ctrl-a c (to create anothe screen)
106 production 1.2 To toggle between screens do: Ctrl-a space 107 Or to go to screen 2: Ctrl-a 2 108 To disconnect from 'screen' do: Ctrl-a d 109 (You will still see a process running called SCREEN). 110 To reconnect to a previous screen (after disconnect or machine restart or 111 from another workstation from a cl1n001 window): 112 > screen -r 113 To kill a screen window: Ctrl-a K (on some machine small 'k') 114 To see all your screens created and which on you're on now: Ctrl-a w 115 To see all screen key bindings: Ctrl-a ?
116 production 1.1
117 production 1.2 In screen 0 do:
118 production 1.1 119 !!Make sure run Phil's script for watchlev0 in the background on cl1n001: 120 /home/production/cvs/JSOC/base/sums/scripts/get_dcs_times.csh 121
122 production 1.2 > cd /home/jsocprod 123 > doingestlev0_HMI.pl & 124 > doingestlev0_AIA.pl & 125 126 On cl1n002: Start lev1 processing for HMI (if needed) as jsocprod 127 ----------------------------------------------------------------- 128 Run a 'screen' on cl1n002: 129 xim:/home/jim> ssh jsocprod@j1 130 Password: 131 Last login: Thu Apr 26 14:54:53 2012 from xim.stanford.edu 132 Directory: /home/jsocprod 133 j1:/home/jsocprod> ssh cl1n002 134 > screen 135 > cd /home/jsocprod
136 production 1.1 > build_lev1_mgr mode=recnum
137 production 1.2 instru=hmi dsin=hmi.lev0a dsout=hmi.lev1_nrt brec=0 erec=0
138 production 1.1
139 production 1.2 On cl1n003: Start lev1 processing for AIA (if needed) as jsocprod 140 ----------------------------------------------------------------- 141 Run a 'screen' on cl1n003 142 xim:/home/jim> ssh jsocprod@j1 143 Password: 144 Last login: Thu Apr 26 14:54:53 2012 from xim.stanford.edu 145 Directory: /home/jsocprod 146 j1:/home/jsocprod> ssh cl1n003 147 > screen 148 > cd /home/jsocprod
149 production 1.1 > build_lev1_mgr mode=recnum 150 instru=aia dsin=aia.lev0 dsout=aia.lev1_nrt2 brec=0 erec=0 151 152 =========================================================================== 153 For data capture machines see: 154 155 > cd /home/production/cvs/JSOC/doc 156 157 whattodo_dcs.txt 158 (dsc0_just_rebooted.txt may be needed if you want to do Pg warmstart) 159 160 If it's dcs1 that you need, then you have to transcribe a bit. 161 Remember that dcs0 and dcs1 databases are in warm standby on dcs2. 162 If you want to ignore the instructions for this for now, I'll take care 163 of it when I'm back. Just let me know that you did so. 164 165 Info on log files and cron jobs is in logs_cron_etc_admin.txt 166

Karen Tian