1 production 1.3   /home/production/cvs/JSOC/doc/whattodo_start_stop_lev1_0_sums.txt 28Jun2012

  2 production 1.1 
  3                

  4 production 1.2 Do as user production or jsocprod as indicated. Do in this order:

  5 production 1.1 
  6                NOTE: Now the drms/sums restart feature is working, there is no need 
  7                to do the "Stop lev1" and "Start lev0" sections below.
  8                
  9                STOP:
 10                =====
 11                

 12 production 1.2 On cl1n003: Stop lev1 processing for AIA as jsocprod
 13                ----------------------------------------------------

 14 production 1.1 > touch /usr/local/logs/lev1/build_mgr_stop_aia
 15                
 16                Wait until you see on the screen running build_lev1_mgr on cl1n003:
 17                
 18                Stop file /usr/local/logs/lev1/build_mgr_stop_aia seen.
 19                Wait until all children are done and exit...
 20                
 21                Of course, if you not local with the screen, use ps.
 22                
 23                (note: ignore any core dump)
 24                

 25 production 1.3 

 26 production 1.2 On cl1n002: Stop lev1 processing for HMI as jsocprod
 27                ----------------------------------------------------

 28 production 1.1 > touch /usr/local/logs/lev1/build_mgr_stop_hmi
 29                
 30                Wait until you see on the screen running build_lev1_mgr on cl1n002:
 31                
 32                Stop file /usr/local/logs/lev1/build_mgr_stop_hmi seen.
 33                Wait until all children are done and exit...
 34                
 35                Of course, if you not local with the screen, use ps.
 36                
 37                (note: ignore any core dump)
 38                
 39                

 40 production 1.2 On cl1n001: Stop lev0 processing as jsocprod
 41                --------------------------------------------

 42 production 1.1 

 43 production 1.2 > cd /home/jsocprod

 44 production 1.1 > stop_lev0_HMI.pl
 45                
 46                This may take a few minutes while the processing of ingest_lev0 for
 47                VC02 and VC05 completes to a commit point. 

 48 production 1.3 If the ingest_lev0 process is hung, it may not see the stop file signal.
 49                Then do a kill -9 of the ingest_lev0 processes.

 50 production 1.4 Make sure ingest_lev0 is not running before you try to restart it.

 51 production 1.3 

 52 production 1.1 Likewise for AIA VC01 and VC04:
 53                
 54                > stop_lev0_AIA.pl
 55                

 56 production 1.3 

 57 production 1.2 On j1: Stop SUMS as production

 58 production 1.1 -------------------------------------
 59                
 60                > cd /home/production/cvs/JSOC/base/sums/scripts

 61 production 1.2 > sum_stop_j1.MULTI -f		(note: may need the -f(orce) flag)

 62 production 1.1 
 63                +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

 64 production 1.3 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

 65 production 1.1 
 66                START:
 67                =====
 68                

 69 production 1.2 On j1: Start SUMS as production

 70 production 1.1 -------------------------------------
 71                
 72                > cd /home/production/cvs/JSOC/base/sums/scripts

 73 production 1.2 > sum_start_j1.MULTI

 74 production 1.1 

 75 production 1.3 

 76 production 1.2 On d02: Restart t950view as production
 77                --------------------------------------

 78 production 1.1 This is optional if you want to view the tape activity.
 79                Restore the minimized t950view window and hit the Exit button (upper right),
 80                or use ps and kill.
 81                > /home/production/cvs/JSOC/base/sums/scripts/t950view -i jsoc_sums
 82                (password: hmi4sdo)
 83                
 84                

 85 production 1.2 On cl1n001: Start lev0 processing (if needed) as jsocprod
 86                ---------------------------------------------------------
 87                This should be done through 'screen' running on cl1n001.
 88                For "How to Use 'screen'" see: 
 89                http://linux.about.com/od/Bash_Scripting_Solutions/a/How-To-Use-The-Gnu-Linux-Screen-Terminal-Window-Manager.htm
 90                This will allow the lev1 processing to continue if your w/s (xim) crashes, 
 91                and the screens to be re-attached to, when xim reboots, or from another w/s.
 92                
 93                xim:/home/jim> ssh jsocprod@j1
 94                Password: 
 95                Last login: Fri Apr 27 12:14:41 2012 from xim.stanford.edu
 96                Fri Apr 27 12:20:49 PDT 2012
 97                j1:/home/jsocprod>  ssh cl1n001
 98                cl1n001:/home/jsocprod> screen
 99                Directory: /home/jsocprod
100                csh> cd

101 production 1.3 cl1n001:/home/jsocprod> Ctrl-a c  (to create anothe screen)

102 production 1.2 To toggle between screens do: Ctrl-a space
103                Or to go to screen 2: Ctrl-a 2
104                To disconnect from 'screen' do: Ctrl-a d
105                (You will still see a process running called SCREEN).
106                To reconnect to a previous screen (after disconnect or machine restart or
107                from another workstation from a cl1n001 window):
108                > screen -r
109                To kill a screen window: Ctrl-a K  (on some machine small 'k')
110                To see all your screens created and which on you're on now: Ctrl-a w
111                To see all screen key bindings: Ctrl-a ?

112 production 1.1 

113 production 1.2 In screen 0 do:

114 production 1.1 
115                !!Make sure run Phil's script for watchlev0 in the background on cl1n001:
116                /home/production/cvs/JSOC/base/sums/scripts/get_dcs_times.csh
117                

118 production 1.2 > cd /home/jsocprod
119                > doingestlev0_HMI.pl &
120                > doingestlev0_AIA.pl &
121                
122                On cl1n002: Start lev1 processing for HMI (if needed) as jsocprod
123                -----------------------------------------------------------------
124                Run a 'screen' on cl1n002:
125                xim:/home/jim> ssh jsocprod@j1
126                Password:
127                Last login: Thu Apr 26 14:54:53 2012 from xim.stanford.edu
128                Directory: /home/jsocprod
129                j1:/home/jsocprod> ssh cl1n002
130                > screen
131                > cd /home/jsocprod

132 production 1.1 > build_lev1_mgr mode=recnum

133 production 1.2 instru=hmi dsin=hmi.lev0a dsout=hmi.lev1_nrt brec=0 erec=0

134 production 1.1 

135 production 1.2 On cl1n003: Start lev1 processing for AIA (if needed) as jsocprod
136                -----------------------------------------------------------------
137                Run a 'screen' on cl1n003
138                xim:/home/jim> ssh jsocprod@j1
139                Password:
140                Last login: Thu Apr 26 14:54:53 2012 from xim.stanford.edu
141                Directory: /home/jsocprod
142                j1:/home/jsocprod> ssh cl1n003
143                > screen
144                > cd /home/jsocprod

145 production 1.1 > build_lev1_mgr mode=recnum
146                instru=aia dsin=aia.lev0 dsout=aia.lev1_nrt2 brec=0 erec=0
147                
148                ===========================================================================
149                For data capture machines see:
150                
151                > cd /home/production/cvs/JSOC/doc
152                
153                whattodo_dcs.txt
154                (dsc0_just_rebooted.txt may be needed if you want to do Pg warmstart)
155                
156                If it's dcs1 that you need, then you have to transcribe a bit.
157                Remember that dcs0 and dcs1 databases are in warm standby on dcs2.
158                If you want to ignore the instructions for this for now, I'll take care 
159                of it when I'm back. Just let me know that you did so.
160                
161                Info on log files and cron jobs is in logs_cron_etc_admin.txt
162