Return to whattodo_start_stop_lev1_0_sums.txt CVS log

Up to [Development] / JSOC / doc

*** empty log message ***

  /home/production/cvs/JSOC/doc/whattodo_start_stop_lev1_0_sums.txt 28Jun2012
  /home/production/cvs/JSOC/doc/whattodo_start_stop_lev1_0_sums.txt 25Oct2013


Do as user production or jsocprod as indicated. Do in this order:

NOTE: Now the drms/sums restart feature is working, there is no need 
to do the "Stop lev1" and "Start lev0" sections below.

STOP:
=====

On cl1n003: Stop lev1 processing for AIA as jsocprod
----------------------------------------------------
> touch /usr/local/logs/lev1/build_mgr_stop_aia

Wait until you see on the screen running build_lev1_mgr on cl1n003:

Stop file /usr/local/logs/lev1/build_mgr_stop_aia seen.
Wait until all children are done and exit...

Of course, if you not local with the screen, use ps.

(note: ignore any core dump)


On cl1n002: Stop lev1 processing for HMI as jsocprod
----------------------------------------------------
> touch /usr/local/logs/lev1/build_mgr_stop_hmi

Wait until you see on the screen running build_lev1_mgr on cl1n002:

Stop file /usr/local/logs/lev1/build_mgr_stop_hmi seen.
Wait until all children are done and exit...

Of course, if you not local with the screen, use ps.

(note: ignore any core dump)


On cl1n001: Stop lev0 processing as jsocprod
--------------------------------------------

> cd /home/jsocprod
> stop_lev0_HMI.pl

This may take a few minutes while the processing of ingest_lev0 for
VC02 and VC05 completes to a commit point. 
If the ingest_lev0 process is hung, it may not see the stop file signal.
Then do a kill -9 of the ingest_lev0 processes.
Make sure ingest_lev0 is not running before you try to restart it.

Likewise for AIA VC01 and VC04:

> stop_lev0_AIA.pl


On j1: Stop SUMS as production
-------------------------------------

> cd /home/production/cvs/JSOC/base/sums/scripts
> sum_stop_j1.MULTI -f		(note: may need the -f(orce) flag)

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

START:
=====

On j1: Start SUMS as production
-------------------------------------

!!!NOTE: sums will be moved to k1. Use these:
sum_start_k1.MULTI  sum_stop_k1.MULTI

> cd /home/production/cvs/JSOC/base/sums/scripts
> sum_start_j1.MULTI


On d02: Restart t950view as production
--------------------------------------
This is optional if you want to view the tape activity.
Restore the minimized t950view window and hit the Exit button (upper right),
or use ps and kill.
> /home/production/cvs/JSOC/base/sums/scripts/t950view -i jsoc_sums
(password: hmi4sdo)


On cl1n001: Start lev0 processing (if needed) as jsocprod
---------------------------------------------------------
This should be done through 'screen' running on cl1n001.
For "How to Use 'screen'" see: 
http://linux.about.com/od/Bash_Scripting_Solutions/a/How-To-Use-The-Gnu-Linux-Screen-Terminal-Window-Manager.htm
This will allow the lev1 processing to continue if your w/s (xim) crashes, 
and the screens to be re-attached to, when xim reboots, or from another w/s.

xim:/home/jim> ssh jsocprod@j1
Password: 
Last login: Fri Apr 27 12:14:41 2012 from xim.stanford.edu
Fri Apr 27 12:20:49 PDT 2012
j1:/home/jsocprod>  ssh cl1n001
cl1n001:/home/jsocprod> screen
Directory: /home/jsocprod
csh> cd
cl1n001:/home/jsocprod> Ctrl-a c  (to create anothe screen)
To toggle between screens do: Ctrl-a space
Or to go to screen 2: Ctrl-a 2
To disconnect from 'screen' do: Ctrl-a d
(You will still see a process running called SCREEN).
To reconnect to a previous screen (after disconnect or machine restart or
from another workstation from a cl1n001 window):
> screen -r
To kill a screen window: Ctrl-a K  (on some machine small 'k')
To see all your screens created and which on you're on now: Ctrl-a w
To see all screen key bindings: Ctrl-a ?

In screen 0 do:

!!Make sure run Phil's script for watchlev0 in the background on cl1n001:
/home/production/cvs/JSOC/base/sums/scripts/get_dcs_times.csh

> cd /home/jsocprod
> doingestlev0_HMI.pl &
> doingestlev0_AIA.pl &

On cl1n002: Start lev1 processing for HMI (if needed) as jsocprod
-----------------------------------------------------------------
Run a 'screen' on cl1n002:
xim:/home/jim> ssh jsocprod@j1
Password:
Last login: Thu Apr 26 14:54:53 2012 from xim.stanford.edu
Directory: /home/jsocprod
j1:/home/jsocprod> ssh cl1n002
> screen
> cd /home/jsocprod
> build_lev1_mgr mode=recnum
instru=hmi dsin=hmi.lev0a dsout=hmi.lev1_nrt brec=0 erec=0

On cl1n003: Start lev1 processing for AIA (if needed) as jsocprod
-----------------------------------------------------------------
Run a 'screen' on cl1n003
xim:/home/jim> ssh jsocprod@j1
Password:
Last login: Thu Apr 26 14:54:53 2012 from xim.stanford.edu
Directory: /home/jsocprod
j1:/home/jsocprod> ssh cl1n003
> screen
> cd /home/jsocprod
> build_lev1_mgr mode=recnum
instru=aia dsin=aia.lev0 dsout=aia.lev1_nrt2 brec=0 erec=0

===========================================================================
For data capture machines see:

> cd /home/production/cvs/JSOC/doc

whattodo_dcs.txt
(dsc0_just_rebooted.txt may be needed if you want to do Pg warmstart)

If it's dcs1 that you need, then you have to transcribe a bit.
Remember that dcs0 and dcs1 databases are in warm standby on dcs2.
If you want to ignore the instructions for this for now, I'll take care 
of it when I'm back. Just let me know that you did so.

Info on log files and cron jobs is in logs_cron_etc_admin.txt