(file) Return to whattodolev0.txt CVS log (file) (dir) Up to [Development] / JSOC / doc

File: [Development] / JSOC / doc / whattodolev0.txt (download)
Revision: 1.5, Thu Oct 16 17:46:58 2008 UTC (14 years, 11 months ago) by production
Branch: MAIN
CVS Tags: Ver_4-7, NetDRMS_Ver_1-1
Changes since 1.4: +3 -2 lines
*** empty log message ***

		/home/production/cvs/JSOC/doc/whattodolev0.txt  16Oct2008

	Running Datacapture & Pipeline Backend lev0 Processing

NOTE: For now, this is all done from the xim w/s (Jim's office)


NOTE:IMPORTANT: Please keep in mind that each data capture machine has its
own independent /home/production.

1. The Datacapture system for aia/hmi is by convention dcs0/dcs1 respectively. 
If the spare dcs2 is to be put in place, it is renamed dcs0 or dcs1, and the
original machine is renamed dcs2.

1a. The spare dcs2 normally servers as a backup destination of the postgres
running on dcs0 and dcs1. You should see this postgres cron job on dcs0
and dcs1, respectively:

0,20,40 * * * * /var/lib/pgsql/rsync_pg_dcs0_to_dcs2.pl
0,20,40 * * * * /var/lib/pgsql/rsync_pg_dcs1_to_dcs2.pl

For this to work, this must be done on dcs0, dcs1 and dcs2, as user
postgres, after any reboot:

> ssh-agent | head -2 > /var/lib/pgsql/ssh-agent.env
> chmod 600 /var/lib/pgsql/ssh-agent.env
> source /var/lib/pgsql/ssh-agent.env
> ssh-add
(The password is written on my whiteboard (same as production's))

2. Login as user production via j0. (password is on Jim's whiteboard).

3. The Postgres must be running and is started automatically on boot:

> ps -ef |grep pg
postgres  4631     1  0 Mar11 ?        00:06:21 /usr/bin/postmaster -D /var/lib/pgsql/data

4. The root of the datacapture tree is /home/production/cvs/JSOC.
The producton runs as user id 388.

5. The sum_svc is normally running:

> ps -ef |grep sum_svc
388      26958     1  0 Jun09 pts/0    00:00:54 sum_svc jsocdc

Note the SUMS database is jsocdc. This is a separate DB on each dcs.

6. To start/restart the sum_svc and related programs (e.g. tape_svc) do:

> sum_start_dc
sum_start at 2008.06.16_13:32:23
** NOTE: "soc_pipe_scp jsocdc" still running
Do you want me to do a sum_stop followed by a sum_start for you (y or n):

You would normally answer 'y' here.

7. To run the datacapture gui that will display the data, mark it for archive,
optionally extract lev0 and send it on the the pipeline backend, do this:

> cd /home/production/cvs/JSOC/proj/datacapture/scripts>
> ./socdc

All you would normally do is hit "Start Instances for HMI" or AIA for
what datacapture machine you are on.

8. To optionally extract lev0 do this:

> touch /usr/local/logs/soc/LEV0FILEON

To stop lev0:

> /bin/rm /usr/local/logs/soc/LEV0FILEON

The last 100 images for each VC are kept in /tmp/jim.

NOTE: If you turn lev0 on, you are going to be data sensitive and you
may see things like this, in which case you have to restart socdc:

ingest_tlm: /home/production/cvs/EGSE/src/libhmicomp.d/decompress.c:1385: decompress_undotransform: Assertion `N>=(6) && N<=(16)' failed.
kill: no process ID specified

9. The datacapture machines automatically copies DDS input data to the 
pipeline backend on /dds/socdc living on d01. This is done by the program:

>  ps -ef |grep soc_pipe_scp
388      21529 21479  0 Jun09 pts/0    00:00:13 soc_pipe_scp /dds/soc2pipe/hmi /dds/socdc/hmi d01i 30

This requires that an ssh-agent be running. If you reboot a dcs machine do:

> ssh-agent | head -2 > /tmp/ssh-agent.env
> chmod 600 /tmp/ssh-agent.env
> source /tmp/ssh-agent.env
> ssh-add
(The password is written on my whiteboard)

NOTE: cron jobs use this /tmp/ssh-agent.env file

If you want another window to use the ssh-agent that is already running do:
> source /tmp/ssh-agent.env

NOTE: on any one machine for user production there s/b just one ssh-agent

If you see that a dcs has asked for a password, the ssh-agent has failed.
You can probably find an error msg on d01 like 'invalid user production'.
You should exit the socdc. Make sure there is no soc_pipe_scp still running.
Restart the socdc.

If you find that there is a hostname for production that is not in the 
/home/production/.ssh/authorized_keys file then do this on the host that
you want to add:

Pick up the entry in /home/production/.ssh/id_rsa.pub
and put it in this file on the host that you want to have access to
(make sure that it's all one line):


NOTE: DO NOT do a ssh-keygen or you will have to update all the host's
authorized_keys with the new public key you just generated.

If not already active, then do what's shown above for the ssh-agent.

10. There should be a cron job running that will archive to the T50 tapes.
Note the names are asymmetric for dcs0 and dcs1.

30 0-23 * * * /home/production/cvs/jsoc/scripts/tapearc_do

00 0-23 * * * /home/production/cvs/jsoc/scripts/tapearc_do_dcs1

11. There should be running the t50view program to display/control the
tape operations.

> t50view -i jsocdc

The -i means interactive mode, which will allow you to change tapes.

12. Every 2 days, inspect the t50 display for the button on the top row
called 'Imp/Exp'. If it is non 0 (and yellow), then some full tapes can be
exported from the T50 and new tapes put in for further archiving.
Hit the 'Imp/Exp' button. 
Follow explicitly all the directions.
The blank L4 tapes are in the tape room in the computer room.

13. Other background info is in:


Level 0 Backend:

1. As mentioned above, the datacapture machines automatically copies DDS input 
data to the pipeline backend on /dds/socdc living on d01. 

2. The lev0 code runs as ingest_lev0 on the cluster machine cl1n001,
which has d01:/dds mounted. cl1n001 can be accessed through j1.

3. All 4 instances of ingest_lev0 for the 4 VCs are controlled by

If you want to start afresh, kill any ingest_lev0 running (will later be
automated). Then do:

> cd /home/production/cvs/JSOC/proj/lev0/apps
> start_lev0.pl

You will see 4 instances started and the log file names can be seen.
You will be advised that to cleanly stop the lev0 processing, run:

> stop_lev0.pl

It may take awhile for all the ingest_lev0 processes to get to a point
where they can stop cleanly.

For now, every hour, the ingest_lev0 processes are automatically restarted.

4. The output is for the series:


#It is all save in DRMS and  archived.
Only the tlmd is archived. (see below if you want to change the
archiving status of a dataseries)

5. If something in the backend goes down such that you can't run 
ingest_lev0, then you may want to start this cron job that will
periodically clean out the /dds/socdc dir of the files that are
coming in from the datacapture systems.

> crontab -l
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.XXXXVnxDO9 installed on Mon Jun 16 16:38:46 2008)
# (Cron version V5.0 -- $Id: whattodolev0.txt,v 1.5 2008/10/16 18:46:58 production Exp $)
#0,20,40 * * * * /home/jim/cvs/jsoc/scripts/pipefe_rm


Starting and stoping SUMS on d02:

Login as production on d02

(if sums is already running it will ask you if you want to halt it.
you normally say 'y'.)

if you just want to stop sums.


SUMS archiving:

Currently SUM is archiving continuously. The script is:


To halt it do:

touch /usr/local/logs/tapearc/TAPEARC_ABORT

Try to keep it running, as there is still much to be archived.


Change archiving status of a dataseries:

> psql -h hmidb jsoc

jsoc=> update hmi.drms_series set archive=0 where seriesname='hmi.lev0c';
jsoc=> \q

Karen Tian
Powered by
ViewCVS 0.9.4