Return to release.notes CVS log

Up to [Development] / JSOC / CM / V4.6

Addition of release notes for the upcoming Version 4.6 JSOC release.

                       Release Notes JSOC V4.6         02Sep2008
                       -----------------------         ---------

IMPORTANT: If you use the scriptj jsoc_update.pl to get the latest release,
please note that there was a critical bug fixed in this script.  However, 
previous versions of JSOC (before 4.6) still have this bug - most likely
you have this bug, so if you run jsoc_update.pl, your update will not
work properly.  To fix this, first cd to your JSOC tree root, then run 
"cvs update jsoc_update.pl".  Then you may run jsoc_update.pl to get the 
latest release.

A release is a set of files, each having a specific version.  And a release typcially
has a version number because over time you have newer and newer releases of the 
same product.  For example, a hypothetical 1.3 release may contain fileA#1.8, 
fileB#1.2, fileC#2.2 and a 1.4 release may contain fileA#2.5, fileB#2.1, fileC#2.9. 
JSOC releases are similarly versioned and contain a set of such files.  JSOC release
code is guaranteed to compile on cluster nodes (eg., n00, n02).  The resulting binaries
have been minimally tested.  At the time of the creation of the release, the
release versions of each file will be the most recent.  But as time passes, newer versions 
of some files will be made, and there is no guarantee that these changes will
not destabilize JSOC (ie., they may cause JSOC to no longer compile or execute
properly).  

There are several ways to use this release.  If you wish to simply use pre-built
binaries, you can simply use the production binaries, which are located at 
/home/production/cvs/JSOC.  Every time a release is created, the binaries in
this location get updated.  Only the production user can update these binaries.
So, you could run /home/production/cvs/JSOC/bin/linux_x86_64/show_keys, for example.
If instead you want to work with stable source files, then you must have a sandbox,
which is a local copy (in your home directory) of the files in the cvs depot.  
You would probably want to work with a sandbox if you plan on making eventual 
changes to the depot files.  Changes you make to your sandbox files are not visible 
to other users until you "commit" those changes back to the cvs depot.  Please see
"If You Don't Have a Sandbox" below for more information on how to create a sandbox.  
There is also a "working" release which resides in in /home/jsoc/cvs/JSOC.  New 
files may be placed here and existing files may be edited for common use before the 
next official release.  Each time a release gets created, the source and binaries of 
the working release get updated.  WARNING: the files you see here may not be stable 
since by the time you see them, another user may have edited them. Only the production 
release is guaranteed to be stable and unchanged between releases.

Obtaining the Release
---------------------
To update your working directory to this release, or to check-out this release anew, 
please visit http://jsoc.stanford.edu/jsocwiki/CvsInit. Please keep in mind that
users may have modified files since the release was created, so use of the 
scripts documented in the web page may result in a working directory whose
content is not identical to the release.  If updating, you can supply 
the flag "-R" to the jsoc_update.pl and jsoc_sync.pl scripts to download the
latest release.  This will ensure that your working directory has the exact, latest
release versions of the files (eg., jsoc_sync.csh -R). If checking-out, 
you can supply the argument "-r Ver_LATEST" to the "cvs checkout" command
to achieve the analogous result, but for a fresh checkout.  WARNING: if you use 
the "-R" or "-r" flags, please use only jsoc_update.pl or jsoc_sync.pl to update 
your sources thereafter.  Use of "-R" or "-r Ver_LATEST" will result in a cvs
"sticky flag" being set.  jsoc_update.pl and jsoc_sync.pl clear this sticky flag.

Additional Info
---------------
If you are unfamiliar with the use of cvs see the file:
JSOC/CM/working_with_sandbox.txt.

There's a linux4 cvs gui at xim:/usr/bin/lincvs
Also on our jsoc web page:

http://jsoc.stanford.edu/cvs/JSOC/

Use the Apache cvs gui to see the diffs. For example, go to
http://jsoc.stanford.edu/cvs/JSOC/base/drms/
and click on the name in the File column and then click on
"diffs to previous #" to see the diffs.

Changes since previous release (V4.5 - July 16, 2008)
-----------------------------------------------------

NEW FEATURES: 

* FITS implementation of TAS files - TAS (Tiled-Array
Storage) is a DRMS segment protocol whereby data from more than one
DRMS record are stored to the same file. In previous JSOC releases,
the file was a homegrown file format (that didn't work 100% and it was
not cross-platorm compatible). In this release, TAS uses a FITS file
format.  The TAS file has an extra dimension (one more than the number
of dimensions of the data) - each value along this dimension
corresponds to one DRMS record. The TAS file can be compressed for
non-floating point data. For floating-point data, it cannot be
compressed (compression would be lossy if this were allowed). Open TAS
files are cached so that multiple reads/writes to the same file do not
incur unneccessary file open/close overhead. In order to use this
feature, the .jsd version that creates the series that uses a TAS file
must be at least version 2.1
* Support for "dsds.XXX" and "ds_mdi.XXX" JSOC data series - These are
special data series that contain MDI data. The MDI data have been
ingested as is into these series. These JSOC series have a
generic-protocol data segment that points to a VDS
directory. drms_open_records() will now recognize these special series
and return a temporary record-set that contains DRMS records that
represent the ingested MDI data. This temporary series will contain
FITS protocol segments
* Implementation of data slicing of FITS and TAS segment protocols
* Overload the DRMS_Segment_t structure's per_segment field, and
rename to DRMS_Keyword_t::kwflags. This was needed so that we can
differentiate between keywords that existed in the original .jsd, and
ones that were implicitly created. We don't always want to show the
implicitly created keywords in a .jsd generated by the "show_info -j"
command.
* tdsignal.c - Implemented a threaded version of alarms. You can now
set an alarm, and receive it, despite being in a multi-threaded
environment. Added a sample module, proj/example/apps/threadsigs.c, to
demonstrate how this works.
* SUMS - Use the multi-threaded signal to make a new log file
* tapearcinfo - New module.
* SUM_alloc2() - This API allows you to specify a SUNUM to allocate
space for. This differs from SUM_alloc(), which picks the SUNUM for
you (the next in an increasing series).
* lookdata.html - Added default seriesname filter to exclude dsds.
* imgdecode - separate HMI/AIA crop/lookup tables into their own
subdirs; swap words before masking out top byte to find FID; read
lookup/crop table as needed even when the IMG struct has
initialized=1; Add FID (low 3 bytes only) to img struct.
* write_hk_to_drms - Changed status check from checking record pointer
to checking status after calling drms create record. If status less
than 0, print error message to log showing error code from drms. Added
check for Unknown series when writing to drms, if true, write status
code and message to log. Updated Debugf message to show write to
keyword failure when -D flag is turned on for DEBUG_WRITE_HK_TO_DRMS
in the make basic file like this:ICC_CF_ICCCOMP = -DICCCOMP
$(D_GCC_FORT) -DDEBUG_WRITE_HK_TO_DRMS
* time_lag_monitor.pl - Add command line args parsing and Help menu
* apid_list_day_file_hsb - added apids(529,536,540) to load hsb
* generated dayfiles into drms data series on production.
* getdf.pl - Updated start and end dates to avoid getting beginning of
next day's dayfile for HSB dayfiles. Added count of files loaded in
log. Added comments on limitations, description, etc.
* movedf.pl - Updated script to move files(really remove
files). Updated script to show number of hkt and xml files found and
log values in log-df-moc log file. Add code to bypass calling
ingest_dayfile.pl script if there are no files to process.

BUG FIXES: 

* jsoc_update.pl - This script was not first cd'ing to the
JSOC tree on the remote machine (so it was trying to do a make from
the directory that becomes the working directory when you ssh into the
remote machine).
* cfitsio - By default, the cfitsio library will be named
"libcfitsio.a". However, for 64-bit machines that compile with gcc,
this can be overridden in the configure script by setting the
CFITSIOFNAME_GCC_X86_64 make variable.
* DSDS support - The path to a dsds datadube dataset resides in
vds->filename, not sds->filename. Previously, libdsds.a code was using
sds->filename only, which was NULL. This caused opening of datacubes
to fail.
* show_info -j - Prevent segment-specific keywords from being expanded
into multiple keywords when creating the jsd entries. The jsd should
have the original keyword (eg., bzero), not a proliferation of
segment-specific keywords (eg., bzero_000, bzero_001, bzero_002,
etc.). The latter causes recursion (if you create a series from a .jsd
that contains bzero_000, bzero_001, bzero_002, etc., then you end up
with a series that has bzero_000_000, bzero_000_001, bzero_000_002,
etc., bzero_001_000, bzero_001_0001, etc.).
* drms_export - Fix mapping from DRMS to FITS keywords that have
missing values. For certain data types, conversion between the type
used by DRMS and the corresponding type used by FITS was done
improperly.
* drms_segment_read() - Fix for reading of binary-protocol data
segments. The faulty code assumed that the binary reader was filling
in bzero/bscale values, but our binary files have no bzero/bscale
values. So the faulty code got garbage bzero/bscale values and used
those when converting data, which led to garbage data.
* drms_stage_records() - Loop over record-set subsets (which are
specific to a series) because drms_getunits() is
series-specific. Also, if, within a subset, more than the maximum
allowed number of storage units are requested, loop.
* create_tables.sql - removed inoperable script with name conflicting
with a valid one, commented out comments and replaced suggested
actions with real actions.
* jsoc_info - Fixed fetch of log directory, now gives dir if online;
Added virtual keyword *dir_mtime* which returns the mod time of the
record directory.
* add_small_image - Modified 99% scaling to use full min-max if most values clumped to few bins.

OTHER CHANGES
* SUMS (driven_svc) - change from return(drive); to return(tapefilenum);
* SUMS (tape_svc) - change TIMEOUT to 30
* SUMS (tape_svc_proc) - add tapenxtfn
* SUMS (tapearc) - reject partial blocks. special handling of group_id for T950 initial archive
* SUMS (tapearcinfo, find_dir_sum_partn_alloc_dc, build_arc_to_dds.pl, ) - initial
* datacapture (rsync_prod.pl, sync_prod_dcs1.pl, tapearc_do_dc,
tapearc_do_dc0, tapearc_do_dc1, doingestlev0.pl) - initial
* SUMS (xsum_svc) - like changes to sum_svc.c
* SUMS (sum_rm.cfg.jsoc_sums) - change SLEEP, change MAX_FREE_0
* SUMS (SUM.h) - change max fn t0 7000, add MAX_TAPE_FN
* SUMS (sum_rpc.h) - update SUMLIB_TapeUpdate() prototype
* SUMS (SUMLIB_PavailGet.pgc) - fix printk() call for err msg
* SUMS (SUMLIB_TapeUpdate.pgc) - use tapenxtfn
* SUMS (SUMLIB_drop_series.pgc) - add SET STATUS=2
* SUMS (build_parc_file.pl) - get ssh info from /tmp/ssh-agent.env,
upgrade w/ssh stuff, use /usr/bin/perl
* SUMS (find_dir_sum_partn_alloc) - update to jsoc_sums and PGPORT
* SUMS (find_dir_sum_partn_alloc.README) - update for JSOC
* SUMS (sum_start_d02) - add /usr/local/logs/SUM/RESTART_AFTER_REBOOT
* SUMS (sum_stop_d02) - call perl in jsoc
* show_series - probably staging reenabled.
* jsoc_info - probably staging reenabled.
* dsview - show dsds name space and non-dsds namespace toggle
* Datacapture (ingest_tlm) - add some comment about build_parc_file.pl
* Datacapture (build_arc_to_dds.pl) - fix date on file name
* Datacapture (tapearc_do_dc) - *** empty log message ***
* store_dsds_migrate - change cp -rp dir to dir/*
* lev0 -  add xingest_lev0
* doingestlev0.pl - update to wait for them all to stop
* hmi_time_setting.c - elim sprint_time_ISO(); put in t_obs for darks
* ingest_lev0 - add do_quallev0(); put in time stamp when process .qac file
* time_lag_monitor.pl - Update ds_names after reading args; Use
/home/jsoc/bin/linux_x86_64/perl; Bring check_interval after command
line args parsing. Simplify alert message
* movedf.pl - Added | to script find statement to lookup dayfiles and xml files for day of year 200-299.