Return to release.notes CVS log

Up to [Development] / JSOC / CM / V5.10

Various random changes to make code build on icc/gcc/ia32/x86_64

                       Release Notes JSOC V5.10         27AUG2010
                       -----------------------         ---------


A release is a set of files, each having a specific version.  And a release typcially
has a version number because over time you have newer and newer releases of the 
same product.  For example, a hypothetical 1.3 release may contain fileA#1.8, 
fileB#1.2, fileC#2.2 and a 1.4 release may contain fileA#2.5, fileB#2.1, fileC#2.9. 
JSOC releases are similarly versioned and contain a set of such files.  JSOC release
code is guaranteed to compile on cluster nodes (eg., n00, n02).  The resulting binaries
have been minimally tested.  At the time of the creation of the release, the
release versions of each file will be the most recent.  But as time passes, newer versions 
of some files will be made, and there is no guarantee that these changes will
not destabilize JSOC (ie., they may cause JSOC to no longer compile or execute
properly).  

There are several ways to use this release.  If you wish to simply use pre-built
binaries, you can simply use the production binaries, which are located at 
/home/production/cvs/JSOC.  Every time a release is created, the binaries in
this location get updated.  Only the production user can update these binaries.
So, you could run /home/production/cvs/JSOC/bin/linux_x86_64/show_keys, for example.
If instead you want to work with stable source files, then you must have a sandbox,
which is a local copy (in your home directory) of the files in the cvs depot.  
You would probably want to work with a sandbox if you plan on making eventual 
changes to the depot files.  Changes you make to your sandbox files are not visible 
to other users until you "commit" those changes back to the cvs depot.  Please see
"If You Don't Have a Sandbox" below for more information on how to create a sandbox.  
There is also a "working" release which resides in in /home/jsoc/cvs/JSOC.  New 
files may be placed here and existing files may be edited for common use before the 
next official release.  Each time a release gets created, the source and binaries of 
the working release get updated.  WARNING: the files you see here may not be stable 
since by the time you see them, another user may have edited them. Only the production 
release is guaranteed to be stable and unchanged between releases.

Obtaining the Release
---------------------
To update your working directory to this release, or to check-out this release anew, 
please visit http://jsoc.stanford.edu/jsocwiki/CvsInit. Please keep in mind that
users may have modified files since the release was created, so use of the 
scripts documented in the web page may result in a working directory whose
content is not identical to the release.  If updating, you can supply 
the flag "-R" to the jsoc_update.pl and jsoc_sync.pl scripts to download the
latest release.  This will ensure that your working directory has the exact, latest
release versions of the files (eg., jsoc_sync.csh -R). If checking-out, 
you can supply the argument "-r Ver_LATEST" to the "cvs checkout" command
to achieve the analogous result, but for a fresh checkout.  WARNING: if you use 
the "-R" or "-r" flags, please use only jsoc_update.pl or jsoc_sync.pl to update 
your sources thereafter.  Use of "-R" or "-r Ver_LATEST" will result in a cvs
"sticky flag" being set.  jsoc_update.pl and jsoc_sync.pl clear this sticky flag.

Additional Info
---------------
If you are unfamiliar with the use of cvs see the file:
JSOC/CM/working_with_sandbox.txt.

There's a linux4 cvs gui at xim:/usr/bin/lincvs
Also on our jsoc web page:

http://jsoc.stanford.edu/cvs/JSOC/

Use the Apache cvs gui to see the diffs. For example, go to
http://jsoc.stanford.edu/cvs/JSOC/base/drms/
and click on the name in the File column and then click on
"diffs to previous #" to see the diffs.

Changes since previous release (V5.9 - Mar 16, 2010)
--------------------------------------------------------


NEW FEATURES:
- modify DRMS to use SUM_infoEx() - this allows modules to make
SUNUM-batched requests of SUMS 
- SUMS now supports multi-SUNUM SUM_put() requests.
- modify DRMS to take advantage of SUM's new multi-SUNUM version of
SUM_put(). 
- Change jsoc_export_as_fits so that it reports export-payload size in
MB, not bytes 
- Add code to the series publication code to properly identify which
transaction is blocking a publication. Add some more logging to the
unpublish code.
- New slony-replication code for merging replication sets, monitoring
slon daemons, publishing series, unpublishing series, and auditing the
publication process.
- New database failover code/scripts.
- Changes to subscribe_series to, by default, not use the ssstate.txt
file (to use that, you need to use the -r flag) 
- Combine redundant variables shared by get_slony_logs.pl and
subscribe_series. Also, put these variables into a single
configuration file - repclient.template.cfg. And organize the
configuration file too.
- Modify archivelogs.pl (the script that tars-up and archives slony
logs) to use cmd-line tar, not the Archive::tar perl 
module. The latter needs to hold the entire tar contents in memory. We
have lots of data in our tar files, and we cannot hold all the data in
memory. 
- Remove much of the hard-coding from subscribe_manage and
subscribe_series
- Add the slon -x command to the slave slon (slony replication). This
causes a script to be run that says which slony logs have been
completely written and are ready for use.
- The slony log parser was replaced with a more efficient one.
- Cleaned up CVS directories for all the slony-replication code
- New flag to delete_series (-k). This effect is that delete_series
will no longer cause SUMS to delete the SUdirs.
- unpublish_series.sh will now call delete_series (to delete the
series from hmidb2) using the -k flag. We don't want to delete the SUs
since hmidb is still using them.
- document drms_run
- Make the subscribe_manage code runnable from any working directory.
- Clean up slony replication subscribe_manage code - ensure all
scripts take a config-file argument. 
- Don't ask user if they want to delete a series that they are trying
to subscribe to but in fact they already have such a series on their
site. Instead, tell them that they cant resubscribe to a series if
they already are subscribed to it, and that they should unsubscribe
first if they want to re-subscribe to it.
- In subscribe_series, add a message telling the user to not change
the subcription list file if they want to resume a previously running
subscription 
- In jsoc_export_as_is, report payload size in MB.
- In jsoc_export_make_index, fixed method=tar case to remove tar'ed
files and not make links in index.html
- Add "_IN" to end of RequestID for internal JSOC_DBHOST, i.e. hmidb
requests.
- Modify cmdparams_set so that it re-runs parse_array,
parse_numerated, parse_range_float if necessary (for example, if
cmdparams_set is being called on an ARG_INTS array).   
- Add logging of the arguments when jsoc_fetch is invoked by HTTP POST.
- Modify jsoc_info and jsoc_fetch to use SUM_infoEx() and batch
SUNUMs, instead of asking for them one at a time
- Added construct {#} to produce incrementing number for name part in
export file name template.
- Made sum_export localizable.
- In show_info, if an sunum isnt known to the local SUMS, print a
warning.
- In lookdata, added show_info option for savable page of keyword
values table.
- In the globalhs pipeline code, added a stand-alone program for
converting durations to numbers of seconds 
- In jv2helio, change keyword requirements to match current HMI
document
- In dogapfill, new shell script to submit jtsfiddle jobs to cluster
- In dogfidl, new shell script to make gap files from gapfilled
timeseries
- In doglobalhs, new shell script to run entire globalhs pipeline
- In dopow, new shell script to submit jtsfiddle jobs to cluster to
output power spectra
- In dopow3d, new shell script to submit jtsslice jobs to cluster to
output 3 day power spectra
- In doretidl, new shell script to make gap and section files from raw
timeseries
- In doretile1, new shell script to submit jretile jobs to cluster for
a single output timeseries
- In doretilen, new shell script to submit jretile jobs to cluster for
multiple output timeseries
- In dosht, new shell script for automatically submitting jv2ts jobs
to cluster
- In ingestgaps, new shell script to ingest gap files into drms/sums
- In ingestsecs, new shell script to ingest section files into drms
- In globalhs, new idl functions, procedures, scripts, and input files
needed by doretidl, dogfidl, and dopkgn
- Added show_info option for savable page of keyword values table.
- Modified lookdata.html to support forwarding of public requests to
the public database and private requests to the private database.
- Added wavelength display, change to hmi.lev0a
- Flatfield project added to CVS tree.
- Lots of changes to the code that generates level 1 images.
- Improved the timing of the MOC product file processing code so that
no unnecessary delays are introduced in the generation of level 1.
- Modified all code to use alternatives to /tmp21.
- Added all of I. Scholl's limbfit code to a CVS project directory.
- Added the 'ident', 'ambig', and 'pfss' magnetic projects to CVS.
- Added basic inversion module.
- Added MPI version of inversion.
- Made a new working directory to hold 'production' slony-replication
code and scripts. Update configuration files to point to this
production location.
- Additions to the rings project directory.


DEFECTS FIXED:
- plugged some critical leaks in sum_svc.
- memory leak in accessreplogs plugged 
- use case-insenstive comparisons of series names in
createtabstructure, fix the index creation statement, do not
automatically attempt to override the archive, tapegroup, owner, and
retention values for the series - only do so if the user provides
values for those fields 
- fix memory corruption in base_strcatalloc
- jsoc_export_as_fits was deriving the FITS keyword name and the FITS
keyword value from keyword structures from the target of linked
series. Instead, it needed to use the target keyword structures for
the FITS keyword values, but use the source keyword structures for the
FITS keyword names. 
- Modify DRMSKeyValToFITSKeyVal() and cfitsio_append_key() to pass the
keyword format string from the DRMS keyword struct to the FITS keyword
struct. Then use the keyword format field when printing FITS
floating-point keyword values into the FITS header. 
- The duration notation wasn't working for TS_SLOT keywords - problem
was a mixup of roundstep and stepsecs. 
- Changed SetKeyInternal() so that it returns an error if an attempt is
made to set a constant keyword. Changed drms_copykeys() so that it skips
target keywords that are constant. 
- Fix order of keyword, segment, and link names when modify_series is
run - adding new keywords should go at the end of keyword list 
- Needed to allocate a keymap structure before calling
drms_keymap_parsefile() 
- Fix record-set-query parser so that it properly tests for mixed
query case - at least one non-prime-key query + at least one prime-key
query 
- When parsing record-set query skip values, for slotted keywords,
must convert skip value into a whole number of slots to skip 
-  Modify drms_delete_series to take a new parameter, keepsums, that,
if set, will skip the call to SUMS that causes the SUdirs to be marked
for deletion. Many fixes for modify_series.
- Fix a crash in drms_record_directory - dont try to use record->su if
it is NULL (because the record has no SU associated with it).
- Fix export problem where not able to select a subset of segments to
export
- Fix numerous memory leaks in DRMS and a few modules
- Move the SUNUM chunking in DRMS (these chunks are passed to
SUM_infoEx) down to drms_storageunit.c. Higer-level code that used to
chunk to 512-byte chunks no longer does so. Higher-level code simply
passes all sunums to the lower-level code 
- Fix for a hang that resulted when parsing an invalid record-set
query with a backslash 
- Fix 2 segmentation faults that can occur if the caller provides an
invalid where clause in a drms_open_records() call 
- Fix crash in show_info - a deleted SU will cause rec->su to be
NULL. Code wasnt checking rec->su and was dereferencing it.
- Fixed uninitialized variables in the drms segment code.
- In jsoc_export_as_fits, call drms_stage_records() before calling the
lower-level seg-specific export function so that we don't have to call
SUM_get() on each record - just do a multi-SUNUM SUM_get() one time
for all records; drms_segment_filename no longer assumes that rec->su
exists - if it doesnt, it calls SUM_get(), and if it still doesnt, it
sets the filename to the empty string 
- Fix for jsoc_export_as_fits bailing out with an error if a segment
file is missing from an SU of a multi-segment record. 
- Fix to handle bad SUNUMs in a SUM_infoEx() call; also fix show_info
to properly do the remote sums call when the sunum isnt known at
Stanford 
- Fix bug involving the SUMS slots at the end of the file list
- get_slony_logs.pl was looking in the wrong place for the die file;
lean up downloaded tar files and unneeded log files extracted from tar
files. 
- Make subscribe_series clean up the get_slony_logs.pl die file when
it exits.
- subscription_cleanup must obtain a parser lock before proceeding;
add an arg to the config files 
- Make publish series code path-independent and dont hard-code paths
in scripts 
-  Changes for fixing race condition between
sdo_slony1_pg_dump.sh and log parser and for allowing the ssh agent
file to be in bash or csh syntax 
- make the publish_series.sh script use lower-case names for the
schema and table names. 
- subscribetemplate.sh doesnt need to source the config file since the
config file is prepended to the contents of the template. 
- slon_log_list.lst was using an incorrect hard-coded path.
- Fix the slon start/stop scripts to first synchronize their access to
the slon PID files, and second to wait until the PID file appears
(during a start) or wait until it disappears (during a shut down)
- In the slony-log archiver, handle case where there are no logs to
archive - which wont happen in practice, unless the parser is disabled
for some length of time
- Add the file locking code to parse_slon_logs.pl - it still has a
race condition with manage_logs.pl and archivelogs.pl. 
- In archivelogs.pl, check for accessreplogs return codes correctly
- make slony log parser reject insert lines that dont end in right
parenthesis semicolon; this will need to be changed again since we
need to allow the presence of newlines in embedded keyword string
values 
- Do not check for the existence of the subscribe lock-file after
releasing the lock - to do that would require having the lock as
another part of the code could have acquired it in the meantime
- Make parse_slon_logs.pl case-insensitive when doing comparisons,
make sql_gen also store the schema and table in lower-case 
- Added a concurrency mechanism for parse_slon_logs.pl,
editlstfile.pl, and various subscribe series scripts.
- unpublish_series.sh now does a case-insensitive search for schema
and table names.
- Always call sql_gen from subscription_update - the client needs the
.sql.done file and the .sql file (even if it is empty) to work; also,
change a message with the word ERROR to be a simple NOTE, since there
are cases where the branch that is writing ERROR is not an error
- in subscription_manage, force all ns and series names to be lower
case
- Fix the subscription retry code - it was using the ps -p command to
get the basename of the command that started the update process, but
this basename is truncated to 15 chars. The result was a failed
comparison, and the execution of multiple update processes for the
same subscription file, which is bad.  Also, modify subscribe_series
to disallow the runing of simultaneous instances 
- Remove not-needed run_get_slony_logs.sh, which has hard-coding in it
- In subscribe_series, fix for incomplete change of download location
of the sql tarball from the server; put the downloaded tarball, which
contains the sql created by the server, into the local working
directory
- In subscribe_series, fix an error message to reflect an accurate
number of failed attempts; add a message to help the user recover from
what is possibly too short a timeout 
- In subscribe_series, check for already existing series before
subscribing to them 
- In subscribe_series, fix the line that checks the dbase for the
existence of the series being subscribed to 
- In manage_logs.pl, the script that tars-up site-specific slony logs,
fix bug in ValidateTar() - just pass in number of files tarred, dont
try and calculate it based on the counter values of the files
contained 
- In manage_logs.pl, if no counter file is provided, then simply
delete old files - ones that have been on the server for greater than
10 days. 
- In remotesums_master.pl, use the sunum argument as the sunum list
argument for exp_su mode - there was a bug in the ds argument, which
will be fixed, but for now, use sunum, which makes more sense anyway.
- In jsoc_export_as_is, manage log files better and catch more errors
- In jsoc_export_manage, added failure checks for each command in the
drms_run script.
- Escape all % before dumping output to the debug log.
- In jsoc_info, fixed scope reporting for per-segment keywords.
- In jsoc_info, fixed op=series_struct to work with series with huge
number of same first or last records. 
- In jsoc_info, fix segfault on case of invalid links.
- Fix function signatures in hash tables
- In cfitsio.h, fix for too-short a path - this variables holds a
path, not a file basename 
- In the SUMS make Rules.mk file, restore -lssl flag - needed for MD5.
- Fixed show_info documentation.
- In show_info, corrected -l action for per-segment keywords.
- In show_info, fixed query for first and last rec, -s flag, to work
with huge number of same first or last rec.
- In show_info, fix -t option in presence of invalid linked keywords.

EXISTING DEFECTS:
- Please see http://jsoc.stanford.edu/trac/report/1 for a list and
description of most known bugs.