Release Notes JSOC V5.5 9OCT2009 ----------------------- --------- A release is a set of files, each having a specific version. And a release typcially has a version number because over time you have newer and newer releases of the same product. For example, a hypothetical 1.3 release may contain fileA#1.8, fileB#1.2, fileC#2.2 and a 1.4 release may contain fileA#2.5, fileB#2.1, fileC#2.9. JSOC releases are similarly versioned and contain a set of such files. JSOC release code is guaranteed to compile on cluster nodes (eg., n00, n02). The resulting binaries have been minimally tested. At the time of the creation of the release, the release versions of each file will be the most recent. But as time passes, newer versions of some files will be made, and there is no guarantee that these changes will not destabilize JSOC (ie., they may cause JSOC to no longer compile or execute properly). There are several ways to use this release. If you wish to simply use pre-built binaries, you can simply use the production binaries, which are located at /home/production/cvs/JSOC. Every time a release is created, the binaries in this location get updated. Only the production user can update these binaries. So, you could run /home/production/cvs/JSOC/bin/linux_x86_64/show_keys, for example. If instead you want to work with stable source files, then you must have a sandbox, which is a local copy (in your home directory) of the files in the cvs depot. You would probably want to work with a sandbox if you plan on making eventual changes to the depot files. Changes you make to your sandbox files are not visible to other users until you "commit" those changes back to the cvs depot. Please see "If You Don't Have a Sandbox" below for more information on how to create a sandbox. There is also a "working" release which resides in in /home/jsoc/cvs/JSOC. New files may be placed here and existing files may be edited for common use before the next official release. Each time a release gets created, the source and binaries of the working release get updated. WARNING: the files you see here may not be stable since by the time you see them, another user may have edited them. Only the production release is guaranteed to be stable and unchanged between releases. Obtaining the Release --------------------- To update your working directory to this release, or to check-out this release anew, please visit http://jsoc.stanford.edu/jsocwiki/CvsInit. Please keep in mind that users may have modified files since the release was created, so use of the scripts documented in the web page may result in a working directory whose content is not identical to the release. If updating, you can supply the flag "-R" to the jsoc_update.pl and jsoc_sync.pl scripts to download the latest release. This will ensure that your working directory has the exact, latest release versions of the files (eg., jsoc_sync.csh -R). If checking-out, you can supply the argument "-r Ver_LATEST" to the "cvs checkout" command to achieve the analogous result, but for a fresh checkout. WARNING: if you use the "-R" or "-r" flags, please use only jsoc_update.pl or jsoc_sync.pl to update your sources thereafter. Use of "-R" or "-r Ver_LATEST" will result in a cvs "sticky flag" being set. jsoc_update.pl and jsoc_sync.pl clear this sticky flag. Additional Info --------------- If you are unfamiliar with the use of cvs see the file: JSOC/CM/working_with_sandbox.txt. There's a linux4 cvs gui at xim:/usr/bin/lincvs Also on our jsoc web page: http://jsoc.stanford.edu/cvs/JSOC/ Use the Apache cvs gui to see the diffs. For example, go to http://jsoc.stanford.edu/cvs/JSOC/base/drms/ and click on the name in the File column and then click on "diffs to previous #" to see the diffs. Changes since previous release (V5.3 - Aug 5, 2009) -------------------------------------------------------- NEW FEATURES: + Cleaned up make system and configuration script. lib_third_party removed. Instead, config.local contains paths to the third-part libraries, which are placed into make variables that are used during compilation and linking. Added support for machine-specific output directories (so you can generate binaries in directories other than linux_x86_64 and linux_ia32). + Reworked config.local/gen_init.csh/customizedefs.pl to provide additional parameters that can be customized, including fields that allow the user to customize the cmd-line used to copy remote storage units locally (the remoteSUMS process). + icc compilation/linking displays default warnings, remarks, and error messages (no -W flags supplied to icc cmd-line). + Addition of support for JSOC_WARNICC environment variable. The string assigned to this variable will be passed onto the icc compile cmd-line. + Move remotesums_master.pl and remotesums_ingest from Stanford-specific locations in the CVS tree (part of the full JSOC release), to the NetDRMS part of the CVS tree. + Modify remotesums export so that sites can override the sums_url field in jsoc.drms_sites to specify their own copy program cmd-line. + Added SUMS_MANAGER_UID, the SUMS manager's UID, to config.local (actually, there is code that runs that will generate this from the SUMS_MANAGER field, which contains the linux-account name of the SUMS manager). + Change from caching ALL series in the series_cache during module startup to lazy loading of the cache (on-demand). Stanford has over 43K series and caching all of them, despite most of them never being used, was inefficient. + Certain keywords can be marked 'reserved' so that they don't get exported or imported. + There are two new functions in the API - base_fitskeycheck and base_drmskeycheck - that check for invalid fits and drms keywords, and check to see if keywords are reserved. + Implementation of drms_segment_fopen() and drms_segment_fclose() - two new API functions to open and close a FILE * to the segment's file, so that, especially for the generic segment protocol, you can write to a FILE * directly (and not have to COPY a generic file into a storage unit). + When writing data slices to compressed data files belonging to vardims segments use the dimensions of the array to be written as the compression-tile size. + SUMS now uses a sum_chmown script to change ownership of files in SUMS to production. Previously a sudo was used, which, under certain circumstances, could require user interaction. This would cause sum_svc to stop responding. sum_chmown has setuid set to run as root. + New utility show_coverage added. For certain types of series, displays how complete the data coverage is (if data are expected to exist at a certain cadence, shows how many expected items are missing). + jsoc_fetch now returns more information, as soon as it has it, when exporting storage units. The information is provided per storage-unit, as a list. + jsoc_export_SU_as_is now will not fail if an SUNUM is unknown. Instead, it will simply set an appropriate status string in the status field of the information returned to the user. + ingest_from_fits added, a simple program to ingest FITS files that contain single image extensions. + Add a new flag, p, to time_convert that allows the user to specify the precisions of seconds. BUG FIXES: + Remove the compilation of the DRMS fortran interface from the NetDRMS build if no acceptable fortran compiler is found. + Remove the ricecomp library - it was unused. + Code that writes images with all blank values - shorts, ints, and long longs - was not working properly. + Add a workaround to overcome a fitsio bug where it was not properly updating the TFORM1 keyword when writing compressed images. The workaround is to manually write the TFORM1 keyword. + Modify drms_copy_keys to not fail if one or more of the target keywords cannot be set because they are not present in the set of source keywords. + timeio library fixes: 1. Fix parse_zone so it handle timezone strings that have trailing non-timezone chars, 2. Another fix to parse_zone to handle +- time offsets, 3. Fix for time parser looking for JD anywhere in timestring when checking for a Julian Day time string (should only have MJD or JD at the beginning of the time string), 4. In DRMS, when parsing a time range, if a time string is of the form 2009.02.05_12:00:00-2009.02.05_15:00:00, associate -2009 with the second time string, not the time zone of the first, 5. Handle invalid time strings properly + drms_ismissing_time wasn't properly comparing against the missing time (JD_0) + cmdparams library fixes: 1. cmdparams had the wrong value for JD_0, 2. memory corruption fixed (appeared with sufficiently long argument string lengths), 3. buffer overrun fixed. + Don't copy the keyword description from a slotted key to the associated index key. + Modify the default file extension for generic-protocol-segment files - it used to be .generic, but now it is the empty string. There is no such thing as a .generic file type. + When converting integer keyword values that are represented as ascii strings to integers, properly supports hexidecimal strings (0x08af35de is properly recognized now). + Fix to script that ingests slony logs into slony log-shipping receiver. + In database connection code, initialize variable "on" to 1 in db_tcp_listen(). NOTE: + If using icc/ifort to compile and link, you must use icc version 11, otherwise use gcc/gfortran. icc compilation and linking flags were modified to support version 11 only.