NetDRMS Release Notes

Version 2.0

New Features since previous release (V 1.1):

• RemoteSUMS implemented. If an attempt it made to access a storage unit not known to the local SUMS, but owned by a remote SUMS, a request is sent to the remote SUMS to transfer the storage unit to the local SUMS. When Stanford is the source of the storage unit, the transfer uses scp (or high-speed scp) via Stanford's j0.stanford.edu scp server. This requires the use of ssh-agent which entails some initialization at the receiving site (details here). The scp of files can happen either synchronously or asynchronously, depending on the provider's configuration (if the payload is above a certain size, the copy happens asynchronously).
• Implementation of SUM_delete_series(). This provides support for removal of DRMS records when a series is deleted with delete_series. To enable this feature, the .jsd must specify an archive value of -1.
• New localization support for NetDRMS sites. Users must now provide a simple text site localization file, config.local, for which a template file (config.local.template) is provided. This file specifies various settings appropriate to the site and should obviate the need for any manual modification of code or scripts for supported architectures. The configure script will apply those settings to the DRMS build environment. N.B. The required contents of the config.local file have changed substantially since the α releases. Saved versions should be compared with and modifieed as necessary to conform with the structure of the distributed template file.
• Implemented [!...!] query construct. ... represents a PSQL where clause. If any part of a record-set query contains such a construct, then the "prime-key" logic is disabled. In other words, if two PSQL rows contain the exact same values for the prime key, then two DRMS records are created, not one as the prime-key log would dictate.
• It is now possible to specify an archive value (via the jsd, the cmd-line, or the environment) of -1. Any storage unit thus labeled will cause all records whose data are contained in that storage unit to be deleted when the storage unit is deleted.
• It is no longer possible to change series data retention via the environment. It was decidedly too dangerous to allow – a user could unknowingly run in a shell where the retention was set to 0. Also, don't allow users who don't own a series to reduce the retention time (they can increase it however). Retention time is specified in the jsd, and overridden with the DRMS_RETENTION=XXX cmd-line argument. Also, change the archive flag to an archive cmd-line argument (DRMS_ARCHIVE=XXX).
• New show_info functionality: can provide an SUNUM and show_info will provide information about that storage unit, and the records whose data it contains. Also, new flags were added (eg, -z) that will provide storage unit information.
• Added SUM_info() - this provides information about a storage unit.
• Added module-wide flags (--version, --ver, --vers, --vn, --about) that when used will cause the module to print out general module information, like version number. After printing this information, the module will terminate.
• Added support for allowing a socket-module connection to a designated PSQL port.
• Added drms_open_nrecords(), an API function that efficiently limits the number of records requested from the PSQL database.
• Added new cmdparams API function that allows jsoc_main to reserve a set of cmd-line arguments. This ensure that module writers don't accidentally try to use an argument that jsoc_main reserves for use (like DRMS_ARCHIVE).
• Added the ability to specify integer cmd-line argument ranges in the module's module_args structure
• Cleaned up make system and configuration script. lib_third_party removed. Instead, config.local contains paths to the third-party libraries, which are placed into make variables that are used during compilation and linking. Added support for machine-specific output directories (so you can generate binaries in directories other than linux_x86_64 and linux_ia32). Reworked config.local, gen_init.csh, and customizedefs.pl to provide additional parameters that can be customized, including fields that allow the user to customize the command line used to copy remote storage units locally (the remoteSUMS process). (previously this was limited to floats and doubles).
• Cleaned up make system and configuration script. lib_third_party removed. Instead, config.local contains paths to the third-party libraries, which are placed into make variables that are used during compilation and linking. Added support for machine-specific output directories (so you can generate binaries in directories other than linux_x86_64 and linux_ia32). Reworked config.local, gen_init.csh, and customizedefs.pl to provide additional parameters that can be customized, including fields that allow the user to customize the command line used to copy remote storage units locally (the remoteSUMS process).
• Added scripts remotesums_master.pl and remotesums_ingest
• Added two new functions in the API: base_fitskeycheck() and base_drmskeycheck(), that check for invalid fits and drms keywords, and check to see if keywords are reserved.
• Implementation of drms_segment_fopen() and drms_segment_fclose(), two new API functions to open and close a FILE* to the segment's file, so that, especially for the generic segment protocol, you can write to a FILE* directly (and not have to copy a generic file into a storage unit).
• When writing data slices to compressed data files belonging to vardims segments, use the dimensions of the array to be written as the compression-tile size.
• SUMS now uses a sum_chmown script to change ownership of files in SUMS to the SUMS administrator account. Previously a sudo was used, which, under certain circumstances, could require user interaction. This would cause sum_svc to stop responding. sum_chmown must be setuid to run as root.
• New utility show_coverage added. For certain types of series, displays how complete the data coverage is (if data are expected to exist at a certain cadence, shows how many expected items are missing).
• jsoc_fetch now returns more information, as soon as it has it, when exporting storage units. The information is provided per storage unit, as a list.
• ingest_from_fits added, a simple program to ingest FITS files that contain single image extensions.

Bug Fixes:

• Fixes that allow DRMS to build under gcc compilation with few or no warnings.
• When creating an "empty" TAS file, instead of attempting to store the entire desired array of missing values in memory so that the entire array is written, store a smaller block in memory, and then write that block several times.
• When printing a jsd with show_info -j (or describe_series -j), if the series for which the jsd is being printed has a dbindex that contains an index keyword, then print the corresponding slotted keyword name, not the index keyword name.
• show_info -j fixes: make cparms and bzero/bscale keywords implicit, so they aren't printed in the jsd; the output jsd is always a current-version jsd that has no version field (jsd's that have no version field refer to current-version series - a version field exists only if the series to be created is to be compatible with old code).
• It is no longer required for there to be a space in otherwise empty strings in jsds.
• Removed restriction on the number of characters in a record-set query, and on the number of lines in an @file record-set query. Fixed problem recognizing of not finding the terminating ']' of a record-set query under atypical situations.
• Tests of time values equivalent to JD_0 were failing - they were using comparisons of doubles. Changed to do a range check (+- epsilon).
• Remove the compilation of the DRMS fortran interface from the NetDRMS build if no acceptable fortran compiler is found.
• Modify drms_copy_keys to not fail if one or more of the target keywords cannot be set because they are not present in the set of source keywords.
• timeio library fixes: 1. Fix parse_zone so it handles timezone strings that have trailing non-timezone chars, 2. Another fix to parse_zone to handle +- time offsets, 3. Fix for time parser looking for JD anywhere in timestring when checking for a Julian Day time string (should only have MJD or JD at the beginning of the time string), 4. In DRMS, when parsing a time range, if a time string is of the form 2009.02.05_12:00:00-2009.02.05_15:00:00, associate -2009 with the second time string, not the time zone of the first, 5. Handle invalid time strings properly. 6. drms_ismissing_time wasn't properly comparing against the missing time (JD_0)
• cmdparams library fixes: 1. cmdparams had the wrong value for JD_0, 2. memory corruption fixed (appeared with sufficiently long argument string lengths), 3. buffer overrun fixed.
• Modify the default file extension for generic-protocol-segment files - it used to be .generic, but now it is the empty string. There is no such thing as a .generic file type.
• When converting integer keyword values that are represented as ascii strings to integers, properly supports hexidecimal strings (0x08af35de is properly recognized now).

NetDRMS 2.0 is a subset of JSOC Version 5.6, q.v. for Full release notes.

12 Mar 2010, 15:25-0800