Release Notes JSOC V6.03 16MAY2012 ------------------------ --------- A release is a set of files, each having a specific version. And a release typcially has a version number because over time you have newer and newer releases of the same product. For example, a hypothetical 1.3 release may contain fileA#1.8, fileB#1.2, fileC#2.2 and a 1.4 release may contain fileA#2.5, fileB#2.1, fileC#2.9. JSOC releases are similarly versioned and contain a set of such files. JSOC release code is guaranteed to compile on cluster nodes (eg., n00, n02). At the time of the creation of the release, the release versions of each file will be the most recent. But as time passes, newer versions of some files will be made, and there is no guarantee that these changes will not destabilize JSOC (ie., they may cause JSOC to no longer compile or execute properly). There are several ways to use this release. If you wish to simply use pre-built binaries, you can simply use the production binaries, which are located at /home/jsoc/cvs/Development/JSOC (The release binaries are actually in /home/jsoc/cvs/JSOC, but as our production code changes emore quickly than releases are generated, we put critical fixes in the "Development" tree. To be sure you use code that has been built with these critical fixes, you'll need to use the "Development" tree. As time passes our production code will stabilize. When that happens, you should use /home/jsoc/cvs/JSOC. But for now, you should use the "Development" tree.). Every time a release is created, the binaries in this location get updated. Only the jsoc user can update these binaries. You could run /home/production/cvs/JSOC/Development/bin/linux_x86_64/show_keys, for example. If instead you want to work with stable source files, then you must have a sandbox, which is a local copy (in your home directory) of the files in the cvs depot. You would probably want to work with a sandbox if you plan on making eventual changes to the depot files. Changes you make to your sandbox files are not visible to other users until you "commit" those changes back to the cvs depot. Please see "If You Don't Have a Sandbox" below for more information on how to create a sandbox. Obtaining the Release --------------------- To update your working directory to this release, or to check-out this release anew, please visit http://jsoc.stanford.edu/jsocwiki/CvsInit. Please keep in mind that users may have modified files since the release was created, so use of the scripts documented in the web page may result in a working directory whose content is not identical to the release. Additional Info --------------- Use the Apache cvs gui to see the diffs between file revisions. For example, go to http://jsoc.stanford.edu/cvs/JSOC/base/drms/ and click on the name in the File column and then click on "diffs to previous #" to see the diffs. Changes since previous release (V6.01 - December 6, 2011) ---------------------------------------------------------- NEW FEATURES: + Mike Turmon's HARP project was committed into the CVS code tree. + The configuration sysem was modified so that the it is now posssible to set defineds that are not configrable. This makes it possible to maintain a set of defines for NetDRMS sites that differs from the set defined for the full JSOC release so that the NetDRMS defines are not overridable. + Socket-connect modules can now make direct database connections. This was needed for the export system. It allows certain export modules to query the database that drms_server is not connected to. Sometimes a module will be running in the context of one database, but need to collect information from another. + The file-IO error logging of accessreplogs was improved. + Production users can now reduce the retention any series now. + A new DRMS API function was added: drms_writeslice_ext(). It takes the final size of the final dimension of the image as an argument, allowing the caller to pre-specify the final dimensions of the image. The caler can then "fill-in" the pixels by writing slices as desired. + The export code that manages processing steps is now "table-driven". This means that we can add additional processing steps by merely adding new records to a database table, instead of having to modify C code and rebuild binaries. + A new hcontainer function was added: hcon_map_ext(). It alllows the caller to call an arbitrary function on every element of the hcontainer, allowing the caller to also pass data to this function. + A new script was added that removes a replication node from the slony-replication subscription service. + A new parameter was added to the slony-log parser (parse_slon_logs.pl) to allow the caller to specify the set of replication nodes for which parsing should be performed. This feature was used to re-process logs for a single node. + The configuration-file reader used by the slony-log parser was swapped out for a version that allows nested parameters (a config-file parameter whose value contains another config-file parameter). + Some perl scripts were written to manage the list of replication nodes that are subscribed to series. The list of series to which nodes are subscribed is now maintained in a database table. The code that reads or writes this information was modified to use these new scripts/tables. The old method of using text files to store this information has not been removed yet, so updates to this release will be backward compatible. + Support for "table-driven" export processing was added. It is now possible to add new image processing steps to the export process by simply adding a record to a database table (a DRMS series). The export-management code now uses this table plus user input to derive intermediate processing-step script lines that are written to the drms-run script. + For debugging purposes, jsoc_fetch was modified to force the processing of an HTTP GET request if the caller provides a QUERY_STRING cmd-line argument. + jsoc_info was modified to add the series owner to the series struct it provides in its series_struct call. + Additional error logging was added to set_info to assist in the tracking-down of an issue. + Perl functions that were used in many modules were put into perl modules in JSOC/base/libs/perl. + A TAI offset was added for UT leap second on 2012.07.01. + SUMS was modified to start delete-pending removal at 98% percent capacity, and stop removal at 95% capacity. + When the usage of a SUMS parition that had been nearly full drops to 95% capacity, the partition gets automaticaly re-enabled. + The SUMS tape-read system time-out value was adjusted. + A SUMS API call was added: SUM_archSU(). This allows production/owner callers to modify the archive-pending disposition of SUs. The main use is for series owners to retro-actively archive storage units. The user can now create data in a non-archive mode, and when they are happy with the data, they can then tell SUMS to archive those data. + A new module, dscp, was added. It copies one DRMS data series to another. The destination series must already exist, and its definition largely controls how dscp operates. This module will duplicate all segment data (FITS files). + Support for "autobang" was added to show_info. The '-b' flag will cause the automatic addition of "[! 1=1 !]" to all SQL queries. The effect of this is to disable the prime-key logic so that queries return all obsolete DRMS records, in addition to the current versions of these records. + show_series can now show the series database owner. + Support for rebin processing was added to the export system. A user can now rebin images during export. To do this, they select the "rebin" item from the processing pull-down, and fill-in some options. + Some enhancements to the global-HS pipeline were made (e.g., retiling was optimized, multiple records can be processed at once, set_info is used in place of set_keys, etc.). + Stanford now downloads product files for APIDs 193, 213, 283, 287, 303, 307, 2002, 2003, 2004, 2005, 2008, 2024, 20248 from the MOC server. + Support for writing FITS files with headers was added to aia_lev1p5.c. + Support for accepting a 'requestid' argument was added to aia_lev1p5.c. This was needed so that this module could be part of the export system. + aia_slot now uses drms_copykeys() to set output record keyword values. + A new version of limbfit code was committed into our production code tree. + Mike Turmon's HARP code was committed into our production code tree. + The pfss_back module was added. + Support for accepting a 'requestid' argument was added to jsoc_rebin for use with the export system. Also add an option that forces the writing of full FITS headers. + A Gaussian smoothing option was added to jsoc_rebin. + dlsource.pl now supports the use of command-line file specifications for updates. A user can now provide a comma-separated list of files in the '-s' argument and dlsource.pl -o update will restrict updates to those files. + The "procser" parameter was added to exportmanage.pl. The argument identifies the export-processing-table series needed by jsoc_export_manage. DEFECTS FIXED: + Datacapture binaries were removed from the JSOC/baes/scripts links. + DSDS support is now completely disabled from NetDRMS releases. It was only partially disabled before. + The nofor-main and static-intel flags are now disabled when the user is building with gcc/f77. + The DRMS-to-DSDS interface code now looks for libdsds.so relative to the running binary. The relative location of this library was previously determined at compile time, which made it impossible to move the whole code tree to a different directory. + A bug in accessreplogs involving file copy size was fixed. + A leak in drms_keyword.c was fixed. + A leak in drms_record.c was fixed. + A leak in aia_slot was fixed. + A bug in the record-set specification parser was fixed. + The create_series jsd parser was modified so that it explicitly allows '*' and 'NA' for certain VARDIM segment fields where actual numerical values make no sense. + Crashing code in lib DRMS was fixed. It now rejects invalid record-set queries that contain a segment-list specifier, but no filter. + A problem parsing "@file" record-set specifications was fixed. There was a problem parsing the series names in @files when there was in fact no seriesname on an @file line. + The drms_create_records() code now correctly avoids creating a record's storage unit, unless there is a least one non-linked DRMS segment. Previously, the storage unit was created, even if all DRMS segments were linked segments. + show_info now properly initiates remote-sums only if the SUNUM in question is a remote SUNUM. Originally, there was no good way of determing what a remote SUNUM was, so show_info would intiate remote-sums when it encountered any kind of SUNUM that SUMS didn't recognize, including bad local SUNUMS. + The export code that extracts series names from record-set specifications was failing when the specification had no filter. This has been fixed. + A lib DRMS crash was fixed. Previously, it crashed when a record-set specification resolved into zero records. + A crash was fixed in drms_open_records() that was triggered by an empty record-set specification. + A bug in drms_record.c was fixed. The record-set-specification-parsing code had negelected to reset a variable to NULL in a loop. + A crash was fixed in jsoc_export_as_fits that occurred in the record-set-specification parsing code. + A bug in show_info was fixed. It was not correctly attempting to determine if an @file record-set query was present. + drms_server.c was modified so that it handles a NULL pointer returned by Sum_infoEx(). + More error logging was added to the script that starts the replication slon daemons - if the daemons cannot be started because the PID files already exist, then print a warning. + In the slony-replication logging module, autoflush was enabled so that tail -f works when reading from the log. + The slony-log parser was modified to error-out if it encounters file-write errors. If a problem happens when creating a log, the parser needs to be stopped and the issue needs to be resolved before continuing. + The slon-log parser was writing to a log file in the current directory. Because the current directory could be nfs-mounted, and the parser calls flock() and flock() isn't supported on NFS mounted directories, the parser was hanging. The log-file directory was moved to a local file system. + A bug in the subscription code was fixed. Previously, this code was not correctly testing for the existence of a to-be-subscribed-to series. + A bug in jsoc_export_manage.c was fixed. A container that gets re-used for each processing step was not being re-initialized between steps. + A bug in jsoc_export_manage.c was fixed. The module had been writing to the drms-run script an output series record-set specification, but it should have been writing a series name. + A bug in jsoc_export_manage.c was fixed. If an export-processing step lacks the "n=XX" value, then the record-limit should be set to zero. However, it was being set to the NULL string. + A crash in jsoc_export_manage.c was fixed. The code was trying to free a function! + A bug in jsoc_export_manage.c was fixed. This module was confusing the databases in one case. A module is connected to one database on start-up. However, jsoc_export_manage needed to communicate with a different database, but it was communicating with the start-up one. + A crash in jsoc_export_manage.c was fixed. This module was crashing when it encountered an unknown series. + A crash in jsoc_export_manage.c was fixed. There is a processing step called "Not Specified". When the export-management code was modified for performing table-driven processing, it stopped handling "Not Specified" properly. + A crash in jsoc_export_manage.c was fixed. There was an attempt to read the hg_patch.log file before it existed. + The jsoc.export Size column, which is supposed to contain the size in bytes of the export, did not have correct value. The export manager now collects the informaton from the index.json file that the export system creates. All export modules now write the size of the data they export and write that size information into index.json. + A bug in jsoc_fetch was fixed. There was a typo in a snprintf() format string. + A bug in jsoc_fetch was fixed. There was a cast of the export size value from a long long to an int. However, export sizes can be sufficiently large that an int cannot contain the value. So the cast was removed and support for long long was added. + A work-around in jsoc_info was added. There was some issue with IDL's json parser. jsoc_info's -o flag code was modified to compensate. + jsoc_export_as_images and jsoc_export_as_movie were modified so that they properly wrote the export size to the index.txt file. + A bug was fixed that existed in the script that manages the table-drive-export process table. + A bug was fixed that existed in the script that manages the table-drive-export process table. Processing-program arguments that begin with '-' (flags) were not being handled properly. + A bug in the fits-writing/reading code of DRMS was fixed. This code had been confusing the status codes for FITSIO calls with the status for non-FITSIO related calls. As a result, a non-FITSIO call error had been generating a FITSIO error message that basically meant that FITSIO had not recognize the error code, because the error code had been generated by a non-FITSIO function call. + A bug in base_strcatalloc() was fixed. + A "missing comma" bug in timeio was fixed. + SUMS was failing to close a file used for transferring SUNUMS for SUM_infoArray() handling. The fclose() was added. + A bug in SUMS was fixed. Code that updates the sum_parn_alloc table was not properly setting the partition set number. + SUMS now examines the last 600 (instead of 6000) records of delete-pending SUs to determine which SUs to delete. + A bug in dscp was fixed. It was not properly calculating a record chunk-size to use as it copies data from one series to another series. + A bug in show_coverage was fixed. + A bug in exportdata.html was fixed. The export-data website was, in rare cases, mislocating the [$] that it automatically adds when the user does not provide a filter or a record-limit to their query. + Some global-HS bugs were fixed. + The path of the JSOC tree is now passed to the interpolation code. Originally a hard-coded relative path was used, but this prevented production people from moving the disk location of the production code tree around. + Some bug fixes in build_lev1_aia.c were made. + Support for handling missing AIA thermal response was added. + A bug in build_lev1_hmi was fixed. The was image corruption due to a bad skip value in the crop table between Dec 2011 and Jan 2012. + Bugs in ingest_lev0 were fixed. + A minor bug in the lev1 production pipeline workflow was fixed. A module needed to be called with the JSOC_DBUSER argument. + The observable-processing code was updated a few times, probably to fix bugs. + The observable-processing code was modified to use a standard set of data files, instead of its own version of these files that was identical to the original set, except that comments were removed. The comment-less files were removed from the JSOC code tree. + aia_slot was added to the default make target. + Several bugs fixes to the HARP code were made. + Some bug fixes to the disambig and ambig mag code were made. + Fixed spurious make warnings. + A bug fix to the ident mag code was made. + A bug fix to the bmap mag code was made. + The call to sprint_time_ISO() was replaced with a call to drms_setkey_time() in pfss_backup. + render_image was "cleaned up". + A bug in dlsource.pl was fixed. A call to chdir() was not changing the value of $ENV{PWD}. The original chdir() doesn't do this, but by overriding the original with the Cwd module's version, chdir() does modify this environment variable correctly. + dlsource.pl now uses realpath to get the current working directory. The original use of catdir() was causing problems in one person's environment. + A bug fix in exportmanage.pl was made. A value was being used before it was checked to be defined. + exportmanage.pl sends email notifications when jsoc_export_manage crashes or otherwise has issues. It was sending email messages too frequently, so code was added to control the rate. + A change in exportmanage.pl accidentally removed the writing of output to a log file. The log-writing behavior was restored. + exportmanage.pl was exiting if jsoc_export_manage failed to run. This behavior was removed so that exportmanage.pl will try running jsoc_export_manage again if it fails previously. + A bug was fixed that had occurred when exportmanage.pl called make_jsoc.pl. + sumstapestat.pl was using the wrong where clause. It was modified to be consistent with what the sumlookgroup.pl does in its FULL mode.