Release Notes JSOC V6.00 15NOV2011 ------------------------ --------- A release is a set of files, each having a specific version. And a release typcially has a version number because over time you have newer and newer releases of the same product. For example, a hypothetical 1.3 release may contain fileA#1.8, fileB#1.2, fileC#2.2 and a 1.4 release may contain fileA#2.5, fileB#2.1, fileC#2.9. JSOC releases are similarly versioned and contain a set of such files. JSOC release code is guaranteed to compile on cluster nodes (eg., n00, n02). At the time of the creation of the release, the release versions of each file will be the most recent. But as time passes, newer versions of some files will be made, and there is no guarantee that these changes will not destabilize JSOC (ie., they may cause JSOC to no longer compile or execute properly). There are several ways to use this release. If you wish to simply use pre-built binaries, you can simply use the production binaries, which are located at /home/jsoc/cvs/Development/JSOC (The release binaries are actually in /home/jsoc/cvs/JSOC, but as our production code changes emore quickly than releases are generated, we put critical fixes in the "Development" tree. To be sure you use code that has been built with these critical fixes, you'll need to use the "Development" tree. As time passes our production code will stabilize. When that happens, you should use /home/jsoc/cvs/JSOC. But for now, you should use the "Development" tree.). Every time a release is created, the binaries in this location get updated. Only the jsoc user can update these binaries. You could run /home/production/cvs/JSOC/Development/bin/linux_x86_64/show_keys, for example. If instead you want to work with stable source files, then you must have a sandbox, which is a local copy (in your home directory) of the files in the cvs depot. You would probably want to work with a sandbox if you plan on making eventual changes to the depot files. Changes you make to your sandbox files are not visible to other users until you "commit" those changes back to the cvs depot. Please see "If You Don't Have a Sandbox" below for more information on how to create a sandbox. Obtaining the Release --------------------- To update your working directory to this release, or to check-out this release anew, please visit http://jsoc.stanford.edu/jsocwiki/CvsInit. Please keep in mind that users may have modified files since the release was created, so use of the scripts documented in the web page may result in a working directory whose content is not identical to the release. Additional Info --------------- Use the Apache cvs gui to see the diffs between file revisions. For example, go to http://jsoc.stanford.edu/cvs/JSOC/base/drms/ and click on the name in the File column and then click on "diffs to previous #" to see the diffs. Changes since previous release (V5.14 - Aug 16, 2011) -------------------------------------------------------- NEW FEATURES: - The SUMS tape-system has been localized to remove some of the unnecessary code from remote sites' NetDRMSs (since remote sites will not use our very Stanford-specific tape system). - All references to the Perl SWIG interface to SUMS have been removed from the make system. - The database-user owner of the series database table (e.g., hmi.m_45s) is the only user who can reduce a dataseries' retention. Previously, any db user who could write (i.e., insert into) to the series database table was able to reduce retention. - The ability to dynamically load strings from a defs file has been removed. It made it difficult to relocate binaries, which made it difficult to update release code. - libDRMS now calls SUM_infoArray() instead of SUM_infoEx() to obtain per-SU information. This call should be several times faster than SUM_infoEx(). - We ported the bash publication script to Perl to make it easier to implement a couple of fixes/features. - We implemented some Perl subroutines, in subscribe_series.pl, to popluate the jmd queue from a just-subscribed-to series, and to install a jmd-queue-populating trigger on this just-subscribed-to series. - We added better error messages for the case where the user puts an invalid keyword name into a filename format. - We now save the hg_patch.log file in the export SU when a tar is performed. - The export server, jsoc_export_manage, now supports the sequential processing of multiple processing steps. - The export system now supports export to the mp4 movie format. - There were changes to the image- and movie-export code. - Added check for "solarmail" in notify address, if found uses @spd.aas.org for . - The Requestor field must now be provided by export-system users (using the internal export web page). - Increase the maximum number of open fits files allowed in libDRMS to 300 - to match lib cfitsio. - Bunch of changes to the specialized sum_svc processes. - We eliminated the sum_rm start from NetDRMS. - New SUMS API - SUM_infoArray(). This call should be several times faster than SUM_infoEx() because it avoids the inefficient 'key' linked-list code used to pass information between DRMS and SUMS and within SUMS. - Bunch of changes and bug fixes to the export system. - Major changes to the graphing functions of lookdata.html. - Added the ability select NOAA active regions in the hg_patch section of the export page. - build_lev1_mgr module now takes the linux user name from the environment, and not from the command-line. - Changes to the Level-0 housekeeping scripts. - We now subscribe to GONG data produced by NSO. To do that, we created several configuration files used by the subscriptio service. - Various changes and bug fixes for hg_patch, the region extraction code used by the export system. - Made some changes to the source-file checkout and make system. We now maintain a black list of files to exclude from releases. With this change, it becomes much easier to specify the complete sets of source files that comprise the full JSOC and NetDRMS releases. - New features for the ring-diagram project (proj/rings): scripts/ReadMe Added new script to listing: scripts/avgpsbycr New script to form rotational averages of power spectrs; see documentation at http://hmi.stanford.edu/teams/rings/pipe_avgpsbycr.html apps/gentargs Added option timed+ for 29-region time-distance synoptic series; added -B option for latitudes relative to disc center rather than equator (for tdsynop options only); generate fewer line of target info for rdsyn05 and rdsyn15 cases to take advantage of increased number of open files allowed. See documentation at http://hmi.stanford.edu/teams/rings/mod_gentargs.html apps/mtrack Changed test open recordset from input to first recnum rather than first record; removed -G flag, added -Z flag; bug fix in determination of segment from multiple candidates; see documentation at http://hmi.stanford.edu/teams/rings/mod_mtrack.html apps/rdfitc Changed output format (fixed D-nu estimator); included autoweeding of velocities to set quality values; fixed bug in determination of default input segment from multiple possible; added optional rectangular filtering on spectrum; see documentation at http://hmi.stanford.edu/teams/rings/mod_rdfitc.html apps/rdvinv Changes to file names and Makefile in preparation for next release; see documentation at http://hmi.stanford.edu/teams/rings/mod_rdvinv.html - The time-distance project was added to CVS (in proj/timed): ReadMe apps/Makefile apps/ReadMe apps/invert_td_hr apps/travel_times scripts/ReadMe scripts/tdsynop scripts/runtdsynop - New features for the global helioseismology project (proj/globalhs): incorporated use of the data statistics library, re-organized re-tiling from many inputs to few outputs, re-tile script now runs multiple qsub jobs, supports an arbitrary number of days for time slices, added processing template, and a few other features. DEFECTS FIXED: - There was a conflict between a make variable name and a C #define. This has been fixed. - A Fortran link flag was added to the make system. This flag causes the binaries' symbols to be loaded into dynamic symbol table. This was necessary so that the libdsds.so shared library can call functions in the binaries loading the dynamic-link library. - There was a crash that occurred when DRMS records were freed if there was an overlap in records between record-set subsets. This was fixed by implementing a refcounting strategy so that a record was not truly freed until all references to it were removed. - Socket-connect modules were not properly determining the owner of dataseries. They were using the database user string from the socket's environment, but instead they should have been using the user string from the drms_server module to which they were connected. Fixed. - drms_copykeys() was not properly "following links" when it was fetching keyword values that it intended to copy to another series. Fixed. - Modules were hanging if they passed the empty string to drms_open_records(). The record-set-query-string parser has been modified to detect the empty string and return an empty recordset and an DRMS_ERROR_INVALIDDATA error code. - In some cases, the DSDS libraries do not convert some integer FITS keyword values to floating-point values. This should never be the case. To work around this bug, libDRMS does the conversion when the error conditions exist. - We fixed a race condition during shutdown/calling SUM_close(). The 'empty' variable was not being updated appropriately. drms_server.c thought the queue was not empty as a result, when in fact it was. This was causing a hang (infinite loop). - Fix for deadlock when a module calls the exit() function, which triggers drms_free_env() to be called, which causes drms_lock_server() to be called a second time in a row. - For the -L runtime logs, set the tape group argument in the SUM_alloc() call to 1. - Don't allow non-owners of a series to set its retention to 0 during a SUM_get() call. - We updated string-to-integer conversion functions to properly interpret hexadecimal strings. - We added the -h flag to all psql commands in all replication scripts that call psql. The writers of these scripts assumed that the database host was the local machine, but this was not true for Stanford's use. - There were several issues preventing Stanford from acting as a replication subscription client. Fixed. - The data-replication subscription manager code was trying to write 'pid' and 'die' files into a directory for which it didn't have write-permission. Fixed. - We removed some superfluous logging from subscription_manager that was happening every 10 seconds. - In the export-to-fits code, we fixed the accidental demotion of unsigned long long to int. - Fixed a memory-corruption crasher in jsoc_export_manage. - Fix hmi noLimbdark max values. - Fix AIA scale and background color. - Make sure that maximum number of open FITS file descriptors is not reached in libDRMS. Also, fix the use of the status variables - one for tasrw.c errors, and one for cfitsio errors. - Bunch of changes to the specialized sum_svc processes. - We fixed an improper #idef check for the presence of multiple sums partition sets. - We fixed SUM_close() to close all sockets opened by SUM_open(). - Bug fixes for imageinterp module. - In the make system, the -L flag that points to the location of the gsl library was incorrect. Fixed. - Ring-diagram project (proj/rings) modifications: ReadMe Corrected a URL error apps/datavg Minor modifications for arguments and keywords; see documentation at http://hmi.stanford.edu/teams/rings/mod_datavg.html - The Makefile for the farside project (proj/farside) was update in CVS. - Farside project (proj/farside) additions: apps/drms_rebin Added collapse option; Fixed setting of array bzero; Added stubs for optional pre-filtering scripts/ReadMe New script listing scripts/fsbin_nrt New in CVS - Pass back to caller of cfitsio_read_keylist_and_image_info a NULL-ed out image-info struct when an error occurs. Also, print out a useful error message. - Segmentation fault in jv2helio fixed (occurred when no input records were found). - A bug in the statistics library having to do with calculating max and median was fixed. EXISTING DEFECTS: - Please see http://jsoc.stanford.edu/trac/report/1 for a list and description of most known bugs. - The DRMS code that recovers from SUMS failures when the tape system goes down is not yet working properly. SUMS needs to be modified to provide the correct error code to DRMS - until then, if the tape services crashes or is terminated, then DRMS modules can hang indefinitely.