Release Notes JSOC V5.10 27AUG2010 ----------------------- --------- A release is a set of files, each having a specific version. And a release typcially has a version number because over time you have newer and newer releases of the same product. For example, a hypothetical 1.3 release may contain fileA#1.8, fileB#1.2, fileC#2.2 and a 1.4 release may contain fileA#2.5, fileB#2.1, fileC#2.9. JSOC releases are similarly versioned and contain a set of such files. JSOC release code is guaranteed to compile on cluster nodes (eg., n00, n02). The resulting binaries have been minimally tested. At the time of the creation of the release, the release versions of each file will be the most recent. But as time passes, newer versions of some files will be made, and there is no guarantee that these changes will not destabilize JSOC (ie., they may cause JSOC to no longer compile or execute properly). There are several ways to use this release. If you wish to simply use pre-built binaries, you can simply use the production binaries, which are located at /home/production/cvs/JSOC. Every time a release is created, the binaries in this location get updated. Only the production user can update these binaries. So, you could run /home/production/cvs/JSOC/bin/linux_x86_64/show_keys, for example. If instead you want to work with stable source files, then you must have a sandbox, which is a local copy (in your home directory) of the files in the cvs depot. You would probably want to work with a sandbox if you plan on making eventual changes to the depot files. Changes you make to your sandbox files are not visible to other users until you "commit" those changes back to the cvs depot. Please see "If You Don't Have a Sandbox" below for more information on how to create a sandbox. There is also a "working" release which resides in in /home/jsoc/cvs/JSOC. New files may be placed here and existing files may be edited for common use before the next official release. Each time a release gets created, the source and binaries of the working release get updated. WARNING: the files you see here may not be stable since by the time you see them, another user may have edited them. Only the production release is guaranteed to be stable and unchanged between releases. Obtaining the Release --------------------- To update your working directory to this release, or to check-out this release anew, please visit http://jsoc.stanford.edu/jsocwiki/CvsInit. Please keep in mind that users may have modified files since the release was created, so use of the scripts documented in the web page may result in a working directory whose content is not identical to the release. If updating, you can supply the flag "-R" to the jsoc_update.pl and jsoc_sync.pl scripts to download the latest release. This will ensure that your working directory has the exact, latest release versions of the files (eg., jsoc_sync.csh -R). If checking-out, you can supply the argument "-r Ver_LATEST" to the "cvs checkout" command to achieve the analogous result, but for a fresh checkout. WARNING: if you use the "-R" or "-r" flags, please use only jsoc_update.pl or jsoc_sync.pl to update your sources thereafter. Use of "-R" or "-r Ver_LATEST" will result in a cvs "sticky flag" being set. jsoc_update.pl and jsoc_sync.pl clear this sticky flag. Additional Info --------------- If you are unfamiliar with the use of cvs see the file: JSOC/CM/working_with_sandbox.txt. There's a linux4 cvs gui at xim:/usr/bin/lincvs Also on our jsoc web page: http://jsoc.stanford.edu/cvs/JSOC/ Use the Apache cvs gui to see the diffs. For example, go to http://jsoc.stanford.edu/cvs/JSOC/base/drms/ and click on the name in the File column and then click on "diffs to previous #" to see the diffs. Changes since previous release (V5.9 - Mar 16, 2010) -------------------------------------------------------- NEW FEATURES: - modify DRMS to use SUM_infoEx() - this allows modules to make SUNUM-batched requests of SUMS - SUMS now supports multi-SUNUM SUM_put() requests. - modify DRMS to take advantage of SUM's new multi-SUNUM version of SUM_put(). - Change jsoc_export_as_fits so that it reports export-payload size in MB, not bytes - Add code to the series publication code to properly identify which transaction is blocking a publication. Add some more logging to the unpublish code. - New slony-replication code for merging replication sets, monitoring slon daemons, publishing series, unpublishing series, and auditing the publication process. - New database failover code/scripts. - Changes to subscribe_series to, by default, not use the ssstate.txt file (to use that, you need to use the -r flag) - Combine redundant variables shared by get_slony_logs.pl and subscribe_series. Also, put these variables into a single configuration file - repclient.template.cfg. And organize the configuration file too. - Modify archivelogs.pl (the script that tars-up and archives slony logs) to use cmd-line tar, not the Archive::tar perl module. The latter needs to hold the entire tar contents in memory. We have lots of data in our tar files, and we cannot hold all the data in memory. - Remove much of the hard-coding from subscribe_manage and subscribe_series - Add the slon -x command to the slave slon (slony replication). This causes a script to be run that says which slony logs have been completely written and are ready for use. - The slony log parser was replaced with a more efficient one. - Cleaned up CVS directories for all the slony-replication code - New flag to delete_series (-k). This effect is that delete_series will no longer cause SUMS to delete the SUdirs. - unpublish_series.sh will now call delete_series (to delete the series from hmidb2) using the -k flag. We don't want to delete the SUs since hmidb is still using them. - document drms_run - Make the subscribe_manage code runnable from any working directory. - Clean up slony replication subscribe_manage code - ensure all scripts take a config-file argument. - Don't ask user if they want to delete a series that they are trying to subscribe to but in fact they already have such a series on their site. Instead, tell them that they cant resubscribe to a series if they already are subscribed to it, and that they should unsubscribe first if they want to re-subscribe to it. - In subscribe_series, add a message telling the user to not change the subcription list file if they want to resume a previously running subscription - In jsoc_export_as_is, report payload size in MB. - In jsoc_export_make_index, fixed method=tar case to remove tar'ed files and not make links in index.html - Add "_IN" to end of RequestID for internal JSOC_DBHOST, i.e. hmidb requests. - Modify cmdparams_set so that it re-runs parse_array, parse_numerated, parse_range_float if necessary (for example, if cmdparams_set is being called on an ARG_INTS array). - Add logging of the arguments when jsoc_fetch is invoked by HTTP POST. - Modify jsoc_info and jsoc_fetch to use SUM_infoEx() and batch SUNUMs, instead of asking for them one at a time - Added construct {#} to produce incrementing number for name part in export file name template. - Made sum_export localizable. - In show_info, if an sunum isnt known to the local SUMS, print a warning. - In lookdata, added show_info option for savable page of keyword values table. - In the globalhs pipeline code, added a stand-alone program for converting durations to numbers of seconds - In jv2helio, change keyword requirements to match current HMI document - In dogapfill, new shell script to submit jtsfiddle jobs to cluster - In dogfidl, new shell script to make gap files from gapfilled timeseries - In doglobalhs, new shell script to run entire globalhs pipeline - In dopow, new shell script to submit jtsfiddle jobs to cluster to output power spectra - In dopow3d, new shell script to submit jtsslice jobs to cluster to output 3 day power spectra - In doretidl, new shell script to make gap and section files from raw timeseries - In doretile1, new shell script to submit jretile jobs to cluster for a single output timeseries - In doretilen, new shell script to submit jretile jobs to cluster for multiple output timeseries - In dosht, new shell script for automatically submitting jv2ts jobs to cluster - In ingestgaps, new shell script to ingest gap files into drms/sums - In ingestsecs, new shell script to ingest section files into drms - In globalhs, new idl functions, procedures, scripts, and input files needed by doretidl, dogfidl, and dopkgn - Added show_info option for savable page of keyword values table. - Modified lookdata.html to support forwarding of public requests to the public database and private requests to the private database. - Added wavelength display, change to hmi.lev0a - Flatfield project added to CVS tree. - Lots of changes to the code that generates level 1 images. - Improved the timing of the MOC product file processing code so that no unnecessary delays are introduced in the generation of level 1. - Modified all code to use alternatives to /tmp21. - Added all of I. Scholl's limbfit code to a CVS project directory. - Added the 'ident', 'ambig', and 'pfss' magnetic projects to CVS. - Added basic inversion module. - Added MPI version of inversion. - Made a new working directory to hold 'production' slony-replication code and scripts. Update configuration files to point to this production location. - Additions to the rings project directory. DEFECTS FIXED: - plugged some critical leaks in sum_svc. - memory leak in accessreplogs plugged - use case-insenstive comparisons of series names in createtabstructure, fix the index creation statement, do not automatically attempt to override the archive, tapegroup, owner, and retention values for the series - only do so if the user provides values for those fields - fix memory corruption in base_strcatalloc - jsoc_export_as_fits was deriving the FITS keyword name and the FITS keyword value from keyword structures from the target of linked series. Instead, it needed to use the target keyword structures for the FITS keyword values, but use the source keyword structures for the FITS keyword names. - Modify DRMSKeyValToFITSKeyVal() and cfitsio_append_key() to pass the keyword format string from the DRMS keyword struct to the FITS keyword struct. Then use the keyword format field when printing FITS floating-point keyword values into the FITS header. - The duration notation wasn't working for TS_SLOT keywords - problem was a mixup of roundstep and stepsecs. - Changed SetKeyInternal() so that it returns an error if an attempt is made to set a constant keyword. Changed drms_copykeys() so that it skips target keywords that are constant. - Fix order of keyword, segment, and link names when modify_series is run - adding new keywords should go at the end of keyword list - Needed to allocate a keymap structure before calling drms_keymap_parsefile() - Fix record-set-query parser so that it properly tests for mixed query case - at least one non-prime-key query + at least one prime-key query - When parsing record-set query skip values, for slotted keywords, must convert skip value into a whole number of slots to skip - Modify drms_delete_series to take a new parameter, keepsums, that, if set, will skip the call to SUMS that causes the SUdirs to be marked for deletion. Many fixes for modify_series. - Fix a crash in drms_record_directory - dont try to use record->su if it is NULL (because the record has no SU associated with it). - Fix export problem where not able to select a subset of segments to export - Fix numerous memory leaks in DRMS and a few modules - Move the SUNUM chunking in DRMS (these chunks are passed to SUM_infoEx) down to drms_storageunit.c. Higer-level code that used to chunk to 512-byte chunks no longer does so. Higher-level code simply passes all sunums to the lower-level code - Fix for a hang that resulted when parsing an invalid record-set query with a backslash - Fix 2 segmentation faults that can occur if the caller provides an invalid where clause in a drms_open_records() call - Fix crash in show_info - a deleted SU will cause rec->su to be NULL. Code wasnt checking rec->su and was dereferencing it. - Fixed uninitialized variables in the drms segment code. - In jsoc_export_as_fits, call drms_stage_records() before calling the lower-level seg-specific export function so that we don't have to call SUM_get() on each record - just do a multi-SUNUM SUM_get() one time for all records; drms_segment_filename no longer assumes that rec->su exists - if it doesnt, it calls SUM_get(), and if it still doesnt, it sets the filename to the empty string - Fix for jsoc_export_as_fits bailing out with an error if a segment file is missing from an SU of a multi-segment record. - Fix to handle bad SUNUMs in a SUM_infoEx() call; also fix show_info to properly do the remote sums call when the sunum isnt known at Stanford - Fix bug involving the SUMS slots at the end of the file list - get_slony_logs.pl was looking in the wrong place for the die file; lean up downloaded tar files and unneeded log files extracted from tar files. - Make subscribe_series clean up the get_slony_logs.pl die file when it exits. - subscription_cleanup must obtain a parser lock before proceeding; add an arg to the config files - Make publish series code path-independent and dont hard-code paths in scripts - Changes for fixing race condition between sdo_slony1_pg_dump.sh and log parser and for allowing the ssh agent file to be in bash or csh syntax - make the publish_series.sh script use lower-case names for the schema and table names. - subscribetemplate.sh doesnt need to source the config file since the config file is prepended to the contents of the template. - slon_log_list.lst was using an incorrect hard-coded path. - Fix the slon start/stop scripts to first synchronize their access to the slon PID files, and second to wait until the PID file appears (during a start) or wait until it disappears (during a shut down) - In the slony-log archiver, handle case where there are no logs to archive - which wont happen in practice, unless the parser is disabled for some length of time - Add the file locking code to parse_slon_logs.pl - it still has a race condition with manage_logs.pl and archivelogs.pl. - In archivelogs.pl, check for accessreplogs return codes correctly - make slony log parser reject insert lines that dont end in right parenthesis semicolon; this will need to be changed again since we need to allow the presence of newlines in embedded keyword string values - Do not check for the existence of the subscribe lock-file after releasing the lock - to do that would require having the lock as another part of the code could have acquired it in the meantime - Make parse_slon_logs.pl case-insensitive when doing comparisons, make sql_gen also store the schema and table in lower-case - Added a concurrency mechanism for parse_slon_logs.pl, editlstfile.pl, and various subscribe series scripts. - unpublish_series.sh now does a case-insensitive search for schema and table names. - Always call sql_gen from subscription_update - the client needs the .sql.done file and the .sql file (even if it is empty) to work; also, change a message with the word ERROR to be a simple NOTE, since there are cases where the branch that is writing ERROR is not an error - in subscription_manage, force all ns and series names to be lower case - Fix the subscription retry code - it was using the ps -p command to get the basename of the command that started the update process, but this basename is truncated to 15 chars. The result was a failed comparison, and the execution of multiple update processes for the same subscription file, which is bad. Also, modify subscribe_series to disallow the runing of simultaneous instances - Remove not-needed run_get_slony_logs.sh, which has hard-coding in it - In subscribe_series, fix for incomplete change of download location of the sql tarball from the server; put the downloaded tarball, which contains the sql created by the server, into the local working directory - In subscribe_series, fix an error message to reflect an accurate number of failed attempts; add a message to help the user recover from what is possibly too short a timeout - In subscribe_series, check for already existing series before subscribing to them - In subscribe_series, fix the line that checks the dbase for the existence of the series being subscribed to - In manage_logs.pl, the script that tars-up site-specific slony logs, fix bug in ValidateTar() - just pass in number of files tarred, dont try and calculate it based on the counter values of the files contained - In manage_logs.pl, if no counter file is provided, then simply delete old files - ones that have been on the server for greater than 10 days. - In remotesums_master.pl, use the sunum argument as the sunum list argument for exp_su mode - there was a bug in the ds argument, which will be fixed, but for now, use sunum, which makes more sense anyway. - In jsoc_export_as_is, manage log files better and catch more errors - In jsoc_export_manage, added failure checks for each command in the drms_run script. - Escape all % before dumping output to the debug log. - In jsoc_info, fixed scope reporting for per-segment keywords. - In jsoc_info, fixed op=series_struct to work with series with huge number of same first or last records. - In jsoc_info, fix segfault on case of invalid links. - Fix function signatures in hash tables - In cfitsio.h, fix for too-short a path - this variables holds a path, not a file basename - In the SUMS make Rules.mk file, restore -lssl flag - needed for MD5. - Fixed show_info documentation. - In show_info, corrected -l action for per-segment keywords. - In show_info, fixed query for first and last rec, -s flag, to work with huge number of same first or last rec. - In show_info, fix -t option in presence of invalid linked keywords. EXISTING DEFECTS: - Please see http://jsoc.stanford.edu/trac/report/1 for a list and description of most known bugs.