Release Notes JSOC V9.1 31JUL2017 ------------------------ --------- A release is a set of binary and text files, each having a specific version. The release itself is also versioned to facilitate the identification of the release. For example, release 1.3 may contain fileA#1.8, fileB#1.2, and fileC#2.2 and release 1.4 may contain fileA#2.5, fileB#2.1, and fileC#2.9. JSOC releases are similarly versioned and contain a set of such files. JSOC release code is guaranteed to compile on cluster nodes (e.g., n04 and solar3). At the time of the creation of the release, the versions of each file will be very recent - most will be the "head" versions. The set of files is stable. They have been tested and verified to work properly on the supported systems. As time passes, newer versions of files will be committed to the code repository and there is no guarantee that these changes will not destabilize JSOC (i.e., they may cause JSOC to no longer compile or execute properly). There are several ways to use this release. If you wish to simply use pre-built binaries, you can use the production binaries, which are located at /home/jsoc/cvs/Development/JSOC (The release binaries are actually in /home/jsoc/cvs/JSOC, but as our production code changes more quickly than releases are generated, we put critical fixes in the "Development" tree. To be sure you use code that has been built with these critical fixes, you'll need to use the "Development" tree. As time passes our production code will stabilize. When that happens, you should use /home/jsoc/cvs/JSOC. But for now, you should use the "Development" tree.). Every time a release is created, the binaries in this location get updated by the jsoc user. To use these binaries, you can run them directly, "/home/jsoc/cvs/JSOC/Development/bin/linux_x86_64/show_info -j hmi.M_45s", for example. Or you can put /home/jsoc/cvs/JSOC/Development/bin/linux_x86_64 in your path environment variable. If instead you want to work with stable source files, then you must have a local copy (e.g., in your home directory) of the source code in the CVS code repository. You must "build" or "make" the code this local code before it can be used. You will want to work with a local copy if you plan on making changes to the repository files (i.e., you plan on doing development). Changes you make to your local copy are not visible to other users until you commit those changes back to the CVS repository. Obtaining the Release --------------------- To update your working directory to this release, or to check-out this release anew, please visit http://jsoc.stanford.edu/jsocwiki/CvsInit (please see the section entitled Update Your Existing Working Directory ("Sandbox")). Please keep in mind that users may have modified files since the release was created, so use of the scripts documented in the web page may result in a working directory whose content is not identical to the release. Additional Info --------------- Use the Apache CVS gui to see diffs between file revisions. For example, go to http://jsoc2.stanford.edu/cvs/JSOC/base/drms/ and click on the name in the File column and then click on "diffs to previous #" to see the diffs. Changes since previous release (V9.0 - February 28, 2017) ---------------------------------------------------------- LOCALIZATION: + added two config.local parameters: DRMS_LOG_DIR (a path where DRMS can write logs - new remote SUMS code uses this, but the location of previously existing logs was not changed), and DRMS_LOCK_DIR (a place for lock files). NEW FEATURES: + added a DRMS-wide flag, DRMS_DBUTF8CLIENTENCODING, that forces the libpq client to use UTF8 encoding; there was an issue with a Latin-1 char being interpreted as UTF8 character, which caused an encoding exception in a Python script; this script operates at many sites that do not use Latin-1 - UTF8 is a better choice that works universally. + added a function to create the JSD versions of template keywords (so, for per-segment keywords, this function produces a single, non-expanded keyword, not the one-per-segment keywords). + added a method by which remote sites can automatically schedule SUs for ("mirror") download; by enabling this feature, sites can commence with automatically pre-fetching SUs as soon as a subscription successfully completes; SUs of series subscribed to will also automatically be downloaded as soon as the DRMS DB info has been ingested. + created an InitializationException subclass to facilitate initialization handling in the remote sums pre-fetch glue code. + added a parameter, --dbuser, to deldbrecs.py, that allows the user to override the DB account to use (the default is the account of the user whose name matches the linux account name). + in the remote-SUMS daemon, changed all exception-handling code to use class-based exceptions. + reorganized logging in the thread-halting code so that every scp worker thread displays a message when it terminates. + created a program that maps an export-system file-name-format string to a full file path; currently, this code will bring offline SUs online, but a future iteration will skip this (in exchange for outputting a base file name only). + implemented a plan to save CGI interface usage. + added a flag to jsoc_info to print FITS keyword names, not DRMS keyword names, in jsoc_info output. + wrote a script to test a large part of the jsoc_info argument-space (by using random argument values). + upgraded some internal CGI scripts to use Python 3, not Python 2. + checked-in more scattered-light work and clean-up. + added the aia limb-fit project. + added the set_gaps_missing DRMS module. DEFECTS FIXED: + forced use of shared crypto library for DRMS build (instead of static library); there was a conflict at the JSOC when running production code on users' machines otherwise, and the original fix for that conflicted with the flags needed for remote sites. + changed the case-sensitive comparisons, to case-insensitive ones, of DRMS DB table namespace / table in drms_query_tabexists(). + modified drms_record_getinfo() to no longer assume that the input record-set has at least one SU associated with it. + fixed a memory leak in drms_destroy_jsdtemplate_record(). + fixed, in drms_retrieve_records_internal(), omission of limit not being set when qoverride provided. + modified drms_addkeys_toseries(): conditionally run code thats add columns to the series table; there was an attempt to add columns to the series table, even if there were no non-constant keywords in the specification. + removed unused and osolete defines in drms_storage_unit.c. + added a marker to the end of the site-specific Slony logs, so the client does not attempt to download partially written files; removed the use of slon_counter.txt in the client - use the log number from db instead. + fixed a problem in the subscription manager in which a status was being changed before the code looking for the status could be read. + modified the subscription manager to check for client errors periodically during dump-file creation (otherwise, it would be possible for a very long dump would continue, despite an early client error). + modified the subscription client-side capturesunum database procedure to remove duplicate SUNUMs. + fixed goofed-up syntax of arguments being passed to the getProcessing() funcction. + in the subscription server code, made all client- and series-name comparisons case-insensitive. + fixed a typo in URL-error exception handler in subscription client (subscribe.py). + fixed a potential infinite loop in the remote-SUMS daemon shut-down code. + fix lock-handling and remove deadlock in shut-down code of remote SUMS. + modified the export manager to cope with spaces in between record-set specification filters. + modified the email-registration-checking code to perform case-insensitive email domain comparison (and not case-sensitive comparisons). + modified the email-registration code to use a random UUID to identify a registration attempt (instead of using a UUID that could not be unique because it was based upon the requestor's email address). + modified the jsoc info CGI to use UTF8 encoding for all input/output streams to avoid issues with Latin-1 characters in the JSOC DB. + modified the jsocextinfo CGI to swallow all lower-level output (i.e., from C jsoc_info) so that there is no duplication in error messages in the resulting dynamic HTML. + modified the email-registration code to fix an issue where the user's email address had the local name and the domain name switched during logging. + revamped the email-registration code to properly parse the incoming email responses from requesters (replaced the home-grown parser with calls into a library that truly knows how to parse the various formats an email message can assume - and there are many). + fixed a syntax error in the email-registration code in a couple of error-handling print statements. + added SU dir group write permissions in the multithreaded SUMS code so that users in the SU-creation linux group can write in the SUs created by SUMS. + fixed code in the multithreaded SUMS alloc function that maps tape group to sums set. + created an index on the wd column of sum_partn_alloc when this table is first created; SUMS needs to query on this column, so let's make it a fast query. + switched to using a library to parse and URL-decode CGI arguments to show_info; the home-grown method performs the URL-decode first, then the argument extraction; the order of these operations needed to be reversed. + fixed the rings project Rules.mk make file; a newer version of the OLA module must be built. + fixed an unintialized variable in the im_patch export processing DRMS module.