Release Notes JSOC V4.6 02Sep2008 ----------------------- --------- IMPORTANT: If you use the scriptj jsoc_update.pl to get the latest release, please note that there was a critical bug fixed in this script. However, previous versions of JSOC (before 4.6) still have this bug - most likely you have this bug, so if you run jsoc_update.pl, your update will not work properly. To fix this, first cd to your JSOC tree root, then run "cvs update jsoc_update.pl". Then you may run jsoc_update.pl to get the latest release. A release is a set of files, each having a specific version. And a release typcially has a version number because over time you have newer and newer releases of the same product. For example, a hypothetical 1.3 release may contain fileA#1.8, fileB#1.2, fileC#2.2 and a 1.4 release may contain fileA#2.5, fileB#2.1, fileC#2.9. JSOC releases are similarly versioned and contain a set of such files. JSOC release code is guaranteed to compile on cluster nodes (eg., n00, n02). The resulting binaries have been minimally tested. At the time of the creation of the release, the release versions of each file will be the most recent. But as time passes, newer versions of some files will be made, and there is no guarantee that these changes will not destabilize JSOC (ie., they may cause JSOC to no longer compile or execute properly). There are several ways to use this release. If you wish to simply use pre-built binaries, you can simply use the production binaries, which are located at /home/production/cvs/JSOC. Every time a release is created, the binaries in this location get updated. Only the production user can update these binaries. So, you could run /home/production/cvs/JSOC/bin/linux_x86_64/show_keys, for example. If instead you want to work with stable source files, then you must have a sandbox, which is a local copy (in your home directory) of the files in the cvs depot. You would probably want to work with a sandbox if you plan on making eventual changes to the depot files. Changes you make to your sandbox files are not visible to other users until you "commit" those changes back to the cvs depot. Please see "If You Don't Have a Sandbox" below for more information on how to create a sandbox. There is also a "working" release which resides in in /home/jsoc/cvs/JSOC. New files may be placed here and existing files may be edited for common use before the next official release. Each time a release gets created, the source and binaries of the working release get updated. WARNING: the files you see here may not be stable since by the time you see them, another user may have edited them. Only the production release is guaranteed to be stable and unchanged between releases. Obtaining the Release --------------------- To update your working directory to this release, or to check-out this release anew, please visit http://jsoc.stanford.edu/jsocwiki/CvsInit. Please keep in mind that users may have modified files since the release was created, so use of the scripts documented in the web page may result in a working directory whose content is not identical to the release. If updating, you can supply the flag "-R" to the jsoc_update.pl and jsoc_sync.pl scripts to download the latest release. This will ensure that your working directory has the exact, latest release versions of the files (eg., jsoc_sync.csh -R). If checking-out, you can supply the argument "-r Ver_LATEST" to the "cvs checkout" command to achieve the analogous result, but for a fresh checkout. WARNING: if you use the "-R" or "-r" flags, please use only jsoc_update.pl or jsoc_sync.pl to update your sources thereafter. Use of "-R" or "-r Ver_LATEST" will result in a cvs "sticky flag" being set. jsoc_update.pl and jsoc_sync.pl clear this sticky flag. Additional Info --------------- If you are unfamiliar with the use of cvs see the file: JSOC/CM/working_with_sandbox.txt. There's a linux4 cvs gui at xim:/usr/bin/lincvs Also on our jsoc web page: http://jsoc.stanford.edu/cvs/JSOC/ Use the Apache cvs gui to see the diffs. For example, go to http://jsoc.stanford.edu/cvs/JSOC/base/drms/ and click on the name in the File column and then click on "diffs to previous #" to see the diffs. Changes since previous release (V4.5 - July 16, 2008) ----------------------------------------------------- NEW FEATURES: * FITS implementation of TAS files - TAS (Tiled-Array Storage) is a DRMS segment protocol whereby data from more than one DRMS record are stored to the same file. In previous JSOC releases, the file was a homegrown file format (that didn't work 100% and it was not cross-platorm compatible). In this release, TAS uses a FITS file format. The TAS file has an extra dimension (one more than the number of dimensions of the data) - each value along this dimension corresponds to one DRMS record. The TAS file can be compressed for non-floating point data. For floating-point data, it cannot be compressed (compression would be lossy if this were allowed). Open TAS files are cached so that multiple reads/writes to the same file do not incur unneccessary file open/close overhead. In order to use this feature, the .jsd version that creates the series that uses a TAS file must be at least version 2.1 * Support for "dsds.XXX" and "ds_mdi.XXX" JSOC data series - These are special data series that contain MDI data. The MDI data have been ingested as is into these series. These JSOC series have a generic-protocol data segment that points to a VDS directory. drms_open_records() will now recognize these special series and return a temporary record-set that contains DRMS records that represent the ingested MDI data. This temporary series will contain FITS protocol segments * Implementation of data slicing of FITS and TAS segment protocols * Overload the DRMS_Segment_t structure's per_segment field, and rename to DRMS_Keyword_t::kwflags. This was needed so that we can differentiate between keywords that existed in the original .jsd, and ones that were implicitly created. We don't always want to show the implicitly created keywords in a .jsd generated by the "show_info -j" command. * tdsignal.c - Implemented a threaded version of alarms. You can now set an alarm, and receive it, despite being in a multi-threaded environment. Added a sample module, proj/example/apps/threadsigs.c, to demonstrate how this works. * SUMS - Use the multi-threaded signal to make a new log file * tapearcinfo - New module. * SUM_alloc2() - This API allows you to specify a SUNUM to allocate space for. This differs from SUM_alloc(), which picks the SUNUM for you (the next in an increasing series). * lookdata.html - Added default seriesname filter to exclude dsds. * imgdecode - separate HMI/AIA crop/lookup tables into their own subdirs; swap words before masking out top byte to find FID; read lookup/crop table as needed even when the IMG struct has initialized=1; Add FID (low 3 bytes only) to img struct. * write_hk_to_drms - Changed status check from checking record pointer to checking status after calling drms create record. If status less than 0, print error message to log showing error code from drms. Added check for Unknown series when writing to drms, if true, write status code and message to log. Updated Debugf message to show write to keyword failure when -D flag is turned on for DEBUG_WRITE_HK_TO_DRMS in the make basic file like this:ICC_CF_ICCCOMP = -DICCCOMP $(D_GCC_FORT) -DDEBUG_WRITE_HK_TO_DRMS * time_lag_monitor.pl - Add command line args parsing and Help menu * apid_list_day_file_hsb - added apids(529,536,540) to load hsb * generated dayfiles into drms data series on production. * getdf.pl - Updated start and end dates to avoid getting beginning of next day's dayfile for HSB dayfiles. Added count of files loaded in log. Added comments on limitations, description, etc. * movedf.pl - Updated script to move files(really remove files). Updated script to show number of hkt and xml files found and log values in log-df-moc log file. Add code to bypass calling ingest_dayfile.pl script if there are no files to process. BUG FIXES: * jsoc_update.pl - This script was not first cd'ing to the JSOC tree on the remote machine (so it was trying to do a make from the directory that becomes the working directory when you ssh into the remote machine). * cfitsio - By default, the cfitsio library will be named "libcfitsio.a". However, for 64-bit machines that compile with gcc, this can be overridden in the configure script by setting the CFITSIOFNAME_GCC_X86_64 make variable. * DSDS support - The path to a dsds datadube dataset resides in vds->filename, not sds->filename. Previously, libdsds.a code was using sds->filename only, which was NULL. This caused opening of datacubes to fail. * show_info -j - Prevent segment-specific keywords from being expanded into multiple keywords when creating the jsd entries. The jsd should have the original keyword (eg., bzero), not a proliferation of segment-specific keywords (eg., bzero_000, bzero_001, bzero_002, etc.). The latter causes recursion (if you create a series from a .jsd that contains bzero_000, bzero_001, bzero_002, etc., then you end up with a series that has bzero_000_000, bzero_000_001, bzero_000_002, etc., bzero_001_000, bzero_001_0001, etc.). * drms_export - Fix mapping from DRMS to FITS keywords that have missing values. For certain data types, conversion between the type used by DRMS and the corresponding type used by FITS was done improperly. * drms_segment_read() - Fix for reading of binary-protocol data segments. The faulty code assumed that the binary reader was filling in bzero/bscale values, but our binary files have no bzero/bscale values. So the faulty code got garbage bzero/bscale values and used those when converting data, which led to garbage data. * drms_stage_records() - Loop over record-set subsets (which are specific to a series) because drms_getunits() is series-specific. Also, if, within a subset, more than the maximum allowed number of storage units are requested, loop. * create_tables.sql - removed inoperable script with name conflicting with a valid one, commented out comments and replaced suggested actions with real actions. * jsoc_info - Fixed fetch of log directory, now gives dir if online; Added virtual keyword *dir_mtime* which returns the mod time of the record directory. * add_small_image - Modified 99% scaling to use full min-max if most values clumped to few bins. OTHER CHANGES * SUMS (driven_svc) - change from return(drive); to return(tapefilenum); * SUMS (tape_svc) - change TIMEOUT to 30 * SUMS (tape_svc_proc) - add tapenxtfn * SUMS (tapearc) - reject partial blocks. special handling of group_id for T950 initial archive * SUMS (tapearcinfo, find_dir_sum_partn_alloc_dc, build_arc_to_dds.pl, ) - initial * datacapture (rsync_prod.pl, sync_prod_dcs1.pl, tapearc_do_dc, tapearc_do_dc0, tapearc_do_dc1, doingestlev0.pl) - initial * SUMS (xsum_svc) - like changes to sum_svc.c * SUMS (sum_rm.cfg.jsoc_sums) - change SLEEP, change MAX_FREE_0 * SUMS (SUM.h) - change max fn t0 7000, add MAX_TAPE_FN * SUMS (sum_rpc.h) - update SUMLIB_TapeUpdate() prototype * SUMS (SUMLIB_PavailGet.pgc) - fix printk() call for err msg * SUMS (SUMLIB_TapeUpdate.pgc) - use tapenxtfn * SUMS (SUMLIB_drop_series.pgc) - add SET STATUS=2 * SUMS (build_parc_file.pl) - get ssh info from /tmp/ssh-agent.env, upgrade w/ssh stuff, use /usr/bin/perl * SUMS (find_dir_sum_partn_alloc) - update to jsoc_sums and PGPORT * SUMS (find_dir_sum_partn_alloc.README) - update for JSOC * SUMS (sum_start_d02) - add /usr/local/logs/SUM/RESTART_AFTER_REBOOT * SUMS (sum_stop_d02) - call perl in jsoc * show_series - probably staging reenabled. * jsoc_info - probably staging reenabled. * dsview - show dsds name space and non-dsds namespace toggle * Datacapture (ingest_tlm) - add some comment about build_parc_file.pl * Datacapture (build_arc_to_dds.pl) - fix date on file name * Datacapture (tapearc_do_dc) - *** empty log message *** * store_dsds_migrate - change cp -rp dir to dir/* * lev0 - add xingest_lev0 * doingestlev0.pl - update to wait for them all to stop * hmi_time_setting.c - elim sprint_time_ISO(); put in t_obs for darks * ingest_lev0 - add do_quallev0(); put in time stamp when process .qac file * time_lag_monitor.pl - Update ds_names after reading args; Use /home/jsoc/bin/linux_x86_64/perl; Bring check_interval after command line args parsing. Simplify alert message * movedf.pl - Added | to script find statement to lookup dayfiles and xml files for day of year 200-299.