Welcome to the Wiki for JSOC software development
For general information including the basic setup to get started see the Users Guide.
JSOC 9.1 released -- August 16, 2017 See release notes
JSOC 8.12 released -- December 8, 2016 See release notes
JSOC 8.11 released -- June 3, 2016 See release notes
JSOC 8.10 released -- September 2, 2015 See release notes
JSOC 8.9 released -- July 8, 2015 See release notes THIS RELEASE WAS RETRACTED
JSOC 8.8 released -- April 13, 2015 See release notes
JSOC 8.7 released -- February 19, 2015 See release notes
JSOC 8.6 released -- September 26, 2014 See release notes
JSOC 8.5 released -- June 23, 2014 See release notes
JSOC 8.4 released -- April 1, 2014 See release notes
JSOC 8.3 released -- February 7, 2014 See release notes
- JSOC 8.2 released -- November 20, 2013 (release notes not prepared)
JSOC 8.1 released -- October 10, 2013 See release notes
JSOC 8.0 released -- April 22, 2013 See release notes
JSOC 7.0 released -- August 28, 2012 See release notes
JSOC 6.3 released -- May 18, 2012 See release notes
JSOC 6.1 released -- December 6, 2011 See release notes
JSOC 6.0 released -- November 16, 2011 See release notes
JSOC 5.14 released -- August 16, 2011 See release notes
JSOC 5.13 released -- June 2, 2011 See release notes
JSOC 5.12 released -- December 15, 2010 See release notes
JSOC 5.11 released -- October 6, 2010 See release notes
JSOC 5.10 released -- September 1, 2010 See release notes
JSOC 5.8 released -- Mar 16, 2010 See release notes
JSOC 5.7 released -- Mar 12, 2010 See release notes
- JSOC 5.6 released -- Jan 28, 2010 (quick rel notes: jsoc_sync.pl should work better, remove a use /auto/homeX paths, added qdecoder lib (so modules can parse HTTP input), cmdparams - fix mem corruption, track which arguments actually, used, slony subscribe_series, subscribe_manage, and server code, pass original cmd args not used by drms_run to the drms_run script, cntrl-C now shuts down modue quickly, sort keywords, segments according to jsd order (in code that lists those items, like show_info -j), improve hash-table implementation (hash function wasn't working, removed unnecessary complexity that was running inefficiently), don't call SUM_put() on a SU if there are no files in the SU, fixes for VARDIM bugs, delete_series checks for series existence before doing anything, modify_series won't change series that are slonied, remove CFITSIO TFORM1 hack (this workaround was no longer necessary as we incorporated a new CFITSIO that had a bug fixed), some jsoc_fetch bugs fixed, several enhancements to lookdata and exportdata (tar export, upload of file containing rec-queries, etc.), completion of iorbit (lev0) code, addition of the rings project.)
JSOC 5.5 released -- Oct 09, 2009 See release notes
JSOC 5.3 released -- Aug 05, 2009 See release notes
JSOC 5.2 released -- Aug 04, 2009 See release notes
JSOC 5.1 released -- May 01, 2009 See release notes
JSOC 5.0 released -- Jan 30, 2009 See release notes
JSOC 4.7 released -- Oct 16, 2008 See release notes
JSOC 4.6 released -- Sep 03, 2008 See release notes
JSOC 4.5 released -- Jul 16, 2008 See release notes
JSOC 4.4 released -- Jun 09, 2008 See release notes
JSOC 4.3 released -- May 26, 2008 See release notes
JSOC 4.2 released -- Apr 08, 2008 See release notes
JSOC 4.1 released -- Mar 03, 2008 See release notes
JSOC 4.0 released -- Dec 07, 2007 See release notes
JSOC 3.8 released -- Sep 12, 2007 See release notes
JSOC 3.3 released -- Nov 03, 2006 See release notes
JSOC 3.2 released -- Sep 15, 2006 See release notes
JSOC 3.1 released -- Aug 24, 2006 See release notes
JSOC 3.0 released -- Aug 14, 2006. See release notes
JSOC 2.3 released -- Aug 10, 2006. See release notes
JSOC 2.2 released -- May 30, 2006. See release notes
JSOC 2.1 released -- May 24, 2006. See release notes
JSOC 2.0 released -- Feb 03, 2006. See release notes
JSOC 1.0 released -- Oct 19, 2005. See release notes
DRMS Data Series
DRMS Names Summary -- DRMS Dataset Names and Queries BNF summary
DRMS Dataset Names -- Full description (pdf)
DRMS Series Names -- Data Series Reserved Names
- SUMS Info -- Data Storage, Archive, On-Line Retention, Etc.
JSOC Sessions, Pipelines, and Modules (Oh my!)
JSOC programs that use DRMS to operate on DataSeries are called "modules". Modules are run in "sessions". HMI and AIA major processing tasks are accomplished in "pipelines" consisting of one or more sessions. Pipelines are started by "PUI" (Pipeline User Interface) usually by the JSOC production team. Pipelines may also be initiated by users requesting DataSets via the web or by team members running locally or remotely. A DataSet is a collection of records selected by a query. In essence a dataset name is simply the query that describes it.
- A DRMS Session is the basic unit of computing that interracts with DRMS and SUMS. At the start of a session the user connects to the DRMS database. During the session the user runs one or more modules which read or create
DataRecords in DataSeries. Access to the actual data stored in SUMS is accomplished within a module via the DRMS API. At the end of a session, SUMS is notified to save any new records online and/or on tape, or to delete records marked temporary to the session.
- Actually using the JSOC DRMS requires running a program or module. By "program" we mean a normal shell command and by "module" we mean a program built to run within a DRMS session and communication to a drms_server. There are four types of programs/modules:
Modules - Most programs that do the work of the user of JSOC are what we call "modules". On the outside modules look like programs. They must run in a DRMS session. If they are built with the normal jsoc_main program they will use an existing session if they are run from a Session Provider or will start their own use-once session if they are called stand-alone from the shell.
drms_server which connects connects to the database and serves sessions. Most users will not need to start drms_server explicitly.
General Development Information - Using Modules
DRMS Man Pages All the JSOC "man" pages are now maintained with doxygen, a semi-automatic documentation tool. They are available in html form via:
- There are limits ...
- memory limits on number of records in the cache (512Meg / (2.5*record size) ). While this may seem like a lot, for datasets with a lot of keywords (e.g. mdi_vw_V_06h) it can be a real limit to the number of records that can be open at a time. For the vw_V example it means that DRMS_QUERY_MEM should be set to at lest 2500 (yes 2.5 gig) to open 100 days of one-minute data. Modules expecting to need tens of thousands of records opened should arrange to do the work in blocks with drms_close_records used to empty the record cache to free memory.
- length of names of series, keywords, etc. (64 chars)
- length of comma sep list of prime key names (1024 chars)
- length of descriptions of series, keywords, links, and segments (254 chars)
- length of string values of keywords (dont know)
- number of keywords in a record (dont know)
- number of records in a series (no fixed limit)
- length of segment filename (255 chars)
- length of path (511 chars)
Log Files - Processing meta-data
- There are log files. Stdout and Stderr are captured in files as well as shown during processing (depending on module and -v flag). These are all put into a SUMS directory and indexed in DRMS by session ID. The session ID is stored in each record so the log files can be retrieved if/when needed. Unless otherwise specified, the default retention time for log files is 10 days. The log files are archived if any one of the SUs in the current session is to be archived.
- drms_server logs --
- The default is no logging. When the logging option is turned on (-L), stdout and stderr are redirected to files in SU directory.
- module logs --
- The default is no logging. When the logging option is turned on (-L), stdout and stderr are tee-ed to files in SU directory.
Software Development - Building Modules
DRMS Applications Programming Interface] Descriptions of the API developed for the underlying JSOC database, and how to code with it.
JSOC Software Tree
Making a JSOC/DRMS Module
JSOC make system
Web Exports via AJAX API
Developers' Weekly Progress Reports
DeveloperProgressReports -- An archive of developers' weekly progress reports
JSOC Development Projects
DSDS-Data Access from JSOC
- -- Details for builders of modules that will access MDI data directly from DRMS code.
Remote DRMS/SUMS - netDRMS
See Rick Bogart and the collection of web pages on NetDRMS at http://jsoc.stanford.edu/netdrms/
JSOC Operator's Guide
- JSOC Administration
JSOC Administration -- Information for management and installation of DRMS/SUMS
Running Datacapture and lev0 Pipeline during SDI I&T
Database Administration -- Tips and tricks for getting information from the database.