Differences between revisions 8 and 80 (spanning 72 versions)
⇤ ← Revision 8 as of 2007-11-15 14:39:00 →
Size: 9022
Editor: PDhome
Comment:
← Revision 80 as of 2015-09-03 00:16:58 → ⇥
Size: 15322
Editor: ArtAmezcua
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
= Welcome to the Wiki for JSOC software development =
[[TableOfContents]]		 == Welcome to the Wiki for JSOC software development ==
<<TableOfContents>>
Line 4: Line 4:
For general information including the basic setup to get started see the [wiki:JsocUsersGuide Users Guide]. For general information including the basic setup to get started see the [[JsocUsersGuide|Users Guide]].
Line 6: Line 6:
== News ==
 * JSOC 3.2 released -- Sep 15, 2006 See [http://jsoc.stanford.edu/release-3.2.notes release notes]
 * JSOC 3.1 released -- Aug 24, 2006 See [http://jsoc.stanford.edu/release-3.1.notes release notes]
 * JSOC 3.0 released -- Aug 14, 2006. See [http://jsoc.stanford.edu/release-3.0.notes release notes]
 * JSOC 2.3 released -- Aug 10, 2006. See [http://jsoc.stanford.edu/release-2.3.notes release notes]
 * JSOC 2.2 released -- May 30, 2006. See [http://jsoc.stanford.edu/release-2.2.notes release notes]
 * JSOC 2.1 released -- May 24, 2006. See [http://jsoc.stanford.edu/release-2.1.notes release notes]
 * JSOC 2.0 released -- February 03, 2006. See [http://jsoc.stanford.edu/release-2.0.notes release notes]
 * JSOC 1.0 released -- October 19, 2005. See [http://jsoc.stanford.edu/release-1.0.notes release notes]		 == Release History ==
 * JSOC 8.10 released -- September 2, 2015 See [[http://jsoc.stanford.edu/release_notes/release-8.10.notes|release notes]]
 * JSOC 8.9 released -- July 8, 2015 See [[http://jsoc.stanford.edu/release_notes/release-8.9.notes|release notes]] THIS RELEASE WAS RETRACTED
 * JSOC 8.8 released -- April 13, 2015 See [[http://jsoc.stanford.edu/release_notes/release-8.8.notes|release notes]]
 * JSOC 8.7 released -- February 19, 2015 See [[http://jsoc.stanford.edu/release_notes/release-8.7.notes|release notes]]
 * JSOC 8.6 released -- September 26, 2014 See [[http://jsoc.stanford.edu/release_notes/release-8.6.notes|release notes]]
 * JSOC 8.5 released -- June 23, 2014 See [[http://jsoc.stanford.edu/release_notes/release-8.5.notes|release notes]]
 * JSOC 8.4 released -- April 1, 2014 See [[http://jsoc.stanford.edu/release_notes/release-8.4.notes|release notes]]
 * JSOC 8.3 released -- February 7, 2014 See [[http://jsoc.stanford.edu/release_notes/release-8.3.notes|release notes]]
 * JSOC 8.2 released -- November 20, 2013 (release notes not prepared)
 * JSOC 8.1 released -- October 10, 2013 See [[http://jsoc.stanford.edu/release_notes/release-8.1.notes|release notes]]
 * JSOC 8.0 released -- April 22, 2013 See [[http://jsoc.stanford.edu/release_notes/release-8.0.notes|release notes]]
 * JSOC 7.0 released -- August 28, 2012 See [[http://jsoc.stanford.edu/release_notes/release-7.0.notes|release notes]]
 * JSOC 6.3 released -- May 18, 2012 See [[http://jsoc.stanford.edu/release_notes/release-6.3.notes|release notes]]
 * JSOC 6.1 released -- December 6, 2011 See [[http://jsoc.stanford.edu/release_notes/release-6.1.notes|release notes]]
 * JSOC 6.0 released -- November 16, 2011 See [[http://jsoc.stanford.edu/release_notes/release-6.0.notes|release notes]]
 * JSOC 5.14 released -- August 16, 2011 See [[http://jsoc.stanford.edu/release_notes/release-5.14.notes|release notes]]
 * JSOC 5.13 released -- June 2, 2011 See [[http://jsoc.stanford.edu/release_notes/release-5.13.notes|release notes]]
 * JSOC 5.12 released -- December 15, 2010 See [[http://jsoc.stanford.edu/release_notes/release-5.12.notes|release notes]]
 * JSOC 5.11 released -- October 6, 2010 See [[http://jsoc.stanford.edu/release_notes/release-5.11.notes|release notes]]
 * JSOC 5.10 released -- September 1, 2010 See [[http://jsoc.stanford.edu/release_notes/release-5.10.notes|release notes]]
 * JSOC 5.8 released -- Mar 16, 2010 See [[http://jsoc.stanford.edu/release_notes/release-5.8.notes|release notes]]
 * JSOC 5.7 released -- Mar 12, 2010 See [[http://jsoc.stanford.edu/release_notes/release-5.7.notes|release notes]]
 * JSOC 5.6 released -- Jan 28, 2010 (quick rel notes: jsoc_sync.pl should work better, remove a use /auto/homeX paths, added qdecoder lib (so modules can parse HTTP input), cmdparams - fix mem corruption, track which arguments actually, used, slony subscribe_series, subscribe_manage, and server code, pass original cmd args not used by drms_run to the drms_run script, cntrl-C now shuts down modue quickly, sort keywords, segments according to jsd order (in code that lists those items, like show_info -j), improve hash-table implementation (hash function wasn't working, removed unnecessary complexity that was running inefficiently), don't call SUM_put() on a SU if there are no files in the SU, fixes for VARDIM bugs, delete_series checks for series existence before doing anything, modify_series won't change series that are slonied, remove CFITSIO TFORM1 hack (this workaround was no longer necessary as we incorporated a new CFITSIO that had a bug fixed), some jsoc_fetch bugs fixed, several enhancements to lookdata and exportdata (tar export, upload of file containing rec-queries, etc.), completion of iorbit (lev0) code, addition of the rings project.)
 * JSOC 5.5 released -- Oct 09, 2009 See [[http://jsoc.stanford.edu/release_notes/release-5.5.notes|release notes]]
 * JSOC 5.3 released -- Aug 05, 2009 See [[http://jsoc.stanford.edu/release_notes/release-5.3.notes|release notes]]
 * JSOC 5.2 released -- Aug 04, 2009 See [[http://jsoc.stanford.edu/release_notes/release-5.2.notes|release notes]]
 * JSOC 5.1 released -- May 01, 2009 See [[http://jsoc.stanford.edu/release_notes/release-5.1.notes|release notes]]
 * JSOC 5.0 released -- Jan 30, 2009 See [[http://jsoc.stanford.edu/release_notes/release-5.0.notes|release notes]]
 * JSOC 4.7 released -- Oct 16, 2008 See [[http://jsoc.stanford.edu/release_notes/release-4.7.notes|release notes]]
 * JSOC 4.6 released -- Sep 03, 2008 See [[http://jsoc.stanford.edu/release_notes/release-4.6.notes|release notes]]
 * JSOC 4.5 released -- Jul 16, 2008 See [[http://jsoc.stanford.edu/release_notes/release-4.5.notes|release notes]]
 * JSOC 4.4 released -- Jun 09, 2008 See [[http://jsoc.stanford.edu/release_notes/release-4.4.notes|release notes]]
 * JSOC 4.3 released -- May 26, 2008 See [[http://jsoc.stanford.edu/release_notes/release-4.3.notes|release notes]]
 * JSOC 4.2 released -- Apr 08, 2008 See [[http://jsoc.stanford.edu/release_notes/release-4.2.notes|release notes]]
 * JSOC 4.1 released -- Mar 03, 2008 See [[http://jsoc.stanford.edu/release_notes/release-4.1.notes|release notes]]
 * JSOC 4.0 released -- Dec 07, 2007 See [[http://jsoc.stanford.edu/release_notes/release-4.0.notes|release notes]]
 * JSOC 3.8 released -- Sep 12, 2007 See [[http://jsoc.stanford.edu/release_notes/release-3.8.notes|release notes]]
 * JSOC 3.3 released -- Nov 03, 2006 See [[http://jsoc.stanford.edu/release_notes/release-3.3.notes|release notes]]
 * JSOC 3.2 released -- Sep 15, 2006 See [[http://jsoc.stanford.edu/release_notes/release-3.2.notes|release notes]]
 * JSOC 3.1 released -- Aug 24, 2006 See [[http://jsoc.stanford.edu/release_notes/release-3.1.notes|release notes]]
 * JSOC 3.0 released -- Aug 14, 2006. See [[http://jsoc.stanford.edu/release_notes/release-3.0.notes|release notes]]
 * JSOC 2.3 released -- Aug 10, 2006. See [[http://jsoc.stanford.edu/release_notes/release-2.3.notes|release notes]]
 * JSOC 2.2 released -- May 30, 2006. See [[http://jsoc.stanford.edu/release_notes/release-2.2.notes|release notes]]
 * JSOC 2.1 released -- May 24, 2006. See [[http://jsoc.stanford.edu/release_notes/release-2.1.notes|release notes]]
 * JSOC 2.0 released -- Feb 03, 2006. See [[http://jsoc.stanford.edu/release_notes/release-2.0.notes|release notes]]
 * JSOC 1.0 released -- Oct 19, 2005. See [[http://jsoc.stanford.edu/release_notes/release-1.0.notes|release notes]]
Line 18: Line 56:
 * [http://jsoc.stanford.edu/doc/DRMSreport_html/index.html DRMS (html)] -- DRMS Overview[http://hmi.stanford.edu/development/JSOC_Documents/DRMS_V10.pdf (as pdf)] - old doc, contents being verified and included in these pages.
 * [wiki:DrmsNames DRMS Names] -- DRMS Dataset Names and Queries
 * [wiki:DrmsSeriesNames DRMS Series Names] -- Data Series Reserved Names
 * Data Storage, Archive, On-Line Retention, EtC.
   * [wiki:SumsDataModel SUMS Data Storage] -- SUMS Basic Data Storage Concepts
   * [wiki:SumsArchiveTimes SUMS Archive and Retention Table] -- Meaning and implications of the values
 * [wiki:Jsd .jsd] -- JSOC Series Definition Files
 * Data series utilities to be run from a user's shell
   * [wiki:DrmsCreateSeriesCmd create_series] -- Create entries and tables for a new series in the DRMS database
   * [wiki:DrmsDescribeSeriesCmd describe_series] -- Prints a verbose description of the
named series and its current highest record number on stdout.
   * [wiki:DrmsDeleteSeriesCmd delete_series] -- Removes a series and all its associated entries from DRMS.		  A. '''DRMS Overview'''
  * [[http://jsoc.stanford.edu/doc/DRMSreport_html/index.html|DRMS (html)]] -- DRMS Overview[http://hmi.stanford.edu/development
  /JSOC_Documents/DRMS_V10.pdf (as pdf)] - old doc, contents being verified and included in these pages.
Line 31: Line 60:
== JSOC Sessions, Pipelines, and Modules (''Oh my!'') ==  A. '''Names'''
  * [[DrmsNames|DRMS Names Summary]] -- DRMS Dataset Names and Queries BNF summary
  * [[http://hmi.stanford.edu/doc/JSOC/DRMS_dataset_names.pdf|DRMS Dataset Names]] -- Full description (pdf)
  * [[DrmsSeriesNames|DRMS Series Names]] -- Data Series Reserved Names
  * SUMS Info -- Data Storage, Archive, On-Line Retention, Etc.
   * [[SumsDataModel|SUMS Data Storage]] -- SUMS Basic Data Storage Concepts
   * [[SumsArchiveTimes|SUMS Archive and Retention Table]] -- Meaning and implications of the values
 A. '''JSD Format'''
  * [[Jsd|.jsd -- JSOC Series Definition Files]]
Line 33: Line 70:
JSOC programs that use DRMS to operate on DataSeries are called "modules". Modules are run in "sessions". HMI and AIA major processing tasks are accomplished in "pipelines" consisting of one or more sessions. Pipelines are started by "PUI" (Pipeline User Interface) usually by the JSOC production team. Pipelines may also be initiated by users requesting [wiki:DataSet DataSets] via the web or by team members running locally or remotely. A DataSet is a collection of records selected by a query. In essence a dataset name is simply the query that describes it.  A. '''JSOC Sessions, Pipelines, and Modules (''Oh my!'')'''
Line 35: Line 72:
A DRMS Session is the basic unit of computing that interracts with DRMS and SUMS. At the start of a session the user connects to the DRMS database. During the session the user runs one or more modules which read or create [wiki:DataRecord DataRecords] in DataSeries. Access to the actual data stored in SUMS is accomplished within a module via the DRMS API. At the end of a session, SUMS is notified to save any new records online and/or on tape, or to delete records marked temporary to the session.  JSOC programs that use DRMS to operate on DataSeries are called "modules". Modules are run in "sessions". HMI and AIA major
 processing tasks are accomplished in "pipelines" consisting of one or more sessions. Pipelines are started by "PUI"
 (Pipeline User Interface) usually by the JSOC production team. Pipelines may also be initiated by users requesting
 [[DataSet|DataSets]] via the web or by team members running locally or remotely. A DataSet is a collection of records
 selected by a query. In essence a dataset name is simply the query that describes it.
Line 37: Line 78:
Actually using the JSOC DRMS requires running a program or module. By "program" we mean a normal shell command and by "module" we mean a program built to run within a DRMS session and communication to a drms_server. There are four types
of programs/modules:
 * '''Modules''' - Most programs that do the work of the user of JSOC are what we call "modules". On the outside modules look like programs. They must run in a DRMS session. If they are built with the normal jsoc_main program they will use an existing session if they are run from a Session Provider or will start their own use-once session if they are called stand-alone from the shell.
 * '''Utility programs''' like [wiki:DrmsCreateSeriesCmd create_series] and [wiki:DrmsDescribeSeriesCmd describe_series] which are usually used to manage the existence of dataseries, not to use dataseries. These programs talk directly to the database.
 * '''Session Providers''' like [wiki:DrmsRunCmd drms_run] or later the [wiki:JsocPui Pipeline User Interface] start DRMS sessions and execute a script file. They can also be used to execute a single instance of a module.
 * '''[wiki:DrmsServerCmd drms_server]''' which connects connects to the database and serves sessions. Most users will not need to start drms_server explicitly.		  . A DRMS Session is the basic unit of computing that interracts with DRMS and SUMS. At the start of a session the user
 connects to the DRMS database. During the session the user runs one or more modules which read or create
 [[DataRecord|DataRecords]] in DataSeries. Access to the actual data stored in SUMS is accomplished within a
 module via the DRMS API. At the end of a session, SUMS is notified to save any new records online and/or on tape,
 or to delete records marked temporary to the session.
Line 44: Line 84:
 . Actually using the JSOC DRMS requires running a program or module. By "program" we mean a normal shell command and by
 "module" we mean a program built to run within a DRMS session and communication to a drms_server. There are four types
 of programs/modules:
  * '''Modules''' - Most programs that do the work of the user of JSOC are what we call "modules". On the outside modules
  look like programs. They must run in a DRMS session. If they are built with the normal jsoc_main program they will use an
  existing session if they are run from a Session Provider or will start their own use-once session if they are called
  stand-alone from the shell.
  * '''Utility programs''' like [[http://jsoc.stanford.edu/doxygen_html/group__create__series.html|create_series]] and [[http://jsoc.stanford.edu/doxygen_html/group__describe__series.html|describe_series]]
  which are usually used to manage the existence of dataseries, not to use dataseries. These programs talk directly to the database.
  * '''Session Providers''' like [[DrmsRunCmd|drms_run]] or later the [[JsocPui|Pipeline User Interface]] start
  DRMS sessions and execute a script file. They can also be used to execute a single instance of a module.
  * '''[[http://jsoc.stanford.edu/doxygen_html/group__drms__server.html|drms_server]]''' which connects connects to the database and serves sessions.
  Most users will not need to start drms_server explicitly.
Line 45: Line 98:
The benefit of running programs as "modules" will hopefully become apparent when we start running
complex pipelines using hundreds of processors.		  The benefit of running programs as "modules" will hopefully become apparent when we start running
 complex pipelines using hundreds of processors.
Line 48: Line 101:
== Setting up Your Own DRMS == == General Development Information - Using Modules ==
 A. '''DRMS Man Pages'''
 All the JSOC "man" pages are now maintained with [[http://www.stack.nl/~dimitri/doxygen/index.html|doxygen]], a semi-automatic
 documentation tool. They are available in html form via:
  * [[http://jsoc.stanford.edu/doxygen_html/|Drms Developer's Documentation]]<<BR>>
Line 50: Line 107:
Information for developers outside the JSOC who wish to construct an independent data archive
that can work in cooperation with the JSOC and other archives (or completely independently) can be found on the
[wiki:DRMSSetup Setting up Your Own DRMS] page.		  All programs and functions have entries in the doxygen generated pages.
Line 54: Line 109:
= General Information =
== DRMS Man Pages ==		  A. '''Limits'''
 . There are limits ...
  * memory limits on number of records in the cache (512Meg / (2.5*record size) ). While this may seem like a lot, for datasets with a lot of keywords (e.g. mdi_vw_V_06h) it can be a real limit to the number of records that can be open at a time. For the vw_V example it means that DRMS_QUERY_MEM should be set to at lest 2500 (yes 2.5 gig) to open 100 days of one-minute data. Modules expecting to need tens of thousands of records opened should arrange to do the work in blocks with drms_close_records used to empty the record cache to free memory.
  * length of names of series, keywords, etc. (64 chars)
  * length of comma sep list of prime key names (1024 chars)
  * length of descriptions of series, keywords, links, and segments (254 chars)
  * length of string values of keywords (dont know)
  * number of keywords in a record (dont know)
  * number of records in a series (no fixed limit)
  * length of segment filename (255 chars)
  * length of path (511 chars)
Line 57: Line 121:
 * [http://jsoc.stanford.edu/man/index.html#man1 Drms Program/Module Man Pages, man1][[BR]]
 * [http://jsoc.stanford.edu/man/index.html#man3 Drms Subroutine Man Pages, man3]		  A. '''Log Files - Processing meta-data'''
 .There are log files. Stdout and Stderr are captured in files as well as shown during processing (depending on module
 and -v flag). These are all put into a SUMS directory and indexed in DRMS by session ID. The session ID is stored in
 each record so the log files can be retrieved if/when needed. Unless otherwise specified, the default retention time
 for log files is 10 days. The log files are archived if any one of the SUs in the current session is to be archived.
  * drms_server logs --
  . The default is no logging. When the logging option is turned on (-L), stdout and stderr are redirected to files in SU directory.
  * module logs --
  . The default is no logging. When the logging option is turned on (-L), stdout and stderr are tee-ed to files in SU directory.
Line 60: Line 131:
   All programs and modules should have unix-style man pages at [http://jsoc.stanford.edu/man/index.html#man1 man1] but Some do not yet:
   * [wiki:DrmsServerCmd drms_server],
   * [wiki:DrmsQueryCmd drms_query],
   * [wiki:DrmsCreateSeriesCmd create_series],
   * [wiki:DrmsDescribeSeriesCmd describe_series],
   * [wiki:DrmsDeleteSeriesCmd delete_series]		 == Software Development - Building Modules ==
 
 A. '''DRMS Applications Programming Interface]'''
 Descriptions of the API developed for the underlying JSOC database, and how to code with it.
  * [[DrmsApi|DRMS API summary]] - ''old''
  * [[http://jsoc.stanford.edu/doxygen_html/modules.html|DRMS API Manual]]
  -- More detailed information for users and developers of specific JSOC software programs or interfaces using the '''Doxygen''' tools.
  Please use the search engine there to find documentations for any API functions.
Line 67: Line 140:
   Note that all modules built with jsoc_main share a basic set of flags and command line keywords. See module.1 in man1.  A. '''SUMS API'''
  * [[SumsApi]]
 
 A. '''JSOC Software Tree'''
  * [[FileStruct|Software File Tree]] -- Organization of files and cvs tree.
  * [[http://jsoc.stanford.edu/cvs/|GUI for JSOC CVS]] -- GUI access to DRMS and SUMS software
  * [[http://jsoc.stanford.edu/cvs/*checkout*/jsoc/src/base/libdrms/drms_types.h?rev=HEAD&content-type=text/plain|drms_types.h]]
  . -- DRMS types and structure defintions.
Line 69: Line 149:
 A. '''Making a JSOC/DRMS Module'''
  * [[DrmsModule|DRMS Module]] -- DRMS Module Structure and Overview
  * [[DrmsApi|DRMS API]] -- DRMS Data Types and Structures and API
  * [[DrmsMakeModule|DRMS Module Compilation]] -- Running 'make' for modules
Line 70: Line 154:
== Limits ==  A. '''JSOC make system'''
  * [[JsocMakefileBackground|Background/Reference]]
  * [[JsocMakeSystem|JSOC make system understood]]
Line 72: Line 158:
There are limits ...
 * memory limits on number of records in the cache (512Meg / (2.5*record size) ). While this may seem like a lot, for datasets with a lot of keywords (e.g. mdi_vw_V_06h) it can be a real limit to the number of records that can be open at a time. For the vw_V example it means that DRMS_QUERY_MEM should be set to at lest 2500 (yes 2.5 gig) to open 100 days of one-minute data. Modules expecting to need tens of thousands of records opened should arrange to do the work in blocks with drms_close_records used to empty the record cache to free memory.
 * length of names of series, keywords, etc. (31 chars)
 * length of descriptions of series, keywords, links, and segments (254 chars)
 * length of string values of keywords (dont know)
 * number of keywords in a record (dont know)
 * number of records in a series (no fixed limit)
 * length of segment filename (255 chars)
 * length of path (511 chars)		  A. '''Web Exports via AJAX API'''
  * [[AjaxJsocConnect|AJAX Style Web Access for JSOC]] used by [[http://jsoc.stanford.edu/ajax/lookdata.html|lookdata.html]]
Line 82: Line 161:
== Log Files - Processing meta-data == == Developers' Weekly Progress Reports ==
  * DeveloperProgressReports -- An archive of developers' weekly progress reports
Line 84: Line 164:
There are log files. Stdout and Stderr are captured in files as well as shown during processing (depending on module and -v flag). These are all put into a SUMS directory and indexed in DRMS by session ID. The session ID is stored in each record so the log files can be retrieved if/when needed. Unless otherwise specified, the default retention time for log files is the maximum retention time of all SUs processed in the current session. The log files are archived if any one of the SUs in the current session is to be archived.
 * drms_server logs --
   The default is no logging. When the logging option is turned on (-L), stdout and stderr are redirected to files in SU directory.
 * module logs --
   The default is no logging. When the logging option is turned on (-L), stdout and stderr are tee-ed to files in SU directory.
= Software Development - Building Modules =
 
== JSOC Software Tree ==
 * [wiki:FileStruct Software File Tree] -- Organization of files and cvs tree.
 * [http://jsoc.stanford.edu/cvs/ GUI for JSOC CVS] -- GUI access to DRMS and SUMS software
 * [http://jsoc.stanford.edu/cvs/*checkout*/jsoc/src/base/libdrms/drms_types.h?rev=HEAD&content-type=text/plain drms_types.h] -- DRMS types and structure defintions.
== Making a JSOC/DRMS Module ==
 * [wiki:DrmsModule DRMS Module] -- DRMS Module Structure and Overview
 * [wiki:DrmsApi DRMS API] -- DRMS Data Types and Structures and API
 * [wiki:DrmsMakeModule DRMS Module Compilation] -- Running 'make' for modules		 == JSOC Development Projects ==
Line 100: Line 166:
== Making a JSOC/DRMS Library ==
 * [wiki:JsocLibrary JSOC LIbrary] -- Creating and using a JSOC library		  A. '''DSDS-Data Access from JSOC'''
  * [[http://hmi.stanford.edu/development/JSOC_Documents/JSOC_DSDS_Interface_Specification.pdf|Interface Specification (.pdf)]]
  . -- Details for builders of modules that will access MDI data directly from DRMS code.
Line 103: Line 170:
== Notes on JSOC Makefile ==
 * [wiki:JsocMakefileBackground Background/Reference]
 * [wiki:JsocMakefileAdd How to add a sub directory]		  A. '''Remote DRMS/SUMS - netDRMS'''
  * See Rick Bogart and the collection of web pages on NetDRMS at http://jsoc.stanford.edu/netdrms/
Line 107: Line 173:
== Development Notes ==
 * [wiki:DevNotes Development Notes Page]		 == JSOC Operator's Guide ==
Line 110: Line 175:
== Database Administration ==
 * [wiki:PgDBAdmin PostgreSQL]
 
== JSOC Backup and Restore ==
 * [wiki:JSOC Backup/Restore Notes Page]		  A. JSOC Administration
 '''[[JsocAdministration| JSOC Administration]]''' -- Information for management and installation of DRMS/SUMS
Line 116: Line 178:
== SUM API ==
 * [http://sun.stanford.edu/web.hmi/development/SU_Development_Plan/SUM_API.html API web page]		  A. '''Running Datacapture and lev0 Pipeline during SDI I&T'''
  * [[http://jsoc.stanford.edu/production/whattodolev0.txt]]<<BR>>
Line 119: Line 181:
== DSDS-Data Access from JSOC ==
 * [http://hmi.stanford.edu/development/JSOC_Documents/JSOC_DSDS_Interface_Specification.pdf Interface Specification (.pdf)]		 == Database Administration ==
 '''[[Dba| Database Administration]]''' -- Tips and tricks for getting information from the database.

Welcome to the Wiki for JSOC software development

Contents

  1. Welcome to the Wiki for JSOC software development
  2. Release History
  3. DRMS Data Series
  4. General Development Information - Using Modules
  5. Software Development - Building Modules
  6. Developers' Weekly Progress Reports
  7. JSOC Development Projects
  8. JSOC Operator's Guide
  9. Database Administration

For general information including the basic setup to get started see the Users Guide.

Release History

  • JSOC 8.10 released -- September 2, 2015 See release notes

  • JSOC 8.9 released -- July 8, 2015 See release notes THIS RELEASE WAS RETRACTED

  • JSOC 8.8 released -- April 13, 2015 See release notes

  • JSOC 8.7 released -- February 19, 2015 See release notes

  • JSOC 8.6 released -- September 26, 2014 See release notes

  • JSOC 8.5 released -- June 23, 2014 See release notes

  • JSOC 8.4 released -- April 1, 2014 See release notes

  • JSOC 8.3 released -- February 7, 2014 See release notes

  • JSOC 8.2 released -- November 20, 2013 (release notes not prepared)

  • JSOC 8.1 released -- October 10, 2013 See release notes

  • JSOC 8.0 released -- April 22, 2013 See release notes

  • JSOC 7.0 released -- August 28, 2012 See release notes

  • JSOC 6.3 released -- May 18, 2012 See release notes

  • JSOC 6.1 released -- December 6, 2011 See release notes

  • JSOC 6.0 released -- November 16, 2011 See release notes

  • JSOC 5.14 released -- August 16, 2011 See release notes

  • JSOC 5.13 released -- June 2, 2011 See release notes

  • JSOC 5.12 released -- December 15, 2010 See release notes

  • JSOC 5.11 released -- October 6, 2010 See release notes

  • JSOC 5.10 released -- September 1, 2010 See release notes

  • JSOC 5.8 released -- Mar 16, 2010 See release notes

  • JSOC 5.7 released -- Mar 12, 2010 See release notes

  • JSOC 5.6 released -- Jan 28, 2010 (quick rel notes: jsoc_sync.pl should work better, remove a use /auto/homeX paths, added qdecoder lib (so modules can parse HTTP input), cmdparams - fix mem corruption, track which arguments actually, used, slony subscribe_series, subscribe_manage, and server code, pass original cmd args not used by drms_run to the drms_run script, cntrl-C now shuts down modue quickly, sort keywords, segments according to jsd order (in code that lists those items, like show_info -j), improve hash-table implementation (hash function wasn't working, removed unnecessary complexity that was running inefficiently), don't call SUM_put() on a SU if there are no files in the SU, fixes for VARDIM bugs, delete_series checks for series existence before doing anything, modify_series won't change series that are slonied, remove CFITSIO TFORM1 hack (this workaround was no longer necessary as we incorporated a new CFITSIO that had a bug fixed), some jsoc_fetch bugs fixed, several enhancements to lookdata and exportdata (tar export, upload of file containing rec-queries, etc.), completion of iorbit (lev0) code, addition of the rings project.)

  • JSOC 5.5 released -- Oct 09, 2009 See release notes

  • JSOC 5.3 released -- Aug 05, 2009 See release notes

  • JSOC 5.2 released -- Aug 04, 2009 See release notes

  • JSOC 5.1 released -- May 01, 2009 See release notes

  • JSOC 5.0 released -- Jan 30, 2009 See release notes

  • JSOC 4.7 released -- Oct 16, 2008 See release notes

  • JSOC 4.6 released -- Sep 03, 2008 See release notes

  • JSOC 4.5 released -- Jul 16, 2008 See release notes

  • JSOC 4.4 released -- Jun 09, 2008 See release notes

  • JSOC 4.3 released -- May 26, 2008 See release notes

  • JSOC 4.2 released -- Apr 08, 2008 See release notes

  • JSOC 4.1 released -- Mar 03, 2008 See release notes

  • JSOC 4.0 released -- Dec 07, 2007 See release notes

  • JSOC 3.8 released -- Sep 12, 2007 See release notes

  • JSOC 3.3 released -- Nov 03, 2006 See release notes

  • JSOC 3.2 released -- Sep 15, 2006 See release notes

  • JSOC 3.1 released -- Aug 24, 2006 See release notes

  • JSOC 3.0 released -- Aug 14, 2006. See release notes

  • JSOC 2.3 released -- Aug 10, 2006. See release notes

  • JSOC 2.2 released -- May 30, 2006. See release notes

  • JSOC 2.1 released -- May 24, 2006. See release notes

  • JSOC 2.0 released -- Feb 03, 2006. See release notes

  • JSOC 1.0 released -- Oct 19, 2005. See release notes

DRMS Data Series

  1. DRMS Overview

  2. Names

  3. JSD Format

  4. JSOC Sessions, Pipelines, and Modules (Oh my!)

    JSOC programs that use DRMS to operate on DataSeries are called "modules". Modules are run in "sessions". HMI and AIA major processing tasks are accomplished in "pipelines" consisting of one or more sessions. Pipelines are started by "PUI" (Pipeline User Interface) usually by the JSOC production team. Pipelines may also be initiated by users requesting DataSets via the web or by team members running locally or remotely. A DataSet is a collection of records selected by a query. In essence a dataset name is simply the query that describes it.

  5. A DRMS Session is the basic unit of computing that interracts with DRMS and SUMS. At the start of a session the user connects to the DRMS database. During the session the user runs one or more modules which read or create

    DataRecords in DataSeries. Access to the actual data stored in SUMS is accomplished within a module via the DRMS API. At the end of a session, SUMS is notified to save any new records online and/or on tape, or to delete records marked temporary to the session.

  6. Actually using the JSOC DRMS requires running a program or module. By "program" we mean a normal shell command and by "module" we mean a program built to run within a DRMS session and communication to a drms_server. There are four types of programs/modules:

    • Modules - Most programs that do the work of the user of JSOC are what we call "modules". On the outside modules look like programs. They must run in a DRMS session. If they are built with the normal jsoc_main program they will use an existing session if they are run from a Session Provider or will start their own use-once session if they are called stand-alone from the shell.

    • Utility programs like create_series and describe_series which are usually used to manage the existence of dataseries, not to use dataseries. These programs talk directly to the database.

    • Session Providers like drms_run or later the Pipeline User Interface start DRMS sessions and execute a script file. They can also be used to execute a single instance of a module.

    • drms_server which connects connects to the database and serves sessions. Most users will not need to start drms_server explicitly.

    The benefit of running programs as "modules" will hopefully become apparent when we start running complex pipelines using hundreds of processors.

General Development Information - Using Modules

  1. DRMS Man Pages All the JSOC "man" pages are now maintained with doxygen, a semi-automatic documentation tool. They are available in html form via:

    All programs and functions have entries in the doxygen generated pages.

  2. Limits

  3. There are limits ...
    • memory limits on number of records in the cache (512Meg / (2.5*record size) ). While this may seem like a lot, for datasets with a lot of keywords (e.g. mdi_vw_V_06h) it can be a real limit to the number of records that can be open at a time. For the vw_V example it means that DRMS_QUERY_MEM should be set to at lest 2500 (yes 2.5 gig) to open 100 days of one-minute data. Modules expecting to need tens of thousands of records opened should arrange to do the work in blocks with drms_close_records used to empty the record cache to free memory.
    • length of names of series, keywords, etc. (64 chars)
    • length of comma sep list of prime key names (1024 chars)
    • length of descriptions of series, keywords, links, and segments (254 chars)
    • length of string values of keywords (dont know)
    • number of keywords in a record (dont know)
    • number of records in a series (no fixed limit)
    • length of segment filename (255 chars)
    • length of path (511 chars)

  4. Log Files - Processing meta-data

  5. There are log files. Stdout and Stderr are captured in files as well as shown during processing (depending on module and -v flag). These are all put into a SUMS directory and indexed in DRMS by session ID. The session ID is stored in each record so the log files can be retrieved if/when needed. Unless otherwise specified, the default retention time for log files is 10 days. The log files are archived if any one of the SUs in the current session is to be archived.
    • drms_server logs --
    • The default is no logging. When the logging option is turned on (-L), stdout and stderr are redirected to files in SU directory.
    • module logs --
    • The default is no logging. When the logging option is turned on (-L), stdout and stderr are tee-ed to files in SU directory.

Software Development - Building Modules

  1. DRMS Applications Programming Interface] Descriptions of the API developed for the underlying JSOC database, and how to code with it.

    • DRMS API summary - old

    • DRMS API Manual -- More detailed information for users and developers of specific JSOC software programs or interfaces using the Doxygen tools. Please use the search engine there to find documentations for any API functions.

  2. SUMS API

  3. JSOC Software Tree

  4. Making a JSOC/DRMS Module

  5. JSOC make system

  6. Web Exports via AJAX API

Developers' Weekly Progress Reports

JSOC Development Projects

  1. DSDS-Data Access from JSOC

  2. Remote DRMS/SUMS - netDRMS

JSOC Operator's Guide

  1. JSOC Administration

    JSOC Administration -- Information for management and installation of DRMS/SUMS

  2. Running Datacapture and lev0 Pipeline during SDI I&T

Database Administration

JsocWiki: JsocDevelopersGuide (last edited 2021-11-16 01:34:32 by ArtAmezcua)