Differences between revisions 34 and 43 (spanning 9 versions)
⇤ ← Revision 34 as of 2009-02-17 09:41:20 →
Size: 10389
Editor: DNab42dffa
Comment:
← Revision 43 as of 2009-04-08 09:08:26 → ⇥
Size: 10902
Editor: phil
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
~+ '''Welcome to the Wiki for JSOC software development''' +~ == Welcome to the Wiki for JSOC software development ==
Line 29: Line 29:
 * [http://jsoc.stanford.edu/doc/DRMSreport_html/index.html DRMS (html)] -- DRMS Overview[http://hmi.stanford.edu/development/JSOC_Documents/DRMS_V10.pdf (as pdf)] - old doc, contents being verified and included in these pages.
 * [wiki:DrmsNames DRMS Names Summary] -- DRMS Dataset Names and Queries BNF summary
 * [http://hmi.stanford.edu/doc/JSOC/DRMS_dataset_names.pdf DRMS Dataset Names] -- Full description (pdf)
 * [wiki:DrmsSeriesNames DRMS Series Names] -- Data Series Reserved Names
 * Data Storage, Archive, On-Line Retention, EtC.		  A. '''DRMS Overview'''
  * [http://jsoc.stanford.edu/doc/DRMSreport_html/index.html DRMS (html)] -- DRMS Overview[http://hmi.stanford.edu/development    /JSOC_Documents/DRMS_V10.pdf (as pdf)] - old doc, contents being verified and included in these pages.

 A. '''Names'''
  * [wiki:DrmsNames DRMS Names Summary] -- DRMS Dataset Names and Queries BNF summary
  * [http://hmi.stanford.edu/doc/JSOC/DRMS_dataset_names.pdf DRMS Dataset Names] -- Full description (pdf)
  * [wiki:DrmsSeriesNames DRMS Series Names] -- Data Series Reserved Names
  * SUMS Info -- Data Storage, Archive, On-Line Retention, Etc.
Line 36: Line 40:
 * [wiki:Jsd .jsd] -- JSOC Series Definition Files  A. '''JSD Format'''
  * [wiki:Jsd .jsd -- JSOC Series Definition Files]
Line 38: Line 43:
=== JSOC Sessions, Pipelines, and Modules (''Oh my!'') ===  A. '''JSOC Sessions, Pipelines, and Modules (''Oh my!'')'''
Line 40: Line 45:
JSOC programs that use DRMS to operate on DataSeries are called "modules". Modules are run in "sessions". HMI and AIA major processing tasks are accomplished in "pipelines" consisting of one or more sessions. Pipelines are started by "PUI" (Pipeline User Interface) usually by the JSOC production team. Pipelines may also be initiated by users requesting [wiki:DataSet DataSets] via the web or by team members running locally or remotely. A DataSet is a collection of records selected by a query. In essence a dataset name is simply the query that describes it.  JSOC programs that use DRMS to operate on DataSeries are called "modules". Modules are run in "sessions". HMI and AIA major
 processing tasks are accomplished in "pipelines" consisting of one or more sessions. Pipelines are started by "PUI"
 (Pipeline User Interface) usually by the JSOC production team. Pipelines may also be initiated by users requesting
 [wiki:DataSet DataSets] via the web or by team members running locally or remotely. A DataSet is a collection of records
 selected by a query. In essence a dataset name is simply the query that describes it.
Line 42: Line 51:
A DRMS Session is the basic unit of computing that interracts with DRMS and SUMS. At the start of a session the user connects to the DRMS database. During the session the user runs one or more modules which read or create [wiki:DataRecord DataRecords] in DataSeries. Access to the actual data stored in SUMS is accomplished within a module via the DRMS API. At the end of a session, SUMS is notified to save any new records online and/or on tape, or to delete records marked temporary to the session.  . A DRMS Session is the basic unit of computing that interracts with DRMS and SUMS. At the start of a session the user
 connects to the DRMS database. During the session the user runs one or more modules which read or create   [wiki:DataRecord DataRecords] in DataSeries. Access to the actual data stored in SUMS is accomplished within a   module via the DRMS API. At the end of a session, SUMS is notified to save any new records online and/or on tape,
 or to delete records marked temporary to the session.
Line 44: Line 57:
Actually using the JSOC DRMS requires running a program or module. By "program" we mean a normal shell command and by "module" we mean a program built to run within a DRMS session and communication to a drms_server. There are four types
of programs/modules:
 * '''Modules''' - Most programs that do the work of the user of JSOC are what we call "modules". On the outside modules look like programs. They must run in a DRMS session. If they are built with the normal jsoc_main program they will use an existing session if they are run from a Session Provider or will start their own use-once session if they are called stand-alone from the shell.
 * '''Utility programs''' like [wiki:DrmsCreateSeriesCmd create_series] and [wiki:DrmsDescribeSeriesCmd describe_series] which are usually used to manage the existence of dataseries, not to use dataseries. These programs talk directly to the database.
 * '''Session Providers''' like [wiki:DrmsRunCmd drms_run] or later the [wiki:JsocPui Pipeline User Interface] start DRMS sessions and execute a script file. They can also be used to execute a single instance of a module.
 * '''[wiki:DrmsServerCmd drms_server]''' which connects connects to the database and serves sessions. Most users will not need to start drms_server explicitly.		  . Actually using the JSOC DRMS requires running a program or module. By "program" we mean a normal shell command and by
 "module" we mean a program built to run within a DRMS session and communication to a drms_server. There are four types 
 of programs/modules:
  * '''Modules''' - Most programs that do the work of the user of JSOC are what we call "modules". On the outside modules
  look like programs. They must run in a DRMS session. If they are built with the normal jsoc_main program they will use an
  existing session if they are run from a Session Provider or will start their own use-once session if they are called    stand-alone from the shell.
  * '''Utility programs''' like [wiki:DrmsCreateSeriesCmd create_series] and [wiki:DrmsDescribeSeriesCmd describe_series]
  which are usually used to manage the existence of dataseries, not to use dataseries. These programs talk directly to the database.
  * '''Session Providers''' like [wiki:DrmsRunCmd drms_run] or later the [wiki:JsocPui Pipeline User Interface] start    DRMS sessions and execute a script file. They can also be used to execute a single instance of a module.
  * '''[wiki:DrmsServerCmd drms_server]''' which connects connects to the database and serves sessions.
  Most users will not need to start drms_server explicitly.
Line 51: Line 71:
The benefit of running programs as "modules" will hopefully become apparent when we start running
complex pipelines using hundreds of processors.		  The benefit of running programs as "modules" will hopefully become apparent when we start running
 complex pipelines using hundreds of processors.
Line 54: Line 74:
== General Information ==
=== DRMS Man Pages ===
All the JSOC "man" pages are now maintained with [http://www.stack.nl/~dimitri/doxygen/index.html doxygen], a semi-automatic documentation tool. They are available in html form via:
 * [http://jsoc.stanford.edu/doxygen_html/ Drms Developer's Documentation][[BR]]		 == General Development Information - Using Modules ==
 A. '''DRMS Man Pages'''
 All the JSOC "man" pages are now maintained with [http://www.stack.nl/~dimitri/doxygen/index.html doxygen], a semi-automatic
 documentation tool. They are available in html form via:
  * [http://jsoc.stanford.edu/doxygen_html/ Drms Developer's Documentation][[BR]]
Line 59: Line 80:
All programs and functions have entries in the doxygen generated pages. Old documentation still exists for these:
   
 * [wiki:DrmsServerCmd drms_server],
 * [wiki:DrmsQueryCmd drms_query],
 * [wiki:DrmsCreateSeriesCmd create_series],
 * [wiki:DrmsDescribeSeriesCmd describe_series],
 * [wiki:DrmsDeleteSeriesCmd delete_series] 		 All programs and functions have entries in the doxygen generated pages.
Line 67: Line 82:
=== Limits ===  A. '''Limits'''
 . There are limits ...
  * memory limits on number of records in the cache (512Meg / (2.5*record size) ). While this may seem like a lot, for datasets with a lot of keywords (e.g. mdi_vw_V_06h) it can be a real limit to the number of records that can be open at a time. For the vw_V example it means that DRMS_QUERY_MEM should be set to at lest 2500 (yes 2.5 gig) to open 100 days of one-minute data. Modules expecting to need tens of thousands of records opened should arrange to do the work in blocks with drms_close_records used to empty the record cache to free memory.
  * length of names of series, keywords, etc. (64 chars)
  * length of comma sep list of prime key names (1024 chars)
  * length of descriptions of series, keywords, links, and segments (254 chars)
  * length of string values of keywords (dont know)
  * number of keywords in a record (dont know)
  * number of records in a series (no fixed limit)
  * length of segment filename (255 chars)
  * length of path (511 chars)
Line 69: Line 94:
There are limits ...
 * memory limits on number of records in the cache (512Meg / (2.5*record size) ). While this may seem like a lot, for datasets with a lot of keywords (e.g. mdi_vw_V_06h) it can be a real limit to the number of records that can be open at a time. For the vw_V example it means that DRMS_QUERY_MEM should be set to at lest 2500 (yes 2.5 gig) to open 100 days of one-minute data. Modules expecting to need tens of thousands of records opened should arrange to do the work in blocks with drms_close_records used to empty the record cache to free memory.
 * length of names of series, keywords, etc. (64 chars)
 * length of comma sep list of prime key names (1024 chars)
 * length of descriptions of series, keywords, links, and segments (254 chars)
 * length of string values of keywords (dont know)
 * number of keywords in a record (dont know)
 * number of records in a series (no fixed limit)
 * length of segment filename (255 chars)
 * length of path (511 chars)		  A. '''Log Files - Processing meta-data'''
 .There are log files. Stdout and Stderr are captured in files as well as shown during processing (depending on module
 and -v flag). These are all put into a SUMS directory and indexed in DRMS by session ID. The session ID is stored in
 each record so the log files can be retrieved if/when needed. Unless otherwise specified, the default retention time
 for log files is 10 days. The log files are archived if any one of the SUs in the current session is to be archived.
  * drms_server logs --
  . The default is no logging. When the logging option is turned on (-L), stdout and stderr are redirected to files in SU directory.
  * module logs --
  . The default is no logging. When the logging option is turned on (-L), stdout and stderr are tee-ed to files in SU directory.
Line 80: Line 104:
=== Log Files - Processing meta-data ===

There are log files. Stdout and Stderr are captured in files as well as shown during processing (depending on module and -v flag). These are all put into a SUMS directory and indexed in DRMS by session ID. The session ID is stored in each record so the log files can be retrieved if/when needed. Unless otherwise specified, the default retention time for log files is 10 days. The log files are archived if any one of the SUs in the current session is to be archived.
 * drms_server logs --
   The default is no logging. When the logging option is turned on (-L), stdout and stderr are redirected to files in SU directory.
 * module logs --
   The default is no logging. When the logging option is turned on (-L), stdout and stderr are tee-ed to files in SU directory.
Line 89: Line 106:
=== JSOC Software Tree ===
 * [wiki:FileStruct Software File Tree] -- Organization of files and cvs tree.
 * [http://jsoc.stanford.edu/cvs/ GUI for JSOC CVS] -- GUI access to DRMS and SUMS software
 * [http://jsoc.stanford.edu/cvs/*checkout*/jsoc/src/base/libdrms/drms_types.h?rev=HEAD&content-type=text/plain drms_types.h] -- DRMS types and structure defintions.
=== Making a JSOC/DRMS Module ===
 * [wiki:DrmsModule DRMS Module] -- DRMS Module Structure and Overview
 * [wiki:DrmsApi DRMS API] -- DRMS Data Types and Structures and API
 * [wiki:DrmsMakeModule DRMS Module Compilation] -- Running 'make' for modules		  A. '''DRMS Applications Programming Interface]'''
 Descriptions of the API developed for the underlying JSOC database, and how to code with it.
  * [:DrmsApi:DRMS API summary] - ''old''
  * [http://jsoc.stanford.edu/doxygen_html/modules.html DRMS API Manual]
  -- More detailed information for users and developers of specific JSOC software programs or interfaces using the '''Doxygen''' tools.
  Please use the search engine there to find documentations for any API functions.
Line 98: Line 113:
=== Making a JSOC/DRMS Library ===
 * [wiki:JsocLibrary JSOC LIbrary] -- Creating and using a JSOC library		  A. '''SUMS API'''
  * [http://sun.stanford.edu/web.hmi/development/SU_Development_Plan/SUM_API.html API web page]
Line 101: Line 116:
=== Notes on JSOC Makefile ===
 * [wiki:JsocMakefileBackground Background/Reference]
 * [wiki:JsocMakefileAdd How to add a sub directory]		  A. '''Developers' Weekly Progress Reports'''
  * DeveloperProgressReports -- An archive of developers' weekly progress reports
 
 A. '''JSOC Software Tree'''
  * [wiki:FileStruct Software File Tree] -- Organization of files and cvs tree.
  * [http://jsoc.stanford.edu/cvs/ GUI for JSOC CVS] -- GUI access to DRMS and SUMS software
  * [http://jsoc.stanford.edu/cvs/*checkout*/jsoc/src/base/libdrms/drms_types.h?rev=HEAD&content-type=text/plain drms_types.h]
  . -- DRMS types and structure defintions.
Line 105: Line 125:
=== Development Notes (old) ===
 * [wiki:DevNotes Development Notes Page]		  A. '''Making a JSOC/DRMS Module'''
  * [wiki:DrmsModule DRMS Module] -- DRMS Module Structure and Overview
  * [wiki:DrmsApi DRMS API] -- DRMS Data Types and Structures and API
  * [wiki:DrmsMakeModule DRMS Module Compilation] -- Running 'make' for modules
Line 108: Line 130:
=== Database Administration ===
 * [wiki:PgDBAdmin PostgreSQL]
 
=== JSOC Backup and Restore ===
 * [wiki:JSOC Backup/Restore Notes Page]		  A. '''Making a JSOC/DRMS Library'''
  * [wiki:JsocLibrary JSOC LIbrary] -- Creating and using a JSOC library
Line 114: Line 133:
=== SUM API ===
 * [http://sun.stanford.edu/web.hmi/development/SU_Development_Plan/SUM_API.html API web page]		  A. '''Notes on JSOC Makefile'''
  * [wiki:JsocMakefileBackground Background/Reference]
  * [wiki:JsocMakefileAdd How to add a sub directory]

 A. '''Web Exports via AJAX API'''
  * [wiki:AjaxJsocConnect AJAX Style Web Access for JSOC] used by [http://jsoc.stanford.edu/ajax/lookdata.html lookdata.html]
Line 119: Line 142:
=== DSDS-Data Access from JSOC ===
 * [http://hmi.stanford.edu/development/JSOC_Documents/JSOC_DSDS_Interface_Specification.pdf Interface Specification (.pdf)]		  A. '''DSDS-Data Access from JSOC'''
  * [http://hmi.stanford.edu/development/JSOC_Documents/JSOC_DSDS_Interface_Specification.pdf Interface Specification (.pdf)]
  . -- Details for builders of modules that will access MDI data directly from DRMS code.
Line 122: Line 146:
=== Exports ===
 * [wiki:AjaxJsocConnect AJAX Style Web Access for JSOC]

=== Remote DRMS/SUMS - netDRMS ===
 * See Rick Bogart
 ( who might not remember that he has a collection of web pages on NetDRMS at http://jsoc.stanford.edu/netdrms/ )		  A. '''Remote DRMS/SUMS - netDRMS'''
 * See Rick Bogart and the collection of web pages on NetDRMS at http://jsoc.stanford.edu/netdrms/
Line 131: Line 151:
== Running Datacapture and lev0 Pipeline during SDI I&T ==
 * [http://jsoc.stanford.edu/production/whattodolev0.txt][[BR]]		  A. JSOC Administration
 '''[:JsocAdministration: JSOC Administration]''' -- Information for management and installation of DRMS/SUMS
Line 134: Line 154:
== Convert the Spare dcs2 machine to an active AIA (dcs0) or HMI (dcs1) ==
 * [http://jsoc.stanford.edu/production/dcs2_convert_to_0_or_1.txt][[BR]]		  A. '''Running Datacapture and lev0 Pipeline during SDI I&T'''
 * [http://jsoc.stanford.edu/production/whattodolev0.txt][[BR]]

Welcome to the Wiki for JSOC software development

TableOfContents

For general information including the basic setup to get started see the [wiki:JsocUsersGuide Users Guide].

News

DRMS Data Series

  1. DRMS Overview

  2. Names

  3. JSD Format

    • [wiki:Jsd .jsd -- JSOC Series Definition Files]

  4. JSOC Sessions, Pipelines, and Modules (Oh my!)

    JSOC programs that use DRMS to operate on DataSeries are called "modules". Modules are run in "sessions". HMI and AIA major processing tasks are accomplished in "pipelines" consisting of one or more sessions. Pipelines are started by "PUI" (Pipeline User Interface) usually by the JSOC production team. Pipelines may also be initiated by users requesting [wiki:DataSet DataSets] via the web or by team members running locally or remotely. A DataSet is a collection of records selected by a query. In essence a dataset name is simply the query that describes it.

  5. A DRMS Session is the basic unit of computing that interracts with DRMS and SUMS. At the start of a session the user connects to the DRMS database. During the session the user runs one or more modules which read or create

    [wiki:DataRecord DataRecords] in DataSeries. Access to the actual data stored in SUMS is accomplished within a module via the DRMS API. At the end of a session, SUMS is notified to save any new records online and/or on tape, or to delete records marked temporary to the session.

  6. Actually using the JSOC DRMS requires running a program or module. By "program" we mean a normal shell command and by "module" we mean a program built to run within a DRMS session and communication to a drms_server. There are four types of programs/modules:

    • Modules - Most programs that do the work of the user of JSOC are what we call "modules". On the outside modules look like programs. They must run in a DRMS session. If they are built with the normal jsoc_main program they will use an existing session if they are run from a Session Provider or will start their own use-once session if they are called stand-alone from the shell.

    • Utility programs like [wiki:DrmsCreateSeriesCmd create_series] and [wiki:DrmsDescribeSeriesCmd describe_series] which are usually used to manage the existence of dataseries, not to use dataseries. These programs talk directly to the database.

    • Session Providers like [wiki:DrmsRunCmd drms_run] or later the [wiki:JsocPui Pipeline User Interface] start DRMS sessions and execute a script file. They can also be used to execute a single instance of a module.

    • [wiki:DrmsServerCmd drms_server] which connects connects to the database and serves sessions. Most users will not need to start drms_server explicitly.

    The benefit of running programs as "modules" will hopefully become apparent when we start running complex pipelines using hundreds of processors.

General Development Information - Using Modules

  1. DRMS Man Pages All the JSOC "man" pages are now maintained with [http://www.stack.nl/~dimitri/doxygen/index.html doxygen], a semi-automatic documentation tool. They are available in html form via:

    All programs and functions have entries in the doxygen generated pages.

  2. Limits

  3. There are limits ...
    • memory limits on number of records in the cache (512Meg / (2.5*record size) ). While this may seem like a lot, for datasets with a lot of keywords (e.g. mdi_vw_V_06h) it can be a real limit to the number of records that can be open at a time. For the vw_V example it means that DRMS_QUERY_MEM should be set to at lest 2500 (yes 2.5 gig) to open 100 days of one-minute data. Modules expecting to need tens of thousands of records opened should arrange to do the work in blocks with drms_close_records used to empty the record cache to free memory.
    • length of names of series, keywords, etc. (64 chars)
    • length of comma sep list of prime key names (1024 chars)
    • length of descriptions of series, keywords, links, and segments (254 chars)
    • length of string values of keywords (dont know)
    • number of keywords in a record (dont know)
    • number of records in a series (no fixed limit)
    • length of segment filename (255 chars)
    • length of path (511 chars)

  4. Log Files - Processing meta-data

  5. There are log files. Stdout and Stderr are captured in files as well as shown during processing (depending on module and -v flag). These are all put into a SUMS directory and indexed in DRMS by session ID. The session ID is stored in each record so the log files can be retrieved if/when needed. Unless otherwise specified, the default retention time for log files is 10 days. The log files are archived if any one of the SUs in the current session is to be archived.
    • drms_server logs --
    • The default is no logging. When the logging option is turned on (-L), stdout and stderr are redirected to files in SU directory.
    • module logs --
    • The default is no logging. When the logging option is turned on (-L), stdout and stderr are tee-ed to files in SU directory.

Software Development - Building Modules

  1. DRMS Applications Programming Interface] Descriptions of the API developed for the underlying JSOC database, and how to code with it.

    • [:DrmsApi:DRMS API summary] - old

    • [http://jsoc.stanford.edu/doxygen_html/modules.html DRMS API Manual] -- More detailed information for users and developers of specific JSOC software programs or interfaces using the Doxygen tools. Please use the search engine there to find documentations for any API functions.

  2. SUMS API

  3. Developers' Weekly Progress Reports

  4. JSOC Software Tree

  5. Making a JSOC/DRMS Module

    • [wiki:DrmsModule DRMS Module] -- DRMS Module Structure and Overview

    • [wiki:DrmsApi DRMS API] -- DRMS Data Types and Structures and API

    • [wiki:DrmsMakeModule DRMS Module Compilation] -- Running 'make' for modules

  6. Making a JSOC/DRMS Library

    • [wiki:JsocLibrary JSOC LIbrary] -- Creating and using a JSOC library

  7. Notes on JSOC Makefile

  8. Web Exports via AJAX API

JSOC Development Projects

  1. DSDS-Data Access from JSOC

  2. Remote DRMS/SUMS - netDRMS

JSOC Operator's Guide

  1. JSOC Administration

    [:JsocAdministration: JSOC Administration] -- Information for management and installation of DRMS/SUMS

  2. Running Datacapture and lev0 Pipeline during SDI I&T

JsocWiki: JsocDevelopersGuide (last edited 2021-11-16 01:34:32 by ArtAmezcua)