Differences between revisions 2 and 3
⇤ ← Revision 2 as of 2007-10-24 15:36:59 →
Size: 8884
Editor: solpc2
Comment:
← Revision 3 as of 2007-10-25 09:28:31 → ⇥
Size: 8964
Editor: solpc2
Comment:
Deletions are marked like this. Additions are marked like this.
Line 91: Line 91:
 * [wiki:FileStruct Software File Tree] -- Organization of files and cvs tree.

Welcome to Trac for JSOC software development

TableOfContents

For general information including the basic setup to get started see the [wiki:JsocUsersGuide Users Guide].

News

DRMS Data Series

named series and its current highest record number on stdout.

  • [wiki:DrmsDeleteSeriesCmd delete_series] -- Removes a series and all its associated entries from DRMS.

JSOC Sessions, Pipelines, and Modules ("''Oh my!''")

JSOC programs that use DRMS to operate on DataSeries are called "modules". Modules are run in "sessions". HMI and AIA major processing tasks are accomplished in "pipelines" consisting of one or more sessions. Pipelines are started by "PUI" (Pipeline User Interface) usually by the JSOC production team. Pipelines may also be initiated by users requesting [wiki:DataSet DataSets] via the web or by team members running locally or remotely. A DataSet is a collection of records selected by a query. In essence a dataset name is simply the query that describes it.

A DRMS Session is the basic unit of computing that interracts with DRMS and SUMS. At the start of a session the user connects to the DRMS database. During the session the user runs one or more modules which read or create [wiki:DataRecord DataRecords] in DataSeries. Access to the actual data stored in SUMS is accomplished within a module via the DRMS API. At the end of a session, SUMS is notified to save any new records online and/or on tape, or to delete records marked temporary to the session.

Actually using the JSOC DRMS requires running a program or module. By "program" we mean a normal shell command and by "module" we mean a program built to run within a DRMS session and communication to a drms_server. There are four types of programs/modules:

  • Modules - Most programs that do the work of the user of JSOC are what we call "modules". On the outside modules look like programs. They must run in a DRMS session. If they are built with the normal jsoc_main program they will use an existing session if they are run from a Session Provider or will start their own use-once session if they are called stand-alone from the shell.

  • Utility programs like [wiki:DrmsCreateSeriesCmd create_series] and [wiki:DrmsDescribeSeriesCmd describe_series] which are usually used to manage the existence of dataseries, not to use dataseries. These programs talk directly to the database.

  • Session Providers like [wiki:DrmsRunCmd drms_run] or later the [wiki:JsocPui Pipeline User Interface] start DRMS sessions and execute a script file. They can also be used to execute a single instance of a module.

  • [wiki:DrmsServerCmd drms_server] which connects connects to the database and serves sessions. Most users will not need to start drms_server explicitly.

The benefit of running programs as "modules" will hopefully become apparent when we start running complex pipelines using hundreds of processors.

[wiki:DRMSSetup Setting up Your Own DRMS]

Information for developers outside the JSOC who wish to construct an independent data archive that can work in cooperation with the JSOC and other archives (or completely independently)

General Information

DRMS Man Pages

Limits

There are limits ...

  • memory limits on number of records in the cache (512Meg / (2.5*record size) ). While this may seem like a lot, for datasets with a lot of keywords (e.g. mdi_vw_V_06h) it can be a real limit to the number of records that can be open at a time. For the vw_V example it means that DRMS_QUERY_MEM should be set to at lest 2500 (yes 2.5 gig) to open 100 days of one-minute data. Modules expecting to need tens of thousands of records opened should arrange to do the work in blocks with drms_close_records used to empty the record cache to free memory.
  • length of names of series, keywords, etc. (31 chars)
  • length of descriptions of series, keywords, links, and segments (254 chars)
  • length of string values of keywords (dont know)
  • number of keywords in a record (dont know)
  • number of records in a series (no fixed limit)
  • length of segment filename (255 chars)
  • length of path (511 chars)

Log Files - Processing meta-data

There are log files. Stdout and Stderr are captured in files as well as shown during processing (depending on module and -v flag). These are all put into a SUMS directory and indexed in DRMS by session ID. The session ID is stored in each record so the log files can be retrieved if/when needed. Unless otherwise specified, the default retention time for log files is the maximum retention time of all SUs processed in the current session. The log files are archived if any one of the SUs in the current session is to be archived.

  • drms_server logs --
    • The default is no logging. When the logging option is turned on (-L), stdout and stderr are redirected to files in SU directory.
  • module logs --
    • The default is no logging. When the logging option is turned on (-L), stdout and stderr are tee-ed to files in SU directory.

Software Development - Building Modules

JSOC Software Tree

Making a JSOC/DRMS Module

  • [wiki:DrmsModule DRMS Module] -- DRMS Module Structure and Overview

  • [wiki:DrmsApi DRMS API] -- DRMS Data Types and Structures and API

  • [wiki:DrmsMakeModule DRMS Module Compilation] -- Running 'make' for modules

Making a JSOC/DRMS Library

  • [wiki:JsocLibrary JSOC LIbrary] -- Creating and using a JSOC library

Notes on JSOC Makefile

Development Notes

[wiki:PgDBAdmin PostgreSQL Database Administration]

JSOC Backup and Restore

  • [wiki:JSOC Backup/Restore Notes Page]

SUM API

DSDS-Data Access from JSOC

JsocWiki: JsocDevelopersGuide (last edited 2021-11-16 01:34:32 by ArtAmezcua)