|
Size: 6433
Comment:
|
← Revision 28 as of 2013-05-01 04:35:25 ⇥
Size: 5906
Comment: converted to 1.6 markup
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 2: | Line 2: |
| [[TableOfContents]] | <<TableOfContents>> |
| Line 4: | Line 4: |
| This guide contains information of interest to all users of the JSOC software. Information for making new software is in the [:JsocDevelopersGuide:Developers Guide]. | This guide contains information of interest to all users of the JSOC software. Information for making new software is in the [[JsocDevelopersGuide|Developers Guide]]. |
| Line 6: | Line 6: |
| Hello Brian Yeah, some access control is generally considered a *good* thing. ---- /!\ '''Edit conflict - other version:''' ---- Yeah, some access control is generally considered a *good* thing. ---- /!\ '''Edit conflict - your version:''' ---- ---- /!\ '''End of edit conflict''' ---- |
|
| Line 21: | Line 9: |
| I.e. SDO HMI/AIA JSOC DRMS/SUMS online tools. | I.e. '''SDO HMI/AIA JSOC DRMS/SUMS''' online document tools. |
| Line 23: | Line 11: |
| If you ''only'' want to use some JSOC programs you will need to '''do''' the steps described in 'Initial Steps' and be familiar with the topics below down through the 'JSOC Man Pages (man 1)' sections below. Basic unix/linux familiarity is assumed. | If you ''only'' want to use some JSOC programs you will need to '''do''' the steps described in 'Initial Steps' and be familiar with the topics below down through the 'JSOC Man Pages (man 1)' sections below. Basic unix/linux familiarity is assumed. The following is intended for persons who will be working on JSOC and with its data regularly. If you have a user account at SDO, you may find you need to use JSOC programs such as "show_keys" and "create_series". To set up your user environment, please follow the instructions below. If you are a JSOC software developer, you may have some different needs from regular users, as noted below. |
| Line 26: | Line 14: |
| First read the following and decide which machines you plan to use for JSOC work. | These steps are to setup access to run or create programs in the JSOC-SDP environment. |
| Line 28: | Line 16: |
| * '''ALL USERS:''' [:SystemEnvironment:System Environment] -- OS constraints, machine requirements, etc. Do the following three steps in this order: |
* '''Setup Your JSOC Account''' [[JsocSetup|JSOC New User Setup]] -- Initial steps for a new user |
| Line 31: | Line 18: |
| * '''ALL USERS:''' [:DrmsEnvironmentVars:DRMS Environment Variables] -- Set the DRMS required shell environment variables * '''DEVELOPERS ONLY:''' [:CvsInit:Make Your CVS Work Area] -- Set up your work area for making new code * '''ALL USERS:''' [:DrmsPassword:DRMS Password] -- Set up your DRMS password file * '''ALL USERS:''' [:DrmsPermission:DRMS Permission] -- Things you should know about permissions in DRMS Now you are all setup and ready to go. The following contains useful information: |
OK, now that you are a JSOC user or developer, try these [[SampleRunsNewUsers|Sample Commands]]. |
| Line 37: | Line 20: |
| * '''DEVELOPERS INFO:''' [:CvsSandbox:Working in Your CVS "Sandbox"] -- Concurrent Versions System '''(CVS)''' for software configuration management OK, now try these [:SampleRunsNewUsers:Sample Commands]. |
= General Information = |
| Line 41: | Line 23: |
| * [:DataSeries:Overview] -- What is a Data Series and how do DRMS and SUMS work (and what are they) * [http://hmi.stanford.edu/development/JSOC_Documents/DRMSreport_html/index.html DRMS (html)] -- DRMS Overview[http://hmi.stanford.edu/development/JSOC_Documents/DRMS_V10.pdf (as pdf)] - old doc, contents being verified and included in these pages. * [:DrmsNames:DRMS Names] -- DRMS Dataset Names and Queries * [:DrmsSeriesNames:DRMS Series Names] -- Data Series Reserved Names * Data Storage, Archive, On-Line Retention, EtC. * [:SumsDataModel:SUMS Data Storage] -- SUMS Basic Data Storage Concepts * [:SumsArchiveTimes:SUMS Archive and Retention Table] -- Meaning and implications of the values * [:Jsd:.jsd] -- JSOC Series Definition Files |
* [[DataAccessPolicy]] -- SDO Data Use Policy * [[DataSeries|Overview]] -- What is a Data Series and how do DRMS and SUMS work (and what are they) * [[AllAboutJsocNames|Names]] -- DRMS Dataset Names and Queries * [[SumsDataModel|SUMS Data Storage]] -- SUMS Basic Data Storage Concepts == JSOC Programs == * JSOC Sessions, Pipelines, and Modules ("Oh my!") * JSOC programs that use DRMS to operate on Data Series are called "modules". Modules are run in "sessions". HMI and AIA major processing tasks are accomplished in "pipelines" consisting of one or more sessions. Pipelines are started by "PUI" (Pipeline User Interface) usually by the JSOC production team. Pipelines may also be initiated by users requesting [[DataSet|DataSets]] via the web or by team members running locally or remotely. A DataSet is a collection of records selected by a query. In essence a dataset name is simply the query that describes it. * All DRMS & SUMS connections and transactions occur inside a session. A DRMS Session is the basic unit of computing for interacting with DRMS and SUMS. At the start of a session the user connects to the DRMS database. During the session the user runs one or more modules which read or create [[DataRecord|DataRecords]] in one or more DataSeries. Access to the actual data stored in SUMS is accomplished within a module via the DRMS API. At the end of a session, SUMS is notified to save any new records online and/or on tape, or to delete records marked temporary to the session. * Actually using the JSOC DRMS requires running a program or module. By "program" we mean a normal shell command and by "module" we mean a program built to run within a DRMS session and communication to a drms_server. There are four types of programs/modules: * '''Modules''' - Most programs that do the work of a JSOC user are what we call "modules". On the outside modules look like programs. They must run in a DRMS session. If they are built with the normal jsoc_main program they will use an existing session if they are run from a Session Provider. Or modules will start their own use-once session if they are called stand-alone from the shell. * '''Utility programs''' like [[http://jsoc.stanford.edu/doxygen_html/group__create__series.html|create_series]] and [[http://jsoc.stanford.edu/doxygen_html/group__describe__series.html|describe_series]] which are usually used to manage the existence of DataSeries; they are not intended to pick up and use DataSeries. These programs talk directly to the database. * '''Session Providers''' like [[DrmsRunCmd|drms_run]] or later the [[JsocPui|Pipeline User Interface]] start DRMS sessions and execute a script file. They can also be used to execute a single instance of a module. * '''[[http://jsoc.stanford.edu/doxygen_html/group__drms__server.html|drms_server]]''' which connects to the database and serves sessions. Most users will not need to start drms_server explicitly. * The benefit of running programs as "modules" will hopefully become apparent when we start running complex pipelines using hundreds of processors. == DRMS Man Pages == * [[http://jsoc.stanford.edu/doxygen_html|Drms Program/Module Man Pages]]<<BR>> . All programs and modules should have unix-style man pages at [[http://jsoc.stanford.edu/doxygen_html/group__a__programs.html|man1]]. Note that all modules built with jsoc_main share a basic set of flags and command line keywords. See [[http://jsoc.stanford.edu/doxygen_html/group__common__main.html|module.1]]. For specific programming information about a module, such as input and output parameters, formats, etc, the man page is expected to be the primary user resource. |
| Line 50: | Line 42: |
| * [:DrmsCreateSeriesCmd:create_series] -- Create entries and tables for a new series in the DRMS database * [:DrmsDescribeSeriesCmd:describe_series] -- Prints a verbose description of the named series and its current highest record number on stdout. * [:DrmsDeleteSeriesCmd:delete_series] -- Removes a series and all its associated entries from DRMS. == JSOC Sessions, Pipelines, and Modules ("''Oh my!''") == JSOC programs that use DRMS to operate on DataSeries are called "modules". Modules are run in "sessions". HMI and AIA major processing tasks are accomplished in "pipelines" consisting of one or more sessions. Pipelines are started by "PUI" (Pipeline User Interface) usually by the JSOC production team. Pipelines may also be initiated by users requesting [:DataSet:DataSets] via the web or by team members running locally or remotely. A DataSet is a collection of records selected by a query. In essence a dataset name is simply the query that describes it. |
* [[http://jsoc.stanford.edu/doxygen_html/group__create__series.html|create_series]] -- Create entries and tables for a new series in the DRMS database * [[http://jsoc.stanford.edu/doxygen_html/group__describe__series.html|describe_series]] -- Prints a verbose description of the named series and its current highest record number on stdout. * [[http://jsoc.stanford.edu/doxygen_html/group__delete__series.html|delete_series]] -- Removes a series and all its associated entries from DRMS. |
| Line 56: | Line 46: |
| A DRMS Session is the basic unit of computing that interracts with DRMS and SUMS. At the start of a session the user connects to the DRMS database. During the session the user runs one or more modules which read or create [:DataRecord:DataRecords] in DataSeries. Access to the actual data stored in SUMS is accomplished within a module via the DRMS API. At the end of a session, SUMS is notified to save any new records online and/or on tape, or to delete records marked temporary to the session. Actually using the JSOC DRMS requires running a program or module. By "program" we mean a normal shell command and by "module" we mean a program built to run within a DRMS session and communication to a drms_server. There are four types of programs/modules: * '''Modules''' - Most programs that do the work of the user of JSOC are what we call "modules". On the outside modules look like programs. They must run in a DRMS session. If they are built with the normal jsoc_main program they will use an existing session if they are run from a Session Provider or will start their own use-once session if they are called stand-alone from the shell. * '''Utility programs''' like [:DrmsCreateSeriesCmd:create_series] and [:DrmsDescribeSeriesCmd:describe_series] which are usually used to manage the existence of dataseries, not to use dataseries. These programs talk directly to the database. * '''Session Providers''' like [:DrmsRunCmd:drms_run] or later the [:JsocPui:Pipeline User Interface] start DRMS sessions and execute a script file. They can also be used to execute a single instance of a module. * '''[:DrmsServerCmd:drms_server]''' which connects connects to the database and serves sessions. Most users will not need to start drms_server explicitly. The benefit of running programs as "modules" will hopefully become apparent when we start running complex pipelines using hundreds of processors. = General Information = == JSOC File Structure == * [:FileStruct:Tree Diagram] == DRMS Man Pages == * [http://jsoc.stanford.edu/man/index.html#man1 Drms Program/Module Man Pages, man1][[BR]] . All programs and modules should have unix-style man pages at [http://jsoc.stanford.edu/man/index.html#man1 man1] but Some do not yet: [:DrmsServerCmd:drms_server], [:DrmsQueryCmd:drms_query], [:DrmsCreateSeriesCmd:create_series], [:DrmsDescribeSeriesCmd:describe_series], [:DrmsDeleteSeriesCmd:delete_series] Note that all modules built with jsoc_main share a basic set of flags and command line keywords. See module.1 in man1. |
== JSOC Data Access == * [[JsocDataAccess| How to find and use JSOC data]] |
Welcome to the JSOC Users Guide
Contents
This guide contains information of interest to all users of the JSOC software. Information for making new software is in the Developers Guide.
Getting Started
Welcome to the Solar Dynamics Observatory Helioseismic and Magnetic Imager and Atmospheric Imaging Array Joint Science Operations Center Data Record Management System and Storage Unit Management System online document tools.
I.e. SDO HMI/AIA JSOC DRMS/SUMS online document tools.
If you only want to use some JSOC programs you will need to do the steps described in 'Initial Steps' and be familiar with the topics below down through the 'JSOC Man Pages (man 1)' sections below. Basic unix/linux familiarity is assumed. The following is intended for persons who will be working on JSOC and with its data regularly. If you have a user account at SDO, you may find you need to use JSOC programs such as "show_keys" and "create_series". To set up your user environment, please follow the instructions below. If you are a JSOC software developer, you may have some different needs from regular users, as noted below.
Initial Steps
These steps are to setup access to run or create programs in the JSOC-SDP environment.
Setup Your JSOC Account JSOC New User Setup -- Initial steps for a new user
OK, now that you are a JSOC user or developer, try these Sample Commands.
General Information
DRMS Data Series
DataAccessPolicy -- SDO Data Use Policy
Overview -- What is a Data Series and how do DRMS and SUMS work (and what are they)
Names -- DRMS Dataset Names and Queries
SUMS Data Storage -- SUMS Basic Data Storage Concepts
JSOC Programs
- JSOC Sessions, Pipelines, and Modules ("Oh my!")
JSOC programs that use DRMS to operate on Data Series are called "modules". Modules are run in "sessions". HMI and AIA major processing tasks are accomplished in "pipelines" consisting of one or more sessions. Pipelines are started by "PUI" (Pipeline User Interface) usually by the JSOC production team. Pipelines may also be initiated by users requesting DataSets via the web or by team members running locally or remotely. A DataSet is a collection of records selected by a query. In essence a dataset name is simply the query that describes it.
All DRMS & SUMS connections and transactions occur inside a session. A DRMS Session is the basic unit of computing for interacting with DRMS and SUMS. At the start of a session the user connects to the DRMS database. During the session the user runs one or more modules which read or create DataRecords in one or more DataSeries. Access to the actual data stored in SUMS is accomplished within a module via the DRMS API. At the end of a session, SUMS is notified to save any new records online and/or on tape, or to delete records marked temporary to the session.
- Actually using the JSOC DRMS requires running a program or module. By "program" we mean a normal shell command and by "module" we mean a program built to run within a DRMS session and communication to a drms_server. There are four types of programs/modules:
Modules - Most programs that do the work of a JSOC user are what we call "modules". On the outside modules look like programs. They must run in a DRMS session. If they are built with the normal jsoc_main program they will use an existing session if they are run from a Session Provider. Or modules will start their own use-once session if they are called stand-alone from the shell.
Utility programs like create_series and describe_series which are usually used to manage the existence of DataSeries; they are not intended to pick up and use DataSeries. These programs talk directly to the database.
Session Providers like drms_run or later the Pipeline User Interface start DRMS sessions and execute a script file. They can also be used to execute a single instance of a module.
drms_server which connects to the database and serves sessions. Most users will not need to start drms_server explicitly.
- The benefit of running programs as "modules" will hopefully become apparent when we start running complex pipelines using hundreds of processors.
DRMS Man Pages
All programs and modules should have unix-style man pages at man1. Note that all modules built with jsoc_main share a basic set of flags and command line keywords. See module.1. For specific programming information about a module, such as input and output parameters, formats, etc, the man page is expected to be the primary user resource.
- Data series utilities to be run from a user's shell
create_series -- Create entries and tables for a new series in the DRMS database
describe_series -- Prints a verbose description of the named series and its current highest record number on stdout.
delete_series -- Removes a series and all its associated entries from DRMS.