JSOC Series Definition Files

The preferred way to create and manage JSOC data series is via a ".jsd" file. These files are read by the "create_series" program to make the required database tables.

At present the create_series program provides only terse messages when it can not parse a .jsd file so the format here should be followed closely. In the format rules here items in angle brackets "<>" are words, strings, or numbers to be provided by the series creator. Words are user provided single word parameters, then can not contain blanks. They may be given with or without quotation marks. Strings may also be given with or without quotation marks but must be enclosed in quote marks if they contain blanks. A string must contain at least one character so an empty comment, for ecample, must be at least one blank character. In order to differentiate between words and strings, strings begin with "str". Choices in curly brackets and delimited by vertical bars are exact words that must be supplied. Choices are shown in bold (where I can figure out how). Any line beginning with a "#" is a comment and is not examined by create_series. The tokens (words, strings_, numbers, must be comma "," separated.

Names for series, keywords, links, and segments are implemented as PostgreSQL column or table names and are not allowed to collide with Postgres reserved names.

The .jsd file consists of four sections containing definitions of global information, keywords, links, and data segments. Only the global information section is required although a series with only global information has limited use.

Global series information

This section contains global information about the JSOC data series, that applies to all records and storage units belonging to the series. It is the information kept in the "drms_series" table. The .jsd syntax takes the form: 

 Seriesname:     <str series name>
 Description:    <str description>
 Author:         <str author name>
 Unitsize:       <storage unit size>
 Retention:      <default retention time>
 Archive:        <archive flag>
 Tapegroup:      <tape group>
 Index:          <primary keyword list>

where:

Keywords

The keyword section contains declarations of all keywords present in records belonging to the series. Each keyword description consist of "Keyword:" followed by a comma-separated list of attributes describing the keyword. Space and tab characters are ignored. The attributes list must be contained in a single line. [To remove this restriction, must modify the function getnextline() in src/base/libdrms/drms_parser.c] The lists of keywords allowed for each series are maintained in the drms_keyword table. A keyword can be simple or a link. A simple keyword can be fixed or variable and will contain an explicit value for the entire series (fixed) or for each record (variable). A link keyword describes a pointer to a keyword in some other series. 

Keyword: <name>, <datatype>, {constant | variable}, {record | segment}, <default value>, <format>, <unit>, <str description>

where:

For example, to declare a keyword containing a floating point value describing velocity associated with a data record, the keyword section could contain a line like 

Keyword: v, float, variable, record, 0.0f, %12g, m/s, "velocity"
 - or -
Keyword: "v", float, variable, record, "0.0f", "%12g", "m/s", "velocity"
 - or -
Keyword: Velocity,float,variable,record,NaN,"%12g m/s","meters per second" ,"velocity [m/s]"

An example Time type keyword might be like 

Keyword: t_obs, time, variable, record, 1993.01.01, TAI, "TAI", "Time of observation"

A link keyword has <datatype> equal to link and takes the form the form: 

Keyword: <name>, link, <linkname>, <target keyword>, <str description>

Keyword: Tilt, link, HMI_ORBIT, P_Angle, "Link to current best p_angle"

Links are pointers to records in (usually) other series. Links make it possible to effectively merge elements from other data records eliminating the necessity to make redundant copies. Links can also capture dependencies between data records such as processing history. For example a data record can contain links to the data records that were used in creating it, such as a dopplergram data record pointing to the filtergrams from which is was created. Links link records. A keyword link points to a particular keyword in a link record. A segment link points to a particular target segment in a link record. Links come in two varieties, static links and dynamic links. 

Link: <name>, <target series>, <link_type>, <str description>

where:

Data Segments

The data segment section describes the data which is associated with records in the series being defined in the JSD but is stored in a file in SUMS rather than in the DRMS database tables. A segment may be direct or linked. A direct segment descriptor specifies the storage scope, protocol, and associated information such as data types, array sizes, etc., depending on the protocol used. A linked segment descriptor specifies the name of a record link descriptor and the name of a target segment held in the target record. 

Data: <name>, <scope>, <datatype>, <naxis>, <axis dims>, <unit>, <protocol>, {<protocol special>,} <str comment>

Data: <name>, <scope>, <datatype>, 0, <unit>, <protocol>, <comment>
Data: <name>, <scope>, <datatype>, <naxis>, <axis dims>, <unit>, <protocol>, <comment>
Data: <name>, <scope>, <datatype>, <naxis>, <axis dims>, <unit>, tas, <tile sizes>, <comment>

where:

Similar to keyword links, the linked segment descriptor contains only the information necessary to identify the target segment. At present the data type, and units will be taken from the target segment. While the naxis and dimensions are specified in the jsd they are also ignored at present. The protocol of course must be specified only in the target. The form is as follows:

The number of fields required depends on the value of <naxis> so the following forms are valid 

Data: <name>, link, <linkname>, <target segment>, 0, <comment>
Data: <name>, link, <linkname>, <target segment>, <naxis>, <axis dims>, <str comment>

JSD Grammar

The complete grammar that describes the set of acceptable strings follows. Keep in mind that the only required section is Series Information. The other sections are optional.

Each symbol of the grammar is enclosed in angled brackets ("<" and ">") and its definition lies to the right of the '='. Literal strings are enclosed in single quotes ("'"). Text enclosed in square brackets ("[" and "]") represents any single character from the set of characters specified in the brackets. A '-' within square brackets denotes a range of characters. For example, [A-Za-z] denotes any one of the 26 upper-case or 26 lower-case letters. S1 | S2 denotes alternation between the symbols S1 and S2 (S1 and S2 may be literals or symbols defined in a different rule). For example, 'Sanjaya' | 'Melinda' means either the string 'Sanjaya' or the string 'Melinda' satisfies the rule containing the alternation. Finally, curly brackets ("{" and "}") denote optional elements. 

### Series Information ###

<SeriesName>
        = 'Seriesname:' <ascii string>

<SeriesDescription>
        = 'Description:' <ascii string>

<Owner>
        = 'Owner:' <ascii string>

<Author>
        = 'Author:' <ascii string>

<ArchiveFlag>
        = 'Archive:' '0' | '1'

<UnitSize>
        = 'Unitsize:' <string representing unsigned 32-bit integer [1, INT_MAX]>

<TapeGroup>
        = 'Tapegroup:' <string representing unsigned 32-bit integer [0, INT_MAX]>

<Retention>
        = 'Retention:' <string representing unsigned 32-bit integer [0, INT_MAX]>

<PrimaryIndex>
        = 'Index:' <KeywordList>

<KeywordList>
        = { <KeywordName> ',' } <KeywordName>

<KeywordName>
        = <Name>

<Name>
        = [A-Za-z] { <NameEnd> }

<NameEnd>
        = [A-Za-z0-9_] { <NameEnd> }



*** Keywords ***

<Keyword>
        = 'Keyword:' <KeyName> ',' <TypeAndFields>

<KeyName>
        = <Name>

<Name>
        = [A-Za-z] { <NameEnd> }

<NameEnd>
        = [A-Za-z0-9_] { <NameEnd> }

<TypeAndFields>
        = 'link,' <LinkName> ',' <TargetKey> ',' <Description>
        | 'char,' <RecordScope> ',' <SegmentScope> ',' <DefChar> ',' 
          <Format> ',' <Unit> ',' <Description>
        | 'short,' <RecordScope> ',' <SegmentScope> ',' <DefShort> ','
          < Format> ','< Unit> ',' <Description>
        | 'int,' <RecordScope> ',' <SegmentScope> ',' <DefInt> ','
          < Format> ',' <Unit> ',' <Description>
        | 'longlong,' <RecordScope> ',' <SegmentScope> ',' <DefLongLong> ','    
           <Format> ',' <Unit> ',' <Description>
        | 'float,' <RecordScope> ',' <SegmentScope> ',' <DefFloat> ','
           <Format> ',' <Unit> ',' <Description>
        | 'double,' <RecordScope> ',' <SegmentScope> ',' <DefDouble> ',
          <Format> ',' <Unit> ',' <Description>
        | 'time,' <RecordScope> ',' <SegmentScope> ',' <DefTime> ','
          <Format> ',' <Unit> ',' <Description>
        | 'string,' <RecordScope> ',' <SegmentScope> ',' <DefString> ','
          <Format> ',' <Unit> ',' <Description>

<LinkName>
        = <Name>

<TargetKey>
        = <KeyName>

<RecordScope>
        = 'constant' |'variable'

<SegmentScope>
        = 'segment' | 'record'

<DefChar>
        = <string representing signed 8-bit integer [SCHAR_MIN, SCHAR_MAX]>

<DefShort>
        = <string representing signed 16-bit integer [SHRT_MIN, SHRT_MAX]>

<DefInt>
        = <string representing signed 32-bit integer [INT_MIN, INT_MAX]>

<DefLongLong>
        = <string representing signed 64-bit integer [LLONG_MIN, LLONG_MAX]>

<DefFloat>
        = <string representing single-precision, floating-point number>
          NOTE: numbers larger in magnitude than what a float can represent
                will become infinite.

<DefDouble>
        = <string representing double-precision, floating-point number>
          NOTE: numbers larger in magnitude than what a double can represent
                will become infinite.
<DefTime>
        = <string representing time - see JSOC Time Format Notes>

<DefString>
        = <ascii string no longer than DRMS_MAXPATHLEN - 1 characters, MUST HAVE LENGTH >= 1>

<Format>
        = <printf conversion specification, e.g., %d, %c>

<Unit>
        = <ascii string>

<Description>
        = <ascii string>



### Data Segments ###

<DataSegment>
        = 'Data:' <SegmentName> ',' <TypeAndFields>

<SegmentName>
        = <ascii string>

<TypeAndFields>
        = 'link,' <LinkName> ',' <TargetSeg> ',' <Naxis> ',' <Axes> ',' <Description>
        | <Scope> ',' <Type> ',' <Naxis> ',' <Axes> ','
          <Unit> ',' <Protocol> ',' <Description>

<LinkName>
        = <Name>

<Name>
        = [A-Za-z] { <NameEnd> }

<NameEnd>
        = [A-Za-z0-9_] { <NameEnd> }

<TargetSeg>
        = <SegmentName>

<Scope>
        = 'constant' | 'variable' | 'vardim'

<Type>
        = 'char' | 'short' | 'int' | 'longlong' | 'float' | 'double' | 
          'time' | 'string'

<Naxis>
        = <string representing an unsigned 32-bit integer [1, INT_MAX]>

<Axes>
        = <AxisDim1> ',' <AxisDim2> ',' ... ',' <AxisDimN>
          where N = the integer represented by <Naxis>

<AxisDim>
        = <string representing an unsigned 32-bit integer [1, INT_MAX]>

<Unit>
        = <ascii string>

<Protocol>
        = 'bin' | 'zip' | 'fits' | 'fitz' | 'msi' | 'tas' , <TasProtocol> | 'generic'

<TasProtocol>
        = <TileSize1> ',' <TileSize2> ',' ... ',' <TileSizeN>
          where N = the integer represented by <Naxis>

<TileSize>
        = <string representing an unsigned 32-bit integer [1, INT_MAX]>

<Description>
        = <ascii string>



### Links ###

<Link>
        = 'Link:' <LinkName> ',' <TargetSeries> ',' <Type> ',' <Description>

<LinkName>
        = <Name>

<Name>
        = [A-Za-z] { <NameEnd> }

<NameEnd>
        = [A-Za-z0-9_] { <NameEnd> }

<TargetSeries>
        = <ascii string>

<Type>
        = 'static' | 'dynamic'

<Description>
        = <ascii string>

Example .jsd File

P% cat testseries5.jsd
#======================= Global series information  ============================

Seriesname:     su_rmunk.testseries5
Author:         "Rasmus Munk Larsen"
Owner:          rmunk
Unitsize:       10
Archive:        1
Retention:      1
Tapegroup:      1
Index:  
Description:    "Another simple testclass"

#============================ Keywords =========================================
# Format:
#   Keyword: <name>, link, <linkname>, <target keyword name>, <comment>
# or
#
#   Keyword: <name>, <datatype>, {constant | variable}, {record | segment}, 
#            <default value>, <format>, <unit>, <comment>
       
Keyword: counter,    int, variable, record,  1,  %d, unit3, "blah"
Keyword: bzero,    double, variable, segment, 0.0, %lf, none, "Zero point for pixel scaling"
Keyword: bscale,   double, variable, segment, 1.0, %lf, none, "Scale factor for pixel scaling"



#============================ Segments =========================================
#
# Format:
#
# Data: <name>, {constant | variable | vardim}, <type>,
#       <naxis>, <axis_0>, <axis_1>,...,<axis_{naxis-1}>, 
#       <unit>, <protocol>, <comment> 
Data: char_image,variable,char,2,1024,1024,m/s, tas, 128, 128,"blah"
Data: short_image,variable,short,2,1024,1024,m/s, tas, 128, 128,"blah"
Data: int_image,variable,int,2,1024,1024,m/s, tas, 128, 128,"blah"
Data: longlong_image,variable,longlong,2,1024,1024,m/s, tas, 128, 128,"blah"
Data: float_image,variable,float,2,1024,1024,m/s, tas, 128, 128,"blah"
Data: double_image,variable,double,2,1024,1024,m/s, tas, 128, 128, "blah"
P%