JSOC Series Definition Files

NEWS!

There have been a number of bug fixes regarding "show_info -j" and "create_series" since the last JSOC release (version 4.7). If those commands have been troubling you, please try upgrading to the head CVS file revisions.


Introduction

The preferred way to create and manage JSOC data series is via a ".jsd" file. These files are read by the "create_series" program to make the required database tables.

At present the create_series program provides only terse messages when it can not parse a .jsd file so the format here should be followed closely. In the format rules here items in angle brackets "<>" are words, strings, or numbers to be provided by the series creator. Words are user provided single word parameters, they can not contain blanks. They may be given with or without quotation marks. Strings may also be given with or without quotation marks but must be enclosed in quote marks if they contain blanks. A string must contain at least one character so an empty comment, for example, must be at least one blank character. In order to differentiate between words and strings, strings begin with "str". Choices in curly brackets and delimited by vertical bars are exact words that must be supplied. Choices are shown in bold (where I can figure out how). Any line beginning with a "#" is a comment and is not examined by create_series. The tokens ( words, strings, numbers, must be comma "," separated.

Names for series, keywords, links, and segments are implemented as PostgreSQL column or table names and are not allowed to collide with Postgres reserved names.

The .jsd file consists of four sections containing definitions of global information, keywords, links, and data segments. Only the global information section is required although a series with only global information has limited use.

Global series information

This section contains global information about the JSOC data series, that applies to all records and storage units belonging to the series. It is the information kept in the "drms_series" table. The .jsd syntax takes the form:

 Seriesname:       <str series name>
 Description:      <str description>
 Author:           <str author name>
 Unitsize:         <storage unit size>
 Retention:        <retention for new SUs>
 StagingRetention: <retention for retrieved SUs>
 Archive:          <archive flag>
 Tapegroup:        <tape group>
 PrimeKeys:        <primary keyword list>
 DBIndex:          <index keyword list>
 Version:          <version number>

where:

Keywords

The keyword section contains declarations of all keywords present in records belonging to the series. Each keyword description consist of "Keyword:" followed by a comma-separated list of attributes describing the keyword. Space and tab characters are ignored. The attributes list must be contained in a single line. [To remove this restriction, must modify the function getnextline() in src/base/libdrms/drms_parser.c] The lists of keywords allowed for each series are maintained in the drms_keyword table. A keyword can be simple or a link. A simple keyword can be fixed or variable and will contain an explicit value for the entire series (fixed) or for each record (variable). A link keyword describes a pointer to a keyword in some other series.

Keyword: <name>, <datatype>, <recordscope>, <segmentscope>, <default value>, <format>, <unit>, <str description>

where:

For example, to declare a keyword containing a floating point value describing velocity associated with a data record, the keyword section could contain a line like

Keyword: v, float, variable, record, 0.0, %12g, m/s, "velocity"
 - or -
Keyword: "v", float, variable, record, "0.0", "%12g", "m/s", "velocity"
 - or -
Keyword: Velocity,float,variable,record,NaN,"%12g m/s","meters per second" ,"velocity [m/s]"

An example Time type keyword might be like

Keyword: t_obs, time, variable, record, 1993.01.01_TAI, 0, TAI, "Time of observation"

An example slotted time type keyword and associated keywords might be like

Keyword: t_rec,       time, ts_eq, record, 1993.01.01_TAI, 0, TAI, "Time index of observation"
Keyword: t_rec_epoch, time, constant, record, 1993.01.01, 0, TAI, "Reference epoch for time index"
Keyword: t_rec_step,  double, constant, record, 60.0, %lf, "s", "Index increment time span"

A link keyword has <datatype> equal to link and takes the form the form:

Keyword: <name>, link, <linkname>, <target keyword>, <str description>

where:

Keyword: ROLL, link, HMI_ORBIT, P_Angle, "Link to current best p_angle"

Links are pointers to records in (usually) other series. Links make it possible to effectively merge elements from other data records eliminating the necessity to make redundant copies. Links can also capture dependencies between data records such as processing history. For example a data record can contain links to the data records that were used in creating it, such as a dopplergram data record pointing to the filtergrams from which is was created. Links link records. A keyword link points to a particular keyword in a link record. A segment link points to a particular target segment in a link record. Links come in two varieties, static links and dynamic links.

Link: <name>, <target series>, <link_type>, <str description>

where:

Data Segments

The data segment section describes the data which is associated with records in the series being defined in the JSD but is stored in a file in SUMS rather than in the DRMS database tables. A segment may be direct or linked. A direct segment descriptor specifies the storage scope, protocol, and associated information such as data types, array sizes, etc., depending on the protocol used. A linked segment descriptor specifies the name of a record link descriptor and the name of a target segment held in the target record.

direct data segments

Each direct data segment describes a chunk of data stored in a file in SUMS. Various external storage formats of the data segments are available, and are specified by the <protocol> field. Raw level data is often stored simply as files of unspecified generic type. Processed data is usually stored in an n-dimensional array of a simple type where the external storage protocol is not of interest to the user. The general form of a data segment description is

Data: <name>, <scope>, <datatype>, <naxis>, <axis dims>, <unit>, <protocol>, {<protocol special>,} <str comment>

Data: <name>, <scope>, <datatype>, 0, <unit>, <protocol>, <comment>
Data: <name>, <scope>, <int datatype>, <naxis>, <axis dims>, <unit>, fits, <tile-compression arguments>, <bzero>, <bscale>, <comment>
Data: <name>, <scope>, <int datatype>, <naxis>, <axis dims>, <unit>, fitz, <bzero>, <bscale>, <comment>
Data: <name>, <scope>, <int datatype>, <naxis>, <axis dims>, <unit>, bin, <bzero>, <bscale>, <comment>
Data: <name>, <scope>, <int datatype>, <naxis>, <axis dims>, <unit>, bin.gz, <bzero>, <bscale>, <comment>
Data: <name>, <scope>, <int datatype>, <naxis>, <axis dims>, <unit>, tas, <tile-compression arguments>, <bzero>, <bscale>, <comment>
Data: <name>, <scope>, <float datatype>, <naxis>, <axis dims>, <unit>, fits, <tile-compression arguments>, <comment>
Data: <name>, <scope>, <float datatype>, <naxis>, <axis dims>, <unit>, fitz, <comment>
Data: <name>, <scope>, <float datatype>, <naxis>, <axis dims>, <unit>, bin, <comment>
Data: <name>, <scope>, <float datatype>, <naxis>, <axis dims>, <unit>, bin.gz, <comment>
Data: <name>, <scope>, <float datatype>, <naxis>, <axis dims>, <unit>, tas, <tile-compression arguments>, <comment>

where:

linked segments

A linked segment is a pointer to a segment belonging to another record in another series. Similar to keyword links, the linked segment descriptor contains only the information necessary to identify the target segment. At present the data type, and units will be taken from the target segment. While the naxis and dimensions are specified in the jsd they are also ignored at present. The protocol of course must be specified only in the target. The form is as follows:

The number of fields required depends on the value of <naxis> so the following forms are valid

Data: <name>, link, <linkname>, <target segment>, 0, <str description>
Data: <name>, link, <linkname>, <target segment>, <naxis>, <axis dims>, <str description>

data slices

If the user intends to write slices of data to each image, instead of writing whole images, then the fields <naxis> and <axis dims> are used to specify the default dimensions of the final, overall image. The code that writes data slices uses these values to determine the image's final dimensions, but these dimensions can be overridden. To override the default values, the first time a slice is to be written the user provides a slice of data with the desired dimensions to a data-slice-writing library call. The appropriate values for <naxis> and <axis dims> depends on the value of the <scope> field as well. A scope of vardim still allows the user to create images whose dimensions vary across records, but to do so the user must override, for each image, the default dimensions present in the segment specification in the .jsd file.

Please see DataSlices for a detailed description about data slices work and how to create images by writing slices.

JSD Grammar

The complete grammar that describes the set of acceptable strings follows. Keep in mind that the only required section is Series Information. The other sections are optional. This grammar is applicable to the most recent jsd version, not older ones.

Each symbol of the grammar is enclosed in angled brackets ("<" and ">") and its definition lies to the right of the '='. Literal strings are enclosed in single quotes ("'"). Text enclosed in square brackets ("[: and :and ]") represents any single character from the set of characters specified in the brackets. A '-' within square brackets denotes a range of characters. For example, [A-Za-z] denotes any one of the 26 upper-case or 26 lower-case letters. S1 | S2 denotes alternation between the symbols S1 and S2 (S1 and S2 may be literals or symbols defined in a different rule). For example, 'Sanjaya' | 'Melinda' means either the string 'Sanjaya' or the string 'Melinda' satisfies the rule containing the alternation. Finally, curly brackets ("{" and "}") denote optional elements.

### Series Information ###
<SeriesName>
        = 'Seriesname:' <ascii string>
<SeriesDescription>
        = 'Description:' <ascii string>
<Owner>
        = 'Owner:' <ascii string>
<Author>
        = 'Author:' <ascii string>
<ArchiveFlag>
        = 'Archive:' '0' | '1'
<UnitSize>
        = 'Unitsize:' <string representing unsigned 32-bit integer [1, INT_MAX]>
<TapeGroup>
        = 'Tapegroup:' <string representing unsigned 32-bit integer [0, INT_MAX]>
<Retention>
        = 'Retention:' <string representing unsigned 16-bit integer [0, SHRT_MAX]>
<StagingRetention>
        = 'StagingRetention:' <string representing unsigned 16-bit integer [0, SHRT_MAX]>
<PrimaryIndex>
        = 'Index:' <KeywordList>
<KeywordList>
        = { <KeywordName> ',' } <KeywordName>
<KeywordName>
        = <Name>
<Name>
        = [A-Za-z] { <NameEnd> }
<NameEnd>
        = [A-Za-z0-9_] { <NameEnd> }
*** Keywords ***
<Keyword>
        = 'Keyword:' <KeyName> ',' <TypeAndFields>
<KeyName>
        = <Name>
<Name>
        = [A-Za-z] { <NameEnd> }
<NameEnd>
        = [A-Za-z0-9_] { <NameEnd> }
<TypeAndFields>
        = 'link,' <LinkName> ',' <TargetKey> ',' <Description>
        | 'char,' <RecordScope> ',' <SegmentScope> ',' <DefChar> ','
          <Format> ',' <Unit> ',' <Description>
        | 'short,' <RecordScope> ',' <SegmentScope> ',' <DefShort> ','
          < Format> ','< Unit> ',' <Description>
        | 'int,' <RecordScope> ',' <SegmentScope> ',' <DefInt> ','
          < Format> ',' <Unit> ',' <Description>
        | 'longlong,' <RecordScope> ',' <SegmentScope> ',' <DefLongLong> ','
           <Format> ',' <Unit> ',' <Description>
        | 'float,' <RecordScope> ',' <SegmentScope> ',' <DefFloat> ','
           <Format> ',' <Unit> ',' <Description>
        | 'double,' <RecordScope> ',' <SegmentScope> ',' <DefDouble> ',
          <Format> ',' <Unit> ',' <Description>
        | 'time,' <RecordScope> ',' <SegmentScope> ',' <DefTime> ','
          <Format> ',' <Unit> ',' <Description>
        | 'string,' <RecordScope> ',' <SegmentScope> ',' <DefString> ','
          <Format> ',' <Unit> ',' <Description>
<LinkName>
        = <Name>
<TargetKey>
        = <KeyName>
<RecordScope>
        = 'constant' |'variable'
<SegmentScope>
        = 'segment' | 'record'
<DefChar>
        = <string representing signed 8-bit integer [SCHAR_MIN, SCHAR_MAX]>
<DefShort>
        = <string representing signed 16-bit integer [SHRT_MIN, SHRT_MAX]>
<DefInt>
        = <string representing signed 32-bit integer [INT_MIN, INT_MAX]>
<DefLongLong>
        = <string representing signed 64-bit integer [LLONG_MIN, LLONG_MAX]>
<DefFloat>
        = <string representing single-precision, floating-point number>
          NOTE: numbers larger in magnitude than what a float can represent
                will become infinite.
<DefDouble>
        = <string representing double-precision, floating-point number>
          NOTE: numbers larger in magnitude than what a double can represent
                will become infinite.
<DefTime>
        = <string representing time - see JSOC Time Format Notes>
<DefString>
        = <ascii string no longer than DRMS_MAXPATHLEN - 1 characters, MUST HAVE LENGTH >= 1>
<Format>
        = <printf conversion specification, e.g., %d, %c>
<Unit>
        = <ascii string>
<Description>
        = <ascii string>
### Data Segments ###
<DataSegment>
        = 'Data:' <SegmentName> ',' <TypeAndFields>
<SegmentName>
        = <ascii string>
<TypeAndFields>
        = 'link,' <LinkName> ',' <TargetSeg> ',' <Naxis> ',' <Axes> ',' <Description>
        | <Scope> ',' <Type> ',' <Naxis> ',' <Axes> ','
          <Unit> ',' <Protocol> ',' <Description>
<LinkName>
        = <Name>
<Name>
        = [A-Za-z] { <NameEnd> }
<NameEnd>
        = [A-Za-z0-9_] { <NameEnd> }
<TargetSeg>
        = <SegmentName>
<Scope>
        = 'constant' | 'variable' | 'vardim'
<Type>
        = 'char' | 'short' | 'int' | 'longlong' | 'float' | 'double' |
          'time' | 'string'
<Naxis>
        = <string representing an unsigned 32-bit integer [1, INT_MAX]>
<Axes>
        = <AxisDim1> ',' <AxisDim2> ',' ... ',' <AxisDimN>
          where N = the integer represented by <Naxis>
<AxisDim>
        = <string representing an unsigned 32-bit integer [1, INT_MAX]>
<Unit>
        = <ascii string>
<Protocol>
        = 'bin' | 'zip' | 'fits', <FitsProtocol> | 'fitz', <FitzProtocol> | 'msi' | 'tas' , <TasProtocol> | 'generic'
<FitsProtocol>
        = <tile-compression arguments> ',' <bzero> ', ' <bscale> (see above for discussion on tile-compression arguments)
<FitzProtocol>
        = <bzero> ', ' <bscale>
<TasProtocol>
        = <tile-compression arguments> ',' <bzero> ', ' <bscale> (see above for discussion on tile-compression arguments)
<Description>
        = <ascii string>
### Links ###
<Link>
        = 'Link:' <LinkName> ',' <TargetSeries> ',' <Type> ',' <Description>
<LinkName>
        = <Name>
<Name>
        = [A-Za-z] { <NameEnd> }
<NameEnd>
        = [A-Za-z0-9_] { <NameEnd> }
<TargetSeries>
        = <ascii string>
<Type>
        = 'static' | 'dynamic'
<Description>
        = <ascii string>

Example .jsd File

#=====General Series Information=====
Seriesname:             su_arta.testfits_int
Author:                 "arta"
Owner:                  arta
Unitsize:               1440
Archive:                0
Retention:              7
StagingRetention:       3
Tapegroup:              1
PrimeKeys:              T_REC
DBIndex:                T_REC
Description:            MDI
#=====Keywords=====
Keyword:DATE, string, variable, record, " ", %s, none, " "
Keyword:DATE__OBS, string, variable, record, " ", %s, none, "[DATE-OBS] "
Keyword:EXPTIME, double, variable, record, nan, %f, s, " "
Keyword:MJD, double, variable, record, nan, %f, none, " "
Keyword:TIME, double, variable, record, nan, %f, none, " "
Keyword:DSUN_OBS, double, variable, record, nan, %f, m, " "
Keyword:R_SUN, double, variable, record, nan, %f, pixels, " "
Keyword:R_SUN_REF, double, constant, record, 696000000.000000, %f, m, " "
Keyword:CRLN_OBS, double, variable, record, nan, %f, m, " "
Keyword:CRLT_OBS, double, variable, record, nan, %f, m, " "
Keyword:CAR_ROT, int, variable, record, -2147483648, %d, none, " "
Keyword:SAT_ROT, double, variable, record, nan, %f, none, " "
Keyword:INST_ROT, double, variable, record, nan, %f, none, " "
Keyword:X0, double, variable, record, nan, %f, none, " "
Keyword:Y0, double, variable, record, nan, %f, none, " "
Keyword:IM_SCALE, double, variable, record, nan, %f, none, " "
Keyword:DATASIGN, int, variable, record, -2147483648, %d, none, " "
Keyword:OBS_VR, double, variable, record, nan, %f, none, " "
Keyword:OBS_VW, double, variable, record, nan, %f, none, " "
Keyword:OBS_VN, double, variable, record, nan, %f, none, " "
Keyword:series_num, int, variable, record, 0, %d, none, "Identifies dataset."
Keyword:ds, int, variable, record, 0, %d, none, "Identifies virtual dataset."
Keyword:rn, int, variable, record, 0, %d, none, "Identifies record within dataset."
Keyword:DSNAME, string, variable, record, " ", %s, none, ""
Keyword:RUNTIME, time, variable, record, -4712.01.01_12:00:00_UTC, 0, UTC, " "
Keyword:XY_QUAL, int, variable, record, 0, %d, none, " "
Keyword:EPH_QUAL, int, variable, record, 0, %d, none, " "
Keyword:T_FIRST, time, variable, record, -4712.01.01_12:00:00_TAI, 0, TAI, " "
Keyword:T_LAST, time, variable, record, -4712.01.01_12:00:00_TAI, 0, TAI, " "
Keyword:VDS_VERS, string, variable, record, " ", %s, none, " "
Keyword:DNAME0, string, variable, record, " ", %s, none, " "
Keyword:SOURCE, string, variable, record, " ", %s, none, " "
Keyword:SRC_ADJ, string, variable, record, " ", %s, none, " "
Keyword:BLDVER00, int, variable, record, 0, %d, none, " "
Keyword:BLDVER10, int, variable, record, 0, %d, none, " "
Keyword:CALTBLS, string, variable, record, " ", %s, none, " "
Keyword:QUALITY, string, variable, record, " ", %s, none, " "
Keyword:DPC, string, variable, record, " ", %s, none, " "
Keyword:T_REF, time, variable, record, -4712.01.01_12:00:00_TAI, 0, TAI, " "
Keyword:T_EARTH, time, variable, record, -4712.01.01_12:00:00_TAI, 0, TAI, " "
Keyword:BUNIT, string, variable, record, " ", %s, none, " "
Keyword:S_MAJOR, double, variable, record, nan, %f, none, " "
Keyword:S_MINOR, double, variable, record, nan, %f, none, " "
Keyword:S_ANGLE, double, variable, record, nan, %f, none, " "
Keyword:MAGNIFY, double, variable, record, nan, %f, none, " "
Keyword:ORIENT, string, variable, record, " ", %s, none, " "
Keyword:FD_SCALE, double, variable, record, nan, %f, none, " "
Keyword:DATAMIN, double, variable, record, nan, %f, none, " "
Keyword:DATAMAX, double, variable, record, nan, %f, none, " "
Keyword:DATA_RMS, double, variable, record, nan, %f, none, " "
Keyword:DATAMEAN, double, variable, record, nan, %f, none, " "
Keyword:DATAMEDN, double, variable, record, nan, %f, none, " "
Keyword:DATASKEW, double, variable, record, nan, %f, none, " "
Keyword:DATAKURT, double, variable, record, nan, %f, none, " "
Keyword:DATAVALS, int, variable, record, -2147483648, %d, none, " "
Keyword:MISSVALS, int, variable, record, -2147483648, %d, none, " "
Keyword:MISSPKTS, int, variable, record, -2147483648, %d, none, ""
Keyword:CMPERRS, int, variable, record, -2147483648, %d, none, " "
Keyword:SIZERRS, string, variable, record, " ", %s, none, ""
Keyword:EARTH_B0, double, variable, record, nan, %f, none, " "
Keyword:EARTH_L0, double, variable, record, nan, %f, none, " "
Keyword:EARTH_CR, int, variable, record, -2147483648, %d, none, " "
Keyword:EARTH_D, double, variable, record, nan, %f, none, " "
Keyword:EARTH_VR, double, variable, record, nan, %f, none, " "
Keyword:EARTH_VW, double, variable, record, nan, %f, none, " "
Keyword:EARTH_VN, double, variable, record, nan, %f, none, " "
Keyword:EARTH_R0, double, variable, record, nan, %f, none, " "
Keyword:EARTH_DT, double, variable, record, nan, %f, none, " "
Keyword:MDI_Y0, double, variable, record, nan, %f, none, " "
Keyword:MDI_X0, double, variable, record, nan, %f, none, " "
Keyword:TBLXYUPD, time, variable, record, -4712.01.01_12:00:00_UTC, 0, UTC, " "
Keyword:P_ANGLE, double, variable, record, nan, %f, none, " "
Keyword:FNDLMBXC, double, variable, record, nan, %f, none, " "
Keyword:FNDLMBYC, double, variable, record, nan, %f, none, " "
Keyword:FNDLMBMA, double, variable, record, nan, %f, none, " "
Keyword:FNDLMBMI, double, variable, record, nan, %f, none, " "
Keyword:FNDLMBAN, double, variable, record, nan, %f, none, " "
Keyword:BLDVER15, int, variable, record, 0, %d, none, " "
Keyword:DATE_OBS, string, variable, record, " ", %s, DRMS_MISSING_VALUE, " "
Keyword:T_REC, time, ts_eq, record, -4712.01.01_12:00:00_TAI, 0, TAI, " "
Keyword:T_REC_step, double, constant, record, 60.000000, %lf, minute, "T_REC step "
Keyword:T_REC_epoch, time, constant, record, 1993.01.01_00:04:56_TAI, 0, TAI, ""
Keyword:TELESCOP, string, constant, record, "SOHO", %s, none, " "
Keyword:INSTRUME, string, constant, record, "MDI", %s, none, " "
Keyword:ORIGIN, string, constant, record, "SOI Science Support Center", %s, none, " "
Keyword:CTYPE1, string, constant, record, "SOLARX", %s, none, " "
Keyword:CRPIX1, double, variable, record, nan, %f, none, " "
Keyword:CRVAL1, double, variable, record, nan, %f, none, " "
Keyword:CROTA1, double, variable, record, nan, %f, none, " "
Keyword:CDELT1, double, variable, record, nan, %f, none, " "
Keyword:CUNIT1, string, constant, record, "arcsecond", %s, none, " "
Keyword:XCEN, double, variable, record, nan, %f, none, " "
Keyword:CTYPE2, string, constant, record, "SOLARY", %s, none, " "
Keyword:CRPIX2, double, variable, record, nan, %f, none, " "
Keyword:CRVAL2, double, variable, record, nan, %f, none, " "
Keyword:CROTA2, double, variable, record, nan, %f, none, " "
Keyword:CDELT2, double, variable, record, nan, %f, none, " "
Keyword:CUNIT2, string, constant, record, "arcsecond", %s, none, " "
Keyword:YCEN, double, variable, record, nan, %f, none, " "
Keyword:T_OBS, time, variable, record, -4712.01.01_12:00:00_TAI, 0, TAI, " "
#=====Segments=====
Data: rice_img, variable, int, 2, 192, 192, m/s, fits, "compress Rice", 2.0, 0.35, "Fits data segment with Rice row compression"
Data: rice_tile_img, variable, int, 2, 192, 192, m/s, fits, "compress Rice 96,96,1", 0.0, 1.0, "Fits data segment with Rice tile compression"
Data: fitz_img, variable, int, 2, 192, 192, m/s, fitz, 0.0, 1.0, "Fitz data segment (Rice row compression)"
Data: tas_img, variable, int, 2, 192, 192, m/s, tas, "compress Rice 96,96,1", 0.5, 0.015, "Tas data segment with Rice tile compression"

JsocWiki: Jsd (last edited 2018-12-18 02:46:01 by ArtAmezcua)