JSOC Series Definition Files
NEWS!
There have been a number of bug fixes regarding "show_info -j" and "create_series" since the last JSOC release (version 4.7). If those commands have been troubling you, please try upgrading to the head CVS file revisions.
Introduction
The preferred way to create and manage JSOC data series is via a ".jsd" file. These files are read by the "create_series" program to make the required database tables.
At present the create_series program provides only terse messages when it can not parse a .jsd file so the format here should be followed closely. In the format rules here items in angle brackets "<>" are words, strings, or numbers to be provided by the series creator. Words are user provided single word parameters, they can not contain blanks. They may be given with or without quotation marks. Strings may also be given with or without quotation marks but must be enclosed in quote marks if they contain blanks. A string must contain at least one character so an empty comment, for example, must be at least one blank character. In order to differentiate between words and strings, strings begin with "str". Choices in curly brackets and delimited by vertical bars are exact words that must be supplied. Choices are shown in bold (where I can figure out how). Any line beginning with a "#" is a comment and is not examined by create_series. The tokens ( words, strings, numbers, must be comma "," separated.
Names for series, keywords, links, and segments are implemented as PostgreSQL column or table names and are not allowed to collide with Postgres reserved names.
The .jsd file consists of four sections containing definitions of global information, keywords, links, and data segments. Only the global information section is required although a series with only global information has limited use.
Global series information
This section contains global information about the JSOC data series, that applies to all records and storage units belonging to the series. It is the information kept in the "drms_series" table. The .jsd syntax takes the form:
Seriesname: <str series name> Description: <str description> Author: <str author name> Unitsize: <storage unit size> Retention: <retention for new SUs> StagingRetention: <retention for retrieved SUs> Archive: <archive flag> Tapegroup: <tape group> PrimeKeys: <primary keyword list> DBIndex: <index keyword list> Version: <version number>
where:
<str series name> is a string containing the name of the JSOC dataseries defined in this file. The series name has to be unique, i.e. there can be no two data series in the JSOC database with the same name. The series name will be used as a database table name so it is constrained to letters, digits, and underscores. The first character must be a letter. Case is preserved but ignored. The seriesname must include both the project and data name parts with the "." separator. See also PostgreSqlReservedNames.
Some good series names include:
su_bugsbunny.elmer_fudd
su_bugsbunny.carrots
warner.cartoons
su_elmer.hunt_Apr2007
Some series names that will not work:
su_bugsbunny.4myself [Starts with a number, 4]
warner.carrot.r*bbit [More than one dot (.) and contains an asterisk (*)]
su_elmer.sat-sun-activity [Uses a hyphen]
[All examples assume that the name spaces su_bugsbunny, su_elmer and warner exist, which is not true.]
See DrmsSeriesNames for the current valid list.
<str description> is a string containing a description of the data series.
<str author name> is a string containing the name of the author who created the data series.
<storage unit size> is a non-zero integer indicating the number of data records that go into forming a storage unit for this series. Unless the data segments are to be stored in the Tiled Array Storage or equivalent FITS based format there is no requirement to include multiple records in a single storage unit. However there may be an efficiency aspect since records created at the same time may be used at the same time and if needed records that share a storage unit will be retrieved from tape as a whole reducing retrieval latency for nearby records. There is no disadvantage to making unitsize larger than 1.
<retention for new SUs> is an integer specifying the minimum number of days that a newly created Storage Units will remain online AKA the Storage Unit's retention. Each newly created Storage Unit will be assigned this value, but the optional, global DRMS parameter DRMS_NEWSURETENTION will override this value. After a Storage Unit has been created, its retention value can be modified with the set_suretention program. A short retention combined with an archive flag of 0 will result in a short-lived Storage Unit that will be quickly purged from SUMS. Some example retention values may be found in DrmsSeriesNames.
<retention for retrieved SUs> is an integer specifying the retention, in days, of Storage Units restored from tape to disk. This value is used when a SUMS call (i.e., SUM_get()) results in the retrieval of an offline Storage Unit that gets restored to disk. Each Storage Unit retrieved from tape will be assigned this value, but the optional, global DRMS parameter DRMS_RETENTION will override this value. After a Storage Unit has been restored from tape, its retention value can be modified with the set_suretention program.
<archive flag> is an integer either 0 or 1, describing whether the data segments associated with the records of this series are to be archived on permanent media. A value of 1 means that they are to be archived. If the archive flag is 0 then the <tape group> field is ignored.
<tape group> is an integer identifying the tape group to which this data series belongs. This is used to group together data storage units belonging to related data series when storing them on tape. The tape group convention is described in DrmsSeriesNames.
<primary keyword list> A list of zero or more keyword names constituting the primary key of the series. These keywords must also be declared in the keyword section of the JSD. If PrimeKeys is present, then a query for DRMS records where a value is specified for each prime-key constituent MUST return at most one DRMS record. At the PostgreSQL level, there can be more than one record that matches such a query. These PostgreSQL records form a set of versions of a single DRMS record, however, and the DRMS query will return the most-recent PostgreSQL record in this set. "Index" is a synonym for PrimeKeys - one can specify the primary key with the Index parameter in exactly the same way as with the PrimeKeys parameter. The Index parameter was in wide use prior to March 2008. If PrimeKeys and Index are both missing, then there will be no primary key for the DRMS records (although at the PostgreSQL level, there is a "recnum" column that can be used to uniquely identify a single record). Note that both 'P' and 'K' are capital letters in "PrimeKeys".
<index keyword list> If the DBIndex parameter, a comma-separated list of keyword names, is specified, then for each keyword in this list a database index will be generated. If DBIndex is not present, but if Index or PrimeKeys is present, then a COMPOSITE index of all prime-key constituents will be generated. A composite index is beneficial in only one case: when a search involves sorting by all members of the index. However, this will generally be true for many of our queries. It is true for DRMS queries that perform the "prime-key logic" - the logic that results in the selection of the current version of a DRMS record. It is NOT true for "bang" queries where the user desires to select ALL versions of one or more DRMS records. If DBIndex, Index, and PrimeKeys are all missing, then there will be no indexes created. Indexed keywords may lead to much faster data retrieval. Generally keywords used for prime keys should also be index keywords. Note that 'D', 'B', and 'I' are all capital letters in "DBIndex".
<version number> is the version number of the .jsd file. Use "2.1" for new series.
Keywords
The keyword section contains declarations of all keywords present in records belonging to the series. Each keyword description consist of "Keyword:" followed by a comma-separated list of attributes describing the keyword. Space and tab characters are ignored. The attributes list must be contained in a single line. [To remove this restriction, must modify the function getnextline() in src/base/libdrms/drms_parser.c] The lists of keywords allowed for each series are maintained in the drms_keyword table. A keyword can be simple or a link. A simple keyword can be fixed or variable and will contain an explicit value for the entire series (fixed) or for each record (variable). A link keyword describes a pointer to a keyword in some other series.
Simple Keywords: If the keyword is a simple value this takes the following form:
Keyword: <name>, <datatype>, <recordscope>, <segmentscope>, <default value>, <format>, <unit>, <str description>
where:
<name> is a word specifying the name of the keyword. Keyword names cannot exceed 24 characters in length. They must begin with an alphabetic character and they may contain only alphanumeric characters or the underscore "_". Although keyword names are case-insensitive in SQL queries, they are case-sensitive in the DRMS API; therefore they should be considered case-sensitive, but with the caveat that a series cannot contain multiple keyword names that would not strcasecmp. See also PostgreSqlReservedNames.
<datatype> specifies the data type of the keyword value. The following types are recognized: char, short, int, longlong, float, double, time, and string.
<recordscope> specifies the variability of the keyword. Keytypes are constant, variable or slotted.
constant means that the keyword has a fixed value set by the default value.
variable means that the keyword can take on a different value for each record (or segment). If not specified it will contain the default value.
Slotted recordscopes include ts_eq, slot, enum, and carr. Slotted keywords used as Primekeys provide a clean mapping between floating point or other non-definite value sets to well defined 'slots' in a virtual axis. Slotted keywords require the existence of several additional, associated keywords whose names are formed by adding a suffix to the slotted keyword name (the 'base' name). The exact set of associated keywords needed depends on the type of slotted keyword. One additional keyword, <base>_index, is implicitly created and cannot be specified in the jsd. When the slotted keyword is specified as a Primekey, the associated <base>_index key will be the actual indexed keyword. See DRMS Dataset Names for a more detailed discussion. Also see examples below.
<segmentscope> specifies the association of the keyword as per record or per segment within the record. Values for 'scope' are:
record means that there is an instance of the keyword for each record in the series.
segment means that there is a value for the keyword for each segment for each record. The actual keyword names will have an underscore and a three digit segment number appended to the given name. Thus to reference say, "bzero" for the third segment of each record in a series one would use the name "bzero_002". (what happens if the .jsd has explicit entries for each segment using this convention??) The way to access the keyword value its to treat it as an array, i.e. "bzero" for the third segment would be "bzero[2]" or with the suffix style name.
<default value> is the default value to be assigned to this keyword. A value must be supplied in the appropriate format. Sample default values are 0 for fixed-point numerical types, 0.0 or NaN for floating-point types, a one blank character string for string types and 1970.01.01 for the time type. In order to specify a null string for values of type char or string it is necessary to use numeric 0; "" will not work. The word DRMS_MISSING_VALUE may be used for all types and will be replaced with the standard value for that type. A constant keyword has the default value for all records. When a new record is first created its keywords are populated with the default values. The default value specification can be up to 4096 characters long. [Does this imply that a string keyword value is restricted to 4096 chars?]
<format> is a format word as used by the printf family of functions for formatting an object of type <datatype>. The length of the format string is restricted to 24 characters. It must be quoted in either single or double quotes if it contains embedded white space. Times are stored internally as doubles but may be shown as familiar yyyy.mm.dd_hh:mm:ss.ss strings. Note: For time keywords, the format field should contain an integer, and specifies the precision of the seconds field in the printed time string. It will be used as the precision parameter to the sprint_time() function (see http://jsoc.stanford.edu/doxygen_html/timeio_8h.html). Also, see JSOC Time Format Notes.
<unit> is a word specifying the physical unit of the keyword value. If no "unit" is to be specified, the word none should be used. The length of the unit string is restricted to 24 characters. It must be quoted if it contains embedded white space. Note: For time keywords, the unit field is a time-zone string, such as "PDT". It will be used as the zone parameter to the sprint_time() function. Special "zones" of "TAI" and "ISO" are a;so recognized.
<str description> is a string describing the meaning of the keyword. It can be up to 4096 characters in length, and must be quoted if it contains embedded white space.
For example, to declare a keyword containing a floating point value describing velocity associated with a data record, the keyword section could contain a line like
Keyword: v, float, variable, record, 0.0, %12g, m/s, "velocity" - or - Keyword: "v", float, variable, record, "0.0", "%12g", "m/s", "velocity" - or - Keyword: Velocity,float,variable,record,NaN,"%12g m/s","meters per second" ,"velocity [m/s]"
An example Time type keyword might be like
Keyword: t_obs, time, variable, record, 1993.01.01_TAI, 0, TAI, "Time of observation"
An example slotted time type keyword and associated keywords might be like
Keyword: t_rec, time, ts_eq, record, 1993.01.01_TAI, 0, TAI, "Time index of observation" Keyword: t_rec_epoch, time, constant, record, 1993.01.01, 0, TAI, "Reference epoch for time index" Keyword: t_rec_step, double, constant, record, 60.0, %lf, "s", "Index increment time span"
Link Keywords A keyword can also be a link pointing to a keyword in a record in a different series, making it possible for data records to inherit keyword values from each other.
A link keyword has <datatype> equal to link and takes the form the form:
Keyword: <name>, link, <linkname>, <target keyword>, <str description>
where:
<name> is a word specifying the name of the keyword.
<linkname> is a word that specifies the name of the link, which must be defined elsewhere in the same JSD. The link specification (see below) will identify a record in another series from which to take the value.
<target keyword> is a word that specifies the name of the target keyword from which to get the value for this keyword from the record pointed to by the link.
<str description> is a string describing the meaning of the keyword. For example:
Keyword: ROLL, link, HMI_ORBIT, P_Angle, "Link to current best p_angle"
This definition means that to get the value of the keyword "ROLL", follow the link named "HMI_ORBIT" to the data record it points to and get the value of keyword named "P_Angle" from that data record. Note that linked keywords may not be used to form "where" clauses in record queries. Note that the create_series program does not complain about a link keyword without the matching record link yet, but it should...
Links
Links are pointers to records in (usually) other series. Links make it possible to effectively merge elements from other data records eliminating the necessity to make redundant copies. Links can also capture dependencies between data records such as processing history. For example a data record can contain links to the data records that were used in creating it, such as a dopplergram data record pointing to the filtergrams from which is was created. Links link records. A keyword link points to a particular keyword in a link record. A segment link points to a particular target segment in a link record. Links come in two varieties, static links and dynamic links.
Link: <name>, <target series>, <link_type>, <str description>
where:
<name> is word giving the symbolic name of the link. See also PostgreSqlReservedNames.
<target series> is a word containing the name of the series from which come the record pointed to by the link.
<link_type> can either take the value static which indicates that the link is static, i.e. based on record number, or the value dynamic which indicates that the link is dynamic, i.e. based on primary index.
<str description> is a string documenting the link for the user.
A static link points to a single data record from the target series. The link must be established/bound by specifying the record number of the record pointed to. This is done by calling the JSOC function
- int drms_setlink_static(DRMS_Record_t *rec, const char *linkname, long long recnum)
from a process running within the JSOC environment. record is the record containing the link linkname and recnum is the target record number in the target series.
A dynamic link points to the data record from the target series with the highest record number for a given primary index. This is done by calling the JSOC function:
- int drms_setlink_dynamic(DRMS_Record_t *rec, const char *linkname, DRMS_Type_t *types, DRMS_Type_Value_t *values)
Where again rec and linkname define the link and types and values specify the target record in the target series that will be linked. A linked record is used by e.g.
- DRMS_Record_t *drms_link_follow(DRMS_Record_t *rec, const char *linkname, int *status)
where rec is the record containing link linkname. drms_link_follow returns the target record in the target series. See the drms_link(3) man page for details.
Data Segments
The data segment section describes the data which is associated with records in the series being defined in the JSD but is stored in a file in SUMS rather than in the DRMS database tables. A segment may be direct or linked. A direct segment descriptor specifies the storage scope, protocol, and associated information such as data types, array sizes, etc., depending on the protocol used. A linked segment descriptor specifies the name of a record link descriptor and the name of a target segment held in the target record.
direct data segments
Each direct data segment describes a chunk of data stored in a file in SUMS. Various external storage formats of the data segments are available, and are specified by the <protocol> field. Raw level data is often stored simply as files of unspecified generic type. Processed data is usually stored in an n-dimensional array of a simple type where the external storage protocol is not of interest to the user. The general form of a data segment description is
Data: <name>, <scope>, <datatype>, <naxis>, <axis dims>, <unit>, <protocol>, {<protocol special>,} <str comment>The number of fields required depends on the value of <naxis> and <protocol>, so the following forms are valid
Data: <name>, <scope>, <datatype>, 0, <unit>, <protocol>, <comment> Data: <name>, <scope>, <int datatype>, <naxis>, <axis dims>, <unit>, fits, <tile-compression arguments>, <bzero>, <bscale>, <comment> Data: <name>, <scope>, <int datatype>, <naxis>, <axis dims>, <unit>, fitz, <bzero>, <bscale>, <comment> Data: <name>, <scope>, <int datatype>, <naxis>, <axis dims>, <unit>, bin, <bzero>, <bscale>, <comment> Data: <name>, <scope>, <int datatype>, <naxis>, <axis dims>, <unit>, bin.gz, <bzero>, <bscale>, <comment> Data: <name>, <scope>, <int datatype>, <naxis>, <axis dims>, <unit>, tas, <tile-compression arguments>, <bzero>, <bscale>, <comment> Data: <name>, <scope>, <float datatype>, <naxis>, <axis dims>, <unit>, fits, <tile-compression arguments>, <comment> Data: <name>, <scope>, <float datatype>, <naxis>, <axis dims>, <unit>, fitz, <comment> Data: <name>, <scope>, <float datatype>, <naxis>, <axis dims>, <unit>, bin, <comment> Data: <name>, <scope>, <float datatype>, <naxis>, <axis dims>, <unit>, bin.gz, <comment> Data: <name>, <scope>, <float datatype>, <naxis>, <axis dims>, <unit>, tas, <tile-compression arguments>, <comment>
where:
<name> is a word containing the name of the data segment. See also PostgreSqlReservedNames.
<scope> can take the value constant, variable, or vardim. The names of these scopes can be confusing.
A constant segment is one which points to the same file in SUMS for all records. It is essentially a link to the first segment (with the same name) written to the series. The constant segment will be set to point to the recnum of the first, and only, record which has had a drms_segment_write or drms_segment_write_from_file call for the constant segment.
Variable and vardims segments are in many senses normal segments which have different files in SUMS for each record.
If the scope is variable then while the record contents may vary from one record to the next the dimensions are fixed by the constants in the JSD.
If the scope is vardim then the dimensions of the data segment arrays may vary from record to record. Naxis does not change but axis dims are allowed to vary. For vardims the default are those from the JSD.
<int datatype> is the external storage type of the data in the data segment. It can be one of the integer data types: char, short, int, or longlong. Not all file formats support all these data types, see discussion for <protocol> below. [SHOULD ADD FIXED LENGTH STRINGS]
<float datatype> is the external storage type of the data in the data segment. It can be one of the floating-point data types: float, double, or time. Not all file formats support all these data types, see discussion for <protocol> below. [SHOULD ADD FIXED LENGTH STRINGS]
<naxis> is an integer and is the number of dimensions of an array data segment array, e.g. 1 is is a vector, 2 is a matrix, 3 is a cube etc. naxis may be 0 in which case no <axis dims> are required.
<axis dims> is a comma-space separated list of naxis positive integers giving the dimensions of the data segment array.
<unit> is a word describing the physical unit of the data segment values. If no units are to be specified use the word none.
<protocol> is the protocol used to store the data segments on disk. Supported protocols are:
bin - Binary array data in a DRMS specific internal format. The format consists of the 8-byte string "DRMS RAW", a 4-byte integer code for the data type, a 4-byte integer giving the number of axes (dimensions), a 4-byte integer for each of the dimensions giving the corresponding length of its axis, two 8-byte integers for bscale and bzero respectively (only present if the data type is fixed-point, i.e. char, short, int, longlong), an 8-byte integer containing the total length in bytes of the following data, and the raw data in native format. All data are little-endian, and floating point numbers are IEEE, so the format is very similar to a minimal FITS file, with the stripped header information in binary format.
bingz - "gzipped" file containing the binary array data in the same format as above.
fits - Standard FITS format. Supports all data types.
fitz - FITS format compressed losslessly with first differencing plus Rice/Golomb entropy coding. Currently supports 8, 16 and 32 bit integer 1d and 2d arrays.
msi - Multi Scale Image format. note: this format is presently not implemented although "msi" is allowed by the JSD parser. Uses a lossless compression algorithm based on wavelet transformation followed by Rice/Golomb entropy coding, storing the bitstream in resolution progressive order. Allows lower resolution (downsampled by powers of two) versions of the image to be extracted rapidly by simply truncating the file. Currently supports 16 bit integer 1d and 2d images with dimensions that are powers of 2.
tas - Special DRMS format allows segments from multiple records to be stored in the same file and for efficient access to sub-arrays within a given segment. If the protocol is tas the AND the TAS file is to be tile-compressed, TAS tile sizes must also be specified in the <tile-compression arguments> field. TAS files may also be uncompressed, in which case, there is no tiling.
generic - Generic data. For this type of segment no processing is done by the DRMS library. The user is expected to use the Storage Unit directory provided for the record for her/his own purposes. The only DRMS API functions likely to be of use for generic data are drms_segment_write_from_file or drms_record_directory. [probably more formats to come...]
<tile-compression arguments> are the CFITSIO tiled-compression parameters that specify a compression algorithm and, optionally, tile sizes. In general, the string looks like "compress <algorithm> <tilesize1>,<tilesize2>...<tilesizeN>". To specify NO compression, this .jsd field should contain the empty string (""). Tile-compression of floating point data is not recommended (since it would most likely result in lossy compression). Also, compression is not supported for 64-bit data types (i.e., longlong) - CFITSIO cannot compress 64-bit data. For float, double, and longlong, the value of this field should be the empty string ("") to denote no compression. <algorithm> refers to the compression algorithm desired. Permissible values are "GZIP", "Rice", or "PLIO". <tilesize1>...<tilesizeN> refer to the size of the tiles (<tilesizeN> refers to the same dimension that NAXISN refers to). If the string is not empty, then the string takes the form "compress" or "compress <algorithm> {<tilesize1>,<tilesize2>...<tilesizeN>}", where {} implies the tile sizes are optional. If the string is "compress" then by default the algorithm is Rice and the data are row-compressed. If the string contains tile sizes, then the size specified overrides the default row-compression. For the TAS protocol, in most cases the user should specify tile sizes and not rely upon row-compression.
<bzero> is the value of the offset that must be applied to the scaled BIN, BINZ, FITS, FITSZ, or TAS file data to retrieve the data in physical units (the "real" values).
<bscale> is the scaling that must be applied to the raw BIN, BINZ, FITS, FITSZ, or TAS file data to retrieve the data in physical units (the "real" values).
- Data segments are numbered in the order they appear in the JSD.
linked segments
A linked segment is a pointer to a segment belonging to another record in another series. Similar to keyword links, the linked segment descriptor contains only the information necessary to identify the target segment. At present the data type, and units will be taken from the target segment. While the naxis and dimensions are specified in the jsd they are also ignored at present. The protocol of course must be specified only in the target. The form is as follows:
The number of fields required depends on the value of <naxis> so the following forms are valid
Data: <name>, link, <linkname>, <target segment>, 0, <str description> Data: <name>, link, <linkname>, <target segment>, <naxis>, <axis dims>, <str description>
- where:
<name> is a word specifying the name of the keyword.
<linkname> is a word that specifies the name of the link, which must be defined elsewhere in the same JSD. The link specification (see above) will identify a record in another series from which to find the target segment.
<target segment> is a word that specifies the name of the target segment from which to get the data for this segment from the record pointed to by the link.
<naxis> is an integer and is the number of dimensions of an array data segment array, e.g. 1 is is a vector, 2 is a matrix, 3 is a cube etc. naxis may be 0 in which case no <axis dims> are required or allowed.
<axis dims> is a comma-space separated list of naxis positive integers giving the dimensions of the data segment array.
<str description> is a string describing the content of the segment. Since naxis and axis dims are ignored in the present implementation the first form is preferred.
data slices
If the user intends to write slices of data to each image, instead of writing whole images, then the fields <naxis> and <axis dims> are used to specify the default dimensions of the final, overall image. The code that writes data slices uses these values to determine the image's final dimensions, but these dimensions can be overridden. To override the default values, the first time a slice is to be written the user provides a slice of data with the desired dimensions to a data-slice-writing library call. The appropriate values for <naxis> and <axis dims> depends on the value of the <scope> field as well. A scope of vardim still allows the user to create images whose dimensions vary across records, but to do so the user must override, for each image, the default dimensions present in the segment specification in the .jsd file.
Please see DataSlices for a detailed description about data slices work and how to create images by writing slices.
JSD Grammar
The complete grammar that describes the set of acceptable strings follows. Keep in mind that the only required section is Series Information. The other sections are optional. This grammar is applicable to the most recent jsd version, not older ones.
Each symbol of the grammar is enclosed in angled brackets ("<" and ">") and its definition lies to the right of the '='. Literal strings are enclosed in single quotes ("'"). Text enclosed in square brackets ("[: and :and ]") represents any single character from the set of characters specified in the brackets. A '-' within square brackets denotes a range of characters. For example, [A-Za-z] denotes any one of the 26 upper-case or 26 lower-case letters. S1 | S2 denotes alternation between the symbols S1 and S2 (S1 and S2 may be literals or symbols defined in a different rule). For example, 'Sanjaya' | 'Melinda' means either the string 'Sanjaya' or the string 'Melinda' satisfies the rule containing the alternation. Finally, curly brackets ("{" and "}") denote optional elements.
### Series Information ###
<SeriesName>
= 'Seriesname:' <ascii string>
<SeriesDescription>
= 'Description:' <ascii string>
<Owner>
= 'Owner:' <ascii string>
<Author>
= 'Author:' <ascii string>
<ArchiveFlag>
= 'Archive:' '0' | '1'
<UnitSize>
= 'Unitsize:' <string representing unsigned 32-bit integer [1, INT_MAX]>
<TapeGroup>
= 'Tapegroup:' <string representing unsigned 32-bit integer [0, INT_MAX]>
<Retention>
= 'Retention:' <string representing unsigned 16-bit integer [0, SHRT_MAX]>
<StagingRetention>
= 'StagingRetention:' <string representing unsigned 16-bit integer [0, SHRT_MAX]>
<PrimaryIndex>
= 'Index:' <KeywordList>
<KeywordList>
= { <KeywordName> ',' } <KeywordName>
<KeywordName>
= <Name>
<Name>
= [A-Za-z] { <NameEnd> }
<NameEnd>
= [A-Za-z0-9_] { <NameEnd> }
*** Keywords ***
<Keyword>
= 'Keyword:' <KeyName> ',' <TypeAndFields>
<KeyName>
= <Name>
<Name>
= [A-Za-z] { <NameEnd> }
<NameEnd>
= [A-Za-z0-9_] { <NameEnd> }
<TypeAndFields>
= 'link,' <LinkName> ',' <TargetKey> ',' <Description>
| 'char,' <RecordScope> ',' <SegmentScope> ',' <DefChar> ','
<Format> ',' <Unit> ',' <Description>
| 'short,' <RecordScope> ',' <SegmentScope> ',' <DefShort> ','
< Format> ','< Unit> ',' <Description>
| 'int,' <RecordScope> ',' <SegmentScope> ',' <DefInt> ','
< Format> ',' <Unit> ',' <Description>
| 'longlong,' <RecordScope> ',' <SegmentScope> ',' <DefLongLong> ','
<Format> ',' <Unit> ',' <Description>
| 'float,' <RecordScope> ',' <SegmentScope> ',' <DefFloat> ','
<Format> ',' <Unit> ',' <Description>
| 'double,' <RecordScope> ',' <SegmentScope> ',' <DefDouble> ',
<Format> ',' <Unit> ',' <Description>
| 'time,' <RecordScope> ',' <SegmentScope> ',' <DefTime> ','
<Format> ',' <Unit> ',' <Description>
| 'string,' <RecordScope> ',' <SegmentScope> ',' <DefString> ','
<Format> ',' <Unit> ',' <Description>
<LinkName>
= <Name>
<TargetKey>
= <KeyName>
<RecordScope>
= 'constant' |'variable'
<SegmentScope>
= 'segment' | 'record'
<DefChar>
= <string representing signed 8-bit integer [SCHAR_MIN, SCHAR_MAX]>
<DefShort>
= <string representing signed 16-bit integer [SHRT_MIN, SHRT_MAX]>
<DefInt>
= <string representing signed 32-bit integer [INT_MIN, INT_MAX]>
<DefLongLong>
= <string representing signed 64-bit integer [LLONG_MIN, LLONG_MAX]>
<DefFloat>
= <string representing single-precision, floating-point number>
NOTE: numbers larger in magnitude than what a float can represent
will become infinite.
<DefDouble>
= <string representing double-precision, floating-point number>
NOTE: numbers larger in magnitude than what a double can represent
will become infinite.
<DefTime>
= <string representing time - see JSOC Time Format Notes>
<DefString>
= <ascii string no longer than DRMS_MAXPATHLEN - 1 characters, MUST HAVE LENGTH >= 1>
<Format>
= <printf conversion specification, e.g., %d, %c>
<Unit>
= <ascii string>
<Description>
= <ascii string>
### Data Segments ###
<DataSegment>
= 'Data:' <SegmentName> ',' <TypeAndFields>
<SegmentName>
= <ascii string>
<TypeAndFields>
= 'link,' <LinkName> ',' <TargetSeg> ',' <Naxis> ',' <Axes> ',' <Description>
| <Scope> ',' <Type> ',' <Naxis> ',' <Axes> ','
<Unit> ',' <Protocol> ',' <Description>
<LinkName>
= <Name>
<Name>
= [A-Za-z] { <NameEnd> }
<NameEnd>
= [A-Za-z0-9_] { <NameEnd> }
<TargetSeg>
= <SegmentName>
<Scope>
= 'constant' | 'variable' | 'vardim'
<Type>
= 'char' | 'short' | 'int' | 'longlong' | 'float' | 'double' |
'time' | 'string'
<Naxis>
= <string representing an unsigned 32-bit integer [1, INT_MAX]>
<Axes>
= <AxisDim1> ',' <AxisDim2> ',' ... ',' <AxisDimN>
where N = the integer represented by <Naxis>
<AxisDim>
= <string representing an unsigned 32-bit integer [1, INT_MAX]>
<Unit>
= <ascii string>
<Protocol>
= 'bin' | 'zip' | 'fits', <FitsProtocol> | 'fitz', <FitzProtocol> | 'msi' | 'tas' , <TasProtocol> | 'generic'
<FitsProtocol>
= <tile-compression arguments> ',' <bzero> ', ' <bscale> (see above for discussion on tile-compression arguments)
<FitzProtocol>
= <bzero> ', ' <bscale>
<TasProtocol>
= <tile-compression arguments> ',' <bzero> ', ' <bscale> (see above for discussion on tile-compression arguments)
<Description>
= <ascii string>
### Links ###
<Link>
= 'Link:' <LinkName> ',' <TargetSeries> ',' <Type> ',' <Description>
<LinkName>
= <Name>
<Name>
= [A-Za-z] { <NameEnd> }
<NameEnd>
= [A-Za-z0-9_] { <NameEnd> }
<TargetSeries>
= <ascii string>
<Type>
= 'static' | 'dynamic'
<Description>
= <ascii string>
Example .jsd File
#=====General Series Information===== Seriesname: su_arta.testfits_int Author: "arta" Owner: arta Unitsize: 1440 Archive: 0 Retention: 7 StagingRetention: 3 Tapegroup: 1 PrimeKeys: T_REC DBIndex: T_REC Description: MDI #=====Keywords===== Keyword:DATE, string, variable, record, " ", %s, none, " " Keyword:DATE__OBS, string, variable, record, " ", %s, none, "[DATE-OBS] " Keyword:EXPTIME, double, variable, record, nan, %f, s, " " Keyword:MJD, double, variable, record, nan, %f, none, " " Keyword:TIME, double, variable, record, nan, %f, none, " " Keyword:DSUN_OBS, double, variable, record, nan, %f, m, " " Keyword:R_SUN, double, variable, record, nan, %f, pixels, " " Keyword:R_SUN_REF, double, constant, record, 696000000.000000, %f, m, " " Keyword:CRLN_OBS, double, variable, record, nan, %f, m, " " Keyword:CRLT_OBS, double, variable, record, nan, %f, m, " " Keyword:CAR_ROT, int, variable, record, -2147483648, %d, none, " " Keyword:SAT_ROT, double, variable, record, nan, %f, none, " " Keyword:INST_ROT, double, variable, record, nan, %f, none, " " Keyword:X0, double, variable, record, nan, %f, none, " " Keyword:Y0, double, variable, record, nan, %f, none, " " Keyword:IM_SCALE, double, variable, record, nan, %f, none, " " Keyword:DATASIGN, int, variable, record, -2147483648, %d, none, " " Keyword:OBS_VR, double, variable, record, nan, %f, none, " " Keyword:OBS_VW, double, variable, record, nan, %f, none, " " Keyword:OBS_VN, double, variable, record, nan, %f, none, " " Keyword:series_num, int, variable, record, 0, %d, none, "Identifies dataset." Keyword:ds, int, variable, record, 0, %d, none, "Identifies virtual dataset." Keyword:rn, int, variable, record, 0, %d, none, "Identifies record within dataset." Keyword:DSNAME, string, variable, record, " ", %s, none, "" Keyword:RUNTIME, time, variable, record, -4712.01.01_12:00:00_UTC, 0, UTC, " " Keyword:XY_QUAL, int, variable, record, 0, %d, none, " " Keyword:EPH_QUAL, int, variable, record, 0, %d, none, " " Keyword:T_FIRST, time, variable, record, -4712.01.01_12:00:00_TAI, 0, TAI, " " Keyword:T_LAST, time, variable, record, -4712.01.01_12:00:00_TAI, 0, TAI, " " Keyword:VDS_VERS, string, variable, record, " ", %s, none, " " Keyword:DNAME0, string, variable, record, " ", %s, none, " " Keyword:SOURCE, string, variable, record, " ", %s, none, " " Keyword:SRC_ADJ, string, variable, record, " ", %s, none, " " Keyword:BLDVER00, int, variable, record, 0, %d, none, " " Keyword:BLDVER10, int, variable, record, 0, %d, none, " " Keyword:CALTBLS, string, variable, record, " ", %s, none, " " Keyword:QUALITY, string, variable, record, " ", %s, none, " " Keyword:DPC, string, variable, record, " ", %s, none, " " Keyword:T_REF, time, variable, record, -4712.01.01_12:00:00_TAI, 0, TAI, " " Keyword:T_EARTH, time, variable, record, -4712.01.01_12:00:00_TAI, 0, TAI, " " Keyword:BUNIT, string, variable, record, " ", %s, none, " " Keyword:S_MAJOR, double, variable, record, nan, %f, none, " " Keyword:S_MINOR, double, variable, record, nan, %f, none, " " Keyword:S_ANGLE, double, variable, record, nan, %f, none, " " Keyword:MAGNIFY, double, variable, record, nan, %f, none, " " Keyword:ORIENT, string, variable, record, " ", %s, none, " " Keyword:FD_SCALE, double, variable, record, nan, %f, none, " " Keyword:DATAMIN, double, variable, record, nan, %f, none, " " Keyword:DATAMAX, double, variable, record, nan, %f, none, " " Keyword:DATA_RMS, double, variable, record, nan, %f, none, " " Keyword:DATAMEAN, double, variable, record, nan, %f, none, " " Keyword:DATAMEDN, double, variable, record, nan, %f, none, " " Keyword:DATASKEW, double, variable, record, nan, %f, none, " " Keyword:DATAKURT, double, variable, record, nan, %f, none, " " Keyword:DATAVALS, int, variable, record, -2147483648, %d, none, " " Keyword:MISSVALS, int, variable, record, -2147483648, %d, none, " " Keyword:MISSPKTS, int, variable, record, -2147483648, %d, none, "" Keyword:CMPERRS, int, variable, record, -2147483648, %d, none, " " Keyword:SIZERRS, string, variable, record, " ", %s, none, "" Keyword:EARTH_B0, double, variable, record, nan, %f, none, " " Keyword:EARTH_L0, double, variable, record, nan, %f, none, " " Keyword:EARTH_CR, int, variable, record, -2147483648, %d, none, " " Keyword:EARTH_D, double, variable, record, nan, %f, none, " " Keyword:EARTH_VR, double, variable, record, nan, %f, none, " " Keyword:EARTH_VW, double, variable, record, nan, %f, none, " " Keyword:EARTH_VN, double, variable, record, nan, %f, none, " " Keyword:EARTH_R0, double, variable, record, nan, %f, none, " " Keyword:EARTH_DT, double, variable, record, nan, %f, none, " " Keyword:MDI_Y0, double, variable, record, nan, %f, none, " " Keyword:MDI_X0, double, variable, record, nan, %f, none, " " Keyword:TBLXYUPD, time, variable, record, -4712.01.01_12:00:00_UTC, 0, UTC, " " Keyword:P_ANGLE, double, variable, record, nan, %f, none, " " Keyword:FNDLMBXC, double, variable, record, nan, %f, none, " " Keyword:FNDLMBYC, double, variable, record, nan, %f, none, " " Keyword:FNDLMBMA, double, variable, record, nan, %f, none, " " Keyword:FNDLMBMI, double, variable, record, nan, %f, none, " " Keyword:FNDLMBAN, double, variable, record, nan, %f, none, " " Keyword:BLDVER15, int, variable, record, 0, %d, none, " " Keyword:DATE_OBS, string, variable, record, " ", %s, DRMS_MISSING_VALUE, " " Keyword:T_REC, time, ts_eq, record, -4712.01.01_12:00:00_TAI, 0, TAI, " " Keyword:T_REC_step, double, constant, record, 60.000000, %lf, minute, "T_REC step " Keyword:T_REC_epoch, time, constant, record, 1993.01.01_00:04:56_TAI, 0, TAI, "" Keyword:TELESCOP, string, constant, record, "SOHO", %s, none, " " Keyword:INSTRUME, string, constant, record, "MDI", %s, none, " " Keyword:ORIGIN, string, constant, record, "SOI Science Support Center", %s, none, " " Keyword:CTYPE1, string, constant, record, "SOLARX", %s, none, " " Keyword:CRPIX1, double, variable, record, nan, %f, none, " " Keyword:CRVAL1, double, variable, record, nan, %f, none, " " Keyword:CROTA1, double, variable, record, nan, %f, none, " " Keyword:CDELT1, double, variable, record, nan, %f, none, " " Keyword:CUNIT1, string, constant, record, "arcsecond", %s, none, " " Keyword:XCEN, double, variable, record, nan, %f, none, " " Keyword:CTYPE2, string, constant, record, "SOLARY", %s, none, " " Keyword:CRPIX2, double, variable, record, nan, %f, none, " " Keyword:CRVAL2, double, variable, record, nan, %f, none, " " Keyword:CROTA2, double, variable, record, nan, %f, none, " " Keyword:CDELT2, double, variable, record, nan, %f, none, " " Keyword:CUNIT2, string, constant, record, "arcsecond", %s, none, " " Keyword:YCEN, double, variable, record, nan, %f, none, " " Keyword:T_OBS, time, variable, record, -4712.01.01_12:00:00_TAI, 0, TAI, " " #=====Segments===== Data: rice_img, variable, int, 2, 192, 192, m/s, fits, "compress Rice", 2.0, 0.35, "Fits data segment with Rice row compression" Data: rice_tile_img, variable, int, 2, 192, 192, m/s, fits, "compress Rice 96,96,1", 0.0, 1.0, "Fits data segment with Rice tile compression" Data: fitz_img, variable, int, 2, 192, 192, m/s, fitz, 0.0, 1.0, "Fitz data segment (Rice row compression)" Data: tas_img, variable, int, 2, 192, 192, m/s, tas, "compress Rice 96,96,1", 0.5, 0.015, "Tas data segment with Rice tile compression"