|
Size: 34032
Comment:
|
Size: 34382
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 42: | Line 42: |
| The operations now supported are: * series_list -- Implemented within "show_series". Expects a single parameter "filter" which is used in regex to limit the number of seriesnames returned. See the man page for show_series. If e.g. all series with the word "mdi" are desired then the URL will be: http://jsoc.stanford.edu/cgi-bin/ajax/show_series?filter=mdi * series_struct -- Implemented in jsoc_info. Expects parameters as described below. Example URL for e.g. the series mdi.vw_V_lev18 is: http://jsoc.stanford.edu/cgi-bin/ajax/jsoc_info?op=series_struct&ds=mdi.vw_V_lev18 which returns the JSON version of the information in the shell commands "show_info -l mdi.vw_V_lev18" and "show_info -s mdi.vw_V_lev18" combined. * rs_summary -- Implemented in jsoc_info. Expects parameters as below. Example URL is http://jsoc.stanford.edu/cgi-bin/ajax/jsoc_info?op=rs_summary&ds=mdi.vw_V_lev18 * rs_list -- Implemented in jsoc_info. Expects parameters as below. The lookdata call limits the recordset to 10,000 records. The present implementation (due to mjson library performance) can take a minute or more for 10,000 records. Use care. This and following functions expect an explicit recordset query which will resolve to a subset of the records in the given series. The example URL here returns a few keywords for 5 minutes (5 records) from the mdi.vw_V_lev18 data. http://jsoc.stanford.edu//cgi-bin/ajax/jsoc_info.csh?ds=mdi.vw_V_lev18[1996.05.01_12:00/5m&op=rs_list&key=DATAMEAN,T_OBS&seg=**NONE** [http://jsoc.stanford.edu//cgi-bin/ajax/jsoc_info.csh?ds=mdi.vw_V_lev18[1996.05.01_12:00/5m]&op=rs_list&key=DATAMEAN,T_OBS&seg=**NONE** http://jsoc.stanford.edu//cgi-bin/ajax/jsoc_info.csh?ds=mdi.vw_V_lev18%5B1996.05.01_12%3A00%2F5m%5D&op=rs_list&key=DATAMEAN%2CT_OBS&seg=**NONE**]] |
The operations now supported at http://jsoc.stanford.edu/cgi-bin/ajax are: * Implemented via show_series * series_list -- Implemented within "show_series". Expects a single parameter "filter" which is used in regex to limit the number of seriesnames returned. See the man page for show_series. If e.g. all series with the word "mdi" are desired then the URL will be: {{{ http://jsoc.stanford.edu/cgi-bin/ajax/show_series?filter=mdi%5C. }}} * Implemented via jsoc_info * series_struct -- Like "show_info -l -s" Example URL for e.g. the series mdi.vw_V_lev18 is: {{{ http://jsoc.stanford.edu/cgi-bin/ajax/jsoc_info?op=series_struct&ds=mdi.vw_V_lev18 }}} which returns the JSON version of the information in the shell commands "show_info -l mdi.vw_V_lev18" and "show_info -s mdi.vw_V_lev18" combined. * rs_summary -- Implemented in jsoc_info. Example URL is {{{ http://jsoc.stanford.edu/cgi-bin/ajax/jsoc_info?op=rs_summary&ds=mdi.vw_V_lev18 }}} At present returns first and last record information. * rs_list -- Implemented in jsoc_info. Functions similar to the shell command "show_info" to get dataseries record attributes. * Implemented via jsoc_fetch * exp_request -- Primary data export request tool. Expects a recordset specification and desired export method. Exported data will be either delivered immediately or kept online for 7 days depending on calling parameters and online status of the data. If a the data is not immediately available a "RequestID" is returned to the user for later access to the exported data. * exp_repeat -- Repeats the request for an expired (export data package is gone after 7 days). Expects RequestID. * exp_status -- Checks on status of prior request, expects RequestID. * exp_su -- Special tool for export of SUMS "Storage Units" to remote DRMS sites. The details are below, but some examples here may help. The lookdata call limits the recordset to 10,000 records. The present implementation can take a minute or more for 10,000 records. Use care. This and following functions expect an explicit recordset query which will resolve to a subset of the records in the given series. The example URL here returns a few keywords for 5 minutes (5 records) from the mdi.vw_V_lev18 data. {{{ http://jsoc.stanford.edu/cgi-bin/ajax/jsoc_info_test?ds=mdi.vw_V_lev18%5B1996.05.01%2F5m%5D&op=rs_list&key=DATAMEAN%2CT_OBS }}} Note the excape characters for URL encoding: '[' to '%5B', ']' to '%5D', '/' to %2F, ',' to '%2C' etc. The return json text from this query, with white space added, is: {{{{ "keywords":[{"name":"DATAMEAN", "values":["321.424972", "320.428355", "320.296396", "319.591784", "320.101514"] }, {"name":"T_OBS", "values":["1996.05.01_00:01:00_TAI", "1996.05.01_00:02:00_TAI", "1996.05.01_00:03:00_TAI", "1996.05.01_00:04:00_TAI", "1996.05.01_00:05:00_TAI"] } ], "segments":[], "links":[], "count":5, "status":0 } }}} Note that the same query can be made to the shell command, 'show_info' which is also available as a cgi-bin call alongside jsoc_info. Show_info returns text while jsoc_info returns json. In this example "wget" is used to get the response as plain text in the file "pt" {{{ wget -O pt 'http://jsoc.stanford.edu/cgi-bin/ajax/show_info?ds=mdi.vw_V_lev18%5B1996.05.01%2F5m%5D&key=DATAMEAN%2CT_OBS' }}} with the response in pt: {{{ DATAMEAN T_OBS 321.424972 1996.05.01_00:01:00_TAI 320.428355 1996.05.01_00:02:00_TAI 320.296396 1996.05.01_00:03:00_TAI 319.591784 1996.05.01_00:04:00_TAI 320.101514 1996.05.01_00:05:00_TAI }}} The prototype access web tool "http://jsoc.stanford.edu/ajax/lookdata.html" uses show_series and jsoc_info. The prototype export tool "http://jsoc.stanford.edu/ajax/exportdata.html" uses jsoc_fetch. |
| Line 249: | Line 298: |
| Method: GET - user should protect against multiple identical requests. | Method: POST or GET - user should protect against multiple identical requests. |
| Line 262: | Line 311: |
| * file - uploaded file in case ds == "*file*" | |
| Line 276: | Line 326: |
| * rcount - number of records found in the recordset. | |
| Line 287: | Line 338: |
| {{{ op=exp_requestlist - request export where recordset in uploaded file Same expected quantities as exp_request except recordset is "*file*" which triggers export request setup but process does not continue until a subsequent exp_upload is sent at which time the process can proceed. All of the export setup parameters are supplied on the exp_requestlist call and the requestid is returned. The subsequent POST of the file must include the requestid. Usage: alternate form of exp_request where a 2-step process is used to tie requestid to uploaded file Method: GET Expects: About the data: * ds - must contain "*file*" * process - Requested processing prior to export in desired protocol, default is "no_op". See below. * protocol - file conversion request. At present options are: fits, as_is. See Note re compression settings. * filenamefmt - rule for export filenames, default: {seriesname}.{recnum:%ld}.{segment_filename} About the communication: * format - format of returned information, defaults to "json", options will be: json, txt, html, maybe xml * method - name of export method, e.g.: url, url_quick, and later: ftp, http, email, tape. About the user: * requestor - ID of user, can be random or known Requestor. * notify - email address of user. May be omitted unless method is "email" or "tape". Returns in the specified format: * status - 4=request not formed correctly, bad series, etc. 7=request info saved, waiting subsequent POST of uploaded file. if status is 4 returns element "error" containing error message * requestid - RequestID of record in jsoc.exports * error - message, only present if status = 4 }}} Note: being tested in jsoc_fetch_test {{{ op=exp_fileupload - second part of recordset by file upload pair This opcode should be used as the second part of a pair with exp_reqlist which will have first provided all info except the recordset info in the "ds" parameter. Usage: second part of export a list of records provided in a file Method: POST Expects: * requestid - RequestID of pending request * file - upload file containing list of recordsets for export Returns in the specified format: * status - 0=OK immediate data available or queue managed data is complete 1=request received and action is pending, i.e. in processing 2=queued for processing 3=request too large for automatic requests 4=request not formed correctly, bad series, etc. 5=request old, results requested after data timed out. if status is > 2 returns element "error" containing error message * requestid - RequestID of record in jsoc.exports * dir - Directory in the JSOC system where exported data is located. * data - an array of count objects containing: * record - name of record as query with segment name as suffix * filename - file or link name of the requested data. * size - bytes of data to be returned if positive, or -1 if not known yet. * count - number of files returned in the data array. * method - copied from input, but url_quick may be reported as "url" if applicable. * protocol - copied from input. * wait - estimated seconds until data is available if status==1. * error - message, only present if status > 2 * contact - email address, name to contact if status >2. user should contact with RequestID. }}} Note: testing in jsoc_fetch_test |
Option for providing the desired RecordSet via an uploaded file is not implemented. For this case the "ds" param must be "*file* and the POST method must be used. In this case the assumption is that a possibly hidden iframe is used and the returned json script will be in the implied <body> tag so content type will be text/html rather than json. The uploaded file should be provided with the field name "file" and should consist of lines of RecordSet specs. Note: presently testing file upload in jsoc_fetch_test and exportdata2.html. Present limit of 8192 chars in the uploaded file. |
| Line 381: | Line 375: |
| op=exp_repeat - This call initiates a re-export of a prior export which has expired. Only the RequestID need be provided, and optionally new requestor contact info for email notification. Usage: Used to renew an expired export, results in repeat of any needed calculations or processing. Mdthod: GET or POST Expects: * requestid - RequestID of record in jsoc.exports from prior successful export request. * notify - email address of user. May be omitted. Returns: Same as for exp_status. }}} Note: If RequestID corresponds to a current online request, the retention will be extended and no further action will be taken, return will be as if exp_status was requested. Otherwise, an export record (in jsoc.export) with a status==0 will be updated in the corresponding record in jsoc.export_new with a status=2 and a possibly new notify address. The re-exported product will have the same RequestID but a new sunum. Implemented in jsoc_fetch_test and exportdata2.html. {{{ |
|
| Line 388: | Line 396: |
| * protocol - specifies the type of files that the exported storage unit should contain (as-is or FITS) | * protocol - specifies the type of files that the exported storage unit should contain (as-is or FITS) |
AJAX Interface for JSOC
Plan:
- Implement basic access to keywords and segment information via CGI-BIN GET and POST server programs to allow Javascript access to basic JSOC functions.
- Provide basic browser side html to demonstrate capability and serve as a starting point for simple use and development.
- Publish the spec for the interface to allow others to build better Client-side tools.
Overview of Export Management
Exports will be managed in the series jsoc.exports Each request will be given a RequestID which can be used to track status. Data to be exported can be requested via several methods. The data requested will be collected (e.g. tar) and placed in a data segment in a record in jsoc.exports. It will be exported (i.e. picked up by the requestor) directly from the SUMS directory.
Series: jsoc.exports will not be archived and retention time of 7 days. Thus the Requestor will have one week to fetch the data. The export record will be retained for a history log of exports.
The jsoc.exports series looks like:
P% show_info jsoc.exports -l
Prime Keys are:
RequestID
DB Index Keys are:
RequestID
All Keywords for series jsoc.exports:
DataSet (string) Dataset requested
ExpTime (time) Time of export
FilenameFmt (string) File basname format
Format (int) Data format code
Notify (string) Notification address
ReqTime (time) Time of request
RequestID (int) Export request identifier
Requestor (string) Name of requestor
Size (int) Volume of data requested
Status (int) Status of request
Segments for series jsoc.exports:
Data NA generic VAR Exported data{ - need to add keywords for export method (URL, tape, etc) and export function requested (i.e. convert to fits, rebin, extract, etc.)}
Status of prototype implementation of the draft plan
Many of the functions described below are now implemented in series_info and in a new module called jsoc_info which has heritage in show_info but emits JSON responses. Several open-source tools have been used in developing jsoc_info as well as http://jsoc.stanford.edu/ajax/lookdata.html which is a test client-side javascript code to use jsoc_info. These include:
Prototype Javascript Framework at http://www.prototypejs.org/ which provides AJAX style javascript functions enabling much easier jsvascript coding.
The [http://www.json.org/ JSON home] which describes the XML-like JSON data protocols with links to software.
[http://mjson.sourceforge.net/index.html M'sJSON] JSON parsing and generating code which is used in jsoc_info.
And hints for compatibility found in [http://www.w3schools.com/ w3schools] DOM information pages.
The operations now supported at http://jsoc.stanford.edu/cgi-bin/ajax are:
- Implemented via show_series
series_list -- Implemented within "show_series". Expects a single parameter "filter" which is used in regex to limit the number of seriesnames returned. See the man page for show_series. If e.g. all series with the word "mdi" are desired then the URL will be: http://jsoc.stanford.edu/cgi-bin/ajax/show_series?filter=mdi%5C.
- Implemented via jsoc_info
series_struct -- Like "show_info -l -s" Example URL for e.g. the series mdi.vw_V_lev18 is: http://jsoc.stanford.edu/cgi-bin/ajax/jsoc_info?op=series_struct&ds=mdi.vw_V_lev18 which returns the JSON version of the information in the shell commands "show_info -l mdi.vw_V_lev18" and "show_info -s mdi.vw_V_lev18" combined.
rs_summary -- Implemented in jsoc_info. Example URL is http://jsoc.stanford.edu/cgi-bin/ajax/jsoc_info?op=rs_summary&ds=mdi.vw_V_lev18 At present returns first and last record information.
- rs_list -- Implemented in jsoc_info. Functions similar to the shell command "show_info" to get dataseries record attributes.
- Implemented via jsoc_fetch
- exp_request -- Primary data export request tool. Expects a recordset specification and desired export method. Exported data will be either delivered immediately or kept online for 7 days depending on calling parameters and online status of the data. If a the data is not immediately available a "RequestID" is returned to the user for later access to the exported data.
- exp_repeat -- Repeats the request for an expired (export data package is gone after 7 days). Expects RequestID.
- exp_status -- Checks on status of prior request, expects RequestID.
- exp_su -- Special tool for export of SUMS "Storage Units" to remote DRMS sites.
The details are below, but some examples here may help. The lookdata call limits the recordset to 10,000 records. The present implementation can take a minute or more for 10,000 records. Use care. This and following functions expect an explicit recordset query which will resolve to a subset of the records in the given series.
The example URL here returns a few keywords for 5 minutes (5 records) from the mdi.vw_V_lev18 data.
http://jsoc.stanford.edu/cgi-bin/ajax/jsoc_info_test?ds=mdi.vw_V_lev18%5B1996.05.01%2F5m%5D&op=rs_list&key=DATAMEAN%2CT_OBS
Note the excape characters for URL encoding: '[' to '%5B', ']' to '%5D', '/' to %2F, ',' to '%2C' etc. The return json text from this query, with white space added, is: