Export System

The Way Things Work: File Structure

The export system implementation involves many components. Starting at the "highest" level, the web pages http://jsoc.stanford.edu/ajax/exportdata.html and http://jsoc2.stanford.edu/ajax/exportdata.html provide interfaces that allow users to initiate export requests, as well as check-on the status of pending requests. The first URL provides access to the external, or public, export system, whereas the second URL provides access to the internal, or private, one. The public system provides access to public data, which is a subset of the data available from the private system.

The machine that serves as the web server for both sites is solarweb. It serves the /web filesystem that is generally mounted as "/web" on many machines. The files that are relevant to exports live in three places on this filesystem: 1. /web/jsoc/htdocs/ajax contains links to the exportdata.html source, which is shared by both the external and internal system; 2. /web/jsoc/cgi-bin/ajax contains the cgi programs, and links to programs, used by the external export system; and 3. /web/jsoc2/cgi-bin/ajax contains the cgi programs, and links to programs, used by the internal system. There is also a link from /web/jsoc/htdocs2 to /web/jsoc/htdocs - this construct is what allows the external and internal systems to share exportdata.html.

The ultimate source for exportdata.html is /home/jsoc/cvs/Development/JSOC/proj/export/webapps/exportdata.html. This is the file that /web/jsoc/htdocs/ajax/exportdata.html links to. In exportdata.html are references to files containing javascript. These files live in the /home/jsoc/cvs/Development/JSOC/proj/export/webapps/exportdata.d directory. The html in exportdata.html accesses these files by first resolving the URL http://jsoc.stanford.edu/ajax/exportdata (in the case of the external system), then looking in the exportdata.d in this resolved directory. In /web/jsoc/htdocs/ajax there is a link named exportdata which links to /home/jsoc/cvs/Development/JSOC/proj/export/webapps, so http://jsoc.stanford.edu/ajax/exportdata resolves to /home/jsoc/cvs/Development/JSOC/proj/export/webapps. The javascript is therefore in /home/jsoc/cvs/Development/JSOC/proj/export/webapps/exportdata.d. Given this structure, it is easy to set-up a development environment by creating two links (discussed in "How to Develop").

At the next lower level are the cgi programs. These programs, which are called via the ajax in exportdata.html, reside in /web/jsoc/cgi-bin/ajax (for external exports) and /web/jsoc2/cgi-bin/ajax (for internal exports). These programs are actually c-shell wrapper scripts that call programs in /home/jsoc/cvs/Development/bin/linux_x86_64. The scripts call the actual programs with the appropriate JSOC_DBHOST cmd-line argument. For external requests, the argument is set to hmidb2, and for internal requests, it is set to hmidb. This argument causes the program to connect to the appropriate database server when it accesses DRMS - hmidb2 contains public data series, and hmidb contains private data series.

Databases

The export system accesses various tables in the two databases (the jsoc database on hmidb, and the jsoc database on hmidb2) to perform an export. As previously mentioned, there are public and private data. These data are organized into data series, and some series are public, and others are private. Public data live in data series on hmidb, and private data live in data series on hmidb2. But there are other tables used by the export system. Each record in jsoc.export_new contains the data for a single, new export request. For each of these records, there is a corresponding record in jsoc.export, which contains data describing the final disposition of the export processing. Each time a user clicks on the "Submit Export Request" button in exportdata.html, a new record is inserted into jsoc.export_new. And when a record gets added to jsoc.export_new, a daemon sees this record and processes it, initiating the internal workflow for a single request. This workflow will create the corresponding record in jsoc.export, and modify it as the workflow proceeds. Upon completion of the workflow, the record in jsoc.export will contain the final status of the export, either "success", or some kind of failure. The databases contain a third table - jsoc.export_user. This table contains one record per export requestor. Each time a user submits a new export request, this table is updated. If the user is a repeat visitor, their record will be updated with the information provided on exportdata.html. This information is used during export processing, mostly as a way to send a notification to the user upon export completion.

These three database tables, jsoc.export_new, jsoc.export, and jsoc.export_user exist in the jsoc databases on both hmidb and hmidb2, for a total of six tables. Requests submitted from the external site result in the access of the three tables in hmidb2, whereas internal requests result in the access of the tables in hmidb.

The Way Things Work: Export-Request Workflow

When a user initiates an export request from one of the exportdata.html pages, the jsoc_info cgi-program is run. This program creates a record in the jsoc.export_new table in the

Supporting Javascript

In the exportdata.d directory are three files containing javascript: addOnTips.js, processing.js, and protocols.js. addOnTips.js contains the "tool tips" for HTML elements supported in other javascript files in this directory. The first argument to the Tip class constructor is the HTML element ID for the element to which the tip is tied. If a user "mouses over" the element, then the tied tool tip appears. Developers can add new tool tips as needed by adding new Tip constructors. processing.js contains the javascript callbacks for the HTML elements that are germane to the export processing steps (aia_scale, hg_patch, rebin). For example, if a user chooses hg_patch processing from the Processing drop-down list, an element is rendered that displays a list of input series. The javascript function that populates this list resides in processing.js. protocols.js serves an analogous purpose for the Protocol drop-down list.

To add a new processing step, a developer must create a minimum of three functions in the processing.js file: XxxxInit(), XxxxCheck(), XxxxSet(). For the rebin processing step, the functions are named "RebinInit", "RebinCheck", and "RebinSet". XxxxInit() is called when it is necessary to reset the variables associated with the processing step to their initial, default values. For example, it will be called when exportdata.html first loads in the browser. It is also called when the user switches between processing steps. Generally, a user will need to specify parameters for each processing step in sequence. They will need to select a processing step, parameterize it, then select the second processing step, and parameterize it. XxxxInit() is called when the user switches to the second processing step. At this point, XxxxInit() may need to reset certain variables or perform certain actions, such as hiding the HTML elements associated with the processing step the user switched away from. XxxxCheck() is called when the user clicks on the "Submit Export Request" button. It creates a comma-separated list of processing-program arguments for the currently selected processing step. XxxxSet() is called when the user changes a processing HTML, or clicks on it. For example, for the rebin processing step, the user can choose a rebin method (i.e., FWHM or nvector). When the user chooses a specific method, the UI must change since each method has a unique set of parameters that must be specified. So when the user chooses FWHM, the RebinInit() method is called, and the code disables the UI specific to the nvector rebin method, and enables the UI for the FWHM method.

The user must also add eleven lines of code to the ProcessingInit() function in processing.js. Here are the lines specific to the rebin processing step: 

  iOpt++;
  ProcessingOptionsHTML +=
    '<input type="checkbox" checked="false" value="rebin" id="OptionRebin" onChange="SetProcessing('+iOpt+');" /> ' +
    'rebin - Rebin with boxcar or gaussian smoothing<br>';
  ExpOpt = new Object();
  ExpOpt.id = "OptionRebin";
  ExpOpt.rowid = "ProcessRebin";
  ExpOpt.Init = RebinInit;
  ExpOpt.Check = RebinCheck;
  ExpOpt.Set = RebinSet;
  ExportProcessingOptions[iOpt] = ExpOpt;

iOpt is the index into the global array variable ExportProcessingOptions, which contains a list of processing-step objects. ProcessingOptionsHTML contains the HTML element descriptions that create checkboxes, one for each processing step. The user can select any subset of these processing steps with the UI created by ProcessingOptionsHTML.

Development

JsocWiki: ExportSystem (last edited 2013-05-01 04:35:24 by localhost)