• JsocDataAccess

User Access to JSOC Managed Data

Web access to data

The lookdata.html and exportdata.html web tools allow browsing metadata, image data, and export from JSOC to the user. There is a tutorial document also linked to the main JSOC page: tour.pdf. (An older doc remains at [LookData lookdata wiki page]). Lookdata can be used to browse names of DataSeries, lists of available keyword metadata for each series, to examine metadata for specifiable RecordSets, and ilink to exportdata to export user selected RecordSets in several forms. lookdata.html written in javascript Ajax style and the cgi-bin access API described at AjaxJsocConnect in this Wiki.

• Note that there are two entry points into the JSOC, jsoc2.stanford.edu for development which allows access to all levels of processing and test seriesa and thus needs a password. The other at jsoc.stanford.edu contains the "published" data series that include the SDO mission primary data products. Both access ports use the same lookdata and exportdata tools.

Querying JSOC data with Python

The easiest way to query JSOC data via Python is by using a module called drms.

• The source code for drms is available on Github, where there are also many examples included in the examples subdirectory, or on PyPI.

• The drms manual is available at Read The Docs, which also contains a detailed tutorial.

• There is also an ipython notebook that shows how to use the drms module to plot, query, visualize, and export the data in a variety of formats.

Using JSOC data in IDL programs

• If you wish to use IDL SolarSoft Sam Freeland has developed tools for access JSOC data. This interface is described at http://www.lmsal.com/solarsoft/jsoc/ssw_jsoc_routines.html.

• If you wish to access HMI or AIA data via SolarSoft from the remote DRMS/SUMS sites that are also VSO sites these notes may help:

  • Access to HMI and AIA data are also available from remote sites by using the keyword 'site' in IDL vso_get command. For example, site= 'cfa' will get them from CfA, Harvard, site='rob' will get data from ROB. European users should see also http://wissdom.oma.be.

• If you need to run in IDL on a local (to Stanford or Lockheed) machine, see Jesper Schou who has developed techniques for accessing the data via shell commands.
• Note, we no longer plan a full API for IDL. But there are several approaches to use for access to JSOC data from within IDL programs. At present there is no way to insert data into DRMS/SUMS from IDL other than shell level commands. There is now no plan to run IDL programs in the JSOC "Pipeline" processing. There can never be interactive IDL jobs in the main JSOC computing cluster since the only way to run programs is via the batch queue system.

Local access to data

JSOC data and metadata can be examined at the "shell" level via several general purpose commands. show_series and show_info are the most common and provide access to data series lists and all the information contained in a particular series respectively. There are "man" pages for both of these and other commands in the Doxygen generated documents at http://jsoc.stanford.edu/doxygen_html/ In the future there will be examples here.

Becoming a Remote DRMS

For some institutions, setting up a system or local network running NetDRMS may be beneficial. This is an involved commitment, and requires the establishment and configuration of a local Postgres database server, and the bulk data storage system. These should be one-time tasks for someone with administrative privileges on the systems on which these services will run. Once the services have been established, any user should be able to set up their own NetDRMS development environment. Becoming a netDRMS would be beneficial in cases where automated, regular data delivery for specific data series is needed, or if the institution plans to become a local download hub for others interested in JSOC data.

Learn more about netDRMS here: http://jsoc.stanford.edu/netdrms/
If you are already a netDRMS and need to get data clusters, please read: SetupForSlonyData

Using JSOC data in c programs

The primary way to access JSOC data is via c programs using the DRMS API. There are a growing list of examples in the CVS software tree. Applications are found in the JSOC/proj subtree. In the future we will have a link to tutorials and examples here.

Using JSOC data in FORTRAN programs

We do support a FORTRAN API for DRMS. It provides most basic functions. One can write FORTRAN main programs and use this API or (and probably preferable) write a c main program using the c DRMS API and call FORTRAN computational functions. Examples of both methods will be linked here in the future.

Using JSOC data in MATLAB programs

We have a basic set of matlab tools to allow access to JSOC data via the cgi-bin ajax style interface used for the web tools and IDL interface. A description of the Matlab tools is at access_jsoc_from_matlab.pdf. The matlab .m files described in the above document are in http://jsoc.stanford.edu/matlab/m/.