= HMI/AIA Data Products =
== Data Product Information ==
 * '''[[Lev0Doc|Raw Telemetry and Level 0 Processing]]''' -- Raw images
 * '''[[Lev1Doc|Level 1+ Processing]]''' -- Level-1 filtergrams through level-1.5 == Observables
 * '''[[SpaceWeatherProducts|Browse and Quick-look Space Weather Products]]'''
 * '''Level 2 Pipeline Processing ''' -- Science Data Products
  * '''[[http://hmi.stanford.edu/teams/rings/|HMI Local Helioseismology: Ring Diagrams]]'''
  * '''[[MagneticField|HMI Magnetic Field Products]]'''
   * '''[[http://jsoc.stanford.edu/data/cov.html|Summary Data-Coverage Plots]]'''
  * '''[[HARPDataSeries|HARPs - HMI Active Region Patches]]'''
 * '''[[EventsOfInterest|Events of Special Interest]]''' -- Comets, Transits, very Active Regions, etc.
 * '''[[HMICalibrationInfo|HMI Calibration Information]]''' -- References relevant to calibration of the HMI data products
  * '''[[ModLRecalibration|Recalibration of Mod-L Vector Products Since 2016.04.13]]''' -- hmi.S_720s-Based Data Products Are Being Fixed to Account for Misalignment of Mod-L Filtergrams 
  * '''[[HMI2018AugustAnomaly|HMI instrument anomaly that appeared on 2018.08.17]]''' -- All HMI data products are to some extend changed by a change in the instrument.

== JSOC Processing Dataflow ==
=== HMI Pipeline Overview ===
[[http://jsoc.stanford.edu/Cmaps/web/HMI_Products.png|{{http://jsoc.stanford.edu/Cmaps/web/HMI_Products.png||width="500"}}]]

=== AIA Pipeline Overview ===
[[http://jsoc.stanford.edu/Cmaps/web/AIA_FORR.png|{{http://jsoc.stanford.edu/Cmaps/web/AIA_FORR.png||width="500"}}]]

=== Detailed JSOC Processing Flow Maps ===
The full processing pipeline can be followed via links on these flow maps:<<BR>>

 . [[http://jsoc.stanford.edu/Cmaps/web/JSOC_SDP_Data_Flow.html|{{http://jsoc.stanford.edu/Cmaps/web/JSOC_SDP_Data_Flow.jpg|alttag|width="400"}}]]

== Data Definitions ==
Most spacecraft mission data is typically divided into "levels" of processing and refinement.  We do the same.

 * '''"Raw"''' data are the basic telemetry transferred from the spacecraft to the ground station, packaged as a file, with only basic r/f integrity checks run, and no processing done to it.  Raw data are only useful as input to the "Level-0" extraction programs.
 * '''Level Zero''' refers to the lowest level of accessible data for a given instrument or spacecraft.
 . Telemetry Checking, APID (Application ID) Lookup, Housekeeping telemetry processing and Image Reconstruction are performed at this level.  The only processing done is typically the removal of duplicate data points (due to multiple file sends or a solid state recorder overlap), the removal or minimal repair of bad r/f data, the sorting of the different APIDs, and the matching of the end of one packet to the beginning of the next.
 . The images taken by an instrument are left "as is," i.e. not manipulated, not flat-fielded, nor otherwise calibrated.  In principle the level-0 data could be converted back to telemetry formatted data and is thus redundant with it, but is in a more easily usable form using standard data storage protocols.  All of the Level Zero data are recorded in the JSOC DRMS (database).  JSOC image data are stored in compressed FITS files.  JSOC metadata are stored in DRMS records but may be exported in several standard protocols (e.g. as part of exported FITS format images or tab delimited text tables).
 * '''Level One''' data are calibrated Level Zero data.  JSOC supports several steps of "Level-One" processing.
  * '''Level-1.0''' data are typically images converted from on-board form to regular array formats; bad pixels are removed and gains calibrated (i.e. flat-fielded).  Level 1.0 data are the cleanest state possible while remaining "pure," i.e. undergoingn no significant, irreversible data alterations.  Level 1.0 data are time-sequenced, and quality-filtered.
  * '''Level-1.5 data''' (for HMI) have been converted to physical observables such as Dopplergrams and Magnetograms which are constructed from many individual Filtergrams.
  * AIA '''Level-1''' data will be scaled and flat fielded with MTF corrections for vignetting and other such "always needed" processing applied.
 * '''Level Two''' data have been irrevocably filtered, time-sequence-merged, Fourier-transformed or otherwise changed from Level 0ne in a way that is irreversible.   Level Two data can be re-created from Level One data, but one cannot create Level One data from Level Two data.  Level-2 data are often used as intermediate products for later production of mission science data products such as helioseismic inferrences of solar subsurface flows.