• ME720sfd10new

Additional info/descriptions regarding hmi.ME_720s_fd10 HMI Magnetic Field Data (VFISV), between June 2 and July 12, 2012

Release Notes for hmi.ME_720s_fd10 HMI Magnetic Field Data (VFISV)

Definitive, science-grade, HMI vector magnetic field data are available via JSOC under the series name hmi.ME_720s_fd10.

hmi.ME_720s_fd10 ([http://jsoc.stanford.edu/ajax/lookdata.html?ds=hmi.ME_720s_fd10 click here for definitive data])

*) Please note that, we had merged the full-disk data and partial (HARP) data to one series, after the announcement made on June 1st. We also made a few changes in the keyword (at FITS header field). The changes are limited to the keyword field, and the data values stored in these two series must be same as those released/announced on 2012 June 1.

This data series contains ME inversion data processed using VFISV ([http://hmi.stanford.edu/doc/magnetic/section4.pdf details]) using the VFISV codes implemented on April 13, 2012. There are known issues with these versions of VFISV, but this remains the best ME release version as of the date of implementation. We consider this a beta-test version of the HMI vector field inversion.

• The prime data products are field strength, inclination, and azimuth. Errors and other quantities are included.
• No disambiguation has been performed, so azimuths are in the range 0-180, not 0-360.
• fd10 in the series name is an index of the choice we think best among those we had tested, since [:ME720se15w1332: the last release].
• Synoptic data, on 4 per day basis at 1, 7, 13 and 19UT daily since 2010 May 1st, are stored in this series. Since 2012 April 25, the latest version of VFISV is being used to extend this series using the latest definitive Stokes data, on one-per-12 minute basis, usually about a week after the observation. Data for scattered additional time intervals, one of which is from 2011 February 1st to March 10, are also available.

• Many of the active regions were recognized automatically by the [:HARPDataSeries:HARP module]. This series includes ME data for the HARP-recognized active regions, on a one per hour basis, for most of the period from 2010 May 1st until 2012 April 25, to supplement full-disk data. The ME VFISV for HARP regions are more intensively coducted, on 12-minute (720s) cadence, for several of the ARs that created X-class flares in 2011, including the one occurring on 2011 February 15 (at NOAA AR11158).

Our efforts to optimize the VFISV for the HMI data pipeline are on going, and we will create the series to fill the data processed by the newer codes when ready. The eventual pipeline code for HMI vector inversions will be different. We will announce the release of the new data through Solar News.

Major differences between 2011-Oct/Nov version (_e15w1332 series) and the FD10 version (announced on June 1, 2012)

• VFISV Algorithm Change
  1. A regularization term, whose purpose is to penalize high values of the parameter eta_0, was added to the merit function in order to tackle a double-minima problem.
  2. The selection process that decides when the inversion of a given pixel needs to be re-started with a different (randomized or tailored) initial guess atmosphere, has been implemented and fine-tuned.
  3. We have settled on a convergence criterion based on the value of the Levenberg-Marquardt Lambda parameter. The algorithm exits the iteration loop when Lambda becomes too large and no re-start of the inversion is enforced for any other reason. The average number of iterations per pixel is now about 30.
• Implementation/Computing Change
  1. Parallelism, by means of MPI, has been modified and optimized to allocate the tasks to each Process Element as evenly as possible.

Major differences between the latest version (as of July 12) and the one announced on June 1

• NO changes in VFISV algorithm, thus, NO changes in the data values.
• The full-disk and HAPR data that used to be stored at separate series are now saved in one series
• The FITS files generated at export.html website will have 4 additional keywords:
  1. Three keywords, DATAVALS, DATAMEAN, and DATARMS, are for number of data pixel, mean of data values and the RMS of data values, respectively. These three will be calculated for each data segment.
  2. One keyword, NHARP, is for number of HARP rectangle boxes for which the ME VFISV processed the input Stokes data. Zero stands for full-disk data, to make which no HARP info. is used.

Temporal Coverage Charts for Three Series below

[http://jsoc.stanford.edu/doc/data/hmi/coverage_maps/fd10/ Monthly charts] show the 12-minute time slots for which the ME inversion data processed with the VFISV module are available. The coverage charts also contain the status flags of the HMI observation, thus the quality and availability of input [http://jsoc.stanford.edu/ajax/lookdata.html?ds=hmi.S_720s Stokes] data. See [:ME720se15w1332CoverChart: details] for an explanation of the coverage charts (* the color and legends in the coverage charts for version FD10 are same as those for the [:ME720se15w1332: previous version] of ME data).

Data files generated before June 1, with old set of keywords

Having consolidated the full-disk and partial data to one series, we will no longer fill two HARP-only series: hmi.ME_720s_fd10_HARP ([http://jsoc.stanford.edu/ajax/lookdata.html?ds=hmi.ME_720s_fd10_HARP click here for data]), and hmi.ME_720s_fd10_HARP_nrt ([http://jsoc.stanford.edu/ajax/lookdata.html?ds=hmi.ME_720s_fd10_HARP_nrt click here for data]). For a while, we will keep these two series available online for user's convenience. Here again, the data values stored in these two series must be same as those released/announced on 2012 June 1.

Versions of VFISV in the FITS files exported at JSOC database

• The VFISV version is stored as a string value in the keyword INVCODEV.

  • The ME data for the series above were processed with the latest version, labeled with a date, "Apr 13, 2012" or later.

  • In IDL, for example, if the exported data is read with [http://hmi.stanford.edu/doc/magnetic/fitsio.pdf fitsio_read_image()] then, the version keyword can be printed with sxpar():

  •     IDL> img=fitsio_read_image('img.fits',header)
        IDL> print, sxpar(header,'INVCODEV') 

Finding the *full-disk* data out of the marged data set

For the database interface to distinguish whether a data record contains the full disk data or the partial one, we prepared a few ways:

First, goto the [http://jsoc.stanford.edu/ajax/lookdata.html?ds=hmi.ME_720s_fd10 website interface for ME definitive data]

A method we recommend: At the 3rd box for "Select Records and Get Record Count", append a conditional filter to pick up the data records with NHARP (number of HARP regions processed) of zero value:

   hmi.ME_720s_fd10[2011.11.11_11:00:00_TAI/3d@1h][? nharp=0 ?]

An alternative method: Similarily to the recommended method, append a conditional filter with INVNPRCS (number of pixels processed by ME VFISV) greater than, for example, 10 million:

   hmi.ME_720s_fd10[2012.03.04_05:00:00_TAI/2h][? invnprcs > 1e7 ?]