Differences between revisions 3 and 19 (spanning 16 versions)
⇤ ← Revision 3 as of 2012-04-16 10:21:23 →
Size: 4438
Editor: lilo
Comment:
← Revision 19 as of 2014-05-23 06:53:52 → ⇥
Size: 6306
Editor: ToddHoeksema
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
== Release Notes for Initial Milne Eddington Inversion of HMI Magnetic Field Data (VFISV) ==
 
'''hmi.ME_720s_fd10''' ([http://jsoc.stanford.edu/ajax/lookdata.html?ds=hmi.ME_720s_fd10 click here for data]),
'''hmi.ME_720s_fd10_HARP''' ([http://jsoc.stanford.edu/ajax/lookdata.html?ds=hmi.ME_720s_fd10_HARP click here for data]), and
'''hmi.ME_720s_fd10_HARP_nrt''' ([http://jsoc.stanford.edu/ajax/lookdata.html?ds=hmi.ME_720s_fd10_HARP_nrt click here for data]).		 == Release Notes for hmi.ME_720s_fd10 HMI Magnetic Field Data (VFISV) ==
Definitive, science-grade, HMI vector magnetic field data are available as of May 2012 via JSOC under the series name hmi.ME_720s_fd10.
Line 7: Line 4:
These three data series contain ME inversion data processed using VFISV ([http://hmi.stanford.edu/doc/magnetic/section4.pdf details])
using the codes implemented on April 13, 2012.
There are known issues with these versions of VFISV, but this remains the best ME release version as of the date of implementation.
We consider this a beta-test version of the HMI vector field inversion.		 This release has now merged the full-disk and partial-disk ME data to one data series.
Line 12: Line 6:
 . The prime data products are field strength, inclination, and azimuth. Errors and other quantities are included.
 . No disambiguation has been performed, so azimuths are in the range 0-180, not 0-360.
 . fd10 in the series name is an index of the choice we think best among those we had tested, since [:ME720se15w1332: the last release].		 ==== Major differences between 2011-Oct/Nov version (_e15w1332 series) and the FD10 version of the VFISV Milne-Eddington Inversion ====
 * VFISV Algorithm Change
  1. A regularization term, whose purpose is to penalize high values of the parameter eta_0, was added to the merit function in order to tackle a double-minima problem.
  1. The selection process that decides when the inversion of a given pixel needs to be re-started with a different (randomized or tailored) initial guess atmosphere, has been implemented and fine-tuned.
  1. We have settled on a convergence criterion based on the value of the Levenberg-Marquardt Lambda parameter. The algorithm exits the iteration loop when Lambda becomes too large and no re-start of the inversion is enforced for any other reason. The average number of iterations per pixel is now about 30.
 * Implementation/Computing Change
  1. Parallelism, by means of MPI, has been modified and optimized to allocate the tasks to each Process Element as evenly as possible.
Line 16: Line 14:
Our efforts to optimize the VFISV for the HMI data pipeline are on going, and
we will create the series to fill the data processed by the newer codes when ready.
The eventual pipeline code for HMI vector inversions will be different.
We will announce the release of the new data through Solar News.		 ==== Temporal Coverage for Three Series below ====
'''[[http://jsoc.stanford.edu/doc/data/hmi/coverage_maps/fd10/|Monthly charts]]''' show the 12-minute time slots for which the ME inversion data processed with the VFISV module are available. The coverage charts also contain the status flags of the HMI observation, thus the quality and availability of input [[http://jsoc.stanford.edu/ajax/lookdata.html?ds=hmi.S_720s|Stokes]] data. See [[ME720se15w1332CoverChart|details]] for an explanation of the coverage charts (* the color and legends in the coverage charts for version FD10 are same as those for the [[ME720se15w1332|previous version]] of ME data).
Line 21: Line 17:
==== Temporal Coverage for These Three Series ====
'''[http://jsoc.stanford.edu/doc/data/hmi/coverage_maps/inversion/ Monthly charts]''' show the 12-minute time slots for which the ME inversion data processed with the VFISV module are available. The coverage charts also contain the status flags of the HMI observation, thus the quality and availability of input [http://jsoc.stanford.edu/ajax/lookdata.html?ds=hmi.S_720s Stokes] data.
See [:ME720se15w1332CoverChart: details] for an explanation of the coverage charts.		  . ''' [[http://jsoc.stanford.edu/ajax/lookdata.html?ds=hmi.ME_720s_fd10|hmi.ME_720s_fd10]] '''
 . ''' [[http://jsoc.stanford.edu/ajax/lookdata.html?ds=hmi.ME_720s_fd10_nrt|hmi.ME_720s_fd10_nrt]] '''
Line 25: Line 20:
[http://sun.stanford.edu/~keiji/HMI/nprcs/invnprcs.html A temporal version] is avaiable, until we will fully adjust scripts making the formal, informative monthly chart. These are now obsolete:
Line 27: Line 22:
 * '''hmi.ME720s_fd10 - ''Full Disk'' '''
  . Synoptic data, on twice per day basis at 7 and 19UT daily, are stored in this series. The latest version of VFISV is being used to extend this series using the latest definitive Stokes data, on the 2 per day basis, usually about a week after the observation. Data for scattered additional time intervals are also available. Checking the version of VFISV codes is strongly recommended.		  . ''' [[http://jsoc.stanford.edu/ajax/lookdata.html?ds=hmi.ME_720s_fd10_HARP|hmi.ME_720s_fd10_HARP]]) '''
 . ''' [[http://jsoc.stanford.edu/ajax/lookdata.html?ds=hmi.ME_720s_fd10_HARP_nrt|hmi.ME_720s_fd10_HARP_nrt]] '''
Line 30: Line 25:
 * '''hmi.ME720s_fd10_HARP - ''HMI Active Region Patches'' '''
  . The latest version of VFISV has been used to generate the ME data stored in this series. Many of the active regions were recognized automatically by the [:HARPDataSeries:HARP module], while some were defined manually. This series includes 12-minute (720s) cadence data for several of the ARs that created X-class flares in 2011, including the one occurring on 2011 February 15 (at NOAA AR11158). Additional time intervals are described at ReleaseNotes2.		 These data series contain ME inversion data processed using an adaptation of VFISV ([[http://link.springer.com/article/10.1007/s11207-014-0497-7|Solar Physics Paper]]) using the codes implemented on April 13, 2012 and modified as described below. There are known issues with these versions of VFISV, but this remains the preferred ME release version as of the date of implementation.
Line 33: Line 27:
 * '''hmi.ME720s_fd10_HARP_nrt - ''Near-Real-Time HMI Active Region Patches'' '''
  . The latest version of VFISV has been used to generate the ME data stored in this series. The input Stokes data and the HARP info. are of preliminary though the time lag between observation and ME process is minimized. Current setting is to start, at 7PM Pacific Time, one-per-hour run for the same day.		  * The prime data products are field strength, inclination, and azimuth. Errors and other quantities are included.
 * No disambiguation has been performed, so azimuths are in the range 0-180, not 0-360 (See FullDiskDisamb).
 * fd10 in the series name is an index of the choice we think best among those we have tested, since [[ME720se15w1332|the last release]].
Line 37: Line 32:
Line 39: Line 33:
  * The ME data for the series above were processed with the latest version, labeled with a date, "Apr 13, 2012".
  * In IDL, for example, if the exported data is read with [http://hmi.stanford.edu/doc/magnetic/fitsio.pdf fitsio_read_image()] then, the version keyword can be printed with sxpar(): 		   * The ME data for the series above were processed with the latest version, labeled with a date, e.g. "Apr 13, 2012".
  * In IDL, for example, if the exported data is read with [[http://hmi.stanford.edu/doc/magnetic/fitsio.pdf|fitsio_read_image()]] then, the version keyword can be printed with sxpar():
Line 43: Line 37:
    IDL> print, sxpar(header,'INVCODEV') }}}     IDL> print, sxpar(header,'INVCODEV')
}}}
Line 45: Line 40:
 * '''Major differences between 2011-October-14 (or 2011-November-27) version and the current FD10 version'''
  * VFISV core part
   1. Regularization of residual.
   2.
   3. Parallelism, by means of MPI, had been modified, optimized to allocate the tasks to each Process Element as evenly as possible.		 === Changes in VFISV, since the "April 13 2012" version. ===
As results of continuing improvements, there are some updates in the VFISV codes since the "April 13, 2012" version was used to make regular data production. Below are summarized the changes. All changes were made in the C-wrapper, and no changes have been made in the VFISV core logic or procedures. The date of the change appears in the INVCODEV keyword as a code version identifier.

 * 2012 Jun. 23
  . The wrapper was modified to host a new run-time option "-f": This option forces the VFISV module to run even when a processed data for a time slot exists. This option was made to process the full disk VFISV for a time slot for which the VFISV had been applied to the HARP (part of the fulldisk) and the partially processed data resided on the target data slot. No changes in the VFISV code itself.
 * 2012 Nov. 28
  . A few minor changes in C-language to adapt to the latest version of the intel compiler. Information of a paper regarding the VFISV implemented to the JSOC pipeline is updated, i.e."Solar Physics 2011, vol 273 pp 267-293, Borrero et.al." No changes in the VFISV itself.
 * On 2013 Mar. 13
  . A few minor changes, for preparing for very very rare cases in the DSDS data file system when the input Stokes data files are not on the disk. No changes in the VFISV itself.
 * On 2013 Apr. 30
  . Two changes. The first one might affect the output from the VFISV. (1) The instrumental phasemap determined from the calibration process and used in VFISV now depends on the time of the observation. This change results in systematic differences between the data made with different phasemaps, though well within the estimated error level. A string keyword INVPHMAP tells which phasemap is used in the VFISV processesing. The VFISV wrapper versions earlier than "2013 Apr. 30", this keyword had been left blank (or may appear as a one-character space). In the new version, the string keyword will have a value of 7 or 8-digit number identifying the phasemap data stored in another JSOC internal data series, hmi.phasemaps_corrected[]. (2) The string keywords for unit of physical variables had been updated: For example, from "degree" to "deg", from "gauss" to "Mx/cm^2", from "adim" to "dimensionless".

=== References ===
 . [[http://link.springer.com/article/10.1007/s11207-014-0516-8|Vector Pipeline]]
 . [[http://link.springer.com/article/10.1007/s11207-014-0529-3|SHARPs]]

=== Links ===
 . PipelineCode
 . MagneticField
 . JSOC FrontPage

Release Notes for hmi.ME_720s_fd10 HMI Magnetic Field Data (VFISV)

Definitive, science-grade, HMI vector magnetic field data are available as of May 2012 via JSOC under the series name hmi.ME_720s_fd10.

This release has now merged the full-disk and partial-disk ME data to one data series.

Major differences between 2011-Oct/Nov version (_e15w1332 series) and the FD10 version of the VFISV Milne-Eddington Inversion

  • VFISV Algorithm Change
    1. A regularization term, whose purpose is to penalize high values of the parameter eta_0, was added to the merit function in order to tackle a double-minima problem.
    2. The selection process that decides when the inversion of a given pixel needs to be re-started with a different (randomized or tailored) initial guess atmosphere, has been implemented and fine-tuned.
    3. We have settled on a convergence criterion based on the value of the Levenberg-Marquardt Lambda parameter. The algorithm exits the iteration loop when Lambda becomes too large and no re-start of the inversion is enforced for any other reason. The average number of iterations per pixel is now about 30.
  • Implementation/Computing Change
    1. Parallelism, by means of MPI, has been modified and optimized to allocate the tasks to each Process Element as evenly as possible.

Temporal Coverage for Three Series below

Monthly charts show the 12-minute time slots for which the ME inversion data processed with the VFISV module are available. The coverage charts also contain the status flags of the HMI observation, thus the quality and availability of input Stokes data. See details for an explanation of the coverage charts (* the color and legends in the coverage charts for version FD10 are same as those for the previous version of ME data).

These are now obsolete:

These data series contain ME inversion data processed using an adaptation of VFISV (Solar Physics Paper) using the codes implemented on April 13, 2012 and modified as described below. There are known issues with these versions of VFISV, but this remains the preferred ME release version as of the date of implementation.

  • The prime data products are field strength, inclination, and azimuth. Errors and other quantities are included.

  • No disambiguation has been performed, so azimuths are in the range 0-180, not 0-360 (See FullDiskDisamb).

  • fd10 in the series name is an index of the choice we think best among those we have tested, since the last release.

Versions of VFISV in the FITS files exported at JSOC database

  • The VFISV version is stored as a string value in the keyword INVCODEV.

    • The ME data for the series above were processed with the latest version, labeled with a date, e.g. "Apr 13, 2012".

    • In IDL, for example, if the exported data is read with fitsio_read_image() then, the version keyword can be printed with sxpar(): 

    •     IDL> img=fitsio_read_image('img.fits',header)
    IDL> print, sxpar(header,'INVCODEV')

Changes in VFISV, since the "April 13 2012" version.

As results of continuing improvements, there are some updates in the VFISV codes since the "April 13, 2012" version was used to make regular data production. Below are summarized the changes. All changes were made in the C-wrapper, and no changes have been made in the VFISV core logic or procedures. The date of the change appears in the INVCODEV keyword as a code version identifier.

  • 2012 Jun. 23
    • The wrapper was modified to host a new run-time option "-f": This option forces the VFISV module to run even when a processed data for a time slot exists. This option was made to process the full disk VFISV for a time slot for which the VFISV had been applied to the HARP (part of the fulldisk) and the partially processed data resided on the target data slot. No changes in the VFISV code itself.
  • 2012 Nov. 28
    • A few minor changes in C-language to adapt to the latest version of the intel compiler. Information of a paper regarding the VFISV implemented to the JSOC pipeline is updated, i.e."Solar Physics 2011, vol 273 pp 267-293, Borrero et.al." No changes in the VFISV itself.
  • On 2013 Mar. 13
    • A few minor changes, for preparing for very very rare cases in the DSDS data file system when the input Stokes data files are not on the disk. No changes in the VFISV itself.
  • On 2013 Apr. 30
    • Two changes. The first one might affect the output from the VFISV. (1) The instrumental phasemap determined from the calibration process and used in VFISV now depends on the time of the observation. This change results in systematic differences between the data made with different phasemaps, though well within the estimated error level. A string keyword INVPHMAP tells which phasemap is used in the VFISV processesing. The VFISV wrapper versions earlier than "2013 Apr. 30", this keyword had been left blank (or may appear as a one-character space). In the new version, the string keyword will have a value of 7 or 8-digit number identifying the phasemap data stored in another JSOC internal data series, hmi.phasemaps_corrected[]. (2) The string keywords for unit of physical variables had been updated: For example, from "degree" to "deg", from "gauss" to "Mx/cm^2", from "adim" to "dimensionless".

References

JsocWiki: ME720sfd10 (last edited 2014-05-23 06:53:52 by ToddHoeksema)