Differences between revisions 25 and 26

Calibration Version keywords - CALVER32 and CALVER64

Keywords describing the HMI calibration version have been included in level-1 data (CALVER32) and higher level products (CALVER64) starting in August 2012. This keyword is a set of fields consisting of single hexidecimal digits. Each nibble has a meaning described in the following table.

The presently assigned values are from least significant nibble:

Field	bits	mask	name	Note
0	0-3	0x0F	HFCORRVR	Height-of-formation code version used to find disk center. Vers 2 for all data to date.
1	4-7	0xF0	CROTA2VR	Version of CROTA2 in the Master pointing table. Vers 0 prior to Transit of Venus, then 1.
2	8-11	0xF00	N/A	If > 0, then smooth look-up tables were used to produce observables
3	12-15	0xF000	N/A	If > 0, then a correction for non-linearity of the CCDs was applied. If 0x1000, the polynomial coefficients used for this correction are: -8.2799134,0.017660396,-3.7157499e-06,9.0137137e-11 (for the side camera) and -11.081771,0.017383740,-2.7165221e-06,6.9233459e-11 (for the front camera). If 0x2000, the polynomial coefficients used for this correction are: 0.0,0.025409177,-4.0088672e-06,1.0615198e-10 (side camera) and 0.0,0.020677687,-3.1873243e-06,8.7536678e-11 (front camera)
4	16-19	0xF0000	N/A	If 0, then observing sequence was 'mod C'; mod C was the standard observing sequence before 2016.04.13_19:12:55.11_UTC, FSN=104683793.
See Calver64ModLNote				If 2 then original deprecated Mod L processing
Also see Note 2				If 3 then mod-L misalignment corrected
				If 4 then misalignment and filtergram selection correct
5	20-23	0xF00000	N/A	If > 0 then PSF/scattered light deconvolution has been done, 1==CUDA version, 2== C version
6	24-27	0xF000000	N/A	If > 0, then a rotational flat field was used. The code in HMI_observables and HMI_IQUVaveraging checks only on the bit 0x1000000 (originally field 4 was used for this, changed in Nov 2016)

The default value of CALVER64 is 0 and should never be seen so long as QUALITY >= 0. The concept of a missing value for a CALVERxx field does not make sense since CALVERxx refers to the version of code or source of values vs the decisions made by that code. If for instance the limb fit fails in lev1 processing, HFCORRVR still should contain the code version but the appropriate QUALITY bit should show that that code could not produce a useful value. Higher level products may propagate CALVER64. If a new product record is made from multiple input records then that code must decide the range of input CALVER that are acceptable and if different versions are OK for the output product, which value to forward. A good choice would be to send forward the lowest level from the input records to indicate that older versions were used in the output product. This could be used later to decide which products should be updated when the input series itself is updated.

Notes

, Data products above hmi.lev1 do not have CALVER64 set for records with no usable data. So if QUALITY < 0 you should expect CALVER64 to be 0.
Due to a bug in the program that updated existing records, for data made prior to 5 Sept 2012 the HFCORRVR values was not set, so values of 0 and 2 in that field should be taken to mean 2. I.e. since to date, only one version of the height of formation code was in use, HFCORRVR of 0 and 2 both mean the same thing. After a lot of discussion we agreed to not update the older ~10M records at this time. They will be corrected when the next calibration update is applied. (PHS 1 Oct 2012).

JSD entries for CALVER32 and CALVER64:

Keyword:CALVER32, int, variable, record, 0, "0x%08X", "none", "Calibration Version lev1"
Keyword:CALVER64, longlong, variable, record, 0, "0x%016llX", "none", "Calibration Version"

Prior to Aug 2012 hmi.lev1 had a keyword named HFCORRVR which can be taken as correct for fields 0 and 1 except that a 0 in the first nibble (field 0) should be changed to 2 to match the other records made with same code.

In the process of adding CALVER64 and CALVER32 for existing records these keywords should be set to 0x12 when updated with the post-Transit-of-Venus CROTA2 values. The plan for doing the August 2012 update is described at Crota2UpdatePlan.

Starting 1 Oct 2012 fields 2 and 3 are populated. The older data have not been reprocessed (as of 15 Jan 2013).

Note 2: Mod C and Mod L Processing

hmi.S_720s users:

Check the value of CALVER64 && 0xF0000 to make sure that you use data with the appropriate processing! Good data has a value for that nibble of 0 or 4.

Regular use of the HMI mod-L frame list began at 2016.04.13_19:12:55 (fns=104683793). Mod-L requires use of filtergrams from both HMI cameras to compute the Stokes vector products, hmi.S_720s. Errors in the Mod-L processing were discovered in July and September 2017 that were fixed in September and October 2017. Reprocessing of the data is underway. Details are provided at Calver64ModLNote.

-  ⇤ ← Revision 25 as of 2017-10-13 09:23:31 → 
  Size: 7502
  Editor: ToddHoeksema
  Comment:
+   ← Revision 26 as of 2017-10-14 02:11:50 → ⇥
  Size: 5235
  Editor: ToddHoeksema
  Comment:
-Deletions are marked like this.
+Additions are marked like this.
 Line 1:
-Calibration Version keywords
+= Calibration Version keywords - CALVER32 and CALVER64 =
 Line 3:
-A keyword for calibration version keywords are included in level-1 data (CALVER32) through higher level products (CALVER64) starting in August 2012.  This keyword is, at the start, considered to be a set of fields consisting of single hexidecimal digits.
+Keywords describing the HMI calibration version have been included in level-1 data (CALVER32) and higher level products (CALVER64) starting in August 2012.  This keyword is a set of fields consisting of single hexidecimal digits. Each nibble has a meaning described in the following table.
 Line 7:
-||0 ||0-3 ||0x0F ||HFCORRVR ||Height of formation code version used to find disk center. Vers 2 for all data to date. ||
||1 ||4-7 ||0xF0 ||CROTA2VR ||Version of CROTA2 in the Master pointing table.  Vers 0 prior to Transit of Venus, then 1. ||
||2 ||8-11 ||0xF00 ||N/A ||If > 0, then smooth look-up tables were used to produce observables ||
||3 ||12-15 ||0xF000 ||N/A ||If > 0, then a correction for non-linearity of the CCDs was applied. If 0x1000, the polynomial coefficients used for this correction are:   -8.2799134,0.017660396,-3.7157499e-06,9.0137137e-11 (for the side camera) and -11.081771,0.017383740,-2.7165221e-06,6.9233459e-11 (for the front camera). If 0x2000, the polynomial coefficients used for this correction are:     0.0,0.025409177,-4.0088672e-06,1.0615198e-10 (side camera)     and 0.0,0.020677687,-3.1873243e-06,8.7536678e-11 (front camera) ||
||4 ||16-19 ||0xF0000 ||N/A ||If > 0, then observing sequence not 'modC'; mod C was the standard sequence until 2016.04.13_19:12:55.11_UTC, FSN=104683793. ||
|||||||| See Note 2 || If 2 then original deprecated Mod L processing ||
|||||||| See Note 2 || If 3 then mod-L misalignment corrected ||
|||||||| See Note 2 || If 4 then misalignment and filtergram selection correct ||
||5 ||20-23 ||0xF00000 ||N/A ||If > 0 then PSF/scattered light deconvolution has been done, 1==CUDA version, 2== C version ||
||6 ||24-27 ||0xF000000 ||N/A || If > 0, then a rotational flat field was used. The code in HMI_observables and HMI_IQUVaveraging checks only on the bit 0x1000000 (originally field 4 was used for this, changed in Nov 2016)  ||
+||0 ||  0-3 ||    0x0F ||HFCORRVR ||Height-of-formation code version used to find disk center. Vers 2 for all data to date. ||
||1 ||  4-7 ||    0xF0 ||CROTA2VR ||Version of CROTA2 in the Master pointing table.  Vers 0 prior to Transit of Venus, then 1. ||
||2 || 8-11 ||   0xF00 ||N/A || If > 0, then smooth look-up tables were used to produce observables ||
||3 ||12-15 ||  0xF000 ||N/A || If > 0, then a correction for non-linearity of the CCDs was applied. If 0x1000, the polynomial coefficients used for this correction are:   -8.2799134,0.017660396,-3.7157499e-06,9.0137137e-11 (for the side camera) and -11.081771,0.017383740,-2.7165221e-06,6.9233459e-11 (for the front camera). If 0x2000, the polynomial coefficients used for this correction are:     0.0,0.025409177,-4.0088672e-06,1.0615198e-10 (side camera)     and 0.0,0.020677687,-3.1873243e-06,8.7536678e-11 (front camera) ||
||4 || 16-19 || 0xF0000 ||N/A ||If 0, then observing sequence was 'mod C'; mod C was the standard observing sequence before 2016.04.13_19:12:55.11_UTC, FSN=104683793. ||
|||||||| See [[Calver64ModLNote]] || If 2 then original deprecated Mod L processing ||
|||||||| Also see Note 2 || If 3 then mod-L misalignment corrected ||
|||||||| || If 4 then misalignment and filtergram selection correct ||
||5 ||20-23 || 0xF00000  ||N/A ||If > 0 then PSF/scattered light deconvolution has been done, 1==CUDA version, 2== C version ||
||6 ||24-27 || 0xF000000 ||N/A || If > 0, then a rotational flat field was used. The code in HMI_observables and HMI_IQUVaveraging checks only on the bit 0x1000000 (originally field 4 was used for this, changed in Nov 2016)  ||
 Line 19:
+The default value of CALVER64 is 0 and should never be seen so long as QUALITY >= 0.
The concept of a missing value for a CALVERxx field does not make sense since CALVERxx refers to the version of code or source of values vs the decisions made by that code.
If for instance the limb fit fails in lev1 processing, HFCORRVR still should contain the code version but the appropriate QUALITY bit should show that that code could not produce a useful value. 
Higher level products may propagate CALVER64.
If a new product record is made from multiple input records then that code must decide the range of input CALVER that are acceptable and if different versions are OK for the output product, which value to forward.
A good choice would be to send forward the lowest level from the input records to indicate that older versions were used in the output product.
This could be used later to decide which products should be updated when the input series itself is updated.
-Line 20:
+Line 27:
+=== Notes ===
-Line 21:
+Line 29:
-The default value of CALVER64 is 0 and should never be seen so long as QUALITY >= 0.  The concept of a missing value for a CALVERxx field does not make sense since CALVERxx refers to the version of code or source of values vs the decisions made by that code.  If for instance the limb fit fails in lev1 processing, HFCORRVR still should contain the code version but the appropriate QUALITY bit should show that that code could not produce a useful value. Higher level products may propagate CALVER64. If a new product record is made from multiple input records then that code must decide the range of input CALVER that are acceptable and if different versions are OK for the output product, which value to forward.  A good choice would be to send forward the lowest level from the input records to indicate that older versions were used in the output product.  This could be used later to decide which products should be updated when the input series itself is updated.

'''NOTE:''' Data products above hmi.lev1 do not have CALVER64 set for records with no usable data.  So if QUALITY < 0 you should expect CALVER64 to be 0.  Also due to a bug my program that updated existing records, for data made prior to 5 Sept 2012 the HFCORRVR values was not set, so values of 0 and 2 in that field should be taken to mean 2.  I.e. since to date, only one version of the height of formation code was in use, HFCORRVR of 0 and 2 both mean the same thing.  After a lot of discussion we agreed to not update the older ~10M records at this time.  They will be corrected when the next calibration update is applied.  (PHS 1 Oct 2012).
+ , Data products above hmi.lev1 do not have CALVER64 set for records with no usable data.  So if QUALITY < 0 you should expect CALVER64 to be 0.  
 . Due to a bug in the program that updated existing records, for data made prior to 5 Sept 2012 the HFCORRVR values was not set, so values of 0 and 2 in that field should be taken to mean 2.  I.e. since to date, only one version of the height of formation code was in use, HFCORRVR of 0 and 2 both mean the same thing.  After a lot of discussion we agreed to not update the older ~10M records at this time.  They will be corrected when the next calibration update is applied.  (PHS 1 Oct 2012).
-Line 27:
+Line 34:
-'''Keyword:CALVER32, int, variable, record, 0, "0x%08X", "none", "Calibration Version lev1" '''
+ . '''Keyword:CALVER32, int, variable, record, 0, "0x%08X", "none", "Calibration Version lev1" '''
-Line 29:
+Line 36:
-'''Keyword:CALVER64, longlong, variable, record, 0, "0x%016llX", "none", "Calibration Version" '''
+ . '''Keyword:CALVER64, longlong, variable, record, 0, "0x%016llX", "none", "Calibration Version" '''
-Line 37:
+Line 44:
-=== Note 2: Mod C and Mod L Processing ===
Regular use of the HMI mod-L frame list began at 2016.04.13_19:12:55 (fns=104683793). Mod-L requires use of filtergrams from both HMI cameras to compute the Stokes vector products. Errors in the processing were discovered in July and September 2017 that were fixed in September and October 2017. Reprocessing of the data are underway. Details are provided below.
+==== Note 2: Mod C and Mod L Processing ====
-Line 40:
+Line 46:
-    '''Check the value of CALVER64 to make sure that you use data with the appropriate processing!'''
+hmi.S_720s users:
   '''Check the value of CALVER64 && 0xF0000 to make sure that you use data with the appropriate processing!'''
   '''Good data has a value for that nibble of 0 or 4.'''
-Line 42:
+Line 50:
-The first bug discovered in Mod-L processing was the more serious because filtergrams were not properly aligned; the difference in p-angle rotation of about 0.08 degrees between the two cameras was not corrected. This resulted in misalignment of about 3 pixels at the limbs and most severely contaminated the polar field and other limb measurements that combine data from the two cameras. Data with this error should only be used with extreme caution.
+Regular use of the HMI mod-L frame list began at 2016.04.13_19:12:55 (fns=104683793). Mod-L requires use of filtergrams from both HMI cameras to compute the Stokes vector products, hmi.S_720s. Errors in the Mod-L processing were discovered in July and September 2017 that were fixed in September and October 2017. Reprocessing of the data is underway. Details are provided at [[Calver64ModLNote]].
-Line 44:
+Line 52:
-The second bug resulted in the use of only half of the available 'I+V' and 'I-V' filtergrams. In mod-L two complete sets of 'I +/- V' filtergrams are collected each 90s, but the filtergram selection was not correct; only half of the available I+V and I-V filtergrams were used and they were used twice. As a consequence the expected noise in the computed values was lower than the actual noise; the 90-s mod-L values that depend on V had the same noise as the original 135-s mod-C. Data processed after 10 October 2017 has 720s values that depend on 'V' have been computed with a filtergram set obtained every 45s rather than 90s for mod-L and 135s for mod C.

How can you tell which data have been processed properly?

Bits 16-19 of CALVER64 in vector products indicate the observing sequence and basic processing applied to create the vector field products hmi.S and subsequent dependent products. Thus the 4-bit nibble to check is 0xF0000

If (CALVER64 & 0xF0000) >> 16 equals                                       %% {comparison is bitwise, so for IDL use 'AND'}

 .   0x0 -> data were collected using mod-C [no known processing issues]
 .   0x1 -> not defined
 .   0x2 -> mod-L data processed with incorrect alignment of filtergrams from the two cameras and duplicated 'V' filtergrams
 .   0x3 -> partially corrected mod-L processing with proper alignment but still using only half the available 'I +/- V' filtergrams
 .   0x4 -> mod-L with proper alignment using all appropriate filtergrams

Comments:

 .   The observed filtergrams actually collected with mod-L have been correct from the start.
 .   All mod-L data were processed incorrectly until 2017.10.10_XX-CONFIRM PRECISE TIME
 .   The misalignment bug was corrected in data processed after 1 September 2017 for data with T_REC 2017.08.27_00:00:00 and after
 .   The filtergram-selection bug was corrected in data processed after 10 October 2017 for data beginning with T_REC 2017:10:XX-CONFIRM PRECISE TIME
 .   Earlier mod-L data are being reprocessed to make both corrections.