HMI Flatfield Processing

(FF Processing Sequence)

Data Series Frequency Created By Used By Applied To
su_production.flatfield_fid Per FID module_flatfield module_flatfield_combine -
hmi.flatfield Daily module_flatfield_combine lev1 processing hmi.lev1
hmi.flatfield_update Daily module_flatfield_combine not currently used future observables reprocessing
hmi.offpoint_flatfield Quarterly/
Weekly
flat_inorbit.pro/
lev0a2deflev1_FULL_PZT_FSN_cron.pl
module_flatfield_combine -

Description:
su_production.flatfield_fid:
hmi.flatfield:
hmi.flatfield_update:
hmi.offpoint_flatfield:

Source Code:
~/cvs/JSOC/proj/flatfield/apps/module_flatfield
~/cvs/JSOC/proj/flatfield/apps/module_flatfield_combine
~/cvs/JSOC/proj/flatfield/off_flat_IDL/flat_inorbit.pro
~/cvs/JSOC/proj/lev0/scripts/lev0a2deflev1_FULL_PZT_FSN_cron.pl
~/cvs/JSOC/proj/lev0/scripts/pzt_flat_cron.pl
~/cvs/JSOC/proj/lev0/apps/pztflat/pzt_flatfield.pro

To run quarterly hmi.offpoint_flatfield

  1.  See /home/jeneen/flats/do_flat.  You can edit this to run it for
      the time(s) you're interested in, or just run the commands by hand.

  2.  The script outputs binary flats, darks, and bad pixel files for 
      both camera 1 and camera 2.  Use IDL to look at the results and
      run the ingest scripts if you're happy with them.


Troubleshooting

hmi.offpoint_flatfield is not being updated weekly

 Processing logs can be found at /scr21/production/qsub/flat/pzt2.*.log, 
 though they are scrubbed weekly and failures may not be caught in time.  You'll
 likely need to rerun by hand:
  1.  make sure the lev0a data are online.  You are looking for hftsacid=3021 for
      2 runs taken 24 hours apart, the last one being on the day you expect the
      update.  You need both of those runs.  This is how I find and retrieve them:

      @ fsn = `/home/jeneen/campaigns/scripts/hmi/tai2fsn 2011.10.01
      @ fsn1 = $fsn - 46100
      @ fsn2 = $fsn + 46100
      show_info -q "hmi.lev0a[$fsn1-$fsn][? hftsacid = 3021 ?]" key=fsn > /tmp/pzt.find
      show_info -q "hmi.lev0a[$fsn-$fsn2][? hftsacid = 3021 ?]" key=fsn > /tmp/pzt.find2
      @ start = `head -1 /tmp/pzt.find`
      @ end = `tail -1 /tmp/pzt.find`
      @ start2 = `head -1 /tmp/pzt.find2`
      @ end2 = `tail -1 /tmp/pzt.find2`
      echo "$i  $start'-'$end  $start2'-'$end2"
      echo "$start'-'$end," >> get_lev0a
      echo "$start2'-'$end2," >> get_lev0a
      ex -s get_lev0a'<<'FOO
      g/,$/s//]/
      FOO
 
      /home0/phil/cvs/JSOC/proj/workflow/scripts/stage_tapes_in_order.csh 5 `cat get_lev0a`

  2.  Run the PZT processing in IDL (it takes awhile to run - 1/2 hour?).
      cd /home/production/cvs/JSOC/proj/lev0/apps/pztflat
      IDL> .run pzt_flatfield
            pzt_flatfield, 2011,10,01
  
  3.  This creates binary files and ingest scripts.  Use IDL to load the binary files and
      look at them.  If they look OK, run the ingest scripts.