Differences between revisions 12 and 26 (spanning 14 versions)
Revision 12 as of 2012-04-19 06:33:08
Size: 5923
Editor: glimmer
Comment: status report April 12-18th,2012.
Revision 26 as of 2012-06-26 02:24:04
Size: 11113
Editor: glimmer
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
4/18/2012:
June 14-24: vacation
Status for June10-14:
  * At AAS meeting in Anchorage. met with Hubble/JWST folks to talk about/see how they catalog/export/visualize their data to get more ideas on our data catalog and possibly the flares catalog that Rick has in mind.
  * met and chatted with some of the VSO/SDO feature detection folks to try and understand what they do and how they do it.
  * Attended the SDO/Solar talks,
   
Status for May31-June 9th:
  *submitted paper. YAY!
  * worked on Poster
  * get ready for AAS/SPD meeting

Status for May 24-30:
 worked on more paper edits. All the co-I's have good and valid suggestion for improvement.As of yesterday, Rick gave his last set of comments and the green light. Need to finish and send it.
 
Started work for the poster.

Status for May 17-24:

1.Turns out colortable AIA for 304 was not quite correct. Phil noticed that the colors were off. I fixed it and rechecked the colortables for the other wvlngths. Thsi time heard back from the AIA people and actually got the clipping values for each wvl. SO need to put that into the jsoc_export_as movies. Tye existing min/max values are our estimates and not quite correct.
Barbara Thompson at Goddard was very helpful. We also realized that the aia fits files are not exposure time corrected. So for ALL aia images/movies that we make, we need to grep for the exp time, divide the image by that and then apply min/max/ scaling and colortable!!
2. Having issues with the pre-postflare errorbars and sinusoidal underlying function folded through the LCP/RCP values. Need to detrend the data form spacecraft motion andmark the pixel location of the Ic images. Sigh.

Status for May 10-17:

 1.Realized that our plan for using a drms color table with inseries as a "prime key" to determine the jsoc_export_as_movie/images parameter will NOT work . We will lose some of the functionality that we have right now. For example, right now  you could give exportdata a series like aia.lev1[2011.07.11_12:00:00/2m] and ask for images and it will do that by looking for *ALL* aia.lev1 images in that time interval including euv and uv wavelengths, find the correct colortable corresponding to each wavelength, and make the images accordingly. We will lose the ability to make images for such requests and AIA requests made using only FSN's as prime keys.
2.So the new plan to enhance existing capability ie . add the ability for the user to choose their own parameters will be done by adding a "Choose your own parameters checkbox" to exportdata.html (like the tracking box in hgpatch). The default will be that jsoc_export_as_movie will run as it does currently. If the "use your own params" is checked; then a set of new parameter options: Colortable, min/max,scaling and size options will be appear and those variables will be passed on to jsoc_export_as movie.
In jsoc_export_as movie; we will add a third part to the if loop. As of now the if loop parses the first 3 characters to check if it is HMI or AIA. If the "choose your own params " is checked; it will go into the 3rd option and the variables passed will be used as input for render_image.
I think we should add some chks in the exporthtml javascript itself, so that the series passes actually can be made into jpgs.

2. Another improvement that will be appreciated by some(Rick in particular) is to be able to read in datacubes to make a movie.
For that, renderimage needed to be edited to chk for no of elements in naxis. If number of elements is 3, then its a datacube and then we need to slice up the datacube into separate fits files and the feed it into the jpg making machine.
 3. Mostly worked on the paper. It is taking forever..
  
 
paper

. Tasks for May 10-17:
 . 1. Get jsoc_export_as_movie working. Get it to the point where all parameters can be passed on the command line.
 . 2. Get started with the poster for AAS. Figure out an outline and realistic scope of what can be done till June 3rd at which point I
 . "freeze" the project and start putting stuff on the poster.
 . 3.Get atleast 2/3 more slide shows up on the dataprods page. Need to ping other people who are supposed to write up and give me stuff.
 . 4.Mail Phil the javascript to check for IE and pop up alert and the " i " and flash bulb symbols as alternatives for the highlighted questionmark in exportdata and lookdata.
 . . . __May 9th: Status Update:__
    1. Finished the paper- and sent to coauthors . Major rewrite is done. hopefully all edits/suggestions will be minor and not a rethink of the focus of the paper.

    2. Found out what Phil wants regarding datapage: He liked the idea of slide shows that launch/start OnMouseOver/or OnMouseClick. Have not had time to work on it any more.

    3. jsoc_export_as_movie does read in the parameters in the form in=inseries colortable=ctable etc with the edits that Phil put in. But am not sure if the ctable and minmax is getting passed Need to check the drms_run log. Havent done much else yet. Was out on Monday and just focused on the paper on Fri/Tue/Wed as I really didnt want it to be on my plate till next week !

 . Tasks for May 3-10th:
 1. work on rewriting the bash script for jsoc_export as movie and image -we no longer need the case loop as the parameters will be read in from the drms table. jsoc_export_manage.c will also need to be changed for the movies/images protocols.

 1. get the paper done! 3.see what Phil wants regarding the dataproducts webpage.
 . __May 3rd,2012:Status Update__
 . 1. exportdataPD.html now has the correct colortables and scaling menus available as pulldown. Also fixed the javascript so only unique values of colortable and scaling show up. (there were many duplicates). right now still going with su_priya.ColorTables . Took me a while to figure out how to do this- and learned about the "hash-sieving" method. However as Art pointed out, the exportdata still does not quite "reset" if you just change one option say, you pick another colortable or something. You have to explicitly click "reset". That should be cleaned up.
 . . 2. Added an associated "text panel" that changes with every changing image in the slide show. Havent heard back from Phil if this is along the lines of what he want: will bug him today -before I spend tons of more time trying to make more slide shows.
 . . 3. Most of the figures are done -yesterday Rick suggested that I get some more filtergrams data -pre and post flare to get a better handle on the statistics. WIll ask Seb to process them for me. Also got the I0-I5 positions image for the paper. This stuff takes so long. Sigh!
 . . __April 25th, 2012 :Status update: __
 . 1>jsoc.ColorTables is created and populated . Art's idea was to have two drms tables: a stand alone colortable with just ct name and format ie .sao or .lut and another tables mapping which hmi series goes with which table and what are the default min/max/scaling values. But as of now, Phil populates the pull down menu from the existing su_priya.ColorTables and that works . So, for now the plan is to not worry about the new colortable, but to edit jsoc_export as movie to not have any of the parameters hard coded but to accept min/max/scaling variables on the command line; if not supplied then to use show_info to determine the defaults and use them.
2>exportdataPD.html has scaling and size pull down menus put in..but I have not succeeded in populating them from the su_priya.ColorTables series.

3>Data products webpage: spent a lot of time making the left navbar with a rollover effect..but decided that it looked very crowded and confusing. spent some time trying to decide how to display the "Visual Data Products" page which needs to give equal status to all the missions ..its a lot of products! Played around with the idea of a carousel ..but finally decided on something simpler . The layout is a tablewith the five missions and then each product type will have its own slide show that will be activated when you roll over the mouse.As of now ONLY the HMI Observables slide how works . If this type of visual is ok with Phil; I'll go ahead.

4>Paper: Got the script for generating the figures using Linecore proxies(using I3 and I4 filtergrams values)working and the plots look right. Need to make 4 sets for the four flares and spend some time making them look pretty. Edited a big chunk based on revised plan. More edits due.would like to be done with it by end of next week.

To do in week of Apr19-25

 1. Make sure all the color Tables including mag is in the DRMS. Figure out if WVL is enough as a prime key? Or do we need to add a "string" as a prime key. make it work with jsocexport manage as a variable.
 1. The link now opens in a new tab..but doesnt close the tab when the focus changes. and a new link is clicked ,need to figure that one out.
 1. Ask Art what he wants for the tables- put in the javascript required to make the scaling/min/max as variables. Make sure you put an alert to pop up when the user chooses a different scaling.
 1. make the figures for the paper. Put in text to explain the reasons for using and focusing on filtergrams.
 1. The Dataproducts webpage: Phil wants a more instrument to go on there.see ~/priya/JSOC_webpages_design.pdf for the flowchart
some other ideas: maybe the ? on the lookdata/exportdata page should be an i symbol or a flashbulb? 4/18/2012:__ __

__April 18 Status update:__
Line 4: Line 79:
 1> '''The JSOC Data products web page:''' [http://jsoc.stanford.edu/Priya/HMI/ http://jsoc.stanford.edu/Priya/HMI/ ]
  *  Have updated most of the child links with info that people sent me.
  * figured out wher to put in the javascript so the difference between jsoc and jsoc2 is maintained. Turns out ..it need to go into a window.OnLoad function no bodyOnload .
 '''1>''' '''The JSOC Data products web page:''' [http://jsoc.stanford.edu/Priya/HMI/ http://jsoc.stanford.edu/Priya/HMI/ ]
  * Have updated most of the child links with info that people sent me.
  * figured out where to put in the javascript so the difference between jsoc and jsoc2 is maintained. Turns out ..it need to go into a window.OnLoad function no bodyOnload .
Line 10: Line 85:
''' 2> Flares paper '''
  * Have determined what all needs to be rewritten- but just starting on it today.
  * Emailed Hugh that I will get back to the paper.
  3> '''Making the Colortables/scale a variable that can be used during exporting data as jpgs/mpgs.'''
''' 2> Flares paper '''
Line 15: Line 87:
  * Have not done much about it yet  * Have determined what all needs to be rewritten- but just starting on it today.
 * Emailed Hugh that I will get back to the paper.
''' '''
Line 17: Line 91:











    3/11/11

 . Worked with Art and Phil to get the hg_patch /AIA to work as export as movie and images. Finally everything seems to be working.Phil wants have the option in hg_patch to "not track" so the AIA people can monitor (take a patch of) a limb phenomenon. ALso we need to put in option to export FITS/ movie and Image in one request. Also the colortables need to
go into a JSOC series??

 . Need to come up with a series of tall the M and C class flares from the past month.
2/16/11

 . Phil pointed out that the hg_patch part of jsoc_export_as_movie/image is not yet working. Tracked down how to make legit hg_patches from JSOC_drms runlogs.hg_patch was not working because of a bug in show_info. Also needed a special case for hg_patch as it should not be scaled andone needed to to go into the @hg_patch.log files to figure out what kind of data it was-M, Icor V etc. Turns out Ic and Ic_flat are also different and need to be treated differently. Pit in log and sqrt scaling in render_image.The cropping at the limb was not happening correctly: turned out that we need to "not invert" the
image forhg_crop (ie. we wanted as-is) and there was a bug in the way crrot was being calculated.

2/14/11 Wrote a script to extract M and C class data from the ftp link Taeil sent. Can extract all the relevant data and run drms_compare on them and store as a series. Came up with 3/4 color tables as per Todds specs using Xpallete in IDL

1/26/11 got started with Solar flares stuff. Need to get a list of known h-alpha and xray flare dates above a certain threshold and see if i can detect anything in the level1.5 files(Ic,Lw,Ld) and calculate Li(Line Intensity) which is Ic-Ld? use ricks drms_compare module.

1/24/10 The aia_make_images module works fine. It chooses the correct color table based on the record: and makes multiple colortable-images for a series with multiple wavelengths. Need to figure out how to put in log scaling in render_image. Also we need a faciltiy to be able to export images/fits and movie ALL at one go. Spent some time just brushing up on C.

1/10/10

Converted all the AIA color tables to .lut. Familiarized myself with the AIA data. We will need lev1 and lev1_hg_patch and lev 1.5 data to be made into movies.

1/4/10

 . 1> Art suggested he would make the edits to the jsoc_export_manage
 . 2> found the colortables for AIA images
 . 3> For the Flares Project: turns out that getting the list of flares may be a bit elaborate: grepping and awking though lists at NOAA. Spent a lot of time trying to find a site with location and flare categories.
 .
12/15/10

1> Bash script is done and works 2> script interacts ok with the qsub call ie works fine on the cluster. The exportdataPD.html and exportdataPD2.html pages includ ethe drop down menu (in js)to include image and movie. 3> met with Phil: 3 new issues:

 * AIA images/movies also need to be made from from the jsioc page. They have specific requirements for their colors.
 * jsoc_export_manage need to be reorganized so that calls are not replicated in the (as of now) 2 possible calles :1) to jsoc_export_as_is:
  . 2) to jsoc_export_as_fits but all possiblre common pre-processing is done.
 * Phil would like the option of choosing the scaling/colortable/size for the images/movie
 * edited the Camilla_sdo card to put on sdo facebook page
##master-page:HomepageTemplate
#format wik

== Priya Desai ==
Email: [[MailTo( priya@sun.stanford.edu ]

11/16/10

1> Met with Art to discuss integrating the image and movie option in the JSOC_export page.The image option will be a bash script~module) with the name "jsoc_export_as_image" to maintain consistency. At least initially all default values such as scaling/colortables etc should be provided within the module (not given as options to the user).

 * Use the colortables from the DRMS series.
 * Have 5 possible types of files: hmi.M_ (magentogram), hmi.V_ (Dopplergram), hmi.Ic_ (Continuum), hmi.Ld_(Line Depth), hmi.Lw_(Line width)
2> As a next step, need to insert *ALL* IDL colortables into the DRMS series for colortables that has been created.

3> Met with Todd to talk about creating a better colortable for the Public Images -something that will show the structure in the penumbra. and has a pale yellow background.
 . '''3> ''''''Making the Colortables/scale a variable that can be used during exporting data as jpgs/mpgs.''' * Have not done much about it yet.
Line 81: Line 93:
 . CategoryHomepage  . CategoryCategory
----
 CategoryCategory

June 14-24: vacation Status for June10-14:

  • At AAS meeting in Anchorage. met with Hubble/JWST folks to talk about/see how they catalog/export/visualize their data to get more ideas on our data catalog and possibly the flares catalog that Rick has in mind.
  • met and chatted with some of the VSO/SDO feature detection folks to try and understand what they do and how they do it.
  • Attended the SDO/Solar talks,

Status for May31-June 9th:

  • submitted paper. YAY!
  • worked on Poster
  • get ready for AAS/SPD meeting

Status for May 24-30:

  • worked on more paper edits. All the co-I's have good and valid suggestion for improvement.As of yesterday, Rick gave his last set of comments and the green light. Need to finish and send it.

Started work for the poster.

Status for May 17-24:

1.Turns out colortable AIA for 304 was not quite correct. Phil noticed that the colors were off. I fixed it and rechecked the colortables for the other wvlngths. Thsi time heard back from the AIA people and actually got the clipping values for each wvl. SO need to put that into the jsoc_export_as movies. Tye existing min/max values are our estimates and not quite correct. Barbara Thompson at Goddard was very helpful. We also realized that the aia fits files are not exposure time corrected. So for ALL aia images/movies that we make, we need to grep for the exp time, divide the image by that and then apply min/max/ scaling and colortable!! 2. Having issues with the pre-postflare errorbars and sinusoidal underlying function folded through the LCP/RCP values. Need to detrend the data form spacecraft motion andmark the pixel location of the Ic images. Sigh.

Status for May 10-17:

  • 1.Realized that our plan for using a drms color table with inseries as a "prime key" to determine the jsoc_export_as_movie/images parameter will NOT work . We will lose some of the functionality that we have right now. For example, right now  you could give exportdata a series like aia.lev1[2011.07.11_12:00:00/2m] and ask for images and it will do that by looking for *ALL* aia.lev1 images in that time interval including euv and uv wavelengths, find the correct colortable corresponding to each wavelength, and make the images accordingly. We will lose the ability to make images for such requests and AIA requests made using only FSN's as prime keys.

2.So the new plan to enhance existing capability ie . add the ability for the user to choose their own parameters will be done by adding a "Choose your own parameters checkbox" to exportdata.html (like the tracking box in hgpatch). The default will be that jsoc_export_as_movie will run as it does currently. If the "use your own params" is checked; then a set of new parameter options: Colortable, min/max,scaling and size options will be appear and those variables will be passed on to jsoc_export_as movie. In jsoc_export_as movie; we will add a third part to the if loop. As of now the if loop parses the first 3 characters to check if it is HMI or AIA. If the "choose your own params " is checked; it will go into the 3rd option and the variables passed will be used as input for render_image. I think we should add some chks in the exporthtml javascript itself, so that the series passes actually can be made into jpgs.

2. Another improvement that will be appreciated by some(Rick in particular) is to be able to read in datacubes to make a movie. For that, renderimage needed to be edited to chk for no of elements in naxis. If number of elements is 3, then its a datacube and then we need to slice up the datacube into separate fits files and the feed it into the jpg making machine.

  1. Mostly worked on the paper. It is taking forever..

paper

. Tasks for May 10-17:

  • 1. Get jsoc_export_as_movie working. Get it to the point where all parameters can be passed on the command line.
  • 2. Get started with the poster for AAS. Figure out an outline and realistic scope of what can be done till June 3rd at which point I
  • "freeze" the project and start putting stuff on the poster.
  • 3.Get atleast 2/3 more slide shows up on the dataprods page. Need to ping other people who are supposed to write up and give me stuff.
  • 4.Mail Phil the javascript to check for IE and pop up alert and the " i " and flash bulb symbols as alternatives for the highlighted questionmark in exportdata and lookdata.
  • . . May 9th: Status Update:

    1. Finished the paper- and sent to coauthors . Major rewrite is done. hopefully all edits/suggestions will be minor and not a rethink of the focus of the paper.
    2. Found out what Phil wants regarding datapage: He liked the idea of slide shows that launch/start OnMouseOver/or OnMouseClick. Have not had time to work on it any more.

    3. jsoc_export_as_movie does read in the parameters in the form in=inseries colortable=ctable etc with the edits that Phil put in. But am not sure if the ctable and minmax is getting passed Need to check the drms_run log. Havent done much else yet. Was out on Monday and just focused on the paper on Fri/Tue/Wed as I really didnt want it to be on my plate till next week !
  • Tasks for May 3-10th:
  • work on rewriting the bash script for jsoc_export as movie and image -we no longer need the case loop as the parameters will be read in from the drms table. jsoc_export_manage.c will also need to be changed for the movies/images protocols.
  • get the paper done! 3.see what Phil wants regarding the dataproducts webpage.
  • May 3rd,2012:Status Update

  • 1. exportdataPD.html now has the correct colortables and scaling menus available as pulldown. Also fixed the javascript so only unique values of colortable and scaling show up. (there were many duplicates). right now still going with su_priya.ColorTables . Took me a while to figure out how to do this- and learned about the "hash-sieving" method. However as Art pointed out, the exportdata still does not quite "reset" if you just change one option say, you pick another colortable or something. You have to explicitly click "reset". That should be cleaned up.

  • . 2. Added an associated "text panel" that changes with every changing image in the slide show. Havent heard back from Phil if this is along the lines of what he want: will bug him today -before I spend tons of more time trying to make more slide shows.
  • . 3. Most of the figures are done -yesterday Rick suggested that I get some more filtergrams data -pre and post flare to get a better handle on the statistics. WIll ask Seb to process them for me. Also got the I0-I5 positions image for the paper. This stuff takes so long. Sigh!
  • . April 25th, 2012 :Status update:

  • 1>jsoc.ColorTables is created and populated . Art's idea was to have two drms tables: a stand alone colortable with just ct name and format ie .sao or .lut and another tables mapping which hmi series goes with which table and what are the default min/max/scaling values. But as of now, Phil populates the pull down menu from the existing su_priya.ColorTables and that works . So, for now the plan is to not worry about the new colortable, but to edit jsoc_export as movie to not have any of the parameters hard coded but to accept min/max/scaling variables on the command line; if not supplied then to use show_info to determine the defaults and use them.

2>exportdataPD.html has scaling and size pull down menus put in..but I have not succeeded in populating them from the su_priya.ColorTables series.

3>Data products webpage: spent a lot of time making the left navbar with a rollover effect..but decided that it looked very crowded and confusing. spent some time trying to decide how to display the "Visual Data Products" page which needs to give equal status to all the missions ..its a lot of products! Played around with the idea of a carousel ..but finally decided on something simpler . The layout is a tablewith the five missions and then each product type will have its own slide show that will be activated when you roll over the mouse.As of now ONLY the HMI Observables slide how works . If this type of visual is ok with Phil; I'll go ahead.

4>Paper: Got the script for generating the figures using Linecore proxies(using I3 and I4 filtergrams values)working and the plots look right. Need to make 4 sets for the four flares and spend some time making them look pretty. Edited a big chunk based on revised plan. More edits due.would like to be done with it by end of next week.

To do in week of Apr19-25

  1. Make sure all the color Tables including mag is in the DRMS. Figure out if WVL is enough as a prime key? Or do we need to add a "string" as a prime key. make it work with jsocexport manage as a variable.
  2. The link now opens in a new tab..but doesnt close the tab when the focus changes. and a new link is clicked ,need to figure that one out.
  3. Ask Art what he wants for the tables- put in the javascript required to make the scaling/min/max as variables. Make sure you put an alert to pop up when the user chooses a different scaling.
  4. make the figures for the paper. Put in text to explain the reasons for using and focusing on filtergrams.
  5. The Dataproducts webpage: Phil wants a more instrument to go on there.see ~/priya/JSOC_webpages_design.pdf for the flowchart

some other ideas: maybe the ? on the lookdata/exportdata page should be an i symbol or a flashbulb? 4/18/2012:

April 18 Status update:

  • This past week was the performance review. Goal is to increase efficiency and productivity by taking on fewer well defined tasks and pushing towards completing them. Will with Art to improve programming skills in C and understanding of DRMS and Export System. Am working on 3 main projects+ any exportdata bugs that Art may send my way.

    1> The JSOC Data products web page: [http://jsoc.stanford.edu/Priya/HMI/ http://jsoc.stanford.edu/Priya/HMI/ ]

    • Have updated most of the child links with info that people sent me.
    • figured out where to put in the javascript so the difference between jsoc and jsoc2 is maintained. Turns out ..it need to go into a window.OnLoad function no bodyOnload .

    • put in javascript to make the pages elastic-ie font/images change size based on client browser width.
    • Resized header so it does not wrap around and come to the next level. all fonts/margins etc are now in "em" so that makes it scalable . Again order of functions in the main js calling function seems important. check in with Phil.
    • The link now opens in a new tab..but doesnt close the tab when the focus changes. and a new link is clicked ,need to figure that one out.

2> Flares paper

  • Have determined what all needs to be rewritten- but just starting on it today.
  • Emailed Hugh that I will get back to the paper.

  • 3> Making the Colortables/scale a variable that can be used during exporting data as jpgs/mpgs. * Have not done much about it yet.



JsocWiki: PriyaDesai (last edited 2013-05-01 04:35:23 by localhost)