Phil's archive
July 2008
Developed watch_lev0 in jsoc.stanford.edu/ajax/watch_lev0.html as a tool for viewing level-0 image processing status and small images.
Added code to the level0 ingest program to generate 512x512 pixel small fits files and 256x256 pixel .png images for quicklook purposes.
Further development of jsoc_info and lookdata.html to allow .txt as well as .json format exchanges. this enables wget scripts to fetch series info and data more easily.
Developed export functions and supporting cgi-bin program and modules to do exports. jsoc_fetch is cgi-bin program that works with lookdata.html to queue export requests. jsoc_export_manage and jsoc_export_as_is and Art's jsoc_export_as_fits work with the series jsoc.export, jsoc.export_new, and jsoc.export_user to manage and fulfill requests.
8 May 2008
Implemented test code for prototype clinet side javascript for export as described in the wiki.
see lookdata2.html in my proj/export/webapps if interested.
1 May 2008
Continued some minor updates to jsoc_info and lookdata.html for AJAX access to DRMS/SUMS data.
Implemented way to walk through huge recordsets by first getting all recnums for a large request (using drms_query_string to build a query of just recnum) then making a "getnext" function to read the next record from the list (blocking internally at 100 records). See ~phil/jsoc/proj/myproj/apps/mylib.c for the parts. Same dir pt_average.c tests and makes average images from an input set.
Compared doing a 30-day average of MDI data via both DSDS and DRMS. The DRMS series is mdi.vw_V_lev18 and is stored with unitsize=1440 as FITS. That means one file per record but one day per SU. On the DSDS side, uses series:vw_V_06h which is stored as a FITS_CUBE so one file per 6 hours. So 120 files in the DSDS case and 43200 files in the DRMS case.
Results are not surprising:
DSDS sonar: 3.145u 3.444s 23:29.00 0.4% 0+0k 19+3io 0pf+0w DSDS n02: 0.115u 0.010s 5:14.50 0.0% 0+0k 0+0io 1pf+0w
DRMS qsub2: 24.793u 15.192s 7:42.36 8.6% 0+0k 0+0io 6pf+0w DRMS n02: 30.066u 27.764s 36:35.12 2.6% 0+0k 0+0io 13pf+0w
So, we do need the TAS via FITS for slicing a FITS cube into records.
In all cases the active load was about 2 of 8 or 16 processors.
13 March 2008
This week worked on spec for export cgi-bin basic set and had discussion with Rick, Art, and Karen about overall plan for initial export process. Rick has made series jsoc.exports and I am summarizing the plan.
MOVED DETAILS to: AjaxJsocConnect