Preamble (please read)
- There are a wide range of tools available to access AIA data and images. They were designed to suit users with a wide variety of needs. This page guides a user through the AIA data system based on their needs and interests. First and foremost, a user should understand why a versatile data system is necessary. An AIA full-resolution image has 4096x4096 pixels, and AIA has [#Channels | ten different AIA wave bands, or channels]. A single full-resolution JPEG can be up to 10 MB in size (a single uncompressed science data file can be dozens of MB) and the AIA performs these observations continuously at a rate of approximately 1 image per second. With previous solar missions, it was not unusual for a user to request a few hours of data. In the case of the AIA, this request would amount to over half a terabyte of data. Even one hour of images at a single wavelength is on the order of 10 GB in size. It is clear that a simple data access system is not suitable for the AIA. We have devised a number of tools so that the user can access specific data sets as quickly as possible.
Please note: This guide does not contain the full range of tools and resources available to AIA data users. The user has a great deal of flexibility, and there is no single "correct" way to identify, obtain, and analyze AIA data. It is hoped that this guide will help the user get started with AIA data, and that users will familiarize themselves with the additional resources over time.
Determining which data you want
I need to browse through some data before I can decide what data I'd like to request.
- You can browse full-disk movies of the Sun in all of the AIA wavelengths on the SDO Browse Data page. These movies are lower resolution (512x512 or 1024x1024) and are much lower cadence (15 minutes). There are also daily 1024x1024 movies for each AIA wavelength.
I'd like a more detailed look instead of just a "browse."
- Helioviewer is a versatile tool that allows the user to view images and movies from multiple data sources, zoom in on specific regions and view zoomed-in movies, and overlay the images in multiple layers to compare features and structure. Helioviewer also has the capability to track a region, meaning that the viewing box moves with the Sun's rotation. The result can be saved as a link and accessed at a later date. A full-resolution AIA image is 4096x4096 pixels, and standard displays are not able to show anything near that size. Helioviewer allows the user to choose between viewing the whole Sun in (much) lower resolution, or viewing a subfield at better or even full resolution. A desktop version, called JHelioviewer, can be downloaded onto your computer. The user is able to create movies and export them in MPEG or Quicktime format.
I'm looking for a specific feature, event or phenomenon but I haven't determined the date/times of the images I need.
Most of the features or events a user would need have already been assembled in the Heliophysics Events Knowledgebase. The HEK is a comprehensive catalog of many items, including Active Regions, CMEs, Coronal Dimmings, Coronal Holes, Coronal Jets, Coronal Waves, Emerging Fluxs, Filaments, Filament Eruptions, Filament Activations, Flares, Loops, Oscillations, Sigmoids, SpraySurges, Sunspots, and Plages (complete list here). It also contains data, images and movies that have already been assembled for a given event, as well as comments and annotations entered by other users. Prior to using the HEK, it is highly recommended that you view the brief HEK tutorial to familiarize yourself with all of the features and options.
I know the start time and end time of the event or phenomenon I'd like to study, but I don't know which wavelength(s).
- The AIA has seven EUV channels and three UV-Visible channels. These channels correspond to different characteristic temperatures and different regions of the solar atmosphere, ranging from 5000 degrees to over 10 million degrees. To see what each of the wavelengths look like and view a recent movie, go to helioviewer or the SDO Browse Data page. However, browsing the Heliophysics Events Knowledgebase is probably the best way to determine which wavelength(s) are most appropriate for your topic of interest.
Accessing the data
I want images and/or a movie, and I don't need the full science data files.
- Science data files are in the FITS format. Each uncompressed full-res FITS file is dozens of MB in size, whereas a full-res JPEG on the order of 10 MB or less. Therefore, it is to your advantage if jpegs or movies are sufficient. The tools described above in Section I should be able to provide you with any images or movies you need.
I need FITS data files, and I think I know the time range and wavelengths of the images I want.
- First and foremost, you must make certain that the data you've identified is manageable in volume. Full-resolution uncompressed FITS files are dozens of MB in size, and the AIA makes approximately one observation per second. A request for all of the 304 Å files for the duration of a typical eruption, for example, can result in a file tens of gigabytes large, which can take upwards of a day to download with a standard DSL or cable internet connection. You may want to use a tool such as the Internet Connection Speedometer to determine your download speed before you decide how large a reasonable data request would be. Another consideration is how much data your computer will be able to store and analyze once the data has been downloaded.
I need FITS data files, I know the time range and wavelengths of the images I want, but the available data volume is overwhelming. Case 1: Using subfields/cutouts
- More often than not, this is the situation of an AIA data user. There are four options: reduce cadence, reduce the number of wavelengths, reduce resolution, or request only a subfield of the images rather than the full solar disk. For the vast majority of analysis efforts, the full 4096x4096 image is not needed. Subfield data utilities allow the user to zooming in on only the portion of the image they need, thereby drastically reducing the volume of data.
- Each Heliophysics Events Knowledgebase event listing links to the Cutout Service, which allows the user to choose a subfield of data and request only that data. This can drastically reduce the volume of data the user will download. The figure at left shows a window of the cutout service. Users identify the region of the Sun they'd like to examine, shown as a pink outline box in the upper left portion of the figure. For this example, the subfield is less than 10% of the size of the total image making the data volume much more tractable. The Cutout Service also has a "tracking" option that automatically moves the observation boundaries to compensate for solar rotation. This is very useful when the user wants to studying a feature (such as an active region) for an extended period.
I need FITS data files, I know the time range and wavelengths of the images I want, but the available volume is overwhelming. Case 2: Decreasing the temporal or spatial resolution
The Virtual Solar Observatory search tools provide the user with a great deal of flexibility in accessing solar data. There is a search interface on the VSO website, (Joe & Joe, which link do you prefer?) but there is also an IDL interface provided within the SolarSoft library that allows the user to integrate their data search, retrieval, and analysis efforts.
The two key SolarSoft functions are vso_search.pro and vso_get.pro. vso_search.pro allows the user to query a vast database of solar data and set requirements regarding timing, data source, data type, resolution, as well as other options. This is an example of a vso_search.pro query:
IDL> searchfile = vso_search('2010-jul-20 14:00','2010-jul-20 14:10', instr='aia', wave='304')
Records Returned : JSOC : 50/50
Records Returned : JSOC : 7/7
- The search query returned records for 57 AIA 304 Å images that are available. (The SDO JSOC, or Joint Science Operations Center, provides AIA and HMI data.) 50 of the records are full resolution (4096x4096 pixels), and 7 of the records are quarter resolution (1024x1024 pixels). To choose only one or the other, the query can be further refined:
IDL> searchfile = vso_search('2010-jul-20 14:00','2010-jul-20 14:10', instr='aia', wave='304', pixel=4096)
Records Returned : JSOC : 50/50
IDL> searchfile = vso_search('2010-jul-20 14:00','2010-jul-20 14:10', instr='aia', wave='304', pixel=1024)
Records Returned : JSOC : 7/7- The search returned fewer quarter-resolution images than there are full-res images because the sampling rate of the lower resolution images is approximately 1/6 slower than the nominal AIA image cadence of 12 seconds per image.
Note: All of the AIA images are available at the full resolution. Quarter-resolution images are created by binning down the 4096x4096 images. The quarter-res images are being made available in order to serve users who would like to work with AIA data but do not require full resolution. If you would like to obtain data at a specific time cadence, you can tell vso_search.pro to filter the query at a lower sampling rate (in this case, 3 minutes):
IDL> searchfile = vso_search('2010-jul-20 14:00','2010-jul-20 14:10', instr='aia', wave='304', pixel=4096, sample=180)
Records Returned : JSOC : 4/4- The query returns full-resolution 304 Å image spaced 3 minutes apart, instead of the nominal one image per twelve seconds. There are a great deal more keywords and options available. Please see Appendix II of this document for a more complete (yet still partial) set of options and examples. Once you've identified which files in the query you'd like to obtain, vso_get.pro transfers the data to your local directory:
IDL> getfile = vso_get(searchfile(6),filelist=filelist) VSO_GET: This will download 1 file(s) 1 : http://vso.tuc.noao.edu/cgi-bin/drms_test/drms_export.cgi?series=aia_synoptic2;record=304_11763447-11763447 % RDWRT_BUFF: Please wait. Downloading... % File: /cgi-bin/drms_test/drms_export.cgi?series=aia_synoptic2;record=304_11763447-11763447 % Size: 4210560 bytes % From: vso.tuc.noao.edu % To: /Users/barbara/data % HTTP::COPY: 4210560 bytes of 4210560 total bytes copied in 2.73 seconds % HTTP::COPY: Wrote 4210560 bytes to file /Users/barbara/data/aia_test.synoptic2.304A_2010-07-20T14_10_02.12Z.image_lev1p5.fits Downloading completed
- The keyword "filelist" provides the local name of the file(s):
IDL> print, filelist(0) /Users/barbara/data/aia_test.synoptic2.304A_2010-07-20T14_10_02.12Z.image_lev1p5.fits
Analyzing the data
Well, well, well. So what have we here?
- Level 1.5 - repointed rotated but not exposure renormalized May be compressed, if so you readfits doesn't work, use read_sdo.pro
level 1.0 & 0 definitions, /uncomp_delete
What analysis tools are available?
- dem stuff, mapping, pfss overlays
- color tables
Section III.c: What are the limitations of the data?
- error sources psf instrument information for writing papers
Section III.d: I'm ready to publish. Who do I acknowledge?
Nothing is required, but especially for early papers it would be nice to acknowledge data provider(s) and any AIA team members who helped.
Appendix I: IDL and the SolarSoft software tree
Appendix I.a: What is IDL?
In this guide, we assume that all manipulation of FITS files will be done using IDL software and SolarSoft software packages. IDL stands for Interactive Data Language, a data analysis programming language with a syntax descended from Fortran. The software is sold by ITT Visual Information Solutions.
Appendix I.b: What is SolarSoft?
The SolarSoft system is a set of integrated software libraries, data bases, and system utilities which provide a common programming and data analysis environment for solar physics. The software in the SolarSoft library has been contributed by people and projects throughout the solar physics community.
The entire SolarSoft suite is free to users. See the SolarSoft website for an overview, installation instructions, user guides, tutorials and documentation.
Appendix I.c: I have IDL, but I don't have the SolarSoft software tree on my computer.
Go to the SolarSoft website and go to "Installing and Configuring Solarsoft" in the menu. If the menu doesn't appear, click the "Open Menu Window" button at the top. The guide will take you through the steps to install and configure SolarSoft on your system.
When installing SolarSoft, there are a variety of software sub-libraries availability. This guide requires the "AIA", "VSO", and "ONTOLOGY" software, so please select those options in addition to any other packages you may need.
Appendix I.d: I have the SolarSoft software tree on my computer, but I'm not sure if I have the appropriate software packages installed.
Go to the SolarSoft website and go to "Upgrading a SolarSoft Installation" in the menu. If the menu doesn't appear, click the "Open Menu Window" button at the top. The guide will take you through the steps to upgrade your SolarSoft library and add new packages. This guide assumes you have the "AIA", "VSO", and "ONTOLOGY" packages installed.
Appendix II: Identifying and retrieving data using the Virtual Solar Observatory's IDL interface
Appendix II.a: Searching the VSO catalog using vso_search.pro
AIA data can be accessed in a variety of ways. One point of access is provided by the Virtual Solar Observatory, or VSO. The VSO's search tools provide the user with a great deal of flexibility in accessing solar data. There is a search interface on the VSO website, but there is also an IDL interface provided within the SolarSoft library that allows the user to integrate their data search, retrieval, and analysis efforts.
Please note that VSO's IDL interface programs, such as vso_search.pro and vso_get.pro, are updated on a regular basis. The examples in this guide do not reflect the full range of capabilities of the program, and to be sure that you're aware of all that has been changed or added you should check the program documentation directly.
We'll proceed with a series of examples that the user can try on their own computer. We'll start by searching the catalog with vso_search.pro, and the query's result can be passed to vso_get.pro to retrieve the data to your local directory.
IDL input code is shown next to the IDL> prompt, and the text response to the input (if any) is shown below the command line. We'll start by defining a time range to work with:
IDL> time1 = '2010-jul-17 00:00' IDL> time2 = '2010-jul-17 01:00'
Example 1. Find all of the AIA images available for a given date range:
IDL> searchfile = vso_search(time1, time2, instr='aia') Records Returned : JSOC : 2306/2306 Records Returned : JSOC : 300/300 IDL> print_struct, searchfile(0) DETECTOR INSTRUMENT SOURCE PROVIDER INFO PHYSOBS FILEID SIZE URL GETINFO AIA SDO JSOC AIA level 14096x4096 intensity aia_lev1:304:1058400038 6.620E+04 IDL> print_struct, searchfile(0).tim START _END 2010-07-17T00:00:02 2010-07-17T00:00:03- Note: The search returned 2606 images for a time range spanning one hour. Ordinarily, one could enter just a single date into vso_search and get the listing for the full day, but SDO's data volume prohibits this.
Appendix II.b: Retrieving data with vso_get.pro
Email: MailTo(you AT SPAMFREE example DOT com)
...