Pipeline processing in the JSOC will occur in batches or session, which consist of the execution of a sequence of modules, see Figure 4. A session is an atomic transaction in the sense that either all or none of the data records created in the session are archived. This is achieved by controlling all database access and creation of data records and units in a single DRMS instance (similar to the PE process in MDI), which acts as the session master process.
The master DRMS process begins a new session by opening a connection to the Oracle database server and issuing a BEGIN TRANSACTION command. All subsequent database operations (reading, modifying or creating data records) performed by DRMS on behalf on client modules will take place within a single database transaction. At the end of a successful session DRMS will issue a COMMIT command and the changes made to the database will become visible to other users of the database. However, all data record created or modified by a module will be immediately visible to subsequent modules executed within the same session.
If an error occurs or one of the client modules finishes with an abort request the DRMS will issue a ROLLBACK command and any changes made to the database will be undone, such that to other users the database will appear as if the session never took place.
The master DRMS process keeps track of all storage units being read and created within a session in an in-memory table.
When an existing data record is opened for reading DRMS will query the database determine which storage unit it belongs to and ask SUMS for a path to the directory where that unit resides (possibly the unit has to be read from tape first). DRMS will then insert an entry in its internal table stating that the unit storage unit is open for reading.
When a new data record is created DRMS checks its internal table to see if it has any data units belonging to the same series open for creation which still have empty slots left. If so it assigns and empty slot to the data record. If not it asks SUMS to allocate a new data unit for the series and inserts an entry for it into its internal table.
In Figure 6 is an illustration of what the internal table in DRMS might look like during the execution of an imaginary pipeline session containing four modules executed sequentially. The two first modules read data records from the series hmi_lev0_cam1_fg and write records to the series hmi_lev1_fd_V and hmi_lev1_fd_V_quick respectively. The two last modules read data records from the series hmi_lev0_cam2_fg and write records to the series hmi_lev1_fd_M and hmi_lev1_fd_M_quick respectively. It is assumed that the unit size for hmi_lev1_fd_V and hmi_lev1_fd_V_quick is 90 (would correspond to one hour of data at a 40 second cadence).
