Housekeeping Telemetry Time Averaging

The Housekeeping Telemetry Time Averaging is calculated from level 0 housekeeping day file data. The mean, maximum, minimum, and standard deviation values are calculated and loaded into a output DRMS data series. An instruction file is created to specify range of time to use to calculate averages, keywords, output DRMS series name, author and owner of DRMS series.

Time Interval Value

An important setting is the interval parameter in the instruction file. This is the range of time to calculate the averaging of keyword values. The tested and verified interval ranges are 60 seconds to 7200 seconds (2 hours) for interval ranges. For example if a user uses 600 for interval. The mean, maximum, minimum and standard deviation for a keyword value will be calculated for all points within 600 second(or 10 minute) blocks. The T_START keyword name is a time variable that is automatically created to represents these blocks of time. For example if 75 data points are available in dayfile data for October 26, 2009 for every 600 seconds. The first block or first T_START value will be 2009.10.26_00:00:00_UTC. The following blocks will be increment every 600 seconds or 10 minutes (i.e., 2009.10.26_00:10:00_UTC, 2009.10.26_00:20:00_UTC, 2009.10.26_00:30:00_UTC, etc.) until the end of the day.

Keyword Names

The long and short keyword names are setup by using the keyword names in the housekeeping configuration files. To view housekeeping configuration files, view CVS repository at http://jsoc.stanford.edu/cvs/TBL_JSOC/lev0/hk_config_file/1.163. The user must use the long keyword name for the third field in the instruction file. The short keyword name can be a user created name or the user can select the short keyword names in the housekeeping configuration files. The short keyname names used in the instruction file should be restricted to 15 characters. For more details, view below the information on setting up the instruction file. The user created keywords or short keyword names set up in the instruction file are created in jsd file with a _MEAN,_MAX, _MIN and _SD suffixes(i.e., <user-name>_MEAN,<user-name>_MAX, <user-name>_MIN and ,<user-name-1>_SD). For more details, view below information on creating JSD file.

Basic Steps To Create Housekeeping Telemetry Time Average DRMS Series With Data

The basic steps for creating and loading a Housekeeping Telemetry Time Averaging DRMS series are discussed below. The overview of the basic steps is as follows.

1. Create Instruction File
2. Create JSD file for Mean, Maximum ,Minimum and Standard Deviation(M3SD) using the instruction file.
3. Create M3SD DRMS data series based on JSD file.
4. Load the M3SD DRMS series using a dayfile and instruction file.

Create Instruction file

The format of the instruction file is shown below.

##########  Instruction File Format  ##################
###comments can be added after # at beginning of line
TemplateName:<name of instruction file. Suggested name for filename should be seriesname.txt >
SeriesName:<name of dataseries to create or use (without .jsd suffix)>
Description:<description of series in quotes>
Author:<name in quotes>
Owner:<name in quotes>
Interval:<in seconds only>
keyword:<apid in decimal format>,<long keyword name>,<user created character name-restricted size to 15 characters>
.
.
.
keyword:<apid in decimal format>,<long keyword name>,<user created character name-restricted size to 15 characters>

An example instruction file is shown below.

###### Example 1 using apid 18 temperature values using users created names for short keyword name
TemplateName:hmi.temp_summary.txt
Seriesname:hmi.temp_summary
Description:"Temperature Statistics using usernames from HMI Housekeeping - Thermal/PS voltage packet"
Author:"rock"
Owner:"rock"
Interval:600
keyword:19, HMI_TS19_OPT_INSIDE_FRONT,  HTS19_INSD_FRT
keyword:19, HMI_TS21_OPT_NEAR_CEB_INT,  HTS21_NEAR_CEB
keyword:19, HMI_TS15_OVN_PREAMP,        HTS15_OVEN_PAMP
###### End Example 1 

Create Maxium, Minimum, Mean and Standard Deviation JSD file

The create mean, maximum, minimum, and standard deviation jsd(cm3sd_jsd.pl) script is use to create the jsd file for the output drms series in DRMS. The script uses each keywords in the instruction file to create a keywords with a MAX, MIN, MEAN, and SD suffix. These keywords are defined in the jsd file as floats. There will be T_START and T_START_step (Interval value), T_START_epoch keywords automatically created to represent the start packet time and interval covered in seconds. The T_START keyword will be the index. The T_START and T_START_epoch values will use UTC in units field. T_START_step will use the interval value in the instructions file where the interval value is in seconds. The time interval or range is use to create a step in the slotted drms series. The author and owner are used to create the corresponding values in jsd file. This script does attempt to locate errors in instruction file and displays error messages to users. For example, DOS special characters are detected and removed from instructions files and a message is displayed to users. The updated instruction file can be used to correctly create a jsd file and later can be used by the C executable load_m3sd to correctly load the M3SD DRMS data series.

An example jsd file created is shown below. This jsd file was created using the example instruction file.

#====Global Series Information ==============
SeriesName:  hmi.temp_summary
Description: "Temperature Statistics from HMI Housekeeping - Thermal/PS
voltage packet"
Author:      "rock"
Owner:       "rock"
Unitsize:    1
Archive:     0
Retention:   0
Tapegroup:   0
Index:       T_START
#== Keywords Series Information ==============
Keyword:T_START,time,ts_eq,record,TSEQ_EPOCH,2,UTC,"T_START"
Keyword:T_START_epoch,time,constant,record,TSEQ_EPOCH,2,UTC,"T_START_epoch"
Keyword:T_START_step,float,constant,record,600,%f,"600 sec","T_START_step"
Keyword:HTS21_NEAR_CEB_MIN,float,variable,record,DRMS_MISSING_VALUE,%f,none,"HTS21_NEAR_CEB_MIN"
Keyword:HTS21_NEAR_CEB_MAX,float,variable,record,DRMS_MISSING_VALUE,%f,none,"HTS21_NEAR_CEB_MAX"
Keyword:HTS21_NEAR_CEB_MEAN,float,variable,record,DRMS_MISSING_VALUE,%f,none,"HTS21_NEAR_CEB_MEAN"
Keyword:HTS21_NEAR_CEB_SD,float,variable,record,DRMS_MISSING_VALUE,%f,none,"HTS21_NEAR_CEB_SD" 
.
.
.

The command format and example of running the cm3sd_jsd_file.pl script is shown below. The isf= is followed by path to the instruction file. Some example instruction files are in software repository at http://jsoc.stanford.edu/cvs/TBL_JSOC/lev1/instruction_file/su_carl . Some example jsd files created from example instruction files are shown at http://jsoc.stanford.edu/cvs/TBL_JSOC/lev1/hk_jsd_file/su_carl

Command Format:
      cm3sd_jsd_file.pl isf=<instruction-file>


Example of Running:
     cm3sd_jsd_file.pl isf=/home/carl/cvs/TBL_JSOC/lev1/instruction_file/su_carl/hmitest1200_thermal_template.txt

The environment variable in script, HK_CM3SD_JSD_DIRECTORY, is defined at top of script. The jsd is created in the location defined in script. The jsd currently is created in the prod directory as shown below. Users can easily switch directory to put jsd file by commenting out current setting and adding a user setting using the ideas shown in the example below.

#PRODUCTION SETTINGS to send jsd files to production directory
$ENV{'HK_CM3SD_JSD_DIRECTORY'}="$hm/cvs/TBL_JSOC/lev1/hk_jsd_file/prod";
#USER SETTING to send jsd files to user directory 
#turn off production by commenting out, turn on what is need below
#su_carl setting for test creating instruction file.
#$ENV{'HK_CM3SD_JSD_DIRECTORY'}="$hm/cvs/TBL_JSOC/lev1/hk_jsd_file/su_carl";
#su_rock setting for test creating instruction file.
#$ENV{'HK_CM3SD_JSD_DIRECTORY'}="$hm/cvs/TBL_JSOC/lev1/hk_jsd_file/su_rock";

Create Maxium, Minimum, Mean and Standard Deviation DRMS Data Series

Utilize the create_series utility to create series in DRMS. The format and example are shown below.

Command Format:
create_series <directory-path>/<jsd file>

Example:
create_series /home/production/cvs/TBL_JSOC/lev1/hk_jsd_file/prod/hmi.temp_summary.jsd

Load Maxium, Minimum, Mean and Standard Deviation in DRMS Data Series

The load mean, maximum, minimum, and standard deviation executable(load_m3sd) is used to get keyword data from housekeeping day file, calculate minimum, maximum, mean and standard deviations keyword values based on time interval and write keywords to defined DRMS data series.

This executable takes as input the housekeeping day file and the instruction file. The dayfile contains packets which contain the keyword values outlined in the instructions files. The housekeeping day file also contains the T_START values which can be obtain by getting TIMECODE keyword data(packet time values) per packet. The instructions file contains information for executable on which keywords to decode(see long keyword names), which keywords to calculate minimum, maximum, stand deviation, and mean values, and which keywords to load(see user created or short keyword names) into M3SD DRMS data series. The arguments passed to program are instruction filename and the housekeeping day file. The command format and example of executing load_m3sd is shown below.

Command Format:
  load_m3sd  [-h] in=<day filename> isf=<instruction file> 

 details are:
  -h: help - show this message then exit(optional field)
  in=<day file name> -use full path to day file(required field)
  isf=<instruction file name> -use full path to instruction file(required field)
  Need data series already created for this to program to load keywords in data series.
  Setup interval time in this file where range of intervals tested are 60 seconds to 7200 seconds.
  Setup keywords to calculate mean, max, min and standard deviation values in this instruction file.
  Setup seriesname in this instruction file for writing keyword min,max,mean and standard deviation values.

Example of running:
  load_m3sd  in=/home/carl/cvs/myprod/JSOC/proj/lev1_hmi/apps/20080918.0x0013 
             isf=/home/carl/cvs/TBL_JSOC/lev1/instruction_file/hmitest1200_thermal_template.txt

Limitations

1. There is a limitation of 15 characters on the user created or short keyword names in the instruction file. Any short names greater than 15 will get trucated by the cm3sd_jsd_file.pl script and be trucated in the M3SD jsd files. The trucation occurs without any warning from cm3sd_jsd_file.pl script.
2. Since the short names can use up to 15 characters, the use of keywords from M3SD DRMS data series in fits files is limited. Note that an additional 3-5 characters(i.e. these suffixes are added _MEAN,_MAX,_MIN,_SD) are added to the 15 character short names from the instruction file. Therefore the maximum number of character in a M3SD DRMS data series for a short keyword name can be 20 characters.
3. It is suggested to use a ranges of 60(seconds) to 7200(seconds) for the interval value in the instruction file. This is suggested because these were the ranges that were tested and verified to work.
4. The instruction file should contain long keyword names from one APID. The mixing of keywords with different APID's in the instruction file should not be done and will not work when using load_m3sd executable.
5. There is no check if a record exists already. Therefore records can get overwritten in M3SD DRMS data series.
6. There needs to be created a M3SD DRMS data series using create_series utility before running load_m3sd executable.

   • Back to ["Lev0Doc"]

   • Back to FrontPage