CVS Initialization

First time user?

If all you need to do right now is to finish your first JSOC setup, just run the commands here then (when they finish) go back to the previous page and take the next step. 

% cd
% mkdir cvs
% cd cvs
% /home/jsoc/dlsource.pl 
% cd JSOC
% ./configure
% make

The last step will pause for you to confirm all is OK. It probably is, so just type "cont". When the above finishes go back a page and proceed to the next "new user" step.

Since you will have some time while jsoc_update is running, feel free to read the following for background information about CVS: (All the steps that need to be taken to get started are in the above scripts, so just read on or skip for now).

What is CVS

All of the software for the JSOC-SDP is maintained in the "CVS" software configuration management system. In order to make new code using the DRMS you will need to establish a CVS work area, download the JSOC software tree into it, configure it for the machine you are using for code development, and build the libraries and programs. Then you will be able to add your own pipeline modules or other tools.

What is CVS anyway? It's a code manager that allows check-in and check-out of programs from the same directory structure. In a nutshell, there is one official set of source-code files that resides in one or more file directories in a remote CVS server. This set is called the "repository". A developer who wishes to edit these files must first "checkout" (download) the files to a "working directory" on their local work machine. This working directory, which is simply a user's local directory containing a copy of the respository files, is sometimes called a "sandbox". The developer makes changes, then "commits" (uploads) the files back to the CVS server. Commiting is not the same thing as copying, since that would overwrite changes made by other developers who made changes between the checkout and commit of a file by another developer. Instead, CVS will MERGE changes back to the repository, i.e., it will determine exactly what code has changed and insert that change into the respository file. This works as long as two developers do not make conflicting changes (e.g., changing the same line of code in different ways). If a conflict occurs, the developer who is attempting to commit a change that conflicts with a previous change needs to manually edit the file in their working directory (ensuring that both their change and the previous changes are both contained within the working-directory file), and then commit the file. To learn more about CVS, read: a CVS Wiki

The base JSOC system is revised often. The instructions for the most recent release should be consulted to make sure the following is still correct. See the Developer's Guide for a list of recent releases.

Create a CVS Working Directory

This step only needs to be done once. The normal place to put your JSOC working directory is in a "cvs" subdirectory in your home directory. E.g.: 

% cd
% mkdir cvs
% mkdir cvs/JSOC

Many users find it convenient to make a link to the JSOC tree. E.g.: 

% cd
% ln -s cvs/JSOC .

If you had an old "jsoc" link it might help old fingers to rename the old cvs/jsoc to cvs/old_jsoc then link ~/jsoc to ~/cvs/JSOC

Set Your Environment

If you followed our advice, this step was accomplished by including "source /home/jsoc/.setJSOCenv" in your .cshrc before you got to this page. This step also needs to be done but once. To use CVS and JSOC, you must set four environment variables:

In greater detail, a Stanford user should do the following. Set the CVSROOT environment variable to contain ":ext:sunroom.stanford.edu:/home/cvsuser/cvsroot". Set the CVS_RSH environment variable to contain "ssh". Set the JSOCROOT environment variable to contain "$HOME/cvs/JSOC". Set the JSOC_MACHINE environment variable to contain a string that correctly identifies your machine architecture (one of "linux_ia32", "linux_ia64", or "linux_x86_64"). It is best to let a script do this for you (see below).

Regardless of user type, you should create an alias to "$JSOCROOT/base/util/scripts/cvsstatus.pl" named "cvsstatus". This script prints the status of all CVS repository files rooted at the current working directory.

It is best to put all these commands in a script that gets called from one of your initialization scripts (e.g., .login, .csh). Your script should look similar to the following: 

setenv CVSROOT :ext:sunroom.stanford.edu:/home/cvsuser/cvsroot
setenv CVS_RSH ssh
setenv JSOCROOT $HOME/cvs/JSOC

set OS = `uname -s`
switch("$OS")
  case "Linux*":
    set CPU = `uname -m`
    breaksw
  default:
    set CPU = `uname -p`
    breaksw
endsw

if ( ! $?JSOC_MACHINE ) then
  if ( $OS == "Linux" ) then
    switch("$CPU")
    case "i686"
    case "i386"
    case "ia32"
      set JSOC_MACHINE = "linux_ia32"
      breaksw
    case "ia64"
      set JSOC_MACHINE = "linux_ia64"
      breaksw
    case "x86_64"
    case "em64t"
      set JSOC_MACHINE = "linux_x86_64"
      breaksw
    default:
      set JSOC_MACHINE = "custom"
      breaksw
    endsw
  else
     set JSOC_MACHINE = "custom"
  endif
endif

setenv JSOC_MACHINE $JSOC_MACHINE
alias cvsstatus "$JSOCROOT/base/util/scripts/cvsstatus.pl"

In addition, in order to run JSOC programs and scripts without having to explicitly provide an absolute path to these executables, add JSOCROOT/bin/$JSOC_MACHINE and $JSOCROOT/scripts to your path environment variable.

IMPORTANT TIP: There is actually a cookie-cutter scripts that you can use that will automatically perform all of these task. To use this, put "source /home/jsoc/.setJSOCenv" in your .cshrc (or similar file).

Download the JSOC Source-Code Tree

The JSOC CVS respository is divided into two sets of files. In the "base" subdirectory of the tree are lower-level source files used to build code that provides access to the DRMS database and data series. The DRMS library is built from the source files in this subdirectory. They are also used to build utility programs that perform common database-access tasks (e.g., querying the database to find information on a particular data series). In the "proj" subdirectory of the source tree are source files that are primarily used to build code that perform JSOC-SDP-specific actions e.g., parsing telemetry packets and ingesting the extracted data into data series). Each JSOC-SDP project has a subdirectory in this "proj" subdirectory. This code frequently contains constructs that will cause it to only run at the JSOC-SDP; however certain projects do not have this restriction. The proj subdirectory also contains a very small set of projects that are not JSOC-SDP-specific (e.g., the example project, which provides some example modules).

We provide support for three types of users:

Each type of user must "check out" the appropriate set of JSOC CVS files (i.e., download the files from the remote CVS server to their working directory on their local machine). This need be done only once (barring major changes to the JSOC CVS repository). Thereafter, the user can "update" their local source tree (see below).

To check out all the JSOC-SDP-specific set of files (including the base DRMS system files), a JSOC-SDP user does the following: 

% cd $JSOCROOT/..
% /home/jsoc/dlsource.pl

To check out the base DRMS system, a NetDRMS user does the following: 

% cd $JSOCROOT/..
% /home/jsoc/dlsource.pl -o checkout -f net

To perform a custom checkout, a Custom user needs to first create a configuration file that specifies the desired project directories. Then the user does the following: 

% cd $JSOCROOT/..
% /home/jsoc/dlsource.pl -o checkout -f <config file>

where <config file> is the path to the configuration file.

Creating a configuration file

The configuration file used for Custom checkouts is a text file that contains three sections. An example of this configuration file is located here. The first section, delimited by the tags __MAKE__ and __END__, contains global make variables that apply to all projects. The various project Rules.mk files can refer to these variables as needed.

The second section, delimited by the tags __PROJ__ and __END__, contains xml that specifies the desired set of project directories. The example configuration file includes all project directories currently available. If desired, it can be used as a template to be modified. The user can delete undesired project directories.

The configuration file contains one "proj" element for each project directory. Each such element includes the name of the project directory, a list of any subdirectories in that directory, and a list of filters: 

<proj>
    <name>mag</name>
    <subdirs>
        <subdir>apps</subdir>
        <subdir>pfss/apps</subdir>
        <subdir>ambig/apps</subdir>
        <subdir>ident/apps</subdir>
        <subdir>ident/libs/mex2c</subdir>
        <subdir>ident/libs/mexfunctions</subdir>
        <subdir>ident/libs/util</subdir>
        <subdir>patch/apps</subdir>
    </subdirs>
    <filters>
        <filter>
            <name>COMPILER</name>
            <value>icc</value>
        </filter>
    </filters>
</proj>

The name of the project directory is simply the case-sensitive name of the subdirectory under $JSOCROOT/proj. The set of such subdirectories is implemented as an xml element that contains one "subdir" sub-element per subdirectory. Each subdir element is the case-sensitive name of a subdirectory under $JSOCROOT/proj/<projname>. Every subdirectory which the user would like to build must have an associated subdir element. Finally, each filter describes the conditions (make variables actually) under which the project will build. In the xml snippet above, the filter in the "mag" project will ensure that the project "mag" will only build on a machine where the compiler is "icc". These filters ensure that there is no attempt to build code that does not build on certain platforms, or that does not build with certain compilers. For example, the following filter will ensure that a project is not built on the ia32 platform: 

<filter>
    <name>JSOC_MACHINE</name>
    <value>linux_x86_64</value>
</filter>

Please contact Art if you'd like help creating filters.

The third section contains a project-specific script that that will be run by the master configure script in $JSOCROOT. In the example configuration file, this script creates links from the $JSOCROOT/scripts directory to the $JSOCROOT/proj/lev0 and $JSOCROOT/proj/datacapture scripts. Because $JSOCROOT/scripts is in the user's path environment variable (if the user follows this set-up documentation), the effect of this project-specific script is to make lev0 and datacapture scripts available via the the user's path variable.

Tables

In addition to the DRMS system, a large set of JSOC-specific data tables exist. These files are not needed by most users. For example, a large subset of these tables is relevant to Level-0 processing. To check out a SUBSET of these JSOC-specific data tables, a user does one of the following: 

% cd $JSOCROOT/..
% /home/jsoc/checkoutJSOC.pl <subset>

or 

% cd $JSOCROOT/..
% cvs checkout -AP <subset>

where <subset> is one of the following: LEV0TBLS (tables relevant to Level-0 processing) and TBL_JSOC (all tables).

The Stanford user 'production' uses some of these tables in a read-only fashion. <subset>, for 'production' only, is one of the following: PROD_LEV0TBLS. The files for 'production' always get downloaded to a directory named 'lev0'. Therefore, production should do one of the following to obtain these files: 

% cd /home/production
% /home/jsoc/checkoutJSOC.pl <subset>

% cd /home/production
% cvs checkout -AP <subset>

The volume of these files is very large, ~200MB. The ONLY users who should download these files in this manner are people who will be modifying these files.

Note: If CVS asks you for a password or fails to checkout the code, make sure you have created a public and private ssh key under your ~/.ssh directory. You can ask our sys-admin for help (action@sun).

Update Your Existing Working Directory ("Sandbox")

**IMPORTANT NOTE FOR JSOC-SDP USERS**

If jsoc_update.pl does not work please use jsoc_sync.pl. In contrast to jsoc_update.pl, which updates your sandbox with changes to the server repository, and then builds the out-of-date binaries, jsoc_sync.pl simply updates sandbox files. It does not build new binaries. To simulate the behavior of jsoc_update.pl with jsoc_sync.pl do the following: 

% cd $JSOCROOT
% /home/jsoc/jsoc_sync.pl
% ./configure
% make

Although jsoc_sync.pl appears to be more stable than jsoc_update.pl, it is also possible that jsoc_sync.pl will fail. This is usually due to a corrupt CVS sandbox. If jsoc_sync.pl fails, the best thing to do is: 

% cd $JSOCROOT/..
% mv JSOC JSOC_old
% /home/jsoc/dlsource.pl
% cd JSOC
% ./configure
% make

Any uncommitted changes will reside in JSOC_old. You will have to find all edited files in JSOC_old and copy them into JSOC before you can commit them into CVS.

**END NOTE**

As CVS is a multi-user system, other users will change the repository files after your initial checkout of these files. To get these changes incorporated into your working directory, you need to "update" your working directory. Although you can do this using CVS's update command, a couple of scripts that reside inside your working directory are available. It is highly recommended that these scripts be used to update your working directory. Improper use of "cvs update" could result in an unstable working directory.

To update the source code in $JSOCROOT and build the default binaries for all supported architectures, use jsoc_update.pl. This works for any user, Stanford and non-Stanford alike. As this builds every executable on every architecture, it takes longer to run than jsoc_sync.pl.[Actually since we now support only one architecture, linux_x86_64, the time-to run will be comparable]. However, if you simply want to run JSOC programs, but not develop new code, then this script is sufficent. 

% cd $JSOCROOT
% /home/jsoc/jsoc_update.pl [ <config file> ]

To update your source-code without building programs, use the jsoc_sync.pl script. Again, regardless of the type of user, this works. 

% cd $JSOCROOT
% /home/jsoc/jsoc_sync.pl [ <config file> ]

<config file> is optional, and is used only when updating a custom checkout, as described above. If the user is updating a custom checkout, and <config file> is not provided, then the set of files to be updated will be equivalent to the set of files last updated, or if an update has never been performed, the set updated with match the set of files previously checked out. If <config file> is provided, then the files specified in <config file> will be the files updated.

A good time to run one of these scripts is when a major release occurs (you will be notified by email, provided you are on the jsoc_dev mail list).

NOTE - If jsoc_update.pl or jsoc_sync.pl indicates a CVS file conflict (check $JSOCROOT/cvsupdate.log, which is generated by these scripts, to see if a line of the form "C <filename>" exists), then you'll need to resolve the conflict before continuing (see the CM if necessary).

Running 'make'

After checking-out the CVS tree for the first time, and after every update of the tree, the user must run the configure script. The JSOC-SDP and NetDRMS users should run the following: 

% cd $JSOCROOT
% ./configure

Users of a custom checkout should run the following: 

% cd $JSOCROOT
% ./configure <config file>

Once again <config file> is used for a Custom checkout only. However, this time it is required. The file is necessary because it provides the configure script with information necessary for creating several make files needed by the make system.

Running 'make' will perform a 'default make' which builds a subset of all possible binaries that can be built. The binaries will be optimized for speed and contain no debug information. To read how to tune this behavior click here.

To manually build the JSOC executables using the default make, run the following: 

% cd $JSOCROOT
% make

If the make file fails because it doesn't find the postgres 'ecpg' preprocessor, then you'll need to install the postgresql-devel package. E.g. "sudo yum install postgresql-devel".

JSOC compilation assumes the following RPMs are installed:

Notes

A new make system was put in place after Ver_3-5-PreRelease. If you have a CVS working directory based on that version or an earlier version, you'll need to run "make clean" before running jsoc_update.pl, jsoc_sync.pl, or cvs update.

JsocWiki: CvsInit (last edited 2021-03-24 04:05:39 by PhilScherrer)