Using the make system |
The JSOC make system uses what is known as “nonrecursive make”. “Recursive make”, by contrast, refers to a hierarchy of directories, each of which contains source files and a file named Makefile. To make all projects, the user would run make on Makefile at the root of the tree, which would then cause make to cd to each subdirectory in the tree, operating on each Makefile throughout the tree. We use the nonrecursive method because full builds complete much faster than full builds using the recursive method, and because in the latter, it is difficult to maintain inter-directory dependencies (please see item E on http://jsoc.stanford.edu/jsocwiki/JsocDevelopersGuide for more information about the rationale for using recursive make).
Nonrecursive make entails a single Makefile file, a make_basic.mk file, a target.mk file, and a custom.mk, all of which live in the root of each user’s CVS tree (typically $HOME/cvs/JSOC, aka the source root). It also entails one Rules.mk file in each subdirectory. make_basic.mk contains variables and make rules that are common to all subdirectories, and the Rules.mk files contain variables and make rules that are specific to the directory in which they lie. target.mk is responsible for creating the “target directories”. When you check-out the JSOC source code from CVS, you end up with a tree whose root is …/JSOC. Source code is contained within the base and proj subdirectories. However, when you run make additional directories are created. One such directory is the root target directory that has a platform-dependent name: …/JSOC/_$ JSOC_MACHINE. Within this directory are found compiled binary files: .o, .o.d, executables, and libraries. For each source subdirectory, there is a parallel target subdirectory (eg, …/JSOC/base/drms/libs/api/client/ --> …/JSOC/_linux_x86_64/base/drms/libs/api/client/). For each source subdirectory, target.mk has one directive that will make the corresponding target subdirectory. Importantly, target.mk contains target-directory directives for only the base JSOC code (subdirectories of …/JSOC/base). custom.mk is an optional file created by the configure script to allow for site-dependent additions to make_basic.mk.
Inside the …/JSOC/proj directory there is also a target.mk, which serves the same purpose as target.mk in the root directory, but it contains target-directory directives for the subdirectories within …/JSOC/proj – the root-level target.mk does not contain this information. Target-directory directives were split between …/JSOC/target.mk and …/JSOC/proj/target.mk to facilitate the distinction between the full JSOC code and NetDRMS (which does not contain any Stanford-specific projects).
These make files are all linked together via the make ‘include’ directive to create essentially one big make file. Makefile includes the make_basic.mk and target.mk contents, and make_basic.mk includes the Rules.mk in the root directory. Then that Rules.mk includes the Rules.mk files in each subdirectory. Each Rules.mk file always includes the contents of the Rules.mk files in each child subdirectory. target.mk also includes …/JSOC/proj/target.mk (if it exists) .
A module writer will generally focus on the Rules.mk files. These are the files that contain module-specific rules. One job the module writer has is to maintain the parent-child link between Rules.mk files. If you add a new subdirectory, you are going to have to add a Rules.mk file for that subdirectory, and a link from the parent directory’s Rules.mk to the new subdirectory’s Rules.mk. It is beneficial to examine a snippet from …/JSOC/proj/Rules.mk to understand how the linking is specified:
dir := $(d)/datacapture -include $(SRCDIR)/$(dir)/Rules.mk dir := $(d)/dsdsmigr -include $(SRCDIR)/$(dir)/Rules.mk dir := $(d)/example
where $(SRCDIR) is the root directory …/JSOC, and $(d) is the current subdirectory. This shows that if …/JSOC/proj/datacapture/Rules.mk exists, it will be included into the current …/proj/Rules.mk file.
Creating a Rules.mk file
At the top and bottom of each Rules.mk file are special variables and directives needed for the non-recursive make to work. You should never need to edit these sections:
# Standard things at the top of the file sp := $(sp).x dirstack_$(sp) := $(d) d := $(dir) ... # Standard things at the bottom of the file d := $(dirstack_$(sp)) sp := $(basename $(sp))
These keep track of the current directory on which make is operating. ‘d’ is a special variable that contains the current path (of the current Rules.mk) relative to the root source directory. Between these two sections resides variable assignments and rules that vary from Rules.mk to Rules.mk. Although the following discusses the basics of three different types of Rules.mk files, in reality a Rules.mk may include elements of more than one type.
Linking Rules.mk
The purpose of some Rules.mk files is to simply link the root Rules.mk file to a lower-level Rules.mk file. For example, …/JSOC/base/Rules.mk contains several “include” directives to include the Rules.mk files from …/JSOC/base/sums, …/JSOC/base/drms, etc. The Rules.mk file has no other purpose. It doesn’t specify how to compile or link code. These types of Rules.mk, contain a single additional section that has these include directives:
dir := $(d)/sums -include $(SRCDIR)/$(dir)/Rules.mk dir := $(d)/drms -include $(SRCDIR)/$(dir)/Rules.mk ...
Rules.mk Containing Executable Rules
Many Rules.mk files contain the rules for compiling and linking executables. These files contain several additional sections between the standard beginning and ending sections. The first additional section of such Rules.mk files contains variable assignments. There are two types of variables – global and local variables. The intent behind this dichotomy is to allow some variables to be visible outside of the file in which they are defined (global variables), but permit others to be visible only within the file in which they are defined (local variables). In particular, global variables are usually referenced from within make_basic.mk. In general, for each output binary file, like a library or executable, there is a global variable to hold the name of the binary:
LIBDRMSCLIENT := $(d)/libdrmsclient.a
Local variables contain a ‘_$(d)’ in their name. $(d) gets evaluated to the current path relative to the root directory. By directly placing this path in the name of the variable it is clear which Rules.mk ‘owns’ the variable.
MODEXE_$(d) := $(addprefix $(d)/, hello_world threadsigs threadalrm)
Global and local variables are used to hold lists of files on which specific prerequisites are established or specific actions are run. This is achieved by placing these lists into the appropriate part of a rule. Remember, rules can contain not only prerequisites, but also actions, where the action can be one of many different types of commands, like compiling, linking, removing files, creating symbolic links, etc. As an example, consider the following canned rule (from make_basic.mk):
$(CEXE): %: %.o $(EXELIBS)
$(LINK)
$(SLBIN)This rule specifies that each file name in the list contained in the variable $(CEXE) has a prerequisite object file whose name is formed by taking the file name and appending a “.o”. It also has as prerequisites the files whose names are listed in the $(EXELIBS) variable. And the actions to execute when make is run on one of these targets are specified by the $(LINK) and $(SLBIN) variables – the former provides the linker command that will link together the .o files and libraries into an executable, and the latter contains the ln command that will make a symbolic link from …/JSOC/bin to the executable in the target directory (LINK is not shown here – it is a bit complicated):
SLBIN = ln -sf ../../_$(JSOC_MACHINE)/$@ ../bin/$(JSOC_MACHINE)/
This canned rule “knows” how to make a C executable – it knows how what libraries to link against, what compiler-linker command to run, and how to create a symbolic link from …/JSOC/_linux_x86_64/bin to the actual executable. Several common “types” of executables such as this one have been recognized. Instead of requiring users to define the rules to create such an executable in every Rules.mk where needed, the variable CEXE was placed in make_basic.mk so that it is available to all Rules.mk files. If one were to assign to CEXE, in a Rules.mk file, the name of an executable, like “base/drms/apps/drms_run” (file paths are relative to the source root), then the make system would understand that a prerequisite for …/JSOC/_linux_x86_64/base/drms/apps/drms_run is …/JSOC/_linux_x86_64/base/drms/apps/drms_run.o, and the library files listed in EXELIBS. The make system would then find the rules to make …/JSOC/_linux_x86_64/base/drms/apps/drms_run.o and the library files. Then it would run the compiler to link together …/JSOC/_linux_x86_64/base/drms/apps/drms_run. Then it would create a link, JSOC/bin/drms_run that points to the actual built executable.
To use this rule to make a “CEXE” executable, simply append the relative path to the executable to the global CEXE variable in a Rules.mk:
CEXE := $(CEXE) proj/myproj/myexe
It is important to not simply assign proj/myproj/myexe to the global variable CEXE, as this would completely overwrite CEXE, even if it already contained a list of executables. In addition to CEXE, make_basic.mk defines rules for several common types of executable (see Table 1).
Table 1 |
|
Target Variable |
Description |
CEXE |
C executable (not a DRMS module) that links against all client libraries (libdrmsclient.a, libdefsclient.a, libdbclient.a, libthreadutil.a, libricecomp.a, libcmdparams.a, libdstruct.a, libtimeio.a, libfitsrw.a, liberrlog.a, libmisc.a). Use this rule if you want to make a C executable that has its own main() function, not the one provided by jsoc_main_sock (which operates through drms_server). CEXE executables can optionally connect to a drms_server process, but typically this is not the case. |
FEXE |
FORTRAN executable (not a DRMS module) that links against FORTRAN math libraries. FEXE executables cannot connect to any drms_server process. |
SERVEREXE |
C executable (not a DRMS module) that links against all server libraries (libdrms.a, libdefsserver.a, libdb.a, lubsumsapi.a, libthreadutil.a, libricecomp.a, libcmdparams.a, libdstruct.a, libtimeio.a, libfitsrw.a, liberrlog.a, libmisc.a). Use this rule if you want to make a C executable that has its own main() function, not the one provided by jsoc_main. This executable can optionally connect directly to the DRMS database, but it must the executable creator must call the correct DRMS API functions to do so. |
MODEXE |
C executable that is also a direct-connect DRMS module (links with jsoc_main, which establishes a direct DRMS-database connection). It links against all server libraries (libdrms.a, libdefsserver.a, libdb.a, lubsumsapi.a, libthreadutil.a, libricecomp.a, libcmdparams.a, libdstruct.a, libtimeio.a, libfitsrw.a, liberrlog.a, libmisc.a). |
MODEXE_SOCK |
C executable that is also a socket-connect DRMS module (links with jsoc_main_sock, must connect to drms_server, which in turn connects to DRMS database; can ‘self-start’, which means that the module can fork a drms_server process and use it to access the DRMS database). It links against all client libraries (libdrmsclient.a, libdefsclient.a, libdbclient.a, libthreadutil.a, libricecomp.a, libcmdparams.a, libdstruct.a, libtimeio.a, libfitsrw.a, liberrlog.a, libmisc.a). |
FMODEXE_SOCK |
FORTRAN executable that is also a socket-connect DRMS module (links with jsoc_main_sock, must connect to drms_server, which in turn connects to DRMS database; can ‘self-start’, which means that the module can fork a drms_server process and use it to access the DRMS database). It links against an interface library, libinthandles_f.a , and all client libraries (libdrmsclient.a, libdefsclient.a, libdbclient.a, libthreadutil.a, libricecomp.a, libcmdparams.a, libdstruct.a, libtimeio.a, libfitsrw.a, liberrlog.a, libmisc.a). |
MODEXE_USEF |
Like MODEXE, except that is linked with a FORTRAN linker, and links against FORTRAN math libraries. |
MODEXE_USEF_SOCK |
Like MODEXE_SOCK, except that is linked with a FORTRAN linker, and links against FORTRAN math libraries. |
Generally, these list-variables contain object files and binaries, and it is possible, in fact common, for a particular file to reside in more than one list. If a file belongs to more than one list, then it is possible to put it into one list to cause it to be the object of a certain action, and put it into a second list to define a set of prerequisites for it. For example, if there were two global rules:
$(MODEXESUMS) : $(LIBSUMSAPI) $(LIBSUM) $(MODEXE): LL_TGT := $(LL_TGT) -lpq $(CFITSIOLIBS)
and you placed “base/sums/apps/sum_get” into both $(MODEXESUMS) and $(MODEXE), then you would define the libraries in $(LIBSUMSAPI) and $(LIBSUM) as prerequisites for sum_get, you would append “-lpq” to the LL_TGT variable for sum_get, and you would define the libraries in $(CFITSIOLIBS) as prerequisites for sum_get.
Getting back to the business of writing a Rules.mk file, the executable names are often placed into local-variable lists, that are in turn placed into one of the eight predefined global executable variables (see Table 1):
MODEXE_$(d) := $(addprefix $(d)/, hello_world threadsigs threadalrm) MODEXE := $(MODEXE) $(MODEXE_$(d))
All .o files are typically placed into local-variable lists:
OBJ_$(d) := $(MODEXE_$(d):%=%.o) $(MODEXE_SOCK_$(d):%_sock=%.o) OBJUSEF_$(d) := $(MODEXE_USEF_SOCK_$(d):%_sock=%.o) OBJF_$(d) := $(FMODEXE_SOCK_$(d):%_sock=%.o)
Then these various object-file lists are combined into a local variable named DEP_$(d):
DEP_$(d) := $(OBJ_$(d):%=%.d) $(OBJUSEF_$(d):%=%.d) $(OBJF_$(d):%=%.d)
which is then included into the current Rules.mk:
-include $(DEP_$(d))
make is configured to track dependencies between .o and .c/.h files – this line includes those dependencies into the Rules.mk so that if one modifies one of the .c or .h files, the .o file gets rebuilt.
All .o, binary, and dependency files are then added to the global CLEAN target variable. Doing so causes a ‘make clean’ to remove all these files from the target directories:
CLEAN := $(CLEAN) \
$(OBJ_$(d)) \
$(OBJF_$(d)) \
$(MODEXE_$(d)) \
$(MODEXE_USEF_SOCK_$(d)) \
$(MODEXE_SOCK_$(d))\
$(FMODEXE_SOCK_$(d) \
$(FMODEXE_GONG_$(d)) \
$(DEP_$(d))Another global variable, TGT_BIN, holds the paths to all binaries that should, by default, be built when a user runs ‘make’ or ‘make all’:
TGT_BIN := $(TGT_BIN) $(MODEXE_$(d)) $(MODEXE_SOCK_$(d))
A specific target can be built by running “make <target>“, where target is the relative path to the desired target binary (eg., “make base/sums/apps/sum_get”). To facilitate typing make commands, a special variable S_$(d), in conjunction with the following rules:
.PHONY: $(S_$(d)) $(S_$(d)): %: $(d)/%
is used to create target shortcuts. By placing “sum_get” into into S_$(d), you can build sum_get by typing “make sum_get”.
The next and final section contains localized rules. In contrast to the rules defined in make_basic.mk, these apply to the current Rules.mk and nowhere else. This section typically contains prerequisite definitions, such as dependencies on third-party libraries, and compile and link flags (like “-L” and “-l” flags). The canned rules in Table 1 use the variables CF_TGT and FF_TGT, which contain C and FORTRAN compile flags. They also contain LF_TGT and LL_TGT, which contain general link flags and strings containing “-L” and “-l” flags, respectively. If in a Rules.mk file flags are assigned to these variables, then the canned rules will use those flags. Notice, however, that these variables appear to be global variables. If a Rules.mk were to contain:
LL_TGT := -L/usr/local/lib -lmkl_em64t
then the link command of ALL targets would use this value as a link flag. To avoid this, Rules.mk files generally use target-specific variables:
$(MYEXE) : LL_TGT := $(LL_TGT) -L /usr/local/lib -lmkl_em64t
This rule will append to LL_TGT “-L /usr/local/lib -lmkl_em64t”, but only for the targets listed in $(MYEXE).
Rules.mk Containing Library Rules
Many Rules.mk files contain the rules for building libraries. Like Rules.mk files for executables, these Rules.mk contain several additional sections between the standard beginning and ending sections.
Similar to a Rules.mk for executables, a Rules.mk for libraries typically contains a global variable that holds the relative path (relative to the target root) to the library:
LIBASTRO := $(d)/libastro.a
Object files are also stored in one or more local variables, and the DEP_$(d), CLEAN, TGT_LIB, and S_$(d) variables are defined as before (see Rules.mk Containing Executable Rules for more information). As before, the following typically exist immediately before the last section:
.PHONY: $(S_$(d)) $(S_$(d)): %: $(d)/% -include $(DEP_$(d))
Unlike the case for executable rules, there are no canned rules in make_basic.mk to assist with the creation of libraries. A library rule looks like the following:
$(LIBASTRO):$(OBJ_$(d))
$(ARCHIVE)
$(SLLIB)The object files are the prerequisites for the library file. The action to take is entirely up to the Rules.mk writer, but there are a couple of standard actions the writer can use if desired. The ARCHIVE variable runs the ar command to produce the library file. It adds all the prerequisite files (.o files) as members of the archive:
ARCHIVE = ar crus $@ $^
The SLLIB simply creates a symbolic link from …/JSOC/lib to the library.
SLLIB = ln -sf ../../_$(JSOC_MACHINE)/$@ ../lib/$(JSOC_MACHINE)/
Adding a subdirectory
To add a subdirectory, you need to add a Rules.mk file to the subdirectory, and then modify the Rules.mk file in the parent directory to contain a link to the child Rules.mk. The entry in the parent Rules.mk looks like:
dir := $(d)/newdirectory -include $(SRCDIR)/$(dir)/Rules.mk
In addition, you must add an entry to the appropriate target.mk file (for subdirectories of …/JSOC/proj, use …/JSOC/proj/target.mk). make will read this entry and create the corresponding target subdirectory. Assuming you are working in the …/JSOC/proj subtree, you would add a line like this to …/JSOC/proj/target.mk:
$(PROJOBJDIR):
...
+@[ -d $@/projdir/apps ] || mkdir -p $@/projdir/apps$@/projdir/apps refers to the subdirectory …/JSOC/proj/projdir/apps.
It is important to keep in mind the general code-tree structure of projects (see http://jsoc.stanford.edu/jsocwiki/FileStruct for more information). For example, you might need to add not a single directory, but multiple directories: …/JSOC/proj/newproj/apps and …/JSOC/proj/newproj/libs. In this case, you would need to add three new directories, and three new Rules.mk files properly linked to the parent Rules.mk files.
How To...
Add a compile flag to the source of an executable
- Create a local-variable containing a list of the object files that require the flag:
OBJ_$(d):= $(addprefix $(d)/, procFdsData.o interp.o obs2helio.o)
- In the local rules section, append the flag to the CF_TGT target-specific variable (for C code), or the FF_TGT target-specific variable (for FORTRAN code):
$(OBJ_$(d)): CF_TGT := $(CF_TGT) -DCDIR="\"$(SRCDIR)/$(d)\""
Add a link flag to the source of an executable
- Create a local-variable containing a list of the executables that require the flag:
EXE_$(d):= $(addprefix $(d)/, procFdsData interp obs2helio)
- In the local rules section, append the flag to the LF_TGT target-specific variable:
$(EXE_$(d)): LF_TGT := $(LF_TGT) -nofor_main -no-ipo
Add a library dependency to link against
- There are a few global variables defined that will provide linking against fftw3 (FFTWH and FFTW3LIBS), fftw3f (FFTWH and FFTW3FLIBS), cfitsio (CFITSIOH and CFITSIOLIBS), gsl (GSLH and GSLLIBS), and tar (LIBTARH and LIBTARLIBS). To link against gsl for example:
- Create a local-variable containing a list of the executables that require the gsl libraries:
GSLEXES_$(d) := $(addprefix $(d)/, mod1 mod2)
- Create a local variable containing a list of the object files that use gsl:
OBJ_$(d) := $(GSLEXES_$(d):%=%.o)
- In the local-rules section, append the path to the library headers to the target-specific CF_TGT variable (for C code):
OBJ_$(d) : CF_TGT := $(CF_TGT) $(GSLH)
- Append the complete library link flags to the target-specific LL_TGT variable:
GSLEXES_$(d) : LL_TGT := $(LL_TGT) $(GSLLIBS)
- To link against a library built by the DRMS make system:
- Follow 1.a. and 1.b above.
- In the local rules section, append the relative paths to the library headers to the target-specific CF_TGT variable (for C code):
$(OBJ_$(d)): CF_TGT := $(CF_TGT) -I$(SRCDIR)/$(d)/../../libs/json -I$(SRCDIR)/$(d)/../../libs/jsmn -I$(SRCDIR)/$(d)/../libs/util
- Add the make variables holding the DRMS library paths to the list of dependencies for the DRMS executables:
$(MODEXE_$(d)): $(LIBJSON) $(LIBJSMN) $(LIBEXPDRMS) $(LIBEXPUTL) $(LIBQDECODER)
- To link against a pre-built system library:
- Follow 1.a. and 1.b above.
- In the local rules section, append the path to the library headers to the target-specific CF_TGT variable (for C code):
OBJ_$(d) : CF_TGT := $(CF_TGT) -I/home/jsoc/include
- Append the complete library link flags to the target-specific LL_TGT variable:
GSLEXES_$(d) : LL_TGT := $(LL_TGT) -L/home/jsoc/lib/linux_x86_64 -ljsoclib
Add a binary to the list of binaries built by default
- Append the relative path (relative to the target root) of the binary to the TGT_BIN global variable:
SERVEREXE_$(d) := $(addprefix $(d)/, drms_sever) TGT_BIN := $(TGT_BIN) $(SERVEREXE)
Add a binary to a shortcut
- Add the filename (not the path) of the binary to the S_$(d) variable:
S_$(d) := $(notdir $(MODEXE_$(d)))
- Ensure that the following rules follow the S_$(d) variable definition:
.PHONY: $(S_$(d)) $(S_$(d)): %: $(d)/%
Use make’s dependency tracker
If you use the dependency tracker, then if any dependency of the target of interest is edited or touched in any way, the targets that depend on the dependency will get rebuilt the next time make is called. But only targets that need to be rebuilt get rebuilt.
- Create a local-variable containing a list of the object files whose dependencies you want to track:
OBJ_$(d):= $(addprefix $(d)/, procFdsData.o interp.o obs2helio.o)
- Put this list of object files into the DEP_$(d) variable:
DEP_$(d) := $(OBJ_$(d) : %=%.d)
- Ensure the following exists after the definition of the first two variables:
-include $(DEP_$(d))