Before you can use any of the installed software packages (e.g. compilers, libraries) you will need to setup the correct environment (e.g. set PATH, LD_LIBRARY_PATH, etc). On curie this is done using the "module" system.
To load a module use: module load PACKAGE, where PACKAGE is the name of the software you want to setup. For example, to use the xlf compiler type, module load xlf. To see all available packages type the module spider command. To see all installed versions for a specific package type, module spider PACKAGE.
To see a list of all installed packages type: module list For more information about modules, please visit, https://hprc.tamu.edu/wiki/Curie:Computing_Environment#Modules
The preferred compilers on curie are the IBM XL Compilers. The table below shows the various compilers:
The compilers with the "_r" extension will generate threadsafe code and are used to compile threaded programs (e.g., programs containing OpenMP directives/constructs)
By default, the XL compilers generate 32-bit code. To generate 64-bit code you can use the -q64 compiler flag or set the environmental variable OBJECT_MODE to 64 (NOTE, when you load any of the XL compiler modules OBJECT_MODE is set to 64 automatically).
Below are some of the most common compiler flags (these flags are accepted by all XL compilers with some notable exceptions):
Basic compiler flags
The next sections introduce some of the most common compiler flags. For a full description of all the compiler flags please consult the appropriate man pages.
|-o <file>||name of output file|
|-c||only compile, skip linking|
|-L <dir>||include <dir> in library search path|
|-l<name>||searches file for libname.so or libname.a for linking|
|-q64 (-q32)||instruct the compiler to generate 64- or 32-bit code|
|-O[n]||level of optimization; range from O0 to O5 (O5 highest level)|
|-qarch=auto||specifies architecture for which the code will be generated|
|-qtune=auto||specifies architecture for which the program is OPTIMIZED|
|-qsmp=auto||enables automatic shared-memory parallelization (very conservative)|
|-qhot||perform certain high-order transformations during optimization|
|-qessl||use the intrinsics of ESSL library instead of Fortran's (FORTRAN ONLY)|
The table below shows some compiler flags that can be useful in debugging your program.
|-g||generate debugging info for use by a symbolic debugger|
|-qcheck||add runtime checks for array bounds, initialized variables, etc|
Compiling OpenMP code
To compile programs containing OpenMP parallel directives/pragmas the following flag needs to be used
|-qsmp=omp||Enables strict OpenMP compliance. Only OpenMP parallelization directives are recognized.|
NOTE: when compiling OpenMP programs the appropriate compiler with the "_r" extension needs to be used otherwise the program will not work and most likely crash.
Example: compile OpenMP program hello_omp.c and name if hello_omp.x
xlc_r -qsmp=omp -o hello_omp.x hello_omp.c
Running OpenMP Code
OpenMP programs can be run exactly the same as serial programs. Use OpenMP environment variables to control the OpenMP behavior
Example: run the program hello_omp.x and set the number of threads to 8
export OMP_NUM_THREADS=8 ./hello_omp.x
Curie has multiple MPI stacks installed (mpich, mpich2, openmpi). To see a complete list type "module spider mpi". In this README file we will use OpenMPI but the instructions also apply to any of the other mpi stacks (there might be a few exceptions). To load the latest OpenMPI stack (including xlf compilers) type "module load xlompi".
To compile you will need to use the appropriate wrapper:
NOTE: these wrappers use the non-thread safe compilers underneath (i.e. without the "_r" extension). We will provide thread safe wrappers asap.
To see the full compiler command for any of the mpi wrappers use the "-show flag" (e.g. mpifort -show). This will show a full listing including compiler flags, LIBRARY paths, and INCLUDE paths and might be useful for debugging purposes.
Example: compile MPI program hello_mpi.c and name if hello_mpi.x
mpicc -o hello_mpi.x hello_mpi.c
Running MPI code
Unlike running serial or OpenMP code, running mpi code requires a launcher. The launcher will set up the environment and start the requested number of tasks. On curie the launcher is named "mpirun" and has the following syntax:
mpirun [mpi_flags] <executable> [exec params]
The following table shows some of the more common flags (for a full listing type "man mpirun")
|-np||number of tasks to start|
|-hosts <list>||comma separated list of hostsnames to start tasks on|
|-hostfile <file>||specifies hostfile (containing list of hosts to use)|
|-map-by ppr:N:node||places N processes on each node.|
Example: run the program hello_mpi.x using 8 tasks, 4 on curie1, 4 on curie2.
mpirun -np 8 -hosts curie1,curie2 -map-by ppr:4:node ./hello_mpi.x