Hprc banner tamu.png


Revision as of 14:53, 23 October 2015 by J-perdue (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Getting Started

Before you can use any of the installed software packages (e.g. compilers, libraries) you will need to setup the correct environment (e.g. set PATH, LD_LIBRARY_PATH, etc). On curie this is done using the "module" system.

To load a module use: module load PACKAGE, where PACKAGE is the name of the software you want to setup. For example, to use the xlf compiler type, module load xlf. To see all available packages type the module spider command. To see all installed versions for a specific package type, module spider PACKAGE.

To see a list of all installed packages type: module list For more information about modules, please visit, https://hprc.tamu.edu/wiki/Curie:Computing_Environment#Modules


The preferred compilers on curie are the IBM XL Compilers. The table below shows the various compilers:

Compilers Commands
Fortran compilers xlf/xlf_r
C compilers xlc/xlc_r
C++ compilers xlc++/xlc++_r

The compilers with the "_r" extension will generate threadsafe code and are used to compile threaded programs (e.g., programs containing OpenMP directives/constructs)

By default, the XL compilers generate 32-bit code. To generate 64-bit code you can use the -q64 compiler flag or set the environmental variable OBJECT_MODE to 64 (NOTE, when you load any of the XL compiler modules OBJECT_MODE is set to 64 automatically).

Serial Programs

Below are some of the most common compiler flags (these flags are accepted by all XL compilers with some notable exceptions):

Basic compiler flags

The next sections introduce some of the most common compiler flags. For a full description of all the compiler flags please consult the appropriate man pages.

Flag Description
-o <file> name of output file
-c only compile, skip linking
-L <dir> include <dir> in library search path
-l<name> searches file for libname.so or libname.a for linking
-q64 (-q32) instruct the compiler to generate 64- or 32-bit code

Optimization flags

Flag Description
-O[n] level of optimization; range from O0 to O5 (O5 highest level)
-qarch=auto specifies architecture for which the code will be generated
-qtune=auto specifies architecture for which the program is OPTIMIZED
-qsmp=auto enables automatic shared-memory parallelization (very conservative)
-qhot perform certain high-order transformations during optimization
-qessl use the intrinsics of ESSL library instead of Fortran's (FORTRAN ONLY)

Debugging flags

The table below shows some compiler flags that can be useful in debugging your program.

Flag Description
-g generate debugging info for use by a symbolic debugger
-qcheck add runtime checks for array bounds, initialized variables, etc

OpenMP Programs

Compiling OpenMP code

To compile programs containing OpenMP parallel directives/pragmas the following flag needs to be used

openmp flags:

Flag Description
-qsmp=omp Enables strict OpenMP compliance. Only OpenMP parallelization directives are recognized.

NOTE: when compiling OpenMP programs the appropriate compiler with the "_r" extension needs to be used otherwise the program will not work and most likely crash.

Example: compile OpenMP program hello_omp.c and name if hello_omp.x

xlc_r -qsmp=omp -o hello_omp.x hello_omp.c

Running OpenMP Code

OpenMP programs can be run exactly the same as serial programs. Use OpenMP environment variables to control the OpenMP behavior

Example: run the program hello_omp.x and set the number of threads to 8


MPI programs


Curie has multiple MPI stacks installed (mpich, mpich2, openmpi). To see a complete list type "module spider mpi". In this README file we will use OpenMPI but the instructions also apply to any of the other mpi stacks (there might be a few exceptions). To load the latest OpenMPI stack (including xlf compilers) type "module load xlompi".

To compile you will need to use the appropriate wrapper:

MPI wrapper Commands
Fortran wrapper mpifort
C compilers mpicc
C++ compilers mpic++

NOTE: these wrappers use the non-thread safe compilers underneath (i.e. without the "_r" extension). We will provide thread safe wrappers asap.

To see the full compiler command for any of the mpi wrappers use the "-show flag" (e.g. mpifort -show). This will show a full listing including compiler flags, LIBRARY paths, and INCLUDE paths and might be useful for debugging purposes.

Example: compile MPI program hello_mpi.c and name if hello_mpi.x

mpicc -o hello_mpi.x hello_mpi.c

Running MPI code

Unlike running serial or OpenMP code, running mpi code requires a launcher. The launcher will set up the environment and start the requested number of tasks. On curie the launcher is named "mpirun" and has the following syntax:

mpirun [mpi_flags] <executable> [exec params]

The following table shows some of the more common flags (for a full listing type "man mpirun")

Flags Descriptions
-np number of tasks to start
-hosts <list> comma separated list of hostsnames to start tasks on
-hostfile <file> specifies hostfile (containing list of hosts to use)
-map-by ppr:N:node places N processes on each node.

Example: run the program hello_mpi.x using 8 tasks, 4 on curie1, 4 on curie2.

mpirun -np 8 -hosts curie1,curie2 -map-by ppr:4:node ./hello_mpi.x