Hprc banner tamu.png

Curie:Computing Environment

Jump to: navigation, search


All nodes on Curie operate under the control of the LINUX operating system. The installed version is the Red Hat Enterprise Linux 6.6 implementation. The global file systems (accessible to all nodes) are organized by IBM's General Parallel File System (GPFS). These are /home and /scratch which are shared with Ada Cluster. Both are hosted on the 4PB (raw) disk space provided by the IBM GSS26 mass storage appliance.

The batch system, although an important part of the computing environment, is covered in a separate section of its own.

The Login Part

The default shell for Curie (all nodes) is bash. Its environment is defined in the startup file, .bash_profile. It is read and executed when you login, and it resides in your home directory. The command,

$ env

when executed right after login, displays the default environment settings.

Many applications use the TMPDIR environment variable, when set, as the area for storing temporary files. By default TMPDIR is not set, whereupon the application programs will typically use the /tmp area on the local drive, not always a safe option because of its limited capacity. Setting it to a directory in your scratch area is a good solution.

The use of the login nodes is for small-to-medium code development and processing. The most important processing limits here are:

  • Cpu time - 1 hour per login session.
  • Cores for Concurrent Use - 8 per login, on the same node or (cummulatively) across all login nodes.

File Space

Immediately upon logging in to Curie, the following message about the status of your disk space use greets you:

Your current disk quotas are:
Disk       Disk Usage      Limit    File Usage      Limit
/home           2.49G        10G           113      10000
/scratch        1.25G         1T            40      50000
Type 'showquota' to view these quotas again. 

The gist of the information is that the file space allocated for your home directory, /home/NetID, is just 10GB, while the allocation for scratch area, /scratch/user/NetID, is 1TB.

Your scratch space is extensible for justifiable needs. It is NOT backed up. The home area, on the other hand, is backed up nightly, but cannot be extended. The following information captures most of the pertinent aspects of file space.

Global file space on Curie, accessible to users on all compute, login and ada fast transfer nodes, is organized by two (GPFS) file systems showing in the table below: /home and /scratch. Access to the allocated file space is through the indicated directories that are setup automatically for every user.

The /tiered file system is accessible to users on login and ada fast transfer nodes but not the compute nodes.

File System Directory Environment Variable Storage Hardware File Space Quota File Counts Quota Comments
Home /home/$USER $HOME GSS26 10 GB 10000 Upon login, you will be situated in /home/$USER, where $USER is your Net ID. Use of this area should take into account its small size with fixed (cannot change) space and file count limits.
Scratch /scratch/user/$USER $SCRATCH GSS26 1 TB 50000 This is a high performance storage, intended to temporarily hold rather large files, and only for on-going processing that uses them. It is NOT backed up nor is it intended as long-term storage area. Please delete or move out of these area any files that are not frequently used.
Tiered /tiered/user/$USER $ARCHIVE GSS26 10TB 50000 This is an archival file system intended to hold valuable data files that are not frequently used. Please delete or move elsewhere files that have been inactive for extended periods. /tiered is not accessible from the compute nodes.


The Modules system organizes the multitude of packages we have installed on Curie so that they can be easily maintained and used. Some of these have several versions. More specifically, Modules configures appropriately the execution environment of each package (and version).

On Curie most non-OS software is organized by the Modules system. That includes all the compilers and associated libraries. Each package has a corresponding modulefile or simply a module where the appropriate actions are specified (in a prescribed format) in setting up its execution environment.

For the latest list of modules on Curie, see our Curie module list.

On Curie, the design and construction of the Modules system is based on the use of the bash shell. A few applications set up their environment with wrapper scripts that do not use the bash shell or insist on doing things outside Modules. Notify the help desk if you cannot locate the module appropriate for an application.

To find out what software packages are available under the Module system on Curie, enter:

$ module spider [packageName]          # List all versions of all packages or just for packageName.

To set up the shell's enviroment so it can run desired packages, you enter:

$ module load packageName1 packageName2 ...

Note that the specification of packageName is case sensitive and it can, optionally, include a specific version. On Ada, there are no packages that are preloaded by default.

To find out what packages/modules you have loaded onto the current shell, enter:

$ module list


$ module load xlf
$ module list

Currently Loaded Modules:
  1) ibmat/7.0   2) GCC/4.8.5-at7.0   3) xlmass/   4) xlsmp/   5) xlfbase/   6) xlf/

The above output illustrates the fact that a modulefile's complete name includes its version. An installed application can have several versions. When the version is omitted in the load action, the system selects the latest installed version for the associated package.

Please also note that loading a package is required in order to access the man pages, if available, associated with the package.

To remove/unload packages from a shell's environment, enter:

$ module unload packageName1 packageName2 ...

To remove ALL loaded modules from the current environemnt, enter:

$ module purge

To search and list what packages/modules are available on Ada, enter any suitable combination of the following:

$ module spider [packageName]          # List all versions of all packages or just for packageName.
$ module spider  string                # List all modules that contain the "string".
$ module spider  packageName/version   # List detailed information about that version of packageName
$ module keyword string                # list all modules and whatis that contain string.

$ module help                          # lists all the subcommands under the module command.
$ module help packageName              # Lists information about packageName


$ module spider xlf

      IBM XL FORTRAN for Linux (SLES11/RHEL6) with IBM Advanced Toolchain - Homepage: http://www.ibm.com/us/en/ 

        xlf/   <---- latest is the default

     Other possible modules matches:
        xlcxlf  xlfbase
   . . .

To remove (unload) a module and load another, use the swap subcommand:

$ module swap packageOUT packageIN

The above command is short for: $ module unload packageOUT; module load packageIN


$ module swap xlf/ xlf/

The following have been reloaded with a version change:
  1) GCC/4.8.5-at7.0 => GCC/4.4.7

$ module list

Currently Loaded Modules:
  1) ibmat/7.0   2) xlmass/   3) xlsmp/   4) xlfbase/   5) GCC/4.4.7   6) xlf/