Ada:Computing Environment

Computing Environment

General Information

All nodes on Ada operate under the control of the LINUX operating system. The installed version is the CentOS 6 implementation. The global file systems (accessible to all nodes) are organized by IBM's General Parallel File System (GPFS). These are /home and /scratch. Both are hosted on the 4PB (raw) disk space provided by the IBM GSS26 mass storage appliance.

The batch system, although an important part of the computing environment, is covered in a separate section of its own.

The Login Part

The default shell for ada (all nodes) is bash. Its environment is defined in the startup file, .bash_profile. It is read and executed when you login, and it resides in your home directory. The command,

[ NetID@ada ~]$ env

when executed right after login, displays the default environment settings.

Many applications use the TMPDIR environment variable, when set, as the area for storing temporary files. By default TMPDIR is not set, whereupon the application programs will typically use the /tmp area on the local drive, not always a safe option because of its limited capacity. Setting it to a directory in your scratch area is a good solution.

The use of the login nodes is for small-to-medium code development and processing. The most important processing limits here are:

• CPU time - ONE HOUR per login session.

• Cores for Concurrent Use - EIGHT CORES per login session, on the same node or (cumulatively) across all login nodes.

The 8 login nodes have a number of diverse features making them suitable platforms for different types of code development, testing and limited computation. Below we display their main hardware features.

Main Features of Login Nodes

	Single NVIDIA K20 GPU	Dual NVIDIA K20 GPUs	Dual Intel Xeon Phi 5110P
HostNames	ada1.tamu.edu ada2.tamu.edu	ada3.tamu.edu ada4.tamu.edu ada5.tamu.edu	ada6.tamu.edu ada7.tamu.edu ada8.tamu.edu
Processor Type	Intel Xeon E5-2670 v2, 10-core, 2.5GHz
Total Nodes	2	3	3
Cores/Node	20
Interconnect	FDR-10 Infiniband
Network	1 GigE for external connectivity
Local Disk Space	per node: 4 834 giga-byte 10K rpm SAS drives

To insure access to a specific node because, say, it must have a Phi coprocessor you must use its specific hostname (e.g., ada8.tamu.edu).

For large data transfers to/from other hosts you should definitely consider the use of the fast data transfer nodes described in a previous section.

File Space

Immediately upon logging in to Ada, the following message about the status of your disk space use greets you:

Your current disk quotas are:
Disk       Disk Usage      Limit    File Usage      Limit
/home           2.49G        10G           113      10000
/scratch        1.25G         1T            40      50000
/tiered          4.8G        10T            69      50000
Type 'showquota' to view these quotas again. 

The gist of the information is that the file space allocated for your home directory, /home/NetID, is just 10GB, while the allocation for scratch area, /scratch/user/NetID, is 1TB.

Your scratch space is extensible for justifiable needs. It is NOT backed up. The home area, on the other hand, is backed up nightly, but cannot be extended. The following information captures most of the pertinent aspects of file space.

Global file space on Ada, accessible to users on all compute, login, and extra-fast data transfer nodes, is organized by the three (GPFS) file systems showing in the table below. Access to the allocated file space is through the indicated directories that are setup automatically for every user.

File System	Directory	Environment Variable	Storage Hardware	File Space Quota	File Counts Quota	Comments
Home	/home/$USER	$HOME	GSS26	10 GB	10000	Upon login, you will be situated in /home/$USER, where $USER is your Net ID. Use of this area should take into account its small size with fixed (cannot change) space and file count limits.
Scratch	/scratch/user/$USER	$SCRATCH	GSS26	1 TB	50000	This is a high performance storage, intended to temporarily hold rather large files, and only for on-going processing that uses them. It is NOT backed up nor is it intended as long-term storage area. Please delete or move out of these area any files that are not frequently used.
Tiered	/tiered/user/$USER	$ARCHIVE	flash+disks+tapes	10 TB	50000	This is an archival file system intended to hold valuable data files that are not frequently used. Please delete or move elsewhere files that have been inactive for extended periods.

Modules

The Modules system organizes the multitude of packages we have installed on Ada so that they can be easily maintained and used. Some of these have several versions. More specifically, Modules configures appropriately the execution environment of each package (and version).

On Ada most non-OS software is organized by the Modules system. That includes all the compilers and associated libraries. Each package has a corresponding modulefile or simply a module where the appropriate actions are specified (in a prescibed format) in setting up its execution environment.

On Ada, the design and construction of the Modules system is based on the use of the bash shell. A few applications set up their environment with wrapper scripts that do not use the bash shell or insist on doing things outside Modules. Notify the help desk if you cannot locate the module appropriate for an application.
To find out what software packages are available under the Modules system use one of the following two commands:

[NetID@cluster ~]$ module spider [packageName]

[NetID@cluster ~]$ module avail [packageName] 

In order to use some software on our clusters, the module for the software must be loaded first. To load the module for a software package, use the following command:

[NetID@cluster ~]$ module load packageName/version

Note: The specification of packageName in the load command is case sensitive and it must include a specific version. No packages are preloaded by default.
Note: Loading a package is required in order to access the man pages associated with the package if they are available.

To find out what packages/modules you have loaded on your current session, use the following command:

[NetID@cluster ~]$ module list

To remove one module from your current session, use the following command:

[NetID@cluster ~]$ module unload [packageName]

To remove ALL modules from your current session, use the following command:

[NetID@cluster ~]$ module purge

Note: It is very important to remove all modules when switching between software and compilers. Mixing modules and versions is NOT a good idea and can cause many problems.

For more specific information as well as examples of using modules, please see our Modules System page.