Hprc banner tamu.png

Ada:Batch Memory Specs

From TAMU HPRC
Jump to: navigation, search

Clarification on Memory, Core, and Node Specifications

With LSF, you can specify the number of cores, the number of cores per node, and memory per core. The number of nodes that the job uses is determined by the core count divided by the number of cores per node.

Basic Ada (LSF) Memory/Core Specifications
Specification Option Example Example-Purpose
Core count -n ## -n 20 Assigns 20 job slots/cores.
Cores per node -R "span[ptile=##]" -R "span[ptile=5]" Request 5 cores per node.
Memory Per Core -M [MB] -M 2560 Sets the per process memory limit to 2560 MB.
Memory Per Core -R "rusage[mem=[MB]]" -R "rusage[mem=2560]" Schedules job on nodes that have at
least 2560 MBs available per core.

Note: There are two memory per core specification options listed above. In any job file, it is recommended that both are used and they should always match.

Memory Per Node

Another important number to keep in mind is memory per node. On Ada, 800+ nodes have 64GB of memory (54GB usable) and 26 nodes have 256GB of memory (245GB usable). A small selection of 1TB and 2TB nodes are available for large jobs as well. More hardware information can be found on the Ada Hardware Summary page.

Memory per node is important to keep in mind because it is one of the main factors that dictate which node(s) a job is placed on. A job that requires less than 54GB of memory per node can be placed on any of the 800+ nodes on Ada. Whereas a job requiring more than 54GB of memory per node has only 26 nodes that it can be placed on.

Examples

Below you will find examples of the above specifications as well as calculations to find the memory per node and number of nodes.
Memory Example 1:

#BSUB -n 1                   #Request 1 core
#BSUB -R "span[ptile=1]"     #Request 1 core per node.
#BSUB -R "rusage[mem=5000]"  #Request 5000MB per core for the job
#BSUB -M 5000                #Set the per process enforceable memory limit to 5000MB.

This results in:

1 Core / 1 Core per Node = 1 Node
5000MB per core * 1 Core per Node = 5000MB per Node

Memory Example 2:

#BSUB -n 10                  #Request 10 cores
#BSUB -R "span[ptile=10]"    #Request 10 cores per node.
#BSUB -R "rusage[mem=10000]"  #Request 10000MB per core for the job
#BSUB -M 10000                #Set the per process enforceable memory limit to 10000MB.

This results in:

10 Cores / 10 Cores per Node = 1 Node
~10GB per core * 10 Cores per Node = ~100GB per Node

Note: This is an example of a job that would require one of the 26 256GB nodes to run.

Memory Example 3:

#BSUB -n 20                  #Request 20 cores
#BSUB -R "span[ptile=10]"    #Request 10 cores per node.
#BSUB -R "rusage[mem=5000]"  #Request 5000MB per core for the job
#BSUB -M 5000                #Set the per process enforceable memory limit to 5000MB.

This results in:

20 Cores / 10 Cores per Node = 2 Nodes
~5GB per core * 10 Cores per Node = ~50GB per Node