Hprc banner tamu.png

Difference between revisions of "Ada:Batch Memory Specs"

From TAMU HPRC
Jump to: navigation, search
(Created page with "== Clarification on Memory, Core, and Node Specifications == <pre> #BSUB -R "rusage[mem=process_alloc_size]" -M process_size_limit </pre> Both of these should be set in a job...")
 
(Examples)
 
(8 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
== Clarification on Memory, Core, and Node Specifications ==
 
== Clarification on Memory, Core, and Node Specifications ==
<pre>
+
With LSF, you can specify the '''number of cores''', the number of cores '''per node''', and '''memory''' per core. The number of nodes that the job uses is determined by the core count divided by the number of cores per node.
#BSUB -R "rusage[mem=process_alloc_size]" -M process_size_limit
 
</pre>
 
  
Both of these should be set in a job and, typically, to the same amount.  
+
{| class="wikitable" style="text-align: center;"
* The '''process_alloc_size''' specifies the amount of memory to allocate/reserve per process on a node. Omitting this specification, <br>
+
|+ Basic Ada (LSF) Memory/Core Specifications
will cause LSF to select nodes on the basis of available cores only, regardless of whether such nodes have sufficient memory to<br>
+
|-
run the job efficiently. Hence, this omission can cause the job to be '''swapped out''' onto the local drive (big delay) and/or experience memory contention<br>
+
! style="width: 150pt;" | Specification
from other jobs running on the same node, thereby bringing a general and dramatic slowdown.
+
! style="width: 130pt;" | Option
 +
! style="width: 170pt;" | Example
 +
! style="width: 230pt;" | Example-Purpose
 +
|-
 +
| Core count
 +
| -n ##
 +
| -n 20
 +
| Assigns 20 job slots/cores.
 +
|-
 +
| Cores per node
 +
| -R "span[ptile=##]"
 +
| -R "span[ptile=5]"
 +
| Request 5 cores per node.
 +
|-
 +
| Memory Per Core
 +
| -M [MB]
 +
| -M 2560
 +
| Sets the per process memory limit to 2560 MB.
 +
|-
 +
| Memory Per Core
 +
| -R "rusage[mem=[MB]]"
 +
| -R "rusage[mem=2560]"
 +
| Schedules job on nodes that have at<br>least 2560 MBs available per core.
 +
|}
  
*If, in addition to the '''process_alloc_size''' option, one specifies a value (=core_count_per_node) for the '''ptile''' parameter,<br>
+
<font color=teal>'''Note:''' There are two '''memory per core''' specification options listed above. In any job file, it is recommended that '''both''' are used and they should '''always''' match. </font>
LSF will allocate/reserve '''core_count_per_node * process_alloc_size''' MB of memory per node.
 
  
* The '''-M process_size_limit''' setting specifies the memory size limit (in MBs) per process, which when exceeded will cause the process and<br>
+
=== Memory Per Node ===
the job to fail. The default value for '''process_size_limit''' is 2.5 GB. Both of these settings should reflect the run-time needs of your job.
 
  
* The total available memory per node for jobs is about 10 GB less than the maximum: 54 GB for '''nxt''' type nodes; 246 GB for the '''mem256gb''' type; etc.
+
Another important number to keep in mind is '''memory per node'''. On Ada, '''800+''' nodes have '''64GB''' of memory (''54GB usable'') and '''26''' nodes have '''256GB''' of memory (''245GB usable''). A small selection of 1TB and 2TB nodes are available for large jobs as well. More hardware information can be found on the [[Ada:Intro | Ada Hardware Summary page]].
  
One should not rely on default memory limit settings. The latter may be too large or too small. A realistic picture of job's memory can be obtained<br>
+
Memory per node is important to keep in mind because it is one of the main factors that dictate which node(s) a job is placed on. A job that requires less than '''54GB''' of memory per node can be placed on any of the 800+ nodes on Ada. Whereas a job requiring more than '''54GB''' of memory per node has only 26 nodes that it can be placed on.  
from the information given by one (e.g., '''bjobs''') or more job tracking commands. For more information, please see the [[Ada:Batch_Job_Submission#Job_tracking_and_control_commands | Job tracking and control commands]] subsection.
 
  
'''Five important job parameters:'''
+
=== Examples ===
<pre>
+
Below you will find examples of the above specifications as well as calculations to find the '''memory per node''' and number of nodes. <br>
#BSUB -n NNN                    # NNN: total number of cores/jobslots to allocate for the job
+
<span style="font-size:120%;">'''Memory Example 1:'''</span> <br />
#BSUB -R "span[ptile=XX]"       # XX:  number of cores/jobslots per node to use. Also, a node selection criterion
+
#BSUB -n 1                  #Request 1 core
#BSUB -R "select[node-type]"    # node-type: nxt, mem256gb, gpu, phi, mem1t, mem2t ...
+
#BSUB -R "span[ptile=1]"     #Request 1 core per node.
#BSUB -R "rusage[mem=nnn]"     # reserves nnn MBs per process/CPU for the job
+
#BSUB -R "rusage[mem=5000]" #Request 5000MB per core for the job
#BSUB -M mmm                    # sets the per process enforceable memory limit to mmm MB
+
#BSUB -M 5000                #Set the per process enforceable memory limit to 5000MB.
</pre>
 
  
We list these together because in many jobs they can be closely related and, therefore, must be
+
This results in:
consistently set. We recommend their adoption in all jobs, serial, single-node and multi-node.
+
1 Core / 1 Core per Node = '''1''' Node
The '''rusage[mem=nnn]''' setting causes LSF to select nodes that can each allocate '''XX * nnn''' MBs for the execution of the job.
+
5000MB per core * 1 Core per Node = '''5000MB''' per Node
The '''-M mm''' sets and enforces the process memory size limit. When this limit is violated the job will abort. Omitting this specification, causes LSF to assume '''the'''
 
'''default memory limit, which by configuration is set to 2.5 giga-bytes (2500 MB) per process.''' The following examples, with some commentary,  illustrate the
 
use of these options.
 
  
'''Important:''' if the process memory limit, default (2500 MB)  or specified, is exceeded during execution the job will fail with a '''memory violation''' error.
+
<span style="font-size:120%;">'''Memory Example 2:'''</span> <br />
 +
#BSUB -n 10                  #Request 10 cores
 +
#BSUB -R "span[ptile=10]"    #Request 10 cores per node.
 +
#BSUB -R "rusage[mem=10000]"  #Request 10000MB per core for the job
 +
#BSUB -M 10000                #Set the per process enforceable memory limit to 10000MB.
  
<pre>
+
This results in:
#BSUB -n 900                    # 900: number of cores/jobslots to allocate for the job
+
10 Cores / 10 Cores per Node = '''1''' Node
#BSUB -R "span[ptile=20]"      # 20:  number of cores per node to use
+
~10GB per core * 10 Cores per Node = '''~100GB''' per Node
#BSUB -R "select[nxt]"          # Allocates NeXtScale type nodes
+
<font color=teal>'''Note:''' This is an example of a job that would require one of the 26 '''256GB nodes''' to run.</font><br>
</pre>
 
  
The above specifications will allocate 45 (=900/20) whole nodes. In many parallel jobs the selection
+
<span style="font-size:120%;">'''Memory Example 3:'''</span> <br />
of NeXtScale nodes at 20 cores per node is the best choice. Here, the maximum memory per process
+
#BSUB -n 20                  #Request 20 cores
is set to 2500 MB. Here, we're just illustrating what happens when you omit the memory-related options.
+
#BSUB -R "span[ptile=10]"    #Request 10 cores per node.
We definitely urge that you specify them. '''The memory enforceable limit per process here is 2.5 MB, the default setting.'''
+
#BSUB -R "rusage[mem=5000]"  #Request 5000MB per core for the job
 +
#BSUB -M 5000                #Set the per process enforceable memory limit to 5000MB.
  
<pre>
+
This results in:
#BSUB -n 900                    # 900: total number of cores/jobslots to allocate for the job
+
  20 Cores / 10 Cores per Node = '''2''' Nodes
#BSUB -R "span[ptile=16]"      # 16: number of cores/jobslots per node to use
+
~5GB per core * 10 Cores per Node = '''~50GB''' per Node
#BSUB -R "select[nxt]"          # allocates NeXtScale type nodes
 
#BSUB -R "rusage[mem=3600]"    # schedules on nodes that have at least 3600 MB per process/CPU avail
 
#BSUB -M 3600                  # enforces 3600 MB memory use per process
 
</pre>
 
  
The above specifications will allocate 57 (= ceiling(900/16)) nodes. The decision to only apply XX (here 16) number cores
 
per node, and not the maximum 20, for a computation requires some judgement. The execution profile of the job is
 
important. Typically, some experimentation is required in finding the optimal tile number for a given code.
 
  
<pre>
+
[[ Category:Ada ]]
#BSUB -n 1                    # Allocate a total of 1 cpu/core for the job, appropriate for serial processing.
 
#BSUB -R "span[ptile=1]"      # Allocate 1 core per node.
 
#BSUB -R "select[gpu]"        # Allocate a node that has gpus (of 64GB or 256GB memory). A "select[phi]"
 
                              # specification would allocate a node with phi coprocessors.
 
</pre>
 
 
 
Omitting the last two options in the above will cause LSF to place the job on any conveniently available
 
core on any node, idle or (partially) busy, of any type, except on those with 1TB or 2TB memory.<br>
 
 
 
It is worth emphasizing that, under the current LSF setup, only the '''-x''' option and a ptile value equal to the node's
 
core limit will prevent LSF from scheduling jobs that match the balance of unreserved cores.
 
 
 
====Inhomogeneous Node Selection====
 
<pre>
 
#BSUB -n 900
 
#BSUB -R "600*{ select[nxt] rusage[mem=3000] span[ptile=20]} + 300*{ select[gpu] rusage[mem=3000] span[ptile=20] }"
 
#BSUB -M 3000
 
</pre>
 
 
 
The above specification will allocate 30 (600/20) NeXtScale and 15 (300/20) iDataPlex nodes, the latter with GPUs, at 20 cores per node. Note that the enforceable memory limit here 3000 MB per process. In the '''Examples''' section, we provide an illustration of the usefulness of inhomogeneous node selection when the MPMD parallelization model is to be used.
 

Latest revision as of 13:10, 22 March 2017

Clarification on Memory, Core, and Node Specifications

With LSF, you can specify the number of cores, the number of cores per node, and memory per core. The number of nodes that the job uses is determined by the core count divided by the number of cores per node.

Basic Ada (LSF) Memory/Core Specifications
Specification Option Example Example-Purpose
Core count -n ## -n 20 Assigns 20 job slots/cores.
Cores per node -R "span[ptile=##]" -R "span[ptile=5]" Request 5 cores per node.
Memory Per Core -M [MB] -M 2560 Sets the per process memory limit to 2560 MB.
Memory Per Core -R "rusage[mem=[MB]]" -R "rusage[mem=2560]" Schedules job on nodes that have at
least 2560 MBs available per core.

Note: There are two memory per core specification options listed above. In any job file, it is recommended that both are used and they should always match.

Memory Per Node

Another important number to keep in mind is memory per node. On Ada, 800+ nodes have 64GB of memory (54GB usable) and 26 nodes have 256GB of memory (245GB usable). A small selection of 1TB and 2TB nodes are available for large jobs as well. More hardware information can be found on the Ada Hardware Summary page.

Memory per node is important to keep in mind because it is one of the main factors that dictate which node(s) a job is placed on. A job that requires less than 54GB of memory per node can be placed on any of the 800+ nodes on Ada. Whereas a job requiring more than 54GB of memory per node has only 26 nodes that it can be placed on.

Examples

Below you will find examples of the above specifications as well as calculations to find the memory per node and number of nodes.
Memory Example 1:

#BSUB -n 1                   #Request 1 core
#BSUB -R "span[ptile=1]"     #Request 1 core per node.
#BSUB -R "rusage[mem=5000]"  #Request 5000MB per core for the job
#BSUB -M 5000                #Set the per process enforceable memory limit to 5000MB.

This results in:

1 Core / 1 Core per Node = 1 Node
5000MB per core * 1 Core per Node = 5000MB per Node

Memory Example 2:

#BSUB -n 10                  #Request 10 cores
#BSUB -R "span[ptile=10]"    #Request 10 cores per node.
#BSUB -R "rusage[mem=10000]"  #Request 10000MB per core for the job
#BSUB -M 10000                #Set the per process enforceable memory limit to 10000MB.

This results in:

10 Cores / 10 Cores per Node = 1 Node
~10GB per core * 10 Cores per Node = ~100GB per Node

Note: This is an example of a job that would require one of the 26 256GB nodes to run.

Memory Example 3:

#BSUB -n 20                  #Request 20 cores
#BSUB -R "span[ptile=10]"    #Request 10 cores per node.
#BSUB -R "rusage[mem=5000]"  #Request 5000MB per core for the job
#BSUB -M 5000                #Set the per process enforceable memory limit to 5000MB.

This results in:

20 Cores / 10 Cores per Node = 2 Nodes
~5GB per core * 10 Cores per Node = ~50GB per Node