Hprc banner tamu.png

Difference between revisions of "Ada:Batch Job Files"

From TAMU HPRC
Jump to: navigation, search
m (Environment Variables)
(Environment Variables)
 
(11 intermediate revisions by the same user not shown)
Line 12: Line 12:
 
An example of a resource specification is given below:
 
An example of a resource specification is given below:
 
  #BSUB -J MyExample  #Set the job name to "MyExample"
 
  #BSUB -J MyExample  #Set the job name to "MyExample"
 +
Note: Comments in a job file also begin with a '''#''' but LSF recognizes '''#BSUB''' as a directive.<br>
 +
 
A list of the most commonly used and important options for these job files are given in the following section of this wiki. Full job file examples are given [[Ada:Batch#Job_File_Examples | below]].
 
A list of the most commonly used and important options for these job files are given in the following section of this wiki. Full job file examples are given [[Ada:Batch#Job_File_Examples | below]].
  
Line 103: Line 105:
 
|
 
|
 
| Assigns a whole node exclusively for the job.
 
| Assigns a whole node exclusively for the job.
 +
|-
 +
| Specific node type
 +
| <nowiki>-R "select[gpu|phi]"</nowiki>
 +
| -R "select[gpu]"
 +
| Requests a node with a GPU to be used for the job.
 
|}
 
|}
  
 
=== Environment Variables ===  
 
=== Environment Variables ===  
All the nodes enlisted for the execution of a job carry most of the environment variables the login process created: HOME, PWD, PATH, USER, etc.
+
All the nodes enlisted for the execution of a job carry most of the environment variables the login process created: '''HOME, SCRATCH, PWD, PATH, USER,''' etc. In addition, LSF defines new ones in the environment of an executing job. Below is a list of most commonly used environment variables.
In addition, LSF defines new ones in the environment of an executing job. Below is a list of most commonly used environment variables.
 
  
 
{| class="wikitable" style="text-align: center;"
 
{| class="wikitable" style="text-align: center;"
Line 138: Line 144:
 
| Hosts I
 
| Hosts I
 
| $LSB_HOSTS
 
| $LSB_HOSTS
| The list of nodes that are used to run the batch job, <br>repeated according to ptile value.
+
| The list of nodes that are used to run the batch job, <br>repeated according to ptile value.<br> *The character limit of LSB_HOSTS variable is 4096.
 
|-
 
|-
 
| Hosts II
 
| Hosts II
Line 148: Line 154:
 
| The hostfile containing the list of nodes<br> that are used to run the batch job.
 
| The hostfile containing the list of nodes<br> that are used to run the batch job.
 
|}
 
|}
 +
 +
'''Note:''' To see all relevant LSF environment variables for a job, add the following line to the '''executable section''' of a job file and submit that job. All the variables will be printed in the output file.
 +
env | grep LSB
  
 
=== Clarification on Memory, Core, and Node Specifications ===
 
=== Clarification on Memory, Core, and Node Specifications ===
Line 154: Line 163:
 
For examples on calculating memory, core, and/or node specifications on Ada: [[:Ada:Batch_Memory_Specs | Specification Clarification]].
 
For examples on calculating memory, core, and/or node specifications on Ada: [[:Ada:Batch_Memory_Specs | Specification Clarification]].
  
 +
=== Executable Commands ===
 +
 +
After the resource specification section of a job file comes the executable section. This executable section contains all the necessary UNIX, Linux, and program commands that will be run in the job. <br>
 +
Some commands that may go in this section include, but are not limited to:
 +
* Changing directories
 +
* Loading, unloading, and listing modules
 +
* Launching software
 +
 +
An example of a possible executable section is below:
 +
cd $SCRATCH      # Change current directory to ''/scratch/user/[netID]/''
 +
ml purge        # Purge all modules
 +
ml intel/2016b  # Load the intel/2016b module
 +
ml              # List all currently loaded modules
 +
 +
./myProgram.o    # Run "myProgram.o"
 +
 +
For information on the module system or specific software, visit our [[SW:Modules | Modules]] page and our [[SW | Software]] page.
 
[[Category: Ada]]
 
[[Category: Ada]]

Latest revision as of 12:27, 10 February 2017

Building Job files

While not the only method of submitted programs to be executed, job files fulfill the needs of most users.

Job files consist of two main parts:

  • Resource Specifications
  • Executable commands

In a job file, resource specification options are preceded by a script directive. For each batch system, this directive is different. On Ada (LSF) this directive is #BSUB.
For every line of resource specifications, this directive must be the first text of the line, and all specifications must come before any executable lines. An example of a resource specification is given below:

#BSUB -J MyExample  #Set the job name to "MyExample"

Note: Comments in a job file also begin with a # but LSF recognizes #BSUB as a directive.

A list of the most commonly used and important options for these job files are given in the following section of this wiki. Full job file examples are given below.

Basic Job Specifications

Several of the most important options are described below. These basic options are typically all that is needed to run a job on Ada.

Basic Ada (LSF) Job Specifications
Specification Option Example Example-Purpose
Job Name -J [SomeText] -J MyJob1 Set the job name to "MyJob1"
Shell -L [Shell] -L /bin/bash Uses the bash shell to initialize
the job's execution environment.
Wall Clock Limit -W [hh:mm] -W 1:15 Set wall clock limit to 1 hour 15 min
Core count -n ## -n 20 Assigns 20 job slots/cores.
Cores per node -R "span[ptile=##]" -R "span[ptile=5]" Request 5 cores per node.
Memory Per Core -M [MB] -M 2560 Sets the per process memory limit to 2560 MB.
Memory Per Core -R "rusage[mem=[MB]]" -R "rusage[mem=2560]" Schedules job on nodes that have at
least 2560 MBs available per core.
Combined stdout and stderr -o [OutputName].%j -o stdout1.%j Collect stdout/err in stdout.[JobID]

Optional Job Specifications

A variety of optional specifications are available to customize your job. The table below lists the specifications which are most useful for users of Ada.

Optional Ada (LSF) Job Specifications
Specification Option Example Example-Purpose
Set Allocation -P ###### -P 274839 Set allocation to charge to 274839
Email Notification I -u [email-address] -u howdy@tamu.edu Send emails to howdy@tamu.edu.
Email Notification II -[B|N] -B -N Send email on beginning (-B) and end (-N) of job.
Specify Queue -q [queue] -q xlarge Request only nodes in xlarge subset.
Exclusive Node Usage -x Assigns a whole node exclusively for the job.
Specific node type -R "select[gpu|phi]" -R "select[gpu]" Requests a node with a GPU to be used for the job.

Environment Variables

All the nodes enlisted for the execution of a job carry most of the environment variables the login process created: HOME, SCRATCH, PWD, PATH, USER, etc. In addition, LSF defines new ones in the environment of an executing job. Below is a list of most commonly used environment variables.

Basic Ada (LSF) Environment Variables
Variable Usage Description
Job ID $LSB_JOBID Batch job ID assigned by LSF.
Job Name $LSB_JOBNAME The name of the Job.
Queue $LSB_QUEUE The name of the queue the job is dispatched from.
Error File $LSB_ERRORFILE Name of the error file specified with a bsub -e.
Submit Directory $LSB_SUBCWD The directory the job was submitted from.
Hosts I $LSB_HOSTS The list of nodes that are used to run the batch job,
repeated according to ptile value.
*The character limit of LSB_HOSTS variable is 4096.
Hosts II $LSB_MCPU_HOSTS The list of nodes and the specified or default
ptile value per node to run the batch job.
Host file $LSB_DJOB_HOSTFILE The hostfile containing the list of nodes
that are used to run the batch job.

Note: To see all relevant LSF environment variables for a job, add the following line to the executable section of a job file and submit that job. All the variables will be printed in the output file.

env | grep LSB

Clarification on Memory, Core, and Node Specifications

Memory Specifications are IMPORTANT.
For examples on calculating memory, core, and/or node specifications on Ada: Specification Clarification.

Executable Commands

After the resource specification section of a job file comes the executable section. This executable section contains all the necessary UNIX, Linux, and program commands that will be run in the job.
Some commands that may go in this section include, but are not limited to:

  • Changing directories
  • Loading, unloading, and listing modules
  • Launching software

An example of a possible executable section is below:

cd $SCRATCH      # Change current directory to /scratch/user/[netID]/
ml purge         # Purge all modules
ml intel/2016b   # Load the intel/2016b module
ml               # List all currently loaded modules

./myProgram.o    # Run "myProgram.o"

For information on the module system or specific software, visit our Modules page and our Software page.