Difference between revisions of "Ada:QuickStart"
(→Submitting and Monitoring Jobs) |
|||
| Line 22: | Line 22: | ||
== Navigating Ada & Storage Quotas == | == Navigating Ada & Storage Quotas == | ||
| + | When users first access Ada, they will be within their ''home'' directory. This directory has smaller storage quotas and should not be used for general purpose. | ||
| − | + | Users can navigate to their ''home'' directory with the following command: | |
| − | |||
| − | |||
[NetID@ada1 ~]$ '''cd /home/''[NetID]''''' | [NetID@ada1 ~]$ '''cd /home/''[NetID]''''' | ||
| − | + | A User's ''scratch'' directory has more storage space than their ''home'' directory and is recommended for general purpose use. Users can navigate to their ''scratch'' directory with the following command: | |
[NetID@ada1 ~]$ '''cd /scratch/user/''[NetID]''''' | [NetID@ada1 ~]$ '''cd /scratch/user/''[NetID]''''' | ||
| − | + | Users can navigate to ''scratch'' or ''home'' easily by using their respective environment variables. | |
Navigate to ''scratch'' with the following command: | Navigate to ''scratch'' with the following command: | ||
| Line 40: | Line 39: | ||
<font color=purple> | <font color=purple> | ||
| − | + | A user's ''scratch'' directory is restricted to 1TB/50,000 files of storage. This storage quota is '''expandable''' upon request. | |
| − | + | A user's ''home'' directory is restricted to 10GB/10,000 files of storage. This storage quota is '''not expandable'''. | |
</font> | </font> | ||
| − | + | Users can see the current status of their storage quotas with: | |
[NetID@ada1 ~]$ '''showquota''' | [NetID@ada1 ~]$ '''showquota''' | ||
| Line 56: | Line 55: | ||
More information on '''LSF''' can be found in the [[Ada:Batch | Ada Batch]] page. | More information on '''LSF''' can be found in the [[Ada:Batch | Ada Batch]] page. | ||
| − | == Running | + | == Running A Program / Preparing a Job File == |
| − | + | In order to properly run a program on Ada, users will need to create a job file and submit a job. | |
| − | In order to properly run a program on Ada, | ||
The simple example job file below requests 1 core on 1 node with 2.5GB of RAM for 1.5 hour. Note that most nodes (>800) on Ada have 20 cores with 54GB of usable memory. Some nodes (~26) have 20 cores with 245GB of usable memory. A few special nodes have more cores and memory than this. Please ensure that your job requirements will fit within these restrictions. | The simple example job file below requests 1 core on 1 node with 2.5GB of RAM for 1.5 hour. Note that most nodes (>800) on Ada have 20 cores with 54GB of usable memory. Some nodes (~26) have 20 cores with 245GB of usable memory. A few special nodes have more cores and memory than this. Please ensure that your job requirements will fit within these restrictions. | ||
| Line 85: | Line 83: | ||
== Submitting and Monitoring Jobs == | == Submitting and Monitoring Jobs == | ||
| − | Once | + | Once a job file is ready, it is time to submit the job. Users can submit their jobs to LSF with the following command: |
[NetID@ada1 ~]$ '''bsub < ''MyJob.lsf''''' | [NetID@ada1 ~]$ '''bsub < ''MyJob.lsf''''' | ||
Verifying job submission parameters... | Verifying job submission parameters... | ||
| Line 94: | Line 92: | ||
Job <0000000> is submitted to default queue <sn_short>. | Job <0000000> is submitted to default queue <sn_short>. | ||
| − | After the job has been submitted, | + | After the job has been submitted, users are able to monitor their own jobs with several methods. |
To see the status of all of your jobs, use the following command: | To see the status of all of your jobs, use the following command: | ||
[NetID@ada1 ~]$ '''bjobs''' | [NetID@ada1 ~]$ '''bjobs''' | ||
| Line 124: | Line 122: | ||
Software on Ada is loaded using '''modules'''. | Software on Ada is loaded using '''modules'''. | ||
| − | + | A list of the most popular software on our systems is available on the [[:SW | HPRC Available Software]] page. | |
| − | + | To '''find''' ''most'' available software on Ada, use the following command: | |
[NetID@ada1 ~]$ '''module avail''' | [NetID@ada1 ~]$ '''module avail''' | ||
| − | + | To '''search for''' particular software by keyword, use: | |
[NetID@ada1 ~]$ '''module spider ''keyword''''' | [NetID@ada1 ~]$ '''module spider ''keyword''''' | ||
| − | + | To load a module, use: | |
[NetID@ada1 ~]$ '''module load ''moduleName''''' | [NetID@ada1 ~]$ '''module load ''moduleName''''' | ||
| − | + | To list all currently loaded modules, use: | |
[NetID@ada1 ~]$ '''module list''' | [NetID@ada1 ~]$ '''module list''' | ||
| − | + | To remove all currently loaded modules, use: | |
[NetID@ada1 ~]$ '''module purge''' | [NetID@ada1 ~]$ '''module purge''' | ||
| Line 157: | Line 155: | ||
<!-- See our [Ada-Filezilla example video] for a demonstration of this process. --> | <!-- See our [Ada-Filezilla example video] for a demonstration of this process. --> | ||
| − | <font color=teal>'''Advice:''' while GUIs are acceptable for file transfers, the cp and scp commands are much quicker and may significantly benefit | + | <font color=teal>'''Advice:''' while GUIs are acceptable for file transfers, the cp and scp commands are much quicker and may significantly benefit workflow.</font> |
==== Reliably Transferring Large Files ==== | ==== Reliably Transferring Large Files ==== | ||
| − | For files larger than several GB, | + | For files larger than several GB, users will want to consider the use of a more fault-tolerant utility such as rsync. |
[NetID@ada1 ~]$ '''rsync -av [-z] ''localdir/ userid@remotesystem:/path/to/remotedir/''''' | [NetID@ada1 ~]$ '''rsync -av [-z] ''localdir/ userid@remotesystem:/path/to/remotedir/''''' | ||
Revision as of 12:17, 12 May 2017
Ada Quick Start Guide
Contents
Ada Usage Policies
Access to Ada is granted with the condition that you will understand and adhere to all TAMU HPRC and Ada-specific policies.
General policies can be found on the HPRC Policies page.
Ada-specific policies can be found on the Ada Policies page.
Accessing Ada
Most access to Ada is done via a secure shell session.
Users on Windows computers use either PuTTY or MobaXterm. If MobaXterm works on your computer, it is usually easier to use.
Users on Mac and Linux/Unix should use whatever SSH-capable terminal is available on their system.
The command to connect to Ada is as follows. Be sure to replace [NetID] with your TAMU NetID.
[user1@localhost ~]$ ssh [NetID]@ada.tamu.edu
Note: In this example [user1@localhost ~]$ represents the command prompt on your local machine.
Your login password is the same that used on Howdy. You will not see your password as your type it into the login prompt.
When users first access Ada, they will be within their home directory. This directory has smaller storage quotas and should not be used for general purpose.
Users can navigate to their home directory with the following command:
[NetID@ada1 ~]$ cd /home/[NetID]
A User's scratch directory has more storage space than their home directory and is recommended for general purpose use. Users can navigate to their scratch directory with the following command:
[NetID@ada1 ~]$ cd /scratch/user/[NetID]
Users can navigate to scratch or home easily by using their respective environment variables.
Navigate to scratch with the following command:
[NetID@ada1 ~]$ cd $SCRATCH
Navigate to home with the following command:
[NetID@ada1 ~]$ cd $HOME
A user's scratch directory is restricted to 1TB/50,000 files of storage. This storage quota is expandable upon request.
A user's home directory is restricted to 10GB/10,000 files of storage. This storage quota is not expandable.
Users can see the current status of their storage quotas with:
[NetID@ada1 ~]$ showquota
If you need a storage quota increase, please contact us with justification and the expected length of time that you will need the quota increase.
The Batch System
The batch system is a load distribution implementation that ensures convenient and fair use of a shared resource. Submitting jobs to a batch system allows a user to reserve specific resources with minimal interference to other users. All users are required to submit resource-intensive processing to the compute nodes through the batch system - attempting to circumvent the batch system is not allowed.
On Ada, LSF is the batch system that provides job management. More information on LSF can be found in the Ada Batch page.
Running A Program / Preparing a Job File
In order to properly run a program on Ada, users will need to create a job file and submit a job.
The simple example job file below requests 1 core on 1 node with 2.5GB of RAM for 1.5 hour. Note that most nodes (>800) on Ada have 20 cores with 54GB of usable memory. Some nodes (~26) have 20 cores with 245GB of usable memory. A few special nodes have more cores and memory than this. Please ensure that your job requirements will fit within these restrictions.
Any modules that need to be loaded or executable commands will replace the "#First Executable Line" in this example.
##NECESSARY JOB SPECIFICATIONS #BSUB -J JobExample1 #Set the job name to "JobExample1" #BSUB -L /bin/bash #Uses the bash login shell to initialize the job's execution environment. #BSUB -W 1:30 #Set the wall clock limit to 1hr and 30min #BSUB -n 1 #Request 1 task #BSUB -R "span[ptile=1]" #Request 1 task/core per node #BSUB -R "rusage[mem=2560]" #Request 2560MB (2.5GB) per node #BSUB -M 2560 #Set the per process enforceable memory limit to 2560MB #BSUB -o Example1Out.%J #Send stdout/err to "Example1Out.[jobID]" #First Executable Line
Note: If your job file has been written on an older Mac or DOS workstation, you will need to use "dos2unix" to remove certain characters that interfere with parsing the script.
[NetID@ada1 ~]$ dos2unix MyJob.lsf
More information on job options can be found in the Building Job Files section of the Ada Batch page.
More information on dos2unix can be found on the dos2unix section of the HPRC Available Software page.
Submitting and Monitoring Jobs
Once a job file is ready, it is time to submit the job. Users can submit their jobs to LSF with the following command:
[NetID@ada1 ~]$ bsub < MyJob.lsf
Verifying job submission parameters...
Verifying project account...
Account to charge: 000000000000
Balance (SUs): 5000.0000
SUs to charge: 1.5000
Job <0000000> is submitted to default queue <sn_short>.
After the job has been submitted, users are able to monitor their own jobs with several methods. To see the status of all of your jobs, use the following command:
[NetID@ada1 ~]$ bjobs JOBID STAT USER QUEUE JOB_NAME NEXEC_HOST SLOTS RUN_TIME TIME_LEFT 0000000 RUN NetID sn_short JobExample1 1 1 0 second(s) 1:30 L 0000001 PEND NetID sn_short JobExample1 1 1 0 second(s) 1:30 L
To see the status of one job, use the following command, where XXXXXXX is the JobID:
[NetID@ada1 ~]$ bjobs XXXXXXX JOBID STAT USER QUEUE JOB_NAME NEXEC_HOST SLOTS RUN_TIME TIME_LEFT XXXXXXX RUN NetID sn_short JobExample1 1 1 0 second(s) 1:30 L
To cancel a job, use the following command, where XXXXXXX is the JobID:
[NetID@ada1 ~]$ bkill XXXXXXX Job <XXXXXXX> is being terminated
More information on submitting and monitoring LSF jobs can be found in the Job Submission section of the Ada Batch System page.
Additional Topics
Translating Ada/LSF <--> Terra/Slurm
The HPRC Batch Translation page contains information on converting between LSF, PBS, and Slurm.
Our staff has also written some example jobs for specific software. These software-specific examples can be seen on the Individual Software Pages where available.
Finding Software
Software on Ada is loaded using modules.
A list of the most popular software on our systems is available on the HPRC Available Software page.
To find most available software on Ada, use the following command:
[NetID@ada1 ~]$ module avail
To search for particular software by keyword, use:
[NetID@ada1 ~]$ module spider keyword
To load a module, use:
[NetID@ada1 ~]$ module load moduleName
To list all currently loaded modules, use:
[NetID@ada1 ~]$ module list
To remove all currently loaded modules, use:
[NetID@ada1 ~]$ module purge
If you need new software or an update, please contact us with your request.
There are restrictions on what software we can install. There is also regularly a queue of requested software installations.
Please account for delays in your installation request timeline.
Transferring Files
Files can be transferred to Ada using the scp command or a file transfer program.
Our users most commonly utilize:
- WinSCP - Straightforward, legacy
- FileZilla Client - Easy to use, additional features, available on most platforms
- MobaXterm Graphical SFTP - Included with MobaXterm
Advice: while GUIs are acceptable for file transfers, the cp and scp commands are much quicker and may significantly benefit workflow.
Reliably Transferring Large Files
For files larger than several GB, users will want to consider the use of a more fault-tolerant utility such as rsync.
[NetID@ada1 ~]$ rsync -av [-z] localdir/ userid@remotesystem:/path/to/remotedir/
An rsync example can be seen on the Ada Fast Transfer page.
[Insert info on glob, ftn]
Graphic User Interfaces (Visualization)
The use of GUIs on Ada is a more complicated process than running non-interactive jobs or doing resource-light interactive processing.
Users have two options for using GUIs on Ada.
The first option is to run on the login node. When doing this, users must observe the fair-use policy of login node usage. Users commonly violate these policies by accident, resulting in terminated processes, confusion, and warnings from our admins.
The second option is to use a VNC job or the new Visualization Portal (viz-portal). In order to use the viz-portal, users must first be added manually to it. If you would like to have access to the viz-portal, please Contact Us. The documentation from the viz-portal short course including usage instructions can be found on the respective short course page.
The VNC method is outside the scope of this guide. See the Ada Remote Visualization page for more information.
