Hprc banner tamu.png


Revision as of 11:07, 19 August 2019 by Tmarkhuang (talk | contribs) (Access Using SSH)
Jump to: navigation, search

Ada Introductory Exercises

Before cluster use, please review our policy page.

Log in to Ada

1. This process can vary based on where you're accessing the cluster from as well as the machine you're using to do so. A couple of common methods are given below.

Access Using SSH

To connect to HPRC clusters, you must use SSH (Secure Shell). SSH is a client-server software that provides for secure (by encryption) logins and other communication between two hosts. SSH is freely available on the Internet for Linux/Unix and PC Windows (in the guise of MobaXterm).

To initiate SSH connection to target cluster's login node, find the hostname and credential (userID/password) info from Table 1 below. For example, if you are connecting to Ada cluster from a terminal enter:

[user1@localhost ~]$ ssh NetID@ada.tamu.edu

where the ada.tamu.edu address is a DNS round-robin alias for ada[1-8].tamu.edu. You will be prompted for your password in order to establish authentication. Once you login into one of the login nodes, the shell's prompt will be [NetID].ada[1-8].

If, however, you are connecting for the very first time, you will see a message similar to the following before arriving at the password prompt:

The authenticity of host 'ada (' can't be established.
ECDSA key fingerprint is SHA256:SfQPtDJW30sj4kG2c4KGFw7LcEduSOFeXGIlsf4WhEA.
ECDSA key fingerprint is MD5:9c:ea:ba:22:0f:6f:1e:b9:0c:21:d4:b6:70:0f:a0:d5.
Are you sure you want to continue connecting (yes/no)? 

Type yes and you will then be presented with the password prompt.

Warning: Permanently added 'ada' (ECDSA) to the list of known hosts.
NetID@ada.tamu.edu's password:

Access from Windows

Using MobaXterm (Recommended)

MobaXterm is an enhanced terminal for Windows with a built-in X11 server, tabbed SSH client, built-in file editor, SFTP functionality, and other useful features. It is available from: http://mobaxterm.mobatek.net/download.html

You will need to choose which license (free Home edition, or professional) and then select the Portable or Installer edition. The Installer edition works best on your personal machine, when you have the privileges to install software. The portable version may be necessary when using a lab workstation, for example. (Be sure to check if MobaXterm is already installed in the Windows Start menu.)

Local Terminal
  1. Open MobaXterm and start a local terminal:
  2. Moba Start-local.png
  3. From the local terminal, connect to the cluster using SSH:
  4. Moba from-local.png

If you encounter a permission error like this:

MobaXterm permission error message

see the special instructions here.

SSH Session
  1. Open MobaXterm and open the Session settings:
  2. Session.png
  3. Enter the hostname in the "Remote Host" field, check the box for "Specify username" and enter the proper username for the cluster you are connecting to, ada.tamu.edu in this case:
    Moba ssh.png
  4. Once you click OK you will be prompted to enter your password. Enter the proper password for the cluster you are connecting to.

Please visit the detailed access wiki page for the Ada cluster for more information.

After logging in, please take careful notice of the message of the day, which will include help resources, announcements, and policy information.

Login Environment Information

When you first access Ada, you will be within your home directory. This directory has smaller storage quotas and should not be used for general purpose. You will also see your current disk and file quota's printed along with the Message of the Day. The Message of the Day includes important policy information and announcements.

You can check your quota with the following command:

[NetID@ada1 ~]$ showquota

You can view the Message of the Day with the following command:

[NetID@ada1 ~]$ motd

You can see your current location in the system with the "print work directory" command:

[NetID@ada1 ~]$ pwd

By default, user's will be placed in their home directory. The home directory has a limit of 10,000 files or 10GB of data. Per our policy, the home directory quota cannot be expanded

[NetID@ada1 ~]$ cd $SCRATCH

Navigate to home with the following command:

[NetID@ada1 ~]$ cd $HOME

Your scratch directory is restricted to 1TB/250,000 files of storage. This storage quota is expandable upon request.

Your home directory is restricted to 10GB/10,000 files of storage. This storage quota is not expandable.

You can see the current status of your storage quotas with:

[NetID@ada1 ~]$ showquota

If you need a storage quota increase, please contact us with justification and the expected length of time that you will need the quota increase.

The Batch System

The batch system is a load distribution implementation that ensures convenient and fair use of a shared resource. Submitting jobs to a batch system allows a user to reserve specific resources with minimal interference to other users. All users are required to submit resource-intensive processing to the compute nodes through the batch system - attempting to circumvent the batch system is not allowed.

On Ada, LSF is the batch system that provides job management. More information on LSF can be found in the Ada Batch page.

Login Nodes are access points to the cluster, and like other nodes are shared by all users.
Use the who command to see all users logged into the login node. You can see which login host you're using based on the information following the @ sign next to your NetID at the beginning of each command line (eg. ada1.tamu.edu)

Note that of the 8 login nodes, there are some important hardware distinctions.

Use the env command to view environment information.

Remember that login nodes are not intended for heavy computing use, and sessions will be terminated after one hour.

Job Information

Computing resources are allocated to users through a job submission platform called LSF. To request resources using LSF, you must first create a job file.

This can be done with any text editor. For this example, we'll use vim. LSF job files begin with setting job parameters which tell the machine how many cores, nodes, and time to allot to your job, as well as how much memory will be required. Each of these parameters begins with #BSUB. After all parameters have been provided, the job file can continue into using regular UNIX commands.

Job Submission

First, create a job file in the text editor of your choice.

Files can be uploaded to the cluster from your local machine using the rsync command.

rsync /local_path_to_file NetID@ada.tamu.edu:/scratch/user/NetID/

If the connection is established, you will be prompted for your NetID password. When entered successfully, file transfer will begin.

Upon completion, your file will be available for use directly on the cluster.

For now, it is ok to base your jobs from the examples on our Ada batch processing page. The most basic job example is used below.

This job will output text that reads "hello world".

#BSUB -J ExampleJob1         #Set the job name to "ExampleJob1"
#BSUB -L /bin/bash           #Uses the bash login shell to initialize the job's execution environment.
#BSUB -W 2:00                #Set the wall clock limit to 2hr
#BSUB -n 1                   #Request 1 core
#BSUB -R "span[ptile=1]"     #Request 1 core per node.
#BSUB -R "rusage[mem=5000]"  #Request 5000MB per process (CPU) for the job
#BSUB -M 5000                #Set the per process enforceable memory limit to 5000MB.
#BSUB -o Example1Out.%J      #Send stdout and stderr to "Example1Out.[jobID]" 
##OPTIONAL JOB SPECIFICATIONS ##BSUB -P 123456 #[commented out] Set billing account to 123456. Without this, it will use the default allocation account. #BSUB -u email_address #Send all emails to email_address #BSUB -B -N #Send email on job begin (-B) and end (-N)

#First Executable Line: echo "hello world"

Save the file, and now it's ready to be submitted.

bsub < testjob.lsf

The job will be added to a queue where it will run as soon as the resources are available. If a default queue is not fitting for your job, a queue can be specified with a #BSUB command like so:

#BSUB -R "select[gpu]" #selects a node with a GPU

Job Monitoring

After a job has been submitted, you may want to check on its progress or cancel it. Below is a list of the most used job monitoring and control commands for jobs on Ada.

Ada (LSF) Job Monitoring and Control Commands
Function Command Example
Submit a job bsub < [script_file] bsub < MyJob.LSF
Cancel/Kill a job bkill [Job_ID] bkill 101204
Check summary status of a single job bjobs [job_id] bjobs 101204
Check summary status of all
jobs for a user
bjobs -u [user_name] bjobs -u adaUser1
Check detailed status of a single job bjobs -l [job_id] bjobs -l 101204
Modify job submission options bmod [bsub_options] [job_id] bmod -W 2:00 101204