Hprc banner tamu.png

Difference between revisions of "SW:Matlab"

From TAMU HPRC
Jump to: navigation, search
(matlabsubmit)
(Retrieving fully populated Cluster Profile Object)
(245 intermediate revisions by 5 users not shown)
Line 1: Line 1:
== Starting Matlab==
 
  
To start matlab you need to load the Matlab module first. You can do this using the command
+
__TOC__
 +
= Running Matlab interactively =
 +
Matlab is accessible to all HPRC users within the terms of our license agreement. If you have particular concerns about whether specific usage falls within the TAMU HPRC license, please send an email to HPRC Helpdesk. You can start a Matlab session either directly on a login node or through our portal
  
<pre>
+
== Running Matlab on a login node ==
  -bash-4.1$ module load Matlab
 
</pre>
 
 
 
This will setup the environment for the latest Matlab version. To see a list of all installed versions, use the command '''module spider Matlab''' command. 
 
 
 
To start matlab, type:
 
  
<pre>
+
To be able to use Matlab, the Matlab module needs to be loaded first. This can be done using the following command:
  -bash-4.1$ matlab
+
[ netID@cluster ~]$ '''module load Matlab/R2019a'''
</pre>
 
  
depending on your X server settings this will start the Matlab GUI or the Matlab command line interface. To force matlab to start in command line interface mode, type:
+
This will setup the environment for Matlab version R2019a. To see a list of all installed versions, use the following command:
 +
[ netID@cluster ~]$ '''module spider Matlab'''
 +
<font color=teal>'''Note:''' New versions of software become available periodically. Version numbers may change.</font>
  
<pre>
+
To start matlab, use the following command:
  -bash-4.1$ matlab -nosplash -nodisplay
+
[ netID@cluster ~]$ '''matlab'''
</pre>
 
  
By default Matlab will enable its multi-threading capabilities and will use as many threads (cores) as available. Since our login nodes are shared we '''don't''' allow individual users to use  more than 8 cores on a node. For that reason we added Matlab code to the system startup script to limit the number of threads. Currently we set it to 4 threads. To explicitly disable multi threading, type:
+
Depending on your X server settings, this will start either the Matlab GUI or the Matlab command-line interface. To start Matlab in command-line interface mode, use the following command with the appropriate flags:
 +
[ netID@cluster ~]$ '''matlab -nosplash -nodisplay'''
  
<pre>
+
By default, Matlab will execute a large number of built-in operators and functions multi-threaded and will use as many threads (i.e. cores) as are available on the node. Since login nodes are shared among all users, HPRC restricts the number of computational threads to 8. This should suffice for most cases. Speedup achieved through multi-threading depends on many factors and in certain cases. To explicitly change the number of computational threads, use the following Matlab command:
  -bash-4.1$ matlab -singleCompThread
+
>>feature('NumThreads',4);
</pre>
 
  
NOTE, when you run interactively (i.e. on the login nodes) there are some other limits as well. For example, CPU time is limited to 1 hour ( your  matlab run will get killed by the system automatically after that). For other interactive limits, please see the appropriate pages for the system your are running on.
+
This will set the number of computational threads to 4.
  
==Batch Mode==
+
To completely disable multi-threading, use the -singleCompThread option when starting Matlab:
 +
[ netID@cluster ~]$ '''matlab -singleCompThread'''
  
When your sequential matlab needs more than 1 hour to execute (or requires more memory than allowed in an interactive session) you need to run your matlab job using the batch system. You can either write you own batch script and submit it manually to the batch scheduler or you can use the matlabsubmit script to run your matlab scripts automatically.
+
{{:SW:Login_Node_Warning}}
  
 +
== Running Matlab through the hprc portal ==
  
===matlabsubmit===
+
HPRC provides a portal through which users can start an interactive Matlab GUI session inside a web browser. For more information how to use the portal see our [[SW:Portal | HPRC OnDemand Portal]] section
  
 +
= Running Matlab through the batch system =
  
TAMU HPRC provides a tool named '''matlabsubmit''' to automate the process of running Matlab simulations on the compute nodes without the need to create your own batch script. This is the recommended way of running Matlab simulations on the compute nodes since it guarantees all resources are set correctly.
 
  
 +
HPRC developed a tool named '''matlabsubmit''' to run Matlab simulations on the HPRC compute nodes without the need to create your own batch script and without the need to start a Matlab session.  '''matlabsubmit''' will automatically generate a batch script with the correct requirements. In addition, '''matlabsubmit''' will also generate boilerplate Matlab code to set up the environment (e.g. the number of computational threads) and, if needed, will start a ''parpool'' using the correct Cluster Profile (''local'' if all workers fit on a single node and a cluster profile when workers are distribued over multiple nodes)
  
To submit your Matlab script just type:
+
To submit your Matlab script, use the following command:
 
<pre>
 
<pre>
  matlabsubmit myscript.m
+
[ netID@cluster ~]$ matlabsubmit myscript.m
 
</pre>
 
</pre>
  
When executing, matlabsubmit will do the following:
+
In the above example, '''matlabsubmit''' will use all default values for runtime, memory requirements, the number of workers, etc. To specify resources, you can use the command-line options of '''matlabsubmmit'''. For example:
 
 
- generate boiler plate Matlab code to setup the matlab environment (e.g. #threads, #workers) <br>
 
- generate a batch script with all resources set correctly and the command to run matlab <br>
 
- submit the generated batch script to the batch scheduler <br>
 
 
 
In Addition, matlabsubmit will also save the complete workspace after the matlab script finishes executing.
 
 
 
 
==== Example 1 ====
 
 
 
The following example shows the simplest use of matlabsubmit. It will execute matlab script '''test.m''' using all default values for batch resources. It will also show the output printed out by matlabsubmit. The output shows the resources requested, the actual '''bsub''' commands used to submit the job, the actual LSF JobID, and the location of output generated by Matlab and LSF.
 
  
 
<pre>
 
<pre>
-bash-4.1$ matlabsubmit test.m
+
[ netID@cluster ~]$ matlabsubmit -t 07:00 -s 4 myscript.m
===============================================
 
Running Matlab script with following parameters
 
-----------------------------------------------
 
Script    : test.m
 
Workers    : 0
 
Nodes      : 1
 
Mem/proc  : 2500
 
#threads  : 8
 
===============================================
 
 
 
bsub  -e MatlabSubmitLOG1/lsf.err -o MatlabSubmitLOG1/lsf.out 
 
      -L /bin/bash -n 8 -R span[ptile=8] -W 02:00 -M 2500
 
      -R rusage[mem=2500]     
 
      -J test1 MatlabSubmitLOG1/submission_script
 
 
 
Verifying job submission parameters...
 
Verifying project account...
 
    Account to charge:  082839397478
 
        Balance (SUs):    81535.6542
 
        SUs to charge:        16.0000
 
Job <2847580> is submitted to default queue <sn_regular>.
 
 
 
-----------------------------------------------
 
matlabsubmit ID        : 1
 
matlab output file    : MatlabSubmitLOG1/matlab.log
 
LSF/matlab output file : MatlabSubmitLOG1/lsf.out
 
LSF/matlab error file  : MatlabSubmitLOG1/lsf.err
 
-bash-4.1$
 
 
</pre>
 
</pre>
  
 +
will set the wall-time to 7 hours and makes sure Matlab will use 4 computational threads for its run ( '''matlabsubmit''' will also request 4 cores).
  
 
+
To see all options for '''matlabsubmit''' use the '''-h''' flag
 
 
This will execute matlab script myscript.m in batch mode. The matlab script myscript.m has to be in the current directory. After printing information about the job to run (including jobid) control is returned immediately.Screen output will be redirected to file
 
After your job finishes matlab will exit and all your results will be lost. If you want to save the results make sure you add the save command to your matlab code. You can use the load command later to load the results into matlab.
 
 
 
====Options with matlabsubmit====
 
 
 
Below is the output of running "matlabsubmit -h"
 
  
 
<pre>
 
<pre>
 +
[ netID@cluster ~]$ matlabsubmit -h
 +
Usage: /sw/hprc/sw/Matlab/bin/matlabsubmit [options] SCRIPTNAME
  
 
+
This tool automates the process of running Matlab codes on the compute nodes.
Usage: /software/tamusc/Matlab/bin/matlabsubmit [options] SCRIPTNAME
 
 
 
This script submits matlab script SCRIPTNAME to the job scheduler
 
  
 
OPTIONS:
 
OPTIONS:
Line 112: Line 64:
 
   -t sets the walltime; form hh:mm (e.g. -t 03:27)
 
   -t sets the walltime; form hh:mm (e.g. -t 03:27)
 
   -w sets the number of ADDITIONAL workers
 
   -w sets the number of ADDITIONAL workers
  -n sets the number of nodes to assign the workers to
 
 
   -g indicates script needs GPU  (no value needed)
 
   -g indicates script needs GPU  (no value needed)
   -b sets the billing account to use
+
   -b sets the billing account to use  
  -q sets the queue to be used
+
   -s set number of threads for multithreading (default: 8 ( 1 when -w > 0)
   -s set number of threads for multithreading (defaults on ada: 20 ( 1 when -w > 0)
+
   -p set number of workers per node
   -r reservation id
+
   -f run function call instead of script
   -f inidicates it will be a function call instead of script
+
  -x add explicit batch scheduler option
 
    
 
    
 
DEFAULT VALUES:
 
DEFAULT VALUES:
   memory  : 24000
+
   memory  : 2000 per core
 
   time    : 02:00
 
   time    : 02:00
 
   workers  : 0
 
   workers  : 0
 
   gpu      : no gpu  
 
   gpu      : no gpu  
   threading: on using 20 threads
+
   threading: on, 8 threads
 +
 
 
</pre>
 
</pre>
  
For example the command matlabsubmit -t "03:27:00" -m 17gb myscript.m will request 17gb of memory and 3 hours and 27 minutes of computing time and executes Matlab script myscript.m.
 
matlabsubmit will scan your script for matlabpool open/close commands and adjust them to use the correct cluster profile and requested number of workers so there is no need to change your script manually.
 
  
====Output files====
+
'''NOTE''' when using the '''-f''' flag to execute a function instead of a script, the function call must be enclosed with double quotes when it contains parentheses. For example: '''matlabsubmit -f "myfunc(21)"'''
  
matlabsubmit will create a directory named MatlabSubmitLOGN (where N is the jobid printed by the matlabsubmit command) where it stores output from the matlab window (matlab.log) and error messages from matlab and PBS (matlab.err). This directory will also contain the adjusted matlab script and the actual submission script.
+
<br>
NOTE: if your job needs more than 1 node, it will also create a number of other directories and files named JobNxxx (where N is matlab batch job id). Do not delete these directories when the computation is not finished yet. Matlab needs these directories and files for execution. After computation is finished you can safely remove them.
 
  
====Checking run status of matlabsubmit====
+
When executing, '''matlabsubmit''' will do the following:
 +
* generate boilerplate Matlab code to setup the Matlab environment (e.g. #threads, #workers) <br>
 +
* generate a batch script with all resources set correctly and the command to run Matlab <br>
 +
* submit the generated batch script to the batch scheduler and return control back to the user <br>
  
To check the status of the matlabsubmit command you can use the standard "qstat -u user" command (where user is your login id). The name of the job is the same as the name of the script you are executing (NOTE: when you need more than 1 node the name will be JobN).
 
  
===Writing your own batch file===
+
For detailed examples on using matlabsubmit see the [[ SW:Matlab_matlabsubmit | examples ]] section.
  
You can also write your own batch file and submit it manually to the batch system. Below is a simple example of a batch script
 
  
<pre>
 
#PBS -l nodes=1:ppn=1,walltime=02:00:00,mem=12gb
 
#PBS -N mymatlab
 
#PBS -S /bin/bash
 
#PBS -j oe
 
  
# load the matlab module
+
= Using Matlab Parallel Toolbox on HPRC Resources=
module load matlab
 
  
#$PBS_O_WORKDIR is the directory from which the job was submitted
 
cd $PBS_O_WORKDIR
 
  
## run the matlab command
+
<font color=red> ''THIS SECTION IS UNDER CONSTRUCTION'' </font><br>
matlab -nosplash -nodesktop << EOF
 
a = zeros(10,1);
 
for i=1:10
 
  a(i) = a(i) + i;
 
end
 
a
 
exit
 
EOF
 
  
</pre>
+
In this section, we will focus on utilizing the Parallel toolbox on HPRC cluster. For a general intro to the Parallel Toolbox see the  [https://www.mathworks.com/help/parallel-computing/index.html?s_tid=CRUX_lftnav  parallel toolbox ] section on the Mathworks website. Here we will discuss how to use Matlab Cluster profiles to distribute workers over multiple nodes.
  
The code between the EOF markers is the actual sample matlab snippet. You will replace this with your own matlab code. Always add the exit command at the end of your matlab code, it tells matlab to exit cleanly.
+
The central concept in most of the discussion below is the '''TAMUClusterProperties''' object which we will discuss in more detail in the next section
Instead of writing the actual matlab code in your batch script you can also create a separate Matlab script file (needs to have extension .m) where you write your Matlab code. As mentioned before, make sure the last matlab command in your matlab script will be the exit command. Suppose you name your matlab script myscript.m you can replace the matlab command above with:
 
  
<pre> 
 
matlab -nosplash -nodesktop -r myscript
 
</pre>
 
  
Screen output will be redirected to file as in any EOS batch run (in the sample batch sript it will be redirected to file mymatlab.oXXX, where XXX is the jobid).
+
== Cluster Profiles ==
 +
Matlab Cluster Profiles provide an interface to define properties of how and where to start Matlab workers. There are two kinds of profiles.
  
==Parallel Matlab==
+
* local profiles: parallel processing is limited to the same node the Matlab client is running.
 
+
* cluster profiles: parallel processing can span multiple nodes; profile interacts with a batch scheduler (e.g. SLURM on terra).
Matlab has excellent capabilities to run your matlab code in parallel; either using multiple cpus or using available gpus. Currently, the Supercomputing Facility has a license for 96 tokens (which means you can run your matlab code in parallel using 96 workers). For parallel computing capabilities Matlab uses Cluster profiles. A cluster profile lets you define certain properties of your cluster (e.g. how to submit jobs, submission parameters, job requirements, etc). Matlab will use the cluster profile to offload parallel (or sequential) matlab code to one or more worker and run it there.
+
Cluster Profiles
+
'''NOTE:''' we will not discuss ''local profiles'' any further here. Processing using a local profile is exactly the same as processing using cluster profiles.
 
 
NOTE: Instructions below are specific for Matlab version r2012b and r2013b. Later version use a little bit different format but the current format will still work.  
 
 
 
===Importing profile===
 
  
The Supercomputing facility already created a template Cluster Profile for your convenience. Before you can use this profile you need to import it first (you only need to do this once).
 
  
<pre>
+
=== Importing Cluster Profile ===
1. In the Matlab window, click 'Parallel/Manage Cluster Profiles/Import'
 
2. Choose '/software/tamuhprc/Matlab/profiles/TAMU.settings' file to import
 
</pre>
 
  
You can also just type on the matlab commandline:
+
For your convenience, HPRC already created a custom Cluster Profile. Using the profile, you can define how many workers you want, how you want to distribute the workers over the nodes, how many computational threads to use, how long to run, etc. Before you can use this profile you need to import it first. This can be done using by calling the following Matlab function.
  
 
<pre>
 
<pre>
cpname = parallel.importProfile('/software/tamuhprc/Matlab/profiles/TAMU');
+
>>tamuprofile.importProfile()
 
</pre>
 
</pre>
  
(where '''cpname''' is the variable that holds the cluster object
 
  
====Setting default profile====
+
This function imports the cluster profile and it creates a directory structure in your scratch directory where Matlab will store meta-information during parallel processing. The default location is ''/scratch/$USER/MatlabJobs/TAMU<VERSION'', where <VERSION> represents the Matlab version. For example, for Matlab R2019b it will be ''/scratch/$USER/MatlabJobs/TAMU2019b''
  
 +
<!--
 +
'''NOTE:''' function '''tamuprofile.clusterprofile''' is a wrapper around the Matlab function
 +
[https://www.mathworks.com/help/distcomp/parallel.importprofile.html parallel.importprofile]
 +
-->
  
To set the default profile do the following:
+
'''NOTE:''' For Matlab versions before R2019b, use the following function
 
<pre>
 
<pre>
1. In the Matlab window, click 'Parallel/Set Default'
+
>>tamu_import_TAMU_clusterprofile()
2. select TAMU.
 
</pre>
 
You can also type on the command line:
 
<pre>
 
parallel.defaultClusterProfile(cpname)
 
 
</pre>
 
</pre>
  
 +
In this case, Matlab will store meta-information in directory ''/scratch/$USER/MatlabJobs/TAMU''
  
====Changing Profile properties====
+
=== Getting Cluster Profile Object ===
  
Currently the default job requirements specified for the template cluster profile (imported above) are:  
+
To return a fully completed cluster object (i.e. with attached resource information) HPRC created the '''tamu_set_profile_properties''' convenience function. There are two steps to follow:
  
'-l nodes=^N^,mem=12gb,walltime=4:00:00'  
+
* define the properties using the TAMUClusterProperties class
 
+
* call '''tamu_set_profile_properties''' using the created TAMUClusterProperties object.
(the ^N^ will be replaced automatically by Matlab with the actual number of workers you specify).
 
 
 
If your job requirements are different (e.g. walltime, amount of memory) you need to modify them
 
  
 +
For example, suppose you have Matlab code and want to use 4 workers for parallel processing.
 +
 
<pre>
 
<pre>
1. In the Matlab window, click 'Parallel/Manage Profile/'
+
>> tp=TAMUClusterProperties;
2. select the profile you want to adjust
+
>> tp.workers(4);
3. click "Edit"
+
>> clusterObject=tamu_set_profile_properties(tp);
4. make the changes.
 
5. click "Done"
 
 
</pre>
 
</pre>
  
You can also change it on the command line but this requires a few more steps.  
+
Variable ''clusterObject'' is a fully populated cluster object that can be used for parallel processing.  
  
<pre>
+
'''NOTE:''' convenience function '''tamu_set_profile_properties''' is a wrapper around Matlab function
STEP 1: get a cluster object from the default cluster:
+
[https://www.mathworks.com/help/distcomp/parcluster.html parcluster]. It also uses HPRC convenience function '''tamu_import_TAMU_clusterprofile''' to check if the '''TAMU''' profile has been imported already.
dcluster=parcluster
 
STEP 2: set the job requirements using the ResourceTemplate property:
 
dcluster.ResourceTemplate='-l nodes=^N^,walltime=02:00:00,mem=20gb'
 
In this case your matlab job would request 2 hours, 20GB.  
 
STEP 3: if needed, add aditional parameters (e.g. billing account) using the SubmitArguments property:
 
dcluster.SubmitArguments='-l billto=XXX'
 
STEP 4: Save the changes to the cluster:
 
dcluster.saveProfile
 
</pre>
 
  
Note: although not as flexible, you can set the number of nodes explicetely by replacing ^N^ with actual values (e.g. 2::ppn=8).
+
== Starting a Parallel Pool ==
  
Now the default profile has been updated with the above changes.  
+
To start a parallel pool you can use the HPRC convenience function '''tamu_parpool'''. It takes as argument a '''TAMUClustrerProperties''' object that specifies all the resources that are requested.  
  
In case you want to save it as a different profile:
+
The '''parpool''' functions enables the full functionality of the parallel language features (parfor and spmd, will be discussed below). A parpool creates a special job on a pool of workers, and connects the pool to the MATLAB client. For example:
dcluster.saveAsProfile('NEWNAME')
+
<pre>
Where NEWNAME is the name you choose for the profile.
+
mypool = parpool 4
Parallel constructs
+
    :
 +
delete(mypool)
 +
</pre>
  
We will discuss briefly some of the most common parallel matlab concepts. For more detailed information about these constructs, as well as additional parallel constructs consult the Parallel Computing Toolbox User Guide
 
matlabpool
 
 
The matlabpool functions enables the full functionality of the parallel language features (parfor and spmd, will be discussed below). matlabpool creates a special job on a pool of workers, and connects the pool to the MATLAB client. For example:
 
matlabpool open 4
 
    :
 
    :
 
matlabpool close
 
 
This code starts a worker pool using the default cluster profile, with 4 additional workers.  
 
This code starts a worker pool using the default cluster profile, with 4 additional workers.  
  
 
NOTE: only instructions within parfor and spmd blocks are executed on the workers. All other instructions are executed on the client.  
 
NOTE: only instructions within parfor and spmd blocks are executed on the workers. All other instructions are executed on the client.  
  
NOTE: all variables declared inside the matlabpool block will be destroyed once the block is finished.
+
NOTE: all variables declared inside the matlabpool block will be destroyed once the block is finished.
 
 
For more detailed information please visit the Matlab matlabpool page.
 
parfor
 
 
 
The concept of a parfor-loop is similar to the standard Matlab for-loop. The difference is that parfor partitions the iterations among the available
 
workers to run in parallel. For example:
 
 
 
<pre>
 
matlabpool open  2
 
parfor i=1:1024
 
  A(i)=sin((i/1024)*2*pi);
 
end
 
matlabpool close
 
</pre>
 
This code will open a matlab pool with 2 workers using the default cluster profile and execute the loop in parallel.
 
 
 
For more information please visit the Matlab parfor page.
 
spmd
 
 
 
spmd runs the same program on all workers concurrently. A typical use of spmd is when you need to run the same program on multiple sets of input. For example, Suppose you have 4 inputs named data1,data2,data3,data4 and you want run funcion myfun on all of them:
 
 
 
<pre>
 
matlabpool open  4
 
spmd (4)
 
    data = load(['data' num2str(labindex)])
 
    myresult = myfun(data)
 
end
 
matlabpool close
 
</pre>
 
NOTE: labindex is a Matlab variable and is set to the worker id, values range from 1 to number of workers.  
 
  
Every worker will have its own version of variable myresult. To access these variables outside the spmd block you append {i} to the variable name, e.g. myresult{3} represents variable myresult from worker 3.
+
== Using GPU ==
  
For more information please visit the Matlab spmd page.
+
Normally all variables reside in the client workspace and matlab operations are executed on the client machine. However, Matlab also provides options to utilize available GPUs to run code faster.
batch
 
 
 
The parallel constructs we discussed so far are all interactive, meaning that the client starts the workers and then waits for completion of the job before accepting any other input. The batch command will submit a job and return control back to the client immediately. For example, suppose we want to run the parfor loop from above without waiting for the result. First create a matlab function myloop.m
 
 
 
<pre>
 
parfor i=1:1024
 
  A(i)=sin((i/1024)*2*pi);
 
end
 
</pre>
 
 
 
To run using the batch command:
 
myjob = batch('myloop','matlabpool',4)
 
This will start the parallel job on the workers and control is returned to the client immediately. To see all your running jobs click on Parallel/Monitor Jobs. Use the wait command, e.g. wait(myjob), to wait for the job to finish, use the load command, e.g. load(myjob), to load all variables from the job into the client workspace.
 
 
 
For more information please visit the Matlab batch page.
 
Using GPU
 
 
 
Normally all variables reside in the client workspace and matlab operations are executed on the client machine (e.g. your desktop, or an eos login node). However, Matlab also provides options to utilize available GPUs to run code faster.
 
 
Running code on the gpu is actually very straightforward. Matlab provides GPU versions for many build-in operations. These operations are executed on the GPU automatically when the variables involved reside on the GPU. The results of these operations will also reside on the GPU. To see what functions can be run on the GPU type:
 
Running code on the gpu is actually very straightforward. Matlab provides GPU versions for many build-in operations. These operations are executed on the GPU automatically when the variables involved reside on the GPU. The results of these operations will also reside on the GPU. To see what functions can be run on the GPU type:
  
Line 331: Line 187:
 
gpuDevice
 
gpuDevice
 
This functions shows all the properties of the GPU. When this function is called from the client (or a node without a GPU) it will just print an error message.
 
This functions shows all the properties of the GPU. When this function is called from the client (or a node without a GPU) it will just print an error message.
Adjusting Cluster Profile
 
  
to use the gpus on EOS we need to adjust the job requirements to make sure the job is scheduled on a node with a gpu, the same way you would do it with a regular eos job.
 
<pre>
 
dcluster = parcluster
 
dcluster.ResourceTemplate='-l nodes=1:ppn=1:gpus=1,walltime=02:00:00,mem=20gb'
 
</pre>
 
 
The above job requirements are just an example. You can adjust the various properties to suit your needs.
 
More detailed information about changing Profile Properties can be found here
 
Copying between client and GPU
 
  
 
To copy variables from the client workspace to the GPU, you can use the gpuArray command. For example:
 
To copy variables from the client workspace to the GPU, you can use the gpuArray command. For example:
Line 349: Line 195:
 
</pre>
 
</pre>
  
will copy variable carr to the GPU wit name garr. If variable carr is not used in the client workspace you can write it as:
+
will copy variable carr to the GPU wit name garr.  
  
<pre>
+
In the example above the 1000x1000 matrix needs to be copied from the client workspace to the GPU. There is a significant overhead involved in doing this.
garr = gpuArray(ones(1000));
 
</pre>
 
  
The two versions have the same problem. They both need to copy the 1000x1000 matrix from client workspace to the GPU. We mentioned above that Matlab provides methods to create data directly on the GPU to avoid the overhead of copying data to the GPU. For example:
+
To create the variables directly on the GPU, Matlab provides a number of convenience functions. For example:
 
<pre>
 
<pre>
 
garr=gpuArray.ones(1000)
 
garr=gpuArray.ones(1000)
Line 361: Line 205:
  
 
This will create a 1000x1000 matrix directly on the GPU consisting of all ones.  
 
This will create a 1000x1000 matrix directly on the GPU consisting of all ones.  
 
You can find a list of all methods to create data directly on the GPU here.
 
  
  
Line 371: Line 213:
  
 
This will copy the array garr on the GPU back to variable carr2 in the client workspace.
 
This will copy the array garr on the GPU back to variable carr2 in the client workspace.
Overhead
 
  
As mentioned before there is considerable overhead involved when using the GPU. Actually, there are two types of overhead.
+
The next example performs a matrix multiplication on the client, a matrix multiplication on the GPU, and prints out elapsed times for both. The actual cpu-gpu matrix multiplication code can be written as:
Warming up GPU (first time GPU is used).
+
<pre>
Data transfer.
+
ag = gpuArray.rand(1000);
Warming up
+
bg = ag*ag;
 +
c = gather(bg);
 +
</pre>
 +
 
 +
= Running (parallel) Matlab Scripts on HPRC compute nodes =
 +
 
 +
'''NOTE:''' Due to the new 2-factor authentication mechanism, this method does not work at the moment. We will update this wiki page when this is fixed.
 +
 
 +
 
 +
For detailed information how to submit Matlab codes remotely, click [[SW:Matlab_app | here]]
 +
 
 +
== Submit Matlab Scripts Remotely or Locally From the Matlab Command Line  ==
 +
 
 +
'''NOTE:''' Due to the new 2-factor authentication mechanism, remote submission method does not work at the moment. We will update this wiki page when this is fixed.
 +
 
 +
Instead of using the App you can also call Matlab functions (developed by HPRC)  directly to run your Matlab script on HPRC compute nodes. There are two steps involved in submitting your Matlab script:
 +
 
 +
* Define the properties for your Matlab script (e.g. #workers).  HPRC created a class named '''TAMUClusterProperties''' for this
 +
* Submit the Matlab script to run on HPRC compute nodes. HPRC created a function named '''tamu_run_batch''' for this.
 +
 
 +
For example, suppose you have a script named ''mysimulation.m'', you want to use 4 workers and estimate it will need less than 7 hours of computing time:
 +
 
 +
<pre>
 +
>> tp=TAMUClusterProperties();
 +
>> tp.workers(4);
 +
>> tp.walltime('07:00');
 +
>> myjob=tamu_run_batch(tp,'mysimulation.m');
 +
</pre>
  
When the GPU is just starting up computation, there are many things that need to be done, both on the Matlab part and the GPU device itself (e.g. loading libraries, initializing the GPU state, etc). For example:
+
'''NOTE:''' '''TAMUClusterProperties''' will use all default values for any of the properties that have not been set explicitly.  
matlabpool open 1
 
spmd 1
 
tic
 
gpuArray.ones(10,1);
 
toc
 
end
 
This code only creates a 10x1 array of ones on the GPU device. The first run takes an astounding 21.5 seconds to execute while every successive run only needs about 0.00017 seconds. This shows the huge cost of warming up the GPU.  
 
  
NOTE:These are running times on EOS. Other systems might have very different timing results.
+
In case you want to submit your Matlab script remotely from your local Matlab GUI, you also have to specify the HPRC cluster name you want to run on and your username.  
Data transfer
+
For example, suppose you have a script that uses Matlab GPU functions and you want to run it on terra:
 +
<pre>
 +
>> tp=TAMUClusterProperties();
 +
>> tp.gpu(1);
 +
>> tp.hostname('terra.tamu.edu');
 +
>> tp.user('<USERNAME>'); 
 +
>> myjob=tamu_run_batch(tp,'mysimulation.m');
 +
</pre>
  
GPU operations in Matlab can only be done when the data is physically located on the GPU device. Therefore data might need to be transferred to the GPU device (and vice versa). This is a significant overhead. For example:
+
To see all available methods on objects of type '''TAMUClusterProperties''' you can use the Matlab '''help''' or '''doc''' functions: E.g.
spmd 1
 
tic;ag=gpuArray(ones(10000));toc;
 
end
 
The above code only copies a 10000x10000 matrix from client workspace to GPU device. The time it takes is almost 0.6 seconds. This is a significant overhead.
 
Example
 
  
Here is a little example that performs a matrix multiplication on the client, a matrix multiplication on the GPU, and prints out elapsed times for both. The actual cpu-gpu matrix multiplication code can be written as:
+
  >> help TAMUClusterProperties/doc
a =  rand(1000);
 
tic; b = a*a; toc;
 
tic; ag = gpuArray(a); bg = ag*ag; toc;
 
c = gather(cg)
 
Almost no additional steps are required to use the gpu. Actually, copying the results to the client workspace is not even needed. Variables that reside on the gpu can be printed or plotted just like variables in the client workspace.
 
  
The above code will run without problems if Matlab is installed on a computer with a gpu attached. Since EOS does not have gpus attached to the login nodes (where the client is running) we need to ensure the above code is run on a gpu node. We will show how to do it in interactive mode (using matlabpool), and by using the Matlab batch command.
+
To see help page for '''tamu_run_batch''', use:
  
For convenience the code above is saved as mymatrixmult.m
+
<pre>
Interactive using matlabpool
+
  >> help tamu_run_batch
 +
      tamu_run_batch  runs Matlab script on worker(s).
 +
 +
        j = TAMU_RUN_BATH(tp,'script') runs the script
 +
        script.m on the worker(s) using the TAMUClusterProperties object tp.
 +
        Returns j, a handle to the job object that runs the script.
 +
 
 +
 
 +
</pre>
  
A matlabpool needs to be opened since a gpu node is needed and the client is running on one of the login nodes (no gpu available) and mymatrixmult needs to be inside a spmd block to ensure code will actually run on the worker instead of the client (see matlabpool section). The code will be as follows:
 
matlabpool open 1
 
spmd 1
 
mymatrixmult
 
end
 
matlabpool close
 
Using Matlab batch command
 
  
This example is a basic sequential code (i.e. uses only one cpu core), so in this case a matlabpool is not even needed. The Matlab batch command will start the job on one of the workers (which has a gpu). The code will look as follows:
+
'''tamu_run_batch''' returns a variable of type '''Job'''. See the section ''"Retrieve results and information from Submitted Job"'' how to get results and information from the submitted job.
batch('mymatrixmult')
 
Warming up the GPU
 
  
there is considerable overhead involved when using the GPU. Besides the data transfer overhead mentioned before, there is another kind of overhead; warming up time. When the GPU is just starting up computation, there are many things that need to be done, both on the Matlab part and the GPU device itself (e.g. loading libraries, initializing the GPU state, etc). To get an indication how much time is needed look at the following example:
 
                                                                                   
 
matlabpool open 1
 
spmd 1
 
tic
 
gpuArray.ones(10,1);
 
toc
 
end
 
This code only creates a 10x1 array of ones on the GPU device. The first run takes 0.026 seconds to execute while every sucessive run only needs about 0.00017 seconds (of course different runs will produce slightly different results). This shows the huge cost of warming up the GPU .
 
  
NOTE:These are running times on EOS. Other systems might have very different timing results.
+
[[Category:Software]]

Revision as of 23:58, 24 February 2020

Running Matlab interactively

Matlab is accessible to all HPRC users within the terms of our license agreement. If you have particular concerns about whether specific usage falls within the TAMU HPRC license, please send an email to HPRC Helpdesk. You can start a Matlab session either directly on a login node or through our portal

Running Matlab on a login node

To be able to use Matlab, the Matlab module needs to be loaded first. This can be done using the following command:

[ netID@cluster ~]$ module load Matlab/R2019a

This will setup the environment for Matlab version R2019a. To see a list of all installed versions, use the following command:

[ netID@cluster ~]$ module spider Matlab

Note: New versions of software become available periodically. Version numbers may change.

To start matlab, use the following command:

[ netID@cluster ~]$ matlab

Depending on your X server settings, this will start either the Matlab GUI or the Matlab command-line interface. To start Matlab in command-line interface mode, use the following command with the appropriate flags:

[ netID@cluster ~]$ matlab -nosplash -nodisplay

By default, Matlab will execute a large number of built-in operators and functions multi-threaded and will use as many threads (i.e. cores) as are available on the node. Since login nodes are shared among all users, HPRC restricts the number of computational threads to 8. This should suffice for most cases. Speedup achieved through multi-threading depends on many factors and in certain cases. To explicitly change the number of computational threads, use the following Matlab command:

>>feature('NumThreads',4);

This will set the number of computational threads to 4.

To completely disable multi-threading, use the -singleCompThread option when starting Matlab:

[ netID@cluster ~]$ matlab -singleCompThread

Usage on the Login Nodes

Please limit interactive processing to short, non-intensive usage. Use non-interactive batch jobs for resource-intensive and/or multiple-core processing. Users are requested to be responsible and courteous to other users when using software on the login nodes.

The most important processing limits here are:

  • ONE HOUR of PROCESSING TIME per login session.
  • EIGHT CORES per login session on the same node or (cumulatively) across all login nodes.

Anyone found violating the processing limits will have their processes killed without warning. Repeated violation of these limits will result in account suspension.
Note: Your login session will disconnect after one hour of inactivity.

Running Matlab through the hprc portal

HPRC provides a portal through which users can start an interactive Matlab GUI session inside a web browser. For more information how to use the portal see our HPRC OnDemand Portal section

Running Matlab through the batch system

HPRC developed a tool named matlabsubmit to run Matlab simulations on the HPRC compute nodes without the need to create your own batch script and without the need to start a Matlab session. matlabsubmit will automatically generate a batch script with the correct requirements. In addition, matlabsubmit will also generate boilerplate Matlab code to set up the environment (e.g. the number of computational threads) and, if needed, will start a parpool using the correct Cluster Profile (local if all workers fit on a single node and a cluster profile when workers are distribued over multiple nodes)

To submit your Matlab script, use the following command:

[ netID@cluster ~]$ matlabsubmit myscript.m

In the above example, matlabsubmit will use all default values for runtime, memory requirements, the number of workers, etc. To specify resources, you can use the command-line options of matlabsubmmit. For example:

[ netID@cluster ~]$ matlabsubmit -t 07:00 -s 4 myscript.m

will set the wall-time to 7 hours and makes sure Matlab will use 4 computational threads for its run ( matlabsubmit will also request 4 cores).

To see all options for matlabsubmit use the -h flag

[ netID@cluster ~]$ matlabsubmit -h
Usage: /sw/hprc/sw/Matlab/bin/matlabsubmit [options] SCRIPTNAME

This tool automates the process of running Matlab codes on the compute nodes.

OPTIONS:
  -h Shows this message
  -m set the amount of requested memory in MEGA bytes(e.g. -m 20000)
  -t sets the walltime; form hh:mm (e.g. -t 03:27)
  -w sets the number of ADDITIONAL workers
  -g indicates script needs GPU  (no value needed)
  -b sets the billing account to use 
  -s set number of threads for multithreading (default: 8 ( 1  when -w > 0)
  -p set number of workers per node
  -f run function call instead of script
  -x add explicit batch scheduler option
  
DEFAULT VALUES:
  memory   : 2000 per core 
  time     : 02:00
  workers  : 0
  gpu      : no gpu 
  threading: on, 8 threads


NOTE when using the -f flag to execute a function instead of a script, the function call must be enclosed with double quotes when it contains parentheses. For example: matlabsubmit -f "myfunc(21)"


When executing, matlabsubmit will do the following:

  • generate boilerplate Matlab code to setup the Matlab environment (e.g. #threads, #workers)
  • generate a batch script with all resources set correctly and the command to run Matlab
  • submit the generated batch script to the batch scheduler and return control back to the user


For detailed examples on using matlabsubmit see the examples section.


Using Matlab Parallel Toolbox on HPRC Resources

THIS SECTION IS UNDER CONSTRUCTION

In this section, we will focus on utilizing the Parallel toolbox on HPRC cluster. For a general intro to the Parallel Toolbox see the parallel toolbox section on the Mathworks website. Here we will discuss how to use Matlab Cluster profiles to distribute workers over multiple nodes.

The central concept in most of the discussion below is the TAMUClusterProperties object which we will discuss in more detail in the next section


Cluster Profiles

Matlab Cluster Profiles provide an interface to define properties of how and where to start Matlab workers. There are two kinds of profiles.

  • local profiles: parallel processing is limited to the same node the Matlab client is running.
  • cluster profiles: parallel processing can span multiple nodes; profile interacts with a batch scheduler (e.g. SLURM on terra).

NOTE: we will not discuss local profiles any further here. Processing using a local profile is exactly the same as processing using cluster profiles.


Importing Cluster Profile

For your convenience, HPRC already created a custom Cluster Profile. Using the profile, you can define how many workers you want, how you want to distribute the workers over the nodes, how many computational threads to use, how long to run, etc. Before you can use this profile you need to import it first. This can be done using by calling the following Matlab function.

>>tamuprofile.importProfile()


This function imports the cluster profile and it creates a directory structure in your scratch directory where Matlab will store meta-information during parallel processing. The default location is /scratch/$USER/MatlabJobs/TAMU<VERSION, where <VERSION> represents the Matlab version. For example, for Matlab R2019b it will be /scratch/$USER/MatlabJobs/TAMU2019b


NOTE: For Matlab versions before R2019b, use the following function

>>tamu_import_TAMU_clusterprofile()

In this case, Matlab will store meta-information in directory /scratch/$USER/MatlabJobs/TAMU

Getting Cluster Profile Object

To return a fully completed cluster object (i.e. with attached resource information) HPRC created the tamu_set_profile_properties convenience function. There are two steps to follow:

  • define the properties using the TAMUClusterProperties class
  • call tamu_set_profile_properties using the created TAMUClusterProperties object.

For example, suppose you have Matlab code and want to use 4 workers for parallel processing.

>> tp=TAMUClusterProperties;
>> tp.workers(4);
>> clusterObject=tamu_set_profile_properties(tp);

Variable clusterObject is a fully populated cluster object that can be used for parallel processing.

NOTE: convenience function tamu_set_profile_properties is a wrapper around Matlab function parcluster. It also uses HPRC convenience function tamu_import_TAMU_clusterprofile to check if the TAMU profile has been imported already.

Starting a Parallel Pool

To start a parallel pool you can use the HPRC convenience function tamu_parpool. It takes as argument a TAMUClustrerProperties object that specifies all the resources that are requested.

The parpool functions enables the full functionality of the parallel language features (parfor and spmd, will be discussed below). A parpool creates a special job on a pool of workers, and connects the pool to the MATLAB client. For example:

mypool = parpool 4
    :
delete(mypool)

This code starts a worker pool using the default cluster profile, with 4 additional workers.

NOTE: only instructions within parfor and spmd blocks are executed on the workers. All other instructions are executed on the client.

NOTE: all variables declared inside the matlabpool block will be destroyed once the block is finished.

Using GPU

Normally all variables reside in the client workspace and matlab operations are executed on the client machine. However, Matlab also provides options to utilize available GPUs to run code faster. Running code on the gpu is actually very straightforward. Matlab provides GPU versions for many build-in operations. These operations are executed on the GPU automatically when the variables involved reside on the GPU. The results of these operations will also reside on the GPU. To see what functions can be run on the GPU type:

methods('gpuArray') This will show a list of all available functions that can be run on the GPU, as well as a list of available static functions to create data on the GPU directly (will be discussed later).

NOTE: There is significant overhead of executing code on the gpu because of memory transfers.

Another useful function is: gpuDevice This functions shows all the properties of the GPU. When this function is called from the client (or a node without a GPU) it will just print an error message.


To copy variables from the client workspace to the GPU, you can use the gpuArray command. For example:

carr = ones(1000);
garr = gpuArray(carr);

will copy variable carr to the GPU wit name garr.

In the example above the 1000x1000 matrix needs to be copied from the client workspace to the GPU. There is a significant overhead involved in doing this.

To create the variables directly on the GPU, Matlab provides a number of convenience functions. For example:

garr=gpuArray.ones(1000)

This will create a 1000x1000 matrix directly on the GPU consisting of all ones.


To copy data back to the client workspace Matlab provides the gather operation.

carr2 = gather(garr)

This will copy the array garr on the GPU back to variable carr2 in the client workspace.

The next example performs a matrix multiplication on the client, a matrix multiplication on the GPU, and prints out elapsed times for both. The actual cpu-gpu matrix multiplication code can be written as:

ag = gpuArray.rand(1000); 
bg = ag*ag;
c = gather(bg); 

Running (parallel) Matlab Scripts on HPRC compute nodes

NOTE: Due to the new 2-factor authentication mechanism, this method does not work at the moment. We will update this wiki page when this is fixed.


For detailed information how to submit Matlab codes remotely, click here

Submit Matlab Scripts Remotely or Locally From the Matlab Command Line

NOTE: Due to the new 2-factor authentication mechanism, remote submission method does not work at the moment. We will update this wiki page when this is fixed.

Instead of using the App you can also call Matlab functions (developed by HPRC) directly to run your Matlab script on HPRC compute nodes. There are two steps involved in submitting your Matlab script:

  • Define the properties for your Matlab script (e.g. #workers). HPRC created a class named TAMUClusterProperties for this
  • Submit the Matlab script to run on HPRC compute nodes. HPRC created a function named tamu_run_batch for this.

For example, suppose you have a script named mysimulation.m, you want to use 4 workers and estimate it will need less than 7 hours of computing time:

>> tp=TAMUClusterProperties();
>> tp.workers(4);
>> tp.walltime('07:00');
>> myjob=tamu_run_batch(tp,'mysimulation.m');

NOTE: TAMUClusterProperties will use all default values for any of the properties that have not been set explicitly.

In case you want to submit your Matlab script remotely from your local Matlab GUI, you also have to specify the HPRC cluster name you want to run on and your username. For example, suppose you have a script that uses Matlab GPU functions and you want to run it on terra:

>> tp=TAMUClusterProperties();
>> tp.gpu(1);
>> tp.hostname('terra.tamu.edu');
>> tp.user('<USERNAME>');  
>> myjob=tamu_run_batch(tp,'mysimulation.m');

To see all available methods on objects of type TAMUClusterProperties you can use the Matlab help or doc functions: E.g.

  >> help TAMUClusterProperties/doc 

To see help page for tamu_run_batch, use:

   >> help tamu_run_batch
      tamu_run_batch  runs Matlab script on worker(s). 
 
         j = TAMU_RUN_BATH(tp,'script') runs the script
         script.m on the worker(s) using the TAMUClusterProperties object tp.
         Returns j, a handle to the job object that runs the script.



tamu_run_batch returns a variable of type Job. See the section "Retrieve results and information from Submitted Job" how to get results and information from the submitted job.