SW:Comsol

When a model is built in Comsol GUI, the next step is to compute the model for a solution, which is often time-consuming. A job script must be created to run the model in batch to have controllable duration and computational resources. This tutorial illustrates how to create Comsol LSF batch scripts on Ada.

All solvers in Comsol can run in parallel in one of three parallel modes: shared memory mode, distributed mode, or hybrid mode. By default, a Comsol solver runs in shared memory mode. This is the same as OpenMP where the parallelism is limited by total number of CPU cores available in one compute node in a cluster.

A Complete Example

Example 1: solving a model in shared memory using 20 cores in one cluster node.

 #BSUB -J comsoltest
 #BSUB -n 20 -R "span[ptile=20]"
 #BSUB -M 2800 -R "rusage[mem=2800]"
 #BSUB -o output.%J
 #BSUB -L /bin/bash
 #BSUB -W 2:00
 
 module load Comsol/xxx 
 comsol -np 20 batch -inputfile in.mph -outputfile out.mph

Note: xxx represents a Comsol version. You need to pick the version you need to access.

Running Comsol in Different Parallel Mode

Assuming other things are the same as in Example 1, we will see additional examples running in different parallel modes by changing number of cores and Comsol command line parameters.

Shared Memory Mode

Example 2: Solving a model in shared mode and using 10 cores in one cluster node. This is the similar as Example 1.

 #BSUB -n 10 -R "span[ptile=10]"
 comsol -np 10 batch -inputfile input.mph -outputfile output.mph

Distributed Mode

Comsol solvers can also run in distributed mode by checking the "distributed computing" checkbox of the solver when building the model. In this mode, the solver runs on multiple nodes and uses MPI for communication. Except PARDISO, all solvers support distributed mode. However, PARDISO also has a check box for distributed computing. If selected, the actual solver used is MUMPS.

Example 3: Solving a model in distributed mode on two cluster nodes with a total of 40 cores

 #BSUB -n 40 -R "span[ptile=20]"
 comsol -simplecluster -inputfile input.mph -outputfile output.mph

This is the same as:

 #BSUB -n 40 -R "span[ptile=20]"
 cat $LSB_DJOB_HOSTFILE  > hostfile.$JOB_ID
 comsol -f ./hostfile.$JOB_ID -nn 40 batch -inputfile input.mph -outputfile output.mph

Hybrid Mode

Either mode has its pros and cons. Shared mode utilizes CPU cores better than distributed mode but can only run in one node, while distributed mode can utilize more than one node. It is usually best to run a solver in a way to take advantage of both modes. This can be done easily at the command line through fine tuning of the options -nn, -nnhost, -np.

Example 4: Solving a model in hybrid mode on 2 cluster nodes with 40 cores. In this example, Comsol will spawn 2 MPI tasks in total (one on each cluster node). Each MPI task will be running with 20 threads on 20 cores.

 #BSUB -n 40 -R "span[ptile=20]"
 cat $LSB_DJOB_HOSTFILE |uniq  > hostfile.$JOB_ID
 comsol batch -f ./hostfile.$JOB_ID -nn 2 -nnhost 1 -np 20 -inputfile input.mph -outputfile output.mph

Example 5: Solving a model in hybrid mode on 2 cluster nodes with 40 cores. In this example, Comsol will spawn 4 MPI tasks in total (one on each cluster node). Each MPI task will be running with 10 threads on 10 cores.

 #BSUB -n 40 -R "span[ptile=20]"
 cat $LSB_DJOB_HOSTFILE |uniq  > hostfile.$JOB_ID
 comsol batch -f ./hostfile.$JOB_ID -nn 4 -nnhost 2 -np 10 -inputfile input.mph -outputfile output.mph

Parameter Sweep

Comsol models configured with parameter sweep can also benefit from parallel computing in different ways. A model configured with parameter sweep needs to run under a range of parameters or combinations of parameters, and each set of parameters can be calculated independently. Once a model with parameter sweep node created in Comsol GUI, it must be also configured with cluster sweep to distribute the parameters to be processes in parallel.

Example 6: Solving a parameter sweep model on 40 cores. In this example, 10 combinations of parameters will be running in concurrently on two cluster node with 5 combinations of parameters on each cluster node. Each combination of parameters will be running with 4 threads on 4 cores.

 #BSUB -n 40 -R "span[ptile=20]"
 cat $LSB_DJOB_HOSTFILE |uniq  > hostfile.$JOB_ID
 comsol -f  ./hostfile.$JOB_ID -nn 10 -nnhost 5 -np 4 -inputfile input.mph -outputfile output.mph

If each combination of parameters requires large amount of memory to solve, then we can specify one combination of parameters per node such that the entire memory on the node will be used for solving one combination of parameters.

Example 7: Solving a parameter sweep model (with 10 parameter combinations) on 200 cores with each parameter combination taking an entire cluster node.

 #BSUB -n 200 -R "span[ptile=20]"
 cat $LSB_DJOB_HOSTFILE |uniq  > hostfile.$JOB_ID
 comsol -f ./hostfile.$JOB_ID -nn 10 -nnhost 1 -np 20 -inputfile input.mph -outputfile output.mph

Common problems

1. Disk quota exceeded in home directory.

By default, comsol stores all temparary files in your home directory. For large models, are likely to get "Disk quota exceeded" error due to huge amount of temporary files dumped into y our home directory. To resolve this issue, you need to redirect temporary files to your scratch directory.

  comsole -tmpdir /scratch/user/username/cosmol/tmp -recoverydir /scratch/user/username/comsol/recovery ...