OpenFOAM
OpenFOAM is the free, open source CFD software first released by OpenCFD Ltd in 2004. It has a large user base across most areas of engineering and science, from both commercial and academic organisations. OpenFOAM has an extensive range of features to solve anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to acoustics, solid mechanics and electromagnetism.
The FOAM in OpenFOAM stands for Field Operation And Manipulation, since the software is used to calculate velocity, pressure, density, temperature, concentration, etc. fields. OpenFOAM includes more than 100 readily available solvers and many pre- and post-processing utilities. For example, simpleFoam, sonicFoam and rhoReactingFoam are three popular included solvers, and blockMesh and snappyHexMesh are two mesh generation tools.
There are two major branches of OpenFOAM development:
- An ESI-OpenCFD branch developed by OpenFOAM trademark owners OpenCFD Limited which is based at www.openfoam.com. These versions are named using the form
OpenFOAM-vYYMMwhere YY and MM are, respectively, the year and month of the released version. The most recent EasyBuild version as of 2023/04 isOpenFOAM-v2206 - An OpenFOAM Foundation branch started in 2011 that uses the OpenFOAM trademark with permission, and which is based at www.openfoam.org. These versions are named with the form
Openfoam-Nwhere N is the release number. The most recent EasyBuild version as of 2023/04 isOpenFoam-11.
There is also a foam-extend branch that attempts to integrate community contributions to OpenFOAM. It is based at sourceforge.net/projects/foam-extend/ and is also supported by EasyBuild, although the latest available version is three years old.
The history of and details about all releases can be found at the Unofficial OpenFOAM Wiki.
The branch with which you will work can be based on a recommendation, a need to replicate previous work, the need for a feature contained within only one branch, or some other factor. Or you can try both branches.
Environment Set-Up
OpenFOAM is available on the Grace, FASTER, and ACES clusters.
Modules
⚠️ It's recommended to load a versioned module versus the latest available (e.g. OpenFOAM/v1912 instead of just OpenFOAM) to ensure compatibility with your workload, reproducibility, and record-keeping
Grace
View available versions of OpenFOAM:
Select a version, and pass it the the "module spider" command to see any prerequisite modules required:
This particular version allows two different sets of prerequisite modules, only one set needs to be loaded:
Load the OpenFOAM module to configure your shell environment for access to OpenFOAM:
module purge # ensure your working environment is clean
module load GCC/8.3.0 OpenMPI/3.1.4 # load prerequisite modules
module load OpenFOAM/v1912 # load OpenFOAM/v1912
Setting Up OpenFOAM Variables
Once the OpenFOAM module is loaded, the environment variables needed to use OpenFOAM can be sourced:
The shell is now ready for use with OpenFOAM.
Usage
Interactive
This method is recommended for quick testing that '''require only a small amount of computational resources''', as this method takes place on the login nodes. Please see our policy regarding login node usage for additional information.
Load the modules for the appropriate cluster:
# Grace
module purge # ensure your working environment is clean
module load GCC/8.3.0 OpenMPI/3.1.4 # load prerequisite modules
module load OpenFOAM/v1912 # load OpenFOAM/v1912
Source the environment script:
Copy the tutorial and update file permissions:
mkdir -p ${SCRATCH}/openfoam-examples
cp -R ${FOAM_TUTORIALS}/basic/laplacianFoam/flange ${SCRATCH}/openfoam-examples
chmod -R +w ${SCRATCH}/openfoam-examples
Change into the tutorial directory:
Run the tutorial:
Batch/Job Submission
⚠️ Please note. OpenFOAM jobs can cause an unnecessary use of storage capacity if not configured carefully. For large jobs, this can lead to excessive loads on the file system, causing slowdowns for other users. But it can also unnecessarily fill up your own quota. Be mindful of the number of files created, as well as the total size of the files, as both put a burden on the file system and your own quota.
Please review the data management section below before submitting OpenFOAM jobs, and before asking for extensions of your storage quota. After reviewing, you should be able to answer the following questions:
- Are my output controls appropriate for the data I need?
- Is my purgeWrite setting appropriate for the data I need?
- Could I make use of function objects to improve the efficiency of my workflow?
- What data can I clean up after the simulation finishes successfully?
Example 1: General Usage
This method is recommend for workloads that '''require a large amount of computational resources''', and takes place on the compute nodes by accessing the job submission system.
Grace
#!/bin/bash
#SBATCH -J openfoam-sample-1-grace # set the job name to "openfoam-sample1-grace"
#SBATCH -t 1:00:00 # set the wall clock limit to 1hr
#SBATCH -N 1 # request 1 node
#SBATCH --ntasks-per-node=8 # request 8 tasks per node
#SBATCH --mem=16G # request 16G per node
#SBATCH -o %x.%j # send stdout/err to "moose-sample-1-grace.[jobID]"
# environment set-up
module purge # ensure your working environment is clean
module load GCC/8.3.0 OpenMPI/3.1.4 # load prerequisite modules
module load OpenFOAM/v1912 # load OpenFOAM/v1912
source ${FOAM_BASH} # set up OpenFOAM environment variables
# run OpenFOAM example 1
mkdir -p ${SCRATCH}/openfoam-examples # create a destination directory for a copy of the tutorial
cp -R ${FOAM_TUTORIALS}/basic/laplacianFoam/flange ${SCRATCH}/openfoam-examples # copy the tutorial to your newly created directory
chmod -R +w ${SCRATCH}/openfoam-examples # add write permissions to the copied tutorial
cd ${SCRATCH}/openfoam-examples/flange # navigate into the tutorial directory
./Allrun # run the tutorial
Data management with OpenFOAM jobs
OpenFOAM saves the field variables (velocity, pressure, etc.) at intervals that you specify. For steady-state simulations, only the final converged fields are usually needed. For unsteady simulations, you may need to save the fields at certain intervals to create animations etc. But bear in mind that, depending on what you want to animate/analyse, an OpenFOAM function object to extract that particular information during runtime is much more efficient than saving all fields. By default, OpenFOAM tutorials tend to save way too much data and easily become unworkable when scaling to large amounts of processors.
How to reduce the storage requirement of OpenFOAM jobs
Only save field data when necessary
Field data is usually the most space consuming, and if running in parallel, creating the most files. For example, if you have a simulation with 7 field variables (e.g. U,p,phi,k,omega,nut,T), you decompose the domain into 128 processor directories, and you write every 0.1s for a 10s simulation, that generates 7*128*10/0.1=89600 files. If you only needed the final time step, that would only be 896 files, and if you reconstructed the domain after completion, that would only leave 7 files, a reduction by 99.99% with the same useful data being kept. OpenFOAM tutorials, by default, usually do not control the output in such an efficient manner.
The main ways to control this is by:
- Modifying the outputControl/writeControl options in system/controlDict
- Using the settings of outputControl/writeControl, you can ask the solver to write field data either every N time steps (e.g. save every 1000 time steps), or specify specific simulation time intervals (e.g. save every 10s of simulation time).
- Modifying the purgeWrite setting in system/controlDict
- purgeWrite deletes old time steps as the simulation progresses. For example, purgeWrite 10 means the 10 last time steps will be kept. purgeWrite 0 means that ALL time steps will be kept (no purging). Oftentimes, purgeWrite 1 is appropriate as it saves only the latest time step. Bear in mind that purgeWrite will not delete time steps from before a restart. So if you are restarting from e.g. 10s, and running to 20s, two times (10s and 20s) will be saved.
Use function objects
Function objects are a great way to reduce storage requirements, and improve the efficiency of your workflow. With function objects, you can specify reduced data (velocity in a set of points, an isosurface of a threshold, velocity contours in a plane, surface pressure contours etc.) to be saved at frequent intervals without saving the entire flow field.
Clean the case after finishing
Even with careful control of storage needs during a simulation, it is worth going over every case after it finishes to determine what data needs to be saved for the future. If you make a habit of doing this immediately, it avoids going back over the case months later trying to remember what needs to be saved. You can check what time steps are currently saved by browsing the contents of the processor0 folder (or the root case folder if not running in parallel).
Please consider if:
- You used some restart points that no longer need to be saved.
- There are other time steps saved that you don't need
- You can use reconstructPar to reconstruct the time steps needed
- You can tarball the entire case for now since you may need it in the future but you are not sure. That reduces it from 1000's of files to a single one.
Use collated file format
Most OpenFOAM versions allow for the use of the collated file format. This does not reduce the total size of the files created, but it does reduce the number of files created. Therefore, using collation should not be seen as a substitute for good data management as outlined in the previous sections. The use of file collation is not default in OpenFOAM and there is no clear documentation how it affects I/O performance. To read more about collation in OpenFOAM, see: https://openfoam.org/guides/parallel-io/