SW:EasyBuild:Building Your Own Software With EasyBuild

This article is meant to provide a step-by-step guide on how to compile your own software on the TAMU HPRC clusters using EasyBuild (EB). This is particularly useful if the software/version of software you're interested in does not already exist on the clusters. For more in-depth information about EasyBuild, please see the official EasyBuild documentation.

Prepare Your Environment

Ada [WIP]

You must load the EasyBuild module. This can be done simply by running the following command:

ml EasyBuild-ada-SCRATCH/

This will set up the necessary directories in your $SCRATCH folder and the environment variables for the software build and installation.

Terra [WIP]

You must be logged onto terra2 to access the EB build directories. You can check whether you are on the correct node or not based on the primary prompt string:

[netid@terra2 netid]$

If your primary prompt string does not say you're logged into terra2, you can switch to terra2from your current login node by using SSH:

ssh netid@terra2

Next, you must load the EasyBuild module. This can be done simply by running the following command:

ml EasyBuild-terra-SCRATCH/

This will set up the necessary directories in your $SCRATCH folder and the environment variables for the software build and installation.

Find Your Software Configuration File

The EasyBuild community has created a series of configuration files (EasyConfigs) spanning a variety of software and versions. Simply navigate to the EasyBuild EasyConfigs repository on GitHub and find the EasyConfig you need for your software. Once you've located it, click the Raw button to access the file in your browser.

Copy the URL of the page, and use the following command to download it directly onto the cluster:

wget url-you-just-copied

Build Your Software

Before starting the build, we advise you to edit the description of the EasyConfig file as a best practice. This will allow you to distinguish the module you've built from the ones already on the cluster (if they are identical):

### EXAMPLE ###

...

easyblock = 'ConfigureMake'

name = 'libyaml'
version = '0.2.2'

homepage = 'https://pyyaml.org/wiki/LibYAML'

description = """LibYAML is a YAML parser and emitter written in C. Built by username"""   # this line is where you can update the description

toolchain = {'name': 'GCCcore', 'version': '8.3.0'}

...

After saving your changes to the EasyConfig, just run it using the following command:

eb name-of-your-easyconfig.eb

Using Your Software

To use your newly built software, run the following command:

ml myEB

This will update your module path to include the directories containing your software installations. From here, you would just load the software module like any other module on the clusters:

### EXAMPLE ###
# note that the module name is just the name of the file with the first hyphen (-) replaced with a forward slash (/)
# also, the file extension is removed
# e.g. libyaml-0.2.2-GCCcore-8.3.0.eb becomes libyaml/0.2.2-GCCcore-8.3.0

ml libyaml/0.2.2-GCCcore-8.3.0

Build Example: Ada

Lines with a dollar sign ($) in front denote commands to be input in the shell prompt. Lines without demonstrate command output.

$ cd $SCRATCH
$ echo $MODULEPATH
    /sw/hprc/mods/all:/sw/eb/mods/all:/sw/lmod/8.4/modulefiles/Core
$ wget https://raw.githubusercontent.com/easybuilders/easybuild-easyconfigs/develop/easybuild/easyconfigs/l/libyaml/libyaml-0.2.2-GCCcore-8.3.0.eb

Build Example: Terra

Lines with a dollar sign ($) in front denote commands to be input in the shell prompt. Lines without demonstrate command output.

$ cd $SCRATCH
$ echo $MODULEPATH
    /sw/hprc/mods/all:/sw/eb/mods/all:/sw/lmod/8.4/modulefiles/Core
$ wget https://raw.githubusercontent.com/easybuilders/easybuild-easyconfigs/develop/easybuild/easyconfigs/l/libyaml/libyaml-0.2.2-GCCcore-8.3.0.eb
$ ml EasyBuild-terra-SCRATCH/
$ eb -f libyaml-0.2.2-GCCcore-8.3.0.eb
    == temporary log file in case of crash /scratch/user/$USER/eb/tmp/logs/easybuild-ZVslcV.log
    
    WARNING: Deprecated functionality, will no longer work in v5.0: Running EasyBuild with Python 2.6 is deprecated; see http://easybuild.readthedocs.org/en/latest/Deprecated-functionality.html for more information
    
    
    WARNING: Deprecated functionality, will no longer work in v5.0: Use of 'dummy' toolchain is deprecated, use 'system' toolchain instead; see http://easybuild.readthedocs.org/en/latest/Deprecated-functionality.html for more information
    
    
    WARNING: Use of 1 unknown easyconfig parameters detected in GCCcore-8.3.0.eb: mpfr_version
    If these are just local variables please rename them to start with 'local_', or try using --fix-deprecated-easyconfigs to do this automatically.
    For more information, see https://easybuild.readthedocs.io/en/latest/Easyconfig-files-local-variables.html .
    
    
    WARNING: Use of 1 unknown easyconfig parameters detected in GCCcore-8.3.0.eb: mpfr_version
    If these are just local variables please rename them to start with 'local_', or try using --fix-deprecated-easyconfigs to do this automatically.
    For more information, see https://easybuild.readthedocs.io/en/latest/Easyconfig-files-local-variables.html .
    
    == resolving dependencies ...
    == processing EasyBuild easyconfig /scratch/user/$USER/libyaml-0.2.2-GCCcore-8.3.0.eb
    == building and installing libyaml/0.2.2-GCCcore-8.3.0...
    == fetching files...
    == creating build dir, resetting environment...
    == unpacking...
    == patching...
    == preparing...
    == configuring...
    == building...
    == testing...
    == installing...
    == taking care of extensions...
    == restore after iterating...
    == postprocessing...
    == sanity checking...
    == cleaning up...
    == creating module...
    == permissions...
    == packaging...
    == COMPLETED: Installation ended successfully (took 1 min 7 sec)
    == Results of the build can be found in the log file(s) /scratch/user/$USER/eb/x86_64/sw/libyaml/0.2.2-GCCcore-8.3.0/easybuild/easybuild-libyaml-0.2.2-20200930.121751.log
    == Build succeeded for 1 out of 1
    == Temporary log file(s) /scratch/user/$USER/eb/tmp/logs/easybuild-ZVslcV.log* have been removed.
    == Temporary directory /scratch/user/$USER/eb/tmp/eb-LtQefN has been removed.
$ ml myEB
$ echo $MODULEPATH
    /scratch/user/$USER/eb/x86_64/mods/all:/sw/hprc/mods/all:/sw/eb/mods/all:/sw/lmod/8.4/modulefiles/Core
$ ml libyaml/0.2.2-GCCcore-8.3.0
$ ml show libyaml/0.2.2-GCCcore-8.3.0
    --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
       /scratch/user/$USER/eb/x86_64/mods/all/libyaml/0.2.2-GCCcore-8.3.0.lua:
    --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    help([[
    Description
    ===========
    LibYAML is a YAML parser and emitter written in C. Built by $USER.
    
    
    More information
    ================
     - Homepage: https://pyyaml.org/wiki/LibYAML
    ]])
    whatis("Description: LibYAML is a YAML parser and emitter written in C. Built by $USER.")
    whatis("Homepage: https://pyyaml.org/wiki/LibYAML")
    whatis("URL: https://pyyaml.org/wiki/LibYAML")
    conflict("libyaml")
    prepend_path("CMAKE_PREFIX_PATH","/scratch/user/$USER/eb/x86_64/sw/libyaml/0.2.2-GCCcore-8.3.0")
    prepend_path("CPATH","/scratch/user/$USER/eb/x86_64/sw/libyaml/0.2.2-GCCcore-8.3.0/include")
    prepend_path("LD_LIBRARY_PATH","/scratch/user/$USER/eb/x86_64/sw/libyaml/0.2.2-GCCcore-8.3.0/lib")
    prepend_path("LIBRARY_PATH","/scratch/user/$USER/eb/x86_64/sw/libyaml/0.2.2-GCCcore-8.3.0/lib")
    prepend_path("PKG_CONFIG_PATH","/scratch/user/$USER/eb/x86_64/sw/libyaml/0.2.2-GCCcore-8.3.0/lib/pkgconfig")
    setenv("EBROOTLIBYAML","/scratch/user/$USER/eb/x86_64/sw/libyaml/0.2.2-GCCcore-8.3.0")
    setenv("EBVERSIONLIBYAML","0.2.2")
    setenv("EBDEVELLIBYAML","/scratch/user/$USER/eb/x86_64/sw/libyaml/0.2.2-GCCcore-8.3.0/easybuild/libyaml-0.2.2-GCCcore-8.3.0-easybuild-devel")