Hprc banner tamu.png

Difference between revisions of "SW:EasyBuild:Building Your Own Software:Another Example - LLVM"

Jump to: navigation, search
(Update cmake options to use the Clang compilers)
(Load the EasyBuild-cluster-SCRATCH environment)
Line 36: Line 36:
List loaded modules. e.g.
List loaded modules. e.g.
[j-perdue@login2 (20:15) tmp]$ ml
[j-perdue@login2 (20:15) tmp]$ ml
Line 43: Line 41:
Currently Loaded Modules:
Currently Loaded Modules:
   1) EasyBuild/4.3.0  2) EasyBuild-ada/0  3) EasyBuild-ada-SCRATCH/0
   1) EasyBuild/4.3.0  2) EasyBuild-ada/0  3) EasyBuild-ada-SCRATCH/0
ml # list modules
* the "EasyBuild" module sets default environment variables using the actual EasyBuild software
* the "EasyBuild" module sets default environment variables using the the lastest version of the actual EasyBuild software
* the "EasyBuild-ada" module that sets a number of environment variables for use on ada
* the "EasyBuild-ada" module that sets a number of environment variables for use on ada
* the "EasyBuild-ada-SCRATCH" module that sets/overrides environment variables for installing into the user's $SCRATCH directory (/scratch/user/<NetID>)
* the "EasyBuild-ada-SCRATCH" module that sets/overrides environment variables for installing into the user's $SCRATCH directory (/scratch/user/<NetID>)

Revision as of 20:37, 3 October 2020

This example will cover building a very recent version of LLVM using the CLang compiler. Users may first want to visit HPRC's EasyBuild page and a previous example of modifying an existing easyconfig. At some point in the process, users might want to also review the official EasyBuild documentation.


A user wants to build a modified version of the LLVM module that includes:

  1. a recent source from the LLVM Github repository
  2. adds support for CLang

LLVM is a rather long build (several hours using a single-processor). Additionally, the currently installed versions do not include CLang support. They are also rather dated (v9.01 released December 20, 2019). Newer releases include v10.0.1, released August 6, 2020. An even newer release candidate includes v11.0.0-rc5 released September 30, 2020 but as of this writing on October 3, 2020, there are already 7681 commits to master since this release. So, for this example we will use a recent (10/3/2020) .zip file of the master repo. Notes for using a bleeding edge (as of this moment) clone of the master repo are included below and are left as an exercise to the reader.

Important note on using EasyBuild to build software on terra

The instructions below are for building on any ada (and previously curie) login node. At present, on terra, the needed directories are only mounted on terra2/login2, so if you want to build for terra you will need to build there. Some things to consider:

  1. If ada2 is currently, or was recently, down (e.g. for system maintenance), the needed directories may not be available on terra2. Additionally, if terra2 was recently rebooted they may not be available. If ada2 (and terra2) is up and available for users and these directories are not yet available on terra2, please email help@hprc.tamu.edu to request they be mounted
  2. If the build requires a GPU at compile (build) time instead of just run time, you will need to use terra3/login3 (the only terra login node with a GPU). If you find your programs has this compile-time requirement, please email help@hprc.tamu.edu and someone should be able to mount the needed directories there. However, for most builds, users should use terra2/login2.

Some additional notes:

  1. Once you have built your software on terra using EasyBuild and only want to use the software you have built, the status of ada2 or terra2 is of no concern. Any software you build should a be available in your $SCRATCH directory using the 'myEB' module.
  1. At some point we hope to set up an independent | CVMFS environment for hosting, and building, all our EasyBuild software such as | the one in use up at Compute Canada. This will eliminate our dependency on any particular cluster to build/deploy software. At present, however, we do not presently have the resources to do so (they have a few more people than TAMU HPRC does and it will take a bit of planning and work to do properly here).

Converting an existing LLVM EasyBuild "easyconfig" using GCC to a much newer LLVM built using the CLang

At some point, you may want to write your own easyconfig from scratch for software not yet built on HPRC systems. That is left an exercise to the reader after they have become familiar with using EasyBuild. For now, we will stick to modifying an existing easyconfig that includes some "simple" user modifications.

Selecting an existing LLVM easyconfig

Before we get started, we'll want to find an easyconfig to convert. In this case, the user wants the most recent version of LLVM. The user also desires to use the currently HPRC-recommended 2019b toolchain which is built on GCCcore/8.3.0. See SW:Toolchains for details on toolchains currently used on HPRC clusters.

Load the EasyBuild-cluster-SCRATCH environment

First, load the "SCRATCH" configuration for using EasyBuild on the cluster you are building on. First, use "module purge" (we use the shorthand "ml purge" below where "ml" stands for "module") to unload all modules (EasyBuild prefers no other modules to be loaded to not interfere with the build).

ml purge

Use "module list" to verify no modules are loaded (here, "ml" is shorthand for "module list"). e.g.


Now load the latest EasyBuild environment to build for your $SCRATCH directory on ada (change to "ada" to "terra" if building there). ml EasyBuild-ada-SCRATCH

List loaded modules. e.g.

[j-perdue@login2 (20:15) tmp]$ ml

Currently Loaded Modules:
  1) EasyBuild/4.3.0   2) EasyBuild-ada/0   3) EasyBuild-ada-SCRATCH/0


  • the "EasyBuild" module sets default environment variables using the the lastest version of the actual EasyBuild software
  • the "EasyBuild-ada" module that sets a number of environment variables for use on ada
  • the "EasyBuild-ada-SCRATCH" module that sets/overrides environment variables for installing into the user's $SCRATCH directory (/scratch/user/<NetID>)

Search for candidate easyconfigs

To search the files currently available via $EASYBUILD_ROBOTPATH, run:

eb --search LLVM

This will provide a list of easyconfigs either provided by EasyBuild or built on the current cluster

  • the files in /sw/eb/sw/EasyBuild/<version>/easybuild/easyconfigs are the ones provided by EasyBuild in their unaltered form
  • the files in /sw/eb/ebfiles_repo_cleaned/<cluster> are the files actually used to build the software on the cluster

Usually, if a file occurs in both locations, then have the some contents. However, sometimes we've made changes to the easyconfig to adapt it to the current cluster. For example, for the Python build on ada, we uncomment the line for OpenSSL because the system openssl-devel on ada is too old for the latest Pythons.

We recommend the user start with the official EasyBuild easyconfig and then consult the local easyconfig if they run into problems.

In this case, we will use the official EasyBuild easyconfig of the latest version of the available LLVM easyconfigs using the desired toolchain, which is /sw/eb/software/EasyBuild/4.3.0/easybuild/easyconfigs/l/LLVM/LLVM-9.0.1-GCCcore-9.3.0.eb

Note: on ada, /sw/eb/sw is the same as /sw/eb/software. We (will) use /sw/eb/sw on terra and all new clusters (using "sw" is easier/better than "software", but ada was originally deployed using "software" so we are stuck with it for now)

Copy to a work directory so you can make modifications

mkdir $SCRATCH/tmp # if you don't already have such a directory
cp /sw/eb/sw/EasyBuild/4.3.0/easybuild/easyconfigs/l/LLVM/LLVM-9.0.1-GCCcore-9.3.0.eb $SCRATCH/tmp # note we use the shorthand "sw" here
cd $SCRATCH/tmp # change to work directory so we can make changes
ls -al LLVM* # verify the file was copied

Making the desired changes to the easyconfig

Begin modifications

First, rename the file to reflect the version we plan to use.

mv LLVM-9.0.1-GCCcore-8.3.0.eb LLVM-master-GCCcore-8.3.0.eb

Then change the file permissions so that we have write permission.

chmod 600 LLVM-master-GCCcore-8.3.0.eb

Now edit the file using your favorite editor. "vi" is guaranteed to be on every Unix system and is highly recommended but involves a bit of a learning curve if you aren't familiar with it. "gedit" provides a nice GUI editor that should be usable by most users but does require you enabled X11 when you connected. "nano" is a very friendly text-based editor popular with many users. "emacs" is an overly complicated editor with lots of bells and whistles not needed by even the most seasoned user but is available via the module system. (Note: religious wars over "vi vs. emacs" have waged for decades and will likely continue as long as Unix exist. Yours truly (the author of this page) is a long time (30+ years) "vi" user so is obviously biased towards using "vi".)

Begin editing the file with:

vi LLVM-master-GCCcore-8.3.0.eb # replace "vi" as desired

Update source specifications

Change source_urls (line 16) to the GitHub directory containing the latest zip file e.g.

source_urls = ['https://github.com/llvm/llvm-project/archive']

Change the name of the source to match the name provided at the link above

sources = ['master.zip']

Delete the "checksums" line (line 19). The checksum will change whenever a new master is downloaded and isn't needed or used for this build (it is typically used to verify official releases).

Add the following line just after the "sources" so cmake knows where to find the CMakeLists.txt file defining the build options.

srcdir = '%(namelower)s'

("%(namelower)s" is the lowercase version of the "name" defined at the top of the file [in this case, "llvm"])

Update the cmake build type

If you want a CMAKE_BUILD_TYPE=RelWithDebInfo build, add the following just after the "build_shared_libs" line

build_type = 'RelWithDebInfo'

If you instead want a CMAKE_BUILD_TYPE=Debug build (default is Release), change line 15 from:

toolchainopts = {'cstd': 'gnu++11'}


toolchainopts = {'cstd': 'gnu++11', 'debug': True}

Note: the cstd option indicates we will use the 2011 version of the C++ standard with GNU gcc extensions.

Save your file and exit

Make changes to support using Clang to build

Find an appropriate Clang module

Use the local "mla" script to find a Clang version that matches the toolchain we are using. ("mla" is a TAMU HPRC script which runs "module avail" and stores a cached list so that subsequent runs are MUCH faster than "module avail").

[j-perdue@login2 (19:55) tmp]$ mla | grep -i clang | grep GCCcore-8.3.0

Add the Clang module to the list of dependencies

Add the Clang version to use (we'll use the latest available) to the list of dependencies. e.g.

dependencies = [
    ('Clang', '10.0.0'),
    ('ncurses', '6.1'),
    ('zlib', '1.2.11'),

Update cmake options to use the Clang compilers

Update the options that will be used with "cmake" to use the Clang compilers by adding the following line after the "build_type" line.

configopts = '-DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++'

Update to build the LLVM projects desired

configopts += ' -DLLVM_ENABLE_PROJECTS="clang;libcxx;libcxxabi;openmp"'

Note: the leading space in the quoted string is important since this will be appended to the previous "configopts" line

The resulting easyconfig

Building the new easyconfig