View source for SW:EasyBuild:Building Your Own Software:Another Example - LLVM

This example will cover building a very recent version of LLVM with Clang support using a previously built Clang frontend. Users may first want to visit [[SW:EasyBuild | HPRC's EasyBuild]] page and [[SW:EasyBuild:Building_Your_Own_Software_With_EasyBuild | a previous example of modifying an existing easyconfig]].  At some point in the process, users might want to also review [https://easybuild.readthedocs.io/en/latest/ the official EasyBuild documentation].

= Motivation = 
A user wants to build a modified version of the LLVM module that includes:
# a recent source from the LLVM Github repository
# adds support for the Clang frontend
LLVM is a rather long build (several hours using a single-processor).  Additionally, the currently installed versions do not include Clang support.  They are also rather dated (v9.01 released December 20, 2019).  Newer releases include v10.0.1, released August 6, 2020. An even newer release candidate includes v11.0.0-rc5 released September 30, 2020 but as of this writing on October 3, 2020, there are already 7681 commits to master since this release.  So, for this example we will use a recent (10/3/2020) .zip file of the master repo. Notes for using a bleeding edge (as of this moment) clone of the master repo are included below and are left as an exercise to the reader.

= Important note on using EasyBuild to build software on terra =
The instructions below are for building on any ada (and previously curie) login node.  At present, on terra, the needed directories are only mounted on terra2/login2, so if you want to build for terra you will need to build there.  Some things to consider:
# If ada2 is currently, or was recently, down (e.g. for system maintenance), the needed directories may not be available on terra2. Additionally, if terra2 was recently rebooted they may not be available. If ada2 (and terra2) is up and available for users and these directories are not yet available on terra2, please email help@hprc.tamu.edu to request they be mounted
# If the build requires a GPU at compile (build) time instead of just run time, you will need to use terra3/login3 (the only terra login node with a GPU).  If you find your programs has this compile-time requirement, please email help@hprc.tamu.edu and someone should be able to mount the needed directories there.  However, for most builds, users should use terra2/login2.

Some additional notes:
# Once you have built your software on terra using EasyBuild and only want to use the software you have built, the status of ada2 or terra2 is of no concern.  Any software you build should a be available in your $SCRATCH directory using the 'myEB' module.

# At some point we hope to set up an independent [https://cernvm.cern.ch/fs/ | CVMFS] environment for hosting, and building, all our EasyBuild software such as [https://docs.computecanada.ca/wiki/Accessing_CVMFS | the one in use up at Compute Canada]. This will eliminate our dependency on any particular cluster to build/deploy software.  At present, however, we do not presently have the resources to do so (they have a few more people than TAMU HPRC does and it will take a bit of planning and work to do properly here).

= Converting an existing LLVM EasyBuild "easyconfig" using GCC to a much newer LLVM built using the Clang =
At some point, you may want to write your own easyconfig from scratch for software not yet built on HPRC systems.  That is left an exercise to the reader after they have become familiar with using EasyBuild.  For now, we will stick to modifying an existing easyconfig that includes some "simple" user modifications.
== Selecting an existing LLVM easyconfig ==
Before we get started, we'll want to find an easyconfig to convert. In this case, the user wants the most recent version of LLVM.  The user also desires to use the currently HPRC-recommended 2019b toolchain which is built on GCCcore/8.3.0.  See [[SW:Toolchains]] for details on toolchains currently used on HPRC clusters.


=== Load the EasyBuild-cluster-SCRATCH environment ===
First, load the "SCRATCH" configuration for using EasyBuild on the cluster you are building on.  First, use "module purge" (we use the shorthand "ml purge" below where "ml" stands for "module") to unload all modules (EasyBuild prefers no other modules to be loaded to not interfere with the build).
<pre>
ml purge
</pre>
Use "module list" to verify no modules are loaded (here, "ml" is shorthand for "module list"). e.g.
<pre>
ml
</pre>
Now load the latest EasyBuild environment to build for your $SCRATCH directory on ada (change to "ada" to "terra" if building there).
<pre>
ml EasyBuild-ada-SCRATCH
</pre>
(here "ml" is shorthand for "module load". The point being that "ml" can mean different things depending upon how its used).

Now, list loaded modules. e.g.
<pre>
[j-perdue@login2 (20:15) tmp]$ ml

Currently Loaded Modules:
  1) EasyBuild/4.3.0   2) EasyBuild-ada/0   3) EasyBuild-ada-SCRATCH/0
</pre>
where:
* the "EasyBuild" module sets default environment variables using the the lastest version of the actual EasyBuild software
* the "EasyBuild-ada" module that sets a number of environment variables for use on ada
* the "EasyBuild-ada-SCRATCH" module that sets/overrides environment variables for installing into the user's $SCRATCH directory (/scratch/user/<NetID>)

=== Search for candidate easyconfigs ===
To search the files currently available via $EASYBUILD_ROBOTPATH, run:
<pre>
eb --search LLVM
</pre>
This will provide a list of easyconfigs either provided by EasyBuild or built on the current cluster
* the files in /sw/eb/sw/EasyBuild/<version>/easybuild/easyconfigs are the ones provided by EasyBuild in their unaltered form
* the files in /sw/eb/ebfiles_repo_cleaned/<cluster> are the files actually used to build the software on the cluster
Usually, if a file occurs in both locations, then have the some contents.  However, sometimes we've made changes to the easyconfig to adapt it to the current cluster.  For example, for the Python build on ada, we uncomment the line for OpenSSL because the system openssl-devel on ada is too old for the latest Pythons.

We recommend the user start with the official EasyBuild easyconfig and then consult the local easyconfig if they run into problems.

In this case, we will use the official EasyBuild easyconfig of the latest version of the available LLVM easyconfigs using the desired toolchain, which is /sw/eb/software/EasyBuild/4.3.0/easybuild/easyconfigs/l/LLVM/LLVM-9.0.1-GCCcore-9.3.0.eb

Note: on ada, /sw/eb/sw is the same as /sw/eb/software.  We (will) use /sw/eb/sw on terra and all new clusters (using "sw" is easier/better than "software", but ada was originally deployed using "software" so we are stuck with it for now)

=== Copy to a work directory so you can make modifications ===
<pre>
mkdir $SCRATCH/tmp # if you don't already have such a directory
cp /sw/eb/sw/EasyBuild/4.3.0/easybuild/easyconfigs/l/LLVM/LLVM-9.0.1-GCCcore-9.3.0.eb $SCRATCH/tmp # note we use the shorthand "sw" here
cd $SCRATCH/tmp # change to work directory so we can make changes
ls -al LLVM* # verify the file was copied
</pre>

== Making the desired changes to the easyconfig ==
=== Begin modifications ===
First, rename the file to reflect the version we plan to use.
<pre>
mv LLVM-9.0.1-GCCcore-8.3.0.eb LLVM-master-GCCcore-8.3.0.eb
</pre>
Then change the file permissions so that we have write permission.
<pre>
chmod 600 LLVM-master-GCCcore-8.3.0.eb
</pre>

Now edit the file using your favorite editor.  "vi" is guaranteed to be on every Unix system and is highly recommended but involves a  bit of a learning curve if you aren't familiar with it.  "gedit" provides a nice GUI editor that should be usable by most users but does require you enabled X11 when you connected.  "nano" is a very friendly text-based editor popular with many users.  "emacs" is an overly complicated editor with lots of bells and whistles not needed by even the  most seasoned user but is available via the module system. (Note: religious wars over "vi vs. emacs" have waged for decades and will likely continue as long as Unix exist.  Yours truly (the author of this page) is a long time (30+ years) "vi" user so is obviously biased towards using "vi".)

Begin editing the file with:
<pre>
vi LLVM-master-GCCcore-8.3.0.eb # replace "vi" as desired
</pre>

=== Update source specifications ===
Change source_urls (line 16) to the GitHub directory containing the latest zip file e.g.
<pre>
source_urls = ['https://github.com/llvm/llvm-project/archive']
</pre>
Change the name of the source to match the name provided at the link above
<pre>
sources = ['master.zip']
</pre>
Delete the "checksums" line (line 19).  The checksum will change whenever a new master is downloaded and isn't needed or used for this build (it is typically used to verify official releases).

Add the following line just after the "sources" so cmake knows where to find the CMakeLists.txt file defining the build options.
<pre>
srcdir = '%(namelower)s'
</pre>
("%(namelower)s" is the lowercase version of the "name" defined at the top of the file [in this case, "llvm"])

=== Update the cmake build type ===
If you want a CMAKE_BUILD_TYPE=RelWithDebInfo build, add the following just after the "build_shared_libs" line
<pre>
build_type = 'RelWithDebInfo'
</pre>
If you instead want a CMAKE_BUILD_TYPE=Debug build (default is Release), change line 15 from:
<pre>
toolchainopts = {'cstd': 'gnu++11'}
</pre>
to:
<pre>
toolchainopts = {'cstd': 'gnu++11', 'debug': True}
</pre>
Note: the cstd option indicates we will use the 2011 version of the C++ standard with GNU gcc extensions. 

Save your file and exit

=== Make changes to support using Clang to build ===
==== Find an appropriate Clang module ====
Use the local "mla" script to find a Clang version that matches the toolchain we are using. ("mla" is a TAMU HPRC script which runs "module avail" and stores a cached list so that subsequent runs are MUCH faster than "module avail").
<pre>
[j-perdue@login2 (19:55) tmp]$ mla | grep -i clang | grep GCCcore-8.3.0
Clang/9.0.1-GCCcore-8.3.0
Clang/10.0.0-GCCcore-8.3.0
</pre>
==== Add the Clang module to the list of dependencies ====
Add the Clang version to use (we'll use the latest available) to the list of dependencies. e.g. 
<pre>
dependencies = [
    ('Clang', '10.0.0'),
    ('ncurses', '6.1'),
    ('zlib', '1.2.11'),
]
</pre>
=== Update cmake options to use the Clang compilers ===
Update the options that will be used with "cmake" to use the Clang compilers by adding the following line after the "build_type" line.
<pre>
configopts = '-DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++'
</pre>

=== Update to build the LLVM projects desired ===
<pre>
configopts += ' -DLLVM_ENABLE_PROJECTS="clang;libcxx;libcxxabi;openmp"'
</pre>
Note: the leading space in the quoted string is important since this will be appended to the previous "configopts" line

== The resulting easyconfig ==
<pre>
name = 'LLVM'
version = 'master'

homepage = "https://llvm.org/"
description = """The LLVM Core libraries provide a modern source- and target-independent
 optimizer, along with code generation support for many popular CPUs
 (as well as some less common ones!) These libraries are built around a well
 specified code representation known as the LLVM intermediate representation
 ("LLVM IR"). The LLVM Core libraries are well documented, and it is
 particularly easy to invent your own language (or port an existing compiler)
 to use LLVM as an optimizer and code generator."""

toolchain = {'name': 'GCCcore', 'version': '8.3.0'}
#toolchainopts = {'cstd': 'gnu++11', 'debug': True}
toolchainopts = {'debug': True}

source_urls = ['https://github.com/llvm/llvm-project/archive']
sources = ['master.zip']
srcdir = '%(namelower)s'

builddependencies = [
    ('binutils', '2.32'),
    ('CMake', '3.15.3'),
    ('Python', '3.7.4'),
]

dependencies = [
    ('Clang', '10.0.0'), 
    ('ncurses', '6.1'),
    ('zlib', '1.2.11'),
]

build_shared_libs = True
build_type = 'RelWithDebInfo' # this overrides debug=True above

configopts = '-DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++'
configopts += ' -DLLVM_ENABLE_PROJECTS="clang;libcxx;libcxxabi;openmp"' # leading space is important

sanity_check_paths = {
    'files': ['bin/llvm-ar', 'bin/FileCheck'],
    'dirs': ['include/llvm', 'include/llvm-c'],
}

moduleclass = 'compiler'
</pre>

= Building the new easyconfig =
= Notes =