Hprc banner tamu.png


Revision as of 07:54, 20 September 2018 by J-perdue (talk | contribs) (Others)
Jump to: navigation, search



A toolchain on a TAMU HPRC cluster is a collection of tools used for building software. They typically include:

  • a compiler collection providing basic language support (C/Fortran/C++)
  • a MPI implementation for multi-node communication
  • a set of linear algebra libraries (FFTW/BLAS/LAPACK) for accelerated math

Most modules on our clusters include the toolchain that was used to build them in their version tag (e.g. .Python/3.5.2-intel-2016b was built with the intel/2016b toolchain below).

Mixing components from different toolchains almost always leads to problems. For example, if you mix Intel MPI modules with OpenMPI modules you can guarantee your program will not run (even if you manage to get it to compile). We recommend you always use modules from the same (sub)toolchains. [Since late 2016 we've been looking at EasyBuild's --minimal-toolchain option to cut down on module count, so "GCCcore" is now a common new "subtoolchain"]

Currently Supported

Toolchain Compiler(s) MPI Linear Algebra Available on:
binutils GCC(core) Intel Compilers
Intel MPI
OpenMPI Intel MKL
LAPACK OpenBLAS ScaLAPACK FFTW ada curie terra
intel Composed of the Intel Cluster Toolkit built upon a particular version of GCC. These are the recommended toolchains for HPRC Intel-based clusters.
2017b 2.28 6.4.0 2017.4.196 2017.3.196 - 2017.3.196 - - - - X - X
2017A 2.2x (system) 6.3.0 2017.1.132 2017.1.132 - 2017.1.132 - - - - X - X
2016b 2.26 5.4.0 2016.3.210 - - - - - X - X
2016a 2.25 4.9.3 2016.1.150 - - - - - X - X
2015B 2.2x (system) 4.8.4 2015.3.187 - - - - - X - -
iomkl A combination of the Intel compiler and math kernel library components with OpenMPI.
2017b 2.28 6.4.0 2017.4.196 - 2.1.1 2017.3.196 - - - - X - X
2017A 2.2x (system) 6.3.0 2017.1.132 - 2.0.2 2017.1.132 - - - - X - X
2016.07 2.26 5.4.0 2016.3.210 - 1.10.3 - - - - X - -
2015B 2.2x (system) 4.8.4 2015.3.187 - 1.8.8 - - - - X - -
foss Based upon "Free and Open-Source Software". These toolchains produce slightly slower code but do provide code portability to other platforms (e.g. those that don't have an Intel compiler).
2017b 2.28 6.4.0 - - 2.1.1 - 3.7.0 0.2.20 2.0.2 3.3.6 X X X
2017A 2.2x (system) 6.3.0 - - 2.0.2 - 3.7.0 0.2.19 2.0.2 3.3.6 X X X
2016b 2.26 5.4.0 - - 1.10.3 - 3.6.1 0.2.18 2.0.2 3.3.4 X X X
2016a 2.25 4.9.3 - - 1.10.2 - 3.6.0 0.2.15 2.0.2 3.3.4 - X -
2015b 2.25 4.9.3 - - 1.8.8 - 3.5.0 0.2.14 2.0.2 3.3.4 - X -
2015a 2.2x (system) 4.9.2 - - 1.8.4 - 3.5.0 0.2.13 2.0.2 3.3.4 - X -
  1. Two new/additional toolchains, goolfc and fosscuda are essentially the foss toolchain with OpenMPI compiled with support for CUDA (9.0.176. for 2017b).
  2. For details on using Intel MKL's BLAS and ScaLAPACK (and at some point FFTW), see our current Intel MKL page.
  3. Note: OpenMPI 1.10.x is largely incompatibale with all previous and later versions and should probably be avoided.

Future (currently unsupported)


  • binutils/2.28
  • CUDA/9.1.85
  • FFTW/3.3.7
  • GCC/6.4.0
  • icc/2018.1.163
  • ifort/2018.1.163
  • imkl/2018.1.163
  • impi/2018.1.163
  • OpenBLAS/0.2.20
  • OpenMPI/2.1.2
  • ScaLAPACK/2.0.2


  • binutils/2.30
  • CUDA/9.2.88
  • FFTW/3.3.8
  • GCC/7.3.0
  • icc/2016.3.210
  • ifort/2016.3.210
  • imkl/
  • impi/
  • OpenBLAS/0.3.1
  • OpenMPI/3.1.1
  • ScaLAPACK/2.0.2

Our recommended toolchains

Last updated: September 3, 2018

  • In general, we recommend the intel toolchain with Intel compilers, MPI and linear algebra (MKL) for the best performance on HPRC clusters.
  • In many cases, using icc instead of gcc takes a lot of effort so you may find things easier with the foss toolchain.
  • Versions:
    • 2017A - these currently have the richest collection of libraries/tools/modules
    • 2017b - these contain updated versions, and address a deficiency of the 2017A toolchain (see below). Additionally, we've tried to learn from our experiences with 2017A. This is not as rich in terms of modules, but may have the ones you need. This toolchain is recommended if you can find all you need in 2017b, othwerwise you might be bettere off with 2017A.

A breakdown of our 2017A toolchains

For the past few years, TAMU HPRC has been using EasyBuild to deploy commonly used HPC libraries. In the past we've tried to use what EasyBuild contributors (who have HPC systems of their own) use/deploy as a guide for software deployment. For 2017A, we've set off on a new path.

With the recent addition of the --minimal-toolchain option, which allows us to minimize the overall module count and share common modules among many toolchains, we've tried to trim things down. Additionally, in an attempt to keep as closely aligned with the the Linux distribution in use, we've deferred to many of the distribution-provided build tools like autoconf/automake/binutils/cmake/make/libtool/pkgconfig/etc.

As such, the modules in the 2017A toolchain aren't as easy to determine what works with what.

Components by version suffix

  • -GCCcore-6.3.0 - these were buit with gcc 6.3.0 using the system binutils (which is a deviation from how EasyBuild does it but we thought best to try here).
  • -GCCcore-6.3.0-Python-2.7.12-bare - these were built with the most basic of Python (only what was required) but do not include a proper Python module. If you use these, you must load a Python based on a full toolchain (intel/iomkl/foss). IF you don't load a full Python module and attempt to use these with the system python, THEN it will likely fail.
  • -iimpi/iompi/gompi-2017A - these are packages that required MPI, but didn't necessarily require linear algebra packages like MKL or BLAS/FFTW/LAPACK. These are useful if you want to, for example, use the Intel compilers and MPI but want to use the non-MKL FFTW. [Note: The first letter, 'i' or 'g', indicates whether the compiler was Intel or GCC. The second letter, 'i' or 'o' indicates whether the MPI was Intel or OpenMPI.]
  • -intel/iomkl/foss-2017A - these are the full blown toolchains with compilers/MPI/math. See the table above for details.

There may be variations on that in the future. But that covers most of 2017A for now.


  1. minimizing module count - for example, it makes no sense to have three versions of bzip2 (intel/iomkl/foss) when a single version built with GCC can be used for all.
  2. more closely aligning with Linux distribution provided build tools - in addition to the above, we wanted to make sure core utilities like binutils were well suited for the C library (glibc) installed on the cluster involved. Beyond that, we found that in most cases, the provided build tools like autoconf/automake/cmake/make/libtool/pkgconfig were sufficient, we tried to use the system-provided ones where possible. We also use the system-provided openssl-devel (for timely security updates) and zlib-devel (which is required by openssl-devel).


  • by relying upon the system binutils, we were:
    • unable to fully use AVX2 and AVX512 vector extensions.
    • unable to build certain programs due to missing tools provided by newer binutils

Our newest toolchains - 2017b

Other than newer versions, the breakdown for 2017b is similar to 2017A. Key differences include:

  • using an updated version of binutils. For 2017A, we went with the system binutils in order to eliminate it as the cause of some of the problems we were having on terra. It turned out not to be a factor. However, we also found that it precluded us from taking advantage of such features like AVX2 (and AVX512) on terra and made it impossible to install TensorFlow on ada (missing needed command). So we've gone back to using the binutils recommended upstream by the EasyBuild people.
  • unfortunately the "Python-bare" packages are much less prevalent. The motivation for this was to reduce the total number of modules. For example, there is no reason to build an intel, iomkl and foss version of YAML when the GCCcore version would work for all three. However, we've not had much luck convincing the upstream that the -bare option is either viable (which we've demonstrated it is) or clear to users (they have a point there... there has certainly been confusion at times with 2017A). So, for the most part, you won't see as many Python-bare packages this time around (though there are still a few).


In the past, we've offered different combinations including some that use the Portland Groups compilers and some that use different variants of MPI (e.g. MVAPICH, MPICH, MPICH2). If the need arises to build such toolchains in the future, we will consider it. But for now, users are recommended to use one of the toolchains above (preferably the most recent).

The buildenv modules

For each toolchain above there exists a buildenv module that: 1) loads the submodules making up the toolchain 2) sets a number of useful flags used when compiling/linking programs


For example, here is what buildenv provides for a couple of our current toolchains.


$ ml -t show buildenv/default-intel-2017b
This module sets a group of environment variables for compilers, linkers,
 maths libraries, etc., that you can use to easily transition between
 toolchains when building your software. To query the variables being set
 please use: module show <this module name>

More information
 - Homepage: None
 This module sets a group of environment variables for compilers, linkers,
 maths libraries, etc., that you can use to easily transition between
 toolchains when building your software. To query the variables being set
 please use: module show <this module name>
whatis("Homepage: None")
setenv("CFLAGS","-O2 -xHost -ftz -fp-speculation=safe -fp-model source")
setenv("CXXFLAGS","-O2 -xHost -ftz -fp-speculation=safe -fp-model source")
setenv("F90FLAGS","-O2 -xHost -ftz -fp-speculation=safe -fp-model source")
setenv("FCFLAGS","-O2 -xHost -ftz -fp-speculation=safe -fp-model source")
setenv("FFLAGS","-O2 -xHost -ftz -fp-speculation=safe -fp-model source")
setenv("FFTW_STATIC_LIBS_MT","-fftw3xc_intel -mkl_intel_lp64 -mkl_sequential -mkl_core")
setenv("LDFLAGS","-L/sw/eb/sw/icc/2017.4.196-GCC-6.4.0-2.28/lib/intel64 -L/sw/eb/sw/imkl/2017.3.196-iimpi-2017b/lib -L/sw/eb/sw/imkl/2017.3.196-iimpi-2017b/mkl/lib/intel64 -L/sw/eb/sw/imkl/2017.3.196-iimpi-2017b/lib")
setenv("LIBBLACS","-Wl,-Bstatic -Wl,--start-group -lmkl_blacs_intelmpi_lp64 -Wl,--end-group -Wl,-Bdynamic")
setenv("LIBBLACS_MT","-Wl,-Bstatic -Wl,--start-group -lmkl_blacs_intelmpi_lp64 -Wl,--end-group -Wl,-Bdynamic")
setenv("LIBBLAS","-Wl,-Bstatic -Wl,--start-group -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -Wl,--end-group -Wl,-Bdynamic")
setenv("LIBBLAS_MT","-Wl,-Bstatic -Wl,--start-group -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -Wl,--end-group -Wl,-Bdynamic -liomp5 -lpthread")
setenv("LIBFFT","-Wl,-Bstatic -Wl,--start-group -lfftw3xc_intel -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -Wl,--end-group -Wl,-Bdynamic")
setenv("LIBFFT_MT","-Wl,-Bstatic -Wl,--start-group -lfftw3xc_intel -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -Wl,--end-group -Wl,-Bdynamic")
setenv("LIBLAPACK","-Wl,-Bstatic -Wl,--start-group -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -Wl,--end-group -Wl,-Bdynamic")
setenv("LIBLAPACK_MT","-Wl,-Bstatic -Wl,--start-group -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -Wl,--end-group -Wl,-Bdynamic -liomp5 -lpthread")
setenv("LIBLAPACK_MT_ONLY","-Wl,-Bstatic -Wl,--start-group -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -Wl,--end-group -Wl,-Bdynamic -liomp5 -lpthread")
setenv("LIBLAPACK_ONLY","-Wl,-Bstatic -Wl,--start-group -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -Wl,--end-group -Wl,-Bdynamic")
setenv("LIBS","-liomp5 -lpthread")
setenv("LIBSCALAPACK","-Wl,-Bstatic -Wl,--start-group -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -Wl,--end-group -Wl,-Bdynamic")
setenv("LIBSCALAPACK_MT","-Wl,-Bstatic -Wl,--start-group -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -Wl,--end-group -Wl,-Bdynamic -liomp5 -lpthread")
setenv("LIBSCALAPACK_MT_ONLY","-Wl,-Bstatic -Wl,--start-group -lmkl_scalapack_lp64 -Wl,--end-group -Wl,-Bdynamic -liomp5 -lpthread")
setenv("LIBSCALAPACK_ONLY","-Wl,-Bstatic -Wl,--start-group -lmkl_scalapack_lp64 -Wl,--end-group -Wl,-Bdynamic")
setenv("OPTFLAGS","-O2 -xHost")
setenv("PRECFLAGS","-ftz -fp-speculation=safe -fp-model source")