Hprc banner tamu.png


Revision as of 06:38, 6 December 2018 by J-perdue (talk | contribs) (Magenta on HPRC clusters)

Jump to: navigation, search


"Magenta is distributed as an open source Python library, powered by TensorFlow. This library includes utilities for manipulating source data (primarily music and images), using this data to train machine learning models, and finally generating new content from these models."

Magenta on HPRC clusters

Note: Since the Python pip-installed version of Magenta recommended below requires GLIBC >= 2.17, Magenta can currently only be deployed on terra or curie which have CentOS 7 (ada is CentOS 6).

Magenta module

Ideally we would rebuild everything needed for Magenta to optimize for our clusters but this is a significant and less than straightforward task. So for now we recommend the Python solution below.

Magenta via an Anaconda virtual environment

A lot of what is described below is probably easily adapted to Anaconda, but we are going to use a plain Python for now

Magenta via a Python virtual environment

In order to get the most out of Magenta you'll need the following modules:

  1. cuDNN/7.0.5-CUDA-9.0.176 - the version of Magenta installed by pip strictly requires CUDA 9.0 and cuDNN 7.0
  2. an "oldish" Python. Python 3.6 is too new. At the moment we recommend a Python 3.5.2 from the 2017A toolchain.

Verify the needed sound library development files are installed (currently only on ada2 and terra2):

rpm -q alsa-lib-devel

Using the steps here, and replacing 'venv' with '$SCRATCH/mymagenta' we can create a virtual environment to install magenta-gpu (we want the GPU version given our TensorFlow benchmarks. We'll start with the 2017A, GCC-based, 'foss' toolchain to keep things simple.

ml purge # clean module environment
ml Python/3.5.2-foss-2017A # load an older Python
ml cuDNN/7.0.5-CUDA-9.0.176 # load the needed CUDA/cuDNN
ml # list modules

Those steps will be required everytime you wish to access Magenta. If you load other modules, make sure they all comply with foss-2017A toolchain.

In order to install Magenta, we'll need to use the steps above to create a virtual environment and install Magenta.

rm -rf $SCRATCH/mymagenta # remove previous attempts (???)
virtualenv $SCRATCH/mymagenta # create virtual environment
source $SCRATCH/mymagenta/bin/activate # activate venv
pip install magenta-gpu # install magenta/tensorflow w/GPU

A quick test is:

  1. run the 'ml' commands above to load the needed modules
  2. run: python -c 'import magenta; magenta.__version__'

Note that you will get a lot of 'warnings' about the precision of numpy. These are only warnings (not errors) and for now can be safely disregarded. You will likely get these everytime you do 'import magenta'. If you come up with a solution, please email help@hprc.tamu.edu.


Once you've installed Magenta, you can use it one of two ways. In both ways, you first need to run the 'ml' commands to load the needed modules. After that, you can either:

  1. activate your virtual environment with 'source $SCRATCH/mymagenta/bin/activate'. This is probably the recommended course but may not be necessary.
  2. simply add the virtual environment libraries to your PYTHONPATH with something like 'export PYTHONPATH=$PYTHONPATH:$SCRATCH/mymagenta/lib/python3.5/site-packages/' (needs checking)