Hprc banner tamu.png

SW:Magenta

From TAMU HPRC
Revision as of 06:54, 6 December 2018 by J-perdue (talk | contribs) (Using)

Jump to: navigation, search

Magenta

"Magenta is distributed as an open source Python library, powered by TensorFlow. This library includes utilities for manipulating source data (primarily music and images), using this data to train machine learning models, and finally generating new content from these models."

Magenta on HPRC clusters

Note: Since the Python pip-installed version of Magenta recommended below requires GLIBC >= 2.17, Magenta can currently only be deployed on terra or curie which have CentOS 7 (ada is CentOS 6).

Magenta module

Ideally we would rebuild everything needed for Magenta to optimize for our clusters but this is a significant and less than straightforward task. So for now we recommend the Python solution below.

Magenta via an Anaconda virtual environment

A lot of what is described below is probably easily adapted to Anaconda, but we are going to use a plain Python for now

Magenta via a Python virtual environment

In order to get the most out of Magenta you'll need the following modules:

  1. cuDNN/7.0.5-CUDA-9.0.176 - the version of Magenta installed by pip strictly requires CUDA 9.0 and cuDNN 7.0
  2. an "oldish" Python. Python 3.6 is too new. At the moment we recommend a Python 3.5.2 from the 2017A toolchain.

Verify the needed sound library development files are installed (currently only on terra2):

rpm -q alsa-lib-devel
Installing

Using the steps here, and replacing 'venv' with '$SCRATCH/mymagenta' we can create a virtual environment to install magenta-gpu (we want the GPU version given our TensorFlow benchmarks. We'll start with the 2017A, GCC-based, 'foss' toolchain to keep things simple.

ml purge # clean module environment
ml Python/3.5.2-foss-2017A # load an older Python
ml cuDNN/7.0.5-CUDA-9.0.176 # load the needed CUDA/cuDNN
ml # list modules

Those steps will be required everytime you wish to access Magenta. If you load other modules, make sure they all comply with foss-2017A toolchain.

In order to install Magenta, we'll need to use the steps above to create a virtual environment and install Magenta.

rm -rf $SCRATCH/mymagenta # remove previous attempts (???)
virtualenv $SCRATCH/mymagenta # create virtual environment
source $SCRATCH/mymagenta/bin/activate # activate venv
pip install magenta-gpu # install magenta/tensorflow w/GPU
Testing

A quick test is:

  1. run the 'ml' commands above to load the needed modules
  2. test it with something like this:
(mymagenta) [j-perdue@tlogin-0502 (05:49) ~]$ python
Python 3.5.2 (default, Mar 29 2017, 10:01:31) 
[GCC 6.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import magenta; # ignore warnings
(lots of warnings to ignore)
>>> magenta.__version__
'0.4.0'

Note that you will get a lot of 'warnings' about the precision of numpy. These are only warnings (not errors) and for now can be safely disregarded. You will likely get these everytime you do 'import magenta'. If you come up with a solution, please email help@hprc.tamu.edu.

Using

Once you've installed Magenta, you can use it one of two ways. In both ways, you first need to run the 'ml' commands to load the needed modules. After that, you can either:

  1. activate your virtual environment with 'source $SCRATCH/mymagenta/bin/activate'. This is probably the recommended course but may not be necessary.
  2. simply add the virtual environment libraries to your PYTHONPATH with something like 'export PYTHONPATH=$PYTHONPATH:$SCRATCH/mymagenta/lib/python3.5/site-packages/' (needs checking)
Running batch jobs

For now, given that Magenta relies upon TensorFlow, the batch files on our TensorFlow benchmarks page might be the best examples.