"Magenta is distributed as an open source Python library, powered by TensorFlow. This library includes utilities for manipulating source data (primarily music and images), using this data to train machine learning models, and finally generating new content from these models."
Magenta on HPRC clusters
Note: Since the Python pip-installed version of Magenta recommended below requires GLIBC >= 2.17, Magenta can currently only be deployed on terra or curie which have CentOS 7 (ada is CentOS 6).
Ideally we would rebuild everything needed for Magenta to optimize for our clusters (and be compatible with the system GLIBC) but this is a significant and less than straightforward task. So for now we recommend the Python solution below.
Magenta via an Anaconda virtual environment
A lot of what is described below is probably easily adapted to Anaconda, but we are going to use a plain Python for now
Magenta via a Python virtual environment
In order to get the most out of Magenta you'll need the following modules:
- cuDNN/7.0.5-CUDA-9.0.176 - the version of Magenta installed by pip strictly requires CUDA 9.0 and cuDNN 7.0
- an "oldish" Python. Python 3.6 is too new. At the moment we recommend a Python 3.5.2 from the 2017A toolchain.
Verify the needed sound library development files are installed (currently only on terra2):
rpm -q alsa-lib-devel
Using the steps here, and replacing 'venv' with '$SCRATCH/mymagenta' we can create a virtual environment to install magenta-gpu (we want the GPU version given our TensorFlow benchmarks. We'll start with the 2017A, GCC-based, 'foss' toolchain to keep things simple.
ml purge # clean module environment ml Python/3.5.2-foss-2017A # load an older Python ml cuDNN/7.0.5-CUDA-9.0.176 # load the needed CUDA/cuDNN ml # list modules
Those steps will be required everytime you wish to access Magenta. If you load other modules, make sure they all comply with foss-2017A toolchain.
In order to install Magenta, we'll need to use the steps above to create a virtual environment and install Magenta.
rm -rf $SCRATCH/mymagenta # remove previous attempts (???) virtualenv $SCRATCH/mymagenta # create virtual environment source $SCRATCH/mymagenta/bin/activate # activate venv pip install magenta-gpu # install magenta/tensorflow w/GPU
A quick test is:
- run the 'ml' commands above to load the needed modules
- 'activate' the environment or add it to your PYTHONPATH (see below)
- test it with something like this:
(mymagenta) [j-perdue@tlogin-0502 (05:49) ~]$ python Python 3.5.2 (default, Mar 29 2017, 10:01:31) [GCC 6.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import magenta; # ignore warnings (lots of warnings to ignore) >>> magenta.__version__ '0.4.0'
Note that you will get a lot of 'warnings' about the precision of numpy. These are only warnings (not errors) and for now can be safely disregarded. You will likely get these everytime you do 'import magenta'. If you come up with a solution, please email email@example.com.
Once you've installed Magenta, you can use it one of two ways. In both ways, you first need to run the 'ml' commands to load the needed modules. After that, you can either:
- activate your virtual environment with 'source $SCRATCH/mymagenta/bin/activate'. This is probably the recommended course but may not be necessary.
- simply add the virtual environment libraries to your PYTHONPATH with something like 'export PYTHONPATH=$PYTHONPATH:$SCRATCH/mymagenta/lib/python3.5/site-packages/' (needs checking)
Running batch jobs
For now, given that Magenta relies upon TensorFlow, the batch files on our TensorFlow benchmarks page might be the best examples (see the Terra GPU example).