SW:myPython myAnaconda23
Contents
User installed using virtual environments
There are often times when a user wants a Python library that isn't already available on the system. All the versions of Python above provide a "virtual environment" that the user can use to install whatever libraries they need using tools like pip and easy_install (and conda with Anaconda).
Due to the differences between Python and Anaconda (and differences between Python 2.x and Python 3.x). we've installed some helper modules (myPython and myAnaconda2/myAnaconda3) to help users manage their own set of Python libraries. [Note: we haven't created these yet for all versions of Python/Anaconda. If you would like us to add one for the version you are using, please email a request to help@hprc.tamu.edu]
These modules will set the following environment variables:
- $MYPYTHONDIR - references your virtual environment in $SCRATCH
- $MYCREATEVIRTENV - should only be used once to create a virtual environment
- $MYACTIVATE - should be used whenever you want to add new Python libraries
- $MYDEACTIVATE - deactivate the virtual environment
Additionally, $PYTHONPATH is updated so your libraries can be used with non-virtual-activated (regular) Python/Anaconda(2/3).
Creating your virtual environment
Creating your virtual environment, which only needs to be done once, consists of two steps.
- load the myPython or myAnaconda2/myAnaconda3 module that matches the version of Python or Anaconda2/Anaconda3 module you want to use
- create the environment by running the $MYCREATEVIRTENV command. [Note: if using Anaconda3/4-x.y on terra, ignore the "lock file" warnings for now]
This will create a $SCRATCH/myPython/<version> (or $SCRATCH/myAnaconda/<version>) directory which will allow you to install whatever Python libraries you need.
Using your virtual environment
Once you've created your virtual environment, you can use it by:
- loading the same myPython or myAnaconda2/3 version that you used in the create step above.
- activate your environment by running $MYACTIVATE. If successful, your prompt should change to reflect the version of Python/Anaconda you are using.
When you've finished using your virtual environment, run the $MYDEACTIVATE command.
Installing Python libraries in your virtual environment
NOTE: You generally only need to install the package once for each virtual environment. It should still be there the next time you "activate" that environment.
If the library you want to install is available at PyPi, then you can use easy_install or pip to install the package by name. e.g.
pip install Theano
or
easy_install Theano
If the library you want is hosted by Anaconda, you can use the conda command (Anaconda only). e.g.
conda install -c rdkit rdkit
Note: All commands above have a --help option that provides more details.
If the libary you want isn't hosted by one of the above, most Python libraries come with a setup.py script. You can use this (after activating your virtual environment) with something like:
python setup.py build python setup.py install
Depending upon its dependencies, you may need to load needed modules first.
Anaconda (legacy)
NOTE: At some point (once we document the process) we recommend all users of the Legacy Anaconda to myAnaconda so they can maintain their Python environments without requiring HPRC staff intervention.
Anaconda2/Anaconda3 (new approach to installation)
This is a new (EasyBuild) approach that uses a slightly different naming scheme as it includes the major number of the Python it supports (either 2 or 3) in the actual name. These modules have been updated with the latest version of conda (at the time of install) to support the myAnaconda modules
myAnaconda2/myAnaconda3 (new approach for virtual environments)
See the Python page for an introduction to user defined virtual environments using the my modules. See below for details on how to replicate your current virtual environment in your $SCRATCH directory so you can update as needed.
For an overview, see the Python page.
For those currently using a HPRC maintained virtual environment (which requires HPRC intervention to update/modify) that want to maintain their own virtual environment in $SCRATCH, we have a collection of HPRC created VEs in /sw/local/etc/Anaconda/virtenvs/
To replicate a (legacy) Anaconda VE in $SCRATCH (so you can maintain it yourself), use the --file option with the $MYCREATEVIRTENV command.
For example:
ml purge # clear modules ml myAnaconda3/4.4.0 $MYCREATEVIRTENV --file /sw/local/etc/Anaconda/virtenvs/3-4-tensorflow.ada
Here is a list of some of the VEs available:
$ ls /sw/local/etc/Anaconda/virtenvs/ 2-4-Aby-bare.ada 3-4-Aby-needswork.ada 3-4-michael-nopango.ada 3-4-shyamprabhakar92.terra 2-4-anaconda-accelerate.ada 3-4-chsu1.ada 3-4-mostafa_karimi-noosprey.ada 3-4-tensorflow.ada 2-4-cpattison.ada 3-4-fangchao118.ada 3-4-mostafa_karimi-nordkit.terra 3-4-tensorflow-cuda-nocuda.ada 2-4-futurejl_1993.ada 3-4-fangchao118.terra 3-4-mpi4py.terra 3-4-tensorflow-cuda.terra 2-4-han820124.ada 3-4-han820124.ada 3-4-mridulgarg11.terra 3-4-vinicius.goecks.ada 2-4-michael-noglib.ada 3-4-han820124.terra 3-4-nlm_maheshwari-needswork.ada 3-4-vinicius.goecks.terra 2-4-OscarGuerrero.ada 3-4-keras-2.0.3.terra 3-4-OscarGuerrero.ada CHANNELS_NEEDED.ada 2-4-OscarGuerrero.terra 3-4-keras.terra 3-4-OscarGuerrero.terra 2-4-python-2-6-8.ada 3-4-lixin.ada 3-4-shyamprabhakar92-noopencv.ada
More info
Here are some references on using virtual environments in Python.
