Hprc banner tamu.png

Difference between revisions of "SW:Python Implementations"

From TAMU HPRC
Jump to: navigation, search
Line 1: Line 1:
 
<!-- == Python == -->
 
<!-- == Python == -->
 
At present, we have three "flavors" of Python.
 
At present, we have three "flavors" of Python.
* '''Python''' modules that were built from source on our systems
+
=== High performance, locally built ===
* '''Anaconda(2/3)''' modules that were built elsewhere but include a number of precompiled Python libraries.  More information on using prebuilt virtual environments can be found on our [[SW:Anaconda | Anaconda]] page.
+
'''Python''' modules that were built from source on our systems
* Two '''Python''' modules (Python/2.7.12-2017.0.035 and Python/3.5.2-2017.0.035)  built by Intel that include a number of Python libraries, all compiled with Intel optimizations.  These include Anaconda and a number of other utiliies.  More information can be found at [https://software.intel.com/en-us/distribution-for-python Intel's Python distribution page].
+
 
 +
=== High performance, prebuilt by Intel ===
 +
Two '''Python''' modules (Python/2.7.12-2017.0.035 and Python/3.5.2-2017.0.035)  built by Intel that include a number of Python libraries, all compiled with Intel optimizations.  These include Anaconda and a number of other utiliies.  More information can be found at [https://software.intel.com/en-us/distribution-for-python Intel's Python distribution page].
 +
 
 +
=== Low performance, provided by Anaconda ===
 +
'''Anaconda(2/3)''' modules that were built elsewhere but include a number of precompiled Python libraries.  More information on using prebuilt virtual environments can be found on our [[SW:Anaconda | Anaconda]] page.
 +
 
  
 
User's wanting to use Python with other applications on the system that were built with Python (they will have '''Python''' in the version),  should use the Python that matches their application.  This generally precludes the use of '''Anaconda''' or the Intel '''Python''' modules
 
User's wanting to use Python with other applications on the system that were built with Python (they will have '''Python''' in the version),  should use the Python that matches their application.  This generally precludes the use of '''Anaconda''' or the Intel '''Python''' modules
Line 16: Line 22:
  
 
For '''Anaconda''' and the Intel '''Python'''s, there are a number of libraries already included.
 
For '''Anaconda''' and the Intel '''Python'''s, there are a number of libraries already included.
 
====User installed using virtual environments====
 
There are often times when a user wants a Python library that isn't already available on the system.  All the versions of Python above provide a "virtual environment" that the user can use to install whatever libraries they need using tools like '''pip''' and '''easy_install''' (and '''conda''' with Anaconda).
 
 
Due to the differences between Python and Anaconda (and differences between Python 2.x and Python 3.x). we've installed some helper modules ('''myPython''' and '''myAnaconda2'''/'''myAnaconda3''') to help users manage their own set of Python libraries.  [Note: we haven't created these yet for all versions of Python/Anaconda.  If you would like us to add one for the version you are using, please email a request to help@hprc.tamu.edu]
 
 
These modules will set the following environment variables:
 
 
* '''$MYPYTHONDIR''' - references your virtual environment in $SCRATCH
 
* '''$MYCREATEVIRTENV''' - should only be used once to create a virtual environment
 
* '''$MYACTIVATE''' - should be used whenever you want to add new Python libraries
 
* '''$MYDEACTIVATE''' - deactivate the virtual environment
 
 
Additionally, '''$PYTHONPATH''' is updated so your libraries can be used
 
with non-virtual-activated (regular) Python/Anaconda(2/3).
 
 
===== Creating your virtual environment =====
 
 
Creating your virtual environment, which only needs to be done once, consists of two steps.
 
 
# load the '''myPython''' or '''myAnaconda2'''/'''myAnaconda3''' module that matches the version of '''Python''' or '''Anaconda2'''/'''Anaconda3''' module you want to use
 
# create the environment by running the '''$MYCREATEVIRTENV''' command.  [Note: if using Anaconda3/4-x.y on terra, ignore the "lock file" warnings for now]
 
 
This will create a $SCRATCH/myPython/<version> (or $SCRATCH/myAnaconda/<version>) directory which will allow you to install whatever Python libraries you need.
 
 
===== Using your virtual environment =====
 
Once you've created your virtual environment, you can use it by:
 
# loading the same '''myPython''' or '''myAnaconda2/3''' version that you used in the create step above.
 
# activate your environment by running '''$MYACTIVATE'''.  If successful, your prompt should change to reflect the version of Python/Anaconda you are using.
 
 
When you've finished using your virtual environment, run the '''$MYDEACTIVATE''' command.
 
 
===== Installing Python libraries in your virtual environment =====
 
NOTE: You generally only need to install the package once for each virtual environment.  It should still be there the next time you "activate" that environment.
 
 
If the library you want to install is available at [https://pypi.python.org/pypi PyPi], then you can use '''easy_install''' or '''pip''' to install the package by name.  e.g.
 
<pre>
 
  pip install Theano
 
</pre>
 
or
 
<pre>
 
  easy_install Theano
 
</pre>
 
 
If the library you want is hosted by Anaconda, you can use the '''conda''' command (Anaconda only).  e.g.
 
<pre>
 
conda install -c rdkit rdkit
 
</pre>
 
 
Note: All commands above have a '''--help''' option that provides more details.
 
 
If the libary you want isn't hosted by one of the above, most Python libraries come with a '''setup.py''' script.  You can use this (after activating your virtual environment) with something like:
 
<pre>
 
python setup.py build
 
python setup.py install
 
</pre>
 
Depending upon its dependencies, you may need to load needed modules first.
 
 
==== Anaconda (legacy) ====
 
 
NOTE: At some point (once we document the process) we recommend all users of the Legacy Anaconda to myAnaconda so they can maintain their Python environments without requiring HPRC staff intervention.
 
 
==== Anaconda2/Anaconda3 (new approach to installation) ====
 
This is a new (EasyBuild) approach that uses a slightly different naming scheme as it includes the major number of the Python it supports (either 2 or 3) in the actual name.  These modules have been updated with the latest version of '''conda''' (at the time of install) to support the '''myAnaconda''' modules
 
 
==== myAnaconda2/myAnaconda3 (new approach for virtual environments) ====
 
See the [[SW:Python | Python page]] for an introduction to user defined virtual environments using the '''my''' modules.  See below for details on how to replicate your current virtual environment in your $SCRATCH directory so you can update as needed.
 
 
For an overview, see the [[SW:Python | Python page]].
 
 
For those currently using a HPRC maintained virtual environment (which requires HPRC intervention to update/modify) that want to maintain their own virtual environment in $SCRATCH, we have a collection of HPRC created VEs in '''/sw/local/etc/Anaconda/virtenvs/'''
 
 
To replicate a (legacy) Anaconda VE in $SCRATCH (so you can maintain it yourself), use the '''--file''' option with the '''$MYCREATEVIRTENV''' command.
 
 
For example:
 
 
<pre>
 
  ml purge # clear modules
 
  ml myAnaconda3/4.4.0
 
  $MYCREATEVIRTENV --file /sw/local/etc/Anaconda/virtenvs/3-4-tensorflow.ada
 
</pre>
 
 
Here is a list of some of the VEs available:
 
 
<pre>
 
$ ls /sw/local/etc/Anaconda/virtenvs/
 
2-4-Aby-bare.ada            3-4-Aby-needswork.ada  3-4-michael-nopango.ada            3-4-shyamprabhakar92.terra
 
2-4-anaconda-accelerate.ada  3-4-chsu1.ada          3-4-mostafa_karimi-noosprey.ada    3-4-tensorflow.ada
 
2-4-cpattison.ada            3-4-fangchao118.ada    3-4-mostafa_karimi-nordkit.terra  3-4-tensorflow-cuda-nocuda.ada
 
2-4-futurejl_1993.ada        3-4-fangchao118.terra  3-4-mpi4py.terra                  3-4-tensorflow-cuda.terra
 
2-4-han820124.ada            3-4-han820124.ada      3-4-mridulgarg11.terra            3-4-vinicius.goecks.ada
 
2-4-michael-noglib.ada      3-4-han820124.terra    3-4-nlm_maheshwari-needswork.ada  3-4-vinicius.goecks.terra
 
2-4-OscarGuerrero.ada        3-4-keras-2.0.3.terra  3-4-OscarGuerrero.ada              CHANNELS_NEEDED.ada
 
2-4-OscarGuerrero.terra      3-4-keras.terra        3-4-OscarGuerrero.terra
 
2-4-python-2-6-8.ada        3-4-lixin.ada          3-4-shyamprabhakar92-noopencv.ada
 
</pre>
 
===== More info =====
 
Here are some references on using virtual environments in Python.
 
 
* https://docs.python.org/3/tutorial/venv.html
 
* http://python-guide-pt-br.readthedocs.io/en/latest/dev/virtualenvs/
 
* https://realpython.com/blog/python/python-virtual-environments-a-primer/
 

Revision as of 12:27, 12 April 2018

At present, we have three "flavors" of Python.

High performance, locally built

Python modules that were built from source on our systems

High performance, prebuilt by Intel

Two Python modules (Python/2.7.12-2017.0.035 and Python/3.5.2-2017.0.035) built by Intel that include a number of Python libraries, all compiled with Intel optimizations. These include Anaconda and a number of other utiliies. More information can be found at Intel's Python distribution page.

Low performance, provided by Anaconda

Anaconda(2/3) modules that were built elsewhere but include a number of precompiled Python libraries. More information on using prebuilt virtual environments can be found on our Anaconda page.


User's wanting to use Python with other applications on the system that were built with Python (they will have Python in the version), should use the Python that matches their application. This generally precludes the use of Anaconda or the Intel Python modules

User's only wanting to use Python in isolation or with just Python modules, might find the Anaconda versions easier to use.

User's demanding the best optimized Python might want to use the Intel provided versions. Note, however, that HPRC staff has limited experience with these versions.

Python libraries

Prebuilt

For the first Python above, there are a number of tools/libraries already built to support applications that needed them. You can use the module command to search for those available. NOTE: Be sure to match Python versions.... e.g. don't try to mix Python-2.7.10 modules with Python-2.7.12 or Python-3.5.3. For more details, see our Toolchains page.

For Anaconda and the Intel Pythons, there are a number of libraries already included.