Hprc banner tamu.png

Difference between revisions of "SW:Python Implementations"

From TAMU HPRC
Jump to: navigation, search
(Fastest, locally built)
(Intel optimized, provided by Intel)
 
(42 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
__TOC__
 
<!-- == Python == -->
 
<!-- == Python == -->
At present, we have three "flavors" of Python.
+
=== Implementations ===
=== Fastest, locally built ===
+
At present, we have three "flavors"/implementations of Python.
'''Python''' modules that were built from source on our systems. These will be named Python/<version>-<toolchain>.  For example Python/2.7.14-intel-2017b can be used with modules with intel-2017b-Python-2.7.14 in their name.  Be sure not to mix versions or toolchains.
 
* these use the latest CPU capibilities for the system(s) they were built on.  These include Intel's Advanced Vector Extensions (AVX).
 
* these are compatible with a other Python modules on the system built using the same [[SW:Toolchains | toolchain]] and the same version of Python
 
  
=== Prebuilt by Intel ===
+
==== Most optimized, locally built ====
Two '''Python''' modules (Python/2.7.12-2017.0.035 and Python/3.5.2-2017.0.035) built by Intel that include a number of Python libraries, all compiled with Intel optimizations.  These include Anaconda and a number of other utiliies.  More information can be found at [https://software.intel.com/en-us/distribution-for-python Intel's Python distribution page]. These can '''NOT''' be used with other Python modules on the system.
+
'''Python''' modules that were built from source on our systems. These will be named Python/<version>-<toolchain>.  For example Python/2.7.14-intel-2017b can be used with modules with intel-2017b-Python-2.7.14 in their name. Be sure not to mix Python versions or [[SW:Toolchains | toolchains]].
 +
* these use the latest CPU capabilities for the system(s) they were built on.  These include [https://en.wikipedia.org/wiki/Advanced_Vector_Extensions Intel/AMD's Advanced Vector Extensions (AVX)].
 +
* these are compatible with all other Python modules on the system built using the same [[SW:Toolchains | toolchain]] and the same version of Python
  
=== Slowest, provided by Anaconda ===
+
==== Intel optimized, provided by Intel ====
'''Anaconda(2/3)''' modules that were built elsewhere but include a number of precompiled Python libraries.  More information on using prebuilt virtual environments can be found on our [[SW:Anaconda | Anaconda]] page. Note that these do not include optimizations from Intel. Also note these can '''NOT''' be used with other Python modules on the system.
+
A number of '''IntelPython''' modules built by Intel that include a number of Python libraries, all compiled with Intel optimizations.  These include Anaconda and a number of other utilities (e.g. Jupyter).  More information can be found at [https://software.intel.com/en-us/distribution-for-python Intel's Python distribution page].
 +
* These were built Intel using Intel compilers but do not necessarily include optimizations for the current generation of HPC CPUs. What level of features like AVX is supported varies upon version (see Note: below).
 +
* These can not be used with other Python modules on the system.
 +
 
 +
==== Anaconda ====
 +
'''Anaconda(2/3)''' modules that were built elsewhere but include a number of precompiled Python libraries.  More information on using Anaconda virtual environments can be found on our [[SW:Anaconda | Anaconda page]].  
 +
* These were not built by Intel compilers. Furthermore, they are only optimized for older architectures. They will not take advantages of modern HPC CPU features like AVX.
 +
* These can not be used with other '''Python''' modules on the system.
 +
* Note that we have two variations of Anaconda currently installed.
 +
** Anaconda/<python-version-major>-<anaconda-release> (e.g. Anaconda/3-5.0.0.1) provides an Anaconda environment that has been updated for use on HPRC clusters.  For details on using them, see our [[SW:Anaconda | Anaconda page]].
 +
** Anaconda<python-version-major>/<anaconda-release> (e.g. Anaconda3/5.0.0.1) provides a pristine Anaconda with no alterations for compatibility with Anaconda environments on other systems.  To see how they can be used to simplify Python and Anaconda virtual environments, see [[SW:myPython_myAnaconda23 | our "my" wrappers for Python and Anaconda]]
  
 
=== Selecting which to use ===
 
=== Selecting which to use ===
User's wanting to use Python with other applications on the system that were built with Python (they will have '''Python''' in the version),  should use the Python that matches their application.  This generally precludes the use of '''Anaconda''' or the Intel '''Python''' modules
+
User's wanting to use Python with other applications on the system that were built with Python (they will have '''Python''' in the version),  should use the Python that matches their application.  This generally precludes/excludes the use of '''Anaconda''' or the Intel '''Python''' modules
  
User's only wanting to use Python in isolation or with just Python modules, might find the '''Anaconda''' versions easier to use.
+
User's only wanting to use Python in isolation or with just Python modules, might find the '''Anaconda''' versions easier to use. (see our [[SW:Anaconda | Anaconda page]])
  
User's demanding the best optimized '''Python''' might want to use the Intel provided versions.  Note, however, that HPRC staff has limited experience with these versions.
+
User's wanting an Intel-optimized '''Python''' might want to use the Intel provided versions.  Note, however, that HPRC staff has limited experience with these versions.
  
 
===Python libraries===
 
===Python libraries===
Line 23: Line 33:
 
For the first '''Python''' above, there are a number of tools/libraries already built to support applications that needed them.  You can use the '''module''' command to search for those available.  NOTE: Be sure to match Python versions.... e.g. don't try to mix Python-2.7.10 modules with Python-2.7.12 or Python-3.5.3.  For more details, see our [[SW:Toolchains | Toolchains]] page.
 
For the first '''Python''' above, there are a number of tools/libraries already built to support applications that needed them.  You can use the '''module''' command to search for those available.  NOTE: Be sure to match Python versions.... e.g. don't try to mix Python-2.7.10 modules with Python-2.7.12 or Python-3.5.3.  For more details, see our [[SW:Toolchains | Toolchains]] page.
  
For '''Anaconda''' and the Intel '''Python'''s, there are a number of libraries already included.
+
For '''Anaconda''' and the Intel '''Python'''s, there are a number of libraries already included.  Others can be installed with the "conda" command into a "virtual environment".  See the [[SW:Anaconda | Anaconda page]] for more details.

Latest revision as of 11:20, 19 April 2018

Implementations

At present, we have three "flavors"/implementations of Python.

Most optimized, locally built

Python modules that were built from source on our systems. These will be named Python/<version>-<toolchain>. For example Python/2.7.14-intel-2017b can be used with modules with intel-2017b-Python-2.7.14 in their name. Be sure not to mix Python versions or toolchains.

Intel optimized, provided by Intel

A number of IntelPython modules built by Intel that include a number of Python libraries, all compiled with Intel optimizations. These include Anaconda and a number of other utilities (e.g. Jupyter). More information can be found at Intel's Python distribution page.

  • These were built Intel using Intel compilers but do not necessarily include optimizations for the current generation of HPC CPUs. What level of features like AVX is supported varies upon version (see Note: below).
  • These can not be used with other Python modules on the system.

Anaconda

Anaconda(2/3) modules that were built elsewhere but include a number of precompiled Python libraries. More information on using Anaconda virtual environments can be found on our Anaconda page.

  • These were not built by Intel compilers. Furthermore, they are only optimized for older architectures. They will not take advantages of modern HPC CPU features like AVX.
  • These can not be used with other Python modules on the system.
  • Note that we have two variations of Anaconda currently installed.
    • Anaconda/<python-version-major>-<anaconda-release> (e.g. Anaconda/3-5.0.0.1) provides an Anaconda environment that has been updated for use on HPRC clusters. For details on using them, see our Anaconda page.
    • Anaconda<python-version-major>/<anaconda-release> (e.g. Anaconda3/5.0.0.1) provides a pristine Anaconda with no alterations for compatibility with Anaconda environments on other systems. To see how they can be used to simplify Python and Anaconda virtual environments, see our "my" wrappers for Python and Anaconda

Selecting which to use

User's wanting to use Python with other applications on the system that were built with Python (they will have Python in the version), should use the Python that matches their application. This generally precludes/excludes the use of Anaconda or the Intel Python modules

User's only wanting to use Python in isolation or with just Python modules, might find the Anaconda versions easier to use. (see our Anaconda page)

User's wanting an Intel-optimized Python might want to use the Intel provided versions. Note, however, that HPRC staff has limited experience with these versions.

Python libraries

Prebuilt

For the first Python above, there are a number of tools/libraries already built to support applications that needed them. You can use the module command to search for those available. NOTE: Be sure to match Python versions.... e.g. don't try to mix Python-2.7.10 modules with Python-2.7.12 or Python-3.5.3. For more details, see our Toolchains page.

For Anaconda and the Intel Pythons, there are a number of libraries already included. Others can be installed with the "conda" command into a "virtual environment". See the Anaconda page for more details.