Skip to content

ModuLair

Description

ModuLair is a Python virtual environment (venv) management solution developed here at HPRC. It empowers users to streamline their Python workflows without having to spend time configuring third-party solutions to integrate with HPC. ModuLair is pre-installed on our clusters and provides seamless integration with our systems.

Command Line Interface

ModuLair's Command Line Interface offers five commands that compose the backbone of the software's utility.

  1. create_venv - Create a new virtual environment in any available Python version.

    • Add a description of what the environment is for and/or a custom module toolchain tailored to your workflow.
    • -p flag: Select a specific Python version.
    • -t flag: Define a custom toolchain.
      • Accepts a space-separated list of modules.
      • Available toolchains can be viewed using the toolchains command.
    • -g flag: Create a shared environment.
      • Stores the environment in the group’s $SCRATCH directory.
      • Makes the environment accessible to all group members.

  2. list_venvs - List all of the environments that you have created with the ModuLair suite. This is your "Lair." All environments are stored in $SCRATCH/virtual_envs.

  3. activate_venv - Activate any of the environments in your Lair with one command.

  4. delete_venv - Delete all of the data of any of the environments in your Lair.

  5. add_venv - Add any existing Python environment into virtual environment manager.

Use -h flag with any of these commands to see how they can be used.

Example environment life-cycle

This example shows the life-cycle of a venv for a deep learning project using gpu accelration. This environment will be created with a custom toolchain by leveraging the -t flag. This option accepts a space-separated list of modules as an input. 

$ create_venv dl_train -d "Environment for training my deep learning model" -t "CUDA/12.8.0 GCCcore/13.3.0 Python/3.12.3"
NOTE: When providing a custom toolchain, make sure dependencies of a module are written before it. For example, if Python was provided before GCCcore, this command would fail, because the Python module relies on the GCCcore module. 
$ list_venvs
Name                Description              Python Version    GCC Version    Toolchain
------------------------------------------------------------------------------------------------------------
dl_train            Environment for training my deep learning model    N/A               N/A            CUDA/12.8.0 GCCcore/13.3.0 Python/3.12.3

$ source activate_venv dl_train
(dl_train) $ ... your commands

$ delete_venv dl_train
Removing {'name': 'dl_train', 'python_version': 'N/A', 'GCCcore_version': 'N/A', 'description': 'Environment for training my deep learning model', 'toolchain': 'CUDA/12.8.0 GCCcore/13.3.0 Python/3.12.3'}'s files...

Successfully removed {'name': 'dl_train', 'python_version': 'N/A', 'GCCcore_version': 'N/A', 'description': 'Environment for training my deep learning model', 'toolchain': 'CUDA/12.8.0 GCCcore/13.3.0 Python/3.12.3'}

OnDemand Portal

Each of HPRC's clusters has an OnDemand Portal which contains many tools to manage a user's HPRC resources. ModuLair's graphical interface can be found as an available tool here.

The GUI offers the create, delete, and list functionality provided by the CLI. The GUI makes it easier for users that are unfamiliar with the command line to set up their workflows.