Skip to content

Docker

Introduction

Containers are a way to package an environment along with an executable so that no additional installation or setup is required to run it on a different machine. Docker is a container solution that is designed for use on personal desktops and virtual machines. If you have previously used Docker, you can use Docker on ACES to help you make the transition to HPC.

The basic element of a container solution is an image. An image is a bundle of files that includes a self-contained environment with both installed executables and the system libraries they depend on. The container runtime mediates between the libraries in the image and the libraries on the host system. In the case of HPRC, the container runtime software options we support are Charliecloud and Singularity. They can both read many common container image file formats, including Docker. The Docker runtime is provided on ACES to help you make the transition to a supported HPC runtime.

This page describe how to fetch and run Docker containers on ACES.

Why use Containers

  • Shareability: you can share your container image file with others by uploading it to a public repository, and download files shared by others.
  • Portability: you can use image files made for any computer with the same architecture (x84-64).
  • Reproducibility: cluster environments can change whenever the locally installed software gets updated. Container users are largely unaffected by this.

Why use Docker

  • Community: DockerHub boasts the largest group of existing container users.

Docker Tutorial

The topics covered in this documentation are minimal. Visit Dockers's documentation for a more complete tutorial.

Docker at HPRC

Rootless Docker is installed on the ACES cluster. (It not installed on Grace, FASTER, or Launch)

Rootless mode allows running the Docker daemon and containers as a non-root user to mitigate potential vulnerabilities in the daemon and the container runtime. Rootless mode executes the Docker daemon and containers inside a user namespace.

Docker Software

Docker is provided through our module system. The version may vary. 

module load docker-user

Additionally, docker-user has a user-specific setup step. This step is needed to install Docker configuration files in your home directory, which is present on all nodes. It is recommended to do this step on a login node. The setup tool will configure the default Docker context based on the current value of XDG_RUNTIME_DIR, which is well-defined on login nodes.

To enable docker-user for the current user:

dockerd-rootless-setuptool.sh install --skip-iptables

To disable docker-user for the current user:

dockerd-rootless-setuptool.sh uninstall --skip-iptables

Cluster Node Usage

Docker is available on login nodes and compute nodes. It is recommended to perform Docker tasks for large images on a compute node because they may be too resource-intensive for login nodes.

  • For resource-heavy activities, access a compute node using srun or the VNC app from the Interactive Apps menu in the Portal.
    • Docker is not recommended for use in batch jobs.
  • For resource-light activities, access a login node using the _cluster Shell access app from the Clusters menu in the Portal.

A Docker process (for example, the Docker Daemon process) that is started on one node may have undefined behavior if Docker is already running on a different node, since they will both be trying to read and write from the shared home directory. It is recommended to stop your Docker processes on each node when it is not in use, so that when you start again on a different node there is not a collision.

Docker Data

Docker saves data on disk in your home directory.

    $ docker info
    ...
    Docker Root Dir: /home/username/.local/share/docker 
    ...

Docker creates a lot of data and may easily overfill your file quota. Consider migrating this directory to the /scratch or /tmp filesystem. See our FAQ for more tips about this.

Docker Daemon

Docker commands do not function unless a Docker Daemon process is running on the current node. Otherwise, you may see an error like this:

ERROR: Cannot connect to the Docker daemon at unix:///run/user/5000/docker.sock. Is the docker daemon running?

When you first setup Docker configuration in your home directory using the dockerd-rootless-setuptool.sh script, it will attempt to start the Docker Daemon on the current node for your convenience.

At all other times, you must manage your own Docker Daemon process.

Warning: It is recommended to only have Docker Daemon running on one node at a time.

Managing Docker Daemon on Login Nodes

Docker Daemon on login nodes is managed by the systemctl tool in --user mode.

To check if Docker Daemon is running on the current node:

systemctl --user status

Compare the output to the following table:

Docker Daemon running on login node HOSTNAME Docker Daemon not running on login node HOSTNAME 
● HOSTNAME
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: ...
   CGroup: /user.slice/user-50000.slice/user@50000.service
           ├─docker.service
           │ ├─2994871 rootlesskit ...
           │ ├─2994878 /proc/self/exe  ...
           │ ├─2994888 vpnkit  ...
           │ ├─2994906 dockerd --iptables=false
           │ └─2994954 containerd ...
           └─init.scope
             ├─2783501 /usr/lib/systemd/systemd --user
             └─2783503 (sd-pam)

● HOSTNAME
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: ...
   CGroup: /user.slice/user-50000.slice/user@50000.service
           └─init.scope
             ├─2783501 /usr/lib/systemd/systemd --user
             └─2783503 (sd-pam)

To start the Docker Daemon on the current node:

systemctl --user start docker.service

To stop the Docker Daemon on the current node:

systemctl --user stop docker.service

Warning: It is recommended to only have Docker Daemon running on one node at a time.

Managing Docker Daemon on Compute Nodes (VNC method)

You will need two terminal windows. One to manage the Docker Daemon, and another to execute Docker commands. Right-click (or long click) on the background outside a terminal window top open the Fluxbox menu and click 'xterm' to create an additional terminal window. It may be necessary to close the browser window and reconnect to the VNC session if it doesn't work right away.

Compute nodes don't have systemctl --user enabled, so you must start the Docker Daemon manually. Some extra variables are required to set this up. In one terminal window: 

export XDG_RUNTIME_DIR=$HOME/.docker/run
export DOCKER_HOST=unix://${XDG_RUNTIME_DIR}/docker.sock
module load docker-user
dockerd-rootless.sh

The process will produce a lot of output. The Daemon is ready when you see a line like this:

    INFO[...] API listen on /home/username/.docker/run/docker.sock

In the other terminal window, simply load the module. 

module load docker-user

To stop the Docker Daemon manually, select its terminal window and use ctrl+c to terminate.

Warning: It is recommended to only have Docker Daemon running on one node at a time.

Managing Docker Daemon on Compute Nodes (srun method)

To reach a compute node interactively, use the slurm command srun with the interactive arguments. Read more about ACES slurm options. 

srun --pty bash -i

Compute nodes don't have systemctl --user enabled, so you must start the Docker Daemon manually. Some extra variables are required to set this up. Start the Docker Daemon as a background process using &. 

mkdir -p $HOME/.docker/run
export XDG_RUNTIME_DIR=$HOME/.docker/run
export DOCKER_HOST=unix://${XDG_RUNTIME_DIR}/docker.sock
module load docker-user
dockerd-rootless.sh &

The process will produce a lot of output. The Daemon is ready when you see a line like this:

    INFO[...] API listen on /home/username/.docker/run/docker.sock

Press ↵ Enter or return to recover your shell prompt.

To stop the Docker Daemon manually, use fg to bring the process out of the background and then ctrl+c to terminate.

Warning: It is recommended to only have Docker Daemon running on one node at a time.

Running with Docker

Note: ensure you have loaded the docker-user module and the Daemon is running on the current node.

  • Reminder: module load docker-user
  • Reminder: systemctl --user start docker.service on login nodes (recommended)

To test if Docker runs, 

docker run hello-world

Interactive Example

To execute the default executable for a given container image,

docker run <image-name>

To execute a specific executable within a container, use the --entrypoint option.

To start an interactive session with a container, add the -it option.

For example, an interactive shell using the almalinux container image:

docker run -it --entrypoint /bin/bash  almalinux

    $ docker run -it --entrypoint /bin/bash  almalinux
    # ls
    afs  bin  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
    # exit
    $

Additional Docker Options

To set additional environment variables, add the -e option for each variable. 

-e VAR=value -e VAR=value ...

To enable use with NVIDIA GPUS, add the --gpus all option.

To mount an external filesystem such as /scratch, use the -v option for each directory. Separete paths with : if the paths outside and inside will be different. 

-v /scratch -v /tmp:/tmp_inside_container

To control memory usage, Docker has these options.

--ulimit memlock=-1 --ulimit stack=67108864 --shm-size=8g --ipc=host

Runtime Limitations

Rootless Docker cannot be used to set a new network address space. You must use the host network address space. If the container does not do this by default, you may need to add the --net=host option. See https://docs.docker.com/engine/security/rootless/#known-limitations for additional details.

Docker can sometimes have permission issues with parallel filesystems such as Lustre, because the user identity in the container may not match the user identity on the remote file server node. This may or may be a problem in your container, depending on how the image was configured.

Container images

Container images are found in both public and private repositories available on the internet.

Caution: Docker Hub and many other container repos are public repositories; do not trust unverified sources!

Warning: downloading a large image file is resource-intensive and takes a long time.

Warning: container images are very large on disk, and may easily overfill your file quota.

Fetching from Docker Hub

Connecting to Docker Hub require internet access. In order to access the internet from compute nodes, use the Web Proxy module (not required on login nodes). 

module load WebProxy

docker pull can automatically download docker images from Docker Hub. Read more about docker repositories.

docker pull <image>

Image storage

Use this command to inspect what images are stored in your local image repository. This repository will be in your home directory, unless you have relocated it (see Docker Data, above). 

docker images

    REPOSITORY        TAG       IMAGE ID        CREATED        SIZE
    hello-world       latest    bf756fb1ae65    5 months ago   13.3kB

Docker to Singularity

Convert a Docker image into a Singularity image (.sif) using Singularity/Apptainer. (For more details, see our Singularity page).

Required: do this on a compute node where your Docker Daemon is running (See Managing Docker Daemon on Compute Nodes, above) 

export SINGULARITY_CACHEDIR=$TMPDIR
singularity build image.sif docker-daemon://<image>

Docker to Charliecloud

Convert a Docker image into a Squashfs image (.sqfs) using Charliecloud. (For more details, see our Charliecloud page.)

Required: do this on a node where your Docker Daemon is running. 

module load charliecloud
ch-convert -i docker <image> image.sqfs