Hprc banner tamu.png

Difference between revisions of "SW:Singularity:Examples"

From TAMU HPRC
Jump to: navigation, search
(Create a sanbox)
(Prebuilt images)
 
(48 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== Examples ==
+
== Popular Repositories ==
=== Prebuilt images ====
+
 
HPRC provides (will provide) images using the steps below on ada/terra in /scratch/datasets/Singularity/images if you want to use a prebuilt image.
+
When looking for containerized software, try these repositories:
 +
 
 +
<span style="color:#FF0000">Notation</span>: [ ] means optional, < > means mandatory.
 +
===[https://hub.docker.com/ Docker Hub]===
 +
Most popular general-purpose public repository. Docker is the most common container image type.
 +
 
 +
Repository URLs in the form <nowiki>https://hub.docker.com/r/<source-group>/<source-name></nowiki>
 +
<pre>$ singularity pull [target-filename] docker://<source-group>/<source-name>[:<tag>]</pre>
 +
 
 +
===[https://singularity-hub.org/ Singularity Hub]===
 +
General-purpose public repository for singularity users.
 +
 
 +
Repository URLs in the form <nowiki>https://singularity-hub.org/collections/<collection-number></nowiki>
 +
 
 +
Each collection number corresponds to a [https://github.com/ github] repository of the form <source-group>/<source-name>
 +
<pre>$ singularity pull [target-filename] shub://<source-group>/<source-name>[:<tag>]</pre>
 +
 
 +
===[https://cloud.sylabs.io/library Singularity Library]===
 +
Curated repository of official builds of commonly-used software for singularity users.
 +
 
 +
Repositories are organized by <source-group>/<collection-name>/<source-name>
 +
<pre>$ singularity pull [target-filename] library://<source-group>/<collection-name>/<source-name>[:<tag>]</pre>
 +
Note that you can search the Singularity Library from the command line:
 +
<pre>$ singularity search <source-name></pre>
 +
 
 +
===[https://ngc.nvidia.com/catalog/containers NVIDIA GPU Cloud]===
 +
High performance builds of GPU-enabled software.
 +
 
 +
Repository URLs in the form <nowiki>https://ngc.nvidia.com/catalog/containers/<source-group>:<source-name></nowiki>
 +
<pre>$ singularity pull [target-filename] docker://nvcr.io/<source-group>/<source-name>[:<tag>]</pre>
 +
 
 +
===[https://quay.io/ Quay.io]===
 +
Similar to Dockerhub but with more features for developers.
 +
 
 +
Repository URLs of the form <nowiki>https://quay.io/repository/<source-group>/<source-name></nowiki>
 +
<pre>$ singularity pull [target-filename] docker://quay.io/<source-group>/<source-name>[:<tag>]</pre>
 +
 
 +
===[https://biocontainers.pro/ BioContainers]===
 +
Curated repository of scientific software for biologists.
 +
 
 +
Repository URLs of the form <nowiki>https://biocontainers.pro/tools/<source-name></nowiki>
 +
 
 +
Image files have a <tag> in the form <version>--<nonsense>
 +
<pre>$ singularity pull [target-filename] https://depot.galaxyproject.org/singularity/<source-name>:<tag></pre>
 +
 
 +
Biocontainers can also be found on Dockerhub and Quay.io (above).
 +
 
 +
===[https://hub.docker.com/r/ibmcom/powerai IBM PowerAI] (Traverse only)===
 +
Repository name ibmcom/powerai on dockerhub (see above).
 +
 
 +
== Prebuilt images ==
 +
HPRC provides (will provide) images using the steps below on ada/terra in /scratch/data/Singularity/images if you want to use a prebuilt image.  At present(/soon) they include.
 +
* Fedora28-HPRCLAB-40GB.img - a Fedora 28 image based on HPRC lab workstations with room to work
 +
** Note: this will not work on ada.  Various versions of Fedora from 23 to 28 were tried and all failed on ada due to the older kernel in CentOS6. 
 +
* CentOS6-XXGB.img - a CentOS 6 image populated with packages based on HPRCLAB workstations (where possible) with room to work
 +
* CentOS7-XXGB.img - a CentOS 7 image populated with packages based on HPRClab workstations (where possible) with room to work
 +
* lammps-stable_29Oct2020_ubuntu20.04_openmpi_py3.sif - an MPI-ready LAMMPS image from docker://lammps/lammps
 +
* tensorflow_2.4.1-gpu.sif - a GPU-ready TensorFlow image from docker://tensorflow/tensorflow
 +
* tensorflow_2.4.1-gpu-jupyter.sif - a GPU-ready TensorFlow image from docker://tensorflow/tensorflow
 +
* bcbio-nextgen-1.2.8.sif - a Biocontainer with RNA sequencing tools from docker://quay.io/biocontainers/bcbio-nextgen
 +
Also check out the README on disk in the same location for more information.
 +
 
 +
== Building your own images ==
 +
In order to build Singularity images yourself you will need a system with a modern Linux distribution (e.g. Ubuntu/Fedora/CentOS/etc) that you have sudo/root on.  HPRC users cannot do this on HPRC systems given the lack of sudo.
  
 
=== Fedora 28 ===
 
=== Fedora 28 ===
Note that the final image requires a kernel newer than the one provided by CentOS6.  This means it can not be used on ada (only on terra).  Fedora version 23-28 were attempted but they eventually failed on ada.  So if you plan to use a Fedora-based Singularity image you will need to use terra.
 
 
==== Objective ====
 
==== Objective ====
 
The objective in this case is a Fedora28-HPRCLAB.img file that has space for users to manuever around and install stuff:
 
The objective in this case is a Fedora28-HPRCLAB.img file that has space for users to manuever around and install stuff:
Line 11: Line 73:
 
# has plenty of disk space for the user to add files (as non-root) while on the cluster
 
# has plenty of disk space for the user to add files (as non-root) while on the cluster
 
# Example below was created on HPRCLAB workstations.
 
# Example below was created on HPRCLAB workstations.
==== Building an image on your Fedora 28 workstation ====
+
==== Building an image on your Linux workstation ====
===== Create a "sandbox" =====
+
===== Gather information for package installation =====
* start a root shell, but maintain location (you will need it for most of the next steps so might as well do it now to avoid all the password retypes for sudo)
+
* Before you start, create a list of RPMs (rpmlist) you want to install.  Also make a file (repoURLs) listing the URLs for the repo install RPMs.  Then place both in /root/forFedora28 on the host/build system. These will be used later while in the sandbox shell.  For Fedora 28 we will use the RPMFusion repos and the package list from current HPRC lab workstations.
 
<pre>
 
<pre>
sudo su
+
# cat /root/forFedora28/repoURLs
 +
https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-28.noarch.rpm
 +
https://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-28.noarch.rpm
 +
 
 +
# head -5 /root/forFedora28/rpmlist
 +
a2ps
 +
a52dec
 +
aajohan-comfortaa-fonts
 +
aalib-libs
 +
aalto-xml
 
</pre>
 
</pre>
 +
 +
===== Create/start a shell in a "sandbox" =====
 
* create a Singularity sandbox from one of Fedora's Docker images
 
* create a Singularity sandbox from one of Fedora's Docker images
 
<pre>
 
<pre>
singularity build --sandbox Fedora28-HPRCLAB.sandbox docker://fedora:28
+
sudo singularity build --sandbox Fedora28-HPRCLAB.sandbox docker://fedora:28
 
</pre>
 
</pre>
* create a list of packages to install into the image and place in /root.  In this case we use the HPRCLAB/Fedora28 workstations current list of software.  So on the workstation/host (not in the image):
+
* start a shell in the sandbox for the next steps
 
<pre>
 
<pre>
rpm -qa --qf "%{NAME}\n" | sort > /root/rpmlist
+
sudo singularity shell --writable Fedora28-HPRCLAB.sandbox/
 
</pre>
 
</pre>
which results in something that looks like this:
+
* go ahead and create bind points for HPRC cluster filesystems
 
<pre>
 
<pre>
[root@hprclab4 ~]# head -5 /root/rpmlist
+
mkdir /scratch /work
a2ps
 
a52dec
 
aajohan-comfortaa-fonts
 
aalib-libs
 
aalto-xml
 
</pre>
 
* while still in the host sudo/root shell update the image using the --writable shell in the sandbox (requires sudo for next steps)
 
<pre>
 
singularity shell --writable Fedora28-HPRCLAB.sandbox/
 
 
</pre>
 
</pre>
 +
* do '''NOT''' add users with useradd... if you do, be sure not to add your NetID or things will break on the HPRC clusters later.
 +
 +
===== Populate the sandbox with packages =====
 
* Install the RPMFUSION repos (needed for below)
 
* Install the RPMFUSION repos (needed for below)
 
<pre>
 
<pre>
dnf -y install\
+
dnf -y install `cat /root/forFedora28/repoURLs`
https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-28.noarch.rpm\
 
https://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-28.noarch.rpm
 
 
</pre>
 
</pre>
 
* populate the sandbox with all the RPMs (we do them in groups at first in case there a problems and then do all to see what those problems were. (get a snack, take a walk... it can take awhile)
 
* populate the sandbox with all the RPMs (we do them in groups at first in case there a problems and then do all to see what those problems were. (get a snack, take a walk... it can take awhile)
 
<pre>
 
<pre>
 
for x in {a..z} ; do
 
for x in {a..z} ; do
   dnf -y install `grep ^$x /root/rpmlist`
+
   dnf -y --skip-broken install `grep ^$x /root/forFedora28/rpmlist`
done
 
dnf -y install `cat /root/rpmlist`
 
</pre>
 
or maybe
 
<pre>
 
for x in `cat /root/rpmlist` ; do
 
  dnf -y install $x
 
 
done
 
done
 +
dnf -y --skip-broken install `grep ^[A-Z] root/forFedora28/rpmlist`
 +
dnf -y --skip-broken install `cat /root/forFedora28/rpmlist`
 
</pre>
 
</pre>
 
There will still be packages that have "No match" due to not including the repos for them.
 
There will still be packages that have "No match" due to not including the repos for them.
Line 90: Line 150:
 
</pre>
 
</pre>
  
==== Using your image on HPRC clusters ====
+
=== CentOS/Ubuntu/others ===
 
+
* the steps to build images using other Linux distributions are mostly the same though there are some notable changes
=== CentOS 6 ===
+
==== CentOS ====
==== Create a sanbox ====
+
* you'll need to use 'yum' instead of 'dnf'
* create/start sandbox
+
* you'll need to change the docker reference to something like: docker://centos:6
<pre>
+
* you'll want EPEL instead of RPMFusion. e.g.
sudo singularity build --sandbox CentOS6.sandbox docker://centos:6
 
sudo singularity shell --writable CentOS6.sandbox/
 
<pre>
 
* create bind points for HPRC cluster filesystems
 
<pre>
 
sudo mkdir /scratch /work
 
</pre>
 
* install repos for other RPMS
 
<pre>
 
yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm
 
</pre>
 
* populate sandbox with RPM installs
 
<pre>
 
for x in `cat /root/HPRCLAB-rpmlist` ; do yum -y install $x ; done
 
mkdir /scratch /work
 
exit
 
</pre>
 
* exit sandbox shell
 
 
<pre>
 
<pre>
exit
+
# cat /root/forCentOS6/repoURLs
 +
https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm
 
</pre>
 
</pre>
 +
* you'll have to develop your own rpmlist
 +
==== Ubuntu ====
 +
* you'll need to use 'apt-get' instead of 'dnf'
 +
* you'll need to change the docker reference to something like docker://ubuntu:18 (just a guess)
 +
* not sure you need extra 'repos'
 +
* rpmlist is up to you

Latest revision as of 14:44, 29 April 2021

Popular Repositories

When looking for containerized software, try these repositories:

Notation: [ ] means optional, < > means mandatory.

Docker Hub

Most popular general-purpose public repository. Docker is the most common container image type.

Repository URLs in the form https://hub.docker.com/r/<source-group>/<source-name>

$ singularity pull [target-filename] docker://<source-group>/<source-name>[:<tag>]

Singularity Hub

General-purpose public repository for singularity users.

Repository URLs in the form https://singularity-hub.org/collections/<collection-number>

Each collection number corresponds to a github repository of the form <source-group>/<source-name>

$ singularity pull [target-filename] shub://<source-group>/<source-name>[:<tag>]

Singularity Library

Curated repository of official builds of commonly-used software for singularity users.

Repositories are organized by <source-group>/<collection-name>/<source-name>

$ singularity pull [target-filename] library://<source-group>/<collection-name>/<source-name>[:<tag>]

Note that you can search the Singularity Library from the command line:

$ singularity search <source-name>

NVIDIA GPU Cloud

High performance builds of GPU-enabled software.

Repository URLs in the form https://ngc.nvidia.com/catalog/containers/<source-group>:<source-name>

$ singularity pull [target-filename] docker://nvcr.io/<source-group>/<source-name>[:<tag>]

Quay.io

Similar to Dockerhub but with more features for developers.

Repository URLs of the form https://quay.io/repository/<source-group>/<source-name>

$ singularity pull [target-filename] docker://quay.io/<source-group>/<source-name>[:<tag>]

BioContainers

Curated repository of scientific software for biologists.

Repository URLs of the form https://biocontainers.pro/tools/<source-name>

Image files have a <tag> in the form <version>--<nonsense>

$ singularity pull [target-filename] https://depot.galaxyproject.org/singularity/<source-name>:<tag>

Biocontainers can also be found on Dockerhub and Quay.io (above).

IBM PowerAI (Traverse only)

Repository name ibmcom/powerai on dockerhub (see above).

Prebuilt images

HPRC provides (will provide) images using the steps below on ada/terra in /scratch/data/Singularity/images if you want to use a prebuilt image. At present(/soon) they include.

  • Fedora28-HPRCLAB-40GB.img - a Fedora 28 image based on HPRC lab workstations with room to work
    • Note: this will not work on ada. Various versions of Fedora from 23 to 28 were tried and all failed on ada due to the older kernel in CentOS6.
  • CentOS6-XXGB.img - a CentOS 6 image populated with packages based on HPRCLAB workstations (where possible) with room to work
  • CentOS7-XXGB.img - a CentOS 7 image populated with packages based on HPRClab workstations (where possible) with room to work
  • lammps-stable_29Oct2020_ubuntu20.04_openmpi_py3.sif - an MPI-ready LAMMPS image from docker://lammps/lammps
  • tensorflow_2.4.1-gpu.sif - a GPU-ready TensorFlow image from docker://tensorflow/tensorflow
  • tensorflow_2.4.1-gpu-jupyter.sif - a GPU-ready TensorFlow image from docker://tensorflow/tensorflow
  • bcbio-nextgen-1.2.8.sif - a Biocontainer with RNA sequencing tools from docker://quay.io/biocontainers/bcbio-nextgen

Also check out the README on disk in the same location for more information.

Building your own images

In order to build Singularity images yourself you will need a system with a modern Linux distribution (e.g. Ubuntu/Fedora/CentOS/etc) that you have sudo/root on. HPRC users cannot do this on HPRC systems given the lack of sudo.

Fedora 28

Objective

The objective in this case is a Fedora28-HPRCLAB.img file that has space for users to manuever around and install stuff:

  1. that we can provide to users, a Singularity .img file (self-contained), for them to update as needed (as root on their own workstations) that also runs on our clusters.
  2. is already populated with HPRCLAB RPMs
  3. has plenty of disk space for the user to add files (as non-root) while on the cluster
  4. Example below was created on HPRCLAB workstations.

Building an image on your Linux workstation

Gather information for package installation
  • Before you start, create a list of RPMs (rpmlist) you want to install. Also make a file (repoURLs) listing the URLs for the repo install RPMs. Then place both in /root/forFedora28 on the host/build system. These will be used later while in the sandbox shell. For Fedora 28 we will use the RPMFusion repos and the package list from current HPRC lab workstations.
# cat /root/forFedora28/repoURLs 
https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-28.noarch.rpm
https://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-28.noarch.rpm

# head -5 /root/forFedora28/rpmlist 
a2ps
a52dec
aajohan-comfortaa-fonts
aalib-libs
aalto-xml
Create/start a shell in a "sandbox"
  • create a Singularity sandbox from one of Fedora's Docker images
sudo singularity build --sandbox Fedora28-HPRCLAB.sandbox docker://fedora:28
  • start a shell in the sandbox for the next steps
sudo singularity shell --writable Fedora28-HPRCLAB.sandbox/
  • go ahead and create bind points for HPRC cluster filesystems
mkdir /scratch /work
  • do NOT add users with useradd... if you do, be sure not to add your NetID or things will break on the HPRC clusters later.
Populate the sandbox with packages
  • Install the RPMFUSION repos (needed for below)
dnf -y install `cat /root/forFedora28/repoURLs`
  • populate the sandbox with all the RPMs (we do them in groups at first in case there a problems and then do all to see what those problems were. (get a snack, take a walk... it can take awhile)
for x in {a..z} ; do
  dnf -y --skip-broken install `grep ^$x /root/forFedora28/rpmlist`
done
dnf -y --skip-broken install `grep ^[A-Z] root/forFedora28/rpmlist`
dnf -y --skip-broken install `cat /root/forFedora28/rpmlist`

There will still be packages that have "No match" due to not including the repos for them.

  • do NOT add users with useradd... if you do, be sure not to add your NetID or things will break.
  • create bind directories for interfacing with HPRC clusters (Note: have to do this now while still in sudo shell)
mkdir /scratch /work
  • once you are happy with the sandbox log out from it (logout/exit/Ctrl-D).
Create an image (.img)
  • to make an .img file that provides space for the user to work in (e.g. download/install software). These steps do not require sudo since they don't really create anything in the image.
  • first, determine the installed size of the sandbox
* Given that, add some space for the user to work in.  In this case, we will use 40GB (approx. sandbox size + 15G).
<pre>
singularity image.create Fedora28-HPRCLAB-40GB.img # create .img file
singularity image.expand -s 40960 Fedora28-HPRCLAB-40GB.img # expand to 40GB, get another snack while you wait

Note: works because there is no real filesystem yet [some ext3/4 filesystems seem to have a problem with expand]

  • Now import the sandbox into the 40GB image (more snacks/time)
sudo tar -cvf - -C Fedora28-HPRCLAB.sandbox/ . | sudo singularity image.import Fedora28-HPRCLAB-40GB.img

(see the last/tar example in 'singularity image.import --help')

Test on host build system
  • open a non-sudo shell and see if you can do "things"
singularity shell Fedora28-HPRCLAB-40GB.img

CentOS/Ubuntu/others

  • the steps to build images using other Linux distributions are mostly the same though there are some notable changes

CentOS

  • you'll need to use 'yum' instead of 'dnf'
  • you'll need to change the docker reference to something like: docker://centos:6
  • you'll want EPEL instead of RPMFusion. e.g.
# cat /root/forCentOS6/repoURLs 
https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm
  • you'll have to develop your own rpmlist

Ubuntu

  • you'll need to use 'apt-get' instead of 'dnf'
  • you'll need to change the docker reference to something like docker://ubuntu:18 (just a guess)
  • not sure you need extra 'repos'
  • rpmlist is up to you