Talk:SW:Singularity

j-perdue random links for 8/10/2018

Placeholder

To be added to SW:Singlularity:Examples or something.... just a placeholder for now to dump all my URLs at home for access at work:

https://github.com/singularityware/singularity/issues/1076

https://github.com/singularityware/singularity/issues/1165

http://singularity.lbl.gov/quickstart#build-images-from-scratch

https://hub.docker.com/_/fedora/

https://github.com/NIH-HPC/singularity-examples/blob/master/torch/Singularity.torch

https://www.sylabs.io/docs/

history

• random notes from .bash_history on hprclab2

[j-perdue@hprclab2 ~]$ history | grep docker
  460  git clone https://github.com/fedora-cloud/docker-brew-fedora.git
  466  less prep-docker-brew-branch.sh 
  468  ./prep-docker-brew-branch.sh 28
  470  sudo docker pull fedora:28
  472  sudo docker run --rm -ti fedora:28 bash
  484  sudo docker run  -ti fedora:28 bash
  494  sudo docker run  -ti fedora:28 bash
  495  man docker
  496  man docker-run
  497  sudo docker build --writable Fedora28.img fedora:28 
  515  sudo singularity build --writable FedoraLatest.img docker://fedora:latest
  516  sudo singularity build --writable Fedora28.img docker://fedora:28
  523  # sudo singularity build --writable --size Fedora28.img docker://fedora:28
  541  history | grep docker
[j-perdue@hprclab2 ~]$ history | grep singularit
  510  sudo singularity build --writable Fedora28.img fedora:28 
  511  sudo singularity build --writable Fedora28.img fedora:27
  512  sudo singularity build --writable Fedora28.img 
  XXX  sudo singularity shell --writable Fedora28-HPRCLAB.img

nearing in on objective

The objective in this case is a Fedora28-HPRCLAB.img file that has space for users to manuever around and install stuff:

1. that we can provide to users, a Singularity .img file (self-contained), for them to update as needed (as root on their own workstations) that also runs on our clusters.
2. is already populated with HPRCLAB RPMs
3. has plenty of disk space for the user to add files (as non-root) while on the cluster
4. Example below was created on HPRCLAB workstations.

Things not included yet include:

1. setting mount/bind points for HPRC clusters

Latest

• start a root shell, but maintain location (you will need it for most of the next steps so might as well do it now to avoid all the password retypes for sudo)

sudo su

• create a Singularity sandbox from one of Fedora's Docker images

singularity build --sandbox Fedora28-HPRCLAB.sandbox docker://fedora:28

• create a list of packages to install into the image and place in /root. In this case we use the HPRCLAB/Fedora28 workstations current list of software. So on the workstation/host (not in the image):

rpm -qa --qf "%{NAME}\n" | sort > /root/rpmlist

which results in something that looks like this:

[root@hprclab4 ~]# head -5 /root/rpmlist 
a2ps
a52dec
aajohan-comfortaa-fonts
aalib-libs
aalto-xml

• while still in the host sudo/root shell update the image using the --writable shell in the sandbox (requires sudo for next steps)

singularity shell --writable Fedora28-HPRCLAB.sandbox/

• Install the RPMFUSION repos (needed for below)

dnf -y install\
 https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-28.noarch.rpm\
 https://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-28.noarch.rpm

• populate the sandbox with all the RPMs (we do them in groups at first in case there a problems and then do all to see what those problems were. (get a snack, take a walk... it can take awhile)

for x in {a..z} ; do
  dnf -y install `grep ^$x /root/rpmlist`
done
dnf -y install `cat /root/rpmlist`

or maybe

for x in `cat /root/rpmlist` ; do
  dnf -y install $x
done

There will still be packages that have "No match" due to not including the repos for them.

• do NOT add users with useradd... if you do, be sure not to add your NetID or things will break.

useradd mynetid 

• once you are happy with the sandbox log out from it (logout/exit/Ctrl-D).
• To make an .img file that provides space for the user to work in (e.g. download/install software). These steps do not require sudo since they don't really create anything in the image. This example is for a 20GB /.

singularity image.create Fedora28-HPRCLAB.img # create .img file
singularity image.expand -s 20480 Fedora28-HPRCLAB.img # expand to 20GB, get another snack while you wait

Note: works because there is no real filesystem yet [some ext3/4 filesystems seem to have a problem with expand]

• Now import the sandbox into the 20GB image (more snacks/time)

sudo tar -cvf - -C Fedora28-HPRCLAB.sandbox/ . | sudo singularity image.import Fedora28-HPRCLAB.img

(see the last/tar example in 'singularity image.import --help')

• open a non-sudo shell and see if you can do "things"

singularity shell Fedora28-HPRCLAB.img