Hprc banner tamu.png

Difference between revisions of "Talk:SW:Singularity"

Jump to: navigation, search
(Replaced content with "See SW:Singularity:Examples")
(19 intermediate revisions by the same user not shown)
Line 1: Line 1:
== j-perdue random links for 8/10/2018 ==
See [[SW:Singularity:Examples]]
=== Placeholder ===
To be added to SW:Singlularity:Examples or something.... just a placeholder for now to dump all my URLs at home for access at work:
=== history ===
* random notes from .bash_history on hprclab2
[j-perdue@hprclab2 ~]$ history | grep docker
  460  git clone https://github.com/fedora-cloud/docker-brew-fedora.git
  466  less prep-docker-brew-branch.sh
  468  ./prep-docker-brew-branch.sh 28
  470  sudo docker pull fedora:28
  472  sudo docker run --rm -ti fedora:28 bash
  484  sudo docker run  -ti fedora:28 bash
  494  sudo docker run  -ti fedora:28 bash
  495  man docker
  496  man docker-run
  497  sudo docker build --writable Fedora28.img fedora:28
  515  sudo singularity build --writable FedoraLatest.img docker://fedora:latest
  516  sudo singularity build --writable Fedora28.img docker://fedora:28
  523  # sudo singularity build --writable --size Fedora28.img docker://fedora:28
  541  history | grep docker
[j-perdue@hprclab2 ~]$ history | grep singularit
  510  sudo singularity build --writable Fedora28.img fedora:28
  511  sudo singularity build --writable Fedora28.img fedora:27
  512  sudo singularity build --writable Fedora28.img
  XXX  sudo singularity shell --writable Fedora28-HPRCLAB.img
=== nearing in on objective ===
The objective in this case is a Fedora28-HPRCLAB.img file that has space for users to manuever around and install stuff:
# that we can provide to users, a Singularity .img file (self-contained), for them to update as needed (as root on their own workstations) that also runs on our clusters.
# is already populated with HPRCLAB RPMs
# has plenty of disk space for the user to add files (as non-root) while on the cluster
# Example below was created on HPRCLAB workstations.
Things not included yet include:
# setting mount/bind points for HPRC clusters
==== Latest ====
* create a Singularity sandbox from one of Fedora's Docker images
sudo singularity build --sandbox Fedora28-HPRCLAB.sandbox docker://fedora:28
* create a list of packages to install into the image and place in /root.  In this case we use the HPRCLAB/Fedora28 workstations current list of software.  So on the workstation/host (not in the image):
sudo su - # need root for redirect
rpm -qa --qf "%{NAME}\n" | sort > /root/rpmlist
which results in something that looks like this:
[root@hprclab4 ~]# head -5 rpmlist
Log out of root shell with exit/logout/Ctrl-D.
* open a sudo --writable shell in the sandbox (requires sudo for next steps)
sudo singularity shell --writable Fedora28-HPRCLAB.sandbox/
* Install the RPMFUSION repos (needed for below)
dnf -y install\
* populate the sandbox with all the RPMs (we do them in groups at first in case there a problems and then do all to see what those problems were. (get a snack, take a walk... it can take awhile)
for x in {a..z} ; do
  dnf -y install `grep ^$x /root/rpmlist`
dnf -y install `cat /root/rpmlist`
or maybe
for x in `cat /root/rpmlist` ; do
  dnf -y install $x
There will still be packages that have "No match" due to not including the repos for them.
* add users (best done after populating with software in case anything added something to /etc/skel) '''ACTUALLY THIS LOOKS TO BE A BAD IDEA... SKIP FOR NOW'''
useradd mynetid
* once you are happy with the sandbox log out from it (logout/exit/Ctrl-D).
* To make an .img file that provides space for the user to work in (e.g. download/install software).  These steps do not require sudo since they don't really create anything in the image.  This example is for a 20GB /.
singularity image.create Fedora28-HPRCLAB.img # create .img file
singularity image.expand -s 20480 Fedora28-HPRCLAB.img # expand to 20GB (works because there is no real filesystem yet [some ext3/4 systems seem to have a problem with expand once the image has filesystems?]... also, get another snack while you wait)
* Now import the sandbox into the 20GB image (more snacks/time)
sudo tar -cvf - -C Fedora28-HPRCLAB.sandbox/ . | sudo singularity image.import Fedora28-HPRCLAB.img # see the last/tar example in 'singularity image.import --help'
* open a non-sudo shell and see if you can do "things" '''GIVING ME PROBLEMS AT THE MOMENT... I THINK DUE TO TO THE useradd ABOVE'''
singularity shell Fedora28-HPRCLAB.img

Latest revision as of 13:24, 14 August 2018