Hprc banner tamu.png

Difference between revisions of "Talk:SW:Singularity"

From TAMU HPRC
Jump to: navigation, search
(Latest)
(Replaced content with "See SW:Singularity:Examples")
 
(15 intermediate revisions by the same user not shown)
Line 1: Line 1:
== j-perdue random links for 8/10/2018 ==
+
See [[SW:Singularity:Examples]]
=== Placeholder ===
 
To be added to SW:Singlularity:Examples or something.... just a placeholder for now to dump all my URLs at home for access at work:
 
 
 
https://github.com/singularityware/singularity/issues/1076
 
 
 
https://github.com/singularityware/singularity/issues/1165
 
 
 
http://singularity.lbl.gov/quickstart#build-images-from-scratch
 
 
 
https://hub.docker.com/_/fedora/
 
 
 
https://github.com/NIH-HPC/singularity-examples/blob/master/torch/Singularity.torch
 
 
 
https://www.sylabs.io/docs/
 
 
 
=== history ===
 
 
 
* random notes from .bash_history on hprclab2
 
 
 
<pre>
 
[j-perdue@hprclab2 ~]$ history | grep docker
 
  460  git clone https://github.com/fedora-cloud/docker-brew-fedora.git
 
  466  less prep-docker-brew-branch.sh
 
  468  ./prep-docker-brew-branch.sh 28
 
  470  sudo docker pull fedora:28
 
  472  sudo docker run --rm -ti fedora:28 bash
 
  484  sudo docker run  -ti fedora:28 bash
 
  494  sudo docker run  -ti fedora:28 bash
 
  495  man docker
 
  496  man docker-run
 
  497  sudo docker build --writable Fedora28.img fedora:28
 
  515  sudo singularity build --writable FedoraLatest.img docker://fedora:latest
 
  516  sudo singularity build --writable Fedora28.img docker://fedora:28
 
  523  # sudo singularity build --writable --size Fedora28.img docker://fedora:28
 
  541  history | grep docker
 
[j-perdue@hprclab2 ~]$ history | grep singularit
 
  510  sudo singularity build --writable Fedora28.img fedora:28
 
  511  sudo singularity build --writable Fedora28.img fedora:27
 
  512  sudo singularity build --writable Fedora28.img
 
  XXX  sudo singularity shell --writable Fedora28-HPRCLAB.img
 
</pre>
 
 
 
=== nearing in on objective ===
 
The objective in this case is a Fedora28-HPRCLAB.img file that has space for users to manuever around and install stuff:
 
# that we can provide to users, a Singularity .img file (self-contained), for them to update as needed (as root on their own workstations) that also runs on our clusters.
 
# is already populated with HPRCLAB RPMs
 
# has plenty of disk space for the user to add files (as non-root) while on the cluster
 
# Example below was created on HPRCLAB workstations.
 
Things not included yet include:
 
# setting mount/bind points for HPRC clusters
 
==== Latest ====
 
 
 
* create a Singularity sandbox from one of Fedora's Docker images
 
<pre>
 
sudo singularity build --sandbox Fedora28-HPRCLAB.sandbox docker://fedora:28
 
</pre>
 
* create a list of packages to install into the image and place in /root.  In this case we use the HPRCLAB/Fedora28 workstations current list of software.  So on the workstation/host (not in the image):
 
<pre>
 
sudo su - # need root for redirect and other stuff below
 
rpm -qa --qf "%{NAME}\n" | sort > /root/rpmlist
 
</pre>
 
which results in something that looks like this:
 
<pre>
 
[root@hprclab4 ~]# head -5 /root/rpmlist
 
a2ps
 
a52dec
 
aajohan-comfortaa-fonts
 
aalib-libs
 
aalto-xml
 
</pre>
 
* while still in the host sudo/root shell update the image using the --writable shell in the sandbox (requires sudo for next steps)
 
<pre>
 
singularity shell --writable Fedora28-HPRCLAB.sandbox/
 
</pre>
 
* Install the RPMFUSION repos (needed for below)
 
<pre>
 
dnf -y install\
 
https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-28.noarch.rpm\
 
https://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-28.noarch.rpm
 
</pre>
 
* populate the sandbox with all the RPMs (we do them in groups at first in case there a problems and then do all to see what those problems were. (get a snack, take a walk... it can take awhile)
 
<pre>
 
for x in {a..z} ; do
 
  dnf -y install `grep ^$x /root/rpmlist`
 
done
 
dnf -y install `cat /root/rpmlist`
 
</pre>
 
or maybe
 
<pre>
 
for x in `cat /root/rpmlist` ; do
 
  dnf -y install $x
 
done
 
</pre>
 
There will still be packages that have "No match" due to not including the repos for them.
 
* add users (best done after populating with software in case anything added something to /etc/skel) '''ACTUALLY THIS LOOKS TO BE A BAD IDEA... SKIP FOR NOW'''
 
<pre>
 
useradd mynetid
 
</pre>
 
* once you are happy with the sandbox log out from it (logout/exit/Ctrl-D).
 
* To make an .img file that provides space for the user to work in (e.g. download/install software).  These steps do not require sudo since they don't really create anything in the image.  This example is for a 20GB /.
 
<pre>
 
singularity image.create Fedora28-HPRCLAB.img # create .img file
 
singularity image.expand -s 20480 Fedora28-HPRCLAB.img # expand to 20GB (works because there is no real filesystem yet [some ext3/4 systems seem to have a problem with expand once the image has filesystems?]... also, get another snack while you wait)
 
</pre>
 
* Now import the sandbox into the 20GB image (more snacks/time)
 
<pre>
 
sudo tar -cvf - -C Fedora28-HPRCLAB.sandbox/ . | sudo singularity image.import Fedora28-HPRCLAB.img # see the last/tar example in 'singularity image.import --help'
 
</pre>
 
* open a non-sudo shell and see if you can do "things" '''GIVING ME PROBLEMS AT THE MOMENT... I THINK DUE TO TO THE useradd ABOVE'''
 
<pre>
 
singularity shell Fedora28-HPRCLAB.img
 
</pre>
 

Latest revision as of 13:24, 14 August 2018