Hprc banner tamu.png

Difference between revisions of "Talk:SW:Singularity"

From TAMU HPRC
Jump to: navigation, search
(Latest)
(Replaced content with "See SW:Singularity:Examples")
 
(40 intermediate revisions by the same user not shown)
Line 1: Line 1:
== j-perdue random links for 8/10/2018 ==
+
See [[SW:Singularity:Examples]]
=== Placeholder ===
 
To be added to SW:Singlularity:Examples or something.... just a placeholder for now to dump all my URLs at home for access at work:
 
 
 
https://github.com/singularityware/singularity/issues/1076
 
 
 
https://github.com/singularityware/singularity/issues/1165
 
 
 
http://singularity.lbl.gov/quickstart#build-images-from-scratch
 
 
 
https://hub.docker.com/_/fedora/
 
 
 
https://github.com/NIH-HPC/singularity-examples/blob/master/torch/Singularity.torch
 
 
 
https://www.sylabs.io/docs/
 
 
 
=== history ===
 
 
 
* random notes from .bash_history on hprclab2
 
 
 
<pre>
 
[j-perdue@hprclab2 ~]$ history | grep docker
 
  460  git clone https://github.com/fedora-cloud/docker-brew-fedora.git
 
  466  less prep-docker-brew-branch.sh
 
  468  ./prep-docker-brew-branch.sh 28
 
  470  sudo docker pull fedora:28
 
  472  sudo docker run --rm -ti fedora:28 bash
 
  484  sudo docker run  -ti fedora:28 bash
 
  494  sudo docker run  -ti fedora:28 bash
 
  495  man docker
 
  496  man docker-run
 
  497  sudo docker build --writable Fedora28.img fedora:28
 
  515  sudo singularity build --writable FedoraLatest.img docker://fedora:latest
 
  516  sudo singularity build --writable Fedora28.img docker://fedora:28
 
  523  # sudo singularity build --writable --size Fedora28.img docker://fedora:28
 
  541  history | grep docker
 
[j-perdue@hprclab2 ~]$ history | grep singularit
 
  510  sudo singularity build --writable Fedora28.img fedora:28
 
  511  sudo singularity build --writable Fedora28.img fedora:27
 
  512  sudo singularity build --writable Fedora28.img
 
  XXX  sudo singularity shell --writable Fedora28-HPRCLAB.img
 
</pre>
 
 
 
=== nearing in on objective ===
 
 
 
The objective in this case is a Fedora28-HPRCLAB.img file that has space for users to manuever around and install stuff:
 
# that we can provide to users, a Singularity .img file (self-contained), for them to update as needed (as root on their own workstations) that also runs on our clusters.
 
# is already populated with HPRCLAB RPMs
 
# has plenty of disk space for the user to add files (as non-root) while on the cluster
 
# Example below was created on HPRCLAB workstations.
 
 
 
==== Latest ====
 
 
 
* create a Singularity sandbox from one of Fedora's Docker images
 
<pre>
 
sudo singularity build --sandbox Fedora28-HPRCLAB.sandbox docker://fedora:28
 
</pre>
 
* create a list of packages to install into the image and place in /root.  In this case we use the HPRCLAB/Fedora28 workstations current list of software.  So on the workstation (not in the image):
 
<pre>
 
sudo su - # need root for redirect
 
rpm -qa --qf "%{NAME}\n" | sort > /root/rpmlist
 
</pre>
 
which results in something that looks like this:
 
<pre>
 
[root@hprclab4 ~]# head /root/rpmlist
 
a2ps
 
a52dec
 
aajohan-comfortaa-fonts
 
aalib-libs
 
aalto-xml
 
abattis-cantarell-fonts
 
abrt-addon-coredump-helper
 
abrt-gui-libs
 
abrt-libs
 
accountsservice
 
</pre>
 
* open a sudo --writable shell in the sandbox (requires sudo for next steps)
 
<pre>
 
sudo singularity shell --writable Fedora28-HPRCLAB.sandbox/
 
</pre>
 
* populate the sandbox with all the RPMs (we do them in groups at first in case there a problems and then do all to see what those problems were. (get a snack, take a walk... it can take awhile)
 
<pre>
 
for x in {a..z} ; do
 
  dnf -y install `grep ^$x /root/rpmlist`
 
done
 
</pre>
 
* add users
 
<pre>
 
useradd mynetid
 
</pre>
 
* once you are happy with that, time to make an .img
 
* singularity image.create my20Gimage.img # create .img file
 
* singularity image.expand -s 20480 my20Gimage.img # expand to 20GB (works because there is no real file system yet)
 
*  sudo tar -cvf - -C mysandbox/ . | sudo singularity image.import my20Gimage.img  # see the last/tar example in 'singularity image.import --help'
 

Latest revision as of 13:24, 14 August 2018