|
|
| Line 51: |
Line 51: |
| | # setting mount/bind points for HPRC clusters | | # setting mount/bind points for HPRC clusters |
| | ==== Latest ==== | | ==== Latest ==== |
| − | ===== Create a "sandbox" =====
| + | Now at [[SW:Singlularity:Examples]] |
| − | * start a root shell, but maintain location (you will need it for most of the next steps so might as well do it now to avoid all the password retypes for sudo)
| |
| − | <pre>
| |
| − | sudo su
| |
| − | </pre>
| |
| − | * create a Singularity sandbox from one of Fedora's Docker images
| |
| − | <pre>
| |
| − | singularity build --sandbox Fedora28-HPRCLAB.sandbox docker://fedora:28
| |
| − | </pre>
| |
| − | * create a list of packages to install into the image and place in /root. In this case we use the HPRCLAB/Fedora28 workstations current list of software. So on the workstation/host (not in the image):
| |
| − | <pre>
| |
| − | rpm -qa --qf "%{NAME}\n" | sort > /root/rpmlist
| |
| − | </pre>
| |
| − | which results in something that looks like this:
| |
| − | <pre>
| |
| − | [root@hprclab4 ~]# head -5 /root/rpmlist
| |
| − | a2ps
| |
| − | a52dec
| |
| − | aajohan-comfortaa-fonts
| |
| − | aalib-libs
| |
| − | aalto-xml
| |
| − | </pre>
| |
| − | * while still in the host sudo/root shell update the image using the --writable shell in the sandbox (requires sudo for next steps)
| |
| − | <pre>
| |
| − | singularity shell --writable Fedora28-HPRCLAB.sandbox/
| |
| − | </pre>
| |
| − | * Install the RPMFUSION repos (needed for below)
| |
| − | <pre>
| |
| − | dnf -y install\
| |
| − | https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-28.noarch.rpm\
| |
| − | https://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-28.noarch.rpm
| |
| − | </pre>
| |
| − | * populate the sandbox with all the RPMs (we do them in groups at first in case there a problems and then do all to see what those problems were. (get a snack, take a walk... it can take awhile)
| |
| − | <pre>
| |
| − | for x in {a..z} ; do
| |
| − | dnf -y install `grep ^$x /root/rpmlist`
| |
| − | done
| |
| − | dnf -y install `cat /root/rpmlist`
| |
| − | </pre>
| |
| − | or maybe
| |
| − | <pre>
| |
| − | for x in `cat /root/rpmlist` ; do
| |
| − | dnf -y install $x
| |
| − | done
| |
| − | </pre>
| |
| − | There will still be packages that have "No match" due to not including the repos for them.
| |
| − | * do '''NOT''' add users with useradd... if you do, be sure not to add your NetID or things will break.
| |
| − | * create bind directories for interfacing with HPRC clusters (Note: have to do this now while still in sudo shell)
| |
| − | <pre>
| |
| − | mkdir /scratch /work
| |
| − | </pre>
| |
| − | * once you are happy with the sandbox log out from it (logout/exit/Ctrl-D).
| |
| | | | |
| − | ===== Create an image (.img) =====
| |
| − | * To make an .img file that provides space for the user to work in (e.g. download/install software). These steps do not require sudo since they don't really create anything in the image. This example is for a 20GB /.
| |
| − | <pre>
| |
| − | singularity image.create Fedora28-HPRCLAB.img # create .img file
| |
| − | singularity image.expand -s 20480 Fedora28-HPRCLAB.img # expand to 20GB, get another snack while you wait
| |
| − | </pre>
| |
| − | Note: works because there is no real filesystem yet [some ext3/4 filesystems seem to have a problem with expand]
| |
| − | * Now import the sandbox into the 20GB image (more snacks/time)
| |
| − | <pre>
| |
| − | sudo tar -cvf - -C Fedora28-HPRCLAB.sandbox/ . | sudo singularity image.import Fedora28-HPRCLAB.img
| |
| − | </pre>
| |
| − | (see the last/tar example in 'singularity image.import --help')
| |
| − | ===== Test on host build system =====
| |
| − | * open a non-sudo shell and see if you can do "things"
| |
| − | <pre>
| |
| − | singularity shell Fedora28-HPRCLAB.img
| |
| − | </pre>
| |
| | ==== Copy to HPRC cluster ==== | | ==== Copy to HPRC cluster ==== |
j-perdue random links for 8/10/2018
Placeholder
To be added to SW:Singlularity:Examples or something.... just a placeholder for now to dump all my URLs at home for access at work:
https://github.com/singularityware/singularity/issues/1076
https://github.com/singularityware/singularity/issues/1165
http://singularity.lbl.gov/quickstart#build-images-from-scratch
https://hub.docker.com/_/fedora/
https://github.com/NIH-HPC/singularity-examples/blob/master/torch/Singularity.torch
https://www.sylabs.io/docs/
history
- random notes from .bash_history on hprclab2
[j-perdue@hprclab2 ~]$ history | grep docker
460 git clone https://github.com/fedora-cloud/docker-brew-fedora.git
466 less prep-docker-brew-branch.sh
468 ./prep-docker-brew-branch.sh 28
470 sudo docker pull fedora:28
472 sudo docker run --rm -ti fedora:28 bash
484 sudo docker run -ti fedora:28 bash
494 sudo docker run -ti fedora:28 bash
495 man docker
496 man docker-run
497 sudo docker build --writable Fedora28.img fedora:28
515 sudo singularity build --writable FedoraLatest.img docker://fedora:latest
516 sudo singularity build --writable Fedora28.img docker://fedora:28
523 # sudo singularity build --writable --size Fedora28.img docker://fedora:28
541 history | grep docker
[j-perdue@hprclab2 ~]$ history | grep singularit
510 sudo singularity build --writable Fedora28.img fedora:28
511 sudo singularity build --writable Fedora28.img fedora:27
512 sudo singularity build --writable Fedora28.img
XXX sudo singularity shell --writable Fedora28-HPRCLAB.img
nearing in on objective
The objective in this case is a Fedora28-HPRCLAB.img file that has space for users to manuever around and install stuff:
- that we can provide to users, a Singularity .img file (self-contained), for them to update as needed (as root on their own workstations) that also runs on our clusters.
- is already populated with HPRCLAB RPMs
- has plenty of disk space for the user to add files (as non-root) while on the cluster
- Example below was created on HPRCLAB workstations.
Things not included yet include:
- setting mount/bind points for HPRC clusters
Latest
Now at SW:Singlularity:Examples
Copy to HPRC cluster
