Singularity: Running Containers on ACCRE
Singularity is a tool that allows containers to be run within a shared high-performance computing environment. This enables users to control the OS environment (including software libraries) that their jobs run within. For example, if a user wishes to run within an Ubuntu 16.04 environment he or she may do so despite the fact that the OS on the ACCRE cluster is a completely different Linux distribution (i.e. CentOS)! Docker containers themselves cannot be run in a shared environment like the ACCRE cluster for security reasons. However, Docker container images can easily be run as Singularity container images, which are safe to run on the cluster.
When running within a Singularity container, a user has the same permissions and privileges that he or she would have outside the container. A Singularity image generally must first be developed and built from a Linux machine where you have administrative access (i.e. a personal machine), although ACCRE makes standard images available to all cluster users at
/scratch/singularity-images . If you do not have administrative access to a Linux machine, you can create a virtual Linux machine using a free tool like Vagrant or VirtualBox. Singularity has some helpful documentation for how to do this on a Mac, Windows, or Linux machine.
A user’s cluster storage may be accessed from within the Singularity container, but no operations (e.g. the installation of system software) that require root/sudo privileges are allowed within the context of the Singularity container when run from the cluster. If you are interested in using Singularity but need assistance creating a custom image to run on the cluster, please schedule an appointment via our Helpdesk. Below are some basic instructions for running Singularity on the cluster. The Singularity documentation is very helpful, so we suggest you invest some time reading through it as well.
ACCRE also hosted a seminar on Singularity in Summer 2017. The recording can be found here. Some of the information in this presentation is out of date as new features have been added to Singularity since this recording, but the same principles apply.
Building a Singularity Image
Once you have installed Singularity on your own Linux machine or virtual machine , you are ready to create your image. First, create a spec file called Singularity that looks like the following:
BootStrap: docker From: ubuntu:16.04 %runscript echo "This is what happens when you run the container..." %post apt-get update apt-get -y install python3 python3-numpy python3-scipy # install any other software you need here... mkdir /scratch /data /gpfs22 /gpfs23
In this file, we are telling Singularity we want to build an image based on the latest version of Ubuntu 16.04. The base image is pulled from Docker Hub (see below). The %runscript section is for defining commands or tasks that you want to run each time you run a container of this image. The %post section contains one-time setup steps that you want to be inside the image. This is where you install your custom software. In this case, we use
apt-get to install Python 3 with NumPy and SciPy included. Finally, it’s important if you want to access all your cluster space from within the container to create the following directories from within the container: /scratch, /data, /gpfs22, and /gpfs23. If you don’t make these directories within the container (inside the %post section), you will get a warning message when you run a container of this image on the cluster. In particular, creating /gpfs22 will ensure you have access to all your /home space from within a container run on the cluster. To build the image, run the following command on your Linux machine where you have admin rights:
# on your personal Linux machine sudo singularity build ubuntu16-accre.simg Singularity
Building Containers Directly from Docker Hub or Singularity Hub
Often it is not necessary to create your own image because someone else has already created one for you! Docker Hub and Singularity Hub contain registries of images that have been contributed by various users. In fact, the first example above pulls its base image from Docker Hub. If you have found an image that you would like to run directly, you have the option of building it locally or running it directly.
Building the image locally is a good option if you intend to run it as a container multiple times. In this way you don’t need to re-download the image from the web repeatedly (which can be slow). Here is an example of how you might build a local version of an image from Docker Hub:
sudo singularity build ubuntu16-accre.simg docker://library/ubuntu:16.04
Note you can also use the
singularity pull subcommand to accomplish the same thing:
singularity pull docker://library/ubuntu:16.04
If you need to modify the container in any way (e.g. adding access to ACCRE storage), we recommend creating a spec file as shown above.
Running a Singularity Image on the Cluster
Once you have successfully created your image, you can shell into it with the following command:
# From the cluster singularity shell ubuntu16-accre.simg
Note: In the past users had to load Singularity using an Lmod command; as of the upgrade of the cluster to CentOS 7, this is no longer needed.
The shell subcommand is useful for interactive work. Note that you can create new data inside the container that will persist outside the context of the container. For batch processing (e.g. within a SLURM job), you should use the exec subcommand instead. For example:
singularity exec ubuntu16-accre.simg /path/to/my/script.sh
Where script.sh script contains the processing steps you want to run from within the batch job. You can also pass a generic Linux command to the exec subcommand. Pipes and redirected output are supported with the exec subcommand. Below is a quick demonstration showing the change in Linux distribution:
[jill@vmps10 ~]$ cat /etc/*release CentOS release 6.9 (Final) LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch CentOS release 6.9 (Final) CentOS release 6.9 (Final)
[jill@vmps10 ~]$ singularity shell ubuntu16-accre.img Singularity: Invoking an interactive shell within container... Singularity ubuntu-16.04.simg:~> cat /etc/*release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=16.04 DISTRIB_CODENAME=xenial DISTRIB_DESCRIPTION="Ubuntu 16.04.4 LTS" NAME="Ubuntu" VERSION="16.04.4 LTS (Xenial Xerus)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 16.04.4 LTS" VERSION_ID="16.04" HOME_URL="http://www.ubuntu.com/" SUPPORT_URL="http://help.ubuntu.com/" BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" VERSION_CODENAME=xenial UBUNTU_CODENAME=xenial Singularity ubuntu-16.04.simg:~> exit exit [jill@vmps10 ~]$
Running GPU-Enabled Containers
Singularity supports running on GPUs from within containers. The
--nv option must be passed to the exec or shell subcommand. Here is an example of a spec file for generating a GPU-enabled Tensorflow/Keras image to be run on ACCRE:
BootStrap: docker From: tensorflow/tensorflow:latest-gpu-py3 %help Singularity image with TensorFlow 1.7 and Keras running on python 3.5 with GPU support. %post export LC_ALL=C export PATH=/bin:/sbin:/usr/bin:/usr/sbin:$PATH pip3 install -U keras mkdir /scratch /data /gpfs22 /gpfs23
The instantiation of a container from such an image would look like:
$ ml GCC Singularity $ singularity exec --nv my-singularity-image.simg python -c "import tensorflow"
Alternatively, you could run a GPU-enabled image directly from Docker Hub. Here is an example SLURM script for running a GPU-enabled Tensorflow container on ACCRE GPUs:
#!/bin/bash #SBATCH --account=MY_GPU_ACCOUNT #SBATCH --partition=pascal #SBATCH --gres=gpu:1 #SBATCH --mem=10G #SBATCH --time=12:00:00 # Clone tensorflow repo to run a tutorial git clone https://github.com/tensorflow/models.git singularity exec --nv docker://tensorflow/tensorflow:latest-gpu \ python ./models/tutorials/image/mnist/convolutional.py
ACCRE Singularity GitHub Repository
ACCRE has a Singularity GitHub repository if you would like to see more examples. We also encourage users to contribute their own examples to the repo!