Docker

Introduction

Docker is a company producing an open-source "software container platform", but the word is also used is also used to refer to the platform itself. Docker allows a user to collect all the code needed to run an application, in our case a grid simulation job, into an image. This image will contain everything from the OS-level utilities and libraries, to the lab UPS products, to the collaboration code, to our grid scripts and environmental variables. This image starts with a OS image, usually provided by the OS experts, and uploaded to the Docker repository. The user then adds layers of software to this base image. The computing division has added the art software layer to an OS image and uploaded this new, larger image to the repository. Mu2e can download the art image and add our own layers. This final mu2e image can be passed around and run on almost any platform with x86_64 architecture.

The Docker image can only be run on a machine with Docker software installed and the Docker daemon running. When the user issues a docker command, the daemon starts the image. The user's command can run in the image's environment, exit, and return control back to the user. The user may also run the image in an interactive mode, gain the image's prompt and issue commands like any linux command line. In effect, the user can fire up their own pseudo-virtual machine. The user's process will have a virtual working directory which can be destroyed, or saved, on exit from the container. File systems mounted on the host machine (your home directory, for example) can be mounted in the pseudo-VM and you can read and write permanently to the file system. This pseudo-VM is called a container because it is isolated in software from other copies of image that may be running.

The main point of the image/container model is that the user brings their entire software stack to the CPU, therefore the execution of the application is very reliable and reproducible. This model differs from the model of running several full virtual machines on the host in an important way. If we start a virtual machine, it requires a separate memory space and will load the shared libraries once in each virtual machine, quickly using up the memory. The Docker daemon knows if the users are running multiple containers for one image and shares as much memory as possible, such as the shared libraries, while keeping each container's data space strictly isolated. If you are running on a machine with 64 GB of memory and 64 cores, you may be able to run 64 containers of one Docker image, but you can't run 64 full virtual machines. Docker is also more CPU efficient than a full VM.

Docker Software

The Docker software can be installed on Mac, Windows and Linux. SL6 has an older kernel which cannot implement all the features of Docker and limits the size of a Docker image to 10GB. Windows (>=10), MacOS (>=10.11), and SL7 are all suitable to run Docker. The code needs to be installed by admin/root because it will need a higher level of access to the kernel than a typical user application, and will run a system daemon. A few collaborators with access to the computing division machine called woof can access a docker server there. From 4/2019, Mu2e has its own docker build machine called mu2eimagegpmv01. This machine is for a few experts to build images - the intention is that most users will only be downloading those images to a laptop or other machine. New users must be added by hand, ask offline management for access.

Creating a Docker image

A Docker image is a file, like a sophisticated tarball, which contains all the code needed to run our executables. This image will contain everything from the OS-level utilities and libraries, to the lab UPS products, to the collaboration code, to our grid scripts. Anything you would use to run the job interactively needs to be provided in the image. The image might be up to 30 GB in size or it might be as small as a few GB if it is carefully pruned.

One could start an image from scratch, but it we will always start with an operating system image, such as an SL6 or CentOS image, provided by experts, and add to that base image. In fact, the computing division has provided an OS image with an added layer of the mu art software manifest. We then have to add a mu2e release and some odds and ends such as the mu2egrid product.

To download the base image, you will need an account on hub.docker.com. Searching for Mu2e, FNAL, or fnalart will show the existing images. Mu2e has an account with username "mu2e" and linked to mu2e-production@fnal.gov. All hub images are public by default, so can be uploaded and shared from any account. The web page will show how to download the image, like:

docker pull fnalart/mu:v2_06_02-s47-e10-prof

Theoretically, the cvmfs client could be added to the image and run in the container to mount and read a standard cvmfs repository. We have heard this has been done, but we don't know of local expertise and haven't tried it yet. Until this is available, we need to include all the software directly in the image, or install cmvfs on the host machine, then mount the cvmfs directory when you start the container, as if it were a simple local disk. (This is how FermiGrid works.)

To build a useful image, you will need to pick a base image (usually a version of linux) then collect all the files you want to add to the image. The files to be added need to all be under a single working directory. There is a control file called Dockerfile, in the workig directory, which defines how to add your files to the base image to create a new image. Here is an example

FROM mu2e/base:grid-00
ADD  setupmu2e-art.sh /setupmu2e-art.sh
ADD  products     /products
ADD  artexternals /artexternals
ADD  DataFiles    /DataFiles
ADD  Offline      /Offline
COPY graphicslibs /graphicslibs
ADD  etc_profile /etc/profile.d/custom.sh
ADD  etc_bashrc   /etc_bashrc
RUN  cat /etc_bashrc >> /etc/bashrc ; rm /etc_bashrc
ENV  UPS_OVERRIDE "-H Linux64bit+2.6-2.12"
ENV  MU2E_DOCKER_IMAGE v6_2_1-00
CMD ["/bin/bash","-l"]

The FROM command defines the base image which you previously "pulled". The ADD command move files into the image. It has two arguments - the first is the local, source directory relative to the location of the Dockerfile and the second is the target directory in the image. The ENV command causes an environmental to be defined every time the image is run. CMD defines what is the default command for the image. This is what is run when you run it interactively. The CMD in the highest layer over-rides any CMD from previous layers.

The UPS_OVERRIDE line tell UPS to always use this as the UPS flavor (a.k.a. the kernel version). This is necessary because UPS will fail to extract a useful flavor from the CentOS kernel.

The pattern docker expects is to put your files to be added in a directory along with the Dockerfile, then tell it to build.

docker build -t <user/repo:tag> <directory with Dockerfile>
docker build -t mu2e/prd:v6_2_1-00 /scratch/rlc/docker/v6_2_1-00

The image itself goes into the docker cache directory which was created when docker product and its server were installed. You generally don't need to know where it is. You will push the image to hub.docker.com when it is ready to use.

If you build many images off of one base image, docker is smart enough to re-use base image and not waste any disk space.

To upload the image to docker hub, which is recommended whenever they need to be shared, you will need an account on [docker.com docker.com]. Then login with a docker.com account username and password, and push the image.

docker login  
docker push mu2e/prd:v6_2_1-00

I couldn't get the the username and password switches so I let it prompt me. Also it seems to require the username, not the user email.

Running a Docker image

There are two basic ways to run a docker image. The first is create a container, run a single command and exit. An example is

docker run --rm <image> <command>
docker run --rm rlcfnal/test:v0 ls; pwd
docker run --rm rlcfnal/test:v0 bash -l -c "ups list -aK+"

The "--rm" says do not save the container after running the command - you want this for any simple single commands or the working disk space will fill up with old non-running containers. If the command prints anything, it will appear on your screen. A switch "-d" will run the container in the background. There seem to be many ways to use docker. One way is to save a container, then "commit" the container, and any new contents, as a new image. You can connect to a container running in the background, or before other actions on it.

You can also run an image interactively.

docker run -it <image>
docker run --rm -it -v /scratch/rlc/docker/v6_2_1-0:/builddir rlcfnal/test:v6_2_1-0

The "--rm" switch says do not save the container after exit. The "-it" says start the interactive container and connects it the terminal. The "-v" option is of the form /localdir:/imagedir and causes the /localdir to be mounted in the container as /imagedir. There doesn't seem to be an x connection since root -l fails to display.

Linux command (from Pasha) which includes mounting cvmfs and setting up xwindows

XSOCK=/tmp/.X11-unix/X0 ; docker run -it -v /projects:/projects -v /cvmfs:/cvmfs:shared -v $XSOCK:$XSOCK -u 5339:5339 -e TZ=America/Chicago mu2e

How to see and delete old containers:

docker ps -a
docker rm $(docker ps -a -q)

Connect to a container that was exited, but not deleted. (Files created in the container will still be there.)

docker ps -a
docker start -ai <container hash or name>

Singularity

Singularity is an alternative virtualization product, orginally developed for HPC centers. Singularity can take a Docker container and modify it in important ways:

convert it to run as a user, not inside the root server. This solves many permissions and security problems.
exposes more of the OS, such as other running process, and mounts some disks by default

docs old docs Our Docker build machine can also build Singularity. SL7 machines can install Singularity and run images.

Start-up scripts

When a docker image is run on a HPC center system, it is often converted to singularity container. This adds security features.

In these tests, based on the CentOS base image, we have two start-up files for all users:

/etc/bashrc
/etc/profile.d/custom.sh

We added setting environmentals to these files so I can tell if they run. We created two test files:

/home/test1 which contains "printenv"
/home/test2 which contains "#!/bin/bash \n printenv"

shifter does not ever run bashrc, even when running interactively. docker does run bashrc, but only when running interactively
both shifter and docker run profile if their command is "/bin/bash" "-l"
if the command is "/home/test1", docker fails to run it, shifter assumes a bash shell and runs it, but does not run profile
if the command is "/bin/bash" "/home/test1" or "/bin/bash" "/home/test2" or "/home/test2", shifter and docker can run it, but do not run profile
is the command is "bash" "-l" "/home/test1" or "/bin/bash" "-l" "/home/test2", docker and shifter run it after running profile

Install on Windows

The basic instructions are provided by Docker windows. There are requirements on Windows versions (Windows Home version won't work) and memory. You will need admin access. You will need an account on docker.com.

After the installation, you can run run powershell or command as admin, then docker should be in your path. Do a "docker login" (I found it would not take email/password, only username/password). At this point you can download and run images. There seems to be no easy way to avoid running as admin, but there were some not-easy ways available as web recipes.

To run a GUI application, you have to set DISPLAY. Run command, and type "ipconfig". Record your I.P. address, (it looks like the from the "DockerNAT" stanza works) and use it to set DISPLAY:

set-variable -name DISPLAY -value 10.11.128.118:0.0
docker run -ti --rm -e DISPLAY=$DISPLAY <image>

After rebooting, I found that I got an error:

No protocol specified
Error: can't open display <ipaddress>

I am using the Xming x server and checked the Xming log by right clicking on the Xming icon in the tray and selecting "show log". I saw errors like :

rejecting request from IP <ipaddress>

I added the ip address to

C: \Program File (x86)\Xming\X0.hosts

In the Xming launch script you can also allow all connections, but that is considered a security risk.

References

From toolkit

These are old scratch notes..

On a windows 10 laptop. The basic download only works for Windows pro, which I didn't have. Switched to downloading Docker toolkit. You need to run as an admin in the command window. Running the first example

docker run hello-world

resulted in errors

docker: error during connect:  ...

I performed a cleanup found on blogs

docker-machine rm default
docker-machine create --driver virtualbox default
docker-machine env --shell cmd default
 (cut and paste printed commands)

and then the hello-world example worked.

docker run -it ubuntu bash

Notes for Mac

Docker provides basic instructions.

A note from Adam Lyon: I do development on my Mac with a Vagrant (Virtual Box) VM and not Docker. Docker i/o performance when accessing the Mac disk volume is atrocious. It's unusable - Docker knows about this problem but can't seem to fix it. With Vagrant, you can mount your Mac disk into the VM with nfs (it sets it all up for you) and it works great. Very fast i/o. You can keep your code and stuff on the Mac side and access it (eg compile) with the VM and it just works. If you try that with docker, it is so slow that you will hate it. I have a VM configuration at https://github.com/lyon-fnal/centos-gm2-dev/blob/master/README.md that I've been using for a long time. With VNC (described in the Readme) it is a very performant and nice to use.

Another note (Pengfei): The slowness of disk access in Mac is caused by osx fuse or osxfs. Turning on “cached” or “delegated" option for the bind-mount might help. The old work around for the slow disk access is to use NFS (e.g. docker-machine + VirtualBox + NFS). But you may not need to do that any more. Since March 2018, Docker for mac starts to support NFS. I personally haven’t tried that though. link

A note from Dave Brown: I was able to access a local disk (/data) through the docker container. To do that I had to:

add /data to the list of exported file systems in the DockerDesktop app
start docker with -v /data (a local disk on my mac)

I was also able to get root to run. For that, I had to:

enable network connections in my local install of XQuartz,
whitelist my local ip address (not sure why that was necessary??).
add '-e DISPLAY=$ip:0' to my docker command

After that I was able to start root and get it to process and display a largish (50MB) .root file living in /data using a standard macro that's in Offline. I timed how long it took to run the macro native on my mac vs through the docker container (the results on the screen were indistinguishable):

docker: 3.42 seconds
native: 1.79 seconds

A procedure on Jenkins

wiki describing a framework for building Docker containers on Jenkins for Continuous Integration. Not sure who is doing this..