diff options
Diffstat (limited to 'lib/spack/docs/containers.rst')
| -rw-r--r-- | lib/spack/docs/containers.rst | 307 |
1 files changed, 307 insertions, 0 deletions
diff --git a/lib/spack/docs/containers.rst b/lib/spack/docs/containers.rst new file mode 100644 index 0000000000..bbb21a2e00 --- /dev/null +++ b/lib/spack/docs/containers.rst @@ -0,0 +1,307 @@ +.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other + Spack Project Developers. See the top-level COPYRIGHT file for details. + + SPDX-License-Identifier: (Apache-2.0 OR MIT) + +.. _containers: + +================ +Container Images +================ + +Spack can be an ideal tool to setup images for containers since all the +features discussed in :ref:`environments` can greatly help to manage +the installation of software during the image build process. Nonetheless, +building a production image from scratch still requires a lot of +boilerplate to: + +- Get Spack working within the image, possibly running as root +- Minimize the physical size of the software installed +- Properly update the system software in the base image + +To facilitate users with these tedious tasks, Spack provides a command +to automatically generate recipes for container images based on +Environments: + +.. code-block:: console + + $ ls + spack.yaml + + $ spack containerize + # Build stage with Spack pre-installed and ready to be used + FROM spack/centos7:latest as builder + + # What we want to install and how we want to install it + # is specified in a manifest file (spack.yaml) + RUN mkdir /opt/spack-environment \ + && (echo "spack:" \ + && echo " specs:" \ + && echo " - gromacs+mpi" \ + && echo " - mpich" \ + && echo " concretization: together" \ + && echo " config:" \ + && echo " install_tree: /opt/software" \ + && echo " view: /opt/view") > /opt/spack-environment/spack.yaml + + # Install the software, remove unecessary deps + RUN cd /opt/spack-environment && spack install && spack gc -y + + # Strip all the binaries + RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \ + xargs file -i | \ + grep 'charset=binary' | \ + grep 'x-executable\|x-archive\|x-sharedlib' | \ + awk -F: '{print $1}' | xargs strip -s + + # Modifications to the environment that are necessary to run + RUN cd /opt/spack-environment && \ + spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh + + + # Bare OS image to run the installed executables + FROM centos:7 + + COPY --from=builder /opt/spack-environment /opt/spack-environment + COPY --from=builder /opt/software /opt/software + COPY --from=builder /opt/view /opt/view + COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh + + RUN yum update -y && yum install -y epel-release && yum update -y \ + && yum install -y libgomp \ + && rm -rf /var/cache/yum && yum clean all + + RUN echo 'export PS1="\[$(tput bold)\]\[$(tput setaf 1)\][gromacs]\[$(tput setaf 2)\]\u\[$(tput sgr0)\]:\w $ \[$(tput sgr0)\]"' >> ~/.bashrc + + + LABEL "app"="gromacs" + LABEL "mpi"="mpich" + + ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"] + + +The bits that make this automation possible are discussed in details +below. All the images generated in this way will be based on +multi-stage builds with: + +- A fat ``build`` stage containing common build tools and Spack itself +- A minimal ``final`` stage containing only the software requested by the user + +----------------- +Spack Base Images +----------------- + +Docker images with Spack preinstalled and ready to be used are +built on `Docker Hub <https://hub.docker.com/u/spack>`_ +at every push to ``develop`` or to a release branch. The OS that +are currently supported are summarized in the table below: + +.. _containers-supported-os: + +.. list-table:: Supported operating systems + :header-rows: 1 + + * - Operating System + - Base Image + - Spack Image + * - Ubuntu 16.04 + - ``ubuntu:16.04`` + - ``spack/ubuntu-xenial`` + * - Ubuntu 18.04 + - ``ubuntu:16.04`` + - ``spack/ubuntu-bionic`` + * - CentOS 6 + - ``centos:6`` + - ``spack/centos6`` + * - CentOS 7 + - ``centos:7`` + - ``spack/centos7`` + +All the images are tagged with the corresponding release of Spack: + +.. image:: dockerhub_spack.png + +with the exception of the ``latest`` tag that points to the HEAD +of the ``develop`` branch. These images are available for anyone +to use and take care of all the repetitive tasks that are necessary +to setup Spack within a container. All the container recipes generated +automatically by Spack use them as base images for their ``build`` stage. + + +------------------------- +Environment Configuration +------------------------- + +Any Spack Environment can be used for the automatic generation of container +recipes. Sensible defaults are provided for things like the base image or the +version of Spack used in the image. If a finer tuning is needed it can be +obtained by adding the relevant metadata under the ``container`` attribute +of environments: + +.. code-block:: yaml + + spack: + specs: + - gromacs+mpi + - mpich + + container: + # Select the format of the recipe e.g. docker, + # singularity or anything else that is currently supported + format: docker + + # Select from a valid list of images + base: + image: "centos:7" + spack: develop + + # Whether or not to strip binaries + strip: true + + # Additional system packages that are needed at runtime + os_packages: + - libgomp + + # Extra instructions + extra_instructions: + final: | + RUN echo 'export PS1="\[$(tput bold)\]\[$(tput setaf 1)\][gromacs]\[$(tput setaf 2)\]\u\[$(tput sgr0)\]:\w $ \[$(tput sgr0)\]"' >> ~/.bashrc + + # Labels for the image + labels: + app: "gromacs" + mpi: "mpich" + +The tables below describe the configuration options that are currently supported: + +.. list-table:: General configuration options for the ``container`` section of ``spack.yaml`` + :header-rows: 1 + + * - Option Name + - Description + - Allowed Values + - Required + * - ``format`` + - The format of the recipe + - ``docker`` or ``singularity`` + - Yes + * - ``base:image`` + - Base image for ``final`` stage + - See :ref:`containers-supported-os` + - Yes + * - ``base:spack`` + - Version of Spack + - Valid tags for ``base:image`` + - Yes + * - ``strip`` + - Whether to strip binaries + - ``true`` (default) or ``false`` + - No + * - ``os_packages`` + - System packages to be installed + - Valid packages for the ``final`` OS + - No + * - ``extra_instructions:build`` + - Extra instructions (e.g. `RUN`, `COPY`, etc.) at the end of the ``build`` stage + - Anything understood by the current ``format`` + - No + * - ``extra_instructions:final`` + - Extra instructions (e.g. `RUN`, `COPY`, etc.) at the end of the ``final`` stage + - Anything understood by the current ``format`` + - No + * - ``labels`` + - Labels to tag the image + - Pairs of key-value strings + - No + +.. list-table:: Configuration options specific to Singularity + :header-rows: 1 + + * - Option Name + - Description + - Allowed Values + - Required + * - ``singularity:runscript`` + - Content of ``%runscript`` + - Any valid script + - No + * - ``singularity:startscript`` + - Content of ``%startscript`` + - Any valid script + - No + * - ``singularity:test`` + - Content of ``%test`` + - Any valid script + - No + * - ``singularity:help`` + - Description of the image + - Description string + - No + +Once the Environment is properly configured a recipe for a container +image can be printed to standard output by issuing the following +command from the directory where the ``spack.yaml`` resides: + +.. code-block:: console + + $ spack containerize + +The example ``spack.yaml`` above would produce for instance the +following ``Dockerfile``: + +.. code-block:: docker + + # Build stage with Spack pre-installed and ready to be used + FROM spack/centos7:latest as builder + + # What we want to install and how we want to install it + # is specified in a manifest file (spack.yaml) + RUN mkdir /opt/spack-environment \ + && (echo "spack:" \ + && echo " specs:" \ + && echo " - gromacs+mpi" \ + && echo " - mpich" \ + && echo " concretization: together" \ + && echo " config:" \ + && echo " install_tree: /opt/software" \ + && echo " view: /opt/view") > /opt/spack-environment/spack.yaml + + # Install the software, remove unecessary deps + RUN cd /opt/spack-environment && spack install && spack gc -y + + # Strip all the binaries + RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \ + xargs file -i | \ + grep 'charset=binary' | \ + grep 'x-executable\|x-archive\|x-sharedlib' | \ + awk -F: '{print $1}' | xargs strip -s + + # Modifications to the environment that are necessary to run + RUN cd /opt/spack-environment && \ + spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh + + + # Bare OS image to run the installed executables + FROM centos:7 + + COPY --from=builder /opt/spack-environment /opt/spack-environment + COPY --from=builder /opt/software /opt/software + COPY --from=builder /opt/view /opt/view + COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh + + RUN yum update -y && yum install -y epel-release && yum update -y \ + && yum install -y libgomp \ + && rm -rf /var/cache/yum && yum clean all + + RUN echo 'export PS1="\[$(tput bold)\]\[$(tput setaf 1)\][gromacs]\[$(tput setaf 2)\]\u\[$(tput sgr0)\]:\w $ \[$(tput sgr0)\]"' >> ~/.bashrc + + + LABEL "app"="gromacs" + LABEL "mpi"="mpich" + + ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"] + +.. note:: + Spack can also produce Singularity definition files to build the image. The + minimum version of Singularity required to build a SIF (Singularity Image Format) + from them is ``3.5.3``.
\ No newline at end of file |