diff options
| -rw-r--r-- | lib/spack/docs/containers.rst | 423 | ||||
| -rw-r--r-- | lib/spack/spack/container/__init__.py | 4 | ||||
| -rw-r--r-- | lib/spack/spack/container/images.json | 75 | ||||
| -rw-r--r-- | lib/spack/spack/container/images.py | 29 | ||||
| -rw-r--r-- | lib/spack/spack/container/writers/__init__.py | 74 | ||||
| -rw-r--r-- | lib/spack/spack/schema/container.py | 66 | ||||
| -rw-r--r-- | lib/spack/spack/tengine.py | 4 | ||||
| -rw-r--r-- | lib/spack/spack/test/container/conftest.py | 4 | ||||
| -rw-r--r-- | lib/spack/spack/test/container/docker.py | 12 | ||||
| -rw-r--r-- | lib/spack/spack/test/container/images.py | 3 | ||||
| -rw-r--r-- | lib/spack/spack/test/container/schema.py | 4 | ||||
| -rw-r--r-- | share/spack/templates/container/Dockerfile | 20 | ||||
| -rw-r--r-- | share/spack/templates/container/singularity.def | 34 |
13 files changed, 514 insertions, 238 deletions
diff --git a/lib/spack/docs/containers.rst b/lib/spack/docs/containers.rst index 2590fec8b6..f5bd5602bc 100644 --- a/lib/spack/docs/containers.rst +++ b/lib/spack/docs/containers.rst @@ -9,28 +9,48 @@ Container Images ================ -Spack can be an ideal tool to setup images for containers since all the -features discussed in :ref:`environments` can greatly help to manage -the installation of software during the image build process. Nonetheless, -building a production image from scratch still requires a lot of -boilerplate to: +Spack :ref:`environments` are a great tool to create container images, but +preparing one that is suitable for production requires some more boilerplate +than just: -- Get Spack working within the image, possibly running as root -- Minimize the physical size of the software installed -- Properly update the system software in the base image +.. code-block:: docker + + COPY spack.yaml /environment + RUN spack -e /environment install + +Additional actions may be needed to minimize the size of the +container, or to update the system software that is installed in the base +image, or to set up a proper entrypoint to run the image. These tasks are +usually both necessary and repetitive, so Spack comes with a command +to generate recipes for container images starting from a ``spack.yaml``. + +-------------------- +A Quick Introduction +-------------------- + +Consider having a Spack environment like the following: + +.. code-block:: yaml + + spack: + specs: + - gromacs+mpi + - mpich -To facilitate users with these tedious tasks, Spack provides a command -to automatically generate recipes for container images based on -Environments: +Producing a ``Dockerfile`` from it is as simple as moving to the directory +where the ``spack.yaml`` file is stored and giving the following command: .. code-block:: console - $ ls - spack.yaml + $ spack containerize > Dockerfile + +The ``Dockerfile`` that gets created uses multi-stage builds and +other techniques to minimize the size of the final image: + +.. code-block:: docker - $ spack containerize # Build stage with Spack pre-installed and ready to be used - FROM spack/centos7:latest as builder + FROM spack/ubuntu-bionic:latest as builder # What we want to install and how we want to install it # is specified in a manifest file (spack.yaml) @@ -45,7 +65,7 @@ Environments: && echo " view: /opt/view") > /opt/spack-environment/spack.yaml # Install the software, remove unnecessary deps - RUN cd /opt/spack-environment && spack env activate . && spack install && spack gc -y + RUN cd /opt/spack-environment && spack env activate . && spack install --fail-fast && spack gc -y # Strip all the binaries RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \ @@ -58,45 +78,34 @@ Environments: RUN cd /opt/spack-environment && \ spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh - # Bare OS image to run the installed executables - FROM centos:7 + FROM ubuntu:18.04 COPY --from=builder /opt/spack-environment /opt/spack-environment COPY --from=builder /opt/software /opt/software COPY --from=builder /opt/view /opt/view COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh - RUN yum update -y && yum install -y epel-release && yum update -y \ - && yum install -y libgomp \ - && rm -rf /var/cache/yum && yum clean all - - RUN echo 'export PS1="\[$(tput bold)\]\[$(tput setaf 1)\][gromacs]\[$(tput setaf 2)\]\u\[$(tput sgr0)\]:\w $ "' >> ~/.bashrc - - - LABEL "app"="gromacs" - LABEL "mpi"="mpich" - ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"] -In order to build and run the image, execute: +The image itself can then be built and run in the usual way, with any of the +tools suitable for the task. For instance, if we decided to use ``docker``: .. code-block:: bash $ spack containerize > Dockerfile $ docker build -t myimage . + [ ... ] $ docker run -it myimage -The bits that make this automation possible are discussed in details -below. All the images generated in this way will be based on -multi-stage builds with: +The various components involved in the generation of the recipe and their +configuration are discussed in details in the sections below. -- A fat ``build`` stage containing common build tools and Spack itself -- A minimal ``final`` stage containing only the software requested by the user +.. _container_spack_images: ------------------ -Spack Base Images ------------------ +-------------------------- +Spack Images on Docker Hub +-------------------------- Docker images with Spack preinstalled and ready to be used are built on `Docker Hub <https://hub.docker.com/u/spack>`_ @@ -131,19 +140,20 @@ All the images are tagged with the corresponding release of Spack: with the exception of the ``latest`` tag that points to the HEAD of the ``develop`` branch. These images are available for anyone to use and take care of all the repetitive tasks that are necessary -to setup Spack within a container. All the container recipes generated -automatically by Spack use them as base images for their ``build`` stage. - +to setup Spack within a container. The container recipes generated +by Spack use them as default base images for their ``build`` stage, +even though handles to use custom base images provided by users are +available to accommodate complex use cases. -------------------------- -Environment Configuration -------------------------- +--------------------------------- +Creating Images From Environments +--------------------------------- Any Spack Environment can be used for the automatic generation of container recipes. Sensible defaults are provided for things like the base image or the -version of Spack used in the image. If a finer tuning is needed it can be -obtained by adding the relevant metadata under the ``container`` attribute -of environments: +version of Spack used in the image. +If a finer tuning is needed it can be obtained by adding the relevant metadata +under the ``container`` attribute of environments: .. code-block:: yaml @@ -157,9 +167,10 @@ of environments: # singularity or anything else that is currently supported format: docker - # Select from a valid list of images - base: - image: "centos:7" + # Sets the base images for the stages where Spack builds the + # software or where the software gets installed after being built.. + images: + os: "centos:7" spack: develop # Whether or not to strip binaries @@ -167,7 +178,8 @@ of environments: # Additional system packages that are needed at runtime os_packages: - - libgomp + final: + - libgomp # Extra instructions extra_instructions: @@ -179,7 +191,210 @@ of environments: app: "gromacs" mpi: "mpich" -The tables below describe the configuration options that are currently supported: +A detailed description of the options available can be found in the +:ref:`container_config_options` section. + +------------------- +Setting Base Images +------------------- + +The ``images`` subsection is used to select both the image where +Spack builds the software and the image where the built software +is installed. This attribute can be set in two different ways and +which one to use depends on the use case at hand. + +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Use Official Spack Images From Dockerhub +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To generate a recipe that uses an official Docker image from the +Spack organization to build the software and the corresponding official OS image +to install the built software, all the user has to do is specify: + +1. An operating system under ``images:os`` +2. A Spack version under ``images:spack`` + +Any combination of these two values that can be mapped to one of the images +discussed in :ref:`container_spack_images` is allowed. For instance, the +following ``spack.yaml``: + +.. code-block:: yaml + + spack: + specs: + - gromacs+mpi + - mpich + + container: + images: + os: centos/7 + spack: 0.15.4 + +uses ``spack/centos7:0.15.4`` and ``centos:7`` for the stages where the +software is respectively built and installed: + +.. code-block:: docker + + # Build stage with Spack pre-installed and ready to be used + FROM spack/centos7:0.15.4 as builder + + # What we want to install and how we want to install it + # is specified in a manifest file (spack.yaml) + RUN mkdir /opt/spack-environment \ + && (echo "spack:" \ + && echo " specs:" \ + && echo " - gromacs+mpi" \ + && echo " - mpich" \ + && echo " concretization: together" \ + && echo " config:" \ + && echo " install_tree: /opt/software" \ + && echo " view: /opt/view") > /opt/spack-environment/spack.yaml + [ ... ] + # Bare OS image to run the installed executables + FROM centos:7 + + COPY --from=builder /opt/spack-environment /opt/spack-environment + COPY --from=builder /opt/software /opt/software + COPY --from=builder /opt/view /opt/view + COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh + + ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"] + +This method of selecting base images is the simplest of the two, and we advise +to use it whenever possible. There are cases though where using Spack official +images is not enough to fit production needs. In these situations users can manually +select which base image to start from in the recipe, as we'll see next. + +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Use Custom Images Provided by Users +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Consider, as an example, building a production grade image for a CUDA +application. The best strategy would probably be to build on top of +images provided by the vendor and regard CUDA as an external package. + +Spack doesn't currently provide an official image with CUDA configured +this way, but users can build it on their own and then configure the +environment to explicitly pull it. This requires users to: + +1. Specify the image used to build the software under ``images:build`` +2. Specify the image used to install the built software under ``images:final`` + +A ``spack.yaml`` like the following: + +.. code-block:: yaml + + spack: + specs: + - gromacs@2019.4+cuda build_type=Release + - mpich + - fftw precision=float + packages: + cuda: + buildable: False + externals: + - spec: cuda%gcc + prefix: /usr/local/cuda + + container: + images: + build: custom/cuda-10.1-ubuntu18.04:latest + final: nvidia/cuda:10.1-base-ubuntu18.04 + +produces, for instance, the following ``Dockerfile``: + +.. code-block:: docker + + # Build stage with Spack pre-installed and ready to be used + FROM custom/cuda-10.1-ubuntu18.04:latest as builder + + # What we want to install and how we want to install it + # is specified in a manifest file (spack.yaml) + RUN mkdir /opt/spack-environment \ + && (echo "spack:" \ + && echo " specs:" \ + && echo " - gromacs@2019.4+cuda build_type=Release" \ + && echo " - mpich" \ + && echo " - fftw precision=float" \ + && echo " packages:" \ + && echo " cuda:" \ + && echo " buildable: false" \ + && echo " externals:" \ + && echo " - spec: cuda%gcc" \ + && echo " prefix: /usr/local/cuda" \ + && echo " concretization: together" \ + && echo " config:" \ + && echo " install_tree: /opt/software" \ + && echo " view: /opt/view") > /opt/spack-environment/spack.yaml + + # Install the software, remove unnecessary deps + RUN cd /opt/spack-environment && spack env activate . && spack install --fail-fast && spack gc -y + + # Strip all the binaries + RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \ + xargs file -i | \ + grep 'charset=binary' | \ + grep 'x-executable\|x-archive\|x-sharedlib' | \ + awk -F: '{print $1}' | xargs strip -s + + # Modifications to the environment that are necessary to run + RUN cd /opt/spack-environment && \ + spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh + + # Bare OS image to run the installed executables + FROM nvidia/cuda:10.1-base-ubuntu18.04 + + COPY --from=builder /opt/spack-environment /opt/spack-environment + COPY --from=builder /opt/software /opt/software + COPY --from=builder /opt/view /opt/view + COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh + + ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"] + +where the base images for both stages are completely custom. + +This second mode of selection for base images is more flexible than just +choosing an operating system and a Spack version, but is also more demanding. +Users may need to generate by themselves their base images and it's also their +responsibility to ensure that: + +1. Spack is available in the ``build`` stage and set up correctly to install the required software +2. The artifacts produced in the ``build`` stage can be executed in the ``final`` stage + +Therefore we don't recommend its use in cases that can be otherwise +covered by the simplified mode shown first. + +---------------------------- +Singularity Definition Files +---------------------------- + +In addition to producing recipes in ``Dockerfile`` format Spack can produce +Singularity Definition Files by just changing the value of the ``format`` +attribute: + +.. code-block:: console + + $ cat spack.yaml + spack: + specs: + - hdf5~mpi + container: + format: singularity + + $ spack containerize > hdf5.def + $ sudo singularity build hdf5.sif hdf5.def + +The minimum version of Singularity required to build a SIF (Singularity Image Format) +image from the recipes generated by Spack is ``3.5.3``. + +.. _container_config_options: + +----------------------- +Configuration Reference +----------------------- + +The tables below describe all the configuration options that are currently supported +to customize the generation of container recipes: .. list-table:: General configuration options for the ``container`` section of ``spack.yaml`` :header-rows: 1 @@ -192,21 +407,41 @@ The tables below describe the configuration options that are currently supported - The format of the recipe - ``docker`` or ``singularity`` - Yes - * - ``base:image`` - - Base image for ``final`` stage + * - ``images:os`` + - Operating system used as a base for the image - See :ref:`containers-supported-os` - - Yes - * - ``base:spack`` - - Version of Spack + - Yes, if using constrained selection of base images + * - ``images:spack`` + - Version of Spack use in the ``build`` stage - Valid tags for ``base:image`` - - Yes + - Yes, if using constrained selection of base images + * - ``images:build`` + - Image to be used in the ``build`` stage + - Any valid container image + - Yes, if using custom selection of base images + * - ``images:final`` + - Image to be used in the ``build`` stage + - Any valid container image + - Yes, if using custom selection of base images * - ``strip`` - Whether to strip binaries - ``true`` (default) or ``false`` - No - * - ``os_packages`` - - System packages to be installed - - Valid packages for the ``final`` OS + * - ``os_packages:command`` + - Tool used to manage system packages + - ``apt``, ``yum`` + - Only with custom base images + * - ``os_packages:update`` + - Whether or not to update the list of available packages + - True or False (default: True) + - No + * - ``os_packages:build`` + - System packages needed at build-time + - Valid packages for the current OS + - No + * - ``os_packages:final`` + - System packages needed at run-time + - Valid packages for the current OS - No * - ``extra_instructions:build`` - Extra instructions (e.g. `RUN`, `COPY`, etc.) at the end of the ``build`` stage @@ -245,74 +480,6 @@ The tables below describe the configuration options that are currently supported - Description string - No -Once the Environment is properly configured a recipe for a container -image can be printed to standard output by issuing the following -command from the directory where the ``spack.yaml`` resides: - -.. code-block:: console - - $ spack containerize - -The example ``spack.yaml`` above would produce for instance the -following ``Dockerfile``: - -.. code-block:: docker - - # Build stage with Spack pre-installed and ready to be used - FROM spack/centos7:latest as builder - - # What we want to install and how we want to install it - # is specified in a manifest file (spack.yaml) - RUN mkdir /opt/spack-environment \ - && (echo "spack:" \ - && echo " specs:" \ - && echo " - gromacs+mpi" \ - && echo " - mpich" \ - && echo " concretization: together" \ - && echo " config:" \ - && echo " install_tree: /opt/software" \ - && echo " view: /opt/view") > /opt/spack-environment/spack.yaml - - # Install the software, remove unnecessary deps - RUN cd /opt/spack-environment && spack env activate . && spack install && spack gc -y - - # Strip all the binaries - RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \ - xargs file -i | \ - grep 'charset=binary' | \ - grep 'x-executable\|x-archive\|x-sharedlib' | \ - awk -F: '{print $1}' | xargs strip -s - - # Modifications to the environment that are necessary to run - RUN cd /opt/spack-environment && \ - spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh - - - # Bare OS image to run the installed executables - FROM centos:7 - - COPY --from=builder /opt/spack-environment /opt/spack-environment - COPY --from=builder /opt/software /opt/software - COPY --from=builder /opt/view /opt/view - COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh - - RUN yum update -y && yum install -y epel-release && yum update -y \ - && yum install -y libgomp \ - && rm -rf /var/cache/yum && yum clean all - - RUN echo 'export PS1="\[$(tput bold)\]\[$(tput setaf 1)\][gromacs]\[$(tput setaf 2)\]\u\[$(tput sgr0)\]:\w $ "' >> ~/.bashrc - - - LABEL "app"="gromacs" - LABEL "mpi"="mpich" - - ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"] - -.. note:: - Spack can also produce Singularity definition files to build the image. The - minimum version of Singularity required to build a SIF (Singularity Image Format) - from them is ``3.5.3``. - -------------- Best Practices -------------- diff --git a/lib/spack/spack/container/__init__.py b/lib/spack/spack/container/__init__.py index fc3750355a..8206efef01 100644 --- a/lib/spack/spack/container/__init__.py +++ b/lib/spack/spack/container/__init__.py @@ -38,11 +38,11 @@ def validate(configuration_file): env_dict = spack.environment.config_dict(config) env_dict.setdefault('container', { 'format': 'docker', - 'base': {'image': 'ubuntu:18.04', 'spack': 'develop'} + 'images': {'os': 'ubuntu:18.04', 'spack': 'develop'} }) env_dict['container'].setdefault('format', 'docker') env_dict['container'].setdefault( - 'base', {'image': 'ubuntu:18.04', 'spack': 'develop'} + 'images', {'os': 'ubuntu:18.04', 'spack': 'develop'} ) # Remove attributes that are not needed / allowed in the diff --git a/lib/spack/spack/container/images.json b/lib/spack/spack/container/images.json index efaedc68b8..cb495908c9 100644 --- a/lib/spack/spack/container/images.json +++ b/lib/spack/spack/container/images.json @@ -1,42 +1,45 @@ { - "ubuntu:18.04": { - "update": "apt-get -yqq update && apt-get -yqq upgrade", - "install": "apt-get -yqq install", - "clean": "rm -rf /var/lib/apt/lists/*", - "environment": [], - "build": "spack/ubuntu-bionic", - "build_tags": { - "develop": "latest" + "images": { + "ubuntu:18.04": { + "os_package_manager": "apt", + "build": "spack/ubuntu-bionic", + "build_tags": { + "develop": "latest" + } + }, + "ubuntu:16.04": { + "os_package_manager": "apt", + "build": "spack/ubuntu-xenial", + "build_tags": { + "develop": "latest" + } + }, + "centos:7": { + "os_package_manager": "yum", + "environment": [], + "build": "spack/centos7", + "build_tags": { + "develop": "latest" + } + }, + "centos:6": { + "os_package_manager": "yum", + "build": "spack/centos6", + "build_tags": { + "develop": "latest" + } } }, - "ubuntu:16.04": { - "update": "apt-get -yqq update && apt-get -yqq upgrade", - "install": "apt-get -yqq install", - "clean": "rm -rf /var/lib/apt/lists/*", - "environment": [], - "build": "spack/ubuntu-xenial", - "build_tags": { - "develop": "latest" - } - }, - "centos:7": { - "update": "yum update -y && yum install -y epel-release && yum update -y", - "install": "yum install -y", - "clean": "rm -rf /var/cache/yum && yum clean all", - "environment": [], - "build": "spack/centos7", - "build_tags": { - "develop": "latest" - } - }, - "centos:6": { - "update": "yum update -y && yum install -y epel-release && yum update -y", - "install": "yum install -y", - "clean": "rm -rf /var/cache/yum && yum clean all", - "environment": [], - "build": "spack/centos6", - "build_tags": { - "develop": "latest" + "os_package_managers": { + "apt": { + "update": "apt-get -yqq update && apt-get -yqq upgrade", + "install": "apt-get -yqq install", + "clean": "rm -rf /var/lib/apt/lists/*" + }, + "yum": { + "update": "yum update -y && yum install -y epel-release && yum update -y", + "install": "yum install -y", + "clean": "rm -rf /var/cache/yum && yum clean all" } } } diff --git a/lib/spack/spack/container/images.py b/lib/spack/spack/container/images.py index f702781d32..32fcff2770 100644 --- a/lib/spack/spack/container/images.py +++ b/lib/spack/spack/container/images.py @@ -38,7 +38,7 @@ def build_info(image, spack_version): """ # Don't handle error here, as a wrong image should have been # caught by the JSON schema - image_data = data()[image] + image_data = data()["images"][image] build_image = image_data['build'] # Try to check if we have a tag for this Spack version @@ -55,19 +55,30 @@ def build_info(image, spack_version): return build_image, build_tag -def package_info(image): - """Returns the commands used to update system repositories, install - system packages and clean afterwards. +def os_package_manager_for(image): + """Returns the name of the OS package manager for the image + passed as argument. Args: image (str): image to be used at run-time. Should be of the form <image_name>:<image_tag> e.g. "ubuntu:18.04" Returns: + Name of the package manager, e.g. "apt" or "yum" + """ + name = data()["images"][image]["os_package_manager"] + return name + + +def commands_for(package_manager): + """Returns the commands used to update system repositories, install + system packages and clean afterwards. + + Args: + package_manager (str): package manager to be used + + Returns: A tuple of (update, install, clean) commands. """ - image_data = data()[image] - update = image_data['update'] - install = image_data['install'] - clean = image_data['clean'] - return update, install, clean + info = data()["os_package_managers"][package_manager] + return info['update'], info['install'], info['clean'] diff --git a/lib/spack/spack/container/writers/__init__.py b/lib/spack/spack/container/writers/__init__.py index a1d2fa3102..176dd7a50c 100644 --- a/lib/spack/spack/container/writers/__init__.py +++ b/lib/spack/spack/container/writers/__init__.py @@ -13,7 +13,8 @@ import spack.schema.env import spack.tengine as tengine import spack.util.spack_yaml as syaml -from spack.container.images import build_info, package_info +from spack.container.images import build_info, commands_for +from spack.container.images import os_package_manager_for #: Caches all the writers that are currently supported _writer_factory = {} @@ -63,21 +64,34 @@ class PathContext(tengine.Context): @tengine.context_property def run(self): """Information related to the run image.""" - image = self.container_config['base']['image'] + images_config = self.container_config['images'] + + # Check if we have custom images + image = images_config.get('final', None) + # If not use the base OS image + if image is None: + image = images_config['os'] + Run = collections.namedtuple('Run', ['image']) return Run(image=image) @tengine.context_property def build(self): """Information related to the build image.""" + images_config = self.container_config['images'] + + # Check if we have custom images + image = images_config.get('build', None) - # Map the final image to the correct build image - run_image = self.container_config['base']['image'] - spack_version = self.container_config['base']['spack'] - image, tag = build_info(run_image, spack_version) + # If not select the correct build image based on OS and Spack version + if image is None: + operating_system = images_config['os'] + spack_version = images_config['spack'] + image_name, tag = build_info(operating_system, spack_version) + image = ':'.join([image_name, tag]) - Build = collections.namedtuple('Build', ['image', 'tag']) - return Build(image=image, tag=tag) + Build = collections.namedtuple('Build', ['image']) + return Build(image=image) @tengine.context_property def strip(self): @@ -116,14 +130,50 @@ class PathContext(tengine.Context): return syaml.dump(manifest, default_flow_style=False).strip() @tengine.context_property - def os_packages(self): + def os_packages_final(self): """Additional system packages that are needed at run-time.""" - package_list = self.container_config.get('os_packages', None) + return self._os_packages_for_stage('final') + + @tengine.context_property + def os_packages_build(self): + """Additional system packages that are needed at build-time.""" + return self._os_packages_for_stage('build') + + @tengine.context_property + def os_package_update(self): + """Whether or not to update the OS package manager cache.""" + os_packages = self.container_config.get('os_packages', {}) + return os_packages.get('update', True) + + def _os_packages_for_stage(self, stage): + os_packages = self.container_config.get('os_packages', {}) + package_list = os_packages.get(stage, None) + return self._package_info_from(package_list) + + def _package_info_from(self, package_list): + """Helper method to pack a list of packages with the additional + information required by the template. + + Args: + package_list: list of packages + + Returns: + Enough information to know how to update the cache, install + a list opf packages, and clean in the end. + """ if not package_list: return package_list - image = self.container_config['base']['image'] - update, install, clean = package_info(image) + image_config = self.container_config['images'] + image = image_config.get('build', None) + + if image is None: + os_pkg_manager = os_package_manager_for(image_config['os']) + else: + os_pkg_manager = self.container_config['os_packages']['command'] + + update, install, clean = commands_for(os_pkg_manager) + Packages = collections.namedtuple( 'Packages', ['update', 'install', 'list', 'clean'] ) diff --git a/lib/spack/spack/schema/container.py b/lib/spack/spack/schema/container.py index d534afee08..7e14efd75c 100644 --- a/lib/spack/spack/schema/container.py +++ b/lib/spack/spack/schema/container.py @@ -4,6 +4,42 @@ # SPDX-License-Identifier: (Apache-2.0 OR MIT) """Schema for the 'container' subsection of Spack environments.""" +_stages_from_dockerhub = { + 'type': 'object', + 'additionalProperties': False, + 'properties': { + 'os': { + 'type': 'string', + 'enum': ['ubuntu:18.04', + 'ubuntu:16.04', + 'centos:7', + 'centos:6'] + }, + 'spack': { + 'type': 'string', + }, + }, + 'required': ['os', 'spack'] +} + +_custom_stages = { + 'type': 'object', + 'additionalProperties': False, + 'properties': { + 'build': {'type': 'string'}, + 'final': {'type': 'string'} + }, + 'required': ['build', 'final'] +} + +#: List of packages for the schema below +_list_of_packages = { + 'type': 'array', + 'items': { + 'type': 'string' + } +} + #: Schema for the container attribute included in Spack environments container_schema = { 'type': 'object', @@ -16,23 +52,7 @@ container_schema = { }, # Describes the base image to start from and the version # of Spack to be used - 'base': { - 'type': 'object', - 'additionalProperties': False, - 'properties': { - 'image': { - 'type': 'string', - 'enum': ['ubuntu:18.04', - 'ubuntu:16.04', - 'centos:7', - 'centos:6'] - }, - 'spack': { - 'type': 'string', - }, - }, - 'required': ['image', 'spack'] - }, + 'images': {'anyOf': [_stages_from_dockerhub, _custom_stages]}, # Whether or not to strip installed binaries 'strip': { 'type': 'boolean', @@ -40,10 +60,14 @@ container_schema = { }, # Additional system packages that are needed at runtime 'os_packages': { - 'type': 'array', - 'items': { - 'type': 'string' - } + 'type': 'object', + 'properties': { + 'command': {'type': 'string', 'enum': ['apt', 'yum']}, + 'update': {'type': 'boolean'}, + 'build': _list_of_packages, + 'final': _list_of_packages + }, + 'additionalProperties': False }, # Add labels to the image 'labels': { diff --git a/lib/spack/spack/tengine.py b/lib/spack/spack/tengine.py index e1e059c6a6..8581c8b41e 100644 --- a/lib/spack/spack/tengine.py +++ b/lib/spack/spack/tengine.py @@ -80,7 +80,9 @@ def make_environment(dirs=None): # Loader for the templates loader = jinja2.FileSystemLoader(dirs) # Environment of the template engine - env = jinja2.Environment(loader=loader, trim_blocks=True) + env = jinja2.Environment( + loader=loader, trim_blocks=True, lstrip_blocks=True + ) # Custom filters _set_filters(env) return env diff --git a/lib/spack/spack/test/container/conftest.py b/lib/spack/spack/test/container/conftest.py index 65a89b0748..70a1ae2547 100644 --- a/lib/spack/spack/test/container/conftest.py +++ b/lib/spack/spack/test/container/conftest.py @@ -18,8 +18,8 @@ def minimal_configuration(): ], 'container': { 'format': 'docker', - 'base': { - 'image': 'ubuntu:18.04', + 'images': { + 'os': 'ubuntu:18.04', 'spack': 'develop' } } diff --git a/lib/spack/spack/test/container/docker.py b/lib/spack/spack/test/container/docker.py index f75ae7de2a..6392f97db2 100644 --- a/lib/spack/spack/test/container/docker.py +++ b/lib/spack/spack/test/container/docker.py @@ -22,20 +22,22 @@ def test_build_and_run_images(minimal_configuration): # Test the output of the build property build = writer.build - assert build.image == 'spack/ubuntu-bionic' - assert build.tag == 'latest' + assert build.image == 'spack/ubuntu-bionic:latest' def test_packages(minimal_configuration): # In this minimal configuration we don't have packages writer = writers.create(minimal_configuration) - assert writer.os_packages is None + assert writer.os_packages_build is None + assert writer.os_packages_final is None # If we add them a list should be returned pkgs = ['libgomp1'] - minimal_configuration['spack']['container']['os_packages'] = pkgs + minimal_configuration['spack']['container']['os_packages'] = { + 'final': pkgs + } writer = writers.create(minimal_configuration) - p = writer.os_packages + p = writer.os_packages_final assert p.update assert p.install assert p.clean diff --git a/lib/spack/spack/test/container/images.py b/lib/spack/spack/test/container/images.py index 6af69e0c83..6cec1cc592 100644 --- a/lib/spack/spack/test/container/images.py +++ b/lib/spack/spack/test/container/images.py @@ -22,7 +22,8 @@ def test_build_info(image, spack_version, expected): 'ubuntu:18.04' ]) def test_package_info(image): - update, install, clean = spack.container.images.package_info(image) + pkg_manager = spack.container.images.os_package_manager_for(image) + update, install, clean = spack.container.images.commands_for(pkg_manager) assert update assert install assert clean diff --git a/lib/spack/spack/test/container/schema.py b/lib/spack/spack/test/container/schema.py index 3f33a3f9f7..e86f9d3fe8 100644 --- a/lib/spack/spack/test/container/schema.py +++ b/lib/spack/spack/test/container/schema.py @@ -10,7 +10,7 @@ import spack.schema.container def test_images_in_schema(): properties = spack.schema.container.container_schema['properties'] allowed_images = set( - properties['base']['properties']['image']['enum'] + properties['images']['anyOf'][0]['properties']['os']['enum'] ) - images_in_json = set(x for x in spack.container.images.data()) + images_in_json = set(x for x in spack.container.images.data()['images']) assert images_in_json == allowed_images diff --git a/share/spack/templates/container/Dockerfile b/share/spack/templates/container/Dockerfile index c65fce5627..3623a7ba0b 100644 --- a/share/spack/templates/container/Dockerfile +++ b/share/spack/templates/container/Dockerfile @@ -1,12 +1,19 @@ # Build stage with Spack pre-installed and ready to be used -FROM {{ build.image }}:{{ build.tag }} as builder +FROM {{ build.image }} as builder + +{% if os_packages_build %} +# Install OS packages needed to build the software +RUN {% if os_package_update %}{{ os_packages_build.update }} \ + && {% endif %}{{ os_packages_build.install }} {{ os_packages_build.list | join | replace('\n', ' ') }} \ + && {{ os_packages_build.clean }} +{% endif %} # What we want to install and how we want to install it # is specified in a manifest file (spack.yaml) RUN mkdir {{ paths.environment }} \ {{ manifest }} > {{ paths.environment }}/spack.yaml -# Install the software, remove unecessary deps +# Install the software, remove unnecessary deps RUN cd {{ paths.environment }} && spack env activate . && spack install --fail-fast && spack gc -y {% if strip %} @@ -34,16 +41,15 @@ COPY --from=builder {{ paths.store }} {{ paths.store }} COPY --from=builder {{ paths.view }} {{ paths.view }} COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh -{% if os_packages %} -RUN {{ os_packages.update }} \ - && {{ os_packages.install }}{% for pkg in os_packages.list %} {{ pkg }}{% endfor %} \ - && {{ os_packages.clean }} +{% if os_packages_final %} +RUN {% if os_package_update %}{{ os_packages_final.update }} \ + && {% endif %}{{ os_packages_final.install }} {{ os_packages_final.list | join | replace('\n', ' ') }} \ + && {{ os_packages_final.clean }} {% endif %} {% if extra_instructions.final %} {{ extra_instructions.final }} {% endif %} - {% for label, value in labels.items() %} LABEL "{{ label }}"="{{ value }}" {% endfor %} diff --git a/share/spack/templates/container/singularity.def b/share/spack/templates/container/singularity.def index 44b22aeae6..33d775b024 100644 --- a/share/spack/templates/container/singularity.def +++ b/share/spack/templates/container/singularity.def @@ -1,8 +1,17 @@ Bootstrap: docker -From: {{ build.image }}:{{ build.tag }} +From: {{ build.image }} Stage: build %post +{% if os_packages_build.list %} + # Update, install and cleanup of system packages needed at build-time + {% if os_package_update %} + {{ os_packages_build.update }} + {% endif %} + {{ os_packages_build.install }} {{ os_packages_build.list | join | replace('\n', ' ') }} + {{ os_packages_build.clean }} + +{% endif %} # Create the manifest file for the installation in /opt/spack-environment mkdir {{ paths.environment }} && cd {{ paths.environment }} cat << EOF > spack.yaml @@ -29,7 +38,6 @@ EOF {{ extra_instructions.build }} {% endif %} - {% if apps %} {% for application, help_text in apps.items() %} @@ -52,39 +60,41 @@ Stage: final {{ paths.environment }}/environment_modifications.sh {{ paths.environment }}/environment_modifications.sh %post -{% if os_packages.list %} - # Update, install and cleanup of system packages - {{ os_packages.update }} - {{ os_packages.install }} {{ os_packages.list | join | replace('\n', ' ') }} - {{ os_packages.clean }} +{% if os_packages_final.list %} + # Update, install and cleanup of system packages needed at run-time + {% if os_package_update %} + {{ os_packages_final.update }} + {% endif %} + {{ os_packages_final.install }} {{ os_packages_final.list | join | replace('\n', ' ') }} + {{ os_packages_final.clean }} {% endif %} # Modify the environment without relying on sourcing shell specific files at startup cat {{ paths.environment }}/environment_modifications.sh >> $SINGULARITY_ENVIRONMENT {% if extra_instructions.final %} {{ extra_instructions.final }} {% endif %} - {% if runscript %} + %runscript {{ runscript }} {% endif %} - {% if startscript %} + %startscript {{ startscript }} {% endif %} - {% if test %} + %test {{ test }} {% endif %} - {% if help %} + %help {{ help }} {% endif %} - {% if labels %} + %labels {% for label, value in labels.items() %} {{ label }} {{ value }} |