diff options
author | Massimiliano Culpo <massimiliano.culpo@gmail.com> | 2023-04-03 21:05:19 +0200 |
---|---|---|
committer | GitHub <noreply@github.com> | 2023-04-03 21:05:19 +0200 |
commit | f91968cf6fac592787f004420cf6c12164c3e66e (patch) | |
tree | 418d745c90358ab1920e0d75ec484ee726628c89 /lib/spack/docs/containers.rst | |
parent | 3d149a7db2d780d48d48fc86c6c43d2490235532 (diff) | |
download | spack-f91968cf6fac592787f004420cf6c12164c3e66e.tar.gz spack-f91968cf6fac592787f004420cf6c12164c3e66e.tar.bz2 spack-f91968cf6fac592787f004420cf6c12164c3e66e.tar.xz spack-f91968cf6fac592787f004420cf6c12164c3e66e.zip |
Improve Dockerfile recipe generation (#35187)
- Update default image to Ubuntu 22.04 (previously was still Ubuntu 18.04)
- Optionally use depfiles to install the environment within the container
- Allow extending Dockerfile Jinja2 template
- Allow extending Singularity definition file Jinja2 template
- Deprecate previous options to add extra instructions
Diffstat (limited to 'lib/spack/docs/containers.rst')
-rw-r--r-- | lib/spack/docs/containers.rst | 126 |
1 files changed, 118 insertions, 8 deletions
diff --git a/lib/spack/docs/containers.rst b/lib/spack/docs/containers.rst index a919db0642..2c097d366c 100644 --- a/lib/spack/docs/containers.rst +++ b/lib/spack/docs/containers.rst @@ -444,6 +444,120 @@ attribute: The minimum version of Singularity required to build a SIF (Singularity Image Format) image from the recipes generated by Spack is ``3.5.3``. +------------------------------ +Extending the Jinja2 Templates +------------------------------ + +The Dockerfile and the Singularity definition file that Spack can generate are based on +a few Jinja2 templates that are rendered according to the environment being containerized. +Even though Spack allows a great deal of customization by just setting appropriate values for +the configuration options, sometimes that is not enough. + +In those cases, a user can directly extend the template that Spack uses to render the image +to e.g. set additional environment variables or perform specific operations either before or +after a given stage of the build. Let's consider as an example the following structure: + +.. code-block:: console + + $ tree /opt/environment + /opt/environment + ├── data + │ └── data.csv + ├── spack.yaml + ├── data + └── templates + └── container + └── CustomDockerfile + +containing both the custom template extension and the environment manifest file. To use a custom +template, the environment must register the directory containing it, and declare its use under the +``container`` configuration: + +.. code-block:: yaml + :emphasize-lines: 7-8,12 + + spack: + specs: + - hdf5~mpi + concretizer: + unify: true + config: + template_dirs: + - /opt/environment/templates + container: + format: docker + depfile: true + template: container/CustomDockerfile + +The template extension can override two blocks, named ``build_stage`` and ``final_stage``, similarly to +the example below: + +.. code-block:: + :emphasize-lines: 3,8 + + {% extends "container/Dockerfile" %} + {% block build_stage %} + RUN echo "Start building" + {{ super() }} + {% endblock %} + {% block final_stage %} + {{ super() }} + COPY data /share/myapp/data + {% endblock %} + +The recipe that gets generated contains the two extra instruction that we added in our template extension: + +.. code-block:: Dockerfile + :emphasize-lines: 4,43 + + # Build stage with Spack pre-installed and ready to be used + FROM spack/ubuntu-jammy:latest as builder + + RUN echo "Start building" + + # What we want to install and how we want to install it + # is specified in a manifest file (spack.yaml) + RUN mkdir /opt/spack-environment \ + && (echo "spack:" \ + && echo " specs:" \ + && echo " - hdf5~mpi" \ + && echo " concretizer:" \ + && echo " unify: true" \ + && echo " config:" \ + && echo " template_dirs:" \ + && echo " - /tmp/environment/templates" \ + && echo " install_tree: /opt/software" \ + && echo " view: /opt/view") > /opt/spack-environment/spack.yaml + + # Install the software, remove unnecessary deps + RUN cd /opt/spack-environment && spack env activate . && spack concretize && spack env depfile -o Makefile && make -j $(nproc) && spack gc -y + + # Strip all the binaries + RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \ + xargs file -i | \ + grep 'charset=binary' | \ + grep 'x-executable\|x-archive\|x-sharedlib' | \ + awk -F: '{print $1}' | xargs strip -s + + # Modifications to the environment that are necessary to run + RUN cd /opt/spack-environment && \ + spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh + + # Bare OS image to run the installed executables + FROM ubuntu:22.04 + + COPY --from=builder /opt/spack-environment /opt/spack-environment + COPY --from=builder /opt/software /opt/software + COPY --from=builder /opt/._view /opt/._view + COPY --from=builder /opt/view /opt/view + COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh + + COPY data /share/myapp/data + + ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l", "-c", "$*", "--" ] + CMD [ "/bin/bash" ] + + .. _container_config_options: ----------------------- @@ -464,6 +578,10 @@ to customize the generation of container recipes: - The format of the recipe - ``docker`` or ``singularity`` - Yes + * - ``depfile`` + - Whether to use a depfile for installation, or not + - True or False (default) + - No * - ``images:os`` - Operating system used as a base for the image - See :ref:`containers-supported-os` @@ -512,14 +630,6 @@ to customize the generation of container recipes: - System packages needed at run-time - Valid packages for the current OS - No - * - ``extra_instructions:build`` - - Extra instructions (e.g. `RUN`, `COPY`, etc.) at the end of the ``build`` stage - - Anything understood by the current ``format`` - - No - * - ``extra_instructions:final`` - - Extra instructions (e.g. `RUN`, `COPY`, etc.) at the end of the ``final`` stage - - Anything understood by the current ``format`` - - No * - ``labels`` - Labels to tag the image - Pairs of key-value strings |