diff options
| -rw-r--r-- | lib/spack/docs/packaging_guide.rst | 152 | ||||
| -rw-r--r-- | lib/spack/spack/cmd/md5.py | 20 | ||||
| -rw-r--r-- | lib/spack/spack/cmd/sha256.py | 40 |
3 files changed, 189 insertions, 23 deletions
diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst index 402de6d8ed..57eafde2dc 100644 --- a/lib/spack/docs/packaging_guide.rst +++ b/lib/spack/docs/packaging_guide.rst @@ -1212,27 +1212,47 @@ structure like this: package.py ad_lustre_rwcontig_open_source.patch -If you supply a URL instead of a filename, you need to supply a checksum, -like this: +If you supply a URL instead of a filename, you need to supply a +``sha256`` checksum, like this: .. code-block:: python - patch('http://www.nwchem-sw.org/images/Tddft_mxvec20.patch.gz', - md5='f91c6a04df56e228fe946291d2f38c9a') + patch('http://www.nwchem-sw.org/images/Tddft_mxvec20.patch', + sha256='252c0af58be3d90e5dc5e0d16658434c9efa5d20a5df6c10bf72c2d77f780866') + +Spack includes the hashes of patches in its versioning information, so +that the same package with different patches applied will have different +hash identifiers. To ensure that the hashing scheme is consistent, you +must use a ``sha256`` checksum for the patch. Patches will be fetched +from their URLs, checked, and applied to your source code. You can use +the ``spack sha256`` command to generate a checksum for a patch file or +URL. -This directive provides an ``md5`` checksum. You can use other hashing -algorihtms like ``sha256`` as well. The patch will be fetched from the -URL, checked, and applied to your source code. You can use the ``spack -md5`` command to generate a checksum for a patch file. +Spack can also handle compressed patches. If you use these, Spack needs +a little more help. Specifically, it needs *two* checksums: the +``sha256`` of the patch and ``archive_sha256`` for the compressed +archive. ``archive_sha256`` helps Spack ensure that the downloaded +file is not corrupted or malicious, before running it through a tool like +``tar`` or ``zip``. The ``sha256`` of the patch is still required so +that it can be included in specs. Providing it in the package file +ensures that Spack won't have to download and decompress patches it won't +end up using at install time. Both the archive and patch checksum are +checked when patch archives are downloaded. -``patch`` can take two options keyword arguments. They are: +.. code-block:: python + + patch('http://www.nwchem-sw.org/images/Tddft_mxvec20.patch.gz', + sha256='252c0af58be3d90e5dc5e0d16658434c9efa5d20a5df6c10bf72c2d77f780866', + archive_sha256='4e8092a161ec6c3a1b5253176fcf33ce7ba23ee2ff27c75dbced589dabacd06e') -"""""""""""""""""""""""""""""""""""""" -``md5``, ``sha256``, ``sha512``, etc. -"""""""""""""""""""""""""""""""""""""" +``patch`` keyword arguments are described below. -Use one of these when you supply a patch to be downloaded from a remote -site. The downloaded file will be validated using the given checksum. +"""""""""""""""""""""""""""""" +``sha256``, ``archive_sha256`` +"""""""""""""""""""""""""""""" + +Hashes of downloaded patch and compressed archive, respectively. Only +needed for patches fetched from URLs. """""""" ``when`` @@ -1309,6 +1329,21 @@ if you run install, hit ctrl-C, and run install again, the code in the patch function is only run once. Also, you can tell Spack to run only the patching part of the build using the :ref:`cmd-spack-patch` command. +.. _patch_dependency_patching: + +^^^^^^^^^^^^^^^^^^^ +Dependency patching +^^^^^^^^^^^^^^^^^^^ + +So far we've covered how the ``patch`` directive can be used by a package +to patch *its own* source code. Packages can *also* specify patches to be +applied to their dependencies, if they require special modifications. As +with all packages in Spack, a patched dependency library can coexist with +other versions of that library. See the `section on depends_on +<dependency_dependency_patching_>`_ for more details. + +.. _handling_rpaths: + --------------- Handling RPATHs --------------- @@ -1482,7 +1517,7 @@ particular constraints, and package authors can use specs to describe relationships between packages. ^^^^^^^^^^^^^^ -Version Ranges +Version ranges ^^^^^^^^^^^^^^ Although some packages require a specific version for their dependencies, @@ -1530,7 +1565,7 @@ correct way to specify this would be: ^^^^^^^^^^^^^^^^ -Dependency Types +Dependency types ^^^^^^^^^^^^^^^^ Not all dependencies are created equal, and Spack allows you to specify @@ -1566,6 +1601,91 @@ inject the dependency's ``prefix/lib`` directory, but the package needs to be in ``PATH`` and ``PYTHONPATH`` during the build process and later when a user wants to run the package. +.. _dependency_dependency_patching: + +^^^^^^^^^^^^^^^^^^^ +Dependency patching +^^^^^^^^^^^^^^^^^^^ + +Some packages maintain special patches on their dependencies, either to +add new features or to fix bugs. This typically makes a package harder +to maintain, and we encourage developers to upstream (contribute back) +their changes rather than maintaining patches. However, in some cases +it's not possible to upstream. Maybe the dependency's developers don't +accept changes, or maybe they just haven't had time to integrate them. + +For times like these, Spack's ``depends_on`` directive can optionally +take a patch or list of patches: + +.. code-block:: python + + class SpecialTool(Package): + ... + depends_on('binutils', patches='special-binutils-feature.patch') + ... + +Here, the ``special-tool`` package requires a special feature in +``binutils``, so it provides an extra ``patches=<filename>`` keyword +argument. This is similar to the `patch directive <patching_>`_, with +one small difference. Here, ``special-tool`` is responsible for the +patch, so it should live in ``special-tool``'s directory in the package +repository, not the ``binutils`` directory. + +If you need something more sophisticated than this, you can simply nest a +``patch()`` directive inside of ``depends_on``: + +.. code-block:: python + + class SpecialTool(Package): + ... + depends_on( + 'binutils', + patches=patch('special-binutils-feature.patch', + level=3, + when='@:1.3'), # condition on binutils + when='@2.0:') # condition on special-tool + ... + +Note that there are two optional ``when`` conditions here -- one on the +``patch`` directive and the other on ``depends_on``. The condition in +the ``patch`` directive applies to ``binutils`` (the package being +patched), while the condition in ``depends_on`` applies to +``special-tool``. See `patch directive <patching_>`_ for details on all +the arguments the ``patch`` directive can take. + +Finally, if you need *multiple* patches on a dependency, you can provide +a list for ``patches``, e.g.: + +.. code-block:: python + + class SpecialTool(Package): + ... + depends_on( + 'binutils', + patches=[ + 'binutils-bugfix1.patch', + 'binutils-bugfix2.patch', + patch('https://example.com/special-binutils-feature.patch', + sha256='252c0af58be3d90e5dc5e0d16658434c9efa5d20a5df6c10bf72c2d77f780866', + when='@:1.3')], + when='@2.0:') + ... + +As with ``patch`` directives, patches are applied in the order they +appear in the package file (or in this case, in the list). + +.. note:: + + You may wonder whether dependency patching will interfere with other + packages that depend on ``binutils``. It won't. + + As described in patching_, Patching a package adds the ``sha256`` of + the patch to the package's spec, which means it will have a + *different* unique hash than other versions without the patch. The + patched version coexists with unpatched versions, and Spack's support + for handling_rpaths_ guarantees that each installation finds the + right version. If two packages depend on ``binutils`` patched *the + same* way, they can both use a single installation of ``binutils``. .. _setup-dependent-environment: diff --git a/lib/spack/spack/cmd/md5.py b/lib/spack/spack/cmd/md5.py index 6f6a4af237..2decbaaee1 100644 --- a/lib/spack/spack/cmd/md5.py +++ b/lib/spack/spack/cmd/md5.py @@ -42,13 +42,15 @@ def setup_parser(subparser): help="files/urls to checksum") -def compute_md5_checksum(url): +def compute_checksum(url, algo): + algo = getattr(hashlib, algo) + if not os.path.isfile(url): with Stage(url) as stage: stage.fetch() - value = spack.util.crypto.checksum(hashlib.md5, stage.archive_file) + value = spack.util.crypto.checksum(algo, stage.archive_file) else: - value = spack.util.crypto.checksum(hashlib.md5, url) + value = spack.util.crypto.checksum(algo, url) return value @@ -61,7 +63,7 @@ def normalized(files): yield value -def md5(parser, args): +def do_checksum(parser, args, algo): if not args.files: setup_parser.parser.print_help() return 1 @@ -70,7 +72,7 @@ def md5(parser, args): results = [] for url in urls: try: - checksum = compute_md5_checksum(url) + checksum = compute_checksum(url, algo) results.append((checksum, url)) except FailedDownloadError as e: tty.warn("Failed to fetch %s" % url) @@ -79,8 +81,12 @@ def md5(parser, args): tty.warn("Error when reading %s" % url) tty.warn("%s" % e) - # Dump the MD5s at last without interleaving them with downloads + # Dump the hashes last, without interleaving them with downloads checksum = 'checksum' if len(results) == 1 else 'checksums' - tty.msg("%d MD5 %s:" % (len(results), checksum)) + tty.msg("%d %s %s:" % (len(results), algo, checksum)) for checksum, url in results: print("{0} {1}".format(checksum, url)) + + +def md5(parser, args): + do_checksum(parser, args, 'md5') diff --git a/lib/spack/spack/cmd/sha256.py b/lib/spack/spack/cmd/sha256.py new file mode 100644 index 0000000000..a5572eade5 --- /dev/null +++ b/lib/spack/spack/cmd/sha256.py @@ -0,0 +1,40 @@ +############################################################################## +# Copyright (c) 2013-2017, Lawrence Livermore National Security, LLC. +# Produced at the Lawrence Livermore National Laboratory. +# +# This file is part of Spack. +# Created by Todd Gamblin, tgamblin@llnl.gov, All rights reserved. +# LLNL-CODE-647188 +# +# For details, see https://github.com/llnl/spack +# Please also see the NOTICE and LICENSE files for our notice and the LGPL. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU Lesser General Public License (as +# published by the Free Software Foundation) version 2.1, February 1999. +# +# This program is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the IMPLIED WARRANTY OF +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the terms and +# conditions of the GNU Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +############################################################################## +import argparse +from spack.cmd.md5 import do_checksum + +description = "calculate sha256 checksums for files/urls" +section = "packaging" +level = "long" + + +def setup_parser(subparser): + setup_parser.parser = subparser + subparser.add_argument('files', nargs=argparse.REMAINDER, + help="files/urls to checksum") + + +def sha256(parser, args): + do_checksum(parser, args, 'sha256') |