summaryrefslogtreecommitdiff
path: root/lib
diff options
context:
space:
mode:
authorTodd Gamblin <tgamblin@llnl.gov>2017-09-27 20:03:30 -0700
committerTodd Gamblin <tgamblin@llnl.gov>2017-09-30 02:06:59 -0700
commit96d2488e0cf04d65448a9a9fb3c7e3198e5dc6ff (patch)
treec88c361a515d9ee160331f5c27aa523bf824661c /lib
parent4f8c7d57eb3839ea866a9fbfb55f9a44af99d6c0 (diff)
downloadspack-96d2488e0cf04d65448a9a9fb3c7e3198e5dc6ff.tar.gz
spack-96d2488e0cf04d65448a9a9fb3c7e3198e5dc6ff.tar.bz2
spack-96d2488e0cf04d65448a9a9fb3c7e3198e5dc6ff.tar.xz
spack-96d2488e0cf04d65448a9a9fb3c7e3198e5dc6ff.zip
Documentation for dependency patching.
Diffstat (limited to 'lib')
-rw-r--r--lib/spack/docs/packaging_guide.rst152
-rw-r--r--lib/spack/spack/cmd/md5.py20
-rw-r--r--lib/spack/spack/cmd/sha256.py40
3 files changed, 189 insertions, 23 deletions
diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst
index 402de6d8ed..57eafde2dc 100644
--- a/lib/spack/docs/packaging_guide.rst
+++ b/lib/spack/docs/packaging_guide.rst
@@ -1212,27 +1212,47 @@ structure like this:
package.py
ad_lustre_rwcontig_open_source.patch
-If you supply a URL instead of a filename, you need to supply a checksum,
-like this:
+If you supply a URL instead of a filename, you need to supply a
+``sha256`` checksum, like this:
.. code-block:: python
- patch('http://www.nwchem-sw.org/images/Tddft_mxvec20.patch.gz',
- md5='f91c6a04df56e228fe946291d2f38c9a')
+ patch('http://www.nwchem-sw.org/images/Tddft_mxvec20.patch',
+ sha256='252c0af58be3d90e5dc5e0d16658434c9efa5d20a5df6c10bf72c2d77f780866')
+
+Spack includes the hashes of patches in its versioning information, so
+that the same package with different patches applied will have different
+hash identifiers. To ensure that the hashing scheme is consistent, you
+must use a ``sha256`` checksum for the patch. Patches will be fetched
+from their URLs, checked, and applied to your source code. You can use
+the ``spack sha256`` command to generate a checksum for a patch file or
+URL.
-This directive provides an ``md5`` checksum. You can use other hashing
-algorihtms like ``sha256`` as well. The patch will be fetched from the
-URL, checked, and applied to your source code. You can use the ``spack
-md5`` command to generate a checksum for a patch file.
+Spack can also handle compressed patches. If you use these, Spack needs
+a little more help. Specifically, it needs *two* checksums: the
+``sha256`` of the patch and ``archive_sha256`` for the compressed
+archive. ``archive_sha256`` helps Spack ensure that the downloaded
+file is not corrupted or malicious, before running it through a tool like
+``tar`` or ``zip``. The ``sha256`` of the patch is still required so
+that it can be included in specs. Providing it in the package file
+ensures that Spack won't have to download and decompress patches it won't
+end up using at install time. Both the archive and patch checksum are
+checked when patch archives are downloaded.
-``patch`` can take two options keyword arguments. They are:
+.. code-block:: python
+
+ patch('http://www.nwchem-sw.org/images/Tddft_mxvec20.patch.gz',
+ sha256='252c0af58be3d90e5dc5e0d16658434c9efa5d20a5df6c10bf72c2d77f780866',
+ archive_sha256='4e8092a161ec6c3a1b5253176fcf33ce7ba23ee2ff27c75dbced589dabacd06e')
-""""""""""""""""""""""""""""""""""""""
-``md5``, ``sha256``, ``sha512``, etc.
-""""""""""""""""""""""""""""""""""""""
+``patch`` keyword arguments are described below.
-Use one of these when you supply a patch to be downloaded from a remote
-site. The downloaded file will be validated using the given checksum.
+""""""""""""""""""""""""""""""
+``sha256``, ``archive_sha256``
+""""""""""""""""""""""""""""""
+
+Hashes of downloaded patch and compressed archive, respectively. Only
+needed for patches fetched from URLs.
""""""""
``when``
@@ -1309,6 +1329,21 @@ if you run install, hit ctrl-C, and run install again, the code in the
patch function is only run once. Also, you can tell Spack to run only
the patching part of the build using the :ref:`cmd-spack-patch` command.
+.. _patch_dependency_patching:
+
+^^^^^^^^^^^^^^^^^^^
+Dependency patching
+^^^^^^^^^^^^^^^^^^^
+
+So far we've covered how the ``patch`` directive can be used by a package
+to patch *its own* source code. Packages can *also* specify patches to be
+applied to their dependencies, if they require special modifications. As
+with all packages in Spack, a patched dependency library can coexist with
+other versions of that library. See the `section on depends_on
+<dependency_dependency_patching_>`_ for more details.
+
+.. _handling_rpaths:
+
---------------
Handling RPATHs
---------------
@@ -1482,7 +1517,7 @@ particular constraints, and package authors can use specs to describe
relationships between packages.
^^^^^^^^^^^^^^
-Version Ranges
+Version ranges
^^^^^^^^^^^^^^
Although some packages require a specific version for their dependencies,
@@ -1530,7 +1565,7 @@ correct way to specify this would be:
^^^^^^^^^^^^^^^^
-Dependency Types
+Dependency types
^^^^^^^^^^^^^^^^
Not all dependencies are created equal, and Spack allows you to specify
@@ -1566,6 +1601,91 @@ inject the dependency's ``prefix/lib`` directory, but the package needs to
be in ``PATH`` and ``PYTHONPATH`` during the build process and later when
a user wants to run the package.
+.. _dependency_dependency_patching:
+
+^^^^^^^^^^^^^^^^^^^
+Dependency patching
+^^^^^^^^^^^^^^^^^^^
+
+Some packages maintain special patches on their dependencies, either to
+add new features or to fix bugs. This typically makes a package harder
+to maintain, and we encourage developers to upstream (contribute back)
+their changes rather than maintaining patches. However, in some cases
+it's not possible to upstream. Maybe the dependency's developers don't
+accept changes, or maybe they just haven't had time to integrate them.
+
+For times like these, Spack's ``depends_on`` directive can optionally
+take a patch or list of patches:
+
+.. code-block:: python
+
+ class SpecialTool(Package):
+ ...
+ depends_on('binutils', patches='special-binutils-feature.patch')
+ ...
+
+Here, the ``special-tool`` package requires a special feature in
+``binutils``, so it provides an extra ``patches=<filename>`` keyword
+argument. This is similar to the `patch directive <patching_>`_, with
+one small difference. Here, ``special-tool`` is responsible for the
+patch, so it should live in ``special-tool``'s directory in the package
+repository, not the ``binutils`` directory.
+
+If you need something more sophisticated than this, you can simply nest a
+``patch()`` directive inside of ``depends_on``:
+
+.. code-block:: python
+
+ class SpecialTool(Package):
+ ...
+ depends_on(
+ 'binutils',
+ patches=patch('special-binutils-feature.patch',
+ level=3,
+ when='@:1.3'), # condition on binutils
+ when='@2.0:') # condition on special-tool
+ ...
+
+Note that there are two optional ``when`` conditions here -- one on the
+``patch`` directive and the other on ``depends_on``. The condition in
+the ``patch`` directive applies to ``binutils`` (the package being
+patched), while the condition in ``depends_on`` applies to
+``special-tool``. See `patch directive <patching_>`_ for details on all
+the arguments the ``patch`` directive can take.
+
+Finally, if you need *multiple* patches on a dependency, you can provide
+a list for ``patches``, e.g.:
+
+.. code-block:: python
+
+ class SpecialTool(Package):
+ ...
+ depends_on(
+ 'binutils',
+ patches=[
+ 'binutils-bugfix1.patch',
+ 'binutils-bugfix2.patch',
+ patch('https://example.com/special-binutils-feature.patch',
+ sha256='252c0af58be3d90e5dc5e0d16658434c9efa5d20a5df6c10bf72c2d77f780866',
+ when='@:1.3')],
+ when='@2.0:')
+ ...
+
+As with ``patch`` directives, patches are applied in the order they
+appear in the package file (or in this case, in the list).
+
+.. note::
+
+ You may wonder whether dependency patching will interfere with other
+ packages that depend on ``binutils``. It won't.
+
+ As described in patching_, Patching a package adds the ``sha256`` of
+ the patch to the package's spec, which means it will have a
+ *different* unique hash than other versions without the patch. The
+ patched version coexists with unpatched versions, and Spack's support
+ for handling_rpaths_ guarantees that each installation finds the
+ right version. If two packages depend on ``binutils`` patched *the
+ same* way, they can both use a single installation of ``binutils``.
.. _setup-dependent-environment:
diff --git a/lib/spack/spack/cmd/md5.py b/lib/spack/spack/cmd/md5.py
index 6f6a4af237..2decbaaee1 100644
--- a/lib/spack/spack/cmd/md5.py
+++ b/lib/spack/spack/cmd/md5.py
@@ -42,13 +42,15 @@ def setup_parser(subparser):
help="files/urls to checksum")
-def compute_md5_checksum(url):
+def compute_checksum(url, algo):
+ algo = getattr(hashlib, algo)
+
if not os.path.isfile(url):
with Stage(url) as stage:
stage.fetch()
- value = spack.util.crypto.checksum(hashlib.md5, stage.archive_file)
+ value = spack.util.crypto.checksum(algo, stage.archive_file)
else:
- value = spack.util.crypto.checksum(hashlib.md5, url)
+ value = spack.util.crypto.checksum(algo, url)
return value
@@ -61,7 +63,7 @@ def normalized(files):
yield value
-def md5(parser, args):
+def do_checksum(parser, args, algo):
if not args.files:
setup_parser.parser.print_help()
return 1
@@ -70,7 +72,7 @@ def md5(parser, args):
results = []
for url in urls:
try:
- checksum = compute_md5_checksum(url)
+ checksum = compute_checksum(url, algo)
results.append((checksum, url))
except FailedDownloadError as e:
tty.warn("Failed to fetch %s" % url)
@@ -79,8 +81,12 @@ def md5(parser, args):
tty.warn("Error when reading %s" % url)
tty.warn("%s" % e)
- # Dump the MD5s at last without interleaving them with downloads
+ # Dump the hashes last, without interleaving them with downloads
checksum = 'checksum' if len(results) == 1 else 'checksums'
- tty.msg("%d MD5 %s:" % (len(results), checksum))
+ tty.msg("%d %s %s:" % (len(results), algo, checksum))
for checksum, url in results:
print("{0} {1}".format(checksum, url))
+
+
+def md5(parser, args):
+ do_checksum(parser, args, 'md5')
diff --git a/lib/spack/spack/cmd/sha256.py b/lib/spack/spack/cmd/sha256.py
new file mode 100644
index 0000000000..a5572eade5
--- /dev/null
+++ b/lib/spack/spack/cmd/sha256.py
@@ -0,0 +1,40 @@
+##############################################################################
+# Copyright (c) 2013-2017, Lawrence Livermore National Security, LLC.
+# Produced at the Lawrence Livermore National Laboratory.
+#
+# This file is part of Spack.
+# Created by Todd Gamblin, tgamblin@llnl.gov, All rights reserved.
+# LLNL-CODE-647188
+#
+# For details, see https://github.com/llnl/spack
+# Please also see the NOTICE and LICENSE files for our notice and the LGPL.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License (as
+# published by the Free Software Foundation) version 2.1, February 1999.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the IMPLIED WARRANTY OF
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the terms and
+# conditions of the GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+##############################################################################
+import argparse
+from spack.cmd.md5 import do_checksum
+
+description = "calculate sha256 checksums for files/urls"
+section = "packaging"
+level = "long"
+
+
+def setup_parser(subparser):
+ setup_parser.parser = subparser
+ subparser.add_argument('files', nargs=argparse.REMAINDER,
+ help="files/urls to checksum")
+
+
+def sha256(parser, args):
+ do_checksum(parser, args, 'sha256')