diff options
Diffstat (limited to 'lib/spack/docs/packaging_guide.rst')
| -rw-r--r-- | lib/spack/docs/packaging_guide.rst | 308 |
1 files changed, 214 insertions, 94 deletions
diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst index 70cd58f6c1..6efb0078f6 100644 --- a/lib/spack/docs/packaging_guide.rst +++ b/lib/spack/docs/packaging_guide.rst @@ -373,6 +373,107 @@ some examples: In general, you won't have to remember this naming convention because :ref:`spack-create` and :ref:`spack-edit` handle the details for you. +----------------- +Trusted Downloads +----------------- + +Spack verifies that the source code it downloads is not corrupted or +compromised; or at least, that it is the same version the author of +the Spack package saw when the package was created. If Spack uses a +download method it can verify, we say the download method is +*trusted*. Trust is important for *all downloads*: Spack +has no control over the security of the various sites from which it +downloads source code, and can never assume that any particular site +hasn't been compromised. + +Trust is established in different ways for different download methods. +For the most common download method --- a single-file tarball --- the +tarball is checksummed. Git downloads using ``commit=`` are trusted +implicitly, as long as a hash is specified. + +Spack also provides untrusted download methods: tarball URLs may be +supplied without a checksum, or Git downloads may specify a branch or +tag instead of a hash. If the user does not control or trust the +source of an untrusted download, it is a security risk. Unless otherwise +specified by the user for special cases, Spack should by default use +*only* trusted download methods. + +Unfortunately, Spack does not currently provide that guarantee. It +does provide the following mechanisms for safety: + +#. By default, Spack will only install a tarball package if it has a + checksum and that checksum matches. You can override this with + ``spack install --no-checksum``. + +#. Numeric versions are almost always tarball downloads, whereas + non-numeric versions not named ``develop`` frequently download + untrusted branches or tags from a version control system. As long + as a package has at least one numeric version, and no non-numeric + version named ``develop``, Spack will prefer it over any + non-numeric versions. + +^^^^^^^^^ +Checksums +^^^^^^^^^ + +For tarball downloads, Spack can currently support checksums using the +MD5, SHA-1, SHA-224, SHA-256, SHA-384, and SHA-512 algorithms. It +determines the algorithm to use based on the hash length. + +----------------------- +Package Version Numbers +----------------------- + +Most Spack versions are numeric, a tuple of integers; for example, +``apex@0.1``, ``ferret@6.96`` or ``py-netcdf@1.2.3.1``. Spack knows +how to compare and sort numeric versions. + +Some Spack versions involve slight extensions of numeric syntax; for +example, ``py-sphinx-rtd-theme@0.1.10a0``. In this case, numbers are +always considered to be "newer" than letters. This is for consistency +with `RPM <https://bugzilla.redhat.com/show_bug.cgi?id=50977>`. + +Spack versions may also be arbitrary non-numeric strings; any string +here will suffice; for example, ``@develop``, ``@master``, ``@local``. +The following rules determine the sort order of numeric +vs. non-numeric versions: + +#. The non-numeric versions ``@develop`` is considered greatest (newest). + +#. Numeric versions are all less than ``@develop`` version, and are + sorted numerically. + +#. All other non-numeric versions are less than numeric versions, and + are sorted alphabetically. + +The logic behind this sort order is two-fold: + +#. Non-numeric versions are usually used for special cases while + developing or debugging a piece of software. Keeping most of them + less than numeric versions ensures that Spack choose numeric + versions by default whenever possible. + +#. The most-recent development version of a package will usually be + newer than any released numeric versions. This allows the + ``develop`` version to satisfy dependencies like ``depends_on(abc, + when="@x.y.z:")`` + + +^^^^^^^^^^^^^ +Date Versions +^^^^^^^^^^^^^ + +If you wish to use dates as versions, it is best to use the format +``@date-yyyy-mm-dd``. This will ensure they sort in the correct +order. If you want your date versions to be numeric (assuming they +don't conflict with other numeric versions), you can use just +``yyyy.mm.dd``. + +Alternately, you might use a hybrid release-version / date scheme. +For example, ``@1.3.2016.08.31`` would mean the version from the +``1.3`` branch, as of August 31, 2016. + + ------------------- Adding new versions ------------------- @@ -459,19 +560,6 @@ it executable, then runs it with some arguments. installer = Executable(self.stage.archive_file) installer('--prefix=%s' % prefix, 'arg1', 'arg2', 'etc.') -^^^^^^^^^ -Checksums -^^^^^^^^^ - -Spack uses a checksum to ensure that the downloaded package version is -not corrupted or compromised. This is especially important when -fetching from insecure sources, like unencrypted http. By default, a -package will *not* be installed if it doesn't pass a checksum test -(though you can override this with ``spack install --no-checksum``). - -Spack can currently support checksums using the MD5, SHA-1, SHA-224, -SHA-256, SHA-384, and SHA-512 algorithms. It determines the algorithm -to use based on the hash length. ^^^^^^^^^^^^^ ``spack md5`` @@ -584,39 +672,6 @@ call to your package with parameters indicating the repository URL and any branch, tag, or revision to fetch. See below for the parameters you'll need for each VCS system. -^^^^^^^^^^^^^^^^^^^^^^^^^ -Repositories and versions -^^^^^^^^^^^^^^^^^^^^^^^^^ - -The package author is responsible for coming up with a sensible name -for each version to be fetched from a repository. For example, if -you're fetching from a tag like ``v1.0``, you might call that ``1.0``. -If you're fetching a nameless git commit or an older subversion -revision, you might give the commit an intuitive name, like ``develop`` -for a development version, or ``some-fancy-new-feature`` if you want -to be more specific. - -In general, it's recommended to fetch tags or particular -commits/revisions, NOT branches or the repository mainline, as -branches move forward over time and you aren't guaranteed to get the -same thing every time you fetch a particular version. Life isn't -always simple, though, so this is not strictly enforced. - -When fetching from from the branch corresponding to the development version -(often called ``master``, ``trunk``, or ``dev``), it is recommended to -call this version ``develop``. Spack has special treatment for this version so -that ``@develop`` will satisfy dependencies like -``depends_on(abc, when="@x.y.z:")``. In other words, ``@develop`` is -greater than any other version. The rationale is that certain features or -options first appear in the development branch. Therefore if a package author -wants to keep the package on the bleeding edge and provide support for new -features, it is advised to use ``develop`` for such a version which will -greatly simplify writing dependencies and version-related conditionals. - -In some future release, Spack may support extrapolating repository -versions as it does for tarball URLs, but currently this is not -supported. - .. _git-fetch: ^^^ @@ -642,8 +697,7 @@ Default branch ... version('develop', git='https://github.com/example-project/example.git') - This is not recommended, as the contents of the default branch - change over time. + This download method is untrusted, and is not recommended. Tags To fetch from a particular tag, use the ``tag`` parameter along with @@ -654,6 +708,8 @@ Tags version('1.0.1', git='https://github.com/example-project/example.git', tag='v1.0.1') + This download method is untrusted, and is not recommended. + Branches To fetch a particular branch, use ``branch`` instead: @@ -662,8 +718,7 @@ Branches version('experimental', git='https://github.com/example-project/example.git', branch='experimental') - This is not recommended, as the contents of branches change over - time. + This download method is untrusted, and is not recommended. Commits Finally, to fetch a particular commit, use ``commit``: @@ -681,6 +736,9 @@ Commits version('2014-10-08', git='https://github.com/example-project/example.git', commit='9d38cd') + This download method *is trusted*. It is the recommended way to + securely download from a Git repository. + It may be useful to provide a saner version for commits like this, e.g. you might use the date as the version, as done above. Or you could just use the abbreviated commit hash. It's up to the package @@ -696,19 +754,24 @@ Submodules version('1.0.1', git='https://github.com/example-project/example.git', tag='v1.0.1', submdoules=True) -^^^^^^^^^^ -Installing -^^^^^^^^^^ -You can fetch and install any of the versions above as you'd expect, -by using ``@<version>`` in a spec: +.. _github-fetch: -.. code-block:: console +"""""" +GitHub +"""""" - $ spack install example@2014-10-08 +If a project is hosted on GitHub, *any* valid Git branch, tag or hash +may be downloaded as a tarball. This is accomplished simply by +constructing an appropriate URL. Spack can checksum any package +downloaded this way, thereby producing a trusted download. For +example, the following downloads a particular hash, and then applies a +checksum. -Git and other VCS versions will show up in the list of versions when -a user runs ``spack info <package name>``. +.. code-block:: python + + version('1.9.5.1.1', 'd035e4bc704d136db79b43ab371b27d2', + url='https://www.github.com/jswhit/pyproj/tarball/0be612cc9f972e38b50a90c946a9b353e2ab140f') .. _hg-fetch: @@ -726,8 +789,7 @@ Default version('develop', hg='https://jay.grs.rwth-aachen.de/hg/example') - Note that this is not recommended; try to fetch a particular - revision instead. + This download method is untrusted, and is not recommended. Revisions Add ``hg`` and ``revision`` parameters: @@ -737,6 +799,8 @@ Revisions version('1.0', hg='https://jay.grs.rwth-aachen.de/hg/example', revision='v1.0') + This download method is untrusted, and is not recommended. + Unlike ``git``, which has special parameters for different types of revisions, you can use ``revision`` for branches, tags, and commits when you fetch with Mercurial. @@ -759,7 +823,7 @@ Fetching the head version('develop', svn='https://outreach.scidac.gov/svn/libmonitor/trunk') - This is not recommended, as the head will move forward over time. + This download method is untrusted, and is not recommended. Fetching a revision To fetch a particular revision, add a ``revision`` to the @@ -770,6 +834,8 @@ Fetching a revision version('develop', svn='https://outreach.scidac.gov/svn/libmonitor/trunk', revision=128) + This download method is untrusted, and is not recommended. + Subversion branches are handled as part of the directory structure, so you can check out a branch or tag by changing the ``url``. @@ -1345,31 +1411,34 @@ Additionally, dependencies may be specified for specific use cases: The dependency types are: - * **"build"**: The dependency package is made available during the - package's build. While the package is built, the dependency - package's install directory will be added to ``PATH``, the - compiler include and library paths, as well as ``PYTHONPATH``. - This only applies during this package's build; other packages - which depend on this one will not know about the dependency - package. In other words, building another project Y doesn't know - about this project X's build dependencies. - * **"link"**: The dependency package is linked against by this - package, presumably via shared libraries. The dependency package - will be added to this package's run-time library search path - ``rpath``. - * **"run"**: The dependency package is used by this package at run - time. The dependency package will be added to both ``PATH`` and - ``PYTHONPATH`` at run time, but not during build time. **"link"** - and **"run"** are similar in that they both describe a dependency - that exists when the package is used, but they differ in the - mechanism: **"link"** is via shared libraries, and **"run"** via - an explicit search. - -If not specified, ``type`` is assumed to be ``("build", "link")``. -This is the common case for compiled language usage. Also available -are the aliases ``"alldeps"`` for all dependency types combined, and -``"nolink"`` (``("build", "run")``) for use by dependencies which are -not expressed via a linker (e.g., Python or Lua module loading). + * **"build"**: made available during the project's build. The package will + be added to ``PATH``, the compiler include paths, and ``PYTHONPATH``. + Other projects which depend on this one will not have these modified + (building project X doesn't need project Y's build dependencies). + * **"link"**: the project is linked to by the project. The package will be + added to the current package's ``rpath``. + * **"run"**: the project is used by the project at runtime. The package will + be added to ``PATH`` and ``PYTHONPATH``. + +Additional hybrid dependency types are (note the lack of quotes): + + * **<not specified>**: ``type`` assumed to be ``("build", + "link")``. This is the common case for compiled language usage. + * **alldeps**: All dependency types. **Note:** No quotes here + * **nolink**: Equal to ``("build", "run")``, for use by dependencies + that are not expressed via a linker (e.g., Python or Lua module + loading). **Note:** No quotes here + +""""""""""""""""""" +Dependency Formulas +""""""""""""""""""" + +This section shows how to write appropriate ``depends_on()`` +declarations for some common cases. + +* Python 2 only: ``depends_on('python@:2.8')`` +* Python 2.7 only: ``depends_on('python@2.7:2.8')`` +* Python 3 only: ``depends_on('python@3:')`` .. _setup-dependent-environment: @@ -1458,6 +1527,17 @@ Now, the ``py-numpy`` package can be used as an argument to ``spack activate``. When it is activated, all the files in its prefix will be symbolically linked into the prefix of the python package. +Some packages produce a Python extension, but are only compatible with +Python 3, or with Python 2. In those cases, a ``depends_on()`` +declaration should be made in addition to the ``extends()`` +declaration: + +.. code-block:: python + + class Icebin(Package): + extends('python', when='+python') + depends_on('python@3:', when='+python') + Many packages produce Python extensions for *some* variants, but not others: they should extend ``python`` only if the appropriate variant(s) are selected. This may be accomplished with conditional @@ -1817,6 +1897,46 @@ See the :ref:`concretization-preferences` section for more details. .. _install-method: +------------------ +Inconsistent Specs +------------------ + +Suppose a user needs to install package C, which depends on packages A +and B. Package A builds a library with a Python2 extension, and +package B builds a library with a Python3 extension. Packages A and B +cannot be loaded together in the same Python runtime: + +.. code-block:: python + + class A(Package): + variant('python', default=True, 'enable python bindings') + depends_on('python@2.7', when='+python') + def install(self, spec, prefix): + # do whatever is necessary to enable/disable python + # bindings according to variant + + class B(Package): + variant('python', default=True, 'enable python bindings') + depends_on('python@3.2:', when='+python') + def install(self, spec, prefix): + # do whatever is necessary to enable/disable python + # bindings according to variant + +Package C needs to use the libraries from packages A and B, but does +not need either of the Python extensions. In this case, package C +should simply depend on the ``~python`` variant of A and B: + +.. code-block:: python + + class C(Package): + depends_on('A~python') + depends_on('B~python') + +This may require that A or B be built twice, if the user wishes to use +the Python extensions provided by them: once for ``+python`` and once +for ``~python``. Other than using a little extra disk space, that +solution has no serious problems. + ----------------------------------- Implementing the ``install`` method ----------------------------------- @@ -3027,15 +3147,15 @@ might write: CXXFLAGS += -I$DWARF_PREFIX/include CXXFLAGS += -L$DWARF_PREFIX/lib ----------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Build System Configuration Support ----------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Imagine a developer creating a CMake or Autotools-based project in a local -directory, which depends on libraries A-Z. Once Spack has installed -those dependencies, one would like to run ``cmake`` with appropriate -command line and environment so CMake can find them. The ``spack -setup`` command does this conveniently, producing a CMake +Imagine a developer creating a CMake or Autotools-based project in a +local directory, which depends on libraries A-Z. Once Spack has +installed those dependencies, one would like to run ``cmake`` with +appropriate command line and environment so CMake can find them. The +``spack setup`` command does this conveniently, producing a CMake configuration that is essentially the same as how Spack *would have* configured the project. This can be demonstrated with a usage example: |