From 7902f3eb49b616c681e6ca69d2ec70d1d1cbaa21 Mon Sep 17 00:00:00 2001 From: "Adam J. Stewart" Date: Wed, 25 Jul 2018 20:35:12 -0500 Subject: Add docs on new top-level VCS attributes --- lib/spack/docs/packaging_guide.rst | 289 +++++++++++++++++++++++-------------- 1 file changed, 181 insertions(+), 108 deletions(-) (limited to 'lib') diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst index 706144403d..12abcfd7ea 100644 --- a/lib/spack/docs/packaging_guide.rst +++ b/lib/spack/docs/packaging_guide.rst @@ -413,27 +413,26 @@ The most straightforward way to add new versions to your package is to add a line like this in the package class: .. code-block:: python - :linenos: class Foo(Package): - url = 'http://example.com/foo-1.0.tar.gz' + + url = "http://example.com/foo-1.0.tar.gz" + version('8.2.1', '4136d7b4c04df68b686570afa26988ac') - ... + version('8.2.0', '1c9f62f0778697a09d36121ead88e08e') + version('8.1.2', 'd47dd09ed7ae6e7fd6f9a816d7f5fdf6') -Versions should be listed with the newest version first. +Versions should be listed in descending order, from newest to oldest. ^^^^^^^^^^^^^ Date Versions ^^^^^^^^^^^^^ If you wish to use dates as versions, it is best to use the format -``@date-yyyy-mm-dd``. This will ensure they sort in the correct -order. If you want your date versions to be numeric (assuming they -don't conflict with other numeric versions), you can use just -``yyyy.mm.dd``. +``@yyyy-mm-dd``. This will ensure they sort in the correct order. Alternately, you might use a hybrid release-version / date scheme. -For example, ``@1.3.2016.08.31`` would mean the version from the +For example, ``@1.3_2016-08-31`` would mean the version from the ``1.3`` branch, as of August 31, 2016. ^^^^^^^^^^^^ @@ -442,7 +441,7 @@ Version URLs By default, each version's URL is extrapolated from the ``url`` field in the package. For example, Spack is smart enough to download -version ``8.2.1.`` of the ``Foo`` package above from +version ``8.2.1`` of the ``Foo`` package above from http://example.com/foo-8.2.1.tar.gz. If the URL is particularly complicated or changes based on the release, @@ -515,44 +514,28 @@ of its versions, you can add an explicit URL for a particular version: When you supply a custom URL for a version, Spack uses that URL -*verbatim* and does not perform extrapolation. +*verbatim* and does not perform extrapolation. The order of precedence +of these methods is: -^^^^^^^^^^^^^^^^^^^^^ -PyPI and version URLs -^^^^^^^^^^^^^^^^^^^^^ - -In addition to their developer websites, many python packages are hosted at the -`Python Package Index (PyPi) `_. Although links to -these individual files are typically `generated using a hash -`_ it is often possible to find a -reliable link of the format - -.. code-block:: sh +#. package-level ``url`` +#. ``url_for_version()`` +#. version-specific ``url`` - https://pypi.python.org/packages/source///-. - -Packages hosted on GitHub and the like are often developer versions that do not -contain all of the files (e.g. configuration scripts) necessary to support -compilation. For this reason it is ideal to link to a repository such as PyPi -if possible. - -More recently, sources are being indexed at `pypi.io `_ as -well. Links obtained from this site follow a similar pattern, namely - -.. code-block:: sh - - https://pypi.io/packages/source///-. - -These links currently redirect back to `pypi.python.org -`_, but this `may change in the future -`_. +so if your package contains a ``url_for_version()``, it can be overridden +by a version-specific ``url``. +If your package does not contain a package-level ``url`` or ``url_for_version()``, +Spack can determine which URL to download from even if only some of the versions +specify their own ``url``. Spack will use the nearest URL *before* the requested +version. This is useful for packages that have an easy to extrapolate URL, but +keep changing their URL format every few releases. With this method, you only +need to specify the ``url`` when the URL changes. ^^^^^^^^^^^^^^^^^^^^^^^^ Skipping the expand step ^^^^^^^^^^^^^^^^^^^^^^^^ -Spack normally expands archives (e.g. `*.tar.gz` and `*.zip`) automatically +Spack normally expands archives (e.g. ``*.tar.gz`` and ``*.zip``) automatically after downloading them. If you want to skip this step (e.g., for self-extracting executables and other custom archive types), you can add ``expand=False`` to a ``version`` directive. @@ -560,7 +543,7 @@ self-extracting executables and other custom archive types), you can add .. code-block:: python version('8.2.1', '4136d7b4c04df68b686570afa26988ac', - url='http://example.com/foo-8.2.1-special-version.tar.gz', expand=False) + url='http://example.com/foo-8.2.1-special-version.sh', expand=False) When ``expand`` is set to ``False``, Spack sets the current working directory to the directory containing the downloaded archive before it @@ -605,7 +588,7 @@ here will suffice; for example, ``@develop``, ``@master``, ``@local``. The following rules determine the sort order of numeric vs. non-numeric versions: -#. The non-numeric versions ``@develop`` is considered greatest (newest). +#. The non-numeric version ``@develop`` is considered greatest (newest). #. Numeric versions are all less than ``@develop`` version, and are sorted numerically. @@ -617,7 +600,7 @@ The logic behind this sort order is two-fold: #. Non-numeric versions are usually used for special cases while developing or debugging a piece of software. Keeping most of them - less than numeric versions ensures that Spack choose numeric + less than numeric versions ensures that Spack chooses numeric versions by default whenever possible. #. The most-recent development version of a package will usually be @@ -679,7 +662,7 @@ example for ``libelf``: ... 0.5.2 http://www.mr511.de/software/libelf-0.5.2.tar.gz - How many would you like to checksum? (default is 5, q to abort) + How many would you like to checksum? (default is 1, q to abort) This does the same thing that ``spack create`` does, but it allows you to go back and add new versions easily as you need them (e.g., as @@ -805,12 +788,12 @@ But its downloads are in many different subdirectories of homepage = "http://www.mpich.org" url = "http://www.mpich.org/static/downloads/3.0.4/mpich-3.0.4.tar.gz" list_url = "http://www.mpich.org/static/downloads/" - list_depth = 2 + list_depth = 1 By default, Spack only looks at the top-level page available at -``list_url``. ``list_depth`` tells it to follow up to 2 levels of -links from the top-level page. Note that here, this implies two -levels of subdirectories, as the ``mpich`` website is structured much +``list_url``. ``list_depth = 1`` tells it to follow up to 1 level of +links from the top-level page. Note that here, this implies 1 +level of subdirectories, as the ``mpich`` website is structured much like a filesystem. But ``list_depth`` really refers to link depth when spidering the page. @@ -823,13 +806,57 @@ Fetching from code repositories For some packages, source code is provided in a Version Control System (VCS) repository rather than in a tarball. Spack can fetch packages from VCS repositories. Currently, Spack supports fetching with `Git -`_, `Mercurial (hg) `_, and `Subversion (SVN) -`_. +`_, `Mercurial (hg) `_, `Subversion (svn) +`_, and `Go `_. + +To fetch a package from a source repository, Spack needs to know which +VCS to use and where to download from. Much like with ``url``, package +authors can specify a class-level ``git``, ``hg``, ``svn``, or ``go`` +attribute containing the correct download location. + +Many packages developed with Git have both a Git repository as well as +release tarballs available for download. Packages can define both a +class-level tarball URL and VCS. For example: + +.. code-block:: python + + class Trilinos(CMakePackage): + + homepage = "https://trilinos.org/" + url = "https://github.com/trilinos/Trilinos/archive/trilinos-release-12-12-1.tar.gz" + git = "https://github.com/trilinos/Trilinos.git" + + version('develop', branch='develop') + version('master', branch='master') + version('12.12.1', 'ecd4606fa332212433c98bf950a69cc7') + version('12.10.1', '667333dbd7c0f031d47d7c5511fd0810') + version('12.8.1', '9f37f683ee2b427b5540db8a20ed6b15') + +If a package contains both a ``url`` and ``git`` class-level attribute, +Spack decides which to use based on the arguments to the ``version()`` +directive. Versions containing a specific branch, tag, or revision are +assumed to be for VCS download methods, while versions containing a +checksum are assumed to be for URL download methods. + +Like ``url``, if a specific version downloads from a different repository +than the default repo, it can be overridden with a version-specific argument. + +.. note:: + + In order to reduce ambiguity, each package can only have a single VCS + top-level attribute in addition to ``url``. In the rare case that a + package uses multiple VCS, a fetch strategy can be specified for each + version. For example, the ``rockstar`` package contains: + + .. code-block:: python + + class Rockstar(MakefilePackage): + + homepage = "https://bitbucket.org/gfcstanford/rockstar" + + version('develop', git='https://bitbucket.org/gfcstanford/rockstar.git') + version('yt', hg='https://bitbucket.org/MatthewTurk/rockstar') -To fetch a package from a source repository, you add a ``version()`` -call to your package with parameters indicating the repository URL and -any branch, tag, or revision to fetch. See below for the parameters -you'll need for each VCS system. .. _git-fetch: @@ -837,11 +864,11 @@ you'll need for each VCS system. Git ^^^ -Git fetching is enabled with the following parameters to ``version``: +Git fetching supports the following parameters to ``version``: -* ``git``: URL of the git repository. -* ``tag``: name of a tag to fetch. -* ``branch``: name of a branch to fetch. +* ``git``: URL of the git repository, if different than the class-level ``git``. +* ``branch``: Name of a branch to fetch. +* ``tag``: Name of a tag to fetch. * ``commit``: SHA hash (or prefix) of a commit to fetch. * ``submodules``: Also fetch submodules recursively when checking out this repository. @@ -853,66 +880,72 @@ Default branch .. code-block:: python class Example(Package): - ... - version('develop', git='https://github.com/example-project/example.git') - This download method is untrusted, and is not recommended. + git = "https://github.com/example-project/example.git" -Tags - To fetch from a particular tag, use the ``tag`` parameter along with - ``git``: + version('develop') + + This download method is untrusted, and is not recommended. Aside from HTTPS, + there is no way to verify that the repository has not been compromised, and + the commit you get when you install the package likely won't be the same + commit that was used when the package was first written. Additionally, the + default branch may change. It is best to at least specify a branch name. + +Branches + To fetch a particular branch, use the ``branch`` parameter: .. code-block:: python - version('1.0.1', git='https://github.com/example-project/example.git', - tag='v1.0.1') + version('experimental', branch='experimental') - This download method is untrusted, and is not recommended. + This download method is untrusted, and is not recommended. Branches are + moving targets, so the commit you get when you install the package likely + won't be the same commit that was used when the package was first written. -Branches - To fetch a particular branch, use ``branch`` instead: +Tags + To fetch from a particular tag, use ``tag`` instead: .. code-block:: python - version('experimental', git='https://github.com/example-project/example.git', - branch='experimental') + version('1.0.1', tag='v1.0.1') - This download method is untrusted, and is not recommended. + This download method is untrusted, and is not recommended. Although tags + are generally more stable than branches, Git allows tags to be moved. + Many developers use tags to denote rolling releases, and may move the + tag when a bug is patched. Commits Finally, to fetch a particular commit, use ``commit``: .. code-block:: python - version('2014-10-08', git='https://github.com/example-project/example.git', - commit='9d38cd4e2c94c3cea97d0e2924814acc') + version('2014-10-08', commit='9d38cd4e2c94c3cea97d0e2924814acc') This doesn't have to be a full hash; you can abbreviate it as you'd expect with git: .. code-block:: python - version('2014-10-08', git='https://github.com/example-project/example.git', - commit='9d38cd') + version('2014-10-08', commit='9d38cd') This download method *is trusted*. It is the recommended way to securely download from a Git repository. It may be useful to provide a saner version for commits like this, - e.g. you might use the date as the version, as done above. Or you - could just use the abbreviated commit hash. It's up to the package - author to decide what makes the most sense. + e.g. you might use the date as the version, as done above. Or, if you + know the commit at which a release was cut, you can use the release + version. It's up to the package author to decide what makes the most + sense. Although you can use the commit hash as the version number, + this is not recommended, as it won't sort properly. Submodules - You can supply ``submodules=True`` to cause Spack to fetch submodules recursively along with the repository at fetch time. For more information about git submodules see the manpage of git: ``man git-submodule``. .. code-block:: python - version('1.0.1', git='https://github.com/example-project/example.git', - tag='v1.0.1', submodules=True) + version('1.0.1', tag='v1.0.1', submodules=True) .. _github-fetch: @@ -921,7 +954,7 @@ Submodules GitHub ^^^^^^ -If a project is hosted on GitHub, *any* valid Git branch, tag or hash +If a project is hosted on GitHub, *any* valid Git branch, tag, or hash may be downloaded as a tarball. This is accomplished simply by constructing an appropriate URL. Spack can checksum any package downloaded this way, thereby producing a trusted download. For @@ -931,7 +964,7 @@ checksum. .. code-block:: python version('1.9.5.1.1', 'd035e4bc704d136db79b43ab371b27d2', - url='https://www.github.com/jswhit/pyproj/tarball/0be612cc9f972e38b50a90c946a9b353e2ab140f') + url='https://www.github.com/jswhit/pyproj/tarball/0be612cc9f972e38b50a90c946a9b353e2ab140f') .. _hg-fetch: @@ -939,34 +972,38 @@ checksum. Mercurial ^^^^^^^^^ -Fetching with mercurial works much like `git `_, but you +Fetching with Mercurial works much like `Git `_, but you use the ``hg`` parameter. -Default - Add the ``hg`` parameter with no ``revision``: +Default branch + Add the ``hg`` attribute with no ``revision`` passed to ``version``: .. code-block:: python - version('develop', hg='https://jay.grs.rwth-aachen.de/hg/example') + class Example(Package): + + hg = "https://bitbucket.org/example-project/example" - This download method is untrusted, and is not recommended. + version('develop') + + This download method is untrusted, and is not recommended. As with + Git's default fetching strategy, there is no way to verify the + integrity of the download. Revisions - Add ``hg`` and ``revision`` parameters: + To fetch a particular revision, use the ``revision`` parameter: .. code-block:: python - version('1.0', hg='https://jay.grs.rwth-aachen.de/hg/example', - revision='v1.0') - - This download method is untrusted, and is not recommended. + version('1.0', revision='v1.0') Unlike ``git``, which has special parameters for different types of revisions, you can use ``revision`` for branches, tags, and commits - when you fetch with Mercurial. + when you fetch with Mercurial. Like Git, fetching specific branches + or tags is an untrusted download method, and is not recommended. + The recommended fetch strategy is to specify a particular commit + hash as the revision. -As with git, you can fetch these versions using the ``spack install -example@`` command-line syntax. .. _svn-fetch: @@ -974,34 +1011,70 @@ example@`` command-line syntax. Subversion ^^^^^^^^^^ -To fetch with subversion, use the ``svn`` and ``revision`` parameters: +To fetch with subversion, use the ``svn`` and ``revision`` parameters. Fetching the head - Simply add an ``svn`` parameter to ``version``: + Simply add an ``svn`` parameter to the package: .. code-block:: python - version('develop', svn='https://outreach.scidac.gov/svn/libmonitor/trunk') + class Example(Package): + + svn = "https://outreach.scidac.gov/svn/example/trunk" - This download method is untrusted, and is not recommended. + version('develop') + + This download method is untrusted, and is not recommended for the + same reasons as mentioned above. Fetching a revision - To fetch a particular revision, add a ``revision`` to the - version call: + To fetch a particular revision, add a ``revision`` argument to the + version directive: .. code-block:: python - version('develop', svn='https://outreach.scidac.gov/svn/libmonitor/trunk', - revision=128) + version('develop', revision=128) This download method is untrusted, and is not recommended. + Unfortunately, Subversion has no commit hashing scheme like Git and + Mercurial do, so there is no way to guarantee that the download you + get is the same as the download used when the package was created. + Use at your own risk. + Subversion branches are handled as part of the directory structure, so -you can check out a branch or tag by changing the ``url``. +you can check out a branch or tag by changing the URL. If you want to +package multiple branches, simply add a ``svn`` argument to each +version directive. + +.. _go-fetch: + +^^ +Go +^^ + +Go isn't a VCS, it is a programming language with a builtin command, +`go get `_, +that fetches packages and their dependencies automatically. +It can clone a Git repository, or download from another source location. +For example: + +.. code-block:: python + + class ThePlatinumSearcher(Package): + + homepage = "https://github.com/monochromegane/the_platinum_searcher" + go = "github.com/monochromegane/the_platinum_searcher/..." + + version('head') + +Go cannot be used to fetch a particular commit or branch, it always +downloads the head of the repository. This download method is untrusted, +and is not recommended. Use another fetch strategy whenever possible. -------------------------------------------------- +------------------------------------ Resources (expanding extra tarballs) -------------------------------------------------- +------------------------------------ Some packages (most notably compilers) provide optional features if additional resources are expanded within their source tree before building. In Spack it is -- cgit v1.2.3-60-g2f50