Add docs on new top-level VCS attributes

1 files changed, 181 insertions, 108 deletions

diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst
index 706144403d..12abcfd7ea 100644
--- a/lib/spack/docs/packaging_guide.rst
+++ b/lib/spack/docs/packaging_guide.rst
@@ -413,27 +413,26 @@ The most straightforward way to add new versions to your package is to
 add a line like this in the package class:
 
 .. code-block:: python
-   :linenos:
 
    class Foo(Package):
-       url = 'http://example.com/foo-1.0.tar.gz'
+
+       url = "http://example.com/foo-1.0.tar.gz"
+
        version('8.2.1', '4136d7b4c04df68b686570afa26988ac')
-       ...
+       version('8.2.0', '1c9f62f0778697a09d36121ead88e08e')
+       version('8.1.2', 'd47dd09ed7ae6e7fd6f9a816d7f5fdf6')
 
-Versions should be listed with the newest version first.
+Versions should be listed in descending order, from newest to oldest.
 
 ^^^^^^^^^^^^^
 Date Versions
 ^^^^^^^^^^^^^
 
 If you wish to use dates as versions, it is best to use the format
-``@date-yyyy-mm-dd``.  This will ensure they sort in the correct
-order.  If you want your date versions to be numeric (assuming they
-don't conflict with other numeric versions), you can use just
-``yyyy.mm.dd``.
+``@yyyy-mm-dd``.  This will ensure they sort in the correct order.
 
 Alternately, you might use a hybrid release-version / date scheme.
-For example, ``@1.3.2016.08.31`` would mean the version from the
+For example, ``@1.3_2016-08-31`` would mean the version from the
 ``1.3`` branch, as of August 31, 2016.
 
 ^^^^^^^^^^^^
@@ -442,7 +441,7 @@ Version URLs
 
 By default, each version's URL is extrapolated from the ``url`` field
 in the package.  For example, Spack is smart enough to download
-version ``8.2.1.`` of the ``Foo`` package above from
+version ``8.2.1`` of the ``Foo`` package above from
 http://example.com/foo-8.2.1.tar.gz.
 
 If the URL is particularly complicated or changes based on the release,
@@ -515,44 +514,28 @@ of its versions, you can add an explicit URL for a particular version:
 
 
 When you supply a custom URL for a version, Spack uses that URL
-*verbatim* and does not perform extrapolation.
+*verbatim* and does not perform extrapolation. The order of precedence
+of these methods is:
 
-^^^^^^^^^^^^^^^^^^^^^
-PyPI and version URLs
-^^^^^^^^^^^^^^^^^^^^^
-
-In addition to their developer websites, many python packages are hosted at the
-`Python Package Index (PyPi) <https://pypi.python.org/pypi>`_. Although links to
-these individual files are typically `generated using a hash
-<https://bitbucket.org/pypa/pypi/issues/438>`_ it is often possible to find a
-reliable link of the format
-
-.. code-block:: sh
+#. package-level ``url``
+#. ``url_for_version()``
+#. version-specific ``url``
 
-  https://pypi.python.org/packages/source/<first letter of package>/<package>/<package>-<version>.<extension>
-
-Packages hosted on GitHub and the like are often developer versions that do not
-contain all of the files (e.g. configuration scripts) necessary to support
-compilation. For this reason it is ideal to link to a repository such as PyPi
-if possible.
-
-More recently, sources are being indexed at `pypi.io <https://pypi.io>`_ as
-well. Links obtained from this site follow a similar pattern, namely
-
-.. code-block:: sh
-
-  https://pypi.io/packages/source/<first letter of package>/<package>/<package>-<version>.<extension>
-
-These links currently redirect back to `pypi.python.org
-<https://pypi.python.org>`_, but this `may change in the future
-<https://bitbucket.org/pypa/pypi/issues/438#comment-27243225>`_.
+so if your package contains a ``url_for_version()``, it can be overridden
+by a version-specific ``url``.
 
+If your package does not contain a package-level ``url`` or ``url_for_version()``,
+Spack can determine which URL to download from even if only some of the versions
+specify their own ``url``. Spack will use the nearest URL *before* the requested
+version. This is useful for packages that have an easy to extrapolate URL, but
+keep changing their URL format every few releases. With this method, you only
+need to specify the ``url`` when the URL changes.
 
 ^^^^^^^^^^^^^^^^^^^^^^^^
 Skipping the expand step
 ^^^^^^^^^^^^^^^^^^^^^^^^
 
-Spack normally expands archives (e.g. `*.tar.gz` and `*.zip`) automatically
+Spack normally expands archives (e.g. ``*.tar.gz`` and ``*.zip``) automatically
 after downloading them. If you want to skip this step (e.g., for
 self-extracting executables and other custom archive types), you can add
 ``expand=False`` to a ``version`` directive.
@@ -560,7 +543,7 @@ self-extracting executables and other custom archive types), you can add
 .. code-block:: python
 
    version('8.2.1', '4136d7b4c04df68b686570afa26988ac',
-           url='http://example.com/foo-8.2.1-special-version.tar.gz', expand=False)
+           url='http://example.com/foo-8.2.1-special-version.sh', expand=False)
 
 When ``expand`` is set to ``False``, Spack sets the current working
 directory to the directory containing the downloaded archive before it
@@ -605,7 +588,7 @@ here will suffice; for example, ``@develop``, ``@master``, ``@local``.
 The following rules determine the sort order of numeric
 vs. non-numeric versions:
 
-#. The non-numeric versions ``@develop`` is considered greatest (newest).
+#. The non-numeric version ``@develop`` is considered greatest (newest).
 
 #. Numeric versions are all less than ``@develop`` version, and are
    sorted numerically.
@@ -617,7 +600,7 @@ The logic behind this sort order is two-fold:
 
 #. Non-numeric versions are usually used for special cases while
    developing or debugging a piece of software.  Keeping most of them
-   less than numeric versions ensures that Spack choose numeric
+   less than numeric versions ensures that Spack chooses numeric
    versions by default whenever possible.
 
 #. The most-recent development version of a package will usually be
@@ -679,7 +662,7 @@ example for ``libelf``:
      ...
      0.5.2     http://www.mr511.de/software/libelf-0.5.2.tar.gz
 
-   How many would you like to checksum? (default is 5, q to abort)
+   How many would you like to checksum? (default is 1, q to abort)
 
 This does the same thing that ``spack create`` does, but it allows you
 to go back and add new versions easily as you need them (e.g., as
@@ -805,12 +788,12 @@ But its downloads are in many different subdirectories of
        homepage   = "http://www.mpich.org"
        url        = "http://www.mpich.org/static/downloads/3.0.4/mpich-3.0.4.tar.gz"
        list_url   = "http://www.mpich.org/static/downloads/"
-       list_depth = 2
+       list_depth = 1
 
 By default, Spack only looks at the top-level page available at
-``list_url``.  ``list_depth`` tells it to follow up to 2 levels of
-links from the top-level page.  Note that here, this implies two
-levels of subdirectories, as the ``mpich`` website is structured much
+``list_url``.  ``list_depth = 1`` tells it to follow up to 1 level of
+links from the top-level page.  Note that here, this implies 1
+level of subdirectories, as the ``mpich`` website is structured much
 like a filesystem.  But ``list_depth`` really refers to link depth
 when spidering the page.
 
@@ -823,13 +806,57 @@ Fetching from code repositories
 For some packages, source code is provided in a Version Control System
 (VCS) repository rather than in a tarball.  Spack can fetch packages
 from VCS repositories. Currently, Spack supports fetching with `Git
-<git-fetch_>`_, `Mercurial (hg) <hg-fetch_>`_, and `Subversion (SVN)
-<svn-fetch_>`_.
+<git-fetch_>`_, `Mercurial (hg) <hg-fetch_>`_, `Subversion (svn)
+<svn-fetch_>`_, and `Go <go-fetch_>`_.
+
+To fetch a package from a source repository, Spack needs to know which
+VCS to use and where to download from. Much like with ``url``, package
+authors can specify a class-level ``git``, ``hg``, ``svn``, or ``go``
+attribute containing the correct download location.
+
+Many packages developed with Git have both a Git repository as well as
+release tarballs available for download. Packages can define both a
+class-level tarball URL and VCS. For example:
+
+.. code-block:: python
+
+   class Trilinos(CMakePackage):
+
+       homepage = "https://trilinos.org/"
+       url      = "https://github.com/trilinos/Trilinos/archive/trilinos-release-12-12-1.tar.gz"
+       git      = "https://github.com/trilinos/Trilinos.git"
+
+       version('develop', branch='develop')
+       version('master',  branch='master')
+       version('12.12.1', 'ecd4606fa332212433c98bf950a69cc7')
+       version('12.10.1', '667333dbd7c0f031d47d7c5511fd0810')
+       version('12.8.1',  '9f37f683ee2b427b5540db8a20ed6b15')
+
+If a package contains both a ``url`` and ``git`` class-level attribute,
+Spack decides which to use based on the arguments to the ``version()``
+directive. Versions containing a specific branch, tag, or revision are
+assumed to be for VCS download methods, while versions containing a
+checksum are assumed to be for URL download methods.
+
+Like ``url``, if a specific version downloads from a different repository
+than the default repo, it can be overridden with a version-specific argument.
+
+.. note::
+
+   In order to reduce ambiguity, each package can only have a single VCS
+   top-level attribute in addition to ``url``. In the rare case that a
+   package uses multiple VCS, a fetch strategy can be specified for each
+   version. For example, the ``rockstar`` package contains:
+
+   .. code-block:: python
+
+      class Rockstar(MakefilePackage):
+
+          homepage = "https://bitbucket.org/gfcstanford/rockstar"
+
+          version('develop', git='https://bitbucket.org/gfcstanford/rockstar.git')
+          version('yt', hg='https://bitbucket.org/MatthewTurk/rockstar')
 
-To fetch a package from a source repository, you add a ``version()``
-call to your package with parameters indicating the repository URL and
-any branch, tag, or revision to fetch.  See below for the parameters
-you'll need for each VCS system.
 
 .. _git-fetch:
 
@@ -837,11 +864,11 @@ you'll need for each VCS system.
 Git
 ^^^
 
-Git fetching is enabled with the following parameters to ``version``:
+Git fetching supports the following parameters to ``version``:
 
-* ``git``: URL of the git repository.
-* ``tag``: name of a tag to fetch.
-* ``branch``: name of a branch to fetch.
+* ``git``: URL of the git repository, if different than the class-level ``git``.
+* ``branch``: Name of a branch to fetch.
+* ``tag``: Name of a tag to fetch.
 * ``commit``: SHA hash (or prefix) of a commit to fetch.
 * ``submodules``: Also fetch submodules recursively when checking out this repository.
 
@@ -853,66 +880,72 @@ Default branch
   .. code-block:: python
 
      class Example(Package):
-         ...
-         version('develop', git='https://github.com/example-project/example.git')
 
-  This download method is untrusted, and is not recommended.
+         git = "https://github.com/example-project/example.git"
 
-Tags
-  To fetch from a particular tag, use the ``tag`` parameter along with
-  ``git``:
+         version('develop')
+
+  This download method is untrusted, and is not recommended. Aside from HTTPS,
+  there is no way to verify that the repository has not been compromised, and
+  the commit you get when you install the package likely won't be the same
+  commit that was used when the package was first written. Additionally, the
+  default branch may change. It is best to at least specify a branch name.
+
+Branches
+  To fetch a particular branch, use the ``branch`` parameter:
 
   .. code-block:: python
 
-     version('1.0.1', git='https://github.com/example-project/example.git',
-             tag='v1.0.1')
+     version('experimental', branch='experimental')
 
-  This download method is untrusted, and is not recommended.
+  This download method is untrusted, and is not recommended. Branches are
+  moving targets, so the commit you get when you install the package likely
+  won't be the same commit that was used when the package was first written.
 
-Branches
-  To fetch a particular branch, use ``branch`` instead:
+Tags
+  To fetch from a particular tag, use ``tag`` instead:
 
   .. code-block:: python
 
-     version('experimental', git='https://github.com/example-project/example.git',
-             branch='experimental')
+     version('1.0.1', tag='v1.0.1')
 
-  This download method is untrusted, and is not recommended.
+  This download method is untrusted, and is not recommended. Although tags
+  are generally more stable than branches, Git allows tags to be moved.
+  Many developers use tags to denote rolling releases, and may move the
+  tag when a bug is patched.
 
 Commits
   Finally, to fetch a particular commit, use ``commit``:
 
   .. code-block:: python
 
-     version('2014-10-08', git='https://github.com/example-project/example.git',
-             commit='9d38cd4e2c94c3cea97d0e2924814acc')
+     version('2014-10-08', commit='9d38cd4e2c94c3cea97d0e2924814acc')
 
   This doesn't have to be a full hash; you can abbreviate it as you'd
   expect with git:
 
   .. code-block:: python
 
-     version('2014-10-08', git='https://github.com/example-project/example.git',
-             commit='9d38cd')
+     version('2014-10-08', commit='9d38cd')
 
   This download method *is trusted*.  It is the recommended way to
   securely download from a Git repository.
 
   It may be useful to provide a saner version for commits like this,
-  e.g. you might use the date as the version, as done above.  Or you
-  could just use the abbreviated commit hash.  It's up to the package
-  author to decide what makes the most sense.
+  e.g. you might use the date as the version, as done above. Or, if you
+  know the commit at which a release was cut, you can use the release
+  version. It's up to the package author to decide what makes the most
+  sense. Although you can use the commit hash as the version number,
+  this is not recommended, as it won't sort properly.
 
 Submodules
-
   You can supply ``submodules=True`` to cause Spack to fetch submodules
   recursively along with the repository at fetch time. For more information
   about git submodules see the manpage of git: ``man git-submodule``.
 
   .. code-block:: python
 
-     version('1.0.1', git='https://github.com/example-project/example.git',
-             tag='v1.0.1', submodules=True)
+     version('1.0.1', tag='v1.0.1', submodules=True)
 
 
 .. _github-fetch:
@@ -921,7 +954,7 @@ Submodules
 GitHub
 ^^^^^^
 
-If a project is hosted on GitHub, *any* valid Git branch, tag or hash
+If a project is hosted on GitHub, *any* valid Git branch, tag, or hash
 may be downloaded as a tarball.  This is accomplished simply by
 constructing an appropriate URL.  Spack can checksum any package
 downloaded this way, thereby producing a trusted download.  For
@@ -931,7 +964,7 @@ checksum.
 .. code-block:: python
 
        version('1.9.5.1.1', 'd035e4bc704d136db79b43ab371b27d2',
-           url='https://www.github.com/jswhit/pyproj/tarball/0be612cc9f972e38b50a90c946a9b353e2ab140f')
+               url='https://www.github.com/jswhit/pyproj/tarball/0be612cc9f972e38b50a90c946a9b353e2ab140f')
 
 .. _hg-fetch:
 
@@ -939,34 +972,38 @@ checksum.
 Mercurial
 ^^^^^^^^^
 
-Fetching with mercurial works much like `git <git-fetch>`_, but you
+Fetching with Mercurial works much like `Git <git-fetch>`_, but you
 use the ``hg`` parameter.
 
-Default
-  Add the ``hg`` parameter with no ``revision``:
+Default branch
+  Add the ``hg`` attribute with no ``revision`` passed to ``version``:
 
   .. code-block:: python
 
-     version('develop', hg='https://jay.grs.rwth-aachen.de/hg/example')
+     class Example(Package):
+
+         hg = "https://bitbucket.org/example-project/example"
 
-  This download method is untrusted, and is not recommended.
+         version('develop')
+
+  This download method is untrusted, and is not recommended. As with
+  Git's default fetching strategy, there is no way to verify the
+  integrity of the download.
 
 Revisions
-  Add ``hg`` and ``revision`` parameters:
+  To fetch a particular revision, use the ``revision`` parameter:
 
   .. code-block:: python
 
-     version('1.0', hg='https://jay.grs.rwth-aachen.de/hg/example',
-             revision='v1.0')
-
-  This download method is untrusted, and is not recommended.
+     version('1.0', revision='v1.0')
 
   Unlike ``git``, which has special parameters for different types of
   revisions, you can use ``revision`` for branches, tags, and commits
-  when you fetch with Mercurial.
+  when you fetch with Mercurial. Like Git, fetching specific branches
+  or tags is an untrusted download method, and is not recommended.
+  The recommended fetch strategy is to specify a particular commit
+  hash as the revision.
 
-As with git, you can fetch these versions using the ``spack install
-example@<version>`` command-line syntax.
 
 .. _svn-fetch:
 
@@ -974,34 +1011,70 @@ example@<version>`` command-line syntax.
 Subversion
 ^^^^^^^^^^
 
-To fetch with subversion, use the ``svn`` and ``revision`` parameters:
+To fetch with subversion, use the ``svn`` and ``revision`` parameters.
 
 Fetching the head
-  Simply add an ``svn`` parameter to ``version``:
+  Simply add an ``svn`` parameter to the package:
 
   .. code-block:: python
 
-     version('develop', svn='https://outreach.scidac.gov/svn/libmonitor/trunk')
+     class Example(Package):
+
+         svn = "https://outreach.scidac.gov/svn/example/trunk"
 
-  This download method is untrusted, and is not recommended.
+         version('develop')
+
+  This download method is untrusted, and is not recommended for the
+  same reasons as mentioned above.
 
 Fetching a revision
-  To fetch a particular revision, add a ``revision`` to the
-  version call:
+  To fetch a particular revision, add a ``revision`` argument to the
+  version directive:
 
   .. code-block:: python
 
-     version('develop', svn='https://outreach.scidac.gov/svn/libmonitor/trunk',
-             revision=128)
+     version('develop', revision=128)
 
   This download method is untrusted, and is not recommended.
 
+  Unfortunately, Subversion has no commit hashing scheme like Git and
+  Mercurial do, so there is no way to guarantee that the download you
+  get is the same as the download used when the package was created.
+  Use at your own risk.
+
 Subversion branches are handled as part of the directory structure, so
-you can check out a branch or tag by changing the ``url``.
+you can check out a branch or tag by changing the URL. If you want to
+package multiple branches, simply add a ``svn`` argument to each
+version directive.
+
+.. _go-fetch:
+
+^^
+Go
+^^
+
+Go isn't a VCS, it is a programming language with a builtin command,
+`go get <https://golang.org/cmd/go/#hdr-Download_and_install_packages_and_dependencies>`_,
+that fetches packages and their dependencies automatically.
+It can clone a Git repository, or download from another source location.
+For example:
+
+.. code-block:: python
+
+   class ThePlatinumSearcher(Package):
+
+       homepage = "https://github.com/monochromegane/the_platinum_searcher"
+       go       = "github.com/monochromegane/the_platinum_searcher/..."
+
+       version('head')
+
+Go cannot be used to fetch a particular commit or branch, it always
+downloads the head of the repository. This download method is untrusted,
+and is not recommended. Use another fetch strategy whenever possible.
 
--------------------------------------------------
+------------------------------------
 Resources (expanding extra tarballs)
--------------------------------------------------
+------------------------------------
 
 Some packages (most notably compilers) provide optional features if additional
 resources are expanded within their source tree before building. In Spack it is