major.minor.micro.dev0 Spack version (#25267)

When you install Spack from a tarball, it will always show an exact
version for Spack itself, even when you don't download a tagged commit:

```
$ wget -q https://github.com/spack/spack/archive/refs/heads/develop.tar.gz
$ tar -xf develop.tar.gz
$ ./spack-develop/bin/spack --version
0.16.2
```

This PR sets the Spack version to `0.18.0.dev0` on develop, following [PEP440](https://github.com/spack/spack/pull/25267#issuecomment-896340234) as
suggested by Adam Stewart.

```
spack (fix/set-dev-version)$ spack --version
0.18.0.dev0 (git 0.17.1-1526-e270464ae0)
spack (fix/set-dev-version)$ mv .git .git_
spack $ spack --version
0.18.0.dev0
```

- [x] Update the release guide
- [x] Add __version__ to spack's __init__.py
- [x] Use PEP 440 canonical version strings
- [x] Make spack --version output [actual version] (git version)

Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>

4 files changed, 91 insertions, 102 deletions

diff --git a/lib/spack/docs/developer_guide.rst b/lib/spack/docs/developer_guide.rst
index 5a3b7a827d..179ec40267 100644
--- a/lib/spack/docs/developer_guide.rst
+++ b/lib/spack/docs/developer_guide.rst
@@ -1057,39 +1057,39 @@ Release branches
 ^^^^^^^^^^^^^^^^
 
 There are currently two types of Spack releases: :ref:`major releases
-<major-releases>` (``0.13.0``, ``0.14.0``, etc.) and :ref:`point releases
-<point-releases>` (``0.13.1``, ``0.13.2``, ``0.13.3``, etc.). Here is a
+<major-releases>` (``0.17.0``, ``0.18.0``, etc.) and :ref:`point releases
+<point-releases>` (``0.17.1``, ``0.17.2``, ``0.17.3``, etc.). Here is a
 diagram of how Spack release branches work::
 
-    o    branch: develop  (latest version)
+    o    branch: develop  (latest version, v0.19.0.dev0)
     |
-    o    merge v0.14.1 into develop
-    |\
-    | o  branch: releases/v0.14, tag: v0.14.1
-    o |  merge v0.14.0 into develop
-    |\|
-    | o  tag: v0.14.0
+    o
+    | o  branch: releases/v0.18, tag: v0.18.1
+    o |
+    | o  tag: v0.18.0
+    o |
+    | o
     |/
-    o    merge v0.13.2 into develop
-    |\
-    | o  branch: releases/v0.13, tag: v0.13.2
-    o |  merge v0.13.1 into develop
-    |\|
-    | o  tag: v0.13.1
-    o |  merge v0.13.0 into develop
-    |\|
-    | o  tag: v0.13.0
+    o
+    |
+    o
+    | o  branch: releases/v0.17, tag: v0.17.2
+    o |
+    | o  tag: v0.17.1
+    o |
+    | o  tag: v0.17.0
     o |
     | o
     |/
     o
 
 The ``develop`` branch has the latest contributions, and nearly all pull
-requests target ``develop``.
+requests target ``develop``. The ``develop`` branch will report that its
+version is that of the next **major** release with a ``.dev0`` suffix.
 
 Each Spack release series also has a corresponding branch, e.g.
-``releases/v0.14`` has ``0.14.x`` versions of Spack, and
-``releases/v0.13`` has ``0.13.x`` versions. A major release is the first
+``releases/v0.18`` has ``0.18.x`` versions of Spack, and
+``releases/v0.17`` has ``0.17.x`` versions. A major release is the first
 tagged version on a release branch. Minor releases are back-ported from
 develop onto release branches. This is typically done by cherry-picking
 bugfix commits off of ``develop``.
@@ -1100,12 +1100,20 @@ packages. They should generally only contain fixes to the Spack core.
 However, sometimes priorities are such that new functionality needs to
 be added to a minor release.
 
-Both major and minor releases are tagged. After each release, we merge
-the release branch back into ``develop`` so that the version bump and any
-other release-specific changes are visible in the mainline. As a
-convenience, we also tag the latest release as ``releases/latest``,
-so that users can easily check it out to get the latest
-stable version. See :ref:`merging-releases` for more details.
+Both major and minor releases are tagged. As a convenience, we also tag
+the latest release as ``releases/latest``, so that users can easily check
+it out to get the latest stable version. See :ref:`updating-latest-release`
+for more details.
+
+.. note::
+
+   Older spack releases were merged **back** into develop so that we could
+   do fancy things with tags, but since tarballs and many git checkouts do
+   not have tags, this proved overly complex and confusing.
+
+   We have since converted to using `PEP 440 <https://peps.python.org/pep-0440/>`_
+   compliant versions.  `See here <https://github.com/spack/spack/pull/25267>`_ for
+   details.
 
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 Scheduling work for releases
@@ -1163,10 +1171,11 @@ completed, the steps to make the major release are:
    ``releases/vX.Y``. That is, you should create a ``releases/vX.Y``
    branch if you are preparing the ``X.Y.0`` release.
 
-#. Bump the version in ``lib/spack/spack/__init__.py``.
+#. Remove the ``dev0`` development release segment from the version tuple in
+   ``lib/spack/spack/__init__.py``.
 
-   See `this example from 0.13.0
-   <https://github.com/spack/spack/commit/8eeb64096c98b8a43d1c587f13ece743c864fba9>`_
+   The version number itself should already be correct and should not be
+   modified.
 
 #. Update ``CHANGELOG.md`` with major highlights in bullet form.
 
@@ -1188,9 +1197,16 @@ completed, the steps to make the major release are:
    is outdated submit pull requests to ``develop`` as normal
    and keep rebasing the release branch on ``develop``.
 
+#. Bump the major version in the ``develop`` branch.
+
+   Create a pull request targeting the ``develop`` branch, bumping the major
+   version in ``lib/spack/spack/__init__.py`` with a ``dev0`` release segment.
+   For instance when you have just released ``v0.15.0``, set the version
+   to ``(0, 16, 0, 'dev0')`` on ``develop``.
+
 #. Follow the steps in :ref:`publishing-releases`.
 
-#. Follow the steps in :ref:`merging-releases`.
+#. Follow the steps in :ref:`updating-latest-release`.
 
 #. Follow the steps in :ref:`announcing-releases`.
 
@@ -1266,9 +1282,6 @@ completed, the steps to make the point release are:
 
 #. Bump the version in ``lib/spack/spack/__init__.py``.
 
-   See `this example from 0.14.1
-   <https://github.com/spack/spack/commit/ff0abb9838121522321df2a054d18e54b566b44a>`_.
-
 #. Update ``CHANGELOG.md`` with a list of the changes.
 
    This is typically a summary of the commits you cherry-picked onto the
@@ -1290,7 +1303,7 @@ completed, the steps to make the point release are:
 
 #. Follow the steps in :ref:`publishing-releases`.
 
-#. Follow the steps in :ref:`merging-releases`.
+#. Follow the steps in :ref:`updating-latest-release`.
 
 #. Follow the steps in :ref:`announcing-releases`.
 
@@ -1351,11 +1364,11 @@ Publishing a release on GitHub
    selectable in the versions menu.
 
 
-.. _merging-releases:
+.. _updating-latest-release:
 
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Updating `releases/latest` and `develop`
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+Updating `releases/latest`
+^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 If the new release is the **highest** Spack release yet, you should
 also tag it as ``releases/latest``. For example, suppose the highest
@@ -1379,40 +1392,6 @@ To tag ``releases/latest``, do this:
 The ``--force`` argument to ``git tag`` makes ``git`` overwrite the existing
 ``releases/latest`` tag with the new one.
 
-We also merge each release that we tag as ``releases/latest`` into ``develop``.
-Make sure to do this with a merge commit:
-
-.. code-block:: console
-
-   $ git checkout develop
-   $ git merge --no-ff -s ours vX.Y.Z  # vX.Y.Z is the new release's tag
-   $ git push
-
-We merge back to ``develop`` because it:
-
-  * updates the version and ``CHANGELOG.md`` on ``develop``; and
-  * ensures that your release tag is reachable from the head of
-    ``develop``.
-
-We *must* use a real merge commit (via the ``--no-ff`` option) to
-ensure that the release tag is reachable from the tip of ``develop``.
-This is necessary for ``spack -V`` to work properly -- it uses ``git
-describe --tags`` to find the last reachable tag in the repository and
-reports how far we are from it. For example:
-
-.. code-block:: console
-
-   $ spack -V
-   0.14.2-1486-b80d5e74e5
-
-This says that we are at commit ``b80d5e74e5``, which is 1,486 commits
-ahead of the ``0.14.2`` release.
-
-We put this step last in the process because it's best to do it only once
-the release is complete and tagged. If you do it before you've tagged the
-release and later decide you want to tag some later commit, you'll need
-to merge again.
-
 
 .. _announcing-releases:
 
diff --git a/lib/spack/spack/__init__.py b/lib/spack/spack/__init__.py
index 74169d5040..810d56164c 100644
--- a/lib/spack/spack/__init__.py
+++ b/lib/spack/spack/__init__.py
@@ -3,10 +3,11 @@
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
-#: major, minor, patch version for Spack, in a tuple
-spack_version_info = (0, 17, 1)
+#: (major, minor, micro, dev release) tuple
+spack_version_info = (0, 18, 0, 'dev0')
 
-#: String containing Spack version joined with .'s
-spack_version = '.'.join(str(v) for v in spack_version_info)
+#: PEP440 canonical <major>.<minor>.<micro>.<devN> string
+spack_version = '.'.join(str(s) for s in spack_version_info)
 
 __all__ = ['spack_version_info', 'spack_version']
+__version__ = spack_version
diff --git a/lib/spack/spack/main.py b/lib/spack/spack/main.py
index 15c970a357..6e72b6a557 100644
--- a/lib/spack/spack/main.py
+++ b/lib/spack/spack/main.py
@@ -26,7 +26,6 @@ from six import StringIO
 
 import archspec.cpu
 
-import llnl.util.filesystem as fs
 import llnl.util.lang
 import llnl.util.tty as tty
 import llnl.util.tty.colify
@@ -123,30 +122,25 @@ def add_all_commands(parser):
 def get_version():
     """Get a descriptive version of this instance of Spack.
 
-    If this is a git repository, and if it is not on a release tag,
-    return a string like:
-
-        release_version-commits_since_release-commit
-
-    If we *are* at a release tag, or if this is not a git repo, return
-    the real spack release number (e.g., 0.13.3).
+    Outputs '<PEP440 version> (<git commit sha>)'.
 
+    The commit sha is only added when available.
     """
+    version = spack.spack_version
     git_path = os.path.join(spack.paths.prefix, ".git")
     if os.path.exists(git_path):
         git = exe.which("git")
-        if git:
-            with fs.working_dir(spack.paths.prefix):
-                desc = git("describe", "--tags", "--match", "v*",
-                           output=str, error=os.devnull, fail_on_error=False)
-
-            if git.returncode == 0:
-                match = re.match(r"v([^-]+)-([^-]+)-g([a-f\d]+)", desc)
-                if match:
-                    v, n, commit = match.groups()
-                    return "%s-%s-%s" % (v, n, commit)
-
-    return spack.spack_version
+        if not git:
+            return version
+        rev = git('-C', spack.paths.prefix, 'rev-parse', 'HEAD',
+                  output=str, error=os.devnull, fail_on_error=False)
+        if git.returncode != 0:
+            return version
+        match = re.match(r"[a-f\d]{7,}$", rev)
+        if match:
+            version += " ({0})".format(match.group(0))
+
+    return version
 
 
 def index_commands():
diff --git a/lib/spack/spack/test/main.py b/lib/spack/spack/test/main.py
index 9917fac6f5..dc6fa8299e 100644
--- a/lib/spack/spack/test/main.py
+++ b/lib/spack/spack/test/main.py
@@ -18,11 +18,11 @@ pytestmark = pytest.mark.skipif(
     reason="Test functionality supported but tests are failing on Win")
 
 
-def test_get_version_no_match_git(tmpdir, working_env):
+def test_version_git_nonsense_output(tmpdir, working_env):
     git = str(tmpdir.join("git"))
     with open(git, "w") as f:
         f.write("""#!/bin/sh
-echo v0.13.3
+echo --|not a hash|----
 """)
     fs.set_executable(git)
 
@@ -30,16 +30,31 @@ echo v0.13.3
     assert spack.spack_version == get_version()
 
 
-def test_get_version_match_git(tmpdir, working_env):
+def test_version_git_fails(tmpdir, working_env):
     git = str(tmpdir.join("git"))
     with open(git, "w") as f:
         f.write("""#!/bin/sh
-echo v0.13.3-912-g3519a1762
+echo 26552533be04e83e66be2c28e0eb5011cb54e8fa
+exit 1
 """)
     fs.set_executable(git)
 
     os.environ["PATH"] = str(tmpdir)
-    assert "0.13.3-912-3519a1762" == get_version()
+    assert spack.spack_version == get_version()
+
+
+def test_git_sha_output(tmpdir, working_env):
+    git = str(tmpdir.join("git"))
+    sha = '26552533be04e83e66be2c28e0eb5011cb54e8fa'
+    with open(git, "w") as f:
+        f.write("""#!/bin/sh
+echo {0}
+""".format(sha))
+    fs.set_executable(git)
+
+    os.environ["PATH"] = str(tmpdir)
+    expected = "{0} ({1})".format(spack.spack_version, sha)
+    assert expected == get_version()
 
 
 def test_get_version_no_repo(tmpdir, monkeypatch):