From 303882834ac1ace64fcad9cebf7d0ffc228d76a9 Mon Sep 17 00:00:00 2001 From: Todd Gamblin Date: Sat, 4 Jul 2020 01:50:55 -0700 Subject: docs: document releases and branches in Spack - [x] Remove references to `master` branch - [x] Document how release branches are structured - [x] Document how to make a major release - [x] Document how to make a point release - [x] Document how to do work in our release projects --- lib/spack/docs/contribution_guide.rst | 21 +- lib/spack/docs/developer_guide.rst | 367 ++++++++++++++++++++++++++++++++++ lib/spack/docs/getting_started.rst | 2 +- lib/spack/docs/images/pr-commit.png | Bin 0 -> 44592 bytes lib/spack/docs/images/projects.png | Bin 0 -> 69075 bytes 5 files changed, 384 insertions(+), 6 deletions(-) create mode 100644 lib/spack/docs/images/pr-commit.png create mode 100644 lib/spack/docs/images/projects.png (limited to 'lib') diff --git a/lib/spack/docs/contribution_guide.rst b/lib/spack/docs/contribution_guide.rst index 9935ec0c83..10c0875e85 100644 --- a/lib/spack/docs/contribution_guide.rst +++ b/lib/spack/docs/contribution_guide.rst @@ -27,11 +27,22 @@ correspond to one feature/bugfix/extension/etc. One can create PRs with changes relevant to different ideas, however reviewing such PRs becomes tedious and error prone. If possible, try to follow the **one-PR-one-package/feature** rule. -Spack uses a rough approximation of the `Git Flow `_ -branching model. The develop branch contains the latest contributions, and -master is always tagged and points to the latest stable release. Therefore, when -you send your request, make ``develop`` the destination branch on the -`Spack repository `_. +-------- +Branches +-------- + +Spack's ``develop`` branch has the latest contributions. Nearly all pull +requests should start from ``develop`` and target ``develop``. + +There is a branch for each major release series. Release branches +originate from ``develop`` and have tags for each point release in the +series. For example, ``releases/v0.14`` has tags for ``0.14.0``, +``0.14.1``, ``0.14.2``, etc. versions of Spack. We backport important bug +fixes to these branches, but we do not advance the package versions or +make other changes that would change the way Spack concretizes +dependencies. Currently, the maintainers manage these branches by +cherry-picking from ``develop``. See :ref:`releases` for more +information. ---------------------- Continuous Integration diff --git a/lib/spack/docs/developer_guide.rst b/lib/spack/docs/developer_guide.rst index de2fe80f85..5c57f28226 100644 --- a/lib/spack/docs/developer_guide.rst +++ b/lib/spack/docs/developer_guide.rst @@ -495,3 +495,370 @@ The bottom of the output shows the top most time consuming functions, slowest on top. The profiling support is from Python's built-in tool, `cProfile `_. + +.. _releases: + +-------- +Releases +-------- + +This section documents Spack's release process. It is intended for +project maintainers, as the tasks described here require maintainer +privileges on the Spack repository. For others, we hope this section at +least provides some insight into how the Spack project works. + +.. _release-branches: + +^^^^^^^^^^^^^^^^ +Release branches +^^^^^^^^^^^^^^^^ + +There are currently two types of Spack releases: :ref:`major releases +` (``0.13.0``, ``0.14.0``, etc.) and :ref:`point releases +` (``0.13.1``, ``0.13.2``, ``0.13.3``, etc.). Here is a +diagram of how Spack release branches work:: + + o branch: develop (latest version) + | + o merge v0.14.1 into develop + |\ + | o branch: releases/v0.14, tag: v0.14.1 + o | merge v0.14.0 into develop + |\| + | o tag: v0.14.0 + |/ + o merge v0.13.2 into develop + |\ + | o branch: releases/v0.13, tag: v0.13.2 + o | merge v0.13.1 into develop + |\| + | o tag: v0.13.1 + o | merge v0.13.0 into develop + |\| + | o tag: v0.13.0 + o | + | o + |/ + o + +The ``develop`` branch has the latest contributions, and nearly all pull +requests target ``develop``. + +Each Spack release series also has a corresponding branch, e.g. +``releases/v0.14`` has ``0.14.x`` versions of Spack, and +``releases/v0.13`` has ``0.13.x`` versions. A major release is the first +tagged version on a release branch. Minor releases are back-ported from +develop onto release branches. This is typically done by cherry-picking +bugfix commits off of ``develop``. + +To avoid version churn for users of a release series, minor releases +should **not** make changes that would change the concretization of +packages. They should generally only contain fixes to the Spack core. + +Both major and minor releases are tagged. After each release, we merge +the release branch back into ``develop`` so that the version bump and any +other release-specific changes are visible in the mainline (see +:ref:`merging-releases-to-develop`). + + +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Scheduling work for releases +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +We schedule work for releases by creating `GitHub projects +`_. At any time, there may be +several open release projects. For example, here are two releases (from +some past version of the page linked above): + +.. image:: images/projects.png + +Here, there's one release in progress for ``0.15.1`` and another for +``0.16.0``. Each of these releases has a project board containing issues +and pull requests. GitHub shows a status bar with completed work in +green, work in progress in purple, and work not started yet in gray, so +it's fairly easy to see progress. + +Spack's project boards are not firm commitments, and we move work between +releases frequently. If we need to make a release and some tasks are not +yet done, we will simply move them to next minor or major release, rather +than delaying the release to complete them. + +For more on using GitHub project boards, see `GitHub's documentation +`_. + +.. _major-releases: + +^^^^^^^^^^^^^^^^^^^^^ +Making Major Releases +^^^^^^^^^^^^^^^^^^^^^ + +Assuming you've already created a project board and completed the work +for a major release, the steps to make the release are as follows: + +#. Create two new project boards: + + * One for the next major release + * One for the next point release + +#. Move any tasks that aren't done yet to one of the new project boards. + Small bugfixes should go to the next point release. Major features, + refactors, and changes that could affect concretization should go in + the next major release. + +#. Create a branch for the release, based on ``develop``: + + .. code-block:: console + + $ git checkout -b releases/v0.15 develop + + For a version ``vX.Y.Z``, the branch's name should be + ``releases/vX.Y``. That is, you should create a ``releases/vX.Y`` + branch if you are preparing the ``X.Y.0`` release. + +#. Bump the version in ``lib/spack/spack/__init__.py``. See `this example from 0.13.0 + `_ + +#. Updaate the release version lists in these files to include the new version: + + * ``lib/spack/spack/schema/container.py`` + * ``lib/spack/spack/container/images.json`` + + **TODO**: We should get rid of this step in some future release. + +#. Update ``CHANGELOG.md`` with major highlights in bullet form. Use + proper markdown formatting, like `this example from 0.15.0 + `_. + +#. Push the release branch to GitHub. + +#. Make sure CI passes on the release branch, including: + * Regular unit tests + * Build tests + * The E4S pipeline at `gitlab.spack.io `_ + + If CI is not passing, submit pull requests to ``develop`` as normal + and keep rebasing the release branch on ``develop`` until CI passes. + +#. Follow the steps in :ref:`publishing-releases`. + +#. Follow the steps in :ref:`merging-releases-to-develop`. + +#. Follow the steps in :ref:`announcing-releases`. + + +.. _point-releases: + +^^^^^^^^^^^^^^^^^^^^^ +Making Point Releases +^^^^^^^^^^^^^^^^^^^^^ + +This assumes you've already created a project board for a point release +and completed the work to be done for the release. To make a point +release: + +#. Create one new project board for the next point release. + +#. Move any cards that aren't done yet to the next project board. + +#. Check out the release branch (it should already exist). For the + ``X.Y.Z`` release, the release branch is called ``releases/vX.Y``. For + ``v0.15.1``, you would check out ``releases/v0.15``: + + .. code-block:: console + + $ git checkout releases/v0.15 + +#. Cherry-pick each pull request in the ``Done`` column of the release + project onto the release branch. + + This is **usually** fairly simple since we squash the commits from the + vast majority of pull requests, which means there is only one commit + per pull request to cherry-pick. For example, `this pull request + `_ has three commits, but + the were squashed into a single commit on merge. You can see the + commit that was created here: + + .. image:: images/pr-commit.png + + You can easily cherry pick it like this (assuming you already have the + release branch checked out): + + .. code-block:: console + + $ git cherry-pick 7e46da7 + + For pull requests that were rebased, you'll need to cherry-pick each + rebased commit individually. There have not been any rebased PRs like + this in recent point releases. + + .. warning:: + + It is important to cherry-pick commits in the order they happened, + otherwise you can get conflicts while cherry-picking. When + cherry-picking onto a point release, look at the merge date, + **not** the number of the pull request or the date it was opened. + + Sometimes you may **still** get merge conflicts even if you have + cherry-picked all the commits in order. This generally means there + is some other intervening pull request that the one you're trying + to pick depends on. In these cases, you'll need to make a judgment + call: + + 1. If the dependency is small, you might just cherry-pick it, too. + If you do this, add it to the release board. + + 2. If it is large, then you may decide that this fix is not worth + including in a point release, in which case you should remove it + from the release project. + + 3. You can always decide to manually back-port the fix to the release + branch if neither of the above options makes sense, but this can + require a lot of work. It's seldom the right choice. + +#. Bump the version in ``lib/spack/spack/__init__.py``. See `this example from 0.14.1 + `_. + +#. Updaate the release version lists in these files to include the new version: + + * ``lib/spack/spack/schema/container.py`` + * ``lib/spack/spack/container/images.json`` + + **TODO**: We should get rid of this step in some future release. + +#. Update ``CHANGELOG.md`` with a list of bugfixes. This is typically just a + summary of the commits you cherry-picked onto the release branch. See + `the changelog from 0.14.1 + `_. + +#. Push the release branch to GitHub. + +#. Make sure CI passes on the release branch, including: + * Regular unit tests + * Build tests + * The E4S pipeline at `gitlab.spack.io `_ + + If CI does not pass, you'll need to figure out why, and make changes + to the release branch until it does. You can make more commits, modify + or remove cherry-picked commits, or cherry-pick **more** from + ``develop`` to make this happen. + +#. Follow the steps in :ref:`publishing-releases`. + +#. Follow the steps in :ref:`merging-releases-to-develop`. + +#. Follow the steps in :ref:`announcing-releases`. + + +.. _publishing-releases: + +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Publishing a release on GitHub +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +#. Go to `github.com/spack/spack/releases + `_ and click ``Draft a new + release``. Set the following: + + * ``Tag version`` should start with ``v`` and contain *all three* + parts of the version, .g. ``v0.15.1``. This is the name of the tag + that will be created. + + * ``Target`` should be the ``releases/vX.Y`` branch (e.g., ``releases/v0.15``). + + * ``Release title`` should be ``vX.Y.Z`` (To match the tag, e.g., ``v0.15.1``). + + * For the text, paste the latest release markdown from your ``CHANGELOG.md``. + + You can save the draft and keep coming back to this as you prepare the release. + +#. When you are done, click ``Publish release``. + +#. Immediately after publishing, go back to + `github.com/spack/spack/releases + `_ and download the + auto-generated ``.tar.gz`` file for the release. It's the ``Source + code (tar.gz)`` link. + +#. Click ``Edit`` on the release you just did and attach the downloaded + release tarball as a binary. This does two things: + + #. Makes sure that the hash of our releases doesn't change over time. + GitHub sometimes annoyingly changes they way they generate + tarballs, and then hashes can change if you rely on the + auto-generated tarball links. + + #. Gets us download counts on releases visible through the GitHub + API. GitHub tracks downloads of artifacts, but *not* the source + links. See the `releases + page `_ and search + for ``download_count`` to see this. + + +.. _merging-releases-to-develop: + +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Merging back into ``develop`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Once each release is complete, make sure that it is merged back into +``develop`` with a merge commit: + +.. code-block:: console + + $ git checkout develop + $ git merge --no-ff releases/v0.15 + $ git push + +We merge back to ``develop`` because it: + + * updates the version and ``CHANGELOG.md`` on ``develop``. + * ensures that your release tag is reachable from the head of + ``develop`` + +We *must* use a real merge commit (via the ``--no-ff`` option) because it +ensures that the release tag is reachable from the tip of ``develop``. +This is necessary for ``spack -V`` to work properly -- it uses ``git +describe --tags`` to find the last reachable tag in the repository and +reports how far we are from it. For example: + +.. code-block:: console + + $ spack -V + 0.14.2-1486-b80d5e74e5 + +This says that we are at commit ``b80d5e74e5``, which is 1,486 commits +ahead of the ``0.14.2`` release. + +We put this step last in the process because it's best to do it only once +the release is complete and tagged. If you do it before you've tagged the +release and later decide you want to tag some later commit, you'll need +to merge again. + + +.. _announcing-releases: + +^^^^^^^^^^^^^^^^^^^^ +Announcing a release +^^^^^^^^^^^^^^^^^^^^ + +We announce releases in all of the major Spack communication channels. +Publishing the release takes care of GitHub. The remaining channels are +Twitter, Slack, and the mailing list. Here are the steps: + +#. Make a tweet to announce the release. It should link to the release's + page on GitHub. You can base it on `this example tweet + `_. + +#. Ping ``@channel`` in ``#general`` on Slack (`spackpm.slack.com + `_) with a link to the tweet. The tweet + will be shown inline so that you do not have to retype your release + announcement. + +#. Email the Spack mailing list to let them know about the release. As + with the tweet, you likely want to link to the release's page on + GitHub. It's also helpful to include some information directly in the + email. You can base yours on this `example email + `_. + +Once you've announced the release, congratulations, you're done! You've +finished making the release! diff --git a/lib/spack/docs/getting_started.rst b/lib/spack/docs/getting_started.rst index 226b1f0883..7b908465f5 100644 --- a/lib/spack/docs/getting_started.rst +++ b/lib/spack/docs/getting_started.rst @@ -818,7 +818,7 @@ Git Some Spack packages use ``git`` to download, which might not work on some computers. For example, the following error was -encountered on a Macintosh during ``spack install julia-master``: +encountered on a Macintosh during ``spack install julia@master``: .. code-block:: console diff --git a/lib/spack/docs/images/pr-commit.png b/lib/spack/docs/images/pr-commit.png new file mode 100644 index 0000000000..a87c800ef5 Binary files /dev/null and b/lib/spack/docs/images/pr-commit.png differ diff --git a/lib/spack/docs/images/projects.png b/lib/spack/docs/images/projects.png new file mode 100644 index 0000000000..bd2971ff62 Binary files /dev/null and b/lib/spack/docs/images/projects.png differ -- cgit v1.2.3-70-g09d2