Docs: Update pipeline ci rebuild to add --tests (plus fixed typos) (#32048)

1 files changed, 64 insertions, 24 deletions

diff --git a/lib/spack/docs/pipelines.rst b/lib/spack/docs/pipelines.rst
index bd695c29ba..419e128fc7 100644
--- a/lib/spack/docs/pipelines.rst
+++ b/lib/spack/docs/pipelines.rst
@@ -5,9 +5,9 @@
 
 .. _pipelines:
 
-=========
-Pipelines
-=========
+============
+CI Pipelines
+============
 
 Spack provides commands that support generating and running automated build
 pipelines designed for Gitlab CI.  At the highest level it works like this:
@@ -168,7 +168,7 @@ which specs are up to date and which need to be rebuilt (it's a good idea for ot
 reasons as well, but those are out of scope for this discussion).  In this case we
 have disabled it (using ``rebuild-index: False``) because the index would only be
 generated in the artifacts mirror anyway, and consequently would not be available
-during subesequent pipeline runs.
+during subsequent pipeline runs.
 
 .. note::
    With the addition of reproducible builds (#22887) a previously working
@@ -267,24 +267,64 @@ generated by jobs in the pipeline.
 ``spack ci rebuild``
 ^^^^^^^^^^^^^^^^^^^^^
 
-The purpose of the ``spack ci rebuild`` is straightforward: take its assigned
-spec job, check whether the target mirror already has a binary for that spec,
-and if not, build the spec from source and push the binary to the mirror.  To
-accomplish this in a reproducible way, the sub-command prepares a ``spack install``
-command line to build a single spec in the DAG, saves that command in a
-shell script, ``install.sh``, in the current working directory, and then runs
-it to install the spec.  The shell script is also exported as an artifact to
-aid in reproducing the build outside of the CI environment.
-
-If it was necessary to install the spec from source, ``spack ci rebuild`` will
-also subsequently create a binary package for the spec and try to push it to the
-mirror.
-
-The ``spack ci rebuild`` sub-command mainly expects its "input" to come either
-from environment variables or from the ``gitlab-ci`` section of the ``spack.yaml``
-environment file.  There are two main sources of the environment variables, some
-are written into ``.gitlab-ci.yml`` by ``spack ci generate``, and some are
-provided by the GitLab CI runtime.
+The purpose of ``spack ci rebuild`` is straightforward: take its assigned
+spec and ensure a binary of a successful build exists on the target mirror.
+If the binary does not already exist, it is built from source and pushed
+to the mirror. The associated stand-alone tests are optionally run against
+the new build. Additionally, files for reproducing the build outside of the
+CI environment are created to facilitate debugging.
+
+If a binary for the spec does not exist on the target mirror, an install
+shell script, ``install.sh``, is created and saved in the current working
+directory. The script is run in a job to install the spec from source. The
+resulting binary package is pushed to the mirror. If ``cdash`` is configured
+for the environment, then the build results will be uploaded to the site.
+
+Environment variables and values in the ``gitlab-ci`` section of the
+``spack.yaml`` environment file provide inputs to this process. The
+two main sources of environment variables are variables written into
+``.gitlab-ci.yml`` by ``spack ci generate`` and the GitLab CI runtime.
+Several key CI pipeline variables are described in
+:ref:`ci_environment_variables`.
+
+If the ``--tests`` option is provided, stand-alone tests are performed but
+only if the build was successful *and* the package does not appear in the
+list of ``broken-tests-packages``. A shell script, ``test.sh``, is created
+and run to perform the tests. On completion, test logs are exported as job
+artifacts for review and to facilitate debugging. If `cdash` is configured,
+test results are also uploaded to the site.
+
+A snippet from an example ``spack.yaml`` file illustrating use of this
+option *and* specification of a package with broken tests is given below.
+The inclusion of a spec for building ``gptune`` is not shown here. Note
+that ``--tests`` is passed to ``spack ci rebuild`` as part of the
+``gitlab-ci`` script.
+
+.. code-block:: yaml
+
+  gitlab-ci:
+    script:
+      - . "./share/spack/setup-env.sh"
+      - spack --version
+      - cd ${SPACK_CONCRETE_ENV_DIR}
+      - spack env activate --without-view .
+      - spack config add "config:install_tree:projections:${SPACK_JOB_SPEC_PKG_NAME}:'morepadding/{architecture}/{compiler.name}-{compiler.version}/{name}-{version}-{hash}'"
+       - mkdir -p ${SPACK_ARTIFACTS_ROOT}/user_data
+       - if [[ -r /mnt/key/intermediate_ci_signing_key.gpg ]]; then spack gpg trust /mnt/key/intermediate_ci_signing_key.gpg; fi
+       - if [[ -r /mnt/key/spack_public_key.gpg ]]; then spack gpg trust /mnt/key/spack_public_key.gpg; fi
+       - spack -d ci rebuild --tests > >(tee ${SPACK_ARTIFACTS_ROOT}/user_data/pipeline_out.txt) 2> >(tee ${SPACK_ARTIFACTS_ROOT}/user_data/pipeline_err.txt >&2)
+
+     broken-tests-packages:
+       - gptune
+
+In this case, even if ``gptune`` is successfully built from source, the
+pipeline will *not* run its stand-alone tests since the package is listed
+under ``broken-tests-packages``.
+
+Spack's cloud pipelines provide actual, up-to-date examples of the CI/CD
+configuration and environment files used by Spack. You can find them
+under Spack's `stacks
+<https://github.com/spack/spack/tree/develop/share/spack/gitlab/cloud_pipelines/stacks>`_ repository directory.
 
 .. _cmd-spack-ci-rebuild-index:
 
@@ -447,7 +487,7 @@ Note about "no-op" jobs
 ^^^^^^^^^^^^^^^^^^^^^^^
 
 If no specs in an environment need to be rebuilt during a given pipeline run
-(meaning all are already up to date on the mirror), a single succesful job
+(meaning all are already up to date on the mirror), a single successful job
 (a NO-OP) is still generated to avoid an empty pipeline (which GitLab
 considers to be an error).  An optional ``service-job-attributes`` section
 can be added to your ``spack.yaml`` where you can provide ``tags`` and
@@ -725,7 +765,7 @@ above with ``git checkout ${SPACK_CHECKOUT_VERSION}``.
 On the other hand, if you're pointing to a spack repository and branch under your
 control, there may be no benefit in using the captured ``SPACK_CHECKOUT_VERSION``,
 and you can instead just clone using the variables you define (``SPACK_REPO``
-and ``SPACK_REF`` in the example aboves).
+and ``SPACK_REF`` in the example above).
 
 .. _custom_workflow: