CI: Add documentation for adding new stacks and runners (#42179)

* CI: Add documentation for adding new stacks and runners
* More docs for runner registration

---------

Co-authored-by: Zack Galbreath <zack.galbreath@kitware.com>
Co-authored-by: Bernhard Kaindl <contact@bernhard.kaindl.dev>

1 files changed, 209 insertions, 0 deletions

diff --git a/lib/spack/docs/contribution_guide.rst b/lib/spack/docs/contribution_guide.rst
index 68a381be9f..97f181b85b 100644
--- a/lib/spack/docs/contribution_guide.rst
+++ b/lib/spack/docs/contribution_guide.rst
@@ -316,6 +316,215 @@ documentation tests to make sure there are no errors. Documentation changes can
 in some obfuscated warning messages. If you don't understand what they mean, feel free
 to ask when you submit your PR.
 
+.. _spack-builders-and-pipelines:
+
+^^^^^^^^^
+GitLab CI
+^^^^^^^^^
+
+""""""""""""""""""
+Build Cache Stacks
+""""""""""""""""""
+
+Spack welcomes the contribution of software stacks of interest to the community. These
+stacks are used to test package recipes and generate publicly available build caches.
+Spack uses GitLab CI for managing the orchestration of build jobs.
+
+GitLab Entry Point
+~~~~~~~~~~~~~~~~~~
+
+Add stack entrypoint to the ``share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml``. There
+are two stages required for each new stack, the generation stage and the build stage.
+
+The generate stage is defined using the job template ``.generate`` configured with
+environment variables defining the name of the stack in ``SPACK_CI_STACK_NAME`` and the
+platform (``SPACK_TARGET_PLATFORM``) and architecture (``SPACK_TARGET_ARCH``) configuration,
+and the tags associated with the class of runners to build on.
+
+.. note::
+
+    The ``SPACK_CI_STACK_NAME`` must match the name of the directory containing the
+    stacks ``spack.yaml``.
+
+
+.. note::
+
+    The platform and architecture variables are specified in order to select the
+    correct configurations from the generic configurations used in Spack CI. The
+    configurations currently available are:
+
+    * ``.cray_rhel_zen4``
+    * ``.cray_sles_zen4``
+    * ``.darwin_aarch64``
+    * ``.darwin_x86_64``
+    * ``.linux_aarch64``
+    * ``.linux_icelake``
+    * ``.linux_neoverse_n1``
+    * ``.linux_neoverse_v1``
+    * ``.linux_neoverse_v2``
+    * ``.linux_power``
+    * ``.linux_skylake``
+    * ``.linux_x86_64``
+    * ``.linux_x86_64_v4``
+
+    New configurations can be added to accommodate new platforms and architectures.
+
+
+The build stage is defined as a trigger job that consumes the GitLab CI pipeline generated in
+the generate stage for this stack. Build stage jobs use the ``.build`` job template which
+handles the basic configuration.
+
+An example entry point for a new stack called ``my-super-cool-stack``
+
+.. code-block:: yaml
+
+    .my-super-cool-stack:
+      extends: [ ".linux_x86_64_v3" ]
+      variables:
+        SPACK_CI_STACK_NAME: my-super-cool-stack
+        tags: [ "all", "tags", "your", "job", "needs"]
+
+    my-super-cool-stack-generate:
+      extends: [ ".generate", ".my-super-cool-stack" ]
+      image: my-super-cool-stack-image:0.0.1
+
+    my-super-cool-stack-build:
+      extends: [ ".build", ".my-super-cool-stack" ]
+      trigger:
+        include:
+          - artifact: jobs_scratch_dir/cloud-ci-pipeline.yml
+            job: my-super-cool-stack-generate
+        strategy: depend
+      needs:
+        - artifacts: True
+          job: my-super-cool-stack-generate
+
+
+Stack Configuration
+~~~~~~~~~~~~~~~~~~~
+
+The stack configuration is a spack environment file with two additional sections added.
+Stack configurations should be located in ``share/spack/gitlab/cloud_pipelines/stacks/<stack_name>/spack.yaml``.
+
+The ``ci`` section is generally used to define stack specific mappings such as image or tags.
+For more information on what can go into the ``ci`` section refer to the docs on pipelines.
+
+The ``cdash`` section is used for defining where to upload the results of builds. Spack configures
+most of the details for posting pipeline results to
+`cdash.spack.io <https://cdash.spack.io/index.php?project=Spack+Testing>`_. The only
+requirement in the stack configuration is to define a ``build-group`` that is unique,
+this is usually the long name of the stack.
+
+An example stack that builds ``zlib``.
+
+.. code-block:: yaml
+
+    spack:
+      view: false
+      packages:
+        all:
+          require: ["%gcc", "target=x86_64_v3"]
+      specs:
+      - zlib
+
+      ci:
+        pipeline-gen
+        - build-job:
+            image: my-super-cool-stack-image:0.0.1
+
+      cdash:
+        build-group: My Super Cool Stack
+
+.. note::
+
+    The ``image`` used in the ``*-generate`` job must match exactly the ``image`` used in the ``build-job``.
+    When the images do not match the build job may fail.
+
+
+"""""""""""""""""""
+Registering Runners
+"""""""""""""""""""
+
+Contributing computational resources to Spack's CI build farm is one way to help expand the
+capabilities and offerings of the public Spack build caches. Currently, Spack utilizes linux runners
+from AWS, Google, and the University of Oregon (UO).
+
+Runners require three key peices:
+* Runner Registration Token
+* Accurate tags
+* OIDC Authentication script
+* GPG keys
+
+
+Minimum GitLab Runner Version: ``16.1.0``
+`Intallation instructions <https://docs.gitlab.com/runner/install/>`_
+
+Registration Token
+~~~~~~~~~~~~~~~~~~
+
+The first step to contribute new runners is to open an issue in the `spack infrastructure <https://github.com/spack/spack-infrastructure/issues/new?assignees=&labels=runner-registration&projects=&template=runner_registration.yml>`_
+project. This will be reported to the spack infrastructure team who will guide users through the process
+of registering new runners for Spack CI.
+
+The information needed to register a runner is the motivation for the new resources, a semi-detailed description of
+the runner, and finallly the point of contact for maintaining the software on the runner.
+
+The point of contact will then work with the infrastruture team to obtain runner registration token(s) for interacting with
+with Spack's GitLab instance. Once the runner is active, this point of contact will also be responsible for updating the
+GitLab runner software to keep pace with Spack's Gitlab.
+
+Tagging
+~~~~~~~
+
+In the initial stages of runner registration it is important to **exclude** the special tag ``spack``. This will prevent
+the new runner(s) from being picked up for production CI jobs while it is configured and evaluated. Once it is determined
+that the runner is ready for production use the ``spack`` tag will be added.
+
+Because gitlab has no concept of tag exclustion, runners that provide specialized resource also require specialized tags.
+For example, a basic CPU only x86_64 runner may have a tag ``x86_64`` associated with it. However, a runner containing an
+CUDA capable GPU may have the tag ``x86_64-cuda`` to denote that it should only be used for packages that will benefit from
+a CUDA capable resource.
+
+OIDC
+~~~~
+
+Spack runners use OIDC authentication for connecting to the appropriate AWS bucket
+which is used for coordinating the communication of binaries between build jobs. In
+order to configure OIDC authentication, Spack CI runners use a python script with minimal
+dependencies. This script can be configured for runners as seen here using the ``pre_build_script``.
+
+.. code-block:: toml
+
+    [[runners]]
+      pre_build_script = """
+      echo 'Executing Spack pre-build setup script'
+
+      for cmd in "${PY3:-}" python3 python; do
+        if command -v > /dev/null "$cmd"; then
+          export PY3="$(command -v "$cmd")"
+          break
+        fi
+      done
+
+      if [ -z "${PY3:-}" ]; then
+        echo "Unable to find python3 executable"
+        exit 1
+      fi
+
+      $PY3 -c "import urllib.request;urllib.request.urlretrieve('https://raw.githubusercontent.com/spack/spack-infrastructure/main/scripts/gitlab_runner_pre_build/pre_build.py', 'pre_build.py')"
+      $PY3 pre_build.py > envvars
+
+      . ./envvars
+      rm -f envvars
+      unset GITLAB_OIDC_TOKEN
+      """
+
+GPG Keys
+~~~~~~~~
+
+Runners that may be utilized for ``protected`` CI require the registration of an intermediate signing key that
+can be used to sign packages. For more information on package signing read :ref:`key_architecture`.
+
 --------
 Coverage
 --------