Add documentation about package signing model (#30939)

Co-authored-by: Scott Wittenburg <scott.wittenburg@kitware.com>
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>

2 files changed, 485 insertions, 0 deletions

diff --git a/lib/spack/docs/index.rst b/lib/spack/docs/index.rst
index 69a5ee4f4f..6a80aa1a24 100644
--- a/lib/spack/docs/index.rst
+++ b/lib/spack/docs/index.rst
@@ -76,6 +76,7 @@ or refer to the full manual below.
    chain
    extensions
    pipelines
+   signing
 
 .. toctree::
    :maxdepth: 2
diff --git a/lib/spack/docs/signing.rst b/lib/spack/docs/signing.rst
new file mode 100644
index 0000000000..f88b87f07e
--- /dev/null
+++ b/lib/spack/docs/signing.rst
@@ -0,0 +1,484 @@
+.. Copyright 2013-2022 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
+
+   SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+.. _signing:
+
+=====================
+Spack Package Signing
+=====================
+
+The goal of package signing in Spack is to provide data integrity
+assurances around official packages produced by the automated Spack CI
+pipelines. These assurances directly address the security of Spack’s
+software supply chain by explaining why a security-conscious user can
+be reasonably justified in the belief that packages installed via Spack
+have an uninterrupted auditable trail back to change management
+decisions judged to be appropriate by the Spack maintainers. This is
+achieved through cryptographic signing of packages built by Spack CI
+pipelines based on code that has been transparently reviewed and
+approved on GitHub. This document describes the signing process for
+interested users.
+
+.. _risks:
+
+------------------------------
+Risks, Impact and Threat Model
+------------------------------
+
+This document addresses the approach taken to safeguard Spack’s
+reputation with regard to the integrity of the package data produced by
+Spack’s CI pipelines. It does not address issues of data confidentiality
+(Spack is intended to be largely open source) or availability (efforts
+are described elsewhere). With that said the main reputational risk can
+be broadly categorized as a loss of faith in the data integrity due to a
+breach of the private key used to sign packages. Remediation of a
+private key breach would require republishing the public key with a
+revocation certificate, generating a new signing key, an assessment and
+potential rebuild/resigning of all packages since the key was breached,
+and finally direct intervention by every spack user to update their copy
+of Spack’s public keys used for local verification.
+
+The primary threat model used in mitigating the risks of these stated
+impacts is one of individual error not malicious intent or insider
+threat. The primary objective is to avoid the above impacts by making a
+private key breach nearly impossible due to oversight or configuration
+error. Obvious and straightforward measures are taken to mitigate issues
+of malicious interference in data integrity and insider threats but
+these attack vectors are not systematically addressed. It should be hard
+to exfiltrate the private key intentionally, and almost impossible to
+leak the key by accident.
+
+.. _overview:
+
+-----------------
+Pipeline Overview
+-----------------
+
+Spack pipelines build software through progressive stages where packages
+in later stages nominally depend on packages built in earlier stages.
+For both technical and design reasons these dependencies are not
+implemented through the default GitLab artifacts mechanism; instead
+built packages are uploaded to AWS S3 mirrors (buckets) where they are
+retrieved by subsequent stages in the pipeline. Two broad categories of
+pipelines exist: Pull Request (PR) pipelines and Develop/Release
+pipelines.
+
+-  PR pipelines are launched in response to pull requests made by
+   trusted and untrusted users. Packages built on these pipelines upload
+   code to quarantined AWS S3 locations which cache the built packages
+   for the purposes of review and iteration on the changes proposed in
+   the pull request. Packages built on PR pipelines can come from
+   untrusted users so signing of these pipelines is not implemented.
+   Jobs in these pipelines are executed via normal GitLab runners both
+   within the AWS GitLab infrastructure and at affiliated institutions.
+-  Develop and Release pipelines **sign** the packages they produce and carry
+   strong integrity assurances that trace back to auditable change management
+   decisions. These pipelines only run after members from a trusted group of
+   reviewers verify that the proposed changes in a pull request are appropriate.
+   Once the PR is merged, or a release is cut, a pipeline is run on protected
+   GitLab runners which provide access to the required signing keys within the
+   job. Intermediary keys are used to sign packages in each stage of the
+   pipeline as they are built and a final job officially signs each package
+   external to any specific packages’ build environment. An intermediate key
+   exists in the AWS infrastructure and for each affiliated instritution that
+   maintains protected runners. The runners that execute these pipelines
+   exclusively accept jobs from protected branches meaning the intermediate keys
+   are never exposed to unreviewed code and the official keys are never exposed
+   to any specific build environment.
+
+.. _key_architecture:
+
+----------------
+Key Architecture
+----------------
+
+Spack’s CI process uses public-key infrastructure (PKI) based on GNU Privacy
+Guard (gpg) keypairs to sign public releases of spack package metadata, also
+called specs. Two classes of GPG keys are involved in the process to reduce the
+impact of an individual private key compromise, these key classes are the
+*Intermediate CI Key* and *Reputational Key*. Each of these keys has signing
+sub-keys that are used exclusively for signing packages. This can be confusing
+so for the purpose of this explanation we’ll refer to Root and Signing keys.
+Each key has a private and a public component as well as one or more identities
+and zero or more signatures.
+
+-------------------
+Intermediate CI Key
+-------------------
+
+The Intermediate key class is used to sign and verify packages between stages
+within a develop or release pipeline. An intermediate key exists for the AWS
+infrastructure as well as each affiliated institution that maintains protected
+runners. These intermediate keys are made available to the GitLab execution
+environment building the package so that the package’s dependencies may be
+verified by the Signing Intermediate CI Public Key and the final package may be
+signed by the Signing Intermediate CI Private Key.
+
+
++---------------------------------------------------------------------------------------------------------+
+| **Intermediate CI Key (GPG)**                                                                           |
++==================================================+======================================================+
+| Root Intermediate CI Private Key (RSA 4096)#     |     Root Intermediate CI Public Key (RSA 4096)       |
++--------------------------------------------------+------------------------------------------------------+
+|   Signing Intermediate CI Private Key (RSA 4096) |        Signing Intermediate CI Public Key (RSA 4096) |
++--------------------------------------------------+------------------------------------------------------+
+| Identity: “Intermediate CI Key <maintainers@spack.io>”                                                  |
++---------------------------------------------------------------------------------------------------------+
+| Signatures: None                                                                                        |
++---------------------------------------------------------------------------------------------------------+
+
+
+The *Root intermediate CI Private Key*\ Is stripped out of the GPG key and
+stored offline completely separate from Spack’s infrastructure. This allows the
+core development team to append revocation certificates to the GPG key and
+issue new sub-keys for use in the pipeline. It is our expectation that this
+will happen on a semi regular basis. A corollary of this is that *this key
+should not be used to verify package integrity outside the internal CI process.*
+
+----------------
+Reputational Key
+----------------
+
+The Reputational Key is the public facing key used to sign complete groups of
+development and release packages. Only one key pair exsits in this class of
+keys. In contrast to the Intermediate CI Key the Reputational Key *should* be
+used to verify package integrity. At the end of develop and release pipeline a
+final pipeline job pulls down all signed package metadata built by the pipeline,
+verifies they were signed with an Intermediate CI Key, then strips the
+Intermediate CI Key signature from the package and re-signs them with the
+Signing Reputational Private Key. The officially signed packages are then
+uploaded back to the AWS S3 mirror. Please note that separating use of the
+reputational key into this final job is done to prevent leakage of the key in a
+spack package. Because the Signing Reputational Private Key is never exposed to
+a build job it cannot accidentally end up in any built package.
+
+
++---------------------------------------------------------------------------------------------------------+
+| **Reputational Key (GPG)**                                                                              |
++==================================================+======================================================+
+| Root Reputational Private Key (RSA 4096)#        |          Root Reputational Public Key (RSA 4096)     |
++--------------------------------------------------+------------------------------------------------------+
+| Signing Reputational Private Key (RSA 4096)      |          Signing Reputational Public Key (RSA 4096)  |
++--------------------------------------------------+------------------------------------------------------+
+| Identity: “Spack Project <maintainers@spack.io>”                                                        |
++---------------------------------------------------------------------------------------------------------+
+| Signatures: Signed by core development team [#f1]_                                                      |
++---------------------------------------------------------------------------------------------------------+
+
+The Root Reputational Private Key is stripped out of the GPG key and stored
+offline completely separate from Spack’s infrastructure. This allows the core
+development team to append revocation certificates to the GPG key in the
+unlikely event that the Signing Reputation Private Key is compromised. In
+general it is the expectation that rotating this key will happen infrequently if
+at all. This should allow relatively transparent verification for the end-user
+community without needing deep familiarity with GnuPG or Public Key
+Infrastructure.
+
+
+.. _build_cache_format:
+
+------------------
+Build Cache Format
+------------------
+
+A binary package consists of a metadata file unambiguously defining the
+built package (and including other details such as how to relocate it)
+and the installation directory of the package stored as a compressed
+archive file. The metadata files can either be unsigned, in which case
+the contents are simply the json-serialized concrete spec plus metadata,
+or they can be signed, in which case the json-serialized concrete spec
+plus metadata is wrapped in a gpg cleartext signature. Built package
+metadata files are named to indicate the operating system and
+architecture for which the package was built as well as the compiler
+used to build it and the packages name and version. For example::
+
+  linux-ubuntu18.04-haswell-gcc-7.5.0-zlib-1.2.12-llv2ysfdxnppzjrt5ldybb5c52qbmoow.spec.json.sig
+
+would contain the concrete spec and binary metadata for a binary package
+of ``zlib@1.2.12``, built for the ``ubuntu`` operating system and ``haswell``
+architecture. The id of the built package exists in the name of the file
+as well (after the package name and version) and in this case begins
+with ``llv2ys``. The id distinguishes a particular built package from all
+other built packages with the same os/arch, compiler, name, and version.
+Below is an example of a signed binary package metadata file. Such a
+file would live in the ``build_cache`` directory of a binary mirror::
+
+  -----BEGIN PGP SIGNED MESSAGE-----
+  Hash: SHA512
+
+  {
+    "spec": {
+      <concrete-spec-contents-omitted>
+    },
+
+    "buildcache_layout_version": 1,
+    "binary_cache_checksum": {
+      "hash_algorithm": "sha256",
+      "hash": "4f1e46452c35a5e61bcacca205bae1bfcd60a83a399af201a29c95b7cc3e1423"
+     },
+
+    "buildinfo": {
+      "relative_prefix":
+      "linux-ubuntu18.04-haswell/gcc-7.5.0/zlib-1.2.12-llv2ysfdxnppzjrt5ldybb5c52qbmoow",
+      "relative_rpaths": false
+    }
+  }
+
+  -----BEGIN PGP SIGNATURE-----
+  iQGzBAEBCgAdFiEETZn0sLle8jIrdAPLx/P+voVcifMFAmKAGvwACgkQx/P+voVc
+  ifNoVgv/VrhA+wurVs5GB9PhmMA1m5U/AfXZb4BElDRwpT8ZcTPIv5X8xtv60eyn
+  4EOneGVbZoMThVxgev/NKARorGmhFXRqhWf+jknJZ1dicpqn/qpv34rELKUpgXU+
+  QDQ4d1P64AIdTczXe2GI9ZvhOo6+bPvK7LIsTkBbtWmopkomVxF0LcMuxAVIbA6b
+  887yBvVO0VGlqRnkDW7nXx49r3AG2+wDcoU1f8ep8QtjOcMNaPTPJ0UnjD0VQGW6
+  4ZFaGZWzdo45MY6tF3o5mqM7zJkVobpoW3iUz6J5tjz7H/nMlGgMkUwY9Kxp2PVH
+  qoj6Zip3LWplnl2OZyAY+vflPFdFh12Xpk4FG7Sxm/ux0r+l8tCAPvtw+G38a5P7
+  QEk2JBr8qMGKASmnRlJUkm1vwz0a95IF3S9YDfTAA2vz6HH3PtsNLFhtorfx8eBi
+  Wn5aPJAGEPOawEOvXGGbsH4cDEKPeN0n6cy1k92uPEmBLDVsdnur8q42jk5c2Qyx
+  j3DXty57
+  =3gvm
+  -----END PGP SIGNATURE-----
+
+If a user has trusted the public key associated with the private key
+used to sign the above spec file, the signature can be verified with
+gpg, as follows::
+
+  $ gpg –verify linux-ubuntu18.04-haswell-gcc-7.5.0-zlib-1.2.12-llv2ysfdxnppzjrt5ldybb5c52qbmoow.spec.json.sig
+
+The metadata (regardless whether signed or unsigned) contains the checksum
+of the ``.spack`` file containing the actual installation. The checksum should
+be compared to a checksum computed locally on the ``.spack`` file to ensure the
+contents have not changed since the binary spec plus metadata were signed. The
+``.spack`` files are actually tarballs containing the compressed archive of the
+install tree.  These files, along with the metadata files, live within the
+``build_cache`` directory of the mirror, and together are organized as follows::
+
+  build_cache/
+    # unsigned metadata (for indexing, contains sha256 of .spack file)
+    <arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.spec.json
+    # clearsigned metadata (same as above, but signed)
+    <arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.spec.json.sig
+    <arch>/
+      <compiler>/
+        <name>-<ver>/
+          # tar.gz-compressed prefix (may support more compression formats later)
+          <arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.spack
+
+Uncompressing and extracting the ``.spack`` file results in the install tree.
+This is in contrast to previous versions of spack, where the ``.spack`` file
+contained a (duplicated) metadata file, a signature file and a nested tarball
+containing the install tree.
+
+.. _internal_implementation:
+
+-----------------------
+Internal Implementation
+-----------------------
+
+The technical implementation of the pipeline signing process includes components
+defined in Amazon Web Services, the Kubernetes cluster, at affilicated
+institutions, and the GitLab/GitLab Runner deployment. We present the techincal
+implementation in two interdependent sections. The first addresses how secrets
+are managed through the lifecycle of a develop or release pipeline. The second
+section describes how Gitlab Runner and pipelines are configured and managed to
+support secure automated signing.
+
+Secrets Management
+^^^^^^^^^^^^^^^^^^
+
+As stated above the Root Private Keys (intermediate and reputational)
+are stripped from the GPG keys and stored outside Spack’s
+infrastructure.
+
+.. warning::
+  **TODO**
+    - Explanation here about where and how access is handled for these keys.
+    - Both Root private keys are protected with strong passwords
+    - Who has access to these and how?
+
+**Intermediate CI Key**
+-----------------------
+
+Multiple intermediate CI signing keys exist, one Intermediate CI Key for jobs
+run in AWS, and one key for each affiliated institution (e.g. Univerity of
+Oregon). Here we describe how the Intermediate CI Key is managed in AWS:
+
+The Intermediate CI Key (including the Signing Intermediate CI Private Key is
+exported as an ASCII armored file and stored in a Kubernetes secret called
+``spack-intermediate-ci-signing-key``. For convenience sake, this same secret
+contains an ASCII-armored export of just the *public* components of the
+Reputational Key. This secret also contains the *public* components of each of
+the affiliated institutions' Intermediate CI Key. These are potentially needed
+to verify dependent packages which may have been found in the public mirror or
+built by a protected job running on an affiliated institution's infrastrcuture
+in an earlier stage of the pipeline.
+
+Procedurally the ``spack-intermediate-ci-signing-key`` secret is used in
+the following way:
+
+1. A ``large-arm-prot`` or ``large-x86-prot`` protected runner picks up
+   a job tagged ``protected`` from a protected GitLab branch. (See
+   `Protected Runners and Reserved Tags <#_8bawjmgykv0b>`__).
+2. Based on its configuration, the runner creates a job Pod in the
+   pipeline namespace and mounts the spack-intermediate-ci-signing-key
+   Kubernetes secret into the build container
+3. The Intermediate CI Key, affiliated institutions' public key and the
+   Reputational Public Key are imported into a keyring by the ``spack gpg …``
+   sub-command. This is initiated by the job’s build script which is created by
+   the generate job at the beginning of the pipeline.
+4. Assuming the package has dependencies those specs are verified using
+   the keyring.
+5. The package is built and the spec.json is generated
+6. The spec.json is signed by the keyring and uploaded to the mirror’s
+   build cache.
+
+**Reputational Key**
+--------------------
+
+Because of the increased impact to end users in the case of a private
+key breach, the Reputational Key is managed separately from the
+Intermediate CI Keys and has additional controls. First, the Reputational
+Key was generated outside of Spack’s infrastructure and has been signed
+by the core development team. The Reputational Key (along with the
+Signing Reputational Private Key) was then ASCII armor exported to a
+file. Unlike the Intermediate CI Key this exported file is not stored as
+a base64 encoded secret in Kubernetes. Instead\ *the key file
+itself*\ is encrypted and stored in Kubernetes as the
+``spack-signing-key-encrypted`` secret in the pipeline namespace.
+
+The encryption of the exported Reputational Key (including the Signing
+Reputational Private Key) is handled by `AWS Key Management Store (KMS) data
+keys
+<https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#data-keys>`__.
+The private key material is decrypted and imported at the time of signing into a
+memory mounted temporary directory holding the keychain. The signing job uses
+the `AWS Encryption SDK
+<https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/crypto-cli.html>`__
+(i.e. ``aws-encryption-cli``) to decrypt the Reputational Key. Permission to
+decrypt the key is granted to the job Pod through a Kubernetes service account
+specifically used for this, and only this, function. Finally, for convenience
+sake, this same secret contains an ASCII-armored export of the *public*
+components of the Intermediate CI Keys and the Reputational Key. This allows the
+signing script to verify that packages were built by the pipeline (both on AWS
+or at affiliated institutions), or signed previously as a part of a different
+pipeline. This is is done *before* importing decrypting and importing the
+Signing Reputational Private Key material and officially signing the packages.
+
+Procedurally the ``spack-singing-key-encrypted`` secret is used in the
+following way:
+
+1.  The ``spack-package-signing-gitlab-runner`` protected runner picks
+    up a job tagged ``notary`` from a protected GitLab branch (See
+    `Protected Runners and Reserved Tags <#_8bawjmgykv0b>`__).
+2.  Based on its configuration, the runner creates a job pod in the
+    pipeline namespace. The job is run in a stripped down purpose-built
+    image ``ghcr.io/spack/notary:latest`` Docker image. The runner is
+    configured to only allow running jobs with this image.
+3.  The runner also mounts the ``spack-signing-key-encrypted`` secret to
+    a path on disk. Note that this becomes several files on disk, the
+    public components of the Intermediate CI Keys, the public components
+    of the Reputational CI, and an AWS KMS encrypted file containing the
+    Singing Reputational Private Key.
+4.  In addition to the secret, the runner creates a tmpfs memory mounted
+    directory where the GnuPG keyring will be created to verify, and
+    then resign the package specs.
+5.  The job script syncs all spec.json.sig files from the build cache to
+    a working directory in the job’s execution environment.
+6.  The job script then runs the ``sign.sh`` script built into the
+    notary Docker image.
+7.  The ``sign.sh`` script imports the public components of the
+    Reputational and Intermediate CI Keys and uses them to verify good
+    signatures on the spec.json.sig files. If any signed spec does not
+    verify the job immediately fails.
+8.  Assuming all specs are verified, the ``sign.sh`` script then unpacks
+    the spec json data from the signed file in preparation for being
+    re-signed with the Reputational Key.
+9.  The private components of the Reputational Key are decrypted to
+    standard out using ``aws-encryption-cli`` directly into a ``gpg
+    –import …`` statement which imports the key into the
+    keyring mounted in-memory.
+10. The private key is then used to sign each of the json specs and the
+    keyring is removed from disk.
+11. The re-signed json specs are resynced to the AWS S3 Mirror and the
+    public signing of the packages for the develop or release pipeline
+    that created them is complete.
+
+Non service-account access to the private components of the Reputational
+Key that are managed through access to the symmetric secret in KMS used
+to encrypt the data key (which in turn is used to encrypt the GnuPG key
+- See:\ `Encryption SDK
+Documentation <https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/crypto-cli-examples.html#cli-example-encrypt-file>`__).
+A small trusted subset of the core development team are the only
+individuals with access to this symmetric key.
+
+.. _protected_runners:
+
+Protected Runners and Reserved Tags
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Spack has a large number of Gitlab Runners operating in its build farm.
+These include runners deployed in the AWS Kubernetes cluster as well as
+runners deployed at affiliated institutions. The majority of runners are
+shared runners that operate across projects in gitlab.spack.io. These
+runners pick up jobs primarily from the spack/spack project and execute
+them in PR pipelines.
+
+A small number of runners operating on AWS and at affiliated institutions are
+registered as specific *protected* runners on the spack/spack project. In
+addition to protected runners there are protected branches on the spack/spack
+project. These are the ``develop`` branch, any release branch (i.e. managed with
+the ``releases/v*`` wildcard) and any tag branch (managed with the ``v*``
+wildcard) Finally Spack’s pipeline generation code reserves certain tags to make
+sure jobs are routed to the correct runners, these tags are ``public``,
+``protected``, and ``notary``. Understanding how all this works together to
+protect secrets and provide integrity assurances can be a little confusing so
+lets break these down:
+
+-  **Protected Branches**- Protected branches in Spack prevent anyone
+   other than Maintainers in GitLab from pushing code. In the case of
+   Spack the only Maintainer level entity pushing code to protected
+   branches is Spack bot. Protecting branches also marks them in such a
+   way that Protected Runners will only run jobs from those branches
+- **Protected Runners**- Protected Runners only run jobs from protected
+   branches. Because protected runners have access to secrets, it's critical
+   that they not run Jobs from untrusted code (i.e. PR branches). If they did it
+   would be possible for a PR branch to tag a job in such a way that a protected
+   runner executed that job and mounted secrets into a code execution
+   environment that had not been reviewed by Spack maintainers. Note however
+   that in the absence of tagging used to route jobs, public runners *could* run
+   jobs from protected branches. No secrets would be at risk of being breached
+   because non-protected runners do not have access to those secrets; lack of
+   secrets would, however, cause the jobs to fail.
+- **Reserved Tags**- To mitigate the issue of public runners picking up
+   protected jobs Spack uses a small set of “reserved” job tags (Note that these
+   are *job* tags not git tags). These tags are “public”, “private”, and
+   “notary.” The majority of jobs executed in Spack’s GitLab instance are
+   executed via a ``generate`` job. The generate job code systematically ensures
+   that no user defined configuration sets these tags. Instead, the ``generate``
+   job sets these tags based on rules related to the branch where this pipeline
+   originated. If the job is a part of a pipeline on a PR branch it sets the
+   ``public`` tag. If the job is part of a pipeline on a protected branch it
+   sets the ``protected`` tag. Finally if the job is the package signing job and
+   it is running on a pipeline that is part of a protected branch then it sets
+   the ``notary`` tag.
+
+Protected Runners are configured to only run jobs from protected branches. Only
+jobs running in pipelines on protected branches are tagged with ``protected`` or
+``notary`` tags. This tightly couples jobs on protected branches to protected
+runners that provide access to the secrets required to sign the built packages.
+The secrets are can **only** be accessed via:
+
+1. Runners under direct control of the core development team.
+2. Runners under direct control of trusted maintainers at affiliated institutions.
+3. By code running the automated pipeline that has been reviewed by the
+   Spack maintainers and judged to be appropriate.
+
+Other attempts (either through malicious intent or incompetence) can at
+worst grab jobs intended for protected runners which will cause those
+jobs to fail alerting both Spack maintainers and the core development
+team.
+
+.. [#f1]
+   The Reputational Key has also cross signed core development team
+   keys.