pipelines: support testing PRs from forks (#19248)

This change makes improvements to the `spack ci rebuild` command
which supports running gitlab pipelines on PRs from forks.  Much
of this has to do with making sure we can run without the secrets
previously required for running gitlab pipelines (e.g signing key,
aws credentials, etc).  Specific improvements in this PR:

Check if spack has precisely one signing key, and use that information
as an additional constraint on whether or not we should attempt to sign
the binary package we create.

Also, if spack does not have at least one public key, add the install
option "--no-check-signature"

If we are running a pipeline without any profile or environment
variables allowing us to push to S3, the pipeline could still
successfully create a buildcache in the artifacts and move on.  So
just print a message and move on if pushing either the buildcache
entry or cdash id file to the remote mirror fails.

When we attempt to generate a pacakge or gpg key index on an S3
mirror, and there is nothing to index, just print a warning and
exit gracefully rather than throw an exception.

Support the use of PR-specific mirrors for temporary binary pkg
storage.  This will allow quality-of-life improvement for developers,
providing a place to store binaries over the lifetime of a PR, so
that they must only wait for packages to rebuild from source when
they push a new commit that causes it to be necessary.

Replace two-pass install with a single pass and the new option:
 --require-full-hash-match.  Doing this also removes the need to
save a copy of the spack.yaml to be copied over the one spack
rewrites in between the two spack install passes.

Work around a mirror configuration issue caused by using
spack.util.executable to do the package installation.

* Update pipeline trigger jobs for PRs from forks

Moving to PRs from forks relies on external synchronization script
pushing special branch names.  Also secrets will only live on the
spack mirror project, and must be propagated to the E4S project via
variables on the trigger jobs.

When this change is merged, pipelines will not run until we update
the "Custom CI configuration path" in the Gitlab CI Settings, as the
name of the file has changed to better reflect its purpose.

* Arg to MirrorCollection is used exclusively, so add main remote mirror to it

* Compute full hash less frequently

* Add tests covering index generation error handling code

2 files changed, 24 insertions, 18 deletions

diff --git a/share/spack/gitlab/cloud_e4s_pipelines.yml b/share/spack/gitlab/cloud_e4s_pipelines.yml
new file mode 100644
index 0000000000..67a74cf971
--- /dev/null
+++ b/share/spack/gitlab/cloud_e4s_pipelines.yml
@@ -0,0 +1,24 @@
+pr_pipeline:
+  only:
+  - /^github\/pr[\d]+_.*$/
+  variables:
+    SPACK_REF: ${CI_COMMIT_SHA}
+    SPACK_PR_BRANCH: ${CI_COMMIT_REF_NAME}
+    SPACK_IS_PR_PIPELINE: "True"
+    AWS_ACCESS_KEY_ID: ${PR_MIRRORS_AWS_ACCESS_KEY_ID}
+    AWS_SECRET_ACCESS_KEY: ${PR_MIRRORS_AWS_SECRET_ACCESS_KEY}
+  trigger:
+    project: spack/e4s
+    strategy: depend
+
+develop_pipeline:
+  only:
+  - /^github\/develop$/
+  variables:
+    SPACK_REF: ${CI_COMMIT_SHA}
+    AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
+    AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
+    SPACK_SIGNING_KEY: ${SPACK_SIGNING_KEY}
+  trigger:
+    project: spack/e4s
+    strategy: depend
diff --git a/share/spack/gitlab/pr_pipeline.yml b/share/spack/gitlab/pr_pipeline.yml
deleted file mode 100644
index 367a4b9077..0000000000
--- a/share/spack/gitlab/pr_pipeline.yml
+++ /dev/null
@@ -1,18 +0,0 @@
-pr_pipeline:
-  only:
-  - external_pull_requests
-  variables:
-    SPACK_REF: ${CI_EXTERNAL_PULL_REQUEST_SOURCE_BRANCH_NAME}
-    SPACK_IS_PR_PIPELINE: "True"
-  trigger:
-    project: spack/e4s
-    strategy: depend
-
-merge_pipeline:
-  only:
-  - develop
-  variables:
-    SPACK_REF: ${CI_COMMIT_SHA}
-  trigger:
-    project: spack/e4s
-    strategy: depend
\ No newline at end of file