From 685e3d7ae9b968aa429c08a642aba4e718e0b419 Mon Sep 17 00:00:00 2001 From: Scott Wittenburg Date: Wed, 30 Mar 2022 18:17:29 -0600 Subject: spack ci: filter untouched pkgs from PR pipelines (#29697) We've previously generated CI pipelines for PRs, and they rebuild any packages that don't have a binary in an existing build cache. The assumption we were making was that ALL prior merged builds would be in cache, but due to the way we do security in the pipeline, they aren't. `develop` pipelines can take a while to catch up with the latest PRs, and while it does that, there may be a bunch of redundant builds on PRs that duplicate things being rebuilt on `develop`. Until we can do better caching of PR builds, we'll have this problem. We can do better in PRs, though, by *only* rebuilding things in the CI environment that are actually touched by the PR. This change computes exactly what packages are changed by a PR branch and *only* includes those packages' dependents and dependencies in the generated pipeline. Other as-yet unbuilt packages are pruned from CI for the PR. For `develop` pipelines, we still want to build everything to ensure that the stack works, and to ensure that `develop` catches up with PRs. This is especially true since we do not do rebuilds for *every* commit on `develop` -- just the most recent one after each `develop` pipeline finishes. Since we skip around, we may end up missing builds unless we ensure that we rebuild everything. We differentiate between `develop` and PR pipelines in `.gitlab-ci.yml` by setting `SPACK_PRUNE_UNTOUCHED` for PRs. `develop` will still have the old behavior. - [x] Add `SPACK_PRUNE_UNTOUCHED` variable to `spack ci` - [x] Refactor `spack pkg` command by moving historical package checking logic to `spack.repo` - [x] Implement pruning logic in `spack ci` to remove untouched packages - [x] add tests --- share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml | 1 + 1 file changed, 1 insertion(+) (limited to 'share') diff --git a/share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml b/share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml index 8ec7060120..bd80a52921 100644 --- a/share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml +++ b/share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml @@ -13,6 +13,7 @@ default: variables: SPACK_PR_BRANCH: ${CI_COMMIT_REF_NAME} SPACK_PIPELINE_TYPE: "spack_pull_request" + SPACK_PRUNE_UNTOUCHED: "True" .develop: only: -- cgit v1.2.3-70-g09d2