summaryrefslogtreecommitdiff
path: root/share
AgeCommit message (Collapse)AuthorFilesLines
2022-10-15CI: allow multiple matches to combine tags (#32290)Jonathon Anderson17-0/+17
Currently "spack ci generate" chooses the first matching entry in gitlab-ci:mappings to fill attributes for a generated build-job, requiring that the entire configuration matrix is listed out explicitly. This unfortunately causes significant problems in environments with large configuration spaces, for example the environment in #31598 (spack.yaml) supports 5 operating systems, 3 architectures and 130 packages with explicit size requirements, resulting in 1300 lines of configuration YAML. This patch adds a configuraiton option to the gitlab-ci schema called "match_behavior"; when it is set to "merge", all matching entries are applied in order to the final build-job, allowing a few entries to cover an entire matrix of configurations. The default for "match_behavior" is "first", which behaves as before this commit (only the runner attributes of the first match are used). In addition, match entries may now include a "remove-attributes" configuration, which allows matches to remove tags that have been aggregated by prior matches. This only makes sense to use with "match_behavior:merge". You can combine "runner-attributes" with "remove-attributes" to effectively override prior tags.
2022-10-12env depfile: allow deps only install (#33245)Harmen Stoppels1-0/+37
* env depfile: allow deps only install - Refactor `spack env depfile` to use a Jinja template, making it a bit easier to follow as a human being. - Add a layer of indirection in the generated Makefile through an `<prefix>/.install-deps/<hash>` target, which allows one to specify different options when installing dependencies. For example, only verbose/debug mode on when installing some particular spec: ``` $ spack -e my_env env depfile -o Makefile --make-target-prefix example $ make example/.install-deps/<hash> -j16 $ make example/.install/<hash> SPACK="spack -d" SPACK_INSTALL_FLAGS=--verbose -j16 ``` This could be used to speed up `spack ci rebuild`: - Parallel install of dependencies from buildcache - Better readability of logs, e.g. reducing verbosity when installing dependencies, and splitting logs into deps.log and current_spec.log * Silence please!
2022-10-11Refactor a few classes related to package repositories (#32273)Massimiliano Culpo2-15/+19
Caches used by repositories don't reference the global spack.repo.path instance anymore, but get the repository they refer to during initialization. Spec.virtual now use the index, and computation done to compute the index use Repository.is_virtual_safe. Code to construct mock packages and mock repository has been factored into a unique MockRepositoryBuilder that is used throughout the codebase. Add debug print for pushing and popping config scopes. Changed spack.repo.use_repositories so that it can override or not previous repos spack.repo.use_repositories updates spack.config.config according to the modifications done Removed a peculiar behavior from spack.config.Configuration where push would always bubble-up a scope named command_line if it existed
2022-10-11e4s ci: add cabana +rocm (#33177)eugeneswalker1-0/+1
2022-10-10e4s: add hypre +rocm (#32148)eugeneswalker1-0/+1
2022-10-10ML CPU pipeline: test py-torch-nvidia-apex (#33158)Adam J. Stewart1-2/+1
2022-10-09Add CI stack for ML packages (#31592)Adam J. Stewart4-0/+553
Basic stack of ML packages we would like to test and generate binaries for in CI. Spack now has a large CI framework in GitLab for PR testing and public binary generation. We should take advantage of this to test and distribute optimized binaries for popular ML frameworks. This is a pretty extensive initial set, including CPU, ROCm, and CUDA versions of a core `x96_64_v4` stack. ### Core ML frameworks These are all popular core ML frameworks already available in Spack. - [x] PyTorch - [x] TensorFlow - [x] Scikit-learn - [x] MXNet - [x] CNTK - [x] Caffe - [x] Chainer - [x] XGBoost - [x] Theano ### ML extensions These are domain libraries and wrappers that build on top of core ML libraries - [x] Keras - [x] TensorBoard - [x] torchvision - [x] torchtext - [x] torchaudio - [x] TorchGeo - [x] PyTorch Lightning - [x] torchmetrics - [x] GPyTorch - [x] Horovod ### ML-adjacent libraries These are libraries that aren't specific to ML but are still core libraries used in ML pipelines - [x] numpy - [x] scipy - [x] pandas - [x] ONNX - [x] bazel Co-authored-by: Jonathon Anderson <17242663+blue42u@users.noreply.github.com>
2022-10-07remove outdated comments (#33123)eugeneswalker1-8/+6
2022-10-07e4s ci: add variorum (#33113)eugeneswalker1-1/+1
2022-10-07e4s ci: add h5bench (#33114)eugeneswalker1-1/+1
2022-10-07e4s ci: add quantum-espresso (#33075)eugeneswalker1-0/+1
2022-10-06tcl module template: automatically unload automatically loaded modules (#32853)Sergey Kosukhin1-1/+1
Remove `module-info mode load` condition that prevents auto-unloading when autoloading is enabled. It looks like this condition was added to work around an issue in environment-modules that is no longer necessary. Add quotes to make is-loaded happy
2022-10-06e4s ci: add charliecloud (#32990)eugeneswalker1-0/+2
2022-10-05e4s ci: add gotcha cpu (#32989)eugeneswalker1-1/+1
2022-10-05e4s ci: add caliper +cuda (#32992)eugeneswalker1-1/+1
2022-10-05e4s ci: enable caliper cpu (#32988)eugeneswalker1-1/+1
2022-10-05e4s: add omega-h +cuda (#32156)eugeneswalker1-0/+1
2022-10-05e4s ci: add unifyfs (#32991)eugeneswalker1-1/+1
2022-10-04e4s: add vtk-m +rocm (#32151)eugeneswalker1-0/+1
2022-09-29Install: Add use-buildcache option to install (#32537)kwryankrattiger1-1/+1
Install: Add use-buildcache option to install * Allow differentiating between top level packages and dependencies when determining whether to install from the cache or not. * Add unit test for --use-buildcache * Use metavar to display use-buildcache options. * Update spack-completion
2022-09-27e4s: use ubuntu 20.04 image and %gcc@9.4.0 (#32795)eugeneswalker2-6/+6
2022-09-26Remove "tut" from the "build-systems" pipelineMassimiliano Culpo1-1/+0
PR #32615 deprecated Python versions up to 3.6.X. Since the "build-systems" pipeline requires Python 3.6.15 to build "tut", it will fail on the first rebuild that involves Python. The "tut" package is meant to perform an end-to-end test of the "Waf" build-system, which is scarcely used. The fix therefore is just to remove it from the pipeline.
2022-09-23gitlab: cuda not compatible amazon linux 2 (#32678)Harmen Stoppels2-17/+0
amazon linux 2 ships a glibc that is too old to work with cuda toolkit for aarch64. For example: `libcurand.so.10.2.10.50` requires the symbol `logf@@GLIBC_2.27`, but glibc is at 2.26. So, these specs are removed.
2022-09-14Revert "e4s ci: restore power builds (#32296)" (#32633)Zack Galbreath3-463/+290
This reverts commit 0d18c32bca8398e2daa736a940154caa235cbacc.
2022-09-12e4s: add py-torch +cuda (#32601)eugeneswalker1-0/+1
2022-09-10ci: restore coverage computation (#32585)Massimiliano Culpo1-5/+17
* ci: restore coverage computation * Mark "test_foreground_background" as xfail * Mark "test_foreground_background_output" as xfail * Make number of processes explicit, remove verbosity on linux * Run coverage on just 3 Python jobs for linux * Run coverage on just 3 Python jobs for linux * Run coverage on just 2 Python jobs for linux * Add back verbose, since before we didn't encounter the xdist internal error * Reduce the workers to 2 * Try to use command line
2022-09-08e4s ci: restore power builds (#32296)eugeneswalker3-290/+463
2022-09-08ci: avoid running coverage on package only PRs (#32573)Massimiliano Culpo1-4/+1
* ci: remove !docs from "core" filters Written like it is now it causes package only PRs to run with coverage. * Try to skip job under condition, see if the workflow proceed * Try to cancel a running CI job * Simplify linux unit-tests, skip windows unit-tests on package PRs * Reduce the inputs to unit-tests workflow * Move control logic to main workflow, remove inputs * Revert "Move control logic to main workflow, remove inputs" This reverts commit 0c46fece4c49eb7a37585ec3ba651a31d7f958af. * Do not compute "with_coverage" since it's always == to "core" * Remove workflow dispatch from unit tests * Revert "Revert "Move control logic to main workflow, remove inputs"" This reverts commit dd4e4a4e61a825901e736348fd044d37e88c90b5. * Try to skip all from the main workflow * Add back bootstrap to needed checks for "all" * Restore the correct logic for conditionals
2022-09-07Make GHA tests parallel by using xdist (#32361)Tom Scogland2-1/+6
* Add two no-op jobs named "all-prechecks" and "all" These are a suggestion from @tgamblin, they are stable named markers we can use from gitlab and possibly for required checks to make CI more resilient to refactors changing the names of specific checks. * Enable parallel testing using xdist for unit testing in CI * Normalize tmp paths to deal with macos * add -u flag compatibility to spack python As of now, it is accepted and ignored. The usage with xdist, where it is invoked specifically by `python -u spack python` which is then passed `-u` by xdist is the entire reason for doing this. It should never be used without explicitly passing -u to the executing python interpreter. * use spack python in xdist to support python 2 When running on python2, spack has many import cycles unless started through main. To allow that, this uses `spack python` as the interpreter, leveraging the `-u` support so xdist doesn't error out when it unconditionally requests unbuffered binary IO. * Use shutil.move to account for tmpdir being in a separate filesystem sometimes
2022-09-02Update bootstrap buildcache to v0.3 (#32262)Harmen Stoppels4-0/+44
This release allow to bootstrap patchelf from binaries.
2022-09-01Fix cause of checksum failures in public binary mirror (#32407)Scott Wittenburg2-4/+6
Move the copying of the buildcache to a root job that runs after all the child pipelines have finished, so that the operation can be coordinated across all child pipelines to remove the possibility of race conditions during potentially simlutandous copies. This lets us ensure the .spec.json.sig and .spack files for any spec in the root mirror always come from the same child pipeline mirror (though which pipeline is arbitrary). It also allows us to avoid copying of duplicates, which we now do.
2022-09-01New command, `spack change`, to change existing env specs (#31995)Peter Scheibel1-1/+10
If you have an environment like ``` $ cat spack.yaml spack: specs: [openmpi@4.1.0+cuda] ``` this PR provides a new command `spack change` that you can use to adjust environment specs from the command line: ``` $ spack change openmpi~cuda $ cat spack.yaml spack: specs: [openmpi@4.1.0~cuda] ``` in other words, this allows you to tweak the details of environment specs from the command line. Notes: * This is only allowed for environments that do not define matrices * This is possible but not anticipated to be needed immediately * If this were done, it should probably only be done for "named"/not-anonymous specs (i.e. we can change `openmpi+cuda` but not spec like `+cuda` or `@4.0.1~cuda`)
2022-09-01Fix CI for package only PRs (#32473)Massimiliano Culpo1-1/+1
2022-08-24oneapi e4s: use require: to force gcc build for some packages (#32309)Robert Cohn1-15/+24
2022-08-23spack ci: add support for running stand-alone tests (#27877)Tamara Dahlgren4-3/+53
This support requires adding the '--tests' option to 'spack ci rebuild'. Packages whose stand-alone tests are broken (in the CI environment) can be configured in gitlab-ci to be skipped by adding them to broken-tests-packages. Highlights include: - Restructured 'spack ci' help to provide better subcommand summaries; - Ensured only one InstallError (i.e., installer's) rather than allowing build_environment to have its own; and - Refactored CI and CDash reporting to keep CDash-related properties and behavior in a separate class. This allows stand-alone tests from `spack ci` to run when the `--tests` option is used. With `--tests`, stand-alone tests are run **after** a **successful** (re)build of the package. Test results are collected and report(able) using CDash. This PR adds the following features: - Adds `-t` and `--tests` to `spack ci rebuild` to run stand-alone tests; - Adds `--fail-fast` to stop stand-alone tests after the first failure; - Ensures a *single* `InstallError` across packages (i.e., removes second class from build environment); - Captures skipping tests for externals and uninstalled packages (for CDash reporting); - Copies test logs and outputs to the CI artifacts directory to facilitate debugging; - Parses stand-alone test results to report outputs from each `run_test` as separate test parts (CDash reporting); - Logs a test completion message to allow capture of timing of the last `run_test` part; - Adds the runner description to the CDash site to better distinguish entries in CDash tables; - Adds `gitlab-ci` `broken-tests-packages` to CI configuration to skip stand-alone testing for packages with known issues; - Changes `spack ci --help` so description of each subcommand is a single line; - Changes `spack ci <subcommand> --help` to provide the full description of each command (versus no description); and - Ensures `junit` test log file ends in an `.xml` extension (versus default where it does not). Tasks: - [x] Include the equivalent of the architecture information, or at least the host target, in the CDash output - [x] Upload stand-alone test results files as `test` artifacts - [x] Confirm tests are run in GitLab - [x] Ensure CDash results are uploaded as artifacts - [x] Resolve issues with CDash build-and test results appearing on same row of the table - [x] Add unit tests as needed - [x] Investigate why some (dependency) packages don't have test results (e.g., related from other pipelines) - [x] Ensure proper parsing and reporting of skipped tests (as `not run`) .. post- #28701 merge - [x] Restore the proper CDash URLand or mirror ONCE out-of-band testing completed
2022-08-22enable packages that use numpy for oneapi e4s stack (#32293)Robert Cohn1-3/+3
* enable geopm oneapi * enable packages that depend on py-numpy * disable nan warnings * undo geopm enable * undo geopm, it has multiple compile issues
2022-08-20spack list: add --tag flag (#32016)sparkyniner1-1/+1
* modified list.py and added functionality for --tag * Removed long and very long, shifted rest of code above return statement * removed results variable * added import statement at top * added the line accidentally deleted * added line accidentally deleted * changed p.name to p, added line inside if statement * line order switched * [@spackbot] updating style on behalf of sparkyniner * ran update completion command * add tests * Update lib/spack/spack/test/cmd/list.py Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> * [@spackbot] updating style on behalf of sparkyniner * changed argument to mock_packages and moved code under filter by tag * removed bad rebase code and added additional test * [@spackbot] updating style on behalf of sparkyniner * added line removed earlier * added line removed earlier * replaced function * added more recommended changes Co-authored-by: sairaj <sairaj@sairajs-MacBook-Pro.local> Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2022-08-19e4s %oneapi: add aml +ze (level zero) (#32161)eugeneswalker1-0/+1
2022-08-19e4s oneapi: compiler environment: prepend lib dir to LD_LIBRARY_PATH (#32227)eugeneswalker1-3/+7
2022-08-18e4s: add dealii +cuda (#32159)eugeneswalker1-0/+1
2022-08-18e4s %oneapi: add amrex +sycl (#32162)eugeneswalker1-1/+2
2022-08-18ascent: fix oneapi build (#32239)Robert Cohn1-1/+1
2022-08-18CI: move nightly macOS builds from GA to pipelines (#32231)Massimiliano Culpo1-0/+5
Modifications: - [x] Delete the `macOS builds nightly` workflow - [x] Removed shield in README.md - [x] Moved corresponding specs to `e4s-macos`
2022-08-17e4s mac ci: try lambda, the new mac studio runner (#32169)eugeneswalker2-7/+9
2022-08-17e4s oneapi stack: remove notes for now-fixed builds (#31839)eugeneswalker1-3/+1
2022-08-17CI: reduce the amount of tests run in the original concretizer (#32179)Massimiliano Culpo1-1/+4
* CI: reduce the amount of tests run in the original concretizer * Don't test Python 3.6 on the original concretizer
2022-08-16e4s: add flecsi +cuda (#32158)eugeneswalker1-0/+1
2022-08-16e4s: add ascent +cuda (#32153)eugeneswalker1-0/+1
2022-08-16e4s: add amrex +cuda (#32152)eugeneswalker1-1/+2
2022-08-16e4s: add papi +rocm (#32150)eugeneswalker1-0/+1