summaryrefslogtreecommitdiff
path: root/share
AgeCommit message (Collapse)AuthorFilesLines
2022-10-28gitlab ci: patched make 4.3.0 (#33583)Harmen Stoppels16-31/+33
2022-10-27Don't fail over cpuinfo (#33546)Harmen Stoppels16-32/+32
2022-10-26gitlab ci: show build machine info (#33523)Harmen Stoppels17-0/+49
2022-10-26Allow for packages with multiple build-systems (#30738)Massimiliano Culpo6-0/+12
This commit extends the DSL that can be used in packages to allow declaring that a package uses different build-systems under different conditions. It requires each spec to have a `build_system` single valued variant. The variant can be used in many context to query, manipulate or select the build system associated with a concrete spec. The knowledge to build a package has been moved out of the PackageBase hierarchy, into a new Builder hierarchy. Customization of the default behavior for a given builder can be obtained by coding a new derived builder in package.py. The "run_after" and "run_before" decorators are now applied to methods on the builder. They can also incorporate a "when=" argument to specify that a method is run only when certain conditions apply. For packages that do not define their own builder, forwarding logic is added between the builder and package (methods not found in one will be retrieved from the other); this PR is expected to be fully backwards compatible with unmodified packages that use a single build system.
2022-10-26gitlab ci: install binary deps faster (#33248)Harmen Stoppels16-0/+46
* Fast Gitlab CI job setup, and better legibility * Use a non-broken, recent GNU Make
2022-10-25depfile: resurrect lost touch (#33504)Harmen Stoppels1-0/+3
2022-10-25feature: Add -x|--explicit option to 'spack test run' (#32910)Tamara Dahlgren1-1/+1
2022-10-25Update Spack Dockerfiles (#33500)Massimiliano Culpo2-2/+1
* Use spack bootstrap now in containers * Fix wrong path glob expression
2022-10-25Remove recursive symbolic link in lib/spack/docs from git repository (#33483)Massimiliano Culpo2-10/+0
Delete code removing the symlink during CI
2022-10-25Improve legibility of Gitlab CI (#33482)Harmen Stoppels17-17/+17
Use --backtrace in ci instead of --debug to reduce verbosity and don't show log on error, since log is already printed
2022-10-24e4s ci: use an appropriate name for cdash build group name (#33494)eugeneswalker1-1/+1
2022-10-24e4s ci: add chai +rocm (#32506)eugeneswalker1-1/+3
2022-10-23backtraces with --backtrace (#33478)Harmen Stoppels1-1/+1
* backtraces without --debug Currently `--debug` is too verbose and not-`--debug` gives to little context about where exceptions are coming from. So, instead, it'd be nice to have `spack --backtrace` and `SPACK_BACKTRACE=1` as methods to get something inbetween: no verbose debug messages, but always a full backtrace. This is useful for CI, where we don't want to drown in debug messages when installing deps, but we do want to get details where something goes wrong if it goes wrong. * completion
2022-10-22Remove x86_64_v4 target from AHUG and ISC stacks (#33464)Zack Galbreath2-3/+0
2022-10-22stop building binaries for the 1% (#33463)Harmen Stoppels3-3/+3
2022-10-21gitlab: Retry protected publish jobs in certain cases (#32496)Scott Wittenburg1-0/+3
When we lose a running pod (possibly loss of spot instance) or encounter some other infrastructure-related failure of this job, we need to retry it. This retries the job the maximum number of times in those cases.
2022-10-21gitlab: when_possible -> false (#33443)Harmen Stoppels2-2/+2
`reuse` and `when_possible` concretization broke the invariant that `spec[pkg_name]` has unique keys. This invariant is relied on in tons of places, such as when setting up the build environment. When using `when_possible` concretization, one may end up with two or more `perl`s or `python`s among the transitive deps of a spec, because concretization does not consider build-only deps of reusable specs. Until the code base is fixed not to rely on this broken property of `__getitem__`, we should disable reuse in CI.
2022-10-21Remove "spack buildcache copy" in v0.19.0 (#33437)Massimiliano Culpo1-5/+1
2022-10-20e4s ci: add raja +rocm (#32505)eugeneswalker1-1/+1
2022-10-19e4s ci: add umpire +rocm (#32504)eugeneswalker1-1/+1
2022-10-19depfile: buildcache support (#33315)Harmen Stoppels2-12/+13
When installing some/all specs from a buildcache, build edges are pruned from those specs. This can result in a much smaller effective DAG. Until now, `spack env depfile` would always generate a full DAG. Ths PR adds the `spack env depfile --use-buildcache` flag that was introduced for `spack install` before. This way, not only can we drop build edges, but also we can automatically set the right buildcache related flags on the specific specs that are gonna get installed. This way we get parallel installs of binary deps without redundancy, which is useful for Gitlab CI.
2022-10-19Add a command to bootstrap Spack right now (#33407)Massimiliano Culpo1-1/+5
2022-10-17e4s ci stack: add trilinos +rocm (#31601)eugeneswalker1-0/+5
2022-10-15CI: allow multiple matches to combine tags (#32290)Jonathon Anderson17-0/+17
Currently "spack ci generate" chooses the first matching entry in gitlab-ci:mappings to fill attributes for a generated build-job, requiring that the entire configuration matrix is listed out explicitly. This unfortunately causes significant problems in environments with large configuration spaces, for example the environment in #31598 (spack.yaml) supports 5 operating systems, 3 architectures and 130 packages with explicit size requirements, resulting in 1300 lines of configuration YAML. This patch adds a configuraiton option to the gitlab-ci schema called "match_behavior"; when it is set to "merge", all matching entries are applied in order to the final build-job, allowing a few entries to cover an entire matrix of configurations. The default for "match_behavior" is "first", which behaves as before this commit (only the runner attributes of the first match are used). In addition, match entries may now include a "remove-attributes" configuration, which allows matches to remove tags that have been aggregated by prior matches. This only makes sense to use with "match_behavior:merge". You can combine "runner-attributes" with "remove-attributes" to effectively override prior tags.
2022-10-12env depfile: allow deps only install (#33245)Harmen Stoppels1-0/+37
* env depfile: allow deps only install - Refactor `spack env depfile` to use a Jinja template, making it a bit easier to follow as a human being. - Add a layer of indirection in the generated Makefile through an `<prefix>/.install-deps/<hash>` target, which allows one to specify different options when installing dependencies. For example, only verbose/debug mode on when installing some particular spec: ``` $ spack -e my_env env depfile -o Makefile --make-target-prefix example $ make example/.install-deps/<hash> -j16 $ make example/.install/<hash> SPACK="spack -d" SPACK_INSTALL_FLAGS=--verbose -j16 ``` This could be used to speed up `spack ci rebuild`: - Parallel install of dependencies from buildcache - Better readability of logs, e.g. reducing verbosity when installing dependencies, and splitting logs into deps.log and current_spec.log * Silence please!
2022-10-11Refactor a few classes related to package repositories (#32273)Massimiliano Culpo2-15/+19
Caches used by repositories don't reference the global spack.repo.path instance anymore, but get the repository they refer to during initialization. Spec.virtual now use the index, and computation done to compute the index use Repository.is_virtual_safe. Code to construct mock packages and mock repository has been factored into a unique MockRepositoryBuilder that is used throughout the codebase. Add debug print for pushing and popping config scopes. Changed spack.repo.use_repositories so that it can override or not previous repos spack.repo.use_repositories updates spack.config.config according to the modifications done Removed a peculiar behavior from spack.config.Configuration where push would always bubble-up a scope named command_line if it existed
2022-10-11e4s ci: add cabana +rocm (#33177)eugeneswalker1-0/+1
2022-10-10e4s: add hypre +rocm (#32148)eugeneswalker1-0/+1
2022-10-10ML CPU pipeline: test py-torch-nvidia-apex (#33158)Adam J. Stewart1-2/+1
2022-10-09Add CI stack for ML packages (#31592)Adam J. Stewart4-0/+553
Basic stack of ML packages we would like to test and generate binaries for in CI. Spack now has a large CI framework in GitLab for PR testing and public binary generation. We should take advantage of this to test and distribute optimized binaries for popular ML frameworks. This is a pretty extensive initial set, including CPU, ROCm, and CUDA versions of a core `x96_64_v4` stack. ### Core ML frameworks These are all popular core ML frameworks already available in Spack. - [x] PyTorch - [x] TensorFlow - [x] Scikit-learn - [x] MXNet - [x] CNTK - [x] Caffe - [x] Chainer - [x] XGBoost - [x] Theano ### ML extensions These are domain libraries and wrappers that build on top of core ML libraries - [x] Keras - [x] TensorBoard - [x] torchvision - [x] torchtext - [x] torchaudio - [x] TorchGeo - [x] PyTorch Lightning - [x] torchmetrics - [x] GPyTorch - [x] Horovod ### ML-adjacent libraries These are libraries that aren't specific to ML but are still core libraries used in ML pipelines - [x] numpy - [x] scipy - [x] pandas - [x] ONNX - [x] bazel Co-authored-by: Jonathon Anderson <17242663+blue42u@users.noreply.github.com>
2022-10-07remove outdated comments (#33123)eugeneswalker1-8/+6
2022-10-07e4s ci: add variorum (#33113)eugeneswalker1-1/+1
2022-10-07e4s ci: add h5bench (#33114)eugeneswalker1-1/+1
2022-10-07e4s ci: add quantum-espresso (#33075)eugeneswalker1-0/+1
2022-10-06tcl module template: automatically unload automatically loaded modules (#32853)Sergey Kosukhin1-1/+1
Remove `module-info mode load` condition that prevents auto-unloading when autoloading is enabled. It looks like this condition was added to work around an issue in environment-modules that is no longer necessary. Add quotes to make is-loaded happy
2022-10-06e4s ci: add charliecloud (#32990)eugeneswalker1-0/+2
2022-10-05e4s ci: add gotcha cpu (#32989)eugeneswalker1-1/+1
2022-10-05e4s ci: add caliper +cuda (#32992)eugeneswalker1-1/+1
2022-10-05e4s ci: enable caliper cpu (#32988)eugeneswalker1-1/+1
2022-10-05e4s: add omega-h +cuda (#32156)eugeneswalker1-0/+1
2022-10-05e4s ci: add unifyfs (#32991)eugeneswalker1-1/+1
2022-10-04e4s: add vtk-m +rocm (#32151)eugeneswalker1-0/+1
2022-09-29Install: Add use-buildcache option to install (#32537)kwryankrattiger1-1/+1
Install: Add use-buildcache option to install * Allow differentiating between top level packages and dependencies when determining whether to install from the cache or not. * Add unit test for --use-buildcache * Use metavar to display use-buildcache options. * Update spack-completion
2022-09-27e4s: use ubuntu 20.04 image and %gcc@9.4.0 (#32795)eugeneswalker2-6/+6
2022-09-26Remove "tut" from the "build-systems" pipelineMassimiliano Culpo1-1/+0
PR #32615 deprecated Python versions up to 3.6.X. Since the "build-systems" pipeline requires Python 3.6.15 to build "tut", it will fail on the first rebuild that involves Python. The "tut" package is meant to perform an end-to-end test of the "Waf" build-system, which is scarcely used. The fix therefore is just to remove it from the pipeline.
2022-09-23gitlab: cuda not compatible amazon linux 2 (#32678)Harmen Stoppels2-17/+0
amazon linux 2 ships a glibc that is too old to work with cuda toolkit for aarch64. For example: `libcurand.so.10.2.10.50` requires the symbol `logf@@GLIBC_2.27`, but glibc is at 2.26. So, these specs are removed.
2022-09-14Revert "e4s ci: restore power builds (#32296)" (#32633)Zack Galbreath3-463/+290
This reverts commit 0d18c32bca8398e2daa736a940154caa235cbacc.
2022-09-12e4s: add py-torch +cuda (#32601)eugeneswalker1-0/+1
2022-09-10ci: restore coverage computation (#32585)Massimiliano Culpo1-5/+17
* ci: restore coverage computation * Mark "test_foreground_background" as xfail * Mark "test_foreground_background_output" as xfail * Make number of processes explicit, remove verbosity on linux * Run coverage on just 3 Python jobs for linux * Run coverage on just 3 Python jobs for linux * Run coverage on just 2 Python jobs for linux * Add back verbose, since before we didn't encounter the xdist internal error * Reduce the workers to 2 * Try to use command line
2022-09-08e4s ci: restore power builds (#32296)eugeneswalker3-290/+463