summaryrefslogtreecommitdiff
path: root/share
AgeCommit message (Collapse)AuthorFilesLines
2022-11-23E4S: Conservatively add ecp-data-vis-sdk (#33621)kwryankrattiger1-19/+9
* E4S: Conservatively add ecp-data-vis-sdk * Remove ascent from CUDA SDK stack to stop hanging on Dray * Adios2: Newer FindPython uses Python_EXECUTABLE
2022-11-22cmd/checksum: allow adding new versions to package (#24532)Michael Kuhn1-1/+1
This adds super-lazy maintainer mode to `spack checksum`: Instead of only printing the new checksums to the terminal, `-a` and `--add-to-package` will add the new checksums to the `package.py` file and open it in the editor afterwards for final checks.
2022-11-22Revert "Warn about removal of deprecated format strings (#33829)" (#34056)Massimiliano Culpo1-2/+2
This reverts commit 7f9af8d4a0bfbb1577e5ac9982624d8d0cb9b9ca.
2022-11-22Warn about removal of deprecated format strings (#33829)Greg Becker1-2/+2
Co-authored-by: becker33 <becker33@users.noreply.github.com>
2022-11-19spack find: remove deprecated "--bootstrap" option (#34015)Massimiliano Culpo1-1/+1
2022-11-17Remove deprecated subcommands from "spack bootstrap" (#33964)Massimiliano Culpo1-19/+1
These commands are slated for removal in v0.20
2022-11-15depfile: improve tab completion (#33773)Harmen Stoppels1-6/+17
This PR allows you to do: ``` spack env create -d . spack -e . add python spack -e . concretize spack -e . env depfile -o Makefile make in<tab> # -> install make install-<tab> # -> install-deps/ make install-deps/py<tab> # -> install-deps/python-x.y.z-hash make install/zl<tab> # -> install/zlib-x.y.z-hash make SP<tab> # -> make SPACK make SPACK_<tab> # -> make SPACK_INSTALL_FLAGS= ```
2022-11-15gitlab: report load in generate job (#33888)Harmen Stoppels17-0/+39
2022-11-11remove activate/deactivate support in favor of environments (#29317)Harmen Stoppels1-20/+2
Environments and environment views have taken over the role of `spack activate/deactivate`, and we should deprecate these commands for several reasons: - Global activation is a really poor idea: - Install prefixes should be immutable; since they can have multiple, unrelated dependents; see below - Added complexity elsewhere: verification of installations, tarballs for build caches, creation of environment views of packages with unrelated extensions "globally activated"... by removing the feature, it gets easier for people to contribute, and we'd end up with fewer bugs due to edge cases. - Environment accomplish the same thing for non-global "activation" i.e. `spack view`, but better. Also we write in the docs: ``` However, Spack global activations have two potential drawbacks: #. Activated packages that involve compiled C extensions may still need their dependencies to be loaded manually. For example, ``spack load openblas`` might be required to make ``py-numpy`` work. #. Global activations "break" a core feature of Spack, which is that multiple versions of a package can co-exist side-by-side. For example, suppose you wish to run a Python package in two different environments but the same basic Python --- one with ``py-numpy@1.7`` and one with ``py-numpy@1.8``. Spack extensions will not support this potential debugging use case. ``` Now that environments are established and views can take over the role of activation non-destructively, we can remove global activation/deactivation.
2022-11-08More jobs in Gitlab CI (#33688)Harmen Stoppels16-0/+31
Use at most 32 jobs when available.
2022-11-08"spack uninstall": don't modify env (#33711)Peter Scheibel1-2/+2
"spack install foo" no longer adds package "foo" to the environment (i.e. to the list of root specs) by default: you must specify "--add". Likewise "spack uninstall foo" no longer removes package "foo" from the environment: you must specify --remove. Generally this means that install/uninstall commands will no longer modify the users list of root specs (which many users found problematic: they had to deactivate an environment if they wanted to uninstall a spec without changing their spack.yaml description). In more detail: if you have environments e1 and e2, and specs [P, Q, R] such that P depends on R, Q depends on R, [P, R] are in e1, and [Q, R] are in e2: * `spack uninstall --dependents --remove r` in e1: removes R from e1 (but does not uninstall it) and uninstalls (and removes) P * `spack uninstall -f --dependents r` in e1: will uninstall P, Q, and R (i.e. e2 will have dependent specs uninstalled as a side effect) * `spack uninstall -f --dependents --remove r` in e1: this uninstalls P, Q, and R, and removes [P, R] from e1 * `spack uninstall -f --remove r` in e1: uninstalls R (so it is "missing" in both environments) and removes R from e1 (note that e1 would still install R as a dependency of P, but it would no longer be listed as a root spec) * `spack uninstall --dependents r` in e1: will fail because e2 needs R Individual unit tests were created for each of these scenarios.
2022-11-05py-transformers: add v4.24.0 (#33716)Adam J. Stewart3-0/+9
* py-transformers: add v4.24.0 * Internet access still required
2022-11-03Update metadata for bootstrapping (#33665)Massimiliano Culpo11-944/+1095
2022-11-01Let pytest-cov create the xml directly (#33619)Massimiliano Culpo1-1/+1
`coverage` sometimes failed to combine, even if there were multiple reports.
2022-10-31Revert "gitlab: when_possible -> false (#33443)" (#33552)Massimiliano Culpo2-2/+2
This reverts commit b1559cc831620ee2b2cf8e57fdecc5bb3bf8edfd.
2022-10-29e4s ci: add mfem +rocm (#31604)eugeneswalker1-0/+1
2022-10-29Deprecate spack bootstrap trust/untrust (#33600)Massimiliano Culpo1-2/+12
* Deprecate spack bootstrap trust/untrust * Update CI * Update tests
2022-10-28gitlab ci: patched make 4.3.0 (#33583)Harmen Stoppels16-31/+33
2022-10-27Don't fail over cpuinfo (#33546)Harmen Stoppels16-32/+32
2022-10-26gitlab ci: show build machine info (#33523)Harmen Stoppels17-0/+49
2022-10-26Allow for packages with multiple build-systems (#30738)Massimiliano Culpo6-0/+12
This commit extends the DSL that can be used in packages to allow declaring that a package uses different build-systems under different conditions. It requires each spec to have a `build_system` single valued variant. The variant can be used in many context to query, manipulate or select the build system associated with a concrete spec. The knowledge to build a package has been moved out of the PackageBase hierarchy, into a new Builder hierarchy. Customization of the default behavior for a given builder can be obtained by coding a new derived builder in package.py. The "run_after" and "run_before" decorators are now applied to methods on the builder. They can also incorporate a "when=" argument to specify that a method is run only when certain conditions apply. For packages that do not define their own builder, forwarding logic is added between the builder and package (methods not found in one will be retrieved from the other); this PR is expected to be fully backwards compatible with unmodified packages that use a single build system.
2022-10-26gitlab ci: install binary deps faster (#33248)Harmen Stoppels16-0/+46
* Fast Gitlab CI job setup, and better legibility * Use a non-broken, recent GNU Make
2022-10-25depfile: resurrect lost touch (#33504)Harmen Stoppels1-0/+3
2022-10-25feature: Add -x|--explicit option to 'spack test run' (#32910)Tamara Dahlgren1-1/+1
2022-10-25Update Spack Dockerfiles (#33500)Massimiliano Culpo2-2/+1
* Use spack bootstrap now in containers * Fix wrong path glob expression
2022-10-25Remove recursive symbolic link in lib/spack/docs from git repository (#33483)Massimiliano Culpo2-10/+0
Delete code removing the symlink during CI
2022-10-25Improve legibility of Gitlab CI (#33482)Harmen Stoppels17-17/+17
Use --backtrace in ci instead of --debug to reduce verbosity and don't show log on error, since log is already printed
2022-10-24e4s ci: use an appropriate name for cdash build group name (#33494)eugeneswalker1-1/+1
2022-10-24e4s ci: add chai +rocm (#32506)eugeneswalker1-1/+3
2022-10-23backtraces with --backtrace (#33478)Harmen Stoppels1-1/+1
* backtraces without --debug Currently `--debug` is too verbose and not-`--debug` gives to little context about where exceptions are coming from. So, instead, it'd be nice to have `spack --backtrace` and `SPACK_BACKTRACE=1` as methods to get something inbetween: no verbose debug messages, but always a full backtrace. This is useful for CI, where we don't want to drown in debug messages when installing deps, but we do want to get details where something goes wrong if it goes wrong. * completion
2022-10-22Remove x86_64_v4 target from AHUG and ISC stacks (#33464)Zack Galbreath2-3/+0
2022-10-22stop building binaries for the 1% (#33463)Harmen Stoppels3-3/+3
2022-10-21gitlab: Retry protected publish jobs in certain cases (#32496)Scott Wittenburg1-0/+3
When we lose a running pod (possibly loss of spot instance) or encounter some other infrastructure-related failure of this job, we need to retry it. This retries the job the maximum number of times in those cases.
2022-10-21gitlab: when_possible -> false (#33443)Harmen Stoppels2-2/+2
`reuse` and `when_possible` concretization broke the invariant that `spec[pkg_name]` has unique keys. This invariant is relied on in tons of places, such as when setting up the build environment. When using `when_possible` concretization, one may end up with two or more `perl`s or `python`s among the transitive deps of a spec, because concretization does not consider build-only deps of reusable specs. Until the code base is fixed not to rely on this broken property of `__getitem__`, we should disable reuse in CI.
2022-10-21Remove "spack buildcache copy" in v0.19.0 (#33437)Massimiliano Culpo1-5/+1
2022-10-20e4s ci: add raja +rocm (#32505)eugeneswalker1-1/+1
2022-10-19e4s ci: add umpire +rocm (#32504)eugeneswalker1-1/+1
2022-10-19depfile: buildcache support (#33315)Harmen Stoppels2-12/+13
When installing some/all specs from a buildcache, build edges are pruned from those specs. This can result in a much smaller effective DAG. Until now, `spack env depfile` would always generate a full DAG. Ths PR adds the `spack env depfile --use-buildcache` flag that was introduced for `spack install` before. This way, not only can we drop build edges, but also we can automatically set the right buildcache related flags on the specific specs that are gonna get installed. This way we get parallel installs of binary deps without redundancy, which is useful for Gitlab CI.
2022-10-19Add a command to bootstrap Spack right now (#33407)Massimiliano Culpo1-1/+5
2022-10-17e4s ci stack: add trilinos +rocm (#31601)eugeneswalker1-0/+5
2022-10-15CI: allow multiple matches to combine tags (#32290)Jonathon Anderson17-0/+17
Currently "spack ci generate" chooses the first matching entry in gitlab-ci:mappings to fill attributes for a generated build-job, requiring that the entire configuration matrix is listed out explicitly. This unfortunately causes significant problems in environments with large configuration spaces, for example the environment in #31598 (spack.yaml) supports 5 operating systems, 3 architectures and 130 packages with explicit size requirements, resulting in 1300 lines of configuration YAML. This patch adds a configuraiton option to the gitlab-ci schema called "match_behavior"; when it is set to "merge", all matching entries are applied in order to the final build-job, allowing a few entries to cover an entire matrix of configurations. The default for "match_behavior" is "first", which behaves as before this commit (only the runner attributes of the first match are used). In addition, match entries may now include a "remove-attributes" configuration, which allows matches to remove tags that have been aggregated by prior matches. This only makes sense to use with "match_behavior:merge". You can combine "runner-attributes" with "remove-attributes" to effectively override prior tags.
2022-10-12env depfile: allow deps only install (#33245)Harmen Stoppels1-0/+37
* env depfile: allow deps only install - Refactor `spack env depfile` to use a Jinja template, making it a bit easier to follow as a human being. - Add a layer of indirection in the generated Makefile through an `<prefix>/.install-deps/<hash>` target, which allows one to specify different options when installing dependencies. For example, only verbose/debug mode on when installing some particular spec: ``` $ spack -e my_env env depfile -o Makefile --make-target-prefix example $ make example/.install-deps/<hash> -j16 $ make example/.install/<hash> SPACK="spack -d" SPACK_INSTALL_FLAGS=--verbose -j16 ``` This could be used to speed up `spack ci rebuild`: - Parallel install of dependencies from buildcache - Better readability of logs, e.g. reducing verbosity when installing dependencies, and splitting logs into deps.log and current_spec.log * Silence please!
2022-10-11Refactor a few classes related to package repositories (#32273)Massimiliano Culpo2-15/+19
Caches used by repositories don't reference the global spack.repo.path instance anymore, but get the repository they refer to during initialization. Spec.virtual now use the index, and computation done to compute the index use Repository.is_virtual_safe. Code to construct mock packages and mock repository has been factored into a unique MockRepositoryBuilder that is used throughout the codebase. Add debug print for pushing and popping config scopes. Changed spack.repo.use_repositories so that it can override or not previous repos spack.repo.use_repositories updates spack.config.config according to the modifications done Removed a peculiar behavior from spack.config.Configuration where push would always bubble-up a scope named command_line if it existed
2022-10-11e4s ci: add cabana +rocm (#33177)eugeneswalker1-0/+1
2022-10-10e4s: add hypre +rocm (#32148)eugeneswalker1-0/+1
2022-10-10ML CPU pipeline: test py-torch-nvidia-apex (#33158)Adam J. Stewart1-2/+1
2022-10-09Add CI stack for ML packages (#31592)Adam J. Stewart4-0/+553
Basic stack of ML packages we would like to test and generate binaries for in CI. Spack now has a large CI framework in GitLab for PR testing and public binary generation. We should take advantage of this to test and distribute optimized binaries for popular ML frameworks. This is a pretty extensive initial set, including CPU, ROCm, and CUDA versions of a core `x96_64_v4` stack. ### Core ML frameworks These are all popular core ML frameworks already available in Spack. - [x] PyTorch - [x] TensorFlow - [x] Scikit-learn - [x] MXNet - [x] CNTK - [x] Caffe - [x] Chainer - [x] XGBoost - [x] Theano ### ML extensions These are domain libraries and wrappers that build on top of core ML libraries - [x] Keras - [x] TensorBoard - [x] torchvision - [x] torchtext - [x] torchaudio - [x] TorchGeo - [x] PyTorch Lightning - [x] torchmetrics - [x] GPyTorch - [x] Horovod ### ML-adjacent libraries These are libraries that aren't specific to ML but are still core libraries used in ML pipelines - [x] numpy - [x] scipy - [x] pandas - [x] ONNX - [x] bazel Co-authored-by: Jonathon Anderson <17242663+blue42u@users.noreply.github.com>
2022-10-07remove outdated comments (#33123)eugeneswalker1-8/+6
2022-10-07e4s ci: add variorum (#33113)eugeneswalker1-1/+1
2022-10-07e4s ci: add h5bench (#33114)eugeneswalker1-1/+1