summaryrefslogtreecommitdiff
path: root/share
AgeCommit message (Collapse)AuthorFilesLines
2022-12-27spack graph: rework to use Jinja templates and builders (#34637)Massimiliano Culpo2-1/+34
`spack graph` has been reworked to use: - Jinja templates - builder objects to construct the template context when DOT graphs are requested. This allowed to add a new colored output for DOT graphs that highlights both the dependency types and the nodes that are needed at runtime for a given spec.
2022-12-22ML CI: Linux x86_64 (#34299)Adam J. Stewart4-256/+283
* ML CI: Linux x86_64 * Update comments * Rename again * Rename comments * Update to match other arches * No compiler * Compiler was wrong anyway * Faster TF
2022-12-17e4s: disable mac stack due to binary relocation issue#32571 (#34560)eugeneswalker1-103/+103
2022-12-15gitlab ci: more resources for slow builds (#34505)Zack Galbreath8-2/+28
2022-12-13Use file paths/urls correctly (#34452)Harmen Stoppels1-1/+1
The main issue that's fixed is that Spack passes paths (as strings) to functions that require urls. That wasn't an issue on unix, since there you can simply concatenate `file://` and `path` and all is good, but on Windows that gives invalid file urls. Also on Unix, Spack would not deal with uri encoding like x%20y for file paths. It also removes Spack's custom url.parse function, which had its own incorrect interpretation of file urls, taking file://x/y to mean the relative path x/y instead of hostname=x and path=/y. Also it automatically interpolated variables, which is surprising for a function that parses URLs. Instead of all sorts of ad-hoc `if windows: fix_broken_file_url` this PR adds two helper functions around Python's own path2url and reverse. Also fixes a bug where some `spack buildcache` commands used `-d` as a flag to mean `--mirror-url` requiring a URL, and others `--directory`, requiring a path. It is now the latter consistently.
2022-12-09gitlab ci: more resources for paraview and py-torch (#34412)Scott Wittenburg4-1/+6
2022-12-09new command: `spack pkg grep` to search package files (#34388)Todd Gamblin2-2/+12
It's very common for us to tell users to grep through the existing Spack packages to find examples of what they want, and it's also very common for package developers to do it. Now, searching packages is even easier. `spack pkg grep` runs grep on all `package.py` files in repos known to Spack. It has no special options other than the search string; all options passed to it are forwarded along to `grep`. ```console > spack pkg grep --help usage: spack pkg grep [--help] ... positional arguments: grep_args arguments for grep options: --help show this help message and exit ``` ```console > spack pkg grep CMakePackage | head -3 /Users/gamblin2/src/spack/var/spack/repos/builtin/packages/3dtk/package.py:class _3dtk(CMakePackage): /Users/gamblin2/src/spack/var/spack/repos/builtin/packages/abseil-cpp/package.py:class AbseilCpp(CMakePackage): /Users/gamblin2/src/spack/var/spack/repos/builtin/packages/accfft/package.py:class Accfft(CMakePackage, CudaPackage): ``` ```console > spack pkg grep -Eho '(\S*)\(PythonPackage\)' | head -3 AwsParallelcluster(PythonPackage) Awscli(PythonPackage) Bueno(PythonPackage) ``` ## Return Value This retains the return value semantics of `grep`: * 0 for found, * 1 for not found * >1 for error ## Choosing a `grep` You can set the ``SPACK_GREP`` environment variable to choose the ``grep`` executable this command should use.
2022-12-09Windows: reenable unit tests (#33385)John W. Parent4-14/+10
Unit tests on Windows are supposed to pass for any PR to pass CI. However, the return code for the unit test command was not being checked, which meant this check was always passing (effectively disabled). This PR * Properly checks the result of the unit tests and fails if the unit tests fail * Fixes (or disables on Windows) a number of tests which have "drifted" out of support on Windows since this check was effectively disabled
2022-12-09bugfix: `spack load` shell test can fail on macos (#34419)Todd Gamblin1-1/+2
At some point the `a` mock package became an `AutotoolsPackage`, and that means it depends on `gnuconfig` on macOS. This was causing one of our shell tests to fail on macOS because it was testing for `{a.prefix.bin}:{b.prefix.bin}` in `PATH`, but `gnuconfig` shows up between them. - [x] simplify the test to check `spack load --sh a` and `spack load --sh b` separately
2022-12-06Bootstrap most of Spack dependencies using environments (#34029)Massimiliano Culpo2-1/+35
This commit reworks the bootstrapping procedure to use Spack environments as much as possible. The `spack.bootstrap` module has also been reorganized into a Python package. A distinction is made among "core" Spack dependencies (clingo, GnuPG, patchelf) and other dependencies. For a number of reasons, explained in the `spack.bootstrap.core` module docstring, "core" dependencies are bootstrapped with the current ad-hoc method. All the other dependencies are instead bootstrapped using a Spack environment that lives in a directory specific to the interpreter and the architecture being used.
2022-12-02py-alphafold: update to 2.2.4, update dependencies (#33876)Andrew W Elble3-9/+6
* py-alphafold: update to 2.2.4, update dependencies * style
2022-12-02patch command: add concretizer args (#34282)Greg Becker1-1/+1
* patch command: add concretizer args * tab completion
2022-11-30e4s ci: use 2022-12-01 runner images (#34212)eugeneswalker2-4/+4
2022-11-30e4s ci: hpx: set max_cpu_count=512 (#33977)eugeneswalker2-4/+4
2022-11-29CI: Update Data and Vis SDK Stack (#34009)kwryankrattiger2-77/+105
* CI: Update Data and Vis SDK Stack * Update image to match target deployments (E4S) * Enable all packages * Test supported variants of ParaView and VisIt * Sensei: Update Python hint for newer cmake * Sensei: add Python3 hint
2022-11-28e4s ci: add hdf5-vol-async; remove expired comments (#34110)eugeneswalker1-4/+1
2022-11-28warn about removal of deprecated format strings (#34101)Greg Becker1-2/+2
* warn about removal of deprecated format strings Co-authored-by: becker33 <becker33@users.noreply.github.com>
2022-11-23E4S: Conservatively add ecp-data-vis-sdk (#33621)kwryankrattiger1-19/+9
* E4S: Conservatively add ecp-data-vis-sdk * Remove ascent from CUDA SDK stack to stop hanging on Dray * Adios2: Newer FindPython uses Python_EXECUTABLE
2022-11-22cmd/checksum: allow adding new versions to package (#24532)Michael Kuhn1-1/+1
This adds super-lazy maintainer mode to `spack checksum`: Instead of only printing the new checksums to the terminal, `-a` and `--add-to-package` will add the new checksums to the `package.py` file and open it in the editor afterwards for final checks.
2022-11-22Revert "Warn about removal of deprecated format strings (#33829)" (#34056)Massimiliano Culpo1-2/+2
This reverts commit 7f9af8d4a0bfbb1577e5ac9982624d8d0cb9b9ca.
2022-11-22Warn about removal of deprecated format strings (#33829)Greg Becker1-2/+2
Co-authored-by: becker33 <becker33@users.noreply.github.com>
2022-11-19spack find: remove deprecated "--bootstrap" option (#34015)Massimiliano Culpo1-1/+1
2022-11-17Remove deprecated subcommands from "spack bootstrap" (#33964)Massimiliano Culpo1-19/+1
These commands are slated for removal in v0.20
2022-11-15depfile: improve tab completion (#33773)Harmen Stoppels1-6/+17
This PR allows you to do: ``` spack env create -d . spack -e . add python spack -e . concretize spack -e . env depfile -o Makefile make in<tab> # -> install make install-<tab> # -> install-deps/ make install-deps/py<tab> # -> install-deps/python-x.y.z-hash make install/zl<tab> # -> install/zlib-x.y.z-hash make SP<tab> # -> make SPACK make SPACK_<tab> # -> make SPACK_INSTALL_FLAGS= ```
2022-11-15gitlab: report load in generate job (#33888)Harmen Stoppels17-0/+39
2022-11-11remove activate/deactivate support in favor of environments (#29317)Harmen Stoppels1-20/+2
Environments and environment views have taken over the role of `spack activate/deactivate`, and we should deprecate these commands for several reasons: - Global activation is a really poor idea: - Install prefixes should be immutable; since they can have multiple, unrelated dependents; see below - Added complexity elsewhere: verification of installations, tarballs for build caches, creation of environment views of packages with unrelated extensions "globally activated"... by removing the feature, it gets easier for people to contribute, and we'd end up with fewer bugs due to edge cases. - Environment accomplish the same thing for non-global "activation" i.e. `spack view`, but better. Also we write in the docs: ``` However, Spack global activations have two potential drawbacks: #. Activated packages that involve compiled C extensions may still need their dependencies to be loaded manually. For example, ``spack load openblas`` might be required to make ``py-numpy`` work. #. Global activations "break" a core feature of Spack, which is that multiple versions of a package can co-exist side-by-side. For example, suppose you wish to run a Python package in two different environments but the same basic Python --- one with ``py-numpy@1.7`` and one with ``py-numpy@1.8``. Spack extensions will not support this potential debugging use case. ``` Now that environments are established and views can take over the role of activation non-destructively, we can remove global activation/deactivation.
2022-11-08More jobs in Gitlab CI (#33688)Harmen Stoppels16-0/+31
Use at most 32 jobs when available.
2022-11-08"spack uninstall": don't modify env (#33711)Peter Scheibel1-2/+2
"spack install foo" no longer adds package "foo" to the environment (i.e. to the list of root specs) by default: you must specify "--add". Likewise "spack uninstall foo" no longer removes package "foo" from the environment: you must specify --remove. Generally this means that install/uninstall commands will no longer modify the users list of root specs (which many users found problematic: they had to deactivate an environment if they wanted to uninstall a spec without changing their spack.yaml description). In more detail: if you have environments e1 and e2, and specs [P, Q, R] such that P depends on R, Q depends on R, [P, R] are in e1, and [Q, R] are in e2: * `spack uninstall --dependents --remove r` in e1: removes R from e1 (but does not uninstall it) and uninstalls (and removes) P * `spack uninstall -f --dependents r` in e1: will uninstall P, Q, and R (i.e. e2 will have dependent specs uninstalled as a side effect) * `spack uninstall -f --dependents --remove r` in e1: this uninstalls P, Q, and R, and removes [P, R] from e1 * `spack uninstall -f --remove r` in e1: uninstalls R (so it is "missing" in both environments) and removes R from e1 (note that e1 would still install R as a dependency of P, but it would no longer be listed as a root spec) * `spack uninstall --dependents r` in e1: will fail because e2 needs R Individual unit tests were created for each of these scenarios.
2022-11-05py-transformers: add v4.24.0 (#33716)Adam J. Stewart3-0/+9
* py-transformers: add v4.24.0 * Internet access still required
2022-11-03Update metadata for bootstrapping (#33665)Massimiliano Culpo11-944/+1095
2022-11-01Let pytest-cov create the xml directly (#33619)Massimiliano Culpo1-1/+1
`coverage` sometimes failed to combine, even if there were multiple reports.
2022-10-31Revert "gitlab: when_possible -> false (#33443)" (#33552)Massimiliano Culpo2-2/+2
This reverts commit b1559cc831620ee2b2cf8e57fdecc5bb3bf8edfd.
2022-10-29e4s ci: add mfem +rocm (#31604)eugeneswalker1-0/+1
2022-10-29Deprecate spack bootstrap trust/untrust (#33600)Massimiliano Culpo1-2/+12
* Deprecate spack bootstrap trust/untrust * Update CI * Update tests
2022-10-28gitlab ci: patched make 4.3.0 (#33583)Harmen Stoppels16-31/+33
2022-10-27Don't fail over cpuinfo (#33546)Harmen Stoppels16-32/+32
2022-10-26gitlab ci: show build machine info (#33523)Harmen Stoppels17-0/+49
2022-10-26Allow for packages with multiple build-systems (#30738)Massimiliano Culpo6-0/+12
This commit extends the DSL that can be used in packages to allow declaring that a package uses different build-systems under different conditions. It requires each spec to have a `build_system` single valued variant. The variant can be used in many context to query, manipulate or select the build system associated with a concrete spec. The knowledge to build a package has been moved out of the PackageBase hierarchy, into a new Builder hierarchy. Customization of the default behavior for a given builder can be obtained by coding a new derived builder in package.py. The "run_after" and "run_before" decorators are now applied to methods on the builder. They can also incorporate a "when=" argument to specify that a method is run only when certain conditions apply. For packages that do not define their own builder, forwarding logic is added between the builder and package (methods not found in one will be retrieved from the other); this PR is expected to be fully backwards compatible with unmodified packages that use a single build system.
2022-10-26gitlab ci: install binary deps faster (#33248)Harmen Stoppels16-0/+46
* Fast Gitlab CI job setup, and better legibility * Use a non-broken, recent GNU Make
2022-10-25depfile: resurrect lost touch (#33504)Harmen Stoppels1-0/+3
2022-10-25feature: Add -x|--explicit option to 'spack test run' (#32910)Tamara Dahlgren1-1/+1
2022-10-25Update Spack Dockerfiles (#33500)Massimiliano Culpo2-2/+1
* Use spack bootstrap now in containers * Fix wrong path glob expression
2022-10-25Remove recursive symbolic link in lib/spack/docs from git repository (#33483)Massimiliano Culpo2-10/+0
Delete code removing the symlink during CI
2022-10-25Improve legibility of Gitlab CI (#33482)Harmen Stoppels17-17/+17
Use --backtrace in ci instead of --debug to reduce verbosity and don't show log on error, since log is already printed
2022-10-24e4s ci: use an appropriate name for cdash build group name (#33494)eugeneswalker1-1/+1
2022-10-24e4s ci: add chai +rocm (#32506)eugeneswalker1-1/+3
2022-10-23backtraces with --backtrace (#33478)Harmen Stoppels1-1/+1
* backtraces without --debug Currently `--debug` is too verbose and not-`--debug` gives to little context about where exceptions are coming from. So, instead, it'd be nice to have `spack --backtrace` and `SPACK_BACKTRACE=1` as methods to get something inbetween: no verbose debug messages, but always a full backtrace. This is useful for CI, where we don't want to drown in debug messages when installing deps, but we do want to get details where something goes wrong if it goes wrong. * completion
2022-10-22Remove x86_64_v4 target from AHUG and ISC stacks (#33464)Zack Galbreath2-3/+0
2022-10-22stop building binaries for the 1% (#33463)Harmen Stoppels3-3/+3
2022-10-21gitlab: Retry protected publish jobs in certain cases (#32496)Scott Wittenburg1-0/+3
When we lose a running pod (possibly loss of spot instance) or encounter some other infrastructure-related failure of this job, we need to retry it. This retries the job the maximum number of times in those cases.