summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2022-12-06Bootstrap most of Spack dependencies using environments (#34029)Massimiliano Culpo18-1149/+1426
This commit reworks the bootstrapping procedure to use Spack environments as much as possible. The `spack.bootstrap` module has also been reorganized into a Python package. A distinction is made among "core" Spack dependencies (clingo, GnuPG, patchelf) and other dependencies. For a number of reasons, explained in the `spack.bootstrap.core` module docstring, "core" dependencies are bootstrapped with the current ad-hoc method. All the other dependencies are instead bootstrapped using a Spack environment that lives in a directory specific to the interpreter and the architecture being used.
2022-12-05vermin: remove all novm comments from code (#34308)Todd Gamblin23-26/+26
All the vermin annotations we were using were for optional features introduced in early Python 3 versions. We no longer need any of them, as we only support Python 3.6+. If we start optionally using features from newer Pythons than 3.6, we will need more vermin annotations. Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com> Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2022-12-05web: remove checks for SSL verification support (#34307)Todd Gamblin1-31/+5
We no longer support Python <3.6, so we don't need to check whether Python supports SSL verification in `spack.util.web`. - [x] Remove a bunch of logic we needed to appease Python 2
2022-12-04typing: move from comment annotations to Python 3.6 annotations (#34305)Todd Gamblin39-134/+118
We've stopped supporting Python 2, and contributors are noticing that our CI no longer allows Python 2.7 comment type hints. They end up having to adapt them, but this adds extra unrelated work to PRs. - [x] Move to 3.6 type hints across the entire code base
2022-12-02patch command: add concretizer args (#34282)Greg Becker1-0/+1
* patch command: add concretizer args * tab completion
2022-12-02Docs: Minor change 'several'->'over a dozen' (#34274)Tamara Dahlgren1-1/+1
2022-12-02concretizer: use only `attr()` for Spec attributes (#31202)Todd Gamblin4-430/+409
All Spec attributes are now represented as `attr(attribute_name, ... args ...)`, e.g. `attr(node, "hdf5")` instead of `node("hdf5")`, as we *have* to maintain the `attr()` form anyway, and it simplifies the encoding to just maintain one form of the Spec information. Background ---------- In #20644, we unified the way conditionals are done in the concretizer, but this introduced a nasty aspect to the encoding: we have to maintain everything we want in general conditions in two forms: `predicate(...)` and `attr("predicate", ...)`. For example, here's the start of the table of spec attributes we had to maintain: ```prolog node(Package) :- attr("node", Package). virtual_node(Virtual) :- attr("virtual_node", Virtual). hash(Package, Hash) :- attr("hash", Package, Hash). version(Package, Version) :- attr("version", Package, Version). ... ``` ```prolog attr("node", Package) :- node(Package). attr("virtual_node", Virtual) :- virtual_node(Virtual). attr("hash", Package, Hash) :- hash(Package, Hash). attr("version", Package, Version) :- version(Package, Version). ... ``` This adds cognitive load to understanding how the concretizer works, as you have to understand the equivalence between the two forms of spec attributes. It also makes the general condition logic in #20644 hard to explain, and it's easy to forget to add a new equivalence to this list when adding new spec attributes (at least two people have been bitten by this). Solution -------- - [x] remove the equivalence list from `concretize.lp` - [x] simplify `spec_clauses()`, `condition()`, and other functions in `asp.py` that need to deal with `Spec` attributes. - [x] Convert all old-form spec attributes in `concretize.lp` to the `attr()` form - [x] Simplify `display.lp`, where we also had to maintain a list of spec attributes. Now we only need to show `attr/2`, `attr/3`, and `attr/4`. - [x] Simplify model extraction logic in `asp.py`. Performance ----------- This seems to result in a smaller grounded problem (as there are no longer duplicated `attr("foo", ...)` / `foo(...)` predicates in the program), but it also adds a slight performance overhead vs. develop. Ultimately, simplifying the encoding will be a win, particularly for improving error messages. Notes ----- This will simplify future node refactors in `concretize.lp` (e.g., not identifying nodes by package name, which we need for separate build dependencies). I'm still not entirely used to reading `attr()` notation, but I thnk it's ultimately clearer than what we did before. We need more uniform naming, and it's now clear what is part of a solution. We should probably continue making the encoding of `concretize.lp` simpler and more self-explanatory. It may make sense to rename `attr` to something like `node_attr` and to simplify the names of node attributes. It also might make sense to do something similar for other types of predicates in `concretize.lp`.
2022-12-02Bugfix: Fetch should not force use of curl to check url existence (#34225)Tamara Dahlgren1-3/+3
* Bugfix: Fetch should not force use of curl to check url existence * Switch type hints from comments to actual hints
2022-12-01conditional variant values: allow boolean (#33939)Greg Becker2-17/+39
2022-11-30Deduplicate code to propagate module changes across MRO (#34157)Massimiliano Culpo2-50/+22
2022-11-30Single pass text replacement (#34180)Harmen Stoppels2-34/+88
2022-11-29debug: move "nonexistent config path" message to much higher verbosity level ↵Todd Gamblin1-2/+2
(#34201) We currently report that searched config paths don't exist at debug level 1, which clutters the output quite a bit: ```console > spack -d solve --fresh --show asp hdf5 > hdf5.lp ==> [2022-11-29-14:18:21.035133] Skipping nonexistent config path /Users/gamblin2/src/spack/etc/spack/darwin/concretizer.yaml ==> [2022-11-29-14:18:21.035151] Skipping nonexistent config path /Users/gamblin2/.spack/concretizer.yaml ==> [2022-11-29-14:18:21.035169] Skipping nonexistent config path /Users/gamblin2/.spack/darwin/concretizer.yaml ==> [2022-11-29-14:18:21.035238] Reading config from file /Users/gamblin2/src/spack/etc/spack/defaults/repos.yaml ==> [2022-11-29-14:18:21.035996] Skipping nonexistent config path /Users/gamblin2/src/spack/etc/spack/defaults/darwin/repos.yaml ==> [2022-11-29-14:18:21.036021] Skipping nonexistent config path /etc/spack/repos.yaml ==> [2022-11-29-14:18:21.036039] Skipping nonexistent config path /etc/spack/darwin/repos.yaml ==> [2022-11-29-14:18:21.036057] Skipping nonexistent config path /Users/gamblin2/src/spack/etc/spack/repos.yaml ==> [2022-11-29-14:18:21.036072] Skipping nonexistent config path /Users/gamblin2/src/spack/etc/spack/darwin/repos.yaml ==> [2022-11-29-14:18:21.036088] Skipping nonexistent config path /Users/gamblin2/.spack/repos.yaml ==> [2022-11-29-14:18:21.036105] Skipping nonexistent config path /Users/gamblin2/.spack/darwin/repos.yaml ==> [2022-11-29-14:18:21.071828] Reading config from file /Users/gamblin2/src/spack/etc/spack/defaults/config.yaml ==> [2022-11-29-14:18:21.081628] Skipping nonexistent config path /Users/gamblin2/src/spack/etc/spack/defaults/darwin/config.yaml ==> [2022-11-29-14:18:21.081669] Skipping nonexistent config path /etc/spack/config.yaml ==> [2022-11-29-14:18:21.081692] Skipping nonexistent config path /etc/spack/darwin/config.yaml ==> [2022-11-29-14:18:21.081712] Skipping nonexistent config path /Users/gamblin2/src/spack/etc/spack/config.yaml ==> [2022-11-29-14:18:21.081731] Skipping nonexistent config path /Users/gamblin2/src/spack/etc/spack/darwin/config.yaml ==> [2022-11-29-14:18:21.081748] Skipping nonexistent config path /Users/gamblin2/.spack/config.yaml ==> [2022-11-29-14:18:21.081764] Skipping nonexistent config path /Users/gamblin2/.spack/darwin/config.yaml ==> [2022-11-29-14:18:21.134909] Reading config from file /Users/gamblin2/src/spack/etc/spack/defaults/packages.yaml ==> [2022-11-29-14:18:21.148695] Reading config from file /Users/gamblin2/src/spack/etc/spack/defaults/darwin/packages.yaml ==> [2022-11-29-14:18:21.152555] Skipping nonexistent config path /etc/spack/packages.yaml ==> [2022-11-29-14:18:21.152582] Skipping nonexistent config path /etc/spack/darwin/packages.yaml ==> [2022-11-29-14:18:21.152601] Skipping nonexistent config path /Users/gamblin2/src/spack/etc/spack/packages.yaml ==> [2022-11-29-14:18:21.152620] Skipping nonexistent config path /Users/gamblin2/src/spack/etc/spack/darwin/packages.yaml ==> [2022-11-29-14:18:21.152637] Skipping nonexistent config path /Users/gamblin2/.spack/packages.yaml ==> [2022-11-29-14:18:21.152654] Skipping nonexistent config path /Users/gamblin2/.spack/darwin/packages.yaml ==> [2022-11-29-14:18:21.853915] Skipping nonexistent config path /Users/gamblin2/src/spack/etc/spack/defaults/compilers.yaml ==> [2022-11-29-14:18:21.853962] Skipping nonexistent config path /Users/gamblin2/src/spack/etc/spack/defaults/darwin/compilers.yaml ==> [2022-11-29-14:18:21.853987] Skipping nonexistent config path /etc/spack/compilers.yaml ==> [2022-11-29-14:18:21.854007] Skipping nonexistent config path /etc/spack/darwin/compilers.yaml ==> [2022-11-29-14:18:21.854025] Skipping nonexistent config path /Users/gamblin2/src/spack/etc/spack/compilers.yaml ==> [2022-11-29-14:18:21.854043] Skipping nonexistent config path /Users/gamblin2/src/spack/etc/spack/darwin/compilers.yaml ==> [2022-11-29-14:18:21.854060] Skipping nonexistent config path /Users/gamblin2/.spack/compilers.yaml ==> [2022-11-29-14:18:21.854093] Reading config from file /Users/gamblin2/.spack/darwin/compilers.yaml ``` It is very rare that I want to know this much information about config search, so I've moved this to level 3. Now at level 1, we can see much more clearly what configs were actually found: ```console > spack -d solve --fresh --show asp hdf5 > hdf5.lp ==> [2022-11-29-14:19:04.035457] Imported solve from built-in commands ==> [2022-11-29-14:19:04.035818] Imported solve from built-in commands ==> [2022-11-29-14:19:04.037626] Reading config from file /Users/gamblin2/src/spack/etc/spack/defaults/concretizer.yaml ==> [2022-11-29-14:19:04.040033] Reading config from file /Users/gamblin2/src/spack/etc/spack/defaults/repos.yaml ==> [2022-11-29-14:19:04.080852] Reading config from file /Users/gamblin2/src/spack/etc/spack/defaults/config.yaml ==> [2022-11-29-14:19:04.133241] Reading config from file /Users/gamblin2/src/spack/etc/spack/defaults/packages.yaml ==> [2022-11-29-14:19:04.147175] Reading config from file /Users/gamblin2/src/spack/etc/spack/defaults/darwin/packages.yaml ==> [2022-11-29-14:19:05.157896] Reading config from file /Users/gamblin2/.spack/darwin/compilers.yaml ``` You can still get the old messages with `spack -ddd` (to activate debug level 3).
2022-11-28warn about removal of deprecated format strings (#34101)Greg Becker4-3/+16
* warn about removal of deprecated format strings Co-authored-by: becker33 <becker33@users.noreply.github.com>
2022-11-28Use a module-like object to propagate changes in the MRO, when setting build ↵Massimiliano Culpo2-18/+77
env (#34059) This fixes an issue introduced in #32340, which changed the semantics of the "module" object passed to the "setup_dependent_package" callback.
2022-11-28Revert "Revert "gitlab: Add shared PR mirror to places pipelines look for ↵Harmen Stoppels3-0/+17
binaries. (#33746)" (#34087)" (#34153) This reverts commit 63e440651436fdc30993ef58875279e71f4851d1.
2022-11-28Add type hints to Prefix class (#34135)Adam J. Stewart1-21/+44
2022-11-26Revert "`url_exists` related improvements (#34095)" (#34144)Satish Balay2-78/+64
This reverts commit d06fd26c9ac8dd525fc129096188e2ea9fd2d0d7. The problem is that Bitbucket's API forwards download requests to an S3 bucket using a temporary URL. This URL includes a signature for the request, which embeds the HTTP verb. That means only GET requests are allowed, and HEAD requests would fail verification, leading to 403 erros. The same is observed when using `curl -LI ...`
2022-11-25Track locks by (dev, ino); close file handlers between tests (#34122)Harmen Stoppels3-21/+37
2022-11-24docs: update info on XCode requirements (#34097)Valentin Volkl1-2/+3
2022-11-23Control Werror by converting to Wno-error (#30882)Tom Scogland5-0/+176
Using `-Werror` is good practice for development and testing, but causes us a great deal of heartburn supporting multiple compiler versions, especially as newer compiler versions add warnings for released packages. This PR adds support for suppressing `-Werror` through spack's compiler wrappers. There are currently three modes for the `flags:keep_werror` setting: * `none`: (default) cancel all `-Werror`, `-Werror=*` and `-Werror-*` flags by converting them to `-Wno-error[=]*` flags * `specific`: preserve explicitly selected warnings as errors, such as `-Werror=format-truncation`, but reverse the blanket `-Werror` * `all`: keeps all `-Werror` flags These can be set globally in config.yaml, through the config command-line flags, or overridden by a particular package (some packages use Werror as a proxy for determining support for other compiler features). We chose to use this approach because: 1. removing `-Werror` flags entirely broke *many* build systems, especially autoconf based ones, because of things like checking `-Werror=feature` and making the assumption that if that did not error other flags related to that feature would also work 2. Attempting to preserve `-Werror` in some phases but not others caused similar issues 3. The per-package setting came about because some packages, even with all these protections, still use `-Werror` unsafely. Currently there are roughly 3 such packages known.
2022-11-23`url_exists` related improvements (#34095)Harmen Stoppels2-64/+78
For reasons beyond me Python thinks it's a great idea to upgrade HEAD requests to GET requests when following redirects. So, this PR adds a better `HTTPRedirectHandler`, and also moves some ad-hoc logic around for dealing with disabling SSL certs verification. Also, I'm stumped by the fact that Spack's `url_exists` does not use HEAD requests at all, so in certain cases Spack awkwardly downloads something first to see if it can download it, and then downloads it again because it knows it can download it. So, this PR ensures that both urllib and botocore use HEAD requests. Finally, it also removes some things that were there to support currently unsupported Python versions. Notice that the HTTP spec [section 10.3.2](https://datatracker.ietf.org/doc/html/rfc2616.html#section-10.3.2) just talks about how to deal with POST request on redirect (whether to follow or not): > If the 301 status code is received in response to a request other > than GET or HEAD, the user agent MUST NOT automatically redirect the > request unless it can be confirmed by the user, since this might > change the conditions under which the request was issued. > Note: When automatically redirecting a POST request after > receiving a 301 status code, some existing HTTP/1.0 user agents > will erroneously change it into a GET request. Python has a comment about this, they choose to go with the "erroneous change". But they then mess up the HEAD request while following the redirect, probably because they were too busy discussing how to deal with POST. See https://github.com/python/cpython/pull/99731
2022-11-23Revert "gitlab: Add shared PR mirror to places pipelines look for binaries. ↵Harmen Stoppels3-17/+0
(#33746)" (#34087) This reverts commit 5c4137baf19b9e271a2f13e886d6b875aab067dd.
2022-11-23rocm: add minimum versions for amdgpu_targets (#34030)Cory Bloor1-0/+23
2022-11-23Build System docs: consistent headers (#34047)Adam J. Stewart8-30/+30
2022-11-22cmd/checksum: allow adding new versions to package (#24532)Michael Kuhn3-8/+98
This adds super-lazy maintainer mode to `spack checksum`: Instead of only printing the new checksums to the terminal, `-a` and `--add-to-package` will add the new checksums to the `package.py` file and open it in the editor afterwards for final checks.
2022-11-22spack uninstall: use topo order (#34053)Harmen Stoppels2-50/+17
2022-11-22Revert "Warn about removal of deprecated format strings (#33829)" (#34056)Massimiliano Culpo2-5/+2
This reverts commit 7f9af8d4a0bfbb1577e5ac9982624d8d0cb9b9ca.
2022-11-22Warn about removal of deprecated format strings (#33829)Greg Becker2-2/+5
Co-authored-by: becker33 <becker33@users.noreply.github.com>
2022-11-22Windows: add registry query and SDK/WDK packages (#33021)John W. Parent8-68/+552
* Add a WindowsRegistryView class, which can query for existing package installations on Windows. This is particularly important because some Windows packages (including those added here) do not allow two simultaneous installs, and this can be queried in order to provide a clear error message. * Consolidate external path detection logic for Windows into WindowsKitExternalPaths and WindowsCompilerExternalPaths objects. * Add external-only packages win-sdk and wgl * Add win-wdk (including external detection) which depends on win-sdk * Replace prior msmpi implementation with a source-based install (depends on win-wdk). This install can control the install destination (unlike the binary installation). * Update MSVC compiler to choose vcvars based on win-sdk dependency * Provide "msbuild" module-level variable to packages during build * When creating symlinks on Windows, need to explicitly specify when a symlink target is a directory * executables_in_path no-longer defaults to using PATH (this is now expected to be taken care of by the caller)
2022-11-21Spec traversal: add option for topological ordering (#33817)Harmen Stoppels2-99/+371
Spec traversals can now specify a topological ordering. A topologically- ordered traversal with input specs X1, X2... will * include all of X1, X2... and their children * be ordered such that a given node is guaranteed to appear before any of its children in the traversal Other notes: * Input specs can be children of other input specs (this is useful if a user specifies a set of specs to uninstall: some of those specs might be children of others) * `direction="parents"` will produce a reversed topological order (children always come before parents). * `cover="edges"` will generate a list of edges L such that (a) input edges will always appear before output edges and (b) if you create a list with the destination of each edge in L the result is topologically ordered
2022-11-21Speed-up a few unit-tests (#34044)Massimiliano Culpo3-3/+3
* test_suite.py: speed up slow test by using mock packages * Don't resolve the sha during unit-tests * Skip long-running test that fails, instead of executing it
2022-11-21ci: restore ability to reproduce gitlab job failures (#33953)Scott Wittenburg1-3/+18
2022-11-21uninstall: fix accidental cubic complexity (#34005)Harmen Stoppels3-17/+66
* uninstall: fix accidental cubic complexity Currently spack uninstall runs in worst case cubic time complexity thanks to traversal during traversal during traversal while collecting the specs to be uninstalled. Also brings down the number of error messages printed to something linear in the amount of matching specs instead of quadratic.
2022-11-19spack find: remove deprecated "--bootstrap" option (#34015)Massimiliano Culpo2-29/+0
2022-11-19package_base.py: Fix #34006: test msg needs to be a string (#34007)Chris Green2-2/+10
2022-11-18PackageBase should not define builder legacy attributes (#33942)Chris Green4-11/+17
* Add a regression test for 33928 * PackageBase should not set `(build|install)_time_test_callbacks` * Fix audits by preserving the current semantic Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-11-18Improve error message for requirements (#33988)Massimiliano Culpo1-1/+1
refers #33985
2022-11-17Hdf5 package: build on Windows (#31141)Jared Popelar2-1/+3
* Enable hdf5 build (including +mpi) on Windows * This includes updates to hdf5 dependencies openssl (minor edit) and bzip2 (more-extensive edits) * Add binary-based installation of msmpi (this is currently the only supported MPI implementation in Spack for Windows). Note that this does not install to the Spack-specified prefix. This implementation will be replaced with a source-based implementation Co-authored-by: John Parent <john.parent@kitware.com>
2022-11-17Waf build system: fix typo in legacy_attributes (#33958)Chris Green1-1/+1
Fix erroneous duplication of `build_time_test_callbacks` in `legacy_attributes`: one of the duplicates should be `install_time_test_callbacks`
2022-11-17docs: fix typo in multiple build systems (#33965)Harmen Stoppels1-1/+1
2022-11-17relocate.py: small refactor for file_is_relocatable (#33967)Harmen Stoppels4-39/+31
2022-11-17Remove deprecated subcommands from "spack bootstrap" (#33964)Massimiliano Culpo2-42/+23
These commands are slated for removal in v0.20
2022-11-17Add sgid notice when running on AFS (#30247)Michael Kuhn1-0/+7
2022-11-17lib/spack/spack/store.py: Fix #28170 for padding relocation (#33122)Brian Vanderwende1-0/+1
2022-11-16initial implementation of slingshot detection (#33793)Greg Becker1-0/+4
2022-11-16Upstreams: add canonicalize path (#33946)psakievich1-1/+3
2022-11-16Windows bootstrapping: remove unneeded call to add dll to PATH (#33622)John W. Parent1-7/+0
#32942 fixed bootstrapping on Windows by having the core Spack code explicitly add the Clingo package bin/ directory as a DLL path. Since then, #33400 has been merged, which ensures that the Python module installed by the Spack `clingo` package can find the DLLs in bin/. Note that this only works for Spack instances which have been bootstrapped after #33400: for installations bootstrapped before then, you will need to run `spack clean -b` (this would only be needed for Spack instances running on Windows).
2022-11-16SIP build system: fix "python not defined in builder" (#33906)Stephen Sachs1-1/+1
Co-authored-by: Stephen Sachs <stesachs@amazon.com>
2022-11-16docs updates for spack env depfile (#33937)Harmen Stoppels1-6/+10
2022-11-16spack env depfile in Gitlab CI should use `install-deps/pkg-version-hash` ↵Harmen Stoppels1-3/+1
target (#33936)
generated by cgit v1.2.3-60-g2f50 (git 2.42.0) at 2024-07-12 07:59:58 +0000