summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2021-02-22Improve error message for inconsistencies in package.py (#21811)Massimiliano Culpo2-5/+40
* Improve error message for inconsistencies in package.py Sometimes directives refer to variants that do not exist. Make it such that: 1. The name of the variant 2. The name of the package which is supposed to have such variant 3. The name of the package making this assumption are all printed in the error message for easier debugging. * Add unit tests
2021-02-22Merge tag 'v0.16.1' into developTodd Gamblin1-2/+2
2021-02-22respect -k/verify-ssl-false in _existing_url method (#21864)Greg Becker1-0/+2
2021-02-19Update CHANGELOG and release versionv0.16.1Tamara Dahlgren1-1/+1
2021-02-19Resolve (post-cherry-picking) flake8 errorsTamara Dahlgren3-4/+8
2021-02-19documentation: correct precedence of included configs in environment ↵Tom Payerle1-2/+2
spack.yaml (#18663) fixes #17993
2021-02-19bugfix: add build deps to 'full hash' (#21735)Greg Becker4-21/+39
The "full hash" was only including the link/run deps, but it should include build deps as well.
2021-02-18Pipelines: Move PR testing stacks (currently only E4S) into spack (#21714)Scott Wittenburg3-37/+61
2021-02-18Fix template for Rpackage in spack create command (#21776)Tom Payerle1-1/+1
The signature for configure_args in the template for new RPackage packages was incorrect (different than what is defined and used in lib/spack/spack/build_systems/r.py) See issue #21774
2021-02-18Testing: use spack.store.use_store everywhere (#21656)Massimiliano Culpo10-175/+176
Keep spack.store.store and spack.store.db consistent in unit tests * Remove calls to monkeypatch for spack.store.store and spack.store.db: tests that used these called one or the other, which lead to inconsistencies (the tests passed regardless but were fragile as a result) * Fixtures making use of monkeypatch with mock_store now use the updated use_store function, which sets store.store and store.db consistently * subprocess_context.TestState now transfers the serializes and restores spack.store.store (without the monkeypatch changes this would have created inconsistencies)
2021-02-18Documentation fix: build_system configure_args for #21760 (#21761)Tom Payerle1-2/+2
Corrects the signature for configure_args (and therefore configure_vars) in documentation on RPackage build system to match the code See issue #21760
2021-02-18bugfix: relax racy test in fg/bg output (#21755)Todd Gamblin1-12/+6
Since signals are fundamentally racy, We can't bound the amount of time that the `test_foreground_background_output` test will take to get to 'on', we can only observe that it transitions to 'on'. So instead of using an arbitrary limit, just adjust the test to allow either 'on' or 'off' followed by 'on'. This should eliminate the spurious errors we see in CI.
2021-02-18Avoid spurious warning from clingo (#21731)Massimiliano Culpo1-0/+1
There's a spurious warning that occurs whenever a spec being concretized does not depend on a virtual provider under any possible configuration.
2021-02-17apple-clang: add correct path to compiler wrappers (#21662)Adam J. Stewart1-1/+1
Follow-up to #17110 ### Before ```bash CC=/Users/Adam/spack/lib/spack/env/clang/clang; export CC SPACK_CC=/usr/bin/clang; export SPACK_CC PATH=...:/Users/Adam/spack/lib/spack/env/apple-clang:/Users/Adam/spack/lib/spack/env/case-insensitive:/Users/Adam/spack/lib/spack/env:...; export PATH ``` ### After ```bash CC=/Users/Adam/spack/lib/spack/env/clang/clang; export CC SPACK_CC=/usr/bin/clang; export SPACK_CC PATH=...:/Users/Adam/spack/lib/spack/env/clang:/Users/Adam/spack/lib/spack/env/case-insensitive:/Users/Adam/spack/lib/spack/env:...; export PATH ``` `CC` and `SPACK_CC` were being set correctly, but `PATH` was using the name of the compiler `apple-clang` instead of `clang`. For most packages, since `CC` was set correctly, nothing broke. But for packages using `Makefiles` that set `CC` based on `which clang`, it was using the system compilers instead of the compiler wrappers. Discovered when working on `py-xgboost@0.90`. An alternative fix would be to copy the symlinks in `env/clang` to `env/apple-clang`. Let me know if you think there's a better way to do this, or to test this.
2021-02-17add intel oneapi to compiler/pkg translations (#21448)Greg Becker1-1/+2
2021-02-17Print groups properly for spack find -d (#20028)Yang Zongze1-1/+1
2021-02-17store sbang_install_path in buildinfo, use for subsequent relocation (#20768)eugeneswalker1-0/+11
2021-02-17[WIP] relocate.py: parallelize test replacement logic (#19690)Nathan Hanford9-210/+281
* sbang pushed back to callers; star moved to util.lang * updated unit test * sbang test moved; local tests pass Co-authored-by: Nathan Hanford <hanford1@llnl.gov>
2021-02-17concretizer: require at least a dependency type to say the dependency holdsMassimiliano Culpo2-0/+11
fixes #20784 Similarly to the previous bug, here we were deducing conditions to be imposed on nodes that were not part of the DAG.
2021-02-17concretizer: dependency conditions cannot hold if package is externalMassimiliano Culpo3-2/+9
fixes #20736 Before this one line fix we were erroneously deducing that dependency conditions hold even if a package was external. This may result in answer sets that contain imposed conditions on a node without the node being present in the DAG, hence #20736.
2021-02-17restore ability of dev-build to skip patches (#20351)Robert Underwood1-0/+1
At some point in the past, the skip_patch argument was removed from the call to package.do_install() this broke the --skip-patch flag on the dev-build command.
2021-02-17concretizer: make rules on virtual packages more linearMassimiliano Culpo1-11/+9
fixes #20679 In this refactor we have a single cardinality rule on the provider, which triggers a rule transforming a dependency on a virtual package into a dependency on the provider of the virtual.
2021-02-17concretizer: use consistent naming for compiler predicates (#20677)Todd Gamblin2-10/+8
Every other predicate in the concretizer uses a `_set` suffix to implement user- or package-supplied settings, but compiler settings use a `_hard` suffix for this. There's no difference in how they're used, so make the names the same. - [x] change `node_compiler_hard` to `node_compiler_set` - [x] change `node_compiler_version_hard` to `node_compiler_version_set`
2021-02-17concretizer: simplify handling of virtual version constraintsTodd Gamblin2-18/+29
Previously, the concretizer handled version constraints by comparing all pairs of constraints and ensuring they satisfied each other. This led to INCONSISTENT ressults from clingo, due to ambiguous semantics like: version_constraint_satisfies("mpi", ":1", ":3") version_constraint_satisfies("mpi", ":3", ":1") To get around this, we introduce possible (fake) versions for virtuals, based on their constraints. Essentially, we add any Versions, VersionRange endpoints, and all such Versions and endpoints from VersionLists to the constraint. Virtuals will have one of these synthetic versions "picked" by the solver. This also allows us to remove a special case from handling of `version_satisfies/3` -- virtuals now work just like regular packages.
2021-02-17concretizer: remove rule generation code from concretizerTodd Gamblin1-70/+1
Our program only generates facts now, so remove all unused code related to generating cardinality constraints and rules.
2021-02-17concretizer: convert virtuals to facts; move all rules to `concretize.lp`Todd Gamblin2-116/+164
This converts the virtual handling in the new concretizer from already-ground rules to facts. This is the last thing that needs to be refactored, and it converts the entire concretizer to just use facts. The previous way of handling virtuals hinged on rules involving `single_provider_for` facts that were tied to the virtual and a version range. The new method uses the condition pattern we've been using for dependencies, externals, and conflicts. To handle virtuals as conditions, we impose constraints on "fake" virtual specs in the logic program. i.e., `version_satisfies("mpi", "2.0:", "2.0")` is legal whereas before we wouldn't have seen something like this. Currently, constriants are only handled on versions -- we don't handle variants or anything else yet, but they key change here is that we *could*. For a long time, virtual handling in Spack has only dealt with versions, and we'd like to be able to handle variants as well. We could easily add an integrity constraint to handle variants like the one we use for versions. One issue with the implementation here is that virtual packages don't actually declare possible versions like regular packages do. To get around that, we implement an integrity constraint like this: :- virtual_node(Virtual), version_satisfies(Virtual, V1), version_satisfies(Virtual, V2), not version_constraint_satisfies(Virtual, V1, V2). This requires us to compare every version constraint to every other, both in program generation and within the concretizer -- so there's a potentially quadratic evaluation time on virtual constraints because we don't have a real version to "anchor" things to. We just say that all the constraints need to agree for the virtual constraint to hold. We can investigate adding synthetic versions for virtuals in the future, to speed this up.
2021-02-17concretizer: consolidate handling of virtuals into spec_clausesTodd Gamblin1-26/+20
2021-02-17concretizer: make _condtion_id_counter an iteratorTodd Gamblin1-18/+18
2021-02-17concretizer: more detailed section headers in concretize.lpTodd Gamblin1-51/+69
2021-02-17bugfix: infinite loop when building a set from incomplete specs (#20649)Todd Gamblin1-1/+6
This code in `SpecBuilder.build_specs()` introduced in #20203, can loop seemingly interminably for very large specs: ```python set([spec.root for spec in self._specs.values()]) ``` It's deceptive, because it seems like there must be an issue with `spec.root`, but that works fine. It's building the set afterwards that takes forever, at least on `r-rminer`. Currently if you try running `spack solve r-rminer`, it loops infinitely and spins up your fan. The issue (I think) is that the spec is not yet complete when this is run, and something is going wrong when constructing and comparing so many values produced by `_cmp_key()`. We can investigate the efficiency of `_cmp_key()` separately, but for now, the fix is: ```python roots = [spec.root for spec in self._specs.values()] roots = dict((id(r), r) for r in roots) ``` We know the specs in `self._specs` are distinct (they just came out of the solver), so we can just use their `id()` to unique them here. This gets rid of the infinite loop.
2021-02-17concretizer: generate facts for externalsMassimiliano Culpo3-41/+42
Generate only facts for external specs. Substitute the use of already grounded rules with non-grounded rules in concretize.lp
2021-02-17bugfix: do not write empty default dicts/lists in envs (#20526)Greg Becker3-47/+38
Environment yaml files should not have default values written to them. To accomplish this, we change the validator to not add the default values to yaml. We rely on the code to set defaults for all values (and use defaulting getters like dict.get(key, default)). Includes regression test.
2021-02-17Add Intel oneAPI packages (#20411)Robert Cohn4-20/+93
This creates a set of packages which all use the same script to install components of Intel oneAPI. This includes: * An inheritable IntelOneApiPackage which knows how to invoke the installation script based on which components are requested * For components which include headers/libraries, an inheritable IntelOneApiLibraryPackage is provided to locate them * Individual packages for DAL, DNN, TBB, etc. * A package for the Intel oneAPI compilers (icx/ifx). This also includes icc/ifortran but these are not currently detected in this PR
2021-02-17concretizer: refactor conditional rules to be less repetitious (#20507)Todd Gamblin2-89/+60
We have to repeat all the spec attributes in a number of places in `concretize.lp`, and Spack has a fair number of spec attributes. If we instead add some rules up front that establish equivalencies like this: ``` node(Package) :- attr("node", Package). attr("node", Package) :- node(Package). version(Package, Version) :- attr("version", Package, Version). attr("version", Package, Version) :- version(Package, Version). ``` We can rewrite most of the repetitive conditions with `attr` and repeat only for each arity (there are only 3 arities for spec attributes so far) as opposed to each spec attribute. This makes the logic easier to read and the rules easier to follow. Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2021-02-17concretizer: optimize loop on compiler versionMassimiliano Culpo2-13/+15
Similar to the optimization on platform
2021-02-17concretizer: optimized loop on node platformsMassimiliano Culpo1-3/+3
We can speed-up the computation by avoiding a double loop in a cardinality constraint and enforcing the rule instead as an integrity constraint.
2021-02-17concretizer: fix failing unit testsMassimiliano Culpo2-4/+18
2021-02-17concretizer: emit facts for integrity constraintsMassimiliano Culpo2-37/+40
2021-02-17concretizer: emit facts for constraints on imposed dependenciesMassimiliano Culpo2-38/+85
2021-02-17concretizer: avoid redundant grounding on dependency typesMassimiliano Culpo2-27/+24
2021-02-17concretizer: move conditional dependency logic into `concretize.lp`Todd Gamblin2-18/+60
Continuing to convert everything in `asp.py` into facts, make the generation of ground rules for conditional dependencies use facts, and move the semantics into `concretize.lp`. This is probably the most complex logic in Spack, as dependencies can be conditional on anything, and we need conditional ASP rules to accumulate and map all the dependency conditions to spec attributes. The logic looks complicated, but essentially it accumulates any constraints associated with particular conditions into a fact associated with the condition by id. Then, if *any* condition id's fact is True, we trigger the dependency. This simplifies the way `declared_dependency()` works -- the dependency is now declared regardless of whether it is conditional, and the conditions are handled by `dependency_condition()` facts.
2021-02-17concretizer: spec_clauses should traverse dependenciesTodd Gamblin1-20/+24
There are currently no places where we do not want to traverse dependencies in `spec_clauses()`, so simplify the logic by consolidating `spec_traverse_clauses()` with `spec_clauses()`.
2021-02-17concretizer: pull _develop_specs_from_env out of main setup loopTodd Gamblin1-7/+10
2021-02-17concretizer: add #defined statements to avoid warnings.Todd Gamblin1-0/+2
`version_satisfies/2` and `node_compiler_version_satisfies/3` are generated but need `#defined` directives to avoid " info: atom does not occur in any rule head:" warnings.
2021-02-17asp: memoize the list of all target_specs to speed-up setup phase (#20473)Massimiliano Culpo1-4/+11
* asp: memoize the list of all target_specs to speed-up setup phase * asp: memoize using a cache per solver object
2021-02-17ci: fixes for compiler bootstrapping (#17563)Scott Wittenburg7-63/+241
This PR addresses a number of issues related to compiler bootstrapping. Specifically: 1. Collect compilers to be bootstrapped while queueing in installer Compiler tasks currently have an incomplete list in their task.dependents, making those packages fail to install as they think they have not all their dependencies installed. This PR collects the dependents and sets them on compiler tasks. 2. allow boostrapped compilers to back off target Bootstrapped compilers may be built with a compiler that doesn't support the target used by the rest of the spec. Allow them to build with less aggressive target optimization settings. 3. Support for target ranges Backing off the target necessitates computing target ranges, so make Spack handle those properly. Notably, this adds an intersection method for target ranges and fixes the way ranges are satisfied and constrained on Spec objects. This PR also: - adds testing - improves concretizer handling of target ranges Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com> Co-authored-by: Gregory Becker <becker33@llnl.gov> Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2021-02-17unit-tests: ensure that installed packages can be reused (#20307)Massimiliano Culpo1-0/+84
refers #20292 Added a unit test that ensures we can reuse installed packages even if in the repository variants have been removed or added.
2021-02-17Fix comparisons for abstract specs (#20341)Greg Becker2-3/+29
bug only relevant for python3
2021-02-17concretizer: don't use one_of_iff for range constraints (#20383)Todd Gamblin2-58/+50
Currently, version range constraints, compiler version range constraints, and target range constraints are implemented by generating ground rules from `asp.py`, via `one_of_iff()`. The rules look like this: ``` version_satisfies("python", "2.6:") :- 1 { version("python", "2.4"); ... } 1. 1 { version("python", "2.4"); ... } 1. :- version_satisfies("python", "2.6:"). ``` So, `version_satisfies(Package, Constraint)` is true if and only if the package is assigned a version that satisfies the constraint. We precompute the set of known versions that satisfy the constraint, and generate the rule in `SpackSolverSetup`. We shouldn't need to generate already-ground rules for this. Rather, we should leave it to the grounder to do the grounding, and generate facts so that the constraint semantics can be defined in `concretize.lp`. We can replace rules like the ones above with facts like this: ``` version_satisfies("python", "2.6:", "2.4") ``` And ground them in `concretize.lp` with rules like this: ``` 1 { version(Package, Version) : version_satisfies(Package, Constraint, Version) } 1 :- version_satisfies(Package, Constraint). version_satisfies(Package, Constraint) :- version(Package, Version), version_satisfies(Package, Constraint, Version). ``` The top rule is the same as before. It makes conditional dependencies and other places where version constraints are used work properly. Note that we do not need the cardinality constraint for the second rule -- we already have rules saying there can be only one version assigned to a package, so we can just infer from `version/2` `version_satisfies/3`. This form is also safe for grounding -- If we used the original form we'd have unsafe variables like `Constraint` and `Package` -- the original form only really worked when specified as ground to begin with. - [x] use facts instead of generating rules for package version constraints - [x] use facts instead of generating rules for compiler version constraints - [x] use facts instead of generating rules for target range constraints - [x] remove `one_of_iff()` and `iff()` as they're no longer needed
2021-02-17package sanity: ensure all variant defaults are allowed values (#20373)Massimiliano Culpo3-5/+16