summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2021-02-17concretizer: make _condtion_id_counter an iteratorTodd Gamblin1-18/+18
2021-02-17concretizer: more detailed section headers in concretize.lpTodd Gamblin1-51/+69
2021-02-17bugfix: infinite loop when building a set from incomplete specs (#20649)Todd Gamblin1-1/+6
This code in `SpecBuilder.build_specs()` introduced in #20203, can loop seemingly interminably for very large specs: ```python set([spec.root for spec in self._specs.values()]) ``` It's deceptive, because it seems like there must be an issue with `spec.root`, but that works fine. It's building the set afterwards that takes forever, at least on `r-rminer`. Currently if you try running `spack solve r-rminer`, it loops infinitely and spins up your fan. The issue (I think) is that the spec is not yet complete when this is run, and something is going wrong when constructing and comparing so many values produced by `_cmp_key()`. We can investigate the efficiency of `_cmp_key()` separately, but for now, the fix is: ```python roots = [spec.root for spec in self._specs.values()] roots = dict((id(r), r) for r in roots) ``` We know the specs in `self._specs` are distinct (they just came out of the solver), so we can just use their `id()` to unique them here. This gets rid of the infinite loop.
2021-02-17concretizer: generate facts for externalsMassimiliano Culpo3-41/+42
Generate only facts for external specs. Substitute the use of already grounded rules with non-grounded rules in concretize.lp
2021-02-17bugfix: do not write empty default dicts/lists in envs (#20526)Greg Becker3-47/+38
Environment yaml files should not have default values written to them. To accomplish this, we change the validator to not add the default values to yaml. We rely on the code to set defaults for all values (and use defaulting getters like dict.get(key, default)). Includes regression test.
2021-02-17Add Intel oneAPI packages (#20411)Robert Cohn4-20/+93
This creates a set of packages which all use the same script to install components of Intel oneAPI. This includes: * An inheritable IntelOneApiPackage which knows how to invoke the installation script based on which components are requested * For components which include headers/libraries, an inheritable IntelOneApiLibraryPackage is provided to locate them * Individual packages for DAL, DNN, TBB, etc. * A package for the Intel oneAPI compilers (icx/ifx). This also includes icc/ifortran but these are not currently detected in this PR
2021-02-17concretizer: refactor conditional rules to be less repetitious (#20507)Todd Gamblin2-89/+60
We have to repeat all the spec attributes in a number of places in `concretize.lp`, and Spack has a fair number of spec attributes. If we instead add some rules up front that establish equivalencies like this: ``` node(Package) :- attr("node", Package). attr("node", Package) :- node(Package). version(Package, Version) :- attr("version", Package, Version). attr("version", Package, Version) :- version(Package, Version). ``` We can rewrite most of the repetitive conditions with `attr` and repeat only for each arity (there are only 3 arities for spec attributes so far) as opposed to each spec attribute. This makes the logic easier to read and the rules easier to follow. Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2021-02-17concretizer: optimize loop on compiler versionMassimiliano Culpo2-13/+15
Similar to the optimization on platform
2021-02-17concretizer: optimized loop on node platformsMassimiliano Culpo1-3/+3
We can speed-up the computation by avoiding a double loop in a cardinality constraint and enforcing the rule instead as an integrity constraint.
2021-02-17concretizer: fix failing unit testsMassimiliano Culpo2-4/+18
2021-02-17concretizer: emit facts for integrity constraintsMassimiliano Culpo2-37/+40
2021-02-17concretizer: emit facts for constraints on imposed dependenciesMassimiliano Culpo2-38/+85
2021-02-17concretizer: avoid redundant grounding on dependency typesMassimiliano Culpo2-27/+24
2021-02-17concretizer: move conditional dependency logic into `concretize.lp`Todd Gamblin2-18/+60
Continuing to convert everything in `asp.py` into facts, make the generation of ground rules for conditional dependencies use facts, and move the semantics into `concretize.lp`. This is probably the most complex logic in Spack, as dependencies can be conditional on anything, and we need conditional ASP rules to accumulate and map all the dependency conditions to spec attributes. The logic looks complicated, but essentially it accumulates any constraints associated with particular conditions into a fact associated with the condition by id. Then, if *any* condition id's fact is True, we trigger the dependency. This simplifies the way `declared_dependency()` works -- the dependency is now declared regardless of whether it is conditional, and the conditions are handled by `dependency_condition()` facts.
2021-02-17concretizer: spec_clauses should traverse dependenciesTodd Gamblin1-20/+24
There are currently no places where we do not want to traverse dependencies in `spec_clauses()`, so simplify the logic by consolidating `spec_traverse_clauses()` with `spec_clauses()`.
2021-02-17concretizer: pull _develop_specs_from_env out of main setup loopTodd Gamblin1-7/+10
2021-02-17concretizer: add #defined statements to avoid warnings.Todd Gamblin1-0/+2
`version_satisfies/2` and `node_compiler_version_satisfies/3` are generated but need `#defined` directives to avoid " info: atom does not occur in any rule head:" warnings.
2021-02-17asp: memoize the list of all target_specs to speed-up setup phase (#20473)Massimiliano Culpo1-4/+11
* asp: memoize the list of all target_specs to speed-up setup phase * asp: memoize using a cache per solver object
2021-02-17ci: fixes for compiler bootstrapping (#17563)Scott Wittenburg7-63/+241
This PR addresses a number of issues related to compiler bootstrapping. Specifically: 1. Collect compilers to be bootstrapped while queueing in installer Compiler tasks currently have an incomplete list in their task.dependents, making those packages fail to install as they think they have not all their dependencies installed. This PR collects the dependents and sets them on compiler tasks. 2. allow boostrapped compilers to back off target Bootstrapped compilers may be built with a compiler that doesn't support the target used by the rest of the spec. Allow them to build with less aggressive target optimization settings. 3. Support for target ranges Backing off the target necessitates computing target ranges, so make Spack handle those properly. Notably, this adds an intersection method for target ranges and fixes the way ranges are satisfied and constrained on Spec objects. This PR also: - adds testing - improves concretizer handling of target ranges Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com> Co-authored-by: Gregory Becker <becker33@llnl.gov> Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2021-02-17unit-tests: ensure that installed packages can be reused (#20307)Massimiliano Culpo1-0/+84
refers #20292 Added a unit test that ensures we can reuse installed packages even if in the repository variants have been removed or added.
2021-02-17Fix comparisons for abstract specs (#20341)Greg Becker2-3/+29
bug only relevant for python3
2021-02-17concretizer: don't use one_of_iff for range constraints (#20383)Todd Gamblin2-58/+50
Currently, version range constraints, compiler version range constraints, and target range constraints are implemented by generating ground rules from `asp.py`, via `one_of_iff()`. The rules look like this: ``` version_satisfies("python", "2.6:") :- 1 { version("python", "2.4"); ... } 1. 1 { version("python", "2.4"); ... } 1. :- version_satisfies("python", "2.6:"). ``` So, `version_satisfies(Package, Constraint)` is true if and only if the package is assigned a version that satisfies the constraint. We precompute the set of known versions that satisfy the constraint, and generate the rule in `SpackSolverSetup`. We shouldn't need to generate already-ground rules for this. Rather, we should leave it to the grounder to do the grounding, and generate facts so that the constraint semantics can be defined in `concretize.lp`. We can replace rules like the ones above with facts like this: ``` version_satisfies("python", "2.6:", "2.4") ``` And ground them in `concretize.lp` with rules like this: ``` 1 { version(Package, Version) : version_satisfies(Package, Constraint, Version) } 1 :- version_satisfies(Package, Constraint). version_satisfies(Package, Constraint) :- version(Package, Version), version_satisfies(Package, Constraint, Version). ``` The top rule is the same as before. It makes conditional dependencies and other places where version constraints are used work properly. Note that we do not need the cardinality constraint for the second rule -- we already have rules saying there can be only one version assigned to a package, so we can just infer from `version/2` `version_satisfies/3`. This form is also safe for grounding -- If we used the original form we'd have unsafe variables like `Constraint` and `Package` -- the original form only really worked when specified as ground to begin with. - [x] use facts instead of generating rules for package version constraints - [x] use facts instead of generating rules for compiler version constraints - [x] use facts instead of generating rules for target range constraints - [x] remove `one_of_iff()` and `iff()` as they're no longer needed
2021-02-17package sanity: ensure all variant defaults are allowed values (#20373)Massimiliano Culpo3-5/+16
2021-02-17concretizer: remove clingo command-line driver (#20362)Todd Gamblin1-216/+0
I was keeping the old `clingo` driver code around in case we had to run using the command line tool instad of through the Python interface. So far, the command line is faster than running through Python, but I'm working on fixing that. I found that if I do this: ```python control = clingo.Control() control.load("concretize.lp") control.load("hdf5.lp") # code from spack solve --show asp hdf5 control.load("display.lp") control.ground([("base", [])]) control.solve(...) ``` It's just as fast as the command line tool. So we can always generate the code and load it manually if we need to -- we don't need two drivers for clingo. Given that the python interface is also the only way to get unsat cores, I think we pretty much have to use it. So, I'm removing the old command line driver and other unused code. We can dig it up again from the history if it is needed.
2021-02-17Tests: enable re-use of post-install tests in smoke tests (#20298)Tamara Dahlgren1-1/+5
2021-02-17concretizer: try hard to obtain all needed variant_possible_value()'s (#20102)Andrew W Elble5-10/+52
Track all the variant values mentioned when emitting constraints, validate them and emit a fact that allows them as possible values. This modification ensures that open-ended variants (variants accepting any string or any integer) are projected to the finite set of values that are relevant for this concretization.
2021-02-17bugfix: work around issue handling packages not in any repoMassimiliano Culpo2-0/+7
2021-02-17concretizer: refactor handling of special variants dev_build and patchesTodd Gamblin2-50/+55
Other parts of the concretizer code build up lists of things we can't know without traversing all specs and packages, and they output these list at the very end. The code for this for variant values from spec literals was intertwined with the code for traversing the input specs. This only covers the input specs and misses variant values that might come from directives in packages. - [x] move ad-hoc value handling code into spec_clauses so we do it in one place for CLI and packages - [x] move handling of `variant_possible_value`, etc. into `concretize.lp`, where we can automatically infer variant existence more concisely. - [x] simplify/clarify some of the code for variants in `spec_clauses()`
2021-02-17concretizer: each external version is allowed by definition (#20247)Massimiliano Culpo3-0/+26
Registering external versions among the lists of allowed ones generates the correct rules for `version_satisfies`
2021-02-17concretizer: restrict maximizing variant values to MV variants (#20194)Massimiliano Culpo2-2/+10
2021-02-17environment installs: fix reporting. (#20004)Matthias Wolf1-1/+1
PR #15702 changed the invocation of the report context when installing specs, do the same when building environments.
2021-02-17avoid circular import (#20236)Greg Becker1-1/+1
2021-02-17concretizer: call inject_patches_variants() on the roots of the specs (#20203)Andrew W Elble3-4/+14
As was done in the old concretizer. Fixes an issue where conditionally patched dependencies did not show up in spec (gdal+jasper)
2021-02-17concretizer: try hard to infer the real version of compilers (#20099)Massimiliano Culpo3-3/+53
fixes #20055 Compiler with custom versions like gcc@foo are not currently matched to the appropriate targets. This is because the version of spec doesn't match the "real" version of the compiler. This PR replicates the strategy used in the original concretizer to deal with that and tries to detect the real version of compilers if the version in the spec returns no results.
2021-02-17Fix hipcc once more (#20095)Harmen Stoppels2-27/+22
2021-02-17concretizer: don't optimize emitting version_satisfies() (#20128)Andrew W Elble2-4/+5
When all versions were allowed a version_satisfies rule was not emitted, and this caused conditional directives to fail.
2021-02-17spec: return early from concretization if a spec is already concrete (#20196)Massimiliano Culpo1-0/+3
2021-02-17concretizer: remove ad-hoc rule for external packages (#20193)Massimiliano Culpo2-7/+10
fixes #20040 Matching compilers among nodes has been prioritized in #20020. Selection of default variants has been tuned in #20182. With this setup there is no need to have an ad-hoc rule for external packages. On the contrary it should be removed to prefer having default variant values over more external nodes in the DAG.
2021-02-17concretizer: swap priority of selecting provider and default variant (#20182)Massimiliano Culpo2-13/+27
refers #20040 Before this PR optimization rules would have selected default providers at a higher priority than default variants. Here we swap this priority and we consider variants that are forced by any means (root spec or spec in depends_on clause) the same as if they were with a default value. This prevents the solver from avoiding expected configurations just because they contain directives like: depends_on('pkg+foo') and `+foo` is not the default variant value for pkg.
2021-02-17Typos: add missing closing parens (#20174)George Hartzell1-3/+3
2021-02-17concretizer: treat target ranges in directives correctly (#19988)Massimiliano Culpo2-1/+59
fixes #19981 This commit adds support for target ranges in directives, for instance: conflicts('+foo', when='target=x86_64:,aarch64:') If any target in a spec body is not a known target the following clause will be emitted: node_target_satisfies(Package, TargetConstraint) when traversing the spec and a definition of the clause will then be printed at the end similarly to what is done for package and compiler versions.
2021-02-17concretizer: prioritize matching compilers over newer versions (#20020)Massimiliano Culpo3-6/+32
fixes #20019 Before this modification having a newer version of a node came at higher priority in the optimization than having matching compilers. This could result in unexpected configurations for packages with conflict directives on compilers of the type: conflicts('%gcc@X.Y:', when='@:A.B') where changing the compiler for just that node is preferred to lower the node version to less than 'A.B'. Now the priority has been switched so the solver will try to lower the version of the nodes in question before changing their compiler.
2021-02-17concretizer: allow a bool to be passed as argument for tests dependencies ↵Massimiliano Culpo3-8/+54
(#20082) refers #20079 Added docstrings to 'concretize' and 'concretized' to document the format for tests. Added tests for the activation of test dependencies.
2021-02-17concretizer: treat conditional providers correctly (#20086)Massimiliano Culpo2-1/+18
refers #20040 This modification emits rules like: provides_virtual("netlib-lapack","blas") :- variant_value("netlib-lapack","external-blas","False"). for packages that provide virtual dependencies conditionally instead of a fact that doesn't account for the condition.
2021-02-17Docs: remove duplication in Command Reference (#20021)Adam J. Stewart2-2/+2
2021-02-17recognize macOS 11.1 as big sur (#20038)Martin Aumüller1-2/+6
Big Sur versions go 11.0, 11.0.1, 11.1 (vs. prior versions that only used the minor component) Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2021-02-17fix error handling for spack test results command (#19987)Greg Becker1-0/+1
2021-02-17hip support for umpire, chai, raja, camp (#19715)Danny Taller2-0/+139
* create HipPackage base class and do some refactoring * comments and added conflict to raja for openmp with hip
2021-02-17apple-clang: add correct path to compiler wrappers (#21662)Adam J. Stewart1-1/+1
Follow-up to #17110 ### Before ```bash CC=/Users/Adam/spack/lib/spack/env/clang/clang; export CC SPACK_CC=/usr/bin/clang; export SPACK_CC PATH=...:/Users/Adam/spack/lib/spack/env/apple-clang:/Users/Adam/spack/lib/spack/env/case-insensitive:/Users/Adam/spack/lib/spack/env:...; export PATH ``` ### After ```bash CC=/Users/Adam/spack/lib/spack/env/clang/clang; export CC SPACK_CC=/usr/bin/clang; export SPACK_CC PATH=...:/Users/Adam/spack/lib/spack/env/clang:/Users/Adam/spack/lib/spack/env/case-insensitive:/Users/Adam/spack/lib/spack/env:...; export PATH ``` `CC` and `SPACK_CC` were being set correctly, but `PATH` was using the name of the compiler `apple-clang` instead of `clang`. For most packages, since `CC` was set correctly, nothing broke. But for packages using `Makefiles` that set `CC` based on `which clang`, it was using the system compilers instead of the compiler wrappers. Discovered when working on `py-xgboost@0.90`. An alternative fix would be to copy the symlinks in `env/clang` to `env/apple-clang`. Let me know if you think there's a better way to do this, or to test this.
2021-02-16Delete unused "mutable_mock_store" fixture (#21632)Massimiliano Culpo1-31/+0
The fixture was introduced in #19690 maybe accidentally. It's not used in unit tests, and though it should be mutable it seems an exact copy of it's immutable version.