summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2020-12-24Remove more variables from build environment (#20156)Omri Mor1-2/+7
GCC looks for included files based on several env vars. Remove C_INCLUDE_PATH, CPLUS_INCLUDE_PATH, and OBJC_INCLUDE_PATH from the build environment to ensure it's clean and prevent accidental clobbering.
2020-12-23bugfix: do not write empty default dicts/lists in envs (#20526)Greg Becker3-47/+38
Environment yaml files should not have default values written to them. To accomplish this, we change the validator to not add the default values to yaml. We rely on the code to set defaults for all values (and use defaulting getters like dict.get(key, default)). Includes regression test.
2020-12-23concretizer: remove vestigial code and commentMassimiliano Culpo1-20/+0
2020-12-23style: ensure that all packages pass `spack style -a`Todd Gamblin3-5/+13
- fix trailing whitespace and other issues uncovered by better flake8 checking. - fix extra whitespace printed by `spack style` command
2020-12-23Add Intel oneAPI packages (#20411)Robert Cohn4-20/+93
This creates a set of packages which all use the same script to install components of Intel oneAPI. This includes: * An inheritable IntelOneApiPackage which knows how to invoke the installation script based on which components are requested * For components which include headers/libraries, an inheritable IntelOneApiLibraryPackage is provided to locate them * Individual packages for DAL, DNN, TBB, etc. * A package for the Intel oneAPI compilers (icx/ifx). This also includes icc/ifortran but these are not currently detected in this PR
2020-12-22add mypy to style checks; rename `spack flake8` to `spack style` (#20384)Tom Scogland42-304/+590
I lost my mind a bit after getting the completion stuff working and decided to get Mypy working for spack as well. This adds a `.mypy.ini` that checks all of the spack and llnl modules, though not yet packages, and fixes all of the identified missing types and type issues for the spack library. In addition to these changes, this includes: * rename `spack flake8` to `spack style` Aliases flake8 to style, and just runs flake8 as before, but with a warning. The style command runs both `flake8` and `mypy`, in sequence. Added --no-<tool> options to turn off one or the other, they are on by default. Fixed two issues caught by the tools. * stub typing module for python2.x We don't support typing in Spack for python 2.x. To allow 2.x to support `import typing` and `from typing import ...` without a try/except dance to support old versions, this adds a stub module *just* for python 2.x. Doing it this way means we can only reliably use all type hints in python3.7+, and mypi.ini has been updated to reflect that. * add non-default black check to spack style This is a first step to requiring black. It doesn't enforce it by default, but it will check it if requested. Currently enforcing the line length of 79 since that's what flake8 requires, but it's a bit odd for a black formatted project to be quite that narrow. All settings are in the style command since spack has no pyproject.toml and I don't want to add one until more discussion happens. Also re-format `style.py` since it no longer passed the black style check with the new length. * use style check in github action Update the style and docs action to use `spack style`, adding in mypy and black to the action even if it isn't running black right now.
2020-12-22concretizer: refactor conditional rules to be less repetitious (#20507)Todd Gamblin2-89/+60
We have to repeat all the spec attributes in a number of places in `concretize.lp`, and Spack has a fair number of spec attributes. If we instead add some rules up front that establish equivalencies like this: ``` node(Package) :- attr("node", Package). attr("node", Package) :- node(Package). version(Package, Version) :- attr("version", Package, Version). attr("version", Package, Version) :- version(Package, Version). ``` We can rewrite most of the repetitive conditions with `attr` and repeat only for each arity (there are only 3 arities for spec attributes so far) as opposed to each spec attribute. This makes the logic easier to read and the rules easier to follow. Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2020-12-22Refactor flake8 handling and tool compatibility (#20376)Tom Scogland1-229/+112
This PR does three related things to try to improve developer tooling quality of life: 1. Adds new options to `.flake8` so it applies the rules of both `.flake8` and `.flake_package` based on paths in the repository. 2. Adds a re-factoring of the `spack flake8` logic into a flake8 plugin so using flake8 directly, or through editor or language server integration, only reports errors that `spack flake8` would. 3. Allows star import of `spack.pkgkit` in packages, since this is now the thing that needs to be imported for completion to work correctly in package files, it's nice to be able to do that. I'm sorely tempted to sed over the whole repository and put `from spack.pkgkit import *` in every package, but at least being allowed to do it on a per-package basis helps. As an example of what the result of this is: ``` ~/Workspace/Projects/spack/spack develop* ⇣ ❯ flake8 --format=pylint ./var/spack/repos/builtin/packages/kripke/package.py ./var/spack/repos/builtin/packages/kripke/package.py:6: [F403] 'from spack.pkgkit import *' used; unable to detect undefined names ./var/spack/repos/builtin/packages/kripke/package.py:25: [E501] line too long (88 > 79 characters) ~/Workspace/Projects/spack/spack refactor-flake8* 1 ❯ flake8 --format=spack ./var/spack/repos/builtin/packages/kripke/package.py ~/Workspace/Projects/spack/spack refactor-flake8* ❯ flake8 ./var/spack/repos/builtin/packages/kripke/package.py ``` * qa/flake8: update .flake8, spack formatter plugin Adds: * Modern flake8 settings for per-path/glob error ignores, allows packages to use the same `.flake8` as the rest of spack * A spack formatter plugin to flake8 that implements the behavior of `spack flake8` for direct invocations. Makes integration with developer tooling nicer, linting with flake8 reports only errors that `spack flake8` would report. Using pyls and pyls-flake8, or any other non-format-dependent flake8 integration, now works with spack's rules. * qa/flake8: allow star import of spack.pkgkit To get working completion of directives and spack components it's necessary to import the contents of spack.pkgkit. At the moment doing this makes flake8 displeased. For now, allow spack.pkgkit and spack both, next step is to ban spack * and require spack.pkgkit *. * first cut at refactoring spack flake8 This version still copies all of the files to be checked as befire, and some other things that probably aren't necessary, but it relies on the spack formatter plugin to implement the ignore logic. * keep flake8 from rejecting itself * remove separate packages flake8 config * fix failures from too many files I ran into this in the PR converting pkgkit to std. The solution in that branch does not work in all cases as it turns out, and all the workarounds I tried to use generated configs to get a single invocation of flake8 with a filename optoion to work failed. It's an astonishingly frustrating config option. Regardless, this removes all temporary file creation from the command and relies on the plugin instead. To work around the huge number of files in spack and still allow the command to control what gets checked, it scans files in batches of 100. This is a completely arbitrary number but was chosen to be safely under common line-length limits. One side-effect of this is that every 100 files the command will produce output, rather than only at the end, which doesn't seem like a terrible thing.
2020-12-22concretizer: optimize loop on compiler versionMassimiliano Culpo2-13/+15
Similar to the optimization on platform
2020-12-22concretizer: optimized loop on node platformsMassimiliano Culpo1-3/+3
We can speed-up the computation by avoiding a double loop in a cardinality constraint and enforcing the rule instead as an integrity constraint.
2020-12-20concretizer: fix failing unit testsMassimiliano Culpo2-4/+18
2020-12-20concretizer: emit facts for integrity constraintsMassimiliano Culpo2-37/+40
2020-12-20concretizer: emit facts for constraints on imposed dependenciesMassimiliano Culpo2-38/+85
2020-12-20concretizer: avoid redundant grounding on dependency typesMassimiliano Culpo2-27/+24
2020-12-20concretizer: move conditional dependency logic into `concretize.lp`Todd Gamblin2-18/+60
Continuing to convert everything in `asp.py` into facts, make the generation of ground rules for conditional dependencies use facts, and move the semantics into `concretize.lp`. This is probably the most complex logic in Spack, as dependencies can be conditional on anything, and we need conditional ASP rules to accumulate and map all the dependency conditions to spec attributes. The logic looks complicated, but essentially it accumulates any constraints associated with particular conditions into a fact associated with the condition by id. Then, if *any* condition id's fact is True, we trigger the dependency. This simplifies the way `declared_dependency()` works -- the dependency is now declared regardless of whether it is conditional, and the conditions are handled by `dependency_condition()` facts.
2020-12-20concretizer: spec_clauses should traverse dependenciesTodd Gamblin1-20/+24
There are currently no places where we do not want to traverse dependencies in `spec_clauses()`, so simplify the logic by consolidating `spec_traverse_clauses()` with `spec_clauses()`.
2020-12-20concretizer: pull _develop_specs_from_env out of main setup loopTodd Gamblin1-7/+10
2020-12-20concretizer: add #defined statements to avoid warnings.Todd Gamblin1-0/+2
`version_satisfies/2` and `node_compiler_version_satisfies/3` are generated but need `#defined` directives to avoid " info: atom does not occur in any rule head:" warnings.
2020-12-19asp: memoize the list of all target_specs to speed-up setup phase (#20473)Massimiliano Culpo1-4/+11
* asp: memoize the list of all target_specs to speed-up setup phase * asp: memoize using a cache per solver object
2020-12-18ci: fixes for compiler bootstrapping (#17563)Scott Wittenburg7-63/+241
This PR addresses a number of issues related to compiler bootstrapping. Specifically: 1. Collect compilers to be bootstrapped while queueing in installer Compiler tasks currently have an incomplete list in their task.dependents, making those packages fail to install as they think they have not all their dependencies installed. This PR collects the dependents and sets them on compiler tasks. 2. allow boostrapped compilers to back off target Bootstrapped compilers may be built with a compiler that doesn't support the target used by the rest of the spec. Allow them to build with less aggressive target optimization settings. 3. Support for target ranges Backing off the target necessitates computing target ranges, so make Spack handle those properly. Notably, this adds an intersection method for target ranges and fixes the way ranges are satisfied and constrained on Spec objects. This PR also: - adds testing - improves concretizer handling of target ranges Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com> Co-authored-by: Gregory Becker <becker33@llnl.gov> Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2020-12-17View location resolve environment variables (#20420)psakievich1-1/+2
2020-12-17docs: add single node concurrent build example (#20416)Tamara Dahlgren1-4/+11
2020-12-17unit-tests: ensure that installed packages can be reused (#20307)Massimiliano Culpo1-0/+84
refers #20292 Added a unit test that ensures we can reuse installed packages even if in the repository variants have been removed or added.
2020-12-16PythonPackage: add import module smoke tests (#20023)Adam J. Stewart5-128/+126
2020-12-16docs: fix spack install debug arg order (#20428)Greg Becker1-4/+3
2020-12-16Docs: add more Command Reference links to spack test (#20413)Adam J. Stewart1-2/+12
2020-12-16docs: fix spack command for unit-test pytest help (#20415)Tamara Dahlgren1-1/+1
2020-12-15Fix comparisons for abstract specs (#20341)Greg Becker2-3/+29
bug only relevant for python3
2020-12-15concretizer: don't use one_of_iff for range constraints (#20383)Todd Gamblin2-58/+50
Currently, version range constraints, compiler version range constraints, and target range constraints are implemented by generating ground rules from `asp.py`, via `one_of_iff()`. The rules look like this: ``` version_satisfies("python", "2.6:") :- 1 { version("python", "2.4"); ... } 1. 1 { version("python", "2.4"); ... } 1. :- version_satisfies("python", "2.6:"). ``` So, `version_satisfies(Package, Constraint)` is true if and only if the package is assigned a version that satisfies the constraint. We precompute the set of known versions that satisfy the constraint, and generate the rule in `SpackSolverSetup`. We shouldn't need to generate already-ground rules for this. Rather, we should leave it to the grounder to do the grounding, and generate facts so that the constraint semantics can be defined in `concretize.lp`. We can replace rules like the ones above with facts like this: ``` version_satisfies("python", "2.6:", "2.4") ``` And ground them in `concretize.lp` with rules like this: ``` 1 { version(Package, Version) : version_satisfies(Package, Constraint, Version) } 1 :- version_satisfies(Package, Constraint). version_satisfies(Package, Constraint) :- version(Package, Version), version_satisfies(Package, Constraint, Version). ``` The top rule is the same as before. It makes conditional dependencies and other places where version constraints are used work properly. Note that we do not need the cardinality constraint for the second rule -- we already have rules saying there can be only one version assigned to a package, so we can just infer from `version/2` `version_satisfies/3`. This form is also safe for grounding -- If we used the original form we'd have unsafe variables like `Constraint` and `Package` -- the original form only really worked when specified as ground to begin with. - [x] use facts instead of generating rules for package version constraints - [x] use facts instead of generating rules for compiler version constraints - [x] use facts instead of generating rules for target range constraints - [x] remove `one_of_iff()` and `iff()` as they're no longer needed
2020-12-15Bugfix/docs: correct and expand smoke test documentation (#20278)Tamara Dahlgren1-70/+151
2020-12-15outputs: restore default output of fetch/build/total times (#20394)Tamara Dahlgren1-5/+4
2020-12-15package sanity: ensure all variant defaults are allowed values (#20373)Massimiliano Culpo3-5/+16
2020-12-14concretizer: remove clingo command-line driver (#20362)Todd Gamblin1-216/+0
I was keeping the old `clingo` driver code around in case we had to run using the command line tool instad of through the Python interface. So far, the command line is faster than running through Python, but I'm working on fixing that. I found that if I do this: ```python control = clingo.Control() control.load("concretize.lp") control.load("hdf5.lp") # code from spack solve --show asp hdf5 control.load("display.lp") control.ground([("base", [])]) control.solve(...) ``` It's just as fast as the command line tool. So we can always generate the code and load it manually if we need to -- we don't need two drivers for clingo. Given that the python interface is also the only way to get unsat cores, I think we pretty much have to use it. So, I'm removing the old command line driver and other unused code. We can dig it up again from the history if it is needed.
2020-12-11Debugging support: fix compiler wrapper log on Mac OS (#20333)Ben Cowan1-0/+2
This fixes a logging error observed on macOS 11.0.1 (Big Sur). When performing a Spack install in debugging mode (e.g. `spack -d install py-scipy`) Spack is supposed to write a log of compiler wrapper command line invocations to the current working directory. Due to a regression error introduced by #18205, these files were no-longer generated, and Spack was printing errors such as "No such file or directory: None/." This is because the log file directory gets set from `spack.main.spack_working_dir`, but that variable is not set in the spawned process. This PR ensures that the working directory (at the time of the "spack install" invocation) is persisted to the subprocess.
2020-12-10Tests: enable re-use of post-install tests in smoke tests (#20298)Tamara Dahlgren1-1/+5
2020-12-08Command Reference: add link to spack test docs (#20054)Adam J. Stewart1-0/+2
2020-12-08concretizer: try hard to obtain all needed variant_possible_value()'s (#20102)Andrew W Elble5-10/+52
Track all the variant values mentioned when emitting constraints, validate them and emit a fact that allows them as possible values. This modification ensures that open-ended variants (variants accepting any string or any integer) are projected to the finite set of values that are relevant for this concretization.
2020-12-07Compiler wrapper linting (#20249)Harmen Stoppels1-4/+4
* Fix duplicate entries in case * make sure the arg is not interpreted as two items in a list * use -n over ! -z
2020-12-07bugfix: work around issue handling packages not in any repoMassimiliano Culpo2-0/+7
2020-12-07concretizer: refactor handling of special variants dev_build and patchesTodd Gamblin2-50/+55
Other parts of the concretizer code build up lists of things we can't know without traversing all specs and packages, and they output these list at the very end. The code for this for variant values from spec literals was intertwined with the code for traversing the input specs. This only covers the input specs and misses variant values that might come from directives in packages. - [x] move ad-hoc value handling code into spec_clauses so we do it in one place for CLI and packages - [x] move handling of `variant_possible_value`, etc. into `concretize.lp`, where we can automatically infer variant existence more concisely. - [x] simplify/clarify some of the code for variants in `spec_clauses()`
2020-12-07Add "spack versions --new" flag to only show new versions (#20030)vvolkl2-16/+61
* [cmd versions] add spack versions --new flag to only fetch new versions format [cmd versions] rename --latest to --newest and add --remote-only [cmd versions] add tests for --remote-only and --new format [cmd versions] update shell tab completion [cmd versions] remove test for --remote-only --new which gives empty output [cmd versions] final rename format * add brillig mock package * add test for spack versions --new * [brillig] format * [versions] increase test coverage * Update lib/spack/spack/cmd/versions.py Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com> * Update lib/spack/spack/cmd/versions.py Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com> Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2020-12-06concretizer: each external version is allowed by definition (#20247)Massimiliano Culpo3-0/+26
Registering external versions among the lists of allowed ones generates the correct rules for `version_satisfies`
2020-12-04Also allow --rpath as rpath linker flags (#18473)Harmen Stoppels2-4/+13
2020-12-04concretizer: restrict maximizing variant values to MV variants (#20194)Massimiliano Culpo2-2/+10
2020-12-03allow install of build-deps from cache via --include-build-deps switch (#19955)eugeneswalker2-2/+8
* allow install of build-deps from cache via --include-build-deps switch * make clear that --include-build-deps is useful for CI pipeline troubleshooting
2020-12-03environment installs: fix reporting. (#20004)Matthias Wolf1-1/+1
PR #15702 changed the invocation of the report context when installing specs, do the same when building environments.
2020-12-03avoid circular import (#20236)Greg Becker1-1/+1
2020-12-03concretizer: call inject_patches_variants() on the roots of the specs (#20203)Andrew W Elble3-4/+14
As was done in the old concretizer. Fixes an issue where conditionally patched dependencies did not show up in spec (gdal+jasper)
2020-12-02Add CARE package, fixes for ROCmPackage and subclasses (#20070)Danny Taller1-0/+2
* use develop version of blt with fixes for rocm * package updates for care+rocm * fixes for plain cpu build * add camp dependency on raja
2020-12-02concretizer: try hard to infer the real version of compilers (#20099)Massimiliano Culpo3-3/+53
fixes #20055 Compiler with custom versions like gcc@foo are not currently matched to the appropriate targets. This is because the version of spec doesn't match the "real" version of the compiler. This PR replicates the strategy used in the original concretizer to deal with that and tries to detect the real version of compilers if the version in the spec returns no results.