summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2020-12-14concretizer: remove clingo command-line driver (#20362)Todd Gamblin1-216/+0
I was keeping the old `clingo` driver code around in case we had to run using the command line tool instad of through the Python interface. So far, the command line is faster than running through Python, but I'm working on fixing that. I found that if I do this: ```python control = clingo.Control() control.load("concretize.lp") control.load("hdf5.lp") # code from spack solve --show asp hdf5 control.load("display.lp") control.ground([("base", [])]) control.solve(...) ``` It's just as fast as the command line tool. So we can always generate the code and load it manually if we need to -- we don't need two drivers for clingo. Given that the python interface is also the only way to get unsat cores, I think we pretty much have to use it. So, I'm removing the old command line driver and other unused code. We can dig it up again from the history if it is needed.
2020-12-11Debugging support: fix compiler wrapper log on Mac OS (#20333)Ben Cowan1-0/+2
This fixes a logging error observed on macOS 11.0.1 (Big Sur). When performing a Spack install in debugging mode (e.g. `spack -d install py-scipy`) Spack is supposed to write a log of compiler wrapper command line invocations to the current working directory. Due to a regression error introduced by #18205, these files were no-longer generated, and Spack was printing errors such as "No such file or directory: None/." This is because the log file directory gets set from `spack.main.spack_working_dir`, but that variable is not set in the spawned process. This PR ensures that the working directory (at the time of the "spack install" invocation) is persisted to the subprocess.
2020-12-10Tests: enable re-use of post-install tests in smoke tests (#20298)Tamara Dahlgren1-1/+5
2020-12-08Command Reference: add link to spack test docs (#20054)Adam J. Stewart1-0/+2
2020-12-08concretizer: try hard to obtain all needed variant_possible_value()'s (#20102)Andrew W Elble5-10/+52
Track all the variant values mentioned when emitting constraints, validate them and emit a fact that allows them as possible values. This modification ensures that open-ended variants (variants accepting any string or any integer) are projected to the finite set of values that are relevant for this concretization.
2020-12-07Compiler wrapper linting (#20249)Harmen Stoppels1-4/+4
* Fix duplicate entries in case * make sure the arg is not interpreted as two items in a list * use -n over ! -z
2020-12-07bugfix: work around issue handling packages not in any repoMassimiliano Culpo2-0/+7
2020-12-07concretizer: refactor handling of special variants dev_build and patchesTodd Gamblin2-50/+55
Other parts of the concretizer code build up lists of things we can't know without traversing all specs and packages, and they output these list at the very end. The code for this for variant values from spec literals was intertwined with the code for traversing the input specs. This only covers the input specs and misses variant values that might come from directives in packages. - [x] move ad-hoc value handling code into spec_clauses so we do it in one place for CLI and packages - [x] move handling of `variant_possible_value`, etc. into `concretize.lp`, where we can automatically infer variant existence more concisely. - [x] simplify/clarify some of the code for variants in `spec_clauses()`
2020-12-07Add "spack versions --new" flag to only show new versions (#20030)vvolkl2-16/+61
* [cmd versions] add spack versions --new flag to only fetch new versions format [cmd versions] rename --latest to --newest and add --remote-only [cmd versions] add tests for --remote-only and --new format [cmd versions] update shell tab completion [cmd versions] remove test for --remote-only --new which gives empty output [cmd versions] final rename format * add brillig mock package * add test for spack versions --new * [brillig] format * [versions] increase test coverage * Update lib/spack/spack/cmd/versions.py Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com> * Update lib/spack/spack/cmd/versions.py Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com> Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2020-12-06concretizer: each external version is allowed by definition (#20247)Massimiliano Culpo3-0/+26
Registering external versions among the lists of allowed ones generates the correct rules for `version_satisfies`
2020-12-04Also allow --rpath as rpath linker flags (#18473)Harmen Stoppels2-4/+13
2020-12-04concretizer: restrict maximizing variant values to MV variants (#20194)Massimiliano Culpo2-2/+10
2020-12-03allow install of build-deps from cache via --include-build-deps switch (#19955)eugeneswalker2-2/+8
* allow install of build-deps from cache via --include-build-deps switch * make clear that --include-build-deps is useful for CI pipeline troubleshooting
2020-12-03environment installs: fix reporting. (#20004)Matthias Wolf1-1/+1
PR #15702 changed the invocation of the report context when installing specs, do the same when building environments.
2020-12-03avoid circular import (#20236)Greg Becker1-1/+1
2020-12-03concretizer: call inject_patches_variants() on the roots of the specs (#20203)Andrew W Elble3-4/+14
As was done in the old concretizer. Fixes an issue where conditionally patched dependencies did not show up in spec (gdal+jasper)
2020-12-02Add CARE package, fixes for ROCmPackage and subclasses (#20070)Danny Taller1-0/+2
* use develop version of blt with fixes for rocm * package updates for care+rocm * fixes for plain cpu build * add camp dependency on raja
2020-12-02concretizer: try hard to infer the real version of compilers (#20099)Massimiliano Culpo3-3/+53
fixes #20055 Compiler with custom versions like gcc@foo are not currently matched to the appropriate targets. This is because the version of spec doesn't match the "real" version of the compiler. This PR replicates the strategy used in the original concretizer to deal with that and tries to detect the real version of compilers if the version in the spec returns no results.
2020-12-02Fix hipcc once more (#20095)Harmen Stoppels2-27/+22
2020-12-02concretizer: don't optimize emitting version_satisfies() (#20128)Andrew W Elble2-4/+5
When all versions were allowed a version_satisfies rule was not emitted, and this caused conditional directives to fail.
2020-12-01spec: return early from concretization if a spec is already concrete (#20196)Massimiliano Culpo1-0/+3
2020-12-01AOCC-2.3.0 is now added to spack (#20089)AMD Toolchain Support1-0/+5
* AOCC-2.3.0 is now added to spack Change-Id: I18fd9606e6fd9a288cc7dc6c6ead11ea17839a7c * Added flag and version tests for AOCC-2.3.0 * Addressed review comments Co-authored-by: vkallesh <Vijay-teekinavar.Kallesh@amd.com>
2020-12-01concretizer: remove ad-hoc rule for external packages (#20193)Massimiliano Culpo2-7/+10
fixes #20040 Matching compilers among nodes has been prioritized in #20020. Selection of default variants has been tuned in #20182. With this setup there is no need to have an ad-hoc rule for external packages. On the contrary it should be removed to prefer having default variant values over more external nodes in the DAG.
2020-12-01concretizer: swap priority of selecting provider and default variant (#20182)Massimiliano Culpo2-13/+27
refers #20040 Before this PR optimization rules would have selected default providers at a higher priority than default variants. Here we swap this priority and we consider variants that are forced by any means (root spec or spec in depends_on clause) the same as if they were with a default value. This prevents the solver from avoiding expected configurations just because they contain directives like: depends_on('pkg+foo') and `+foo` is not the default variant value for pkg.
2020-11-30Typos: add missing closing parens (#20174)George Hartzell1-3/+3
2020-11-30libQGLViewer: add new package (#20164)Adam J. Stewart2-4/+31
2020-11-27concretizer: treat target ranges in directives correctly (#19988)Massimiliano Culpo2-1/+59
fixes #19981 This commit adds support for target ranges in directives, for instance: conflicts('+foo', when='target=x86_64:,aarch64:') If any target in a spec body is not a known target the following clause will be emitted: node_target_satisfies(Package, TargetConstraint) when traversing the spec and a definition of the clause will then be printed at the end similarly to what is done for package and compiler versions.
2020-11-26archspec: added support for aocc (#20124)Massimiliano Culpo3-2/+145
2020-11-26concretizer: prioritize matching compilers over newer versions (#20020)Massimiliano Culpo3-6/+32
fixes #20019 Before this modification having a newer version of a node came at higher priority in the optimization than having matching compilers. This could result in unexpected configurations for packages with conflict directives on compilers of the type: conflicts('%gcc@X.Y:', when='@:A.B') where changing the compiler for just that node is preferred to lower the node version to less than 'A.B'. Now the priority has been switched so the solver will try to lower the version of the nodes in question before changing their compiler.
2020-11-26concretizer: allow a bool to be passed as argument for tests dependencies ↵Massimiliano Culpo3-8/+54
(#20082) refers #20079 Added docstrings to 'concretize' and 'concretized' to document the format for tests. Added tests for the activation of test dependencies.
2020-11-25concretizer: treat conditional providers correctly (#20086)Massimiliano Culpo2-1/+18
refers #20040 This modification emits rules like: provides_virtual("netlib-lapack","blas") :- variant_value("netlib-lapack","external-blas","False"). for packages that provide virtual dependencies conditionally instead of a fact that doesn't account for the condition.
2020-11-25concretizer: remove debug statement (#20085)Harmen Stoppels1-1/+0
2020-11-23Docs: remove duplication in Command Reference (#20021)Adam J. Stewart2-2/+2
2020-11-23recognize macOS 11.1 as big sur (#20038)Martin Aumüller1-2/+6
Big Sur versions go 11.0, 11.0.1, 11.1 (vs. prior versions that only used the minor component) Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2020-11-19spack debug report: print concretizer (#19983)Adam J. Stewart2-0/+4
2020-11-19fujitsu compiler: added / fixed support for compiler flags (#19967)Tomoki, Karatsu2-2/+12
Added flags for: - Debug symbols - C++17 standard Fixed the list of flags for generic optimizations
2020-11-19clang/llvm: fix version detection (#19978)Michael Kuhn2-4/+8
This PR fixes two problems with clang/llvm's version detection. clang's version output looks like this: ``` clang version 11.0.0 Target: x86_64-unknown-linux-gnu ``` This caused clang's version to be misdetected as: ``` clang@11.0.0 Target: ``` This resulted in errors when trying to actually use it as a compiler. When using `spack external find`, we couldn't determine the compiler version, resulting in errors like this: ``` ==> Warning: "llvm@11.0.0+clang+lld+lldb" has been detected on the system but will not be added to packages.yaml [reason=c compiler not found for llvm@11.0.0+clang+lld+lldb] ``` Changing the regex to only match until the end of the line fixes these problems. Fixes: #19473
2020-11-18fix error handling for spack test results command (#19987)Greg Becker1-0/+1
2020-11-18hip support for umpire, chai, raja, camp (#19715)Danny Taller2-0/+139
* create HipPackage base class and do some refactoring * comments and added conflict to raja for openmp with hip
2020-11-18bump version number to 0.16.0Todd Gamblin1-1/+1
2020-11-18cmd: add `spack mark` command (#16662)Michael Kuhn7-33/+329
This adds a new `mark` command that can be used to mark packages as either explicitly or implicitly installed. Apart from fixing the package database after installing a dependency manually, it can be used to implement upgrade workflows as outlined in #13385. The following commands demonstrate how the `mark` and `gc` commands can be used to only keep the current version of a package installed: ```console $ spack install pkgA $ spack install pkgB $ git pull # Imagine new versions for pkgA and/or pkgB are introduced $ spack mark -i -a $ spack install pkgA $ spack install pkgB $ spack gc ``` If there is no new version for a package, `install` will simply mark it as explicitly installed and `gc` will not remove it. Co-authored-by: Greg Becker <becker33@llnl.gov>
2020-11-18spack test (#15702)Greg Becker48-620/+2283
Users can add test() methods to their packages to run smoke tests on installations with the new `spack test` command (the old `spack test` is now `spack unit-test`). spack test is environment-aware, so you can `spack install` an environment and then run `spack test run` to run smoke tests on all of its packages. Historical test logs can be perused with `spack test results`. Generic smoke tests for MPI implementations, C, C++, and Fortran compilers as well as specific smoke tests for 18 packages. Inside the test method, individual tests can be run separately (and continue to run best-effort after a test failure) using the `run_test` method. The `run_test` method encapsulates finding test executables, running and checking return codes, checking output, and error handling. This handles the following trickier aspects of testing with direct support in Spack's package API: - [x] Caching source or intermediate build files at build time for use at test time. - [x] Test dependencies, - [x] packages that require a compiler for testing (such as library only packages). See the packaging guide for more details on using Spack testing support. Included is support for package.py files for virtual packages. This does not change the Spack interface, but is a major change in internals. Co-authored-by: Tamara Dahlgren <dahlgren1@llnl.gov> Co-authored-by: wspear <wjspear@gmail.com> Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2020-11-17Improve warning message for deprecated attributes in "packages.yaml"Massimiliano Culpo6-28/+67
The deprecatedProperties custom validator now can accept a function to compute a better error message. Improve error/warning message for deprecated properties
2020-11-17Documentation: spack load/environments prefix inspections (#19961)Peter Scheibel3-4/+34
As of #18260, `spack load` and `spack env activate` now use `prefix_inspections` from the modules configuration to decide how to modify environment variables. This updates the modules configuration documentation to describe how to update environment variables with the `prefix_inspections` section. This also updates the `spack load` and environments documentation to refer to the new `prefix_inspections` documentation.
2020-11-17spack load/environments: allow customization of prefix inspections (#18260)Dr. Christian Tacke1-0/+5
`spack load` and `spack env activate` now use the prefix inspections defined in `modules.yaml`. This allows users to customize/override environment variable modifications if desired. If no `prefix_inspections` configuration is present, Spack uses the values in the default configuration.
2020-11-17spack containerize: allow users to customize the base image (#15028)Massimiliano Culpo11-219/+479
This PR reworks a few attributes in the container subsection of spack.yaml to permit the injection of custom base images when generating containers with Spack. In more detail, users can still specify the base operating system and Spack version they want to use: spack: container: images: os: ubuntu:18.04 spack: develop in which case the generated recipe will use one of the Spack images built on Docker Hub for the build stage and the base OS image in the final stage. Alternatively, they can specify explicitly the two base images: spack: container: images: build: spack/ubuntu-bionic:latest final: ubuntu:18.04 and it will be up to them to ensure their consistency. Additional changes: * This commit adds documentation on the two approaches. * Users can now specify OS packages to install (e.g. with apt or yum) prior to the build (previously this was only available for the finalized image). * Handles to avoid an update of the available system packages have been added to the configuration to facilitate the generation of recipes permitting deterministic builds.
2020-11-17concretizer: modified weights for providers and matching for externalsMassimiliano Culpo4-35/+29
This commit address the case of concretizing a root spec with a transitive conditional dependency on a virtual package, provided by an external. Before these modifications default variant values for the dependency bringing in the virtual package were not respected, and the external package providing the virtual was added to the DAG. The issue stems from two facts: - Selecting a provider has higher precedence than selecting default variants - To ensure that an external is preferred, we used a negative weight To solve it we shift all the providers weight so that: - External providers have a weight of 0 - Non external provider have a weight of 10 or more Using a weight of zero for external providers is such that having an external provider, if present, or not having a provider at all has the same effect on the higher priority minimization. Also fixed a few minor bugs in concretize.lp, that were causing spurious entries in the final answer set. Cleaned concretize.lp from leftover rules.
2020-11-17concretizer: maximize the number of default values used for a single variantMassimiliano Culpo2-0/+17
If a the default of a multi-valued variant is set to multiple values either in package.py or in packages.yaml we need to ensure that all the values are present in the concretized spec. Since each default value has a weight of 0 and the variant value is set implicitly by the concretizer we need to add a rule to maximize on the number of default values that are used.
2020-11-17concretizer: don't require a provider for virtual deps if spec is externalMassimiliano Culpo4-3/+17
This commit introduces a new rule: real_node(Package) :- not external(Package), node(Package). that permits to distinguish between an external node and a real node that shouldn't trim dependency. It solves the case of concretizing ninja with an external Python.
2020-11-17concretizer: spec_clauses() shouldn't emit node_compiler_hard for rule bodies.Todd Gamblin1-2/+0
`node_compiler_hard()` means that something explicitly asked for a node's compiler to be set -- i.e., it's not inherited, it's required. We're generating this in spec_clauses even for specs in rule bodies, which results in conditions like this for optional dependencies: In py-torch/package.py: depends_on('llvm-openmp', when='%apple-clang +openmp') In the generated ASP: declared_dependency("py-torch","llvm-openmp","build") :- node("py-torch"), variant_value("py-torch","openmp","True"), node_compiler("py-torch","apple-clang"), node_compiler_hard("py-torch","apple-clang"), node_compiler_version_satisfies("py-torch","apple-clang",":"). The `node_compiler_hard` there means we would have to *explicitly* set py-torch's compiler to trigger the llvm-openmp dependency, rather than just letting it be set by preferences. This is wrong; the dependency should be there regardless of how the compiler was set. - [x] remove fn.node_compiler_hard() call from spec_clauses when generating rule body clauses.