summaryrefslogtreecommitdiff
AgeCommit message (Collapse)AuthorFilesLines
2020-11-18update CHANGELOG.md for v0.16.0v0.16.0Todd Gamblin1-0/+110
2020-11-18bump version number to 0.16.0Todd Gamblin1-1/+1
2020-11-18clingo: add `master` branch version (#19958)Massimiliano Culpo1-1/+3
* updated @master to point to the master branch * also added a @spack that points to a fixed commit
2020-11-18cmd: add `spack mark` command (#16662)Michael Kuhn8-34/+339
This adds a new `mark` command that can be used to mark packages as either explicitly or implicitly installed. Apart from fixing the package database after installing a dependency manually, it can be used to implement upgrade workflows as outlined in #13385. The following commands demonstrate how the `mark` and `gc` commands can be used to only keep the current version of a package installed: ```console $ spack install pkgA $ spack install pkgB $ git pull # Imagine new versions for pkgA and/or pkgB are introduced $ spack mark -i -a $ spack install pkgA $ spack install pkgB $ spack gc ``` If there is no new version for a package, `install` will simply mark it as explicitly installed and `gc` will not remove it. Co-authored-by: Greg Becker <becker33@llnl.gov>
2020-11-18spack test (#15702)Greg Becker131-676/+3599
Users can add test() methods to their packages to run smoke tests on installations with the new `spack test` command (the old `spack test` is now `spack unit-test`). spack test is environment-aware, so you can `spack install` an environment and then run `spack test run` to run smoke tests on all of its packages. Historical test logs can be perused with `spack test results`. Generic smoke tests for MPI implementations, C, C++, and Fortran compilers as well as specific smoke tests for 18 packages. Inside the test method, individual tests can be run separately (and continue to run best-effort after a test failure) using the `run_test` method. The `run_test` method encapsulates finding test executables, running and checking return codes, checking output, and error handling. This handles the following trickier aspects of testing with direct support in Spack's package API: - [x] Caching source or intermediate build files at build time for use at test time. - [x] Test dependencies, - [x] packages that require a compiler for testing (such as library only packages). See the packaging guide for more details on using Spack testing support. Included is support for package.py files for virtual packages. This does not change the Spack interface, but is a major change in internals. Co-authored-by: Tamara Dahlgren <dahlgren1@llnl.gov> Co-authored-by: wspear <wjspear@gmail.com> Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2020-11-17Added -level_zero -rocm -opencl flags and sha256 for TAU v2.30. (#19962)sameershende1-1/+18
* Added -level_zero -rocm -opencl flags and sha256 for TAU v2.30. * Removed the depends_on clause for OpenCL and added a variant for OneAPI level_zero. * remove depends_on rocm * remove depends_on rocprofiler Co-authored-by: eugeneswalker <eugenesunsetwalker@gmail.com>
2020-11-17Improve warning message for deprecated attributes in "packages.yaml"Massimiliano Culpo6-28/+67
The deprecatedProperties custom validator now can accept a function to compute a better error message. Improve error/warning message for deprecated properties
2020-11-17Documentation: spack load/environments prefix inspections (#19961)Peter Scheibel3-4/+34
As of #18260, `spack load` and `spack env activate` now use `prefix_inspections` from the modules configuration to decide how to modify environment variables. This updates the modules configuration documentation to describe how to update environment variables with the `prefix_inspections` section. This also updates the `spack load` and environments documentation to refer to the new `prefix_inspections` documentation.
2020-11-17spack load/environments: allow customization of prefix inspections (#18260)Dr. Christian Tacke1-0/+5
`spack load` and `spack env activate` now use the prefix inspections defined in `modules.yaml`. This allows users to customize/override environment variable modifications if desired. If no `prefix_inspections` configuration is present, Spack uses the values in the default configuration.
2020-11-17spack containerize: allow users to customize the base image (#15028)Massimiliano Culpo13-238/+514
This PR reworks a few attributes in the container subsection of spack.yaml to permit the injection of custom base images when generating containers with Spack. In more detail, users can still specify the base operating system and Spack version they want to use: spack: container: images: os: ubuntu:18.04 spack: develop in which case the generated recipe will use one of the Spack images built on Docker Hub for the build stage and the base OS image in the final stage. Alternatively, they can specify explicitly the two base images: spack: container: images: build: spack/ubuntu-bionic:latest final: ubuntu:18.04 and it will be up to them to ensure their consistency. Additional changes: * This commit adds documentation on the two approaches. * Users can now specify OS packages to install (e.g. with apt or yum) prior to the build (previously this was only available for the finalized image). * Handles to avoid an update of the available system packages have been added to the configuration to facilitate the generation of recipes permitting deterministic builds.
2020-11-17concretizer: modified weights for providers and matching for externalsMassimiliano Culpo6-35/+56
This commit address the case of concretizing a root spec with a transitive conditional dependency on a virtual package, provided by an external. Before these modifications default variant values for the dependency bringing in the virtual package were not respected, and the external package providing the virtual was added to the DAG. The issue stems from two facts: - Selecting a provider has higher precedence than selecting default variants - To ensure that an external is preferred, we used a negative weight To solve it we shift all the providers weight so that: - External providers have a weight of 0 - Non external provider have a weight of 10 or more Using a weight of zero for external providers is such that having an external provider, if present, or not having a provider at all has the same effect on the higher priority minimization. Also fixed a few minor bugs in concretize.lp, that were causing spurious entries in the final answer set. Cleaned concretize.lp from leftover rules.
2020-11-17concretizer: maximize the number of default values used for a single variantMassimiliano Culpo2-0/+17
If a the default of a multi-valued variant is set to multiple values either in package.py or in packages.yaml we need to ensure that all the values are present in the concretized spec. Since each default value has a weight of 0 and the variant value is set implicitly by the concretizer we need to add a rule to maximize on the number of default values that are used.
2020-11-17concretizer: don't require a provider for virtual deps if spec is externalMassimiliano Culpo5-3/+33
This commit introduces a new rule: real_node(Package) :- not external(Package), node(Package). that permits to distinguish between an external node and a real node that shouldn't trim dependency. It solves the case of concretizing ninja with an external Python.
2020-11-17concretizer: spec_clauses() shouldn't emit node_compiler_hard for rule bodies.Todd Gamblin1-2/+0
`node_compiler_hard()` means that something explicitly asked for a node's compiler to be set -- i.e., it's not inherited, it's required. We're generating this in spec_clauses even for specs in rule bodies, which results in conditions like this for optional dependencies: In py-torch/package.py: depends_on('llvm-openmp', when='%apple-clang +openmp') In the generated ASP: declared_dependency("py-torch","llvm-openmp","build") :- node("py-torch"), variant_value("py-torch","openmp","True"), node_compiler("py-torch","apple-clang"), node_compiler_hard("py-torch","apple-clang"), node_compiler_version_satisfies("py-torch","apple-clang",":"). The `node_compiler_hard` there means we would have to *explicitly* set py-torch's compiler to trigger the llvm-openmp dependency, rather than just letting it be set by preferences. This is wrong; the dependency should be there regardless of how the compiler was set. - [x] remove fn.node_compiler_hard() call from spec_clauses when generating rule body clauses.
2020-11-17concretizer: don't generate rules for empty version listsTodd Gamblin1-0/+4
If the version list passed to one_of_iff is empty, it still generates a rule like this: node_compiler_version_satisfies("fujitsu-mpi", "arm", ":") :- 1 { } 1. 1 { } 1 :- node_compiler_version_satisfies("fujitsu-mpi", "arm", ":"). The cardinality rules on the right and left above are never satisfiale, and these rules do nothing. - [x] Skip generating any rules at all for empty version lists.
2020-11-17concretizer: add a rule to avoid cycles in the graph of dependenciesMassimiliano Culpo2-5/+9
2020-11-17External packages have a consistent hash across different concretizersMassimiliano Culpo2-1/+12
2020-11-17Don't fail if MV variants have a tuple as default valueMassimiliano Culpo1-1/+2
2020-11-17Fixup for target preferencesMassimiliano Culpo2-3/+11
2020-11-17Added unit tests to for regressions on open concretizer bugsMassimiliano Culpo11-8/+306
2020-11-17Changed clingo optionsMassimiliano Culpo1-1/+4
2020-11-17Reworked optimization rulesMassimiliano Culpo1-15/+49
2020-11-17concretizer: set target preference for inheritance from rootMassimiliano Culpo1-0/+7
2020-11-17install: one less concretization when installing from fileMassimiliano Culpo1-2/+3
2020-11-17Fixed branch after rebase (port to archspec)Massimiliano Culpo2-13/+14
TODO: Investigate the need to remove memoization on Spec.patches (infinite recursion when testing `__contains__`)
2020-11-17Add unit tests for dependencies being patched by parentMassimiliano Culpo4-0/+23
2020-11-17concretizer: handle dependencies conditional on other dependenciesMassimiliano Culpo1-5/+10
2020-11-17tests: verify to handle dependencies conditional on other dependenciesMassimiliano Culpo2-0/+39
2020-11-17concretizer: handle conflicts with compiler ranges correctlyMassimiliano Culpo5-21/+70
As reported, conflicts with compiler ranges were not treated correctly. This commit adds tests to verify the expected behavior for the new concretizer. The new rules to enforce a correct behavior involve: - Adding a rule to prefer the compiler selected for the root package, if no other preference is set - Give a strong negative weight to compiler preferences expressed in packages.yaml - Maximize on compiler AND compiler version match
2020-11-17Github actions: add CI for ASP based solverMassimiliano Culpo4-0/+37
2020-11-17Make all tests passMassimiliano Culpo8-28/+41
Fixed a couple of tests and marked a few xfails to solve them later.
2020-11-17concretizer: added handling for dev_path variantMassimiliano Culpo3-10/+77
This variant is currently either set from command line, in which case it enters the concretization, or attached from environment after concretization.
2020-11-17concretizer: ensure upfront that variants are validMassimiliano Culpo3-12/+31
2020-11-17concretizer: account for test dependencies only when requiredMassimiliano Culpo3-39/+38
2020-11-17Fix installer.py unit tests that check outputMassimiliano Culpo1-1/+2
2020-11-17Compute the correct package name for hierarchies that change class namesMassimiliano Culpo1-1/+5
2020-11-17concretizer: handle variants defined through validatorsMassimiliano Culpo2-2/+23
Variant of this kind don't have a list of possible values encoded in the ASP facts. Since all we have is a validator the list of possible values just includes just the default value and possibly the value passed from packages.yaml or cli.
2020-11-17concretizer: account for patches variantMassimiliano Culpo2-44/+55
This is done after the builder has actually built the specs, to respect the semantics use with the old concretizer. Later we could move this to the solver as a multivalued variant.
2020-11-17concretizer: ensure that no deprecated spec is being usedMassimiliano Culpo2-16/+30
This is done after the builder has actually built the specs, to respect the semantics use with the old concretizer. A better approach is to substitute the spec directly in concretization.
2020-11-17conftest: hook the new solver in the config fixtureMassimiliano Culpo6-10/+12
2020-11-17concretizer: handle "none" value and '*' wildcardMassimiliano Culpo2-2/+9
The "none" variant value cannot be combined with other values. The '*' wildcard matches anything, including "none". It's thus relevant in queries, but disregarded in concretization.
2020-11-17Fixed failing unit testsMassimiliano Culpo6-20/+27
- The test on concretization of anonymous dependencies has been fixed by raising the expected exception. - The test on compiler bootstrap has been fixed by updating the version of GCC used in the test. Since gcc@2.0 does not support targets later than x86_64, the new concretizer was looking for a non-existing spec, i.e. it was correctly trying to retrieve 'gcc target=x86_64' instead of 'gcc target=core2'. - The test on gitlab CI needed an update of the target
2020-11-17concretizer: virtual entry in packages.yaml, external modulesMassimiliano Culpo2-11/+54
This commit adds support for specifying rules in packages.yaml that refer to virtual packages. The approach is to normalize in memory each configuration and turn it into an equivalent configuration without rules on virtual. This is possible if the set of packages to be handled is considered fixed.
2020-11-17concretizer: concretize a virtual rootMassimiliano Culpo4-5/+41
Before this modification the root of a DAG has to be a real package. This commit adds rules to concretize virtual roots.
2020-11-17concretizer: handle version preferences from packages.yamlMassimiliano Culpo2-2/+22
2020-11-17concretizer: handle target preferences from packages.yamlMassimiliano Culpo2-9/+45
The weight of the target used in concretization is, in order: 1. A specific per package weight, if set in packages.yaml 2. Inherited from the parent, if possible 3. The default target weight (always set)
2020-11-17concretizer: fixed test on compiler preferencesMassimiliano Culpo3-3/+3
2020-11-17concretizer: added logic for preferred variantsMassimiliano Culpo4-25/+63
If preferred variants are present, they'll set the default value of a variant. Otherwise the default value is what is encoded in package.py
2020-11-17concretizer: refine compiler logicMassimiliano Culpo4-17/+42
Concrete versions for compilers are respected verbatim. Permit to use a non-existing compiler if the appropriate configuration option has been set.
2020-11-17Fixed failing unit testsMassimiliano Culpo2-10/+11
- Tests based on TestArchitecture - Tests on non-buildable external