summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2021-03-16Speed-up CI by reorganizing tests (#22247)Massimiliano Culpo6-12/+18
* unit tests: mark slow tests as "maybeslow" This commit also removes the "network" marker and marks every "network" test as "maybeslow". Tests marked as db are maintained, but they're not slow anymore. * GA: require style tests to pass before running unit-tests * GA: make MacOS unit tests fail fast * GA: move all unit tests into the same workflow, run style tests as a prerequisite All the unit tests have been moved into the same workflow so that a single run of the dorny/paths-filter action can be used to ask for coverage based on the files that have been changed in a PR. The basic idea is that for PRs that introduce only changes to packages coverage is not necessary, this resulting in a faster execution of the tests. Also, for package only PRs slow unit tests are skipped. Finally, MacOS and linux unit tests are now conditional on style tests passing meaning that e.g. we won't waste a MacOS worker if we know that the PR has flake8 issues. * Addressed review comments * Skipping slow tests on MacOS for package only recipes * QA: make tests on changes correct before merging
2021-03-16bugfix: allow imposed constraints to be overridden in special casesTodd Gamblin1-6/+16
In most cases, we want condition_holds(ID) to imply any imposed constraints associated with the ID. However, the dependency relationship in Spack is special because it's "extra" conditional -- a dependency *condition* may hold, but we have decided that externals will not have dependencies, so we need a way to avoid having imposed constraints appear for nodes that don't exist. This introduces a new rule that says that constraints are imposed *unless* we define `do_not_impose(ID)`. This allows rules like dependencies, which rely on more than just spec conditions, to cancel imposed constraints. We add one special case for this: dependencies of externals.
2021-03-16bugfix: do not generate dep conditions when no dependencyTodd Gamblin1-10/+15
We only consider test dependencies some of the time. Some packages are *only* test dependencies. Spack's algorithm was previously generating dependency conditions that could hold, *even* if there was no potential dependency type. - [x] change asp.py so that this can't happen -- we now only generate dependency types for possible dependencies.
2021-03-16concretizer: unify logic for spec conditionalsTodd Gamblin3-183/+119
This builds on #20638 by unifying all the places in the concretizer where things are conditional on specs. Previously, we duplicated a common spec conditional pattern for dependencies, virtual providers, conflicts, and externals. That was introduced in #20423 and refined in #20507, and roughly looked as follows. Given some directives in a package like: ```python depends_on("foo@1.0+bar", when="@2.0+variant") provides("mpi@2:", when="@1.9:") ``` We handled the `@2.0+variant` and `@1.9:` parts by generating generated `dependency_condition()`, `required_dependency_condition()`, and `imposed_dependency_condition()` facts to trigger rules like this: ```prolog dependency_conditions_hold(ID, Parent, Dependency) :- attr(Name, Arg1) : required_dependency_condition(ID, Name, Arg1); attr(Name, Arg1, Arg2) : required_dependency_condition(ID, Name, Arg1, Arg2); attr(Name, Arg1, Arg2, Arg3) : required_dependency_condition(ID, Name, Arg1, Arg2, Arg3); dependency_condition(ID, Parent, Dependency); node(Parent). ``` And we handled `foo@1.0+bar` and `mpi@2:` parts ("imposed constraints") like this: ```prolog attr(Name, Arg1, Arg2) :- dependency_conditions_hold(ID, Package, Dependency), imposed_dependency_condition(ID, Name, Arg1, Arg2). attr(Name, Arg1, Arg2, Arg3) :- dependency_conditions_hold(ID, Package, Dependency), imposed_dependency_condition(ID, Name, Arg1, Arg2, Arg3). ``` These rules were repeated with different input predicates for requirements (e.g., `required_dependency_condition`) and imposed constraints (e.g., `imposed_dependency_condition`) throughout `concretize.lp`. In #20638 it got to be a bit confusing, because we used the same `dependency_condition_holds` predicate to impose constraints on conditional dependencies and virtual providers. So, even though the pattern was repeated, some of the conditional rules were conjoined in a weird way. Instead of repeating this pattern everywhere, we now have *one* set of consolidated rules for conditions: ```prolog condition_holds(ID) :- condition(ID); attr(Name, A1) : condition_requirement(ID, Name, A1); attr(Name, A1, A2) : condition_requirement(ID, Name, A1, A2); attr(Name, A1, A2, A3) : condition_requirement(ID, Name, A1, A2, A3). attr(Name, A1) :- condition_holds(ID), imposed_constraint(ID, Name, A1). attr(Name, A1, A2) :- condition_holds(ID), imposed_constraint(ID, Name, A1, A2). attr(Name, A1, A2, A3) :- condition_holds(ID), imposed_constraint(ID, Name, A1, A2, A3). ``` this allows us to use `condition(ID)` and `condition_holds(ID)` to encapsulate the conditional logic on specs in all the scenarios where we need it. Instead of defining predicates for the requirements and imposed constraints, we generate the condition inputs with generic facts, and define predicates to associate the condition ID with a particular scenario. So, now, the generated facts for a condition look like this: ```prolog condition(121). condition_requirement(121,"node","cairo"). condition_requirement(121,"variant_value","cairo","fc","True"). imposed_constraint(121,"version_satisfies","fontconfig","2.10.91:"). dependency_condition(121,"cairo","fontconfig"). dependency_type(121,"build"). dependency_type(121,"link"). ``` The requirements and imposed constraints are generic, and we associate them with their meaning via the id. Here, `dependency_condition(121, "cairo", "fontconfig")` tells us that condition 121 has to do with the dependency of `cairo` on `fontconfig`, and the conditional dependency rules just become: ```prolog dependency_holds(Package, Dependency, Type) :- dependency_condition(ID, Package, Dependency), dependency_type(ID, Type), condition_holds(ID). ``` Dependencies, virtuals, conflicts, and externals all now use similar patterns, and the logic for generating condition facts is common to all of them on the python side, as well. The more specific routines like `package_dependencies_rules` just call `self.condition(...)` to get an id and generate requirements and imposed constraints, then they generate their extra facts with the returned id, like this: ```python def package_dependencies_rules(self, pkg, tests): """Translate 'depends_on' directives into ASP logic.""" for _, conditions in sorted(pkg.dependencies.items()): for cond, dep in sorted(conditions.items()): condition_id = self.condition(cond, dep.spec, pkg.name) # create a condition and get its id self.gen.fact(fn.dependency_condition( # associate specifics about the dependency w/the id condition_id, pkg.name, dep.spec.name )) # etc. ``` - [x] unify generation and logic for conditions - [x] use unified logic for dependencies - [x] use unified logic for virtuals - [x] use unified logic for conflicts - [x] use unified logic for externals LocalWords: concretizer mpi attr Arg concretize lp cairo fc fontconfig LocalWords: virtuals def pkg cond dep fn refactor github py
2021-03-15Expand relative dev paths in environment files (#22045)Harmen Stoppels6-19/+138
* Rewrite relative dev_spec paths internally to absolute paths in case of relocation of the environment file * Test relative paths for dev_path in environments * Add a --keep-relative flag to spack env create This ensures that relative paths of develop paths are not expanded to absolute paths when initializing the environment in a different location from the spack.yaml init file.
2021-03-15Propagate --test= for environments (#22040)Harmen Stoppels7-27/+125
* Propagate --test= for environments * Improve help comment for spack concretize --test flag * Add tests for --test with environments
2021-03-15Fix use of quotes in Python build system (#22279)Adam J. Stewart1-1/+1
2021-03-14do not validate variants of concrete specs in solver setup (#22272)Vanessasaurus1-10/+11
Currently, regardless of a spec being concrete or not, we validate its variants in `spec_clauses` (part of `SpackSolverSetup`). This PR skips the check if the spec is concrete. The reason we want to do this is so that the solver setup class (really, `spec_clauses`) can be used for cases when we just want the logic statements / facts (is that what they are called?) and we don't need to re-validate an already concrete spec. We can't change existing concrete specs, and we have to be able to handle them *even if they violate constraints in the current spack*. This happens in practice if we are doing the validation for a spec produced by a different spack install. Signed-off-by: vsoch <vsoch@users.noreply.github.com>
2021-03-13adding spack -c to set one off config arguments (#22251)Vanessasaurus5-72/+132
This pull request will add the ability for a user to add a configuration argument on the fly, on the command line, e.g.,: ```bash $ spack -c config:install_tree:root:/path/to/config.yaml -c packages:all:compiler:[gcc] list --help ``` The above command doesn't do anything (I'm just getting help for list) but you can imagine having another root of packages, and updating it on the fly for a command (something I'd like to do in the near future!) I've moved the logic for config_add that used to be in spack/cmd/config.py into spack/config.py proper, and now both the main.py (where spack commands live) and spack/cmd/config.py use these functions. I only needed spack config add, so I didn't move the others. We can move the others if there are also needed in multiple places.
2021-03-12bugfix: ensure spack test list still works (#22203)Tamara Dahlgren2-2/+25
Was getting the following error: ``` $ spack test list ==> Error: issubclass() arg 1 must be a class ``` This PR adds a check in `has_test_method` (in case it is re-used elsewhere such as #22097) and ensures a class is passed to the method from `spack test list`.
2021-03-10Pipelines: Fix issue that allowed failing jobs to pass (#22199)Scott Wittenburg1-1/+3
This is a workaround for an issue with how "spack install" is invoked from within "spack ci rebuild". The fact that we don't get an exception or even the actual returncode when using the object returned by spack.util.executable.which('spack') to install the target spec means we get no indication of failures about the install command itself. Instead we rely on the subsequent buildcache creation failure to fail the job.
2021-03-08Fix `spack graph` when deptypes are filtered (#22121)Harmen Stoppels3-8/+16
2021-03-08Add `--vanilla` flag to r build system (#22166)Glenn Johnson1-0/+1
Unlike the other commands of the `R CMD` interface, the `INSTALL` command will read `Renviron` files. This can potentially break builds of r- packages, depending on what is set in the `Renviron` file. This PR adds the `--vanilla` flag to ensure that neither `Rprofile` nor `Renviron` files are read during Spack builds of r- packages.
2021-03-07spack python: add --path option (#22006)Todd Gamblin2-0/+13
This adds a `--path` option to `spack python` that shows the `python` interpreter that Spack is using. e.g.: ```console $ spack python --path /Users/gamblin2/src/spack/var/spack/environments/default/.spack-env/view/bin/python ``` This is useful for debugging, and we can ask users to run it to understand what python Spack is picking up via preferences in `bin/spack` and via the `SPACK_PYTHON` environment variable introduced in #21222.
2021-03-07add `spack test list --all` (#22032)Todd Gamblin2-5/+38
`spack test list` will show you which *installed* packages can be tested but it won't show you which packages have tests. - [x] add `spack test list --all` to show which packages have test methods - [x] update `has_test_method()` to handle package instances *and* package classes.
2021-03-05Improve R package creation (#21861)Glenn Johnson3-6/+125
* Improve R package creation This PR adds the `list_url` attribute to CRAN R packages when using `spack create`. It also adds the `git` attribute to R Bioconductor packages upon creation. * Switch over to using cran/bioc attributes The cran/bioc entries are set to have the '=' line up with homepage entry, but homepage does not need to exist in the package file. If it does not, that could affect the alignment. * Do not have to split bioc * Edit R package documentation Explain Bioconductor packages and add `cran` and `bioc` attributes. * Update lib/spack/docs/build_systems/rpackage.rst Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> * Update lib/spack/docs/build_systems/rpackage.rst Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> * Simplify the cran attribute The version can be faked so that the cran attribute is simply equal to the CRAN package name. * Edit the docs to reflect new `cran` attribute format * Use the first element of self.versions() for url Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2021-03-04Allow relative paths in config files (relative to file dirname) (#21996)Greg Becker2-8/+63
This allows users to use relative paths for mirrors and repos and other things that may be part of a Spack environment. There are two ways to do it. 1. Relative to the file ```yaml spack: repos: - local_dir/my_repository ``` Which will refer to a repository like this in the directory where `spack.yaml` lives: ``` env/ spack.yaml <-- the config file above local_dir/ my_repository/ <-- this repository repo.yaml packages/ ``` 2. Relative to the environment ```yaml spack: repos: - $env/local_dir/my_repository ``` Both of these would refer to the same directory, but they differ for included files. For example, if you had this layout: ``` env/ spack.yaml repository/ includes/ repos.yaml repository/ ``` And this `spack.yaml`: ```yaml spack: include: includes/repos.yaml ``` Then, these two `repos.yaml` files are functionally different: ```yaml repos: - $env/repository # refers to env/repository/ above repos: - repository # refers to env/includes/repository/ above ``` The $env variable will not be evaluated if there is no active environment. This generally means that it should not be used outside of an environment's spack.yaml file. However, if other aspects of your workflow guarantee that there is always an active environment, it may be used in other config scopes.
2021-03-03Bootstrap clingo from sources (#21446)Massimiliano Culpo10-19/+348
* Allow the bootstrapping of clingo from sources Allow python builds with system python as external for MacOS * Ensure consistent configuration when bootstrapping clingo This commit uses context managers to ensure we can bootstrap clingo using a consistent configuration regardless of the use case being managed. * Github actions: test clingo with bootstrapping from sources * Add command to inspect and clean the bootstrap store Prevent users to set the install tree root to the bootstrap store * clingo: documented how to bootstrap from sources Co-authored-by: Gregory Becker <becker33@llnl.gov>
2021-03-02Assert the use of the patch directive with a non-existing file will fail ↵Adam J. Stewart1-6/+22
(#21524)
2021-02-27Raise InstallError if any explicit specs is skipped (#21980)Tamara Dahlgren4-11/+32
2021-02-26Compilers: make oneapi ifx version_regex tolerant to wrappers (#21742)e4s-21.02Gregory Lee1-1/+1
If a user creates a wrapper for the ifx binary called ifx_orig, this causes the ifx --version command to produce: $ ifx --version ifx_orig (IFORT) 2021.1 Beta 20201113 Copyright (C) 1985-2020 Intel Corporation. All rights reserved. The regex for ifx currently expects the output to begin with "ifx (IFORT)..." so the wrapper would not be detected as ifx. This PR removes the need for the static "ifx" string which allows wrappers to be detected as ifx. In general, the Intel compiler regexes do not include the invoked executable name (i.e., ifort, icc, icx, etc.), so this is not expected to cause any issues.
2021-02-26make `spack fetch` work with environments (#19166)Andreas Baumbach4-20/+117
* make `spack fetch` work with environments * previously: `spack fetch` required the explicit statement of the specs to be fetched, even when in an environment * now: if there is no spec(s) provided to `spack fetch` we check if an environment is active and if yes we fetch all uninstalled specs.
2021-02-25typo fix (#21967)Greg Becker1-1/+1
2021-02-25Old concretizer: prevent unexpected propagation of external config (#20976)Michael Kuron2-0/+31
When using an external package with the old concretizer, all dependencies of that external package were severed. This was not performed bidirectionally though, so for an external package W with a dependency on Z, if some other package Y depended on Z, Z could still pull properties (e.g. compiler) from W since it was not severed as a parent dependency. This performs the severing bidirectionally, and adds tests to confirm expected behavior when using config from DAG-adjacent packages during concretization.
2021-02-24Config prefer upstream (#21487)Paul Ferrell3-2/+138
This allows for quickly configuring a spack install/env to use upstream packages by default. This is particularly important when upstreaming from a set of officially supported spack installs on a production cluster. By configuring such that package preferences match the upstream, you ensure maximal reuse of existing package installations.
2021-02-23Gitlab fix pr workflow (#21786)Scott Wittenburg1-1/+2
Fixes for gitlab pipelines * Remove accidentally retained testing branch name * Generate pipeline w/out debug mode * Make jobs interruptible for auto-cancel pending * Work around concretization conflicts
2021-02-23Updates to support clingo-cffi (#20657)Josh Essman1-9/+19
* Support clingo when used with cffi Clingo recently merged in a new Python module option based on cffi. Compatibility with this module requires a few changes to spack - it does not automatically convert strings/ints/etc to Symbol and clingo.Symbol.string throws on failure. manually convert str/int to clingo.Symbol types catch stringify exceptions add job for clingo-cffi to Spack CI switch to potassco-vendored wheel for clingo-cffi CI on_unsat argument when cffi
2021-02-23New splice method in class Spec. (#20262)Nathan Hanford3-0/+174
* Spec.splice feature Construct a new spec with a dependency swapped out. Currently can only swap dependencies of the same name, and can only apply to concrete specs. This feature is not yet attached to any install functionality, but will eventually allow us to "rewire" a package to depend on a different set of dependencies. Docstring is reformatted for git below Splices dependency "other" into this ("target") Spec, and return the result as a concrete Spec. If transitive, then other and its dependencies will be extrapolated to a list of Specs and spliced in accordingly. For example, let there exist a dependency graph as follows: T | \ Z<-H In this example, Spec T depends on H and Z, and H also depends on Z. Suppose, however, that we wish to use a differently-built H, known as H'. This function will splice in the new H' in one of two ways: 1. transitively, where H' depends on the Z' it was built with, and the new T* also directly depends on this new Z', or 2. intransitively, where the new T* and H' both depend on the original Z. Since the Spec returned by this splicing function is no longer deployed the same way it was built, any such changes are tracked by setting the build_spec to point to the corresponding dependency from the original Spec. Co-authored-by: Nathan Hanford <hanford1@llnl.gov>
2021-02-23"spack build-env" searches env for relevant spec (#21642)Peter Scheibel5-4/+117
If you install packages using spack install in an environment with complex spec constraints, and the install fails, you may want to test out the build using spack build-env; one issue (particularly if you use concretize: together) is that it may be hard to pass the appropriate spec that matches what the environment is attempting to install. This updates the build-env command to default to pulling a matching spec from the environment rather than concretizing what the user provides on the command line independently. This makes a similar change to spack cd. If the user-provided spec matches multiple specs in the environment, then these commands will now report an error and display all matching specs (to help the user specify). Co-authored-by: Gregory Becker <becker33@llnl.gov>
2021-02-23reduce strictness of directory layout spec-equality check (#21869)Peter Scheibel1-1/+7
2021-02-22Improve error message for inconsistencies in package.py (#21811)Massimiliano Culpo2-5/+40
* Improve error message for inconsistencies in package.py Sometimes directives refer to variants that do not exist. Make it such that: 1. The name of the variant 2. The name of the package which is supposed to have such variant 3. The name of the package making this assumption are all printed in the error message for easier debugging. * Add unit tests
2021-02-22Merge tag 'v0.16.1' into developTodd Gamblin1-2/+2
2021-02-22respect -k/verify-ssl-false in _existing_url method (#21864)Greg Becker1-0/+2
2021-02-19Update CHANGELOG and release versionv0.16.1Tamara Dahlgren1-1/+1
2021-02-19Resolve (post-cherry-picking) flake8 errorsTamara Dahlgren3-4/+8
2021-02-19documentation: correct precedence of included configs in environment ↵Tom Payerle1-2/+2
spack.yaml (#18663) fixes #17993
2021-02-19bugfix: add build deps to 'full hash' (#21735)Greg Becker4-21/+39
The "full hash" was only including the link/run deps, but it should include build deps as well.
2021-02-18Pipelines: Move PR testing stacks (currently only E4S) into spack (#21714)Scott Wittenburg3-37/+61
2021-02-18Fix template for Rpackage in spack create command (#21776)Tom Payerle1-1/+1
The signature for configure_args in the template for new RPackage packages was incorrect (different than what is defined and used in lib/spack/spack/build_systems/r.py) See issue #21774
2021-02-18Testing: use spack.store.use_store everywhere (#21656)Massimiliano Culpo10-175/+176
Keep spack.store.store and spack.store.db consistent in unit tests * Remove calls to monkeypatch for spack.store.store and spack.store.db: tests that used these called one or the other, which lead to inconsistencies (the tests passed regardless but were fragile as a result) * Fixtures making use of monkeypatch with mock_store now use the updated use_store function, which sets store.store and store.db consistently * subprocess_context.TestState now transfers the serializes and restores spack.store.store (without the monkeypatch changes this would have created inconsistencies)
2021-02-18Documentation fix: build_system configure_args for #21760 (#21761)Tom Payerle1-2/+2
Corrects the signature for configure_args (and therefore configure_vars) in documentation on RPackage build system to match the code See issue #21760
2021-02-18bugfix: relax racy test in fg/bg output (#21755)Todd Gamblin1-12/+6
Since signals are fundamentally racy, We can't bound the amount of time that the `test_foreground_background_output` test will take to get to 'on', we can only observe that it transitions to 'on'. So instead of using an arbitrary limit, just adjust the test to allow either 'on' or 'off' followed by 'on'. This should eliminate the spurious errors we see in CI.
2021-02-18Avoid spurious warning from clingo (#21731)Massimiliano Culpo1-0/+1
There's a spurious warning that occurs whenever a spec being concretized does not depend on a virtual provider under any possible configuration.
2021-02-17apple-clang: add correct path to compiler wrappers (#21662)Adam J. Stewart1-1/+1
Follow-up to #17110 ### Before ```bash CC=/Users/Adam/spack/lib/spack/env/clang/clang; export CC SPACK_CC=/usr/bin/clang; export SPACK_CC PATH=...:/Users/Adam/spack/lib/spack/env/apple-clang:/Users/Adam/spack/lib/spack/env/case-insensitive:/Users/Adam/spack/lib/spack/env:...; export PATH ``` ### After ```bash CC=/Users/Adam/spack/lib/spack/env/clang/clang; export CC SPACK_CC=/usr/bin/clang; export SPACK_CC PATH=...:/Users/Adam/spack/lib/spack/env/clang:/Users/Adam/spack/lib/spack/env/case-insensitive:/Users/Adam/spack/lib/spack/env:...; export PATH ``` `CC` and `SPACK_CC` were being set correctly, but `PATH` was using the name of the compiler `apple-clang` instead of `clang`. For most packages, since `CC` was set correctly, nothing broke. But for packages using `Makefiles` that set `CC` based on `which clang`, it was using the system compilers instead of the compiler wrappers. Discovered when working on `py-xgboost@0.90`. An alternative fix would be to copy the symlinks in `env/clang` to `env/apple-clang`. Let me know if you think there's a better way to do this, or to test this.
2021-02-17add intel oneapi to compiler/pkg translations (#21448)Greg Becker1-1/+2
2021-02-17Print groups properly for spack find -d (#20028)Yang Zongze1-1/+1
2021-02-17store sbang_install_path in buildinfo, use for subsequent relocation (#20768)eugeneswalker1-0/+11
2021-02-17[WIP] relocate.py: parallelize test replacement logic (#19690)Nathan Hanford9-210/+281
* sbang pushed back to callers; star moved to util.lang * updated unit test * sbang test moved; local tests pass Co-authored-by: Nathan Hanford <hanford1@llnl.gov>
2021-02-17concretizer: require at least a dependency type to say the dependency holdsMassimiliano Culpo2-0/+11
fixes #20784 Similarly to the previous bug, here we were deducing conditions to be imposed on nodes that were not part of the DAG.
2021-02-17concretizer: dependency conditions cannot hold if package is externalMassimiliano Culpo3-2/+9
fixes #20736 Before this one line fix we were erroneously deducing that dependency conditions hold even if a package was external. This may result in answer sets that contain imposed conditions on a node without the node being present in the DAG, hence #20736.