summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2021-04-06spack location: fix usage without args (#22755)Harmen Stoppels2-4/+5
2021-04-06Remove erroneous warnings about quotes for from_source_file (#22767)Adam J. Stewart2-10/+18
2021-04-02Check against a list of known-broken specs in `ci generate` (#22690)Zack Galbreath7-16/+124
* Strip leading slash from S3 key in url_exists() * Check against a list of known-broken specs in `ci generate`
2021-04-02Document unzip (#22723)Harmen Stoppels1-1/+1
2021-04-02concretizer: improve display of optimization criteria (#22433)Todd Gamblin4-18/+78
By default, clingo doesn't show any optimization criteria (maximized or minimized sums) if the set they aggregate is empty. Per the clingo mailing list, we can get around that by adding, e.g.: ``` #minimize{ 0@2 : #true }. ``` for the 2nd criterion. This forces clingo to print out the criterion but does not affect the optimization. This PR adds directives as above for all of our optimization criteria, as well as facts with descriptions of each criterion,like this: ``` opt_criterion(2, "number of non-default variants") ``` We use facts in `concretize.lp` rather than hard-coding these in `asp.py` so that the names can be maintained in the same place as the other optimization criteria. The now-displayed weights and the names are used to display optimization output like this: ```console (spackle):solver> spack solve --show opt zlib ==> Best of 0 answers. ==> Optimization Criteria: Priority Criterion Value 1 version weight 0 2 number of non-default variants (roots) 0 3 multi-valued variants + preferred providers for roots 0 4 number of non-default variants (non-roots) 0 5 number of non-default providers (non-roots) 0 6 count of non-root multi-valued variants 0 7 compiler matches + number of nodes 1 8 version badness 0 9 non-preferred compilers 0 10 target matches 0 11 non-preferred targets 0 zlib@1.2.11%apple-clang@12.0.0+optimize+pic+shared arch=darwin-catalina-skylake ``` Note that this is all hidden behind a `--show opt` option to `spack solve`. Optimization weights are no longer shown by default, but you can at least inspect them and more easily understand what is going on. - [x] always show optimization criteria in `clingo` output - [x] add `opt_criterion()` facts for all optimizationc criteria - [x] make display of opt criteria optional in `spack solve` - [x] rework how optimization criteria are displayed, and add a `--show opt` optiong to `spack solve`
2021-04-01add CachedCMakePackage for using CMake initial config filesGreg Becker2-0/+253
CachedCMakePackage is a CMakePackage subclass for using CMake initial cache. This feature of CMake allows packages to increase reproducibility, especially between spack builds and manual builds. It also allows packages to sidestep certain parsing bugs in extremely long cmake commands, and to avoid system limits on the length of the command line. Co-authored by: Chris White <white238@llnl.gov>
2021-04-01Revert "CachedCMakePackage for using *.cmake initial config files (#19316)""Todd Gamblin2-253/+0
This reverts commit 7daf5823574dc18522f559a084095714cc9f3fb9.
2021-04-01bugfix: compiler wrappers should handle extra spaces between arguments (#22725)Elizabeth Fischer1-1/+9
In the face of two consecutive spaces in the command line, the compiler wrapper would skip all remaining arguments, causing problems building py-scipy with Intel compiler. This PR solves the problem. * Fixed compiler wrapper in the face of extra spaces between arguments Co-authored-by: Elizabeth Fischer <elizabeth.fischer@alaska.edu>
2021-03-31CachedCMakePackage for using *.cmake initial config files (#19316)"Greg Becker2-0/+253
Original commit message: This feature of CMake allows packages to increase reproducibility, especially between Spack- and manual builds. It also allows packages to sidestep certain parsing bugs in extremely long ``cmake`` commands, and to avoid system limits on the length of the command line. Adding: Co-authored by: Chris White <white238@llnl.gov> This reverts commit c4f0a3cf6cd2ab282f6abf20fd72fcb4739a432a.
2021-03-31Revert "CachedCMakePackage for using *.cmake initial config files (#19316)"Chris White2-253/+0
This reverts commit 764c17053041a65f684ce565a2598d705b04a16b.
2021-03-31CachedCMakePackage for using *.cmake initial config files (#19316)Greg Becker2-0/+253
CachedCMakePackage is a specialized class for packages built using CMake initial cache. This feature of CMake allows packages to increase reproducibility, especially between Spack- and manual builds. It also allows packages to sidestep certain parsing bugs in extremely long ``cmake`` commands, and to avoid system limits on the length of the command line.
2021-03-31hotfix: make ifx work with autoconf <= 2.69 in Spack (#22683)Todd Gamblin2-0/+22
Autoconf before 2.70 will erroneously pass ifx's -loopopt argument to the linker, requiring all packages to use autoconf 2.70 or newer to use ifx. This is a hotfix enabling ifx to be used in Spack. Instead of bothering to upgrade autoconf for every package, we'll just strip out the problematic flag if we're in `ld` mode. - [x] Add a conditional to the `cc` wrapper to skip `-loopopt` in `ld` mode. This can probably be generalized in the future to strip more things (e.g., via an environment variable we can constrol from Spack) but it's good enough for now. - [x] Add a test ensuring that `-loopopt` arguments are stripped in link mode, but not in compile mode.
2021-03-31specs: remove "or ''" from Spec comparisonsTodd Gamblin2-4/+8
Since `lazy_lexicographic_ordering` handles `None` comparison for us, we don't need to adjust the spec comparators to return empty strings or other type-specific empty types. We can just leverage the None-awareness of `lazy_lexicographic_ordering`. - [x] remove "or ''" from `_cmp_iter` in `Spec` - [x] remove setting of `self.namespace` to `''` in `MockPackage`
2021-03-31specs: use lazy lexicographic comparison instead of key_orderingTodd Gamblin5-155/+340
We have been using the `@llnl.util.lang.key_ordering` decorator for specs and most of their components. This leverages the fact that in Python, tuple comparison is lexicographic. It allows you to implement a `_cmp_key` method on your class, and have `__eq__`, `__lt__`, etc. implemented automatically using that key. For example, you might use tuple keys to implement comparison, e.g.: ```python class Widget: # author implements this def _cmp_key(self): return ( self.a, self.b, (self.c, self.d), self.e ) # operators are generated by @key_ordering def __eq__(self, other): return self._cmp_key() == other._cmp_key() def __lt__(self): return self._cmp_key() < other._cmp_key() # etc. ``` The issue there for simple comparators is that we have to bulid the tuples *and* we have to generate all the values in them up front. When implementing comparisons for large data structures, this can be costly. This PR replaces `@key_ordering` with a new decorator, `@lazy_lexicographic_ordering`. Lazy lexicographic comparison maps the tuple comparison shown above to generator functions. Instead of comparing based on pre-constructed tuple keys, users of this decorator can compare using elements from a generator. So, you'd write: ```python @lazy_lexicographic_ordering class Widget: def _cmp_iter(self): yield a yield b def cd_fun(): yield c yield d yield cd_fun yield e # operators are added by decorator (but are a bit more complex) There are no tuples that have to be pre-constructed, and the generator does not have to complete. Instead of tuples, we simply make functions that lazily yield what would've been in the tuple. If a yielded value is a `callable`, the comparison functions will call it and recursively compar it. The comparator just walks the data structure like you'd expect it to. The ``@lazy_lexicographic_ordering`` decorator handles the details of implementing comparison operators, and the ``Widget`` implementor only has to worry about writing ``_cmp_iter``, and making sure the elements in it are also comparable. Using this PR shaves another 1.5 sec off the runtime of `spack buildcache list`, and it also speeds up Spec comparison by about 30%. The runtime improvement comes mostly from *not* calling `hash()` `_cmp_iter()`.
2021-03-31specs: speed up traversal by avoiding redundant canonicalizationTodd Gamblin1-1/+10
2021-03-30Make -j flag less exceptional (#22360)Harmen Stoppels7-39/+90
* Make -j flag less exceptional The -j flag in spack behaves differently from make, ctest, ninja, etc, because it caps the number of jobs to an arbitrary number 16. Spack will behave like other tools if `spack install` uses a reasonable default, and `spack install -j <num>` *overrides* that default. This will be particularly useful for Spack usage outside of a traditional HPC context and for HPC centers that encourage users to compile on login nodes with many cores instead of on compute nodes, which has become increasingly common as individual nodes have more cores. This maintains the existing default value of min(num_cpus, 16). However, as it is right now, Spack does a poor job at determining the number of cpus on linux, since it doesn't take cgroups into account. This is particularly problematic when using distributed builds with slurm. This PR also introduces `spack.util.cpus.cpus_available()` to consolidate knowledge on determining the number of available cores, and improves core detection for linux. This should also improve core detection for Docker/ Kubernetes, which also use cgroups.
2021-03-30SpackCommand objects can set global args (#22318)Harmen Stoppels3-5/+31
This commit extends the API of the __call__ method of the SpackCommand class to permit passing global arguments like those interposed between the main "spack" command and the subsequent subcommand. The functionality is used to fix an issue where running ```spack -e . location -b some_package``` ends up printing the name of the environment instead of the build directory of the package, because the location arg parser also stores this value as `arg.env`.
2021-03-30Bootstrapping: swap store before configuration (#22631)Massimiliano Culpo3-6/+45
fixes #22294 A combination of the swapping order for global variables and the fact that most of them are lazily evaluated resulted in custom install tree not being taken into account if clingo had to be bootstrapped. This commit fixes that particular issue, but a broader refactor may be needed to ensure that similar situations won't affect us in the future.
2021-03-30Bootstrap: add _builtin config scope (#22610)Harmen Stoppels1-2/+5
2021-03-30Fix clearing cache of InternalConfigScope (#22609)Harmen Stoppels3-38/+47
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2021-03-29move binary indices are stored into the misc_cache (#22500)Danny McClanahan2-13/+20
Remote buildcache indices need to be stored in a place that does not require writing to the Spack prefix. Move them from the install_tree to the misc_cache.
2021-03-29bugfix for active when pkg is already active error (#22587)Cyrus Harrison1-2/+7
* bugfix for active when pkg is already active error Co-authored-by: Greg Becker <becker33@llnl.gov>
2021-03-29Externals are preferred even when they have non-default variant valuesMassimiliano Culpo3-1/+33
fixes #22596 Variants which are specified in an external spec are not scored negatively if they encode a non-default value.
2021-03-29Enforce uniqueness of the version_weight atom per nodeMassimiliano Culpo4-7/+32
fixes #22565 This change enforces the uniqueness of the version_weight atom per node(Package) in the DAG. It does so by applying FTSE and adding an extra layer of indirection with the possible_version_weight/2 atom. Before this change it may have happened that for the same node two different version_weight/2 were in the answer set, each of which referred to a different spec with the same version, and their weights would sum up. This lead to unexpected result like preferring to build a new version of an external if the external version was older.
2021-03-29Make stage use concrete specs from environment (#22320)Harmen Stoppels2-1/+121
* Make stage use concrete specs from environment Same as in https://github.com/spack/spack/pull/21642, the idea is that we want to easily stage a package that fails to build in a complex environment. Instead of making the user create a spec by hand (basically transforming all the rules in the environment manifest into a spec, defying the purpose of the environment...), use the provided spec as a filter for the already concretized specs. This also speeds up things, cause we don't have to reconcretize.
2021-03-29Add "spack [cd|location] --source-dir" (#22321)Harmen Stoppels2-56/+80
2021-03-26clingo: modify recipe for bootstrapping (#22354)Massimiliano Culpo2-12/+57
* clingo: modify recipe for bootstrapping Modifications: - clingo builds with shared Python only if ^python+shared - avoid building the clingo app for bootstrapping - don't link to libpython when bootstrapping * Remove option that breaks on linux * Give more hints for the current Python * Disable CLINGO_BUILD_PY_SHARED for bootstrapping * bootstrapping: try to detect the current python from std library This is much faster than calling external executables * Fix compatibility with Python 2.6 * Give hints on which compiler and OS to use when bootstrapping This change hints which compiler to use for bootstrapping clingo (either GCC or Apple Clang on MacOS). On Cray platforms it also hints to build for the frontend system, where software is meant to be installed. * Use spec_for_current_python to constrain module requirement
2021-03-26ASP-based solver: model disjoint sets for multivalued variants (#22534)Massimiliano Culpo3-2/+49
* ASP-based solver: avoid adding values to variants when they're set fixes #22533 fixes #21911 Added a rule that prevents any value to slip in a variant when the variant is set explicitly. This is relevant for multi-valued variants, in particular for those that have disjoint sets of values. * Ensure disjoint sets have a clear semantics for external packages
2021-03-26Make SingleFileScope able to repopulate the cache after clearing it (#22559)Massimiliano Culpo2-16/+47
fixes #22547 SingleFileScope was not able to repopulate its cache before this change. This was affecting the configuration seen by environments using clingo bootstrapped from sources, since the bootstrapping operation involved a few cache invalidation for config files.
2021-03-24Add doc for mirror of env (#22525)Frédéric Simonis1-0/+21
2021-03-23Add stdcxx_libs for PGI and Cray compilers (#22491)Sergey Kosukhin3-0/+12
2021-03-23bootstrap: account for platform specific configuration scopes (#22489)Massimiliano Culpo1-7/+18
This change accounts for platform specific configuration scopes, like ~/.spack/linux, during bootstrapping. These scopes were previously not accounted for and that was causing issues e.g. when searching for compilers.
2021-03-22Oneapi packages: update URLs, environment management, and dependencies (#22202)Robert Cohn4-36/+185
* Replace URL computation in base IntelOneApiPackage class with defining URLs in component packages (this is expected to be simpler for now) * Add component_dir property that all oneAPI component packages must define. This property names a directory that should exist after installation completes (useful for making sure the install was successful) and also defines the search location for the component's environment update script. * Add needed dependencies for components (e.g. intel-oneapi-dnn requires intel-oneapi-tbb). The compilers provided by intel-oneapi-compilers need some components under certain circumstances (e.g. when enabling SYCL support) but these were omitted since the libraries should only be linked when a dependent package requests that feature * Remove individual setup_run_environment implementations and use IntelOneApiPackage superclass method which sources vars.sh (located in a subdirectory of component_dir) * Add documentation for IntelOneApiPackge build system Co-authored-by: Vasily Danilin <vasily.danilin@yandex.ru>
2021-03-22use link/run deps only to compare extensions (#22396)Greg Becker1-2/+2
2021-03-21Document cli syntax for environment scopes (#20344)Greg Becker2-4/+18
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2021-03-18Fix broken spack -c flag (#22361)Harmen Stoppels1-1/+1
2021-03-18archspec: update to latest version (#22357)Massimiliano Culpo2-2/+45
2021-03-18Tab to spaces (#22362)Harmen Stoppels1-2/+2
2021-03-17Fix indentation compiler wrapper issue (#22352)Harmen Stoppels1-1/+1
2021-03-17aocc: add support for v3.0 compilers (#22219)AMD Toolchain Support3-0/+29
A mitigation of a known issue that affects v3.0 is added, see https://developer.amd.com/wp-content/resources/AOCC-3.0-Install-Guide.pdf
2021-03-17spack location: bugfix for out of source build dirs (#22348)Harmen Stoppels1-1/+13
2021-03-16fix weird failure in variant values (#22328)Danny McClanahan1-1/+14
2021-03-16containerize: fix typo in documentation (#22331)Wouter Deconinck1-1/+1
Before this fix, `spack containerize` complains that `centos/7` is invalid (should have been `centos:7`)
2021-03-16Speed-up CI by reorganizing tests (#22247)Massimiliano Culpo6-12/+18
* unit tests: mark slow tests as "maybeslow" This commit also removes the "network" marker and marks every "network" test as "maybeslow". Tests marked as db are maintained, but they're not slow anymore. * GA: require style tests to pass before running unit-tests * GA: make MacOS unit tests fail fast * GA: move all unit tests into the same workflow, run style tests as a prerequisite All the unit tests have been moved into the same workflow so that a single run of the dorny/paths-filter action can be used to ask for coverage based on the files that have been changed in a PR. The basic idea is that for PRs that introduce only changes to packages coverage is not necessary, this resulting in a faster execution of the tests. Also, for package only PRs slow unit tests are skipped. Finally, MacOS and linux unit tests are now conditional on style tests passing meaning that e.g. we won't waste a MacOS worker if we know that the PR has flake8 issues. * Addressed review comments * Skipping slow tests on MacOS for package only recipes * QA: make tests on changes correct before merging
2021-03-16bugfix: allow imposed constraints to be overridden in special casesTodd Gamblin1-6/+16
In most cases, we want condition_holds(ID) to imply any imposed constraints associated with the ID. However, the dependency relationship in Spack is special because it's "extra" conditional -- a dependency *condition* may hold, but we have decided that externals will not have dependencies, so we need a way to avoid having imposed constraints appear for nodes that don't exist. This introduces a new rule that says that constraints are imposed *unless* we define `do_not_impose(ID)`. This allows rules like dependencies, which rely on more than just spec conditions, to cancel imposed constraints. We add one special case for this: dependencies of externals.
2021-03-16bugfix: do not generate dep conditions when no dependencyTodd Gamblin1-10/+15
We only consider test dependencies some of the time. Some packages are *only* test dependencies. Spack's algorithm was previously generating dependency conditions that could hold, *even* if there was no potential dependency type. - [x] change asp.py so that this can't happen -- we now only generate dependency types for possible dependencies.
2021-03-16concretizer: unify logic for spec conditionalsTodd Gamblin3-183/+119
This builds on #20638 by unifying all the places in the concretizer where things are conditional on specs. Previously, we duplicated a common spec conditional pattern for dependencies, virtual providers, conflicts, and externals. That was introduced in #20423 and refined in #20507, and roughly looked as follows. Given some directives in a package like: ```python depends_on("foo@1.0+bar", when="@2.0+variant") provides("mpi@2:", when="@1.9:") ``` We handled the `@2.0+variant` and `@1.9:` parts by generating generated `dependency_condition()`, `required_dependency_condition()`, and `imposed_dependency_condition()` facts to trigger rules like this: ```prolog dependency_conditions_hold(ID, Parent, Dependency) :- attr(Name, Arg1) : required_dependency_condition(ID, Name, Arg1); attr(Name, Arg1, Arg2) : required_dependency_condition(ID, Name, Arg1, Arg2); attr(Name, Arg1, Arg2, Arg3) : required_dependency_condition(ID, Name, Arg1, Arg2, Arg3); dependency_condition(ID, Parent, Dependency); node(Parent). ``` And we handled `foo@1.0+bar` and `mpi@2:` parts ("imposed constraints") like this: ```prolog attr(Name, Arg1, Arg2) :- dependency_conditions_hold(ID, Package, Dependency), imposed_dependency_condition(ID, Name, Arg1, Arg2). attr(Name, Arg1, Arg2, Arg3) :- dependency_conditions_hold(ID, Package, Dependency), imposed_dependency_condition(ID, Name, Arg1, Arg2, Arg3). ``` These rules were repeated with different input predicates for requirements (e.g., `required_dependency_condition`) and imposed constraints (e.g., `imposed_dependency_condition`) throughout `concretize.lp`. In #20638 it got to be a bit confusing, because we used the same `dependency_condition_holds` predicate to impose constraints on conditional dependencies and virtual providers. So, even though the pattern was repeated, some of the conditional rules were conjoined in a weird way. Instead of repeating this pattern everywhere, we now have *one* set of consolidated rules for conditions: ```prolog condition_holds(ID) :- condition(ID); attr(Name, A1) : condition_requirement(ID, Name, A1); attr(Name, A1, A2) : condition_requirement(ID, Name, A1, A2); attr(Name, A1, A2, A3) : condition_requirement(ID, Name, A1, A2, A3). attr(Name, A1) :- condition_holds(ID), imposed_constraint(ID, Name, A1). attr(Name, A1, A2) :- condition_holds(ID), imposed_constraint(ID, Name, A1, A2). attr(Name, A1, A2, A3) :- condition_holds(ID), imposed_constraint(ID, Name, A1, A2, A3). ``` this allows us to use `condition(ID)` and `condition_holds(ID)` to encapsulate the conditional logic on specs in all the scenarios where we need it. Instead of defining predicates for the requirements and imposed constraints, we generate the condition inputs with generic facts, and define predicates to associate the condition ID with a particular scenario. So, now, the generated facts for a condition look like this: ```prolog condition(121). condition_requirement(121,"node","cairo"). condition_requirement(121,"variant_value","cairo","fc","True"). imposed_constraint(121,"version_satisfies","fontconfig","2.10.91:"). dependency_condition(121,"cairo","fontconfig"). dependency_type(121,"build"). dependency_type(121,"link"). ``` The requirements and imposed constraints are generic, and we associate them with their meaning via the id. Here, `dependency_condition(121, "cairo", "fontconfig")` tells us that condition 121 has to do with the dependency of `cairo` on `fontconfig`, and the conditional dependency rules just become: ```prolog dependency_holds(Package, Dependency, Type) :- dependency_condition(ID, Package, Dependency), dependency_type(ID, Type), condition_holds(ID). ``` Dependencies, virtuals, conflicts, and externals all now use similar patterns, and the logic for generating condition facts is common to all of them on the python side, as well. The more specific routines like `package_dependencies_rules` just call `self.condition(...)` to get an id and generate requirements and imposed constraints, then they generate their extra facts with the returned id, like this: ```python def package_dependencies_rules(self, pkg, tests): """Translate 'depends_on' directives into ASP logic.""" for _, conditions in sorted(pkg.dependencies.items()): for cond, dep in sorted(conditions.items()): condition_id = self.condition(cond, dep.spec, pkg.name) # create a condition and get its id self.gen.fact(fn.dependency_condition( # associate specifics about the dependency w/the id condition_id, pkg.name, dep.spec.name )) # etc. ``` - [x] unify generation and logic for conditions - [x] use unified logic for dependencies - [x] use unified logic for virtuals - [x] use unified logic for conflicts - [x] use unified logic for externals LocalWords: concretizer mpi attr Arg concretize lp cairo fc fontconfig LocalWords: virtuals def pkg cond dep fn refactor github py
2021-03-15Expand relative dev paths in environment files (#22045)Harmen Stoppels6-19/+138
* Rewrite relative dev_spec paths internally to absolute paths in case of relocation of the environment file * Test relative paths for dev_path in environments * Add a --keep-relative flag to spack env create This ensures that relative paths of develop paths are not expanded to absolute paths when initializing the environment in a different location from the spack.yaml init file.
2021-03-15Propagate --test= for environments (#22040)Harmen Stoppels7-27/+125
* Propagate --test= for environments * Improve help comment for spack concretize --test flag * Add tests for --test with environments
2021-03-15Fix use of quotes in Python build system (#22279)Adam J. Stewart1-1/+1