summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2022-03-04Fix importing Spack packages as Python modules (#29221)Massimiliano Culpo9-53/+82
fixes #29203 This PR fixes a subtle bug we have when importing Spack packages as Python modules that can lead to multiple module objects being created for the same package. It also fixes all the places in unit-tests where "relying" on the old bug was crucial to have a new "clean" state of the package class.
2022-03-03Revert GCS fetch strategy, to remove s3 interface (#28736)Doug Jacobsen1-7/+1
This commit reverts the GCS fetch strategy to before commit: d7596125231e800ca41c60e379be2b8abb47d20d The previous commit added some s3 syntax to handle connections, but added them into the GCS fetch strategy in a way that prevents GCS from working anymore.
2022-03-03rocmcc compiler: initial commit based on aocc and clang (#28575)Greg Becker7-5/+91
* rocmcc compiler: initial commit based on aocc and clang Co-authored-by: luker <luke.roskop@hpe.com> Co-authored-by: Tom Scogland <scogland1@llnl.gov>
2022-03-03installer: Fix cosmetic problem with terminal title (#29070)Michael Kuhn1-4/+8
The status displayed in the terminal title could be wrong when doing distributed builds. For instance, doing `spack install glib` in two different terminals could lead to the current package being reported as `40/29` due to the way Spack handles retrying locks. Work around this by keeping track of the package IDs that were already encountered to avoid counting packages twice.
2022-03-03test/installer.py: remove commented code and inaccurate docstring (#29305)Scott Wittenburg1-21/+1
2022-03-02make @llnl.util.lang.memoized support kwargs (#21722)Danny McClanahan2-10/+96
* make memoized() support kwargs * add testing for @memoized
2022-03-02Fix typos when forwarding arguments to traverse_edges (#29261)Massimiliano Culpo4-4/+11
A few calls use `deptypes=...` instead of `deptype=...`
2022-03-01test_env_install_two_specs_same_dep: properly check installed specs (#29222)Tamara Dahlgren1-4/+4
2022-02-25commands: Propgate command return value as exit code (#29223)Scott Wittenburg1-1/+1
2022-02-23Testing: Summarize test results and add verbose output (#28700)Tamara Dahlgren6-13/+106
2022-02-23Move early exit for setup only argument (#29041)Massimiliano Culpo1-4/+4
See https://github.com/spack/spack/pull/28468/files#r809156986 If we exit before generating the: error("Dependencies must have compatible OS's with their dependents"). ... facts we'll output a problem that is effectively different by the one solved by clingo.
2022-02-23Checksum match (#28989)Tom Scogland2-2/+25
* cmd/checksum: prefer url matching url_from_version This is a minimal change toward getting the right archive from places like github. The heuristic is: * if an archive url exists, take its version * generate a url from the package with pkg.url_from_version * if they match * stop considering other URLs for this version * otherwise, continue replacing the url for the version I doubt this will always work, but it should address a variety of versions of this bug. A good test right now is `spack checksum gh`, which checksums macos binaries without this, and the correct source packages with it. fixes #15985 related to #14129 related to #13940 * add heuristics to help create as well Since create can't rely on an existing package, this commit adds another pair of heuristics: 1. if the current version is a specifically listed archive, don't replace it 2. if the current url matches the result of applying `spack.url.substitute_version(a, ver)` for any a in archive_urls, prefer it and don't replace it fixes #13940 * clean up style and a lingering debug import * ok flake8, you got me * document reference_package argument * Update lib/spack/spack/util/web.py Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> * try to appease sphinx Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2022-02-22Add `spack --bootstrap` option for accessing bootstrap store (#25601)Todd Gamblin5-58/+64
We can see what is in the bootstrap store with `spack find -b`, and you can clean it with `spack clean -b`, but we can't do much else with it, and if there are bootstrap issues they can be hard to debug. We already have `spack --mock`, which allows you to swap in the mock packages from the command line. This PR introduces `spack -b` / `spack --bootstrap`, which runs all of spack with `ensure_bootstrap_configuration()` set. This means that you can run `spack -b find`, `spack -b install`, `spack -b spec`, etc. to see what *would* happen with bootstrap configuration, to remove specific bootstrap packages, etc. This will hopefully make developers' lives easier as they deal with bootstrap packages. This PR also uses a `nullcontext` context manager. `nullcontext` has been implemented in several other places in Spack, and this PR consolidates them to `llnl.util.lang`, with a note that we can delete the function if we ever reqyire a new enough Python. - [x] introduce `spack --bootstrap` option - [x] consolidated all `nullcontext` usages to `llnl.util.lang`
2022-02-21bugfix: Not all concrete versions on the CLI should be considered real (#28620)Todd Gamblin5-24/+69
Some "concrete" versions on the command line, e.g. `qt@5` are really meant to satisfy some actual concrete version from a package. We should only assume the user is introducing a new, unknown version on the CLI if we, well, don't know of any version that satisfies the user's request. So, if we know about `5.11.1` and `5.11.3` and they ask for `5.11.2`, we'd ask the solver to consider `5.11.2` as a solution. If they just ask for `5`, though, `5.11.1` or `5.11.3` are fine solutions, as they satisfy `@5`, so use them. Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2022-02-21Bring back cuda@11.4.0 conflicts for GCC and clang; add 11.4.3:11.4.4 (#29076)Alberto Invernizzi1-0/+2
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2022-02-18spack external find: change default behavior (#29031)Massimiliano Culpo2-4/+22
See https://github.com/spack/spack/issues/25353#issuecomment-1041868116 This commit changes the default behavior of ``` $ spack external find ``` from searching all the possible packages Spack knows about to search only for the ones tagged as being a "build-tool". It also introduces a `--all` option to restore the old behavior.
2022-02-17darwin: robust macos version detection (#28991)Seth R. Johnson2-7/+49
Prefer `sw_vers` to `platform.mac_ver`. In anaconda3 installation, for example, the latter reports 10.16 on Monterey -- I think this is affected by how and where the python instance was built. Use MACOSX_DEPLOYMENT_TARGET if present to override the operating system choice.
2022-02-17gitlab: Propagate stack name to downstream build jobs (#29019)Scott Wittenburg1-0/+4
It will be useful for metrics gathering and possibly debugging to have this environment variable available in the runner pods that do the actual rebuilds.
2022-02-17Testing: optionally run tests on externally installed packages (#28701)Tamara Dahlgren5-8/+64
Since Spack does not install external packages, this commit skips them by default when running stand-alone tests. The assumption is that such packages have likely undergone an acceptance test process. However, the tests can be run against installed externals using ``` % spack test run --externals ... ```
2022-02-17Reduce verbosity of patches=... variant (#29015)Harmen Stoppels1-2/+19
* reduce verbosity of patches=... variant * Special-case prefix-matches for satisfies of patches variant
2022-02-17ASP-based solver: don't sort when defining variant possible values (#29013)Massimiliano Culpo1-2/+3
fixes #28260 Since we iterate over different variants from many packages, the variant values may have types which are not comparable, which causes errors at runtime. This is not a real issue though, since we don't need the facts to be ordered. Thus, to avoid needless sorting, the sorted function has been removed and a comment has been added to tip any developer that might need to inspect these clauses for debugging to add back sorting on the first two items only. It's kind of difficult to add a test for this, since the error depends on whether Python sorting algorithm ever needs to compare the third value of a tuple being ordered.
2022-02-16extensions: allow multiple "extends" directives (#28853)Tom Scogland2-5/+28
* extensions: allow multiple "extends" directives This will allow multiple extends directives in a package as long as only one of them is selected as a dependency in the concrete spec. * document the option to have multiple extends
2022-02-16concretizer: refactor argument passing for reuseTodd Gamblin6-107/+106
Reuse previously was a very invasive change that required parameters to be added to all the methods that called `concretize()` on a `Spec` object. With the addition of concretizer configuration, we can use the config system to simplify this argument passing and keep the code cleaner. We decided that concretizer config options should be read at `Solver` instantiation time, and if config changes between instnatiation of a particular solver and `solve()` invocation, the `Solver` should use the settings from `__init__()`. - [x] remove `reuse` keyword argument from most concretize functions - [x] refactor usages to use `spack.config.override("concretizer:reuse", True)` - [x] rework argument passing in `Solver` so that parameters are set from config at instantiation time
2022-02-16docs: add section on concretizer configurationTodd Gamblin5-13/+59
* Document `concretizer.yaml`, `--reuse`, and `--fresh`.
2022-02-16commands: refactor `--reuse` handling to use configTodd Gamblin11-35/+135
`--reuse` was previously handled individually by each command that needed it. We are growing more concretization options, and they'll need their own section for commands that support them. Now there are two concretization options: * `--reuse`: Attempt to reuse packages from installs and buildcaches. * `--fresh`: Opposite of reuse -- traditional spack install. To handle thes, this PR adds a `ConfigSetAction` for `argparse`, so that you can write argparse code like this: ``` subgroup.add_argument( '--reuse', action=ConfigSetAction, dest="concretizer:reuse", const=True, default=None, help='reuse installed dependencies/buildcaches when possible' ) ``` With this, you don't need to add logic to pull the argument out and handle it; the `ConfigSetAction` just does it for you. This can probably be used to clean up some other commands later, as well. Code that was previously passing `reuse=True` around everywhere has been refactored to use config, and config is set from the CLI using a new `add_concretizer_args()` function in `spack.cmd.common.arguments`. - [x] Add `ConfigSetAction` to simplify concretizer config on the CLI - [x] Refactor code so that it does not pass `reuse=True` to every function. - [x] Refactor commands to use `add_concretizer_args()` and to pass concretizer config using the config system.
2022-02-16tests: consolidate mock scope creation logic in `conftest.py`Todd Gamblin2-22/+18
Config scopes were different for `config` and `mutable_config`, and `mutable_config` did not have a command line scope. - [x] Fix by consolidating the creation logic for the two fixtures.
2022-02-16config: add a new `concretizer` config sectionTodd Gamblin3-2/+41
The concretizer is going to grow to have many more configuration, and we really need some structured config for that. * We have the `config:concretizer` option that chooses the solver, but extending that is awkward (we'd need to replace a string with a `dict`) and the solver choice will be deprecated eventually. * We have the `concretization` option in environments, but it's not a top-level config section -- it's just for environments, and it also only admits a string right now. To avoid overlapping with either of these and to allow the most extensibility in the future, this adds a new `concretizer` config section that can be used in and outside of environments. There is only one option right now: `reuse`. This can expand to include other options later. Likely, we will soon deprecate `config:concretizer` and warn when the user doesn't use `clingo`, and we will eventually (sometime later) move the `together` / `separately` options from `concretization` into the top-level `concretizer` section. This commit just adds the new section and schema. Fully wiring it up is TBD.
2022-02-16refactor: convert spack.solver.asp.solve() to a classTodd Gamblin4-46/+95
The solver has a lot of configuration associated with it. Rather than adding arguments to everything, we should encapsulate that in a class. This is the start of that work; it replaces `solve()` and its kwargs with a class and properties.
2022-02-16Add 'stable' to the list of infinity version names. (#28772)Mark W. Krentel2-2/+3
* Add 'stable' to the list of infinity version names. Rename libunwind 1.5-head to 1.5-stable. * Add stable to the infinite version list in packaging_guide.rst.
2022-02-16find_libraries: search for both .so and .dylib on macOS (#28924)Adam J. Stewart1-4/+10
2022-02-15intel compiler: fix link time error with `LLVMgold.so` (#28731)Stephen Sachs1-0/+38
The Intel compiler will, at link time, call `ld -plugin LLVMgold.so`, which expects libraries like `libimfo.so` to be found either in the `LD_LIBRARY_PATH` or in `LLVMgold.so`s RPATH. As `LLVMgold.so` already uses RUNPATH, I used that to extend this to the necessary library locations. This PR should fix issues: https://github.com/spack/spack/issues/10308 https://github.com/spack/spack/issues/18606 https://github.com/spack/spack/issues/17100 https://github.com/spack/spack/issues/21237 https://github.com/spack/spack/issues/4261 Co-authored-by: Stephen Sachs <stesachs@amazon.com>
2022-02-15Make `spack -e [env] spec` show environment root specs (#25941)Harmen Stoppels2-19/+47
2022-02-15Unalias despacktivate only when alias exists (#28939)Harmen Stoppels1-1/+1
2022-02-15darwin: make sure MACOSX_DEPLOYMENT_TARGET has a minor component (#28926)Seth R. Johnson1-1/+10
2022-02-11move typing_extensions.py back into typing.py =\ (#28549)Danny McClanahan3-29/+26
2022-02-10macOS: always set `MACOSX_DEPLOYMENT_TARGET` (#28797)Seth R. Johnson5-6/+75
* core: Make platform environment an instance not class method In preparation for accessing data constructed in __init__. * macos: set consistent macosx deployment target This should silence numerous warnings from mixed gcc/macos toolchains. * perl: prevent too-new deployment target version ``` *** Unexpected MACOSX_DEPLOYMENT_TARGET=11 *** *** Please either set it to a valid macOS version number (e.g., 10.15) or to empty. ``` * Stylin' * Add deployment target overrides to failing autoconf packages * Move configure workaround to base autoconf package This reverts commit 3c119eaf8b4fb37c943d503beacf5ad2aa513d4c. * Stylin' * macos: add utility functions for SDK These aren't yet used but should probably be added to spack debug report.
2022-02-10ASP-based solver: reduce input facts and add heuristic (#28848)Massimiliano Culpo2-44/+49
* Remove node_target_satisfies/3 in favor of target_satisfies/2 When emitting input facts we don't need to couple target with packages, but we can emit fewer facts independently and let the grounder combine them. * Remove compiler_version_satisfies/4 in favor of compiler_version_satisfies/3 When emitting input facts we don't need to couple compilers with packages, but we can emit fewer facts independently and let the grounder combine them. * Introduce heuristic in the ASP-program With heuristic we can drive clingo to make better initial guesses, which lead to fewer choices and conflicts in the overall solve
2022-02-10Fix CMakePackage.define for libs/headers (#28838)Seth R. Johnson2-1/+7
The 'libs' property returned by a spec is not a list nor tuple. Closes #28836.
2022-02-10spack audit: fix spurious failures for target/platform conflicts (#28860)Greg Becker2-5/+15
2022-02-09Tests: move has_test_method to spack.package (#28813)Tamara Dahlgren4-22/+20
2022-02-09Detecting "Cray" as "linux" during bootstrap (#28726)Massimiliano Culpo3-34/+47
* bootstrap: avoid detecting "Cray" and treat the platform as "linux" * bootstrap: create a proper context manager to disable cray
2022-02-07intel-mkl: BLACS with intel-oneapi-mpi (#28476)Stephen Sachs1-0/+1
Identify the correct BLACS libaries when `intel-oneapi-mpi` is used.
2022-02-07Support config variables in config.yaml extensions paths (#17772)Jordan Galby4-10/+32
2022-02-04database: fix reindex with uninstalled deps (#28764)Harmen Stoppels4-42/+94
* Fix reindex with uninstalled deps When a prefix of a dep is removed, and the db is reindexed, it is added through the dependent, but until now it incorrectly listed the spec as 'installed'. There was also some questionable behavior in the db when the same spec was added multiple times, it would always be marked installed. * Always reserve path * Only add installed spec's prefixes to install prefixes set * Improve warning, and ensure ensure only ensures * test: reindex with every file system remnant removed except for the old index; it should give a database with nothing installed, including records with installed==False,external==False,ref_count==0,explicit=True, and these should be removable from the database
2022-02-04Use Spec.constrain to construct spec lists for stacks (#28783)Massimiliano Culpo4-40/+62
* stacks: add regression tests for matrix expansion * Use constrain semantics to construct spec lists for stacks * Fix semantics for constraining an anonymous spec. Add tests
2022-02-02Add a "sticky" property to variants (#28630)Massimiliano Culpo8-101/+176
* Add sticky variants * Add unit tests for sticky variants * Add documentation for sticky variants * Revert "Revert 19736 because conflicts are avoided by clingo by default (#26721)" This reverts commit 33ef7d57c1a09d6ce454503e9ff5638747e4f4a2. * Add stickiness to "allow-unsupported-compiler"
2022-02-01TermStatusLine: fix python 2.7 and add test (#28715)Harmen Stoppels2-1/+12
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-01-28macholib, altgraph: update vendored dependency (#28664)Massimiliano Culpo25-1253/+1345
2022-01-26Ensure "spack unit-test" can bootstrap clingo (#28572)Massimiliano Culpo2-1/+8
2022-01-26Print 'Waiting for another process to install x, y, z' in distributed builds ↵Harmen Stoppels1-0/+53
(#28535) Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>