summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2023-11-16Use preferred capitalization of "Slurm" (#41109)Tim Wickberg2-2/+2
https://slurm.schedmd.com/faq.html#acronym
2023-11-16Improve the error message for deprecated preferences (#41075)Massimiliano Culpo3-7/+53
Improves the warning for deprecated preferences, and adds a configuration audit to get files:lines details of the issues. Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
2023-11-16Add audit check to spot `when=` arguments using wrong named specs (#41107)Massimiliano Culpo1-1/+48
* Add audit check to spot when= arguments using named specs * Fix package issues caught by the new audit
2023-11-15docs: packages config on separate page, demote bootstrapping (#41085)Harmen Stoppels5-547/+553
2023-11-14Fix infinite recursion when computing concretization errors (#41061)Massimiliano Culpo1-2/+9
2023-11-14modules: unit-tests without polluted user scope (#41041)Massimiliano Culpo8-58/+62
2023-11-14build(deps): bump urllib3 from 2.0.7 to 2.1.0 in /lib/spack/docs (#41055)dependabot[bot]1-1/+1
Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.0.7 to 2.1.0. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](https://github.com/urllib3/urllib3/compare/2.0.7...2.1.0) --- updated-dependencies: - dependency-name: urllib3 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-11-13info: improve coverage (#41001)Todd Gamblin2-22/+23
Tests didn't cover the new `--variants-by-name` parameter in #40998. Add some parameterization to hit that. This changeset makes me think that the main section-printing loop in `spack info` isn't factored so well. It makes it difficult to pass different arguments to different helper functions. I could break it out into if statements if folks think that would be cleaner.
2023-11-13spack diff: allow hashes from mirrors (#41043)Greg Becker1-0/+2
2023-11-13build(deps): bump black from 23.10.1 to 23.11.0 in /lib/spack/docs (#40967)dependabot[bot]1-1/+1
Bumps [black](https://github.com/psf/black) from 23.10.1 to 23.11.0. - [Release notes](https://github.com/psf/black/releases) - [Changelog](https://github.com/psf/black/blob/main/CHANGES.md) - [Commits](https://github.com/psf/black/compare/23.10.1...23.11.0) --- updated-dependencies: - dependency-name: black dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-11-13Compiler.debug_flags: drop -gz (#40900)Harmen Stoppels3-4/+0
That enables compression of the debug symbols, it doesn't toggle them on or off.
2023-11-13build(deps): bump mypy from 1.6.1 to 1.7.0 in /lib/spack/docs (#41020)dependabot[bot]1-1/+1
Bumps [mypy](https://github.com/python/mypy) from 1.6.1 to 1.7.0. - [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md) - [Commits](https://github.com/python/mypy/compare/v1.6.1...v1.7.0) --- updated-dependencies: - dependency-name: mypy dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-11-10`spack deconcretize` command (#38803)Greg Becker6-25/+242
We have two ways to concretize now: * `spack concretize` concretizes only the root specs that are not concrete in the environment. * `spack concretize -f` eliminates all cached concretization data and reconcretizes the *entire* environment. This PR adds `spack deconcretize`, which eliminates cached concretization data for a spec. This allows users greater control over what is preserved from their `spack.lock` file and what is reused when not using `spack concretize -f`. If you want to update a spec installed in your environment, you can call `spack deconcretize` on it, and that spec and any relevant dependents will be removed from the lock file. `spack concretize` has two options: * `--root`: limits deconcretized specs to *specific* roots in the environment. You can use this to deconcretize exactly one root in a `unify: false` environment. i.e., if `foo` root is a dependent of `bar`, both roots, `spack deconcretize bar` will *not* deconcretize `foo`. * `--all`: deconcretize *all* specs that match the input spec. By default `spack deconcretize` will complain about multiple matches, like `spack uninstall`.
2023-11-10gromacs et al: fix ^mkl pattern (#41002)Massimiliano Culpo2-2/+6
The ^mkl pattern was used to refer to three packages even though none of software using it was depending on "mkl". This pattern, which follows Hyrum's law, is now being removed in favor of a more explicit one. In this PR gromacs, abinit, lammps, and quantum-espresso are modified. Intel packages are also modified to provide "lapack" and "blas" together.
2023-11-10env: compute env mods only for installed roots (#40997)Harmen Stoppels3-5/+9
And improve the error message (load vs unload). Of course you could have some uninstalled dependency too, but as long as it doesn't implement `setup_run_environment` etc, I don't think it hurts to attempt to load the root anyways, given that failure to do so is a warning, not a fatal error.
2023-11-10info: rework spack info command to display variants better (#40998)Todd Gamblin2-39/+180
This changes variant display to use a much more legible format, and to use screen space much better (particularly on narrow terminals). It also adds color the variant display to match other parts of `spack info`. Descriptions and variant value lists that were frequently squished into a tiny column before now have closer to the full terminal width. This change also preserves any whitespace formatting present in `package.py`, so package maintainers can make easer-to-read descriptions of variant values if they want. For example, `gasnet` has had a nice description of the `conduits` variant for a while, but it was wrapped and made illegible by `spack info`. That is now fixed and the original newlines are kept. Conditional variants are grouped by their when clauses by default, but if you do not like the grouping, you can display all the variants in order with `--variants-by-name`. I'm not sure when people will prefer this, but it makes it easier to tell that a particular variant is/isn't there. I do think grouping by `when` is the better default.
2023-11-09docs: tweak formatting of `+:` and `-:` operators (#40988)Todd Gamblin1-3/+5
Just trying to make these stand out a bit more in the docs.
2023-11-09DiHydrogen, Hydrogen, and Aluminum CachedCMakePackage (#39714)Brian Van Essen2-1/+16
2023-11-09buildcache: skip unrecognized metadata files (#40941)Scott Wittenburg2-31/+163
This commit improves forward compatibility of Spack with newer build cache metadata formats. Before this commit, invalid or unrecognized metadata would be fatal errors, now they just cause a mirror to be skipped. Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
2023-11-09Set version to 0.22.0.dev0 (#40975)Harmen Stoppels1-1/+1
2023-11-09modules: remove deprecated code and test data (#40966)Massimiliano Culpo7-106/+3
This removes a few deprecated attributes from the schema of the "modules" section. Test data for deprecated options is removed as well.
2023-11-08modules: restore exclude_implicits (#40958)Harmen Stoppels4-99/+19
2023-11-07Propagate variant across nodes that don't have that variant (#38512)Richarda Butler3-18/+71
Before this PR, variant were not propagated to leaf nodes that could accept the propagated value, if some intermediate node couldn't accept it. This PR fixes that issue by marking nodes as "candidate" for propagation and by setting the variant only if it can be accepted by the node. Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2023-11-07Ensure global command line arguments end up in args like before (#40929)Harmen Stoppels1-1/+1
2023-11-07ci: do not retry timed out build jobs (#40936)Scott Wittenburg1-1/+16
2023-11-07Update the branch for the tutorial command (#40934)Massimiliano Culpo1-1/+1
2023-11-07catch exceptions in which_string (#40935)Harmen Stoppels1-2/+5
2023-11-07ASP: targets, compilers and providers soft-preferences are only global (#31261)Massimiliano Culpo11-207/+209
Modify the packages.yaml schema so that soft-preferences on targets, compilers and providers can only be specified under the "all" attribute. This makes them effectively global preferences. Version preferences instead can only be specified under a package specific section. If a preference attribute is found in a section where it should not be, it will be ignored and a warning is printed to screen.
2023-11-07archspec: update to v0.2.2 (#40917)Massimiliano Culpo3-4/+202
Adds support for Neoverse V2
2023-11-07database: optimize query() by skipping unnecessary virtual checks (#40898)Todd Gamblin2-9/+42
Most queries will end up calling `spec.satisfies(query)` on everything in the DB, which will cause Spack to ask whether the query spec is virtual if its name doesn't match the target spec's. This can be expensive, because it can cause Spack to check if any new virtuals showed up in *all* the packages it knows about. That can currently trigger thousands of `stat()` calls. We can avoid the virtual check for most successful queries if we consider that if there *is* a match by name, the query spec *can't* be virtual. This PR adds an optimization to the query loop to save any comparisons that would trigger a virtual check for last. - [x] Add a `deferred` list to the `query()` loop. - [x] First run through the `query()` loop *only* checks for name matches. - [x] Query loop now returns early if there's a name match, skipping most `satisfies()` calls. - [x] Second run through the `deferred()` list only runs if query spec is virtual. - [x] Fix up handling of concrete specs. - [x] Add test for querying virtuals in DB. - [x] Avoid allocating deferred if not necessary. --------- Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
2023-11-06spack compiler find --[no]-mixed-toolchain (#40902)Harmen Stoppels6-72/+139
Currently there's some hacky logic in the AppleClang compiler that makes it also accept `gfortran` as a fortran compiler if `flang` is not found. This is guarded by `if sys.platform` checks s.t. it only applies to Darwin. But on Linux the feature of detecting mixed toolchains is highly requested too, cause it's rather annoying to run into a failed build of `openblas` after dozens of minutes of compiling its dependencies, just because clang doesn't have a fortran compiler. In particular in CI where the system compilers may change during system updates, it's typically impossible to fix compilers in a hand-written compilers.yaml config file: the config will almost certainly be outdated sooner or later, and maintaining one config file per target machine and writing logic to select the correct config is rather undesirable too. --- This PR introduces a flag `spack compiler find --mixed-toolchain` that fills out missing `fc` and `f77` entries in `clang` / `apple-clang` by picking the best matching `gcc`. It is enabled by default on macOS, but not on Linux, matching current behavior of `spack compiler find`. The "best matching gcc" logic and compiler path updates are identical to how compiler path dictionaries are currently flattened "horizontally" (per compiler id). This just adds logic to do the same "vertically" (across different compiler ids). So, with this change on Ubuntu 22.04: ``` $ spack compiler find --mixed-toolchain ==> Added 6 new compilers to /home/harmen/.spack/linux/compilers.yaml gcc@13.1.0 gcc@12.3.0 gcc@11.4.0 gcc@10.5.0 clang@16.0.0 clang@15.0.7 ==> Compilers are defined in the following files: /home/harmen/.spack/linux/compilers.yaml ``` you finally get: ``` compilers: - compiler: spec: clang@=15.0.7 paths: cc: /usr/bin/clang cxx: /usr/bin/clang++ f77: /usr/bin/gfortran fc: /usr/bin/gfortran flags: {} operating_system: ubuntu23.04 target: x86_64 modules: [] environment: {} extra_rpaths: [] - compiler: spec: clang@=16.0.0 paths: cc: /usr/bin/clang-16 cxx: /usr/bin/clang++-16 f77: /usr/bin/gfortran fc: /usr/bin/gfortran flags: {} operating_system: ubuntu23.04 target: x86_64 modules: [] environment: {} extra_rpaths: [] ``` The "best gcc" is automatically default system gcc, since it has no suffixes / prefixes.
2023-11-06docs: expand section about relocation, suggest padding (#40909)Harmen Stoppels1-10/+27
2023-11-06Add support for aliases (#17229)Michael Kuhn5-14/+93
Add a new config section: `config:aliases`, which is a dictionary mapping aliases to commands. For instance: ```yaml config: aliases: sp: spec -I ``` will define a new command `sp` that will execute `spec` with the `-I` argument. Aliases cannot override existing commands, and this is ensured with a test. We cannot currently alias subcommands. Spack will warn about any aliases containing a space, but will not error, which leaves room for subcommand aliases in the future. --------- Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2023-11-06Don't let runtime env variables of compiler like deps leak into the build ↵Harmen Stoppels3-2/+32
environment (#40916) * Test that setup_run_environment changes to CC/CXX/FC/F77 are dropped in build env * compilers set in run env shouldn't impact build Adds `drop` to EnvironmentModifications courtesy of @haampie, and uses it to clear modifications of CC, CXX, F77 and FC made by `setup_{,dependent_}run_environment` routines when producing an environment in BUILD context. * comment / style * comment --------- Co-authored-by: Tom Scogland <scogland1@llnl.gov>
2023-11-06fix prefix_inspections keys in example (#40904)Harmen Stoppels1-4/+4
2023-11-06docs: mention public build cache for GHA (#40908)Harmen Stoppels1-23/+47
2023-11-06Introduce `default_args` context manager (#39964)Harmen Stoppels4-2/+77
This adds a rather trivial context manager that lets you deduplicate repeated arguments in directives, e.g. ```python depends_on("py-x@1", when="@1", type=("build", "run")) depends_on("py-x@2", when="@2", type=("build", "run")) depends_on("py-x@3", when="@3", type=("build", "run")) depends_on("py-x@4", when="@4", type=("build", "run")) ``` can be condensed to ```python with default_args(type=("build", "run")): depends_on("py-x@1", when="@1") depends_on("py-x@2", when="@2") depends_on("py-x@3", when="@3") depends_on("py-x@4", when="@4") ``` The advantage is it's clear for humans, the downside it's less clear for type checkers due to type erasure.
2023-11-06error messages: condition chaining (#40173)Greg Becker6-113/+592
Create chains of causation for error messages. The current implementation is only completed for some of the many errors presented by the concretizer. The rest will need to be filled out over time, but this demonstrates the capability. The basic idea is to associate conditions in the solver with one another in causal relationships, and to associate errors with the proximate causes of their facts in the condition graph. Then we can construct causal trees to explain errors, which will hopefully present users with useful information to avoid the error or report issues. Technically, this is implemented as a secondary solve. The concretizer computes the optimal model, and if the optimal model contains an error, then a secondary solve computes causation information about the error(s) in the concretizer output. Examples: $ spack solve hdf5 ^cmake@3.0.1 ==> Error: concretization failed for the following reasons: 1. Cannot satisfy 'cmake@3.0.1' 2. Cannot satisfy 'cmake@3.0.1' required because hdf5 ^cmake@3.0.1 requested from CLI 3. Cannot satisfy 'cmake@3.18:' and 'cmake@3.0.1 required because hdf5 ^cmake@3.0.1 requested from CLI required because hdf5 depends on cmake@3.18: when @1.13: required because hdf5 ^cmake@3.0.1 requested from CLI 4. Cannot satisfy 'cmake@3.12:' and 'cmake@3.0.1 required because hdf5 depends on cmake@3.12: required because hdf5 ^cmake@3.0.1 requested from CLI required because hdf5 ^cmake@3.0.1 requested from CLI $ spack spec cmake ^curl~ldap # <-- with curl configured non-buildable and an external with `+ldap` ==> Error: concretization failed for the following reasons: 1. Attempted to use external for 'curl' which does not satisfy any configured external spec 2. Attempted to build package curl which is not buildable and does not have a satisfying external attr('variant_value', 'curl', 'ldap', 'True') is an external constraint for curl which was not satisfied 3. Attempted to build package curl which is not buildable and does not have a satisfying external attr('variant_value', 'curl', 'gssapi', 'True') is an external constraint for curl which was not satisfied 4. Attempted to build package curl which is not buildable and does not have a satisfying external 'curl+ldap' is an external constraint for curl which was not satisfied 'curl~ldap' required required because cmake ^curl~ldap requested from CLI $ spack solve yambo+mpi ^hdf5~mpi ==> Error: concretization failed for the following reasons: 1. 'hdf5' required multiple values for single-valued variant 'mpi' 2. 'hdf5' required multiple values for single-valued variant 'mpi' Requested '~mpi' and '+mpi' required because yambo depends on hdf5+mpi when +mpi required because yambo+mpi ^hdf5~mpi requested from CLI required because yambo+mpi ^hdf5~mpi requested from CLI 3. 'hdf5' required multiple values for single-valued variant 'mpi' Requested '~mpi' and '+mpi' required because netcdf-c depends on hdf5+mpi when +mpi required because netcdf-fortran depends on netcdf-c required because yambo depends on netcdf-fortran required because yambo+mpi ^hdf5~mpi requested from CLI required because netcdf-fortran depends on netcdf-c@4.7.4: when @4.5.3: required because yambo depends on netcdf-fortran required because yambo+mpi ^hdf5~mpi requested from CLI required because yambo depends on netcdf-c required because yambo+mpi ^hdf5~mpi requested from CLI required because yambo depends on netcdf-c+mpi when +mpi required because yambo+mpi ^hdf5~mpi requested from CLI required because yambo+mpi ^hdf5~mpi requested from CLI Future work: In addition to fleshing out the causes of other errors, I would like to find a way to associate different components of the error messages with different causes. In this example it's pretty easy to infer which part is which, but I'm not confident that will always be the case. See the previous PR #34500 for discussion of how the condition chains are incomplete. In the future, we may need custom logic for individual attributes to associate some important choice rules with conditions such that clingo choices or other derivations can be part of the explanation. --------- Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2023-11-06Environments: remove environments created with SpackYAMLErrors (#40878)Tamara Dahlgren4-21/+68
2023-11-05spack env activate: create & activate default environment without args (#40756)Harmen Stoppels2-4/+40
This PR implements the concept of "default environment", which doesn't have to be created explicitly. The aim is to lower the barrier for adopting environments. To (create and) activate the default environment, run ``` $ spack env activate ``` This mimics the behavior of ``` $ cd ``` which brings you to your home directory. This is not a breaking change, since `spack env activate` without arguments currently errors. It is similar to the already existing `spack env activate --temp` command which always creates an env in a temporary directory, the difference is that the default environment is a managed / named environment named `default`. The name `default` is not a reserved name, it's just that `spack env activate` creates it for you if you don't have it already. With this change, you can get started with environments faster: ``` $ spack env activate [--prompt] $ spack install --add x y z ``` instead of ``` $ spack env create default ==> Created environment 'default in /Users/harmenstoppels/spack/var/spack/environments/default ==> You can activate this environment with: ==> spack env activate default $ spack env activate [--prompt] default $ spack install --add x y z ``` Notice that Spack supports switching (but not stacking) environments, so the parallel with `cd` is pretty clear: ``` $ spack env activate named_env $ spack env status ==> In environment named_env $ spack env activate $ spack env status ==> In environment default ```
2023-11-05Add command and package suggestions (#40895)Michael Kuhn5-12/+35
* Add command suggestions This adds suggestions of similar commands in case users mistype a command. Before: ``` $ spack spack ==> Error: spack is not a recognized Spack command or extension command; check with `spack commands`. ``` After: ``` $ spack spack ==> Error: spack is not a recognized Spack command or extension command; check with `spack commands`. Did you mean one of the following commands? spec patch ``` * Add package name suggestions * Remove suggestion to run spack clean -m
2023-11-05Hidden modules: always append hash (#40868)Harmen Stoppels10-150/+153
2023-11-05Environments: Add support for including definitions files (#33960)Tamara Dahlgren10-149/+296
This PR adds support for including separate definitions from `spack.yaml`. Supporting the inclusion of files with definitions enables user to make curated/standardized collections of packages that can re-used by others.
2023-11-04spack.modules.commmon: pass spec to SetupContext (#40886)Harmen Stoppels1-1/+3
Currently module globals aren't set before running `setup_[dependent_]run_environment` to compute environment modifications for module files. This commit fixes that.
2023-11-03environment: solve one spec per child process (#40876)Massimiliano Culpo2-3/+12
Looking at the memory profiles of concurrent solves for environment with unify:false, it seems memory is only ramping up. This exchange in the potassco mailing list: https://sourceforge.net/p/potassco/mailman/potassco-users/thread/b55b5b8c2e8945409abb3fa3c935c27e%40lohn.at/#msg36517698 Seems to suggest that clingo doesn't release memory until end of the application. Since when unify:false we distribute work to processes, here we give a maxtaskperchild=1, so we clean memory after each solve.
2023-11-03Bugfix: propagation of multivalued variants (#39833)Richarda Butler2-0/+13
Don't encourage use of default value if propagating a multivalued variant.
2023-11-03clingo-bootstrap: force setuptools through variant (#40866)Massimiliano Culpo1-0/+4
2023-11-03ASP-based solver: fix for unsplittable providers (#40859)Massimiliano Culpo2-1/+16
Some providers must provide virtuals "together", i.e. if they provide one virtual of a set, they must be the providers also of the others. There was a bug though, where we were not checking if the other virtuals in the set were needed at all in the DAG. This commit fixes the bug.
2023-11-03oci parsing: make image name case insensitive (#40858)Harmen Stoppels2-3/+14
2023-11-03fix typo in packaging guide (#40853)Thomas-Ulrich1-1/+1