summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2023-12-26`spack gc`: add options for environments and build dependencies (#41731)Todd Gamblin8-76/+297
This adds a few options to `spack gc`. One to give you a little more control over dependencies: * `-b` / `--keep-build-dependencies`: By default, `spack gc` considers build dependencies to be "no longer needed" once their dependents are installed. With this option, we'll keep build dependencies of needed installations as well. And two more to make working with environments easier: * `-E` / `--except-any-environment`: Garbage collect anything NOT needed by an environment. `spack gc -E` and `spack gc -bE` are now easy ways to get rid of everytihng not used by some environment. * `-e` / `--except-environment` `ENV`: Instead of considering all environments, garbage collect everything not needed by a *specific* environment. Note that you can use this with `-E` to add directory environments to the list of considered envs, e.g.: spack gc -E -e /path/to/direnv1 -e /path/to/direnv2 #... - [x] rework `unused_specs()` method on DB to add options for roots and deptypes - [x] add `all_hashes()` method on DB - [x] rework `spack gc` command to add 3 more options - [x] tests
2023-12-24build(deps): bump mypy from 1.7.1 to 1.8.0 in /lib/spack/docs (#41821)dependabot[bot]1-1/+1
Bumps [mypy](https://github.com/python/mypy) from 1.7.1 to 1.8.0. - [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md) - [Commits](https://github.com/python/mypy/compare/v1.7.1...v1.8.0) --- updated-dependencies: - dependency-name: mypy dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-12-23CI: Disable downloading artifacts from upstream jobs (#41846)kwryankrattiger1-0/+3
2023-12-23Revert "CI: Disable downloading artifacts from upstream jobs (#41432)" (#41843)Harmen Stoppels1-3/+0
This reverts commit a43156a86131a0b138f356a2721709a2070665ca.
2023-12-22Fix a couple typos in the docs (#41822)Alec Scott3-7/+7
2023-12-22Fix variant initialization logic to allow proper handling of values="*" (#40406)Tim Fuller2-1/+38
Co-authored-by: psakiev <psakiev@sandia.gov> Co-authored-by: tjfulle <tjfulle@users.noreply.github.com>
2023-12-21Consolidate definition of Spack's extra `sys.path` components (#41816)Todd Gamblin1-7/+12
To work properly, Spack requires a few directories from its repository to be added to `sys.path`. Previously these were buried in `spack_installable.main.main()`, but it's sometimes useful to get the paths separately, e.g., if you want to set up your own functioning spack environment. With this change, adding the paths is much simpler: ```python import spack_installable sys.path[:0] = get_spack_sys_paths(spack_prefix) ``` - [x] Add `get_spack_sys_paths()` method with extra paths in order. - [x] Refactor `spack_installable.main.main()` to use it.
2023-12-21Improve error reporting when Clingo install is broken (#41181)John W. Parent1-0/+33
With an improper/incomplete/broken installation of Clingo, it can be importable but not have any of the expected attributes Improve error reporting in this case
2023-12-21Compilers can inject first order rules into the solverMassimiliano Culpo5-25/+225
* Restore PackageBase class, and modify only ASP This prevents a noticeable slowdown in concretization due to the number of directives involved. * Fix issue with 'clang' being preferred to 'gcc', due to runtime version weights * Constraints on runtimes are declared by compilers The declaration of available runtime versions, and of their compatibility constraints are in the associated compiler class. Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2023-12-21gcc-runtime: add separate package for gcc runtime libsHarmen Stoppels3-1/+24
The gcc-runtime package adds a separate node for gcc's dynamic runtime libraries. This should help with: 1. binary caches where rpaths for compiler support libs cannot be relocated because the compiler is missing on the target system 2. creating "minimal" container images The package is versioned like `gcc` (in principle it could be unversioned, but Spack doesn't always guarantee not mixing compilers)
2023-12-21specs: Better error messages for badly quoted specs (#41805)Todd Gamblin4-25/+18
If you are calling Spack from the python API, you might have written something like this before #41529: ``` find = SpackCommand("find") find('--format={name}', 'saxpy@1.0.0', '+rocm', 'amdgpu_target="gfx90a"') ``` But with the breaking change in #41529, you should write: ``` find = SpackCommand("find") find('--format={name}', 'gromacs', '+rocm', 'amdgpu_target=gfx90a') ``` Note that we don't need quotes in Python strings, and that this is what would come in via argv if you typed a quoted variant on the CLI. The error messages for strings like this are not great -- you get something like this: ``` ==> No package matches the query: gromacs+rocm amdgpu_target="gfx90a" ``` Which doesn't indicate that the issue might be your quoting. This is because we were simply outputting the argv we got, instead of using spec.format() to output the error message. This PR fixes such errors to use `spec.format()` and to look like this: ``` ==> No package matches the query: gromacs+rocm amdgpu_target='"gfx90a"' ``` So users should have an easier time understanding that Spack considers the variant value to contain quotes here. - [x] update ConstraintAction to store parsed Specs - [x] refactor commands to display formatted parsed Specs instead of raw input
2023-12-20Only reuse externals when configured (#41707)Harmen Stoppels3-33/+78
Users expect that changes to the externals sections in packages.yaml config apply immediately, but reuse concretization caused this not to be the case. With this commit, the concretizer is only allowed to reuse externals previously imported from config if identical config exists.
2023-12-20fix attribute error in perl build-system (#41628)Christopher Christofi1-14/+25
2023-12-20containers.rst: small docs improvement (#41792)Harmen Stoppels1-2/+3
2023-12-20`spack buildcache push --tag`: create container image with multiple roots ↵Harmen Stoppels4-121/+294
(#41077) This PR adds a flag `--tag/-t` to `buildcache push`, which you can use like ``` $ spack mirror add my-oci-registry oci://example.com/hello/world $ spack -e my_env buildcache push --base-image ubuntu:22.04 --tag my_custom_tag my-oci-registry ``` and lets users ship a full, installed environment as a minimal container image where each image layer is one Spack package, on top of a base image of choice. The image can then be used as ``` $ docker run -it --rm example.com/hello/world:my_custom_tag ``` Apart from environments, users can also pick arbitrary installed spec from their database, for instance: ``` $ spack buildcache push --base-image ubuntu:22.04 --tag some_specs my-oci-registry gcc@12 cmake $ docker run -it --rm example.com/hello/world:some_specs ``` It has many advantages over `spack containerize`: 1. No external tools required (`docker`, `buildah`, ...) 2. Creates images from locally installed Spack packages (No need to rebuild inside `docker build`, where troubleshooting build failures is notoriously hard) 3. No need for multistage builds (Spack just tarballs existing installations of runtime deps) 4. Reduced storage size / composability: when pushing multiple environments with common specs, container image layers are shared. 5. Automatic build cache: later `spack install` of the env elsewhere speeds up since the containerized environment is a build cache
2023-12-19"spack diff": add ignore option for dependencies (#41711)Peter Scheibel6-25/+204
* add trim function to `Spec` and `--ignore` option to 'spack diff' Allows user to compare two specs while ignoring the sub-DAG of a particular dependency, e.g. spack diff --ignore=mpi --ignore=zlib trilinos/abcdef trilinos/fedcba to focus on differences closer to the root of the software stack
2023-12-19CI: Disable downloading artifacts from upstream jobs (#41432)kwryankrattiger1-0/+3
* CI: Disable downloading artifacts from upstream jobs * CI: Default .base-jobs are `when:manual`
2023-12-19environment_modifications_for_specs: do not mutate spec.prefix (#41737)Harmen Stoppels1-41/+31
Sometimes env variables computed in `setup_run_environment` depend on tests w.r.t. files in `spec.prefix`, but Spack temporarily projects `spec.prefix` to the view. This is problematic for two reasons: 1. Some packages iterate over `<prefix>/bin`: they expect only the current package's executables, but find all linked in the view, leading to false positives. 2. Some packages test for `os.path.islink(...)`, which is always true in a view `gcc` is an example that does both. This PR lets Spack compute the environment modifications using the original prefix, and projects to the view afterwards
2023-12-19Add an audit to prevent virtual packages with variants specified (#41747)Massimiliano Culpo1-0/+9
Currently, a virtual spec is composed of just a name and a version. When a virtual spec contains other components, such as variants, Spack won't emit warnings or errors but will silently drop them - which is unexpected by users.
2023-12-19build(deps): bump black from 23.11.0 to 23.12.0 in /lib/spack/docs (#41615)dependabot[bot]1-1/+1
Bumps [black](https://github.com/psf/black) from 23.11.0 to 23.12.0. - [Release notes](https://github.com/psf/black/releases) - [Changelog](https://github.com/psf/black/blob/main/CHANGES.md) - [Commits](https://github.com/psf/black/compare/23.11.0...23.12.0) --- updated-dependencies: - dependency-name: black dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-12-19`spack config get`/`blame`: with no args, show entire config Massimiliano Culpo3-42/+43
This PR changes the default behavior of `spack config get` and `spack config blame` to print a flattened version of the entire spack configuration, including any active environment, if the commands are invoked with no section arguments. The new behavior is used in Gitlab CI to help debug CI configuration, but it can also be useful when asking for more information in issues, or when simply debugging Spack.
2023-12-18environment modifications for externals (#41723)Greg Becker4-1/+60
* allow externals to configure environment modifications * docs for external env modification --------- Co-authored-by: becker33 <becker33@users.noreply.github.com>
2023-12-18build(deps): bump isort from 5.12.0 to 5.13.2 in /lib/spack/docs (#41651)dependabot[bot]1-1/+1
Bumps [isort](https://github.com/pycqa/isort) from 5.12.0 to 5.13.2. - [Release notes](https://github.com/pycqa/isort/releases) - [Changelog](https://github.com/PyCQA/isort/blob/main/CHANGELOG.md) - [Commits](https://github.com/pycqa/isort/compare/5.12.0...5.13.2) --- updated-dependencies: - dependency-name: isort dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-12-18spack.config: cleanup and add type hints (#41741)Massimiliano Culpo4-164/+177
2023-12-18asp.py: remove "CLI" reference (#41718)Harmen Stoppels2-6/+6
Can also be an environment root, or programatically `Spec("x").concretized()`.
2023-12-18`spack develop`: convert to config (#35273)Peter Scheibel11-208/+365
Convert the 'develop' section of an environment to a dedicated configuration section. This means for example that instead of having to define `develop` specs in the `spack.yaml`, the environment can `include:` another `develop.yaml` configuration which specifies which specs should be developed in the environment. This change is not expected to be disruptive given that existing environment `spack.yaml` files will conform to the new schema. (Update 11/28/2023) I have implemented the `develop`/`undevelop` commands in terms of more-generic modification functions added to the `config` module: `change_or_add` and `update_all`. It is assumed that the semantics added here (described in 11/18 update) would be desirable to extend to other config update actions (e.g. adding compilers, changing package requirements, adding mirrors). (Update 11/18/2023) I have updated this such that `spack develop`, and `spack undevelop` to potentially modify all writable scopes, like https://github.com/spack/spack/pull/41147. https://github.com/spack/spack/pull/35307 will be useful for modifying included scopes, but generally speaking specifying a `--scope` will not be required for `spack develop`: `spack develop` will add new develop specs to whatever scope already has develop specs defined, or to the highest-priority writable scope (which should be the env scope). TODOs: - [x] If you `spack undevelop` a package which is mentioned at multiple layers of configuration, then currently this would only modify one of them. That's not technically a new issue (has always existed for configuration modification), but may be confusing to users when presented via an interface other than `spack config set` - [x] Need to add (or confirm) the ability to modify individual config files by providing a path (rather than using a scope identifier as a key to retrieve associated config). - [x] `spack develop` adds new develop specs to the scope that defines them (potentially skipping higher priority scopes to e.g. augment included scope files) --------- Co-authored-by: scheibelp <scheibelp@users.noreply.github.com> Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2023-12-14spec parser: precompile quoting-related regular expressions (#41657)Todd Gamblin2-8/+8
This adds a small (~5%) performance improvement to Spec parsing. Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2023-12-13Improve parsing of quoted flags and variants in specs (#41529)Todd Gamblin7-188/+198
This PR does several things: - [x] Allow any character to appear in the quoted values of variants and flags. - [x] Allow easier passing of quoted flags on the command line, e.g. `cflags="-O2 -g"`. - [x] Handle quoting better in spec output, using single quotes around double quotes and vice versa. - [x] Disallow spaces around `=` and `==` when parsing variants and flags. ## Motivation This PR is motivated by the issues above and by ORNL's [tips for launching at scale on Frontier](https://docs.olcf.ornl.gov/systems/frontier_user_guide.html#tips-for-launching-at-scale). ORNL recommends using `sbcast --send-libs` to broadcast executables and their libraries to compute nodes when running large jobs (e.g., 80k ranks). For an executable named `exe`, `sbcast --send-libs` stores the needed libraries in a directory alongside the executable called `exe_libs`. ORNL recommends pointing `LD_LIBRARY_PATH` at that directory so that `exe` will find the local libraries and not overwhelm the filesystem. There are other ways to mitigate this problem: * You could build with `RUNPATH` using `spack config add config:shared_linking:type:runpath`, which would make `LD_LIBRARY_PATH` take precedence over Spack's `RUNPATHs`. I don't recommend this one because `RUNPATH` can cause many other things to go wrong. * You could use `spack config add config:shared_linking:bind:true`, added in #31948, which will greatly reduce the filesystem load for large jobs by pointing `DT_NEEDED` entries in ELF *directly* at the needed `.so` files instead of relying on `RPATH` search via soname. I have not experimented with this at 80,000 ranks, but it should help quite a bit. * You could use [Spindle](https://github.com/hpc/Spindle) (as LLNL does on its machines) which should transparently fix this without any changes to your executable and without any need to use `sbcast` or other tools. But we want to support the `sbcast` use case as well. ## `sbcast` and Spack Spack's `RPATHs` break the `sbcast` fix because they're considered with higher precedence than `LD_LIBRARY_PATH`. So Spack applications will still end up hitting the shared filesystem when searching for libraries. We can avoid this by injecting some `ldflags` in to the build, e.g., if were were going to launch, say, `LAMMPS` at scale, we could add another `RPATH` specifically for use with `sbcast`: spack install lammps ldflags='-Wl,-rpath=$ORIGIN/lmp_libs' This will put the `lmp_libs` directory alongside `LAMMPS`'s `lmp` executable first in the `RPATH`, so it will be searched before any directories on the shared filesystem. ## Issues with quoting Before this PR, the command above would've errored out for two reasons: 1. `$` wasn't an allowed character in our spec parser. 2. You would've had to double quote the flags to get them to pass through correctly: spack install lammps ldflags='"-Wl,-rpath=$ORIGIN/lmp_libs"' This is ugly and I don't think many users will easily figure it out. The behavior was added in #29282, and it improved parsing of specs passed as a single string, e.g.: spack install 'lammps ldflags="-Wl,-rpath=$ORIGIN/lmp_libs"' but a lot of users are naturally going to try to quote arguments *directly* on the command line, without quoting their entire spec. #29282 used a heuristic to detect unquoted flags and warn the user, but the warning could be confusing. In particular, if you wrote `cflags="-O2 -g"` on the command line, it would break the flags up, warn, and tell you that you could fix the issue by writing `cflags="-O2 -g"` even though you just wrote that. It's telling you to *quote* that value, but the user has to know to double quote. ## New heuristic for quoted arguments from the CLI There are only two places where we allow arbitrary quoted strings in specs: flags and variant values, so this PR adds a simpler heuristic to the CLI parser: if an argument in `sys.argv` starts with `name=...`, then we assume the whole argument is quoted. This means you can write: spack install bzip2 cflags="-O2 -g" directly on the command line, without multiple levels of quoting. This also works: spack install 'bzip2 cflags="-O2 -g"' The only place where this heuristic runs into ambiguity is if you attempt to pass anonymous specs that start with `name=...` as one large string. e.g., this will be interpreted as one large flag value: spack find 'cflags="-O2 -g" ~bar +baz' This sets `cflags` to `"-O2 -g" ~bar +baz`, which is likely not what you wanted. You can fix this easily by either removing the quotes: spack find cflags="-O2 -g" ~bar +baz Or by adding a space at the start, which has the same effect: spack find ' cflags="-O2 -g" ~bar +baz' You may wonder why we don't just look for quotes inside of flag arguments, and the reason is that you *might* want them there. If you are passing arguments like: spack install zlib cppflags="-D DEBUG_MSG1='quick fox' -D DEBUG_MSG2='lazy dog'" You *need* the quotes there. So we've opted for one potentially confusing, but easily fixed outcome vs. limiting what you can put in your quoted strings. ## Quotes in formatted spec output In addition to being more lenient about characters accepted in quoted strings, this PR fixes up spec formatting a bit. We now format quoted strings in specs with single quotes, unless the string has a single quote in it, in which case we JSON-escape the string (i.e., we add `\` before `"` and `\`). zlib cflags='-D FOO="bar"' zlib cflags="-D FOO='bar'" zlib cflags="-D FOO='bar' BAR=\"baz\""
2023-12-13spack mirror create --all: include patches (#41579)Harmen Stoppels1-26/+25
2023-12-12mysql: fix issue when using old core API call (#41573)Massimiliano Culpo1-3/+1
MySQL was performing a core API call to `Spec.flat_dependencies` when setting up the build environment. This function is an implementation detail of the old concretizer, where multiple nodes from the same package are not allowed. This PR uses a more idiomatic way to check if "python" is in the DAG. For reference, see #11356 to check why the call was introduced.
2023-12-11Bump up the version for ROCm-5.7.0 and ROCm-5.7.1 releases. (#40724)Sreenivasa Murthy Kolam1-0/+4
* initial commit for rocm-5.7.0 and 5.7.1 releases * bump up ther version for 5.7.0 and 5.7.1 releases * update recipes to support 5.7.0 and 5.7.1 releases * bump up the version for ROCm 5.7.0 and ROCm-5.7.1 releases * bump up the version for composable-kernel amd miopen-hip * fix style errors * fix style errors in hip etc * renaming composable-kernel recipe * changes for composable_kernel * Revert "renaming composable-kernel recipe" This reverts commit 0cf6c6debfc7b12014f514af26144132ae187e71. * Revert "changes for composable_kernel" This reverts commit 05272a10a79cc14dc9c1afbda8fa4de87ea672ad. * bump up the version for hiprand * using the checksum for hiprand-5.7.1 * bump up the version for 5.7.0 and 5.7.1 releases * fix style errors * fix merge conflicts with the develop. * temp workaround for the error seen with rocm-5.7.0 when trying to generate the dependency file for runtime/legion/legion_redop.cu * fix build issue(work around) with legion * add patch for migraphx package to turn off ck * update to hip recipe * fix hip-path detection inside llvm clang driver * update llvm-amdgpu and rocm-validation-suite recipes * fix style errors * bump up the version for amdsmi for rocm-5.7.0 release * add support for gfx941,gfx942 for rocm-5.7.0 release onwards * revert changes to rocm.py file * added gfx941 and gfx942 to rocm.py and add the gfx942 to kokkos and new checksum the new version seem to support gfx942 * bump up the version for rccl for 5.7.1 * update the patch for rocm-openmp-extras for 5.7.0 * update mivisionx recipe for 5.7.0 release * add new dependencies for rocfft tests * port the fix for avx build, the start address of values_ buffer in KernelParameters is not correct as it is computed based on 16-byte alignment * set HIP_PATH=ROCM_PATH for 5.7.0 onwards * address review comments * revert adding xnack- and xnack+ to gfx940,gfx941,gfx942 as the prechecks were failing
2023-12-11Build cache: make signed/unsigned a mirror property (#41507)Harmen Stoppels10-52/+214
* Add `signed` property to mirror config * make unsigned a tri-state: true/false overrides mirror config, none takes mirror config * test commands * Document this * add a test
2023-12-11Add missing build-system/custom phases to the CDash map (#41439)Tamara Dahlgren1-4/+17
2023-12-11Link to GitHub Action spack/setup-spack in docs (#41509)Harmen Stoppels1-78/+4
2023-12-11unit tests: replace /bin/bash with /bin/sh (#41495)Harmen Stoppels3-10/+10
2023-12-11Fix filter_compiler_wrapper where compiler is None (#41502)Dave Keeshan1-1/+1
Fix filer_compiler_wrapper for cases where the compiler returned in None, this happens on some installed gcc systems that do not have fortran built into them as standard, e.g. gcc@11.4.0 on ubuntu 22.04
2023-12-11commands: better install status help formatting (#41527)Todd Gamblin1-4/+5
Before (hard to read, doesn't fit on small terminals): : ```console -I, --install-status show install status of packages packages can be: installed [+], missing and needed by an installed package [-], installed in an upstream instance [^], or not installed (no annotation) ``` After (fits in 80 columns): ```console -I, --install-status show install status of packages [+] installed [^] installed in an upstream - not installed [-] missing dep of installed package ```
2023-12-08Add logic to cache the RPATH variables in CachedCMakePackages. (#41417)Brian Van Essen1-0/+17
2023-12-07cce compiler: remove vestigial compiler names (#41303)Greg Becker1-4/+4
2023-12-07add missing endtime property to CDash (#41498)Tamara Dahlgren1-0/+1
2023-12-07Fix cdash reporter time stamps (#38825)Victor Brunini1-7/+7
* Fix cdash reporter time stamps (#38818). The cdash reporter is created before packages are installed so save the starttime then instead of the endtime. * Use endtime instead of starttime for the endtime of update --------- Co-authored-by: Tamara Dahlgren <dahlgren1@llnl.gov>
2023-12-07freebsd (#41480)Harmen Stoppels1-0/+2
2023-12-07traverse.py: use > 0 instead of >= 0 (#41482)Ataf Fazledin Ahamed1-1/+1
Signed-off-by: fazledyn-or <ataf@openrefactory.com>
2023-12-07Do not use depfile in bootstrap (#41458)Jordan Galby2-32/+10
- we don't have a fallback if make is not installed - we assume file system locking works - we don't verify that make is gnu make (bootstrapping fails on FreeBSD as a result) - there are some weird race conditions in writing spack.yaml on concurrent spack install - the view is updated after every package install instead of post environment install.
2023-12-07audit: forbid nested dependencies in depends_on declarations (#41428)Massimiliano Culpo1-2/+26
Forbid nested dependencies in depends_on declarations, by running an audit in CI. Fix the packages not passing the new audit: - amd-aocl - exago - palace - shapemapper - xsdk-examples ginkgo: add a commit sha to v1.5.0.glu_experimental
2023-12-06bootstrap: Don't catch Ctrl-C (#41449)Jordan Galby3-5/+23
2023-12-06minimal support for freebsd (#41434)Harmen Stoppels6-9/+76
2023-12-06bootstrap status: no bash (#41431)Harmen Stoppels1-1/+0
2023-12-05CDash: Spack dumps stage errors to configure phase (#41436)psakievich1-0/+19
2023-12-05documentation: add instructions on how to use external opengl (#40987)Billae1-0/+25