summaryrefslogtreecommitdiff
path: root/share
AgeCommit message (Collapse)AuthorFilesLines
2023-12-28CI: Fix timing search paths to ignore bootstrap (#41893)kwryankrattiger1-1/+1
2023-12-28CI: Fix timing search paths to ignore bootstrap (#40677)kwryankrattiger2-3/+11
2023-12-26`spack gc`: add options for environments and build dependencies (#41731)Todd Gamblin2-2/+8
This adds a few options to `spack gc`. One to give you a little more control over dependencies: * `-b` / `--keep-build-dependencies`: By default, `spack gc` considers build dependencies to be "no longer needed" once their dependents are installed. With this option, we'll keep build dependencies of needed installations as well. And two more to make working with environments easier: * `-E` / `--except-any-environment`: Garbage collect anything NOT needed by an environment. `spack gc -E` and `spack gc -bE` are now easy ways to get rid of everytihng not used by some environment. * `-e` / `--except-environment` `ENV`: Instead of considering all environments, garbage collect everything not needed by a *specific* environment. Note that you can use this with `-E` to add directory environments to the list of considered envs, e.g.: spack gc -E -e /path/to/direnv1 -e /path/to/direnv2 #... - [x] rework `unused_specs()` method on DB to add options for roots and deptypes - [x] add `all_hashes()` method on DB - [x] rework `spack gc` command to add 3 more options - [x] tests
2023-12-23Revert "CI: Disable downloading artifacts from upstream jobs (#41432)" (#41843)Harmen Stoppels1-1/+0
This reverts commit a43156a86131a0b138f356a2721709a2070665ca.
2023-12-21gcc-runtime: add separate package for gcc runtime libsHarmen Stoppels2-0/+4
The gcc-runtime package adds a separate node for gcc's dynamic runtime libraries. This should help with: 1. binary caches where rpaths for compiler support libs cannot be relocated because the compiler is missing on the target system 2. creating "minimal" container images The package is versioned like `gcc` (in principle it could be unversioned, but Spack doesn't always guarantee not mixing compilers)
2023-12-21ci: use "strong preference" idiom for compilers (#41806)Harmen Stoppels2-2/+5
to avoid duplication of conflicts / requirements in config
2023-12-20`spack buildcache push --tag`: create container image with multiple roots ↵Harmen Stoppels2-6/+10
(#41077) This PR adds a flag `--tag/-t` to `buildcache push`, which you can use like ``` $ spack mirror add my-oci-registry oci://example.com/hello/world $ spack -e my_env buildcache push --base-image ubuntu:22.04 --tag my_custom_tag my-oci-registry ``` and lets users ship a full, installed environment as a minimal container image where each image layer is one Spack package, on top of a base image of choice. The image can then be used as ``` $ docker run -it --rm example.com/hello/world:my_custom_tag ``` Apart from environments, users can also pick arbitrary installed spec from their database, for instance: ``` $ spack buildcache push --base-image ubuntu:22.04 --tag some_specs my-oci-registry gcc@12 cmake $ docker run -it --rm example.com/hello/world:some_specs ``` It has many advantages over `spack containerize`: 1. No external tools required (`docker`, `buildah`, ...) 2. Creates images from locally installed Spack packages (No need to rebuild inside `docker build`, where troubleshooting build failures is notoriously hard) 3. No need for multistage builds (Spack just tarballs existing installations of runtime deps) 4. Reduced storage size / composability: when pushing multiple environments with common specs, container image layers are shared. 5. Automatic build cache: later `spack install` of the env elsewhere speeds up since the containerized environment is a build cache
2023-12-19"spack diff": add ignore option for dependencies (#41711)Peter Scheibel2-2/+4
* add trim function to `Spec` and `--ignore` option to 'spack diff' Allows user to compare two specs while ignoring the sub-DAG of a particular dependency, e.g. spack diff --ignore=mpi --ignore=zlib trilinos/abcdef trilinos/fedcba to focus on differences closer to the root of the software stack
2023-12-19CI: Disable downloading artifacts from upstream jobs (#41432)kwryankrattiger1-0/+1
* CI: Disable downloading artifacts from upstream jobs * CI: Default .base-jobs are `when:manual`
2023-12-19e4s: add julia (#41768)Harmen Stoppels1-0/+1
2023-12-19`spack config get`/`blame`: with no args, show entire config Massimiliano Culpo1-0/+7
This PR changes the default behavior of `spack config get` and `spack config blame` to print a flattened version of the entire spack configuration, including any active environment, if the commands are invoked with no section arguments. The new behavior is used in Gitlab CI to help debug CI configuration, but it can also be useful when asking for more information in issues, or when simply debugging Spack.
2023-12-18`spack develop`: convert to config (#35273)Peter Scheibel1-3/+3
Convert the 'develop' section of an environment to a dedicated configuration section. This means for example that instead of having to define `develop` specs in the `spack.yaml`, the environment can `include:` another `develop.yaml` configuration which specifies which specs should be developed in the environment. This change is not expected to be disruptive given that existing environment `spack.yaml` files will conform to the new schema. (Update 11/28/2023) I have implemented the `develop`/`undevelop` commands in terms of more-generic modification functions added to the `config` module: `change_or_add` and `update_all`. It is assumed that the semantics added here (described in 11/18 update) would be desirable to extend to other config update actions (e.g. adding compilers, changing package requirements, adding mirrors). (Update 11/18/2023) I have updated this such that `spack develop`, and `spack undevelop` to potentially modify all writable scopes, like https://github.com/spack/spack/pull/41147. https://github.com/spack/spack/pull/35307 will be useful for modifying included scopes, but generally speaking specifying a `--scope` will not be required for `spack develop`: `spack develop` will add new develop specs to whatever scope already has develop specs defined, or to the highest-priority writable scope (which should be the env scope). TODOs: - [x] If you `spack undevelop` a package which is mentioned at multiple layers of configuration, then currently this would only modify one of them. That's not technically a new issue (has always existed for configuration modification), but may be confusing to users when presented via an interface other than `spack config set` - [x] Need to add (or confirm) the ability to modify individual config files by providing a path (rather than using a scope identifier as a key to retrieve associated config). - [x] `spack develop` adds new develop specs to the scope that defines them (potentially skipping higher priority scopes to e.g. augment included scope files) --------- Co-authored-by: scheibelp <scheibelp@users.noreply.github.com> Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2023-12-11Build cache: make signed/unsigned a mirror property (#41507)Harmen Stoppels2-11/+23
* Add `signed` property to mirror config * make unsigned a tri-state: true/false overrides mirror config, none takes mirror config * test commands * Document this * add a test
2023-12-07Do not use depfile in bootstrap (#41458)Jordan Galby1-0/+2
- we don't have a fallback if make is not installed - we assume file system locking works - we don't verify that make is gnu make (bootstrapping fails on FreeBSD as a result) - there are some weird race conditions in writing spack.yaml on concurrent spack install - the view is updated after every package install instead of post environment install.
2023-12-05spack buildcache check: use same interface as push (#41378)Harmen Stoppels2-1/+7
2023-12-01py-keras: add v3.0.0 (#41356)Adam J. Stewart4-4/+12
* py-keras: add v3.0.0 * Older keras actually requires protobuf * Correct url_for_version * Capitalization is important * Keep pil and pydot deps
2023-11-30developer tools stack try 2 (#40921)Tom Scogland3-1/+91
* developer tools stack try 2 This version is actually in use locally and has largely stabilized, at least on x86. Some packages are still a challenge on ppc64le, but maybe worth keeping this working as a set. * add packages, try to get container with newer gcc * remove reuse: true * try to get cmake to build on medium, 25 minutes is too long * add lsd package and add to dev tools stack * clean up fzf dependency and sorting * Update share/spack/gitlab/cloud_pipelines/stacks/developer_tools/spack.yaml * cuda: add 12.3.0 (#40827) * Switch to dashes * yet more underscores --------- Co-authored-by: Paul R. C. Kent <kentpr@ornl.gov>
2023-11-29kokkos: add v4.2.00 (#41203)Luc Berger1-4/+4
* Kokkos: adding version 4.2.00 to the package * Kokkos: adding AMD GPU arch * kokkos@4.2.00 +sycl: patch numeric traits unit test --------- Co-authored-by: eugeneswalker <eugenesunsetwalker@gmail.com> Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2023-11-28e4s ci: disable gpu test stack (#41296)eugeneswalker1-17/+17
2023-11-28GoPackage: add new build system for Go packages (#41164)Alec Scott1-1/+1
Co-authored-by: Tom Scogland <scogland1@llnl.gov> Co-authored-by: Wouter Deconinck <wdconinc@gmail.com> Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2023-11-27CargoPackage: add new build system for Cargo packages (#41192)Alec Scott1-1/+1
Co-authored-by: Tom Scogland <scogland1@llnl.gov>
2023-11-27intel-oneapi 2024.0.0: added new version to packages (#41135)Andrey Perestoronin1-0/+3
* oneapi 2024.0.0 release * oneapi v2 directory support and some cleanups * sycl abi change requires 2024 compilers for packages that use sycl --------- Co-authored-by: Robert Cohn <robert.s.cohn@intel.com>
2023-11-18e4s oneapi stack: turn on +sycl: ginkgo, heffte, petsc, upcxx, warpx (#41157)eugeneswalker1-1/+5
* e4s oneapi stack: turn on +sycl: ginkgo, heffte, petsc, upcxx, warpx * comment out warpx; build fails; add note
2023-11-14Add papyrus to the list of broken tests (#40923)Massimiliano Culpo1-0/+1
* Disable papyrus in the neoverse v1 pipeline See https://gitlab.spack.io/spack/spack/-/jobs/8983875 The job is hanging on tests for 6 hrs. * Add papyrus to broken tests instead of removing it
2023-11-10`spack deconcretize` command (#38803)Greg Becker2-1/+23
We have two ways to concretize now: * `spack concretize` concretizes only the root specs that are not concrete in the environment. * `spack concretize -f` eliminates all cached concretization data and reconcretizes the *entire* environment. This PR adds `spack deconcretize`, which eliminates cached concretization data for a spec. This allows users greater control over what is preserved from their `spack.lock` file and what is reused when not using `spack concretize -f`. If you want to update a spec installed in your environment, you can call `spack deconcretize` on it, and that spec and any relevant dependents will be removed from the lock file. `spack concretize` has two options: * `--root`: limits deconcretized specs to *specific* roots in the environment. You can use this to deconcretize exactly one root in a `unify: false` environment. i.e., if `foo` root is a dependent of `bar`, both roots, `spack deconcretize bar` will *not* deconcretize `foo`. * `--all`: deconcretize *all* specs that match the input spec. By default `spack deconcretize` will complain about multiple matches, like `spack uninstall`.
2023-11-10info: rework spack info command to display variants better (#40998)Todd Gamblin2-2/+4
This changes variant display to use a much more legible format, and to use screen space much better (particularly on narrow terminals). It also adds color the variant display to match other parts of `spack info`. Descriptions and variant value lists that were frequently squished into a tiny column before now have closer to the full terminal width. This change also preserves any whitespace formatting present in `package.py`, so package maintainers can make easer-to-read descriptions of variant values if they want. For example, `gasnet` has had a nice description of the `conduits` variant for a while, but it was wrapped and made illegible by `spack info`. That is now fixed and the original newlines are kept. Conditional variants are grouped by their when clauses by default, but if you do not like the grouping, you can display all the variants in order with `--variants-by-name`. I'm not sure when people will prefer this, but it makes it easier to tell that a particular variant is/isn't there. I do think grouping by `when` is the better default.
2023-11-09Revert "Deactivate Cray sles, due to unavailable runner (#40291)" (#40910)eugeneswalker1-12/+12
This reverts commit 4b06862a7f3fee9352cd4834b4de7cb400cd4aa1.
2023-11-08tcl: filter compiler wrappers to avoid pointing to Spack (#40946)Massimiliano Culpo1-0/+1
2023-11-07tutorial stack: update for changes to the basics section for SC23 (#40942)Greg Becker1-6/+6
2023-11-07tutorial: use lmod@8.7.18 because @8.7.19: has bugs (#40939)Harmen Stoppels1-1/+1
2023-11-07tutorial pipeline: force gcc@12.3.0 (#40937)Harmen Stoppels1-1/+1
2023-11-06spack compiler find --[no]-mixed-toolchain (#40902)Harmen Stoppels2-4/+12
Currently there's some hacky logic in the AppleClang compiler that makes it also accept `gfortran` as a fortran compiler if `flang` is not found. This is guarded by `if sys.platform` checks s.t. it only applies to Darwin. But on Linux the feature of detecting mixed toolchains is highly requested too, cause it's rather annoying to run into a failed build of `openblas` after dozens of minutes of compiling its dependencies, just because clang doesn't have a fortran compiler. In particular in CI where the system compilers may change during system updates, it's typically impossible to fix compilers in a hand-written compilers.yaml config file: the config will almost certainly be outdated sooner or later, and maintaining one config file per target machine and writing logic to select the correct config is rather undesirable too. --- This PR introduces a flag `spack compiler find --mixed-toolchain` that fills out missing `fc` and `f77` entries in `clang` / `apple-clang` by picking the best matching `gcc`. It is enabled by default on macOS, but not on Linux, matching current behavior of `spack compiler find`. The "best matching gcc" logic and compiler path updates are identical to how compiler path dictionaries are currently flattened "horizontally" (per compiler id). This just adds logic to do the same "vertically" (across different compiler ids). So, with this change on Ubuntu 22.04: ``` $ spack compiler find --mixed-toolchain ==> Added 6 new compilers to /home/harmen/.spack/linux/compilers.yaml gcc@13.1.0 gcc@12.3.0 gcc@11.4.0 gcc@10.5.0 clang@16.0.0 clang@15.0.7 ==> Compilers are defined in the following files: /home/harmen/.spack/linux/compilers.yaml ``` you finally get: ``` compilers: - compiler: spec: clang@=15.0.7 paths: cc: /usr/bin/clang cxx: /usr/bin/clang++ f77: /usr/bin/gfortran fc: /usr/bin/gfortran flags: {} operating_system: ubuntu23.04 target: x86_64 modules: [] environment: {} extra_rpaths: [] - compiler: spec: clang@=16.0.0 paths: cc: /usr/bin/clang-16 cxx: /usr/bin/clang++-16 f77: /usr/bin/gfortran fc: /usr/bin/gfortran flags: {} operating_system: ubuntu23.04 target: x86_64 modules: [] environment: {} extra_rpaths: [] ``` The "best gcc" is automatically default system gcc, since it has no suffixes / prefixes.
2023-11-05spack env activate: create & activate default environment without args (#40756)Harmen Stoppels3-4/+11
This PR implements the concept of "default environment", which doesn't have to be created explicitly. The aim is to lower the barrier for adopting environments. To (create and) activate the default environment, run ``` $ spack env activate ``` This mimics the behavior of ``` $ cd ``` which brings you to your home directory. This is not a breaking change, since `spack env activate` without arguments currently errors. It is similar to the already existing `spack env activate --temp` command which always creates an env in a temporary directory, the difference is that the default environment is a managed / named environment named `default`. The name `default` is not a reserved name, it's just that `spack env activate` creates it for you if you don't have it already. With this change, you can get started with environments faster: ``` $ spack env activate [--prompt] $ spack install --add x y z ``` instead of ``` $ spack env create default ==> Created environment 'default in /Users/harmenstoppels/spack/var/spack/environments/default ==> You can activate this environment with: ==> spack env activate default $ spack env activate [--prompt] default $ spack install --add x y z ``` Notice that Spack supports switching (but not stacking) environments, so the parallel with `cd` is pretty clear: ``` $ spack env activate named_env $ spack env status ==> In environment named_env $ spack env activate $ spack env status ==> In environment default ```
2023-11-05bugfix: compress aliases for first command in completion (#40890)Todd Gamblin2-2/+2
This completes to `spack concretize`: ``` spack conc<tab> ``` but this still gets hung up on the difference between `concretize` and `concretise`: ``` spack -e . conc<tab> ``` We were checking `"$COMP_CWORD" = 1`, which tracks the word on the command line including any flags and their args, but we should track `"$COMP_CWORD_NO_FLAGS" = 1` to figure out if the arg we're completing is the first real command.
2023-11-05Environments: Add support for including definitions files (#33960)Tamara Dahlgren1-3/+3
This PR adds support for including separate definitions from `spack.yaml`. Supporting the inclusion of files with definitions enables user to make curated/standardized collections of packages that can re-used by others.
2023-11-04hdf5-vol-async: better specify dependency condition (#40882)Massimiliano Culpo1-0/+2
2023-11-04sundials +sycl: add cxxflags=-fsycl via flag_handler (#40845)eugeneswalker1-1/+1
2023-11-03tau: update 2.33 hash, add syscall variant (#40851)eugeneswalker5-9/+9
Co-authored-by: wspear <wjspear@gmail.com>
2023-11-02depfile: deal with empty / non-concrete env (#40816)Harmen Stoppels1-1/+1
2023-10-31force color in subshell if not SPACK_COLOR (#40782)Harmen Stoppels1-4/+4
2023-10-31ci: bump tutorial image and toolchain (#40795)Harmen Stoppels2-8/+8
2023-10-31tutorial: replace zlib -> gmake to avoid deprecated versions (#40769)Harmen Stoppels1-13/+8
2023-10-30ci: print colored specs in concretization progress (#40711)Harmen Stoppels1-2/+2
2023-10-27e4s ci stacks: add exago specs (#40712)eugeneswalker3-0/+6
* e4s ci: add exago +cuda, +rocm builds * exago: rename 5-18-2022-snapshot to snapshot.5-18-2022 * disable exago +rocm for non-external rocm ci install * note that hiop +rocm fails to find hip libraries when they are spack-installed
2023-10-27OCI buildcache (#38358)Harmen Stoppels2-10/+30
Credits to @ChristianKniep for advocating the idea of OCI image layers being identical to spack buildcache tarballs. With this you can configure an OCI registry as a buildcache: ```console $ spack mirror add my_registry oci://user/image # Dockerhub $ spack mirror add my_registry oci://ghcr.io/haampie/spack-test # GHCR $ spack mirror set --push --oci-username ... --oci-password ... my_registry # set login credentials ``` which should result in this config: ```yaml mirrors: my_registry: url: oci://ghcr.io/haampie/spack-test push: access_pair: [<username>, <password>] ``` It can be used like any other registry ``` spack buildcache push my_registry [specs...] ``` It will upload the Spack tarballs in parallel, as well as manifest + config files s.t. the binaries are compatible with `docker pull` or `skopeo copy`. In fact, a base image can be added to get a _runnable_ image: ```console $ spack buildcache push --base-image ubuntu:23.04 my_registry python Pushed ... as [image]:python-3.11.2-65txfcpqbmpawclvtasuog4yzmxwaoia.spack $ docker run --rm -it [image]:python-3.11.2-65txfcpqbmpawclvtasuog4yzmxwaoia.spack ``` which should really be a game changer for sharing binaries. Further, all content-addressable blobs that are downloaded and verified will be cached in Spack's download cache. This should make repeated `push` commands faster, as well as `push` followed by a separate `update-index` command. An end to end example of how to use this in Github Actions is here: **https://github.com/haampie/spack-oci-buildcache-example** TODO: - [x] Generate environment modifications in config so PATH is set up - [x] Enrich config with Spack's `spec` json (this is allowed in the OCI specification) - [x] When ^ is done, add logic to create an index in say `<image>:index` by fetching all config files (using OCI distribution discovery API) - [x] Add logic to use object storage in an OCI registry in `spack install`. - [x] Make the user pick the base image for generated OCI images. - [x] Update buildcache install logic to deal with absolute paths in tarballs - [x] Merge with `spack buildcache` command - [x] Merge #37441 (included here) - [x] Merge #39077 (included here) - [x] #39187 + #39285 - [x] #39341 - [x] Not a blocker: #35737 fixes correctness run env for the generated container images NOTE: 1. `oci://` is unfortunately taken, so it's being abused in this PR to mean "oci type mirror". `skopeo` uses `docker://` which I'd like to avoid, given that classical docker v1 registries are not supported. 2. this is currently `https`-only, given that basic auth is used to login. I _could_ be convinced to allow http, but I'd prefer not to, given that for a `spack buildcache push` command multiple domains can be involved (auth server, source of base image, destination registry). Right now, no urllib http handler is added, so redirects to https and auth servers with http urls will simply result in a hard failure. CAVEATS: 1. Signing is not implemented in this PR. `gpg --clearsign` is not the nicest solution, since (a) the spec.json is merged into the image config, which must be valid json, and (b) it would be better to sign the manifest (referencing both config/spec file and tarball) using more conventional image signing tools 2. `spack.binary_distribution.push` is not yet implemented for the OCI buildcache, only `spack buildcache push` is. This is because I'd like to always push images + deps to the registry, so that it's `docker pull`-able, whereas in `spack ci` we really wanna push an individual package without its deps to say `pr-xyz`, while its deps reside in some `develop` buildcache. 3. The `push -j ...` flag only works for OCI buildcache, not for others
2023-10-27ci: spack compiler find should list extra config scopes (#40727)Harmen Stoppels1-1/+6
otherwise it detected pre-configured compilers in an potentially different way.
2023-10-26modules: no --delim option if separator is colon character (#39010)Xavier Delaruelle1-1/+13
Update Tcl modulefile template to simplify generated `append-path`, `prepend-path` and `remove-path` commands and improve their readability. If path element delimiter is colon character, do not set the `--delim` option as it is the default delimiter value.
2023-10-26spack checksum: show long flags in usage output (#40407)Harmen Stoppels2-9/+9
2023-10-25ci: don't put compilers in config (#40700)Harmen Stoppels7-89/+2
* ci: don't register detectable compilers Cause they go out of sync... * remove intel compiler, it can be detected too * Do not run spack compiler find since compilers are registered in concretize job already * trilinos: work around +stokhos +cuda +superlu-dist bug due to EMPTY macro
2023-10-25ci: darwin aarch64 use apple-clang-15 tag (#40706)Harmen Stoppels2-2/+2