summaryrefslogtreecommitdiff
path: root/share
AgeCommit message (Collapse)AuthorFilesLines
2023-11-10`spack deconcretize` command (#38803)Greg Becker2-1/+23
We have two ways to concretize now: * `spack concretize` concretizes only the root specs that are not concrete in the environment. * `spack concretize -f` eliminates all cached concretization data and reconcretizes the *entire* environment. This PR adds `spack deconcretize`, which eliminates cached concretization data for a spec. This allows users greater control over what is preserved from their `spack.lock` file and what is reused when not using `spack concretize -f`. If you want to update a spec installed in your environment, you can call `spack deconcretize` on it, and that spec and any relevant dependents will be removed from the lock file. `spack concretize` has two options: * `--root`: limits deconcretized specs to *specific* roots in the environment. You can use this to deconcretize exactly one root in a `unify: false` environment. i.e., if `foo` root is a dependent of `bar`, both roots, `spack deconcretize bar` will *not* deconcretize `foo`. * `--all`: deconcretize *all* specs that match the input spec. By default `spack deconcretize` will complain about multiple matches, like `spack uninstall`.
2023-11-10info: rework spack info command to display variants better (#40998)Todd Gamblin2-2/+4
This changes variant display to use a much more legible format, and to use screen space much better (particularly on narrow terminals). It also adds color the variant display to match other parts of `spack info`. Descriptions and variant value lists that were frequently squished into a tiny column before now have closer to the full terminal width. This change also preserves any whitespace formatting present in `package.py`, so package maintainers can make easer-to-read descriptions of variant values if they want. For example, `gasnet` has had a nice description of the `conduits` variant for a while, but it was wrapped and made illegible by `spack info`. That is now fixed and the original newlines are kept. Conditional variants are grouped by their when clauses by default, but if you do not like the grouping, you can display all the variants in order with `--variants-by-name`. I'm not sure when people will prefer this, but it makes it easier to tell that a particular variant is/isn't there. I do think grouping by `when` is the better default.
2023-11-09Revert "Deactivate Cray sles, due to unavailable runner (#40291)" (#40910)eugeneswalker1-12/+12
This reverts commit 4b06862a7f3fee9352cd4834b4de7cb400cd4aa1.
2023-11-08tcl: filter compiler wrappers to avoid pointing to Spack (#40946)Massimiliano Culpo1-0/+1
2023-11-07tutorial stack: update for changes to the basics section for SC23 (#40942)Greg Becker1-6/+6
2023-11-07tutorial: use lmod@8.7.18 because @8.7.19: has bugs (#40939)Harmen Stoppels1-1/+1
2023-11-07tutorial pipeline: force gcc@12.3.0 (#40937)Harmen Stoppels1-1/+1
2023-11-06spack compiler find --[no]-mixed-toolchain (#40902)Harmen Stoppels2-4/+12
Currently there's some hacky logic in the AppleClang compiler that makes it also accept `gfortran` as a fortran compiler if `flang` is not found. This is guarded by `if sys.platform` checks s.t. it only applies to Darwin. But on Linux the feature of detecting mixed toolchains is highly requested too, cause it's rather annoying to run into a failed build of `openblas` after dozens of minutes of compiling its dependencies, just because clang doesn't have a fortran compiler. In particular in CI where the system compilers may change during system updates, it's typically impossible to fix compilers in a hand-written compilers.yaml config file: the config will almost certainly be outdated sooner or later, and maintaining one config file per target machine and writing logic to select the correct config is rather undesirable too. --- This PR introduces a flag `spack compiler find --mixed-toolchain` that fills out missing `fc` and `f77` entries in `clang` / `apple-clang` by picking the best matching `gcc`. It is enabled by default on macOS, but not on Linux, matching current behavior of `spack compiler find`. The "best matching gcc" logic and compiler path updates are identical to how compiler path dictionaries are currently flattened "horizontally" (per compiler id). This just adds logic to do the same "vertically" (across different compiler ids). So, with this change on Ubuntu 22.04: ``` $ spack compiler find --mixed-toolchain ==> Added 6 new compilers to /home/harmen/.spack/linux/compilers.yaml gcc@13.1.0 gcc@12.3.0 gcc@11.4.0 gcc@10.5.0 clang@16.0.0 clang@15.0.7 ==> Compilers are defined in the following files: /home/harmen/.spack/linux/compilers.yaml ``` you finally get: ``` compilers: - compiler: spec: clang@=15.0.7 paths: cc: /usr/bin/clang cxx: /usr/bin/clang++ f77: /usr/bin/gfortran fc: /usr/bin/gfortran flags: {} operating_system: ubuntu23.04 target: x86_64 modules: [] environment: {} extra_rpaths: [] - compiler: spec: clang@=16.0.0 paths: cc: /usr/bin/clang-16 cxx: /usr/bin/clang++-16 f77: /usr/bin/gfortran fc: /usr/bin/gfortran flags: {} operating_system: ubuntu23.04 target: x86_64 modules: [] environment: {} extra_rpaths: [] ``` The "best gcc" is automatically default system gcc, since it has no suffixes / prefixes.
2023-11-05spack env activate: create & activate default environment without args (#40756)Harmen Stoppels3-4/+11
This PR implements the concept of "default environment", which doesn't have to be created explicitly. The aim is to lower the barrier for adopting environments. To (create and) activate the default environment, run ``` $ spack env activate ``` This mimics the behavior of ``` $ cd ``` which brings you to your home directory. This is not a breaking change, since `spack env activate` without arguments currently errors. It is similar to the already existing `spack env activate --temp` command which always creates an env in a temporary directory, the difference is that the default environment is a managed / named environment named `default`. The name `default` is not a reserved name, it's just that `spack env activate` creates it for you if you don't have it already. With this change, you can get started with environments faster: ``` $ spack env activate [--prompt] $ spack install --add x y z ``` instead of ``` $ spack env create default ==> Created environment 'default in /Users/harmenstoppels/spack/var/spack/environments/default ==> You can activate this environment with: ==> spack env activate default $ spack env activate [--prompt] default $ spack install --add x y z ``` Notice that Spack supports switching (but not stacking) environments, so the parallel with `cd` is pretty clear: ``` $ spack env activate named_env $ spack env status ==> In environment named_env $ spack env activate $ spack env status ==> In environment default ```
2023-11-05bugfix: compress aliases for first command in completion (#40890)Todd Gamblin2-2/+2
This completes to `spack concretize`: ``` spack conc<tab> ``` but this still gets hung up on the difference between `concretize` and `concretise`: ``` spack -e . conc<tab> ``` We were checking `"$COMP_CWORD" = 1`, which tracks the word on the command line including any flags and their args, but we should track `"$COMP_CWORD_NO_FLAGS" = 1` to figure out if the arg we're completing is the first real command.
2023-11-05Environments: Add support for including definitions files (#33960)Tamara Dahlgren1-3/+3
This PR adds support for including separate definitions from `spack.yaml`. Supporting the inclusion of files with definitions enables user to make curated/standardized collections of packages that can re-used by others.
2023-11-04hdf5-vol-async: better specify dependency condition (#40882)Massimiliano Culpo1-0/+2
2023-11-04sundials +sycl: add cxxflags=-fsycl via flag_handler (#40845)eugeneswalker1-1/+1
2023-11-03tau: update 2.33 hash, add syscall variant (#40851)eugeneswalker5-9/+9
Co-authored-by: wspear <wjspear@gmail.com>
2023-11-02depfile: deal with empty / non-concrete env (#40816)Harmen Stoppels1-1/+1
2023-10-31force color in subshell if not SPACK_COLOR (#40782)Harmen Stoppels1-4/+4
2023-10-31ci: bump tutorial image and toolchain (#40795)Harmen Stoppels2-8/+8
2023-10-31tutorial: replace zlib -> gmake to avoid deprecated versions (#40769)Harmen Stoppels1-13/+8
2023-10-30ci: print colored specs in concretization progress (#40711)Harmen Stoppels1-2/+2
2023-10-27e4s ci stacks: add exago specs (#40712)eugeneswalker3-0/+6
* e4s ci: add exago +cuda, +rocm builds * exago: rename 5-18-2022-snapshot to snapshot.5-18-2022 * disable exago +rocm for non-external rocm ci install * note that hiop +rocm fails to find hip libraries when they are spack-installed
2023-10-27OCI buildcache (#38358)Harmen Stoppels2-10/+30
Credits to @ChristianKniep for advocating the idea of OCI image layers being identical to spack buildcache tarballs. With this you can configure an OCI registry as a buildcache: ```console $ spack mirror add my_registry oci://user/image # Dockerhub $ spack mirror add my_registry oci://ghcr.io/haampie/spack-test # GHCR $ spack mirror set --push --oci-username ... --oci-password ... my_registry # set login credentials ``` which should result in this config: ```yaml mirrors: my_registry: url: oci://ghcr.io/haampie/spack-test push: access_pair: [<username>, <password>] ``` It can be used like any other registry ``` spack buildcache push my_registry [specs...] ``` It will upload the Spack tarballs in parallel, as well as manifest + config files s.t. the binaries are compatible with `docker pull` or `skopeo copy`. In fact, a base image can be added to get a _runnable_ image: ```console $ spack buildcache push --base-image ubuntu:23.04 my_registry python Pushed ... as [image]:python-3.11.2-65txfcpqbmpawclvtasuog4yzmxwaoia.spack $ docker run --rm -it [image]:python-3.11.2-65txfcpqbmpawclvtasuog4yzmxwaoia.spack ``` which should really be a game changer for sharing binaries. Further, all content-addressable blobs that are downloaded and verified will be cached in Spack's download cache. This should make repeated `push` commands faster, as well as `push` followed by a separate `update-index` command. An end to end example of how to use this in Github Actions is here: **https://github.com/haampie/spack-oci-buildcache-example** TODO: - [x] Generate environment modifications in config so PATH is set up - [x] Enrich config with Spack's `spec` json (this is allowed in the OCI specification) - [x] When ^ is done, add logic to create an index in say `<image>:index` by fetching all config files (using OCI distribution discovery API) - [x] Add logic to use object storage in an OCI registry in `spack install`. - [x] Make the user pick the base image for generated OCI images. - [x] Update buildcache install logic to deal with absolute paths in tarballs - [x] Merge with `spack buildcache` command - [x] Merge #37441 (included here) - [x] Merge #39077 (included here) - [x] #39187 + #39285 - [x] #39341 - [x] Not a blocker: #35737 fixes correctness run env for the generated container images NOTE: 1. `oci://` is unfortunately taken, so it's being abused in this PR to mean "oci type mirror". `skopeo` uses `docker://` which I'd like to avoid, given that classical docker v1 registries are not supported. 2. this is currently `https`-only, given that basic auth is used to login. I _could_ be convinced to allow http, but I'd prefer not to, given that for a `spack buildcache push` command multiple domains can be involved (auth server, source of base image, destination registry). Right now, no urllib http handler is added, so redirects to https and auth servers with http urls will simply result in a hard failure. CAVEATS: 1. Signing is not implemented in this PR. `gpg --clearsign` is not the nicest solution, since (a) the spec.json is merged into the image config, which must be valid json, and (b) it would be better to sign the manifest (referencing both config/spec file and tarball) using more conventional image signing tools 2. `spack.binary_distribution.push` is not yet implemented for the OCI buildcache, only `spack buildcache push` is. This is because I'd like to always push images + deps to the registry, so that it's `docker pull`-able, whereas in `spack ci` we really wanna push an individual package without its deps to say `pr-xyz`, while its deps reside in some `develop` buildcache. 3. The `push -j ...` flag only works for OCI buildcache, not for others
2023-10-27ci: spack compiler find should list extra config scopes (#40727)Harmen Stoppels1-1/+6
otherwise it detected pre-configured compilers in an potentially different way.
2023-10-26modules: no --delim option if separator is colon character (#39010)Xavier Delaruelle1-1/+13
Update Tcl modulefile template to simplify generated `append-path`, `prepend-path` and `remove-path` commands and improve their readability. If path element delimiter is colon character, do not set the `--delim` option as it is the default delimiter value.
2023-10-26spack checksum: show long flags in usage output (#40407)Harmen Stoppels2-9/+9
2023-10-25ci: don't put compilers in config (#40700)Harmen Stoppels7-89/+2
* ci: don't register detectable compilers Cause they go out of sync... * remove intel compiler, it can be detected too * Do not run spack compiler find since compilers are registered in concretize job already * trilinos: work around +stokhos +cuda +superlu-dist bug due to EMPTY macro
2023-10-25ci: darwin aarch64 use apple-clang-15 tag (#40706)Harmen Stoppels2-2/+2
2023-10-23Adios2: add kokkos variant (#40623)Vicente Bolea3-1/+5
* adios2: update variants and dependencies * adios2: add kokkos rocm|cuda|sycl variant * e4s oneapi ci stack: add adios2 +sycl * e4s ci stack: add adios2 +rocm * [@spackbot] updating style on behalf of vicentebolea * Apply suggestions from code review * adios2: fixed cuda variant * update ecp-data-vis-sdk * Update share/spack/gitlab/cloud_pipelines/stacks/e4s-power/spack.yaml --------- Co-authored-by: eugeneswalker <eugenesunsetwalker@gmail.com> Co-authored-by: vicentebolea <vicentebolea@users.noreply.github.com>
2023-10-20concretize separately: show concretization time per spec as they concretize ↵Harmen Stoppels1-2/+2
when verbose (#40634)
2023-10-20gromacs +cp2k: build in CI (#40494)Harmen Stoppels3-0/+3
* gromacs +cp2k: build in CI * libxsmm: x86 only * attempt to fix dbcsr + new mpich * use c11 standard * gromacs: does not depend on dbcsr * cp2k: build with cmake in CI, s.t. dbcsr is a separate package * cp2k: cmake patches for config files and C/C++ std * cp2k: remove unnecessary constraints due to patch
2023-10-19Improve setup build / run / test environment (#35737)Harmen Stoppels2-7/+5
This adds a `SetupContext` class which is responsible for setting package.py module globals, and computing the changes to environment variables for the build, test or run context. The class uses `effective_deptypes` which takes a list of specs (e.g. single item of a spec to build, or a list of environment roots) and a context (build, run, test), and outputs a flat list of specs that affect the environment together with a flag in what way they do so. This list is topologically ordered from root to leaf, so that one can be assured that dependents override variables set by dependencies, not the other way around. This is used to replace the logic in `modifications_from_dependencies`, which has several issues: missing calls to `setup_run_environment`, and the order in which operations are applied. Further, it should improve performance a bit in certain cases, since `effective_deptypes` run in O(v + e) time, whereas `spack env activate` currently can take up to O(v^2 + e) time due to loops over roots. Each edge in the DAG is visited once by calling `effective_deptypes` with `env.concrete_roots()`. By marking and propagating flags through the DAG, this commit also fixes a bug where Spack wouldn't call `setup_run_environment` for runtime dependencies of link dependencies. And this PR ensures that Spack correctly sets up the runtime environment of direct build dependencies. Regarding test dependencies: in a build context they are are build-time test deps, whereas in a test context they are install-time test deps. Since there are no means to distinguish the build/install type test deps, they're both. Further changes: - all `package.py` module globals are guaranteed to be set before any of the `setup_(dependent)_(run|build)_env` functions is called - traversal order during setup: first the group of externals, then the group of non-externals, with specs in each group traversed topological (dependencies are setup before dependents) - modules: only ever call `setup_dependent_run_environment` of *direct* link/run type deps - the marker in `set_module_variables_for_package` is dropped, since we should call the method once per spec. This allows us to set only a cheap subset of globals on the module: for example it's not necessary to compute the expensive `cmake_args` and w/e if the spec under consideration is not the root node to be built. - `spack load`'s `--only` is deprecated (it has no effect now), and `spack load x` now means: do everything that's required for `x` to work at runtime, which requires runtime deps to be setup -- just like `spack env activate`. - `spack load` no longer loads build deps (of build deps) ... - `spack env activate` on partially installed or broken environments: this is all or nothing now. If some spec errors during setup of its runtime env, you'll only get the unconditional variables + a warning that says the runtime changes for specs couldn't be applied. - Remove traversal in upward direction from `setup_dependent_*` in packages. Upward traversal may iterate to specs that aren't children of the roots (e.g. zlib / python have hundreds of dependents, only a small fraction is reachable from the roots. Packages should only modify the direct dependent they receive as an argument)
2023-10-19gitlab ci: Rework how mirrors are configured (#39939)Scott Wittenburg30-81/+77
Improve how mirrors are used in gitlab ci, where we have until now thought of them as only a string. By configuring ci mirrors ahead of time using the proposed mirror templates, and by taking advantage of the expressiveness that spack now has for mirrors, this PR will allow us to easily switch the protocol/url we use for fetching binary dependencies. This change also deprecates some gitlab functionality and marks it for removal in Spack 0.23: - arguments to "spack ci generate": * --buildcache-destination * --copy-to - gitlab configuration options: * enable-artifacts-buildcache * temporary-storage-url-prefix
2023-10-19ci: remove incorrect compilers.yaml (#40610)Harmen Stoppels1-27/+0
2023-10-18AutotoolsPackage / MakefilePackage: add gmake build dependency (#40380)Harmen Stoppels2-36/+36
2023-10-17Support `spack env activate --with-view <name> <env>` (#40549)Harmen Stoppels2-6/+6
Currently `spack env activate --with-view` exists, but is a no-op. So, it is not too much of a breaking change to make this redundant flag accept a value `spack env activate --with-view <name>` which activates a particular view by name. The view name is stored in `SPACK_ENV_VIEW`. This also fixes an issue where deactivating a view that was activated with `--without-view` possibly removes entries from PATH, since now we keep track of whether the default view was "enabled" or not.
2023-10-12Remove deprecated "extra_instructions" option for containers (#40365)Massimiliano Culpo2-13/+0
2023-10-11Update bootstrap buildcache to support Python 3.12 (#40404)Massimiliano Culpo6-479/+650
* Add support for Python 3.12 * Use optimized build of clingo
2023-10-10e4s arm stack: duplicate and target neoverse v1 (#40369)eugeneswalker2-13/+17
* e4s arm stack: duplicate and target both neoverse n1, v1 * remove neoverse_n1 target until issue #40397 is resolved
2023-10-06Change 'exit' to 'return' in setup-env.sh (#36137)Alex Richert1-1/+1
* Change 'exit' to 'return' in `setup-env.sh` to avoid losing shell in some cases when sourcing twice.
2023-10-06Make "minimal" the default duplicate strategy (#39621)Massimiliano Culpo1-10/+6
* Allow branching out of the "generic build" unification set For cases like the one in https://github.com/spack/spack/pull/39661 we need to relax rules on unification sets. The issue is that, right now, nodes in the "generic build" unification set are unified together with their build dependencies. This was done out of caution to avoid the risk of circular dependencies, which would ultimately cause a very slow solve. For build-tools like Cython, however, the build dependencies is masked by a long chain of "build, run" dependencies that belong in the "generic build" unification space. To allow splitting on cases like this, we relax the rule disallowing branching out of the "generic build" unification set. * Fix issue with pure build virtual dependencies Pure build virtual dependencies were not accounted properly in the list of possible virtuals. This caused some facts connecting virtuals to the corresponding providers to not be emitted, and in the end lead to unsat problems. * Fixed a few issues in packages py-gevent: restore dependency on py-cython@3 jsoncpp: fix typo in build dependency ecp-data-vis-sdk: update spack.yaml and cmake recipe py-statsmodels: add v0.13.5 * Make dependency on "blt" of type "build"
2023-10-05Revert "cray rhel: disable due to runner issues (#40324)" (#40335)eugeneswalker1-16/+16
This reverts commit bf7f54449ba8ed157c9ee258007e0a7a509600cf.
2023-10-05cray rhel: disable due to runner issues (#40324)Harmen Stoppels1-16/+16
2023-10-04ci: pull E4S images from github instead of dockerhub (#40307)Scott Wittenburg6-21/+21
2023-10-04e4s arm: disable bricks due to target=aarch64 not being respected (#40308)eugeneswalker1-6/+6
2023-10-03e4s ci stacks: sync with e4s-23.08 (#40263)eugeneswalker6-289/+1212
* e4s amd64 gcc ci stack: sync with e4s-23.08 * e4s amd64 oneapi ci stack: sync with e4s-23.08 * e4s ppc64 gcc ci stack: sync with e4s-23.08 * add new ci stack: e4s amd64 gcc w/ external rocm * add new ci stack: e4s arm gcc ci * updates * py-scipy: -fvisibility issue is resolved in 2023.1.0: #39464 * paraview oneapi fails * comment out pkgs that fail to build on power * fix arm stack name * fix cabana +cuda specification * comment out failing spces * visit fails build on arm * comment out slepc arm builds due to make issue * comment out failing dealii arm builds
2023-10-03Deactivate Cray sles, due to unavailable runner (#40291)Massimiliano Culpo1-12/+12
This reverts commit 027409120408eb4d41d8bdd82a13c34b3ac7fea9.
2023-10-02Implemented +sycl for slate (#39927)G-Ragghianti1-0/+1
* Implemented +sycl for slate * style * style * add slate +sycl to ci * slate +sycl: explicitly specify +mpi * comment out slate+sycl+mpi in e4s oneapi stack * removing requirement of intel-oneapi-mpi * Added slate+sycl to e4s oneapi stack * Removing obsolete comment --------- Co-authored-by: eugeneswalker <eugenesunsetwalker@gmail.com>
2023-10-01py-scipy: -fvisibility issue is resolved in 2023.1.0: (#39464)eugeneswalker1-0/+3
* py-scipy: -fvisibility issue is resolved in 2023.1.0: * e4s oneapi ci: add py-scipy
2023-09-30e4s cray sles ci: expand spec list (#40162)eugeneswalker1-13/+127
* e4s cray sles stack: expand spec list * remove unnecessary packages:trilinos:one_of
2023-09-30e4s ci: add packages: drishti, dxt-explorer (#39597)eugeneswalker3-0/+6
* e4s ci: add packages: drishti, dxt-explorer * e4s oneapi ci: comment out dxt-explorer until r%oneapi issue #40257 is resolved
2023-09-29Test package detection in a systematic way (#18175)Massimiliano Culpo2-1/+19
This PR adds a new audit sub-command to check that detection of relevant packages is performed correctly in a few scenarios mocking real use-cases. The data for each package being tested is in a YAML file called detection_test.yaml alongside the corresponding package.py file. This is to allow encoding detection tests for compilers and other widely used tools, in preparation for compilers as dependencies.