summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2023-11-02bugfix: computing NodeID2 in requirement node_flag_source (#40846)Greg Becker1-1/+1
2023-11-02depfile: deal with empty / non-concrete env (#40816)Harmen Stoppels3-9/+35
2023-11-02spack external find: fix multi-arch troubles (#33973)Harmen Stoppels3-16/+98
2023-11-02env remove: add a unit test removing two environments (#40814)Tamara Dahlgren1-11/+40
2023-11-01Cherry-picking virtual dependencies (#35322)Massimiliano Culpo18-142/+1035
This PR makes it possible to select only a subset of virtual dependencies from a spec that _may_ provide more. To select providers, a syntax to specify edge attributes is introduced: ``` hdf5 ^[virtuals=mpi] mpich ``` With that syntax we can concretize specs like: ```console $ spack spec strumpack ^[virtuals=mpi] intel-parallel-studio+mkl ^[virtuals=lapack] openblas ``` On `develop` this would currently fail with: ```console $ spack spec strumpack ^intel-parallel-studio+mkl ^openblas ==> Error: Spec cannot include multiple providers for virtual 'blas' Requested 'intel-parallel-studio' and 'openblas' ``` In package recipes, virtual specs that are declared in the same `provides` directive need to be provided _together_. This means that e.g. `openblas`, which has: ```python provides("blas", "lapack") ``` needs to provide both `lapack` and `blas` when requested to provide at least one of them. ## Additional notes This capability is needed to model compilers. Assuming that languages are treated like virtual dependencies, we might want e.g. to use LLVM to compile C/C++ and Gnu GCC to compile Fortran. This can be accomplished by the following[^1]: ``` hdf5 ^[virtuals=c,cxx] llvm ^[virtuals=fortran] gcc ``` [^1]: We plan to add some syntactic sugar around this syntax, and reuse the `%` sigil to avoid having a lot of boilerplate around compilers. Modifications: - [x] Add syntax to interact with edge attributes from spec literals - [x] Add concretization logic to be able to cherry-pick virtual dependencies - [x] Extend semantic of the `provides` directive to express when virtuals need to be provided together - [x] Add unit-tests and documentation
2023-11-01Executable.add_default_arg: multiple (#40801)Harmen Stoppels3-7/+5
2023-11-01Parser: fix ambiguity with whitespace in version ranges (#40344)Harmen Stoppels2-14/+27
Allowing white space around `:` in version ranges introduces an ambiguity: ``` a@1: b ``` parses as `a@1:b` but should really be parsed as two separate specs `a@1:` and `b`. With white space disallowed around `:` in ranges, the ambiguity is resolved.
2023-11-01`SetupContext.get_env_modifications` fixes and documentation (#40683)Harmen Stoppels3-74/+83
Call setup_dependent_run_environment on both link and run edges, instead of only run edges, which restores old behavior. Move setup_build_environment into get_env_modifications Also call setup_run_environment on direct build deps, since their run environment has to be set up.
2023-11-01Revert python build isolation & setuptools source install (#40796)Harmen Stoppels1-99/+8
* Revert "Improve build isolation in PythonPipBuilder (#40224)" This reverts commit 0f43074f3e93d13445ea662a13f5672f960947c2. * Revert "py-setuptools: sdist + rpath patch backport (#40205)" This reverts commit 512e41a84aa794ec0cc53872aaa6c228c36e0b49.
2023-11-01Fix env remove indentation (#40811)Tamara Dahlgren1-21/+19
2023-10-31fix create/remove env with invalid spack.yaml (#39898)Luisa Burini3-30/+60
* fix create/remove env with invalid spack.yaml * fix isort error * fix env ident unittests * Fix pull request points
2023-10-31Fix cflags requirements (#40639)Peter Scheibel2-13/+22
2023-10-31Fix interaction of spec literals that propagate variants with unify:false ↵Massimiliano Culpo3-1/+53
(#40789) * Add tests to ensure variant propagation syntax can round-trip to/from string * Add a regression test for the bug in 35298 * Reconstruct the spec constraints in the worker process Specs do not preserve any information on propagation of variants when round-tripping to/from JSON (which we use to pickle), but preserve it when round-tripping to/from strings. Therefore, we pass a spec literal to the worker and reconstruct the Spec objects there.
2023-10-31spack checksum: improve signature (#40800)Harmen Stoppels1-3/+9
2023-10-31spack checksum: fix error when initial filter yields empty list (#40799)Harmen Stoppels1-6/+5
2023-10-30docs: update `license()` docs with examples and links (#40598)Todd Gamblin2-11/+45
- [x] Add links to information people are going to want to know when adding license information to their packages (namely OSI licenses and SPDX identifiers). - [x] Update the packaging docs for `license()` with Spack as an example for `when=`. After all, it's a dual-licensed package that changed once in the past. - [x] Add link to https://spdx.org/licenses/ in the `spack create` boilerplate as well.
2023-10-31build(deps): bump black from 23.9.1 to 23.10.1 in /lib/spack/docs (#40680)dependabot[bot]1-1/+1
Bumps [black](https://github.com/psf/black) from 23.9.1 to 23.10.1. - [Release notes](https://github.com/psf/black/releases) - [Changelog](https://github.com/psf/black/blob/main/CHANGES.md) - [Commits](https://github.com/psf/black/compare/23.9.1...23.10.1) --- updated-dependencies: - dependency-name: black dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-30ci: print colored specs in concretization progress (#40711)Harmen Stoppels3-30/+31
2023-10-30tty: flush immediately (#40774)Harmen Stoppels1-0/+1
2023-10-30binary_distribution.py: fix type annotation singleton (#40572)Harmen Stoppels4-27/+21
Convince the language server it's really just a BinaryCacheIndex, otherwise it defaults to thinking it's Singleton, and can't autocomplete etc.
2023-10-30ASP-based solver: avoid cycles in clingo using hidden directive (#40720)Massimiliano Culpo3-50/+4
The code should be functonally equivalent to what it was before, but now to avoid cycles by design we are using a "hidden" feature of clingo
2023-10-27MSVC: detection from registry (#38500)John W. Parent3-55/+168
Typically MSVC is detected via the VSWhere program. However, this may not be available, or may be installed in an unpredictable location. This PR adds an additional approach via Windows Registry queries to determine VS install location root. Additionally: * Construct vs_install_paths after class-definition time (move it to variable-access time). * Skip over keys for which a user does not have read permissions when performing searches (previously the presence of these keys would have caused an error, regardless of whether they were needed). * Extend helper functionality with option for regex matching on registry keys vs. exact string matching. * Some internal refactoring: remove boolean parameters in some cases where the function was always called with the same value (e.g. `find_subkey`)
2023-10-27External finding: update default paths; treat .bat as executable on Windows ↵John W. Parent4-82/+89
(#39850) .bat or .exe files can be considered executable on Windows. This PR expands the regex for detectable packages to allow for the detection of packages that vendor .bat wrappers (intel mpi for example). Additional changes: * Outside of Windows, when searching for executables `path_hints=None` was used to indicate that default path hints should be provided, and `[]` was taken to mean that no defaults should be chosen (in that case, nothing is searched); behavior on Windows has now been updated to match. * Above logic for handling of `path_hints=[]` has also been extended to library search (for both Linux and Windows). * All exceptions for external packages were documented as timeout errors: this commit adds a distinction for other types of errors in warning messages to the user.
2023-10-27OCI buildcache (#38358)Harmen Stoppels22-232/+3449
Credits to @ChristianKniep for advocating the idea of OCI image layers being identical to spack buildcache tarballs. With this you can configure an OCI registry as a buildcache: ```console $ spack mirror add my_registry oci://user/image # Dockerhub $ spack mirror add my_registry oci://ghcr.io/haampie/spack-test # GHCR $ spack mirror set --push --oci-username ... --oci-password ... my_registry # set login credentials ``` which should result in this config: ```yaml mirrors: my_registry: url: oci://ghcr.io/haampie/spack-test push: access_pair: [<username>, <password>] ``` It can be used like any other registry ``` spack buildcache push my_registry [specs...] ``` It will upload the Spack tarballs in parallel, as well as manifest + config files s.t. the binaries are compatible with `docker pull` or `skopeo copy`. In fact, a base image can be added to get a _runnable_ image: ```console $ spack buildcache push --base-image ubuntu:23.04 my_registry python Pushed ... as [image]:python-3.11.2-65txfcpqbmpawclvtasuog4yzmxwaoia.spack $ docker run --rm -it [image]:python-3.11.2-65txfcpqbmpawclvtasuog4yzmxwaoia.spack ``` which should really be a game changer for sharing binaries. Further, all content-addressable blobs that are downloaded and verified will be cached in Spack's download cache. This should make repeated `push` commands faster, as well as `push` followed by a separate `update-index` command. An end to end example of how to use this in Github Actions is here: **https://github.com/haampie/spack-oci-buildcache-example** TODO: - [x] Generate environment modifications in config so PATH is set up - [x] Enrich config with Spack's `spec` json (this is allowed in the OCI specification) - [x] When ^ is done, add logic to create an index in say `<image>:index` by fetching all config files (using OCI distribution discovery API) - [x] Add logic to use object storage in an OCI registry in `spack install`. - [x] Make the user pick the base image for generated OCI images. - [x] Update buildcache install logic to deal with absolute paths in tarballs - [x] Merge with `spack buildcache` command - [x] Merge #37441 (included here) - [x] Merge #39077 (included here) - [x] #39187 + #39285 - [x] #39341 - [x] Not a blocker: #35737 fixes correctness run env for the generated container images NOTE: 1. `oci://` is unfortunately taken, so it's being abused in this PR to mean "oci type mirror". `skopeo` uses `docker://` which I'd like to avoid, given that classical docker v1 registries are not supported. 2. this is currently `https`-only, given that basic auth is used to login. I _could_ be convinced to allow http, but I'd prefer not to, given that for a `spack buildcache push` command multiple domains can be involved (auth server, source of base image, destination registry). Right now, no urllib http handler is added, so redirects to https and auth servers with http urls will simply result in a hard failure. CAVEATS: 1. Signing is not implemented in this PR. `gpg --clearsign` is not the nicest solution, since (a) the spec.json is merged into the image config, which must be valid json, and (b) it would be better to sign the manifest (referencing both config/spec file and tarball) using more conventional image signing tools 2. `spack.binary_distribution.push` is not yet implemented for the OCI buildcache, only `spack buildcache push` is. This is because I'd like to always push images + deps to the registry, so that it's `docker pull`-able, whereas in `spack ci` we really wanna push an individual package without its deps to say `pr-xyz`, while its deps reside in some `develop` buildcache. 3. The `push -j ...` flag only works for OCI buildcache, not for others
2023-10-26PythonPackage: allow archive_files to be overridden (#40694)Adam J. Stewart1-1/+1
2023-10-26spack checksum pkg@1.2, use as version filter (#39694)Harmen Stoppels3-25/+15
* spack checksum pkg@1.2, use as version filter Currently pkg@1.2 splits on @ and looks for 1.2 specifically, with this PR pkg@1.2 is a filter so any matching 1.2, 1.2.1, ..., 1.2.10 version is displayed. * fix tests * fix style
2023-10-26modules: no --delim option if separator is colon character (#39010)Xavier Delaruelle1-24/+10
Update Tcl modulefile template to simplify generated `append-path`, `prepend-path` and `remove-path` commands and improve their readability. If path element delimiter is colon character, do not set the `--delim` option as it is the default delimiter value.
2023-10-26PythonPackage: nested config_settings (#40693)Adam J. Stewart1-17/+26
* PythonPackage: nested config_settings * flake8
2023-10-26spack checksum: show long flags in usage output (#40407)Harmen Stoppels1-4/+4
2023-10-26modules: hide implicit modulefiles (#36619)Xavier Delaruelle10-20/+407
Renames exclude_implicits to hide_implicits When hide_implicits option is enabled, generate modulefile of implicitly installed software and hide them. Even if implicit, those modulefiles may be referred as dependency in other modulefiles thus they should be generated to make module properly load dependent module. A new hidden property is added to BaseConfiguration class. To hide modulefiles, modulercs are generated along modulefiles. Such rc files contain specific module command to indicate a module should be hidden (for instance when using "module avail"). A modulerc property is added to TclFileLayout and LmodFileLayout classes to get fully qualified path name of the modulerc associated to a given modulefile. Modulerc files will be located in each module directory, next to the version modulefiles. This scheme is supported by both module tool implementations. modulerc_header and hide_cmd_format attributes are added to TclModulefileWriter and LmodModulefileWriter. They help to know how to generate a modulerc file with hidden commands for each module tool. Tcl modulerc file requires an header. As we use a command introduced on Modules 4.7 (module-hide --hidden-loaded), a version requirement is added to header string. For lmod, modules that open up a hierarchy are never hidden, even if they are implicitly installed. Modulerc is created, updated or removed when associated modulefile is written or removed. If an implicit modulefile becomes explicit, hidden command in modulerc for this modulefile is removed. If modulerc becomes empty, this file is removed. Modulerc file is not rewritten when no content change is detected. Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
2023-10-25build(deps): bump pytest from 7.4.2 to 7.4.3 in /lib/spack/docs (#40697)dependabot[bot]1-1/+1
2023-10-24Windows: search PATH for patch utility (#40513)John W. Parent1-4/+7
Previously, we only searched for `patch` inside of whatever Git installation was available because the most common installation of Git available on Windows had `patch`. That's not true for all possible installations of Git though, so this updates the search to also check PATH.
2023-10-23audit: add check for GitLab patches (#40656)Michael Kuhn2-14/+33
GitLab's .patch URLs only provide abbreviated hashes, while .diff URLs provide full hashes. There does not seem to be a parameter to force .patch URLs to also return full hashes, so we should make sure to use the .diff ones.
2023-10-23concretizer verbose: show progress in % too (#40654)Harmen Stoppels1-3/+6
2023-10-23Docs: Update spec variant checks plus python quotes and string formatting ↵Tamara Dahlgren9-164/+176
(#40643)
2023-10-22Docs: Add version range example to conditional dependencies (#40630)Tamara Dahlgren1-9/+36
* Docs: Add version range example to conditional dependencies * Add when context manager example
2023-10-20concretize separately: show concretization time per spec as they concretize ↵Harmen Stoppels3-96/+45
when verbose (#40634)
2023-10-20ASP-based solver: minimize weights over edges (#40632)Massimiliano Culpo2-0/+48
With the introduction of multiple build dependencies from the same package in the DAG, we need to minimize a few weights accounting for edges rather than nodes. If we don't do that we might have multiple "optimal" solutions that differ only in how the same nodes are connected together. This commit ensures optimal versions are picked per parent in case of multiple choices for a dependency.
2023-10-20schema/compilers.py: fix validation of 2+ entries (#40627)Harmen Stoppels1-54/+52
Fix the following syntax which validates only the first array entry: ```python "compilers": { "type": "array", "items": [ { "type": ... } ] } ``` to ```python "compilers": { "type": "array", "items": { "type": ... } } ``` which validates the entire array. Oops...
2023-10-19build(deps): bump mypy from 1.6.0 to 1.6.1 in /lib/spack/docs (#40603)dependabot[bot]1-1/+1
2023-10-19build(deps): bump urllib3 from 2.0.6 to 2.0.7 in /lib/spack/docs (#40583)dependabot[bot]1-1/+1
2023-10-19Improve setup build / run / test environment (#35737)Harmen Stoppels14-372/+535
This adds a `SetupContext` class which is responsible for setting package.py module globals, and computing the changes to environment variables for the build, test or run context. The class uses `effective_deptypes` which takes a list of specs (e.g. single item of a spec to build, or a list of environment roots) and a context (build, run, test), and outputs a flat list of specs that affect the environment together with a flag in what way they do so. This list is topologically ordered from root to leaf, so that one can be assured that dependents override variables set by dependencies, not the other way around. This is used to replace the logic in `modifications_from_dependencies`, which has several issues: missing calls to `setup_run_environment`, and the order in which operations are applied. Further, it should improve performance a bit in certain cases, since `effective_deptypes` run in O(v + e) time, whereas `spack env activate` currently can take up to O(v^2 + e) time due to loops over roots. Each edge in the DAG is visited once by calling `effective_deptypes` with `env.concrete_roots()`. By marking and propagating flags through the DAG, this commit also fixes a bug where Spack wouldn't call `setup_run_environment` for runtime dependencies of link dependencies. And this PR ensures that Spack correctly sets up the runtime environment of direct build dependencies. Regarding test dependencies: in a build context they are are build-time test deps, whereas in a test context they are install-time test deps. Since there are no means to distinguish the build/install type test deps, they're both. Further changes: - all `package.py` module globals are guaranteed to be set before any of the `setup_(dependent)_(run|build)_env` functions is called - traversal order during setup: first the group of externals, then the group of non-externals, with specs in each group traversed topological (dependencies are setup before dependents) - modules: only ever call `setup_dependent_run_environment` of *direct* link/run type deps - the marker in `set_module_variables_for_package` is dropped, since we should call the method once per spec. This allows us to set only a cheap subset of globals on the module: for example it's not necessary to compute the expensive `cmake_args` and w/e if the spec under consideration is not the root node to be built. - `spack load`'s `--only` is deprecated (it has no effect now), and `spack load x` now means: do everything that's required for `x` to work at runtime, which requires runtime deps to be setup -- just like `spack env activate`. - `spack load` no longer loads build deps (of build deps) ... - `spack env activate` on partially installed or broken environments: this is all or nothing now. If some spec errors during setup of its runtime env, you'll only get the unconditional variables + a warning that says the runtime changes for specs couldn't be applied. - Remove traversal in upward direction from `setup_dependent_*` in packages. Upward traversal may iterate to specs that aren't children of the roots (e.g. zlib / python have hundreds of dependents, only a small fraction is reachable from the roots. Packages should only modify the direct dependent they receive as an argument)
2023-10-19spack checksum: restore ability to select top n (#40531)Harmen Stoppels2-6/+40
The ability to select the top N versions got removed in the checksum overhaul, cause initially numbers were used for commands. Now that we settled on characters for commands, let's make numbers pick the top N again.
2023-10-19gitlab ci: Rework how mirrors are configured (#39939)Scott Wittenburg6-69/+196
Improve how mirrors are used in gitlab ci, where we have until now thought of them as only a string. By configuring ci mirrors ahead of time using the proposed mirror templates, and by taking advantage of the expressiveness that spack now has for mirrors, this PR will allow us to easily switch the protocol/url we use for fetching binary dependencies. This change also deprecates some gitlab functionality and marks it for removal in Spack 0.23: - arguments to "spack ci generate": * --buildcache-destination * --copy-to - gitlab configuration options: * enable-artifacts-buildcache * temporary-storage-url-prefix
2023-10-19ASP-based solver: single Spec instance per dag hash (#39590)Massimiliano Culpo2-32/+127
Reused specs used to be referenced directly into the built spec. This might cause issues like in issue 39570 where two objects in memory represent the same node, because two reused specs were loaded from different sources but referred to the same spec by DAG hash. The issue is solved by copying concrete specs to a dictionary keyed by dag hash.
2023-10-19Stand-alone test feature deprecation postponed to v0.22 (#40600)Tamara Dahlgren1-2/+2
2023-10-18unparse: also support generic type aliases (#40328)Harmen Stoppels1-0/+4
2023-10-18AutotoolsPackage / MakefilePackage: add gmake build dependency (#40380)Harmen Stoppels6-40/+33
2023-10-18Fix dev-build keep_stage behavior (#40576)Harmen Stoppels2-52/+29
`spack dev-build` would incorrectly set `keep_stage=True` for the entire DAG, including for non-dev specs, even though the dev specs have a DIYStage which never deletes sources.
2023-10-18Add license directive (#39346)Aiden Grossman7-0/+134
This patch adds in a license directive to get the ball rolling on adding in license information about packages to spack. I'm primarily interested in just adding license into spack, but this would also help with other efforts that people are interested in such as adding license information to the ASP solve for concretization to make sure licenses are compatible. Usage: Specifying the specific license that a package is released under in a project's `package.py` is good practice. To specify a license, find the SPDX identifier for a project and then add it using the license directive: ```python license("<SPDX Identifier HERE>") ``` For example, for Apache 2.0, you might write: ```python license("Apache-2.0") ``` Note that specifying a license without a when clause makes it apply to all versions and variants of the package, which might not actually be the case. For example, a project might have switched licenses at some point or have certain build configurations that include files that are licensed differently. To account for this, you can specify when licenses should be applied. For example, to specify that a specific license identifier should only apply to versionup to and including 1.5, you could write the following directive: ```python license("MIT", when="@:1.5") ```