summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2023-10-20schema/compilers.py: fix validation of 2+ entries (#40627)Harmen Stoppels1-54/+52
Fix the following syntax which validates only the first array entry: ```python "compilers": { "type": "array", "items": [ { "type": ... } ] } ``` to ```python "compilers": { "type": "array", "items": { "type": ... } } ``` which validates the entire array. Oops...
2023-10-19build(deps): bump mypy from 1.6.0 to 1.6.1 in /lib/spack/docs (#40603)dependabot[bot]1-1/+1
2023-10-19build(deps): bump urllib3 from 2.0.6 to 2.0.7 in /lib/spack/docs (#40583)dependabot[bot]1-1/+1
2023-10-19Improve setup build / run / test environment (#35737)Harmen Stoppels14-372/+535
This adds a `SetupContext` class which is responsible for setting package.py module globals, and computing the changes to environment variables for the build, test or run context. The class uses `effective_deptypes` which takes a list of specs (e.g. single item of a spec to build, or a list of environment roots) and a context (build, run, test), and outputs a flat list of specs that affect the environment together with a flag in what way they do so. This list is topologically ordered from root to leaf, so that one can be assured that dependents override variables set by dependencies, not the other way around. This is used to replace the logic in `modifications_from_dependencies`, which has several issues: missing calls to `setup_run_environment`, and the order in which operations are applied. Further, it should improve performance a bit in certain cases, since `effective_deptypes` run in O(v + e) time, whereas `spack env activate` currently can take up to O(v^2 + e) time due to loops over roots. Each edge in the DAG is visited once by calling `effective_deptypes` with `env.concrete_roots()`. By marking and propagating flags through the DAG, this commit also fixes a bug where Spack wouldn't call `setup_run_environment` for runtime dependencies of link dependencies. And this PR ensures that Spack correctly sets up the runtime environment of direct build dependencies. Regarding test dependencies: in a build context they are are build-time test deps, whereas in a test context they are install-time test deps. Since there are no means to distinguish the build/install type test deps, they're both. Further changes: - all `package.py` module globals are guaranteed to be set before any of the `setup_(dependent)_(run|build)_env` functions is called - traversal order during setup: first the group of externals, then the group of non-externals, with specs in each group traversed topological (dependencies are setup before dependents) - modules: only ever call `setup_dependent_run_environment` of *direct* link/run type deps - the marker in `set_module_variables_for_package` is dropped, since we should call the method once per spec. This allows us to set only a cheap subset of globals on the module: for example it's not necessary to compute the expensive `cmake_args` and w/e if the spec under consideration is not the root node to be built. - `spack load`'s `--only` is deprecated (it has no effect now), and `spack load x` now means: do everything that's required for `x` to work at runtime, which requires runtime deps to be setup -- just like `spack env activate`. - `spack load` no longer loads build deps (of build deps) ... - `spack env activate` on partially installed or broken environments: this is all or nothing now. If some spec errors during setup of its runtime env, you'll only get the unconditional variables + a warning that says the runtime changes for specs couldn't be applied. - Remove traversal in upward direction from `setup_dependent_*` in packages. Upward traversal may iterate to specs that aren't children of the roots (e.g. zlib / python have hundreds of dependents, only a small fraction is reachable from the roots. Packages should only modify the direct dependent they receive as an argument)
2023-10-19spack checksum: restore ability to select top n (#40531)Harmen Stoppels2-6/+40
The ability to select the top N versions got removed in the checksum overhaul, cause initially numbers were used for commands. Now that we settled on characters for commands, let's make numbers pick the top N again.
2023-10-19gitlab ci: Rework how mirrors are configured (#39939)Scott Wittenburg6-69/+196
Improve how mirrors are used in gitlab ci, where we have until now thought of them as only a string. By configuring ci mirrors ahead of time using the proposed mirror templates, and by taking advantage of the expressiveness that spack now has for mirrors, this PR will allow us to easily switch the protocol/url we use for fetching binary dependencies. This change also deprecates some gitlab functionality and marks it for removal in Spack 0.23: - arguments to "spack ci generate": * --buildcache-destination * --copy-to - gitlab configuration options: * enable-artifacts-buildcache * temporary-storage-url-prefix
2023-10-19ASP-based solver: single Spec instance per dag hash (#39590)Massimiliano Culpo2-32/+127
Reused specs used to be referenced directly into the built spec. This might cause issues like in issue 39570 where two objects in memory represent the same node, because two reused specs were loaded from different sources but referred to the same spec by DAG hash. The issue is solved by copying concrete specs to a dictionary keyed by dag hash.
2023-10-19Stand-alone test feature deprecation postponed to v0.22 (#40600)Tamara Dahlgren1-2/+2
2023-10-18unparse: also support generic type aliases (#40328)Harmen Stoppels1-0/+4
2023-10-18AutotoolsPackage / MakefilePackage: add gmake build dependency (#40380)Harmen Stoppels6-40/+33
2023-10-18Fix dev-build keep_stage behavior (#40576)Harmen Stoppels2-52/+29
`spack dev-build` would incorrectly set `keep_stage=True` for the entire DAG, including for non-dev specs, even though the dev specs have a DIYStage which never deletes sources.
2023-10-18Add license directive (#39346)Aiden Grossman7-0/+134
This patch adds in a license directive to get the ball rolling on adding in license information about packages to spack. I'm primarily interested in just adding license into spack, but this would also help with other efforts that people are interested in such as adding license information to the ASP solve for concretization to make sure licenses are compatible. Usage: Specifying the specific license that a package is released under in a project's `package.py` is good practice. To specify a license, find the SPDX identifier for a project and then add it using the license directive: ```python license("<SPDX Identifier HERE>") ``` For example, for Apache 2.0, you might write: ```python license("Apache-2.0") ``` Note that specifying a license without a when clause makes it apply to all versions and variants of the package, which might not actually be the case. For example, a project might have switched licenses at some point or have certain build configurations that include files that are licensed differently. To account for this, you can specify when licenses should be applied. For example, to specify that a specific license identifier should only apply to versionup to and including 1.5, you could write the following directive: ```python license("MIT", when="@:1.5") ```
2023-10-18abi.py: fix typo, add type-hints (#38216)Greg Becker2-13/+20
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2023-10-17Allow / in GitVersion (#39398)Peter Scheibel16-27/+214
This commit allows version specifiers to refer to git branches that contain forward slashes. For example, the following is valid syntax now: pkg@git.releases/1.0 It also adds a new method `Spec.format_path(fmt)` which is like `Spec.format`, but also maps unsafe characters to `_` after interpolation. The difference is as follows: >>> Spec("pkg@git.releases/1.0").format("{name}/{version}") 'pkg/git.releases/1.0' >>> Spec("pkg@git.releases/1.0").format_path("{name}/{version}") 'pkg/git.releases_1.0' The `format_path` method is used in all projections. Notice that this method also maps `=` to `_` >>> Spec("pkg@git.main=1.0").format_path("{name}/{version}") 'pkg/git.main_1.0' which should avoid syntax issues when `Spec.prefix` is literally copied into a Makefile as sometimes happens in AutotoolsPackage or MakefilePackage
2023-10-17Support `spack env activate --with-view <name> <env>` (#40549)Harmen Stoppels5-69/+114
Currently `spack env activate --with-view` exists, but is a no-op. So, it is not too much of a breaking change to make this redundant flag accept a value `spack env activate --with-view <name>` which activates a particular view by name. The view name is stored in `SPACK_ENV_VIEW`. This also fixes an issue where deactivating a view that was activated with `--without-view` possibly removes entries from PATH, since now we keep track of whether the default view was "enabled" or not.
2023-10-16Use string representation of deptypes for concrete specs (#40566)Massimiliano Culpo1-1/+5
2023-10-15spack checksum: handle all versions dropped better (#40530)Harmen Stoppels3-3/+21
* spack checksum: fix error when all versions are dropped * add test
2023-10-13spack checksum: improve interactive filtering (#40403)Harmen Stoppels6-55/+352
* spack checksum: improve interactive filtering * fix signature of executable * Fix restart when using editor * Don't show [x version(s) are new] when no known versions (e.g. in spack create <url>) * Test ^D in test_checksum_interactive_quit_from_ask_each * formatting * colorize / skip header on invalid command * show original total, not modified total * use colify for command list * Warn about possible URL changes * show possible URL change as comment * make mypy happy * drop numbers * [o]pen editor -> [e]dit
2023-10-13Expand multiple build systems section (#39589)Harmen Stoppels1-51/+102
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2023-10-13Better error message when wrong platform is used (#40492)Massimiliano Culpo1-1/+2
fixes #40299
2023-10-13containerize: update docs to activate env before using container templates ↵Matthew Chan1-0/+7
(#40493)
2023-10-12modules:prefix_inspections: allow empty dict (#40485)Harmen Stoppels1-2/+2
Currently ``` modules: prefix_inspections:: {} ``` gives you the builtin defaults instead of no mapping.
2023-10-12build(deps): bump python-levenshtein in /lib/spack/docs (#40461)dependabot[bot]1-1/+1
Bumps [python-levenshtein](https://github.com/maxbachmann/python-Levenshtein) from 0.22.0 to 0.23.0. - [Release notes](https://github.com/maxbachmann/python-Levenshtein/releases) - [Changelog](https://github.com/maxbachmann/python-Levenshtein/blob/main/HISTORY.md) - [Commits](https://github.com/maxbachmann/python-Levenshtein/compare/v0.22.0...v0.23.0) --- updated-dependencies: - dependency-name: python-levenshtein dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-12Remove deprecated "extra_instructions" option for containers (#40365)Massimiliano Culpo4-46/+1
2023-10-11buildcache: Tell servers not to cache index or hash (#40339)Scott Wittenburg1-2/+2
2023-10-11cmake: drop CMAKE_STATIC_LINKER_FLAGS (#40423)Victor Brunini2-4/+3
Because those end up being passed to ar which does not understand linker arguments. This was making ldflags largely unusuable for statically linked cmake packages.
2023-10-11Update bootstrap buildcache to support Python 3.12 (#40404)Massimiliano Culpo4-5/+5
* Add support for Python 3.12 * Use optimized build of clingo
2023-10-11spider: respect <base> tag (#40443)Harmen Stoppels1-8/+20
2023-10-11build(deps): bump mypy from 1.5.1 to 1.6.0 in /lib/spack/docs (#40424)dependabot[bot]1-1/+1
Bumps [mypy](https://github.com/python/mypy) from 1.5.1 to 1.6.0. - [Commits](https://github.com/python/mypy/compare/v1.5.1...v1.6.0) --- updated-dependencies: - dependency-name: mypy dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-11Update legacy `.format()` calls to fstrings in installer.py (#40426)Alec Scott1-152/+132
2023-10-11spack buildcache: fix a typo in a function call (#40446)Massimiliano Culpo1-3/+3
fixes #40415
2023-10-10More helpful error when patch lookup fails (#40379)Harmen Stoppels2-10/+21
2023-10-09unparse: drop python 3.3 remnants (#40331)Harmen Stoppels1-20/+8
2023-10-09docs: update Spack prerequisites (#40381)Harmen Stoppels1-2/+1
2023-10-09build(deps): bump python-levenshtein in /lib/spack/docs (#40220)dependabot[bot]1-1/+1
Bumps [python-levenshtein](https://github.com/maxbachmann/python-Levenshtein) from 0.21.1 to 0.22.0. - [Release notes](https://github.com/maxbachmann/python-Levenshtein/releases) - [Changelog](https://github.com/maxbachmann/python-Levenshtein/blob/main/HISTORY.md) - [Commits](https://github.com/maxbachmann/python-Levenshtein/compare/v0.21.1...v0.22.0) --- updated-dependencies: - dependency-name: python-levenshtein dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-09racket packages: fix typo after multiple build systems support (#40088)Thomas Dickerson1-1/+1
2023-10-09parser: use non-capturing groups (#40373)Harmen Stoppels1-23/+23
2023-10-07Remove warning for custom module configuration, when no module is enabled ↵Massimiliano Culpo3-47/+2
(#40358) The warning was added in v0.20 and was slated for removal in v0.21
2023-10-06VersionRange: improve error message for empty range (#40345)Harmen Stoppels4-4/+15
2023-10-06unparse: drop python 2 remnants (#40329)Harmen Stoppels1-40/+0
2023-10-06Make "minimal" the default duplicate strategy (#39621)Massimiliano Culpo8-97/+2930
* Allow branching out of the "generic build" unification set For cases like the one in https://github.com/spack/spack/pull/39661 we need to relax rules on unification sets. The issue is that, right now, nodes in the "generic build" unification set are unified together with their build dependencies. This was done out of caution to avoid the risk of circular dependencies, which would ultimately cause a very slow solve. For build-tools like Cython, however, the build dependencies is masked by a long chain of "build, run" dependencies that belong in the "generic build" unification space. To allow splitting on cases like this, we relax the rule disallowing branching out of the "generic build" unification set. * Fix issue with pure build virtual dependencies Pure build virtual dependencies were not accounted properly in the list of possible virtuals. This caused some facts connecting virtuals to the corresponding providers to not be emitted, and in the end lead to unsat problems. * Fixed a few issues in packages py-gevent: restore dependency on py-cython@3 jsoncpp: fix typo in build dependency ecp-data-vis-sdk: update spack.yaml and cmake recipe py-statsmodels: add v0.13.5 * Make dependency on "blt" of type "build"
2023-10-06python: add 3.12.0 (but keep 3.11 preferred) (#40282)Harmen Stoppels1-1/+8
2023-10-05unparse: drop python 3.4 remnants (#40333)Harmen Stoppels1-15/+0
2023-10-04Improve build isolation in PythonPipBuilder (#40224)Harmen Stoppels1-1/+99
We run pip with `--no-build-isolation` because we don't wanna let pip install build deps. As a consequence, when pip runs hooks, it runs hooks of *any* package it can find in `sys.path`. For Spack-built Python this includes user site packages -- there shouldn't be any system site packages. So in this case it suffices to set the environment variable PYTHONNOUSERSITE=1. For external Python, more needs to be done, cause there is no env variable that disables both system and user site packages; setting the `python -S` flag doesn't work because pip runs subprocesses that don't inherit this flag (and there is no API to know if -S was passed) So, for external Python, an empty venv is created before invoking pip in Spack's build env ensures that pip can no longer see anything but standard libraries and `PYTHONPATH`. The downside of this is that pip will generate shebangs that point to the python executable from the venv. So, for external python an extra step is necessary where we fix up shebangs post install.
2023-10-03ci: Change how job names appear in gitlab (#39963)Scott Wittenburg2-38/+13
2023-10-03build(deps): bump urllib3 from 2.0.5 to 2.0.6 in /lib/spack/docs (#40286)dependabot[bot]1-1/+1
Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.0.5 to 2.0.6. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](https://github.com/urllib3/urllib3/compare/v2.0.5...2.0.6) --- updated-dependencies: - dependency-name: urllib3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-03Buildcache tarballs with rootfs structure (#39341)Harmen Stoppels2-73/+162
Two changes in this PR: 1. Register absolute paths in tarballs, which makes it easier to use them as container image layers, or rootfs in general, outside of Spack. Spack supports this already on develop. 2. Assemble the tarfile entries "by hand", which has a few advantages: 1. Avoid reading `/etc/passwd`, `/etc/groups`, `/etc/nsswitch.conf` which `tar.add(dir)` does _for each file it adds_ 2. Reduce the number of stat calls per file added by a factor two, compared to `tar.add`, which should help with slow, shared filesystems where these calls are expensive 4. Create normalized `TarInfo` entries from the start, instead of letting Python create them and patching them after the fact 5. Don't recurse into subdirs before processing files, to avoid keeping nested directories opened. (this changes the tar entry order slightly, it's like sorting by `(not is_dir, name)`.
2023-10-02buildcache: ignore errors of newer buildcache version (#40279)Harmen Stoppels1-2/+2
Currently buildcaches are forward incompatible in an annoying way: spack errors out when trying to use them. With this change, you just get a warning.
2023-10-02Spack python 3.12: PEP 695 unparse support (#40155)Harmen Stoppels1-0/+28
2023-10-01docs: Replace package list with packages.spack.io (#40251)Todd Gamblin8-86/+30
For a long time, the docs have generated a huge, static HTML package list. It has some disadvantages: * It's slow to load * It's slow to build * It's hard to search We now have a nice website that can tell us about Spack packages, and it's searchable so users can easily find the one or two packages out of 7400 that they're looking for. We should link to this instead of including a static package list page in the docs. - [x] Replace package list link with link to packages.spack.io - [x] Remove `package_list.html` generation from `conf.py`. - [x] Add a new section for "Links" to the docs. - [x] Remove docstring notes from contribution guide (we haven't generated RST for package docstrings for a while) - [x] Remove referencese to `package-list` from docs.