From 34f3b8fdd084754d75d9d8fb9061c278a090e062 Mon Sep 17 00:00:00 2001 From: Harmen Stoppels <me@harmenstoppels.nl> Date: Fri, 28 Jun 2024 13:31:38 +0200 Subject: Add missing v0.22.0 changelog (#44945) --- CHANGELOG.md | 321 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 321 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1f421eb0c3..b20dce89d8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,324 @@ + +# v0.22.0 (2024-05-12) + +`v0.22.0` is a major feature release. + +## Features in this release + +1. **Compiler dependencies** + + We are in the process of making compilers proper dependencies in Spack, and a number + of changes in `v0.22` support that effort. You may notice nodes in your dependency + graphs for compiler runtime libraries like `gcc-runtime` or `libgfortran`, and you + may notice that Spack graphs now include `libc`. We've also begun moving compiler + configuration from `compilers.yaml` to `packages.yaml` to make it consistent with + other externals. We are trying to do this with the least disruption possible, so + your existing `compilers.yaml` files should still work. We expect to be done with + this transition by the `v0.23` release in November. + + * #41104: Packages compiled with `%gcc` on Linux, macOS and FreeBSD now depend on a + new package `gcc-runtime`, which contains a copy of the shared compiler runtime + libraries. This enables gcc runtime libraries to be installed and relocated when + using a build cache. When building minimal Spack-generated container images it is + no longer necessary to install libgfortran, libgomp etc. using the system package + manager. + + * #42062: Packages compiled with `%oneapi` now depend on a new package + `intel-oneapi-runtime`. This is similar to `gcc-runtime`, and the runtimes can + provide virtuals and compilers can inject dependencies on virtuals into compiled + packages. This allows us to model library soname compatibility and allows + compilers like `%oneapi` to provide virtuals like `sycl` (which can also be + provided by standalone libraries). Note that until we have an agreement in place + with intel, Intel packages are marked `redistribute(source=False, binary=False)` + and must be downloaded outside of Spack. + + * #43272: changes to the optimization criteria of the solver improve the hit-rate of + buildcaches by a fair amount. The solver more relaxed compatibility rules and will + not try to strictly match compilers or targets of reused specs. Users can still + enforce the previous strict behavior with `require:` sections in `packages.yaml`. + Note that to enforce correct linking, Spack will *not* reuse old `%gcc` and + `%oneapi` specs that do not have the runtime libraries as a dependency. + + * #43539: Spack will reuse specs built with compilers that are *not* explicitly + configured in `compilers.yaml`. Because we can now keep runtime libraries in build + cache, we do not require you to also have a local configured compiler to *use* the + runtime libraries. This improves reuse in buildcaches and avoids conflicts with OS + updates that happen underneath Spack. + + * #43190: binary compatibility on `linux` is now based on the `libc` version, + instead of on the `os` tag. Spack builds now detect the host `libc` (`glibc` or + `musl`) and add it as an implicit external node in the dependency graph. Binaries + with a `libc` with the same name and a version less than or equal to that of the + detected `libc` can be reused. This is only on `linux`, not `macos` or `Windows`. + + * #43464: each package that can provide a compiler is now detectable using `spack + external find`. External packages defining compiler paths are effectively used as + compilers, and `spack external find -t compiler` can be used as a substitute for + `spack compiler find`. More details on this transition are in + [the docs](https://spack.readthedocs.io/en/latest/getting_started.html#manual-compiler-configuration) + +2. **Improved `spack find` UI for Environments** + + If you're working in an enviroment, you likely care about: + + * What are the roots + * Which ones are installed / not installed + * What's been added that still needs to be concretized + + We've tweaked `spack find` in environments to show this information much more + clearly. Installation status is shown next to each root, so you can see what is + installed. Roots are also shown in bold in the list of installed packages. There is + also a new option for `spack find -r` / `--only-roots` that will only show env + roots, if you don't want to look at all the installed specs. + + More details in #42334. + +3. **Improved command-line string quoting** + + We are making some breaking changes to how Spack parses specs on the CLI in order to + respect shell quoting instead of trying to fight it. If you (sadly) had to write + something like this on the command line: + + ``` + spack install zlib cflags=\"-O2 -g\" + ``` + + That will now result in an error, but you can now write what you probably expected + to work in the first place: + + ``` + spack install zlib cflags="-O2 -g" + ``` + + Quoted can also now include special characters, so you can supply flags like: + + ``` + spack intall zlib ldflags='-Wl,-rpath=$ORIGIN/_libs' + ``` + + To reduce ambiguity in parsing, we now require that you *not* put spaces around `=` + and `==` when for flags or variants. This would not have broken before but will now + result in an error: + + ``` + spack install zlib cflags = "-O2 -g" + ``` + + More details and discussion in #30634. + +4. **Revert default `spack install` behavior to `--reuse`** + + We changed the default concretizer behavior from `--reuse` to `--reuse-deps` in + #30990 (in `v0.20`), which meant that *every* `spack install` invocation would + attempt to build a new version of the requested package / any environment roots. + While this is a common ask for *upgrading* and for *developer* workflows, we don't + think it should be the default for a package manager. + + We are going to try to stick to this policy: + 1. Prioritize reuse and build as little as possible by default. + 2. Only upgrade or install duplicates if they are explicitly asked for, or if there + is a known security issue that necessitates an upgrade. + + With the install command you now have three options: + + * `--reuse` (default): reuse as many existing installations as possible. + * `--reuse-deps` / `--fresh-roots`: upgrade (freshen) roots but reuse dependencies if possible. + * `--fresh`: install fresh versions of requested packages (roots) and their dependencies. + + We've also introduced `--fresh-roots` as an alias for `--reuse-deps` to make it more clear + that it may give you fresh versions. More details in #41302 and #43988. + +5. **More control over reused specs** + + You can now control which packages to reuse and how. There is a new + `concretizer:reuse` config option, which accepts the following properties: + + - `roots`: `true` to reuse roots, `false` to reuse just dependencies + - `exclude`: list of constraints used to select which specs *not* to reuse + - `include`: list of constraints used to select which specs *to* reuse + - `from`: list of sources for reused specs (some combination of `local`, + `buildcache`, or `external`) + + For example, to reuse only specs compiled with GCC, you could write: + + ```yaml + concretizer: + reuse: + roots: true + include: + - "%gcc" + ``` + + Or, if `openmpi` must be used from externals, and it must be the only external used: + + ```yaml + concretizer: + reuse: + roots: true + from: + - type: local + exclude: ["openmpi"] + - type: buildcache + exclude: ["openmpi"] + - type: external + include: ["openmpi"] + ``` + +6. **New `redistribute()` directive** + + Some packages can't be redistributed in source or binary form. We need an explicit + way to say that in a package. + + Now there is a `redistribute()` directive so that package authors can write: + + ```python + class MyPackage(Package): + redistribute(source=False, binary=False) + ``` + + Like other directives, this works with `when=`: + + ```python + class MyPackage(Package): + # 12.0 and higher are proprietary + redistribute(source=False, binary=False, when="@12.0:") + + # can't redistribute when we depend on some proprietary dependency + redistribute(source=False, binary=False, when="^proprietary-dependency") + ``` + + More in #20185. + +7. **New `conflict:` and `prefer:` syntax for package preferences** + + Previously, you could express conflicts and preferences in `packages.yaml` through + some contortions with `require:`: + + ```yaml + packages: + zlib-ng: + require: + - one_of: ["%clang", "@:"] # conflict on %clang + - any_of: ["+shared", "@:"] # strong preference for +shared + ``` + + You can now use `require:` and `prefer:` for a much more readable configuration: + + ```yaml + packages: + zlib-ng: + conflict: + - "%clang" + prefer: + - "+shared" + ``` + + See [the documentation](https://spack.readthedocs.io/en/latest/packages_yaml.html#conflicts-and-strong-preferences) + and #41832 for more details. + +8. **`include_concrete` in environments** + + You may want to build on the *concrete* contents of another environment without + changing that environment. You can now include the concrete specs from another + environment's `spack.lock` with `include_concrete`: + + ```yaml + spack: + specs: [] + concretizer: + unify: true + include_concrete: + - /path/to/environment1 + - /path/to/environment2 + ``` + + Now, when *this* environment is concretized, it will bring in the already concrete + specs from `environment1` and `environment2`, and build on top of them without + changing them. This is useful if you have phased deployments, where old deployments + should not be modified but you want to use as many of them as possible. More details + in #33768. + +9. **`python-venv` isolation** + + Spack has unique requirements for Python because it: + 1. installs every package in its own independent directory, and + 2. allows users to register *external* python installations. + + External installations may contain their own installed packages that can interfere + with Spack installations, and some distributions (Debian and Ubuntu) even change the + `sysconfig` in ways that alter the installation layout of installed Python packages + (e.g., with the addition of a `/local` prefix on Debian or Ubuntu). To isolate Spack + from these and other issues, we now insert a small `python-venv` package in between + `python` and packages that need to install Python code. This isolates Spack's build + environment, isolates Spack from any issues with an external python, and resolves a + large number of issues we've had with Python installations. + + See #40773 for further details. + +## New commands, options, and directives + +* Allow packages to be pushed to build cache after install from source (#42423) +* `spack develop`: stage build artifacts in same root as non-dev builds #41373 + * Don't delete `spack develop` build artifacts after install (#43424) +* `spack find`: add options for local/upstream only (#42999) +* `spack logs`: print log files for packages (either partially built or installed) (#42202) +* `patch`: support reversing patches (#43040) +* `develop`: Add -b/--build-directory option to set build_directory package attribute (#39606) +* `spack list`: add `--namesapce` / `--repo` option (#41948) +* directives: add `checked_by` field to `license()`, add some license checks +* `spack gc`: add options for environments and build dependencies (#41731) +* Add `--create` to `spack env activate` (#40896) + +## Performance improvements + +* environment.py: fix excessive re-reads (#43746) +* ruamel yaml: fix quadratic complexity bug (#43745) +* Refactor to improve `spec format` speed (#43712) +* Do not acquire a write lock on the env post install if no views (#43505) +* asp.py: fewer calls to `spec.copy()` (#43715) +* spec.py: early return in `__str__` +* avoid `jinja2` import at startup unless needed (#43237) + +## Other new features of note + +* `archspec`: update to `v0.2.4`: support for Windows, bugfixes for `neoverse-v1` and + `neoverse-v2` detection. +* `spack config get`/`blame`: with no args, show entire config +* `spack env create <env>`: dir if dir-like (#44024) +* ASP-based solver: update os compatibility for macOS (#43862) +* Add handling of custom ssl certs in urllib ops (#42953) +* Add ability to rename environments (#43296) +* Add config option and compiler support to reuse across OS's (#42693) +* Support for prereleases (#43140) +* Only reuse externals when configured (#41707) +* Environments: Add support for including views (#42250) + +## Binary caches +* Build cache: make signed/unsigned a mirror property (#41507) +* tools stack + +## Removals, deprecations, and syntax changes +* remove `dpcpp` compiler and package (#43418) +* spack load: remove --only argument (#42120) + +## Notable Bugfixes +* repo.py: drop deleted packages from provider cache (#43779) +* Allow `+` in module file names (#41999) +* `cmd/python`: use runpy to allow multiprocessing in scripts (#41789) +* Show extension commands with spack -h (#41726) +* Support environment variable expansion inside module projections (#42917) +* Alert user to failed concretizations (#42655) +* shell: fix zsh color formatting for PS1 in environments (#39497) +* spack mirror create --all: include patches (#41579) + +## Spack community stats + +* 7,994 total packages; 525 since `v0.21.0` + * 178 new Python packages, 5 new R packages +* 358 people contributed to this release + * 344 committers to packages + * 45 committers to core + # v0.21.2 (2024-03-01) ## Bugfixes -- cgit v1.2.3-70-g09d2