summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2023-05-10Windows: Fix external detection for service accounts (#37293)John W. Parent1-2/+4
Prior to this PR, the HOMEDRIVE environment variable was used to detect what drive we are operating in. This variable is not available for service account logins (like what is used for CI), so switch to extracting the drive from PROGRAMFILES (which is more-widely defined).
2023-05-10Windows: prefer Python decompression support (#36507)John W. Parent2-113/+257
On Windows, several commonly available system tools for decompression are unreliable (gz/bz2/xz). This commit refactors `decompressor_for` to call out to a Windows or Unix-specific method: * The decompressor_for_nix method behaves the same as before and generally treats the Python/system support options for decompression as interchangeable (although avoids using Python's built-in tar support since that has had issues with permissions). * The decompressor_for_win method can only use Python support for gz/bz2/xz, although for a tar.gz it does use system support for untar (after the decompression step). .zip uses the system tar utility, and .Z depends on external support (i.e. that the user has installed 7zip). A naming scheme has been introduced for the various _decompression methods: * _system_gunzip means to use a system tool (and fail if it's not available) * _py_gunzip means to use Python's built-in support for decompressing .gzip files (and fail if it's not available) * _gunzip is a method that can do either
2023-05-10Fix compiler version issues (concrete vs range) (#37572)Harmen Stoppels3-13/+15
2023-05-10cuda: add versions 12.0.1, 12.1.0 and 12.1.1 (#37083)matteo-chesi1-1/+2
2023-05-10New option "--first" for "spack location" (#36283)G-Ragghianti2-1/+18
2023-05-10Create include/lib in prefix for oneapi packages (#37552)Robert Cohn1-0/+28
2023-05-10Stand-alone testing: make recipe support and processing spack-/pytest-like ↵Tamara Dahlgren38-1011/+2288
(#34236) This is a refactor of Spack's stand-alone test process to be more spack- and pytest-like. It is more spack-like in that test parts are no longer "hidden" in a package's run_test() method and pytest-like in that any package method whose name starts test_ (i.e., a "test" method) is a test part. We also support the ability to embed test parts in a test method when that makes sense. Test methods are now implicit test parts. The docstring is the purpose for the test part. The name of the method is the name of the test part. The working directory is the active spec's test stage directory. You can embed test parts using the test_part context manager. Functionality added by this commit: * Adds support for multiple test_* stand-alone package test methods, each of which is an implicit test_part for execution and reporting purposes; * Deprecates package use of run_test(); * Exposes some functionality from run_test() as optional helper methods; * Adds a SkipTest exception that can be used to flag stand-alone tests as being skipped; * Updates the packaging guide section on stand-alone tests to provide more examples; * Restores the ability to run tests "inherited" from provided virtual packages; * Prints the test log path (like we currently do for build log paths); * Times and reports the post-install process (since it can include post-install tests); * Corrects context-related error message to distinguish test recipes from build recipes.
2023-05-09CI: Backwards compatibility requires script override behavior (#37015)kwryankrattiger1-4/+23
2023-05-09Use single quotes to inline manifest in Dockerfiles (#37571)Massimiliano Culpo2-3/+19
fixes #22341 Using double quotes creates issues with shell variable substitutions, in particular when the manifest has "definitions:" in it. Use single quotes instead.
2023-05-09spack view copy: relocate symlinks (#32306)Harmen Stoppels1-33/+33
2023-05-08Add a "requires" directive, extend functionality of package requirements ↵Massimiliano Culpo8-110/+587
(#36286) Add a "require" directive to packages, which functions exactly like requirements specified in packages.yaml (uses the same fact-generation logic); update both to allow making the requirement conditional. * Packages may now use "require" to add constraints. This can be useful for something like "require(%gcc)" (where before we had to add a conflict for every compiler except gcc). * Requirements (in packages.yaml or in a "require" directive) can be conditional on a spec, e.g. "require(%gcc, when=@1.0.0)" (version 1.0.0 can only build with gcc). * Requirements may include a message which clarifies why they are needed. The concretizer assigns a high priority to errors which generate these messages (in particular over errors for unsatisfied requirements that do not produce messages, but also over a number of more-generic errors).
2023-05-05Packaging Guide: build-time test updates: option and test logs (#37093)Tamara Dahlgren1-5/+59
* Packaging Guide: build-time test updates: option and test logs * Fix a couple of typos
2023-05-05Improve version, version range, and version list syntax and behavior (#36273)Harmen Stoppels69-1812/+2001
## Version types, parsing and printing - The version classes have changed: `VersionBase` is removed, there is now a `ConcreteVersion` base class. `StandardVersion` and `GitVersion` both inherit from this. - The public api (`Version`, `VersionRange`, `ver`) has changed a bit: 1. `Version` produces either `StandardVersion` or `GitVersion` instances. 2. `VersionRange` produces a `ClosedOpenRange`, but this shouldn't affect the user. 3. `ver` produces any of `VersionList`, `ClosedOpenRange`, `StandardVersion` or `GitVersion`. - No unexpected type promotion, so that the following is no longer an identity: `Version(x) != VersionRange(x, x)`. - `VersionList.concrete` now returns a version if it contains only a single element subtyping `ConcreteVersion` (i.e. `StandardVersion(...)` or `GitVersion(...)`) - In version lists, the parser turns `@x` into `VersionRange(x, x)` instead of `Version(x)`. - The above also means that `ver("x")` produces a range, whereas `ver("=x")` produces a `StandardVersion`. The `=` is part of _VersionList_ syntax. - `VersionList.__str__` now outputs `=x.y.z` for specific version entries, and `x.y.z` as a short-hand for ranges `x.y.z:x.y.z`. - `Spec.format` no longer aliases `{version}` to `{versions}`, but pulls the concrete version out of the list and prints that -- except when the list is is not concrete, then is falls back to `{versions}` to avoid a pedantic error. For projections of concrete specs, `{version}` should be used to render `1.2.3` instead of `=1.2.3` (which you would get with `{versions}`). The default `Spec` format string used in `Spec.__str__` now uses `{versions}` so that `str(Spec(string)) == string` holds. ## Changes to `GitVersion` - `GitVersion` is a small wrapper around `StandardVersion` which enriches it with a git ref. It no longer inherits from it. - `GitVersion` _always_ needs to be able to look up an associated Spack version if it was not assigned (yet). It throws a `VersionLookupError` whenever `ref_version` is accessed but it has no means to look up the ref; in the past Spack would not error and use the commit sha as a literal version, which was incorrect. - `GitVersion` is never equal to `StandardVersion`, nor is satisfied by it. This is such that we don't lose transitivity. This fixes the following bug on `develop` where `git_version_a == standard_version == git_version_b` does not imply `git_version_a == git_version_b`. It also ensures equality always implies equal hash, which is also currently broken on develop; inclusion tests of a set of versions + git versions would behave differently from inclusion tests of a list of the same objects. - The above means `ver("ref=1.2.3) != ver("=1.2.3")` could break packages that branch on specific versions, but that was brittle already, since the same happens with externals: `pkg@1.2.3-external` suffixes wouldn't be exactly equal either. Instead, those checks should be `x.satisfies("@1.2.3")` which works both for git versions and custom version suffixes. - `GitVersion` from commit will now print as `<hash>=<version>` once the git ref is resolved to a spack version. This is for reliability -- version is frozen when added to the database and queried later. It also improves performance since there is no need to clone all repos of all git versions after `spack clean -m` is run and something queries the database, triggering version comparison, such as potentially reuse concretization. - The "empty VerstionStrComponent trick" for `GitVerison` is dropped since it wasn't representable as a version string (by design). Instead, it's replaced by `git`, so you get `1.2.3.git.4` (which reads 4 commits after a tag 1.2.3). This means that there's an edge case for version schemes `1.1.1`, `1.1.1a`, since the generated git version `1.1.1.git.1` (1 commit after `1.1.1`) compares larger than `1.1.1a`, since `a < git` are compared as strings. This is currently a wont-fix edge case, but if really required, could be fixed by special casing the `git` string. - Saved, concrete specs (database, lock file, ...) that only had a git sha as their version, but have no means to look the effective Spack version anymore, will now see their version mapped to `hash=develop`. Previously these specs would always have their sha literally interpreted as a version string (even when it _could_ be looked up). This only applies to databases, lock files and spec.json files created before Spack 0.20; after this PR, we always have a Spack version associated to the relevant GitVersion). - Fixes a bug where previously `to_dict` / `from_dict` (de)serialization would not reattach the repo to the GitVersion, causing the git hash to be used as a literal (bogus) version instead of the resolved version. This was in particularly breaking version comparison in the build process on macOS/Windows. ## Installing or matching specific versions - In the past, `spack install pkg@3.2` would install `pkg@=3.2` if it was a known specific version defined in the package, even when newer patch releases `3.2.1`, `3.2.2`, `...` were available. This behavior was only there because there was no syntax to distinguish between `3.2` and `3.2.1`. Since there is syntax for this now through `pkg@=3.2`, the old exact matching behavior is removed. This means that `spack install pkg@3.2` constrains the `pkg` version to the range `3.2`, and `spack install pkg@=3.2` constrains it to the specific version `3.2`. - Also in directives such as `depends_on("pkg@2.3")` and their when conditions `conflicts("...", when="@2.3")` ranges are ranges, and specific version matches require `@=2.3.`. - No matching version: in the case `pkg@3.2` matches nothing, concretization errors. However, if you run `spack install pkg@=3.2` and this version doesn't exist, Spack will define it; this allows you to install non-registered versions. - For consistency, you can now do `%gcc@10` and let it match a configured `10.x.y` compiler. It errors when there is no matching compiler. In the past it was interpreted like a specific `gcc@=10` version, which would get bootstrapped. - When compiler _bootstrapping_ is enabled, `%gcc@=10.2.0` can be used to bootstrap a specific compiler version. ## Other changes - Externals, compilers, and develop spec definitions are backwards compatible. They are typically defined as `pkg@3.2.1` even though they should be saying `pkg@=3.2.1`. Spack now transforms `pkg@3` into `pkg@=3` in those cases. - Finally, fix strictness of `version(...)` directive/declaration. It just does a simple type check, and now requires strings/integers. Floats are not allowed because they are ambiguous `str(3.10) == "3.1"`.
2023-05-05Fix typos in packaging guide (#37460)Eric Berquist1-3/+3
2023-05-05Add spack buildcache push (alias to buildcache create) (#34861)Harmen Stoppels7-48/+46
`spack buildcache create` is a misnomer cause it's the only way to push to an existing buildcache (and it in fact calls binary_distribution.push). Also we have `spack buildcache update-index` but for create the flag is `--rebuild-index`, which is confusing (and also... why "rebuild" something if the command is "create" in the first place, that implies it wasn't there to begin with). So, after this PR, you can use either ``` spack buildcache create --rebuild-index ``` or ``` spack buildcache push --update-index ``` Also, alias `spack buildcache rebuild-index` to `spack buildcache update-index`.
2023-05-05Relax environment manifest filename requirements and lockfile identification ↵Chris Green6-33/+92
criteria (#37413) * Relax filename requirements and lockfile identification criteria * Tests * Update function docs and help text * Update function documentation * Update Sphinx documentation * Adjustments per https://github.com/spack/spack/pull/37413#pullrequestreview-1413540132 * Further tweaks per https://github.com/spack/spack/pull/37413#pullrequestreview-1413971254 * Doc fixes per https://github.com/spack/spack/pull/37413#issuecomment-1535976068
2023-05-05cc: deal with -Wl,-rpath= without value, deal with NAG (#37215)Harmen Stoppels2-5/+32
Spack never parsed `nagfor` linker arguments put on the compiler line: ``` nagfor -Wl,-Wl,,-rpath,,/path ```` so, let's continue not attempting to parse that.
2023-05-05meson: change default build type to "release" (#37436)Michael Kuhn1-1/+1
The same was done for CMake in #36679.
2023-05-05spack uninstall: reduce verbosity with named environments (#34001)Harmen Stoppels5-192/+96
2023-05-05deprecate buildcache create --rel, buildcache install --allow-root (#37285)Harmen Stoppels4-19/+21
`buildcache create --rel`: deprecate this because there is no point in making things relative before tarballing; on install you need to expand `$ORIGIN` / `@loader_path` / relative symlinks anyways because some dependencies may actually be in an upstream, or have different projections. `buildcache install --allow-root`: this flag was propagated through a lot of functions but was ultimately unused.
2023-05-05Remove "blacklist" and "whitelist" from module configuration (#37432)Massimiliano Culpo6-154/+21
The sections were deprecated in v0.19
2023-05-04Allow choosing the name of the packages subdirectory in repositories (#36643)Greg Becker4-22/+61
Co-authored-by: becker33 <becker33@users.noreply.github.com>
2023-05-04Place an upper bound on urllib3 to build docs (#37433)Massimiliano Culpo1-0/+1
2023-05-04detect ifx 2023.1, add test (#37377)Robert Cohn2-1/+6
2023-05-04cmake build system: change default build type to `Release` (#36679)Massimiliano Culpo1-1/+5
This switches the default Make build type to `build_type=Release`. This offers: - higher optimization level, including loop vectorization on older GCC - adds NDEBUG define, which disables assertions, which could cause speedups if assertions are in loops etc - no `-g` means smaller install size Downsides are: - worse backtraces (though this does NOT strip symbols) - perf reports may be useless - no function arguments / local variables in debugger (could be of course) - no file path / line numbers in debugger The downsides can be mitigated by overriding to `build_type=RelWithDebInfo` in `packages.yaml`, if needed. The upside is that builds will be MUCH smaller (and faster) with this change. --------- Co-authored-by: Gregory Becker <becker33@llnl.gov>
2023-05-04Update vendored ruamel.yaml to v0.17.21 (#37008)Massimiliano Culpo72-8601/+15041
* Vendor ruamel.yaml v0.17.21 * Add unit test for whitespace regression * Add an abstraction layer in Spack to wrap ruamel.yaml All YAML operations are routed through spack.util.spack_yaml The custom classes have been adapted to the new ruamel.yaml class hierarchy. Fixed line annotation issue in "spack config blame"
2023-05-04Remove the old spec format in configuration (#37425)Massimiliano Culpo2-303/+1
The format was deprecated in v0.15
2023-05-04Deprecate `env:` as top level environment key (#37424)Massimiliano Culpo6-61/+68
2023-05-04acfl: add compiler-package mapping and fix version number (#36768)Annop Wongwathanarat3-16/+5
2023-05-04buildcache metadata: store hash -> prefix mapping (#37404)Harmen Stoppels1-18/+22
This ensures that: a) no externals are added to the tarball metadata file b) no externals are added to the prefix to prefix map on install, also for old tarballs that did include externals c) ensure that the prefix -> prefix map is always string to string, and doesn't contain None in case for some reason a hash is missing
2023-05-03docs: update RHEL/CentOS system prerequisites (#36720)Bryce Torcello1-6/+3
2023-05-03Add 'zypper' to the valid container.os_packages options (#36681)Egbert Eich3-2/+27
Signed-off-by: Egbert Eich <eich@suse.com> Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com> Co-authored-by: e4t <e4t@users.noreply.github.com>
2023-05-03Allow adding specs to an environment without the 'specs' attribute (#37378)Massimiliano Culpo2-2/+24
2023-05-03remove unused global in bindist tests (#37358)Harmen Stoppels2-10/+0
* remove unused global in bindist tests * remove unused function
2023-05-03buildcache push: improve printing (#36141)Scott Wittenburg9-224/+284
2023-05-02Disable module file generation by default (#37258)Massimiliano Culpo3-13/+50
* Disable module generation by default (#35564) a) It's used by site administrators, so it's niche b) If it's used by site administrators, they likely need to modify the config anyhow, so the default config only serves as an example to get started c) it's too arbitrary to enable tcl, but disable lmod * Remove leftover from old module file schema * Warn if module file config is detected and generation is disabled --------- Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2023-05-01Factor YAML manifest manipulation out of the Environment class (#36927)Massimiliano Culpo14-502/+978
Change the signature of the Environment.__init__ method to have a single argument, i.e. the directory where the environment manifest is located. Initializing that directory is now delegated to a function taking care of all the error handling upfront. Environment objects require a "spack.yaml" to be available to be constructed. Add a class to manage the environment manifest file. The environment now delegates to an attribute of that class the responsibility of keeping track of changes modifying the manifest. This allows simplifying the updates of the manifest file, and helps keeping in sync the spec lists in memory with the spack.yaml on disk.
2023-04-28Spec.format: fix bug in dependency hash formatting (#37073)Greg Becker2-60/+58
Co-authored-by: becker33 <becker33@users.noreply.github.com>
2023-04-28Reduce the number of stat calls in "spack verify" (#37251)Harmen Stoppels2-66/+47
Spack comes to a crawl post-install of nvhpc, which is partly thanks to this post install hook which has a lot of redundancy, and isn't correct. 1. There's no need to store "type" because that _is_ "mode". 2. There are more file types than "symlink", "dir", "file". 3. Don't checksum device type things 4. Don't run 3 stat calls (exists, stat, isdir/islink), but one lstat call 5. Don't read entire files into memory I also don't know why `spack.crypto` wasn't used for checksumming, but I guess it's too late for that now. Finally md5 would've been the faster algorithm, which would've been fine given that a non cryptographicall checksum was used anyways.
2023-04-28Add container images supporting RHEL alternatives (#36713)Jonathon Anderson1-0/+84
Add container support for AlmaLinux, Fedora 37 and 38 and Rocky Linux
2023-04-25require: do not allow additional properties (#37174)Harmen Stoppels1-0/+1
2023-04-24Copy more logs to CI artifacts (#36783)Adam J. Stewart1-8/+8
* Copy more logs to CI artifacts * Trigger rebuilds again * Remove test variant
2023-04-21Windows: shell variables are case-insensitive (#36813)John W. Parent2-1/+34
If we modify both Path and PATH, on Windows they will clobber one another. This PR updates the shell modification logic to automatically convert variable names to upper-case on Windows.
2023-04-21Enable verify tests on windows (#36975)markus-ferrell1-5/+0
2023-04-21Enable versions cmd tests on windows (#36974)markus-ferrell1-4/+0
2023-04-21Enable test suite tests on windows (#36966)markus-ferrell1-2/+1
2023-04-21Enable database tests for windows (#36968)markus-ferrell1-6/+0
2023-04-21Enable config values tests on windows (#36969)markus-ferrell1-3/+0
2023-04-21Enable graph tests on windows (#36967)markus-ferrell1-4/+0
2023-04-21Update archspec to latest release (#37070)Massimiliano Culpo4-23/+209
Fix -mcpu flags for gcc on neoverse-v1 Add support for NVHPC flags