summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2023-03-31Allow configurable stage names (#36509)John W. Parent4-6/+20
Add `config:stage_name` which is a Spec format string that can customize the names of stages created by Spack. This was primarily created to allow generating shorter stage names on Windows (along with `config:build_stage`, this can be used to create stages with short absolute paths). By default, this is not set and the prior name stage format is used. This also removes the username component that is always added to Stage paths on Windows (if users want to include this, they can add it to the `build_stage`).
2023-03-31compiler wrapper: fix -Xlinker parsing (#35929)Harmen Stoppels2-35/+123
* compiler wrapper: fix -Xlinker parsing * handle the case of -rpath without value; avoid that we drop the flag * also handle the -Xlinker -rpath -Xlinker without further args case... * fix test * get rid of global $rp var, reduce branching
2023-03-31typehint a few globals (#36544)Harmen Stoppels6-8/+20
2023-03-30Add type-hints to spack.bootstrap (#36491)Massimiliano Culpo5-78/+117
2023-03-30Fix a couple of minor bugs with ASP weights (#36522)Massimiliano Culpo2-23/+14
Reorder versions so that deprecated ones are last. Account for default not used when the variant exists.
2023-03-28retry: bugfix: package requirements with git commits (#35057) (#36347)Todd Gamblin4-34/+234
- [x] Specs that define 'new' versions in the require: section need to generate associated facts to indicate that those versions are valid. - [x] add test to verify success with unknown versions. - [x] remove unneeded check that was leading to extra complexity and test failures (at this point, all `hash=version` does not require listing out that version in `packages.yaml`) - [x] unique index for origin (dont reuse 0) Co-authored-by: Peter Josef Scheibel <scheibel1@llnl.gov>
2023-03-28macos: set new ad-hoc signature with codesign after binary string ↵Harmen Stoppels3-10/+31
replacement (#35585)
2023-03-27Add `llnl.util.filesystem.find_first` (#36083)Harmen Stoppels2-1/+168
Add a `find_first` method that locates one instance of a file that matches a specified pattern by recursively searching a directory tree. Unlike other `find` methods, this only locates one file at most, so can use optimizations that avoid searching the entire tree: Typically the relevant files are at low depth, so it makes sense to locate files through iterative deepening and early exit.
2023-03-27modules: enhance help message (#36410)Xavier Delaruelle2-1/+54
Update tcl and lmod modulefile template to provide more information on help message (name, version and target) like done on whatis for lmod modulefiles.
2023-03-24modules: add support for append_flags/remove_flags (#36402)Xavier Delaruelle2-0/+4
Adapt tcl and lmod modulefile templates to generate append-path or remove-path commands in modulefile when respectively append_flags or remove_flags commands are defined in package for run environment. Fixes #10299.
2023-03-24modules tcl: simplify env modification block in template (#36334)Xavier Delaruelle3-8/+27
Simplify environment modification block in modulefile Tcl template by always setting a path delimiter to the prepend-path, append-path and remove-path commands. Remove --delim option to the setenv command as this command does not allow such option. Update test_prepend_path_separator test to explicitly check the 6 path-like commands that should be present in generated modulefile.
2023-03-23unit tests: don't hard-code arch in compiler config (#36360)Harmen Stoppels1-3/+0
This breaks when testing on non-x86_64 machines outside CI
2023-03-23Fix regression on compiler constraint (#36342)Massimiliano Culpo3-3/+68
fixes #36339 We were missing a rule that enforced a match between the `node_compiler` and the compiler used to satisfy a requirement. Fix compiler with custom, made up version too
2023-03-23Revert "Bugfix: package requirements with git commits (#35057)" (#36341)Harmen Stoppels4-215/+11
This reverts commit 3d597e29bee1429d2b8d204016df40b575900d66.
2023-03-23Bugfix: package requirements with git commits (#35057)Peter Scheibel4-11/+215
* Specs that define 'new' versions in the require: section need to generate associated facts to indicate that those versions are valid. * add test to verify success with unknown versions.
2023-03-23modules tcl: switch default all:autoload from none to direct (#36269)Xavier Delaruelle1-4/+7
Since environment-modules has support for autoloading since 4.2, and Spack-builds of it enable it by default, use the same autoload default for tcl as lmod.
2023-03-22docs: mention cuda multi-arch capability (#36321)Leopold Talirz1-1/+4
2023-03-20Package requirements: allow single specs in requirement lists (#36258)Peter Scheibel3-9/+35
If you have a "require:" section in your packages config, and you use it to specify a list of requirements, the list elements can now include strings (before this, each element in the list had to be a `one_of` or `any_of` specification, which is awkward if you wanted to apply just one spec with no alternatives).
2023-03-20spack install: simplify behavior when inside environments (#35206)Harmen Stoppels3-217/+183
Example one: ``` spack install --add x y z ``` is equivalent to ``` spack add x y z spack concretize spack install --only-concrete ``` where `--only-concrete` installs without modifying spack.yaml/spack.lock Example two: ``` spack install ``` concretizes current spack.yaml if outdated and installs all specs. Example three: ``` spack install x y z ``` concretizes current spack.yaml if outdated and installs *only* concrete specs in the environment that match abstract specs `x`, `y`, or `z`.
2023-03-20Remove unused `ignore` parameter of `extends()` directive (#35588)Adam J. Stewart1-23/+0
The `ignore` parameter was only used for `spack activate/deactivate`, and it isn't used by Spack Environments which have their own handling of file conflicts. We should remove it. Everything that handles `ignore=` was removed in #29317 and included in 0.19, when we removed `spack activate` and `spack deactivate` in favor of environments. So all of these usages removed here were already being ignored by Spack.
2023-03-20ci.py: remove redundant wrapper around get (#36188)Harmen Stoppels1-30/+24
2023-03-20modules tcl: fix autoload mechanism in template (#36237)Xavier Delaruelle1-13/+2
Adapt tcl modulefile template to call "module load" on autoload dependency without testing if this dependency is already loaded or not. The is-loaded test is not necessary, as module commands know how to cope with an already loaded module. With environment-modules 4.2+ (released in 2018) it is also important to have this "module load" command even if dependency is already loaded in order to record that the modulefile declares such dependency. This is important if you want to keep a consistent environment when a dependent module is unloaded. The "Autoloading" verbose message is also removed as recent module commands will report such information to the user (depending on the verbosity configured for the module command). Such change has been test successfully with Modules 3.2 (EL7), 4.5 (EL8) and 5.2 (latest) and also with Lmod 7 and 8 (as it is mentionned in Spack docs that Lmod can be used along with tcl modules). Dependencies are correctly loaded or unloaded, whether they are loaded/unloaded or not. This change fixes Tcl quoting issue introduced in #32853. Fixes #19155.
2023-03-19Fix case spelling for Lmod and Tcl (#36215)Xavier Delaruelle17-37/+37
2023-03-18cmake build system: make "generator" a variant (#35552)Massimiliano Culpo3-33/+64
2023-03-18Fix wrong computation of concrete specs due to a bug in intersects (#36194)Massimiliano Culpo2-1/+22
fixes #36190
2023-03-17Improve wording of audit message (#36180)Massimiliano Culpo1-2/+5
2023-03-17Windows: properly handle symlink failures (#36003)John W. Parent2-4/+10
In the Windows filesystem logic for creating a symlink, we intend to fall back to a copy when the symlink cannot be created (for some configuration settings on Windows it is not possible for the user to create a symlink). It turns out we were overly-broad in which exceptions lead to this fallback, and the subsequent copy would also fail: at least one case where this occurred is when we attempted to create a symlink that already existed. The updated logic expressly avoids falling back to a copy when the file/symlink already exists.
2023-03-17Mitigation for GitVersion bug when no `=reference` is given (#36159)Massimiliano Culpo2-1/+39
* ASP-based solver: use satisfies instead of intersects They are semantically equivalent for concrete versions, but the GitVersion.intersects implementation is buggy * Mitigation for git version bug fixes #36134 This commit works around the issue in #36134, by using GitVersion.satisfies instead of GitVersion.intersects There are still underlying issues when trying to infer the "reference version" when no explicit one is given, but: 1. They are not reproducible with our synthetic repo 2. They occur only when the `git.<xxx>` form of Git version is used Here we just work around the user facing issue and ensure the tests are correct with our synthetic repository.
2023-03-17ASP-based solver: tweak heuristic, modify compiler encoding (#35989)Massimiliano Culpo2-158/+187
This PR does 2 unrelated things: 1. It changes the encoding of the compilers 2. It tweaks the heuristic for the solves in a0d88179074f81d13a3fad629a43d86334e7f982 Both were initially motivated by trying to get a performance gain but, while 2 showed significant speed-ups[^1], 1 instead didn't. I kept it anyhow, since I think the code related to compilers is more consolidated with the new encoding and we might get some performance improvement out of it if we can base our errors on the `node_compiler(Package, CompilerID)` atoms instead of `attrs`. [^1]: In general the changes in the heuristic brought a ~10% speed-up on the tests I did. I'll post detailed results below. Add a warning about compilers.yaml that is triggered if there are multiple compilers with the same spec, os and target (since they can't be selected by users with the spec syntax only).
2023-03-16reduce spec.json.sig file size (#36157)Harmen Stoppels1-1/+5
Since GPG clear-sign cannot deal with lines longer than 19995 characters and doesn't even error but simply truncates those linese (don't ask me why...), we have to be careful not to hit that line limit when reducing the filesize. So, instead this PR sets the indent level to 0 and drops the whitespace after `: `, which still reduces file size by 50% or so.
2023-03-16Revert "minify spec.json in buildcache (#36138)" (#36156)Harmen Stoppels1-1/+1
This reverts commit 1a8eefe09b034f8dda564e60b5b1905f53958aa2.
2023-03-15minify spec.json in buildcache (#36138)Harmen Stoppels1-1/+1
saves about 50% of data, which is significant for hundreds of thousands of spec.json files in our buildcaches.
2023-03-14Remove leftover command from documentation (#36116)Shahzeb Siddiqui1-8/+3
The command refers to dotkit files, which are not supported since a long time.
2023-03-14Libogg and libtheora: build on windows (#35099)John W. Parent1-1/+13
Adds builders appropriate for building these packages on Windows. It is intended that builds on other platforms are unaffected (e.g. they build with Autotools as before on Linux).
2023-03-14environment.matching_spec: linear time traversal (#35534)Harmen Stoppels1-35/+37
... and use colors in disambiguate message for clarity. This commit avoids the loop: ``` for root in roots: for dep in deps(root): ... ``` instead it ensures each node is visited once and only once. Also adds a small optimization when searching for concrete specs, since we can assume uniqueness of dag hash, so it's fine to early exit.
2023-03-14Use bfs in get_spec_filter_list (#36093)Harmen Stoppels1-7/+9
2023-03-14Fix typo affecting Gitlab CI (#36103)Harmen Stoppels1-3/+3
Introduced in #35944
2023-03-14use stage dir for buildcache create (#36091)Harmen Stoppels1-26/+42
2023-03-14concretizer: add mode to reuse dependencies only (#30990)Michael Kuhn4-12/+28
This adds a new mode for `concretizer:reuse` called `dependencies`, which only reuses dependencies. Currently, `spack install foo` will reuse older versions of `foo`, which might be surprising to users.
2023-03-12cce compiler: bugfix for version regex to avoid conflation with apple-clang ↵Greg Becker2-1/+26
(#35974) Currently apple-clang is detected as cce, and it should not be. --------- Co-authored-by: becker33 <becker33@users.noreply.github.com>
2023-03-10get --dev and drop set -x (#36010)Harmen Stoppels1-1/+1
2023-03-10Hotfix: Fix CI unit test after CI refactor (#36004)kwryankrattiger1-8/+5
* Hotfix: Fix CI unit test after CI refactor
2023-03-10CI boilerplate reduction (#34272)kwryankrattiger10-612/+1212
* CI configuration boilerplate reduction and refactor Configuration: - New notation for list concatenation (prepend/append) - New notation for string concatenation (prepend/append) - Break out configuration files for: ci.yaml, cdash.yaml, view.yaml - Spack CI section refactored to improve self-consistency and composability - Scripts are now lists of lists and/or lists of strings - Job attributes are now listed under precedence ordered list that are composed/merged using Spack config merge rules. - "service-jobs" are identified explicitly rather than as a batch CI: - Consolidate common, platform, and architecture configurations for all CI stacks into composable configuration files - Make padding consistent across all stacks (256) - Merge all package -> runner mappings to be consistent across all stacks Unit Test: - Refactor CI module unit-tests for refactor configuration Docs: - Add docs for new notations in configuration.rst - Rewrite docs on CI pipelines to be consistent with refactored CI workflow * Script verbose environ, dev bootstrap * Port #35409
2023-03-09reorder_flags: properly handle flags from concrete reused specs (#35951)Greg Becker2-4/+23
2023-03-08add logging to help users debug where external file searches are taking a ↵Peter Scheibel1-0/+13
long time (#35900)
2023-03-08gitlab ci: Provide a knob to control untouched spec pruning (#35274)Scott Wittenburg3-22/+149
When untouched spec pruning is enabled, specs possibly affected by a change cannot be pruned from a pipeline. Previously spack looked at all specs matching changed package names, and traversed dependents of each, all the way to the environment root, to compute the set of environment specs possibly affected by a change (and thus, not candidates for pruning). With this PR, when untouched spec pruning is enabled, a new environment variable can control how far towards the root spack traverses to compute the set of specs possibly affected by a change. SPACK_UNTOUCHED_PRUNING_DEPENDENT_DEPTH can be set to any numeric value before the "spack ci generate" command is called to control this traversal depth parameter. Setting it to "0" traverses only touched specs, setting it to "1" traverses only touched specs and their direct dependents, and so on. Omitting the variable results in the previous behavior of traversing all the way to the root. Setting it to a negative value means no traversal is done, and always yields an empty set of possibly affected specs (which would result in the max pruning possible).
2023-03-08buildcache create: reproducible tarballs (#35623)Harmen Stoppels2-50/+177
Currently `spack buildcache create` creates compressed tarballs that differ between each invocation, thanks to: 1. The gzip header containing mtime set to time.time() 2. The generated buildinfo file which has a different mtime every time. To avoid this, you have to explicitly construct GZipFile yourself, since the Python API doesn't expose the mtime arg, and we have to manually create the tarinfo object for the buildinfo metadata file. Normalize mode: regular files & hardlinks executable by user, dirs, symlinks: set 0o755 permissions in tarfile; other files use 0o644
2023-03-08Split `satisfies(..., strict=True/False)` into two functions (#35681)Massimiliano Culpo33-850/+1030
This commit formalizes `satisfies(lhs, rhs, strict=True/False)` and splits it into two functions: `satisfies(lhs, rhs)` and `intersects(lhs, rhs)`. - `satisfies(lhs, rhs)` means: all concrete specs matching the left hand side also match the right hand side - `intersects(lhs, rhs)` means: there exist concrete specs matching both lhs and rhs. `intersects` now has the property that it's commutative, which previously was not guaranteed. For abstract specs, `intersects(lhs, rhs)` implies that `constrain(lhs, rhs)` works. What's *not* done in this PR is ensuring that `intersects(concrete, abstract)` returns false when the abstract spec has additional properties not present in the concrete spec, but `constrain(concrete, abstract)` will raise an error. To accomplish this, some semantics have changed, as well as bugfixes to ArchSpec: - GitVersion is now interpreted as a more constrained version - Compiler flags are interpreted as strings since their order is important - Abstract specs respect variant type (bool / multivalued)
2023-03-08docs: added platform conflicts example, fix quotes (#35771)Tamara Dahlgren1-384/+394
2023-03-08compiler wrapper: parse Wl and Xlinker properly (#35912)Harmen Stoppels3-50/+57
Two fixes: 1. `-Wl,a,b,c,d` is a comma separated list of linker arguments, we incorrectly assume key/value pairs, which runs into issues with for example `-Wl,--enable-new-dtags,-rpath,/x` 2. `-Xlinker,xxx` is not a think, so it shouldn't be parsed.