summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2023-03-17Windows: properly handle symlink failures (#36003)John W. Parent2-4/+10
In the Windows filesystem logic for creating a symlink, we intend to fall back to a copy when the symlink cannot be created (for some configuration settings on Windows it is not possible for the user to create a symlink). It turns out we were overly-broad in which exceptions lead to this fallback, and the subsequent copy would also fail: at least one case where this occurred is when we attempted to create a symlink that already existed. The updated logic expressly avoids falling back to a copy when the file/symlink already exists.
2023-03-17Mitigation for GitVersion bug when no `=reference` is given (#36159)Massimiliano Culpo2-1/+39
* ASP-based solver: use satisfies instead of intersects They are semantically equivalent for concrete versions, but the GitVersion.intersects implementation is buggy * Mitigation for git version bug fixes #36134 This commit works around the issue in #36134, by using GitVersion.satisfies instead of GitVersion.intersects There are still underlying issues when trying to infer the "reference version" when no explicit one is given, but: 1. They are not reproducible with our synthetic repo 2. They occur only when the `git.<xxx>` form of Git version is used Here we just work around the user facing issue and ensure the tests are correct with our synthetic repository.
2023-03-17ASP-based solver: tweak heuristic, modify compiler encoding (#35989)Massimiliano Culpo2-158/+187
This PR does 2 unrelated things: 1. It changes the encoding of the compilers 2. It tweaks the heuristic for the solves in a0d88179074f81d13a3fad629a43d86334e7f982 Both were initially motivated by trying to get a performance gain but, while 2 showed significant speed-ups[^1], 1 instead didn't. I kept it anyhow, since I think the code related to compilers is more consolidated with the new encoding and we might get some performance improvement out of it if we can base our errors on the `node_compiler(Package, CompilerID)` atoms instead of `attrs`. [^1]: In general the changes in the heuristic brought a ~10% speed-up on the tests I did. I'll post detailed results below. Add a warning about compilers.yaml that is triggered if there are multiple compilers with the same spec, os and target (since they can't be selected by users with the spec syntax only).
2023-03-16reduce spec.json.sig file size (#36157)Harmen Stoppels1-1/+5
Since GPG clear-sign cannot deal with lines longer than 19995 characters and doesn't even error but simply truncates those linese (don't ask me why...), we have to be careful not to hit that line limit when reducing the filesize. So, instead this PR sets the indent level to 0 and drops the whitespace after `: `, which still reduces file size by 50% or so.
2023-03-16Revert "minify spec.json in buildcache (#36138)" (#36156)Harmen Stoppels1-1/+1
This reverts commit 1a8eefe09b034f8dda564e60b5b1905f53958aa2.
2023-03-15minify spec.json in buildcache (#36138)Harmen Stoppels1-1/+1
saves about 50% of data, which is significant for hundreds of thousands of spec.json files in our buildcaches.
2023-03-14Remove leftover command from documentation (#36116)Shahzeb Siddiqui1-8/+3
The command refers to dotkit files, which are not supported since a long time.
2023-03-14Libogg and libtheora: build on windows (#35099)John W. Parent1-1/+13
Adds builders appropriate for building these packages on Windows. It is intended that builds on other platforms are unaffected (e.g. they build with Autotools as before on Linux).
2023-03-14environment.matching_spec: linear time traversal (#35534)Harmen Stoppels1-35/+37
... and use colors in disambiguate message for clarity. This commit avoids the loop: ``` for root in roots: for dep in deps(root): ... ``` instead it ensures each node is visited once and only once. Also adds a small optimization when searching for concrete specs, since we can assume uniqueness of dag hash, so it's fine to early exit.
2023-03-14Use bfs in get_spec_filter_list (#36093)Harmen Stoppels1-7/+9
2023-03-14Fix typo affecting Gitlab CI (#36103)Harmen Stoppels1-3/+3
Introduced in #35944
2023-03-14use stage dir for buildcache create (#36091)Harmen Stoppels1-26/+42
2023-03-14concretizer: add mode to reuse dependencies only (#30990)Michael Kuhn4-12/+28
This adds a new mode for `concretizer:reuse` called `dependencies`, which only reuses dependencies. Currently, `spack install foo` will reuse older versions of `foo`, which might be surprising to users.
2023-03-12cce compiler: bugfix for version regex to avoid conflation with apple-clang ↵Greg Becker2-1/+26
(#35974) Currently apple-clang is detected as cce, and it should not be. --------- Co-authored-by: becker33 <becker33@users.noreply.github.com>
2023-03-10get --dev and drop set -x (#36010)Harmen Stoppels1-1/+1
2023-03-10Hotfix: Fix CI unit test after CI refactor (#36004)kwryankrattiger1-8/+5
* Hotfix: Fix CI unit test after CI refactor
2023-03-10CI boilerplate reduction (#34272)kwryankrattiger10-612/+1212
* CI configuration boilerplate reduction and refactor Configuration: - New notation for list concatenation (prepend/append) - New notation for string concatenation (prepend/append) - Break out configuration files for: ci.yaml, cdash.yaml, view.yaml - Spack CI section refactored to improve self-consistency and composability - Scripts are now lists of lists and/or lists of strings - Job attributes are now listed under precedence ordered list that are composed/merged using Spack config merge rules. - "service-jobs" are identified explicitly rather than as a batch CI: - Consolidate common, platform, and architecture configurations for all CI stacks into composable configuration files - Make padding consistent across all stacks (256) - Merge all package -> runner mappings to be consistent across all stacks Unit Test: - Refactor CI module unit-tests for refactor configuration Docs: - Add docs for new notations in configuration.rst - Rewrite docs on CI pipelines to be consistent with refactored CI workflow * Script verbose environ, dev bootstrap * Port #35409
2023-03-09reorder_flags: properly handle flags from concrete reused specs (#35951)Greg Becker2-4/+23
2023-03-08add logging to help users debug where external file searches are taking a ↵Peter Scheibel1-0/+13
long time (#35900)
2023-03-08gitlab ci: Provide a knob to control untouched spec pruning (#35274)Scott Wittenburg3-22/+149
When untouched spec pruning is enabled, specs possibly affected by a change cannot be pruned from a pipeline. Previously spack looked at all specs matching changed package names, and traversed dependents of each, all the way to the environment root, to compute the set of environment specs possibly affected by a change (and thus, not candidates for pruning). With this PR, when untouched spec pruning is enabled, a new environment variable can control how far towards the root spack traverses to compute the set of specs possibly affected by a change. SPACK_UNTOUCHED_PRUNING_DEPENDENT_DEPTH can be set to any numeric value before the "spack ci generate" command is called to control this traversal depth parameter. Setting it to "0" traverses only touched specs, setting it to "1" traverses only touched specs and their direct dependents, and so on. Omitting the variable results in the previous behavior of traversing all the way to the root. Setting it to a negative value means no traversal is done, and always yields an empty set of possibly affected specs (which would result in the max pruning possible).
2023-03-08buildcache create: reproducible tarballs (#35623)Harmen Stoppels2-50/+177
Currently `spack buildcache create` creates compressed tarballs that differ between each invocation, thanks to: 1. The gzip header containing mtime set to time.time() 2. The generated buildinfo file which has a different mtime every time. To avoid this, you have to explicitly construct GZipFile yourself, since the Python API doesn't expose the mtime arg, and we have to manually create the tarinfo object for the buildinfo metadata file. Normalize mode: regular files & hardlinks executable by user, dirs, symlinks: set 0o755 permissions in tarfile; other files use 0o644
2023-03-08Split `satisfies(..., strict=True/False)` into two functions (#35681)Massimiliano Culpo33-850/+1030
This commit formalizes `satisfies(lhs, rhs, strict=True/False)` and splits it into two functions: `satisfies(lhs, rhs)` and `intersects(lhs, rhs)`. - `satisfies(lhs, rhs)` means: all concrete specs matching the left hand side also match the right hand side - `intersects(lhs, rhs)` means: there exist concrete specs matching both lhs and rhs. `intersects` now has the property that it's commutative, which previously was not guaranteed. For abstract specs, `intersects(lhs, rhs)` implies that `constrain(lhs, rhs)` works. What's *not* done in this PR is ensuring that `intersects(concrete, abstract)` returns false when the abstract spec has additional properties not present in the concrete spec, but `constrain(concrete, abstract)` will raise an error. To accomplish this, some semantics have changed, as well as bugfixes to ArchSpec: - GitVersion is now interpreted as a more constrained version - Compiler flags are interpreted as strings since their order is important - Abstract specs respect variant type (bool / multivalued)
2023-03-08docs: added platform conflicts example, fix quotes (#35771)Tamara Dahlgren1-384/+394
2023-03-08compiler wrapper: parse Wl and Xlinker properly (#35912)Harmen Stoppels3-50/+57
Two fixes: 1. `-Wl,a,b,c,d` is a comma separated list of linker arguments, we incorrectly assume key/value pairs, which runs into issues with for example `-Wl,--enable-new-dtags,-rpath,/x` 2. `-Xlinker,xxx` is not a think, so it shouldn't be parsed.
2023-03-06compiler flags: fix multiple compilers with different flags (#35721)Greg Becker3-2/+27
Currently, if two compilers with the same spec differ on the flags, the concretizer will: 1. mix both sets of flags for the spec in the ASP program 2. error noting that the set of flags from the compiler (both of them) doesn't match the set from the lower priority compiler This PR fixes both -- only flags from the highest priority compiler with a given spec are considered.
2023-03-05windows: use `sys.platform == "win32"` instead of `is_windows` (#35640)Todd Gamblin30-147/+94
`mypy` only understands `sys.platform == "win32"`, not indirect assignments of that value to things like `is_windows`. If we don't use the accepted platform checks, `mypy` registers many Windows-only symbols as not present on Linux, when it should skip the checks for platform-specific code.
2023-03-03Polish `spack.util.environment` (#35812)Massimiliano Culpo7-336/+341
Update `spack.util.environment` to remove legacy idioms. * Remove kwargs from method signature and use a class for traces * Uppercase a few global variables * spack.util.environment: add type-hints * Improve docstrings * Fixed most style issues reported by pylint
2023-03-01Detection of Cray's slingshot detection has relied on the presence of (#35779)Brian Van Essen1-1/+1
a shared library /lib64/libcxi.so, which seems to also appear on other non-slingshot systems. This patch also checks to make sure that there is a Cray programming enviornment in /opt/cray/pe in addition to the shared library.
2023-03-01Simplify `spack help --spec` output (#35626)Adam J. Stewart1-7/+2
2023-03-01Docs: fix link to PythonPackage docs (#35725)Adam J. Stewart1-1/+1
2023-03-01Remove handling of deprecated target names (graviton) (#35537)Massimiliano Culpo1-13/+0
* Update target names for Gitlab pipelines * Remove handling of deprecated names for graviton
2023-02-24fix dump problem (#35673)Alberto Invernizzi1-1/+1
if dump file existed it was not truncating the file, resulting in a file with unaltered filesize, with the new content at the beginning, "padded" with the tail of the old content, since the new content was not enough to overwrite it.
2023-02-23colify.py: get rid of **kwargs (#35641)Todd Gamblin1-37/+62
`colify` is an old module in Spack that still uses `**kwargs` liberally. We should be more explicit. Doing this eliminates the need for many checks (can't pass the wrong arg if it isn't allowed) and makes the function documentation more clear.
2023-02-23buildcache push: ensure bool arguments for include_* (#35632)Harmen Stoppels3-3/+73
Fixes a bug introduced in 44ed0de8c077630148c213d3c7f40a8965eb6f94 where the push method of binary_distribution now takes named args include_root and include_depedencies, to avoid the **kwarg hole. But the call site wasn't update and we passed a dict of keys/values instead of arguments, which resulted in a call like this: ``` push(include_root={"include_root": True, "include_dependencies": False}) ``` This commit fixes that, and adds a test to see if we push the correct packages.
2023-02-22copy_files_to_artifacts: error -> warning (#35613)Harmen Stoppels1-4/+1
This error shows up a lot, typically it's harmless because an error happened before the source build even started, in which case we don't have build logs to copy. So, warn instead of error, cause it distracts from the actual CI error.
2023-02-22add cdash arg to source build only (#35565)Harmen Stoppels1-5/+8
2023-02-22spack build-env: error when deps are not installed (#35533)Harmen Stoppels4-2/+70
Currently we attempt to setup the build environment even when dependencies are not installed, which typically results in error while searching for libraries or executables in a dependency's prefix. With this change, we get a more user friendly error: ``` $ spack build-env perl ==> Error: Not all dependencies of perl are installed, cannot setup build environment: - qpj6dw5 perl@5.36.0%apple-clang@14.0.0+cpanm+open+shared+threads build_system=generic arch=darwin-ventura-m1 - jq2plbe ^berkeley-db@18.1.40%apple-clang@14.0.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=darwin-ventura-m1 ... $ echo $? 1 ```
2023-02-22Allow users to specify root env dir (#32836)psakievich11-29/+92
* Allow users to specify root env dir Environments managed by spack have some advantages over anonymous Environments but they are tucked away inside spack's directory tree. This PR gives users the ability to specify where the environments should live. See #32823 This is also taken as an opportunity to ensure that all references are to "managed environments", rather than "named environments". Prior to this PR some references to the latter persisted. Co-authored-by: Tom Scogland <scogland1@llnl.gov> Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com> Co-authored-by: Gregory Becker <becker33@llnl.gov>
2023-02-21Spec.satisfies should be commutative when strict=False (#35598)Harmen Stoppels2-10/+36
The call: ``` x.satisfies(y[, strict=False]) ``` is commutative, and tests non-empty intersection, whereas: ``` x.satsifies(y, strict=True) ``` is not commutative, and tests set-inclusion. There are 2 fast paths. When strict=False both self and other need to be concrete, when strict=True we can optimize when other is concrete.
2023-02-21nonexistent module concise error message (#35502)simon-Leary421-4/+10
2023-02-20Fix view support for optional Python extensions (#35489)Adam J. Stewart1-0/+3
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2023-02-20environment views: better, earlier error on clash (#35541)Harmen Stoppels4-9/+53
Spack generally ignores file-file projection clashes in environment views, but would eventually error when linking the `.spack` directory for two specs of the same package. This leads to obscure errors where users have no clue what the issue is and how to fix it. On top of that, the error comes very late, since it happens when the .spack dir contents are linked (which happens after everything else) This PR improves that by doing a quick check ahead of time if clashes are going to be anticipated (by simply checking for clashes in the projection of each spec's .spack metadir). If there are clashes, a human-readable error is thrown which shows two of the conflicting specs, and tells users to user unify:true, view:false, or set up custom projections.
2023-02-20Patchel shutil.copystat to avoid PermissionError on Lustre (#27247)Ben Wibking1-0/+68
2023-02-20spack ci: remove 120s db timeout since 60s is the default now (#35566)Harmen Stoppels1-8/+0
2023-02-20Revert "lib/spack/spack/store.py: Fix #28170 for padding relocation ↵Harmen Stoppels1-1/+0
(#33122)" (#35077) This reverts commit c721aab0067e127cd9db6a4eb919747814549b4f.
2023-02-20Pass `tests` argument to solver method when concretizing together (#35290)elenimath1-3/+1
to make the workaround for #29447 work with `concretizer:unify:true` option
2023-02-20Fix broken links in docs (#35582)Adam J. Stewart8-16/+15
2023-02-18Improve error handling in buildcache downloads (#35568)Harmen Stoppels6-44/+67
The checksum exception was not detailed enough and not reraised when using cache only, resulting in useless error messages. Now it dumps the file path, expected hash, computed hash, and the downloaded file summary.
2023-02-17clang/apple-clang correct c/c++ standard flags (#35062)Harmen Stoppels3-17/+83
2023-02-16Style: black 23, skip magic trailing comma (#35351)Adam J. Stewart158-1650/+367
* Style: black 23, skip magic trailing commas * isort should use same line length as black * Fix unused import * Update version of black used in CI * Update new packages * Update new packages