summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2022-11-04fix requires test for aarch64 (#33656)Greg Becker1-2/+2
2022-11-04Add in-place RPATH replacement (#27610)Harmen Stoppels5-52/+155
Whenever the rpath string actually _grows_, it falls back to patchelf, when it stays the same length or gets shorter, we update it in-place, padded with null bytes. This PR only deals with absolute -> absolute rpath replacement. We don't use `_build_tarball(relative=True)` in our CI. If `relative` then it falls back to the old replacement code. With this PR, relocation time goes down significantly, likely because patchelf does some odd things with mmap, causing lots of overhead. Example: - `binutils`: 700MB installed, goes from `1.91s` to `0.57s`, or `3.4x` faster. Relocation time: 27% -> 10% of total install time - `llvm`: 6.8GB installed, goes from `28.56s` to `5.38`, or `5.3x` faster. Relocation time: 44% -> 13% of total install time The bottleneck is now decompression. Note: I'm somewhat confused about the "relative rpath" code paths. Right now this PR only deals with absolute -> absolute replacement. As far as I understand, if you embrace relative rpaths when uploading to the buildcache, the whole point is you _don't_ want to patch rpaths on install? So it seems fine to not expand `$ORIGIN` again imho.
2022-11-04Fix non-parallel make under depfile jobserver (#32862)Jordan Galby2-37/+58
When a package asks for non-parallel make, we need to force `make -j1` because just doing `make` will run in parallel under jobserver (e.g. `spack env depfile`). We now always add `-j1` when asked for a non-parallel execution (even if there is no jobserver). And each `MakeExecutable` can now ask for jobserver support or not. For example: the default `ninja` does not support jobserver so spack applies the default `-j`, but `ninja@kitware` or `ninja-fortran` does, so spack doesn't add `-j`. Tips: you can run `SPACK_INSTALL_FLAGS=-j1 make -f spack-env-depfile.make -j8` to avoid massive job-spawning because of build tools that don't support jobserver (ninja).
2022-11-03propagation: don't count propagated variant values as non-default (#33687)Todd Gamblin1-0/+2
We try to avoid non-default variant values in the concretizer, but this doesn't make sense for variants forced to take some non-default value by variant propagation. Counting this as a penalty effectively biases the concretizer for small specs dependency graphs -- something we try very hard to avoid elsewhere because it can lead to very strange decisions. Example: with the penalty, `spack spec hdf5` will choose the default `openmpi` as its `mpi` provider, but `spack spec hdf5 ~~shared` will choose `mpich` because it has to set fewer non-default variant values because `mpich`'s DAG is smaller. That's not a good reason to prefer a non-default virtual provider. To fix this, if the user explicitly requests a non-default value to be propagated, there shouldn't be a penalty. Variant values set on the CLI already don't count as default; we just need to extend that to propagated values.
2022-11-03Experimental binding of shared ELF libraries (#31948)Harmen Stoppels11-11/+358
Adds another post install hook that loops over the install prefix, looking for shared libraries type of ELF files, and sets the soname to their own absolute paths. The idea being, whenever somebody links against those libraries, the linker copies the soname (which is the absolute path to the library) as a "needed" library, so that at runtime the dynamic loader realizes the needed library is a path which should be loaded directly without searching. As a result: 1. rpaths are not used for the fixed/static list of needed libraries in the dynamic section (only for _actually_ dynamically loaded libraries through `dlopen`), which largely solves the issue that Spack's rpaths are a heuristic (`<prefix>/lib` and `<prefix>/lib64` might not be where libraries really are...) 2. improved startup times (no library search required)
2022-11-03gitlab: Prune untouched specs less aggressively (#33669)Scott Wittenburg3-26/+31
Untouched spec pruning was added to reduce the number of specs developers see getting rebuilt in their PR pipelines that they don't understand. Because the state of the develop mirror lags quite far behind the tip of the develop branch, PRs often find they need to rebuild things untouched by their PR. Untouched spec pruning was previously implemented by finding all specs in the environment with names of packages touched by the PR, traversing in both directions the DAGS of those specs, and adding all dependencies as well as dependents to a list of concrete specs that should not be considered for pruning. We found that this heuristic results in too many pruned specs, and that dependents of touched specs must have all their dependencies added to the list of specs that should not be considered for pruning.
2022-11-03Use spack.traverse.traverse_nodes where useful (#33677)Harmen Stoppels1-45/+33
2022-11-03Update metadata for bootstrapping (#33665)Massimiliano Culpo4-51/+24
2022-11-02Bugfix for spec objects modified by flag handlers (#33682)Greg Becker2-1/+13
This issue was introduced in #29761: ``` ==> Installing ncurses-6.3-22hz6q6cvo3ep2uhrs3erpp2kogxncbn ==> No binary for ncurses-6.3-22hz6q6cvo3ep2uhrs3erpp2kogxncbn found: installing from source ==> Using cached archive: /spack/var/spack/cache/_source-cache/archive/97/97fc51ac2b085d4cde31ef4d2c3122c21abc217e9090a43a30fc5ec21684e059.tar.gz ==> No patches needed for ncurses ==> ncurses: Executing phase: 'autoreconf' ==> ncurses: Executing phase: 'configure' ==> ncurses: Executing phase: 'build' ==> ncurses: Executing phase: 'install' ==> Error: AttributeError: 'str' object has no attribute 'propagate' The 'ncurses' package cannot find an attribute while trying to build from sources. This might be due to a change in Spack's package format to support multiple build-systems for a single package. You can fix this by updating the build recipe, and you can also report the issue as a bug. More information at https://spack.readthedocs.io/en/latest/packaging_guide.html#installation-procedure /spack/lib/spack/spack/build_environment.py:1075, in _setup_pkg_and_run: 1072 tb_string = traceback.format_exc() 1073 1074 # build up some context from the offending package so we can >> 1075 # show that, too. 1076 package_context = get_package_context(tb) 1077 1078 logfile = None ``` It turns out this was caused by a bug that had been around much longer, in which the flags were passed by reference to the flag_handler, and the flag_handler was modifying the spec object, not just the flags given to the build system. The scope of this bug was limited by the forking model in Spack, which is how it went under the radar for so long. PR includes regression test.
2022-11-02remove `deptype_query` remnants and fix incorrect `deptypes` kwarg (#33670)Harmen Stoppels1-1/+1
* remove deptype_query remnants * deptypes -> deptype These arguments haven't existed since 2017, but `traverse` now fails on unknown **kwargs, so they have finally popped up.
2022-11-02propagation: improve performanceGregory Becker3-61/+68
This updates the propagation logic used in `concretize.lp` to avoid rules with `path()` in the body and instead base propagation around `depends_on()`.
2022-11-02flags/variants: Add ++/~~/== syntax for propagation to dependenciesKayla Butler14-118/+426
Currently, compiler flags and variants are inconsistent: compiler flags set for a package are inherited by its dependencies, while variants are not. We should have these be consistent by allowing for inheritance to be enabled or disabled for both variants and compiler flags. - [x] Make new (spec language) operators - [x] Apply operators to variants and compiler flags - [x] Conflicts currently result in an unsatisfiable spec (i.e., you can't propagate two conflicting values) What I propose is using two of the currently used sigils to symbolized that the variant or compiler flag will be inherited: Example syntax: - `package ++variant` enabled variant that will be propagated to dependencies - `package +variant` enabled variant that will NOT be propagated to dependencies - `package ~~variant` disabled variant that will be propagated to dependencies - `package ~variant` disabled variant that will NOT be propagated to dependencies - `package cflags==True` `cflags` will be propagated to dependencies - `package cflags=True` `cflags` will NOT be propagated to dependencies Syntax for string-valued variants is similar to compiler flags.
2022-11-02tests: fix group membership check in sbang tests. (#33658)Todd Gamblin2-4/+13
Fixes an issue on the RHEL8 UBI container where this test would fail because `gr_mem` was empty for every entry in the `grp` DB. You have to check *both* the `pwd` database (which has primary groups) and `grp` (which has other gorups) to do this correctly. - [x] update `llnl.util.filesystem.group_ids()` to do this - [x] use it in the `sbang` test
2022-11-01Consolidate DAG traversal in traverse.py, support DFS/BFS (#33406)Harmen Stoppels5-204/+671
This PR introduces breadth-first traversal, and moves depth-first traversal logic out of Spec's member functions, into `traverse.py`. It introduces a high-level API with three main methods: ```python spack.traverse.traverse_edges(specs, kwargs...) spack.traverse.traverse_nodes(specs, kwags...) spack.traverse.traverse_tree(specs, kwargs...) ``` with the usual `root`, `order`, `cover`, `direction`, `deptype`, `depth`, `key`, `visited` kwargs for the first two. What's new is that `order="breadth"` is added for breadth-first traversal. The lower level API is not exported, but is certainly useful for advanced use cases. The lower level API includes visitor classes for direction reversal and edge pruning, which can be used to create more advanced traversal methods, especially useful when the `deptype` is not constant but depends on the node or depth. --- There's a couple nice use-cases for breadth-first traversal: - Sometimes roots have to be handled differently (e.g. follow build edges of roots but not of deps). BFS ensures that root nodes are always discovered at depth 0, instead of at any depth > 1 as a dep of another root. - When printing a tree, it would be nice to reduce indent levels so it fits in the terminal, and ensure that e.g. `zlib` is not printed at indent level 10 as a dependency of a build dep of a build dep -- rather if it's a direct dep of my package, I wanna see it at depth 1. This basically requires one breadth-first traversal to construct a tree, which can then be printed with depth-first traversal. - In environments in general, it's sometimes inconvenient to have a double loop: first over the roots then over each root's deps, and maintain your own `visited` set outside. With BFS, you can simply init the queue with the environment root specs and it Just Works. [Example here](https://github.com/spack/spack/blob/3ec73046995d9504d6e135f564f1370cfe31ba34/lib/spack/spack/environment/environment.py#L1815-L1816)
2022-11-01Unit tests: make unit tests work for aarch64 machines (#33625)Greg Becker24-91/+340
Currently, many tests hardcode to older versions of gcc for comparisons of concretization among compiler versions. Those versions are too old to concretize for `aarch64`-family targets, which leads to failing tests on `aarch64`. This PR fixes those tests by updating the compiler versions used for testing. Currently, many tests hardcode the expected architecture result in concretization to the `x86_64` family of architectures. This PR generalizes the tests that can be generalized, to cover multiple architecture families. For those that test specific relationships among `x86_64`-family targets, it ensures that concretization uses the `x86_64`-family targets in those cases. Currently, many tests rely on the fact that `AutotoolsPackage` imposes no dependencies on the inheriting package. That is not true on `aarch64`-family architectures. This PR ensures that the fact `AutotoolsPackage` on `aarch64` pulls in a dependency on `gnuconfig` is ignored when testing for the appropriate relationships among dependencies Additionally, 5 tests currently prompt the user for input when `gpg` is available in the user's path. This PR fixes that issue. And 7 tests fail currently when the user has a yubikey available. This PR fixes the incorrect gpg argument causing those issues.
2022-11-01Update command option for example (#33321)Greg Sjaardema1-1/+1
The `spack info <package>` command does not show the `Virtual Packages:` output unless the `--virtuals` command option is passed. Before this changes, the information that the command is supposed to be illustrating is not shown in the example and is confusing.
2022-11-01Allow target requirements in packages.yaml (#32528)Massimiliano Culpo6-14/+32
This PR solves the issue reported in #32471 specifically for targets and operating systems, by avoiding to add a default platform to anonymous specs.
2022-11-01Add elf parsing utility function (#33628)Harmen Stoppels4-46/+635
Introduces `spack.util.elf.parse_elf(file_handle)`
2022-11-01Relocate links using prefix to prefix map (#33636)Harmen Stoppels3-58/+58
Previously symlinks were not relocated when they pointed across packages
2022-10-31Windows: fix library loading and enable Clingo bootstrapping (#33400)John W. Parent5-69/+104
Changes to improve locating shared libraries on Windows, which in turn enables the use of Clingo. This PR attempts to establish a proper distinction between linking on Windows vs. Linux/Mac: on Windows, linking is always done with .lib files (never .dll files). This somewhat complicates the model since the Spec.lib method could return libraries that were used for both linking and loading, but since these are not always the same on Windows, it was decided to treat Spec.libs as being for link-time libraries. Additional functions are added to help dependents locate run-time libraries. * Clingo is now the default concretizer on Windows * Clingo is now the concretizer used for unit tests on Windows * Fix a permissions issue that can occur while moving Git files during fetching/staging * Packages can now implement "win_add_library_dependent" to register files/directories that include libraries that would need to link to dependency dlls * Packages can now implement "win_add_rpath" to register the locations of dlls that dependents would want to load * "Spec.libs" on Windows is updated to return link-time libraries (i.e. .lib files, rather than .dll files) * PackageBase.rpath on Windows is now updated to return the most-likely locations where .dlls will be found (which is generally in the bin/ directory)
2022-10-31Make --backtrace show non-SpackError backtraces (#33540)Harmen Stoppels1-3/+8
2022-10-31remove sequential filter in binary relo (#33608)Harmen Stoppels1-16/+4
Currently there's a slow sequential step in binary relocation where all strings of a binary are collected, with rpaths removed, and then filtered for the old install root. This is completely unnecessary, and also incorrect, since we replace more than just the old install root in the prefix to prefix mapping. And in fact the prefix to prefix mapping is parallel, and a single pass. So even as an optimization, this filter makes no sense anymore. Therefor we remove it
2022-10-31_replace_prefix_bin performance improvements (#33590)Harmen Stoppels2-26/+51
- single pass over the binary data matching all prefixes - collect offsets and replacement strings - do in-place updates with `fseek` / `fwrite`, since typically our replacement touch O(few bytes) while the file is O(many megabytes) - be nice: leave the file untouched if some string can't be replaced
2022-10-29Deprecate spack bootstrap trust/untrust (#33600)Massimiliano Culpo4-57/+90
* Deprecate spack bootstrap trust/untrust * Update CI * Update tests
2022-10-27CachedCMakePackage: Add back initconfig as a defined phase (#33575)Chris White3-4/+7
Also: add type annotation to indicate that "phases" is always a tuple of strings.
2022-10-27LuaPackage: add missing attribute (#33551)Massimiliano Culpo1-0/+3
fixes #33544
2022-10-26Windows: fix bootstrap and package install failure (#32942)百地 希留耶2-1/+10
* Add patches for building clingo with MSVC * Help python find clingo DLL * If an executable is located in "C:\Program Files", Executable was running into issues with the extra space. This quotes the exe to ensure that it is treated as a single value. Signed-off-by: Kiruya Momochi <65301509+KiruyaMomochi@users.noreply.github.com>
2022-10-26Allow for packages with multiple build-systems (#30738)Massimiliano Culpo80-2323/+6226
This commit extends the DSL that can be used in packages to allow declaring that a package uses different build-systems under different conditions. It requires each spec to have a `build_system` single valued variant. The variant can be used in many context to query, manipulate or select the build system associated with a concrete spec. The knowledge to build a package has been moved out of the PackageBase hierarchy, into a new Builder hierarchy. Customization of the default behavior for a given builder can be obtained by coding a new derived builder in package.py. The "run_after" and "run_before" decorators are now applied to methods on the builder. They can also incorporate a "when=" argument to specify that a method is run only when certain conditions apply. For packages that do not define their own builder, forwarding logic is added between the builder and package (methods not found in one will be retrieved from the other); this PR is expected to be fully backwards compatible with unmodified packages that use a single build system.
2022-10-26fix use of non-existing kwarg (#33520)Harmen Stoppels1-1/+1
2022-10-26Relocation regex single pass (#33496)Harmen Stoppels3-21/+36
Instead of looping over multiple regexes and the entire text file contents, create a giant regex with all literal prefixes and do a single pass over files to detect prefixes. Not only is a single pass faster, it's also likely that the regex is compiled better, given that most prefixes share a common ... prefix.
2022-10-26gitlab ci: install binary deps faster (#33248)Harmen Stoppels3-40/+95
* Fast Gitlab CI job setup, and better legibility * Use a non-broken, recent GNU Make
2022-10-25dfs traversal: simplify edges in reverse mode (#33481)Harmen Stoppels1-27/+26
In the dfs code, flip edges so that `parent` means `from` and `spec` means `to` in the direction of traversal. This makes it slightly easier to write generic/composable code. For example when using visitors where one visitor reverses direction, and another only cares about accepting particular edges or not depending on whether the target node is seen before, it would be good if the second visitor didn't have to know whether the order was changed or not.
2022-10-25binary_distribution: compress level 9 -> 6 (#33513)Harmen Stoppels1-1/+5
Use the same compression level as `gzip` (6) instead of what Python uses (9). The LLVM tarball takes 4m instead of 12m to create, and is <1% larger. That's not worth the wait...
2022-10-25feature: Add -x|--explicit option to 'spack test run' (#32910)Tamara Dahlgren2-6/+20
2022-10-25BinaryCacheIndex: track update failures with cooldown (#33509)Jonathon Anderson1-6/+12
#32137 added an option to update() a BinaryCacheIndex with a cooldown: repeated attempts within this cooldown would not actually retry. However, the cooldown was not properly tracked for failures (which is common when the mirror does not store any binaries and therefore has no index.json). This commit ensures that update(..., with_cooldown=True) will also skip the update even if a failure has occurred within the cooldown period.
2022-10-25spec.py: prefer transitive link and direct build/run/test deps (#33498)Harmen Stoppels2-2/+48
Due to reuse concretization, we may generate DAGs with two occurrences of the same package corresponding to distinct specs. This happens when build dependencies are reused, since their dependencies are ignored in concretization. This caused a regression, for example: `spec['openssl']` would take the 'openssl' of the build dep `cmake`, instead of the direct `openssl` dependency, simply because the edge to `cmake` was traversed first and we do depth first traversal. One solution that was discussed is to limit `spec[name]` to just direct deps, or direct deps + transitive link deps, but this is too breaking. Instead, this PR simply prioritizes transitive link and direct build/run/test deps, and then falls back to a full DAG traversal. So, it's just about order of iteration.
2022-10-25Remove recursive symbolic link in lib/spack/docs from git repository (#33483)Massimiliano Culpo2-1/+3
Delete code removing the symlink during CI
2022-10-25Improve legibility of Gitlab CI (#33482)Harmen Stoppels2-4/+4
Use --backtrace in ci instead of --debug to reduce verbosity and don't show log on error, since log is already printed
2022-10-24Add filename to text_to_relocate only if it needs to be relocated (#31074)iarspider4-8/+54
Scan the text files for relocatable prefixes *before* creating a tarball, to reduce the amount of work to be done during install from binary cache. Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2022-10-24redact line numbers from grouped exception message (#33485)Danny McClanahan1-4/+8
2022-10-24locks: improved errors (#33477)Harmen Stoppels6-12/+59
Instead of showing ``` ==> Error: Timed out waiting for a write lock. ``` show ``` ==> Error: Timed out waiting for a write lock after 1.200ms and 4 attempts on file: /some/file ``` s.t. we actually get to see where acquiring a lock failed even when not running in debug mode. And use pretty time units everywhere, so we don't get 1.45e-9 seconds but 1.450ns etc.
2022-10-23backtraces with --backtrace (#33478)Harmen Stoppels1-1/+9
* backtraces without --debug Currently `--debug` is too verbose and not-`--debug` gives to little context about where exceptions are coming from. So, instead, it'd be nice to have `spack --backtrace` and `SPACK_BACKTRACE=1` as methods to get something inbetween: no verbose debug messages, but always a full backtrace. This is useful for CI, where we don't want to drown in debug messages when installing deps, but we do want to get details where something goes wrong if it goes wrong. * completion
2022-10-21Relocation should take hardlinks into account (#33460)Harmen Stoppels7-62/+190
Currently `relocate_text` and `relocate_text_bin` are unsafe in the sense that they run in parallel, and lead to races when modifying different items pointing to the same inode. This leads to the issue observed in #33453. This PR: 1. Renames those functions to `unsafe_*` so people are aware 2. Adds logic to deal with hardlinks in current binary packages 3. Adds logic to deal with hardlinks when creating new binary tarballs, so the install side doesn't have to de-dupe hardlinks. 4. Adds a test for 3 The assumption is that all our relocation logic preserves inodes. That is, we should never copy a file, modify it, and then move it back. I quickly verified, and its seems like this is true for (binary) text relocation, as well as rpath patching in patchelf (even when the file grows) and mach-o fixes.
2022-10-21Remove "spack buildcache copy" in v0.19.0 (#33437)Massimiliano Culpo1-86/+0
2022-10-20gitlab: Do not use root_spec['pkg_name'] anymore (#33403)Scott Wittenburg5-147/+27
* gitlab: Do not use root_spec['pkg_name'] anymore For a long time it was fine to index a concrete root spec with the name of a dependency in order to access the concrete dependency spec. Since pipelines started using `--use-buildcache dependencies:only,package:never` though, it has exposed a scheduling issue in how pipelines are generated. If a concrete root spec depends on two different hashes of `openssl` for example, indexing that root with just the package name is ambiguous, so we should no longer depend on that approach when scheduling jobs. * env: make sure exactly one spec in env matches hash
2022-10-19depfile: buildcache support (#33315)Harmen Stoppels2-35/+206
When installing some/all specs from a buildcache, build edges are pruned from those specs. This can result in a much smaller effective DAG. Until now, `spack env depfile` would always generate a full DAG. Ths PR adds the `spack env depfile --use-buildcache` flag that was introduced for `spack install` before. This way, not only can we drop build edges, but also we can automatically set the right buildcache related flags on the specific specs that are gonna get installed. This way we get parallel installs of binary deps without redundancy, which is useful for Gitlab CI.
2022-10-19Add a command to bootstrap Spack right now (#33407)Massimiliano Culpo1-0/+12
2022-10-19Relocate "run" type dependencies too (#33191)Stephen Sachs1-2/+2
When downloading from binary cache not only replace RPATHs to dependencies, but also text references to dependencies. Example: `autoconf@2.69` contains a text reference to the executable of its dependency `perl`: ``` $ grep perl-5 /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/autoconf-2.69-q3lo/bin/autoreconf eval 'case $# in 0) exec /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/perl-5.34.1-yphg/bin/perl -S "$0";; *) exec /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/perl-5.34.1-yphg/bin/perl -S "$0" "$@";; esac' ``` These references need to be replace or any package using `autoreconf` will fail as it cannot find the installed `perl`. Co-authored-by: Stephen Sachs <stesachs@amazon.com>
2022-10-19Update the binary index before attempting direct fetches (#32137)Jonathon Anderson3-19/+44
"spack install" will not update the binary index if given a concrete spec, which causes it to fall back to direct fetches when a simple index update would have helped. For S3 buckets in particular, this significantly and needlessly slows down the install process. This commit alters the logic so that the binary index is updated whenever a by-hash lookup fails. The lookup is attempted again with the updated index before falling back to direct fetches. To avoid updating too frequently (potentially once for each spec being installed), BinaryCacheIndex.update now includes a "cooldown" option, and when this option is enabled it will not update more than once in a cooldown window (set in config.yaml). Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
2022-10-18spack checksum: warn if version is deprecated (#32438)Tamara Dahlgren5-2/+42
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>