summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2022-11-06solver setup: extract virtual dependencies from reusable specs (#32434)Greg Becker2-0/+28
* extract virtual dependencies from reusable specs * bugfix to avoid establishing new node for virtual
2022-11-06package preferences: allow specs to be configured buildable when their ↵Greg Becker2-9/+26
virtuals are not (#18269) * respect spec buildable that overrides virtual buildable
2022-11-06improve error message for dependency on nonexistant compiler (#32084)Greg Becker1-12/+25
2022-11-06solver: do not punish explicitly requested compiler mismatches (#30074)Greg Becker2-1/+29
2022-11-06MesonPackage: disable automatic download and install of dependencies (#33717)Michael Kuhn1-2/+4
Without this, Meson will use its Wraps to automatically download and install dependencies. We want to manage dependencies explicitly, therefore disable this functionality.
2022-11-06allow multiple compatible deps from CLI (#21262)Greg Becker2-2/+24
Currently, Spack can fail for a valid spec if the spec is constructed from overlapping, but not conflicting, concrete specs via the hash. For example, if abcdef and ghijkl are the hashes of specs that both depend on zlib/mnopqr, then foo ^/abcdef ^/ghijkl will fail to construct a spec, with the error message "Cannot depend on zlib... twice". This PR changes this behavior to check whether the specs are compatible before failing. With this PR, foo ^/abcdef ^/ghijkl will concretize. As a side-effect, so will foo ^zlib ^zlib and other specs that are redundant on their dependencies.
2022-11-06canonicalize_path: add arch information to substitutions (#29810)Greg Becker3-0/+50
Co-authored-by: becker33 <becker33@users.noreply.github.com>
2022-11-05Add support for Python 3.11 (#33505)Massimiliano Culpo4-12/+182
Argparse started raising ArgumentError exceptions when the same parser is added twice. Therefore, we perform the addition only if the parser is not there already Port match syntax to our unparser
2022-11-05Fix relocation to avoid overwriting merged constant strings (#32253)Tom Scogland6-83/+311
Compilers and linker optimize string constants for space by aliasing them when one is a suffix of another. For gcc / binutils this happens already at -O1, due to -fmerge-constants. This means that we have to take care during relocation to always preserve a certain length of the suffix of those prefixes that are C-strings. In this commit we pick length 7 as a safe suffix length, assuming the suffix is typically the 7 characters from the hash (i.e. random), so it's unlikely to alias with any string constant used in the sources. In general we now pad shortened strings from the left with leading dir seperators, but in the case of C-strings that are much shorter and don't share a common suffix (due to projections), we do allow shrinking the C-string, appending a null, and retaining the old part of the prefix. Also when rewiring, we ensure that the new hash preserves the last 7 bytes of the old hash. Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2022-11-05packages.yaml: set url/git (#33275)Peter Scheibel4-0/+113
A user may want to set some attributes on a package without actually modifying the package (e.g. if they want to git pull updates to the package without conflicts). This PR adds a per-package configuration section called "set", which is a dictionary of attribute names to desired values. For example: packages: openblas: package_attributes: submodules: true git: "https://github.com/myfork/openblas" in this case, the package will always retrieve git submodules, and will use an alternate location for the git repo. While git, url, and submodules are the attributes for which we envision the most usage, this allows any attribute to be overridden, and the acceptable values are any value parseable from yaml.
2022-11-05unparser: fix bug in unit test assertion (#33722)Massimiliano Culpo1-5/+3
2022-11-05Fix formatting in packaging guide (#33714)iarspider1-10/+3
2022-11-04demote warning for no source id to debug message (#33657)Greg Becker4-15/+15
* demote warning for no source id to debug message
2022-11-04Cray support: use linux platform for newer craype versions (#29392)Greg Becker7-30/+105
Newer versions of the CrayPE for EX systems have standalone compiler executables for CCE and compiler wrappers for Cray MPICH. With those, we can treat the cray systems as part of the linux platform rather than having a separate cray platform. This PR: - [x] Changes cray platform detection to ignore EX systems with Craype version 21.10 or later - [x] Changes the cce compiler to be detectable via paths - [x] Changes the spack compiler wrapper to understand the executable names for the standalone cce compiler (`craycc`, `crayCC`, `crayftn`).
2022-11-04delegate to cray modules for target args on cray (#17857)Greg Becker2-2/+8
2022-11-04Fix typo in docs (#33662)Matthieu Boileau1-1/+1
2022-11-04Deprecate old YAML format for buildcaches (#33707)Massimiliano Culpo1-2/+17
2022-11-04Updates to stand-alone test documentation (#33703)Tamara Dahlgren2-22/+154
2022-11-04fix requires test for aarch64 (#33656)Greg Becker1-2/+2
2022-11-04Add in-place RPATH replacement (#27610)Harmen Stoppels5-52/+155
Whenever the rpath string actually _grows_, it falls back to patchelf, when it stays the same length or gets shorter, we update it in-place, padded with null bytes. This PR only deals with absolute -> absolute rpath replacement. We don't use `_build_tarball(relative=True)` in our CI. If `relative` then it falls back to the old replacement code. With this PR, relocation time goes down significantly, likely because patchelf does some odd things with mmap, causing lots of overhead. Example: - `binutils`: 700MB installed, goes from `1.91s` to `0.57s`, or `3.4x` faster. Relocation time: 27% -> 10% of total install time - `llvm`: 6.8GB installed, goes from `28.56s` to `5.38`, or `5.3x` faster. Relocation time: 44% -> 13% of total install time The bottleneck is now decompression. Note: I'm somewhat confused about the "relative rpath" code paths. Right now this PR only deals with absolute -> absolute replacement. As far as I understand, if you embrace relative rpaths when uploading to the buildcache, the whole point is you _don't_ want to patch rpaths on install? So it seems fine to not expand `$ORIGIN` again imho.
2022-11-04Fix non-parallel make under depfile jobserver (#32862)Jordan Galby2-37/+58
When a package asks for non-parallel make, we need to force `make -j1` because just doing `make` will run in parallel under jobserver (e.g. `spack env depfile`). We now always add `-j1` when asked for a non-parallel execution (even if there is no jobserver). And each `MakeExecutable` can now ask for jobserver support or not. For example: the default `ninja` does not support jobserver so spack applies the default `-j`, but `ninja@kitware` or `ninja-fortran` does, so spack doesn't add `-j`. Tips: you can run `SPACK_INSTALL_FLAGS=-j1 make -f spack-env-depfile.make -j8` to avoid massive job-spawning because of build tools that don't support jobserver (ninja).
2022-11-03propagation: don't count propagated variant values as non-default (#33687)Todd Gamblin1-0/+2
We try to avoid non-default variant values in the concretizer, but this doesn't make sense for variants forced to take some non-default value by variant propagation. Counting this as a penalty effectively biases the concretizer for small specs dependency graphs -- something we try very hard to avoid elsewhere because it can lead to very strange decisions. Example: with the penalty, `spack spec hdf5` will choose the default `openmpi` as its `mpi` provider, but `spack spec hdf5 ~~shared` will choose `mpich` because it has to set fewer non-default variant values because `mpich`'s DAG is smaller. That's not a good reason to prefer a non-default virtual provider. To fix this, if the user explicitly requests a non-default value to be propagated, there shouldn't be a penalty. Variant values set on the CLI already don't count as default; we just need to extend that to propagated values.
2022-11-03Experimental binding of shared ELF libraries (#31948)Harmen Stoppels11-11/+358
Adds another post install hook that loops over the install prefix, looking for shared libraries type of ELF files, and sets the soname to their own absolute paths. The idea being, whenever somebody links against those libraries, the linker copies the soname (which is the absolute path to the library) as a "needed" library, so that at runtime the dynamic loader realizes the needed library is a path which should be loaded directly without searching. As a result: 1. rpaths are not used for the fixed/static list of needed libraries in the dynamic section (only for _actually_ dynamically loaded libraries through `dlopen`), which largely solves the issue that Spack's rpaths are a heuristic (`<prefix>/lib` and `<prefix>/lib64` might not be where libraries really are...) 2. improved startup times (no library search required)
2022-11-03gitlab: Prune untouched specs less aggressively (#33669)Scott Wittenburg3-26/+31
Untouched spec pruning was added to reduce the number of specs developers see getting rebuilt in their PR pipelines that they don't understand. Because the state of the develop mirror lags quite far behind the tip of the develop branch, PRs often find they need to rebuild things untouched by their PR. Untouched spec pruning was previously implemented by finding all specs in the environment with names of packages touched by the PR, traversing in both directions the DAGS of those specs, and adding all dependencies as well as dependents to a list of concrete specs that should not be considered for pruning. We found that this heuristic results in too many pruned specs, and that dependents of touched specs must have all their dependencies added to the list of specs that should not be considered for pruning.
2022-11-03Use spack.traverse.traverse_nodes where useful (#33677)Harmen Stoppels1-45/+33
2022-11-03Update metadata for bootstrapping (#33665)Massimiliano Culpo4-51/+24
2022-11-02Bugfix for spec objects modified by flag handlers (#33682)Greg Becker2-1/+13
This issue was introduced in #29761: ``` ==> Installing ncurses-6.3-22hz6q6cvo3ep2uhrs3erpp2kogxncbn ==> No binary for ncurses-6.3-22hz6q6cvo3ep2uhrs3erpp2kogxncbn found: installing from source ==> Using cached archive: /spack/var/spack/cache/_source-cache/archive/97/97fc51ac2b085d4cde31ef4d2c3122c21abc217e9090a43a30fc5ec21684e059.tar.gz ==> No patches needed for ncurses ==> ncurses: Executing phase: 'autoreconf' ==> ncurses: Executing phase: 'configure' ==> ncurses: Executing phase: 'build' ==> ncurses: Executing phase: 'install' ==> Error: AttributeError: 'str' object has no attribute 'propagate' The 'ncurses' package cannot find an attribute while trying to build from sources. This might be due to a change in Spack's package format to support multiple build-systems for a single package. You can fix this by updating the build recipe, and you can also report the issue as a bug. More information at https://spack.readthedocs.io/en/latest/packaging_guide.html#installation-procedure /spack/lib/spack/spack/build_environment.py:1075, in _setup_pkg_and_run: 1072 tb_string = traceback.format_exc() 1073 1074 # build up some context from the offending package so we can >> 1075 # show that, too. 1076 package_context = get_package_context(tb) 1077 1078 logfile = None ``` It turns out this was caused by a bug that had been around much longer, in which the flags were passed by reference to the flag_handler, and the flag_handler was modifying the spec object, not just the flags given to the build system. The scope of this bug was limited by the forking model in Spack, which is how it went under the radar for so long. PR includes regression test.
2022-11-02remove `deptype_query` remnants and fix incorrect `deptypes` kwarg (#33670)Harmen Stoppels1-1/+1
* remove deptype_query remnants * deptypes -> deptype These arguments haven't existed since 2017, but `traverse` now fails on unknown **kwargs, so they have finally popped up.
2022-11-02propagation: improve performanceGregory Becker3-61/+68
This updates the propagation logic used in `concretize.lp` to avoid rules with `path()` in the body and instead base propagation around `depends_on()`.
2022-11-02flags/variants: Add ++/~~/== syntax for propagation to dependenciesKayla Butler14-118/+426
Currently, compiler flags and variants are inconsistent: compiler flags set for a package are inherited by its dependencies, while variants are not. We should have these be consistent by allowing for inheritance to be enabled or disabled for both variants and compiler flags. - [x] Make new (spec language) operators - [x] Apply operators to variants and compiler flags - [x] Conflicts currently result in an unsatisfiable spec (i.e., you can't propagate two conflicting values) What I propose is using two of the currently used sigils to symbolized that the variant or compiler flag will be inherited: Example syntax: - `package ++variant` enabled variant that will be propagated to dependencies - `package +variant` enabled variant that will NOT be propagated to dependencies - `package ~~variant` disabled variant that will be propagated to dependencies - `package ~variant` disabled variant that will NOT be propagated to dependencies - `package cflags==True` `cflags` will be propagated to dependencies - `package cflags=True` `cflags` will NOT be propagated to dependencies Syntax for string-valued variants is similar to compiler flags.
2022-11-02tests: fix group membership check in sbang tests. (#33658)Todd Gamblin2-4/+13
Fixes an issue on the RHEL8 UBI container where this test would fail because `gr_mem` was empty for every entry in the `grp` DB. You have to check *both* the `pwd` database (which has primary groups) and `grp` (which has other gorups) to do this correctly. - [x] update `llnl.util.filesystem.group_ids()` to do this - [x] use it in the `sbang` test
2022-11-01Consolidate DAG traversal in traverse.py, support DFS/BFS (#33406)Harmen Stoppels5-204/+671
This PR introduces breadth-first traversal, and moves depth-first traversal logic out of Spec's member functions, into `traverse.py`. It introduces a high-level API with three main methods: ```python spack.traverse.traverse_edges(specs, kwargs...) spack.traverse.traverse_nodes(specs, kwags...) spack.traverse.traverse_tree(specs, kwargs...) ``` with the usual `root`, `order`, `cover`, `direction`, `deptype`, `depth`, `key`, `visited` kwargs for the first two. What's new is that `order="breadth"` is added for breadth-first traversal. The lower level API is not exported, but is certainly useful for advanced use cases. The lower level API includes visitor classes for direction reversal and edge pruning, which can be used to create more advanced traversal methods, especially useful when the `deptype` is not constant but depends on the node or depth. --- There's a couple nice use-cases for breadth-first traversal: - Sometimes roots have to be handled differently (e.g. follow build edges of roots but not of deps). BFS ensures that root nodes are always discovered at depth 0, instead of at any depth > 1 as a dep of another root. - When printing a tree, it would be nice to reduce indent levels so it fits in the terminal, and ensure that e.g. `zlib` is not printed at indent level 10 as a dependency of a build dep of a build dep -- rather if it's a direct dep of my package, I wanna see it at depth 1. This basically requires one breadth-first traversal to construct a tree, which can then be printed with depth-first traversal. - In environments in general, it's sometimes inconvenient to have a double loop: first over the roots then over each root's deps, and maintain your own `visited` set outside. With BFS, you can simply init the queue with the environment root specs and it Just Works. [Example here](https://github.com/spack/spack/blob/3ec73046995d9504d6e135f564f1370cfe31ba34/lib/spack/spack/environment/environment.py#L1815-L1816)
2022-11-01Unit tests: make unit tests work for aarch64 machines (#33625)Greg Becker24-91/+340
Currently, many tests hardcode to older versions of gcc for comparisons of concretization among compiler versions. Those versions are too old to concretize for `aarch64`-family targets, which leads to failing tests on `aarch64`. This PR fixes those tests by updating the compiler versions used for testing. Currently, many tests hardcode the expected architecture result in concretization to the `x86_64` family of architectures. This PR generalizes the tests that can be generalized, to cover multiple architecture families. For those that test specific relationships among `x86_64`-family targets, it ensures that concretization uses the `x86_64`-family targets in those cases. Currently, many tests rely on the fact that `AutotoolsPackage` imposes no dependencies on the inheriting package. That is not true on `aarch64`-family architectures. This PR ensures that the fact `AutotoolsPackage` on `aarch64` pulls in a dependency on `gnuconfig` is ignored when testing for the appropriate relationships among dependencies Additionally, 5 tests currently prompt the user for input when `gpg` is available in the user's path. This PR fixes that issue. And 7 tests fail currently when the user has a yubikey available. This PR fixes the incorrect gpg argument causing those issues.
2022-11-01Update command option for example (#33321)Greg Sjaardema1-1/+1
The `spack info <package>` command does not show the `Virtual Packages:` output unless the `--virtuals` command option is passed. Before this changes, the information that the command is supposed to be illustrating is not shown in the example and is confusing.
2022-11-01Allow target requirements in packages.yaml (#32528)Massimiliano Culpo6-14/+32
This PR solves the issue reported in #32471 specifically for targets and operating systems, by avoiding to add a default platform to anonymous specs.
2022-11-01Add elf parsing utility function (#33628)Harmen Stoppels4-46/+635
Introduces `spack.util.elf.parse_elf(file_handle)`
2022-11-01Relocate links using prefix to prefix map (#33636)Harmen Stoppels3-58/+58
Previously symlinks were not relocated when they pointed across packages
2022-10-31Windows: fix library loading and enable Clingo bootstrapping (#33400)John W. Parent5-69/+104
Changes to improve locating shared libraries on Windows, which in turn enables the use of Clingo. This PR attempts to establish a proper distinction between linking on Windows vs. Linux/Mac: on Windows, linking is always done with .lib files (never .dll files). This somewhat complicates the model since the Spec.lib method could return libraries that were used for both linking and loading, but since these are not always the same on Windows, it was decided to treat Spec.libs as being for link-time libraries. Additional functions are added to help dependents locate run-time libraries. * Clingo is now the default concretizer on Windows * Clingo is now the concretizer used for unit tests on Windows * Fix a permissions issue that can occur while moving Git files during fetching/staging * Packages can now implement "win_add_library_dependent" to register files/directories that include libraries that would need to link to dependency dlls * Packages can now implement "win_add_rpath" to register the locations of dlls that dependents would want to load * "Spec.libs" on Windows is updated to return link-time libraries (i.e. .lib files, rather than .dll files) * PackageBase.rpath on Windows is now updated to return the most-likely locations where .dlls will be found (which is generally in the bin/ directory)
2022-10-31Make --backtrace show non-SpackError backtraces (#33540)Harmen Stoppels1-3/+8
2022-10-31remove sequential filter in binary relo (#33608)Harmen Stoppels1-16/+4
Currently there's a slow sequential step in binary relocation where all strings of a binary are collected, with rpaths removed, and then filtered for the old install root. This is completely unnecessary, and also incorrect, since we replace more than just the old install root in the prefix to prefix mapping. And in fact the prefix to prefix mapping is parallel, and a single pass. So even as an optimization, this filter makes no sense anymore. Therefor we remove it
2022-10-31_replace_prefix_bin performance improvements (#33590)Harmen Stoppels2-26/+51
- single pass over the binary data matching all prefixes - collect offsets and replacement strings - do in-place updates with `fseek` / `fwrite`, since typically our replacement touch O(few bytes) while the file is O(many megabytes) - be nice: leave the file untouched if some string can't be replaced
2022-10-29Deprecate spack bootstrap trust/untrust (#33600)Massimiliano Culpo4-57/+90
* Deprecate spack bootstrap trust/untrust * Update CI * Update tests
2022-10-27CachedCMakePackage: Add back initconfig as a defined phase (#33575)Chris White3-4/+7
Also: add type annotation to indicate that "phases" is always a tuple of strings.
2022-10-27LuaPackage: add missing attribute (#33551)Massimiliano Culpo1-0/+3
fixes #33544
2022-10-26Windows: fix bootstrap and package install failure (#32942)百地 希留耶2-1/+10
* Add patches for building clingo with MSVC * Help python find clingo DLL * If an executable is located in "C:\Program Files", Executable was running into issues with the extra space. This quotes the exe to ensure that it is treated as a single value. Signed-off-by: Kiruya Momochi <65301509+KiruyaMomochi@users.noreply.github.com>
2022-10-26Allow for packages with multiple build-systems (#30738)Massimiliano Culpo80-2323/+6226
This commit extends the DSL that can be used in packages to allow declaring that a package uses different build-systems under different conditions. It requires each spec to have a `build_system` single valued variant. The variant can be used in many context to query, manipulate or select the build system associated with a concrete spec. The knowledge to build a package has been moved out of the PackageBase hierarchy, into a new Builder hierarchy. Customization of the default behavior for a given builder can be obtained by coding a new derived builder in package.py. The "run_after" and "run_before" decorators are now applied to methods on the builder. They can also incorporate a "when=" argument to specify that a method is run only when certain conditions apply. For packages that do not define their own builder, forwarding logic is added between the builder and package (methods not found in one will be retrieved from the other); this PR is expected to be fully backwards compatible with unmodified packages that use a single build system.
2022-10-26fix use of non-existing kwarg (#33520)Harmen Stoppels1-1/+1
2022-10-26Relocation regex single pass (#33496)Harmen Stoppels3-21/+36
Instead of looping over multiple regexes and the entire text file contents, create a giant regex with all literal prefixes and do a single pass over files to detect prefixes. Not only is a single pass faster, it's also likely that the regex is compiled better, given that most prefixes share a common ... prefix.
2022-10-26gitlab ci: install binary deps faster (#33248)Harmen Stoppels3-40/+95
* Fast Gitlab CI job setup, and better legibility * Use a non-broken, recent GNU Make
2022-10-25dfs traversal: simplify edges in reverse mode (#33481)Harmen Stoppels1-27/+26
In the dfs code, flip edges so that `parent` means `from` and `spec` means `to` in the direction of traversal. This makes it slightly easier to write generic/composable code. For example when using visitors where one visitor reverses direction, and another only cares about accepting particular edges or not depending on whether the target node is seen before, it would be good if the second visitor didn't have to know whether the order was changed or not.