summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2022-10-21Relocation should take hardlinks into account (#33460)Harmen Stoppels7-62/+190
Currently `relocate_text` and `relocate_text_bin` are unsafe in the sense that they run in parallel, and lead to races when modifying different items pointing to the same inode. This leads to the issue observed in #33453. This PR: 1. Renames those functions to `unsafe_*` so people are aware 2. Adds logic to deal with hardlinks in current binary packages 3. Adds logic to deal with hardlinks when creating new binary tarballs, so the install side doesn't have to de-dupe hardlinks. 4. Adds a test for 3 The assumption is that all our relocation logic preserves inodes. That is, we should never copy a file, modify it, and then move it back. I quickly verified, and its seems like this is true for (binary) text relocation, as well as rpath patching in patchelf (even when the file grows) and mach-o fixes.
2022-10-21Remove "spack buildcache copy" in v0.19.0 (#33437)Massimiliano Culpo1-86/+0
2022-10-20gitlab: Do not use root_spec['pkg_name'] anymore (#33403)Scott Wittenburg5-147/+27
* gitlab: Do not use root_spec['pkg_name'] anymore For a long time it was fine to index a concrete root spec with the name of a dependency in order to access the concrete dependency spec. Since pipelines started using `--use-buildcache dependencies:only,package:never` though, it has exposed a scheduling issue in how pipelines are generated. If a concrete root spec depends on two different hashes of `openssl` for example, indexing that root with just the package name is ambiguous, so we should no longer depend on that approach when scheduling jobs. * env: make sure exactly one spec in env matches hash
2022-10-19depfile: buildcache support (#33315)Harmen Stoppels2-35/+206
When installing some/all specs from a buildcache, build edges are pruned from those specs. This can result in a much smaller effective DAG. Until now, `spack env depfile` would always generate a full DAG. Ths PR adds the `spack env depfile --use-buildcache` flag that was introduced for `spack install` before. This way, not only can we drop build edges, but also we can automatically set the right buildcache related flags on the specific specs that are gonna get installed. This way we get parallel installs of binary deps without redundancy, which is useful for Gitlab CI.
2022-10-19Add a command to bootstrap Spack right now (#33407)Massimiliano Culpo1-0/+12
2022-10-19Relocate "run" type dependencies too (#33191)Stephen Sachs1-2/+2
When downloading from binary cache not only replace RPATHs to dependencies, but also text references to dependencies. Example: `autoconf@2.69` contains a text reference to the executable of its dependency `perl`: ``` $ grep perl-5 /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/autoconf-2.69-q3lo/bin/autoreconf eval 'case $# in 0) exec /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/perl-5.34.1-yphg/bin/perl -S "$0";; *) exec /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/perl-5.34.1-yphg/bin/perl -S "$0" "$@";; esac' ``` These references need to be replace or any package using `autoreconf` will fail as it cannot find the installed `perl`. Co-authored-by: Stephen Sachs <stesachs@amazon.com>
2022-10-19Update the binary index before attempting direct fetches (#32137)Jonathon Anderson3-19/+44
"spack install" will not update the binary index if given a concrete spec, which causes it to fall back to direct fetches when a simple index update would have helped. For S3 buckets in particular, this significantly and needlessly slows down the install process. This commit alters the logic so that the binary index is updated whenever a by-hash lookup fails. The lookup is attempted again with the updated index before falling back to direct fetches. To avoid updating too frequently (potentially once for each spec being installed), BinaryCacheIndex.update now includes a "cooldown" option, and when this option is enabled it will not update more than once in a cooldown window (set in config.yaml). Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
2022-10-18spack checksum: warn if version is deprecated (#32438)Tamara Dahlgren5-2/+42
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-10-18Docs: Spack info option updates (#33376)Tamara Dahlgren3-10/+27
2022-10-18Reusable --use-buildcache with better validation (#33388)Harmen Stoppels4-31/+76
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-10-17Initial contribution of LibPressio ecosystem (#32630)Robert Underwood1-0/+2
* Add libpressio and dependencies; some of these packages are maintained as forks of the original repositories and in those cases the docstring mentions this. * Add optional dependency in adios2 on libpressio * cub package: set CUB_DIR environment variable for dependent installations * Clear R_HOME/R_ENVIRON before Spack installation (avoid sources outside of Spack from affecting the installation in Spack) * Rename dlib to dorian3d-dlib and update dependents; add new dlib implementation. Pending an official policy on how to handle packages with short names, reviewer unilaterally decided that the rename was acceptable given that the new Spack dlib package is referenced more widely (by orders of magnitude) than the original Co-authored-by: Samuel Li <shaomeng@users.noreply.github.com>
2022-10-17database: don't warn adding missing build deps (#33361)Harmen Stoppels1-27/+48
When installing an individual spec `spack --only=package --cache-only /xyz` from a buildcache, Spack currently issues tons of warnings about missing build deps (and their deps) in the database. This PR disables these warnings, since it's fine to have a spec without its build deps in the db (they are just "missing").
2022-10-17installer.py: traverse_dependencies has local deptype (#33367)Harmen Stoppels1-9/+19
Currently `traverse_dependencies` fixes deptypes to traverse once and for all in the recursion, but this is incorrect, since deptypes depend on the node (e.g. if it's a dependency and cache-only, don't follow build type edges, even if the parent is build from sources and needs build deps.)
2022-10-17Support spackbot rebuilding all specs from source (#32596)Scott Wittenburg2-21/+66
Support spackbot rebuilding all specs from source when asked (with "rebuild everything") - Allow overriding --prune-dag cli opt with env var - Use job variable to optionally prevent rebuild jobs early exit behavior - ci rebuild: Use new install argument to insist deps are always installed from binary, but package is only installed from source. - gitlab: fix bug w/ untouched pruning - ci rebuild: install from hash rather than json file - When doing a "rebuild everything" pipeline, make sure that each install job only consumes binary dependencies from the mirror being populated by the current pipeline. This avoids using, e.g. binaries from develop, when rebuilding everything on a PR. - When running a pipeline to rebuild everything, do not die because we generated a hash on the broken specs list. Instead only warn in that case. - bugfix: Replace broken no-args tty.die() with sys.exit(1)
2022-10-17installer.py: fix/test get_deptypes (#33363)Harmen Stoppels2-1/+40
Fixing an oversight in https://github.com/spack/spack/pull/32537 `get_deptypes` should depend on new `package/dependencies_cache_only` props.
2022-10-17gitlab ci: Do not force protected build jobs to run on aws runners (#33314)Scott Wittenburg1-1/+1
2022-10-17installer.py: show timers for binary install (#33305)Harmen Stoppels1-23/+57
Print a message of the form ``` Fetch mm:ss. Build: mm:ss. Total: mm:ss ``` when installing from buildcache. Previously this only happened for source builds.
2022-10-15CI: allow multiple matches to combine tags (#32290)Jonathon Anderson3-13/+59
Currently "spack ci generate" chooses the first matching entry in gitlab-ci:mappings to fill attributes for a generated build-job, requiring that the entire configuration matrix is listed out explicitly. This unfortunately causes significant problems in environments with large configuration spaces, for example the environment in #31598 (spack.yaml) supports 5 operating systems, 3 architectures and 130 packages with explicit size requirements, resulting in 1300 lines of configuration YAML. This patch adds a configuraiton option to the gitlab-ci schema called "match_behavior"; when it is set to "merge", all matching entries are applied in order to the final build-job, allowing a few entries to cover an entire matrix of configurations. The default for "match_behavior" is "first", which behaves as before this commit (only the runner attributes of the first match are used). In addition, match entries may now include a "remove-attributes" configuration, which allows matches to remove tags that have been aggregated by prior matches. This only makes sense to use with "match_behavior:merge". You can combine "runner-attributes" with "remove-attributes" to effectively override prior tags.
2022-10-15depfile: update docs (#33279)Harmen Stoppels1-9/+37
2022-10-13meson: remove slash in path (#33292)Adam J. Stewart1-1/+1
2022-10-13gitlab ci: Print better information about broken specs (#33124)Scott Wittenburg3-49/+98
When a pipeline generation job is automatically failed because it generated jobs for specs known to be broken on develop, print better information about the broken specs that were encountered. Include at a minimum the hash and the url of the job whose failure caused it to be put on the broken specs list in the first place.
2022-10-13Add missing upperbound to docs/spack.yaml (#33280)Harmen Stoppels1-0/+2
2022-10-12env depfile: allow deps only install (#33245)Harmen Stoppels2-65/+32
* env depfile: allow deps only install - Refactor `spack env depfile` to use a Jinja template, making it a bit easier to follow as a human being. - Add a layer of indirection in the generated Makefile through an `<prefix>/.install-deps/<hash>` target, which allows one to specify different options when installing dependencies. For example, only verbose/debug mode on when installing some particular spec: ``` $ spack -e my_env env depfile -o Makefile --make-target-prefix example $ make example/.install-deps/<hash> -j16 $ make example/.install/<hash> SPACK="spack -d" SPACK_INSTALL_FLAGS=--verbose -j16 ``` This could be used to speed up `spack ci rebuild`: - Parallel install of dependencies from buildcache - Better readability of logs, e.g. reducing verbosity when installing dependencies, and splitting logs into deps.log and current_spec.log * Silence please!
2022-10-12Fix typo in docs (#33182)Massimiliano Culpo1-1/+1
2022-10-11Specify GCC prefix in LLVM-based compilers (#33146)Harmen Stoppels2-0/+29
* spack.compiler.Compiler: introduce prefix property We currently don't really have something that gives the GCC install path, which is used by many LLVM-based compilers (llvm, llvm-amdgpu, nvhpc, ...) to fix the GCC toolchain once and for all. This `prefix` property is dynamic in the sense that it queries the compiler itself. This is necessary because it's not easy to deduce the install path from the `cc` property (might be a symlink, might be a filename like `gcc` which works by having the compiler load a module that sets the PATH variable, might be a generic compiler wrapper based on environment variables like on cray...). With this property introduced, we can clean up some recipes that have the logic repeated for GCC. * intel-oneapi-compilers: set --gcc-sysroot to %gcc prefix
2022-10-11Refactor a few classes related to package repositories (#32273)Massimiliano Culpo27-669/+466
Caches used by repositories don't reference the global spack.repo.path instance anymore, but get the repository they refer to during initialization. Spec.virtual now use the index, and computation done to compute the index use Repository.is_virtual_safe. Code to construct mock packages and mock repository has been factored into a unique MockRepositoryBuilder that is used throughout the codebase. Add debug print for pushing and popping config scopes. Changed spack.repo.use_repositories so that it can override or not previous repos spack.repo.use_repositories updates spack.config.config according to the modifications done Removed a peculiar behavior from spack.config.Configuration where push would always bubble-up a scope named command_line if it existed
2022-10-11`spack diff` any specs you want (#32737)Peter Scheibel1-4/+6
Resolves #31782 With this change, if a spec is concrete after parsing (e.g. spec.yaml or /hash-based), then it is not disambiguated (a process which requires (a) that the spec be installed and (b) that it be part of the currently-active environment). This commit allows you to: * Diff specs from an environment regardless of whether they have been installed (more useful for projection/matrix-based envs) * Diff specs read from .yaml files which may or may not be entirely different installations of Spack Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-10-10Docs: Getting Started Dependencies (#32480)Axel Huebl4-2/+33
* Docs: Getting Started Dependencies Finally document what one needs to install to use Spack on Linux and Mac :-) With <3 for minimal container users and my colleagues with their fancy Macs. * Debian Update Packages: GCC, Python - build-essential: includes gcc, g++ (thx Cory) - Python: add python3-venv, python3-distutils (thx Pradyun) * Add RHEL8 Dependencies
2022-10-10shell prompt: enclose control sequence in brackets (#33079)Brian Vanderwende2-4/+10
When setting `PS1` in Bash, it's required to enclose non-printable characters in square brackets, so that the width of the terminal is handled correctly. See https://www.gnu.org/software/bash/manual/bash.html#Controlling-the-Prompt
2022-10-10PythonPackage: fix libs/headers attributes (#32970)Adam J. Stewart1-3/+3
2022-10-07autotools: extend patching of the libtool script (#30768)Sergey Kosukhin4-34/+208
* filter_file: introduce argument 'start_at' * autotools: extend patching of the libtool script * autotools: refactor _patch_usr_bin_file * autotools: improve readability of the filtering * autotools: keep the modification time of the configure scripts * autotools: do not try to patch directories * autotools: explain libtool patching for posterity Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com> Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2022-10-07add compatibility for rocky8 and rhel8 (#33068)MatthewLieber1-0/+3
2022-10-06Add a warning on Python 2.7 deprecation (#33052)Massimiliano Culpo2-0/+9
Co-authored-by: alalazo <alalazo@users.noreply.github.com> Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2022-10-06tcl module template: automatically unload automatically loaded modules (#32853)Sergey Kosukhin1-1/+1
Remove `module-info mode load` condition that prevents auto-unloading when autoloading is enabled. It looks like this condition was added to work around an issue in environment-modules that is no longer necessary. Add quotes to make is-loaded happy
2022-10-06docs: fix deprecated use of install_tree (#33004)Harmen Stoppels1-3/+3
2022-10-06cuda: add v11.8.0 (#33027)Andrew W Elble1-1/+10
Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
2022-10-05Optimize concurrent misc_cache provider index rebuild (#32874)Jordan Galby1-2/+10
When concurrent misc_cache provider index rebuilds happen, try to rebuild it only once, so we don't exceed misc_cache lock timeout. For example, when using `spack env depfile`, with no previous misc_cache, running `make -f depfile -j8` could run at most 8 concurrent `spack install` locking on misc_cache to rebuild the provider index. If one rebuild takes 30s, before this fix, the "worst" lock could wait up to 30s * 7, easily exceeding misc_cache lock timeout. Now, the "worst" lock would take 30s * 1 + ~1s * 6.
2022-10-05autotools: Filter libtools when building with dpcpp (#32876)Chris MacMackin1-1/+9
Due to a [known issue](https://community.intel.com/t5/Intel-oneAPI-Data-Parallel-C/dpcpp-and-GNU-Autotools/m-p/1296985) with dpcpp, autotool-based builds that try to use it will fail because they try to link against temporary files that no longer exist. This commit filters those files out of the libtools script so that linking can work properly.
2022-10-04BuildEnvironment: accumulate module changes to poke to all relevant modules ↵Greg Becker2-1/+26
(#32340) Currently, module changes from `setup_dependent_package` are applied only to the module of the package class, but not to any parent classes' modules between the package class module and `spack.package_base`. In this PR, we create a custom class to accumulate module changes, and apply those changes to each class that requires it. This design allows us to code for a single module, while applying the changes to multiple modules as needed under the hood, without requiring the user to reason about package inheritance.
2022-10-04`find`/`list`: display package counts last (#32946)Todd Gamblin5-6/+53
* find/list: display package counts last We have over 6,600 packages now, and `spack list` still displays the number of packages before it lists them all. This is useless for large sets of results (e.g., with no args) as the number has scrolled way off the screen before you can see it. The same is true for `spack find` with large installations. This PR changes `spack find` and `spack list` so that they display the package count last. * add some quick testing Co-authored-by: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
2022-10-04Fix typo in documentation (#32984)Adam J. Stewart1-1/+1
2022-10-02Fixup errors introduced by Clingo Pr: (#32905)John W. Parent1-1/+1
* re2c depends on cmake on Windows * Winbison properly added to bootstrap package search list
2022-09-30develop: canonicalize dev paths and base relative paths on `env.path` (#30075)Greg Becker6-12/+60
Allow environment variables and spack-specific path substitution variables (e.g. `$spack`) to be used in the paths associated with develop specs, while maintaining the ability to keep those paths relative to the environment rather than the working directory.
2022-09-29Install: Add use-buildcache option to install (#32537)kwryankrattiger3-8/+166
Install: Add use-buildcache option to install * Allow differentiating between top level packages and dependencies when determining whether to install from the cache or not. * Add unit test for --use-buildcache * Use metavar to display use-buildcache options. * Update spack-completion
2022-09-28Windows: Support for Clingo and dependencies (#30690)John W. Parent4-5/+14
Make it possible to install the Clingo package on Windows; this also provides a means to use Clingo with Spack on Windows. This includes * A new "winbison" package: Windows has a port of bison and flex where the two packages are grouped together. Clingo dependencies have been updated to use winbison on Windows and bison elsewhere (this avoids complicating the existin bison/flex packages until we can add support for implied virtuals). * The CMake build system was incorrectly converting CMAKE_INSTALL_PREFIX to POSIX format. * The re2c package has been modified to use CMake on Windows; for now this is done by overloading the configure/build/install methods to perform CMake-appropriate operations; the package should be refactored once support for multiple build systems in one Package is available.
2022-09-28Remove mentions of "best-effort" matrix expansion in the docs (#32755)Massimiliano Culpo1-25/+0
closes #20340
2022-09-27shell.py: fix undefined variable in csh deactivate (#32816)Alex Hornburg1-2/+2
This commit fixes #27027. The root cause of the issue is that the `SPACK_OLD_PROMPT` variable was evaluated in string interpolation regardless of whether the guard condition above evaluates to true or false. This commit uses the `eval` keyword to defer evaluation until the command is executed. Co-authored-by: Alexander Hornburg <alexande@xilinx.com>
2022-09-26improve lexing error (#32832)Chris White1-1/+3
2022-09-26Fetching/decompressing: use magic numbers (#31589)John W. Parent9-137/+524
Spack currently depends on parsing filenames of downloaded files to determine what type of archive they are and how to decompress them. This commit adds a preliminary check based on magic numbers to determine archive type (but falls back on name parsing if the extension type cannot be determined). As part of this work, this commit also enables decompression of .tar.xz-compressed archives on Windows.
2022-09-23Fix two docstring typos (#32751)Tamara Dahlgren1-2/+2