summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2022-10-29Deprecate spack bootstrap trust/untrust (#33600)Massimiliano Culpo4-57/+90
* Deprecate spack bootstrap trust/untrust * Update CI * Update tests
2022-10-27CachedCMakePackage: Add back initconfig as a defined phase (#33575)Chris White3-4/+7
Also: add type annotation to indicate that "phases" is always a tuple of strings.
2022-10-27LuaPackage: add missing attribute (#33551)Massimiliano Culpo1-0/+3
fixes #33544
2022-10-26Windows: fix bootstrap and package install failure (#32942)百地 希留耶2-1/+10
* Add patches for building clingo with MSVC * Help python find clingo DLL * If an executable is located in "C:\Program Files", Executable was running into issues with the extra space. This quotes the exe to ensure that it is treated as a single value. Signed-off-by: Kiruya Momochi <65301509+KiruyaMomochi@users.noreply.github.com>
2022-10-26Allow for packages with multiple build-systems (#30738)Massimiliano Culpo80-2323/+6226
This commit extends the DSL that can be used in packages to allow declaring that a package uses different build-systems under different conditions. It requires each spec to have a `build_system` single valued variant. The variant can be used in many context to query, manipulate or select the build system associated with a concrete spec. The knowledge to build a package has been moved out of the PackageBase hierarchy, into a new Builder hierarchy. Customization of the default behavior for a given builder can be obtained by coding a new derived builder in package.py. The "run_after" and "run_before" decorators are now applied to methods on the builder. They can also incorporate a "when=" argument to specify that a method is run only when certain conditions apply. For packages that do not define their own builder, forwarding logic is added between the builder and package (methods not found in one will be retrieved from the other); this PR is expected to be fully backwards compatible with unmodified packages that use a single build system.
2022-10-26fix use of non-existing kwarg (#33520)Harmen Stoppels1-1/+1
2022-10-26Relocation regex single pass (#33496)Harmen Stoppels3-21/+36
Instead of looping over multiple regexes and the entire text file contents, create a giant regex with all literal prefixes and do a single pass over files to detect prefixes. Not only is a single pass faster, it's also likely that the regex is compiled better, given that most prefixes share a common ... prefix.
2022-10-26gitlab ci: install binary deps faster (#33248)Harmen Stoppels3-40/+95
* Fast Gitlab CI job setup, and better legibility * Use a non-broken, recent GNU Make
2022-10-25dfs traversal: simplify edges in reverse mode (#33481)Harmen Stoppels1-27/+26
In the dfs code, flip edges so that `parent` means `from` and `spec` means `to` in the direction of traversal. This makes it slightly easier to write generic/composable code. For example when using visitors where one visitor reverses direction, and another only cares about accepting particular edges or not depending on whether the target node is seen before, it would be good if the second visitor didn't have to know whether the order was changed or not.
2022-10-25binary_distribution: compress level 9 -> 6 (#33513)Harmen Stoppels1-1/+5
Use the same compression level as `gzip` (6) instead of what Python uses (9). The LLVM tarball takes 4m instead of 12m to create, and is <1% larger. That's not worth the wait...
2022-10-25feature: Add -x|--explicit option to 'spack test run' (#32910)Tamara Dahlgren2-6/+20
2022-10-25BinaryCacheIndex: track update failures with cooldown (#33509)Jonathon Anderson1-6/+12
#32137 added an option to update() a BinaryCacheIndex with a cooldown: repeated attempts within this cooldown would not actually retry. However, the cooldown was not properly tracked for failures (which is common when the mirror does not store any binaries and therefore has no index.json). This commit ensures that update(..., with_cooldown=True) will also skip the update even if a failure has occurred within the cooldown period.
2022-10-25spec.py: prefer transitive link and direct build/run/test deps (#33498)Harmen Stoppels2-2/+48
Due to reuse concretization, we may generate DAGs with two occurrences of the same package corresponding to distinct specs. This happens when build dependencies are reused, since their dependencies are ignored in concretization. This caused a regression, for example: `spec['openssl']` would take the 'openssl' of the build dep `cmake`, instead of the direct `openssl` dependency, simply because the edge to `cmake` was traversed first and we do depth first traversal. One solution that was discussed is to limit `spec[name]` to just direct deps, or direct deps + transitive link deps, but this is too breaking. Instead, this PR simply prioritizes transitive link and direct build/run/test deps, and then falls back to a full DAG traversal. So, it's just about order of iteration.
2022-10-25Remove recursive symbolic link in lib/spack/docs from git repository (#33483)Massimiliano Culpo2-1/+3
Delete code removing the symlink during CI
2022-10-25Improve legibility of Gitlab CI (#33482)Harmen Stoppels2-4/+4
Use --backtrace in ci instead of --debug to reduce verbosity and don't show log on error, since log is already printed
2022-10-24Add filename to text_to_relocate only if it needs to be relocated (#31074)iarspider4-8/+54
Scan the text files for relocatable prefixes *before* creating a tarball, to reduce the amount of work to be done during install from binary cache. Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2022-10-24redact line numbers from grouped exception message (#33485)Danny McClanahan1-4/+8
2022-10-24locks: improved errors (#33477)Harmen Stoppels6-12/+59
Instead of showing ``` ==> Error: Timed out waiting for a write lock. ``` show ``` ==> Error: Timed out waiting for a write lock after 1.200ms and 4 attempts on file: /some/file ``` s.t. we actually get to see where acquiring a lock failed even when not running in debug mode. And use pretty time units everywhere, so we don't get 1.45e-9 seconds but 1.450ns etc.
2022-10-23backtraces with --backtrace (#33478)Harmen Stoppels1-1/+9
* backtraces without --debug Currently `--debug` is too verbose and not-`--debug` gives to little context about where exceptions are coming from. So, instead, it'd be nice to have `spack --backtrace` and `SPACK_BACKTRACE=1` as methods to get something inbetween: no verbose debug messages, but always a full backtrace. This is useful for CI, where we don't want to drown in debug messages when installing deps, but we do want to get details where something goes wrong if it goes wrong. * completion
2022-10-21Relocation should take hardlinks into account (#33460)Harmen Stoppels7-62/+190
Currently `relocate_text` and `relocate_text_bin` are unsafe in the sense that they run in parallel, and lead to races when modifying different items pointing to the same inode. This leads to the issue observed in #33453. This PR: 1. Renames those functions to `unsafe_*` so people are aware 2. Adds logic to deal with hardlinks in current binary packages 3. Adds logic to deal with hardlinks when creating new binary tarballs, so the install side doesn't have to de-dupe hardlinks. 4. Adds a test for 3 The assumption is that all our relocation logic preserves inodes. That is, we should never copy a file, modify it, and then move it back. I quickly verified, and its seems like this is true for (binary) text relocation, as well as rpath patching in patchelf (even when the file grows) and mach-o fixes.
2022-10-21Remove "spack buildcache copy" in v0.19.0 (#33437)Massimiliano Culpo1-86/+0
2022-10-20gitlab: Do not use root_spec['pkg_name'] anymore (#33403)Scott Wittenburg5-147/+27
* gitlab: Do not use root_spec['pkg_name'] anymore For a long time it was fine to index a concrete root spec with the name of a dependency in order to access the concrete dependency spec. Since pipelines started using `--use-buildcache dependencies:only,package:never` though, it has exposed a scheduling issue in how pipelines are generated. If a concrete root spec depends on two different hashes of `openssl` for example, indexing that root with just the package name is ambiguous, so we should no longer depend on that approach when scheduling jobs. * env: make sure exactly one spec in env matches hash
2022-10-19depfile: buildcache support (#33315)Harmen Stoppels2-35/+206
When installing some/all specs from a buildcache, build edges are pruned from those specs. This can result in a much smaller effective DAG. Until now, `spack env depfile` would always generate a full DAG. Ths PR adds the `spack env depfile --use-buildcache` flag that was introduced for `spack install` before. This way, not only can we drop build edges, but also we can automatically set the right buildcache related flags on the specific specs that are gonna get installed. This way we get parallel installs of binary deps without redundancy, which is useful for Gitlab CI.
2022-10-19Add a command to bootstrap Spack right now (#33407)Massimiliano Culpo1-0/+12
2022-10-19Relocate "run" type dependencies too (#33191)Stephen Sachs1-2/+2
When downloading from binary cache not only replace RPATHs to dependencies, but also text references to dependencies. Example: `autoconf@2.69` contains a text reference to the executable of its dependency `perl`: ``` $ grep perl-5 /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/autoconf-2.69-q3lo/bin/autoreconf eval 'case $# in 0) exec /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/perl-5.34.1-yphg/bin/perl -S "$0";; *) exec /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/perl-5.34.1-yphg/bin/perl -S "$0" "$@";; esac' ``` These references need to be replace or any package using `autoreconf` will fail as it cannot find the installed `perl`. Co-authored-by: Stephen Sachs <stesachs@amazon.com>
2022-10-19Update the binary index before attempting direct fetches (#32137)Jonathon Anderson3-19/+44
"spack install" will not update the binary index if given a concrete spec, which causes it to fall back to direct fetches when a simple index update would have helped. For S3 buckets in particular, this significantly and needlessly slows down the install process. This commit alters the logic so that the binary index is updated whenever a by-hash lookup fails. The lookup is attempted again with the updated index before falling back to direct fetches. To avoid updating too frequently (potentially once for each spec being installed), BinaryCacheIndex.update now includes a "cooldown" option, and when this option is enabled it will not update more than once in a cooldown window (set in config.yaml). Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
2022-10-18spack checksum: warn if version is deprecated (#32438)Tamara Dahlgren5-2/+42
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-10-18Docs: Spack info option updates (#33376)Tamara Dahlgren3-10/+27
2022-10-18Reusable --use-buildcache with better validation (#33388)Harmen Stoppels4-31/+76
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-10-17Initial contribution of LibPressio ecosystem (#32630)Robert Underwood1-0/+2
* Add libpressio and dependencies; some of these packages are maintained as forks of the original repositories and in those cases the docstring mentions this. * Add optional dependency in adios2 on libpressio * cub package: set CUB_DIR environment variable for dependent installations * Clear R_HOME/R_ENVIRON before Spack installation (avoid sources outside of Spack from affecting the installation in Spack) * Rename dlib to dorian3d-dlib and update dependents; add new dlib implementation. Pending an official policy on how to handle packages with short names, reviewer unilaterally decided that the rename was acceptable given that the new Spack dlib package is referenced more widely (by orders of magnitude) than the original Co-authored-by: Samuel Li <shaomeng@users.noreply.github.com>
2022-10-17database: don't warn adding missing build deps (#33361)Harmen Stoppels1-27/+48
When installing an individual spec `spack --only=package --cache-only /xyz` from a buildcache, Spack currently issues tons of warnings about missing build deps (and their deps) in the database. This PR disables these warnings, since it's fine to have a spec without its build deps in the db (they are just "missing").
2022-10-17installer.py: traverse_dependencies has local deptype (#33367)Harmen Stoppels1-9/+19
Currently `traverse_dependencies` fixes deptypes to traverse once and for all in the recursion, but this is incorrect, since deptypes depend on the node (e.g. if it's a dependency and cache-only, don't follow build type edges, even if the parent is build from sources and needs build deps.)
2022-10-17Support spackbot rebuilding all specs from source (#32596)Scott Wittenburg2-21/+66
Support spackbot rebuilding all specs from source when asked (with "rebuild everything") - Allow overriding --prune-dag cli opt with env var - Use job variable to optionally prevent rebuild jobs early exit behavior - ci rebuild: Use new install argument to insist deps are always installed from binary, but package is only installed from source. - gitlab: fix bug w/ untouched pruning - ci rebuild: install from hash rather than json file - When doing a "rebuild everything" pipeline, make sure that each install job only consumes binary dependencies from the mirror being populated by the current pipeline. This avoids using, e.g. binaries from develop, when rebuilding everything on a PR. - When running a pipeline to rebuild everything, do not die because we generated a hash on the broken specs list. Instead only warn in that case. - bugfix: Replace broken no-args tty.die() with sys.exit(1)
2022-10-17installer.py: fix/test get_deptypes (#33363)Harmen Stoppels2-1/+40
Fixing an oversight in https://github.com/spack/spack/pull/32537 `get_deptypes` should depend on new `package/dependencies_cache_only` props.
2022-10-17gitlab ci: Do not force protected build jobs to run on aws runners (#33314)Scott Wittenburg1-1/+1
2022-10-17installer.py: show timers for binary install (#33305)Harmen Stoppels1-23/+57
Print a message of the form ``` Fetch mm:ss. Build: mm:ss. Total: mm:ss ``` when installing from buildcache. Previously this only happened for source builds.
2022-10-15CI: allow multiple matches to combine tags (#32290)Jonathon Anderson3-13/+59
Currently "spack ci generate" chooses the first matching entry in gitlab-ci:mappings to fill attributes for a generated build-job, requiring that the entire configuration matrix is listed out explicitly. This unfortunately causes significant problems in environments with large configuration spaces, for example the environment in #31598 (spack.yaml) supports 5 operating systems, 3 architectures and 130 packages with explicit size requirements, resulting in 1300 lines of configuration YAML. This patch adds a configuraiton option to the gitlab-ci schema called "match_behavior"; when it is set to "merge", all matching entries are applied in order to the final build-job, allowing a few entries to cover an entire matrix of configurations. The default for "match_behavior" is "first", which behaves as before this commit (only the runner attributes of the first match are used). In addition, match entries may now include a "remove-attributes" configuration, which allows matches to remove tags that have been aggregated by prior matches. This only makes sense to use with "match_behavior:merge". You can combine "runner-attributes" with "remove-attributes" to effectively override prior tags.
2022-10-15depfile: update docs (#33279)Harmen Stoppels1-9/+37
2022-10-13meson: remove slash in path (#33292)Adam J. Stewart1-1/+1
2022-10-13gitlab ci: Print better information about broken specs (#33124)Scott Wittenburg3-49/+98
When a pipeline generation job is automatically failed because it generated jobs for specs known to be broken on develop, print better information about the broken specs that were encountered. Include at a minimum the hash and the url of the job whose failure caused it to be put on the broken specs list in the first place.
2022-10-13Add missing upperbound to docs/spack.yaml (#33280)Harmen Stoppels1-0/+2
2022-10-12env depfile: allow deps only install (#33245)Harmen Stoppels2-65/+32
* env depfile: allow deps only install - Refactor `spack env depfile` to use a Jinja template, making it a bit easier to follow as a human being. - Add a layer of indirection in the generated Makefile through an `<prefix>/.install-deps/<hash>` target, which allows one to specify different options when installing dependencies. For example, only verbose/debug mode on when installing some particular spec: ``` $ spack -e my_env env depfile -o Makefile --make-target-prefix example $ make example/.install-deps/<hash> -j16 $ make example/.install/<hash> SPACK="spack -d" SPACK_INSTALL_FLAGS=--verbose -j16 ``` This could be used to speed up `spack ci rebuild`: - Parallel install of dependencies from buildcache - Better readability of logs, e.g. reducing verbosity when installing dependencies, and splitting logs into deps.log and current_spec.log * Silence please!
2022-10-12Fix typo in docs (#33182)Massimiliano Culpo1-1/+1
2022-10-11Specify GCC prefix in LLVM-based compilers (#33146)Harmen Stoppels2-0/+29
* spack.compiler.Compiler: introduce prefix property We currently don't really have something that gives the GCC install path, which is used by many LLVM-based compilers (llvm, llvm-amdgpu, nvhpc, ...) to fix the GCC toolchain once and for all. This `prefix` property is dynamic in the sense that it queries the compiler itself. This is necessary because it's not easy to deduce the install path from the `cc` property (might be a symlink, might be a filename like `gcc` which works by having the compiler load a module that sets the PATH variable, might be a generic compiler wrapper based on environment variables like on cray...). With this property introduced, we can clean up some recipes that have the logic repeated for GCC. * intel-oneapi-compilers: set --gcc-sysroot to %gcc prefix
2022-10-11Refactor a few classes related to package repositories (#32273)Massimiliano Culpo27-669/+466
Caches used by repositories don't reference the global spack.repo.path instance anymore, but get the repository they refer to during initialization. Spec.virtual now use the index, and computation done to compute the index use Repository.is_virtual_safe. Code to construct mock packages and mock repository has been factored into a unique MockRepositoryBuilder that is used throughout the codebase. Add debug print for pushing and popping config scopes. Changed spack.repo.use_repositories so that it can override or not previous repos spack.repo.use_repositories updates spack.config.config according to the modifications done Removed a peculiar behavior from spack.config.Configuration where push would always bubble-up a scope named command_line if it existed
2022-10-11`spack diff` any specs you want (#32737)Peter Scheibel1-4/+6
Resolves #31782 With this change, if a spec is concrete after parsing (e.g. spec.yaml or /hash-based), then it is not disambiguated (a process which requires (a) that the spec be installed and (b) that it be part of the currently-active environment). This commit allows you to: * Diff specs from an environment regardless of whether they have been installed (more useful for projection/matrix-based envs) * Diff specs read from .yaml files which may or may not be entirely different installations of Spack Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-10-10Docs: Getting Started Dependencies (#32480)Axel Huebl4-2/+33
* Docs: Getting Started Dependencies Finally document what one needs to install to use Spack on Linux and Mac :-) With <3 for minimal container users and my colleagues with their fancy Macs. * Debian Update Packages: GCC, Python - build-essential: includes gcc, g++ (thx Cory) - Python: add python3-venv, python3-distutils (thx Pradyun) * Add RHEL8 Dependencies
2022-10-10shell prompt: enclose control sequence in brackets (#33079)Brian Vanderwende2-4/+10
When setting `PS1` in Bash, it's required to enclose non-printable characters in square brackets, so that the width of the terminal is handled correctly. See https://www.gnu.org/software/bash/manual/bash.html#Controlling-the-Prompt
2022-10-10PythonPackage: fix libs/headers attributes (#32970)Adam J. Stewart1-3/+3
2022-10-07autotools: extend patching of the libtool script (#30768)Sergey Kosukhin4-34/+208
* filter_file: introduce argument 'start_at' * autotools: extend patching of the libtool script * autotools: refactor _patch_usr_bin_file * autotools: improve readability of the filtering * autotools: keep the modification time of the configure scripts * autotools: do not try to patch directories * autotools: explain libtool patching for posterity Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com> Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>