summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2022-04-12Splice differing virtual packages (#27919)Nathan Hanford3-21/+86
Co-authored-by: Nathan Hanford <hanford1@llnl.gov>
2022-04-12Don't allow replacement of root develop specs with --reuse (#28605)psakievich3-4/+48
* Fix to concretize.lp do not allow dev specs to be reused Co-authored-by: Gregory Becker <becker33@llnl.gov>
2022-04-12Git commit versions bugfix: Environments and Concretization (#29717)Peter Scheibel7-32/+127
Spack added support in #24639 for ad-hoc Git-commit-hash-based versions: A user can install a package x@hash, where X is a package that stores its source code in a Git repository, and the hash refers to a commit in that repository which is not recorded as an explicit version in the package.py file for X. A couple issues were found relating to this: * If an environment defines an alternative package repo (i.e. with repos.yaml), and spack.yaml contains user Specs with ad-hoc Git-commit-hash-based versions for packages in that repo, then as part of retrieving the data needed for version comparisons it will attempt to retrieve the package before the environment's configuration is instantiated. * The bookkeeping information added to compare ad-hoc git versions was being stripped from Specs during concretization (such that user Specs which succeeded before concretizing would then fail after) This addresses the issues: * The first issue is resolved by deferring access to the associated Package until the versions are actually compared to one another. * The second issue is resolved by ensuring that the Git bookkeeping information is explicitly applied to Specs after they are concretized. This also: * Resolves an ambiguity in the mock_git_version_info fixture used to create a tree of Git commits and provide a list where each index maps to a known commit. * Isolates the cache used for Git repositories in tests using the mock_git_version_info fixture * Adds a TODO which points out that if the remote Git repository overwrites tags, that Spack will then fail when using ad-hoc Git-commit-hash-based versions
2022-04-11Update `gpg publish` to work with mirror arguments (#28740)Doug Jacobsen2-3/+50
This commit updates the `gpg publish` command to work with the mirror arguments, when trying to push keys to a mirror. - [x] update `gpg publish command - [x] add test for publishing GPG keys and rebuilding the key index within a mirror
2022-04-11cray platform: unload cray-mpich (#29898)Sergey Kosukhin1-3/+4
2022-04-11macos: fewer calls to sw_vers (#29997)Seth R. Johnson1-0/+5
In a typical call to spack, the OperatingSystem gets instantiated multiple times. For macOS, each one requires a call to `sw_vers`, which is done through the Executable helper class. Memoizing reduces the call count from "spac spec" from three to one.
2022-04-08Environments: unify the spec objects on first concretization (#29948)Massimiliano Culpo4-5/+39
Currently environments are indexed by build hashes. When looking into this bug I noticed there is a disconnect between environments that are concretized in memory for the first time and environments that are read from a `spack.lock`. The issue is that specs read from a `spack.lock` don't have a full hash, since they are indexed by a build hash which is strictly coarser. They are also marked "final" as they are read from a file, so we can't compute additional hashes. This bugfix PR makes "first concretization" equivalent to re-reading the specs from a corresponding `spack.lock`, and doing so unveiled a few tests were we were making wrong assumptions and relying on the fact that a `spack.lock` file was not there already. * Add unit test * Modify mpich to trigger jobs in pipelines * Fix two failing unit tests * Fix another full_hash vs. build_hash mismatch in tests
2022-04-08Deprecate top-level module config (#28659)Harmen Stoppels10-185/+100
* Ignore top-level module config; add auto-update In Spack 0.17 we got module sets (modules:[name]:[prop]), and for backwards compat modules:[prop] was short for modules:default:[prop]. But this makes it awkward to define default config for the "default" module set. Since 0.17 is branched off, we can now deprecate top-level module config (that is, just ignore it with a warning). This PR does that, and it implements `spack config update modules` to make upgrading easy (we should have added that to 0.17 already...) It also removes references to `dotkit` stuff which was already deprecated in 0.13 and could have been removed in 0.14. Prefix inspections are the only exception, since the top-level prefix inspections used for `spack load` and `spack env activate`.
2022-04-08concretize.lp: enforce target compatibility through DAG (#29694)Greg Becker2-8/+40
Spack currently allows dependencies to be concretized for an architecture incompatible with the root. This commit adds rules to make this situation impossible by design.
2022-04-07Use the non-deprecated `MetaPathFinder` interface (#29745)Massimiliano Culpo9-379/+294
* Extract the MetaPathFinder and Loaders for packages in their own classes https://peps.python.org/pep-0451/ Currently, RepoPath and Repo implement the (deprecated) interface of MetaPathFinder (find_module) and of Loader (load_module). This commit extracts both of them and places the code in their own classes. The MetaPathFinder interface is updated to contain both the deprecated "find_module" (for Python 2.7 support) and the recommended "find_spec". Update of the Loader interface is deferred at a subsequent commit. * Move the lines to be prepended inside "RepoLoader" Also adjust the naming of a few variables too * Remove spack.util.imp, since code is only used in spack.repo * Remove support from loading Python modules Python > 3 but < 3.5 * Remove `Repo._create_namespace` This function was interacting badly with the MetaPathFinder and causing issues with "normal" imports. Removing the function allows to do things like: ```python import spack.pkg.builtin.mpich cls = spack.pkg.builtin.mpich.Mpich ``` * Remove code needed to trigger the Singleton evaluation The finder is coded in a way to trigger the Singleton, so we don't need external code now that we register it at module level into `sys.meta_path`. * Add unit tests
2022-04-07ci: clean up env between tests with working_dir (#29807)Scott Wittenburg2-98/+70
2022-04-07ci: do not prune specs when stacks are touched (#29825)Scott Wittenburg1-9/+33
2022-04-07autotools.py: pic flags for %nvhpc (#29920)Harmen Stoppels1-5/+8
2022-04-06web.py: set User-Agent (#29919)Harmen Stoppels1-1/+5
Some servers require `User-Agent` to be set, and otherwise error with access denied. One such example is mpich. To fix this, set `User-Agent: Spackbot/[version]` as a header. Apparently by convention, it should include the word `bot`.
2022-04-06link_tree.py: format conflict error message (#29870)Harmen Stoppels1-2/+2
Show each `[src a] and [src b] both project to [dst]` on a separate line.
2022-04-05Bugfix: CVS fetching (#29793)Erik Schnetter1-2/+7
#27021 broke fetching for CVS-based packages because: - The mirror logic was using URL parsing to extract a path from the CVS repository location - #27021 added sanity checks to enforce that strings passed to the URL parser were actually URLs This replaces the call to "url_util.parse" with logic that is customized for CVS. This implies that VCSFetchStrategy should rename the "url_attr" attribute to something more generic, but that should be handled separately.
2022-04-04Allow conditional possible values in variants (#29530)Massimiliano Culpo6-6/+173
Allow declaring possible values for variants with an associated condition. If the variant takes one of those values, the condition is imposed as a further constraint. The idea of this PR is to implement part of the mechanisms needed for modeling [packages with multiple build-systems]( https://github.com/spack/seps/pull/3). After this PR the build-system directive can be implemented as: ```python variant( 'build-system', default='cmake', values=( 'autotools', conditional('cmake', when='@X.Y:') ), description='...', ) ``` Modifications: - [x] Allow conditional possible values in variants - [x] Add a unit-test for the feature - [x] Add documentation
2022-04-04rewiring of spliced specs (#26873)Nathan Hanford2-0/+270
* tests for rewiring pure specs to spliced specs * relocate text, binaries, and links * using llnl.util.symlink for windows compat. Note: This does not include CLI hooks for relocation. Co-authored-by: Nathan Hanford <hanford1@llnl.gov>
2022-04-04Add support for racket packages (#27564)Thomas Dickerson5-0/+155
- Add variants for various common build flags, including support for both versions of the Racket VM environment. - Prevent `-j` flags to `make`, which has been known to cause problems with Racket builds. - Prefer the minimal release to improve install times. Bells and whistles carry their own runtime dependencies and should be installed via `raco`. An enterprising user may even create a `RacketPackage` class to make spack aware of `raco` installed packages. - Match the official version numbering scheme.
2022-04-01"spack external find": also find library-only packages (#28005)Brian Van Essen6-13/+215
Update "spack external find --all" to also find library-only packages. A Package can add a ".libraries" attribute, which is a list of regular expressions to use to find libraries associated with the Package. "spack external find --all" will search LD_LIBRARY_PATH for potential libraries. This PR adds examples for NCCL, RCCL, and hipblas packages. These examples specify the suffix ".so" for the regular expressions used to find libraries, so generally are only useful for detecting library packages on Linux.
2022-04-01Allow "spack install foo@git-hash" without terminal prompt (#29827)Peter Scheibel1-1/+2
Do not prompt user with checksum warning when using git commit hashes as versions. Spack was incorrectly reporting this as a potential problem: it would display a prompt asking the user whether they want to proceed if Spack was running in a terminal, or it would terminate the running instance of Spack if running as part of a script.
2022-03-31Windows Support: Fixup Perl build (#29711)John W. Parent1-12/+4
* Add pl2bat to PATH: Windows on Perl requires the script pl2bat.bat and Perl to be available to the installer via the PATH. The build and dependent environments of Perl on Windows have the install prefix bin added to the PATH. * symlink with win32file module instead of using Executable to call mklink (mklink is a shell function and so is not accessible in this manner).
2022-03-31Allow use of "git:"-based URLs (#29765)Ivo Steinbrecher1-1/+1
2022-03-30spack ci: filter untouched pkgs from PR pipelines (#29697)Scott Wittenburg6-98/+290
We've previously generated CI pipelines for PRs, and they rebuild any packages that don't have a binary in an existing build cache. The assumption we were making was that ALL prior merged builds would be in cache, but due to the way we do security in the pipeline, they aren't. `develop` pipelines can take a while to catch up with the latest PRs, and while it does that, there may be a bunch of redundant builds on PRs that duplicate things being rebuilt on `develop`. Until we can do better caching of PR builds, we'll have this problem. We can do better in PRs, though, by *only* rebuilding things in the CI environment that are actually touched by the PR. This change computes exactly what packages are changed by a PR branch and *only* includes those packages' dependents and dependencies in the generated pipeline. Other as-yet unbuilt packages are pruned from CI for the PR. For `develop` pipelines, we still want to build everything to ensure that the stack works, and to ensure that `develop` catches up with PRs. This is especially true since we do not do rebuilds for *every* commit on `develop` -- just the most recent one after each `develop` pipeline finishes. Since we skip around, we may end up missing builds unless we ensure that we rebuild everything. We differentiate between `develop` and PR pipelines in `.gitlab-ci.yml` by setting `SPACK_PRUNE_UNTOUCHED` for PRs. `develop` will still have the old behavior. - [x] Add `SPACK_PRUNE_UNTOUCHED` variable to `spack ci` - [x] Refactor `spack pkg` command by moving historical package checking logic to `spack.repo` - [x] Implement pruning logic in `spack ci` to remove untouched packages - [x] add tests
2022-03-30Mixed compilers: drop warning message to debug level (#29748)Adam J. Stewart1-2/+1
2022-03-30Add completion of locals to spack python (#29702)Jörg Behrmann1-0/+4
2022-03-30patch cache: fix bug finding inherited packages (#29574)Greg Becker2-2/+25
2022-03-29cmake: use CMAKE_INSTALL_RPATH_USE_LINK_PATH (#29703)Harmen Stoppels2-31/+80
* cmake: use CMAKE_INSTALL_RPATH_USE_LINK_PATH Spack has a heuristic to add rpaths for packages it knows are required, but it's really a heuristic, and it does not work when the dependencies put their libraries in a different folder than `<prefix>/lib{64,}`. CMake patches binaries after install with the "install rpaths", which by default are provided by Spack and its heuristic through `CMAKE_INSTALL_RPATH`. CMake however knows better what libraries are effectively being linked to, and has an option to include those in the install rpath too, through `CMAKE_INSTALL_RPATH_USE_LINK_PATH`. These two CMake options are complementary, repeated rpaths seem to be filtered, and the "use link path" paths are appended to Spack's heuristic "install rpath". So, it seems like a good idea to enable "use link path" by default, so that: - `dlopen` by library name uses Spack's heuristic search paths - linked libraries in non-standard locations within a prefix get an rpath thanks to CMake. * docs
2022-03-28spack info: make sections optional; add build/stand-alone test information ↵Tamara Dahlgren3-48/+190
(#22097) Add output of build- and install-time tests to info command Enable dependencies, variants, and versions by default (i.e., provide --no* options; add gcc to test_info_fields to increase coverage for c_names->v_names
2022-03-28Add ssh to filter of valid urls (#29749)John W. Parent1-1/+1
2022-03-28Removed unneeded conflicts in CudaPackage and RocmPackage (#29699)Massimiliano Culpo2-8/+0
Since the variant is conditional there's no need to add conflicts too.
2022-03-27autotools doc: add missing right quote (#29734)百地 希留耶1-1/+1
2022-03-25Use the appropriate function to remove files in the stage directory (#29690)Massimiliano Culpo3-4/+29
We shouldn't be using "remove_linked_tree" to remove the lock file, since that function expects to receive a directory path as an argument. Also, as a further measure to avoid regression, this commit restores the "ignore_errors=True" argument on linux and adds a unit test checking that "remove_linked_tree" doesn't change file permissions as a side effect of a failure to remove.
2022-03-24environment.py: ensure view dir does not exist (#29641)Harmen Stoppels4-0/+101
2022-03-24add_files_to_view: flip incorrect default (#29700)Harmen Stoppels3-3/+3
2022-03-24environment.py: concrete root specs to view (#29634)Harmen Stoppels1-5/+6
Slight simplification of the code: just pass the list of concrete root specs to the view
2022-03-24fix --reuse w/ upstreams in env (#29680)Harmen Stoppels1-3/+5
2022-03-24environment views: single pass view generation (#29443)Harmen Stoppels10-24/+885
Reduces the number of stat calls to a bare minimum: - Single pass over src prefixes - Handle projection clashes in memory Symlinked directories in the src prefixes are now conditionally transformed into directories with symlinks in the dst dir. Notably `intel-mkl`, `cuda` and `qt` has top-level symlinked directories that previously resulted in empty directories in the view. We now avoid cycles and possible exponential blowup by only expanding symlinks that: - point to dirs deeper in the folder structure; - are a fixed depth of 2.
2022-03-24environment.py: clean up broken views on failure (#29643)Harmen Stoppels2-13/+46
When view creation fails for some reason, remove it, so that the next time around it can start from scratch.
2022-03-23environment: be more defensive when deleting roots for old views (#29636)Harmen Stoppels2-2/+19
Currently `old_root` is computed by reading the symlink at `self.root`. We should be more defensive in removing it by checking that it is in the same directory as the new root. Otherwise, in the worst case, when someone runs `spack env create --with-view=./view -d .` and `view` already exists and is a symlink to `/`, Spack effectively runs `rm -rf /`.
2022-03-23python: drop dependency on `file` for script check (#29513)Harmen Stoppels3-2/+49
`file` was used to detect Python scripts with shebangs, so that the interpreter could be changed from <python prefix> to <view path>. With this change, we detect shebangs using Python instead, so that `file` is no longer required.
2022-03-23Use stable URLs and `?full_index=1` for all github patches (#29239)Adam J. Stewart2-33/+48
The number of commit characters in patch files fetched from GitHub can change, so we should use `full_index=1` to enforce full commit hashes (and a stable patch `sha256`). Similarly, URLs for branches like `master` don't give us stable patch files, because branches are moving targets. Use specific tags or commits for those. - [x] update all github patch URLs to use `full_index=1` - [x] don't use `master` or other branches for patches - [x] add an audit check and a test for `?full_index=1` Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2022-03-23Remove known issues from documentation (#29664)Massimiliano Culpo2-78/+0
Known issues reports only 2 issues, among the bugs reported on GitHub. One of the two is also outdated, since the issue has been solved with the new concretizer. Thus, this commit removes the section.
2022-03-22sbang.py: single lstat (#29670)Harmen Stoppels1-11/+9
2022-03-21config add: fix parsing of validator error to infer type from oneOf (#29475)Greg Becker2-4/+10
2022-03-20major.minor.micro.dev0 Spack version (#25267)Harmen Stoppels4-102/+91
When you install Spack from a tarball, it will always show an exact version for Spack itself, even when you don't download a tagged commit: ``` $ wget -q https://github.com/spack/spack/archive/refs/heads/develop.tar.gz $ tar -xf develop.tar.gz $ ./spack-develop/bin/spack --version 0.16.2 ``` This PR sets the Spack version to `0.18.0.dev0` on develop, following [PEP440](https://github.com/spack/spack/pull/25267#issuecomment-896340234) as suggested by Adam Stewart. ``` spack (fix/set-dev-version)$ spack --version 0.18.0.dev0 (git 0.17.1-1526-e270464ae0) spack (fix/set-dev-version)$ mv .git .git_ spack $ spack --version 0.18.0.dev0 ``` - [x] Update the release guide - [x] Add __version__ to spack's __init__.py - [x] Use PEP 440 canonical version strings - [x] Make spack --version output [actual version] (git version) Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2022-03-20Enable GCS urls as valid (#29592)Doug Jacobsen3-1/+13
* Add tests to ensure google cloud storage urls work as mirrors This commit adds two tests to track that GCS buckets can work as mirrors, and can be parsed as valid URLs. Currently, gs:// format URLs are not correctly parsed. * Fix URL parsing for GCS buckets This commit adds GCS bucket URLs as valid URLs.
2022-03-19We were doing a check in a metaclass to ensure that Package classes weren't ↵Peter Scheibel1-1/+1
defining a function themselves; this check failed (erroneously) if that Package class subclassed another package class (because the check examined all superclasses and thought the definition we automatically add was an offender) (#29569)
2022-03-19Revert "Add command for reading a json-based DB description (#24894)" (#29603)Nils Vu10-665/+39
This reverts commit 531b1c5c3dcc9bc7bec27e223608aed477e94dbd.
2022-03-18allow ci reproduce to work when spack is cloned to a directory not named ↵Tom Scogland1-1/+1
spack (#29518)