summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2022-04-01"spack external find": also find library-only packages (#28005)Brian Van Essen6-13/+215
Update "spack external find --all" to also find library-only packages. A Package can add a ".libraries" attribute, which is a list of regular expressions to use to find libraries associated with the Package. "spack external find --all" will search LD_LIBRARY_PATH for potential libraries. This PR adds examples for NCCL, RCCL, and hipblas packages. These examples specify the suffix ".so" for the regular expressions used to find libraries, so generally are only useful for detecting library packages on Linux.
2022-04-01Allow "spack install foo@git-hash" without terminal prompt (#29827)Peter Scheibel1-1/+2
Do not prompt user with checksum warning when using git commit hashes as versions. Spack was incorrectly reporting this as a potential problem: it would display a prompt asking the user whether they want to proceed if Spack was running in a terminal, or it would terminate the running instance of Spack if running as part of a script.
2022-03-31Windows Support: Fixup Perl build (#29711)John W. Parent1-12/+4
* Add pl2bat to PATH: Windows on Perl requires the script pl2bat.bat and Perl to be available to the installer via the PATH. The build and dependent environments of Perl on Windows have the install prefix bin added to the PATH. * symlink with win32file module instead of using Executable to call mklink (mklink is a shell function and so is not accessible in this manner).
2022-03-31Allow use of "git:"-based URLs (#29765)Ivo Steinbrecher1-1/+1
2022-03-30spack ci: filter untouched pkgs from PR pipelines (#29697)Scott Wittenburg6-98/+290
We've previously generated CI pipelines for PRs, and they rebuild any packages that don't have a binary in an existing build cache. The assumption we were making was that ALL prior merged builds would be in cache, but due to the way we do security in the pipeline, they aren't. `develop` pipelines can take a while to catch up with the latest PRs, and while it does that, there may be a bunch of redundant builds on PRs that duplicate things being rebuilt on `develop`. Until we can do better caching of PR builds, we'll have this problem. We can do better in PRs, though, by *only* rebuilding things in the CI environment that are actually touched by the PR. This change computes exactly what packages are changed by a PR branch and *only* includes those packages' dependents and dependencies in the generated pipeline. Other as-yet unbuilt packages are pruned from CI for the PR. For `develop` pipelines, we still want to build everything to ensure that the stack works, and to ensure that `develop` catches up with PRs. This is especially true since we do not do rebuilds for *every* commit on `develop` -- just the most recent one after each `develop` pipeline finishes. Since we skip around, we may end up missing builds unless we ensure that we rebuild everything. We differentiate between `develop` and PR pipelines in `.gitlab-ci.yml` by setting `SPACK_PRUNE_UNTOUCHED` for PRs. `develop` will still have the old behavior. - [x] Add `SPACK_PRUNE_UNTOUCHED` variable to `spack ci` - [x] Refactor `spack pkg` command by moving historical package checking logic to `spack.repo` - [x] Implement pruning logic in `spack ci` to remove untouched packages - [x] add tests
2022-03-30Mixed compilers: drop warning message to debug level (#29748)Adam J. Stewart1-2/+1
2022-03-30Add completion of locals to spack python (#29702)Jörg Behrmann1-0/+4
2022-03-30patch cache: fix bug finding inherited packages (#29574)Greg Becker2-2/+25
2022-03-29cmake: use CMAKE_INSTALL_RPATH_USE_LINK_PATH (#29703)Harmen Stoppels2-31/+80
* cmake: use CMAKE_INSTALL_RPATH_USE_LINK_PATH Spack has a heuristic to add rpaths for packages it knows are required, but it's really a heuristic, and it does not work when the dependencies put their libraries in a different folder than `<prefix>/lib{64,}`. CMake patches binaries after install with the "install rpaths", which by default are provided by Spack and its heuristic through `CMAKE_INSTALL_RPATH`. CMake however knows better what libraries are effectively being linked to, and has an option to include those in the install rpath too, through `CMAKE_INSTALL_RPATH_USE_LINK_PATH`. These two CMake options are complementary, repeated rpaths seem to be filtered, and the "use link path" paths are appended to Spack's heuristic "install rpath". So, it seems like a good idea to enable "use link path" by default, so that: - `dlopen` by library name uses Spack's heuristic search paths - linked libraries in non-standard locations within a prefix get an rpath thanks to CMake. * docs
2022-03-28spack info: make sections optional; add build/stand-alone test information ↵Tamara Dahlgren3-48/+190
(#22097) Add output of build- and install-time tests to info command Enable dependencies, variants, and versions by default (i.e., provide --no* options; add gcc to test_info_fields to increase coverage for c_names->v_names
2022-03-28Add ssh to filter of valid urls (#29749)John W. Parent1-1/+1
2022-03-28Removed unneeded conflicts in CudaPackage and RocmPackage (#29699)Massimiliano Culpo2-8/+0
Since the variant is conditional there's no need to add conflicts too.
2022-03-27autotools doc: add missing right quote (#29734)百地 希留耶1-1/+1
2022-03-25Use the appropriate function to remove files in the stage directory (#29690)Massimiliano Culpo3-4/+29
We shouldn't be using "remove_linked_tree" to remove the lock file, since that function expects to receive a directory path as an argument. Also, as a further measure to avoid regression, this commit restores the "ignore_errors=True" argument on linux and adds a unit test checking that "remove_linked_tree" doesn't change file permissions as a side effect of a failure to remove.
2022-03-24environment.py: ensure view dir does not exist (#29641)Harmen Stoppels4-0/+101
2022-03-24add_files_to_view: flip incorrect default (#29700)Harmen Stoppels3-3/+3
2022-03-24environment.py: concrete root specs to view (#29634)Harmen Stoppels1-5/+6
Slight simplification of the code: just pass the list of concrete root specs to the view
2022-03-24fix --reuse w/ upstreams in env (#29680)Harmen Stoppels1-3/+5
2022-03-24environment views: single pass view generation (#29443)Harmen Stoppels10-24/+885
Reduces the number of stat calls to a bare minimum: - Single pass over src prefixes - Handle projection clashes in memory Symlinked directories in the src prefixes are now conditionally transformed into directories with symlinks in the dst dir. Notably `intel-mkl`, `cuda` and `qt` has top-level symlinked directories that previously resulted in empty directories in the view. We now avoid cycles and possible exponential blowup by only expanding symlinks that: - point to dirs deeper in the folder structure; - are a fixed depth of 2.
2022-03-24environment.py: clean up broken views on failure (#29643)Harmen Stoppels2-13/+46
When view creation fails for some reason, remove it, so that the next time around it can start from scratch.
2022-03-23environment: be more defensive when deleting roots for old views (#29636)Harmen Stoppels2-2/+19
Currently `old_root` is computed by reading the symlink at `self.root`. We should be more defensive in removing it by checking that it is in the same directory as the new root. Otherwise, in the worst case, when someone runs `spack env create --with-view=./view -d .` and `view` already exists and is a symlink to `/`, Spack effectively runs `rm -rf /`.
2022-03-23python: drop dependency on `file` for script check (#29513)Harmen Stoppels3-2/+49
`file` was used to detect Python scripts with shebangs, so that the interpreter could be changed from <python prefix> to <view path>. With this change, we detect shebangs using Python instead, so that `file` is no longer required.
2022-03-23Use stable URLs and `?full_index=1` for all github patches (#29239)Adam J. Stewart2-33/+48
The number of commit characters in patch files fetched from GitHub can change, so we should use `full_index=1` to enforce full commit hashes (and a stable patch `sha256`). Similarly, URLs for branches like `master` don't give us stable patch files, because branches are moving targets. Use specific tags or commits for those. - [x] update all github patch URLs to use `full_index=1` - [x] don't use `master` or other branches for patches - [x] add an audit check and a test for `?full_index=1` Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2022-03-23Remove known issues from documentation (#29664)Massimiliano Culpo2-78/+0
Known issues reports only 2 issues, among the bugs reported on GitHub. One of the two is also outdated, since the issue has been solved with the new concretizer. Thus, this commit removes the section.
2022-03-22sbang.py: single lstat (#29670)Harmen Stoppels1-11/+9
2022-03-21config add: fix parsing of validator error to infer type from oneOf (#29475)Greg Becker2-4/+10
2022-03-20major.minor.micro.dev0 Spack version (#25267)Harmen Stoppels4-102/+91
When you install Spack from a tarball, it will always show an exact version for Spack itself, even when you don't download a tagged commit: ``` $ wget -q https://github.com/spack/spack/archive/refs/heads/develop.tar.gz $ tar -xf develop.tar.gz $ ./spack-develop/bin/spack --version 0.16.2 ``` This PR sets the Spack version to `0.18.0.dev0` on develop, following [PEP440](https://github.com/spack/spack/pull/25267#issuecomment-896340234) as suggested by Adam Stewart. ``` spack (fix/set-dev-version)$ spack --version 0.18.0.dev0 (git 0.17.1-1526-e270464ae0) spack (fix/set-dev-version)$ mv .git .git_ spack $ spack --version 0.18.0.dev0 ``` - [x] Update the release guide - [x] Add __version__ to spack's __init__.py - [x] Use PEP 440 canonical version strings - [x] Make spack --version output [actual version] (git version) Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2022-03-20Enable GCS urls as valid (#29592)Doug Jacobsen3-1/+13
* Add tests to ensure google cloud storage urls work as mirrors This commit adds two tests to track that GCS buckets can work as mirrors, and can be parsed as valid URLs. Currently, gs:// format URLs are not correctly parsed. * Fix URL parsing for GCS buckets This commit adds GCS bucket URLs as valid URLs.
2022-03-19We were doing a check in a metaclass to ensure that Package classes weren't ↵Peter Scheibel1-1/+1
defining a function themselves; this check failed (erroneously) if that Package class subclassed another package class (because the check examined all superclasses and thought the definition we automatically add was an offender) (#29569)
2022-03-19Revert "Add command for reading a json-based DB description (#24894)" (#29603)Nils Vu10-665/+39
This reverts commit 531b1c5c3dcc9bc7bec27e223608aed477e94dbd.
2022-03-18allow ci reproduce to work when spack is cloned to a directory not named ↵Tom Scogland1-1/+1
spack (#29518)
2022-03-18R versions (#29258)Tom Scogland5-43/+129
* lower priority of package-provided urls This change favors urls found in a scraped page over those provided by the package from `url_for_version`. In most cases this doesn't matter, but R specifically returns known bad URLs in some cases, and the fallback path for a failed fetch uses `fetch_remote_versions` to find a substitute. This fixes that problem. fixes #29204 * consider what links actually exist in all cases Checksum was only actually scraping when called with no versions. It now always scrapes and then selects URLs from the set of URLs known to exist whenever possible. fixes #25831 * bow to the wrath of flake8 * test-fetch urls from package, prefer if successful * Update lib/spack/spack/package.py Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov> * reword as suggested * re-enable mypy specific ignore and ignore pyflakes * remove flake8 ignore from .flake8 * address review comments * address comments * add sneaky missing substitute I missed this one because we call substitute on a URL that doesn't contain a version component. I'm not sure how that's supposed to work, but apparently it's required by at least one mock package, so back in it goes. Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
2022-03-18Add command for reading a json-based DB description (#24894)Peter Scheibel10-39/+665
Adds `spack external read-cray-manifest`, which reads a json file that describes a set of package DAGs. The parsed results are stored directly in the database. A user can see these installed specs with `spack find` (like any installed spec). The easiest way to use them right now as dependencies is to run `spack spec ... ^/hash-of-external-package`. Changes include: * `spack external read-cray-manifest --file <path/to/file>` will add all specs described in the file to Spack's installation DB and will also install described compilers to the compilers configuration (the expected format of the file is described in this PR as well including examples of the file) * Database records now may include an "origin" (the command added in this PR registers the origin as "external-db"). In the future, it is assumed users may want to be able to treat installs registered with this command differently (e.g. they may want to uninstall all specs added with this command) * Hash properties are now always preserved when copying specs if the source spec is concrete * I don't think the hashes of installed-and-concrete specs should change and this was the easiest way to handle that * also specs that are concrete preserve their `.normal` property when copied (external specs may mention compilers that are not registered, and without this change they would fail in `normalize` when calling `validate_or_raise`) * it might be this should only be the case if the spec was installed - [x] Improve testing - [x] Specifically mark DB records added with this command (so that users can do something like "uninstall all packages added with `spack read-external-db`) * This is now possible with `spack uninstall --all --origin=external-db` (this will remove all specs added from manifest files) - [x] Strip variants that are listed in json entries but don't actually exist for the package Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2022-03-18environment: Don't print 'updating view at' when no specs (#29536)Harmen Stoppels1-1/+2
2022-03-18Split the workflow section and remove outdated advices (#29344)Massimiliano Culpo7-1248/+287
This PR removes a few outdated sections from the "Basics" part of the documentation. It also makes a few topic under the environment section more prominent by removing an unneeded spack.yaml subsection and promoting everything under it.
2022-03-17Add deprecation warnings for activate/deactivate/view (#29430)Harmen Stoppels2-0/+8
2022-03-17Remove references to `features/windows-support` branch (#29565)Zack Galbreath1-1/+1
2022-03-17Windows Support: Testing Suite integrationJohn Parent91-995/+725
Broaden support for execution of the test suite on Windows. General bug and review fixups
2022-03-17Failed _write now track that the DB is inconsistentPeter Josef Scheibel1-0/+15
Consistency is restored on next transaction
2022-03-17"spack commands --update-completion"John Parent96-923/+558
2022-03-17Path handling (#28402)John W. Parent36-232/+493
Consolidate Spack's internal filepath logic to a select few places and refactor to consistent internal useage of os.path utilities. Creates a prefix, and a series of utilities in the path utility module that facilitate handling paths in a platform agnostic manner. Convert Windows paths to posix paths internally Prefer posixpath.join instead of os.path.join Updated util/ directory to account for Windows integration Co-authored-by: Stephen Crowell <stephen.crowell@khq.kitware.com> Co-authored-by: John Parent <john.parent@kitware.com> Module template format for windows (#23041)
2022-03-17Expand external find for Windows (#27588)John Parent50-423/+504
* Incorporate new search location * Add external user option * proper doc string * Explicit commands in getting started * raise during chgrp on Win recover installer changes Notate admin privleges Windows phase install hooks Find external python and install ninja (#23496) Allow external find python to find windows python and spack install ninja Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> Co-authored-by: Betsy McPhail <betsy.mcphail@kitware.com>
2022-03-17Update tests support for WindowsBetsy McPhail124-193/+1918
Fixup common tests * Remove requirement for Python 2.6 * Skip new failing test Windows: Update url util to handle Windows paths (#27959) * update url util to handle windows paths * Update tests to handle fixed url handling * canonicalize path only when the path type matches the host platform * Skip some url tests on Windows Co-authored-by: Omar Padron <omar.padron@kitware.com> Use threading.TIMEOUT_MAX when available (#24246) This value was introduced in Python 3.2. Specifying a timeout greater than this value will raise an OverflowError. Co-authored-by: Lou Lawrence <lou.lawrence@kitware.com> Co-authored-by: John Parent <john.parent@kitware.com> Co-authored-by: Betsy McPhail <betsy.mcphail@kitware.com>
2022-03-17Add Windows to platform and target changesJohn Parent12-23/+47
Add compiler hint to the root spec for Windows Reporters on Windows (#26038) Reporters use Jinja2 as the templating engine, and Jinja2 indexes templates by Unix separators, even on Windows, so search using Unix paths on all systems. Support patching on win via git (#25871) Handle GRP on windows
2022-03-17Spack on Windows package portsJohn Parent15-58/+93
CMake - Windows Bootstrap (#25825) Remove hardcoded cmake compiler (#26410) Revert breaking cmake changes Ensure no autotools on Windows Perl on Windows (#26612) Python source build windows (#26313) Reconfigure sysconf for Windows Python2.6 compatibility Fxixup new sbang tests for windows Ruby support (#28287) Add NASM support (#28319) Add mock Ninja package for testing
2022-03-17Relocate spack_cmd and scripts from installer to root bin (#24651)John W. Parent6-374/+52
Relocate spack_cmd and scripts from installer to root bin Refactor documentation, installer, and launcher to facilitate that change
2022-03-17Use Python's zipfile, if available (#24556)Betsy McPhail1-3/+21
* Style fixes * Use Python's zipfile, if available The compression libs are optional in Python. Rely on python as a first attempt then fall back to `unzip`
2022-03-17Windows: VS and CMake supportBetsy McPhail3-23/+29
Fix: Building packages with CMake is broken (#24241) vsInstallPaths needs to be renamed vs_install_paths (#24297)
2022-03-17Packaging of netlib-lapack for windows (#24993)Jared Popelar7-103/+190
MSVC's internal CMake and Ninja now detected by spack external find and added to packages.yaml Saving progress on packaging zlib for Windows Fixing the shared CMake flag * Loading Intel's ifx Fortran compiler into MSVC; if there are multiple versions of MSVC installed and detected, ifx will only be placed into the first block written in compilers.yaml. The version number of ifx can be detected using MSVC's version flag (instead of /QV) by using ignore_version_errors. This commit also provides support for detection of Intel compilers in their own compiler block by adding ifx.exe to the fc/f77_name blocks inside intel.py * Giving CMake a Fortran compiler argument * Adding patch file for removing duplicated mangling header for versions 3.9.1 and older; static and shared now successfully building on Windows * Have netlib-lapack depend on ninja@1.10 Co-authored-by: John R. Cary <cary@txcorp.com> Co-authored-by: Jared Popelar <jpopelar@txcorp.com> Making a default config.yaml for Windows Small path length for build_stage Provide more prerequisite details, mention default config.yaml Killing an unnecessary setvars call Replacing some lost changes, proofreading, updating windows-supported package list Co-authored-by: John Parent <john.parent@kitware.com>
2022-03-17Windows: Create installer and environmentlou.lawrence@kitware.com18-11/+987
* Add 'make-installer' command for Windows * Add '--bat' arg to env activate, env deactivate and unload commands * An equivalent script to setup-env on linux: spack_cmd.bat. This script has a wrapper to evaluate cd, load/unload, env activate/deactivate.(#21734) * Add spacktivate and config editor (#22049) * spack_cmd: will find python and spack on its own. It preferentially tries to use python on your PATH (#22414) * Ignore Windows python installer if found (#23134) * Bundle git in windows installer (#23597) * Add Windows section to Getting Started document (#23131), (#23295), (#24240) Co-authored-by: Stephen Crowell <stephen.crowell@kitware.com> Co-authored-by: lou.lawrence@kitware.com <lou.lawrence@kitware.com> Co-authored-by: Betsy McPhail <betsy.mcphail@kitware.com> Co-authored-by: Jared Popelar <jpopelar@txcorp.com> Co-authored-by: Ben Cowan <benc@txcorp.com> Update Installer CI Co-authored-by: John Parent <john.parent@kitware.com>