summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2022-10-06Add a warning on Python 2.7 deprecation (#33052)Massimiliano Culpo2-0/+9
Co-authored-by: alalazo <alalazo@users.noreply.github.com> Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2022-10-06tcl module template: automatically unload automatically loaded modules (#32853)Sergey Kosukhin1-1/+1
Remove `module-info mode load` condition that prevents auto-unloading when autoloading is enabled. It looks like this condition was added to work around an issue in environment-modules that is no longer necessary. Add quotes to make is-loaded happy
2022-10-06docs: fix deprecated use of install_tree (#33004)Harmen Stoppels1-3/+3
2022-10-06cuda: add v11.8.0 (#33027)Andrew W Elble1-1/+10
Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
2022-10-05Optimize concurrent misc_cache provider index rebuild (#32874)Jordan Galby1-2/+10
When concurrent misc_cache provider index rebuilds happen, try to rebuild it only once, so we don't exceed misc_cache lock timeout. For example, when using `spack env depfile`, with no previous misc_cache, running `make -f depfile -j8` could run at most 8 concurrent `spack install` locking on misc_cache to rebuild the provider index. If one rebuild takes 30s, before this fix, the "worst" lock could wait up to 30s * 7, easily exceeding misc_cache lock timeout. Now, the "worst" lock would take 30s * 1 + ~1s * 6.
2022-10-05autotools: Filter libtools when building with dpcpp (#32876)Chris MacMackin1-1/+9
Due to a [known issue](https://community.intel.com/t5/Intel-oneAPI-Data-Parallel-C/dpcpp-and-GNU-Autotools/m-p/1296985) with dpcpp, autotool-based builds that try to use it will fail because they try to link against temporary files that no longer exist. This commit filters those files out of the libtools script so that linking can work properly.
2022-10-04BuildEnvironment: accumulate module changes to poke to all relevant modules ↵Greg Becker2-1/+26
(#32340) Currently, module changes from `setup_dependent_package` are applied only to the module of the package class, but not to any parent classes' modules between the package class module and `spack.package_base`. In this PR, we create a custom class to accumulate module changes, and apply those changes to each class that requires it. This design allows us to code for a single module, while applying the changes to multiple modules as needed under the hood, without requiring the user to reason about package inheritance.
2022-10-04`find`/`list`: display package counts last (#32946)Todd Gamblin5-6/+53
* find/list: display package counts last We have over 6,600 packages now, and `spack list` still displays the number of packages before it lists them all. This is useless for large sets of results (e.g., with no args) as the number has scrolled way off the screen before you can see it. The same is true for `spack find` with large installations. This PR changes `spack find` and `spack list` so that they display the package count last. * add some quick testing Co-authored-by: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
2022-10-04Fix typo in documentation (#32984)Adam J. Stewart1-1/+1
2022-10-02Fixup errors introduced by Clingo Pr: (#32905)John W. Parent1-1/+1
* re2c depends on cmake on Windows * Winbison properly added to bootstrap package search list
2022-09-30develop: canonicalize dev paths and base relative paths on `env.path` (#30075)Greg Becker6-12/+60
Allow environment variables and spack-specific path substitution variables (e.g. `$spack`) to be used in the paths associated with develop specs, while maintaining the ability to keep those paths relative to the environment rather than the working directory.
2022-09-29Install: Add use-buildcache option to install (#32537)kwryankrattiger3-8/+166
Install: Add use-buildcache option to install * Allow differentiating between top level packages and dependencies when determining whether to install from the cache or not. * Add unit test for --use-buildcache * Use metavar to display use-buildcache options. * Update spack-completion
2022-09-28Windows: Support for Clingo and dependencies (#30690)John W. Parent4-5/+14
Make it possible to install the Clingo package on Windows; this also provides a means to use Clingo with Spack on Windows. This includes * A new "winbison" package: Windows has a port of bison and flex where the two packages are grouped together. Clingo dependencies have been updated to use winbison on Windows and bison elsewhere (this avoids complicating the existin bison/flex packages until we can add support for implied virtuals). * The CMake build system was incorrectly converting CMAKE_INSTALL_PREFIX to POSIX format. * The re2c package has been modified to use CMake on Windows; for now this is done by overloading the configure/build/install methods to perform CMake-appropriate operations; the package should be refactored once support for multiple build systems in one Package is available.
2022-09-28Remove mentions of "best-effort" matrix expansion in the docs (#32755)Massimiliano Culpo1-25/+0
closes #20340
2022-09-27shell.py: fix undefined variable in csh deactivate (#32816)Alex Hornburg1-2/+2
This commit fixes #27027. The root cause of the issue is that the `SPACK_OLD_PROMPT` variable was evaluated in string interpolation regardless of whether the guard condition above evaluates to true or false. This commit uses the `eval` keyword to defer evaluation until the command is executed. Co-authored-by: Alexander Hornburg <alexande@xilinx.com>
2022-09-26improve lexing error (#32832)Chris White1-1/+3
2022-09-26Fetching/decompressing: use magic numbers (#31589)John W. Parent9-137/+524
Spack currently depends on parsing filenames of downloaded files to determine what type of archive they are and how to decompress them. This commit adds a preliminary check based on magic numbers to determine archive type (but falls back on name parsing if the extension type cannot be determined). As part of this work, this commit also enables decompression of .tar.xz-compressed archives on Windows.
2022-09-23Fix two docstring typos (#32751)Tamara Dahlgren1-2/+2
2022-09-23Skip all tests using problematic fixture on python 2.7 (#32768)Peter Scheibel1-0/+6
2022-09-22Fetching: log more exceptions (#32736)Peter Scheibel2-1/+3
Include exception info related to url retrieval in debug messages which otherwise would be swallowed. This is intended to be useful for detecting if CA configuration interferes with downloads from HTTPS links.
2022-09-22Add hash hint to multi-spec message (#32652)Tamara Dahlgren3-25/+22
2022-09-21Allow conditional variants as first values in a variant directive (#32740)Massimiliano Culpo2-3/+10
2022-09-21Docs: Update pipeline ci rebuild to add --tests (plus fixed typos) (#32048)Tamara Dahlgren1-24/+64
2022-09-17Add skip_import to PythonPackage and use it in py-nilearn (#32664)Manuela Kuhn2-1/+30
* Add skip_import to PythonPackage and use it in py-nilearn * Fix dependencies
2022-09-16bugfix: package hash should affect process, dag, and dunder hashes (#32516)Todd Gamblin5-1/+40
This fixes a bug where two installations that differ only by package hash will not show up in `spack find`. The bug arose because `_cmp_node` on `Spec` didn't include the package hash in its yielded fields. So, any two `Spec` objects that were only different by package hash would appear to be equal and would overwrite each other when inserted into the same `dict`. Note that we could still *install* specs with different package hashes, and they would appear in the database, but we code that needed to put them into data structures that use `__hash__` would have issues. This PR makes `Spec.__hash__` and `Spec.__eq__` include the `process_hash()`, and it makes `Spec._cmp_node` include the package hash. All of these *should* include all information in a spec so that we don't end up in a situation where we are blind to particular field differences. Eventually, we should unify the `_cmp_*` methods with `to_node_dict` so there aren't two sources of truth, but this needs some thought, since the `_cmp_*` methods exist for speed. We should benchmark whether it's really worth having two types of hashing now that we use `json` instead of `yaml` for spec hashing. - [x] Add `package_hash` to `Spec._cmp_node` - [x] Add `package_hash` to `spack.solve.asp.spec_clauses` so that the `package_hash` will show up in `spack diff`. - [x] Add `package_hash` to the `process_hash` (which doesn't affect abstract specs but will make concrete specs correct) - [x] Make `_cmp_iter` report the dag_hash so that no two specs with different process hashes will be considered equal.
2022-09-14Bugfix: find_libraries (#32653)John W. Parent2-9/+32
53a7b49 created a reference error which broke `.libs` (and `find_libraries`) for many packages. This fixes the reference error and improves the testing for `find_libraries` by actually checking the extension types of libraries that are retrieved by the function.
2022-09-14Manifest parsing: skip invalid manifest files (#32467)Peter Scheibel3-4/+57
* catch json schema errors and reraise as property of SpackError * no need to catch subclass of given error * Builtin json library for Python 2 uses more generic type * Correct instantiation of SpackError (requires a string rather than an exception) * Use exception chaining (where possible)
2022-09-14Allow version to accept the '=' token without activating lexer switch (#32257)psakievich2-54/+126
2022-09-14Update module_file_support.rst (#32629)Sam Broderick1-1/+1
Missing lead in "This may to inconsistent modulfiles if..."
2022-09-13Windows rpath support (#31930)John W. Parent14-40/+271
Add a post-install step which runs (only) on Windows to modify an install prefix, adding symlinks to all dependency libraries. Windows does not have the same concept of RPATHs as Linux, but when resolving symbols will check the local directory for dependency libraries; by placing a symlink to each dependency library in the directory with the library that needs it, the package can then use all Spack-built dependencies. Note: * This collects dependency libraries based on Package.rpath, which includes only direct link dependencies * There is no examination of libraries to check what dependencies they require, so all libraries of dependencies are symlinked into any directory of the package which contains libraries
2022-09-13Added install phase test methods for CachedCMakePackage (inherited) and ↵Tamara Dahlgren1-0/+6
WafPackage (#32627)
2022-09-10ci: restore coverage computation (#32585)Massimiliano Culpo2-1/+5
* ci: restore coverage computation * Mark "test_foreground_background" as xfail * Mark "test_foreground_background_output" as xfail * Make number of processes explicit, remove verbosity on linux * Run coverage on just 3 Python jobs for linux * Run coverage on just 3 Python jobs for linux * Run coverage on just 2 Python jobs for linux * Add back verbose, since before we didn't encounter the xdist internal error * Reduce the workers to 2 * Try to use command line
2022-09-09GitRef Versions: Bug Fixes (#32437)psakievich4-3/+47
* Fix a version cmp bug in asp.py * Fix submodule bug for git refs * Add branch in logic for submodules * Fix git version comparisons main does not satisfy git.foo=main git.foo=main does satisfy main
2022-09-08CMakePackage: allow custom CMAKE_PREFIX_PATH entries (#32547)Peter Scheibel3-2/+17
* define `cmake_prefix_paths` property for packages add to CMake arguments via CMakePackage Co-authored-by: Robert Underwood <runderwood@anl.gov>
2022-09-08Skip test which fails randomly on Python 2.7 (#32569)Peter Scheibel1-0/+3
2022-09-07Make GHA tests parallel by using xdist (#32361)Tom Scogland14-79/+114
* Add two no-op jobs named "all-prechecks" and "all" These are a suggestion from @tgamblin, they are stable named markers we can use from gitlab and possibly for required checks to make CI more resilient to refactors changing the names of specific checks. * Enable parallel testing using xdist for unit testing in CI * Normalize tmp paths to deal with macos * add -u flag compatibility to spack python As of now, it is accepted and ignored. The usage with xdist, where it is invoked specifically by `python -u spack python` which is then passed `-u` by xdist is the entire reason for doing this. It should never be used without explicitly passing -u to the executing python interpreter. * use spack python in xdist to support python 2 When running on python2, spack has many import cycles unless started through main. To allow that, this uses `spack python` as the interpreter, leveraging the `-u` support so xdist doesn't error out when it unconditionally requests unbuffered binary IO. * Use shutil.move to account for tmpdir being in a separate filesystem sometimes
2022-09-06Fix spack locking on some NFS systems (#32426)Seth R. Johnson1-2/+6
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2022-09-06Mark a test xfail on Python 2.7 (#32526)Massimiliano Culpo1-0/+3
refers #32470
2022-09-02Reorder workflow execution in GHA (#32183)Tom Scogland1-1/+1
This patchset refactors our GitHub actions into a single top-level ci workflow that invokes a series of reusable actions. The main goal of this is to be able to easily control which tests run and in what order based on the success or failure of top-level prechecks. Our previous workflows ran in three sets: * nix tests: style and verification first, then linux and macos tests if successful * windows tests: style and verification first, then linux and macos tests if successful * bootstrap tests As a result, the bootstrap tests ran even if the style failed, and style and verification had to run on two different platforms despite running identical checks. I'm relatively sure that's because of the limitation on dependencies between steps in the jobs. Reusable workflows allow us to run the style, verification and now audit checks once, then depending on the results, and the files changed, run the appropriate nix, windows and bootstrap tests. While it saves only a few minutes by itself, this makes it easier to refactor checks to subset tests without having to replicate tests or other workflow components in the future. Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-09-02Update bootstrap buildcache to v0.3 (#32262)Harmen Stoppels2-15/+44
This release allow to bootstrap patchelf from binaries.
2022-09-01Fix cause of checksum failures in public binary mirror (#32407)Scott Wittenburg2-45/+113
Move the copying of the buildcache to a root job that runs after all the child pipelines have finished, so that the operation can be coordinated across all child pipelines to remove the possibility of race conditions during potentially simlutandous copies. This lets us ensure the .spec.json.sig and .spack files for any spec in the root mirror always come from the same child pipeline mirror (though which pipeline is arbitrary). It also allows us to avoid copying of duplicates, which we now do.
2022-09-01New command, `spack change`, to change existing env specs (#31995)Peter Scheibel7-3/+263
If you have an environment like ``` $ cat spack.yaml spack: specs: [openmpi@4.1.0+cuda] ``` this PR provides a new command `spack change` that you can use to adjust environment specs from the command line: ``` $ spack change openmpi~cuda $ cat spack.yaml spack: specs: [openmpi@4.1.0~cuda] ``` in other words, this allows you to tweak the details of environment specs from the command line. Notes: * This is only allowed for environments that do not define matrices * This is possible but not anticipated to be needed immediately * If this were done, it should probably only be done for "named"/not-anonymous specs (i.e. we can change `openmpi+cuda` but not spec like `+cuda` or `@4.0.1~cuda`)
2022-09-01Port package sanity unit tests to audits (#32405)Massimiliano Culpo4-310/+213
2022-08-31ASP-based solver: allow to reuse installed externals (#31558)Massimiliano Culpo4-31/+146
fixes #31484 Before this change if anything was matching an external condition, it was considered "external" and thus something to be "built". This was happening in particular to external packages that were re-read from the DB, which then couldn't be reused, causing the problems shown in #31484. This PR fixes the issue by excluding specs with a "hash" from being considered "external" * Test that users have a way to select a virtual This ought to be solved by extending the "require" attribute to virtual packages, so that one can: ```yaml mpi: require: 'multi-provider-mpi' ``` * Prevent conflicts to be enforced on specs that can be reused. * Rename the "external_only" fact to "buildable_false", to better reflect its origin
2022-08-30Environments: Add support for include URLs (#29026)Tamara Dahlgren22-221/+919
* Preliminary support for include URLs in spack.yaml (environment) files This commit adds support in environments for external configuration files obtained from a URL with a preference for grabbing raw text from GitHub and gitlab for efficient downloads of the relevant files. The URL can also be a link to a directory that contains multiple configuration files. Remote configuration files are retrieved and cached for the environment. Configuration files with the same name will not be overwritten once cached.
2022-08-26Use threading.TIMEOUT_MAX when available (#32399)Betsy McPhail1-1/+8
This value was introduced in Python 3.2. Specifying a timeout greater than this value will raise an OverflowError.
2022-08-26Configuration: Allow requirements for virtual packages (#32369)Massimiliano Culpo4-22/+149
Extend the semantics of package requirements to allow using them also under a virtual package attribute in packages.yaml These requirements are enforced whenever that virtual spec is present in the DAG.
2022-08-26Update archspec to latest commit (#32368)Massimiliano Culpo3-6/+105
Modifications: - [x] Add graviton3 - [x] Optimize __eq__ for microarchitectures
2022-08-26Improve error message for yaml config file (#32382)Chris White1-1/+9
* improve error message * Update lib/spack/spack/config.py Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com> Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
2022-08-25Python: add new maintainer (#31755)Adam J. Stewart1-1/+1