summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2022-09-16bugfix: package hash should affect process, dag, and dunder hashes (#32516)Todd Gamblin5-1/+40
This fixes a bug where two installations that differ only by package hash will not show up in `spack find`. The bug arose because `_cmp_node` on `Spec` didn't include the package hash in its yielded fields. So, any two `Spec` objects that were only different by package hash would appear to be equal and would overwrite each other when inserted into the same `dict`. Note that we could still *install* specs with different package hashes, and they would appear in the database, but we code that needed to put them into data structures that use `__hash__` would have issues. This PR makes `Spec.__hash__` and `Spec.__eq__` include the `process_hash()`, and it makes `Spec._cmp_node` include the package hash. All of these *should* include all information in a spec so that we don't end up in a situation where we are blind to particular field differences. Eventually, we should unify the `_cmp_*` methods with `to_node_dict` so there aren't two sources of truth, but this needs some thought, since the `_cmp_*` methods exist for speed. We should benchmark whether it's really worth having two types of hashing now that we use `json` instead of `yaml` for spec hashing. - [x] Add `package_hash` to `Spec._cmp_node` - [x] Add `package_hash` to `spack.solve.asp.spec_clauses` so that the `package_hash` will show up in `spack diff`. - [x] Add `package_hash` to the `process_hash` (which doesn't affect abstract specs but will make concrete specs correct) - [x] Make `_cmp_iter` report the dag_hash so that no two specs with different process hashes will be considered equal.
2022-09-14Bugfix: find_libraries (#32653)John W. Parent2-9/+32
53a7b49 created a reference error which broke `.libs` (and `find_libraries`) for many packages. This fixes the reference error and improves the testing for `find_libraries` by actually checking the extension types of libraries that are retrieved by the function.
2022-09-14Manifest parsing: skip invalid manifest files (#32467)Peter Scheibel3-4/+57
* catch json schema errors and reraise as property of SpackError * no need to catch subclass of given error * Builtin json library for Python 2 uses more generic type * Correct instantiation of SpackError (requires a string rather than an exception) * Use exception chaining (where possible)
2022-09-14Allow version to accept the '=' token without activating lexer switch (#32257)psakievich2-54/+126
2022-09-14Update module_file_support.rst (#32629)Sam Broderick1-1/+1
Missing lead in "This may to inconsistent modulfiles if..."
2022-09-13Windows rpath support (#31930)John W. Parent14-40/+271
Add a post-install step which runs (only) on Windows to modify an install prefix, adding symlinks to all dependency libraries. Windows does not have the same concept of RPATHs as Linux, but when resolving symbols will check the local directory for dependency libraries; by placing a symlink to each dependency library in the directory with the library that needs it, the package can then use all Spack-built dependencies. Note: * This collects dependency libraries based on Package.rpath, which includes only direct link dependencies * There is no examination of libraries to check what dependencies they require, so all libraries of dependencies are symlinked into any directory of the package which contains libraries
2022-09-13Added install phase test methods for CachedCMakePackage (inherited) and ↵Tamara Dahlgren1-0/+6
WafPackage (#32627)
2022-09-10ci: restore coverage computation (#32585)Massimiliano Culpo2-1/+5
* ci: restore coverage computation * Mark "test_foreground_background" as xfail * Mark "test_foreground_background_output" as xfail * Make number of processes explicit, remove verbosity on linux * Run coverage on just 3 Python jobs for linux * Run coverage on just 3 Python jobs for linux * Run coverage on just 2 Python jobs for linux * Add back verbose, since before we didn't encounter the xdist internal error * Reduce the workers to 2 * Try to use command line
2022-09-09GitRef Versions: Bug Fixes (#32437)psakievich4-3/+47
* Fix a version cmp bug in asp.py * Fix submodule bug for git refs * Add branch in logic for submodules * Fix git version comparisons main does not satisfy git.foo=main git.foo=main does satisfy main
2022-09-08CMakePackage: allow custom CMAKE_PREFIX_PATH entries (#32547)Peter Scheibel3-2/+17
* define `cmake_prefix_paths` property for packages add to CMake arguments via CMakePackage Co-authored-by: Robert Underwood <runderwood@anl.gov>
2022-09-08Skip test which fails randomly on Python 2.7 (#32569)Peter Scheibel1-0/+3
2022-09-07Make GHA tests parallel by using xdist (#32361)Tom Scogland14-79/+114
* Add two no-op jobs named "all-prechecks" and "all" These are a suggestion from @tgamblin, they are stable named markers we can use from gitlab and possibly for required checks to make CI more resilient to refactors changing the names of specific checks. * Enable parallel testing using xdist for unit testing in CI * Normalize tmp paths to deal with macos * add -u flag compatibility to spack python As of now, it is accepted and ignored. The usage with xdist, where it is invoked specifically by `python -u spack python` which is then passed `-u` by xdist is the entire reason for doing this. It should never be used without explicitly passing -u to the executing python interpreter. * use spack python in xdist to support python 2 When running on python2, spack has many import cycles unless started through main. To allow that, this uses `spack python` as the interpreter, leveraging the `-u` support so xdist doesn't error out when it unconditionally requests unbuffered binary IO. * Use shutil.move to account for tmpdir being in a separate filesystem sometimes
2022-09-06Fix spack locking on some NFS systems (#32426)Seth R. Johnson1-2/+6
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2022-09-06Mark a test xfail on Python 2.7 (#32526)Massimiliano Culpo1-0/+3
refers #32470
2022-09-02Reorder workflow execution in GHA (#32183)Tom Scogland1-1/+1
This patchset refactors our GitHub actions into a single top-level ci workflow that invokes a series of reusable actions. The main goal of this is to be able to easily control which tests run and in what order based on the success or failure of top-level prechecks. Our previous workflows ran in three sets: * nix tests: style and verification first, then linux and macos tests if successful * windows tests: style and verification first, then linux and macos tests if successful * bootstrap tests As a result, the bootstrap tests ran even if the style failed, and style and verification had to run on two different platforms despite running identical checks. I'm relatively sure that's because of the limitation on dependencies between steps in the jobs. Reusable workflows allow us to run the style, verification and now audit checks once, then depending on the results, and the files changed, run the appropriate nix, windows and bootstrap tests. While it saves only a few minutes by itself, this makes it easier to refactor checks to subset tests without having to replicate tests or other workflow components in the future. Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-09-02Update bootstrap buildcache to v0.3 (#32262)Harmen Stoppels2-15/+44
This release allow to bootstrap patchelf from binaries.
2022-09-01Fix cause of checksum failures in public binary mirror (#32407)Scott Wittenburg2-45/+113
Move the copying of the buildcache to a root job that runs after all the child pipelines have finished, so that the operation can be coordinated across all child pipelines to remove the possibility of race conditions during potentially simlutandous copies. This lets us ensure the .spec.json.sig and .spack files for any spec in the root mirror always come from the same child pipeline mirror (though which pipeline is arbitrary). It also allows us to avoid copying of duplicates, which we now do.
2022-09-01New command, `spack change`, to change existing env specs (#31995)Peter Scheibel7-3/+263
If you have an environment like ``` $ cat spack.yaml spack: specs: [openmpi@4.1.0+cuda] ``` this PR provides a new command `spack change` that you can use to adjust environment specs from the command line: ``` $ spack change openmpi~cuda $ cat spack.yaml spack: specs: [openmpi@4.1.0~cuda] ``` in other words, this allows you to tweak the details of environment specs from the command line. Notes: * This is only allowed for environments that do not define matrices * This is possible but not anticipated to be needed immediately * If this were done, it should probably only be done for "named"/not-anonymous specs (i.e. we can change `openmpi+cuda` but not spec like `+cuda` or `@4.0.1~cuda`)
2022-09-01Port package sanity unit tests to audits (#32405)Massimiliano Culpo4-310/+213
2022-08-31ASP-based solver: allow to reuse installed externals (#31558)Massimiliano Culpo4-31/+146
fixes #31484 Before this change if anything was matching an external condition, it was considered "external" and thus something to be "built". This was happening in particular to external packages that were re-read from the DB, which then couldn't be reused, causing the problems shown in #31484. This PR fixes the issue by excluding specs with a "hash" from being considered "external" * Test that users have a way to select a virtual This ought to be solved by extending the "require" attribute to virtual packages, so that one can: ```yaml mpi: require: 'multi-provider-mpi' ``` * Prevent conflicts to be enforced on specs that can be reused. * Rename the "external_only" fact to "buildable_false", to better reflect its origin
2022-08-30Environments: Add support for include URLs (#29026)Tamara Dahlgren22-221/+919
* Preliminary support for include URLs in spack.yaml (environment) files This commit adds support in environments for external configuration files obtained from a URL with a preference for grabbing raw text from GitHub and gitlab for efficient downloads of the relevant files. The URL can also be a link to a directory that contains multiple configuration files. Remote configuration files are retrieved and cached for the environment. Configuration files with the same name will not be overwritten once cached.
2022-08-26Use threading.TIMEOUT_MAX when available (#32399)Betsy McPhail1-1/+8
This value was introduced in Python 3.2. Specifying a timeout greater than this value will raise an OverflowError.
2022-08-26Configuration: Allow requirements for virtual packages (#32369)Massimiliano Culpo4-22/+149
Extend the semantics of package requirements to allow using them also under a virtual package attribute in packages.yaml These requirements are enforced whenever that virtual spec is present in the DAG.
2022-08-26Update archspec to latest commit (#32368)Massimiliano Culpo3-6/+105
Modifications: - [x] Add graviton3 - [x] Optimize __eq__ for microarchitectures
2022-08-26Improve error message for yaml config file (#32382)Chris White1-1/+9
* improve error message * Update lib/spack/spack/config.py Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com> Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
2022-08-25Python: add new maintainer (#31755)Adam J. Stewart1-1/+1
2022-08-25ASP-based solver: prevent the use of hashes that are not available (#32354)Massimiliano Culpo1-0/+5
fixes #32348
2022-08-24Allow default requirements in packages.yaml (#32260)Massimiliano Culpo3-3/+81
Allow users to express default requirements in packages.yaml. These requirements are overridden if more specific requirements are present for a given package.
2022-08-23New package: py-griffe and new build backend: py-pdm-pep517 (#32335)Ryan Marcellino1-2/+14
* New package: py-griffe and new build backend: py-pdm-pep517 * add pdm to build backend docs
2022-08-23`spack -e x config --scope=y add z` add to scope (#31746)Harmen Stoppels2-1/+9
* `spack -e x config --scope=y add z` add to scope instead of to the environment file.
2022-08-23meson: create unique names for build directory (#32062)Simon Pintarelli1-1/+9
Taken from CMakePackage
2022-08-23spack ci: add support for running stand-alone tests (#27877)Tamara Dahlgren22-445/+1669
This support requires adding the '--tests' option to 'spack ci rebuild'. Packages whose stand-alone tests are broken (in the CI environment) can be configured in gitlab-ci to be skipped by adding them to broken-tests-packages. Highlights include: - Restructured 'spack ci' help to provide better subcommand summaries; - Ensured only one InstallError (i.e., installer's) rather than allowing build_environment to have its own; and - Refactored CI and CDash reporting to keep CDash-related properties and behavior in a separate class. This allows stand-alone tests from `spack ci` to run when the `--tests` option is used. With `--tests`, stand-alone tests are run **after** a **successful** (re)build of the package. Test results are collected and report(able) using CDash. This PR adds the following features: - Adds `-t` and `--tests` to `spack ci rebuild` to run stand-alone tests; - Adds `--fail-fast` to stop stand-alone tests after the first failure; - Ensures a *single* `InstallError` across packages (i.e., removes second class from build environment); - Captures skipping tests for externals and uninstalled packages (for CDash reporting); - Copies test logs and outputs to the CI artifacts directory to facilitate debugging; - Parses stand-alone test results to report outputs from each `run_test` as separate test parts (CDash reporting); - Logs a test completion message to allow capture of timing of the last `run_test` part; - Adds the runner description to the CDash site to better distinguish entries in CDash tables; - Adds `gitlab-ci` `broken-tests-packages` to CI configuration to skip stand-alone testing for packages with known issues; - Changes `spack ci --help` so description of each subcommand is a single line; - Changes `spack ci <subcommand> --help` to provide the full description of each command (versus no description); and - Ensures `junit` test log file ends in an `.xml` extension (versus default where it does not). Tasks: - [x] Include the equivalent of the architecture information, or at least the host target, in the CDash output - [x] Upload stand-alone test results files as `test` artifacts - [x] Confirm tests are run in GitLab - [x] Ensure CDash results are uploaded as artifacts - [x] Resolve issues with CDash build-and test results appearing on same row of the table - [x] Add unit tests as needed - [x] Investigate why some (dependency) packages don't have test results (e.g., related from other pipelines) - [x] Ensure proper parsing and reporting of skipped tests (as `not run`) .. post- #28701 merge - [x] Restore the proper CDash URLand or mirror ONCE out-of-band testing completed
2022-08-20spack list: add --tag flag (#32016)sparkyniner2-0/+21
* modified list.py and added functionality for --tag * Removed long and very long, shifted rest of code above return statement * removed results variable * added import statement at top * added the line accidentally deleted * added line accidentally deleted * changed p.name to p, added line inside if statement * line order switched * [@spackbot] updating style on behalf of sparkyniner * ran update completion command * add tests * Update lib/spack/spack/test/cmd/list.py Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> * [@spackbot] updating style on behalf of sparkyniner * changed argument to mock_packages and moved code under filter by tag * removed bad rebase code and added additional test * [@spackbot] updating style on behalf of sparkyniner * added line removed earlier * added line removed earlier * replaced function * added more recommended changes Co-authored-by: sairaj <sairaj@sairajs-MacBook-Pro.local> Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2022-08-19docs: add a note about an issue being solved on `develop` (#32261)Harmen Stoppels1-1/+7
2022-08-19Add messages to assertions in asp.py (#32237)psakievich1-9/+40
Assertions without messages if/when hit create a blank error message for users. This PR adds error messages to all assertions in asp.py even if it seems unlikely they will ever be needed.
2022-08-18spack.util.package_hash: parametrize unit-tests (#32240)Massimiliano Culpo1-68/+29
* spack.util.package_hash: parametrize unit-tests * Fix comment
2022-08-18Remove unused argument in test fixture (#32236)Massimiliano Culpo1-1/+1
The argument is very likely a typo, and was meant to be given to the fixture decorator. Since the value being passed is the default, let's just remove it.
2022-08-18spec: fix typo in Spec._finalize_concretization (#32210)Massimiliano Culpo1-1/+1
2022-08-17bugfix: use cmake version from dependency (#31739)Tom Scogland2-7/+11
Ensure that build tools with module-level commands in spack use the version built as part of their build graph if one exists. This is now also required for mesa, scons, cmake and ctest, out of graph versions of these tools in path will not be found unless added as an external. This bug appeared because a new version of rocprim needs cmake 3.16, while I have 3.14 in my path I had added an external for cmake 3.20 to the dag, but 3.14 was still used to configure rocprim causing it to fail. As far as I can tell, all the build tools added in build_environment.py had this problem, despite the fact that they should have been resolving these tools by name with a path search and find the one in the dag that way. I'm still investigating why the path searching and Executable logic didn't do it, but this makes three of the build systems much more explicit, and leaves only gmake and ninja as dependencies from out in the system while ensuring the version in the dag is used if there is one. The additional sqlite version is to perturb the hash of python to work around a relocation bug which will be fixed in a subsequent PR.
2022-08-17Add c++14 support flag for xl compiler (#31826)Peter Scheibel1-0/+8
2022-08-16filesystem: in recursive mtime, check only files that exist (#32175)Wouter Deconinck1-1/+1
* filesystem: use lstat in recursive mtime When a `develop` path contains a dead symlink, the `os.stat` in the recursive `mtime` determination trips up over it. Closes #32165.
2022-08-16bugfix: silence `clingo` warning about `requirement_policy/3`Todd Gamblin1-0/+1
`requirement_policy/3` is generated and may not be in Spack's inputs to Clingo. Currently this is causing warnings like: ``` $ spack spec zlib /global/u2/t/tgamblin/src/spack/lib/spack/spack/solver/concretize.lp:510:3-43: info: atom does not occur in any rule head: requirement_policy(Package,X,"one_of") /global/u2/t/tgamblin/src/spack/lib/spack/spack/solver/concretize.lp:517:3-43: info: atom does not occur in any rule head: requirement_policy(Package,X,"one_of") /global/u2/t/tgamblin/src/spack/lib/spack/spack/solver/concretize.lp:523:3-43: info: atom does not occur in any rule head: requirement_policy(Package,X,"any_of") /global/u2/t/tgamblin/src/spack/lib/spack/spack/solver/concretize.lp:534:3-43: info: atom does not occur in any rule head: requirement_policy(Package,X,"any_of") Input spec -------------------------------- zlib Concretized -------------------------------- zlib@1.2.11%gcc@7.5.0+optimize+pic+shared arch=cray-sles15-haswell ``` - [x] Silence warning with `#defined requirement_policy/3`
2022-08-16Configuration: allow users to enforce hard spec constraints (#27987)Peter Scheibel7-75/+545
Spack doesn't have an easy way to say something like "If I build package X, then I *need* version Y": * If you specify something on the command line, then you ensure that the constraints are applied, but the package is always built * Likewise if you `spack add X...`` to your environment, the constraints are guaranteed to hold, but the environment always builds the package * You can add preferences to packages.yaml, but these are not guaranteed to hold (Spack can choose other settings) This commit adds a 'require' subsection to packages.yaml: the specs added there are guaranteed to hold. The commit includes documentation for the feature. Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-08-15CI/documentation: constrain pygments to avoid latest version (2.13.0) (#32163)Peter Scheibel1-0/+1
All PRs are failing the docs build on account of an error with pygments. These errors coincide with a new release of pygments (2.13.0) and restricting to < 2.13 allows the doc tests to pass, so this commit enforces that constraint for the docs build. Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2022-08-15Check that no reserved names are overridden by package recipes (#32092)Massimiliano Culpo1-0/+40
A few attribute in packages are meant to be reserved for Spack internal use. This audit checks packages to ensure none of these attributes are overridden. - [x] add additional audit check
2022-08-12rocm: add v5.2 for the entire stack (#31591)renjithravindrankannath1-8/+0
2022-08-11Fix performance regression with `spack mirror create --all` (#32005)Massimiliano Culpo6-169/+401
This PR fixes the performance regression reported in #31985 and a few other issues found while refactoring the spack mirror create command. Modifications: * (Primary) Do not require concretization for `spack mirror create --all` * Forbid using --versions-per-spec together with --all * Fixed a few issues when reading specs from input file (specs were not concretized, command would fail when trying to mirror dependencies) * Fix issue with default directory for spack mirror create not being canonicalized * Add more unit tests to poke spack mirror create * Skip externals also when mirroring environments * Changed slightly the wording for reporting (it was mentioning "Successfully created" even in presence of errors) * Fix issue with colify (was not called properly during error reporting)
2022-08-11CachedCMakePackage: Port over logic from CMake build system for ldlibs and ↵Chris White1-5/+26
ldflags (#31750) * port over logic from cmake build system for ldlibs and ldflags * follow new style standards * escape quotes * escape quotes * nevermind going back
2022-08-11Don't set `LD_LIBRARY_PATH` by default on Linux (#28354)Harmen Stoppels4-15/+26
`LD_LIBRARY_PATH` can break system executables (e.g., when an enviornment is loaded) and isn't necessary thanks to `RPATH`s. Packages that require `LD_LIBRARY_PATH` can set this in `setup_run_environment`. - [x] Prefix inspections no longer set `LD_LIBRARY_PATH` by default - [x] Document changes and workarounds for people who want `LD_LIBRARY_PATH`
2022-08-11Add missing info to diagnostic for circular provides (#32027)Chris Green1-1/+1