summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2022-05-03OpenCV and OpenBLAS: add external find support (#30240)Brian Van Essen3-8/+20
Added support for finding the OpenCV package via the find external command. Included support for identifying variants based on available shared libraries. Added support to finding the OpenBLAS package via the find external command. Enabled packages to show that they can be discovered via the find external command in the info message. Updated the OpenCV and OpenBLAS packages to use the extensible search mechanism for library extensions on multiple OS platforms. Corrected how find externals works on Darwin for OpenCV and OpenBLAS to accommodate that the version numbers are placed before the file extension instead of after it, as on Linux. Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-05-02spack ci: Fix typo in error message (#30438)Ken Raffenetti1-1/+1
2022-05-01bugfix: don't calculate `spack tutorial` version from version infoTodd Gamblin1-1/+1
A recent switch to the way we do `develop` versioning broke this. We should hard-code the latest tutorial version.
2022-04-29spack.yaml: add concretizer.yaml to the schema (#30376)Massimiliano Culpo1-0/+2
2022-04-28Add command for reading JSON-based DB description (now with more tests) (#29652)Peter Scheibel14-53/+764
This is an amended version of https://github.com/spack/spack/pull/24894 (reverted in https://github.com/spack/spack/pull/29603). https://github.com/spack/spack/pull/24894 broke all instances of `spack external find` (namely when it is invoked without arguments/options) because it was mandating the presence of a file which most systems would not have. This allows `spack external find` to proceed if that file is not present and adds tests for this. - [x] Add a test which confirms that `spack external find` successfully reads a manifest file if present in the default manifest path --- Original commit message --- Adds `spack external read-cray-manifest`, which reads a json file that describes a set of package DAGs. The parsed results are stored directly in the database. A user can see these installed specs with `spack find` (like any installed spec). The easiest way to use them right now as dependencies is to run `spack spec ... ^/hash-of-external-package`. Changes include: * `spack external read-cray-manifest --file <path/to/file>` will add all specs described in the file to Spack's installation DB and will also install described compilers to the compilers configuration (the expected format of the file is described in this PR as well including examples of the file) * Database records now may include an "origin" (the command added in this PR registers the origin as "external-db"). In the future, it is assumed users may want to be able to treat installs registered with this command differently (e.g. they may want to uninstall all specs added with this command) * Hash properties are now always preserved when copying specs if the source spec is concrete * I don't think the hashes of installed-and-concrete specs should change and this was the easiest way to handle that * also specs that are concrete preserve their `.normal` property when copied (external specs may mention compilers that are not registered, and without this change they would fail in `normalize` when calling `validate_or_raise`) * it might be this should only be the case if the spec was installed - [x] Improve testing - [x] Specifically mark DB records added with this command (so that users can do something like "uninstall all packages added with `spack read-external-db`) * This is now possible with `spack uninstall --all --origin=external-db` (this will remove all specs added from manifest files) - [x] Strip variants that are listed in json entries but don't actually exist for the package
2022-04-28ASP-based solver: discard unknown packages from reuse (#30357)Massimiliano Culpo2-3/+25
* ASP-based solver: discard unknown packages from reuse This is an add-on to #28259 that cover for the case of a single package.py being removed from a repository, rather than an entire custom repository being removed. * Add unit test
2022-04-28Add BUILD_TESTING to standard CMake arguments (#30374)Frédéric Simonis1-0/+2
CTest determines whether to enable tests using the BUILD_TESTING variable. This should be used by projects to conditionally enable the compilation of tests. Spack knowns which packages have to run tests and can thus automatically define this variable.
2022-04-28build_env/test_env: add concretizer args (#30289)Greg Becker1-0/+2
2022-04-28install --overwrite: use rename instead of tmpdir (#29746)Harmen Stoppels4-54/+45
I tried to use --overwrite on nvhpc, but nvhpc's install size is 16GB. Seems better to do os.rename in the same directory than moving the directory to `/tmp`. - [x] install --overwrite: use rename instead of tmpdir - [x] use tempfile
2022-04-27ASP-based solver: handle installed specs from unknown namespaces (#30092)Massimiliano Culpo2-0/+44
fixes #28259 This commit discard specs from unknown namespaces from the ones that can be "reused" during concretization. Previously Spack would just error out when encountering them.
2022-04-26Feature: Allow re-use of run_test() in install_time_test_callbacks (#26594)Tamara Dahlgren4-54/+109
Allow re-use of run_test() in install_time_test_callbacks Co-authored-by: Greg Becker <becker33@llnl.gov>
2022-04-26docs: Fix ROCmPackage example syntax (#30168)Ken Raffenetti1-1/+1
2022-04-26old concretizer: update deprecation warning (#30227)Greg Becker1-2/+2
2022-04-26Windows: fix termination of process output redirection (#29923)John W. Parent2-25/+20
The parent thread in the process stdout redirection logic on Windows was closing a file that was being read in child thread, which lead to error-based termination of the reader thread. This updates the interaction to avoid the error.
2022-04-26spack spec: add '--format' argument (#27908)lorddavidiii2-3/+14
2022-04-26setup_package: don't unload modules on Cray when run with --dirty (#30261)Mark W. Krentel2-1/+40
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-04-25ASP-based solver: allow configuring target selection (#29835)Massimiliano Culpo5-43/+149
* ASP-based solver: allow configuring target selection This commit adds a new "concretizer:targets" configuration section, and two options under it. - "concretizer:targets:granularity" allows switching from considering only generic targets to consider all possible microarchitectures. - "concretizer:targets:host_compatible" instead controls whether we can concretize for microarchitectures that are incompatible with the current host. * Add documentation * Add unit-tests
2022-04-25Environments: add flag to skip printing concretized specs (#30272)iarspider1-1/+5
With an active environment, you can now run "spack concretize --quiet" and it will suppress printing the concretized specs.
2022-04-25ASP-based solver: always consider version of installed packages (#29933)Massimiliano Culpo3-19/+115
* ASP-based solver: always consider version of installed packages fixes #29201 Explicitly add facts for versions of installed software when using the --reuse option, so that we could consider versions that are not declared in package.py
2022-04-25spec.py: make parser select from environment (#30276)Harmen Stoppels3-2/+29
The parser is already committing a crime of querying the database for specs when it encounters a `/hash`. It's helpful, but unfortunately not helpful when trying to install a specific spec in an environment by hash. Therefore, consider the environment first, then the database. This allows the following: ```console $ spack -e . concretize ==> Starting concretization ==> Environment concretized in 0.27 seconds. ==> Concretized diffutils - 7vangk4 diffutils@3.8%gcc@10.3.0 arch=linux-ubuntu20.04-zen2 - hyb7ehx ^libiconv@1.16%gcc@10.3.0 libs=shared,static arch=linux-ubuntu20.04-zen2 $ spack -e . install /hyb7ehx ==> Installing libiconv-1.16-hyb7ehxxyqqp2hiw56bzm5ampkw6cxws ... ==> libiconv: Successfully installed libiconv-1.16-hyb7ehxxyqqp2hiw56bzm5ampkw6cxws Fetch: 0.01s. Build: 17.54s. Total: 17.55s. [+] /tmp/tmp.VpvYApofVm/store/linux-ubuntu20.04-zen2/gcc-10.3.0/libiconv-1.16-hyb7ehxxyqqp2hiw56bzm5ampkw6cxws ```
2022-04-25bugfix: `installed` and `installed_upstream` should not assert (#30271)Todd Gamblin4-12/+63
Fix bug introduced in #30191. `Spec.installed` and `Spec.installed_upstream` should just return `False` for abstract specs, as they can be called in that context. - [x] `Spec.installed` returns `False` now instead of asserting that the `Spec` is concrete. - [x] `Spec.installed_upstream` returns `False` now instead of asserting that the `Spec` is concrete. - [x] `Spec.installed_upstream` no longer caches its result, as install status seems like a bad thing to cache -- it can easily be invalidated. Calling code should use transactions if there are peformance issues, as with other places in Spack. - [x] add tests for `Spec.installed` and `Spec.installed_upstream`
2022-04-22Move "installed" and "installed_upstream" from PackageBase to Spec (#30191)Massimiliano Culpo20-94/+121
This PR moves the `installed` and `installed_upstream` properties from `PackageBase` to `Spec` and is a step towards being able to reuse specs for which we don't have a `package.py` available. It _should_ be sufficient to complete the concretization step and see the spec in the concretized DAG. To fully reuse a spec without a package.py though we need a way to serialize enough data to reconstruct the results of calls to: - `Spec.libs`, `Spec.headers` and `Spec.ommand` - `Package.setup_dependent_*_environment` and `Package.setup_run_environment` - [x] Add stub methods to packages with warnings - [x] Add a missing "root=False" in cmd/fetch.py - [x] Assert that a spec is concrete before checking installation status
2022-04-22Package-level submodule attribute: support explicit versions (#30085)Peter Scheibel3-1/+51
2022-04-22Update Dockerfiles and images for Spack v0.18.0 (#30216)Massimiliano Culpo2-14/+27
This PR updates the list of images we build nightly, deprecating Ubuntu 16.04 and CentOS 8 and adding Ubuntu 20.04, Ubuntu 22.04 and CentOS Stream. It also removes a lot of duplication by generating the Dockerfiles during the CI workflow and uploading them as artifacts for later inspection or reuse.
2022-04-20docs: add (config.yaml) to sections for faster lookup by config file (#30157)Harmen Stoppels6-18/+18
2022-04-20PythonPackage docs: Spack now supports Windows (#30177)Adam J. Stewart1-2/+1
2022-04-19ci: Make test independent from any changes in spack repo (#30150)Scott Wittenburg1-0/+53
Fix test_ci_generate_prune_untouched(), which would fail if run when the latest commit changed the .gitlab-ci.yml. This change mocks the get_stack_changed() method in that test to disregard the state of the current spack repo in favor of a mock repo under test control.
2022-04-19environment.py: write lockfile last (#30039)Harmen Stoppels1-2/+6
This makes it easier to write `spack.lock: spack.yaml` type of rules in `Makefiles`.
2022-04-15Remove assert in favor of error message (#29797)John W. Parent3-20/+21
2022-04-14Windows platform: don't allow MakefilePackage (#29904)John W. Parent1-0/+2
2022-04-14spack ci: remove relate-CDash-builds functionality (#29950)Zack Galbreath7-359/+16
gitlab ci: Remove code for relating CDash builds Relating CDash builds to their dependencies was a seldom used feature. Removing it will make it easier for us to reorganize our CDash projects & build groups in the future by eliminating the needs to keep track of CDash build ids in our binary mirrors.
2022-04-13Ad-hoc Git commit versions: support submodules (#30037)Peter Scheibel3-11/+83
* Allow packages to add a 'submodules' property that determines when ad-hoc Git-commit-based versions should initialize submodules * add support for ad-hoc git-commit-based versions to instantiate submodules if the associated package has a 'submodules' property and it indicates this should happen for the associated spec * allow Package-level submodule request to influence all explicitly-defined version() in the Package * skip test on windows which fails because of long paths
2022-04-13Add support for Python 3.10 (#29581)Massimiliano Culpo3-8/+9
* Add support for Python 3.10 * Update unit-tests to use 3.10 * Update Getting started section of the docs * Update bootstrap action
2022-04-12Splice differing virtual packages (#27919)Nathan Hanford3-21/+86
Co-authored-by: Nathan Hanford <hanford1@llnl.gov>
2022-04-12Don't allow replacement of root develop specs with --reuse (#28605)psakievich3-4/+48
* Fix to concretize.lp do not allow dev specs to be reused Co-authored-by: Gregory Becker <becker33@llnl.gov>
2022-04-12Git commit versions bugfix: Environments and Concretization (#29717)Peter Scheibel7-32/+127
Spack added support in #24639 for ad-hoc Git-commit-hash-based versions: A user can install a package x@hash, where X is a package that stores its source code in a Git repository, and the hash refers to a commit in that repository which is not recorded as an explicit version in the package.py file for X. A couple issues were found relating to this: * If an environment defines an alternative package repo (i.e. with repos.yaml), and spack.yaml contains user Specs with ad-hoc Git-commit-hash-based versions for packages in that repo, then as part of retrieving the data needed for version comparisons it will attempt to retrieve the package before the environment's configuration is instantiated. * The bookkeeping information added to compare ad-hoc git versions was being stripped from Specs during concretization (such that user Specs which succeeded before concretizing would then fail after) This addresses the issues: * The first issue is resolved by deferring access to the associated Package until the versions are actually compared to one another. * The second issue is resolved by ensuring that the Git bookkeeping information is explicitly applied to Specs after they are concretized. This also: * Resolves an ambiguity in the mock_git_version_info fixture used to create a tree of Git commits and provide a list where each index maps to a known commit. * Isolates the cache used for Git repositories in tests using the mock_git_version_info fixture * Adds a TODO which points out that if the remote Git repository overwrites tags, that Spack will then fail when using ad-hoc Git-commit-hash-based versions
2022-04-11Update `gpg publish` to work with mirror arguments (#28740)Doug Jacobsen2-3/+50
This commit updates the `gpg publish` command to work with the mirror arguments, when trying to push keys to a mirror. - [x] update `gpg publish command - [x] add test for publishing GPG keys and rebuilding the key index within a mirror
2022-04-11cray platform: unload cray-mpich (#29898)Sergey Kosukhin1-3/+4
2022-04-11macos: fewer calls to sw_vers (#29997)Seth R. Johnson1-0/+5
In a typical call to spack, the OperatingSystem gets instantiated multiple times. For macOS, each one requires a call to `sw_vers`, which is done through the Executable helper class. Memoizing reduces the call count from "spac spec" from three to one.
2022-04-08Environments: unify the spec objects on first concretization (#29948)Massimiliano Culpo4-5/+39
Currently environments are indexed by build hashes. When looking into this bug I noticed there is a disconnect between environments that are concretized in memory for the first time and environments that are read from a `spack.lock`. The issue is that specs read from a `spack.lock` don't have a full hash, since they are indexed by a build hash which is strictly coarser. They are also marked "final" as they are read from a file, so we can't compute additional hashes. This bugfix PR makes "first concretization" equivalent to re-reading the specs from a corresponding `spack.lock`, and doing so unveiled a few tests were we were making wrong assumptions and relying on the fact that a `spack.lock` file was not there already. * Add unit test * Modify mpich to trigger jobs in pipelines * Fix two failing unit tests * Fix another full_hash vs. build_hash mismatch in tests
2022-04-08Deprecate top-level module config (#28659)Harmen Stoppels10-185/+100
* Ignore top-level module config; add auto-update In Spack 0.17 we got module sets (modules:[name]:[prop]), and for backwards compat modules:[prop] was short for modules:default:[prop]. But this makes it awkward to define default config for the "default" module set. Since 0.17 is branched off, we can now deprecate top-level module config (that is, just ignore it with a warning). This PR does that, and it implements `spack config update modules` to make upgrading easy (we should have added that to 0.17 already...) It also removes references to `dotkit` stuff which was already deprecated in 0.13 and could have been removed in 0.14. Prefix inspections are the only exception, since the top-level prefix inspections used for `spack load` and `spack env activate`.
2022-04-08concretize.lp: enforce target compatibility through DAG (#29694)Greg Becker2-8/+40
Spack currently allows dependencies to be concretized for an architecture incompatible with the root. This commit adds rules to make this situation impossible by design.
2022-04-07Use the non-deprecated `MetaPathFinder` interface (#29745)Massimiliano Culpo9-379/+294
* Extract the MetaPathFinder and Loaders for packages in their own classes https://peps.python.org/pep-0451/ Currently, RepoPath and Repo implement the (deprecated) interface of MetaPathFinder (find_module) and of Loader (load_module). This commit extracts both of them and places the code in their own classes. The MetaPathFinder interface is updated to contain both the deprecated "find_module" (for Python 2.7 support) and the recommended "find_spec". Update of the Loader interface is deferred at a subsequent commit. * Move the lines to be prepended inside "RepoLoader" Also adjust the naming of a few variables too * Remove spack.util.imp, since code is only used in spack.repo * Remove support from loading Python modules Python > 3 but < 3.5 * Remove `Repo._create_namespace` This function was interacting badly with the MetaPathFinder and causing issues with "normal" imports. Removing the function allows to do things like: ```python import spack.pkg.builtin.mpich cls = spack.pkg.builtin.mpich.Mpich ``` * Remove code needed to trigger the Singleton evaluation The finder is coded in a way to trigger the Singleton, so we don't need external code now that we register it at module level into `sys.meta_path`. * Add unit tests
2022-04-07ci: clean up env between tests with working_dir (#29807)Scott Wittenburg2-98/+70
2022-04-07ci: do not prune specs when stacks are touched (#29825)Scott Wittenburg1-9/+33
2022-04-07autotools.py: pic flags for %nvhpc (#29920)Harmen Stoppels1-5/+8
2022-04-06web.py: set User-Agent (#29919)Harmen Stoppels1-1/+5
Some servers require `User-Agent` to be set, and otherwise error with access denied. One such example is mpich. To fix this, set `User-Agent: Spackbot/[version]` as a header. Apparently by convention, it should include the word `bot`.
2022-04-06link_tree.py: format conflict error message (#29870)Harmen Stoppels1-2/+2
Show each `[src a] and [src b] both project to [dst]` on a separate line.
2022-04-05Bugfix: CVS fetching (#29793)Erik Schnetter1-2/+7
#27021 broke fetching for CVS-based packages because: - The mirror logic was using URL parsing to extract a path from the CVS repository location - #27021 added sanity checks to enforce that strings passed to the URL parser were actually URLs This replaces the call to "url_util.parse" with logic that is customized for CVS. This implies that VCSFetchStrategy should rename the "url_attr" attribute to something more generic, but that should be handled separately.
2022-04-04Allow conditional possible values in variants (#29530)Massimiliano Culpo6-6/+173
Allow declaring possible values for variants with an associated condition. If the variant takes one of those values, the condition is imposed as a further constraint. The idea of this PR is to implement part of the mechanisms needed for modeling [packages with multiple build-systems]( https://github.com/spack/seps/pull/3). After this PR the build-system directive can be implemented as: ```python variant( 'build-system', default='cmake', values=( 'autotools', conditional('cmake', when='@X.Y:') ), description='...', ) ``` Modifications: - [x] Allow conditional possible values in variants - [x] Add a unit-test for the feature - [x] Add documentation