summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2023-09-08Speed-up `spack external find` execution (#39843)Massimiliano Culpo9-340/+314
* Perform external spec detection with multiple workers The logic to perform external spec detection has been refactored into classes. These classes use the GoF "template" pattern to account for the small differences between searching for "executables" and for "libraries", while unifying the larger part of the algorithm. A ProcessPoolExecutor is used to parallelize the work. * Speed-up external find by tagging detectable packages automatically Querying packages by tag is much faster than inspecting the repository, since tags are cached. This commit adds a "detectable" tag to every package that implements the detection protocol, and external detection uses it to search for packages. * Pass package names instead of package classes to workers The slowest part of the search is importing the Python modules associated with candidate packages. The import is done serially before we distribute the work to the pool of executors. This commit pushes the import of the Python module to the job performed by the workers, and passes just the name of the packages to the executors. In this way imports can be done in parallel. * Rework unit-tests for Windows Some unit tests were doing a full e2e run of a command just to check a input handling. Make the test more focused by just stressing a specific function. Mark as xfailed 2 tests on Windows, that will be fixed by a PR in the queue. The tests are failing because we monkeypatch internals in the parent process, but the monkeypatching is not done in the "spawned" child process.
2023-09-08build(deps): bump pytest from 7.4.1 to 7.4.2 in /lib/spack/docs (#39883)dependabot[bot]1-1/+1
Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.4.1 to 7.4.2. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst) - [Commits](https://github.com/pytest-dev/pytest/compare/7.4.1...7.4.2) --- updated-dependencies: - dependency-name: pytest dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-07Docs/Packaging guide: Add BundlePackage (#39691)Tamara Dahlgren2-12/+80
* Docs/Packaging guide: Add BundlePackage * Adjusted version ordering note to reflect convention.
2023-09-07CI Timing Statistics (#38598)kwryankrattiger5-38/+113
* Write timing information for installs from cache * CI: aggregate and upload install_times.json to artifacts * CI: Don't change root directory for artifact generation * Flat event based timer variation Event based timer allows for easily starting and stopping timers without wiping sub-timer data. It also requires less branching logic when tracking time. The json output is non-hierarchical in this version and hierarchy is less rigidly enforced between starting and stopping. * Add and write timers for top level install * Update completion * remove unused subtimer api * Fix unit tests * Suppress timing summary option * Save timers summaries to user_data artifacts * Remove completion from fish * Move spack python to script section * Write timer correctly for non-cache installs * Re-add hash to timer file * Fish completion updates * Fix null timer yield value * fix type hints * Remove timer-summary-file option * Add "." in front of non-package timer name --------- Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com> Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
2023-09-07move determine_number_of_jobs into spack.util.cpus, use it in concretize ↵Harmen Stoppels7-46/+86
(#37620)
2023-09-07CUDA 10.1/10.2: Fix conflicts with PGI 19+. (#39747)Rémi Lacroix1-1/+1
The condition probably did not get updated when the behavior of the version specifier changed.
2023-09-06Handles case of empty craype_dir (#39371)Dominic Hofer1-0/+2
The craype_dir folder can be empty. It happens to be the case on Arolla and Tsa at CSCS.
2023-09-06"spack config add": support values with ":" (#39831)John W. Parent2-8/+44
This is a fixed version of b72a268 * That commit would discard the final key component (so if you set "config:install_tree:root", it would discard "root" and just set install tree). * When setting key:"value", with the quotes, that commit would discard the quotes, which would confuse the system if adding a value like "{example}" (the "{" character indicates a dictionary). This commit retains the quotes.
2023-09-05CMakePackage: remove -DBUILD_TESTING flag (#37967)Adam J. Stewart1-2/+0
2023-09-05Revert ""spack config add": allow values with a ":" (#39279)" (#39825)Harmen Stoppels2-30/+7
This reverts commit b72a268bc565d370bd64589e86929acfd73c08a1.
2023-09-05compilers/gcc.py: support cxx{20,23}_flag (#39777)Wouter Deconinck1-0/+22
2023-09-05Add type-hints to the `spack.detection` package (#39803)Massimiliano Culpo2-67/+99
2023-09-05build(deps): bump pytest from 7.4.0 to 7.4.1 in /lib/spack/docs (#39790)dependabot[bot]1-1/+1
Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.4.0 to 7.4.1. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst) - [Commits](https://github.com/pytest-dev/pytest/compare/7.4.0...7.4.1) --- updated-dependencies: - dependency-name: pytest dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-04glibc: add package (#39695)Harmen Stoppels1-0/+41
2023-09-01Windows: fix pwsh env activate/deactivate; load/unload (#39118)John W. Parent6-11/+31
These commands are currently broken on powershell (Windows) due to improper use of the InvokeCommand commandlet and a lack of direct support for the `--pwsh` argument in `spack load`, `spack unload`, and `spack env deactivate`.
2023-09-01"spack config add": allow values with a ":" (#39279)John W. Parent2-7/+30
If you wanted to set a configuration option like `config:install_tree:root` to "C:/path/to/config.yaml", Spack had trouble parsing this because of the ":" in the value. This adds logic to allow using quotes to enclose the value, so you can add `config:install_tree:root:"C:/path/to/config.yaml"`. Configuration keys should never contain a quote character, so the presence of any quote is taken to mean that the rest of the string is specifying the value.
2023-08-31environment: improve spack remove matching (#39390)Massimiliano Culpo3-5/+95
search for equivalent specs, not for equal strings when selecting a spec to remove.
2023-08-31build(deps): bump sphinx from 7.2.4 to 7.2.5 in /lib/spack/docs (#39716)dependabot[bot]1-1/+1
Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 7.2.4 to 7.2.5. - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/master/CHANGES) - [Commits](https://github.com/sphinx-doc/sphinx/compare/v7.2.4...v7.2.5) --- updated-dependencies: - dependency-name: sphinx dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-31Remove a few local imports in tests (#39719)Harmen Stoppels1-12/+0
2023-08-31ASP-based solver: add hidden mode to ignore versions that are moving ↵Harmen Stoppels5-134/+194
targets, use that in CI (#39611) Setting the undocumented variable SPACK_CONCRETIZER_REQUIRE_CHECKSUM now causes the solver to avoid accounting for versions that are not checksummed. This feature is used in CI to avoid spurious concretization against e.g. develop branches.
2023-08-30Windows: oneapi/msvc consistency (#39180)John W. Parent1-36/+145
Currently, OneAPI's setvars scripts effectively disregard any arguments we're passing to the MSVC vcvars env setup script, and additionally, completely ignore the requested version of OneAPI, defaulting to whatever the latest installed on the system is. This leads to a scenario where we have improperly constructed Windows native development environments, with potentially multiple versions of MSVC and OneAPI being loaded or called in the same env. Obviously this is far from ideal and leads to some fairly inscrutable errors such as overlapping header files between MSVC and OneAPI and a different version of OneAPI being called than the env was setup for. This PR solves this issue by creating a structured invocation of each relevant script in an order that ensures the correct values are set in the resultant build env. The order needs to be: 1. MSVC vcvarsall 2. The compiler specific env.bat script for the relevant version of the oneapi compiler we're looking for. The root setvars scripts seems to respect this as well, although it is less explicit 3. The root oneapi setvars script, which sets up everything else the oneapi env needs and seems to respect previous env invocations.
2023-08-30completion: add alias handlingTodd Gamblin1-0/+3
Bash completion is now smarter about handling aliases. In particular, if all completions for some input command are aliased to the same thing, we'll just complete with that thing. If you've already *typed* the full alias for a command, we'll complete the alias. So, for example, here there's more than one real command involved, so all aliases are shown: ```console $ spack con concretise concretize config containerise containerize ``` Here, there are two possibilities: `concretise` and `concretize`, but both map to `concretize` so we just complete that: ```console $ spack conc concretize ``` And here, the user has already typed `concretis`, so we just go with it as there is only one option: ```console spack concretis concretise ```
2023-08-30Add `concretise` and `containerise` aliases for our UK usersTodd Gamblin1-1/+1
From a user: > Aargh. > ``` > ==> Error: concretise is not a recognized Spack command or extension command; check with `spack commands`. > ``` To make things easier for our friends in the UK, this adds `concretise` and `containerise` aliases for the `spack concretize` and `spack containerize` commands. - [x] add aliases - [x] update completions
2023-08-29Revert "Add style tool to fix fish file formatting (#39155)" (#39680)Harmen Stoppels3-37/+13
This reverts commit 70c71e8f935b3c5152036a993295df13c5fe69ca.
2023-08-29build(deps): bump sphinx from 7.2.3 to 7.2.4 in /lib/spack/docs (#39668)dependabot[bot]1-1/+1
Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 7.2.3 to 7.2.4. - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/master/CHANGES) - [Commits](https://github.com/sphinx-doc/sphinx/compare/v7.2.3...v7.2.4) --- updated-dependencies: - dependency-name: sphinx dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-28Add style tool to fix fish file formatting (#39155)Adam J. Stewart3-13/+37
2023-08-28modules: copy matched config to prevent bleed (#39421)Jonathon Anderson1-1/+1
2023-08-26External ROCm: add example configuration (#39602)Peter Scheibel2-0/+114
* add an example of an external rocm configuration * include more info * generalize section to all GPU support
2023-08-25Windows symlinking support (#38599)James Smillie11-161/+787
This reapplies 66f7540, which adds supports for hardlinks/junctions on Windows systems where developer mode is not enabled. The commit was reverted on account of multiple issues: * Checks added to prevent dangling symlinks were interfering with existing CI builds on Linux (i.e. builds that otherwise succeed were failing for creating dangling symlinks). * The logic also updated symlinking to perform redirection of relative paths, which lead to malformed symlinks. This commit fixes these issues.
2023-08-24rebuild-index: fix race condition by avoiding parallelism (#39594)Harmen Stoppels1-25/+10
2023-08-24Remove leftover attributes from parser (#39574)Massimiliano Culpo2-27/+1
#35042 introduced lazy hash parsing, but didn't remove a few attributes from the parser that were needed only for concrete specs This commit removes them, since they are effectively dead code.
2023-08-24build(deps): bump sphinx from 7.2.2 to 7.2.3 in /lib/spack/docs (#39603)dependabot[bot]1-1/+1
Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 7.2.2 to 7.2.3. - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/master/CHANGES) - [Commits](https://github.com/sphinx-doc/sphinx/compare/v7.2.2...v7.2.3) --- updated-dependencies: - dependency-name: sphinx dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-23Uppercase global constants in spec.py (#39573)Massimiliano Culpo9-38/+38
* Uppercase global constants in spec.py Co-authored-by: Bernhard Kaindl <bernhardkaindl7@gmail.com>
2023-08-23do not warn for invalid package name on repo.yaml if subdirectory: '' (#39439)Greg Becker1-1/+1
2023-08-23ASP-based solver: split heuristic for duplicate nodes (#39593)Massimiliano Culpo3-16/+27
The heuristic for duplicate nodes contains a few typos, and apparently slows down the solve for specs that have a lot of sub-optimal choices to be taken. This is likely because with a lot of sub-optimal choices, the low priority, flawed heuristic is being used by clingo. Here I split the heuristic, so complex rules that matter only if we allow multiple nodes from the same package are used only in that case.
2023-08-23Add --fail-fast option for generating build caches (#38496)Dom Heinzeller1-1/+25
Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
2023-08-23docs: API, f-string, and path updates to test section (#39584)Tamara Dahlgren1-38/+42
2023-08-22spack.caches: make fetch_cache_location lowercase (#39575)Kamil Iskra1-1/+1
fetch_cache_location was erroneously renamed to FETCH_cache_location as part of #39428, breaking "spack module create".
2023-08-22ASP-based solver: use edge properties for reused specs (#39508)Massimiliano Culpo3-31/+33
Since #34821 we are annotating virtual dependencies on DAG edges, and reconstructing virtuals in memory when we read a concrete spec from previous formats. Therefore, we can remove a TODO in asp.py, and rely on "virtual_on_edge" facts to be imposed.
2023-08-22Bump sphinx from 6.2.1 to 7.2.2 in /lib/spack/docs (#39502)dependabot[bot]1-1/+1
Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 6.2.1 to 7.2.2. - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/master/CHANGES) - [Commits](https://github.com/sphinx-doc/sphinx/compare/v6.2.1...v7.2.2) --- updated-dependencies: - dependency-name: sphinx dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-22Bump sphinx-rtd-theme from 1.2.2 to 1.3.0 in /lib/spack/docs (#39562)dependabot[bot]1-1/+1
Bumps [sphinx-rtd-theme](https://github.com/readthedocs/sphinx_rtd_theme) from 1.2.2 to 1.3.0. - [Changelog](https://github.com/readthedocs/sphinx_rtd_theme/blob/master/docs/changelog.rst) - [Commits](https://github.com/readthedocs/sphinx_rtd_theme/compare/1.2.2...1.3.0) --- updated-dependencies: - dependency-name: sphinx-rtd-theme dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-18Fix Spack freeze on install child process unexpected exit (#39015)Jordan Galby1-8/+27
* Fix spack frozen on child process defunct * Rename parent/child pipe to read/write to emphasize non-duplex mode
2023-08-17ASP-based solver: optimize key to intermediate dicts (#39471)Massimiliano Culpo1-8/+15
Computing str(spec) is faster than computing hash(spec), and since all the abstract specs we deal with come from user configuration they cannot cover DAG structures that are not captured by str() but are captured by hash()
2023-08-17Bump mypy from 1.5.0 to 1.5.1 in /lib/spack/docs (#39478)dependabot[bot]1-1/+1
Bumps [mypy](https://github.com/python/mypy) from 1.5.0 to 1.5.1. - [Commits](https://github.com/python/mypy/compare/v1.5.0...v1.5.1) --- updated-dependencies: - dependency-name: mypy dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-17Add debugging statements to file search (#39121)Peter Scheibel1-2/+7
Co-authored-by: Scheibel <scheibel1@ml-9983616.the-lab.llnl.gov>
2023-08-17Delay abstract hashes lookup (#39251)Harmen Stoppels7-68/+92
Delay lookup for abstract hashes until concretization time, instead of until Spec comparison. This has a few advantages: 1. `satisfies` / `intersects` etc don't always know where to resolve the abstract hash (in some cases it's wrong to look in the current env, db, buildcache, ...). Better to let the call site dictate it. 2. Allows search by abstract hash without triggering a database lookup, causing quadratic complexity issues (accidental nested loop during search) 3. Simplifies queries against the buildcache, they can now use Spec instances instead of strings. The rules are straightforward: 1. a satisfies b when b's hash is prefix of a's hash 2. a intersects b when either a's or b's hash is a prefix of b's or a's hash respectively
2023-08-16Extract Package from PackageNode for error messagesMassimiliano Culpo1-72/+72
2023-08-16Avoid sort on singleton list during edge insertion (#39458)Harmen Stoppels1-9/+7
The median length of this list of 1. For reasons I don't know, `.sort()` still like to call the key function. This saves ~9% of total database read time, and the number of calls goes from 5305 -> 1715.
2023-08-16Do not impose provider conditions, if the node is not a provider (#39456)Massimiliano Culpo2-8/+30
* Do not impose provider conditions, if the node is not a provider fixes #39455 When a node can be a provider of a spec, but is not selected as a provider, we should not be imposing provider conditions on the virtual. * Adjust the integrity constraint, by using the correct atom
2023-08-16json: minify by default (#39457)Harmen Stoppels1-1/+1