summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2023-10-03ci: Change how job names appear in gitlab (#39963)Scott Wittenburg2-38/+13
2023-10-03build(deps): bump urllib3 from 2.0.5 to 2.0.6 in /lib/spack/docs (#40286)dependabot[bot]1-1/+1
Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.0.5 to 2.0.6. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](https://github.com/urllib3/urllib3/compare/v2.0.5...2.0.6) --- updated-dependencies: - dependency-name: urllib3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-03Buildcache tarballs with rootfs structure (#39341)Harmen Stoppels2-73/+162
Two changes in this PR: 1. Register absolute paths in tarballs, which makes it easier to use them as container image layers, or rootfs in general, outside of Spack. Spack supports this already on develop. 2. Assemble the tarfile entries "by hand", which has a few advantages: 1. Avoid reading `/etc/passwd`, `/etc/groups`, `/etc/nsswitch.conf` which `tar.add(dir)` does _for each file it adds_ 2. Reduce the number of stat calls per file added by a factor two, compared to `tar.add`, which should help with slow, shared filesystems where these calls are expensive 4. Create normalized `TarInfo` entries from the start, instead of letting Python create them and patching them after the fact 5. Don't recurse into subdirs before processing files, to avoid keeping nested directories opened. (this changes the tar entry order slightly, it's like sorting by `(not is_dir, name)`.
2023-10-02buildcache: ignore errors of newer buildcache version (#40279)Harmen Stoppels1-2/+2
Currently buildcaches are forward incompatible in an annoying way: spack errors out when trying to use them. With this change, you just get a warning.
2023-10-02Spack python 3.12: PEP 695 unparse support (#40155)Harmen Stoppels1-0/+28
2023-10-01docs: Replace package list with packages.spack.io (#40251)Todd Gamblin8-86/+30
For a long time, the docs have generated a huge, static HTML package list. It has some disadvantages: * It's slow to load * It's slow to build * It's hard to search We now have a nice website that can tell us about Spack packages, and it's searchable so users can easily find the one or two packages out of 7400 that they're looking for. We should link to this instead of including a static package list page in the docs. - [x] Replace package list link with link to packages.spack.io - [x] Remove `package_list.html` generation from `conf.py`. - [x] Add a new section for "Links" to the docs. - [x] Remove docstring notes from contribution guide (we haven't generated RST for package docstrings for a while) - [x] Remove referencese to `package-list` from docs.
2023-09-29Windows: detect all available SDK versions (#39823)John W. Parent1-28/+35
Currently, Windows SDK detection will only pick up SDK versions related to the current version of Windows Spack is running on. However, in some circumstances, we want to detect other version of the SDK, for example, for compiling on Windows 11 for Windows 10 to ensure an API is compatible with Win10.
2023-09-29Cray manifest: compiler handling fixes (#40061)Peter Scheibel5-131/+215
* Make use of `prefix` in the Cray manifest schema (prepend it to the relative CC etc.) - this was a Spack error. * Warn people when wrong-looking compilers are found in the manifest (i.e. non-existent CC path). * Bypass compilers that we fail to add (don't allow a single bad compiler to terminate the entire read-cray-manifest action). * Refactor Cray manifest tests: module-level variables have been replaced with fixtures, specifically using the `test_platform` fixture, which allows the unit tests to run with the new concretizer. * Add unit test to check case where adding a compiler raises an exception (check that this doesn't prevent processing the rest of the manifest).
2023-09-29Print error when missing git (#40254)Jordan Galby1-1/+5
Like a missing curl.
2023-09-29Don't drop build deps on overwrite install (#40252)Harmen Stoppels2-4/+26
If you `spack install x ^y` where `y` is a pure build dep of `x`, and then uninstall `y`, and then `spack install --overwrite x ^y`, the build fails because `y` is not re-installed. Same can happen when you install a develop spec, run `spack gc`, modify sources, and install again; develop specs rely on overwrite install to work correctly.
2023-09-29Test package detection in a systematic way (#18175)Massimiliano Culpo10-23/+457
This PR adds a new audit sub-command to check that detection of relevant packages is performed correctly in a few scenarios mocking real use-cases. The data for each package being tested is in a YAML file called detection_test.yaml alongside the corresponding package.py file. This is to allow encoding detection tests for compilers and other widely used tools, in preparation for compilers as dependencies.
2023-09-28Partial removal of circular dependencies between `spack` and `llnl` (#40090)Massimiliano Culpo30-258/+297
Modifications: - [x] Move `spack.util.string` to `llnl.string` - [x] Remove dependency of `llnl` on `spack.error` - [x] Move path of `spack.util.path` to `llnl.path` - [x] Move `spack.util.environment.get_host_*` to `spack.spec`
2023-09-28Use st_nlink in hardlink tracking (#39328)Harmen Stoppels2-11/+12
Only add potential hardlinks to a set/dict, instead of each file. This should be much cheaper, since hardlinks are very rare.
2023-09-28Support optional vars script arguments (#40064)Brian Vanderwende1-1/+6
2023-09-26spack --profile: dump to stderr (#40209)Harmen Stoppels1-1/+1
2023-09-24build(deps): bump urllib3 from 2.0.4 to 2.0.5 in /lib/spack/docs (#40119)dependabot[bot]1-1/+1
Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.0.4 to 2.0.5. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](https://github.com/urllib3/urllib3/compare/2.0.4...v2.0.5) --- updated-dependencies: - dependency-name: urllib3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-22Remove distutils dependency in Spack (#40153)Harmen Stoppels1-3/+2
* msvc.py: don't import distutils Introduced in #27021, makes Spack forward incompatible with Python. The module was already deprecated at the time of the PR. * update spack package
2023-09-22Add support for macOS Sonoma (#40115)Adam J. Stewart1-0/+1
2023-09-21Harden compiler detection against errors (#39736)John W. Parent4-36/+106
Fixes #39622 Add a timeout to compiler detection and allow Spack to proceed when this timeout occurs. In all cases, the timeout is 120 seconds: it is assumed any compiler invocation we do for the purposes of verifying it would resolve in that amount of time. Also refine executables that are tested as being possible MSVC instances, and limit where we try to detect MSVC. In more detail: * Compiler detection should timeout after a certain period of time. Because compiler detection executes arbitrary executables on the system, we could encounter a program that just hangs, or even a compiler that hangs on a license key or similar. A timeout prevents this from hanging Spack. * Prevents things like cl-.* from being detected as potential MSVC installs. cl is always just cl in all cases that Spack supports. Change the MSVC class to indicate this. * Prevent compilers unsupported on certain platforms from being detected there (i.e. don't look for MSVC on systems other than Windows). The first point alone is sufficient to address #39622, but the next two reduce the likelihood of timeouts (which is useful since those slow down the user even if they are survivable).
2023-09-21Restore virtuals normalization on edge construction (#40130)Massimiliano Culpo3-1/+19
Put back normalization of the "virtuals" input as a sorted tuple. Without this we might get edges that differ just for the order of virtuals, or that have lists, which are not hashable. Add unit-tests to prevent regressions.
2023-09-20Fix a leftover typo from depflag rework (#40101)Massimiliano Culpo1-1/+1
2023-09-20Add details on error messages from requirements (#40092)Marc Mengel1-2/+6
2023-09-19Use process pool executors for web-crawling and retrieving archives (#39888)Massimiliano Culpo18-278/+288
Fix a race condition when searching urls, and updating a shared set '_visited'.
2023-09-19ASP-based solver: declare deprecated versions iff config:deprecated:true ↵Massimiliano Culpo8-41/+146
(#40011) By default, do not let deprecated versions enter the solve. Previously you could concretize to something deprecated, only to get errors on install. With this commit, we get errors on concretization, so the issue is caught earlier.
2023-09-18Better detection of Python libs/headers (#39308)Adam J. Stewart1-8/+12
2023-09-18checksum: use FIXME instead of FIX ME when adding versions (#40050)Michael Kuhn1-1/+1
The former is highlighted by editors, while the latter is not.
2023-09-15CI: add spec to job vars (#39905)Caetano Melone1-0/+5
* CI: add details about built spec to ci job variables Co-authored-by: Alec Scott <alec@bcs.sh> Co-authored-by: Alec Scott <hi@alecbcs.com>
2023-09-15Windows RPATHing: fix symlink error (#39933)John W. Parent2-23/+38
With 349ba83, you cannot symlink() if the link already exists. Update the simulated RPATHing logic on Windows to account for that.
2023-09-15Group primitive url/path handling functions together (#40028)Massimiliano Culpo20-884/+901
2023-09-15build(deps): bump sphinx from 7.2.5 to 7.2.6 in /lib/spack/docs (#40029)dependabot[bot]1-1/+1
Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 7.2.5 to 7.2.6. - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/master/CHANGES.rst) - [Commits](https://github.com/sphinx-doc/sphinx/compare/v7.2.5...v7.2.6) --- updated-dependencies: - dependency-name: sphinx dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-14Ensure PythonExtension is consistent when finding externals (#40012)Massimiliano Culpo1-41/+41
PythonExtension is a base class for PythonPackage, and is meant to be used for any package that is a Python extension but is not built using "python_pip". The "update_external_dependency" method in the base class calls another method that is defined in the derived class. Push "get_external_python_for_prefix" up in the hierarchy to make method calls consistent.
2023-09-14Add efficient `deptype` flag and `spack.deptypes` module (#39472)Harmen Stoppels33-455/+529
This commit replaces the internal representation of deptypes with `int`, which is more compact and faster to operate with. Double loops like: ``` any(x in ys for x in xs) ``` are replaced by constant operations bool(xs & ys), where xs and ys are dependency types. Global constants are exposed for convenience in `spack.deptypes`
2023-09-13Revert "ASP-based solver: don't declare deprecated versions unless required ↵Harmen Stoppels8-91/+42
(#38181)" (#40010) This reverts commit babd29da5015d740748329006487b7d7f96df169.
2023-09-13ASP-based solver: don't declare deprecated versions unless required (#38181)Massimiliano Culpo8-42/+91
Currently, the concretizer emits facts for all versions known to Spack, including deprecated versions, and has a specific optimization objective to minimize their use. This commit simplifies how deprecated versions are handled by considering possible versions for a spec only if they appear in a spec literal, or if the `config:deprecated:true` is set directly or through the `--deprecated` flag. The optimization objective has also been removed, in favor of just ordering versions and having deprecated ones last. This results in: a) no delayed errors on install, but concretization errors when deprecated versions would be the only option. This is in particular relevant for CI where it's better to get errors early b) a slight concretization speed-up due to fewer facts c) a simplification of the logic program. Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
2023-09-13Spec.tree: add type hints (#39948)Massimiliano Culpo2-21/+47
2023-09-12Fix typo in metaclasses (#39947)Massimiliano Culpo1-1/+1
2023-09-12asp.py: fix deprecation warning (#39943)Harmen Stoppels1-3/+2
2023-09-12build(deps): bump black from 23.7.0 to 23.9.1 in /lib/spack/docs (#39937)dependabot[bot]1-1/+1
Bumps [black](https://github.com/psf/black) from 23.7.0 to 23.9.1. - [Release notes](https://github.com/psf/black/releases) - [Changelog](https://github.com/psf/black/blob/main/CHANGES.md) - [Commits](https://github.com/psf/black/compare/23.7.0...23.9.1) --- updated-dependencies: - dependency-name: black dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-11Update archspec to latest commit (#39920)Massimiliano Culpo4-10/+119
- [x] Intel flags for old architectures - [x] Support for Sapphire Rapids - [x] Cache the "ancestors" computation
2023-09-08NMake Builder: change property name (#39824)John W. Parent1-5/+5
NMake makefiles are still called makefiles. The corresponding builder variable was called "nmakefile", which is a bit unintuitive and lead to a few easy-to-make, hard-to-notice mistakes when creating packages. This commit renames the builder property to be "makefile"
2023-09-08Windows decompression: fix removal of intermediate file (#38958)Dan Lipsa1-6/+7
Extensionless archives requiring two-stage decompression and extraction require intermediate archives to be renamed after decompression/extraction to prevent collision. Prior behavior attempted to cleanup the intermediate archive with the original name, this PR ensures the renamed folder is cleaned instead. Co-authored-by: Dan Lipsa <dan.lipsa@khq.kitware.com> Co-authored-by: John Parent <john.parent@kitware.com>
2023-09-08Speed-up `spack external find` execution (#39843)Massimiliano Culpo9-340/+314
* Perform external spec detection with multiple workers The logic to perform external spec detection has been refactored into classes. These classes use the GoF "template" pattern to account for the small differences between searching for "executables" and for "libraries", while unifying the larger part of the algorithm. A ProcessPoolExecutor is used to parallelize the work. * Speed-up external find by tagging detectable packages automatically Querying packages by tag is much faster than inspecting the repository, since tags are cached. This commit adds a "detectable" tag to every package that implements the detection protocol, and external detection uses it to search for packages. * Pass package names instead of package classes to workers The slowest part of the search is importing the Python modules associated with candidate packages. The import is done serially before we distribute the work to the pool of executors. This commit pushes the import of the Python module to the job performed by the workers, and passes just the name of the packages to the executors. In this way imports can be done in parallel. * Rework unit-tests for Windows Some unit tests were doing a full e2e run of a command just to check a input handling. Make the test more focused by just stressing a specific function. Mark as xfailed 2 tests on Windows, that will be fixed by a PR in the queue. The tests are failing because we monkeypatch internals in the parent process, but the monkeypatching is not done in the "spawned" child process.
2023-09-08build(deps): bump pytest from 7.4.1 to 7.4.2 in /lib/spack/docs (#39883)dependabot[bot]1-1/+1
Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.4.1 to 7.4.2. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst) - [Commits](https://github.com/pytest-dev/pytest/compare/7.4.1...7.4.2) --- updated-dependencies: - dependency-name: pytest dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-07Docs/Packaging guide: Add BundlePackage (#39691)Tamara Dahlgren2-12/+80
* Docs/Packaging guide: Add BundlePackage * Adjusted version ordering note to reflect convention.
2023-09-07CI Timing Statistics (#38598)kwryankrattiger5-38/+113
* Write timing information for installs from cache * CI: aggregate and upload install_times.json to artifacts * CI: Don't change root directory for artifact generation * Flat event based timer variation Event based timer allows for easily starting and stopping timers without wiping sub-timer data. It also requires less branching logic when tracking time. The json output is non-hierarchical in this version and hierarchy is less rigidly enforced between starting and stopping. * Add and write timers for top level install * Update completion * remove unused subtimer api * Fix unit tests * Suppress timing summary option * Save timers summaries to user_data artifacts * Remove completion from fish * Move spack python to script section * Write timer correctly for non-cache installs * Re-add hash to timer file * Fish completion updates * Fix null timer yield value * fix type hints * Remove timer-summary-file option * Add "." in front of non-package timer name --------- Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com> Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
2023-09-07move determine_number_of_jobs into spack.util.cpus, use it in concretize ↵Harmen Stoppels7-46/+86
(#37620)
2023-09-07CUDA 10.1/10.2: Fix conflicts with PGI 19+. (#39747)Rémi Lacroix1-1/+1
The condition probably did not get updated when the behavior of the version specifier changed.
2023-09-06Handles case of empty craype_dir (#39371)Dominic Hofer1-0/+2
The craype_dir folder can be empty. It happens to be the case on Arolla and Tsa at CSCS.
2023-09-06"spack config add": support values with ":" (#39831)John W. Parent2-8/+44
This is a fixed version of b72a268 * That commit would discard the final key component (so if you set "config:install_tree:root", it would discard "root" and just set install tree). * When setting key:"value", with the quotes, that commit would discard the quotes, which would confuse the system if adding a value like "{example}" (the "{" character indicates a dictionary). This commit retains the quotes.
2023-09-05CMakePackage: remove -DBUILD_TESTING flag (#37967)Adam J. Stewart1-2/+0