summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2023-02-16Style: black 23, skip magic trailing comma (#35351)Adam J. Stewart158-1650/+367
* Style: black 23, skip magic trailing commas * isort should use same line length as black * Fix unused import * Update version of black used in CI * Update new packages * Update new packages
2023-02-16view: fix issue with non-contributing specs (#34661)Harmen Stoppels3-13/+68
Specs that did not contribute any files to an env view caused a problem where zip(specs, files grouped by prefix) got "out of sync", causing the wrong merge map to be passed to a package's `add_files_to_view`, which specifically caused an issue where *sometimes* bin/python ended up as a symlink instead of a copy. One such example is kokkos + kokkos-nvcc-wrapper, as the latter package only provides the file bin/nvcc_wrapper, which is also added to view by kokkos, causing kokkos-nvcc-wrapper to contribute 0 files. The test feels a bit contrived, but it captures the problem... pkg a is added first and has 0 files to contribute, pkg b adds a single file, and we check if pkg b receives a merge map (and a does not).
2023-02-16simplify cdash filename (#35509)Harmen Stoppels1-8/+1
2023-02-16spack buildcache create: push all deps / cleanup (#34860)Harmen Stoppels2-38/+28
2023-02-16spack uninstall: follow run/link edges on --dependents (#34058)Harmen Stoppels2-42/+44
`spack gc` removes build deps of explicitly installed specs, but somehow if you take one of the specs that `spack gc` would remove, and feed it to `spack uninstall /<hash>` by hash, it complains about all the dependents that still rely on it. This resolves the inconsistency by only following run/link type deps in spack uninstall. That way you can finally do `spack uninstall cmake` without having to remove all packages built with cmake.
2023-02-16Avoid verifying variants in default package requirements (#35037)Massimiliano Culpo3-23/+58
Default package requirements might contain variants that are not defined in each package, so we shouldn't verify them when emitting facts for the ASP solver. Account for group when enforcing requirements packages:all : don't emit facts for requirement conditions that can't apply to current spec
2023-02-16buildcache create: avoid prefix copy (#35173)Harmen Stoppels1-37/+40
2023-02-15Use the `maintainers` directive in all base classes (#35324)Adam J. Stewart5-13/+22
* Use the `maintainers` directive in all base classes * Update unit tests
2023-02-14Update cce.py (#35469)luker1-0/+5
* Update cce.py adding c++17 flag for cce * Update cce.py
2023-02-10MSBuilder/MSVC: correct toolchain ver (#35424)John W. Parent1-1/+1
#35098 added the correct extraction of toolset version for the MSVC compiler. This updates the associated method in MSBuilder to retrieve the (now correct) property.
2023-02-10Restore our ability to submit build/test results to CDash from GitLab CI ↵Zack Galbreath3-14/+36
(#35328) * Restore our ability to submit build/test results to CDash from GitLab CI * Don't use CDash upload URL as report filename
2023-02-08spack help --spec: fix indentation (#35383)Adam J. Stewart1-2/+2
2023-02-08Ensure we print the correct branch number for tutorials (#35371)Massimiliano Culpo1-4/+4
2023-02-08bindist: use append as a method, not assignable attr (#35379)Scott Wittenburg1-4/+6
2023-02-07Windows: Fix spack.bat handling of env commands (#35143)Dan Lipsa1-9/+7
This PR enables the successful execution of the spack binary cache tutorial on Windows. It assumes gnupg and file are available (they can be installed with choco). * Fix handling of args with quotes in spack.bat * `file` utility can be installed on Windows (e.g. with choco): update error message accordingly
2023-02-06Fix path handling in prefix inspections (#35318)Matthias Wolf1-1/+1
At least with ZSH, prefix inspections containing `./bin` result in a `$PREFIX/./bin` and result in strange `$PATH` handling. I.e., `module load git` will prepend `/path/to/git/./bin`, `which git` will find the right executable, but `git --version` will print the system one. Normalize the relative path to avoid this behavior. See also spack/spack#31867.
2023-02-06GenericBuilder: facilitate post-install phase test callbacks (#35314)Tamara Dahlgren1-2/+11
2023-02-05Don't use CDash upload URL as report filename (#35338)Massimiliano Culpo4-31/+55
fixes #35337
2023-02-03MSVC compiler: add platform toolset version (#35098)John W. Parent1-2/+13
2023-02-02NMake builder: fix incorrect variable reference (#34937)John W. Parent1-1/+1
2023-02-01Use the `maintainers` directive in all packages (#35201)Massimiliano Culpo3-2/+8
2023-02-01Pin black to 22.12.0 to avoid spurious style changes (#35282)Massimiliano Culpo1-1/+1
We need to sync changes on major versions with spackbot, to avoid the bot saying everything is allright and have CI failing nonetheless.
2023-01-27Add a `maintainers` directive (#35083)Massimiliano Culpo6-24/+95
fixes #34879 This commit adds a new maintainer directive, which by default extend the list of maintainers for a given package. The directive is backward compatible with the current practice of having a "maintainers" list declared at the class level.
2023-01-26Cleanup of binary text relocation (#34188)Harmen Stoppels8-484/+561
Move the relocation of binary text in its own class Drop threaded text replacement, since the current bottleneck is decompression. It would be better to parallellize over packages, instead of over files per package. A small improvement with separate classes for text replacement is that we now compile the regex in the constructor; previously it was compiled per binary to be relocated.
2023-01-26Extract functions to read spec files from different format (#35094)Massimiliano Culpo17-256/+374
This commit makes explicit the format version of the spec file we are reading from. Before there were different functions capable of reading some part of the spec file at multiple format versions. The decision was implicit, since checks were based on the structure of the JSON without ever checking a format version number. The refactor makes also explicit which spec file format is used by which database and lockfile format, since the information is stored in global mappings. To ensure we don't change the hash of old specs, JSON representations of specs have been added as data. A unit tests checks that we read the correct hash in, and that the hash stays the same when we re-serialize the spec using the most recent format version. Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
2023-01-26test/config.py: fix import issues (#35076)Harmen Stoppels1-3/+5
2023-01-25Print file summary on checksum validation failure (#35161)Harmen Stoppels2-4/+35
Currently we print "sha256 checksum failed for [file]. Expected X but got Y". This PR extends that message with file size and contents info: "... but got Y. File size = 123456 bytes. Contents = b'abc...def'" That way we can immediately see if the file was downloaded only partially, or if we downloaded a text page instead of a binary, etc. Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2023-01-24hip: add cuda variant (#33872)Cory Bloor1-1/+1
2023-01-24Avoid creating help.sh and test.sh in cwd (#35123)Massimiliano Culpo1-42/+35
When running unit-test the test/ci.py module is leaving garbage (help.sh, test.sh files) in the current working directory. This commit changes the current working directory to a temporary path before those files are created.
2023-01-23ParaView/VTK: Patch xdmf2 for HDF5 1.13 (#33930)kwryankrattiger1-1/+5
* ParaView/VTK: Patch xdmf2 for HDF5 1.13 * Meson: update meson build system for 0.64 * ParaView: Change HDF5 1.13 patch to 1.13.1/2 patchs * Mesa: Remove legacy mesa option from meson_args * Use append to assemble meson args
2023-01-24Update: spack help --spec (#35115)Richarda Butler1-0/+9
* Update variant & compiler propagation * Add the info for non boolean variants * Add example for multi variants
2023-01-23environments: don't replace relative view path with absolute path on ↵Harmen Stoppels4-4/+48
concretize/install (#34958) * environments: don't rewrite relative view path, expand path on cli ahead of time Currently if you have a spack.yaml that specifies a view by relative path, Spack expands it to an absolute path on `spack -e . install` and persists that to disk. This is rather annoying when you have a `spack.yaml` file inside a git repo, cause you want to use relative paths to make it relocatable, but you constantly have to undo the changes made to spack.yaml by Spack. So, as an alternative: 1. Always stick to paths as they are provided in spack.yaml, never replace them with a canonicalized version 2. Turn relative paths on the command line into absolute paths before storing to spack.yaml. This way you can do `spack env create --dir ./env --with-view ./view` and both `./env` and `./view` are resolved to the current working dir, as expected (not `./env/view`). This corresponds to the old behavior of `spack env create`. * create --with-view always takes a value
2023-01-21Add --exclude option to 'spack external find' (#35013)Alex Richert2-0/+24
* Add --exclude option to 'spack external find' to ignore user-specified external packages * Update bash completion arg order for external find Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2023-01-20Windows/testing: enable tests for "spack find" (#33588)markus-ferrell1-9/+21
2023-01-20Identify Windows support with tags (#35027)John W. Parent1-21/+8
All packages with explicit Windows support can be found with `spack list --tags=windows`. This also removes the documentation which explicitly lists supported packages on Windows (which is currently out of date and is now unnecessary with the added tags). Note that if a package does not appear in this list, it *may* still build on Windows, but it likely means that no explicit attempt has been made to support it.
2023-01-20Add type hints to DependencySpec (#35021)Massimiliano Culpo2-16/+18
* DependencySpec: add type hints * DependencySpec: make deptypes a keyword only argument
2023-01-19buildcache create: make "file exists" less verbose (#35019)Harmen Stoppels1-3/+3
Currently we print 3 lines of "file exist" warning per tarball, this is a bit excessive. Instead, it can be a simple single-line tty.warn message.
2023-01-19[py-numpy, py-scipy] Enable MKL & ARMpl (#34979)Stephen Sachs1-0/+2
2023-01-19depfile: --make-target-prefix -> --make-prefix (#35009)Harmen Stoppels3-21/+21
Since SPACK_PACKAGE_IDS is now also "namespaced" with <prefix>, it makes more sense to call the flag `--make-prefix` and alias the old flag `--make-target-prefix` to it.
2023-01-18Bugfix: External Python Extensions (#34202)Greg Becker4-20/+207
Normally when using external packages in concretization, Spack ignores all dependencies of the external. #33777 updated this logic to attach a Python Spec to external Python extensions (most py-* packages), but as implemented there were a couple issues: * this did not account for concretization groups and could generate multiple different python specs for a single DAG * in some cases this created a fake Python spec with insufficient details to be usable (concretization/installation of the extension would fail) This PR addresses both of these issues: * For environment specs that are concretized together, external python extensions in those specs will all be assigned the same Python spec * If Spack needs to "invent" a Python spec, then it will have all the needed details (e.g. compiler/architecture)
2023-01-18license year bump (#34921)Harmen Stoppels571-578/+578
* license bump year * fix black issues of modified files * mypy * fix 2021 -> 2023
2023-01-18depfile: variable with all identifiers (#34678)Harmen Stoppels3-1/+112
With the new variable [prefix/]SPACK_PACKAGE_IDS you can conveniently execute things after each successful install. For example push just-built packages to a buildcache ``` SPACK ?= spack export SPACK_COLOR = always MAKEFLAGS += -Orecurse MY_BUILDCACHE := $(CURDIR)/cache .PHONY: all clean all: push ifeq (,$(filter clean,$(MAKECMDGOALS))) include env.mk endif # the relevant part: push has *all* example/push/<pkg identifier> as prereqs push: $(addprefix example/push/,$(example/SPACK_PACKAGE_IDS)) $(SPACK) -e . buildcache update-index --directory $(MY_BUILDCACHE) $(info Pushed everything, yay!) # and each example/push/<pkg identifier> has the install target as prereq, # and the body can use target local $(HASH) and $(SPEC) variables to do # things, such as pushing to a build cache example/push/%: example/install/% @mkdir -p $(dir $@) $(SPACK) -e . buildcache create --allow-root --only=package --unsigned --directory $(MY_BUILDCACHE) /$(HASH) # push $(SPEC) @touch $@ spack.lock: spack.yaml $(SPACK) -e . concretize -f env.mk: spack.lock $(SPACK) -e . env depfile -o $@ --make-target-prefix example clean: rm -rf spack.lock env.mk example/ ``
2023-01-18Remove verbose warning message from _try_install_from_binary_cache (#34994)Harmen Stoppels2-38/+18
In the past we checked remote binary mirrors for existence of a spec before attempting to download it. That changed to only checking local copies of index.jsons (if available) to prioritize certain mirrors where we expect to find a tarball. That was faster for CI since fetching index.json and loading it just to order mirrors takes more time than just attempting to fetch tarballs -- and also if we have a direct hit there's no point to look at other mirrors. Long story short: the info message only makes sense in the old version of Spack, so it's better to remove it.
2023-01-18Bug fix for duplicate rpath errors on macOS when creating build caches (#34375)Dom Heinzeller1-3/+11
2023-01-18spack mirror create takes local paths only (#34992)Harmen Stoppels2-43/+14
spack mirror create cannot work with urls, so it shouldn't promote local paths to file urls.
2023-01-17CDashReporter: remove unused argument (#34869)Massimiliano Culpo4-82/+7
* CDashReporter: remove unused argument * Removed extract_ctest_test_data
2023-01-17Forward lookup of "test_log_file" and "test_failures" (#34882)Massimiliano Culpo4-22/+69
* Forward lookup of "test_log_file" and "test_failures" refers #34531 closes #34487 fixes #34440 * Add unit test * py-libensemble: fix tests * Support stand-alone tests with cached files as install-time tests Co-authored-by: Tamara Dahlgren <dahlgren1@llnl.gov>
2023-01-17Reduce verbosity in mirrors.yaml (#34210)roottreej4-24/+42
Ensure `spack mirror add <name> <url/path>` without further arguments translates to `<name>: <url>` key value pairs in mirrors.yaml. If --s3-* flags are provided, only store the provided ones. Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
2023-01-16Refer to mirrors by name, path, or url (#34891)Harmen Stoppels16-313/+340
With this change we get the invariant that `mirror.fetch_url` and `mirror.push_url` return valid URLs, even when the backing config file is actually using (relative) paths with potentially `$spack` and `$env` like variables. Secondly it avoids expanding mirror path / URLs too early, so if I say `spack mirror add name ./path`, it stays `./path` in my config. When it's retrieved through MirrorCollection() we exand it to say `file://<env dir>/path` if `./path` was set in an environment scope. Thirdly, the interface is simplified for the relevant buildcache commands, so it's more like `git push`: ``` spack buildcache create [mirror] [specs...] ``` `mirror` is either a mirror name, a path, or a URL. Resolving the relevant mirror goes as follows: - If it contains either / or \ it is used as an anonymous mirror with path or url. - Otherwise, it's interpreted as a named mirror, which must exist. This helps to guard against typos, e.g. typing `my-mirror` when there is no such named mirror now errors with: ``` $ spack -e . buildcache create my-mirror ==> Error: no mirror named "my-mirror". Did you mean ./my-mirror? ``` instead of creating a directory in the current working directory. I think this is reasonable, as the alternative (requiring that a local dir exists) feels a bit pendantic in the general case -- spack is happy to create the build cache dir when needed, saving a `mkdir`. The old (now deprecated) format will still be available in Spack 0.20, but is scheduled to be removed in 0.21: ``` spack buildcache create (--directory | --mirror-url | --mirror-name) [specs...] ``` This PR also touches `tmp_scope` in tests, because it didn't really work for me, since spack fixes the possible --scope values once and for all across tests, so tests failed when run out of order.
2023-01-16fix libtool filter for Fujitsu compilers (#34916)Gilles Gouaillardet1-8/+8
Since legit regular expressions are matched, do not pass string=True to the filter Refs #34897