summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2023-01-21Add --exclude option to 'spack external find' (#35013)Alex Richert2-0/+24
* Add --exclude option to 'spack external find' to ignore user-specified external packages * Update bash completion arg order for external find Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2023-01-20Windows/testing: enable tests for "spack find" (#33588)markus-ferrell1-9/+21
2023-01-20Identify Windows support with tags (#35027)John W. Parent1-21/+8
All packages with explicit Windows support can be found with `spack list --tags=windows`. This also removes the documentation which explicitly lists supported packages on Windows (which is currently out of date and is now unnecessary with the added tags). Note that if a package does not appear in this list, it *may* still build on Windows, but it likely means that no explicit attempt has been made to support it.
2023-01-20Add type hints to DependencySpec (#35021)Massimiliano Culpo2-16/+18
* DependencySpec: add type hints * DependencySpec: make deptypes a keyword only argument
2023-01-19buildcache create: make "file exists" less verbose (#35019)Harmen Stoppels1-3/+3
Currently we print 3 lines of "file exist" warning per tarball, this is a bit excessive. Instead, it can be a simple single-line tty.warn message.
2023-01-19[py-numpy, py-scipy] Enable MKL & ARMpl (#34979)Stephen Sachs1-0/+2
2023-01-19depfile: --make-target-prefix -> --make-prefix (#35009)Harmen Stoppels3-21/+21
Since SPACK_PACKAGE_IDS is now also "namespaced" with <prefix>, it makes more sense to call the flag `--make-prefix` and alias the old flag `--make-target-prefix` to it.
2023-01-18Bugfix: External Python Extensions (#34202)Greg Becker4-20/+207
Normally when using external packages in concretization, Spack ignores all dependencies of the external. #33777 updated this logic to attach a Python Spec to external Python extensions (most py-* packages), but as implemented there were a couple issues: * this did not account for concretization groups and could generate multiple different python specs for a single DAG * in some cases this created a fake Python spec with insufficient details to be usable (concretization/installation of the extension would fail) This PR addresses both of these issues: * For environment specs that are concretized together, external python extensions in those specs will all be assigned the same Python spec * If Spack needs to "invent" a Python spec, then it will have all the needed details (e.g. compiler/architecture)
2023-01-18license year bump (#34921)Harmen Stoppels571-578/+578
* license bump year * fix black issues of modified files * mypy * fix 2021 -> 2023
2023-01-18depfile: variable with all identifiers (#34678)Harmen Stoppels3-1/+112
With the new variable [prefix/]SPACK_PACKAGE_IDS you can conveniently execute things after each successful install. For example push just-built packages to a buildcache ``` SPACK ?= spack export SPACK_COLOR = always MAKEFLAGS += -Orecurse MY_BUILDCACHE := $(CURDIR)/cache .PHONY: all clean all: push ifeq (,$(filter clean,$(MAKECMDGOALS))) include env.mk endif # the relevant part: push has *all* example/push/<pkg identifier> as prereqs push: $(addprefix example/push/,$(example/SPACK_PACKAGE_IDS)) $(SPACK) -e . buildcache update-index --directory $(MY_BUILDCACHE) $(info Pushed everything, yay!) # and each example/push/<pkg identifier> has the install target as prereq, # and the body can use target local $(HASH) and $(SPEC) variables to do # things, such as pushing to a build cache example/push/%: example/install/% @mkdir -p $(dir $@) $(SPACK) -e . buildcache create --allow-root --only=package --unsigned --directory $(MY_BUILDCACHE) /$(HASH) # push $(SPEC) @touch $@ spack.lock: spack.yaml $(SPACK) -e . concretize -f env.mk: spack.lock $(SPACK) -e . env depfile -o $@ --make-target-prefix example clean: rm -rf spack.lock env.mk example/ ``
2023-01-18Remove verbose warning message from _try_install_from_binary_cache (#34994)Harmen Stoppels2-38/+18
In the past we checked remote binary mirrors for existence of a spec before attempting to download it. That changed to only checking local copies of index.jsons (if available) to prioritize certain mirrors where we expect to find a tarball. That was faster for CI since fetching index.json and loading it just to order mirrors takes more time than just attempting to fetch tarballs -- and also if we have a direct hit there's no point to look at other mirrors. Long story short: the info message only makes sense in the old version of Spack, so it's better to remove it.
2023-01-18Bug fix for duplicate rpath errors on macOS when creating build caches (#34375)Dom Heinzeller1-3/+11
2023-01-18spack mirror create takes local paths only (#34992)Harmen Stoppels2-43/+14
spack mirror create cannot work with urls, so it shouldn't promote local paths to file urls.
2023-01-17CDashReporter: remove unused argument (#34869)Massimiliano Culpo4-82/+7
* CDashReporter: remove unused argument * Removed extract_ctest_test_data
2023-01-17Forward lookup of "test_log_file" and "test_failures" (#34882)Massimiliano Culpo4-22/+69
* Forward lookup of "test_log_file" and "test_failures" refers #34531 closes #34487 fixes #34440 * Add unit test * py-libensemble: fix tests * Support stand-alone tests with cached files as install-time tests Co-authored-by: Tamara Dahlgren <dahlgren1@llnl.gov>
2023-01-17Reduce verbosity in mirrors.yaml (#34210)roottreej4-24/+42
Ensure `spack mirror add <name> <url/path>` without further arguments translates to `<name>: <url>` key value pairs in mirrors.yaml. If --s3-* flags are provided, only store the provided ones. Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
2023-01-16Refer to mirrors by name, path, or url (#34891)Harmen Stoppels16-313/+340
With this change we get the invariant that `mirror.fetch_url` and `mirror.push_url` return valid URLs, even when the backing config file is actually using (relative) paths with potentially `$spack` and `$env` like variables. Secondly it avoids expanding mirror path / URLs too early, so if I say `spack mirror add name ./path`, it stays `./path` in my config. When it's retrieved through MirrorCollection() we exand it to say `file://<env dir>/path` if `./path` was set in an environment scope. Thirdly, the interface is simplified for the relevant buildcache commands, so it's more like `git push`: ``` spack buildcache create [mirror] [specs...] ``` `mirror` is either a mirror name, a path, or a URL. Resolving the relevant mirror goes as follows: - If it contains either / or \ it is used as an anonymous mirror with path or url. - Otherwise, it's interpreted as a named mirror, which must exist. This helps to guard against typos, e.g. typing `my-mirror` when there is no such named mirror now errors with: ``` $ spack -e . buildcache create my-mirror ==> Error: no mirror named "my-mirror". Did you mean ./my-mirror? ``` instead of creating a directory in the current working directory. I think this is reasonable, as the alternative (requiring that a local dir exists) feels a bit pendantic in the general case -- spack is happy to create the build cache dir when needed, saving a `mkdir`. The old (now deprecated) format will still be available in Spack 0.20, but is scheduled to be removed in 0.21: ``` spack buildcache create (--directory | --mirror-url | --mirror-name) [specs...] ``` This PR also touches `tmp_scope` in tests, because it didn't really work for me, since spack fixes the possible --scope values once and for all across tests, so tests failed when run out of order.
2023-01-16fix libtool filter for Fujitsu compilers (#34916)Gilles Gouaillardet1-8/+8
Since legit regular expressions are matched, do not pass string=True to the filter Refs #34897
2023-01-16Fix incorrect configuration file name in docs (#34925)Jack Morrison1-1/+1
2023-01-16Drop Python 3.6 again (#34442)Adam J. Stewart1-1/+1
2023-01-14spack list: add `--count` option (#34950)Todd Gamblin2-8/+31
Sometimes I just want to know how many packages of a certain type there are. - [x] add `--count` option to `spack list` that output the number of packages that *would* be listed. ```console > spack list --count 6864 > spack list --count py- 2040 > spack list --count r- 1162 ```
2023-01-14Support windows paths in a spec (#34405)John W. Parent2-21/+146
Refactor Spack's spec syntax parsing to handle Windows style paths Amend unit tests to reflect this updated behavior.
2023-01-13config: make path replacements lazy (#34758)Todd Gamblin1-23/+23
Currently, all of the replacements in `spack.util.path.replacements()` get evaluated for each replacement. This makes it easy to get bootstrap issues, because config is used very early on in Spack. Right now, if I run `test_autotools_gnuconfig_replacement_no_gnuconfig` on my M1 mac, I get the circular reference error below. This fixes the issue by making all of the path replacements lazy lambdas. As a bonus, this cleans up the way we do substitution for `$env` -- it's consistent with other substitutions now. - [x] make all path `replacements()` lazy - [x] clean up handling of `$env` ```console > spack unit-test -k test_autotools_gnuconfig_replacement_no_gnuconfig ... ==> [2022-12-31-15:44:21.771459] Error: AttributeError: The 'autotools-config-replacement' package cannot find an attribute while trying to build from sources. This might be due to a change in Spack's package format to support multiple build-systems for a single package. You can fix this by updating the build recipe, and you can also report the issue as a bug. More information at https://spack.readthedocs.io/en/latest/packaging_guide.html#installation-procedure /Users/gamblin2/src/spack/lib/spack/spack/package_base.py:1332, in prefix: 1330 @property 1331 def prefix(self): >> 1332 """Get the prefix into which this package should be installed.""" 1333 return self.spec.prefix Traceback (most recent call last): File "/Users/gamblin2/src/spack/lib/spack/spack/build_environment.py", line 1030, in _setup_pkg_and_run kwargs["env_modifications"] = setup_package( ^^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/build_environment.py", line 757, in setup_package set_module_variables_for_package(pkg) File "/Users/gamblin2/src/spack/lib/spack/spack/build_environment.py", line 596, in set_module_variables_for_package m.std_cmake_args = spack.build_systems.cmake.CMakeBuilder.std_args(pkg) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/build_systems/cmake.py", line 241, in std_args define("CMAKE_INSTALL_PREFIX", pkg.prefix), ^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/package_base.py", line 1333, in prefix return self.spec.prefix ^^^^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/spec.py", line 1710, in prefix self.prefix = spack.store.layout.path_for_spec(self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/directory_layout.py", line 336, in path_for_spec path = self.relative_path_for_spec(spec) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/directory_layout.py", line 106, in relative_path_for_spec projection = spack.projections.get_projection(self.projections, spec) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/projections.py", line 13, in get_projection if spec.satisfies(spec_like, strict=True): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/spec.py", line 3642, in satisfies if not self.virtual and other.virtual: ^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/spec.py", line 1622, in virtual return spack.repo.path.is_virtual(self.name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/repo.py", line 890, in is_virtual return have_name and pkg_name in self.provider_index ^^^^^^^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/repo.py", line 770, in provider_index self._provider_index.merge(repo.provider_index) ^^^^^^^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/repo.py", line 1096, in provider_index return self.index["providers"] ~~~~~~~~~~^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/repo.py", line 592, in __getitem__ self._build_all_indexes() File "/Users/gamblin2/src/spack/lib/spack/spack/repo.py", line 607, in _build_all_indexes self.indexes[name] = self._build_index(name, indexer) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/repo.py", line 616, in _build_index index_mtime = self.cache.mtime(cache_filename) ^^^^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/llnl/util/lang.py", line 826, in __getattr__ return getattr(self.instance, name) ^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/llnl/util/lang.py", line 825, in __getattr__ raise AttributeError() AttributeError ```
2023-01-12Documentation: Add installation policy to packaging guide (#34878)Tamara Dahlgren1-0/+9
2023-01-11Fix issue when generating compiler hierarchy (#34873)Massimiliano Culpo1-1/+1
- [x] Fix bug affecting generation of Lmod modules
2023-01-10Build systems: add MSBuild and update NMake (#34659)John W. Parent4-53/+225
Add/update build systems used to build packages on Windows.
2023-01-10Remove **kwargs from function signatures in llnl.util.filesystem (#34804)Massimiliano Culpo1-69/+119
Since we dropped support for Python 2.7, we can embrace using keyword only arguments for many functions in Spack that use **kwargs in the function signature. Here this is done for the llnl.util.filesystem module. There were a couple of bugs lurking in the code related to typo-like errors when retrieving from kwargs. Those have been fixed as well.
2023-01-10style: Run mypy last (#34687)Jonathon Anderson1-1/+1
2023-01-10FileCache: Delete the new cache file on exception (#34623)Jonathon Anderson2-2/+22
The code in FileCache for write_transaction attempts to delete the temporary file when an exception occurs under the context by calling shutil.rmtree. However, rmtree only operates on directories while the rest of FileCache uses normal files. This causes an empty file to be written to the cache key when unneeded. Use os.remove instead which operates on normal files. Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2023-01-10Add CUDA 12.0 (#34664)Wileam Y. Phan1-1/+2
2023-01-09Support ASCI controlJohn Parent2-0/+62
Windows CMD prompt does not automatically support ASCI color control characters on the console from Python. Enable this behavior by accessing the current console and allowing the interpreation of ASCI control characters from Python via the win32 API.
2023-01-09Revert "Allow spec.json files to be clearsigned, use transparent compression ↵Harmen Stoppels5-255/+158
for *.json (#34490)" (#34856) This reverts commit 8a1b81797834dd04de883c767867e2f85d45c9c4.
2023-01-07Allow spec.json files to be clearsigned, use transparent compression for ↵Harmen Stoppels5-158/+255
*.json (#34490) This commit allows (remote) spec.json files to be clearsigned and gzipped. The idea is to reduce the number of requests and number of bytes transferred
2023-01-06Fix PrgEnv detection (#34845)QuellynSnead1-1/+1
* Fix PrgEnv detection for Cray compiler link paths
2023-01-05spack.dependency: add type hints (#34807)Massimiliano Culpo2-18/+27
2023-01-04Fix building docsMassimiliano Culpo2-2/+9
2023-01-04Add vendored packages backMassimiliano Culpo154-0/+41107
2023-01-04Delete outdated externalsMassimiliano Culpo187-56425/+0
2023-01-04Use "vendoring" to manage 3rd party dependenciesMassimiliano Culpo9-66/+50
2023-01-04Remove dead code that was needed for the old parser (#34792)Massimiliano Culpo1-7/+0
The old to token definitions and spec_id_re regular expression are not used anymore
2023-01-02environment view use new traversal (#34662)Harmen Stoppels1-11/+11
2022-12-30Simplify creation of test and install reports (#34712)Massimiliano Culpo11-322/+368
The code in Spack to generate install and test reports currently suffers from unneeded complexity. For instance, we have classes in Spack core packages, like `spack.reporters.CDash`, that need an `argparse.Namespace` to be initialized and have "hard-coded" string literals on which they branch to change their behavior: ```python if do_fn.__name__ == "do_test" and skip_externals: package["result"] = "skipped" else: package["result"] = "success" package["stdout"] = fetch_log(pkg, do_fn, self.dir) package["installed_from_binary_cache"] = pkg.installed_from_binary_cache if do_fn.__name__ == "_install_task" and installed_already: return ``` This PR attempt to polish the major issues encountered in both `spack.report` and `spack.reporters`. Details: - [x] `spack.reporters` is now a package that contains both the base class `Reporter` and all the derived classes (`JUnit` and `CDash`) - [x] Classes derived from `spack.reporters.Reporter` don't take an `argparse.Namespace` anymore as argument to `__init__`. The rationale is that code for commands should be built upon Spack core classes, not vice-versa. - [x] An `argparse.Action` has been coded to create the correct `Reporter` object based on command line arguments - [x] The context managers to generate reports from either `spack install` or from `spack test` have been greatly simplified, and have been made less "dynamic" in nature. In particular, the `collect_info` class has been deleted in favor of two more specific context managers. This allows for a simpler structure of the code, and less knowledge required to client code (in particular on which method to patch) - [x] The `InfoCollector` class has been turned into a simple hierarchy, so to avoid conditional statements within methods that assume a knowledge of the context in which the method is called.
2022-12-30bugfix: setgid tests fail when primary group is unknown (#34729)Todd Gamblin1-0/+14
On systems with remote groups, the primary user group may be remote and may not exist on the local system (i.e., it might just be a number). On the CLI, it looks like this: ```console > touch foo > l foo -rw-r--r-- 1 gamblin2 57095 0 Dec 29 22:24 foo > chmod 2000 foo chmod: changing permissions of 'foo': Operation not permitted ``` Here, the local machine doesn't know about per-user groups, so they appear as gids in `ls` output. `57095` is also `gamblin2`'s uid, but the local machine doesn't know that `gamblin2` is in the `57095` group. Unfortunately, it seems that Python's `os.chmod()` just fails silently, setting permissions to `0o0000` instead of `0o2000`. We can avoid this by ensuring that the file has a group the user is known to be a member of. - [x] Add `ensure_known_group()` in the permissions tests. - [x] Call `ensure_known_group()` on tempfile in `test_chmod_real_entries_ignores_suid_sgid`.
2022-12-29docs: avoid errors by using type hints instead of doc types (#34707)Todd Gamblin13-83/+133
There are a number of places in our docstrings where we write "list of X" as the type, even though napoleon doesn't actually support this. It ends up causing warnings when generating docs. Now that we require Python 3, we don't have to rely on type hints in docs -- we can just use Python type hints and omit the types of args and return values from docstrings. We should probably do this for all types in docstrings eventually, but this PR focuses on the ones that generate warnings during doc builds. Some `mypy` annoyances we should consider in the future: 1. Adding some of these type annotations gets you: ``` note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] ``` because they are in unannotated functions (like constructors where we don't really need any annotations). You can silence these with `disable_error_code = "annotation-unchecked"` in `pyproject.toml` 2. Right now we support running `mypy` in Python `3.6`. That means we have to support `mypy` `.971`, which does not support `disable_error_code = "annotation-unchecked"`, so I just filter `[annotation-unchecked]` lines out in `spack style`. 3. I would rather just turn on `check_untyped_defs` and get more `mypy` coverage everywhere, but that will require about 1,000 fixes. We should probably do that eventually. 4. We could also consider only running `mypy` on newer python versions. This is not easy to do while supporting `3.6`, because you have to use `if TYPE_CHECKING` for a lot of things to ensure that 3.6 still parses correctly. If we only supported `3.7` and above we could use [`from __future__ import annotations`](https://mypy.readthedocs.io/en/stable/runtime_troubles.html#future-annotations-import-pep-563), but we have to support 3.6 for now. Sigh. - [x] Convert a number of docstring types to Python type hints - [x] Get rid of "list of" wherever it appears
2022-12-28Consolidate how Spack uses `git` (#34700)Todd Gamblin24-154/+138
Local `git` tests will fail with `fatal: transport 'file' not allowed` when using git 2.38.1 or higher, due to a fix for `CVE-2022-39253`. This was fixed in CI in #33429, but that doesn't help the issue for anyone's local environment. Instead of fixing this with git config in CI, we should ensure that the tests run anywhere. - [x] Introduce `spack.util.git`. - [x] Use `spack.util.git.get_git()` to get a git executable, instead of `which("git")` everywhere. - [x] Make all `git` tests use a `git` fixture that goes through `spack.util.git.get_git()`. - [x] Add `-c protocol.file.allow=always` to all `git` invocations under `pytest`. - [x] Revert changes from #33429, which are no longer needed.
2022-12-27spack graph: rework to use Jinja templates and builders (#34637)Massimiliano Culpo4-213/+219
`spack graph` has been reworked to use: - Jinja templates - builder objects to construct the template context when DOT graphs are requested. This allowed to add a new colored output for DOT graphs that highlights both the dependency types and the nodes that are needed at runtime for a given spec.
2022-12-26types: fix type annotations and remove novm annootations for llnl moduleTodd Gamblin2-18/+13
Apparently I forgot to do this in #34305.
2022-12-26timer: pick a single unit based on max duration.Harmen Stoppels3-17/+25
2022-12-26`spack solve`: use consistent units for timeTodd Gamblin2-5/+5
`spack solve` is supposed to show you times you can compare. setup, ground, solve, etc. all in a list. You're also supposed to be able to compare easily across runs. With `pretty_seconds()` (introduced in #33900), it's easy to miss the units, e.g., spot the bottleneck here: ```console > spack solve --timers tcl setup 22.125ms load 16.083ms ground 8.298ms solve 848.055us total 58.615ms ``` It's easier to see what matters if these are all in the same units, e.g.: ``` > spack solve --timers tcl setup 0.0147s load 0.0130s ground 0.0078s solve 0.0008s total 0.0463s ``` And the units won't fluctuate from run to run as you make changes. -[x] make `spack solve` timings consistent like before
2022-12-23QMakeBuilder: fix bug introduced during multi-bs refactor (#34683)Adam J. Stewart1-1/+1