summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2023-01-14Support windows paths in a spec (#34405)John W. Parent2-21/+146
Refactor Spack's spec syntax parsing to handle Windows style paths Amend unit tests to reflect this updated behavior.
2023-01-13config: make path replacements lazy (#34758)Todd Gamblin1-23/+23
Currently, all of the replacements in `spack.util.path.replacements()` get evaluated for each replacement. This makes it easy to get bootstrap issues, because config is used very early on in Spack. Right now, if I run `test_autotools_gnuconfig_replacement_no_gnuconfig` on my M1 mac, I get the circular reference error below. This fixes the issue by making all of the path replacements lazy lambdas. As a bonus, this cleans up the way we do substitution for `$env` -- it's consistent with other substitutions now. - [x] make all path `replacements()` lazy - [x] clean up handling of `$env` ```console > spack unit-test -k test_autotools_gnuconfig_replacement_no_gnuconfig ... ==> [2022-12-31-15:44:21.771459] Error: AttributeError: The 'autotools-config-replacement' package cannot find an attribute while trying to build from sources. This might be due to a change in Spack's package format to support multiple build-systems for a single package. You can fix this by updating the build recipe, and you can also report the issue as a bug. More information at https://spack.readthedocs.io/en/latest/packaging_guide.html#installation-procedure /Users/gamblin2/src/spack/lib/spack/spack/package_base.py:1332, in prefix: 1330 @property 1331 def prefix(self): >> 1332 """Get the prefix into which this package should be installed.""" 1333 return self.spec.prefix Traceback (most recent call last): File "/Users/gamblin2/src/spack/lib/spack/spack/build_environment.py", line 1030, in _setup_pkg_and_run kwargs["env_modifications"] = setup_package( ^^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/build_environment.py", line 757, in setup_package set_module_variables_for_package(pkg) File "/Users/gamblin2/src/spack/lib/spack/spack/build_environment.py", line 596, in set_module_variables_for_package m.std_cmake_args = spack.build_systems.cmake.CMakeBuilder.std_args(pkg) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/build_systems/cmake.py", line 241, in std_args define("CMAKE_INSTALL_PREFIX", pkg.prefix), ^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/package_base.py", line 1333, in prefix return self.spec.prefix ^^^^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/spec.py", line 1710, in prefix self.prefix = spack.store.layout.path_for_spec(self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/directory_layout.py", line 336, in path_for_spec path = self.relative_path_for_spec(spec) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/directory_layout.py", line 106, in relative_path_for_spec projection = spack.projections.get_projection(self.projections, spec) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/projections.py", line 13, in get_projection if spec.satisfies(spec_like, strict=True): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/spec.py", line 3642, in satisfies if not self.virtual and other.virtual: ^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/spec.py", line 1622, in virtual return spack.repo.path.is_virtual(self.name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/repo.py", line 890, in is_virtual return have_name and pkg_name in self.provider_index ^^^^^^^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/repo.py", line 770, in provider_index self._provider_index.merge(repo.provider_index) ^^^^^^^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/repo.py", line 1096, in provider_index return self.index["providers"] ~~~~~~~~~~^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/repo.py", line 592, in __getitem__ self._build_all_indexes() File "/Users/gamblin2/src/spack/lib/spack/spack/repo.py", line 607, in _build_all_indexes self.indexes[name] = self._build_index(name, indexer) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/spack/repo.py", line 616, in _build_index index_mtime = self.cache.mtime(cache_filename) ^^^^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/llnl/util/lang.py", line 826, in __getattr__ return getattr(self.instance, name) ^^^^^^^^^^^^^ File "/Users/gamblin2/src/spack/lib/spack/llnl/util/lang.py", line 825, in __getattr__ raise AttributeError() AttributeError ```
2023-01-12Documentation: Add installation policy to packaging guide (#34878)Tamara Dahlgren1-0/+9
2023-01-11Fix issue when generating compiler hierarchy (#34873)Massimiliano Culpo1-1/+1
- [x] Fix bug affecting generation of Lmod modules
2023-01-10Build systems: add MSBuild and update NMake (#34659)John W. Parent4-53/+225
Add/update build systems used to build packages on Windows.
2023-01-10Remove **kwargs from function signatures in llnl.util.filesystem (#34804)Massimiliano Culpo1-69/+119
Since we dropped support for Python 2.7, we can embrace using keyword only arguments for many functions in Spack that use **kwargs in the function signature. Here this is done for the llnl.util.filesystem module. There were a couple of bugs lurking in the code related to typo-like errors when retrieving from kwargs. Those have been fixed as well.
2023-01-10style: Run mypy last (#34687)Jonathon Anderson1-1/+1
2023-01-10FileCache: Delete the new cache file on exception (#34623)Jonathon Anderson2-2/+22
The code in FileCache for write_transaction attempts to delete the temporary file when an exception occurs under the context by calling shutil.rmtree. However, rmtree only operates on directories while the rest of FileCache uses normal files. This causes an empty file to be written to the cache key when unneeded. Use os.remove instead which operates on normal files. Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2023-01-10Add CUDA 12.0 (#34664)Wileam Y. Phan1-1/+2
2023-01-09Support ASCI controlJohn Parent2-0/+62
Windows CMD prompt does not automatically support ASCI color control characters on the console from Python. Enable this behavior by accessing the current console and allowing the interpreation of ASCI control characters from Python via the win32 API.
2023-01-09Revert "Allow spec.json files to be clearsigned, use transparent compression ↵Harmen Stoppels5-255/+158
for *.json (#34490)" (#34856) This reverts commit 8a1b81797834dd04de883c767867e2f85d45c9c4.
2023-01-07Allow spec.json files to be clearsigned, use transparent compression for ↵Harmen Stoppels5-158/+255
*.json (#34490) This commit allows (remote) spec.json files to be clearsigned and gzipped. The idea is to reduce the number of requests and number of bytes transferred
2023-01-06Fix PrgEnv detection (#34845)QuellynSnead1-1/+1
* Fix PrgEnv detection for Cray compiler link paths
2023-01-05spack.dependency: add type hints (#34807)Massimiliano Culpo2-18/+27
2023-01-04Fix building docsMassimiliano Culpo2-2/+9
2023-01-04Add vendored packages backMassimiliano Culpo154-0/+41107
2023-01-04Delete outdated externalsMassimiliano Culpo187-56425/+0
2023-01-04Use "vendoring" to manage 3rd party dependenciesMassimiliano Culpo9-66/+50
2023-01-04Remove dead code that was needed for the old parser (#34792)Massimiliano Culpo1-7/+0
The old to token definitions and spec_id_re regular expression are not used anymore
2023-01-02environment view use new traversal (#34662)Harmen Stoppels1-11/+11
2022-12-30Simplify creation of test and install reports (#34712)Massimiliano Culpo11-322/+368
The code in Spack to generate install and test reports currently suffers from unneeded complexity. For instance, we have classes in Spack core packages, like `spack.reporters.CDash`, that need an `argparse.Namespace` to be initialized and have "hard-coded" string literals on which they branch to change their behavior: ```python if do_fn.__name__ == "do_test" and skip_externals: package["result"] = "skipped" else: package["result"] = "success" package["stdout"] = fetch_log(pkg, do_fn, self.dir) package["installed_from_binary_cache"] = pkg.installed_from_binary_cache if do_fn.__name__ == "_install_task" and installed_already: return ``` This PR attempt to polish the major issues encountered in both `spack.report` and `spack.reporters`. Details: - [x] `spack.reporters` is now a package that contains both the base class `Reporter` and all the derived classes (`JUnit` and `CDash`) - [x] Classes derived from `spack.reporters.Reporter` don't take an `argparse.Namespace` anymore as argument to `__init__`. The rationale is that code for commands should be built upon Spack core classes, not vice-versa. - [x] An `argparse.Action` has been coded to create the correct `Reporter` object based on command line arguments - [x] The context managers to generate reports from either `spack install` or from `spack test` have been greatly simplified, and have been made less "dynamic" in nature. In particular, the `collect_info` class has been deleted in favor of two more specific context managers. This allows for a simpler structure of the code, and less knowledge required to client code (in particular on which method to patch) - [x] The `InfoCollector` class has been turned into a simple hierarchy, so to avoid conditional statements within methods that assume a knowledge of the context in which the method is called.
2022-12-30bugfix: setgid tests fail when primary group is unknown (#34729)Todd Gamblin1-0/+14
On systems with remote groups, the primary user group may be remote and may not exist on the local system (i.e., it might just be a number). On the CLI, it looks like this: ```console > touch foo > l foo -rw-r--r-- 1 gamblin2 57095 0 Dec 29 22:24 foo > chmod 2000 foo chmod: changing permissions of 'foo': Operation not permitted ``` Here, the local machine doesn't know about per-user groups, so they appear as gids in `ls` output. `57095` is also `gamblin2`'s uid, but the local machine doesn't know that `gamblin2` is in the `57095` group. Unfortunately, it seems that Python's `os.chmod()` just fails silently, setting permissions to `0o0000` instead of `0o2000`. We can avoid this by ensuring that the file has a group the user is known to be a member of. - [x] Add `ensure_known_group()` in the permissions tests. - [x] Call `ensure_known_group()` on tempfile in `test_chmod_real_entries_ignores_suid_sgid`.
2022-12-29docs: avoid errors by using type hints instead of doc types (#34707)Todd Gamblin13-83/+133
There are a number of places in our docstrings where we write "list of X" as the type, even though napoleon doesn't actually support this. It ends up causing warnings when generating docs. Now that we require Python 3, we don't have to rely on type hints in docs -- we can just use Python type hints and omit the types of args and return values from docstrings. We should probably do this for all types in docstrings eventually, but this PR focuses on the ones that generate warnings during doc builds. Some `mypy` annoyances we should consider in the future: 1. Adding some of these type annotations gets you: ``` note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] ``` because they are in unannotated functions (like constructors where we don't really need any annotations). You can silence these with `disable_error_code = "annotation-unchecked"` in `pyproject.toml` 2. Right now we support running `mypy` in Python `3.6`. That means we have to support `mypy` `.971`, which does not support `disable_error_code = "annotation-unchecked"`, so I just filter `[annotation-unchecked]` lines out in `spack style`. 3. I would rather just turn on `check_untyped_defs` and get more `mypy` coverage everywhere, but that will require about 1,000 fixes. We should probably do that eventually. 4. We could also consider only running `mypy` on newer python versions. This is not easy to do while supporting `3.6`, because you have to use `if TYPE_CHECKING` for a lot of things to ensure that 3.6 still parses correctly. If we only supported `3.7` and above we could use [`from __future__ import annotations`](https://mypy.readthedocs.io/en/stable/runtime_troubles.html#future-annotations-import-pep-563), but we have to support 3.6 for now. Sigh. - [x] Convert a number of docstring types to Python type hints - [x] Get rid of "list of" wherever it appears
2022-12-28Consolidate how Spack uses `git` (#34700)Todd Gamblin24-154/+138
Local `git` tests will fail with `fatal: transport 'file' not allowed` when using git 2.38.1 or higher, due to a fix for `CVE-2022-39253`. This was fixed in CI in #33429, but that doesn't help the issue for anyone's local environment. Instead of fixing this with git config in CI, we should ensure that the tests run anywhere. - [x] Introduce `spack.util.git`. - [x] Use `spack.util.git.get_git()` to get a git executable, instead of `which("git")` everywhere. - [x] Make all `git` tests use a `git` fixture that goes through `spack.util.git.get_git()`. - [x] Add `-c protocol.file.allow=always` to all `git` invocations under `pytest`. - [x] Revert changes from #33429, which are no longer needed.
2022-12-27spack graph: rework to use Jinja templates and builders (#34637)Massimiliano Culpo4-213/+219
`spack graph` has been reworked to use: - Jinja templates - builder objects to construct the template context when DOT graphs are requested. This allowed to add a new colored output for DOT graphs that highlights both the dependency types and the nodes that are needed at runtime for a given spec.
2022-12-26types: fix type annotations and remove novm annootations for llnl moduleTodd Gamblin2-18/+13
Apparently I forgot to do this in #34305.
2022-12-26timer: pick a single unit based on max duration.Harmen Stoppels3-17/+25
2022-12-26`spack solve`: use consistent units for timeTodd Gamblin2-5/+5
`spack solve` is supposed to show you times you can compare. setup, ground, solve, etc. all in a list. You're also supposed to be able to compare easily across runs. With `pretty_seconds()` (introduced in #33900), it's easy to miss the units, e.g., spot the bottleneck here: ```console > spack solve --timers tcl setup 22.125ms load 16.083ms ground 8.298ms solve 848.055us total 58.615ms ``` It's easier to see what matters if these are all in the same units, e.g.: ``` > spack solve --timers tcl setup 0.0147s load 0.0130s ground 0.0078s solve 0.0008s total 0.0463s ``` And the units won't fluctuate from run to run as you make changes. -[x] make `spack solve` timings consistent like before
2022-12-23QMakeBuilder: fix bug introduced during multi-bs refactor (#34683)Adam J. Stewart1-1/+1
2022-12-22Fix combine_phase_logs text encoding issues (#34657)Harmen Stoppels2-5/+20
Avoid text decoding and encoding when combining log files, instead combine in binary mode. Also do a buffered copy which is sometimes faster for large log files.
2022-12-22Add `--fresh` to docs to actually upgrade spack environments (#34433)Alec Scott1-5/+40
2022-12-22docs: show module documentation before submodules (#34258)Todd Gamblin1-0/+1
Currently, the Spack docs show documentation for submodules *before* documentation for submodules on package doc pages. This means that if you put docs in `__init__.py` in some package, the docs in there will be shown *after* the docs for all submodules of the package instead of at the top as an intro to the package. See, e.g., [the lockfile docs](https://spack.readthedocs.io/en/latest/spack.environment.html#module-spack.environment), which should be at the [top of that page](https://spack.readthedocs.io/en/latest/spack.environment.html). - [x] add the `--module-first` option to sphinx so that it generates module docs at top of page.
2022-12-22docs: remove monitors and analyzers (#34358)Todd Gamblin3-434/+0
These experimental features were removed in #31130, but the docs were not. - [x] remove the `spack monitor` and `spack analyze` docs
2022-12-22index.json.hash, no fatal error if key cannot be fetched (#34643)Harmen Stoppels2-19/+20
2022-12-21json: remove python 2 only code (#34615)Harmen Stoppels3-45/+5
2022-12-21rocm: make amdgpu_target sticky (#34591)Cory Bloor1-0/+1
The sticky property will prevent clingo from changing the amdgpu_target to work around conflicts. This is the same behaviour as was adopted for cuda_arch in 055c9d125d29606947f2fd88a06e279667a74648.
2022-12-21urlopen: handle timeout in opener (#34639)Harmen Stoppels1-4/+4
2022-12-21etags for index.json invalidation, test coverage (#34641)Harmen Stoppels4-97/+442
Implement an alternative strategy to do index.json invalidation. The current approach of pairs of index.json / index.json.hash is problematic because it leads to races. The standard solution for cache invalidation is etags, which are supported by both http and s3 protocols, which allows one to do conditional fetches. This PR implements that for the http/https schemes. It should also work for s3 schemes, but that requires other prs to be merged. Also it improves unit tests for index.json fetches.
2022-12-20archspec: add support for zen4 (#34609)Massimiliano Culpo7-33/+98
Also add: - Upper bound for Xeon Phi compiler support - Better detection for a64fx
2022-12-17fix != -> == typo (#34568)Harmen Stoppels1-1/+1
2022-12-16Compiler detection: avoid false recognition of MSVC (#34574)John W. Parent1-0/+2
Interim fix for #34559 Spack's MSVC compiler definition uses ifx as the Fortran compiler. Prior to #33385, the Spack MSVC compiler definition required the executable to be called "ifx.exe"; #33385 replaced this with just "ifx", which inadvertently led to ifx falsely indicating the presence of MSVC on non-Windows systems (which leads to future errors when attempting to query/use those compiler objects). This commit applies a short-term fix by updating MSVC Fortran version detection to always indicate a failure on non-Windows.
2022-12-15Propagate exceptions from Spack python console (#34547)Massimiliano Culpo1-0/+17
fixes #34489 Customize sys.excepthook to raise SystemExit when any unhandled exception reaches the hook.
2022-12-15IntelOneApiPackage: add envmods variant to toggle environment modifications ↵Sean Koyama1-5/+14
by oneapi packages (#34253) Co-authored-by: Sean Koyama <skoyama@anl.gov> Co-authored-by: Robert Cohn <robert.s.cohn@intel.com>
2022-12-15Add a proper deprecation warning for update-index -d (#34520)Harmen Stoppels4-5/+38
2022-12-15Remove warning in download_tarball (#34549)Harmen Stoppels1-4/+0
2022-12-15Forward lookup of the "run_tests" attribute (#34531)Massimiliano Culpo2-1/+20
fixes #34518 Fix an issue due to the MRO chain of the package wrapper during build. Before this PR we were always returning False when the builder object was created before the run_tests method was monkey patched.
2022-12-14Revert "Revert "Use `urllib` handler for `s3://` and `gs://`, improve ↵Harmen Stoppels4-159/+92
`url_exists` through HEAD requests (#34324)"" (#34498) This reverts commit 8035eeb36d5068fcbae613e51dd13cb1ae9f4888. And also removes logic around an additional HEAD request to prevent a more expensive GET request on wrong content-type. Since large files are typically an attachment and only downloaded when reading the stream, it's not an optimization that helps much, and in fact the logic was broken since the GET request was done unconditionally.
2022-12-14Use update-index --mirror-url <url> instead of -d <url> (#34519)Harmen Stoppels3-3/+3
2022-12-13Use file paths/urls correctly (#34452)Harmen Stoppels28-471/+259
The main issue that's fixed is that Spack passes paths (as strings) to functions that require urls. That wasn't an issue on unix, since there you can simply concatenate `file://` and `path` and all is good, but on Windows that gives invalid file urls. Also on Unix, Spack would not deal with uri encoding like x%20y for file paths. It also removes Spack's custom url.parse function, which had its own incorrect interpretation of file urls, taking file://x/y to mean the relative path x/y instead of hostname=x and path=/y. Also it automatically interpolated variables, which is surprising for a function that parses URLs. Instead of all sorts of ad-hoc `if windows: fix_broken_file_url` this PR adds two helper functions around Python's own path2url and reverse. Also fixes a bug where some `spack buildcache` commands used `-d` as a flag to mean `--mirror-url` requiring a URL, and others `--directory`, requiring a path. It is now the latter consistently.
2022-12-13Don't fetch to order mirrors (#34359)Harmen Stoppels2-11/+15
When installing binary tarballs, Spack has to download from its binary mirrors. Sometimes Spack has cache available for these mirrors. That cache helps to order mirrors to increase the likelihood of getting a direct hit. However, currently, when Spack can't find a spec in any local cache of mirrors, it's very dumb: - A while ago it used to query each mirror to see if it had a spec, and use that information to order the mirror again, only to go about and do exactly a part of what it just did: fetch the spec from that mirror confused - Recently, it was changed to download a full index.json, which can be multiple dozens of MBs of data and may take a minute to process thanks to the blazing fast performance you get with Python. In a typical use case of concretizing with reuse, the full index.json is already available, and it likely that the local cache gives a perfect mirror ordering on install. (There's typically no need to update any caches). However, in the use case of Gitlab CI, the build jobs don't have cache, and it would be smart to just do direct fetches instead of all the redundant work of (1) and/or (2). Also, direct fetches from mirrors will soon be fast enough to prefer these direct fetches over the excruciating slowness of index.json files.