summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2022-08-03Match documentation of spack create with actual behavior. (#31892)dunatotatos1-1/+1
2022-08-02spack ci rebuild: Don't install in a login shell (#31771)Jonathon Anderson1-3/+2
On some systems the shell in login mode wipes important parts of the environment, such as PATH. This causes the build to fail since it can't find `spack`. For better robustness, don't use a login shell. In a full CI job the final spack install is run in an environment formed by scripts running in this order: export AWS_SECRET=... # 1. Load environment from GitLab project variables source spack/share/spack/setup-env.sh # 2. Load Spack into the environment (PATH) spack env activate -V concrete_env # 3. Activate the concrete environment source /etc/profile # 4. Bash login shell (from -l) spack install ... Whereas when a user launches their own container with (docker|podman) run -it, they end up running spack install in an environment formed in this order: source /etc/bash.bashrc # (not 4). Bash interactive shell (default with TTY) export AWS_SECRET=... #~1. Manually load environment from GitLab project variables source spack/share/spack/setup-env.sh # 2. Load Spack into the environment (PATH) spack env activate -V concrete_env # 3. Activate the concrete environment spack install ... The big problem being that (4) has a completely different position and content (on Leap 15 and possibly other containers). So in context, this PR removes (4) from the CI job case, leaving us with the simpler: export AWS_SECRET=... # 1. Load environment from GitLab project variables source spack/share/spack/setup-env.sh # 2. Load Spack into the environment (PATH) spack env activate -V concrete_env # 3. Activate the concrete environment spack install ...
2022-08-02Optimize reuse from buildcaches (#30806)Massimiliano Culpo2-11/+13
* database: don't sort on return from query_local * ASP-based solver: don't build the hash-lookup dictionary twice Building this dictionary twice and traversing all the specs might be time-consuming for large buildcaches.
2022-08-02Style fixesMassimiliano Culpo3-10/+11
2022-08-02Remove line that sets non-existing attribute?Massimiliano Culpo1-1/+0
2022-08-02Use __slots__ for fast attribute accessMassimiliano Culpo2-0/+12
2022-08-02Use __slots__ for fast attribute access during parsingMassimiliano Culpo2-0/+8
2022-08-02Document that reuse is higher priority than preferences in packages.yaml ↵Harmen Stoppels2-0/+35
(#31864) Resurrect Known issues, since users ask frequently about that.
2022-08-02docs: Fix outdated pre-0.18 text on build dependencies (#31860)Todd Gamblin1-5/+13
2022-08-02Add meson Python build backend (#31809)Adam J. Stewart1-36/+51
2022-08-02spack checksum: black compliance (#31851)Adam J. Stewart1-1/+1
2022-08-01Add c++14 support flag for xl compiler (#31826)Peter Scheibel1-0/+8
2022-08-01Add support for macOS Ventura (#31812)Adam J. Stewart1-0/+1
2022-08-01CUDA: add new GCC conflict (#31820)Adam J. Stewart1-0/+1
2022-07-31tests: remove old `changed_files()` testTodd Gamblin1-10/+0
This test relied on an old version of the `flake8_package` fixture that modified the spack repository, but it doesn't do that anymore. There are other tests for `changed_files()` that do a better job of mocking up a git repository with changes, so we can just delete this one.
2022-07-31style: simplify arguments with `--tool TOOL` and `--skip TOOL`Todd Gamblin2-58/+68
`spack style` tests were annoyingly brittle because we could not easily be specific about which tools to run (we had to use `--no-black`, `--no-isort`, `--no-flake8`, and `--no-mypy`). We should be able to specify what to run OR what to skip. Now you can run, e.g.: spack style --tool black,flake8 or: spack style --skip black,isort - [x] Remove `--no-black`, `--no-isort`, `--no-flake8`, and `--no-mypy` args. - [x] Add `--tool TOOL` argument. - [x] Add `--skip TOOL` argument. - [x] Allow either `--tool black --tool flake8` or `--tool black,flake8` syntax.
2022-07-31black: ensure that `spack create` templates are black-compliantTodd Gamblin2-114/+101
- [x] remove alignment spaces from tempaltes - [x] replace single with double quotes - [x] Makefile template now generates parsable code (function body is `pass` instead of just a comment) - [x] template checks now run black to check output
2022-07-31black: do not align `sha56`'s with spaces in `spack checksum` outputTodd Gamblin1-9/+1
2022-07-31black: bootstrap if black in `PATH` is too newTodd Gamblin1-4/+27
Previously we'd accept any version for bootstrapping black, but we need <= 21. - [x] modify bootstrapping code to check black version before accepting an executable from `PATH`.
2022-07-31black: add .git-blame-ignore-revs toignore black reformatting in blameTodd Gamblin1-2/+12
- [x] add `.git-blame-ignore-revs` to ignore black reformatting - [x] make `spack blame` respect `.git-blame-ignore-revs` (even if the user hasn't configured git to do so)
2022-07-31black: fix format-sensitive testsTodd Gamblin3-7/+7
Some of our tests rely on single vs. double quotes, and others rely on specific line numbers in the source. These needed fixing after the switch to Black.
2022-07-31black: reformat entire repository with blackTodd Gamblin462-35680/+36530
2022-07-31black: fix style check package and flake8 formatting for blackTodd Gamblin3-42/+33
Black will automatically fix a lot of the exceptions we previously allowed for directives, so we don't need them in our custom `flake8_formatter` anymore. - [x] remove `E501` (long line) exceptions for directives from `flake8_formatter`, as they won't help us now. - [x] Refine exceptions for long URLs in the `flake8_formatter`. - [x] Adjust the mock `flake8-package` to exhibit the exceptions we still allow. - [x] Update style tests for new `flake8-package`. - [x] Blacken style test.
2022-07-31black: clean up noqa comments from most of the codeTodd Gamblin27-71/+103
Many noqa's in the code are no longer necessary now that the column limit is 99 characters. Others can easily be eliminated, and still more can just be made more specific if they do not have to do with line length. The only E501's still in the code are in the tests for `spack.util.path` and the tests for `spack style`.
2022-07-31black: break up long strings that black cannot fixTodd Gamblin6-107/+160
2022-07-31black: configurationTodd Gamblin2-39/+56
This adds necessary configuration for flake8 and black to work together. This also sets the line length to 99, per the data here: * https://github.com/spack/spack/pull/24718#issuecomment-876933636 Given the data and the spirit of black's 88-character limit, we set the limit to 99 characters for all of Spack, because: * 99 is one less than 100, a nice round number, and all lines will fit in a 100-character wide terminal (even when the text editor puts a \ at EOL). * 99 is just past the knee the file size curve for packages, and it means that packages remain readable and not significantly longer than they are now. * It doesn't seem to hurt core -- files in core might change length by a few percent but seem like they'll be mostly the same as before -- just a bit more roomy. - [x] set line length to 99 - [x] remove most exceptions from `.flake8` and add the ones black cares about - [x] add `[tool.black]` to `pyproject.toml` - [x] make `black` run if available in `spack style --fix` Co-Authored-By: Tom Scogland <tscogland@llnl.gov>
2022-07-30MACOSX_DEPLOYMENT_TARGET: MAJOR.MINOR (#31811)Adam J. Stewart1-1/+2
2022-07-28environment.py: only install root specs explicitly (#31645)Harmen Stoppels2-3/+6
2022-07-28Fix parsing of clean_url (#31783)Stephen Sachs1-2/+2
In #31618 the idea was to determine the file extension heuristically by dropping query params etc from a url and then consider it as a file path. That broke for URLs that only have query params like http://example.com/?patch=x as it would result in empty string as basename. This PR reverts to the old behavior of saving files as ?patch=x in that case. Co-authored-by: Stephen Sachs <stesachs@amazon.com>
2022-07-26environment.py: reduce # of locks further (#31643)Harmen Stoppels2-27/+49
* environment.py: reduce # of locks further
2022-07-26containerize: fix concretization -> concretizer (#31594)Harmen Stoppels3-8/+8
* containerize: fix concretization -> concretizer * fix test
2022-07-26llvm: Variant and dependency cleanup (#31331)Chuck Atkins2-111/+171
* llvm: Use variant when clauses for many of the expressed conflicts * llvm: Remove the shared variant as it wasn't really used * llvm: Remove unnecessary deps and make explicit the ones that are * llvm: Cleanup patch conditions * pocl: Update for llvm cleanup * unit-test: update unparse package hash with the updated llvm package * llvm: Fix ppc long double patching and add clarifying comments
2022-07-26Fix `self.extension` for symlinks (#31743)Stephen Sachs1-1/+3
`self.archive_file` is (among others) a symlink to a tarball. `extension()` on a symlink will result in no extension. This patch fixes the behavior introduced in https://github.com/spack/spack/pull/31618. Co-authored-by: Stephen Sachs <stesachs@amazon.com>
2022-07-25Fix stage with uri params (#31618)Harmen Stoppels2-3/+3
* don't determine extension of a url * parse url to drop query params from filename
2022-07-24Sphinx 5.1.0 breaks doc builds (#31709)Adam J. Stewart1-1/+1
2022-07-22Mark external as explicit only when installing explicitly (#31665)Harmen Stoppels2-5/+18
2022-07-22autotools: add -I flag when non-standard libtool (#31677)Harmen Stoppels2-6/+99
When 1. Spack installs libtool, 2. system libtool is installed too, and 3. system automake is used Spack passes system automake's `-I <prefix>` flag to itself, even though it's a default search path. This takes precedence over spack's libtool prefix dir. This causes the wrong `libtool.m4` file to be used (since system libtool is in the same prefix as system automake). And that leads to error messages about incompatible libtool, something something LT_INIT.
2022-07-22Use pkg_cls in spack.mirror.get_all_versions (#31636)Massimiliano Culpo2-7/+22
fixes #31627 spack.mirror.get_all_versions now uses the package class instead of the package object in its implementation. Ensure spec is concrete before staging for mirrors
2022-07-20Don't restage develop specs when a patch fails (#31593)psakievich2-3/+55
* make develop specs not restage when a patch fails * add a unit test
2022-07-20spack stage: add missing --fresh and --reuse (#31626)Harmen Stoppels2-0/+12
2022-07-20installer.py: require "explicit: True" in the install arguments to mark a ↵Harmen Stoppels2-1/+14
package "explicit" (#31646)
2022-07-18Only hack botocore when needed (#31610)Jonathon Anderson1-6/+10
Newer versions of botocore (>=1.23.47) support the full IOBase interface, so the hacks added to supplement the missing attributes are no longer needed. Conditionally disable the hacks if they appear to be unnecessary based on the class hierarchy found at runtime.
2022-07-15Use lexists instead of exists during fetch (#31509)Harmen Stoppels1-11/+11
2022-07-14Bug/fix credentials s3 buildcache update (#31391)Joseph Snyder3-5/+14
* Add connection information to buildcache update command Ensure that the s3 connection made when attempting to update the content of a buildcache attempts to use the extra connection information from the mirror creation. * Add unique help for endpoint URL argument Fix copy/paste error for endpoint URL help which was the same as the access token * Re-work URL checking for S3 mirrors Due to the fact that nested bucket URLs would never match the string used for checking that the mirror is the same, switch the check used. Sort all mirror URLs by length to have the most specific cases first and see if the desired URL "starts with" the mirror URL. * Long line style fixes Add execptions for long lines and fix other style errors * Use format() function to rebuild URL Use the format command to rebuild the url instead of crafing a formatted string out of known values * Add early exit for URL checking When a valid mirror is found, break from the loop
2022-07-14Deprecate `blacklist`/`whitelist` in favor of `include`/`exclude` (#31569)Todd Gamblin23-135/+375
For a long time the module configuration has had a few settings that use `blacklist`/`whitelist` terminology. We've been asked by some of our users to replace this with more inclusive language. In addition to being non-inclusive, `blacklist` and `whitelist` are inconsistent with the rest of Spack, which uses `include` and `exclude` for the same concepts. - [x] Deprecate `blacklist`, `whitelist`, `blacklist_implicits` and `environment_blacklist` in favor of `exclude`, `include`, `exclude_implicits` and `exclude_env_vars` in module configuration, to be removed in Spack v0.20. - [x] Print deprecation warnings if any of the deprecated names are in module config. - [x] Update tests to test old and new names. - [x] Update docs. - [x] Update `spack config update` to fix this automatically, and include a note in the error that you can use this command.
2022-07-13Decompression: use tar exe vs. built-in Python tarfile support (#31563)Peter Scheibel1-19/+4
Python's built-in tarfile support doesn't address some general cases of malformed tarfiles that are already handled by the system 'tar' utility; until these can be addressed, use that exclusively.
2022-07-12`spack.repo.get()` can only be called on concrete specs (#31411)Massimiliano Culpo76-1397/+1065
The goal of this PR is to make clearer where we need a package object in Spack as opposed to a package class. We currently instantiate a lot of package objects when we could make do with a class. We should use the class when we only need metadata, and we should only instantiate and us an instance of `PackageBase` at build time. Modifications: - [x] Remove the `spack.repo.get` convenience function (which was used in many places, and not really needed) - [x] Use `spack.repo.path.get_pkg_class` wherever possible - [x] Try to route most of the need for `spack.repo.path.get` through `Spec.package` - [x] Introduce a non-data descriptor, that can be used as a decorator, to have "class level properties" - [x] Refactor unit tests that had to be modified to reduce code duplication - [x] `Spec.package` and `Repo.get` now require a concrete spec as input - [x] Remove `RepoPath.all_packages` and `Repo.all_packages`
2022-07-12file_cache.py: idempotent remove without races (#31477)Harmen Stoppels2-4/+19
There's a race condition in `remove()` as the lockfile is removed after releasing the lock, which is a problem when another process acquires a write lock during deletion. Also simplify life a bit in multiprocessing when a file is possibly removed multiple times, which currently is an error on the second deletion, so the proposed fix is to make remove(...) idempotent and not error when deleting non-existing cache entries. Don't tests for existence of lockfile, cause windows/linux behavior is different
2022-07-12compression.py: buffered copy (#31533)Harmen Stoppels1-3/+4
2022-07-12environment.py: only acquire write lock when necessary (#31493)Harmen Stoppels1-3/+4