summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2023-08-03repo cache: use -inf default instead of 0 (#39214)Harmen Stoppels2-28/+24
FastPackageChecker.modified_since should use a default number < 0 When the repo cache does not exist, Spack uses mtime 0. This causes the repo cache not to be generated when the repo has mtime 0. Some popular package managers such as spack use 0 mtime normalization for reproducible tarballs. So when installing spack with spack from a buildcache, the repo cache doesn't generate Also add some typehints
2023-08-03Buildcache commands cleanup, again... (#39203)Harmen Stoppels6-158/+114
* Inform mypy that tty.die is noreturn * avoid temporary allocation in env * update spack buildcache save-specfile * fix spack buildcache check/download/get-buildcache-name - ensure that required args and mutually exclusive ones are marked as such in argparse for better error messages - deprecate --spec-file everywhere - use disambiguate for better error messages
2023-08-02Add qmake virtual provider (#38848)Adam J. Stewart1-1/+1
2023-08-02CI: Refactor ci reproducer (#37088)kwryankrattiger3-93/+184
* CI: Refactor ci reproducer * Autostart container * Reproducer paths match CI paths * Generate start scripts for docker and reproducer * CI: Add interactive and gpg options to reproduce-build * Interactive will determine if the docker container persists after running reproduction. * GPG path/url allow downloading GPG keys needed for binary cache download validation. This is important for running reproducer for protected CI jobs. * Add exit_on_failure option to CI scripts * CI: Add runtime option for reproducer
2023-08-02version: move to module, avoid circular imports (#39077)Harmen Stoppels8-268/+358
2023-08-02buildcache extractall: extract directly into spec.prefix (#37441)Harmen Stoppels5-60/+159
- Run `mkdirp` on `spec.prefix` - Extract directly into `spec.prefix` 1. No need for `$store/tmp.xxx` where we extract the tarball directly, pray that it has one subdir `<name>-<version>-<hash>`, and then `rm -rf` the package prefix followed by `mv`. 2. No need to clean up this temp dir in `spack clean`. 3. Instead figure out package directory prefix from the tarball contents, and strip the tarinfo entries accordingly (kinda like tar --strip-components but more strict) - Set package dir permissions - Don't error during error handling when files cannot removed - No need to "enrich" spec.json with this tarball-toplevel-path After this PR, we can in fact tarball packages relative to `/` instead of `spec.prefix/..`, which makes it possible to use Spack tarballs as container layers, where relocation is impossible, and rootfs tarballs are expected.
2023-08-02feat: move -N/--namespace(s) to common args, allow in `buildcache list` (#36719)Wouter Deconinck7-31/+25
`spack buildcache list` did not have a way to display the namespace of packages in the buildcache. This PR adds that functionality. For consistency's sake, it moves the `-N/--namespace` arg definition to the `common/arguments.py` and modifies `find`, `solve`, `spec` to use the common definition. Previously, `find` was using `--namespace` (singular) to control whether to display the namespace (it doesn't restrict the search to that namespace). The other commands were using `--namespaces` (plural). For backwards compatibility and for consistency with `--deps`, `--tags`, etc, the plural `--namespaces` was chosen. The argument parser ensures that `find --namespace` will continue to behave as before. Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
2023-08-01remove CMAKE_GENERATOR from the host-config because it cannot be overwritten ↵Chris White1-11/+0
on the command line (#39044)
2023-07-31Add `spack checksum --verify`, fix `--add` (#38458)Alec Scott6-113/+266
* Add rewrite of spack checksum to include --verify and better add versions to package.py files * Fix formatting and remove unused import * Update checksum unit-tests to correctly test multiple versions and add to package * Remove references to latest in stage.py * Update bash-completion scripts to fix unit tests failures * Fix docs generation * Remove unused url_dict argument from methods * Reduce chance of redundant remote_versions work * Add print() before tty.die() to increase error readablity * Update version regular expression to allow for multi-line versions * Add a few unit tests to improve test coverage * Update command completion * Add type hints to added functions and fix a few py-lint suggestions * Add @no_type_check to prevent mypy from failing on pkg.versions * Add type hints to format.py and fix unit test * Black format lib/spack/spack/package_base.py * Attempt ignoring type errors * Add optional dict type hint and declare versions in PackageBase * Refactor util/format.py to allow for url_dict as an optional parameter * Directly reference PackageBase class instead of using TypeVar * Fix comment typo --------- Co-authored-by: Tamara Dahlgren <dahlgren1@llnl.gov>
2023-07-31Remove Xcode mock-up (#39020)Adam J. Stewart3-227/+0
* Remove Xcode mock-up * Remove unused imports
2023-07-26ci: Make signing requirement explicit (#38995)Scott Wittenburg4-9/+50
Instead of inferring whether to sign binaries, make it explicit, and fail rebuild jobs early if signing is required but cannot be accomplished.
2023-07-24Always set workflow:rules in spack ci pipelines (#38921)Jonathon Anderson2-13/+20
2023-07-24Fix typo: binary_caches.rst (#39064)Ashwin Kumar Karnad1-1/+1
2023-07-24Don't add `.spack/binary_distribution` twice to the tarball when ↵Harmen Stoppels2-2/+44
re-distributing (#39042) Previously, spack would list the ./spack/binary_distribution file twice when pushing a package that was installed from a binary tarball itself.
2023-07-24Fix broken `sanitize_file_path` (#38926)Harmen Stoppels6-35/+52
The sanitization function is completely bogus as it tries to replace / on unix after ... splitting on it. The way it's implemented is very questionable: the input is a file name, not a path. It doesn't make sense to interpret the input as a path and then make the components valid -- you'll interpret / in a filename as a dir separator. It also fails to deal with path components that contain just unsupported characters (resulting in empty component). The correct way to deal with this is to have a function that takes a potential file name and replaces unsupported characters. I'm not going to fix the other issues on Windows, such as reserved file names, but left a note, and hope that @johnwparent can fix that separately. (Obviously we wouldn't have this problem at all if we just fixed the filename in a safe way instead of trying to derive something from the url; we could use the content digest when available for example)
2023-07-22Implement fish completion (#29549)百地 希留耶5-165/+563
* commands: provide more information to Command * fish: Add script to generate fish completion * fish: auto prepend `spack` command to avoid duplication * fish: impove completion generation code readability * commands: replace match-case with if-else * fish: fix optspec variable name prefix * fish: fix return value in get_optspecs * fish: fix return value in get_optspecs * format: split long line and trim trailing space * bugfix: replace f-string with interpolation * fish: compete more specs and some fixes * fish: complete hash spec starts with / * fish: improve compatibility * style: trim trailing whitespace * commands: add fish to update args and update tests * commands: add fish completion file * style: merge imports * fish: source completion in setup-env * fish: caret only completes dependencies * fish: make sure we always get same order of output * fish: spack activate only show installed packages that have extensions * fish: update completion file * fish: make dict keys sorted * Blacken code * Fix bad merge * Undo style changes to setup-env.fish * Fix unit tests * Style fix * Compatible with fish_indent * Use list for stability of order * Sort one more place * Sort more things * Sorting unneeded * Unsort * Print difference * Style fix * Help messages need quotes * Arguments to -a must be quoted * Update types * Update types * Update types * Add type hints * Change order of positionals * Always expand help * Remove shared base class * Fix type hints * Remove platform-specific choices * First line of help only * Remove unused maps * Remove suppress * Remove debugging comments * Better quoting * Fish completions have no double dash * Remove test for deleted class * Fix grammar in header file * Use single quotes in most places * Better support for remainder nargs * No magic strings * * and + can also complete multiple * lower case, no period --------- Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2023-07-21Remove spack.repo.IndexError (#39029)Massimiliano Culpo1-4/+0
This exception is never used and overrides a built-in.
2023-07-21binary_caches.rst: fix typo (#39030)Ashwin Kumar Karnad1-1/+1
2023-07-20nag: update the versioning scheme (#35457)Sergey Kosukhin2-2/+11
* nag: append build number to the version * nag: add version 7.1.7125 * nag: deprecate unavailable versions
2023-07-20spack.util.lock: add type-hints, remove **kwargs in method signatures (#39011)Massimiliano Culpo4-23/+37
2023-07-19Respect custom user store when bootstrapping (#39001)Massimiliano Culpo3-1/+26
The user store is lazily evaluated. The change in #38975 made it such that the first evaluation was happening in the middle of swapping to user configuration. Ensure we construct the user store before that.
2023-07-19modules: use curly braces to enclose value in Tcl modulefile (#38375)Xavier Delaruelle3-33/+65
Use curly braces instead of quotes to enclose value or text in Tcl modulefile. Within curly braces Tcl special characters like [, ] or $ are treated verbatim whereas they are evaluated within quotes. Curly braces is Tcl recommended way to enclose verbatim content [1]. Note: if curly braces charaters are used within content, they must be balanced. This point has been checked against current repository and no unbalanced curly braces has been spotted. Fixes #24243 [1] https://wiki.tcl-lang.org/page/Tcl+Minimal+Escaping+Style
2023-07-19Fetch & patch: actually acquire stage lock, and many more issues (#38903)Harmen Stoppels17-219/+210
* Fetching patches wouldn't result in acquiring a stage lock during install * The installer would acquire a stage lock *after* fetching instead of before, leading to races * The name of the stage for patches was random, so on build failure (where stage dirs are not removed), these directories would continue to exist after a second successful install. * There was this redundant "composite fetch" object -- there's already a composite stage. Remove this. * For some reason we do *double* shasum validation of patches, before and after compression -- that's just too much? I removed it.
2023-07-19drop redundant rpaths post install (#38976)Harmen Stoppels2-0/+152
Spack heuristically adds `<install prefix>/lib` and `<install prefix>/lib64` as rpath entries, as it doesn't know what the install dir is going to be ahead of the build. This PR cleans up non-existing, absolute paths[^1], which 1. avoids redundant stat calls at runtime 2. drops redundant rpaths in `patchelf`, making it relocatable -- you don't need patchelf recursively then. [^1]: It also removes relative paths not starting with `$` (so, `$ORIGIN/../lib` is retained -- we _could_ interpolate `$ORIGIN`, but that's hard to get right when symlinks have to be taken into account). Relative paths _are_ supported in glibc, but are relative to _the current working directory_, which is madness, and it would be better to drop those paths.
2023-07-19llnl.util.lock: add type-hints (#38977)Massimiliano Culpo3-71/+114
Also uppercase global variables in the module
2023-07-19Remove `LazyReference` from code (#38944)Massimiliano Culpo72-425/+376
A LazyReference object is a reference to an attribute of a lazily evaluated singleton. Its only purpose is to let developers use shorter names to refer to such attribute. This class does more harm than good, as it obfuscates the fact that we are using the attribute of a global object. Also, it can easily go out of sync with the singleton it refers to if, for instance, the singleton is updated but the references are not. This commit removes the LazyReference class entirely, and access the attributes explicitly passing through the global value to which they are attached.
2023-07-18Windows testing: enable tests for installer components (#36970)markus-ferrell1-8/+0
These tests now work without any changes to core. Furthermore, it is surprising that they had to be disabled (at least, as long as the installer.py tests are run on Windows: these tests are more-basic and their functionality would have been exercised automatically).
2023-07-18Ensure the bootstrap store has a padding length of zero (#38975)Massimiliano Culpo2-17/+26
Without this PR, padded length was propagating from user configuration to bootstrap configuration, and was causing the issue reported in #38963
2023-07-18Windows testing: enable bootstrap test (#36972)markus-ferrell1-2/+0
2023-07-18test_clear_failures_success: run on Windows too (#36792)markus-ferrell1-2/+3
2023-07-18binary cache docs: remove redundant flag and comment (#38960)Harmen Stoppels1-5/+1
2023-07-18Windows testing: enable architecture test (#36973)markus-ferrell1-2/+0
Works out of the box: remove skip.
2023-07-18Windows testing: enable "spack clean" tests (#36840)markus-ferrell1-3/+0
They work out out of the box on windows. Simply removing skips.
2023-07-18buildcache push: make --allow-root the default and deprecate the option (#38878)Harmen Stoppels14-218/+36
Without --allow-root spack cannot push binaries that contain paths in binaries. This flag is almost always needed, so there is no point of requiring users to spell it out. Even without --allow-root, rpaths would still have to be patched, so the flag is not there to guarantee binaries are not modified on install. This commit makes --allow-root the default, and drops the code required for it. It also deprecates `spack buildcache preview`, since the command made sense only with --allow-root. As a side effect, Spack no longer depends on binutils for relocation
2023-07-18Pin Spack dev dependencies on RtD (#38950)Massimiliano Culpo1-0/+5
2023-07-18Fix default construction of locks (#38953)Massimiliano Culpo1-1/+1
This fixes a typo introduced in a refactor
2023-07-18modules: add support for conflict in lua modulefile (#36701)Xavier Delaruelle8-59/+116
Add support for conflict directives in Lua modulefile like done for Tcl modulefile. Note that conflicts are correctly honored on Lmod and Environment Modules <4.2 only if mutually expressed on both modulefiles that conflict with each other. Migrate conflict code from Tcl-specific classes to the common part. Add tests for Lmod and split the conflict test case in two. Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2023-07-18"spack develop": always pull full history for git repos (#38343)Peter Scheibel4-9/+66
2023-07-17Decompression: fix naming issues (#37749)Dan Lipsa2-35/+76
* When using system tools to unpack a .gz file, the input file needs a different name than the output file. Normally, we generate this new name by stripping off the .gz extension off of the file name. This was not sufficient if the file name did not have an extension, so we temporarily rename the file in that case. * When using system tar utility to untar on Windows, we were (erroneously) skipping the actual untar step if the filename was lacking a .tar extension * For foo.txz, we were not changing the extension of the decompressed file (i.e. we would decompress foo.txz to foo.txz). This did not cause any problems, but is confusing, so has been updated such that the output filename reflects its decompressed state (i.e. foo.tar). * Added test for strip_compression_extension * Update test_native_unpacking to test each archive type with and without an extension as part of the file name (i.e. we test "foo.tar.gz", but also make sure we decompress properly if it is named "foo").
2023-07-17Fix wrong StageComposite keep override (#38938)Harmen Stoppels2-2/+38
`Stage(keep=True)` was ignored when put in a composite that doesn't override the value.
2023-07-17Fix python import tests (#38928)Manuela Kuhn1-1/+1
Running `spack test run <python package>` resulted in the error ``` 'str' object is not callable ``` because the python executable was not set correctly.
2023-07-17Lock, database and store don't need global configuration on construction ↵Massimiliano Culpo24-337/+431
(#33495) Lock objects can now be instantiated independently, without being tied to the global configuration. The same is true for database and store objects. The database __init__ method has been simplified to take a single lock configuration object. Some common lock configurations (e.g. NO_LOCK or NO_TIMEOUT) have been named and are provided as globals. The use_store context manager keeps the configuration consistent by pushing and popping an internal scope. It can also be tuned by passing extra data to set up e.g. upstreams or anything else that might be related to the store.
2023-07-17Update new PythonPackage template to prefer --config-settings (#38918)Adam J. Stewart3-17/+12
2023-07-17Environment/depfile: fix bug with Git hash versions (attempt #2) (#37560)Peter Scheibel2-12/+91
Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
2023-07-17Minimal cleanup of a few tests in `test/packaging.py` (#38880)Massimiliano Culpo1-162/+125
* Minimal cleanup of a few tests in packaging.py * Use f-strings
2023-07-14Remove unused context manager (#38897)Harmen Stoppels1-8/+0
2023-07-14Fix issue on cray with super call (#38895)Rocco Meli1-1/+1
2023-07-13Bugfix/spack spec: read and use the environment concretizer:unification ↵Tamara Dahlgren2-4/+62
option (#38248) * Bugfix: spack.yaml concretizer:unify needs to be read and used * Optional: add environment test to ensure configuration scheme is used * Activate environment in unit tests A more proper solution would be to keep an environment instance configuration as an attribute, but that is a bigger refactor * Delay evaluation of Environment.unify * Slightly simplify unit tests --------- Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2023-07-13spack buildcache push: improve argparse (#38876)Harmen Stoppels1-8/+11
2023-07-13mirrors: distinguish between source/binary mirror; simplify schema (#34523)Harmen Stoppels11-298/+469
Allow the following formats: ```yaml mirrors: name: <url> ``` ```yaml mirrors: name: url: s3://xyz access_pair: [x, y] ``` ```yaml mirrors: name: fetch: http://xyz push: url: s3://xyz access_pair: [x, y] ``` And reserve two new properties to indicate the mirror type (e.g. mirror.spack.io is a source mirror, not a binary cache) ```yaml mirrors: spack-public: source: true binary: false url: https://mirror.spack.io ```