summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2022-06-09Allow more fine-grained control over what submodules are updated (#27293)Tim Fuller5-6/+78
The "submodules" argument of the "version" directive can now accept a callable that returns a list of submodules, in addition to the usual Boolean values
2022-06-08bootstrap: account for disabled sources (#31042)Massimiliano Culpo2-4/+22
* bootstrap: account for disabled sources Fix a bug introduced in #30192, which effectively skips any prescription on disabled bootstrapping sources. * Add unit test to avoid regression
2022-06-07Staging: determine extensions from nonstandard URLs (#31027)Peter Scheibel2-5/+2
Fixes #31021 With #25185, we no longer default to using tar when we can't determine the extension type, opting to fail instead. This broke fetching for the pcre package, where we couldn't determine the extension. To determine the extension, we were attempting to extract it from the destination filename; however, this file name may omit details of the origin URL that are required to determine the extension, so instead we examine the URL directly. This also updates the decompressor_for method not to set ext=None by default: it must now always be set by the caller.
2022-06-07docs: update the list of Docker images with Spack preinstalled (#31003)Massimiliano Culpo3-7/+17
Also, update the image in the docs and use ghcr.io
2022-06-07Fix empty install prefix post-install sanity check (#30983)Jordan Galby2-7/+17
2022-06-07archspec: bump to v0.1.4 (#30856)Massimiliano Culpo2-10/+49
Fixes compiler flags for oneapi and dpcpp
2022-06-06Update decompression support on Windows (#25185)John W. Parent19-61/+397
Most package installations include compressed source files. This adds support for common archive types on Windows: * Add support for using system 7zip functionality to decompress .Z files when available (and on Windows, use 7zip for .xz archives) * Default to using built-in Python support for tar/bz2 decompression (note that Python tar documentation mentions preservation of file permissions) * Add tests for decompression support * Extract logic for handling exploding archives (i.e. compressed archives that expand to more than one base file) into an exploding_archive_catch context manager in the filesystem module
2022-06-06Package: Don't warn for missing source on bundle packages without code (#30913)Chuck Atkins2-1/+19
2022-06-04Sanitize filepath from URL (#30625)John W. Parent3-75/+123
Spack's staging logic constructs a file path based on a URL. The URL may contain characters which are not allowed in valid file paths on the system (e.g. Windows prohibits ':' and '?' among others). This commit adds a function to remove such offending characters (but otherwise preserves the URL string when constructing a file path).
2022-06-02Use stable URLs for patch-diff GitHub patches (#30953)Adam J. Stewart1-1/+2
2022-05-31Fix module support for oneapi compilers (#28901)Robert Cohn1-0/+8
Updates to improve Spack-generated modules for Intel oneAPI compilers: * intel-oneapi-compilers set CC etc. * Add a new package intel-oneapi-compilers-classic which can be used to generate a module which sets CC etc. to older compilers (e.g. icc) * lmod module logic now updated to treat the intel-oneapi-compilers* packages as compilers
2022-05-30CUDA: make cuda_arch sticky (#30910)Adam J. Stewart1-0/+1
2022-05-280.18.0.dev0 -> 0.19.0.dev0 (#30907)Todd Gamblin1-1/+1
2022-05-28refactor: packages import `spack.package` explicitly (#30404)Tom Scogland80-3362/+3420
Explicitly import package utilities in all packages, and corresponding fallout. This includes: * rename `spack.package` to `spack.package_base` * rename `spack.pkgkit` to `spack.package` * update all packages in builtin, builtin_mock and tutorials to include `from spack.package import *` * update spack style * ensure packages include the import * automatically add the new import and remove any/all imports of `spack` and `spack.pkgkit` from packages when using `--fix` * add support for type-checking packages with mypy when SPACK_MYPY_CHECK_PACKAGES is set in the environment * fix all type checking errors in packages in spack upstream * update spack create to include the new imports * update spack repo to inject the new import, injection persists to allow for a deprecation period Original message below: As requested @adamjstewart, update all packages to use pkgkit. I ended up using isort to do this, so repro is easy: ```console $ isort -a 'from spack.pkgkit import *' --rm 'spack' ./var/spack/repos/builtin/packages/*/package.py $ spack style --fix ``` There were several line spacing fixups caused either by space manipulation in isort or by packages that haven't been touched since we added requirements, but there are no functional changes in here. * [x] add config to isort to make sure this is maintained going forward
2022-05-27target optimization: re-norm optimization scale so that 0 is best. (#29926)Greg Becker3-16/+34
referred targets are currently the only minimization criteria for Spack for which we allow negative values. That means Spack may be incentivized to add nodes to the DAG if they match the preferred target. This PR re-norms the minimization criteria so that preferred targets are weighted from 0, and default target weights are offset by the number of preferred targets per-package to calculate node_target_weight. Also fixes a bug in the test for preferred targets that was making the test easier to pass than it should be.
2022-05-28update tutorial command for v0.18.0 and new gpg key (#30904)Greg Becker1-1/+1
2022-05-26Revert "strip -Werror: all specific or none (#30284)" (#30878)Scott Wittenburg5-127/+3
This reverts commit 330832c22cfa59554f6681a570bdec24ca46e79b.
2022-05-26Remove the warning that Spack prints at each spec (#30872)Massimiliano Culpo2-3/+7
Add instead a warning box in the documentation
2022-05-26ci: Support secure binary signing on protected pipelines (#30753)Scott Wittenburg5-78/+322
This PR supports the creation of securely signed binaries built from spack develop as well as release branches and tags. Specifically: - remove internal pr mirror url generation logic in favor of buildcache destination on command line - with a single mirror url specified in the spack.yaml, this makes it clearer where binaries from various pipelines are pushed - designate some tags as reserved: ['public', 'protected', 'notary'] - these tags are stripped from all jobs by default and provisioned internally based on pipeline type - update gitlab ci yaml to include pipelines on more protected branches than just develop (so include releases and tags) - binaries from all protected pipelines are pushed into mirrors including the branch name so releases, tags, and develop binaries are kept separate - update rebuild jobs running on protected pipelines to run on special runners provisioned with an intermediate signing key - protected rebuild jobs no longer use "SPACK_SIGNING_KEY" env var to obtain signing key (in fact, final signing key is nowhere available to rebuild jobs) - these intermediate signatures are verified at the end of each pipeline by a new signing job to ensure binaries were produced by a protected pipeline - optionallly schedule a signing/notary job at the end of the pipeline to sign all packges in the mirror - add signing-job-attributes to gitlab-ci section of spack environment to allow configuration - signing job runs on special runner (separate from protected rebuild runners) provisioned with public intermediate key and secret signing key
2022-05-26bugfix: do not compute `package_hash` for old concrete specs (#30861)Todd Gamblin2-35/+50
Old concrete specs were slipping through in `_assign_hash`, and `package_hash` was attempting to recompute a package hash when we could not know the package a time of concretization. Part of this was that the logic for `_assign_hash` was hard to understand -- it was called twice from `_finalize_concretization` and had special cases for both args it was called with. It's much easier to understand the logic here if we just inline it. - [x] Get rid of `_assign_hash` and just integrate it with `_finalize_concretization` - [x] Don't call `_package_hash` at all for already-concrete specs. - [x] Add regression test.
2022-05-24buildcache: Update layout and signing (#30750)Scott Wittenburg12-137/+549
This PR introduces a new build cache layout and package format, with improvements for both efficiency and security. ## Old Format Currently a binary package consists of a `spec.json` file at the root and a `.spack` file, which is a `tar` archive containing a copy of the `spec.json` format, possibly a detached signature (`.asc`) file, and a tar-gzip compressed archive containing the install tree. ``` build_cache/ # metadata (for indexing) <arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.spec.json <arch>/ <compiler>/ <name>-<ver>/ # tar archive <arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.spack # tar archive contents: # metadata (contains sha256 of internal .tar.gz) <arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.spec.json # signature <arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.spec.json.asc # tar.gz-compressed prefix <arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.tar.gz ``` After this change, the nesting has been removed so that the `.spack` file is the compressed archive of the install tree. Now signed binary packages, will take the form of a clearsigned `spec.json` file (a `spec.json.sig`) at the root, while unsigned binary packages will contain a `spec.json` at the root. ## New Format ``` build_cache/ # metadata (for indexing, contains sha256 of .spack file) <arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.spec.json # clearsigned spec.json metadata <arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.spec.json.sig <arch>/ <compiler>/ <name>-<ver>/ # tar.gz-compressed prefix (may support more compression formats later) <arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.spack ``` ## Benefits The major benefit of this change is that the signatures on binary packages can be verified without: 1. Having to download the tarball, or 2. having to extract an unknown tarball. (1) is an improvement in efficiency; (2) is a security fix: we now ensure that we trust the binary before we try to run it through `tar`, which avoids potential attacks. ## Backward compatibility Also after this change, spack should still be able to handle the previous buildcache structure and binary mirrors with mixed layouts.
2022-05-24Add a command to generate a local mirror for bootstrapping (#28556)Massimiliano Culpo7-45/+523
This PR builds on #28392 by adding a convenience command to create a local mirror that can be used to bootstrap Spack. This is to overcome the inconvenience in setting up this mirror manually, which has been reported when trying to setup Spack on air-gapped systems. Using this PR the user can create a bootstrapping mirror, on a machine with internet access, by: % spack bootstrap mirror --binary-packages /opt/bootstrap ==> Adding "clingo-bootstrap@spack+python %apple-clang target=x86_64" and dependencies to the mirror at /opt/bootstrap/local-mirror ==> Adding "gnupg@2.3: %apple-clang target=x86_64" and dependencies to the mirror at /opt/bootstrap/local-mirror ==> Adding "patchelf@0.13.1:0.13.99 %apple-clang target=x86_64" and dependencies to the mirror at /opt/bootstrap/local-mirror ==> Adding binary packages from "https://github.com/alalazo/spack-bootstrap-mirrors/releases/download/v0.1-rc.2/bootstrap-buildcache.tar.gz" to the mirror at /opt/bootstrap/local-mirror To register the mirror on the platform where it's supposed to be used run the following command(s): % spack bootstrap add --trust local-sources /opt/bootstrap/metadata/sources % spack bootstrap add --trust local-binaries /opt/bootstrap/metadata/binaries The mirror has to be moved over to the air-gapped system, and registered using the commands shown at prompt. The command has options to: 1. Add pre-built binaries downloaded from Github (default is not to add them) 2. Add development dependencies for Spack (currently the Python packages needed to use spack style) * bootstrap: refactor bootstrap.yaml to move sources metadata out * bootstrap: allow adding/removing custom bootstrapping sources This operation can be performed from the command line since new subcommands have been added to `spack bootstrap` * Add --trust argument to spack bootstrap add * Add a command to generate a local mirror for bootstrapping * Add a unit test for mirror creation
2022-05-24Best effort co-concretization (iterative algorithm) (#28941)Massimiliano Culpo10-204/+530
Currently, environments can either be concretized fully together or fully separately. This works well for users who create environments for interoperable software and can use `concretizer:unify:true`. It does not allow environments with conflicting software to be concretized for maximal interoperability. The primary use-case for this is facilities providing system software. Facilities provide multiple MPI implementations, but all software built against a given MPI ought to be interoperable. This PR adds a concretization option `concretizer:unify:when_possible`. When this option is used, Spack will concretize specs in the environment separately, but will optimize for minimal differences in overlapping packages. * Add a level of indirection to root specs This commit introduce the "literal" atom, which comes with a few different "arities". The unary "literal" contains an integer that id the ID of a spec literal. Other "literals" contain information on the requests made by literal ID. For instance zlib@1.2.11 generates the following facts: literal(0,"root","zlib"). literal(0,"node","zlib"). literal(0,"node_version_satisfies","zlib","1.2.11"). This should help with solving large environments "together where possible" since later literals can be now solved together in batches. * Add a mechanism to relax the number of literals being solved * Modify spack solve to display the new criteria Since the new criteria is above all the build criteria, we need to modify the way we display the output. Originally done by Greg in #27964 and cherry-picked to this branch by the co-author of the commit. Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com> * Inject reusable specs into the solve Instead of coupling the PyclingoDriver() object with spack.config, inject the concrete specs that can be reused. A method level function takes care of reading from the store and the buildcache. * spack solve: show output of multi-rounds * add tests for best-effort coconcretization * Enforce having at least a literal being solved Co-authored-by: Greg Becker <becker33@llnl.gov>
2022-05-24environments: fail gracefully on missing keys (#26378)Seth R. Johnson1-1/+8
2022-05-24Update regex to correctly identify quoted args (#23494)edwardsp1-1/+1
Previously the regex was only checking for presence of quotes as a beginning or end character and not a matching set. This erroneously identified the following *single* argument as being quoted: source bashenvfile &> /dev/null && python3 -c "import os, json; print(json.dumps(dict(os.environ)))"
2022-05-24compiler flags: imposed hashes impose the lack of additional compiler flags ↵Greg Becker1-0/+5
(#30797)
2022-05-24strip -Werror: all specific or none (#30284)Tom Scogland5-3/+127
Add a config option to strip `-Werror*` or `-Werror=*` from compile lines everywhere. ```yaml config: keep_werror: false ``` By default, we strip all `-Werror` arguments out of compile lines, to avoid unwanted failures when upgrading compilers. You can re-enable `-Werror` in your builds if you really want to, with either: ```yaml config: keep_werror: all ``` or to keep *just* specific `-Werror=XXX` args: ```yaml config: keep_werror: specific ``` This should make swapping in newer versions of compilers much smoother when maintainers have decided to enable `-Werror` by default.
2022-05-24specs: emit better parsing errors for specs. (#24860)Todd Gamblin5-27/+33
Parse error information is kept for specs, but it doesn't seem like we propagate it to the user when we encounter an error. This fixes that. e.g., for this error in a package: ```python depends_on("python@:3.8", when="0.900:") ``` Before, with no context and no clue that it's even from a particular spec: ``` ==> Error: Unexpected token: ':' ``` With this PR: ``` ==> Error: Unexpected token: ':' Encountered when parsing spec: 0.900: ^ ```
2022-05-23Documentation and new method for `CachedCMakePackage` build system (#22706)Greg Becker3-0/+128
Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
2022-05-23archspec: add oneapi and dpcpp flag support (#30783)Massimiliano Culpo2-1/+323
2022-05-23Deprecate `spack:concretization` over `concretizer:unify` (#30038)Harmen Stoppels11-41/+138
* Introduce concretizer:unify option to replace spack:concretization * Deprecate concretization * Make spack:concretization overrule concretize:unify for now * Add environment update logic to move from spack:concretization to spack:concretizer:reuse * Migrate spack:concretization to spack:concretize:unify in all locations * For new environments make concretizer:unify explicit, so that defaults can be changed in 0.19
2022-05-21Fix toolchain detection for oneapi/dpcpp compilers (#30775)Glenn Johnson1-1/+2
The oneapi and dpcpp compilers are essentially the same except for which binary is used foc CXX. Spack will detect them as "mixed toolchain" and not inject compiler optimization flags. This will be needed once archspec has entries for the oneapi and dpcpp compilers. This PR detects when dpcpp and oneapi are in the toolchains list and explicitly sets `is_mixed_toolchain` to `False`.
2022-05-20errors: model error messages as an optimization problem (#30669)Greg Becker8-244/+387
Error messages for the clingo concretizer have proven challenging. The current messages are incredibly vague and often don't help users at all. Unsat cores in clingo are not guaranteed to be minimal, and lead to cores that are either not useful or need to be post-processed for hours to reach a minimal core. Following up on an idea from a slack conversation with kwryankrattiger on slack, this PR takes a new approach. We eliminate most integrity constraints and minima/maxima on choice rules in clingo, and instead force invalid states to imply an error predicate. The error predicate can include context on the cause of the error (Package, Version, etc). These error predicates are then heavily optimized against, to ensure that we do not include error facts in the solution when a solution with no error facts could be generated. When post-processing the clingo solution to construct specs, any error facts cause the program to raise an exception. This leads to much more legible error messages. Each error predicate includes a priority and an error message. The error message is formatted by the remaining arguments to produce the error message. The priority is used to ensure that when clingo has a choice of which rules to violate, it chooses the one which will be most informative to the user. Performance: "fresh" concretizations appear to suffer a ~20% performance penalty under this branch, while "reuse" concretizations see a speedup of around 33%. Possible optimizations if users still see unhelpful messages: There are currently 3 levels of priority of the error messages. Additional priorities are possible, and can allow us finer granularity to ensure more informative error messages are provided in lieu of less informative ones. Future work: Improve tests to ensure that every possible rule implying an error message is exercised
2022-05-19Non-existent upstream is not fatal (#30746)Jordan Galby1-3/+1
A non-existent upstream should not be fatal: it could only mean it is not deployed yet. In the meantime, it should not block the user to rebuild anything it needs. A warning is still emitted, to let the user decide if this is ok or not.
2022-05-19Fix spack install chgrp on symlinks (#30743)Jordan Galby3-4/+7
Fixes missing chgrp on symlinks in package installations, and errors on symlinks referencing non-existent or non-writable locations. Note: `os.chown(.., follow_symlinks=False)` is python3 only, but `os.lchown` exists in both versions.
2022-05-19Don't try to mkdir upstream directory when nonexistent (#30744)Jordan Galby1-2/+2
When an upstream is specified but the directory does not exist, don't create the directory for it, it might not be yours.
2022-05-18Add license dir to config (#30135)robgics8-16/+40
* Change license dir from hard-coded to a configurable item * Change config item to be a string not an array Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2022-05-18bugfix: handle new `dag_hash()` on old concrete specs gracefully. (#30678)Todd Gamblin4-21/+124
Trying to compute `dag_hash()` or `package_hash()` on a concrete spec that doesn't have a `_package_hash` attribute would attempt to recompute the package hash. This most commonly manifests as a failed lookup of a namespace if you attempt to uninstall or compute the hashes of packages in exsternal repositories that aren't registered, e.g.: ```console > spack spec --json c/htno ==> Error: Unknown namespace: myrepo ``` While it wouldn't change the already-assigned `dag_hash` value, this behavior is incorrect, since the package file for a previously concrete spec: 1. might have changed since concretization, 2. might not exist anymore, or 3. might just not be findable by Spack. This PR ensures that the package hash can't be computed on older concrete specs. Instead of calling `package_hash()` from within `to_node_dict()`, we now check for the `_package_hash` attribute and only add the package_hash to the spec record if it's there. This PR also handles the tricky semantics of computing `package_hash()` at concretization time. We have to compute it *before* marking the spec concrete so that `to_node_dict` can use it. But this means that the logic for `package_hash()` can't rely on `spec.concrete`, as it is called *during* concretization. Instead of checking for concreteness, `package_hash()` now checks `_patches_assigned()` to determine whether it should add them to the package hash. - [x] Add an assert to `package_hash()` so it can't be called on specs for which it would be wrong. - [x] Add an `_assign_hash()` method to handle tricky semantics of `package_hash` and `dag_hash`. - [x] Rework concretization to call `_assign_hash()` before and after marking specs concrete. - [x] Rework content hash part of package hash to check for `_patches_assigned()` instead of `spec.concrete`. - [x] regression test
2022-05-18vendored externals: update archspec (#30683)Massimiliano Culpo4-38/+109
- Better support for 164fx - Better support for Apple M1(pro)
2022-05-18Compiler wrapper: fix globbing and debug out.log bell chars (#30699)Harmen Stoppels1-2/+4
* Disable globbing * Split on bell char when dumping cmd to out.log
2022-05-17Mark test_repo_last_mtime xfail on Python < 3.5 (#30696)Massimiliano Culpo1-1/+4
2022-05-16Avoid calling a method on a NoneType object (#30637)andymwood1-1/+2
2022-05-16bugfix: use deterministic edge order for `spack graph` (#30681)Todd Gamblin2-4/+6
Previously we sorted by hash values for `spack graph`, but changing hashes can make the test brittle and the node order seem nondeterministic to users. - [x] Sort nodes in `spack graph` by the default edge order, which takes into account parent and child names as well as dependency types. - [x] Update ASCII test output for new order.
2022-05-15Introduce GroupedExceptionHandler and use it to simplify bootstrap error ↵Danny McClanahan4-37/+143
handling (#30192)
2022-05-15Fix for `spack stage` command not extracting packages in custom paths (#30448)Alberto Invernizzi2-7/+4
2022-05-14uninstall: fix dependency check (#30674)Michael Kuhn1-1/+1
The dependency check currently checks whether there are only build dependencies left for a particular package. However, the database also contains uninstalled packages, which can cause the check to fail. For instance, with `bison` and `flex` having already been uninstalled, `m4` will have the following dependents: ``` bison ('build', 'run')--> m4 flex ('build',)--> m4 libnl ('build',)--> m4 ``` `bison` and `flex` should be ignored in this case because they are not installed anymore. Fixes #30673
2022-05-13Preserve Permissions on .zip extraction (#30407)John W. Parent1-9/+11
#24556 merged in support for Python's .zip file support via ZipFile. However as per #30200 ZipFile does not preserve file permissions of the extracted contents. This PR returns to using the `unzip` executable on non-Windows systems (as was the case before #24556) and now uses `tar` on Windows to extract .zip files.
2022-05-13directory_layout: remove outdated checks for old DAG hashTodd Gamblin2-34/+6
We previously had checks in `directory_layout` to check for build-dependency conflicts when we weren't storing build dependencies. We don't need those anymore; we can just rely on the DAG hash now that it includes everything we know about each spec. - [x] Remove vestigial code for checking installed spec against concrete spec in `ensure_installed()` - [x] Remove `SpecHashCollisionError` -- if specs have the same hash now, they're the same as far as `DirectoryLayout` should be concerned. - [x] Convert spec comparison to `dag_hash()` comparison when adding extensions.
2022-05-13full hash: fix uninstall and gc with full hash DBTodd Gamblin3-5/+26
The database now stores full hashes, so we need to adjust the criteria we use to determine if something can be uninstalled. Specifically, it's ok to uninstall thing that have remaining build-only dependents.
2022-05-13concretizer: enable hash reuse with full hashTodd Gamblin7-87/+156
With the original DAG hash, we did not store build dependencies in the database, but with the full DAG hash, we do. Previously, we'd never tell the concretizer about build dependencies of things used by hash, because we never had them. Now, we have to avoid telling the concretizer about them, or they'll unnecessarily constrain build dependencies for new concretizations. - [x] Make database track all dependencies included in the `dag_hash` - [x] Modify spec_clauses so that build dependency information is optional and off by default. - [x] `spack diff` asks `spec_clauses` for build dependencies for completeness - [x] Modify `concretize.lp` so that reuse optimization doesn't affect fresh installations. - [x] Modify concretizer setup so that it does *not* prioritize installed versions over package versions. We don't need this with reuse, so they're low priority. - [x] Fix `test_installed_deps` for full hash and new concretizer (does not work for old concretizer with full hash -- leave this for later if we need it) - [x] Move `test_installed_deps` mock packages to `builtin.mock` for easier debugging with `spack -m`. - [x] Fix `test_reuse_installed_packages_when_package_def_changes` for full hash