summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2021-07-30adding spack diff command (#22283)Vanessasaurus4-5/+438
A `spack diff` will take two specs, and then use the spack.solver.asp.SpackSolverSetup to generate lists of facts about each (e.g., nodes, variants, etc.) and then take a set difference between the two to show the user the differences. Example output: $ spack diff python@2.7.8 python@3.8.11 ==> Warning: This interface is subject to change. --- python@2.7.8/tsxdi6gl4lihp25qrm4d6nys3nypufbf +++ python@3.8.11/yjtseru4nbpllbaxb46q7wfkyxbuvzxx @@ variant_value @@ - python patches a8c52415a8b03c0e5f28b5d52ae498f7a7e602007db2b9554df28cd5685839b8 + python patches 0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87 @@ version @@ - openssl Version(1.0.2u) + openssl Version(1.1.1k) - python Version(2.7.8) + python Version(3.8.11) Currently this uses diff-like output but we will attempt to improve on this in the future. One use case for `spack diff` is whenever a user has a disambiguate situation and cannot remember how two different installs are different. The command can also output `--json` in the case of a more analysis type use case where we want to save complete data with all diffs and the intersection. However, the command is really more intended for a command line use case, and we likely will have an analyzer more suited to saving data Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com> Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2021-07-29ci: automatically prune the broken-specs list (#24809)Zack Galbreath1-0/+20
When a develop pipeline successfully finishes a `spack install`, check if the spec that was just built is on the broken-specs list. If so, remove it.
2021-07-29Catch ConnectionError from CDash reporter (#24818)Zack Galbreath3-25/+30
* Catch ConnectionError from CDash reporter Catch ConnectionError when attempting to upload the results of `spack install` to CDash. This follows in the spirit of #24299. We do not want `spack install` to exit with a non-zero status when something goes wrong while attempting to report results to CDash. * Catch HTTP Error 400 (Bad Request) in relate_cdash_builds()
2021-07-27bugfix: be careful about GITHUB_BASE_REF in `spack style`Todd Gamblin1-8/+11
`spack style` previously used a Travis CI variable to figure out what the base branch of a PR was, and this was apparently also set on `develop`. We switched to `GITHUB_BASE_REF` to support GitHub Actions, but it looks like this is set to `""` in pushes to develop, so `spack style` breaks there. This PR does two things: - [x] Remove `GITHUB_BASE_REF` knowledge from `spack style` entirely - [x] Handle `GITHUB_BASE_REF` in style scripts instead, and explicitly pass the base ref if it is present, but don't otherwise. This makes `spack style` *not* dependent on the environment and fixes handling of the base branch in the right place.
2021-07-27`spack style`: add `--root` option (#25085)Todd Gamblin4-50/+285
This adds a `--root` option so that `spack style` can check style for a spack instance other than its own. We also change the inner workings of `spack style` so that `--config FILE` (and similar options for the various tools) options are used. This ensures that when `spack style` runs, it always uses the config from the running spack, and does *not* pick up configuration from the external root. - [x] add `--root` option to `spack style` - [x] add `--config` (or similar) option when invoking style tools - [x] add a test that verifies we can check an external instance
2021-07-26Documentation does not build with Sphinx 4.1.2Adam J. Stewart3-4/+12
2021-07-26Confirm that the docstring is the issueAdam J. Stewart1-6/+6
2021-07-26Docs: attempt to fix doc tests for sphinx 4.1.2Adam J. Stewart1-5/+5
2021-07-26intel-oneapi: fix parallel installer errors (#24911)Dylan Simon1-0/+3
Intel oneAPI installs maintain a lock file in XDG_RUNTIME_DIR, which by default exists in /tmp (and is shared by all component installs). This prevented multiple oneAPI components from being installed in parallel. This commit sets XDG_RUNTIME_DIR to exist within Spack's installation Stage, so allows multiple components to be installed at the same time.
2021-07-24`spack style`: automatically bootstrap dependenciesvsoch2-8/+43
This uses our bootstrapping logic to automatically install dependencies for `spack style`. Users should no longer have to pre-install all of the tools (`isort`, `mypy`, `black`, `flake8`). The command will do it for them. - [x] add logic to bootstrap specs with specific version requirements in `spack style` - [x] remove style tools from CI requirements (to ensure we test bootstrapping) - [x] rework dependencies for `mypy` and `py-typed-ast` - `py-typed-ast` needs to be a link dependency - it needs to be at 1.4.1 or higher to work with python 3.9 Signed-off-by: vsoch <vsoch@users.noreply.github.com>
2021-07-24Bump codecov/action to v2.0.2 (#24983)Massimiliano Culpo1-1/+0
* build(deps): bump codecov/codecov-action from 1 to 2.0.1 Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 1 to 2.0.1. - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md) - [Commits](https://github.com/codecov/codecov-action/compare/v1...v2.0.1) --- updated-dependencies: - dependency-name: codecov/codecov-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Update arguments to codecov action * Try to delete the symbolic link to root folder Hopefully this should get rid of the ELOOP error * Delete also for shell tests and MacOS * Bump to v2.0.2 Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-07-21Fix bootstrap from sourcesMassimiliano Culpo1-6/+4
#24095 introduced a couple of bugs, which are fixed here: 1. The module path is computed incorrectly for bootstrapped clingo 2. We remove too many paths for `sys.path` in case of failures
2021-07-19Fix S3 urlparse scheme error and add test (#24952)Harmen Stoppels2-4/+13
2021-07-16Sphinx 3.4+ required for correct reference target linkingAdam J. Stewart3-3/+3
2021-07-16API Docs: fix broken reference targetsAdam J. Stewart57-471/+510
2021-07-16Python: query distutils to find site-packages directory (#24095)Adam J. Stewart3-43/+32
Third-party Python libraries may be installed in one of several directories: 1. `lib/pythonX.Y/site-packages` for Spack-installed Python 2. `lib64/pythonX.Y/site-packages` for system Python on RHEL/CentOS/Fedora 3. `lib/pythonX/dist-packages` for system Python on Debian/Ubuntu Previously, Spack packages were hard-coded to use the (1). Now, we query the Python installation itself and ask it which to use. Ever since #21446 this is how we've been determining where to install Python libraries anyway. Note: there are still many packages that are hard-coded to use (1). I can change them in this PR, but I don't have the bandwidth to test all of them. * Python: handle dist-packages and site-packages * Query Python to find site-packages directory * Add try-except statements for when distutils isn't installed * Catch more errors * Fix root directory used in import tests * Rely on site_packages_dir property
2021-07-15archspec: added support for arm compiler on graviton2 (#24904)Massimiliano Culpo2-1/+7
2021-07-14spec may be a string, use precomputed namespace (#24867)Greg Becker1-1/+1
2021-07-13binary_distribution: relocate x-pie-executable files (#24854)eugeneswalker1-1/+1
2021-07-13Fix KeyboardInterrupt signal for Python 2Adam J. Stewart1-1/+4
2021-07-12Enable/disable bootstrapping and customize store location (#23677)Massimiliano Culpo11-10/+299
* Permit to enable/disable bootstrapping and customize store location This PR adds configuration handles to allow enabling and disabling bootstrapping, and to customize the store location. * Move bootstrap related configuration into its own YAML file * Add a bootstrap command to manage configuration
2021-07-12build output: filter padding out of console output when `padded_length` is ↵Todd Gamblin5-7/+188
used (#24514) Spack allows users to set `padded_length` to pad out the installation path in build farms so that any binaries created are more easily relocatable. The issue with this is that the padding dominates installation output and makes it difficult to see what is going on. The padding also causes logs to easily exceed size limits for things like GitLab artifacts. This PR fixes this by adding a filter in the logger daemon. If you use a setting like this: config: install_tree: padded_length: 512 Then lines like this in the output: ==> [2021-06-23-15:59:05.020387] './configure' '--prefix=/Users/gamblin2/padding-log-test/opt/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_pla/darwin-bigsur-skylake/apple-clang-12.0.5/zlib-1.2.11-74mwnxgn6nujehpyyalhwizwojwn5zga will be replaced with the much more readable: ==> [2021-06-23-15:59:05.020387] './configure' '--prefix=/Users/gamblin2/padding-log-test/opt/[padded-to-512-chars]/darwin-bigsur-skylake/apple-clang-12.0.5/zlib-1.2.11-74mwnxgn6nujehpyyalhwizwojwn5zga You can see that the padding has been replaced with `[padded-to-512-chars]` to indicate the total number of characters in the padded prefix. Over a long log file, this should save a lot of space and allow us to see error messages in GitHub/GitLab log output. The *actual* build logs still have full paths in them. Also lines that are output by Spack and not by a package build are not filtered and will still display the fully padded path. There aren't that many of these, so the change should still help reduce file size and readability quite a bit.
2021-07-11docs: fix reference to count of system packages (#24821)Matthew Fernandez1-1/+1
015e29efe105ddd039e8b395e12cf78a3787ebb3 that introduced this section to the documentation said “two” here instead of the actual count, three. 9f54cea5c500bb19d2301149460c741e96be1f8b then added a fourth, BLAS/LAPACK. Rather than trying to keep this leading count in sync, this change just replaces the wording with something more generic/stable.
2021-07-10Fix out of date remote spec.yamls while updating the index (#24811)Scott Wittenburg2-10/+130
2021-07-09debug: initial global debug flag support (#24285)Vanessasaurus2-0/+74
The developer can export environment variables that are seen by the compiler wrapper to then add the flags
2021-07-10spec.splice: properly handle cached hash invalidations (#24795)Greg Becker3-18/+91
* spec.splice: properly handle cached hash invalidations * make package_hash a cached hash on the spec
2021-07-09ci: tolerate 'InvalidAccessKeyId' (#24741)Zack Galbreath1-1/+1
Add 'InvalidAccessKeyId' to the list of error messages we expect and tolerate in push_mirror_contents()
2021-07-09Remove unnecessary compiler id override for XL (#24799)Chris White1-15/+0
2021-07-08imports: sort imports everywhere in Spack (#24695)Todd Gamblin327-863/+1045
* fix remaining flake8 errors * imports: sort imports everywhere in Spack We enabled import order checking in #23947, but fixing things manually drives people crazy. This used `spack style --fix --all` from #24071 to automatically sort everything in Spack so PR submitters won't have to deal with it. This should go in after #24071, as it assumes we're using `isort`, not `flake8-import-order` to order things. `isort` seems to be more flexible and allows `llnl` mports to be in their own group before `spack` ones, so this seems like a good switch.
2021-07-07cvs tests: don't use dateutil at allTodd Gamblin1-18/+21
`dateutil.parser` was an optional dependency for CVS tests. It was failing on macOS beacuse the dateutil types were not being installed, and mypy was failing *even when the CVS tests were skipped*. This seems like it was an oversight on macOS -- `types-dateutil-parser` was not installed there, though it was on Linux unit tests. It takes 6 lines of YAML and some weird test-skipping logic to get `python-dateutil` and `types-python-dateutil` installed in all the tests where we need them, but it only takes 4 lines of code to write the date parser we need for CVS, so I just did that instead. Note that CVS date format can vary from system to system, but it seems like it's always pretty similar for the parts we care about. - [x] Replace dateutil.parser with a simpler date regex - [x] Lose the dependency on `dateutil.parser`
2021-07-07style: get close to full coverage of `spack style`Todd Gamblin3-95/+182
Previous tests of `spack style` didn't really run the tools -- they just ensure that the commands worked enough to get coverage. This adds several real tests and ensures that we hit the corner cases in `spack style`. This also tests sucess as well as failure cases.
2021-07-07style: clean up and restructure `spack style` commandTodd Gamblin1-93/+77
This consolidates code across tools in `spack style` so that each `run_<tool>` function can be called indirecty through a dictionary of handlers, and os that checks like finding the executable for the tool can be shared across commands. - [x] rework `spack style` to use decorators to register tools - [x] define tool order in one place in `spack style` - [x] fix python 2/3 issues to Get `isort` checks working - [x] make isort error regex more robust across versions - [x] remove unused output option - [x] change vestigial `TRAVIS_BRANCH` to `GITHUB_BASE_REF` - [x] update completion
2021-07-07style: Move isort configuration to pyproject.tomlTodd Gamblin1-4/+0
- [x] Remove flake8-import-order checks, as we only need isort for this - [x] Clean up configuration and requirements
2021-07-07style: add support for `isort` and `--fix`Danny McClanahan1-9/+55
2021-07-06oneAPI packages: add 2021.3 release (#24617)Vasily Danilin2-12/+35
2021-07-06fix buffered download (#24623)Harmen Stoppels1-8/+8
* Use shutil to do a buffered copy from http response to file * Fix flake8... * Somehow flake8 still complains about unrelated files
2021-07-06installer: fix double print of exception (#24697)eugeneswalker1-0/+1
2021-07-02Fix fetching for Python 3.9.6 (#24686)Adam J. Stewart1-6/+6
When using Python 3.9.6, Spack is no longer able to fetch anything. Commands like `spack fetch` and `spack install` all break. Python 3.9.6 includes a [new change](https://github.com/python/cpython/pull/25853/files#diff-b3712475a413ec972134c0260c8f1eb1deefb66184f740ef00c37b4487ef873eR462) that means that `scheme` must be a string, it cannot be None. The solution is to use an empty string like the method default. Fixes #24644. Also see https://github.com/Homebrew/homebrew-core/pull/80175 where this issue was discovered by CI. Thanks @branchvincent for reporting such a serious issue before any actual users encountered it! Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2021-07-02Pipelines: Improve broken specs check (#24643)Scott Wittenburg2-21/+25
We should not fail the generate stage simply due to the presence of a broken-spec somewhere in the DAG. Only fail if the known broken spec needs to be rebuilt.
2021-07-02Add `when` context manager to group common constraints in packages (#24650)Massimiliano Culpo4-63/+183
This PR adds a context manager that permit to group the common part of a `when=` argument and add that to the context: ```python class Gcc(AutotoolsPackage): with when('+nvptx'): depends_on('cuda') conflicts('@:6', msg='NVPTX only supported in gcc 7 and above') conflicts('languages=ada') conflicts('languages=brig') conflicts('languages=go') ``` The above snippet is equivalent to: ```python class Gcc(AutotoolsPackage): depends_on('cuda', when='+nvptx') conflicts('@:6', when='+nvptx', msg='NVPTX only supported in gcc 7 and above') conflicts('languages=ada', when='+nvptx') conflicts('languages=brig', when='+nvptx') conflicts('languages=go', when='+nvptx') ``` which needs a repetition of the `when='+nvptx'` argument. The context manager might help improving readability and permits to group together directives related to the same semantic aspect (e.g. all the directives needed to model the behavior of `gcc` when `+nvptx` is active). Modifications: - [x] Added a `when` context manager to be used with package directives - [x] Add unit tests and documentation for the new feature - [x] Modified `cp2k` and `gcc` to show the use of the context manager
2021-07-01meson: fix typo (libs instead of default_library) (#24649)Harmen Stoppels1-2/+2
2021-06-30ci: only write to broken-specs list on SpackError (#24618)Zack Galbreath2-18/+20
ci: only write to broken-specs list on SpackError Only write to the broken-specs list when `spack install` raises a SpackError, instead of writing to this list unnecessarily when infrastructure-related problems prevent a develop job from completing successfully.
2021-06-30ci: Don't raise an exception if we fail to relate CDash builds (#24299)Zack Galbreath4-73/+96
ci: Do not allow cdash-related errors to cause pipeline failure
2021-06-29spack style: warn if flake8-import-order is missing (#24590)Adam J. Stewart1-4/+7
2021-06-29Fix prefix-collision detection for projections (#24049)Harmen Stoppels4-32/+80
If two Specs have the same hash (and prefix) but are not equal, Spack originally had logic to detect this and raise an error (since both cannot be installed in the same place). Recently this has eroded and the check no-longer works; moreover, when defining projections (which may truncate the hash or other distinguishing properties from the prefix) Spack was also failing to detect collisions (in both of these cases, Spack would overwrite the old prefix with the new Spec). This PR maintains a list of all "taken" prefixes: if a hash is not registered (i.e. recorded as installed in the database) but the prefix is occupied, that is a collision. This can detect collisions created by defining projections (specifically when they omit the hash). The PR does not detect collisions where specs have the same hash (and prefix) but are not equal.
2021-06-28Use flake8-import-order to enforce PEP-8 compliance (#23947)Adam J. Stewart1-9/+9
2021-06-28CUDA: 11.0.3 (11.0 "Update 1") (#24481)Axel Huebl1-4/+4
Add CUDA 11.0.3. This release adds new features such as NVCC flags `--forward-unknown-to-host-compiler` and `--forward-unknown-to-host-linker`
2021-06-27docs: link projections docs to spec format (#24478)Dylan Simon1-1/+2
2021-06-26Update archspec to support arm compiler on a64fx (#24524)Massimiliano Culpo2-1/+13
2021-06-25Doc: Note on required changes after merge of reproducible builds (#24347)Adrien Bernede1-1/+18
* Suggestion of a note for conversion of existing pipelines. * Wording * Fix format in .rst note * Wording