summaryrefslogtreecommitdiff
path: root/share
AgeCommit message (Collapse)AuthorFilesLines
2021-10-25containerize: pin the Spack version used in a container (#21910)Massimiliano Culpo11-4/+181
This PR permits to specify the `url` and `ref` of the Spack instance used in a container recipe simply by expanding the YAML schema as outlined in #20442: ```yaml container: images: os: amazonlinux:2 spack: ref: develop resolve_sha: true ``` The `resolve_sha` option, if true, verifies the `ref` by cloning the Spack repository in a temporary directory and transforming any tag or branch name to a commit sha. When this new ability is leveraged an additional "bootstrap" stage is added, which builds an image with Spack setup and ready to install software. The Spack repository to be used can be customized with the `url` keyword under `spack`. Modifications: - [x] Permit to pin the version of Spack, either by branch or tag or sha - [x] Added a few new OSes (centos:8, amazonlinux:2, ubuntu:20.04, alpine:3, cuda:11.2.1) - [x] Permit to print the bootstrap image as a standalone - [x] Add documentation on the new part of the schema - [x] Add unit tests for different use cases
2021-10-22E4S amd64 CI: add parsec (#26906)eugeneswalker1-0/+1
2021-10-21update E4S CI environments in preparation for 21.11 release (#26826)eugeneswalker2-283/+34
* update E4S CI environments in preparation for 21.11 release * e4s ci env: use clingo
2021-10-20Add --preferred and --latest to`spack checksum` (#25830)Tamara Dahlgren1-1/+1
2021-10-19Relax os constraints in e4s pipelines (#26547)Massimiliano Culpo3-3/+3
2021-10-19Gitlab pipelines: use images from the Spack organization (#26796)Massimiliano Culpo6-16/+13
2021-10-11Add `spack env activate --temp` (#25388)Harmen Stoppels3-3/+15
Creates an environment in a temporary directory and activates it, which is useful for a quick ephemeral environment: ``` $ spack env activate -p --temp [spack-1a203lyg] $ spack add zlib ==> Adding zlib to environment /tmp/spack-1a203lyg ==> Updating view at /tmp/spack-1a203lyg/.spack-env/view ```
2021-10-11Remove unused --dependencies flag (#25731)Harmen Stoppels1-1/+1
2021-10-01Retry pipeline generation jobs in certain casesScott Wittenburg1-0/+5
2021-10-01Add DAG scheduling to child pipelinesScott Wittenburg1-0/+30
2021-10-01Service jobs do not need an active environmentScott Wittenburg5-10/+0
2021-10-01Activate concrete env dir in all service jobsScott Wittenburg5-10/+6
2021-10-01Use same image to build e4s as to generate itScott Wittenburg1-3/+3
2021-10-01Use same image to build dvsdk as to generate itScott Wittenburg1-2/+2
2021-10-01Use default runner image for radiussScott Wittenburg1-6/+2
2021-09-30Build container images on Github Actions and push to multiple registries ↵Massimiliano Culpo5-28/+23
(#26247) Modifications: - Modify the workflow to build container images without pushing when the workflow file itself is modified - Strip the leading ghcr.io/spack/ from env.container env.versioned to prepare pushing to multiple registries - Fixed CentOS 7 and Amazon Linux builds - Login and push to Docker Hub as well as Github Action - Add a badge to README.md with the status of docker images
2021-09-24Pipelines: Disable ppc builds until we have resources or make it smaller ↵Scott Wittenburg1-24/+24
(#26238)
2021-09-24Remove centos:6 image references (#26095)Harmen Stoppels1-74/+0
This was EOL November 30th, 2020. I believe the "builds" are failing on develop because of it.
2021-09-14Make clingo the default solver (#25502)Massimiliano Culpo1-1/+1
Modifications: - [x] Change `defaults/config.yaml` - [x] Add a fix for bootstrapping patchelf from sources if `compilers.yaml` is empty - [x] Make `SPACK_TEST_SOLVER=clingo` the default for unit-tests - [x] Fix package failures in the e4s pipeline Caveats: 1. CentOS 6 still uses the original concretizer as it can't connect to the buildcache due to issues with `ssl` (bootstrapping from sources requires a C++14 capable compiler) 1. I had to update the image tag for GitlabCI in e699f14. 1. libtool v2.4.2 has been deprecated and other packages received some update
2021-09-14Pipelines: (Re)enable E4S on Power stack (#25921)Scott Wittenburg2-21/+21
Pipelines: (Re)enable E4S on Power stack
2021-09-13ci: Add ecp-data-vis-sdk CI pipeline (#22179)Chuck Atkins2-0/+92
* ci: Add a minimal subset of the ECP Data & Vis SDK CI pipeline * ci: Expand the ECP Data & Vis SDK pipeline with more variants
2021-09-10Pipelines: build kokkos-kernels on bigger instance (#25845)Scott Wittenburg1-0/+1
2021-09-09specs: move to new spec.json format with build provenance (#22845)Nathan Hanford1-8/+8
This is a major rework of Spack's core core `spec.yaml` metadata format. It moves from `spec.yaml` to `spec.json` for speed, and it changes the format in several ways. Specifically: 1. The spec format now has a `_meta` section with a version (now set to version `2`). This will simplify major changes like this one in the future. 2. The node list in spec dictionaries is no longer keyed by name. Instead, it is a list of records with no required key. The name, hash, etc. are fields in the dictionary records like any other. 3. Dependencies can be keyed by any hash (`hash`, `full_hash`, `build_hash`). 4. `build_spec` provenance from #20262 is included in the spec format. This means that, for spliced specs, we preserve the *full* provenance of how to build, and we can reproduce a spliced spec from the original builds that produced it. **NOTE**: Because we have switched the spec format, this PR changes Spack's hashing algorithm. This means that after this commit, Spack will think a lot of things need rebuilds. There are two major benefits this PR provides: * The switch to JSON format speeds up Spack significantly, as Python's builtin JSON implementation is orders of magnitude faster than YAML. * The new Spec format will soon allow us to represent DAGs with potentially multiple versions of the same dependency -- e.g., for build dependencies or for compilers-as-dependencies. This PR lays the necessary groundwork for those features. The old `spec.yaml` format continues to be supported, but is now considered a legacy format, and Spack will opportunistically convert these to the new `spec.json` format.
2021-09-08url stats: add `--show-issues` option (#25792)Todd Gamblin1-1/+1
* tests: make `spack url [stats|summary]` work on mock packages Mock packages have historically had mock hashes, but this means they're also invalid as far as Spack's hash detection is concerned. - [x] convert all hashes in mock package to md5 or sha256 - [x] ensure that all mock packages have a URL - [x] ignore some special cases with multiple VCS fetchers * url stats: add `--show-issues` option `spack url stats` tells us how many URLs are using what protocol, type of checksum, etc., but it previously did not tell us which packages and URLs had the issues. This adds a `--show-issues` option to show URLs with insecure (`http`) URLs or `md5` hashes (which are now deprecated by NIST).
2021-09-06Update pinned OpenSSL version to 1.1.1l (#25787)Scott Wittenburg2-2/+2
Update to the latest version of openssl, as the previous one (1.1.1k) is now deprecated, so spack can no longer rebuild it from source.
2021-09-02start of work to add spack audit packages-https checker (#25670)Vanessasaurus1-1/+10
This PR will add a new audit, specifically for spack package homepage urls (and eventually other kinds I suspect) to see if there is an http address that can be changed to https. Usage is as follows: ```bash $ spack audit packages-https <package> ``` And in list view: ```bash $ spack audit list generic: Generic checks relying on global variables configs: Sanity checks on compilers.yaml Sanity checks on packages.yaml packages: Sanity checks on specs used in directives packages-https: Sanity checks on https checks of package urls, etc. ``` I think it would be unwise to include with packages, because when run for all, since we do requests it takes a long time. I also like the idea of more well scoped checks - likely there will be other addresses for http/https within a package that we eventually check. For now, there are two error cases - one is when an https url is tried but there is some SSL error (or other error that means we cannot update to https): ```bash $ spack audit packages-https zoltan PKG-HTTPS-DIRECTIVES: 1 issue found 1. Error with attempting https for "zoltan": <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'www.cs.sandia.gov'. (_ssl.c:1125)> ``` This is either not fixable, or could be fixed with a change to the url or (better) contacting the site owners to ask about some certificate or similar. The second case is when there is an http that needs to be https, which is a huge issue now, but hopefully not after this spack PR. ```bash $ spack audit packages-https xman Package "xman" uses http but has a valid https endpoint. ``` And then when a package is fixed: ```bash $ spack audit packages-https zlib PKG-HTTPS-DIRECTIVES: 0 issues found. ``` And that's mostly it. :) Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com>
2021-09-01Update versions for RAJA, CHAI, Umpire and camp (#25528)David Beckingsale2-4/+4
2021-08-30Pipelines: disable power builds (#25704)Scott Wittenburg1-18/+18
2021-08-27Fix fish test "framework" (#25242)Harmen Stoppels1-34/+16
Remove broken test, see #21699
2021-08-24fixing small bug that a line of spack monitor commands are still produced ↵Vanessasaurus1-1/+4
(#25366) Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com>
2021-08-19Use kcov from official Ubuntu 20.04 repository (#25385)Massimiliano Culpo3-5/+8
* Ubuntu 20.04 provides kcov, so don't build from source * Use two undocumented options for kcov v3.8
2021-08-19buildcache: Add environment-aware buildcache sync command (#25470)Scott Wittenburg1-1/+5
2021-08-18Bootstrap clingo from binaries (#22720)Massimiliano Culpo2-2/+281
* Bootstrap clingo from binaries * Move information on clingo binaries to a JSON file * Add support to bootstrap on Cray Bootstrapping on Cray requires, at the moment, to swap the platform when looking for binaries - due to #22800. * Add SHA256 verification for bootstrapped software Use sha256 verification for binaries necessary to bootstrap the concretizer and gpg for signature verification * patchelf: use Spec._old_concretize() to bootstrap As noted in #24450 we may happen to need the concretizer when bootstrapping clingo. In that case only the old concretizer is available. * Add a schema for bootstrapping methods Two fields have been added to bootstrap.yaml: "sources" which lists the methods available for bootstrapping software "trusted" which records if a source is trusted or not A subcommand has been added to "spack bootstrap" to list the sources currently available. * Methods used for bootstrapping are configurable from bootstrap:sources The function that tries to ensure a given Python module is importable now tries bootstrapping methods in the same order as they are defined in `bootstrap.yaml` * Permit to trust/untrust bootstrapping methods * Add binary tests for MacOS, Ubuntu * Add documentation * Add a note on bash
2021-08-16e4s ci: further expand power stack (#25405)eugeneswalker1-23/+22
2021-08-16Second pass at increasing RADIUSS cloud CI packages (#25321)Tamara Dahlgren1-7/+7
2021-08-09ci pipelines: expand the list of RADIUSS packages (#25282)Tamara Dahlgren1-10/+10
2021-08-06Add New Build Containers Workflow (#24257)Alec Scott1-1/+1
This pull request adds a new workflow to build and deploy Spack Docker containers from GitHub Actions. In comparison with our current system where we use Dockerhub's CI to build our Docker containers, this workflow will allow us to now build for multiple architectures and deploy to multiple registries. (At the moment x86_64 and Arm64 because ppc64le is throwing an error within archspec.) As currently set up, the PR will build all of the current containers (minus Centos6 because those yum repositories are no longer available?) as both x86_64 and Arm64 variants. The workflow is currently setup to build and deploy containers nightly from develop as well as on tagged releases. The workflow will also build, but NOT deploy containers on a pull request for the purposes of testing this PR. At the moment it is setup to deploy the built containers to GitHub's Container Registry although, support for also uploading to Dockerhub/Quay can be included easily if we decide to keep releasing on Dockerhub/want to begin releasing on Quay.
2021-08-03e4s ci stack: update package preferences (#25163)eugeneswalker1-4/+17
2021-08-02ci: Add RADIUSS stack to cloud CI (#23922)Tamara Dahlgren2-2/+131
Add RADIUSS software stack to gitlab PR testing pipelines
2021-07-30CI: capture stdout/stderr output to artifact files (#24401)Scott Wittenburg1-1/+2
Gitlab truncates job trace output (even the complete raw output) at 4MB, so this change captures it to a file under "user_data" artifacts as well, to make sure we can debug output from the end of the rebuild job.
2021-07-30adding spack diff command (#22283)Vanessasaurus1-1/+10
A `spack diff` will take two specs, and then use the spack.solver.asp.SpackSolverSetup to generate lists of facts about each (e.g., nodes, variants, etc.) and then take a set difference between the two to show the user the differences. Example output: $ spack diff python@2.7.8 python@3.8.11 ==> Warning: This interface is subject to change. --- python@2.7.8/tsxdi6gl4lihp25qrm4d6nys3nypufbf +++ python@3.8.11/yjtseru4nbpllbaxb46q7wfkyxbuvzxx @@ variant_value @@ - python patches a8c52415a8b03c0e5f28b5d52ae498f7a7e602007db2b9554df28cd5685839b8 + python patches 0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87 @@ version @@ - openssl Version(1.0.2u) + openssl Version(1.1.1k) - python Version(2.7.8) + python Version(3.8.11) Currently this uses diff-like output but we will attempt to improve on this in the future. One use case for `spack diff` is whenever a user has a disambiguate situation and cannot remember how two different installs are different. The command can also output `--json` in the case of a more analysis type use case where we want to save complete data with all diffs and the intersection. However, the command is really more intended for a command line use case, and we likely will have an analyzer more suited to saving data Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com> Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2021-07-29Move build tests from GA to Gitlab (#25120)Massimiliano Culpo2-1/+92
Modifications: - Remove the "build tests" workflow from GitHub Actions - Setup a similar e2e test on Gitlab In this way we'll reduce load on GitHub Actions workflows and for e2e tests will benefit from the buildcache reuse granted by pipelines.
2021-07-27bugfix: be careful about GITHUB_BASE_REF in `spack style`Todd Gamblin1-1/+6
`spack style` previously used a Travis CI variable to figure out what the base branch of a PR was, and this was apparently also set on `develop`. We switched to `GITHUB_BASE_REF` to support GitHub Actions, but it looks like this is set to `""` in pushes to develop, so `spack style` breaks there. This PR does two things: - [x] Remove `GITHUB_BASE_REF` knowledge from `spack style` entirely - [x] Handle `GITHUB_BASE_REF` in style scripts instead, and explicitly pass the base ref if it is present, but don't otherwise. This makes `spack style` *not* dependent on the environment and fixes handling of the base branch in the right place.
2021-07-27`spack style`: add `--root` option (#25085)Todd Gamblin1-2/+2
This adds a `--root` option so that `spack style` can check style for a spack instance other than its own. We also change the inner workings of `spack style` so that `--config FILE` (and similar options for the various tools) options are used. This ensures that when `spack style` runs, it always uses the config from the running spack, and does *not* pick up configuration from the external root. - [x] add `--root` option to `spack style` - [x] add `--config` (or similar) option when invoking style tools - [x] add a test that verifies we can check an external instance
2021-07-24`spack style`: automatically bootstrap dependenciesvsoch1-1/+0
This uses our bootstrapping logic to automatically install dependencies for `spack style`. Users should no longer have to pre-install all of the tools (`isort`, `mypy`, `black`, `flake8`). The command will do it for them. - [x] add logic to bootstrap specs with specific version requirements in `spack style` - [x] remove style tools from CI requirements (to ensure we test bootstrapping) - [x] rework dependencies for `mypy` and `py-typed-ast` - `py-typed-ast` needs to be a link dependency - it needs to be at 1.4.1 or higher to work with python 3.9 Signed-off-by: vsoch <vsoch@users.noreply.github.com>
2021-07-24Bump codecov/action to v2.0.2 (#24983)Massimiliano Culpo2-0/+13
* build(deps): bump codecov/codecov-action from 1 to 2.0.1 Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 1 to 2.0.1. - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md) - [Commits](https://github.com/codecov/codecov-action/compare/v1...v2.0.1) --- updated-dependencies: - dependency-name: codecov/codecov-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Update arguments to codecov action * Try to delete the symbolic link to root folder Hopefully this should get rid of the ELOOP error * Delete also for shell tests and MacOS * Bump to v2.0.2 Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-07-16trilinos: simplify some variants (#24820)Seth R. Johnson2-2/+2
* trilinos: rename basker variant The Basker solver is part of amesos2 but is clearer without the extra scoping. * trilinos: automatically enable teuchos and remove variant Basically everything in trilinos needs teuchos * trilinos: group top-level dependencies * trilinos: update dependencies, removing unused - GLM, X11 are unused (x11 lacks dependency specs too) - Python variant is more like a TPL so rearrange that - Gtest internal package shouldn't be compiled or exported - Add MPI4PY requirement for pytrilinos * trilinos: remove package meta-options - XSDK settings and "all opt packages" are not used anywhere - all optional packages are dangerous * trilinos: Use hwloc iff kokkos See #19119, also the HWLOC tpl name was misspelled so this was being ignored before. * Flake * Fix trilinos +netcdf~mpi * trilinos: default to disabling external dependencies * Remove teuchos from downstream dependencies * fixup! trilinos: Use hwloc iff kokkos * Add netcdf requirements to packages with ^trilinos+exodus * trilinos: disable exodus by default * fixup! Add netcdf requirements to packages with ^trilinos+exodus * trilinos: only enable hwloc when @13: +kokkos * xyce: propagate trilinos dependencies more simply * dtk: fix missing boost dependency * trilinos: remove explicit metis dependency * trilinos: require metis/parmetis for zoltan Disable zoltan by default to minimize default dependencies * trilinos: mark mesquite disabled and fix kokkos arch * xsdk: fix trilinos to also list zoltan [with zoltan2] * ci: remove nonexistent variant from trilinos * trilinos: add missing boost dependency Co-authored-by: Satish Balay <balay@mcs.anl.gov>
2021-07-15ci: build trilinos with all variants on (#24908)eugeneswalker2-3/+2
2021-07-12Enable/disable bootstrapping and customize store location (#23677)Massimiliano Culpo1-1/+31
* Permit to enable/disable bootstrapping and customize store location This PR adds configuration handles to allow enabling and disabling bootstrapping, and to customize the store location. * Move bootstrap related configuration into its own YAML file * Add a bootstrap command to manage configuration
2021-07-11Expat: add version 2.4.0, 2.4.1; fix CVE-2013-0340 (#24669)Christoph Conrads2-2/+2
* Expat: add version 2.4.0, 2.4.1; fix CVE-2013-0340 fixes #24628 * E4S pipeline: update pinned Expat version
generated by cgit v1.2.3-60-g2f50 (git 2.42.0) at 2024-07-03 06:13:09 +0000