summaryrefslogtreecommitdiff
path: root/share
AgeCommit message (Collapse)AuthorFilesLines
2021-09-10Pipelines: build kokkos-kernels on bigger instance (#25845)Scott Wittenburg1-0/+1
2021-09-09specs: move to new spec.json format with build provenance (#22845)Nathan Hanford1-8/+8
This is a major rework of Spack's core core `spec.yaml` metadata format. It moves from `spec.yaml` to `spec.json` for speed, and it changes the format in several ways. Specifically: 1. The spec format now has a `_meta` section with a version (now set to version `2`). This will simplify major changes like this one in the future. 2. The node list in spec dictionaries is no longer keyed by name. Instead, it is a list of records with no required key. The name, hash, etc. are fields in the dictionary records like any other. 3. Dependencies can be keyed by any hash (`hash`, `full_hash`, `build_hash`). 4. `build_spec` provenance from #20262 is included in the spec format. This means that, for spliced specs, we preserve the *full* provenance of how to build, and we can reproduce a spliced spec from the original builds that produced it. **NOTE**: Because we have switched the spec format, this PR changes Spack's hashing algorithm. This means that after this commit, Spack will think a lot of things need rebuilds. There are two major benefits this PR provides: * The switch to JSON format speeds up Spack significantly, as Python's builtin JSON implementation is orders of magnitude faster than YAML. * The new Spec format will soon allow us to represent DAGs with potentially multiple versions of the same dependency -- e.g., for build dependencies or for compilers-as-dependencies. This PR lays the necessary groundwork for those features. The old `spec.yaml` format continues to be supported, but is now considered a legacy format, and Spack will opportunistically convert these to the new `spec.json` format.
2021-09-08url stats: add `--show-issues` option (#25792)Todd Gamblin1-1/+1
* tests: make `spack url [stats|summary]` work on mock packages Mock packages have historically had mock hashes, but this means they're also invalid as far as Spack's hash detection is concerned. - [x] convert all hashes in mock package to md5 or sha256 - [x] ensure that all mock packages have a URL - [x] ignore some special cases with multiple VCS fetchers * url stats: add `--show-issues` option `spack url stats` tells us how many URLs are using what protocol, type of checksum, etc., but it previously did not tell us which packages and URLs had the issues. This adds a `--show-issues` option to show URLs with insecure (`http`) URLs or `md5` hashes (which are now deprecated by NIST).
2021-09-06Update pinned OpenSSL version to 1.1.1l (#25787)Scott Wittenburg2-2/+2
Update to the latest version of openssl, as the previous one (1.1.1k) is now deprecated, so spack can no longer rebuild it from source.
2021-09-02start of work to add spack audit packages-https checker (#25670)Vanessasaurus1-1/+10
This PR will add a new audit, specifically for spack package homepage urls (and eventually other kinds I suspect) to see if there is an http address that can be changed to https. Usage is as follows: ```bash $ spack audit packages-https <package> ``` And in list view: ```bash $ spack audit list generic: Generic checks relying on global variables configs: Sanity checks on compilers.yaml Sanity checks on packages.yaml packages: Sanity checks on specs used in directives packages-https: Sanity checks on https checks of package urls, etc. ``` I think it would be unwise to include with packages, because when run for all, since we do requests it takes a long time. I also like the idea of more well scoped checks - likely there will be other addresses for http/https within a package that we eventually check. For now, there are two error cases - one is when an https url is tried but there is some SSL error (or other error that means we cannot update to https): ```bash $ spack audit packages-https zoltan PKG-HTTPS-DIRECTIVES: 1 issue found 1. Error with attempting https for "zoltan": <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'www.cs.sandia.gov'. (_ssl.c:1125)> ``` This is either not fixable, or could be fixed with a change to the url or (better) contacting the site owners to ask about some certificate or similar. The second case is when there is an http that needs to be https, which is a huge issue now, but hopefully not after this spack PR. ```bash $ spack audit packages-https xman Package "xman" uses http but has a valid https endpoint. ``` And then when a package is fixed: ```bash $ spack audit packages-https zlib PKG-HTTPS-DIRECTIVES: 0 issues found. ``` And that's mostly it. :) Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com>
2021-09-01Update versions for RAJA, CHAI, Umpire and camp (#25528)David Beckingsale2-4/+4
2021-08-30Pipelines: disable power builds (#25704)Scott Wittenburg1-18/+18
2021-08-27Fix fish test "framework" (#25242)Harmen Stoppels1-34/+16
Remove broken test, see #21699
2021-08-24fixing small bug that a line of spack monitor commands are still produced ↵Vanessasaurus1-1/+4
(#25366) Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com>
2021-08-19Use kcov from official Ubuntu 20.04 repository (#25385)Massimiliano Culpo3-5/+8
* Ubuntu 20.04 provides kcov, so don't build from source * Use two undocumented options for kcov v3.8
2021-08-19buildcache: Add environment-aware buildcache sync command (#25470)Scott Wittenburg1-1/+5
2021-08-18Bootstrap clingo from binaries (#22720)Massimiliano Culpo2-2/+281
* Bootstrap clingo from binaries * Move information on clingo binaries to a JSON file * Add support to bootstrap on Cray Bootstrapping on Cray requires, at the moment, to swap the platform when looking for binaries - due to #22800. * Add SHA256 verification for bootstrapped software Use sha256 verification for binaries necessary to bootstrap the concretizer and gpg for signature verification * patchelf: use Spec._old_concretize() to bootstrap As noted in #24450 we may happen to need the concretizer when bootstrapping clingo. In that case only the old concretizer is available. * Add a schema for bootstrapping methods Two fields have been added to bootstrap.yaml: "sources" which lists the methods available for bootstrapping software "trusted" which records if a source is trusted or not A subcommand has been added to "spack bootstrap" to list the sources currently available. * Methods used for bootstrapping are configurable from bootstrap:sources The function that tries to ensure a given Python module is importable now tries bootstrapping methods in the same order as they are defined in `bootstrap.yaml` * Permit to trust/untrust bootstrapping methods * Add binary tests for MacOS, Ubuntu * Add documentation * Add a note on bash
2021-08-16e4s ci: further expand power stack (#25405)eugeneswalker1-23/+22
2021-08-16Second pass at increasing RADIUSS cloud CI packages (#25321)Tamara Dahlgren1-7/+7
2021-08-09ci pipelines: expand the list of RADIUSS packages (#25282)Tamara Dahlgren1-10/+10
2021-08-06Add New Build Containers Workflow (#24257)Alec Scott1-1/+1
This pull request adds a new workflow to build and deploy Spack Docker containers from GitHub Actions. In comparison with our current system where we use Dockerhub's CI to build our Docker containers, this workflow will allow us to now build for multiple architectures and deploy to multiple registries. (At the moment x86_64 and Arm64 because ppc64le is throwing an error within archspec.) As currently set up, the PR will build all of the current containers (minus Centos6 because those yum repositories are no longer available?) as both x86_64 and Arm64 variants. The workflow is currently setup to build and deploy containers nightly from develop as well as on tagged releases. The workflow will also build, but NOT deploy containers on a pull request for the purposes of testing this PR. At the moment it is setup to deploy the built containers to GitHub's Container Registry although, support for also uploading to Dockerhub/Quay can be included easily if we decide to keep releasing on Dockerhub/want to begin releasing on Quay.
2021-08-03e4s ci stack: update package preferences (#25163)eugeneswalker1-4/+17
2021-08-02ci: Add RADIUSS stack to cloud CI (#23922)Tamara Dahlgren2-2/+131
Add RADIUSS software stack to gitlab PR testing pipelines
2021-07-30CI: capture stdout/stderr output to artifact files (#24401)Scott Wittenburg1-1/+2
Gitlab truncates job trace output (even the complete raw output) at 4MB, so this change captures it to a file under "user_data" artifacts as well, to make sure we can debug output from the end of the rebuild job.
2021-07-30adding spack diff command (#22283)Vanessasaurus1-1/+10
A `spack diff` will take two specs, and then use the spack.solver.asp.SpackSolverSetup to generate lists of facts about each (e.g., nodes, variants, etc.) and then take a set difference between the two to show the user the differences. Example output: $ spack diff python@2.7.8 python@3.8.11 ==> Warning: This interface is subject to change. --- python@2.7.8/tsxdi6gl4lihp25qrm4d6nys3nypufbf +++ python@3.8.11/yjtseru4nbpllbaxb46q7wfkyxbuvzxx @@ variant_value @@ - python patches a8c52415a8b03c0e5f28b5d52ae498f7a7e602007db2b9554df28cd5685839b8 + python patches 0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87 @@ version @@ - openssl Version(1.0.2u) + openssl Version(1.1.1k) - python Version(2.7.8) + python Version(3.8.11) Currently this uses diff-like output but we will attempt to improve on this in the future. One use case for `spack diff` is whenever a user has a disambiguate situation and cannot remember how two different installs are different. The command can also output `--json` in the case of a more analysis type use case where we want to save complete data with all diffs and the intersection. However, the command is really more intended for a command line use case, and we likely will have an analyzer more suited to saving data Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com> Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2021-07-29Move build tests from GA to Gitlab (#25120)Massimiliano Culpo2-1/+92
Modifications: - Remove the "build tests" workflow from GitHub Actions - Setup a similar e2e test on Gitlab In this way we'll reduce load on GitHub Actions workflows and for e2e tests will benefit from the buildcache reuse granted by pipelines.
2021-07-27bugfix: be careful about GITHUB_BASE_REF in `spack style`Todd Gamblin1-1/+6
`spack style` previously used a Travis CI variable to figure out what the base branch of a PR was, and this was apparently also set on `develop`. We switched to `GITHUB_BASE_REF` to support GitHub Actions, but it looks like this is set to `""` in pushes to develop, so `spack style` breaks there. This PR does two things: - [x] Remove `GITHUB_BASE_REF` knowledge from `spack style` entirely - [x] Handle `GITHUB_BASE_REF` in style scripts instead, and explicitly pass the base ref if it is present, but don't otherwise. This makes `spack style` *not* dependent on the environment and fixes handling of the base branch in the right place.
2021-07-27`spack style`: add `--root` option (#25085)Todd Gamblin1-2/+2
This adds a `--root` option so that `spack style` can check style for a spack instance other than its own. We also change the inner workings of `spack style` so that `--config FILE` (and similar options for the various tools) options are used. This ensures that when `spack style` runs, it always uses the config from the running spack, and does *not* pick up configuration from the external root. - [x] add `--root` option to `spack style` - [x] add `--config` (or similar) option when invoking style tools - [x] add a test that verifies we can check an external instance
2021-07-24`spack style`: automatically bootstrap dependenciesvsoch1-1/+0
This uses our bootstrapping logic to automatically install dependencies for `spack style`. Users should no longer have to pre-install all of the tools (`isort`, `mypy`, `black`, `flake8`). The command will do it for them. - [x] add logic to bootstrap specs with specific version requirements in `spack style` - [x] remove style tools from CI requirements (to ensure we test bootstrapping) - [x] rework dependencies for `mypy` and `py-typed-ast` - `py-typed-ast` needs to be a link dependency - it needs to be at 1.4.1 or higher to work with python 3.9 Signed-off-by: vsoch <vsoch@users.noreply.github.com>
2021-07-24Bump codecov/action to v2.0.2 (#24983)Massimiliano Culpo2-0/+13
* build(deps): bump codecov/codecov-action from 1 to 2.0.1 Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 1 to 2.0.1. - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md) - [Commits](https://github.com/codecov/codecov-action/compare/v1...v2.0.1) --- updated-dependencies: - dependency-name: codecov/codecov-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Update arguments to codecov action * Try to delete the symbolic link to root folder Hopefully this should get rid of the ELOOP error * Delete also for shell tests and MacOS * Bump to v2.0.2 Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-07-16trilinos: simplify some variants (#24820)Seth R. Johnson2-2/+2
* trilinos: rename basker variant The Basker solver is part of amesos2 but is clearer without the extra scoping. * trilinos: automatically enable teuchos and remove variant Basically everything in trilinos needs teuchos * trilinos: group top-level dependencies * trilinos: update dependencies, removing unused - GLM, X11 are unused (x11 lacks dependency specs too) - Python variant is more like a TPL so rearrange that - Gtest internal package shouldn't be compiled or exported - Add MPI4PY requirement for pytrilinos * trilinos: remove package meta-options - XSDK settings and "all opt packages" are not used anywhere - all optional packages are dangerous * trilinos: Use hwloc iff kokkos See #19119, also the HWLOC tpl name was misspelled so this was being ignored before. * Flake * Fix trilinos +netcdf~mpi * trilinos: default to disabling external dependencies * Remove teuchos from downstream dependencies * fixup! trilinos: Use hwloc iff kokkos * Add netcdf requirements to packages with ^trilinos+exodus * trilinos: disable exodus by default * fixup! Add netcdf requirements to packages with ^trilinos+exodus * trilinos: only enable hwloc when @13: +kokkos * xyce: propagate trilinos dependencies more simply * dtk: fix missing boost dependency * trilinos: remove explicit metis dependency * trilinos: require metis/parmetis for zoltan Disable zoltan by default to minimize default dependencies * trilinos: mark mesquite disabled and fix kokkos arch * xsdk: fix trilinos to also list zoltan [with zoltan2] * ci: remove nonexistent variant from trilinos * trilinos: add missing boost dependency Co-authored-by: Satish Balay <balay@mcs.anl.gov>
2021-07-15ci: build trilinos with all variants on (#24908)eugeneswalker2-3/+2
2021-07-12Enable/disable bootstrapping and customize store location (#23677)Massimiliano Culpo1-1/+31
* Permit to enable/disable bootstrapping and customize store location This PR adds configuration handles to allow enabling and disabling bootstrapping, and to customize the store location. * Move bootstrap related configuration into its own YAML file * Add a bootstrap command to manage configuration
2021-07-11Expat: add version 2.4.0, 2.4.1; fix CVE-2013-0340 (#24669)Christoph Conrads2-2/+2
* Expat: add version 2.4.0, 2.4.1; fix CVE-2013-0340 fixes #24628 * E4S pipeline: update pinned Expat version
2021-07-10docker: remove boto3 from CentOS 6 since it requires and updated pip (#24813)Massimiliano Culpo1-1/+0
2021-07-09docker: Fix CentOS 6 build on Docker Hub (#24804)Massimiliano Culpo1-0/+3
This change make yum usable again on CentOS 6
2021-07-08imports: sort imports everywhere in Spack (#24695)Todd Gamblin1-1/+2
* fix remaining flake8 errors * imports: sort imports everywhere in Spack We enabled import order checking in #23947, but fixing things manually drives people crazy. This used `spack style --fix --all` from #24071 to automatically sort everything in Spack so PR submitters won't have to deal with it. This should go in after #24071, as it assumes we're using `isort`, not `flake8-import-order` to order things. `isort` seems to be more flexible and allows `llnl` mports to be in their own group before `spack` ones, so this seems like a good switch.
2021-07-08setup-env: allow users to skip slow parts (#24545)Adam J. Stewart2-87/+91
2021-07-07style: clean up and restructure `spack style` commandTodd Gamblin2-3/+3
This consolidates code across tools in `spack style` so that each `run_<tool>` function can be called indirecty through a dictionary of handlers, and os that checks like finding the executable for the tool can be shared across commands. - [x] rework `spack style` to use decorators to register tools - [x] define tool order in one place in `spack style` - [x] fix python 2/3 issues to Get `isort` checks working - [x] make isort error regex more robust across versions - [x] remove unused output option - [x] change vestigial `TRAVIS_BRANCH` to `GITHUB_BASE_REF` - [x] update completion
2021-07-07add e4s-on-power stack (#24734)eugeneswalker2-0/+406
2021-06-29pipelines: build warpx on instance with more memory (#24592)Scott Wittenburg1-0/+1
2021-06-24Pipelines: Set a pipeline type variable (#24505)Scott Wittenburg1-2/+2
Spack pipelines need to take specific actions internally that depend on whether the pipeline is being run on a PR to spack or a merge to the develop branch. Pipelines can also run in other repositories, which represents other possible use cases than just the two mentioned above. This PR creates a "SPACK_PIPELINE_TYPE" gitlab variable which is propagated to rebuild jobs, and is also used internally to determine which pipeline-specific tasks to run. One goal of the PR is fix an issue where rebuild jobs which failed on develop pipelines did not properly report the broken full hash to the "broken-specs-url".
2021-06-22Axom: Remove blueos check on cuda variant (#24349)Chris White1-2/+1
* remove blueos check on cuda variant, fix typo * restore necessary compiler guard * remove axom+cuda from testing because it only partially works outside ppc systems
2021-06-18Removed unofficial MAGMA release and enabled MAGMA in e4s (#24400)G-Ragghianti1-1/+1
2021-06-18Add an audit system to Spack (#23053)Massimiliano Culpo1-1/+27
Add a new "spack audit" command. This command can check for issues with configuration or with packages and is intended to help a user debug a failed Spack build. In some cases the reported issues are always errors but are too costly to check for (e.g. packages that specify missing variants on dependencies). In other cases the issues may be legitimate but uncommon usage of Spack and we want to be sure the user intended the behavior (e.g. duplicate compiler definitions). Audits are grouped by theme, and for now the two themes are packages and configuration. For example you can run all available audits on packages with "spack audit packages". It is intended that in the future users will be able to define their own audits. The package audits are good candidates for running in package_sanity (i.e. they could catch bugs in user-submitted packages before they are merged) but that is left for a later PR.
2021-06-18Disable magma in the E4S pipeline (#24395)Massimiliano Culpo1-1/+1
Building magma has been failing consistently and is currently blocking PRs from being merged. Disable that spec while we investigate the failure and work on a fix.
2021-06-17Adding support for spack monitor with containerize (#23777)Vanessasaurus3-3/+3
This should get us most of the way there to support using monitor during a spack container build, for both Singularity and Docker. Some quick notes: ### Docker Docker works by way of BUILDKIT and being able to specify --secret. What this means is that you can prefix a line with a mount of type secret as follows: ```bash # Install the software, remove unnecessary deps RUN --mount=type=secret,id=su --mount=type=secret,id=st cd /opt/spack-environment && spack env activate . && export SPACKMON_USER=$(cat /run/secrets/su) && export SPACKMON_TOKEN=$(cat /run/secrets/st) && spack install --monitor --fail-fast && spack gc -y ``` Where the id for one or more secrets corresponds to the file mounted at `/run/secrets/<name>`. So, for example, to build this container with su (spackmon user) and sv (spackmon token) defined I would export them on my host and do: ```bash $ DOCKER_BUILDKIT=1 docker build --network="host" --secret id=st,env=SPACKMON_TOKEN --secret id=su,env=SPACKMON_USER -t spack/container . ``` And when we add `env` to the secret definition that tells the build to look for the secret with id "st" in the environment variable `SPACKMON_TOKEN` for example. If the user is building locally with a local spack monitor, we also need to set the `--network` to be the host, otherwise you can't connect to it (a la isolation of course.) ## Singularity Singularity doesn't have as nice an ability to clearly specify secrets, so (hoping this eventually gets implemented) what I'm doing now is providing the user instructions to write the credentials to a file, add it to the container to source, and remove when done. ## Tags Note that the tags PR https://github.com/spack/spack/pull/23712 will need to be merged before `--monitor-tags` will actually work because I'm checking for the attribute (that doesn't exist yet): ```bash "tags": getattr(args, "monitor_tags", None) ``` So when that PR is merged to update the argument group, it will work here, and I can either update the PR here to not check if the attribute is there (it will be) or open another one in the case this PR is already merged. Finally, I added a bunch of documetation for how to use monitor with containerize. I say "mostly working" because I can't do a full test run with this new version until the container base is built with the updated spack (the request to the monitor server for an env install was missing so I had to add it here). Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com>
2021-06-17e4s ci env: package preferences: use newer versions (#24371)eugeneswalker1-14/+14
2021-06-15e4s ci: specs: add datatransferkit (#24325)eugeneswalker1-0/+1
2021-06-15adding spack upload command (#24321)Vanessasaurus1-1/+5
this will first support uploads for spack monitor, and eventually could be used for other kinds of spack uploads Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com>
2021-06-14e4s ci environment: packages: update to newer versions (#24308)eugeneswalker1-6/+6
2021-06-11setup-env: allow users to skip module function setup (#24236)Adam J. Stewart2-2/+2
* setup-env: allow users to skip module function setup * Add documentation on SPACK_SKIP_MODULES
2021-06-09Update of Flecsi Spackage (#24106)Robert Pavel1-1/+1
* Update of Flecsi Spackage Update of flecsi spackage to reconcile differences between flecsi@1:1.9 and flecsi@2: for future support purposes * Removing Unnecessary Conditional Removing unused conditional. Initially the plan was to switch based on version in `cmake_args` but this was not necessary as build system variable names remained mostly the same and conflicts prevent the rest. For the most part, if a variant is there it does not need to check against what version of the code is being built. * Updated CI To Reconcile Flecsi Changes Updated CI to target flecsi@1.4.2 which best matches the previous release version and reconciled change in variant name
2021-06-08e4s ci: re-enable veloc builds after recent fixes (#24190)eugeneswalker1-1/+1
2021-05-30CI: E4S: enable full E4S (#24011)eugeneswalker1-78/+295
* e4s ci: enable full e4s * add llvm-amdgpu to list of specs needing an xlarge tagged runner * comment out qt and qwt because of intermittent build failures * remove +rocm specs because rocblas job consistently fails due to infrastructure