summaryrefslogtreecommitdiff
path: root/share
AgeCommit message (Collapse)AuthorFilesLines
2021-05-17Revert "Separable module configurations (#22588)" (#23674)Harmen Stoppels1-3/+3
This reverts commit cefbe48c89209dc3df654795644973b1885cdea4.
2021-05-14Separable module configurations (#22588)Greg Becker1-3/+3
Currently, module configurations are inconsistent because modulefiles are generated with the configs for the active environment, but are shared among all environments (and spack outside any environment). This PR fixes that by allowing Spack environments (or other spack config scopes) to define additional sets of modules to generate. Each set of modules can enable either lmod or tcl modules, and contains all of the previously available module configuration. The user defines the name of each module set -- the set configured in Spack by default is named "default", and is the one returned by module manipulation commands in the absence of user intervention. As part of this change, the module roots configuration moved from the `config` section to inside each module configuration. Additionally, it adds a feature that the modulefiles for an environment can be configured to be relative to an environment view rather than the underlying prefix. This will not be enabled by default, as it should only be enabled within an environment and for non-default views constructed with separate projections per-spec. TODO: - [x] code changes to support multiple module sets - [x] code changes to support modules relative to a view - [x] Tests for multiple module configurations - [x] Tests for modules relative to a view - [x] Backwards compatibility for module roots from config section - [x] Backwards compatibility for default module set without the name specified - [x] Tests for backwards compatibility
2021-05-07install cmd: --no-add in an env installs w/out concretize and addScott Wittenburg1-1/+1
In an active concretize environment, support installing one or more cli specs only if they are already present in the environment. The `--no-add` option is the default for root specs, but optional for dependency specs. I.e. if you `spack install <depspec>` in an environment, the dependency-only spec `depspec` will be added as a root of the environment before being installed. In addition, `spack install --no-add <spec>` fails if it does not find an unambiguous match for `spec`.
2021-04-27Dyninst: add elfutils versioning (#19648)Tim Haines1-1/+1
2021-04-26Dyninst: Add dependencies for v11.0.0 (#23121)Tim Haines1-1/+1
Also update the mpileaks unit test to avoid a conflict on CentOS 6 where Dyninst >=11.0.0 no longer builds due to a compiler version conflict.
2021-04-26ci: Remove leftover duplicate gitlab yaml (#23248)Chuck Atkins1-125/+0
2021-04-26ci: Generalize the GitLab CI pipeline yaml (#23225)Chuck Atkins3-34/+224
* ci: Generalize the GitLab CI pipeline yaml * ci: Rename cloud_e4s_pipelines to the more general cloud_pipelines
2021-04-15Merge pull request #21930 from vsoch/add/spack-monitorVanessasaurus1-3/+25
This provides initial support for [spack monitor](https://github.com/spack/spack-monitor), a web application that stores information and analysis about Spack installations. Spack can now contact a monitor server and upload analysis -- even after a build is already done. Specifically, this adds: - [x] monitor options for `spack install` - [x] `spack analyze` command - [x] hook architecture for analyzers - [x] separate build logs (in addition to the existing combined log) - [x] docs for spack analyze - [x] reworked developer docs, with hook docs - [x] analyzers for: - [x] config args - [x] environment variables - [x] installed files - [x] libabigail There is a lot more information in the docs contained in this PR, so consult those for full details on this feature. Additional tests will be added in a future PR.
2021-04-14update tutorial public keyGregory Becker1-35/+26
2021-03-29Add "spack [cd|location] --source-dir" (#22321)Harmen Stoppels1-2/+2
2021-03-19CI: drastically reduce the number of tests for package only PRs (#22410)Massimiliano Culpo1-1/+1
PRs that change only package recipes will only run tests under "package_sanity.py" and without coverage. This should result in a huge drop the cpu-time spent in CI for most PRs.
2021-03-18Tab to spaces (#22362)Harmen Stoppels2-6/+6
2021-03-16Speed-up CI by reorganizing tests (#22247)Massimiliano Culpo2-1/+10
* unit tests: mark slow tests as "maybeslow" This commit also removes the "network" marker and marks every "network" test as "maybeslow". Tests marked as db are maintained, but they're not slow anymore. * GA: require style tests to pass before running unit-tests * GA: make MacOS unit tests fail fast * GA: move all unit tests into the same workflow, run style tests as a prerequisite All the unit tests have been moved into the same workflow so that a single run of the dorny/paths-filter action can be used to ask for coverage based on the files that have been changed in a PR. The basic idea is that for PRs that introduce only changes to packages coverage is not necessary, this resulting in a faster execution of the tests. Also, for package only PRs slow unit tests are skipped. Finally, MacOS and linux unit tests are now conditional on style tests passing meaning that e.g. we won't waste a MacOS worker if we know that the PR has flake8 issues. * Addressed review comments * Skipping slow tests on MacOS for package only recipes * QA: make tests on changes correct before merging
2021-03-15Expand relative dev paths in environment files (#22045)Harmen Stoppels1-1/+1
* Rewrite relative dev_spec paths internally to absolute paths in case of relocation of the environment file * Test relative paths for dev_path in environments * Add a --keep-relative flag to spack env create This ensures that relative paths of develop paths are not expanded to absolute paths when initializing the environment in a different location from the spack.yaml init file.
2021-03-15Propagate --test= for environments (#22040)Harmen Stoppels1-1/+1
* Propagate --test= for environments * Improve help comment for spack concretize --test flag * Add tests for --test with environments
2021-03-13adding spack -c to set one off config arguments (#22251)Vanessasaurus1-1/+1
This pull request will add the ability for a user to add a configuration argument on the fly, on the command line, e.g.,: ```bash $ spack -c config:install_tree:root:/path/to/config.yaml -c packages:all:compiler:[gcc] list --help ``` The above command doesn't do anything (I'm just getting help for list) but you can imagine having another root of packages, and updating it on the fly for a command (something I'd like to do in the near future!) I've moved the logic for config_add that used to be in spack/cmd/config.py into spack/config.py proper, and now both the main.py (where spack commands live) and spack/cmd/config.py use these functions. I only needed spack config add, so I didn't move the others. We can move the others if there are also needed in multiple places.
2021-03-10fix setup-env.sh on older linux zsh (#21721)Danny McClanahan2-2/+2
2021-03-07spack python: add --path option (#22006)Todd Gamblin1-1/+1
This adds a `--path` option to `spack python` that shows the `python` interpreter that Spack is using. e.g.: ```console $ spack python --path /Users/gamblin2/src/spack/var/spack/environments/default/.spack-env/view/bin/python ``` This is useful for debugging, and we can ask users to run it to understand what python Spack is picking up via preferences in `bin/spack` and via the `SPACK_PYTHON` environment variable introduced in #21222.
2021-03-07add `spack test list --all` (#22032)Todd Gamblin1-1/+1
`spack test list` will show you which *installed* packages can be tested but it won't show you which packages have tests. - [x] add `spack test list --all` to show which packages have test methods - [x] update `has_test_method()` to handle package instances *and* package classes.
2021-03-03Bootstrap clingo from sources (#21446)Massimiliano Culpo1-2/+2
* Allow the bootstrapping of clingo from sources Allow python builds with system python as external for MacOS * Ensure consistent configuration when bootstrapping clingo This commit uses context managers to ensure we can bootstrap clingo using a consistent configuration regardless of the use case being managed. * Github actions: test clingo with bootstrapping from sources * Add command to inspect and clean the bootstrap store Prevent users to set the install tree root to the bootstrap store * clingo: documented how to bootstrap from sources Co-authored-by: Gregory Becker <becker33@llnl.gov>
2021-02-27Temporarily reduce pr stack size (#21998)Scott Wittenburg2-73/+73
Gitlab pipelines fixes * add arch tag to avoid picking up UO power9 runners * temporarily reduce PR workload
2021-02-24Config prefer upstream (#21487)Paul Ferrell1-1/+5
This allows for quickly configuring a spack install/env to use upstream packages by default. This is particularly important when upstreaming from a set of officially supported spack installs on a production cluster. By configuring such that package preferences match the upstream, you ensure maximal reuse of existing package installations.
2021-02-23Gitlab fix pr workflow (#21786)Scott Wittenburg2-5/+4
Fixes for gitlab pipelines * Remove accidentally retained testing branch name * Generate pipeline w/out debug mode * Make jobs interruptible for auto-cancel pending * Work around concretization conflicts
2021-02-23Drop compiler variables from spack load (#21699)Harmen Stoppels1-1/+0
Drops: * C_INCLUDE_PATH * CPLUS_INCLUDE_PATH * LIBRARY_PATH * INCLUDE We already decided to use C_INCLUDE_PATH, CPLUS_INCLUDE_PATH, INCLUDE over CPATH here: https://github.com/spack/spack/pull/14749 However, none of these flags apply to Fortran on Linux. So for consistency it seems better to make the user use -I and -L flags by hand or through pkgconfig.
2021-02-18Pipelines: Move PR testing stacks (currently only E4S) into spack (#21714)Scott Wittenburg3-24/+212
2021-02-18Fixed conditional in match_flag for fish env (#21679)Severin Strobl1-1/+1
An attempt to fix the conditional was made in 5a771bc8ad, yet this broke the conditional completely.
2021-02-16Pipelines: Temporary buildcache storage (#21474)Scott Wittenburg1-1/+5
Before this change, in pipeline environments where runners do not have access to persistent shared file-system storage, the only way to pass buildcaches to dependents in later stages was by using the "enable-artifacts-buildcache" flag in the gitlab-ci section of the spack.yaml. This change supports a second mechanism, named "temporary-storage-url-prefix", which can be provided instead of the "enable-artifacts-buildcache" feature, but the two cannot be used at the same time. If this prefix is provided (only "file://" and "s3://" urls are supported), the gitlab "CI_PIPELINE_ID" will be appended to it to create a url for a mirror where pipeline jobs will write buildcache entries for use by jobs in subsequent stages. If this prefix is provided, a cleanup job will be generated to run after all the rebuild jobs have finished that will delete the contents of the temporary mirror. To support this behavior a new mirror sub-command has been added: "spack mirror destroy" which can take either a mirror name or url. This change also fixes a bug in generation of "needs" list for each job. Each jobs "needs" list is supposed to only contain direct dependencies for scheduling purposes, unless "enable-artifacts-buildcache" is specified. Only in that case are the needs lists supposed to contain all transitive dependencies. This changes fixes a bug that caused the needs lists to always contain all transitive dependencies, regardless of whether or not "enable-artifacts-buildcache" was specified.
2021-02-16Add RHEL8 Universal Base Image with platform-python to CI unit tests (#21655)Chuck Atkins1-1/+1
2021-02-16Pipelines: DAG Pruning (#20435)Scott Wittenburg1-2/+6
Pipelines: DAG pruning During the pipeline generation staging process we check each spec against all configured mirrors to determine whether it is up to date on any of the mirrors. By default, and with the --prune-dag argument to "spack ci generate", any spec already up to date on at least one remote mirror is omitted from the generated pipeline. To generate jobs for up to date specs instead of omitting them, use the --no-prune-dag argument. To speed up the pipeline generation process, pass the --check-index-only argument. This will cause spack to check only remote buildcache indices and avoid directly fetching any spec.yaml files from mirrors. The drawback is that if the remote buildcache index is out of date, spec rebuild jobs may be scheduled unnecessarily. This change removes the final-stage-rebuild-index block from gitlab-ci section of spack.yaml. Now rebuilding the buildcache index of the mirror specified in the spack.yaml is the default, unless "rebuild-index: False" is set. Spack assigns the generated rebuild-index job runner attributes from an optional new "service-job-attributes" block, which is also used as the source of runner attributes for another generated non-build job, a no-op job, which spack generates to avoid gitlab errors when DAG pruning results in empty pipelines.
2021-02-12Introduce a SPACK_PYTHON environment variable (#21222)Chuck Atkins3-1/+34
The SPACK_PYTHON environment variable can be set to a python interpreter to be used by the spack command. This allows the spack command itself to use a consistent and separate interpreter from whatever python might be used for package building.
2021-02-09Procedure to deprecate old versions of software (#20767)Adam J. Stewart1-6/+6
* Procedure to deprecate old versions of software * Add documentation * Fix bug in logic * Update tab completion * Deprecate legacy packages * Deprecate old mxnet as well * More explicit docs
2021-02-04spack external find: allow to search by tags (#21407)Massimiliano Culpo1-3/+3
This commit adds an option to the `external find` command that allows it to search by tags. In this way group of executables with common purposes can be grouped under a single name and a simple command can be used to detect all of them. As an example introduce the 'build-tools' tag to search for common development tools on a system
2021-01-27spack setup: remove the command for v0.17.0 (#20277)Adam J. Stewart1-10/+1
spack setup was deprecated in 0.16 and will be removed in 0.17 Follow-up to #18240
2021-01-08Remove ascent gitlab trigger (#20755)Jamie Finney1-9/+0
Remove the ORNL Ascent gitlab trigger CI will now be done internally via periodic builds.
2021-01-05spack python: allow use of IPython (#20329)Vanessasaurus1-1/+1
This adds a -i option to "spack python" which allows use of the IPython interpreter; it can be used with "spack python -i ipython". This assumes it is available in the Python instance used to run Spack (i.e. that you can "import IPython").
2021-01-02copyrights: update all files with license headers for 2021Todd Gamblin23-23/+23
- [x] add `concretize.lp`, `spack.yaml`, etc. to licensed files - [x] update all licensed files to say 2013-2021 using `spack license update-copyright-year` - [x] appease mypy with some additions to package.py that needed for oneapi.py
2021-01-02commands: add `spack license update-copyright-year`Todd Gamblin1-3/+7
This adds a new subcommand to `spack license` that automatically updates the copyright year in files that should have a license header. - [x] add `spack license update-copyright-year` command - [x] add test
2020-12-29PythonPackage: url -> pypi (#20610)Adam J. Stewart1-0/+1
* Convert all `url` attributes in `PythonPackage`s to `pypi` attributes * add `pypi =` to flake8 exceptions
2020-12-22add mypy to style checks; rename `spack flake8` to `spack style` (#20384)Tom Scogland3-4/+17
I lost my mind a bit after getting the completion stuff working and decided to get Mypy working for spack as well. This adds a `.mypy.ini` that checks all of the spack and llnl modules, though not yet packages, and fixes all of the identified missing types and type issues for the spack library. In addition to these changes, this includes: * rename `spack flake8` to `spack style` Aliases flake8 to style, and just runs flake8 as before, but with a warning. The style command runs both `flake8` and `mypy`, in sequence. Added --no-<tool> options to turn off one or the other, they are on by default. Fixed two issues caught by the tools. * stub typing module for python2.x We don't support typing in Spack for python 2.x. To allow 2.x to support `import typing` and `from typing import ...` without a try/except dance to support old versions, this adds a stub module *just* for python 2.x. Doing it this way means we can only reliably use all type hints in python3.7+, and mypi.ini has been updated to reflect that. * add non-default black check to spack style This is a first step to requiring black. It doesn't enforce it by default, but it will check it if requested. Currently enforcing the line length of 79 since that's what flake8 requires, but it's a bit odd for a black formatted project to be quite that narrow. All settings are in the style command since spack has no pyproject.toml and I don't want to add one until more discussion happens. Also re-format `style.py` since it no longer passed the black style check with the new length. * use style check in github action Update the style and docs action to use `spack style`, adding in mypy and black to the action even if it isn't running black right now.
2020-12-22Refactor flake8 handling and tool compatibility (#20376)Tom Scogland2-1/+147
This PR does three related things to try to improve developer tooling quality of life: 1. Adds new options to `.flake8` so it applies the rules of both `.flake8` and `.flake_package` based on paths in the repository. 2. Adds a re-factoring of the `spack flake8` logic into a flake8 plugin so using flake8 directly, or through editor or language server integration, only reports errors that `spack flake8` would. 3. Allows star import of `spack.pkgkit` in packages, since this is now the thing that needs to be imported for completion to work correctly in package files, it's nice to be able to do that. I'm sorely tempted to sed over the whole repository and put `from spack.pkgkit import *` in every package, but at least being allowed to do it on a per-package basis helps. As an example of what the result of this is: ``` ~/Workspace/Projects/spack/spack develop* ⇣ ❯ flake8 --format=pylint ./var/spack/repos/builtin/packages/kripke/package.py ./var/spack/repos/builtin/packages/kripke/package.py:6: [F403] 'from spack.pkgkit import *' used; unable to detect undefined names ./var/spack/repos/builtin/packages/kripke/package.py:25: [E501] line too long (88 > 79 characters) ~/Workspace/Projects/spack/spack refactor-flake8* 1 ❯ flake8 --format=spack ./var/spack/repos/builtin/packages/kripke/package.py ~/Workspace/Projects/spack/spack refactor-flake8* ❯ flake8 ./var/spack/repos/builtin/packages/kripke/package.py ``` * qa/flake8: update .flake8, spack formatter plugin Adds: * Modern flake8 settings for per-path/glob error ignores, allows packages to use the same `.flake8` as the rest of spack * A spack formatter plugin to flake8 that implements the behavior of `spack flake8` for direct invocations. Makes integration with developer tooling nicer, linting with flake8 reports only errors that `spack flake8` would report. Using pyls and pyls-flake8, or any other non-format-dependent flake8 integration, now works with spack's rules. * qa/flake8: allow star import of spack.pkgkit To get working completion of directives and spack components it's necessary to import the contents of spack.pkgkit. At the moment doing this makes flake8 displeased. For now, allow spack.pkgkit and spack both, next step is to ban spack * and require spack.pkgkit *. * first cut at refactoring spack flake8 This version still copies all of the files to be checked as befire, and some other things that probably aren't necessary, but it relies on the spack formatter plugin to implement the ignore logic. * keep flake8 from rejecting itself * remove separate packages flake8 config * fix failures from too many files I ran into this in the PR converting pkgkit to std. The solution in that branch does not work in all cases as it turns out, and all the workarounds I tried to use generated configs to get a single invocation of flake8 with a filename optoion to work failed. It's an astonishingly frustrating config option. Regardless, this removes all temporary file creation from the command and relies on the plugin instead. To work around the huge number of files in spack and still allow the command to control what gets checked, it scans files in batches of 100. This is a completely arbitrary number but was chosen to be safely under common line-length limits. One side-effect of this is that every 100 files the command will produce output, rather than only at the end, which doesn't seem like a terrible thing.
2020-12-18minimal zsh completion (#20253)Tom Scogland5-35/+74
Since zsh can load bash completion files natively, seems reasonable to just turn this on. The only changes are to switch from `type -t` which zsh doesn't support to using `type` with a regex and adding a new arm to the sourcing of the completions to allow it to work for zsh as well as bash. Could use more bash/dash/etc testing probably, but everything I've thought to try has worked so far. Notes: * unit-test zsh support, fix issues Specifically fixed word splitting in completion-test, use a different method to apply sh emulation to zsh loaded bash completion, and fixed an incompatibility in regex operator quoting requirements. * compinit now ignores insecure directories Completion isn't meant to be enabled in non-interactive environments, so by default compinit will ask the user if they want to ignore insecure directories or load them anyway. To pass the spack unit tests in GH actions, this prompt must be disabled, so ignore explicitly until a better solution can be found. * debug functions test also requires bash emulation COMP_WORDS is a bash-ism that zsh doesn't natively support, turn on emulation for just that section of tests to allow the comparison to work. Does not change the behavior of the functions themselves since they are already pinned to sh emulation elsewhere. * propagate change to .in file * fix comment and update script based on .in
2020-12-11Add amazonlinux (x86/arm) dockerfile (#20320)Christian Kniep1-0/+70
Co-authored-by: Christian Kniep <kniec@amazon.com>
2020-12-07Add "spack versions --new" flag to only show new versions (#20030)vvolkl1-1/+1
* [cmd versions] add spack versions --new flag to only fetch new versions format [cmd versions] rename --latest to --newest and add --remote-only [cmd versions] add tests for --remote-only and --new format [cmd versions] update shell tab completion [cmd versions] remove test for --remote-only --new which gives empty output [cmd versions] final rename format * add brillig mock package * add test for spack versions --new * [brillig] format * [versions] increase test coverage * Update lib/spack/spack/cmd/versions.py Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com> * Update lib/spack/spack/cmd/versions.py Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com> Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2020-12-03allow install of build-deps from cache via --include-build-deps switch (#19955)eugeneswalker1-1/+1
* allow install of build-deps from cache via --include-build-deps switch * make clear that --include-build-deps is useful for CI pipeline troubleshooting
2020-12-01dockerfiles: add dockerfile for opensuse leap 15 (#20091)Christian Goll1-0/+61
* added dockerfile for opensuse leap 15 * updated maintainer info * Update share/spack/docker/leap-15.dockerfile * move copies and symlinks after package install also use ${SPACK_ROOT} for spack calls as this works with buildah Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2020-11-20[WIP] nersc e4s pipeline trigger (#19688)Shahzeb Siddiqui1-0/+9
* nersc e4s pipeline trigger * Update nersc_pipeline.yml * Update nersc_pipeline.yml
2020-11-18cmd: add `spack mark` command (#16662)Michael Kuhn1-1/+10
This adds a new `mark` command that can be used to mark packages as either explicitly or implicitly installed. Apart from fixing the package database after installing a dependency manually, it can be used to implement upgrade workflows as outlined in #13385. The following commands demonstrate how the `mark` and `gc` commands can be used to only keep the current version of a package installed: ```console $ spack install pkgA $ spack install pkgB $ git pull # Imagine new versions for pkgA and/or pkgB are introduced $ spack mark -i -a $ spack install pkgA $ spack install pkgB $ spack gc ``` If there is no new version for a package, `install` will simply mark it as explicitly installed and `gc` will not remove it. Co-authored-by: Greg Becker <becker33@llnl.gov>
2020-11-18spack test (#15702)Greg Becker4-7/+101
Users can add test() methods to their packages to run smoke tests on installations with the new `spack test` command (the old `spack test` is now `spack unit-test`). spack test is environment-aware, so you can `spack install` an environment and then run `spack test run` to run smoke tests on all of its packages. Historical test logs can be perused with `spack test results`. Generic smoke tests for MPI implementations, C, C++, and Fortran compilers as well as specific smoke tests for 18 packages. Inside the test method, individual tests can be run separately (and continue to run best-effort after a test failure) using the `run_test` method. The `run_test` method encapsulates finding test executables, running and checking return codes, checking output, and error handling. This handles the following trickier aspects of testing with direct support in Spack's package API: - [x] Caching source or intermediate build files at build time for use at test time. - [x] Test dependencies, - [x] packages that require a compiler for testing (such as library only packages). See the packaging guide for more details on using Spack testing support. Included is support for package.py files for virtual packages. This does not change the Spack interface, but is a major change in internals. Co-authored-by: Tamara Dahlgren <dahlgren1@llnl.gov> Co-authored-by: wspear <wjspear@gmail.com> Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2020-11-17spack containerize: allow users to customize the base image (#15028)Massimiliano Culpo2-19/+35
This PR reworks a few attributes in the container subsection of spack.yaml to permit the injection of custom base images when generating containers with Spack. In more detail, users can still specify the base operating system and Spack version they want to use: spack: container: images: os: ubuntu:18.04 spack: develop in which case the generated recipe will use one of the Spack images built on Docker Hub for the build stage and the base OS image in the final stage. Alternatively, they can specify explicitly the two base images: spack: container: images: build: spack/ubuntu-bionic:latest final: ubuntu:18.04 and it will be up to them to ensure their consistency. Additional changes: * This commit adds documentation on the two approaches. * Users can now specify OS packages to install (e.g. with apt or yum) prior to the build (previously this was only available for the finalized image). * Handles to avoid an update of the available system packages have been added to the configuration to facilitate the generation of recipes permitting deterministic builds.
2020-11-17concretizer: first working version with pyclingo interfaceTodd Gamblin1-1/+10
- [x] Solver now uses the Python interface to clingo - [x] can extract unsatisfiable cores from problems when things go wrong - [x] use Python callbacks for versions instead of choice rules (this may ultimately hurt performance)