summaryrefslogtreecommitdiff
path: root/share
AgeCommit message (Collapse)AuthorFilesLines
2020-02-03buildcache list: restore original behavior of allowing constraints like ↵Patrick Gartung1-1/+1
@version. (#14732)
2020-01-30`spack containerize` generates containers from envs (#14202)Massimiliano Culpo3-1/+146
This PR adds a new command to Spack: ```console $ spack containerize -h usage: spack containerize [-h] [--config CONFIG] creates recipes to build images for different container runtimes optional arguments: -h, --help show this help message and exit --config CONFIG configuration for the container recipe that will be generated ``` which takes an environment with an additional `container` section: ```yaml spack: specs: - gromacs build_type=Release - mpich - fftw precision=float packages: all: target: [broadwell] container: # Select the format of the recipe e.g. docker, # singularity or anything else that is currently supported format: docker # Select from a valid list of images base: image: "ubuntu:18.04" spack: prerelease # Additional system packages that are needed at runtime os_packages: - libgomp1 ``` and turns it into a `Dockerfile` or a Singularity definition file, for instance: ```Dockerfile # Build stage with Spack pre-installed and ready to be used FROM spack/ubuntu-bionic:prerelease as builder # What we want to install and how we want to install it # is specified in a manifest file (spack.yaml) RUN mkdir /opt/spack-environment \ && (echo "spack:" \ && echo " specs:" \ && echo " - gromacs build_type=Release" \ && echo " - mpich" \ && echo " - fftw precision=float" \ && echo " packages:" \ && echo " all:" \ && echo " target:" \ && echo " - broadwell" \ && echo " config:" \ && echo " install_tree: /opt/software" \ && echo " concretization: together" \ && echo " view: /opt/view") > /opt/spack-environment/spack.yaml # Install the software, remove unecessary deps and strip executables RUN cd /opt/spack-environment && spack install && spack autoremove -y RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \ xargs file -i | \ grep 'charset=binary' | \ grep 'x-executable\|x-archive\|x-sharedlib' | \ awk -F: '{print $1}' | xargs strip -s # Modifications to the environment that are necessary to run RUN cd /opt/spack-environment && \ spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh # Bare OS image to run the installed executables FROM ubuntu:18.04 COPY --from=builder /opt/spack-environment /opt/spack-environment COPY --from=builder /opt/software /opt/software COPY --from=builder /opt/view /opt/view COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh RUN apt-get -yqq update && apt-get -yqq upgrade \ && apt-get -yqq install libgomp1 \ && rm -rf /var/lib/apt/lists/* ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"] ```
2020-01-30Limit the number of spec files downloaded to find matches for buildcaches ↵Patrick Gartung1-1/+1
(#14659) * Limit the number of spec flies downloaded to find matches
2020-01-24Add spack config list command for tab completion (#14474)Adam J. Stewart2-3/+7
* Add spack config list command for tab completion * Update tab completion scripts
2020-01-23commands: add simple `spack commands --update-completion` argument (#14607)Todd Gamblin2-24/+1
Instead of another script, this adds a simple argument to `spack commands` that updates the completion script. Developers can now just run: spack commands --update-completion This should make it simpler for developers to remember to run this *before* the tests fail. Also, this version tab-completes.
2020-01-22shell support: `spack load` no longer needs modules (#14062)Greg Becker4-76/+62
Previously the `spack load` command was a wrapper around `module load`. This required some bootstrapping of modules to make `spack load` work properly. With this PR, the `spack` shell function handles the environment modifications necessary to add packages to your user environment. This removes the dependence on environment modules or lmod and removes the requirement to bootstrap spack (beyond using the setup-env scripts). Included in this PR is support for MacOS when using Apple's System Integrity Protection (SIP), which is enabled by default in modern MacOS versions. SIP clears the `LD_LIBRARY_PATH` and `DYLD_LIBRARY_PATH` variables on process startup for executables that live in `/usr` (but not '/usr/local', `/System`, `/bin`, and `/sbin` among other system locations. Spack cannot know the `LD_LIBRARY_PATH` of the calling process when executed using `/bin/sh` and `/usr/bin/python`. The `spack` shell function now manually forwards these two variables, if they are present, as `SPACK_<VAR>` and recovers those values on startup. - [x] spack load/unload no longer delegate to modules - [x] refactor user_environment modification calculations - [x] update documentation for spack load/unload Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2020-01-22Use `spack commands --format=bash` to generate shell completion (#14393)Adam J. Stewart7-709/+1403
This PR adds a `--format=bash` option to `spack commands` to auto-generate the Bash programmable tab completion script. It can be extended to work for other shells. Progress: - [x] Fix bug in superclass initialization in `ArgparseWriter` - [x] Refactor `ArgparseWriter` (see below) - [x] Ensure that output of old `--format` options remains the same - [x] Add `ArgparseCompletionWriter` and `BashCompletionWriter` - [x] Add `--aliases` option to add command aliases - [x] Standardize positional argument names - [x] Tests for `spack commands --format=bash` coverage - [x] Tests to make sure `spack-completion.bash` stays up-to-date - [x] Tests for `spack-completion.bash` coverage - [x] Speed up `spack-completion.bash` by caching subroutine calls This PR also necessitates a significant refactoring of `ArgparseWriter`. Previously, `ArgparseWriter` was mostly a single `_write` method which handled everything from extracting the information we care about from the parser to formatting the output. Now, `_write` only handles recursion, while the information extraction is split into a separate `parse` method, and the formatting is handled by `format`. This allows subclasses to completely redefine how the format will appear without overriding all of `_write`. Co-Authored-by: Todd Gamblin <tgamblin@llnl.gov>
2020-01-14stop word splitting from leaking out of setup-env (#14472)Tom Scogland1-1/+1
The pathadd function was using setopt to configure zsh for word splitting, which leaks out of the function and breaks default functionality in a number of external zsh plugins and packages. This switches to emulate -L, just as the spack function uses, to keep the setting local to the function.
2020-01-10Reformat Bash tab completion script (#14456)Adam J. Stewart1-306/+224
2020-01-06Fix outdated bash tab completion (#14392)Adam J. Stewart1-201/+321
2020-01-01tests: improved `spack test` command line optionsTodd Gamblin1-1/+2
Previously, `spack test` automatically passed all of its arguments to `pytest -k` if no options were provided, and to `pytest` if they were. `spack test -l` also provided a list of test filenames, but they didn't really let you completely narrow down which tests you wanted to run. Instead of trying to do our own weird thing, this passes `spack test` args directly to `pytest`, and omits the implicit `-k`. This means we can now run, e.g.: ```console $ spack test spec_syntax.py::TestSpecSyntax::test_ambiguous ``` This wasn't possible before, because we'd pass the fully qualified name to `pytest -k` and get an error. Because `pytest` doesn't have the greatest ability to list tests, I've tweaked the `-l`/`--list`, `-L`/`--list-long`, and `-N`/`--list-names` options to `spack test` so that they help you understand the names better. you can combine these options with `-k` or other arguments to do pretty powerful searches. This one makes it easy to get a list of names so you can run tests in different orders (something I find useful for debugging `pytest` issues): ```console $ spack test --list-names -k "spec and concretize" cmd/env.py::test_concretize_user_specs_together concretize.py::TestConcretize::test_conflicts_in_spec concretize.py::TestConcretize::test_find_spec_children concretize.py::TestConcretize::test_find_spec_none concretize.py::TestConcretize::test_find_spec_parents concretize.py::TestConcretize::test_find_spec_self concretize.py::TestConcretize::test_find_spec_sibling concretize.py::TestConcretize::test_no_matching_compiler_specs concretize.py::TestConcretize::test_simultaneous_concretization_of_specs spec_dag.py::TestSpecDag::test_concretize_deptypes spec_dag.py::TestSpecDag::test_copy_concretized ``` You can combine any list option with keywords: ```console $ spack test --list -k microarchitecture llnl/util/cpu.py modules/lmod.py ``` ```console $ spack test --list-long -k microarchitecture llnl/util/cpu.py:: test_generic_microarchitecture modules/lmod.py::TestLmod:: test_only_generic_microarchitectures_in_root ``` Or just list specific files: ```console $ spack test --list-long cmd/test.py cmd/test.py:: test_list test_list_names_with_pytest_arg test_list_long test_list_with_keywords test_list_long_with_pytest_arg test_list_with_pytest_arg test_list_names ``` Hopefully this stuff will help with debugging test issues. - [x] make `spack test` send args directly to `pytest` instead of trying to do fancy things. - [x] rework `--list`, `--list-long`, and add `--list-names` to make searching for tests easier. - [x] make it possible to mix Spack's list args with `pytest` args (they're just fancy parsing around `pytest --collect-only`) - [x] add docs - [x] add tests - [x] update spack completion
2019-12-30copyright: update copyright dates for 2020 (#14328)Todd Gamblin18-18/+18
2019-12-27setup-env-test: fix pipe redirect (#14306)Johannes Blaschke1-1/+1
2019-12-25Migrate build tests from Travis to Github Actions (#13967)Massimiliano Culpo1-3/+16
This PR moves build smoke tests from TravisCI and migrates them to Github Actions. The result is that build tests are performed in parallel with unit tests and they don't hog additional resources on Travis. The workflow will not run if a PR only changes packages in the built-in repository, but will always run on pushes to develop or master. * Removed build tests from Travis and passed them to Github Actions * Store ~/.ccache in Github Actions cache * Add filters on paths and make sure this workflow don't run * Use paths-ignore and exclude only files in the built-in repo * Added a badge to README.md
2019-12-16Harden shell detection when procfs is available (#13950)Massimiliano Culpo1-3/+6
2019-12-16Travis exits at the first failing test, pin codecov at v4.5.4 (#14179)Massimiliano Culpo1-1/+1
Before this commit we used to run the entire unit test suite in the presence of a failure. Since we currently rely a lot on the state of the filesystem etc. the end report was most of the time showing spurious failures that were a consequence of the first failing test. This PR makes unit tests exit at the first failing test Also, pin codecov at v4.5.4 (last one supporting Python 2.6)
2019-12-13docker: add `file`, adjust locale, and use python3 for ubuntu (#13508)Scott Wittenburg2-16/+56
* docker: add missing module to ubuntu images * docker: fix issue with missing locale * docker: one package per line + rm python2 support * docker: ubuntu image also needs 'file' for buildcache creation
2019-12-13Split out CDash options to a separate help document (#13704)Zack Galbreath1-1/+1
Prevent `spack help install` from getting too cluttered with CDash-specific documentation.
2019-11-26Package Index: Build in Dockerhub (#13810)Axel Huebl8-75/+75
* Package Index: Build in Dockerhub Prepare to build the package index service, packages.spack.io, on Dockerhub. Local build (in spack root dir): ``` docker build -t spack/packages.spack.io:latest -f share/spack/docker/package-index/Dockerfile . ``` Local test: ``` docker run -p 8080:80 spack/packages.spack.io:latest ``` * Travis-CI: Remove Docker Remove leftover docker stages from Travis-CI. * Simplify Split Call
2019-11-22cmd/install: remove unused install_status option (#13751)Greg Becker1-6/+7
* cmd/install: remove unused install_status option * update bash completions for spack install
2019-10-25completion: add bash completion for `spack spec --json` (#13433)Todd Gamblin1-1/+2
2019-10-21Bootstrap environment modules optimizing for generic architectures (#13105)Massimiliano Culpo1-0/+37
fixes #13073 Since #3206 was merged bootstrapping environment-modules was using the architecture of the current host or the best match supported by the default compiler. The former case is an issue since shell integration was looking for a spec targeted at the host microarchitecture. 1. Bootstrap an env modules targeted at generic architectures 2. Look for generic targets in shell integration scripts 3. Add a new entry in Travis to test shell integration
2019-10-15lmod: module files are written in a root folder named by target family (#13121)Massimiliano Culpo1-0/+1
fixes #13005 This commit fixes an issue with the name of the root directory for module file hierarchies. Since #3206 the root folder was named after the microarchitecture used for the spec, which is too specific and not backward compatible for lmod hierarchies. Here we compute the root folder name using the target family instead of the target name itself and we add target information in the 'whatis' portion of the module file.
2019-10-02Remove support for generating dotkit files (#11986)Massimiliano Culpo6-166/+25
Dotkit is being used only at a few sites and has been deprecated on new machines. This commit removes all the code that provide support for the generation of dotkit module files. A new validator named "deprecatedProperties" has been added to the jsonschema validators. It permits to prompt a warning message or exit with an error if a property that has been marked as deprecated is encountered. * Removed references to dotkit in the docs * Removed references to dotkit in setup-env-test.sh * Added a unit test for the 'deprecatedProperties' schema validator
2019-10-01Add all compatible system types directory to module pathsMassimiliano Culpo2-2/+7
fixes #12915 closes #12916 Since Spack has support for specific targets it might happen that software is built for targets that are not exactly the host because it was either an explicit user request or the compiler being used is too old to support the host. Modules for different targets are written into different directories and by default Spack was adding to MODULEPATH only the directory corresponding to the current host. This PR modifies this behavior to add all the directories that are **compatible** with the current host.
2019-09-20Add --known-targets to bash completion for arch command (#12887)Glenn Johnson1-1/+1
This PR adds the new --known-targets flag to the `spack arch` command.
2019-09-13Refactor release-jobs cmd based on use of environments (no docker either)Scott Wittenburg1-11/+0
2019-09-12BUG: recursively load modules with tcsh (#12664)Tyler Reddy1-1/+6
* for tcsh and csh, spack load -r package should now correctly load recursively instead of only loading the target package without any dependencies
2019-09-09packages service: fix docker build again (#12774)Axel Huebl1-1/+1
CD is hard.
2019-09-09packages service: fix docker build (#12773)Axel Huebl1-1/+1
The build instructions I cloned from did not work ;)
2019-09-09packages build: fix path to spack (#12772)Axel Huebl1-1/+4
Fix docker build for packages.spack.io
2019-09-09Fix CD: Packages Service First (#12764)Axel Huebl1-81/+2
* Fix CD: Packages Service First Build the packages.spack.io service images first, so they are guaranteed to be pushed even if further images fail to build. Fix the query to the `spack` script executed in later builds. * CD: Remove Spack Images Now done on Dockerhub.
2019-09-08Add CORS for spack.packages.ioAxel Huebl2-0/+4
Add the HTTP header `Access-Control-Allow-Origin: *` for our NGINX service that is serving static JSON content on https://spack.packages.io .
2019-09-02tests and completions for `spack find --json` and `spack find --format`Todd Gamblin1-2/+3
2019-07-24new command: `spack maintainers` queries package maintainersTodd Gamblin1-0/+10
- We don't currently make enough use of the maintainers field on packages, though we could use it to assign reviews. - add a command that allows maintainers to be queried - can ask who is maintaining a package or packages - can ask what packages users are maintaining - can list all maintained or unmaintained packages - add tests for the command
2019-07-20docker: fix builds, remove extra distros, add ci builds (#11621)Omar Padron26-643/+383
* fix docker builds/remove extra builds/add ci builds * preinstall vim in CI builder images * simplify & streamline docker resources * restore os-container-mapping.yaml file
2019-07-20environments: add activate/deactivate tests, work wtih `set -u`Todd Gamblin1-9/+89
- [x] Add shell tests to ensure that `spack env activate`, `spack env deactivate`, and `despacktivate` continue to work. - [x] Also ensure that activate and deactivate both work with `set -u`
2019-07-17feature: Allow developers to use Spack for partial builds (#12006)Greg Becker1-1/+1
Added new diy option.
2019-07-15bugfix: fix module() function in setup-env.sh (#12030)Todd Gamblin1-6/+9
Modify `setup-env.sh` to export fewer variables but to still set the module() function correctly.
2019-07-15Fix typo in module template (#12028)Adam J. Stewart1-1/+1
2019-07-12Copy --{use,no}-cache from install to bootstrap (#11786)Oliver Breitwieser1-1/+2
2019-07-12Add template creation test, --skip-editor option (#11970)Tamara Dahlgren1-1/+1
* Add template creation test * Added --skip-editor option to "spack create": normally "spack create" opens an editor for the user after generating a package file; when the --skip-editor option is used, "spack create" only generates the package file and does not open an editor * Added --skip-editor option to bash completion
2019-07-12bugfix: setup-env.sh detects bash even when bash is run as sh (#11994)Todd Gamblin1-7/+2
- `setup-env.sh` was not properly detecting a bash shell when bash was run as /bin/sh. - Detection routine now always reports bash when bash is run as sh, and no longer parses the path to the executable indicated in `$BASH`.
2019-07-05setup-env.sh: make setup scripts work with `set -u`Todd Gamblin2-4/+11
- Add set -u to the setup-env.sh test script - Refactor lines in setup-env.sh that tested potentially undefined variables to use the `[ -z ${var+x} ]` construct
2019-07-05tests: add tests for setup-env.shTodd Gamblin3-6/+331
- tests use a shell-script harness and test all Spack commands that require special shell support. - tests work in bash, zsh, and dash - run setup-env.sh tests on macos and linux builds. - we run them on macos and linux
2019-07-05setup-env.sh: now compatible with bash, zsh, and dashTodd Gamblin1-65/+114
- replace use of [[ with [ - replace function foo { .. } with foo() { .. } - wrap some long lines - add lsof and /proc/fd magic so that we can find the sourced file even in dash - only do the complicated shell checks in one place; test $_sp_shell elsewhere.
2019-06-29setup-env.sh: send cd output to /dev/null when it affects SpackTodd Gamblin1-9/+12
- We've seen this a few times now where users have set up `cd` to echo the new directory, and it screws up `setup-env.sh` - In the past we've said this is user error. - Here, we just fix it by sending `cd` output to /dev/null where needed. - this works in bash, zsh, and dash
2019-06-10CD: Package Index (#11665)Axel Huebl5-0/+53
Build the docker image for the new package index REST API on each merge/push to develop and master.
2019-06-08tests: restore `spack spec mpileaks` in test output for Python 2 (#11664)Todd Gamblin1-5/+3
- also remove coverage calls of these commands - we JUST want coverage to tell us what is in the test suite.
2019-06-05travis: disable `spack spec` for python 2.7 as well as 2.6Todd Gamblin1-1/+1
- keep this disabled until the `compiler find` fork bomb is fixed.