summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2020-02-06Allow installing unsigned binary packages (#11107)Oliver Breitwieser2-4/+10
This commit introduces a `--no-check-signature` option for `spack install` so that unsigned packages can be installed. It is off by default (signatures required).
2020-02-06Adds fma and vsx features to entire power arch family. (#14759)Matt Belhorn1-0/+14
VSX alitvec extensions are supported by PowerISA from v2.06 (Power7+), but might not be listed in features. FMA has been supported by PowerISA since Power1, but might not be listed in features. This commit adds these features to all the power ISA family sets.
2020-02-03Git fetching: add option to remove submodules (#14370)Andrew W Elble4-4/+95
Add an optional 'submodules_delete' field to Git versions in Spack packages that allows them to remove specific submodules. For example: the nervanagpu submodule has become unavailable for the PyTorch project (see issue 19457 at https://github.com/pytorch/pytorch/issues/). Removing this submodule allows 0.4.1 to build.
2020-02-03buildcache list: restore original behavior of allowing constraints like ↵Patrick Gartung3-22/+15
@version. (#14732)
2020-01-31binary_distribution: Initialize _cached_specs at the module level and ↵Patrick Gartung3-13/+7
only search the mirrors in get_spec if spec is not in _cached_specs. (#14714) * Initialize _cached_specs at the file level and check for spec in it before searching mirrors in try_download_spec. * Make _cached_specs a set to avoid duplicates * Fix packaging test * Ignore build_cache in stage when spec.yaml files are downloaded.
2020-01-31`spack -V` is now more descriptive for dev branchesTodd Gamblin2-1/+94
`spack -V` previously always returned the version of spack from `spack.spack_version`. This gives us a general idea of what version users are on, but if they're on `develop` or on some branch, we have to ask more questions. This PR makes `spack -V` check whether this instance of Spack is a git repository, and if it is, it appends useful information from `git describe --tags` to the version. Specifically, it adds: - number of commits since the last release tag - abbreviated (but unique) commit hash So, if you're on `develop` you might get something like this: $ spack -V 0.13.3-912-3519a1762 This means you're on commit 3519a1762, which is 912 commits ahead of the 0.13.3 release. If you are on a release branch, or if you are using a tarball of Spack, you'll get the usual `spack.spack_version`: $ spack -V 0.13.3 This should help when asking users what version they are on, since a lot of people use the `develop` branch.
2020-01-31Document how to use Spack to replace Homebrew/Conda (#13083)Adam J. Stewart1-0/+242
* Document how to use Spack to replace Homebrew/Conda * Initial draft; can iterate more as features become available
2020-01-30`spack containerize` generates containers from envs (#14202)Massimiliano Culpo21-2/+1092
This PR adds a new command to Spack: ```console $ spack containerize -h usage: spack containerize [-h] [--config CONFIG] creates recipes to build images for different container runtimes optional arguments: -h, --help show this help message and exit --config CONFIG configuration for the container recipe that will be generated ``` which takes an environment with an additional `container` section: ```yaml spack: specs: - gromacs build_type=Release - mpich - fftw precision=float packages: all: target: [broadwell] container: # Select the format of the recipe e.g. docker, # singularity or anything else that is currently supported format: docker # Select from a valid list of images base: image: "ubuntu:18.04" spack: prerelease # Additional system packages that are needed at runtime os_packages: - libgomp1 ``` and turns it into a `Dockerfile` or a Singularity definition file, for instance: ```Dockerfile # Build stage with Spack pre-installed and ready to be used FROM spack/ubuntu-bionic:prerelease as builder # What we want to install and how we want to install it # is specified in a manifest file (spack.yaml) RUN mkdir /opt/spack-environment \ && (echo "spack:" \ && echo " specs:" \ && echo " - gromacs build_type=Release" \ && echo " - mpich" \ && echo " - fftw precision=float" \ && echo " packages:" \ && echo " all:" \ && echo " target:" \ && echo " - broadwell" \ && echo " config:" \ && echo " install_tree: /opt/software" \ && echo " concretization: together" \ && echo " view: /opt/view") > /opt/spack-environment/spack.yaml # Install the software, remove unecessary deps and strip executables RUN cd /opt/spack-environment && spack install && spack autoremove -y RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \ xargs file -i | \ grep 'charset=binary' | \ grep 'x-executable\|x-archive\|x-sharedlib' | \ awk -F: '{print $1}' | xargs strip -s # Modifications to the environment that are necessary to run RUN cd /opt/spack-environment && \ spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh # Bare OS image to run the installed executables FROM ubuntu:18.04 COPY --from=builder /opt/spack-environment /opt/spack-environment COPY --from=builder /opt/software /opt/software COPY --from=builder /opt/view /opt/view COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh RUN apt-get -yqq update && apt-get -yqq upgrade \ && apt-get -yqq install libgomp1 \ && rm -rf /var/lib/apt/lists/* ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"] ```
2020-01-30Bypass build_cache/index.html read when trying to download spec.yaml for ↵Patrick Gartung2-21/+68
concretized spec. (#14698) * Add binary_distribution::get_spec which takes concretized spec Add binary_distribution::try_download_specs for downloading of spec.yaml files to cache get_spec is used by package::try_install_from_binary_cache to download only the spec.yaml for the concretized spec if it exists.
2020-01-30Use non-mutable default for names in binary_distribution::get_specs call ↵Patrick Gartung1-1/+3
(#14696) * Use non-mutable default for names * Make suggested change
2020-01-30Document how to add conditional dependencies (#14694)Peter Scheibel1-0/+22
* add short docs section on conditional dependencies * add reference to spec syntax * add note that conditional dependencies can save time
2020-01-30Bugfix: put environment lock in the right place (#14692)Peter Scheibel1-1/+1
Locate the environment lock in the hidden environment directory rather than the root of the environment.
2020-01-30Limit the number of spec files downloaded to find matches for buildcaches ↵Patrick Gartung4-24/+28
(#14659) * Limit the number of spec flies downloaded to find matches
2020-01-29specs: avoid traversing specs when parsingTodd Gamblin1-8/+13
The Spec parser currently calls `spec.traverse()` after every parse, in order to set the platform if it's not set. We don't need to do a full traverse -- we can just check the platforrm as new specs are parsed. This takes about a second off the time required to import all packages in Spack (from 8s to 7s). - [x] simplify platform-setting logic in `SpecParser`.
2020-01-29repo: avoid unnecessary spec parsing in `filename_for_package_name()`Todd Gamblin1-14/+5
`filename_for_package_name()` and `dirname_for_package_name()` automatically construct a Spec from their arguments, which adds a fair amount of overhead to importing lots of packages. Removing this removes about 11% of the runtime of importing all packages in Spack (9s -> 8s). - [x] `filename_for_package_name()` and `dirname_for_package_name()` now take a string `pkg_name` arguments instead of specs.
2020-01-29environments: synchronize read and uninstall (#14676)Peter Scheibel2-18/+20
* `Environment.__init__` is now synchronized with all writing operations * `spack uninstall` now synchronizes its updates to any associated environment * A side effect of this is that the environment is no longer updated piecemeal as specs are uninstalled - all specs are removed from the environment before they are uninstalled
2020-01-28bugfix: correct exception message matching in tests (#14655)Tamara Dahlgren6-17/+26
This commit makes two fundamental corrections to tests: 1) Changes 'matches' to the correct 'match' argument for 'pytest.raises' (for all affected tests except those checking for 'SystemExit'); 2) Replaces the 'match' argument for tests expecting 'SystemExit' (since the exit code is retained instead) with 'capsys' error message capture. Both changes are needed to ensure the associated exception message is actually checked.
2020-01-28Fujitsu compiler: Defining option that is always added. (#14657)t-karatsu1-0/+4
2020-01-28env: synchronize updates to environments (#14621)Peter Scheibel5-93/+153
Updates to environments were not multi-process safe, which prevented them from taking advantage of parallel builds as implemented in #13100. This is a minimal set of changes to enable `spack install` in an environment to be parallelized: - [x] add an internal lock, stored in the `.spack-env` directory, to synchronize updates to `spack.yaml` and `spack.lock` - [x] add `Environment.write_transaction` interface for this lock - [x] makes use of `Environment.write_transaction` in `install`, `add`, and `remove` commands - `uninstall` is not synchronized yet; that is left for a future PR.
2020-01-28Note about Intel compiler segfault with long paths (#14652)Glenn Johnson1-5/+15
This PR adds a note about segfaults with the Intel compiler when the install paths are long and the dependencies many.
2020-01-28Fix handling of filter_file exceptions (#14651)Greg Becker1-1/+0
2020-01-27make the new 'spack load' faster (#14628)Andrew W Elble1-10/+13
before, a 'time spack load singularity' 4.129u 0.346s 0:04.47 99.7% 0+0k 0+8io 0pf+0w after, a 'time spack load singularity' 0.844u 0.319s 0:01.16 99.1% 0+0k 0+16io 0pf+0w
2020-01-27fix spack env loads example (#14558)Owen Solberg1-2/+3
2020-01-27Fix: hash-based references to upstream specs (#14629)Andrew W Elble2-36/+56
Spack commands referring to upstream-installed specs by hash have been broken since 6b619da (merged September 2019), which added a new Database function specifically for parsing hashes from command-line specs; this function was inappropriately attempting to acquire locks on upstream databases. This PR updates the offending function to avoid locking upstream databases and also updates associated tests to catch regression errors: the upstream database created for these tests was not explicitly set as an upstream (i.e. initialized with upstream=True) so it was not guarding against inappropriate accesses.
2020-01-27Package source ID cannot be determined when the url can't be extrapolated ↵Patrick Gartung1-1/+4
for older version. (#14237)
2020-01-27Follow the example of spack arch (#14642)Patrick Gartung1-4/+8
2020-01-27Only set tcl default. Remove lmod default. (#14640)Patrick Gartung1-1/+0
2020-01-27Fix bug introduced by pull request 14467 being merged (#14639)Patrick Gartung1-2/+8
* Fix bug introduced by pull request 14467 being merged * Only filter on platform and OS
2020-01-27Set module_roots in test/config/config.yaml to defaults. (#14517)Patrick Gartung1-0/+3
2020-01-27Unified environment modifications in config files (#14372)Massimiliano Culpo8-183/+169
* Unified environment modifications in config files fixes #13357 This commit factors all the code that is involved in the validation (schema) and parsing of environment modifications from configuration files in a single place. The factored out code is then used for module files and compiler configuration. Attributes were separated by dashes in `compilers.yaml` files and by underscores in `modules.yaml` files. This PR unifies the syntax on attributes separated by underscores. Unit testing of environment modifications in compilers has been refactored and simplified.
2020-01-26Fix Python version compatibility tests for vermin 0.10.0 (#14632)Adam J. Stewart2-4/+4
2020-01-25When spack install checks for buildcaches only add urls for current arch. ↵Patrick Gartung2-4/+10
(#14467)
2020-01-24Add spack config list command for tab completion (#14474)Adam J. Stewart2-5/+22
* Add spack config list command for tab completion * Update tab completion scripts
2020-01-24bugfix: make `_source_single_file` work in venvs (#14569)Massimiliano Culpo2-6/+10
Using `sys.executable` to run Python in a sub-shell doesn't always work in a virtual environment as the `sys.executable` Python is not necessarily compatible with any loaded spack/other virtual environment. - revert use of sys.executable to print out subshell environment (#14496) - try instead to use an available python, then if there *is not* one, use `sys.executable` - this addresses RHEL8 (where there is no `python` and `PYTHONHOME` issue in a simpler way
2020-01-24Use Spack target architecture to determine OpenBLAS target (#14380)Seth R. Johnson1-0/+11
Openblas target is now determined automatically upon inspection of `TargetList.txt`. If the spack target is a generic architecture family (like x86_64 or aarch64) the DYNAMIC_ARCH setting is used instead of targeting a specific microarchitecture.
2020-01-23commands: add simple `spack commands --update-completion` argument (#14607)Todd Gamblin2-3/+95
Instead of another script, this adds a simple argument to `spack commands` that updates the completion script. Developers can now just run: spack commands --update-completion This should make it simpler for developers to remember to run this *before* the tests fail. Also, this version tab-completes.
2020-01-22shell support: `spack load` no longer needs modules (#14062)Greg Becker13-108/+454
Previously the `spack load` command was a wrapper around `module load`. This required some bootstrapping of modules to make `spack load` work properly. With this PR, the `spack` shell function handles the environment modifications necessary to add packages to your user environment. This removes the dependence on environment modules or lmod and removes the requirement to bootstrap spack (beyond using the setup-env scripts). Included in this PR is support for MacOS when using Apple's System Integrity Protection (SIP), which is enabled by default in modern MacOS versions. SIP clears the `LD_LIBRARY_PATH` and `DYLD_LIBRARY_PATH` variables on process startup for executables that live in `/usr` (but not '/usr/local', `/System`, `/bin`, and `/sbin` among other system locations. Spack cannot know the `LD_LIBRARY_PATH` of the calling process when executed using `/bin/sh` and `/usr/bin/python`. The `spack` shell function now manually forwards these two variables, if they are present, as `SPACK_<VAR>` and recovers those values on startup. - [x] spack load/unload no longer delegate to modules - [x] refactor user_environment modification calculations - [x] update documentation for spack load/unload Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2020-01-22Use `spack commands --format=bash` to generate shell completion (#14393)Adam J. Stewart43-370/+751
This PR adds a `--format=bash` option to `spack commands` to auto-generate the Bash programmable tab completion script. It can be extended to work for other shells. Progress: - [x] Fix bug in superclass initialization in `ArgparseWriter` - [x] Refactor `ArgparseWriter` (see below) - [x] Ensure that output of old `--format` options remains the same - [x] Add `ArgparseCompletionWriter` and `BashCompletionWriter` - [x] Add `--aliases` option to add command aliases - [x] Standardize positional argument names - [x] Tests for `spack commands --format=bash` coverage - [x] Tests to make sure `spack-completion.bash` stays up-to-date - [x] Tests for `spack-completion.bash` coverage - [x] Speed up `spack-completion.bash` by caching subroutine calls This PR also necessitates a significant refactoring of `ArgparseWriter`. Previously, `ArgparseWriter` was mostly a single `_write` method which handled everything from extracting the information we care about from the parser to formatting the output. Now, `_write` only handles recursion, while the information extraction is split into a separate `parse` method, and the formatting is handled by `format`. This allows subclasses to completely redefine how the format will appear without overriding all of `_write`. Co-Authored-by: Todd Gamblin <tgamblin@llnl.gov>
2020-01-22bugfix: gpg2 is called 'gpg' on macOSTodd Gamblin3-13/+89
The gpg2 command isn't always around; it's sometimes called gpg. This is the case with the brew-installed version, and it's breaking our tests. - [x] Look for both 'gpg2' and 'gpg' when finding the command - [x] If we find 'gpg', ensure the version is 2 or higher - [x] Add tests for version detection.
2020-01-22tests: removed code duplication (#14596)Massimiliano Culpo6-83/+37
- [x] Factored to a common place the fixture `testing_gpg_directory`, renamed it as `mock_gnupghome` - [x] Removed altogether the function `has_gnupg2` For `has_gnupg2`, since we were not trying to parse the version from the output of: ```console $ gpg2 --version ``` this is effectively equivalent to check if `spack.util.gpg.GPG.gpg()` was found. If we need to ensure version is `^2.X` it's probably better to do it in `spack.util.gpg.GPG.gpg()` than in a separate function.
2020-01-21bugfix: hashes should use ordered dictionaries (#14390)Todd Gamblin3-8/+81
Despite trying very hard to keep dicts out of our hash algorithm, we seem to still accidentally add them in ways that the tests can't catch. This can cause errors when hashes are not computed deterministically. This fixes an error we saw with Python 3.5, where dictionary iteration order is random. In this instance, we saw a bug when reading Spack environment lockfiles -- The load would fail like this: ``` ... File "/sw/spack/lib/spack/spack/environment.py", line 1249, in concretized_specs yield (s, self.specs_by_hash[h]) KeyError: 'qcttqplkwgxzjlycbs4rfxxladnt423p' ``` This was because the hashes differed depending on whether we wrote `path` or `module` first when recomputing the build hash as part of reading a Spack lockfile. We can fix it by ensuring a determistic iteration order. - [x] Fix two places (one that caused an issue, and one that did not... yet) where our to_node_dict-like methods were using regular python dicts. - [x] Also add a check that statically analyzes our to_node_dict functions and flags any that use Python dicts. The test found the two errors fixed here, specifically: ``` E AssertionError: assert [] == ['Use syaml_dict instead of ...pack/spack/spec.py:1495:28'] E Right contains more items, first extra item: 'Use syaml_dict instead of dict at /Users/gamblin2/src/spack/lib/spack/spack/spec.py:1495:28' E Full diff: E - [] E + ['Use syaml_dict instead of dict at ' E + '/Users/gamblin2/src/spack/lib/spack/spack/spec.py:1495:28'] ``` and ``` E AssertionError: assert [] == ['Use syaml_dict instead of ...ack/architecture.py:359:15'] E Right contains more items, first extra item: 'Use syaml_dict instead of dict at /Users/gamblin2/src/spack/lib/spack/spack/architecture.py:359:15' E Full diff: E - [] E + ['Use syaml_dict instead of dict at ' E + '/Users/gamblin2/src/spack/lib/spack/spack/architecture.py:359:15'] ```
2020-01-21pipelines: `spack ci` command with env-based workflow (#12854)Scott Wittenburg11-261/+2165
Rework Spack's continuous integration workflow to be environment-based. - Add the `spack ci` command, which replaces the many scripts in `bin/` - `spack ci` decouples the CI workflow from the spack instance: - CI is defined in a spack environment - environment is in its own (single) git repository, separate from Spack - spack instance used to run the pipeline is up to the user - A new `gitlab-ci` section in environments allows users to configure how specs in the environment should be mapped to runners - Compilers can be bootstrapped in the new pipeline workflow - Add extensive documentation on pipelines (see `pipelines.rst` for further details) - Add extensive tests for pipeline code
2020-01-21Use util.url.join for URLs in GNU Mirrors / reorder Mirrors (#14395)Dr. Christian Tacke1-4/+4
* Reorder GNU mirrors (#14395) As @adamjstewart commented in #14395, GNU suggests to use their mirror. So reorder the mirror to the top. GNU Doc: https://www.gnu.org/prep/ftp.en.html * Use spack.util.url.join for URLs in GNU mirrors (#14395) One should not use os.path.join for URLs. This does only work on POSIX systems. Instead use spack.util.url.join. So every part in spack uses the same url joining method.
2020-01-16Fix use of sys.executable for module/env commands (#14496)Adam J. Stewart2-4/+6
* Fix use of sys.executable for module/env commands * Fix unit tests * More consistent quotation, less duplication * Fix import syntax
2020-01-15Fix typo in modules docstring (#14521)Adam J. Stewart1-1/+1
2020-01-14Fix parsing of rocketmq URL (#14490)Adam J. Stewart2-0/+3
2020-01-13Fixes #14402 (#14483)Tamara Dahlgren1-1/+1
Check if patchelf is executable, not binary, in case a site is wrapping it.
2020-01-08Remove extensions from view in the correct order (#12961)Jeffrey Salmond2-17/+39
When removing packages from a view, extensions were being deactivated in an arbitrary order. Extensions must be deactivated in preorder traversal (dependents before dependencies), so when this order was violated the view update would fail. This commit ensures that views deactivate extensions based on a preorder traversal and adds a test for it.
2020-01-08spack.compilers.clang: add new version check (#14365)Tim Haines2-1/+11
2020-01-07bugfix: Issue #14346, buildcache create s3 push fails when package w same ↵eugeneswalker1-1/+1
DAG hash already exists at mirror (#14412)