summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2019-12-04version bump: 0.13.2Todd Gamblin1-1/+1
2019-12-04Bugfix: allow missing modules if they are blacklisted (#13540)Peter Scheibel4-30/+115
`spack module loads` and `spack module find` previously failed if any upstream modules were missing. This prevented it from being used with upstreams (or, really, any spack instance) that blacklisted modules. This PR makes module finding is now more lenient (especially for blacklisted modules). - `spack module find` now does not report an error if the spec is blacklisted - instead, it prints a single warning if any modules will be omitted from the loads file - It comments the missing modules out of the loads file so the user can see what's missing - Debug messages are also printed so users can check this with `spack -d...` - also added tests for new functionality
2019-12-02Speedup environment activation (#13557)Massimiliano Culpo2-2/+13
* Add a transaction around repeated calls to `spec.prefix` in the activation process * cache the computation of home in the python package to speed up setting deps * ensure that module-scope variables are only set *once* per module
2019-12-01bugfix: mirror path works for unknown versions (#13626)Todd Gamblin2-2/+8
`mirror_archive_path` was failing to account for the case where the fetched version isn't known to Spack. - [x] don't require the fetched version to be in `Package.versions` - [x] add regression test for mirror paths when package does not have a version
2019-12-01environments: don't try to modify run-env if a spec is not installed (#13589)Adam J. Stewart1-2/+2
Fixes #13529 Fixes #13509
2019-12-01use semicolons instead of newlines in module/python command (#13904)Greg Becker1-1/+1
2019-12-01verify.py: os.path.exists exception handling (#13656)Daryl W. Grunau1-14/+16
2019-11-17Document use of the maintainers field (#13748)Adam J. Stewart2-0/+19
2019-11-15Bugfix/config caching 13754 (#13759)Greg Becker1-2/+3
* remove reference to `spack.store` in method definition Referencing `spack.store` in method definition will cache the `spack.config.config` singleton variable too early, before we have a chance to add command line and environment scopes.
2019-11-14Config option to allow gpg warning suppression (#13744)Greg Becker3-3/+9
Add a configuration option to suppress gpg warnings during binary package verification. This only suppresses warnings: a gpg failure will still fail the install. This allows users who have already explicitly trusted the gpg key they are using to avoid seeing repeated warnings that it is self-signed.
2019-11-14determine target relative to the link directory rather than the full link ↵Peter Scheibel1-3/+3
path (which includes the file name) (#13727)
2019-11-13Allow binary relocation of strings in relative binaries (#13724)Greg Becker2-15/+25
Binaries with relative RPATHS currently do not relocate strings hard-coded in binaries This PR extends the best-effort relocation of strings hard-coded in binaries to those whose RPATHs have been relativized.
2019-11-05`spack find` now displays variants and other spec constraintsv0.13.1Todd Gamblin3-4/+23
If you do this in a spack environment: spack add hdf5+hl hdf5+hl will be the root added to the `spack.yaml` file, and you should really expect `hdf5+hl` to display as a root in the environment. - [x] Add decoration to roots so that you can see the details about what is required to build. - [x] Add a test.
2019-11-05bugfix: uninstall should find concrete specs by DAG hashTodd Gamblin1-1/+9
This fixes a regression introduced in #10792. `spack uninstall` in an environment would not match concrete query specs properly after the index hash of enviroments changed. - [x] Search by DAG hash for specs to remove instead of by build hash
2019-11-04environments: make shell modifications partially unconditional (#13523)Greg Becker2-1/+37
* environments: make shell modifications partially unconditional * flake * missing module name * add regression test * flake
2019-11-04binary distribution: relocate text files properly in relative binaries (#13578)Greg Becker2-10/+12
* Make relative binaries relocate text files properly * rb strings aren't valid in python 2 * move perl to new interface for setup_environment family methods
2019-11-03bugfix: fetch prefers to fetch local mirrors over remote resources (#13545)Omar Padron3-8/+11
- [x] insert at beginning of list so fetch grabs local mirrors before remote resources - [x] update the S3FetchStrategy so that it throws a SpackError if the fetch fails. Before, it was throwing URLError, which was not being caught in stage.py. - [x] move error handling out of S3FetchStrategy and into web_util.read_from_url() - [x] pass string instead of URLError to SpackWebError
2019-11-03environments: only write when necessary (#13546)Greg Becker4-25/+66
This changes Spack environments so that the YAML file associated with the environment is *only* written when necessary (i.e., if it is changed *by spack*). The lockfile is still written out as before. There is a larger question here of which part of Spack should be responsible for setting defaults in config files, and how we can get rid of empty lists and data structures currently cluttering files like `compilers.yaml`. But that probably requires a rework of the default-setting validator in `spack.config`, as well as the code that uses `spack.config`. This will at least help for `spack.yaml`.
2019-11-01version bump: 0.13.1Todd Gamblin1-1/+1
2019-11-01bugfix: spack.util.url.join() now handles absolute paths correctly (#13488)Omar Padron3-2/+379
* fix issue where spack.util.url.join() failed to correctly handle absolute path components * add url util tests
2019-11-01sbang: use utf-8 for encoding when patching (#13490)Gregory Lee1-4/+13
This fixes a UnicodeDecodeError in the sbang patching function.
2019-11-01Specs with quoted flags containing spaces are parsed correctly (#13521)Massimiliano Culpo2-1/+22
2019-11-01targets: print a warning message before downgrading (#13513)Massimiliano Culpo2-25/+57
* Make package preferences a soft failure for targets, instead of a hard failure. * Added unit tests for preferences expressed via packages.yaml
2019-11-01Travis CI: Test Python 3.8 (#13347)Adam J. Stewart3-4/+20
* Travis CI: Test Python 3.8 * Fix use of deprecated cgi.escape method * Fix version comparison * Fix flake8 F811 change in Python 3.8 * Make flake8 happy * Use Python 3.8 for all test categories
2019-11-01Documentation: Database.query methods share docstrings (#13515)Greg Becker1-41/+53
Currently, query arguments in the Spack core are documented on the Database._query method, where the functionality is defined. For users of the spack python command, this makes the python builtin method help less than ideally useful, as help(spack.store.db.query) and help(spack.store.db.query_local) do not show relevant information. This PR updates the doc attributes for the Database.query and Database.query_local arguments to mirror everything after the first line of the Database._query docstring.
2019-11-01cuda: fix conflict statements for x86-64 targets (#13472)Massimiliano Culpo1-4/+6
* cuda: fix conflict statements for x86-64 targets fixes #13462 This build system mixin was not updated after the support for specific targets has been merged. * Updated the version range of cuda that conflicts with gcc@8: * Updated the version range of cuda that conflicts with gcc@8: for ppc64le * Relaxed conflicts for version > 10.1 * Updated versions in conflicts Co-Authored-By: Axel Huebl <axel.huebl@plasma.ninja>
2019-11-01cpu: fix clang flags for generic x86_64 (#13491)Greg Becker2-8/+16
* cpu: differentiate flags used for pristine LLVM vs. Apple's version
2019-11-01syaml_int type should use int.__repr__ rather than str.__repr__ (#13487)Peter Scheibel1-1/+1
2019-10-25version bump: 0.13.0Todd Gamblin1-1/+1
2019-10-25tutorial: move tutorial to standalone site (#13450)Todd Gamblin31-7565/+28
* docs: add a spack environment for building the docs * docs: remove tutorial and link to spack-tutorial.readthedocs.io The tutorial now has its own standalone website, versioned by instances of the tutorial. Link to that instead of versioning it directly with Spack.
2019-10-25`mirror create --all` can mirror everything (#12940)Peter Scheibel11-228/+465
Support mirroring all packages with `spack mirror create --all`. In this mode there is no concretization: * Spack pulls every version of every package into the created mirror. * It also makes multiple attempts for each package/version combination (if there is a temporary connection failure). * Continues if all attempts fail. i.e., this makes its best effort to fetch evrerything, even if all attempts to fetch one package fail. This also changes mirroring logic to prefer storing sources by their hash or by a unique name derived from the source. For example: * Archives with checksums are named by the sha256 sum, i.e., `archive/f6/f6cf3bd233f9ea6147b21c7c02cac24e5363570ce4fd6be11dab9f499ed6a7d8.tar.gz` vs the previous `<package-name>-package-version>.tar.gz` * VCS repositories are stored by a path derived from their URL, e.g. `git/google/leveldb.git/master.tar.gz`. The new mirror layout allows different packages to refer to the same resource or source without duplicating that download in the mirror/cache. This change is not essential to mirroring everything but is expected to save space when mirroring packages that all use the same resource. The new structure of the mirror is: ``` <base directory>/ _source-cache/ <-- the _source-cache directory is new archive/ <-- archives/resources/patches stored by hash 00/ <-- 2-letter sha256 prefix 002748bdd0319d5ab82606cf92dc210fc1c05d0607a2e1d5538f60512b029056.tar.gz 01/ 0154c25c45b5506b6d618ca8e18d0ef093dac47946ac0df464fb21e77b504118.tar.gz 0173a74a515211997a3117a47e7b9ea43594a04b865b69da5a71c0886fa829ea.tar.gz ... git/ OpenFAST/ openfast.git/ master.tar.gz <-- repo by branch name PHASTA/ phasta.git/ 11f431f2d1a53a529dab4b0f079ab8aab7ca1109.tar.gz <-- repo by commit ... svn/ <-- each fetch strategy has its own subdirectory ... openmpi/ <-- the remaining package directories have the old format openmpi-1.10.1.tar.gz <-- human-readable name is symlink to _source-cache ``` In addition to the archive names as described above, `mirror create` now also creates symlinks with the old format to help users understand which package each mirrored archive is associated with, and to allow mirrors to work with old spack versions. The symlinks are relative so the mirror directory can still itself be archived. Other improvements: * `spack mirror create` will not re-download resources that have already been placed in it. * When creating a mirror, the resources downloaded to the mirror will not be cached (things are not stored twice).
2019-10-25bugfix: restore upstream lock safety; update testsPeter Scheibel2-19/+48
Restore upstream lock safety; avoid calling methods directly on upstream DB in test.
2019-10-25Make error msg of test_api_for_build_and_run_environment more informative ↵Andreas Baumbach1-1/+2
(#13435)
2019-10-25bugfix: reindexing is not necessary for DB v0.9.3 to v5 upgrade (#13434)Todd Gamblin1-3/+25
reindexing takes a significant amount of time, and there's no reason to do it from DB version 0.9.3 to version 5. The only difference is that v5 can contain "deprecated_for" fields. - [x] Add a `_skip_reindex` list at the start of `database.py` - [x] Skip the reindex for upgrades in this list. The new version will just be written to the file the first time we actually have to write the DB out (e.g., after an install), and reads will still work fine.
2019-10-24bugfix: allow fetching no-code packages (#13429)Todd Gamblin3-22/+12
Previously, spack would error out if we tried to fetch something with no code, but that would prevent fetching dependencies. In particular, this would fail: spack fetch --dependencies xsdk - [x] Instead of raising an error, just print a message that there is nothing to be fetched for packages like xsdk that do not have code. - [x] Make BundleFetchStrategy a bit more quiet about doing nothing.
2019-10-24commands: Add `--json` argument to `spack spec` (#13431)Todd Gamblin2-5/+23
We've had `spack spec --yaml` for a while, and we've had methods for JSON for a while as well. We just haven't has a `--json` argument for `spack spec`. - [x] Add a `--json` argument to `spack spec`, just like `--yaml`
2019-10-24Improvements to detection of AMD architectures. (#13407)Chris Green6-4/+142
New entry for K10 microarchitecture. Reorder Zen* microarchitectures to avoid triggering as k10. Remove some desktop-specific flags that were preventing Opteron Bulldozer/Piledriver/Steamroller/Excavator CPUs from being recognized as such. Remove one or two flags which weren't produced in /proc/cpuinfo on older OS (RHEL6 and friends).
2019-10-24Correctly identify Skylake CPUs on Darwin. (#13377)Chris Green4-1/+12
* Correctly identify Skylake CPUs on Darwin. * Add a test for haswell on Mojave.
2019-10-24database: update DB version and change to integer database versions (#13410)Greg Becker1-1/+3
Update on version format: change to an integer database format version for simplicity, instead of tracking the Spack version.
2019-10-24Patch libtool when using the Arm compiler (#12004)Nick Forrington1-0/+24
* Patch libtool when using the arm, clang, and fujitsu compilers If libtool does not have values for linker/pic flags, patch them in
2019-10-23add `spack dev-build` command; deprecate `spack diy` (#13374)Greg Becker6-86/+189
Rename the `spack diy` command to `spack dev-build` to make the use case clearer. The `spack diy` command has some useful functionality for developers using Spack to build their dependencies and configure/build/install the code they are developing. Developers do not notice it, partly because of the obscure name. The `spack dev-build` command has a `-u/--until PHASE` option to stop after a given phase of the build. This can be used to configure your project, run cmake on your project, or similarly stop after any stage of the build the user wants. These options are analogous to the existing `spack configure` and `spack build` commands, but for developer builds. To unify the syntax, we have deprecated the `spack configure` and `spack build` commands, and added a `-u/--until PHASE` option to the `spack install` command as well. The functionality in `spack dev-build` (specifically `spack dev-build -u cmake`) may be able to supersede the `spack setup` command, but this PR does not deprecate that command as that will require slightly more thought.
2019-10-23bugfix: web.push_to_url should not format the local path that is passed to ↵Peter Scheibel1-6/+1
it (#13408) fd58c98 formats the `Stage`'s `archive_path` in `Stage.archive` (as part of `web.push_to_url`). This is not needed and if the formatted differs from the original path (for example if the archive file name contains a URL query suffix), then the copy fails. This removes the formatting that occurs in `web.push_to_url`. We should figure out a way to handle bad cases like this *and* to have nicer filenames for downloaded files. One option that would work in this particular case would be to also pass `-J` / `--remote-header-name` to `curl`. We'll need to do follow-up work to determine if we can use `-J` everywhere. See also: https://github.com/spack/spack/pull/11117#discussion_r338301058
2019-10-23syaml.dump now allows representing all syaml_ objects (originally this was ↵Peter Scheibel1-10/+5
only available for syaml.dump_config) (#13403)
2019-10-23Users can configure use of RPATH or RUNPATH (#9168)Massimiliano Culpo8-2/+167
Add a new entry in `config.yaml`: config: shared_linking: 'rpath' If this variable is set to `rpath` (the default) Spack will set RPATH in ELF binaries. If set to `runpath` it will set RUNPATH. Details: * Spack cc wrapper explicitly adds `--disable-new-dtags` when linking * cc wrapper also strips `--enable-new-dtags` from the compile line when disabling (and vice versa) * We specifically do *not* add any dtags flags on macOS, which uses Mach-O binaries, not ELF, so there's no RUNPATH)
2019-10-23commands: add `spack deprecate` command (#12933)Greg Becker13-63/+789
`spack deprecate` allows for the removal of insecure packages with minimal impact to their dependents. It allows one package to be symlinked into the prefix of another to provide seamless transition for rpath'd and hard-coded applications using the old version. Example usage: spack deprecate /hash-of-old-openssl /hash-of-new-openssl The spack deprecate command is designed for use only in extroardinary circumstances. The spack deprecate command makes no promises about binary compatibility. It is up to the user to ensure the replacement is suitable for the deprecated package.
2019-10-23url summary: show right and wrong parse counts for each regexTodd Gamblin1-13/+30
Previously this command only showed total counts for each regular expression. This doesn't give you a sense of which regexes are working well and which ones are not. We now display the number of right, wrong, and total URL parses per regex. It's easier to see where we might improve the URL parsing with this change.
2019-10-23Preserve comments for Spack YAML objects (#11602)Todd Gamblin13-178/+171
This updates the configuration loading/dumping logic (now called load_config/dump_config) in spack_yaml to preserve comments (by using ruamel.yaml's RoundTripLoader). This has two effects: * environment spack.yaml files expect to retain comments, which load_config now supports. By using load_config, users can now use the ':' override syntax that was previously unavailable for environment configs (but was available for other config files). * config files now retain user comments by default (although in cases where Spack updates/overwrites config, the comments can still be removed). Details: * Subclasses `RoundTripLoader`/`RoundTripDumper` to parse yaml into ruamel's `CommentedMap` and analogous data structures * Applies filename info directly to ruamel objects in cases where the updated loader returns those * Copies management of sections in `SingleFileScope` from #10651 to allow overrides to occur * Updates the loader/dumper to handle the processing of overrides by specifically checking for the `:` character * Possibly the most controversial aspect, but without that, the parsed objects have to be reconstructed (i.e. as was done in `mark_overrides`). It is possible that `mark_overrides` could remain and a deep copy will not cause problems, but IMO that's generally worth avoiding. * This is also possibly controversial because Spack YAML strings can include `:`. My reckoning is that this only occurs for version specifications, so it is safe to check for `endswith(':') and not ('@' in string)` * As a consequence, this PR ends up reserving spack yaml functions load_config/dump_config exclusively for the purpose of storing spack config
2019-10-22tests: avoid extra output in `env status` test (#13344)Todd Gamblin1-9/+12
`test_envoronment_status()` was printing extra output during tests. - [x] disable output only for `env('status')` calls instead of disabling it for the whole test.
2019-10-22env activation: use package defined env setup methods (#13249)Greg Becker5-44/+189
This PR ensures that environment activation sets all environment variables set by the equivalent `module load` operations, except that the spec prefixes are "rebased" to the view associated with the environment. Currently, Spack blindly adds paths relative to the environment view root to the user environment on activation. Issue #12731 points out ways in which this behavior is insufficient. This PR changes that behavior to use the `setup_run_environment` logic for each package to augment the prefix inspections (as in Spack's modulefile generation logic) to ensure that all necessary variables are set to make use of the packages in the environment. See #12731 for details on the previous problems in behavior. This PR also updates the `ViewDescriptor` object in `spack.environment` to have a `__contains__` method. This allows for checks like `if spec in self.default_view`. The `__contains__` operator for `ViewDescriptor` objects checks whether the spec satisfies the filters of the View descriptor, not whether the spec is already linked into the underlying `FilesystemView` object.
2019-10-22microarchitectures: look in /sbin and /usr/sbin for sysctl (#13365)Massimiliano Culpo3-22/+30
This PR ensures that on Darwin we always append /sbin and /usr/sbin to PATH, if they are not already present, when looking for sysctl. * Make sure we look into /sbin and /usr/sbin for sysctl * Refactor sysctl for better readability * Remove marker to make test pass