summaryrefslogtreecommitdiff
AgeCommit message (Collapse)AuthorFilesLines
2020-02-07update CHANGELOG.md for 0.13.4v0.13.4Todd Gamblin1-0/+9
2020-02-07version bump: 0.13.4Todd Gamblin1-1/+1
2020-02-07bugfix: make `_source_single_file` work in venvs (#14569)Massimiliano Culpo2-6/+10
Using `sys.executable` to run Python in a sub-shell doesn't always work in a virtual environment as the `sys.executable` Python is not necessarily compatible with any loaded spack/other virtual environment. - revert use of sys.executable to print out subshell environment (#14496) - try instead to use an available python, then if there *is not* one, use `sys.executable` - this addresses RHEL8 (where there is no `python` and `PYTHONHOME` issue in a simpler way
2020-02-07Fix use of sys.executable for module/env commands (#14496)Adam J. Stewart2-4/+6
* Fix use of sys.executable for module/env commands * Fix unit tests * More consistent quotation, less duplication * Fix import syntax
2020-02-07RHEL8 bugfix for module_cmd (#14349)Sajid Ali1-1/+2
2020-02-07Fix outdated bash tab completion (#14392)Adam J. Stewart1-198/+319
2020-02-07Remove extensions from view in the correct order (#12961)Jeffrey Salmond2-17/+39
When removing packages from a view, extensions were being deactivated in an arbitrary order. Extensions must be deactivated in preorder traversal (dependents before dependencies), so when this order was violated the view update would fail. This commit ensures that views deactivate extensions based on a preorder traversal and adds a test for it.
2020-02-07bugfix: hashes should use ordered dictionaries (#14390)Todd Gamblin3-8/+81
Despite trying very hard to keep dicts out of our hash algorithm, we seem to still accidentally add them in ways that the tests can't catch. This can cause errors when hashes are not computed deterministically. This fixes an error we saw with Python 3.5, where dictionary iteration order is random. In this instance, we saw a bug when reading Spack environment lockfiles -- The load would fail like this: ``` ... File "/sw/spack/lib/spack/spack/environment.py", line 1249, in concretized_specs yield (s, self.specs_by_hash[h]) KeyError: 'qcttqplkwgxzjlycbs4rfxxladnt423p' ``` This was because the hashes differed depending on whether we wrote `path` or `module` first when recomputing the build hash as part of reading a Spack lockfile. We can fix it by ensuring a determistic iteration order. - [x] Fix two places (one that caused an issue, and one that did not... yet) where our to_node_dict-like methods were using regular python dicts. - [x] Also add a check that statically analyzes our to_node_dict functions and flags any that use Python dicts. The test found the two errors fixed here, specifically: ``` E AssertionError: assert [] == ['Use syaml_dict instead of ...pack/spack/spec.py:1495:28'] E Right contains more items, first extra item: 'Use syaml_dict instead of dict at /Users/gamblin2/src/spack/lib/spack/spack/spec.py:1495:28' E Full diff: E - [] E + ['Use syaml_dict instead of dict at ' E + '/Users/gamblin2/src/spack/lib/spack/spack/spec.py:1495:28'] ``` and ``` E AssertionError: assert [] == ['Use syaml_dict instead of ...ack/architecture.py:359:15'] E Right contains more items, first extra item: 'Use syaml_dict instead of dict at /Users/gamblin2/src/spack/lib/spack/spack/architecture.py:359:15' E Full diff: E - [] E + ['Use syaml_dict instead of dict at ' E + '/Users/gamblin2/src/spack/lib/spack/spack/architecture.py:359:15'] ```
2019-12-23update CHANGELOG.md for 0.13.3v0.13.3Todd Gamblin1-0/+16
2019-12-23version bump: 0.13.3Todd Gamblin1-1/+1
2019-12-23Travis exits at the first failing test, pin codecov at v4.5.4 (#14179)Massimiliano Culpo2-2/+2
Before this commit we used to run the entire unit test suite in the presence of a failure. Since we currently rely a lot on the state of the filesystem etc. the end report was most of the time showing spurious failures that were a consequence of the first failing test. This PR makes unit tests exit at the first failing test Also, pin codecov at v4.5.4 (last one supporting Python 2.6)
2019-12-23performance: dont' read `spec.yaml` files twice in view regenerationTodd Gamblin2-4/+8
`ViewDescriptor.regenerate()` calls `get_all_specs()`, which reads `spec.yaml` files, which is slow. It's fine to do this once, but `view.remove_specs()` *also* calls it immediately afterwards. - [x] Pass the result of `get_all_specs()` as an optional parameter to `view.remove_specs()` to avoid reading `spec.yaml` files twice.
2019-12-23performance: don't recompute hashes when regenerating environmentsTodd Gamblin2-4/+9
`ViewDescriptor.regenerate()` was copying specs and stripping build dependencies, which clears `_hash` and other cached fields on concrete specs, which causes a bunch of YAML hashes to be recomputed. - [x] Preserve the `_hash` and `_normal` fields on stripped specs, as these will be unchanged.
2019-12-23performance: reduce system calls required for remove_dead_linksTodd Gamblin1-4/+2
`os.path.exists()` will report False if the target of a symlink doesn't exist, so we can avoid a costly call to realpath here.
2019-12-23performance: only regenerate env views once in `spack install`Todd Gamblin2-9/+29
`spack install` previously concretized, writes the entire environment out, regenerated views, then wrote and regenerated views again. Regenerating views is slow, so ensure that we only do that once. - [x] add an option to env.write() to skip view regeneration - [x] add a note on whether regenerate_views() shouldn't just be a separate operation -- not clear if we want to keep it as part of write to ensure consistency, or take it out to avoid performance issues.
2019-12-23performance: add read transactions for `install_all()` and `install()`Todd Gamblin1-31/+35
Environments need to read the DB a lot when installing all specs. - [x] Put a read transaction around `install_all()` and `install()` to avoid repeated locking
2019-12-23lock transactions: avoid redundant reading in write transactionsTodd Gamblin2-1/+62
Our `LockTransaction` class was reading overly aggressively. In cases like this: ``` 1 with spack.store.db.read_transaction(): 2 with spack.store.db.write_transaction(): 3 ... ``` The `ReadTransaction` on line 1 would read in the DB, but the WriteTransaction on line 2 would read in the DB *again*, even though we had a read lock the whole time. `WriteTransaction`s were only considering nested writes to decide when to read, but they didn't know when we already had a read lock. - [x] `Lock.acquire_write()` return `False` in cases where we already had a read lock.
2019-12-23lock transactions: ensure that nested write transactions writeTodd Gamblin2-1/+64
If a write transaction was nested inside a read transaction, it would not write properly on release, e.g., in a sequence like this, inside our `LockTransaction` class: ``` 1 with spack.store.db.read_transaction(): 2 with spack.store.db.write_transaction(): 3 ... 4 with spack.store.db.read_transaction(): ... ``` The WriteTransaction on line 2 had no way of knowing that its `__exit__()` call was the last *write* in the nesting, and it would skip calling its write function. The `__exit__()` call of the `ReadTransaction` on line 1 wouldn't know how to write, and the file would never be written. The DB would be correct in memory, but the `ReadTransaction` on line 4 would re-read the whole DB assuming that other processes may have modified it. Since the DB was never written, we got stale data. - [x] Make `Lock.release_write()` return `True` whenever we release the *last write* in a nest.
2019-12-23lock transactions: fix non-transactional writesTodd Gamblin4-172/+285
Lock transactions were actually writing *after* the lock was released. The code was looking at the result of `release_write()` before writing, then writing based on whether the lock was released. This is pretty obviously wrong. - [x] Refactor `Lock` so that a release function can be passed to the `Lock` and called *only* when a lock is really released. - [x] Refactor `LockTransaction` classes to use the release function instead of checking the return value of `release_read()` / `release_write()`
2019-12-23performance: avoid repeated DB locking on view generationTodd Gamblin1-10/+13
`ViewDescriptor.regenerate()` checks repeatedly whether packages are installed and also does a lot of DB queries. Put a read transaction around the whole thing to avoid repeatedly locking and unlocking the DB.
2019-12-23performance: speed up `spack find` in environmentsTodd Gamblin1-7/+11
`Environment.added_specs()` has a loop around calls to `Package.installed()`, which can result in repeated DB queries. Optimize this with a read transaction in `Environment`.
2019-12-23performance: `spack spec` should use a read transacction with -ITodd Gamblin1-9/+26
`spack spec -I` queries the database for installation status and should use a read transaction around calls to `Spec.tree()`.
2019-12-23concretization: improve performance by avoiding database locksTodd Gamblin1-4/+6
Checks for deprecated specs were repeatedly taking out read locks on the database, which can be very slow. - [x] put a read transaction around the deprecation check
2019-12-23performance: memoize spack.architecture.get_platform()Todd Gamblin2-0/+8
`get_platform()` is pretty expensive and can be called many times in a spack invocation. - [x] memoize `get_platform()`
2019-12-23use `sys.executable` instead of `python` in `_source_single_file` (#14252)Sajid Ali1-1/+1
2019-12-23Patch fetching: remove unnecessary argumentPeter Josef Scheibel3-8/+4
2019-12-23Mirrors: skip attempts to fetch BundlePackagesPeter Josef Scheibel1-0/+10
BundlePackages use a noop fetch strategy. The mirror logic was assuming that the fetcher had a resource to cach after performing a fetch. This adds a special check to skip caching if the stage is associated with a BundleFetchStrategy. Note that this should allow caching resources associated with BundlePackages.
2019-12-23Mirrors: avoid re-downloading patchesPeter Josef Scheibel2-20/+28
When updating a mirror, Spack was re-retrieving all patches (since the fetch logic for patches is separate). This updates the patch logic to allow the mirror logic to avoid this.
2019-12-23Mirrors: perform checksum of fetched sourcesPeter Josef Scheibel1-0/+1
Since cache_mirror does the fetch itself, it also needs to do the checksum itself if it wants to verify that the source stored in the mirror is valid. Note that this isn't strictly required because fetching (including from mirrors) always separately verifies the checksum.
2019-12-23Mirrors: fix cosmetic symlink targetsPeter Josef Scheibel2-1/+34
The targets for the cosmetic paths in mirrrors were being calculated incorrectly as of fb3a3ba: the symlinks used relative paths as targets, and the relative path was computed relative to the wrong directory.
2019-12-23Allow repeated invocations of 'mirror create'Peter Josef Scheibel1-0/+6
When creating a cosmetic symlink for a resource in a mirror, remove it if it already exists. The symlink is removed in case the logic to create the symlink has changed.
2019-12-23mirror bug fixes: symlinks, duplicate patch names, and exception handling ↵Paul Ferrell4-17/+24
(#13789) * Some packages (e.g. mpfr at the time of this patch) can have patches with the same name but different contents (which apply to different versions of the package). This appends part of the patch hash to the cache file name to avoid conflicts. * Some exceptions which occur during fetching are not a subclass of SpackError and therefore do not have a 'message' attribute. This updates the logic for mirroring a single spec (add_single_spec) to produce an appropriate error message in that case (where before it failed with an AttributeError) * In various circumstances, a mirror can contain the universal storage path but not a cosmetic symlink; in this case it would not generate a symlink. Now "spack mirror create" will create a symlink for any package that doesn't have one.
2019-12-04update CHANGELOG.md for 0.13.2v0.13.2Todd Gamblin1-0/+40
2019-12-04version bump: 0.13.2Todd Gamblin1-1/+1
2019-12-04Bugfix: allow missing modules if they are blacklisted (#13540)Peter Scheibel4-30/+115
`spack module loads` and `spack module find` previously failed if any upstream modules were missing. This prevented it from being used with upstreams (or, really, any spack instance) that blacklisted modules. This PR makes module finding is now more lenient (especially for blacklisted modules). - `spack module find` now does not report an error if the spec is blacklisted - instead, it prints a single warning if any modules will be omitted from the loads file - It comments the missing modules out of the loads file so the user can see what's missing - Debug messages are also printed so users can check this with `spack -d...` - also added tests for new functionality
2019-12-02Speedup environment activation (#13557)Massimiliano Culpo3-4/+21
* Add a transaction around repeated calls to `spec.prefix` in the activation process * cache the computation of home in the python package to speed up setting deps * ensure that module-scope variables are only set *once* per module
2019-12-01bugfix: mirror path works for unknown versions (#13626)Todd Gamblin2-2/+8
`mirror_archive_path` was failing to account for the case where the fetched version isn't known to Spack. - [x] don't require the fetched version to be in `Package.versions` - [x] add regression test for mirror paths when package does not have a version
2019-12-01environments: don't try to modify run-env if a spec is not installed (#13589)Adam J. Stewart1-2/+2
Fixes #13529 Fixes #13509
2019-12-01use semicolons instead of newlines in module/python command (#13904)Greg Becker1-1/+1
2019-12-01verify.py: os.path.exists exception handling (#13656)Daryl W. Grunau1-14/+16
2019-11-17Document use of the maintainers field (#13748)Adam J. Stewart2-0/+19
2019-11-15Bugfix/config caching 13754 (#13759)Greg Becker1-2/+3
* remove reference to `spack.store` in method definition Referencing `spack.store` in method definition will cache the `spack.config.config` singleton variable too early, before we have a chance to add command line and environment scopes.
2019-11-15hwloc: added 'master' version pointing at the HEAD of the master branch (#13734)Massimiliano Culpo1-4/+6
2019-11-14Config option to allow gpg warning suppression (#13744)Greg Becker4-3/+17
Add a configuration option to suppress gpg warnings during binary package verification. This only suppresses warnings: a gpg failure will still fail the install. This allows users who have already explicitly trusted the gpg key they are using to avoid seeing repeated warnings that it is self-signed.
2019-11-14determine target relative to the link directory rather than the full link ↵Peter Scheibel1-3/+3
path (which includes the file name) (#13727)
2019-11-13Allow binary relocation of strings in relative binaries (#13724)Greg Becker2-15/+25
Binaries with relative RPATHS currently do not relocate strings hard-coded in binaries This PR extends the best-effort relocation of strings hard-coded in binaries to those whose RPATHs have been relativized.
2019-11-05`spack find` now displays variants and other spec constraintsv0.13.1Todd Gamblin3-4/+23
If you do this in a spack environment: spack add hdf5+hl hdf5+hl will be the root added to the `spack.yaml` file, and you should really expect `hdf5+hl` to display as a root in the environment. - [x] Add decoration to roots so that you can see the details about what is required to build. - [x] Add a test.
2019-11-05bugfix: uninstall should find concrete specs by DAG hashTodd Gamblin1-1/+9
This fixes a regression introduced in #10792. `spack uninstall` in an environment would not match concrete query specs properly after the index hash of enviroments changed. - [x] Search by DAG hash for specs to remove instead of by build hash
2019-11-04environments: make shell modifications partially unconditional (#13523)Greg Becker2-1/+37
* environments: make shell modifications partially unconditional * flake * missing module name * add regression test * flake
2019-11-04binary distribution: relocate text files properly in relative binaries (#13578)Greg Becker3-17/+37
* Make relative binaries relocate text files properly * rb strings aren't valid in python 2 * move perl to new interface for setup_environment family methods