summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2020-01-22shell support: `spack load` no longer needs modules (#14062)Greg Becker13-108/+454
Previously the `spack load` command was a wrapper around `module load`. This required some bootstrapping of modules to make `spack load` work properly. With this PR, the `spack` shell function handles the environment modifications necessary to add packages to your user environment. This removes the dependence on environment modules or lmod and removes the requirement to bootstrap spack (beyond using the setup-env scripts). Included in this PR is support for MacOS when using Apple's System Integrity Protection (SIP), which is enabled by default in modern MacOS versions. SIP clears the `LD_LIBRARY_PATH` and `DYLD_LIBRARY_PATH` variables on process startup for executables that live in `/usr` (but not '/usr/local', `/System`, `/bin`, and `/sbin` among other system locations. Spack cannot know the `LD_LIBRARY_PATH` of the calling process when executed using `/bin/sh` and `/usr/bin/python`. The `spack` shell function now manually forwards these two variables, if they are present, as `SPACK_<VAR>` and recovers those values on startup. - [x] spack load/unload no longer delegate to modules - [x] refactor user_environment modification calculations - [x] update documentation for spack load/unload Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2020-01-22Use `spack commands --format=bash` to generate shell completion (#14393)Adam J. Stewart43-370/+751
This PR adds a `--format=bash` option to `spack commands` to auto-generate the Bash programmable tab completion script. It can be extended to work for other shells. Progress: - [x] Fix bug in superclass initialization in `ArgparseWriter` - [x] Refactor `ArgparseWriter` (see below) - [x] Ensure that output of old `--format` options remains the same - [x] Add `ArgparseCompletionWriter` and `BashCompletionWriter` - [x] Add `--aliases` option to add command aliases - [x] Standardize positional argument names - [x] Tests for `spack commands --format=bash` coverage - [x] Tests to make sure `spack-completion.bash` stays up-to-date - [x] Tests for `spack-completion.bash` coverage - [x] Speed up `spack-completion.bash` by caching subroutine calls This PR also necessitates a significant refactoring of `ArgparseWriter`. Previously, `ArgparseWriter` was mostly a single `_write` method which handled everything from extracting the information we care about from the parser to formatting the output. Now, `_write` only handles recursion, while the information extraction is split into a separate `parse` method, and the formatting is handled by `format`. This allows subclasses to completely redefine how the format will appear without overriding all of `_write`. Co-Authored-by: Todd Gamblin <tgamblin@llnl.gov>
2020-01-22bugfix: gpg2 is called 'gpg' on macOSTodd Gamblin3-13/+89
The gpg2 command isn't always around; it's sometimes called gpg. This is the case with the brew-installed version, and it's breaking our tests. - [x] Look for both 'gpg2' and 'gpg' when finding the command - [x] If we find 'gpg', ensure the version is 2 or higher - [x] Add tests for version detection.
2020-01-22tests: removed code duplication (#14596)Massimiliano Culpo6-83/+37
- [x] Factored to a common place the fixture `testing_gpg_directory`, renamed it as `mock_gnupghome` - [x] Removed altogether the function `has_gnupg2` For `has_gnupg2`, since we were not trying to parse the version from the output of: ```console $ gpg2 --version ``` this is effectively equivalent to check if `spack.util.gpg.GPG.gpg()` was found. If we need to ensure version is `^2.X` it's probably better to do it in `spack.util.gpg.GPG.gpg()` than in a separate function.
2020-01-21bugfix: hashes should use ordered dictionaries (#14390)Todd Gamblin3-8/+81
Despite trying very hard to keep dicts out of our hash algorithm, we seem to still accidentally add them in ways that the tests can't catch. This can cause errors when hashes are not computed deterministically. This fixes an error we saw with Python 3.5, where dictionary iteration order is random. In this instance, we saw a bug when reading Spack environment lockfiles -- The load would fail like this: ``` ... File "/sw/spack/lib/spack/spack/environment.py", line 1249, in concretized_specs yield (s, self.specs_by_hash[h]) KeyError: 'qcttqplkwgxzjlycbs4rfxxladnt423p' ``` This was because the hashes differed depending on whether we wrote `path` or `module` first when recomputing the build hash as part of reading a Spack lockfile. We can fix it by ensuring a determistic iteration order. - [x] Fix two places (one that caused an issue, and one that did not... yet) where our to_node_dict-like methods were using regular python dicts. - [x] Also add a check that statically analyzes our to_node_dict functions and flags any that use Python dicts. The test found the two errors fixed here, specifically: ``` E AssertionError: assert [] == ['Use syaml_dict instead of ...pack/spack/spec.py:1495:28'] E Right contains more items, first extra item: 'Use syaml_dict instead of dict at /Users/gamblin2/src/spack/lib/spack/spack/spec.py:1495:28' E Full diff: E - [] E + ['Use syaml_dict instead of dict at ' E + '/Users/gamblin2/src/spack/lib/spack/spack/spec.py:1495:28'] ``` and ``` E AssertionError: assert [] == ['Use syaml_dict instead of ...ack/architecture.py:359:15'] E Right contains more items, first extra item: 'Use syaml_dict instead of dict at /Users/gamblin2/src/spack/lib/spack/spack/architecture.py:359:15' E Full diff: E - [] E + ['Use syaml_dict instead of dict at ' E + '/Users/gamblin2/src/spack/lib/spack/spack/architecture.py:359:15'] ```
2020-01-21pipelines: `spack ci` command with env-based workflow (#12854)Scott Wittenburg11-261/+2165
Rework Spack's continuous integration workflow to be environment-based. - Add the `spack ci` command, which replaces the many scripts in `bin/` - `spack ci` decouples the CI workflow from the spack instance: - CI is defined in a spack environment - environment is in its own (single) git repository, separate from Spack - spack instance used to run the pipeline is up to the user - A new `gitlab-ci` section in environments allows users to configure how specs in the environment should be mapped to runners - Compilers can be bootstrapped in the new pipeline workflow - Add extensive documentation on pipelines (see `pipelines.rst` for further details) - Add extensive tests for pipeline code
2020-01-21Use util.url.join for URLs in GNU Mirrors / reorder Mirrors (#14395)Dr. Christian Tacke1-4/+4
* Reorder GNU mirrors (#14395) As @adamjstewart commented in #14395, GNU suggests to use their mirror. So reorder the mirror to the top. GNU Doc: https://www.gnu.org/prep/ftp.en.html * Use spack.util.url.join for URLs in GNU mirrors (#14395) One should not use os.path.join for URLs. This does only work on POSIX systems. Instead use spack.util.url.join. So every part in spack uses the same url joining method.
2020-01-16Fix use of sys.executable for module/env commands (#14496)Adam J. Stewart2-4/+6
* Fix use of sys.executable for module/env commands * Fix unit tests * More consistent quotation, less duplication * Fix import syntax
2020-01-15Fix typo in modules docstring (#14521)Adam J. Stewart1-1/+1
2020-01-14Fix parsing of rocketmq URL (#14490)Adam J. Stewart2-0/+3
2020-01-13Fixes #14402 (#14483)Tamara Dahlgren1-1/+1
Check if patchelf is executable, not binary, in case a site is wrapping it.
2020-01-08Remove extensions from view in the correct order (#12961)Jeffrey Salmond2-17/+39
When removing packages from a view, extensions were being deactivated in an arbitrary order. Extensions must be deactivated in preorder traversal (dependents before dependencies), so when this order was violated the view update would fail. This commit ensures that views deactivate extensions based on a preorder traversal and adds a test for it.
2020-01-08spack.compilers.clang: add new version check (#14365)Tim Haines2-1/+11
2020-01-07bugfix: Issue #14346, buildcache create s3 push fails when package w same ↵eugeneswalker1-1/+1
DAG hash already exists at mirror (#14412)
2020-01-07Spack can automatically remove unused specs (#13534)Massimiliano Culpo7-7/+193
* Spack can uninstall unused specs fixes #4382 Added an option to spack uninstall that removes all unused specs i.e. build dependencies or transitive dependencies that are left in the store after the specs that pulled them in have been removed. * Moved the functionality to its own command The command has been named 'spack autoremove' to follow the naming used for the same functionality by other widely known package managers i.e. yum and apt. * Speed-up autoremoving specs by not locking and re-reading the scratch DB * Make autoremove work directly on Spack's store * Added unit tests for the new command * Display a terser output to the user * Renamed the "autoremove" command "gc" Following discussion there's more consensus around the latter name. * Preserve root specs in env contexts * Instead of preserving specs, restrict gc to the active environment * Added docs * Added a unit test for gc within an environment * Updated copyright to 2020 * Updated documentation according to review Rephrased a couple of sentences, added references to `spack find` and dependency types. * Updated function naming and docstrings * Simplified computation of unused specs Since the new approach uses private attributes of the DB it has been coded as a method of that class rather than a freestanding function.
2020-01-06Reference `spack help --spec` in `spack spec --help`Adam J. Stewart1-0/+4
2020-01-02RHEL8 bugfix for module_cmd (#14349)Sajid Ali1-1/+2
2020-01-02MKL: set appropriate CMake env vars (#14274)Adam J. Stewart1-0/+8
2020-01-02refactor: cleanup imports in spec.pyTodd Gamblin9-169/+167
The imports in `spec.py` are getting to be pretty unwieldy. - [x] Remove all of the `import from` style imports and replace them with `import` or `import as` - [x] Remove a number names that were exported by `spack.spec` that weren't even in `spack.spec`
2020-01-02bugfix: cdash tests shoudln't modify working directoryTodd Gamblin1-8/+9
The latest cdash test creates a local cdash_reports directory, but it should do that in a tmpdir.
2020-01-01tests: improved `spack test` command line optionsTodd Gamblin4-61/+264
Previously, `spack test` automatically passed all of its arguments to `pytest -k` if no options were provided, and to `pytest` if they were. `spack test -l` also provided a list of test filenames, but they didn't really let you completely narrow down which tests you wanted to run. Instead of trying to do our own weird thing, this passes `spack test` args directly to `pytest`, and omits the implicit `-k`. This means we can now run, e.g.: ```console $ spack test spec_syntax.py::TestSpecSyntax::test_ambiguous ``` This wasn't possible before, because we'd pass the fully qualified name to `pytest -k` and get an error. Because `pytest` doesn't have the greatest ability to list tests, I've tweaked the `-l`/`--list`, `-L`/`--list-long`, and `-N`/`--list-names` options to `spack test` so that they help you understand the names better. you can combine these options with `-k` or other arguments to do pretty powerful searches. This one makes it easy to get a list of names so you can run tests in different orders (something I find useful for debugging `pytest` issues): ```console $ spack test --list-names -k "spec and concretize" cmd/env.py::test_concretize_user_specs_together concretize.py::TestConcretize::test_conflicts_in_spec concretize.py::TestConcretize::test_find_spec_children concretize.py::TestConcretize::test_find_spec_none concretize.py::TestConcretize::test_find_spec_parents concretize.py::TestConcretize::test_find_spec_self concretize.py::TestConcretize::test_find_spec_sibling concretize.py::TestConcretize::test_no_matching_compiler_specs concretize.py::TestConcretize::test_simultaneous_concretization_of_specs spec_dag.py::TestSpecDag::test_concretize_deptypes spec_dag.py::TestSpecDag::test_copy_concretized ``` You can combine any list option with keywords: ```console $ spack test --list -k microarchitecture llnl/util/cpu.py modules/lmod.py ``` ```console $ spack test --list-long -k microarchitecture llnl/util/cpu.py:: test_generic_microarchitecture modules/lmod.py::TestLmod:: test_only_generic_microarchitectures_in_root ``` Or just list specific files: ```console $ spack test --list-long cmd/test.py cmd/test.py:: test_list test_list_names_with_pytest_arg test_list_long test_list_with_keywords test_list_long_with_pytest_arg test_list_with_pytest_arg test_list_names ``` Hopefully this stuff will help with debugging test issues. - [x] make `spack test` send args directly to `pytest` instead of trying to do fancy things. - [x] rework `--list`, `--list-long`, and add `--list-names` to make searching for tests easier. - [x] make it possible to mix Spack's list args with `pytest` args (they're just fancy parsing around `pytest --collect-only`) - [x] add docs - [x] add tests - [x] update spack completion
2019-12-31Modify create clue list so R packages are detected (#12277)Glenn Johnson1-1/+1
R packages can contain configure scripts so R needs to be before autotools in the clue list.
2019-12-31tests: move mock config.yaml files to common directoryTodd Gamblin5-15/+8
Test configuration files (except modules.yaml) were in the root level of test/data, but should really just be in their own directory. The absence of modules.yaml was also breaking module tests if we got module preferences after tests started, as the mock modules.yaml was not in the test directory.
2019-12-31config: remove all module-scope calls to spack.config.get()Todd Gamblin4-16/+27
This avoids parsing modules.yaml on startup.
2019-12-31modules: make the module hook more robustTodd Gamblin1-8/+12
The module hook would previously fail if there were no enabled module types. - Instead of looking for a `KeyError`, default to empty list when the config variable is not present. - Convert lambdas to real functions for clarity.
2019-12-31hooks: remove pre_run hook to improve startup time.Todd Gamblin3-49/+0
- Remove legacy yaml_version_check() hook - Remove the pre_run hook from `hook/__init__.py` and `main.py` We want to discourage the use of pre-run hooks because they have to run at startup. To keep Spack fast, we should do things like this lazily instead of in hooks that require spidering directories full of modules.
2019-12-30copyright: update copyright dates for 2020 (#14328)Todd Gamblin418-422/+422
2019-12-30bugfix: add required fixture for CDash authentication test (#14325)Todd Gamblin1-1/+1
2019-12-30Add support for authenticated CDash uploads (#14200)Zack Galbreath4-3/+35
2019-12-30refactor: rename mock_config fixture to mock_low_high_configTodd Gamblin4-31/+32
This avoids confusion with mock_configuration.
2019-12-30argparse: lazily construct common argumentsTodd Gamblin3-72/+120
Continuing to shave small bits of time off startup -- `spack.cmd.common.arguments` constructs many `Args` objects at module scope, which has to be done for all commands that import it. Instead of doing this at load time, do it lazily. - [x] construct Args objects lazily - [x] remove the module-scoped argparse fixture - [x] make the mock config scope set dirty to False by default (like the regular scope) This *seems* to reduce load time slightly
2019-12-30tests: refactor tests to avoid persistent global stateTodd Gamblin16-90/+141
Previously, fixtures like `config`, `database`, and `store` were module-scoped, but frequently used as test function arguments. These fixtures swap out global on setup and restore them on teardown. As function arguments, they would do the right set-up, but they'd leave the global changes in place for the whole module the function lived in. This meant that if you use `config` once, other functions in the same module would inadvertently inherit the mock Spack configuration, as it would only be torn down once all tests in the module were complete. In general, we should module- or session-scope the *STATE* required for these global objects (as it's expensive to create0, but we shouldn't module-or session scope the activation/use of them, or things can get really confusing. - [x] Make generic context managers for global-modifying fixtures. - [x] Make session- and module-scoped fixtures that ONLY build filesystem state and create objects, but do not swap out any variables. - [x] Make seeparate function-scoped fixtures that *use* the session scoped fixtures and actually swap out (and back in) the global variables like `config`, `database`, and `store`. These changes make it so that global changes are *only* ever alive for a singlee test function, and we don't get weird dependencies because a global fixture hasn't been destroyed.
2019-12-30tests: make env tests that use configs non-destructiveTodd Gamblin1-10/+10
Environment tests pushed config scopes but didn't properly remove them. - [x] use `with env:` context manager instead of `env.prepare_config_scopes()`
2019-12-30package_prefs: move class-level cache to PackagePref instanceTodd Gamblin5-57/+11
`PackagePrefs` has had a class-level cache of data from `packages.yaml` for a long time, but it complicates testing and leads to subtle errors, especially now that we frequently manipulate custom config scopes and environments. Moving the cache to instance-level doesn't slow down concretization or the test suite, and it just caches for the life of a `PackagePrefs` instance (i.e., for a single cocncretization) so we don't need to worry about global state anymore. - [x] Remove class-level caches from `PackagePrefs` - [x] Add a cached _spec_order object on each `PackagePrefs` instance - [x] Remove all calls to `PackagePrefs.clear_caches()`
2019-12-28externals: avoid importing jinja2 on startup (#14308)Todd Gamblin2-5/+6
Jinja2 costs a tenth to a few tenths of a second to import, so we should avoid importing it on startup. - [x] only import jinja2 within functions
2019-12-28bugfix: colify_table should not revert to 1 column for non-tty (#14307)Todd Gamblin1-4/+13
Commands like `spack blame` were printig poorly when redirected to files, as colify reverts to a single column when redirected. This works for list data but not tables. - [x] Force a table by always passing `tty=True` from `colify_table()`
2019-12-27Improve info variant header (#14275)Dr. Christian Tacke1-1/+3
In "spack info" the Variants header currently has two blank lines under it. That's too much. It looks like the actual content belongs to something else. Instead underline the headers to make things more obvious.
2019-12-24tests: finish removing pyqver from the repository (#14294)Todd Gamblin2-2/+2
Remove a few remaining mentions of the pyqver package, which was removed in #14289.
2019-12-24tests: check min required python version with vermin (#14289)Massimiliano Culpo12-769/+12
This commit removes the `python_version.py` unit test module and the vendored dependencies `pyqver2.py` and `pyqver3.py`. It substitutes them with an equivalent check done using `vermin` that is run as a separate workflow via Github Actions. This allows us to delete 2 vendored dependencies that are unmaintained and substitutes them with a maintained tool. Also, updates the list of vendored dependencies.
2019-12-24a64fx: fix typo in GCC flags (#14286)t-karatsu1-2/+2
2019-12-24Merge branch 'releases/v0.13' into developTodd Gamblin1-1/+1
2019-12-23version bump: 0.13.3Todd Gamblin1-1/+1
2019-12-23performance: dont' read `spec.yaml` files twice in view regenerationTodd Gamblin2-4/+8
`ViewDescriptor.regenerate()` calls `get_all_specs()`, which reads `spec.yaml` files, which is slow. It's fine to do this once, but `view.remove_specs()` *also* calls it immediately afterwards. - [x] Pass the result of `get_all_specs()` as an optional parameter to `view.remove_specs()` to avoid reading `spec.yaml` files twice.
2019-12-23performance: don't recompute hashes when regenerating environmentsTodd Gamblin2-4/+9
`ViewDescriptor.regenerate()` was copying specs and stripping build dependencies, which clears `_hash` and other cached fields on concrete specs, which causes a bunch of YAML hashes to be recomputed. - [x] Preserve the `_hash` and `_normal` fields on stripped specs, as these will be unchanged.
2019-12-23performance: reduce system calls required for remove_dead_linksTodd Gamblin1-4/+2
`os.path.exists()` will report False if the target of a symlink doesn't exist, so we can avoid a costly call to realpath here.
2019-12-23performance: only regenerate env views once in `spack install`Todd Gamblin2-9/+29
`spack install` previously concretized, writes the entire environment out, regenerated views, then wrote and regenerated views again. Regenerating views is slow, so ensure that we only do that once. - [x] add an option to env.write() to skip view regeneration - [x] add a note on whether regenerate_views() shouldn't just be a separate operation -- not clear if we want to keep it as part of write to ensure consistency, or take it out to avoid performance issues.
2019-12-23performance: add read transactions for `install_all()` and `install()`Todd Gamblin1-31/+35
Environments need to read the DB a lot when installing all specs. - [x] Put a read transaction around `install_all()` and `install()` to avoid repeated locking
2019-12-23lock transactions: avoid redundant reading in write transactionsTodd Gamblin2-1/+62
Our `LockTransaction` class was reading overly aggressively. In cases like this: ``` 1 with spack.store.db.read_transaction(): 2 with spack.store.db.write_transaction(): 3 ... ``` The `ReadTransaction` on line 1 would read in the DB, but the WriteTransaction on line 2 would read in the DB *again*, even though we had a read lock the whole time. `WriteTransaction`s were only considering nested writes to decide when to read, but they didn't know when we already had a read lock. - [x] `Lock.acquire_write()` return `False` in cases where we already had a read lock.
2019-12-23lock transactions: ensure that nested write transactions writeTodd Gamblin2-1/+64
If a write transaction was nested inside a read transaction, it would not write properly on release, e.g., in a sequence like this, inside our `LockTransaction` class: ``` 1 with spack.store.db.read_transaction(): 2 with spack.store.db.write_transaction(): 3 ... 4 with spack.store.db.read_transaction(): ... ``` The WriteTransaction on line 2 had no way of knowing that its `__exit__()` call was the last *write* in the nesting, and it would skip calling its write function. The `__exit__()` call of the `ReadTransaction` on line 1 wouldn't know how to write, and the file would never be written. The DB would be correct in memory, but the `ReadTransaction` on line 4 would re-read the whole DB assuming that other processes may have modified it. Since the DB was never written, we got stale data. - [x] Make `Lock.release_write()` return `True` whenever we release the *last write* in a nest.
2019-12-23lock transactions: fix non-transactional writesTodd Gamblin4-172/+285
Lock transactions were actually writing *after* the lock was released. The code was looking at the result of `release_write()` before writing, then writing based on whether the lock was released. This is pretty obviously wrong. - [x] Refactor `Lock` so that a release function can be passed to the `Lock` and called *only* when a lock is really released. - [x] Refactor `LockTransaction` classes to use the release function instead of checking the return value of `release_read()` / `release_write()`