summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2020-06-08view remove: directly check whether specs own files before removing from ↵Greg Becker2-5/+44
view (#16955) Bugfix for hardlinks and copies
2020-06-05spack dev-build: Do not mark -u builds in database (#16333)Greg Becker4-21/+61
Builds can be stopped before the final install phase due to user requests. Those builds should not be registered as installed in the database. We had code intended to handle this but: 1. It caught the wrong type of exception 2. We were catching these exceptions to suppress them at a lower level in the stack This PR allows the StopIteration to propagate through a ChildError, and catches it properly. Also added to an existing test to prevent regression.
2020-06-05commands: use a single ThreadPool for `spack versions` (#16749)Massimiliano Culpo4-251/+228
This fixes a fork bomb in `spack versions`. Recursive generation of pools to scrape URLs in `_spider` was creating large numbers of processes. Instead of recursively creating process pools, we now use a single `ThreadPool` with a concurrency limit. More on the issue: having ~10 users running at the same time spack versions on front-end nodes caused kernel lockup due to the high number of sockets opened (sys-admin reports ~210k distributed over 3 nodes). Users were internal, so they had ulimit -n set to ~70k. The forking behavior could be observed by just running: $ spack versions boost and checking the number of processes spawned. Number of processes per se was not the issue, but each one of them opens a socket which can stress `iptables`. In the original issue the kernel watchdog was reporting: Message from syslogd@login03 at May 19 12:01:30 ... kernel:Watchdog CPU:110 Hard LOCKUP Message from syslogd@login03 at May 19 12:01:31 ... kernel:watchdog: BUG: soft lockup - CPU#110 stuck for 23s! [python3:2756] Message from syslogd@login03 at May 19 12:01:31 ... kernel:watchdog: BUG: soft lockup - CPU#94 stuck for 22s! [iptables:5603]
2020-06-03Fix parsing of EvtGen URL (#16883)Adam J. Stewart2-9/+11
* Fix parsing of EvtGen URL * Flake8 fix
2020-06-03Mirrors: add option to exclude packages from "mirror create" (#14154)Peter Scheibel3-23/+98
* add an --exclude-file option to 'spack mirror create' which allows a user to specify a file of specs to exclude when creating a mirror. this is anticipated to be useful especially when using the '--all' option * allow specifying number of versions when mirroring all packages * when mirroring all specs within an environment, include dependencies of root specs * add '--exclude-specs' option to allow user to specify that specs should be excluded on the command line * add test for excluding specs
2020-06-03Feature: add option to create view by copying/relocating files (#16480)Greg Becker4-14/+87
* add subcommand `spack view copy/relocate` * update bash completions * add copy/relocate commands to view tests * allow copied views to be removed
2020-06-03Implicit rpaths for NAG/GCC mixed toolchain (#14782)Sergey Kosukhin6-38/+139
* Implicit rpaths for NAG. * set up environment when checking for implicit rpaths
2020-06-03spack uninstall: improve help message (#16886)Massimiliano Culpo1-7/+15
fixes #12527 Mention that specs can be uninstalled by hash also in the help message. Reference `spack gc` in case people are looking for ways to clean the store from build time dependencies. Use "spec" instead of "package" to avoid ambiguity in the error message.
2020-06-02Unify tests for compiler command in the same file (#16891)Massimiliano Culpo3-113/+105
* Unify tests for compiler command in the same file Tests for the "spack compiler" command were previously scattered among different files. * Tests should use mutable_config, since they modify the compiler list
2020-06-02Fix satisfaction checks for excluding variants from matrices (#16893)Greg Becker6-49/+79
Because of the way abstract variants are implemented, the following spec matrix does not work as intended: ``` matrix: - [foo] - [bar=a, bar=b] exclude: - bar=a ``` because abstract variants always satisfy any variant of the same name, regardless of values. This PR converts abstract variants to whatever their appropriate type is before running satisfaction checks for the excludes clause in a matrix. fixes #16841
2020-06-01openmpi: add opa-psm2 dependency (#16873)Jonathon Anderson1-1/+118
Also document with_or_without and enable_or_disable, (which are used to configure the opa-psm2 dependency).
2020-06-01Recognize system installed gcc-10 as compilers (#16884)Massimiliano Culpo1-1/+1
Now that the version number of GCC reached double digits, an update to the regex is needed to recognize gcc-10 as an executable to be inspected when searching for compilers.
2020-05-29repo: fix construction of UnknownPackageError (#16865)Sergey Kosukhin2-15/+24
Also improve the error messages
2020-05-29Mention that packages can be uninstalled by hash (#16863)Robert Rosca1-1/+2
2020-05-29spack module_cmd: set LD_LIBRARY_PATH for python in subshell. (#16827)Greg Becker3-8/+46
* Ensure python runs with proper ld_library_path within Spack regardless of environment
2020-05-29Update error message when UnavailableCompilerVersionError is triggered (#16838)Jane Herriman1-1/+3
Co-authored-by: Herriman <herriman1@panorama.llnl.gov>
2020-05-29compilers: add opt_flags and debug_flags properties (#16710)Greg Becker11-0/+122
2020-05-28bugfix: use flags when computing implicit rpaths (#16634)Greg Becker10-25/+113
* make verbose_flag a property * tests
2020-05-27spack containerize: allow 0.14.3 tag for next releaseMassimiliano Culpo2-5/+9
2020-05-27spack containerize: allow 0.14.1 and 0.14.2 tagsMassimiliano Culpo2-5/+16
fixes #16727
2020-05-27concretize: fix UnboundLocalError due to import within a function (#16809)Sergey Kosukhin2-1/+7
2020-05-26backwards compatibility for naming scheme (#16812)Greg Becker4-3/+30
* backwards compatibility for naming scheme
2020-05-26spack.relocate: further coverage for ELF related functions (#16585)Massimiliano Culpo4-141/+258
* make_link_relative: added docstring * make_elf_binaries_relative: added docstring, unit tests * raise_if_not_relocatable: added docstring, added unit test for exceptional case * relocate_links: removed unused arguments, added docstring and comments Also fixed a possible bug that was issuing spurious warning when a file was relocated successfully * relocate_text: added docstring and comments, renamed arguments * relocate_text_bin: added docstring and comments, renamed arguments, unit tests
2020-05-25lua: fix compilation on Cray (#16713)Stephen Herbein1-1/+1
Problem: when calling `static_to_shared_library` on the `cray` arch, it produces a non-sensical compiler command with no input files. For example, when installing lua@5.2.4, it produced: 'gcc -lm -ldl -o /big-long-spack-path/liblua.so.5.2.4' Solution: do the same thing on `cray` that is done for `linux`
2020-05-25bugfix: schema errors without line numbers (#16765)Peter Scheibel1-4/+8
* account for schema validation errors where the associated instance doesn't have a line number * fix unrelated flake error (but it must be fixed because this PR touches this file and the flake rules have been updated since the last edit to this file)
2020-05-21autotools: delete args from postdep objects when %fj (#16274)Tomoki, Karatsu1-0/+3
2020-05-20externals: allow package prefs to configure default not buildable (#16735)Greg Becker2-4/+74
Allows `all` to be configured non-buildable in packages.yaml. The following config would only allow zlib to be built by Spack, all other packages would have to be found as externals. ``` packages: all: buildable: False zlib: buildable: True ```
2020-05-20tests: check presence not equality for rpath changes (#16637)Tamara Dahlgren1-3/+9
Fixes #16636
2020-05-19Pipelines: Ensure consistent spack version for entire pipelineScott Wittenburg1-16/+4
Also retry jobs always, as script failure is not confined to only our spack ci rebuild command exit code.
2020-05-18move source dir to last cmake arg (#16709)Greg Becker1-2/+2
2020-05-16intel-mpi: add libfabrics directory to SPACK_COMPILER_EXTRA_RPATHS (#15214)TZ1-0/+9
workaround for not finding an appropriate libfabrics.so with Intel MPI 2019+. Fixes #11140 and #12493
2020-05-15modules: use projections format for naming schemes (#16629)Greg Becker15-62/+171
* update tcl naming_scheme to use projections * add projections to lmod modules
2020-05-14Pipelines: Support DAG scheduling and dynamic child pipelinesScott Wittenburg4-292/+420
This change also adds a code path through the spack ci pipelines infrastructure which supports PR testing on the Spack repository. Gitlab pipelines run as a result of a PR (either creation or pushing to a PR branch) will only verify that the packages in the environment build without error. When the PR branch is merged to develop, another pipeline will run which results in the generated binaries getting pushed to the binary mirror.
2020-05-14Support os-specific $padding in config:install_treeScott Wittenburg2-11/+108
Providing only $padding or ${padding} results in an attempt to substitute a padding of maximum system path length, while leaving room for the parts of the install path spack generates. Providing $padding-<len> or ${padding-<len>} simply substitutes padding of the specified length.
2020-05-14Feature: Allow lmod configuration to set core specs (#16517)Greg Becker5-3/+30
Packages built with lmod core_compiler are placed in `Core`. Other packages may belong in `Core`. For example, python may be built with a proprietary compiler for performance, but belong on the `Core` directory. With this PR, lmod config can include a `core_specs` list. Any package that satisfies a spec in that list is placed in `Core`, regardless of its compiler or dependencies.
2020-05-13cmake build system: filter system paths from rpaths (#16612)Greg Becker1-1/+1
2020-05-12Update docs on "spack external find" (#16482)Peter Scheibel2-8/+80
This improves the documentation for `spack external find` in several ways: * Provide a code example of implementing `determine_spec_details` for a package * Explain how to define executables to look for (and also e.g. that they are treated as regular expressions and so can pull in unexpected files). * Add the "why" for a couple of constraints (i.e. explain that this logic only works for build/run deps because it examines `PATH` for executables) * Spread the docs between build customization and packaging sections * Add cross-references * Add a label so that `spack external find` is linked from the command reference.
2020-05-12bugfix: reorder variants in Spec strings (#16462)Greg Becker3-18/+17
* change print order for variants to avoid zsh parsing bugs * change tests for new variant parse order
2020-05-11Charm++: fix build prefix; add ucx/pmi support (#15666)Sajid Ali1-2/+1
* Add pmi support (required by ucx, ofi, and gni backends) * Add support for ucx backend * Add dependency on MPI for pmi=simplepmi, slurmpmi, or slurmpmi2 * Remove charmpp as an MPI provider since the changes in this PR can add MPI as a dependency (mentioned previously) * Install into transport_protocol-OS-arch subdirectory to match default charmpp installation behavior (which helps dependents find it)
2020-05-11Remove 'spack bootstrap' and associated docs (#15179)Massimiliano Culpo4-145/+8
fixes #15145 This commit removes the outdated `spack bootstrap` command and any reference to it in the documentation and unit tests.
2020-05-09Increase coverage of spack.relocate (#16475)Massimiliano Culpo2-120/+266
- add docstrings and make parameter names consistent in `relocate.py` - Make `replace_prefix_*` and other functions private (as they are implementation details) - remove unused function _replace_prefix_nullterm() - Add unit tests for `relocate.py` functions - add patchelf to Travis and use it during tests - add hello_world fixture with a compiled binary, so we can test relocation
2020-05-09bugfix: don't use sys.stdout as a default arg value (#16541)Todd Gamblin8-12/+25
After migrating to `travis-ci.com`, we saw I/O issues in our tests -- tests that relied on `capfd` and `capsys` were failing. We've also seen this in GitHub actions, and it's kept us from switching to them so far. Turns out that the issue is that using streams like `sys.stdout` as default arguments doesn't play well with `pytest` and output redirection, as `pytest` changes the values of `sys.stdout` and `sys.stderr`. if these values are evaluated before output redirection (as they are when used as default arg values), output won't be captured properly later. - [x] replace all stream default arg values with `None`, and only assign stream values inside functions. - [x] fix tests we didn't notice were relying on this erroneous behavior
2020-05-08all_urls: add urls[0] for versions (#16435)Axel Huebl1-0/+4
This adds the `url` alternative `urls` to `package.all_urls`. With this addition, one can find again new versions with `spack versions <package>` for packages that are populated with from mixin mirror `urls`. Example: `util-macros` from x.org mixin.
2020-05-07Documentation: typo in option description (#16518)Erik Schnetter1-1/+1
2020-05-07Config option to disable setting S_ISGID bit when creating installation ↵iarspider2-1/+2
directory (#14479) * Add config option to disable setting S_ISGID bit when creating installation directory. Co-authored-by: Ivan Razumov <ivan.razumov@cern.ch>
2020-05-07"spack checksum" QoL (#14311)iarspider3-4/+18
* Non-interactive mode for spack checksum; allow passing 'package@version' to spack checksum * Flake8 fixes * Update checksum.py Fix typo * Update spack-completion script * Automatically set non-interactive mode if more than one version passed * Update lib/spack/spack/cmd/checksum.py Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com> * Add documentation and update spack-completion * Flake8 * Rename option * Update spack-completion * Update lib/spack/spack/cmd/checksum.py Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com> * Update checksum.py * Update stage.py * Update create.py Use batch mode when adding a new package Co-authored-by: Ivan Razumov <ivan.razumov@cern.ch> Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2020-05-07Testing: fix unintended interactions between tests, part 2 (#16429)Peter Scheibel6-12/+47
This fixes some errors with setting up test configuration. These errors do not cause current Spack tests to fail but do create red herring issues elsewhere (see #15666). Fixing these errors leads to more errors in tests that depended on the original misconfigured state, so those are also addressed here. This is an update to #16003 which accounts for some unit tests with conflicting config/mutable_config fixtures. These conflicts were not exposed until the mutable_config fixture was fixed. Details are included below. The change which builds on #16003 is prefixed with "(new)". * For tests that use the real Spack package repository, the config needs to avoid using MPI providers that are not intended to be installed by Spack. Without this, it is possible that Spack tests which concretize the MPI virtual will end up trying to use an implementation that it shouldn't (e.g. one that is always provided externally). See #15666 for an example. * The mutable_config test fixture was not initializing the scope roots to the right directories (so the resulting config was empty). * The current_host fixture in the concretize.py tests was using the config fixture rather than mutable_config, and was polluting the config cache for other tests. * One test in concretize.py was clearing a nonexistent cache (PackagePrefs._packages_config_cache). This reference has been removed. * The test 'test_preferred_compilers' was was depending on cross test config pollution to succeed. The initial spec before concretization has been updated to updated to be explicit about the desired result. * (new) For tests that use install_mockery and mutable_config, replace install_mockery with a separate install_mockery_mutable_config fixture that is exactly the same as install_mockery but uses the mutable_config fixture to avoid conflicts.
2020-05-07 bugfix: spack shouldn't fail in an incomplete environment (#16473)Todd Gamblin2-9/+49
Fixed #15884. Spack asks every package linked into an environment to tell us how environment variables should be modified when a spack environment is activated. As part of this, specs in an environment are symlinked into the environment's view (see #13249), and the package calculates environment modifications with *the default view as the prefix*. All of this works nicely for pointing the user's environment at the view *if* every package is successfully linked. Unfortunately, right now we only track what specs "should" be in a view, not which specs actually are. So we end up calculating environment modifications on things that aren't linked into thee view, and the exception isn't caught, so lots of spack commands end up failing. This fixes the issue by ignoring and warning about specs where calculating environment modifications fails. So we can still keep using Spack even if the current environment is incomplete. We should probably also just avoid computing env modifications *entirely* for unlinked packages, but right now that is a slow operation (requires a lot of YAML parsing). We should revisit that when we have some better state management for views, but the fix adopted here will still be necessary, as we want spack commands to be resilient to other types of bugs in `setup_run_environment()` and friends. That code is in packages and we have to assume it could be buggy when we call it outside of builds (as it might fail more than just the build).
2020-05-06fix underscore/dash problems for mic_knl and skylake_avx512 (#16491)Greg Becker1-1/+2
2020-05-05Automatically find externals (#15158)Peter Scheibel3-0/+479
Add a `spack external find` command that tries to populate `packages.yaml` with external packages from the user's `$PATH`. This focuses on finding build dependencies. Currently, support has only been added for `cmake`. For a package to be discoverable with `spack external find`, it must define: * an `executables` class attribute containing a list of regular expressions that match executable names. * a `determine_spec_details(prefix, specs_in_prefix)` method Spack will call `determine_spec_details()` once for each prefix where executables are found, passing in the path to the prefix and the path to all found executables. The package is responsible for invoking the executables and figuring out what type of installation(s) are in the prefix, and returning one or more specs (each with version, variants or whatever else the user decides to include in the spec). The found specs and prefixes will be added to the user's `packages.yaml` file. Providing the `--not-buildable` option will mark all generated entries in `packages.yaml` as `buildable: False`