summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2020-05-01Revert "Testing: fix unintended interactions between tests (#16003)" (#16420)Peter Scheibel5-23/+8
This was breaking tests on develop.
2020-05-01Testing: fix unintended interactions between tests (#16003)Peter Scheibel5-8/+23
* For tests that use the real Spack package repository, the config needs to avoid using MPI providers that are not intended to be installed by Spack. Without this, it is possible that Spack tests which concretize the MPI virtual will end up trying to use an implementation that it shouldn't (e.g. one that is always provided externally). See #15666 for an example. * The mutable_config test fixture was not initializing the scope roots to the right directories (so the resulting config was empty). * The current_host fixture in the concretize.py tests was using the config fixture rather than mutable_config, and was polluting the config cache for other tests. * One test in concretize.py was clearing a nonexistent cache (PackagePrefs._packages_config_cache). This reference has been removed. * The test 'test_preferred_compilers' was was depending on cross test config pollution to succeed. The initial spec before concretization has been updated to updated to be explicit about the desired result.
2020-05-01dev-build: --drop-in <shell> (#14887)Axel Huebl2-0/+19
* dev-build: --drop-in <shell> Add a `--drop-in <shell>` option to `spack dev-build`. This option will automatically run a `spack build-env <spec> -- <shell>` at the end of a `dev-build`, e.g. to quickly drop-and-devel into a build phase of a package. Example usage: ``` spack dev-build --before cmake --drop-in bash openpmd-api@develop ``` * build_env: drop in unit test Co-authored-by: Greg Becker <becker33@llnl.gov>
2020-04-30Fix typos in new AutotoolsPackage function (#16406)Adam J. Stewart1-2/+2
2020-04-29Spack command extensions: error-handling (#13635)Chris Green5-85/+332
Generally speaking, errors that are encountered when attempting to load command extensions now terminate the running Spack instance. * Added new exceptions `spack.cmd.PythonNameError` and `spack.cmd.CommandNameError`. * New functions `spack.cmd.require_python_name(pname)` and `spack.cmd.require_cmd_name(cname)` check that `pname` and `cname` respectively meet requirements, throwing the appropriate error if not. * `spack.cmd.get_module()` uses `require_cmd_name()` and passes through exceptions from module load attempts. * `spack.cmd.get_command()` uses `require_cmd_name()` and invokes `get_module()` with the correct command-name form rather than the previous (incorrect) Python name. * Added New exceptions `spack.extensions.CommandNotFoundError` and `spack.extensions.ExtensionNamingError`. * `_extension_regexp` has a new leading underscore to indicate expected privacy. * `spack.extensions.extension_name()` raises an `ExtensionNamingError` rather than using `tty.warn()`. * `spack.extensions.load_command_extension()` checks command source existence early and bails out if missing. Also, exceptions raised by `load_module_from_file()` are passed through. * `spack.extensions.get_module()` raises `CommandNotFoundError` as appropriate. * Spack `main()` allows `parser.add_command()` exceptions to cause program end. Tests: * More common boilerplate has been pulled out into fixtures including `sys.modules` dictionary cleanup and resource-managed creation of a simple command extension with specified contents in the source file for a single named command. * "Hello, World!" test now uses a command named `hello-world` instead of `hello` in order to verify correct handling of commands with hyphens. * New tests for: * Missing (or misnamed) command. * Badly-named extension. * Verification that errors encountered during import of a command are propagated upward. Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2020-04-29prefix gets removed when adding additional configure options (#16335)Nathan2-4/+6
* prefix gets removed when adding additional configure options * Update waf documentation to reflect the new change
2020-04-28dev-build: stop before phase (#14699)Axel Huebl4-6/+65
Add `-b,--before` option to dev-build command to stop before the phase in question.
2020-04-28Docs: Limit Microarch on Travis CI (#16200)Axel Huebl1-1/+11
Update the travis CI docs: limiting the microarch is essential for caching.
2020-04-28relocate: remove an assertion in _make_relative (#16351)Massimiliano Culpo1-4/+0
This was an oversight in #15654 since `os.path.isfile` checks that the something is a file in the current filesystem.
2020-04-27Improve the coverage of spack.relocate (#15654)Massimiliano Culpo3-76/+148
This PR introduces trivial refactoring in: - `get_existing_elf_rpaths` - `get_relative_elf_rpaths` - `get_normalized_elf_rpaths` - `set_placeholder` mainly to be more consistent with practices used in other parts of the code and to simplify functions locally. It also adds or reworks unit tests for these functions and extends their docstrings. Co-authored-by: Patrick Gartung <gartung@fnal.gov> Co-authored-by: Peter J. Scheibel <scheibel1@llnl.gov>
2020-04-24spack env: activate note (#16279)Axel Huebl1-0/+4
print a note on how to activate a newly created environment
2020-04-23tests: each mock package now has its own class (#16157)Todd Gamblin9-153/+279
Packages in Spack are classes, and we need to be able to execute class methods on mock packages. The previous design used instances of a single MockPackage class; this version gives each package its own class that can spider depenencies. This allows us to implement class methods like `possible_dependencies()` on mock packages. This design change moves mock package creation into the `MockPackageMultiRepo`, and mock packages now *must* be created from a repo. This is required for us to mock `possible_dependencies()`, which needs to be able to get dependency packages from the package repo. Changes include: * `MockPackage` is now `MockPackageBase` * `MockPackageBase` instances must now be created with `MockPackageMultiRepo.add_package()` * add `possible_dependencies()` method to `MockPackageBase` * refactor tests to use new code structure * move package mocking infrastructure into `spack.util.mock_package`, as it's becoming a more sophisticated class and it gets lots in `conftest.py`
2020-04-23spack info: allow variants section to be as wide as the terminal (#16254)Todd Gamblin1-9/+14
The variants table in `spack info` is cramped, as the *widest* it can be is 80 columns. And that's actually only sort of true -- the padding calculation is off, so it still wraps on terminals of size 80 because it comes out *slightly* wider. This change looks at the terminal size and calculates the width of the description column based on it. On larger terminals, the output looks much nicer, and on small terminals, the output no longer wraps. Here's an example for `spack info qmcpack` with 110 columns. Before: Name [Default] Allowed values Description ==================== ==================== ============================== afqmc [off] on, off Install with AFQMC support. NOTE that if used in combination with CUDA, only AFQMC will have CUDA. build_type [Release] Debug, Release, The build type to build RelWithDebInfo complex [off] on, off Build the complex (general twist/k-point) version cuda [off] on, off Build with CUDA After: Name [Default] Allowed values Description ==================== ==================== ======================================================== afqmc [off] on, off Install with AFQMC support. NOTE that if used in combination with CUDA, only AFQMC will have CUDA. build_type [Release] Debug, Release, The build type to build RelWithDebInfo complex [off] on, off Build the complex (general twist/k-point) version cuda [off] on, off Build with CUDA
2020-04-22spack info: replace "True, False" with "on, off" (#16235)Massimiliano Culpo1-4/+2
fixes #16184
2020-04-22update compiler config with bootstrapped compiler when already installed ↵Greg Becker5-4/+38
(#16221) Update compiler config with bootstrapped compiler when it was already installed and added config defaults to code so mutable_config test fixture works.
2020-04-22spack environments documentation: -E -> -e (#16242)George Hartzell1-2/+2
To specify an environment for a comment, the user can specify "spack -e <env>". The documentation incorrectly specified "-E" (which is actually used to ignore any implicit use of environments).
2020-04-21Doc: Conflict Msg Pkg Guide (#16201)Axel Huebl1-3/+7
Document the nice `msg=` argument in conflicts. Super useful for less generic conflict guidance.
2020-04-21compiler wrappers: don't override -isystem with -I (#16077)Andrew W Elble2-14/+102
If the Spack compiler wrapper encounters any "-isystem" option, then when adding include directories for Spack dependencies, Spack will use "-isystem" instead of "-I". This prevents Spack-generated "-I" options from overriding the "-isystem" options generated by the build system. To ensure that build-system "-isystem" directories are searched first, Spack places all of its inserted "-isystem" directories after. The new ordering of -isystem includes is: * -isystem from build system (not system directories) * -isystem from Spack * -isystem from build system (for directories like /usr/include) The prior order of "-I" arguments is preserved (although as of this commit Spack no longer generates -I if -isystem is detected): * -I from build system (not system directories) * -I from Spack (only if there are no "-isystem" options) * -I from build system (for directories like /usr/include)
2020-04-20autotools: remove FORCE_UNSAFE_CONFIGURE from docs, commands, code (#16145)Todd Gamblin2-11/+4
Since #16132, we've consolidated the setting of FORCE_UNSAFE_CONFIGURE to `autotools.py`, so we don't need to use it in packages like `coreutils`, in our commands, or in our container recipes. - [x] Remove FORCE_UNSAFE_CONFIGURE from packages - [x] Remove FORCE_UNSAFE_CONFIGURE from container recipes - [x] Remove FORCE_UNSAFE_CONFIGURE from `spack ci` command
2020-04-19Typo: vew -> view (#16155)George Hartzell1-1/+1
2020-04-18autotools: set FORCE_UNSAFE_CONFIGURE to 1 in autotools.py (#16132)Diego Magdaleno1-0/+15
This commit sets the `FORCE_UNSAFE_CONFIGURE` environment variable to 1 in autotools builds. We see a lot of builds popping up and complaining about `FORCE_UNSAFE_CONFIGURE`. This behavior is not actually part of `autoconf` per se. It comes from this patch to `mknod.m4`, which is used by a lot of autoconf builds: * https://lists.gnu.org/archive/html/bug-gnulib/2010-07/msg00282.html Which originated from this problem that someone had on AIX: * https://lists.gnu.org/archive/html/bug-gnulib/2010-07/msg00279.html The gist of the problem seems to be that they want to check whether `mknod` can do something as root, but instead of checking whether they're running as root and using `su` or something to test this, they just made it harder to run `configure` as root. This seems very ad hoc and this is one of many checks that are run as root in `configure`. Many of them run before this check, so it's not clear that the `FORCE_UNSAFE_CONFIGURE` thing is even preventing bad things from happening. So: 1. This only happens in `autotools` builds, so we should go ahead and put it into `autotools.py` instead of in the global build environment, and 2. The variable does too little and provides a false sense of security in the first place, so we'll just disable it and avoid the nuisance. If we really feel strongly about this we can put some warnings in Spack about running as root, but at the top level, not in the middle of an already running script like `configure`.
2020-04-17remove files accidentally committed with 0.14.0 (#16138)Todd Gamblin3-11712/+0
2020-04-17AutotoolsPackage: add aclocal for all build deps (#15784)Adam J. Stewart1-5/+5
2020-04-17Language-specific PIC flags (#15474)Adam J. Stewart12-77/+300
* Language-specific PIC flags * Add tests for every compiler flag * Fix bad rebase * pic_flag -> cxx_pic_flag
2020-04-16SourceForge: Mirror Mixin (#16112)Axel Huebl3-0/+64
* SourceForge: Mirror Mixin Add a mixing class for direct `CNAME`s to sourceforge mirrors. Since the main gateway servers are often down, this could reduce timeouts and fetch errors for sourceforge.net hosted software. * SourceForge: unspectacular mirror replacement add mirrors to all sourceforge packages with trivial download logic. tested fetch of latest version of each of these packages with various mirrors before committing. * SourceForge: xz the author homepage is chronocially overrun and this is the offical upload with many mirrors.
2020-04-16macos: use DYLD_FALLBACK_LIBRARY_PATH instead of DYLD_LIBRARY_PATHTodd Gamblin3-2/+6
`DYLD_LIBRARY_PATH` can frequently break builtin macOS software when pointed at Spack libraries. This is because it takes *higher* precedence than the default library search paths, which are used by system software. `DYLD_FALLBACK_LIBRARY_PATH`, on the other hand, takes lower precedence. At first glance, this might seem bad, because the software installed by Spack in an environment needs to find *its* libraries, and it should not use the defaults. However, Spack's isntallations are always `RPATH`'d, so they do not have this problem. `DYLD_FALLBACK_LIBRARY_PATH` is thus useful for things built in an environment that need to use Spack's libraries, that don't set *their* RPATHs correctly for whatever reason. We now prefer it to `DYLD_LIBRARY_PATH` in modules and in environments because it helps a little bit, and it is much less intrusive.
2020-04-16Bugfix: For spack buildcache check, Handle specs where only the hash is (#15663)Marc Allen1-1/+1
provided (#15662). Prior to this fix, the checked Spec object would not be populated, and concretization would fail. Co-authored-by: Marc Allen <mrcall@amazon.com>
2020-04-15Merge branch 'releases/v0.14' into developTodd Gamblin1-1/+1
2020-04-15Make tags case insensitiveMichael Kuhn2-0/+10
Currently, tags are case sensitive, which is unintuitive: ```console $ spack list -t hpc ==> 2 packages. nek5000 nektools $ spack list -t HPC ==> 1 packages. mfem $ spack list -t Hpc ==> 0 packages. ``` This change makes them case insensitive: ```console $ spack list -t hpc ==> 3 packages. mfem nek5000 nektools $ spack list -t HPC ==> 3 packages. mfem nek5000 nektools $ spack list -t Hpc ==> 3 packages. mfem nek5000 nektools ```
2020-04-15version bump: 0.14.2Todd Gamblin1-1/+1
2020-04-15bugfix: spack test should not output [+] for mock installs (#15609)Todd Gamblin1-3/+2
`spack test` has a spurious '[+] ' in the output: ``` lib/spack/spack/test/install.py .........[+] ...... ``` Output is properly suppressed: ``` lib/spack/spack/test/install.py ............... ```
2020-04-15`spack install` terminal output handling in foreground/background (#15723)Todd Gamblin7-224/+1155
Makes the following changes: * (Fixes #15620) tty configuration was failing when stdout was redirected. The implementation now creates a pseudo terminal for stdin and checks stdout properly, so redirections of stdin/out/err should be handled now. * Handles terminal configuration when the Spack process moves between the foreground and background (possibly multiple times) during a build. * Spack adjusts terminal settings to allow users to to enable/disable build process output to the terminal using a "v" toggle, abnormal exit cases (like CTRL-C) could leave the terminal in an unusable state. This is addressed here with a special-case handler which restores terminal settings. Significantly extend testing of process output logger: * New PseudoShell object for setting up a master and child process and configuring file descriptor inheritance between the two * Tests for "v" verbosity toggle making use of the added PseudoShell object * Added `uniq` function which takes a list of elements and replaces any consecutive sequence of duplicate elements with a single instance (e.g. "112211" -> "121") Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2020-04-15database: maintain in-memory consistency on remove (#15777)Massimiliano Culpo2-1/+21
The performance improvements done in #14693 where leaving the DB in an inconsistent state when specs were removed from it. This PR updates the DB internal state whenever the DB is written to a file. Note that we still cannot properly enumerate installed dependents, so there is a TODO in this code. Fixing that will require the dependents dictionaries in specs to be re-keyed (either by hash, or not keyed at all -- a list would do). See #11983 for details.
2020-04-15performance: add a verification file to the database (#14693)Andrew W Elble3-3/+52
Reading the database repeatedly can be quite slow. We need a way to speed up Spack when it reads the DB multiple times, but the DB has not been modified between reads (which is nearly all the time). - [x] Add a file containing a unique uuid that is regenerated at database write time. Use this uuid to suppress re-parsing the database contents if we know a previous uuid and the uuid has not changed. - [x] Fix mutable_database fixture so that it resets the last seen verifier when it resets. - [x] Enable not rereading the database immediately after a write. Make the tests reset the last seen verifier in between tests that use the database fixture. - [x] make presence of uuid module optional
2020-04-15Remove DB conversion of old index.yaml (#15298)Peter Scheibel1-84/+45
Removed the code that was converting the old index.yaml format into index.json. Since the change happened in #2189 it should be considered safe to drop this (untested) code.
2020-04-15Environments/views: only override spec prefix for non-external packages (#15475)Peter Scheibel2-1/+49
* only override spec prefix for non-external packages * add test that environment shell modifications respect explicitly-specified prefixes for external packages * add clarifying comment
2020-04-15Source devnull in environment_after_sourcing_files (closes #15775) (#15791)Jonathon Anderson1-1/+2
spack.util.environment_after_sourcing_files compares the local environment against a shell environment after having sourced a file; but this ends up including the default shell profile and rc, which might differ from the local environment. To change this, compare against the default shell environment, expressed here as 'source /dev/null'.
2020-04-15bugfix: TERM may not be in the environment on Cray (#15630)Todd Gamblin1-3/+7
2020-04-15Blacklist Lmod variable modifications when sourcing files (#15778)Massimiliano Culpo3-1/+24
fixes #15775 Add all the variables listed here: https://lmod.readthedocs.io/en/latest/090_configuring_lmod.html to the list of those blacklisted when constructing environment modifications by sourcing files.
2020-04-15Tests: Mirror Mixin ClassesAxel Huebl1-0/+69
2020-04-15x.org: mirror mixinAxel Huebl2-0/+38
According to my nightly CI/CD tests, x.org is another large provider of software in common build chains that is often down. Added a hand-selected amount of mirrors that is well up-to-sync. Tested with `util-macros` that has a quite "recent" patch release. Other packages to follow in an individual PR.
2020-04-15`spack install` terminal output handling in foreground/background (#15723)Todd Gamblin7-224/+1155
Makes the following changes: * (Fixes #15620) tty configuration was failing when stdout was redirected. The implementation now creates a pseudo terminal for stdin and checks stdout properly, so redirections of stdin/out/err should be handled now. * Handles terminal configuration when the Spack process moves between the foreground and background (possibly multiple times) during a build. * Spack adjusts terminal settings to allow users to to enable/disable build process output to the terminal using a "v" toggle, abnormal exit cases (like CTRL-C) could leave the terminal in an unusable state. This is addressed here with a special-case handler which restores terminal settings. Significantly extend testing of process output logger: * New PseudoShell object for setting up a master and child process and configuring file descriptor inheritance between the two * Tests for "v" verbosity toggle making use of the added PseudoShell object * Added `uniq` function which takes a list of elements and replaces any consecutive sequence of duplicate elements with a single instance (e.g. "112211" -> "121") Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2020-04-14Package extensions: fixed a link in docs (#16040)Massimiliano Culpo3-7/+9
* Moved link to the right place in the docs * Fixed a few minor issues in extensions docs Fixed a typo, added a subsubsection for better navigation, reworded "modules in Python" as "Python packages"
2020-04-14sourceware.org: mirror urls (#15992)Axel Huebl2-0/+38
sourceware.org is often quite overrun and times out or results in certificate errors. Since libffi, bzip2, elfutils, etc. are quite fundamental in build chains, lets add some official mirrors. libffi, bzip2, elfutils, lvm2, valgrind: add mirrors
2020-04-12database: maintain in-memory consistency on remove (#15777)Massimiliano Culpo2-1/+21
The performance improvements done in #14693 where leaving the DB in an inconsistent state when specs were removed from it. This PR updates the DB internal state whenever the DB is written to a file. Note that we still cannot properly enumerate installed dependents, so there is a TODO in this code. Fixing that will require the dependents dictionaries in specs to be re-keyed (either by hash, or not keyed at all -- a list would do). See #11983 for details.
2020-04-10fujitsu compiler: Delete custom environment function. (#15988)Tomoki, Karatsu1-4/+0
2020-04-09NAG compiler fix: skip implicit RPATH detection (#15902)Peter Scheibel1-2/+6
* Skip collection of compiler link paths if compiler does not define a verbose flag * modules config bug: allow user to configure a compiler without an explicit entry for loaded modules
2020-04-07pipelines: Document dynamic generation limitationsScott Wittenburg1-1/+23
2020-04-07Add build number to Arm compiler version identification (#15809)LPettey-Arm2-2/+22
* Add capability for detecting build number for Arm compilers * Fixing fleck8 errors and updating test_arm_version_detection function for more detailed Arm compielr version detection * Ran flake8 locally and corrected errors * Altering Arm compielr version check to remove else clause and be more consistent with other compielr version checks. Added test case so both the 'if' and 'else' conditionals of the Arm compiler version check have a test case Co-authored-by: EC2 Default User <ec2-user@ip-172-31-7-135.us-east-2.compute.internal>
2020-04-06Docs: fix Spack Environments command (#15891)Adam J. Stewart1-1/+1