summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2020-04-16SourceForge: Mirror Mixin (#16112)Axel Huebl3-0/+64
* SourceForge: Mirror Mixin Add a mixing class for direct `CNAME`s to sourceforge mirrors. Since the main gateway servers are often down, this could reduce timeouts and fetch errors for sourceforge.net hosted software. * SourceForge: unspectacular mirror replacement add mirrors to all sourceforge packages with trivial download logic. tested fetch of latest version of each of these packages with various mirrors before committing. * SourceForge: xz the author homepage is chronocially overrun and this is the offical upload with many mirrors.
2020-04-16macos: use DYLD_FALLBACK_LIBRARY_PATH instead of DYLD_LIBRARY_PATHTodd Gamblin3-2/+6
`DYLD_LIBRARY_PATH` can frequently break builtin macOS software when pointed at Spack libraries. This is because it takes *higher* precedence than the default library search paths, which are used by system software. `DYLD_FALLBACK_LIBRARY_PATH`, on the other hand, takes lower precedence. At first glance, this might seem bad, because the software installed by Spack in an environment needs to find *its* libraries, and it should not use the defaults. However, Spack's isntallations are always `RPATH`'d, so they do not have this problem. `DYLD_FALLBACK_LIBRARY_PATH` is thus useful for things built in an environment that need to use Spack's libraries, that don't set *their* RPATHs correctly for whatever reason. We now prefer it to `DYLD_LIBRARY_PATH` in modules and in environments because it helps a little bit, and it is much less intrusive.
2020-04-16Bugfix: For spack buildcache check, Handle specs where only the hash is (#15663)Marc Allen1-1/+1
provided (#15662). Prior to this fix, the checked Spec object would not be populated, and concretization would fail. Co-authored-by: Marc Allen <mrcall@amazon.com>
2020-04-15Merge branch 'releases/v0.14' into developTodd Gamblin1-1/+1
2020-04-15Make tags case insensitiveMichael Kuhn2-0/+10
Currently, tags are case sensitive, which is unintuitive: ```console $ spack list -t hpc ==> 2 packages. nek5000 nektools $ spack list -t HPC ==> 1 packages. mfem $ spack list -t Hpc ==> 0 packages. ``` This change makes them case insensitive: ```console $ spack list -t hpc ==> 3 packages. mfem nek5000 nektools $ spack list -t HPC ==> 3 packages. mfem nek5000 nektools $ spack list -t Hpc ==> 3 packages. mfem nek5000 nektools ```
2020-04-15version bump: 0.14.2Todd Gamblin1-1/+1
2020-04-15bugfix: spack test should not output [+] for mock installs (#15609)Todd Gamblin1-3/+2
`spack test` has a spurious '[+] ' in the output: ``` lib/spack/spack/test/install.py .........[+] ...... ``` Output is properly suppressed: ``` lib/spack/spack/test/install.py ............... ```
2020-04-15`spack install` terminal output handling in foreground/background (#15723)Todd Gamblin7-224/+1155
Makes the following changes: * (Fixes #15620) tty configuration was failing when stdout was redirected. The implementation now creates a pseudo terminal for stdin and checks stdout properly, so redirections of stdin/out/err should be handled now. * Handles terminal configuration when the Spack process moves between the foreground and background (possibly multiple times) during a build. * Spack adjusts terminal settings to allow users to to enable/disable build process output to the terminal using a "v" toggle, abnormal exit cases (like CTRL-C) could leave the terminal in an unusable state. This is addressed here with a special-case handler which restores terminal settings. Significantly extend testing of process output logger: * New PseudoShell object for setting up a master and child process and configuring file descriptor inheritance between the two * Tests for "v" verbosity toggle making use of the added PseudoShell object * Added `uniq` function which takes a list of elements and replaces any consecutive sequence of duplicate elements with a single instance (e.g. "112211" -> "121") Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2020-04-15database: maintain in-memory consistency on remove (#15777)Massimiliano Culpo2-1/+21
The performance improvements done in #14693 where leaving the DB in an inconsistent state when specs were removed from it. This PR updates the DB internal state whenever the DB is written to a file. Note that we still cannot properly enumerate installed dependents, so there is a TODO in this code. Fixing that will require the dependents dictionaries in specs to be re-keyed (either by hash, or not keyed at all -- a list would do). See #11983 for details.
2020-04-15performance: add a verification file to the database (#14693)Andrew W Elble3-3/+52
Reading the database repeatedly can be quite slow. We need a way to speed up Spack when it reads the DB multiple times, but the DB has not been modified between reads (which is nearly all the time). - [x] Add a file containing a unique uuid that is regenerated at database write time. Use this uuid to suppress re-parsing the database contents if we know a previous uuid and the uuid has not changed. - [x] Fix mutable_database fixture so that it resets the last seen verifier when it resets. - [x] Enable not rereading the database immediately after a write. Make the tests reset the last seen verifier in between tests that use the database fixture. - [x] make presence of uuid module optional
2020-04-15Remove DB conversion of old index.yaml (#15298)Peter Scheibel1-84/+45
Removed the code that was converting the old index.yaml format into index.json. Since the change happened in #2189 it should be considered safe to drop this (untested) code.
2020-04-15Environments/views: only override spec prefix for non-external packages (#15475)Peter Scheibel2-1/+49
* only override spec prefix for non-external packages * add test that environment shell modifications respect explicitly-specified prefixes for external packages * add clarifying comment
2020-04-15Source devnull in environment_after_sourcing_files (closes #15775) (#15791)Jonathon Anderson1-1/+2
spack.util.environment_after_sourcing_files compares the local environment against a shell environment after having sourced a file; but this ends up including the default shell profile and rc, which might differ from the local environment. To change this, compare against the default shell environment, expressed here as 'source /dev/null'.
2020-04-15bugfix: TERM may not be in the environment on Cray (#15630)Todd Gamblin1-3/+7
2020-04-15Blacklist Lmod variable modifications when sourcing files (#15778)Massimiliano Culpo3-1/+24
fixes #15775 Add all the variables listed here: https://lmod.readthedocs.io/en/latest/090_configuring_lmod.html to the list of those blacklisted when constructing environment modifications by sourcing files.
2020-04-15Tests: Mirror Mixin ClassesAxel Huebl1-0/+69
2020-04-15x.org: mirror mixinAxel Huebl2-0/+38
According to my nightly CI/CD tests, x.org is another large provider of software in common build chains that is often down. Added a hand-selected amount of mirrors that is well up-to-sync. Tested with `util-macros` that has a quite "recent" patch release. Other packages to follow in an individual PR.
2020-04-15`spack install` terminal output handling in foreground/background (#15723)Todd Gamblin7-224/+1155
Makes the following changes: * (Fixes #15620) tty configuration was failing when stdout was redirected. The implementation now creates a pseudo terminal for stdin and checks stdout properly, so redirections of stdin/out/err should be handled now. * Handles terminal configuration when the Spack process moves between the foreground and background (possibly multiple times) during a build. * Spack adjusts terminal settings to allow users to to enable/disable build process output to the terminal using a "v" toggle, abnormal exit cases (like CTRL-C) could leave the terminal in an unusable state. This is addressed here with a special-case handler which restores terminal settings. Significantly extend testing of process output logger: * New PseudoShell object for setting up a master and child process and configuring file descriptor inheritance between the two * Tests for "v" verbosity toggle making use of the added PseudoShell object * Added `uniq` function which takes a list of elements and replaces any consecutive sequence of duplicate elements with a single instance (e.g. "112211" -> "121") Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2020-04-14Package extensions: fixed a link in docs (#16040)Massimiliano Culpo3-7/+9
* Moved link to the right place in the docs * Fixed a few minor issues in extensions docs Fixed a typo, added a subsubsection for better navigation, reworded "modules in Python" as "Python packages"
2020-04-14sourceware.org: mirror urls (#15992)Axel Huebl2-0/+38
sourceware.org is often quite overrun and times out or results in certificate errors. Since libffi, bzip2, elfutils, etc. are quite fundamental in build chains, lets add some official mirrors. libffi, bzip2, elfutils, lvm2, valgrind: add mirrors
2020-04-12database: maintain in-memory consistency on remove (#15777)Massimiliano Culpo2-1/+21
The performance improvements done in #14693 where leaving the DB in an inconsistent state when specs were removed from it. This PR updates the DB internal state whenever the DB is written to a file. Note that we still cannot properly enumerate installed dependents, so there is a TODO in this code. Fixing that will require the dependents dictionaries in specs to be re-keyed (either by hash, or not keyed at all -- a list would do). See #11983 for details.
2020-04-10fujitsu compiler: Delete custom environment function. (#15988)Tomoki, Karatsu1-4/+0
2020-04-09NAG compiler fix: skip implicit RPATH detection (#15902)Peter Scheibel1-2/+6
* Skip collection of compiler link paths if compiler does not define a verbose flag * modules config bug: allow user to configure a compiler without an explicit entry for loaded modules
2020-04-07pipelines: Document dynamic generation limitationsScott Wittenburg1-1/+23
2020-04-07Add build number to Arm compiler version identification (#15809)LPettey-Arm2-2/+22
* Add capability for detecting build number for Arm compilers * Fixing fleck8 errors and updating test_arm_version_detection function for more detailed Arm compielr version detection * Ran flake8 locally and corrected errors * Altering Arm compielr version check to remove else clause and be more consistent with other compielr version checks. Added test case so both the 'if' and 'else' conditionals of the Arm compiler version check have a test case Co-authored-by: EC2 Default User <ec2-user@ip-172-31-7-135.us-east-2.compute.internal>
2020-04-06Docs: fix Spack Environments command (#15891)Adam J. Stewart1-1/+1
2020-04-06Blacklist Lmod variable modifications when sourcing files (#15778)Massimiliano Culpo3-1/+24
fixes #15775 Add all the variables listed here: https://lmod.readthedocs.io/en/latest/090_configuring_lmod.html to the list of those blacklisted when constructing environment modifications by sourcing files.
2020-04-06Source devnull in environment_after_sourcing_files (closes #15775) (#15791)Jonathon Anderson1-1/+2
spack.util.environment_after_sourcing_files compares the local environment against a shell environment after having sourced a file; but this ends up including the default shell profile and rc, which might differ from the local environment. To change this, compare against the default shell environment, expressed here as 'source /dev/null'.
2020-04-05verify `other.exe` attr exists before use in __eq__ (#15882)Stephen McDowell1-1/+1
2020-04-03Add option "--first" for "spack load" (#15622)G-Ragghianti4-7/+34
* Implemented --first option for "spack load" * added test for "spack load --first" Co-authored-by: gragghia <gragghia@localhost.localdomain>
2020-04-03Environments/views: only override spec prefix for non-external packages (#15475)Peter Scheibel2-1/+49
* only override spec prefix for non-external packages * add test that environment shell modifications respect explicitly-specified prefixes for external packages * add clarifying comment
2020-04-02Add commands to facilitate Spack/Python/OS reporting (#15834)Adam J. Stewart4-2/+46
* Add --version arg to spack python command * Add `spack debug report` command
2020-04-01Spelling fixes (#15805)Greg Sjaardema7-13/+13
* Spelling fixes * spelling fixes * spelling fixes * spelling fixes * spelling fix * spelling fix * spelling fix * spelling fixes * spelling fix
2020-03-31packages.yaml: allow virtuals to specify buildable: false (#14934)Greg Becker9-153/+155
Currently, to force Spack to use an external MPI, you have to specify `buildable: False` for every MPI provider in Spack in your packages.yaml file. This is both tedious and fragile, as new MPI providers can be added and break your workflow when you do a git pull. This PR allows you to specify an entire virtual dependency as non-buildable, and specify particular implementations to be built: ``` packages: all: providers: mpi: [mpich] mpi: buildable: false paths: mpich@3.2 %gcc@7.3.0: /usr/packages/mpich-3.2-gcc-7.3.0 ``` will force all Spack builds to use the specified `mpich` install.
2020-03-31Update link for codecov's browser extensions (#15779)Massimiliano Culpo1-2/+2
2020-03-30CUDA 10.1+ supports ICC 19.0 (#15764)Axel Huebl1-2/+2
Fix the CUDA-ICC conflict check. Simplify the XL 17 check.
2020-03-28Buildcache keys command speed up (#15727)Patrick Gartung1-1/+1
* Limit the spidering to depth=0 for keys * depth=0 is default argument
2020-03-25provider index: removed import from + refactored a few parts (#15570)Massimiliano Culpo2-130/+179
Removed provider_index use of 'import from' and refactored a few routines to a further subclassing of _IndexBase for implementing user defined bindings of provider specs.
2020-03-24spack.repo: remove "import from" statements (#15505)Massimiliano Culpo3-39/+71
spack.repo: remove "import from" statements
2020-03-24spack.relocate: add unit test (#15610)Massimiliano Culpo2-67/+130
* relocate: removed import from statements * relocate: renamed *Exception to *Error This aims at consistency in naming with both the standard library (ValueError, AttributeError, etc.) and other errors in 'spack.error'. Improved existing docstrings * relocate: simplified search function by un-nesting conditionals The search function that searches for patchelf has been refactored to remove deeply nested conditionals. Extended docstring. * relocate: removed a condition specific to unit tests * relocate: added test for _patchelf
2020-03-23Bugfix: remove unit tests that install real packages (#15646)Peter Scheibel1-480/+0
Our unit tests run many times. Any unit test which actually installs a package (which involves fetching code on the internet) is a severe bug because it runs an installation many times (i.e. re-downloading the same package for each version of Python that we run unit tests for). This reverts commit 25893f1, which added tests that install real packages.
2020-03-23Vendoring: remove dependency on Setuptools from vendored pytest (#15612)Todd Gamblin3-51/+17
If the Python used by Spack does not include Setuptools, then 'spack test' will fail because Spack's vendored pytest dependency imports and uses Setuptools in some of its functions. It turns out that Spack doesn't use the functionality those methods enable, so this PR removes those functions and thereby allows 'spack test' to run without Setuptools.
2020-03-23Bugfix (config): enable "spack test" when in an active environment (#15618)Todd Gamblin1-4/+9
For any Spack test using Spack's YAML configuration, avoid using real Spack configuration that has been cached by other tests and Spack startup logic. Previously this was only done for tests using 'mutable_config' (i.e. those which expected to *change* the configuration of Spack), but in fact all tests that read Spack config should use it. This was an issue when running tests within an environment, because compiler configuration ends up being queried earlier, and the user's real config "leaks" into the cache. Outside an environment, the cache is never set until tests touch it, so we weren't seeing this issue.
2020-03-22bugfix: TERM may not be in the environment on Cray (#15630)Todd Gamblin1-3/+7
2020-03-22bugfix: spack test should not output [+] for mock installs (#15609)Todd Gamblin1-3/+2
`spack test` has a spurious '[+] ' in the output: ``` lib/spack/spack/test/install.py .........[+] ...... ``` Output is properly suppressed: ``` lib/spack/spack/test/install.py ............... ```
2020-03-21Buildcache: attempt to build and install buildcaches in test environment ↵Patrick Gartung1-0/+480
(#15577) * Put bindist.py on it's own branch * Working on SL7 with python 3.6 * Flake8 * Put back clearing of compiler cache
2020-03-20Merge branch 'releases/v0.14' into developTodd Gamblin1-1/+1
2020-03-20performance: add a verification file to the database (#14693)Andrew W Elble3-3/+52
Reading the database repeatedly can be quite slow. We need a way to speed up Spack when it reads the DB multiple times, but the DB has not been modified between reads (which is nearly all the time). - [x] Add a file containing a unique uuid that is regenerated at database write time. Use this uuid to suppress re-parsing the database contents if we know a previous uuid and the uuid has not changed. - [x] Fix mutable_database fixture so that it resets the last seen verifier when it resets. - [x] Enable not rereading the database immediately after a write. Make the tests reset the last seen verifier in between tests that use the database fixture. - [x] make presence of uuid module optional
2020-03-20Buildcache: add unit tests for normalized path functions in relocate.py (#15607)Patrick Gartung1-5/+57
2020-03-20fix: change typo dependnecies to dependencies (#15602)Ryan Mast2-2/+2