summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2020-11-09tutorial cmd: fix gpg invocation (#19829)Greg Becker1-1/+1
2020-11-09Fix minor typo in function comment (#19804)Adam J. Stewart1-1/+1
2020-11-09[docs] getting_started.rst: fix typo (#19815)Emir İşman1-1/+1
2020-11-09commands: add `spack tutorial` command (#19808)Todd Gamblin1-0/+85
Added a command to set up Spack for our tutorial at https://spack-tutorial.readthedocs.io. The command does some common operations we need first-time users to do. Specifically: - checks out a particular branch of Spack - deletes spurious configuration in `~/.spack` that might be left over from prior parts of the tutorial - adds a mirror and trusts its public key
2020-11-05Remove hardcoded version numbers from container logic (#19716)Greg Becker5-74/+7
Previously, we hardcoded a list of Spack versions which could be used by the containerize command. This PR removes that list. It's a maintenance burden when cutting a release, and prevents older versions of Spack from creating containers to be used by newer versions.
2020-11-04bug fix: Display error when curl is missing even in non-debug mode (#19695)Tamara Dahlgren2-2/+32
2020-11-03documentation: fix formatting of code-block section (#19693)Shahzeb Siddiqui1-1/+1
2020-11-02Bugfix - hashing: don't recompute full_hash or build_hash (#19672)Todd Gamblin1-3/+40
There was an error introduced in #19209 where `full_hash()` and `build_hash()` are called on older specs that we've read in from the DB; older specs may not be able to compute these hashes (e.g. if they have removed patches used in computing the full_hash). When serializing a Spec, we want to generate the full/build hash when possible, but we need a mechanism to skip it for Specs that have themselves been read from YAML (and may not support this). To get around this ambiguity and to fix the issue, we: - Add an attribute to the spec called `_hashes_final`, that is `True` if we can't lazily compute `build_hash` and `full_hash`. - Set `_hashes_final` to `False` for new specs (i.e., lazily computing hashes is ok) - Set `_hashes_final` to `True` for concrete specs read in via `from_node_dict`, as it may be too late to recompute hashes. - Compute and write out all hashes in `node_dict_with_hashes` *if possible*. Effectively what this means is that we can round-trip specs that are missing `_build_hash` and `_full_hash` without recomputing them, but for all new specs, we'll compute them and store them. So Spack should work fine with old DBs now.
2020-11-01sbang: fixes for sbang relocationTodd Gamblin3-9/+77
This fixes sbang relocation when using old binary packages, and updates code in `relocate.py`. There are really two places where we would want to handle an `sbang` relocation: 1. Installing an old package that uses `sbang` with shebang lines like `#!/bin/bash $spack_prefix/sbang` 2. Installing a *new* package that uses `sbang` with shebang lines like `#!/bin/sh $install_tree/sbang` The second case is actually handled automatically by our text relocation; we don't need any special relocation logic for new shebangs, as our relocation logic already changes references to the build-time `install_tree` to point to the `install_tree` at intall-time. Case 1 was not properly handled -- we would not take an old binary package and point its shebangs at the new `sbang` location. This PR fixes that and updates the code in `relocation.py` with some notes. There is one more case we don't currently handle: if a binary package is created from an installation in a short prefix that does *not* need `sbang` and is installed to a long prefix that *does* need `sbang`, we won't do anything. We should just patch the file as we would for a normal install. In some upcoming PR we should probably change *all* `sbang` relocation logic to be idempotent and to apply to any sort of shebang'd file. Then we'd only have to worry about which files to `sbang`-ify at install time and wouldn't need to care about these special cases.
2020-10-30Update documentation on containers (#19631)Massimiliano Culpo4-207/+61
fixes #15183 - Moved the container related content from workflows.rst into containers.rst - Deleted the docker_for_developers.rst file, since it describes an outdated procedure Co-authored-by: Axel Huebl <a.huebl@hzdr.de> Co-authored-by: Omar Padron <omar.padron@kitware.com>
2020-10-30Config: cache results of get_config (#19605)Massimiliano Culpo4-12/+49
`config.get_config` now caches the results and returns the same configuration if called multiple times with the same arguments (i.e. the same section and scope). As a consequence, it is expected that users will always call update methods provided in the `config` module after changing the configuration (even if manipulating it as a Python nested dictionary). The following two examples should cover most scenarios: * Most configuration update logic in the core (e.g. relating to adding new compiler) should call `Configuration.update_config` * Tests that need to change the global configuration should use the newly-provided `config.replace_config` function. (if neither of these methods apply, then the essential requirement is to use a method marked as `_config_mutator`) Failure to call such a function after modifying the configuration will lead to unexpected results (e.g. calling `get_config` after changing the configuration will not reflect the changes since the first call to get_config).
2020-10-30Make archspec a vendored dependency (#19600)Massimiliano Culpo50-1565/+1887
- Added archspec to the list of vendored dependencies - Removed every reference to llnl.util.cpu - Removed tests from Spack code base
2020-10-30Binary caching: use full hashes (#19209)Scott Wittenburg16-158/+795
* "spack install" now has a "--require-full-hash-match" option, which forces Spack to skip an available binary package when the full hash doesn't match. Normally only a DAG-hash match is required, which ensures equivalent Specs, but does not account for changing logic inside the associated package. * Add a local binary cache index which tracks specs that have a binary install available in a remote binary cache. It is updated with "spack buildcache list" or for a given spec when a binary package is retrieved for that Spec.
2020-10-29CI: disable vermin check for deprecated hash (#19612)Peter Scheibel1-1/+2
Spack has a fallback for hash checking with m55sums that may not be supported in earlier versions of Python 3.x. The comments in the Spack code acknowledge that this is best effort and may fail, but recent vermin checks (running as part of our CI) reject this. This disables vermin checks for that fallback.
2020-10-29Oneapi add compiler (#19330)Frank Willmore7-3/+164
* enable flatcc to be built with gcc/9.X.X * add static option for building libyogrt * cleanup * Initial working version * rework new oneapi wrappers * tested and removed my initials from source * cleanup * Update __init__.py * remove whitespace * working now with mods for testing, detection. Detection for oneapi is working, but entry needs to be modified to add link path for libimf.so. Cleared cruft for old Intel versions * fixed some formatting * cleanup * flake8 cleanup * flake8 * fixed syntax of compiler version detection tests * fixed syntax of compiler version detection tests modified: detection.py * fix typo * fixes for compilers tests * remove erroneous tests for outdated -std= flags, remove ifx version check (output won't parse) Co-authored-by: Frank Willmore <willmore@anl.gov>
2020-10-28sbang: vendor sbangTodd Gamblin2-66/+0
`sbang` now lives at https://github.com/spack/sbang, and it has its own test suite that's more extensive than what's in Spack. We'll leave sbang tests to sbang from now on, and just vendor `bin/sbang` directly. Remaining `sbang` tests have to do with patching files, not with `sbang`'s functionality. This update also fixes a bug with `sbang` and multiple command line arguments that was introduced in #19529. See: * https://github.com/spack/sbang/pull/1 * https://github.com/spack/sbang/pull/2 - [x] include latest `sbang` from https://github.com/spack/sbang - [x] remove old `sbang` tests from Spack - [x] update `COPYRIGHT` and `cmd/license.py`
2020-10-27sbang: convert sbang script to POSIX shellTodd Gamblin2-7/+75
`sbang` was previously a bash script but did not need to be. This converts it to a plain old POSIX shell script and adds some options. This also allows us to simplify sbang shebangs to `#!/bin/sh /path/to/sbang` instead of `#!/bin/bash /path/to/sbang`. The new script passes shellcheck (with a few exceptions noted in the file) - [x] `SBANG_DEBUG` env var enables printing what *would* be executed - [x] `sbang` checks whether it has been passed an option and fails gracefully - [x] `sbang` will now fail if it can't find a second shebang line, or if the second line happens to be sbang (avoid infinite loops) - [x] add more rigorous tests for `sbang` behavior using `SBANG_DEBUG`
2020-10-26sbang: add support for php (#18299)Toyohisa Kameyama2-0/+22
PHP supports an initial shebang, but its comment syntax can't handle our 2-line shebangs. So, we need to embed the 2nd-line shebang comment to look like a PHP comment: <?php #!/path/to/php ?> This adds patching support to the sbang hook and support for instrumenting php shebangs. This also patches `phar`, which is a tool used to create php packages. `phar` itself has to add sbangs to those packages (as phar archives apparently contain UTF-8, as well as binary blobs), and `phar` sets a checksum based on the contents of the package. Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2020-10-26sbang: put sbang in the install_tree (#11598)Patrick Gartung3-15/+88
`sbang` is not always accessible to users of packages, e.g., if Spack is installed in someone's home directory and they deploy software for others. Avoid this by: 1. Always installing the `sbang` script in the `install_tree` 2. Relocating binaries to point to the copy in the `install_tree` and not the one in the Spack installation. This PR also: - ensures that `sbang` is reinstalled if it is modified in Spack - adds tests - updates the way `gobject-introspection` patches Makefiles to support `sbang` Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2020-10-26bugfix: test_push_and_fetch_keys should be skipped w/o gpg (#19511)Todd Gamblin1-0/+2
- [x] add a `@pytest.skipif` decorator
2020-10-24bugfix: fix config merge order for OrderdDicts (#18482)Todd Gamblin5-30/+67
The logic in `config.py` merges lists correctly so that list elements from higher-precedence config files come first, but the way we merge `dict` elements reverses the precedence. Since `mirrors.yaml` relies on `OrderedDict` for precedence, this bug causes mirrors in lower-precedence config scopes to be checked before higher-precedence scopes. We should probably convert `mirrors.yaml` to use a list at some point, but in the meantie here's a fix for `OrderedDict`. - [x] ensuring that keys are ordered correctly in `OrderedDict` by re-inserting keys from the destination `dict` after adding the keys from the source `dict`. - [x] also simplify the logic in `merge_yaml` by always reinserting common keys -- this preserves mark information without all the special cases, and makes it simpler to preserve insertion order. Assuming a default spack configuration, if we run this: ```console $ spack mirror add foo https://bar.com ``` Results before this change: ```console $ spack config blame mirrors --- mirrors: /Users/gamblin2/src/spack/etc/spack/defaults/mirrors.yaml:2 spack-public: https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/ /Users/gamblin2/.spack/mirrors.yaml:2 foo: https://bar.com ``` Results after: ```console $ spack config blame mirrors --- mirrors: /Users/gamblin2/.spack/mirrors.yaml:2 foo: https://bar.com /Users/gamblin2/src/spack/etc/spack/defaults/mirrors.yaml:2 spack-public: https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/ ```
2020-10-23docs: update docs on shell support and using packages (#19486)Todd Gamblin3-337/+344
Shell integration no longer requires setting `SPACK_ROOT`, so we can simplify the documentation on it. The docs on shell support and using packages are getting a bit old, and information on `spack load` (which seems to be everyone's most common way of using packages) is hard to find. This PR simplifies the shell documentation to remove SPACK_ROOT, and also moves some sections around for clearer organization. - [x] make docs on sourcing setup scripts clearer and simpler - [x] introduce `spack load` early in the basic usage guide instead of burying it in the module docs - [x] clean up module docs so that spack module tcl loads comes later - [x] be clear about the different ways to use packages so that the users can find the docs better. Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2020-10-23csh: don't require SPACK_ROOT for sourcing setup-env.csh (#18225)Todd Gamblin8-80/+69
Don't require SPACK_ROOT for sourcing setup-env.csh and make output more consistent
2020-10-22Add scratch module roots to test configuration (#19477)Massimiliano Culpo4-82/+23
fixes #19476 Module file content is written to file in a temporary location and read back to be analyzed by unit tests. The approach to patch "open" and write to a StringIO in memory has been abandoned, since over time other operations insisting on the filesystem have been added to the module file generator.
2020-10-21tests: increase tolerance of termios tests (#19456)Todd Gamblin1-2/+6
Synchronization on GitHub macOS runners seems to be very slow, and frequently the foreground/background tests fail due to the race this causes. This increases the tolerance for slowness a bit more, to allow up to 4 spurious output lines in the tests. This should hopefully result in no more false negatives on these tests for macOS on GitHub.
2020-10-21Added _poll_lock exception tests (#19446)Tamara Dahlgren1-0/+26
2020-10-21Add qgraf (#19404)iarspider1-1/+4
* Add recipe for qgraf * Revert "Add recipe for qgraf" This reverts commit 76783f73867a32b4a96e980e31a433ed3c0037fd. * Add qgraf * Update package.py Changes from review * Changes from MR * Fix for URLs containing @ symbol Co-authored-by: Ivan Razumov <ivan.razumov@cern.ch> Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2020-10-21Skip malformed spec strings when searching for externals (#19438)Massimiliano Culpo1-8/+13
fixes #19266 fzf search method has also been updated Co-authored-by: Tom Scogland <tom.scogland@gmail.com>
2020-10-21CMakePackage: added 'ipo' variant (#18374)Omri Mor1-1/+17
+ipo sets CMAKE_INTERPROCEDURAL_OPTIMIZATION=ON The option is not supported for CMake < 3.9
2020-10-21Make url_fetch test independent of locale settings (#19390)Massimiliano Culpo1-1/+1
2020-10-21Change 'any' to wildcard for variants (#19381)Greg Becker7-103/+49
2020-10-20py-archspec: added new package at v0.1.1 (#19395)Massimiliano Culpo1-2/+0
2020-10-20Adding AOCC compiler to SPACK community (#19345)GaneshPrasadMA9-3/+162
* Adding AOCC compiler to SPACK community The AOCC compiler system offers a high level of advanced optimizations, multi-threading and processor support that includes global optimization, vectorization, inter-procedural analyses, loop transformations, and code generation. AMD also provides highly optimized libraries, which extract the optimal performance from each x86 processor core when utilized. The AOCC Compiler Suite simplifies and accelerates development and tuning for x86 applications. * Added unit tests for detection and flags for AOCC * Addressed reviewers comments w.r.t version checks and url,checksum related line lengths Co-authored-by: Test User <spack@example.com>
2020-10-20Fix python scripts relying on external python in env (#19241)Greg Becker1-1/+3
2020-10-19Fix "buildcache update-index --keys ..." when mirror is S3 (#19141)Scott Wittenburg1-5/+14
2020-10-18filter_compiler_wrappers: a fix for NAG (#17133)Sergey Kosukhin1-1/+7
2020-10-18Add testing option to dev-build command (#17293)elsagermann1-0/+14
* ADD: testing to dev-build command * RM: mutally exclusive group for testing in parser * FIX: test option to subparser and not testing * ADD: spack-completion.bash * RM: local devbuildcosmo cmd * FIX: bad merge --drop-in -b --before options forgotten * FIX: --test place in spack-completion.bash * FIX: typo * FIX: blank line removing * FIX: trailing white space Co-authored-by: Elsa Germann <egermann@tsa-ln002.cm.cluster>
2020-10-18[docs] Pkg list: current version, not latest release (#18213)Wouter Deconinck1-2/+2
The package list at https://spack.readthedocs.io/en/latest/package_list.html claims "it is automatically generated based on the packages in the latest Spack release" but it is actually based on the develop branch. This leads to confusion when users find that e.g. herwigpp is included in the list, but it cannot be found when they install the latest release. That latest release has a package list at https://spack.readthedocs.io/en/stable/package_list.html which does indeed not include herwigpp. Changing the language from "the latest Spack release" to "this Spack version" might make that clearer. Maybe.
2020-10-18Fix for buildcache -o (#19354)Jason Miller1-1/+2
* Fix for buildcache -o The method has more positional arguments than the caller expects. * Address length issue. Fix pylint/flake errors.
2020-10-17Spec: fix multiple generator iteration in satisfies_dependencies (#18527) ↵Omri Mor2-2/+10
(#18559)
2020-10-16New compiler: nvhpc (NVIDIA HPC SDK) (#19294)Scott McMillan7-3/+168
* Add nvhpc compiler definition: "spack compiler add" will now look for instances of the NVIDIA HPC SDK compiler executables (nvc, nvc++, nvfortran) in supplied paths * Add the nvhpc package which installs the nvhpc compiler * Add testing for nvhpc detection and C++-standard/pic flags Co-authored-by: Scott McMillan <smcmillan@nvidia.com>
2020-10-16spack external find: fix debug output (#19342)iarspider1-2/+4
Output was, e.g. `Executables in /bin and /,u,s,r,/,b,i,n are both associated with the same spec xz@5.2.2`, will be `Executables in /bin and /usr/bin are both associated with the same spec xz@5.2.2`.
2020-10-16autotools: recursively patch config.guess and config.sub (#18347)Toyohisa Kameyama1-68/+57
Previously config.guess and config.sub were patched only in the root of the source path. This modification extend the previous behavior to patch every config.guess or config.sub file even in subfolders, if need be. Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2020-10-15Environments: specify packages for developer builds (#15256)Greg Becker24-84/+1058
* allow environments to specify dev-build packages * spack develop and spack undevelop commands * never pull dev-build packges from bincache * reinstall dev_specs when code has changed; reinstall dependents too * preserve dev info paths and versions in concretization as special variant * move install overwrite transaction into installer * move dev-build argument handling to package.do_install now that specs are dev-aware, package.do_install can add necessary args (keep_stage=True, use_cache=False) to dev builds. This simplifies driving logic in cmd and env._install * allow 'any' as wildcard for variants * spec: allow anonymous dependencies raise an error when constraining by or normalizing an anonymous dep refactor concretize_develop to remove dev_build variant refactor tests to check for ^dev_path=any instead of +dev_build * fix variant class hierarchy
2020-10-14autotools.py: fix the list of objects to be removed from libtool (Fujitsu). ↵Tomoki, Karatsu1-1/+1
(#19303)
2020-10-13autotools: add attribute to delete libtool archives .la files (#18850)Massimiliano Culpo2-8/+67
* autotools: add attribute to delete libtool archives .la files According to Autotools Mythbuster (https://autotools.io/libtool/lafiles.html) libtool archive files are mostly vestigial, but they might create issues when relocating binary packages as shown in #18694. For GCC specifically, most distributions remove these files with explicit commands: https://git.stg.centos.org/rpms/gcc/blob/master/f/gcc.spec#_1303 Considered all of that, this commit adds an easy way for each AutotoolsPackage to remove every .la file that has been installed. The default, for the time being, is to maintain them - to be consistent with what Spack was doing previously. * autotools: delete libtool archive files by default Following review this commit changes the default for libtool archive files deletion and adds test to verify the behavior.
2020-10-12autotools: refactor search paths for aclocal in its own method (#19258)Massimiliano Culpo1-5/+10
This commit refactors the computation of the search path for aclocal in its own method, so that it's easier to reuse for packages that need to have a custom autoreconf phase. Co-authored-by: Toyohisa Kameyama <kameyama@riken.jp>
2020-10-12autotools.py: removed some options from libtool only for Fujitsu. (#19217)Tomoki, Karatsu1-1/+6
2020-10-11Add testing for Python 3.9 (#19261)Adam J. Stewart1-5/+5
2020-10-11CUDA: update maintainers (#19262)Adam J. Stewart1-1/+1