summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2021-10-05Use gnuconfig package for config file replacement for RISC-V. (#26364)Kevin Pedretti5-10/+86
* Use gnuconfig package for config file replacement for RISC-V. This extends the changes in #26035 to handle RISC-V. Before this change, many packages fail to configure on riscv64 due to config.guess being too old to know about RISC-V. This is seen out of the box when clingo fails to build from source due to pkgconfig failing to configure, throwing error: "configure: error: cannot guess build type; you must specify one". * Add riscv64 architecture * Update vendored archspec from upstream project. These archspec updates include changes needed to support riscv64. * Update archspec's __init__.py to reflect the commit hash of archspec being used.
2021-10-05Move shell aware env into spack.environment.shell (#25608)Harmen Stoppels12-218/+346
Cherry-picked from #25564 so this is standalone. With this PR we can activate an environment in Spack itself, without computing changes to environment variables only necessary for "shell aware" env activation. 1. Activating an environment: ```python spack.environment.activate(Environment(xyz)) -> None ``` this basically just sets `_active_environment` and modifies some config scopes. 2. Activating an environment **and** getting environment variable modifications for the shell: ```python spack.environment.shell.activate(Environment(xyz)) -> EnvironmentModifications ``` This should make it easier/faster to do unit tests and scripting with spack, without the cli interface.
2021-10-05Isolate bootstrap configuration from user configuration (#26071)Massimiliano Culpo5-27/+96
* Isolate bootstrap configuration from user configuration * Search for build dependencies automatically if bootstrapping from sources The bootstrapping logic will search for build dependencies automatically if bootstrapping anything form sources. Any external spec, if found, is written in a scope that is specific to bootstrapping. * Don't clean the bootstrap store with "spack clean -a" * Copy bootstrap.yaml and config.yaml in the bootstrap area
2021-10-04cc: make error messages more clearTodd Gamblin1-6/+7
- [x] Our wrapper error messages are sometimes hard to differentiate from other build output, so prefix all errors from `die()` with '[spack cc] ERROR:' - [x] The error we raise when running, say, `fc` without a Fortran compiler was not clear enough. Clarify the message and the comment.
2021-10-04cc: convert compiler wrapper to posix shellTodd Gamblin2-292/+434
This converts everything in cc to POSIX sh, except for the parts currently handled with bash arrays. Tests are still passing. This version tries to be as straightforward as possible. Specifically, most conversions are kept simple -- convert ifs to ifs, handle indirect expansion the way we do in `setup-env.sh`, only mess with the logic in `cc`, and don't mess with the python code at all. The big refactor is for arrays. We can't rely on bash's nice arrays and be ignorant of separators anymore. So: 1. To avoid complicated separator logic, there are three types of lists. They are: * `$lsep`-separated lists, which end with `_list`. `lsep` is customizable, but we picked `^G` (alarm bell) for `$lsep` because it's ASCII and it's unlikely that it would actually appear in any arguments. If we need to get fancier (and I will lose faith in the world if we do) then we could consider XON or XOFF. * `:`-separated directory lists, which end with `_dirs`, `_DIRS`, `PATH`, or `PATHS` * Whitespace-separated lists (like flags), which can have any other name. Whitespace and colon-separated lists come with the territory with PATHs from env vars and lists of flags. `^G` separated lists are what we use for most internal variables, b/c it's more likely to work. 2. To avoid subshells, use a bunch of functions that do dirty `eval` stuff instead. This adds 3 functions to deal with lists: * `append LISTNAME ELEMENT [SEP]` will put `ELEMENT` at the end of the list called `LISTNAME`. You can optionally say what separator you expect to use. Note that we are taking advantage of everything being global and passing lists by name. * `prepend LISTNAME ELEMENT [SEP]` like append, but puts `ELEMENT` at the start of `LISTNAME` * `extend LISTNAME1 LISTNAME2 [PREFIX]` appends everything in LISTNAME2 to LISTNAME1, and optionally prepends `PREFIX` to every element (this is useful for things like `-I`, `-isystem `, etc. * `preextend LISTNAME1 LISTNAME2 [PREFIX]` prepends everything in LISTNAME2 to LISTNAME1 in order, and optionally prepends `PREFIX` to every element. The routines determine the separator for each argument by its name, so we don't have to pass around separators everywhere. Amazingly, as long as you do not expand variables' values within an `eval` environment, you can do all this and still preserve quoting. When iterating over lists, the user of this API still has to set and unset `IFS` properly. We ended up having to ignore shellcheck SC2034 (unused variable), because using evals all over the place means that shellcheck doesn't notice that our list variables are actually used. So far this is looking pretty good. I took the most complex unit test I could find (which runs a sample link line) and ran the same command line 200 times in a shell script. Times are roughly as follows: For this invocation: ```console $ bash -c 'time (for i in `seq 1 200`; do ~/test_cc.sh > /dev/null; done)' ``` I get the following performance numbers (the listed shells are what I put in `cc`'s shebang): **Original** * Old version of `cc` with arrays and `bash v3.2.57` (macOS builtin): `4.462s` (`.022s` / call) * Old version of `cc` with arrays and `bash v5.1.8` (Homebrew): `3.267s` (`.016s` / call) **Using many subshells (#26408)** * with `bash v3.2.57`: `25.302s` (`.127s` / call) * with `bash v5.1.8`: `27.801s` (`.139s` / call) * with `dash`: `15.302s` (`.077s` / call) This version didn't seem to work with zsh. **This PR (no subshells)** * with `bash v3.2.57`: `4.973s` (`.025s` / call) * with `bash v5.1.8`: `4.984s` (`.025s` / call) * with `zsh`: `2.995s` (`.015s` / call) * with `dash`: `1.890s` (`.0095s` / call) Dash, with the new posix design, is easily the winner. So there are several interesting things to note here: 1. Running the posix version in `bash` is slower than using `bash` arrays. That is to be expected because it's doing a bunch of string processing where it likely did not have to before, at least in `bash`. 2. `zsh`, at least on macOS, is significantly faster than the ancient `bash` they ship with the system. Using `zsh` with the new version also makes the posix wrappers faster than `develop`. So it's worth preferring `zsh` if we have it. I suppose we should also try this with newer `bash` on Linux. 3. `bash v5.1.8` seems to be significantly faster than the old system `bash v3.2.57` for arrays. For straight POSIX stuff, it's a little slower. It did not seem to matter whether `--posix` was used. 4. `dash` is way faster than `bash` or `zsh`, so the real payoff just comes from being able to use it. I am not sure if that is mostly startup time, but it's significant. `dash` is ~2.4x faster than the original `bash` with arrays. So, doing a lot of string stuff is slower than arrays, but converting to posix seems worth it to be able to exploit `dash`. - [x] Convert all but array-related portions to sh - [x] Fix basic shellcheck issues. - [x] Convert arrays to use a few convenience functions: `append` and `extend` - [x] Get `cc` tests passing. - [x] Add `cc` tests where needed passing. - [x] Benchmarking. Co-authored-by: Tom Scogland <scogland1@llnl.gov> Co-authored-by: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
2021-10-04Stand-alone tests: distinguish NO-TESTS from PASSED (#25880)Tamara Dahlgren2-2/+31
Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
2021-10-04Avoid replacing symlinked spack.yaml when concretizing an environment (#26428)Pedro Demarchi Gomes2-1/+20
2021-10-04Improve an error message in stage.py (#23737)Andreas Baumbach1-1/+2
2021-10-04Fix NonVirtualInHierarchyError message format (#26008)Dylan Simon1-1/+1
2021-10-04Build ppc64le docker images (#26442)Massimiliano Culpo3-5/+32
* Update archspec * Add ppc64le to docker images
2021-10-04Update Spec.full_hash docstring (#26456)Massimiliano Culpo1-4/+1
The docstring is outdated since #21735 when the build hash has been included in the full hash.
2021-10-03Fix JSONDecodeError when using compiler modules (#25624)Jordan Galby2-2/+2
When using modules for compiler (and/or external package), if a package's `setup_[dependent_]build_environment` sets `PYTHONHOME`, it can influence the python subprocess executed to gather module information. The error seen was: ``` json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) ``` But the actual hidden error happened in the `python -c 'import json...'` subprocess, which made it return an empty string as json: ``` ModuleNotFoundError: No module named 'encodings' ``` This fix uses `python -E` to ignore `PYTHONHOME` and `PYTHONPATH`. Should be safe here because the python subprocess code only use packages built-in python. The python subprocess in `environment.py` was also patched to be safe and consistent.
2021-10-03Remove .99 from version ranges (#26422)Harmen Stoppels3-6/+6
In most cases, .99 can be omitted thanks to #26402 .
2021-10-01Simplify setup_package in build environment (#26070)Harmen Stoppels1-29/+22
* Remove redundant preserve environment code in build environment * Remove fix for a bug in a module See https://github.com/spack/spack/issues/3153#issuecomment-280460041, this shouldn't be part of core spack. * Don't module unload cray-libsci on all platforms
2021-10-01Fix error message when test throws AttributeError (#25895)Cory Bloor1-4/+6
Narrow the scope of the try/except block, to avoid a misleading error message if fn() throws an AttributeError.
2021-10-01Allow non-empty ranges 1.1.0:1.1 (#26402)Harmen Stoppels2-2/+23
2021-10-01Spack install: handle failed restore of backup (#25647)Harmen Stoppels3-35/+200
Spack has logic to preserve an installation prefix when it is being overwritten: if the new install fails, the old files are restored. This PR adds error handling for when this backup restoration fails (i.e. the new install fails, and then some unexpected error prevents restoration from the backup).
2021-10-01Service jobs do not need an active environmentScott Wittenburg1-5/+0
2021-10-01Add oneAPI packages from 2021.4 release (#26401)Anna Masalskaya1-0/+12
2021-10-01bootstrapping: improve error messages (#26399)Harmen Stoppels2-8/+26
2021-09-30Add info command tests to increase coverage (#26127)Tamara Dahlgren1-0/+16
2021-09-30Replace spec-related install_test asserts with exceptions; added unit tests ↵Tamara Dahlgren2-6/+57
(#25982)
2021-09-30ArchSpec: minor cleanup of a few methods (#26376)Massimiliano Culpo1-51/+26
* Remove vestigial code to be compatible with Spack v0.9.X * ArchSpec: reworked __repr__ to be more adherent to common Python idioms * ArchSpec: simplified __init__.py and copy()
2021-09-30match .spack literal, not as a regex (#26374)Harmen Stoppels3-5/+6
2021-09-30Bump version from v0.16.2 to v0.16.3 (#26372)Harmen Stoppels1-1/+1
2021-09-30Move new CUDA conflicts inside when('~allow-unsupported-compilers') block ↵David Beckingsale1-22/+21
(#26132) * Move new CUDA conflicts inside when('~allow-unsupported-compilers') block Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
2021-09-29Un-unset TERM and DISPLAY (#26326)Harmen Stoppels1-4/+0
For interactive `spack build-env`'s this is undesired behavior
2021-09-28Move detection logic in its own package (#26119)Massimiliano Culpo5-287/+376
The logic to perform detection of already installed packages has been extracted from cmd/external.py and put into the spack.detection package. In this way it can be reused programmatically for other purposes, like bootstrapping. The new implementation accounts for cases where the executables are placed in a subdirectory within <prefix>/bin
2021-09-27Use gnuconfig package for config file replacement (#26035)Harmen Stoppels3-44/+234
* Use gnuconfig package for config file replacement Currently the autotools build system tries to pick up config.sub and config.guess files from the system (in /usr/share) on arm and power. This is introduces an implicit system dependency which we can avoid by distributing config.guess and config.sub files in a separate package, such as the new `gnuconfig` package which is very lightweight/text only (unlike automake where we previously pulled these files from as a backup). This PR adds `gnuconfig` as an unconditional build dependency for arm and power archs. In case the user needs a system version of config.sub and config.guess, they are free to mark `gnuconfig` as an external package with the prefix pointing to the directory containing the config files: ```yaml gnuconfig: externals: - spec: gnuconfig@master prefix: /tmp/tmp.ooBlkyAKdw/lol buildable: false ``` Apart from that, this PR gives some better instructions for users when replacing config files goes wrong. * Mock needs this package too now, because autotools adds a depends_on * Add documentation * Make patch_config_files a prop, fix the docs, add integrations tests * Make macOS happy
2021-09-27Correct path comparisons for fs view (#25891)psakievich2-6/+13
* Fix path comparisons in copy views * Get correct permissions * Set group id though os Co-authored-by: Philip Sakievich <psakiev@sanida.gov>
2021-09-26log_parser.py: Find failed test case messages in error logs (#25694)bernhardkaindl1-1/+7
- Match failed autotest tests show the word "FAILED" near the end - Match "FAIL: ", "FATAL: ", "failed ", "Failed test" of other suites - autotest " ok"$ means the test passed, independend of text before. - autoconf messages showing missing tools are fatal later, show them.
2021-09-25autotools doc PR: No depends_on('m4') with depends_on('autoconf') (#26101)bernhardkaindl1-8/+32
* autotoolspackage.rst: No depends_on('m4') with depends_on('autoconf') - Remove `m4` from the example depends_on() lines for the autoreconf phase. - Change the branch used as example from develop to master as it is far more common in the packages of spack's builtin repo. - Fix the wrong info that libtoolize and aclocal are run explicitly in the autoreconf phase by default. autoreconf calls these internally as needed, thus autotools.py also does not call them directly. - Add that autoreconf() also adds -I<aclocal-prefix>/share/aclocal. - Add an example how to set autoreconf_extra_args. - Add an example of a custom autoreconf phase for running autogen.sh. Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2021-09-24autotools.py/autoreconf: Show the depends_on()s to add to the package (#26115)bernhardkaindl1-4/+18
This commit shows a template for cut-and-paste into the package to fix it: ```py ==> fast-global-file-status: Executing phase: 'autoreconf' ==> Error: RuntimeError: Cannot generate configure: missing dependencies autoconf, automake, libtool. Please add the following lines to the package: depends_on('autoconf', type='build', when='@master') depends_on('automake', type='build', when='@master') depends_on('libtool', type='build', when='@master') Update the version (when='@master') as needed. ``` Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2021-09-24Remove centos:6 image references (#26095)Harmen Stoppels3-12/+1
This was EOL November 30th, 2020. I believe the "builds" are failing on develop because of it.
2021-09-22spack/build_environment.py: Clean MAKEFLAGS, DISPLAY and TERM (#26092)bernhardkaindl1-0/+7
clean_environment(): Unset three more environment variables: MAKEFLAGS: Affects make, can eg indirectly inhibit enabling parallel build DISPLAY: Tests of GUI widget libraries might try to connect to an X server TERM: Could make testsuites attempt to color their output
2021-09-21Feature: Add deprecated versions section to spack info output (#25972)Tamara Dahlgren1-5/+20
2021-09-21Rename 'variant_name' to 'variant' and document it in autotools build system ↵Harmen Stoppels3-20/+40
(#26064)
2021-09-20Allow setting variant name in AutotoolsPackage._activate_or_not (#26054)iarspider2-15/+29
2021-09-19Python: use platform-specific site packages dir (#25998)Adam J. Stewart2-2/+2
2021-09-17Bootstrap should search for compilers after switching config scopes (#26029)Massimiliano Culpo6-47/+80
fixes #25992 Currently the bootstrapping process may need a compiler. When bootstrapping from sources the need is obvious, while when bootstrapping from binaries it's currently needed in case patchelf is not on the system (since it will be then bootstrapped from sources). Before this PR we were searching for compilers as the first operation, in case they were not declared in the configuration. This fails in case we start bootstrapping from within an environment. The fix is to defer the search until we have swapped configuration.
2021-09-16cc: Use parameter expansion instead of basename (#24509)Michael Kuhn1-1/+1
While debugging #24508, I noticed that we call `basename` in `cc`. The same can be achieved by using Bash's parameter expansion, saving one external process per call. Parameter expansion cannot replace basename for directories in some cases, but is guaranteed to work for executables.
2021-09-16Recommend Git's manyFiles feature (#25977)Michael Kuhn2-2/+2
Git 2.24 introduced a feature flag for repositories with many files, see: https://github.blog/2019-11-03-highlights-from-git-2-24/#feature-macros Since Spack's Git repository contains roughly 8,500 files, it can be worthwhile to enable this, especially on slow file systems such as NFS: ``` $ hyperfine --warmup 3 'cd spack-default; git status' 'cd spack-manyfiles; git status' Benchmark #1: cd spack-default; git status Time (mean ± σ): 3.388 s ± 0.095 s [User: 256.2 ms, System: 625.8 ms] Range (min … max): 3.168 s … 3.535 s 10 runs Benchmark #2: cd spack-manyfiles; git status Time (mean ± σ): 168.7 ms ± 10.9 ms [User: 98.6 ms, System: 126.1 ms] Range (min … max): 144.8 ms … 188.0 ms 19 runs Summary 'cd spack-manyfiles; git status' ran 20.09 ± 1.42 times faster than 'cd spack-default; git status' ```
2021-09-16Add a deprecation warning when using the old concretizer (#25966)Massimiliano Culpo1-0/+8
2021-09-16Improve bootstrapping docs a hair (#25962)Harmen Stoppels2-50/+51
2021-09-16Fix NameError in foreground/background test (#25967)Harmen Stoppels1-2/+4
2021-09-16Inform the user about bootstrapping (#25964)Harmen Stoppels1-0/+3
2021-09-15Raise exception when 1+ stand-alone tests fail (#25857)Tamara Dahlgren1-0/+14
2021-09-14Make clingo the default solver (#25502)Massimiliano Culpo2-1/+11
Modifications: - [x] Change `defaults/config.yaml` - [x] Add a fix for bootstrapping patchelf from sources if `compilers.yaml` is empty - [x] Make `SPACK_TEST_SOLVER=clingo` the default for unit-tests - [x] Fix package failures in the e4s pipeline Caveats: 1. CentOS 6 still uses the original concretizer as it can't connect to the buildcache due to issues with `ssl` (bootstrapping from sources requires a C++14 capable compiler) 1. I had to update the image tag for GitlabCI in e699f14. 1. libtool v2.4.2 has been deprecated and other packages received some update
2021-09-14Add a __reduce__ method to Environment (#25678)Adam J. Stewart2-0/+27
* Add a __reduce__ method to Environment * Add unit test * Convert Path to str
2021-09-14Adding ability to compare git references to spack install (#24639)Vanessasaurus11-43/+614
This will allow a user to (from anywhere a Spec is parsed including both name and version) refer to a git commit in lieu of a package version, and be able to make comparisons with releases in the history based on commits (or with other commits). We do this by way of: - Adding a property, is_commit, to a version, meaning I can always check if a version is a commit and then change some action. - Adding an attribute to the Version object which can lookup commits from a git repo and find the last known version before that commit, and the distance - Construct new Version comparators, which are tuples. For normal versions, they are unchanged. For commits with a previous version x.y.z, d commits away, the comparator is (x, y, z, '', d). For commits with no previous version, the comparator is ('', d) where d is the distance from the first commit in the repo. - Metadata on git commits is cached in the misc_cache, for quick lookup later. - Git repos are cached as bare repos in `~/.spack/git_repos` - In both caches, git repo urls are turned into file paths within the cache If a commit cannot be found in the cached git repo, we fetch from the repo. If a commit is found in the cached metadata, we do not recompare to newly downloaded tags (assuming repo structure does not change). The cached metadata may be thrown out by using the `spack clean -m` option if you know the repo structure has changed in a way that invalidates existing entries. Future work will include automatic updates. # Finding previous versions Spack will search the repo for any tags that match the string of a version given by the `version` directive. Spack will also search for any tags that match `v + string` for any version string. Beyond that, Spack will search for tags that match a SEMVER regex (i.e., tags of the form x.y.z) and interpret those tags as valid versions as well. Future work will increase the breadth of tags understood by Spack For each tag, Spack queries git to determine whether the tag is an ancestor of the commit in question or not. Spack then sorts the tags that are ancestors of the commit by commit-distance in the repo, and takes the nearest ancestor. The version represented by that tag is listed as the previous version for the commit. Not all commits will find a previous version, depending on the package workflow. Future work may enable more tangential relationships between commits and versions to be discovered, but many commits in real world git repos require human knowledge to associate with a most recent previous version. Future work will also allow packages to specify commit/tag/version relationships manually for such situations. # Version comparisons. The empty string is a valid component of a Spack version tuple, and is in fact the lowest-valued component. It cannot be generated as part of any valid version. These two characteristics make it perfect for delineating previous versions from distances. For any version x.y.z, (x, y, z, '', _) will be less than any "real" version beginning x.y.z. This ensures that no distance from a release will cause the commit to be interpreted as "greater than" a version which is not an ancestor of it. Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: Gregory Becker <becker33@llnl.gov> Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>