summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2021-10-01Simplify setup_package in build environment (#26070)Harmen Stoppels1-29/+22
* Remove redundant preserve environment code in build environment * Remove fix for a bug in a module See https://github.com/spack/spack/issues/3153#issuecomment-280460041, this shouldn't be part of core spack. * Don't module unload cray-libsci on all platforms
2021-10-01Fix error message when test throws AttributeError (#25895)Cory Bloor1-4/+6
Narrow the scope of the try/except block, to avoid a misleading error message if fn() throws an AttributeError.
2021-10-01Allow non-empty ranges 1.1.0:1.1 (#26402)Harmen Stoppels2-2/+23
2021-10-01Spack install: handle failed restore of backup (#25647)Harmen Stoppels3-35/+200
Spack has logic to preserve an installation prefix when it is being overwritten: if the new install fails, the old files are restored. This PR adds error handling for when this backup restoration fails (i.e. the new install fails, and then some unexpected error prevents restoration from the backup).
2021-10-01Service jobs do not need an active environmentScott Wittenburg1-5/+0
2021-10-01Add oneAPI packages from 2021.4 release (#26401)Anna Masalskaya1-0/+12
2021-10-01bootstrapping: improve error messages (#26399)Harmen Stoppels2-8/+26
2021-09-30Add info command tests to increase coverage (#26127)Tamara Dahlgren1-0/+16
2021-09-30Replace spec-related install_test asserts with exceptions; added unit tests ↵Tamara Dahlgren2-6/+57
(#25982)
2021-09-30ArchSpec: minor cleanup of a few methods (#26376)Massimiliano Culpo1-51/+26
* Remove vestigial code to be compatible with Spack v0.9.X * ArchSpec: reworked __repr__ to be more adherent to common Python idioms * ArchSpec: simplified __init__.py and copy()
2021-09-30match .spack literal, not as a regex (#26374)Harmen Stoppels3-5/+6
2021-09-30Bump version from v0.16.2 to v0.16.3 (#26372)Harmen Stoppels1-1/+1
2021-09-30Move new CUDA conflicts inside when('~allow-unsupported-compilers') block ↵David Beckingsale1-22/+21
(#26132) * Move new CUDA conflicts inside when('~allow-unsupported-compilers') block Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
2021-09-29Un-unset TERM and DISPLAY (#26326)Harmen Stoppels1-4/+0
For interactive `spack build-env`'s this is undesired behavior
2021-09-28Move detection logic in its own package (#26119)Massimiliano Culpo5-287/+376
The logic to perform detection of already installed packages has been extracted from cmd/external.py and put into the spack.detection package. In this way it can be reused programmatically for other purposes, like bootstrapping. The new implementation accounts for cases where the executables are placed in a subdirectory within <prefix>/bin
2021-09-27Use gnuconfig package for config file replacement (#26035)Harmen Stoppels3-44/+234
* Use gnuconfig package for config file replacement Currently the autotools build system tries to pick up config.sub and config.guess files from the system (in /usr/share) on arm and power. This is introduces an implicit system dependency which we can avoid by distributing config.guess and config.sub files in a separate package, such as the new `gnuconfig` package which is very lightweight/text only (unlike automake where we previously pulled these files from as a backup). This PR adds `gnuconfig` as an unconditional build dependency for arm and power archs. In case the user needs a system version of config.sub and config.guess, they are free to mark `gnuconfig` as an external package with the prefix pointing to the directory containing the config files: ```yaml gnuconfig: externals: - spec: gnuconfig@master prefix: /tmp/tmp.ooBlkyAKdw/lol buildable: false ``` Apart from that, this PR gives some better instructions for users when replacing config files goes wrong. * Mock needs this package too now, because autotools adds a depends_on * Add documentation * Make patch_config_files a prop, fix the docs, add integrations tests * Make macOS happy
2021-09-27Correct path comparisons for fs view (#25891)psakievich2-6/+13
* Fix path comparisons in copy views * Get correct permissions * Set group id though os Co-authored-by: Philip Sakievich <psakiev@sanida.gov>
2021-09-26log_parser.py: Find failed test case messages in error logs (#25694)bernhardkaindl1-1/+7
- Match failed autotest tests show the word "FAILED" near the end - Match "FAIL: ", "FATAL: ", "failed ", "Failed test" of other suites - autotest " ok"$ means the test passed, independend of text before. - autoconf messages showing missing tools are fatal later, show them.
2021-09-25autotools doc PR: No depends_on('m4') with depends_on('autoconf') (#26101)bernhardkaindl1-8/+32
* autotoolspackage.rst: No depends_on('m4') with depends_on('autoconf') - Remove `m4` from the example depends_on() lines for the autoreconf phase. - Change the branch used as example from develop to master as it is far more common in the packages of spack's builtin repo. - Fix the wrong info that libtoolize and aclocal are run explicitly in the autoreconf phase by default. autoreconf calls these internally as needed, thus autotools.py also does not call them directly. - Add that autoreconf() also adds -I<aclocal-prefix>/share/aclocal. - Add an example how to set autoreconf_extra_args. - Add an example of a custom autoreconf phase for running autogen.sh. Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2021-09-24autotools.py/autoreconf: Show the depends_on()s to add to the package (#26115)bernhardkaindl1-4/+18
This commit shows a template for cut-and-paste into the package to fix it: ```py ==> fast-global-file-status: Executing phase: 'autoreconf' ==> Error: RuntimeError: Cannot generate configure: missing dependencies autoconf, automake, libtool. Please add the following lines to the package: depends_on('autoconf', type='build', when='@master') depends_on('automake', type='build', when='@master') depends_on('libtool', type='build', when='@master') Update the version (when='@master') as needed. ``` Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2021-09-24Remove centos:6 image references (#26095)Harmen Stoppels3-12/+1
This was EOL November 30th, 2020. I believe the "builds" are failing on develop because of it.
2021-09-22spack/build_environment.py: Clean MAKEFLAGS, DISPLAY and TERM (#26092)bernhardkaindl1-0/+7
clean_environment(): Unset three more environment variables: MAKEFLAGS: Affects make, can eg indirectly inhibit enabling parallel build DISPLAY: Tests of GUI widget libraries might try to connect to an X server TERM: Could make testsuites attempt to color their output
2021-09-21Feature: Add deprecated versions section to spack info output (#25972)Tamara Dahlgren1-5/+20
2021-09-21Rename 'variant_name' to 'variant' and document it in autotools build system ↵Harmen Stoppels3-20/+40
(#26064)
2021-09-20Allow setting variant name in AutotoolsPackage._activate_or_not (#26054)iarspider2-15/+29
2021-09-19Python: use platform-specific site packages dir (#25998)Adam J. Stewart2-2/+2
2021-09-17Bootstrap should search for compilers after switching config scopes (#26029)Massimiliano Culpo6-47/+80
fixes #25992 Currently the bootstrapping process may need a compiler. When bootstrapping from sources the need is obvious, while when bootstrapping from binaries it's currently needed in case patchelf is not on the system (since it will be then bootstrapped from sources). Before this PR we were searching for compilers as the first operation, in case they were not declared in the configuration. This fails in case we start bootstrapping from within an environment. The fix is to defer the search until we have swapped configuration.
2021-09-16cc: Use parameter expansion instead of basename (#24509)Michael Kuhn1-1/+1
While debugging #24508, I noticed that we call `basename` in `cc`. The same can be achieved by using Bash's parameter expansion, saving one external process per call. Parameter expansion cannot replace basename for directories in some cases, but is guaranteed to work for executables.
2021-09-16Recommend Git's manyFiles feature (#25977)Michael Kuhn2-2/+2
Git 2.24 introduced a feature flag for repositories with many files, see: https://github.blog/2019-11-03-highlights-from-git-2-24/#feature-macros Since Spack's Git repository contains roughly 8,500 files, it can be worthwhile to enable this, especially on slow file systems such as NFS: ``` $ hyperfine --warmup 3 'cd spack-default; git status' 'cd spack-manyfiles; git status' Benchmark #1: cd spack-default; git status Time (mean ± σ): 3.388 s ± 0.095 s [User: 256.2 ms, System: 625.8 ms] Range (min … max): 3.168 s … 3.535 s 10 runs Benchmark #2: cd spack-manyfiles; git status Time (mean ± σ): 168.7 ms ± 10.9 ms [User: 98.6 ms, System: 126.1 ms] Range (min … max): 144.8 ms … 188.0 ms 19 runs Summary 'cd spack-manyfiles; git status' ran 20.09 ± 1.42 times faster than 'cd spack-default; git status' ```
2021-09-16Add a deprecation warning when using the old concretizer (#25966)Massimiliano Culpo1-0/+8
2021-09-16Improve bootstrapping docs a hair (#25962)Harmen Stoppels2-50/+51
2021-09-16Fix NameError in foreground/background test (#25967)Harmen Stoppels1-2/+4
2021-09-16Inform the user about bootstrapping (#25964)Harmen Stoppels1-0/+3
2021-09-15Raise exception when 1+ stand-alone tests fail (#25857)Tamara Dahlgren1-0/+14
2021-09-14Make clingo the default solver (#25502)Massimiliano Culpo2-1/+11
Modifications: - [x] Change `defaults/config.yaml` - [x] Add a fix for bootstrapping patchelf from sources if `compilers.yaml` is empty - [x] Make `SPACK_TEST_SOLVER=clingo` the default for unit-tests - [x] Fix package failures in the e4s pipeline Caveats: 1. CentOS 6 still uses the original concretizer as it can't connect to the buildcache due to issues with `ssl` (bootstrapping from sources requires a C++14 capable compiler) 1. I had to update the image tag for GitlabCI in e699f14. 1. libtool v2.4.2 has been deprecated and other packages received some update
2021-09-14Add a __reduce__ method to Environment (#25678)Adam J. Stewart2-0/+27
* Add a __reduce__ method to Environment * Add unit test * Convert Path to str
2021-09-14Adding ability to compare git references to spack install (#24639)Vanessasaurus11-43/+614
This will allow a user to (from anywhere a Spec is parsed including both name and version) refer to a git commit in lieu of a package version, and be able to make comparisons with releases in the history based on commits (or with other commits). We do this by way of: - Adding a property, is_commit, to a version, meaning I can always check if a version is a commit and then change some action. - Adding an attribute to the Version object which can lookup commits from a git repo and find the last known version before that commit, and the distance - Construct new Version comparators, which are tuples. For normal versions, they are unchanged. For commits with a previous version x.y.z, d commits away, the comparator is (x, y, z, '', d). For commits with no previous version, the comparator is ('', d) where d is the distance from the first commit in the repo. - Metadata on git commits is cached in the misc_cache, for quick lookup later. - Git repos are cached as bare repos in `~/.spack/git_repos` - In both caches, git repo urls are turned into file paths within the cache If a commit cannot be found in the cached git repo, we fetch from the repo. If a commit is found in the cached metadata, we do not recompare to newly downloaded tags (assuming repo structure does not change). The cached metadata may be thrown out by using the `spack clean -m` option if you know the repo structure has changed in a way that invalidates existing entries. Future work will include automatic updates. # Finding previous versions Spack will search the repo for any tags that match the string of a version given by the `version` directive. Spack will also search for any tags that match `v + string` for any version string. Beyond that, Spack will search for tags that match a SEMVER regex (i.e., tags of the form x.y.z) and interpret those tags as valid versions as well. Future work will increase the breadth of tags understood by Spack For each tag, Spack queries git to determine whether the tag is an ancestor of the commit in question or not. Spack then sorts the tags that are ancestors of the commit by commit-distance in the repo, and takes the nearest ancestor. The version represented by that tag is listed as the previous version for the commit. Not all commits will find a previous version, depending on the package workflow. Future work may enable more tangential relationships between commits and versions to be discovered, but many commits in real world git repos require human knowledge to associate with a most recent previous version. Future work will also allow packages to specify commit/tag/version relationships manually for such situations. # Version comparisons. The empty string is a valid component of a Spack version tuple, and is in fact the lowest-valued component. It cannot be generated as part of any valid version. These two characteristics make it perfect for delineating previous versions from distances. For any version x.y.z, (x, y, z, '', _) will be less than any "real" version beginning x.y.z. This ensures that no distance from a release will cause the commit to be interpreted as "greater than" a version which is not an ancestor of it. Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: Gregory Becker <becker33@llnl.gov> Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2021-09-14Update spack monitor to support new spec (#25928)Vanessasaurus1-1/+3
This PR coincides with tiny changes to spack to support spack monitor using the new spec the corresponding spack monitor PR is at https://github.com/spack/spack-monitor/pull/31. Since there are no changes to the database we can actually update the current server fairly easily, so either someone can test locally or we can just update and then test from that (and update as needed). Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com>
2021-09-13Fix environment reading from lockfile to trust written hashes (#25879)Greg Becker3-10/+57
#22845 revealed a long-standing bug that had never been triggered before, because the hashing algorithm had been stable for multiple years while the bug was in production. The bug was that when reading a concretized environment, Spack did not properly read in the build hashes associated with the specs in the environment. Those hashes were recomputed (and as long as we didn't change the algorithm, were recomputed identically). Spack's policy, though, is never to recompute a hash. Once something is installed, we respect its metadata hash forever -- even if internally Spack changes the hashing method. Put differently, once something is concretized, it has a concrete hash, and that's it -- forever. When we changed the hashing algorithm for performance in #22845 we exposed the bug. This PR fixes the bug at its source, but properly reading in the cached build hash attributes associated with the specs. I've also renamed some variables in the Environment class methods to make a mistake of this sort more difficult to make in the future. * ensure environment build hashes are never recomputed * add comment clarifying reattachment of env build hashes * bump lockfile version and include specfile version in env meta * Fix unit-test for v1 to v2 conversion Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2021-09-13Avoid hidden circular dependencies in spack.architecture (#25873)Massimiliano Culpo22-473/+490
* Refactor platform etc. to avoid circular dependencies All the base classes in spack.architecture have been moved to the corresponding specialized subpackages, e.g. Platform is now defined within spack.platforms. This resolves a circular dependency where spack.architecture was both: - Defining the base classes for spack.platforms, etc. - Collecting derived classes from spack.platforms, etc. Now it dopes only the latter. * Move a few platform related functions to "spack.platforms" * Removed spack.architecture.sys_type() * Fixup for docs * Rename Python modules according to review
2021-09-13Bugfix: Correct checksum's sha256 when retrieve from remote (#25831)Tamara Dahlgren1-0/+10
2021-09-13Bugfix: spack test debug requires Spack tty (#25897)Tamara Dahlgren1-1/+1
2021-09-10[docs] document official gfortran macOS precompiled binaries (#25818)Stephen McDowell1-3/+5
* document official gfortran macOS precompiled binaries * compile without -vvv ;) {squash this}
2021-09-10Remove dead code in installer (#24035)Harmen Stoppels3-69/+0
Currently as part of installing a package, we lock a prefix, check if it exists, and create it if not; the logic for creating the prefix included a check for the existence of that prefix (and raised an exception if it did), which was redundant. This also includes removal of tests which were not verifying anything (they pass with or without the modifications in this PR).
2021-09-09CUDA official GCC conflicts (#25054)albestro1-6/+21
* update CUDA 11 / GCC compatibility range * additional unofficial conflict * minor changes to comments
2021-09-09Refactor unit-tests in test/architecture.py (#25848)Massimiliano Culpo5-121/+101
Modifications: - Export platforms from spack.platforms directly, so that client modules don't have to import submodules - Use only plain imports in test/architecture.py - Parametrized test in test/architecture.py and put most of the setup/teardown in fixtures
2021-09-09specs: move to new spec.json format with build provenance (#22845)Nathan Hanford34-674/+1195
This is a major rework of Spack's core core `spec.yaml` metadata format. It moves from `spec.yaml` to `spec.json` for speed, and it changes the format in several ways. Specifically: 1. The spec format now has a `_meta` section with a version (now set to version `2`). This will simplify major changes like this one in the future. 2. The node list in spec dictionaries is no longer keyed by name. Instead, it is a list of records with no required key. The name, hash, etc. are fields in the dictionary records like any other. 3. Dependencies can be keyed by any hash (`hash`, `full_hash`, `build_hash`). 4. `build_spec` provenance from #20262 is included in the spec format. This means that, for spliced specs, we preserve the *full* provenance of how to build, and we can reproduce a spliced spec from the original builds that produced it. **NOTE**: Because we have switched the spec format, this PR changes Spack's hashing algorithm. This means that after this commit, Spack will think a lot of things need rebuilds. There are two major benefits this PR provides: * The switch to JSON format speeds up Spack significantly, as Python's builtin JSON implementation is orders of magnitude faster than YAML. * The new Spec format will soon allow us to represent DAGs with potentially multiple versions of the same dependency -- e.g., for build dependencies or for compilers-as-dependencies. This PR lays the necessary groundwork for those features. The old `spec.yaml` format continues to be supported, but is now considered a legacy format, and Spack will opportunistically convert these to the new `spec.json` format.
2021-09-08Account for bootstrapping from sources niche caseMassimiliano Culpo1-7/+23
This modification accounts for: 1. Bootstrapping from sources using system, non-standard Python 2. Using later an ABI compatible standard Python interpreter
2021-09-08Fix clingo bootstrapping on rhel + ppc64leMassimiliano Culpo1-0/+53
The system Python interpreter on rhel is patched to have slightly different names for some architectures. This makes it incompatible with manylinux generated extensions for ppc64le. To fix this issue when bootstrapping Spack we generate on-the-fly symbolic links to the name expected by the current interpreter if it differs from the default. Links: https://github.com/pypa/manylinux/issues/687 https://src.fedoraproject.org/fork/churchyard/rpms/python3/blame/00274-fix-arch-names.patch?identifier=test_email-mktime
2021-09-08Disable module generation during bootstrappingMassimiliano Culpo5-7/+41