summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2021-09-27Use gnuconfig package for config file replacement (#26035)Harmen Stoppels3-44/+234
* Use gnuconfig package for config file replacement Currently the autotools build system tries to pick up config.sub and config.guess files from the system (in /usr/share) on arm and power. This is introduces an implicit system dependency which we can avoid by distributing config.guess and config.sub files in a separate package, such as the new `gnuconfig` package which is very lightweight/text only (unlike automake where we previously pulled these files from as a backup). This PR adds `gnuconfig` as an unconditional build dependency for arm and power archs. In case the user needs a system version of config.sub and config.guess, they are free to mark `gnuconfig` as an external package with the prefix pointing to the directory containing the config files: ```yaml gnuconfig: externals: - spec: gnuconfig@master prefix: /tmp/tmp.ooBlkyAKdw/lol buildable: false ``` Apart from that, this PR gives some better instructions for users when replacing config files goes wrong. * Mock needs this package too now, because autotools adds a depends_on * Add documentation * Make patch_config_files a prop, fix the docs, add integrations tests * Make macOS happy
2021-09-27Correct path comparisons for fs view (#25891)psakievich2-6/+13
* Fix path comparisons in copy views * Get correct permissions * Set group id though os Co-authored-by: Philip Sakievich <psakiev@sanida.gov>
2021-09-26log_parser.py: Find failed test case messages in error logs (#25694)bernhardkaindl1-1/+7
- Match failed autotest tests show the word "FAILED" near the end - Match "FAIL: ", "FATAL: ", "failed ", "Failed test" of other suites - autotest " ok"$ means the test passed, independend of text before. - autoconf messages showing missing tools are fatal later, show them.
2021-09-25autotools doc PR: No depends_on('m4') with depends_on('autoconf') (#26101)bernhardkaindl1-8/+32
* autotoolspackage.rst: No depends_on('m4') with depends_on('autoconf') - Remove `m4` from the example depends_on() lines for the autoreconf phase. - Change the branch used as example from develop to master as it is far more common in the packages of spack's builtin repo. - Fix the wrong info that libtoolize and aclocal are run explicitly in the autoreconf phase by default. autoreconf calls these internally as needed, thus autotools.py also does not call them directly. - Add that autoreconf() also adds -I<aclocal-prefix>/share/aclocal. - Add an example how to set autoreconf_extra_args. - Add an example of a custom autoreconf phase for running autogen.sh. Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2021-09-24autotools.py/autoreconf: Show the depends_on()s to add to the package (#26115)bernhardkaindl1-4/+18
This commit shows a template for cut-and-paste into the package to fix it: ```py ==> fast-global-file-status: Executing phase: 'autoreconf' ==> Error: RuntimeError: Cannot generate configure: missing dependencies autoconf, automake, libtool. Please add the following lines to the package: depends_on('autoconf', type='build', when='@master') depends_on('automake', type='build', when='@master') depends_on('libtool', type='build', when='@master') Update the version (when='@master') as needed. ``` Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2021-09-24Remove centos:6 image references (#26095)Harmen Stoppels3-12/+1
This was EOL November 30th, 2020. I believe the "builds" are failing on develop because of it.
2021-09-22spack/build_environment.py: Clean MAKEFLAGS, DISPLAY and TERM (#26092)bernhardkaindl1-0/+7
clean_environment(): Unset three more environment variables: MAKEFLAGS: Affects make, can eg indirectly inhibit enabling parallel build DISPLAY: Tests of GUI widget libraries might try to connect to an X server TERM: Could make testsuites attempt to color their output
2021-09-21Feature: Add deprecated versions section to spack info output (#25972)Tamara Dahlgren1-5/+20
2021-09-21Rename 'variant_name' to 'variant' and document it in autotools build system ↵Harmen Stoppels3-20/+40
(#26064)
2021-09-20Allow setting variant name in AutotoolsPackage._activate_or_not (#26054)iarspider2-15/+29
2021-09-19Python: use platform-specific site packages dir (#25998)Adam J. Stewart2-2/+2
2021-09-17Bootstrap should search for compilers after switching config scopes (#26029)Massimiliano Culpo6-47/+80
fixes #25992 Currently the bootstrapping process may need a compiler. When bootstrapping from sources the need is obvious, while when bootstrapping from binaries it's currently needed in case patchelf is not on the system (since it will be then bootstrapped from sources). Before this PR we were searching for compilers as the first operation, in case they were not declared in the configuration. This fails in case we start bootstrapping from within an environment. The fix is to defer the search until we have swapped configuration.
2021-09-16cc: Use parameter expansion instead of basename (#24509)Michael Kuhn1-1/+1
While debugging #24508, I noticed that we call `basename` in `cc`. The same can be achieved by using Bash's parameter expansion, saving one external process per call. Parameter expansion cannot replace basename for directories in some cases, but is guaranteed to work for executables.
2021-09-16Recommend Git's manyFiles feature (#25977)Michael Kuhn2-2/+2
Git 2.24 introduced a feature flag for repositories with many files, see: https://github.blog/2019-11-03-highlights-from-git-2-24/#feature-macros Since Spack's Git repository contains roughly 8,500 files, it can be worthwhile to enable this, especially on slow file systems such as NFS: ``` $ hyperfine --warmup 3 'cd spack-default; git status' 'cd spack-manyfiles; git status' Benchmark #1: cd spack-default; git status Time (mean ± σ): 3.388 s ± 0.095 s [User: 256.2 ms, System: 625.8 ms] Range (min … max): 3.168 s … 3.535 s 10 runs Benchmark #2: cd spack-manyfiles; git status Time (mean ± σ): 168.7 ms ± 10.9 ms [User: 98.6 ms, System: 126.1 ms] Range (min … max): 144.8 ms … 188.0 ms 19 runs Summary 'cd spack-manyfiles; git status' ran 20.09 ± 1.42 times faster than 'cd spack-default; git status' ```
2021-09-16Add a deprecation warning when using the old concretizer (#25966)Massimiliano Culpo1-0/+8
2021-09-16Improve bootstrapping docs a hair (#25962)Harmen Stoppels2-50/+51
2021-09-16Fix NameError in foreground/background test (#25967)Harmen Stoppels1-2/+4
2021-09-16Inform the user about bootstrapping (#25964)Harmen Stoppels1-0/+3
2021-09-15Raise exception when 1+ stand-alone tests fail (#25857)Tamara Dahlgren1-0/+14
2021-09-14Make clingo the default solver (#25502)Massimiliano Culpo2-1/+11
Modifications: - [x] Change `defaults/config.yaml` - [x] Add a fix for bootstrapping patchelf from sources if `compilers.yaml` is empty - [x] Make `SPACK_TEST_SOLVER=clingo` the default for unit-tests - [x] Fix package failures in the e4s pipeline Caveats: 1. CentOS 6 still uses the original concretizer as it can't connect to the buildcache due to issues with `ssl` (bootstrapping from sources requires a C++14 capable compiler) 1. I had to update the image tag for GitlabCI in e699f14. 1. libtool v2.4.2 has been deprecated and other packages received some update
2021-09-14Add a __reduce__ method to Environment (#25678)Adam J. Stewart2-0/+27
* Add a __reduce__ method to Environment * Add unit test * Convert Path to str
2021-09-14Adding ability to compare git references to spack install (#24639)Vanessasaurus11-43/+614
This will allow a user to (from anywhere a Spec is parsed including both name and version) refer to a git commit in lieu of a package version, and be able to make comparisons with releases in the history based on commits (or with other commits). We do this by way of: - Adding a property, is_commit, to a version, meaning I can always check if a version is a commit and then change some action. - Adding an attribute to the Version object which can lookup commits from a git repo and find the last known version before that commit, and the distance - Construct new Version comparators, which are tuples. For normal versions, they are unchanged. For commits with a previous version x.y.z, d commits away, the comparator is (x, y, z, '', d). For commits with no previous version, the comparator is ('', d) where d is the distance from the first commit in the repo. - Metadata on git commits is cached in the misc_cache, for quick lookup later. - Git repos are cached as bare repos in `~/.spack/git_repos` - In both caches, git repo urls are turned into file paths within the cache If a commit cannot be found in the cached git repo, we fetch from the repo. If a commit is found in the cached metadata, we do not recompare to newly downloaded tags (assuming repo structure does not change). The cached metadata may be thrown out by using the `spack clean -m` option if you know the repo structure has changed in a way that invalidates existing entries. Future work will include automatic updates. # Finding previous versions Spack will search the repo for any tags that match the string of a version given by the `version` directive. Spack will also search for any tags that match `v + string` for any version string. Beyond that, Spack will search for tags that match a SEMVER regex (i.e., tags of the form x.y.z) and interpret those tags as valid versions as well. Future work will increase the breadth of tags understood by Spack For each tag, Spack queries git to determine whether the tag is an ancestor of the commit in question or not. Spack then sorts the tags that are ancestors of the commit by commit-distance in the repo, and takes the nearest ancestor. The version represented by that tag is listed as the previous version for the commit. Not all commits will find a previous version, depending on the package workflow. Future work may enable more tangential relationships between commits and versions to be discovered, but many commits in real world git repos require human knowledge to associate with a most recent previous version. Future work will also allow packages to specify commit/tag/version relationships manually for such situations. # Version comparisons. The empty string is a valid component of a Spack version tuple, and is in fact the lowest-valued component. It cannot be generated as part of any valid version. These two characteristics make it perfect for delineating previous versions from distances. For any version x.y.z, (x, y, z, '', _) will be less than any "real" version beginning x.y.z. This ensures that no distance from a release will cause the commit to be interpreted as "greater than" a version which is not an ancestor of it. Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: Gregory Becker <becker33@llnl.gov> Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2021-09-14Update spack monitor to support new spec (#25928)Vanessasaurus1-1/+3
This PR coincides with tiny changes to spack to support spack monitor using the new spec the corresponding spack monitor PR is at https://github.com/spack/spack-monitor/pull/31. Since there are no changes to the database we can actually update the current server fairly easily, so either someone can test locally or we can just update and then test from that (and update as needed). Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com>
2021-09-13Fix environment reading from lockfile to trust written hashes (#25879)Greg Becker3-10/+57
#22845 revealed a long-standing bug that had never been triggered before, because the hashing algorithm had been stable for multiple years while the bug was in production. The bug was that when reading a concretized environment, Spack did not properly read in the build hashes associated with the specs in the environment. Those hashes were recomputed (and as long as we didn't change the algorithm, were recomputed identically). Spack's policy, though, is never to recompute a hash. Once something is installed, we respect its metadata hash forever -- even if internally Spack changes the hashing method. Put differently, once something is concretized, it has a concrete hash, and that's it -- forever. When we changed the hashing algorithm for performance in #22845 we exposed the bug. This PR fixes the bug at its source, but properly reading in the cached build hash attributes associated with the specs. I've also renamed some variables in the Environment class methods to make a mistake of this sort more difficult to make in the future. * ensure environment build hashes are never recomputed * add comment clarifying reattachment of env build hashes * bump lockfile version and include specfile version in env meta * Fix unit-test for v1 to v2 conversion Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2021-09-13Avoid hidden circular dependencies in spack.architecture (#25873)Massimiliano Culpo22-473/+490
* Refactor platform etc. to avoid circular dependencies All the base classes in spack.architecture have been moved to the corresponding specialized subpackages, e.g. Platform is now defined within spack.platforms. This resolves a circular dependency where spack.architecture was both: - Defining the base classes for spack.platforms, etc. - Collecting derived classes from spack.platforms, etc. Now it dopes only the latter. * Move a few platform related functions to "spack.platforms" * Removed spack.architecture.sys_type() * Fixup for docs * Rename Python modules according to review
2021-09-13Bugfix: Correct checksum's sha256 when retrieve from remote (#25831)Tamara Dahlgren1-0/+10
2021-09-13Bugfix: spack test debug requires Spack tty (#25897)Tamara Dahlgren1-1/+1
2021-09-10[docs] document official gfortran macOS precompiled binaries (#25818)Stephen McDowell1-3/+5
* document official gfortran macOS precompiled binaries * compile without -vvv ;) {squash this}
2021-09-10Remove dead code in installer (#24035)Harmen Stoppels3-69/+0
Currently as part of installing a package, we lock a prefix, check if it exists, and create it if not; the logic for creating the prefix included a check for the existence of that prefix (and raised an exception if it did), which was redundant. This also includes removal of tests which were not verifying anything (they pass with or without the modifications in this PR).
2021-09-09CUDA official GCC conflicts (#25054)albestro1-6/+21
* update CUDA 11 / GCC compatibility range * additional unofficial conflict * minor changes to comments
2021-09-09Refactor unit-tests in test/architecture.py (#25848)Massimiliano Culpo5-121/+101
Modifications: - Export platforms from spack.platforms directly, so that client modules don't have to import submodules - Use only plain imports in test/architecture.py - Parametrized test in test/architecture.py and put most of the setup/teardown in fixtures
2021-09-09specs: move to new spec.json format with build provenance (#22845)Nathan Hanford34-674/+1195
This is a major rework of Spack's core core `spec.yaml` metadata format. It moves from `spec.yaml` to `spec.json` for speed, and it changes the format in several ways. Specifically: 1. The spec format now has a `_meta` section with a version (now set to version `2`). This will simplify major changes like this one in the future. 2. The node list in spec dictionaries is no longer keyed by name. Instead, it is a list of records with no required key. The name, hash, etc. are fields in the dictionary records like any other. 3. Dependencies can be keyed by any hash (`hash`, `full_hash`, `build_hash`). 4. `build_spec` provenance from #20262 is included in the spec format. This means that, for spliced specs, we preserve the *full* provenance of how to build, and we can reproduce a spliced spec from the original builds that produced it. **NOTE**: Because we have switched the spec format, this PR changes Spack's hashing algorithm. This means that after this commit, Spack will think a lot of things need rebuilds. There are two major benefits this PR provides: * The switch to JSON format speeds up Spack significantly, as Python's builtin JSON implementation is orders of magnitude faster than YAML. * The new Spec format will soon allow us to represent DAGs with potentially multiple versions of the same dependency -- e.g., for build dependencies or for compilers-as-dependencies. This PR lays the necessary groundwork for those features. The old `spec.yaml` format continues to be supported, but is now considered a legacy format, and Spack will opportunistically convert these to the new `spec.json` format.
2021-09-08Account for bootstrapping from sources niche caseMassimiliano Culpo1-7/+23
This modification accounts for: 1. Bootstrapping from sources using system, non-standard Python 2. Using later an ABI compatible standard Python interpreter
2021-09-08Fix clingo bootstrapping on rhel + ppc64leMassimiliano Culpo1-0/+53
The system Python interpreter on rhel is patched to have slightly different names for some architectures. This makes it incompatible with manylinux generated extensions for ppc64le. To fix this issue when bootstrapping Spack we generate on-the-fly symbolic links to the name expected by the current interpreter if it differs from the default. Links: https://github.com/pypa/manylinux/issues/687 https://src.fedoraproject.org/fork/churchyard/rpms/python3/blame/00274-fix-arch-names.patch?identifier=test_email-mktime
2021-09-08Disable module generation during bootstrappingMassimiliano Culpo5-7/+41
2021-09-08url stats: add `--show-issues` option (#25792)Todd Gamblin4-36/+91
* tests: make `spack url [stats|summary]` work on mock packages Mock packages have historically had mock hashes, but this means they're also invalid as far as Spack's hash detection is concerned. - [x] convert all hashes in mock package to md5 or sha256 - [x] ensure that all mock packages have a URL - [x] ignore some special cases with multiple VCS fetchers * url stats: add `--show-issues` option `spack url stats` tells us how many URLs are using what protocol, type of checksum, etc., but it previously did not tell us which packages and URLs had the issues. This adds a `--show-issues` option to show URLs with insecure (`http`) URLs or `md5` hashes (which are now deprecated by NIST).
2021-09-08lib/spack/env/cc: tolerate trailing / in elements of $PATH (#25733)bernhardkaindl2-1/+35
Fixes removal of SPACK_ENV_PATH from PATH in the presence of trailing slashes in the elements of PATH: The compiler wrapper has to ensure that it is not called nested like it would happen when gcc's collect2 uses PATH to call the linker ld, or else the compilation fails. To prevent nested calls, the compiler wrapper removes the elements of SPACK_ENV_PATH from PATH. Sadly, the autotest framework appends a slash to each element of PATH when adding AUTOTEST_PATH to the PATH for the tests, and some tests like those of GNU bison run cc inside the test. Thus, ensure that PATH cleanup works even with trailing slashes. This fixes the autotest suite of bison, compiling hundreds of bison-generated test cases in a autotest-generated testsuite. Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2021-09-07docs: minor grammar fix (#25814)Stephen McDowell1-1/+1
2021-09-03Always disable leftover active environment after testsHarmen Stoppels1-12/+12
2021-09-03Don't error when removing scope that does not existHarmen Stoppels1-1/+2
2021-09-02start of work to add spack audit packages-https checker (#25670)Vanessasaurus3-4/+92
This PR will add a new audit, specifically for spack package homepage urls (and eventually other kinds I suspect) to see if there is an http address that can be changed to https. Usage is as follows: ```bash $ spack audit packages-https <package> ``` And in list view: ```bash $ spack audit list generic: Generic checks relying on global variables configs: Sanity checks on compilers.yaml Sanity checks on packages.yaml packages: Sanity checks on specs used in directives packages-https: Sanity checks on https checks of package urls, etc. ``` I think it would be unwise to include with packages, because when run for all, since we do requests it takes a long time. I also like the idea of more well scoped checks - likely there will be other addresses for http/https within a package that we eventually check. For now, there are two error cases - one is when an https url is tried but there is some SSL error (or other error that means we cannot update to https): ```bash $ spack audit packages-https zoltan PKG-HTTPS-DIRECTIVES: 1 issue found 1. Error with attempting https for "zoltan": <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'www.cs.sandia.gov'. (_ssl.c:1125)> ``` This is either not fixable, or could be fixed with a change to the url or (better) contacting the site owners to ask about some certificate or similar. The second case is when there is an http that needs to be https, which is a huge issue now, but hopefully not after this spack PR. ```bash $ spack audit packages-https xman Package "xman" uses http but has a valid https endpoint. ``` And then when a package is fixed: ```bash $ spack audit packages-https zlib PKG-HTTPS-DIRECTIVES: 0 issues found. ``` And that's mostly it. :) Signed-off-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com>
2021-09-01Add variant to allow unsupported compiler & CUDA combinations (#19736)David Beckingsale1-84/+82
Sometimes users need to be able to override the conflicts in `CudaPacakge`. This introduces a variant to enable/disable them.
2021-09-01Speed-up two unit tests by using builtin.mock instead of builtin (#25544)Harmen Stoppels1-2/+2
2021-08-30Add documentation on compiler `environment` (#25508)kwryankrattiger1-0/+28
2021-08-28Add a __reduce__ method to Spec (#25658)Massimiliano Culpo1-22/+38
* Add a __reduce__ method to Spec fixes #23892 The recursion limit seems to be due to the default way in which a Spec is serialized, following all the attributes. It's still not clear to me why this is related to being in an environment, but in any case we already have methods to serialize Specs to disk in JSON and YAML format. Here we use them to pickle a Spec instance too. * Downgrade to build-hash Hopefully nothing will change the package in between serializing the spec and sending it to the child process. * Add support for Python 2
2021-08-27Fix: --overwrite backs up old install dir, but it gets destroyed anyways ↵Harmen Stoppels3-5/+30
(#25583) * Make sure PackageInstaller does not remove the just-restored install dir after failure in spack install --overwrite * Remove cryptic error message and rethrow actual error
2021-08-27Load package environment prior to stand-alone/smoke test execution (#25619)Tamara Dahlgren1-0/+9
2021-08-27Add ld.gold and ld.lld compiler wrapper (#25626)Jordan Galby3-1/+3
The gcc compiler can be configured to use `ld.gold` by default. It will then call `ld.gold` explicitly when linking. When so, spack need to have a ld.gold wrapper in PATH to inject rpaths link flags etc... Also I wouldn't be surprised to see some package calling `ld.gold` directly. As for ld.gold, the argument could be made that we want to support any package that could call ld.lld.
2021-08-27Make `SpecBuildInterface` pickleable (#25628)Massimiliano Culpo1-1/+6
* Add a __reduce__ method to SpecBuildInterface This class was confusing pickle when being serialized, due to its scary nature of being an object that disguise as another type. * Add more MacOS tests, switch them to clingo * Fix condition syntax * Remove Python v3.6 and v3.9 with macOS
2021-08-26Make env (de)activate error with -e, -E, -D flags (#25625)Harmen Stoppels2-0/+33
* Make sure that spack -e. env activate b and spack -e. env deactivate error * Add a test