summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2019-10-07Spack environments can concretize specs together (#11372)Massimiliano Culpo4-4/+171
This PR adds a 'concretize' entry to an environment's spec.yaml file which controls how user specs are concretized. By default it is set to 'separately' which means that each spec added by the user is concretized separately (the behavior of environments before this PR). If set to 'together', the environment will concretize all of the added user specs together; this means that all specs and their dependencies will be consistent with each other (for example, a user could develop code linked against the set of libraries in the environment without conflicts). If the environment was previously concretized, this will re-concretize all specs, in which case previously-installed specs may no longer be used by the environment (in this sense, adding a new spec to an environment with 'concretize: together' can be significantly more expensive). The 'concretize: together' setting is not compatible with Spec matrices; this PR adds a check to look for multiple instances of the same package added to the environment and fails early when 'concretize: together' is set (to avoid confusing messages about conflicts later on).
2019-10-05doc: fix #12245 non-functional libdwarf dependency (#12515)Pariksheet Nanda1-4/+4
Applying accepted fix from spack/spack.io#4
2019-10-05Consistently support pkg-config files in share subdirectory (#12838)Michael Kuhn5-2/+7
While the build environment already takes share/pkgconfig into account, the generated module files etc. only consider lib/pkgconfig and lib64/pkgconfig.
2019-10-04bugfix: issue with custom dotkit root in config.yaml (#13046)Massimiliano Culpo2-1/+25
When removing support for dotkit in #11986 the code trying to set the paths of the various module files was not updated to skip it. This results in a failure because of a key error after the deprecation warning is displayed to user. This commit fixes the issue and adds a unit test for regression. Note that code for Spack chains has been updated accordingly but no unit test has been added for that case.
2019-10-03Update compilers.yaml location in Getting Started docs (#13029)Adam J. Stewart1-11/+14
2019-10-03Generic x86_64 code compiled with GCC uses non deprecated mtune flags (#13022)Massimiliano Culpo2-6/+15
fixes #12928
2019-10-02Remove support for generating dotkit files (#11986)Massimiliano Culpo24-256/+95
Dotkit is being used only at a few sites and has been deprecated on new machines. This commit removes all the code that provide support for the generation of dotkit module files. A new validator named "deprecatedProperties" has been added to the jsonschema validators. It permits to prompt a warning message or exit with an error if a property that has been marked as deprecated is encountered. * Removed references to dotkit in the docs * Removed references to dotkit in setup-env-test.sh * Added a unit test for the 'deprecatedProperties' schema validator
2019-10-02fujitsu compiler: Add 'required_libs'. (#13014)t-karatsu1-0/+2
2019-10-02Replace expensive store.reindex() call with db.add() call. (#13021)Patrick Gartung1-1/+1
2019-10-01'spack buildcache list' should show all buildaches available. (#13002)Patrick Gartung1-5/+1
* binary cache: show all packages for compatible differing targets * Don't restrict spack buildcache list to arch or os * Fix from merge conflict
2019-10-01Add all compatible system types directory to module pathsMassimiliano Culpo2-1/+14
fixes #12915 closes #12916 Since Spack has support for specific targets it might happen that software is built for targets that are not exactly the host because it was either an explicit user request or the compiler being used is too old to support the host. Modules for different targets are written into different directories and by default Spack was adding to MODULEPATH only the directory corresponding to the current host. This PR modifies this behavior to add all the directories that are **compatible** with the current host.
2019-10-01binary cache: show all packages for compatible differing targets (#12943)Greg Becker1-3/+6
2019-10-01When removing a file from a view, don't fail if it doesn't exist (#12960)Jeffrey Salmond1-0/+3
Sometimes when remove_file is called on a link, that link is missing (perhaps ctrl-C happened halfway through a previous action). As removing a non-existent file is no problem, this patch changes the behavior so Spack continues rather than stopping with an error. Currently you would see ValueError: /path/to/dir is not a link tree! and now it continues with a warning.
2019-09-29make license check slightly more lenientTodd Gamblin1-1/+1
bin/spack now needs to have a "-*- python -*-" line after the shebang, so that emacs will interpret it as a python file instead of as a shell script. Add one line to the license check limit to accommodate this.
2019-09-28Add all the 'generic' architectures that are mentioned in recipes (#12958)Massimiliano Culpo1-0/+35
LLVM, mesa and other packages check for these generic microarchitectures. One solution is to let Spack know they exist.
2019-09-26Fix perl build when using Build.PLGlenn P Johnson1-0/+12
This fixes #12852 where perl builds that use Build.PL will fail when the shebang of the Build script produced from the configure step is too long.
2019-09-26Relocate mach-o binaries using macholib on linux. (#12946)Patrick Gartung2-95/+74
Changes deps and rpaths for bins and libs, changes id for libs.
2019-09-26add --no-deps opt to `buildcache-create` (#12956)eugeneswalker1-0/+4
2019-09-26External: add macholib and altgraph needed to relocate Mach-o binaries on ↵Patrick Gartung25-0/+5425
Linux (#12909)
2019-09-24Fix "specific target" detection in Python 3 (#12906)Adam J. Stewart1-8/+5
The output of subprocess.check_output is a byte string in Python 3. This causes dictionary lookup to fail later on. A try-except around this function prevented this error from being noticed. Removed this so that more errors can propagate out.
2019-09-24Change get_patchelf to immediately return patchelf path if found (#12925)Patrick Gartung1-7/+7
2019-09-24Fujitsu compilers: added 'verbose_flag' method (#12922)t-karatsu1-0/+4
2019-09-24bugfix: use string keys to set preferred targets (#12921)Todd Gamblin4-9/+32
Preferred targets were failing because we were looking them up by Microarchitecture object, not by string. - [x] Add a call to `str()` to fix target lookup. - [x] Add a test to exercise this part of concretization. - [x] Add documentation for setting `target` in `packages.yaml`
2019-09-23AMD: fix architecture hierarchy (zen) (#12913)Massimiliano Culpo3-17/+45
* microarchitectures: zen starts from x86_64, not from excavator * Unit tests: fixed a test that is wrong with the new modeling * microarchitectures: fixed features and inheritance for 15h family bulldozer doesn't inherit from barcelona (10h) + added xop, lwp and tbm instruction sets to the 15h family (it distinguish the family from 17h)
2019-09-23Fix detection of Apple Clang 11.0.0 (#12912)Adam J. Stewart2-1/+6
2019-09-23tests: more template creation tests (#12882)Tamara Dahlgren1-8/+50
Addresses #12804 This PR adds the creation of the remaining (16) templates to ensure we can create them with expected content. The goal is to facilitate catching during testing.
2019-09-22externals: add note to jsonschema about modifications (#12895)Todd Gamblin1-1/+3
2019-09-21externals: avoid importing requests in jsonschemaTodd Gamblin1-4/+1
Spack doesn't need `requests`, and neither does `jsonschema`, but `jsonschema` tries to import it, and it'll succeed if `requests` is on your machine (which is likely, given how popular it is). This commit removes the import to improve Spack's startup time a bit. On a mac with SSD, the import of requests is ~28% of Spack's startup time when run as `spack --print-shell-vars sh,modules` (.069 / .25 seconds), which is what `setup-env.sh` runs. On a Linux cluster where Python is mounted from NFS, this reduces `setup-env.sh` source time from ~1s to .75s. Note: This issue will be eliminated if we upgrade to a newer `jsonschema` (we'd need to drop Python 2.6 for that). See https://github.com/Julian/jsonschema/pull/388.
2019-09-20Fix how 'gpg --list-secret-keys ...' output is parsedScott Wittenburg2-6/+77
2019-09-20targets: add mic_knl target to microarchitectures.jsonGregory Becker1-0/+37
- This is needed to support Cray machines -- we need an architecture mic_knl > x86_64 - We used Cray's naming scheme for this target to make it work seamlessly with the module-based detection sccheme on Cray. mic_knl is pretty much dead, so this will be the last succh target. We will need to work wtih Cray and other vendors in the future.
2019-09-20targets: adjust packages to use new specific targets semanticsMassimiliano Culpo1-1/+13
Seamless translation from 'target=<generic>' to either - target.family == <generic> (in methods) - 'target=<generic>:' (in directives) Also updated docs to show ranges in directives.
2019-09-20targets: Spack targets can now be fine-grained microarchitecturesMassimiliano Culpo50-540/+3040
Spack can now: - label ppc64, ppc64le, x86_64, etc. builds with specific microarchitecture-specific names, like 'haswell', 'skylake' or 'icelake'. - detect the host architecture of a machine from /proc/cpuinfo or similar tools. - Understand which microarchitectures are compatible with which (for binary reuse) - Understand which compiler flags are needed (for GCC, so far) to build binaries for particular microarchitectures. All of this is managed through a JSON file (microarchitectures.json) that contains detailed auto-detection, compiler flag, and compatibility information for specific microarchitecture targets. The `llnl.util.cpu` module implements a library that allows detection and comparison of microarchitectures based on the data in this file. The `target` part of Spack specs is now essentially a Microarchitecture object, and Specs' targets can be compared for compatibility as well. This allows us to label optimized binary packages at a granularity that enables them to be reused on compatible machines. Previously, we only knew that a package was built for x86_64, NOT which x86_64 machines it was usable on. Currently this feature supports Intel, Power, and AMD chips. Support for ARM is forthcoming. Specifics: - Add microarchitectures.json with descriptions of architectures - Relaxed semantic of compiler's "target" attribute. Before this change the semantic to check if a compiler could be viable for a given target was exact match. This made sense as the finest granularity of targets was architecture families. As now we can target micro-architectures, this commit changes the semantic by interpreting as the architecture family what is stored in the compiler's "target" attribute. A compiler is then a viable choice if the target being concretized belongs to the same family. Similarly when a new compiler is detected the architecture family is stored in the "target" attribute. - Make Spack's `cc` compiler wrapper inject target-specific flags on the command line - Architecture concretization updated to use the same algorithm as compiler concretization - Micro-architecture features, vendor, generation etc. are included in the package hash. Generic architectures, such as x86_64 or ppc64, are still dumped using the name only. - If the compiler for a target is not supported exit with an intelligible error message. If the compiler support is unknown don't try to use optimization flags. - Support and define feature aliases (e.g., sse3 -> ssse3) in microarchitectures.json and on Microarchitecture objects. Feature aliases are defined in targets.json and map a name (the "alias") to a list of rules that must be met for the test to be successful. The rules that are available can be extended later using a decorator. - Implement subset semantics for comparing microarchitectures (treat microarchitectures as a partial order, i.e. (a < b), (a == b) and (b < a) can all be false. - Implement logic to automatically demote the default target if the compiler being used is too old to optimize for it. Updated docs to make this behavior explicit. This avoids surprising the user if the default compiler is older than the host architecture. This commit adds unit tests to verify the semantics of target ranges and target lists in constraints. The implementation to allow target ranges and lists is minimal and doesn't add any new type. A more careful refactor that takes into account the type system might be due later. Co-authored-by: Gregory Becker <becker33.llnl.gov>
2019-09-20targets: first pass at target detection for linuxGregory Becker12-17/+350
Add llnl.util.cpu_name, with initial support for detecting different microarchitectures on Linux. This also adds preliminary changes for compiler support and variants to control the optimizatoin levels by target. This does not yet include translations of targets to particular compilers; that is left to another PR. Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2019-09-19Put back the use of otool and install_name_tool when running on macOS. Only ↵Patrick Gartung4-68/+112
use machotools on linux. (#12867) Move verbose messages to debug level get_patchelf should return None for test platform as well because create_buildinfo invokes patchelf to get rpaths.
2019-09-18Update buildcache creation and installation to allow mach-o binary ↵Patrick Gartung3-138/+200
relocation using py-machotools on linux or macos. (#12858) Update py-machotools dependencies and versions.
2019-09-17Support yaml paths anywhere specs are handled on CLI (#12561)Scott Wittenburg4-19/+318
Update command-line (CLI) parsing to understand references to yaml files that store Spack specs. Where a file reference is encountered, the full Spec in the file will be read in. A file reference may appear anywhere that a spec could appear before. For example, if you write "spack spec -y openmpi > openmpi.yaml" you may then install the spec using the yaml file by running "spack install ./openmpi.yaml"; you can also refer to dependencies in this way (e.g. "spack install foo^./openmpi.yaml"). There are two requirements for file references: * A file path entered on the CLI must include a "/" even if the file exists in your current working directory. For example, if you create an openmpi.yaml file as above and run "spack install openmpi.yaml" from the same directory, it will report an error. * A file path entered on the CLI must end with ".yaml" This commit adds error messages to clearly inform the user of both violations.
2019-09-17implicit rpaths filtering (#12789)Peter Scheibel14-52/+191
* implicit_rpaths are now removed from compilers.yaml config and are always instantiated dynamically, this occurs one time in the build_environment module * per-compiler list required libraries (e.g. libstdc++, libgfortran) and whitelist directories from rpaths including those libraries. Remove non-whitelisted implicit rpaths. Some libraries default for all compilers. * reintroduce 'implicit_rpaths' as a config variable that can be used to disable Spack insertion of compiler RPATHs generated at build time.
2019-09-17Fix generic body during package creation (#12804)Adam J. Stewart1-14/+14
Fixes incomplete change in #11981 Use the proper variable (`body_def`) during package creation for package subclasses.
2019-09-16Allow build to proceed with failed compiler lib detection (#12756)Adam J. Stewart1-0/+4
Fixes #12732 Fixes #12767 c22a145 added automatic detection and RPATHing of compiler libraries to Spack builds. However, in cases where the parsing/detection logic fails this was terminating the build. This makes the compiler library detection "best-effort" and reports an issue when the detection fails rather than terminating the build.
2019-09-16Quote shell variable values (#12759)Adam J. Stewart1-2/+2
This updates logic which sets shell variables to quote the values, which is necessary when the value contains a space (e.g. PATH).
2019-09-13Fixes identified in ecp facilities hackathon fixes:Scott Wittenburg2-3/+2
- improve mirror git repo management - minio s3 implementation needs endpoint_url - Remove image key from rebuild-index job - Remove image, rely on tags instead
2019-09-13Implement an optional compiler bootstrapping phaseScott Wittenburg7-251/+515
2019-09-13Remove CombinatorialSpecSet in favor of environments + stacksScott Wittenburg9-705/+83
2019-09-13Refactor release-jobs cmd based on use of environments (no docker either)Scott Wittenburg2-291/+173
2019-09-13Add schema for new gitlab-ci and cdash additions to environmentScott Wittenburg4-0/+220
2019-09-11module parsing: make heuristic to get paths from modules more robust (#12693)Greg Becker2-36/+87
* module parsing: make heuristic to get paths from modules more robust * refactor module parsing
2019-09-09clang: add another regex for version detection (#12769)Massimiliano Culpo2-0/+5
This is similar to #10191. The Ubuntu package for clang 8.0.0 displays a very unusual version string, and we need this new regex to detect it as just 8.0.0 Unit test have been complemented by the output that was failing detection.
2019-09-08Fix PyQt installation (#12479)Adam J. Stewart2-15/+41
* Fix PyQt installation * Switch dependency type * Replace SIP dependency with resource * Relax py-pyqt4 Qt dependency
2019-09-07flake8: make entire repository pass `spack flake8 -a`Todd Gamblin8-45/+46
- Fix trailing whitespace missed by the bug described in #12755. - Fix other style issues that have crept in over time (this can happen when flake8 adds new checks with new versions)
2019-09-07flake8: only add E501 exemptions when absolutely necessary (#12755)Todd Gamblin1-2/+13
E501 (line too long) exemptions are probably our most common ones -- we add them for directives, URLs, hashes, etc. in packages. But we currently add them even when a line *doesn't* need them, which can mask trailing whitespace errors. This changes `spack flake8` so that it will only add E501 exemptions if the line is *actually* too long. Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>