summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2019-10-23Preserve comments for Spack YAML objects (#11602)Todd Gamblin13-178/+171
This updates the configuration loading/dumping logic (now called load_config/dump_config) in spack_yaml to preserve comments (by using ruamel.yaml's RoundTripLoader). This has two effects: * environment spack.yaml files expect to retain comments, which load_config now supports. By using load_config, users can now use the ':' override syntax that was previously unavailable for environment configs (but was available for other config files). * config files now retain user comments by default (although in cases where Spack updates/overwrites config, the comments can still be removed). Details: * Subclasses `RoundTripLoader`/`RoundTripDumper` to parse yaml into ruamel's `CommentedMap` and analogous data structures * Applies filename info directly to ruamel objects in cases where the updated loader returns those * Copies management of sections in `SingleFileScope` from #10651 to allow overrides to occur * Updates the loader/dumper to handle the processing of overrides by specifically checking for the `:` character * Possibly the most controversial aspect, but without that, the parsed objects have to be reconstructed (i.e. as was done in `mark_overrides`). It is possible that `mark_overrides` could remain and a deep copy will not cause problems, but IMO that's generally worth avoiding. * This is also possibly controversial because Spack YAML strings can include `:`. My reckoning is that this only occurs for version specifications, so it is safe to check for `endswith(':') and not ('@' in string)` * As a consequence, this PR ends up reserving spack yaml functions load_config/dump_config exclusively for the purpose of storing spack config
2019-10-22tests: avoid extra output in `env status` test (#13344)Todd Gamblin1-9/+12
`test_envoronment_status()` was printing extra output during tests. - [x] disable output only for `env('status')` calls instead of disabling it for the whole test.
2019-10-22env activation: use package defined env setup methods (#13249)Greg Becker5-44/+189
This PR ensures that environment activation sets all environment variables set by the equivalent `module load` operations, except that the spec prefixes are "rebased" to the view associated with the environment. Currently, Spack blindly adds paths relative to the environment view root to the user environment on activation. Issue #12731 points out ways in which this behavior is insufficient. This PR changes that behavior to use the `setup_run_environment` logic for each package to augment the prefix inspections (as in Spack's modulefile generation logic) to ensure that all necessary variables are set to make use of the packages in the environment. See #12731 for details on the previous problems in behavior. This PR also updates the `ViewDescriptor` object in `spack.environment` to have a `__contains__` method. This allows for checks like `if spec in self.default_view`. The `__contains__` operator for `ViewDescriptor` objects checks whether the spec satisfies the filters of the View descriptor, not whether the spec is already linked into the underlying `FilesystemView` object.
2019-10-22microarchitectures: look in /sbin and /usr/sbin for sysctl (#13365)Massimiliano Culpo3-22/+30
This PR ensures that on Darwin we always append /sbin and /usr/sbin to PATH, if they are not already present, when looking for sysctl. * Make sure we look into /sbin and /usr/sbin for sysctl * Refactor sysctl for better readability * Remove marker to make test pass
2019-10-22Fixed optimization flags support for old GCC versions (#13362)Massimiliano Culpo4-22/+24
These changes update our gcc microarchitecture descriptions based on manuals found here https://gcc.gnu.org/onlinedocs/ and assuming that new architectures are not added during patch releases.
2019-10-22microarchitectures: add optimization flags for Intel compilers (#13345)Massimiliano Culpo2-4/+134
* Added optimization flags for Intel compilers with Intel CPUs * Added optimization flags for Intel compilers with AMD CPUs
2019-10-22fetching: S3 upload and download (#11117)Omar Padron19-288/+1414
This extends Spack functionality so that it can fetch sources and binaries from-, push sources and binaries to-, and index the contents of- mirrors hosted on an S3 bucket. High level to-do list: - [x] Extend mirrors configuration to add support for `file://`, and `s3://` URLs. - [x] Ensure all fetching, pushing, and indexing operations work for `file://` URLs. - [x] Implement S3 source fetching - [x] Implement S3 binary mirror indexing - [x] Implement S3 binary package fetching - [x] Implement S3 source pushing - [x] Implement S3 binary package pushing Important details: * refactor URL handling to handle S3 URLs and mirror URLs more gracefully. - updated parse() to accept already-parsed URL objects. an equivalent object is returned with any extra s3-related attributes intact. Objects created with urllib can also be passed, and the additional s3 handling logic will still be applied. * update mirror schema/parsing (mirror can have separate fetch/push URLs) * implement s3_fetch_strategy/several utility changes * provide more feature-complete S3 fetching * update buildcache create command to support S3 * Move the core logic for reading data from S3 out of the s3 fetch strategy and into the s3 URL handler. The s3 fetch strategy now calls into `read_from_url()` Since read_from_url can now handle S3 URLs, the S3 fetch strategy is redundant. It's not clear whether the ideal design is to have S3 fetching functionality in a fetch strategy, directly implemented in read_from_url, or both. * expanded what can be passed to `spack buildcache` via the -d flag: In addition to a directory on the local filesystem, the name of a configured mirror can be passed, or a push URL can be passed directly.
2019-10-21Bootstrap environment modules optimizing for generic architectures (#13105)Massimiliano Culpo2-3/+11
fixes #13073 Since #3206 was merged bootstrapping environment-modules was using the architecture of the current host or the best match supported by the default compiler. The former case is an issue since shell integration was looking for a spec targeted at the host microarchitecture. 1. Bootstrap an env modules targeted at generic architectures 2. Look for generic targets in shell integration scripts 3. Add a new entry in Travis to test shell integration
2019-10-21microarchitectures: fix custom compiler versions (#13222)Massimiliano Culpo6-5/+100
Custom string versions for compilers were raising a ValueError on conversion to int. This commit fixes the behavior by trying to detect the underlying compiler version when in presence of a custom string version. * Refactor code that deals with custom versions for better readability * Partition version components with a regex * Fix semantic of custom compiler versions with a suffix * clang@x.y-apple has been special-cased * Add unit tests
2019-10-20spack create: http -> https (#13269)Adam J. Stewart1-1/+1
2019-10-19shebangs: don't warn when patching long shebangs (#13266)Todd Gamblin1-1/+1
We've been doing this for quite a while now, and it does not seem to cause issues. - [x] Switch the noisy warning to a debug to make Spack a bit quieter while building.
2019-10-19Microarchitecture specific optimizations for LLVM (#13250)Massimiliano Culpo9-10/+292
* Added architecture specific optimization flags for Clang / LLVM * Disallow compiler optimizations for mixed toolchains * We emit a warning when building for a mixed toolchain * Fixed issues with suffixed versions of compilers; Apple's Clang will, for the time being, fall back on x86-64 for every compilation.
2019-10-17satisfaction checks on dependencies should be strict when the parent check ↵Peter Scheibel1-1/+1
is strict (#13243)
2019-10-17Separate setting build environment and run environment in packages (#11115)Massimiliano Culpo8-160/+275
* Methods setting the environment now do it separately for build and run Before this commit the `*_environment` methods were setting modifications to both the build-time and run-time environment simultaneously. This might cause issues as the two environments inherently rely on different preconditions: 1. The build-time environment is set before building a package, thus the package prefix doesn't exist and can't be inspected 2. The run-time environment instead is set assuming the target package has been already installed Here we split each of these functions into two: one setting the build-time environment, one the run-time. We also adopt a fallback strategy that inspects for old methods and executes them as before, but prints a deprecation warning to tty. This permits to port packages to use the new methods in a distributed way, rather than having to modify all the packages at once. * Added a test that fails if any package uses the old API Marked the test xfail for now as we have a lot of packages in that state. * Added a test to check that a package modified by a PR is up to date This test can be used any time we deprecate a method call to ensure that during the first modification of the package we update also the deprecated calls. * Updated documentation
2019-10-17multimethods: avoid calling caller_locals() in Python 3 (#13238)Todd Gamblin2-10/+35
Python 3 metaclasses have a `__prepare__` method that lets us save the class's dictionary before it is constructed. In Python 2 we had to walk up the stack using our `caller_locals()` method to get at this. Using `__prepare__` is much faster as it doesn't require us to use `inspect`. This makes multimethods use the faster `__prepare__` method in Python3, while still using `caller_locals()` in Python 2. We try to reduce the use of caller locals using caching to speed up Python 2 a little bit.
2019-10-16importing: make importlib_importer recognize .pyc cache (#13239)Todd Gamblin1-0/+6
Our importer was always parsing from source (which is considerably slower) because the source size recorded in the .pyc file differed from the size of the input file. Override path_stats in the prepending importer to fool it into thinking that the source size is the size *with* the prepended code.
2019-10-16filter_file: fix multiple invocations on the same file (#13234)Michael Kuhn2-2/+33
Since the backup file is only created on the first invocation, it will contain the original file without any modifications. Further invocations will then read the backup file, effectively reverting prior invocations. This can be reproduced easily by trying to install likwid, which will try to install into /usr/local. Work around this by creating a temporary file to read from.
2019-10-16Bugfix: stage directory permissions and cleaning (#12733)Tamara Dahlgren9-83/+297
* This updates stage names to use "spack-stage-" as a prefix. This avoids removing non-Spack directories in "spack clean" as c141e99 did (in this case so long as they don't contain the prefix "spack-stage-"), and also addresses a follow-up issue where Spack stage directories were not removed. * Spack now does more-stringent checking of expected permissions for staging directories. For a given stage root that includes a user component, all directories before the user component that are created by Spack are expected to match the permissions of their parent; the user component and all deeper directories are expected to be accessible to the user (read/write/execute).
2019-10-15Feature: installed file verification (#12841)Greg Becker13-54/+791
This feature generates a verification manifest for each installed package and provides a command, "spack verify", which can be used to compare the current file checksums/permissions with those calculated at installed time. Verification includes * Checksums of files * File permissions * Modification time * File size Packages installed before this PR will be skipped during verification. To verify such a package you must reinstall it. The spack verify command has three modes. * With the -a,--all option it will check every installed package. * With the -f,--files option, it will check some specific files, determine which package they belong to, and confirm that they have not been changed. * With the -s,--specs option or by default, it will check some specific packages that no files havae changed.
2019-10-15lmod: module files are written in a root folder named by target family (#13121)Massimiliano Culpo2-1/+21
fixes #13005 This commit fixes an issue with the name of the root directory for module file hierarchies. Since #3206 the root folder was named after the microarchitecture used for the spec, which is too specific and not backward compatible for lmod hierarchies. Here we compute the root folder name using the target family instead of the target name itself and we add target information in the 'whatis' portion of the module file.
2019-10-14filter_file uses "surrogateescape" error handling (#12765)Massimiliano Culpo4-5/+1600
From Python docs: -- 'surrogateescape' will represent any incorrect bytes as code points in the Unicode Private Use Area ranging from U+DC80 to U+DCFF. These private code points will then be turned back into the same bytes when the surrogateescape error handler is used when writing data. This is useful for processing files in an unknown encoding. -- This will allow us to process files with unknown encodings. To accommodate the case of self-extracting bash scripts, filter_file can now stop filtering text input if a certain marker is found. The marker must be passed at call time via the "stop_at" function argument. At that point the file will be reopened in binary mode and copied verbatim. * use "surrogateescape" error handling to ignore unknown chars * permit to stop filtering if a marker is found * add unit tests for non-ASCII and mixed text/binary files
2019-10-14bugfix: install --only dependencies works in env (#13090)Greg Becker3-18/+70
* bugfix: install --only dependents works in env includes regression testing
2019-10-14commands: add resource stats to `spack url stats` (#13205)Todd Gamblin2-65/+93
`spack url stats` now also looks at packages' resources when outputting statistics. Example: ``` $ spack url stats ==> URL stats for 3531 packages: -------------------------------------------------------------- stat versions % resources % -------------------------------------------------------------- url 8335 89.3% 339 89.0% schemes https 6489 69.5% 93 24.4% ftp 32 0.3% 8 2.1% http 1763 18.9% 237 62.2% file 51 0.5% 1 0.3% checksums md5 26 0.3% 0 0.0% sha256 8306 89.0% 336 88.2% no checksum 3 0.0% 3 0.8% -------------------------------------------------------------- go 1 0.0% 0 0.0% -------------------------------------------------------------- hg 7 0.1% 0 0.0% -------------------------------------------------------------- no code 4 0.0% 0 0.0% -------------------------------------------------------------- svn 4 0.0% 16 4.2% -------------------------------------------------------------- git 981 10.5% 26 6.8% branch 442 4.7% 4 1.0% commit 362 3.9% 14 3.7% no ref 36 0.4% 2 0.5% tag 141 1.5% 6 1.6% -------------------------------------------------------------- ```
2019-10-14Fix svn tests, skip if svnadmin not found (#13137)Adam J. Stewart2-2/+4
2019-10-12install: add --cache-only option (#12729)Greg Becker4-3/+22
* add `--cache-only` option to install * testing for `--cache-only` * remove extraneous stage creation at stage destroy time
2019-10-12checksums: enforce that all mainline packages use sha256 checksumsTodd Gamblin2-2/+45
- Add a test that verifies checksums on all packages - Also add an attribute to packages that indicates whether they need a manual download or not, and add an exception in the tests for these packages until we can verify them.
2019-10-11Autotools build system to patch config.guess based on a range checkMassimiliano Culpo1-2/+3
2019-10-11Fixed options to compile generic code on ppc64 and ppc64leMassimiliano Culpo1-0/+2
2019-10-11Fix python3 errors from string and byte concatenation (#13141)Patrick Gartung2-32/+47
2019-10-10tests: cleanup config:build_stage handling (fixes #12651, #12798)Tamara Dahlgren5-190/+119
2019-10-10Add support for nested "overrides" scopes.Tamara Dahlgren2-2/+72
2019-10-10Added NEON to the list of features required for the aarch64 familyMassimiliano Culpo2-0/+7
Both floating-point and NEON are required in all standard ARMv8 implementations. Theoretically though specialized markets can support no NEON or floating-point at all. Source: https://developer.arm.com/docs/den0024/latest/aarch64-floating-point-and-neon On the other hand the base procedure call standard for Aarch64 "assumes the availability of the vector registers for passing floating-point and SIMD arguments". Further "the Arm 64-bit architecture defines two mandatory register banks: a general-purpose register bank which can be used for scalar integer processing and pointer arithmetic; and a SIMD and Floating-Point register bank". Source: https://developer.arm.com/docs/ihi0055/latest/procedure-call-standard-for-the-arm-64-bit-architecture This makes customization of Aarch64 with no NEON instruction set available so unlikely that we can consider them a feature of the generic family.
2019-10-10ArchSpec: fix constraint satisfaction for targetsMassimiliano Culpo2-0/+9
fixes #13111 Due to a missing case we were treating a single target that was not equal to the one we were comparing to as a range open on the right.
2019-10-09Buildcache: pass string.encode('utf-8') for old_dir and new_dir to ↵Patrick Gartung1-1/+2
replace_prefix_bin. (#13114) This should fix a Python3 error from concatenating strings and bytes.
2019-10-09"No Spack mirror configured": demoted the warning to a debug message (#13082)Massimiliano Culpo1-1/+1
fixes #12010
2019-10-07Add macOS Catalina support (#13070)Adam J. Stewart1-12/+22
2019-10-07Spack environments can concretize specs together (#11372)Massimiliano Culpo4-4/+171
This PR adds a 'concretize' entry to an environment's spec.yaml file which controls how user specs are concretized. By default it is set to 'separately' which means that each spec added by the user is concretized separately (the behavior of environments before this PR). If set to 'together', the environment will concretize all of the added user specs together; this means that all specs and their dependencies will be consistent with each other (for example, a user could develop code linked against the set of libraries in the environment without conflicts). If the environment was previously concretized, this will re-concretize all specs, in which case previously-installed specs may no longer be used by the environment (in this sense, adding a new spec to an environment with 'concretize: together' can be significantly more expensive). The 'concretize: together' setting is not compatible with Spec matrices; this PR adds a check to look for multiple instances of the same package added to the environment and fails early when 'concretize: together' is set (to avoid confusing messages about conflicts later on).
2019-10-05doc: fix #12245 non-functional libdwarf dependency (#12515)Pariksheet Nanda1-4/+4
Applying accepted fix from spack/spack.io#4
2019-10-05Consistently support pkg-config files in share subdirectory (#12838)Michael Kuhn5-2/+7
While the build environment already takes share/pkgconfig into account, the generated module files etc. only consider lib/pkgconfig and lib64/pkgconfig.
2019-10-04bugfix: issue with custom dotkit root in config.yaml (#13046)Massimiliano Culpo2-1/+25
When removing support for dotkit in #11986 the code trying to set the paths of the various module files was not updated to skip it. This results in a failure because of a key error after the deprecation warning is displayed to user. This commit fixes the issue and adds a unit test for regression. Note that code for Spack chains has been updated accordingly but no unit test has been added for that case.
2019-10-03Update compilers.yaml location in Getting Started docs (#13029)Adam J. Stewart1-11/+14
2019-10-03Generic x86_64 code compiled with GCC uses non deprecated mtune flags (#13022)Massimiliano Culpo2-6/+15
fixes #12928
2019-10-02Remove support for generating dotkit files (#11986)Massimiliano Culpo24-256/+95
Dotkit is being used only at a few sites and has been deprecated on new machines. This commit removes all the code that provide support for the generation of dotkit module files. A new validator named "deprecatedProperties" has been added to the jsonschema validators. It permits to prompt a warning message or exit with an error if a property that has been marked as deprecated is encountered. * Removed references to dotkit in the docs * Removed references to dotkit in setup-env-test.sh * Added a unit test for the 'deprecatedProperties' schema validator
2019-10-02fujitsu compiler: Add 'required_libs'. (#13014)t-karatsu1-0/+2
2019-10-02Replace expensive store.reindex() call with db.add() call. (#13021)Patrick Gartung1-1/+1
2019-10-01'spack buildcache list' should show all buildaches available. (#13002)Patrick Gartung1-5/+1
* binary cache: show all packages for compatible differing targets * Don't restrict spack buildcache list to arch or os * Fix from merge conflict
2019-10-01Add all compatible system types directory to module pathsMassimiliano Culpo2-1/+14
fixes #12915 closes #12916 Since Spack has support for specific targets it might happen that software is built for targets that are not exactly the host because it was either an explicit user request or the compiler being used is too old to support the host. Modules for different targets are written into different directories and by default Spack was adding to MODULEPATH only the directory corresponding to the current host. This PR modifies this behavior to add all the directories that are **compatible** with the current host.
2019-10-01binary cache: show all packages for compatible differing targets (#12943)Greg Becker1-3/+6
2019-10-01When removing a file from a view, don't fail if it doesn't exist (#12960)Jeffrey Salmond1-0/+3
Sometimes when remove_file is called on a link, that link is missing (perhaps ctrl-C happened halfway through a previous action). As removing a non-existent file is no problem, this patch changes the behavior so Spack continues rather than stopping with an error. Currently you would see ValueError: /path/to/dir is not a link tree! and now it continues with a warning.
2019-09-29make license check slightly more lenientTodd Gamblin1-1/+1
bin/spack now needs to have a "-*- python -*-" line after the shebang, so that emacs will interpret it as a python file instead of as a shell script. Add one line to the license check limit to accommodate this.