summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2018-10-22Bug fix: Module PATH check (#9574)Satish Balay1-6/+5
#9100 added a warning message when a path extracted from a module file did not appear to be a valid filesystem path. This check was applied to a variable which could be a list of paths, which would erroneously trigger the warning. This commit updates the check to run at the actual point where the path has been extracted.
2018-10-22license: exclude ignored files from the license check (#9578)Todd Gamblin1-1/+1
Previously, if you built the docs, you'd get license errors for generated .rst files. This removes them from the list of licensed files.
2018-10-18Allow setting language of compiler messages (#9486)Michael Kuhn2-0/+8
* Add a build_language config.yaml option which controls the language of compiler messages * build_language defaults to "C", in which case the compiler messages will be in English. This allows Spack log parsing to detect and highlight error messages (since the regular expressions to find error messages are in English) * The user can use the default language in their environment by setting the build_language config variable to null or ''
2018-10-17relicense: add `spack license` commandTodd Gamblin2-0/+224
- `spack license list-files`: list all files that should have license headers - `spack license list-lgpl`: list files still under LGPL-2.1 - `spack license verify`: check that license headers are correct - Added `spack license verify` to style tests
2018-10-17relicense: add Apache2/MIT header to `spack create`Todd Gamblin1-4/+12
2018-10-17relicense: replace LGPL headers with Apache-2.0/MIT SPDX headersTodd Gamblin364-7493/+1532
- remove the old LGPL license headers from all files in Spack - add SPDX headers to all files - core and most packages are (Apache-2.0 OR MIT) - a very small number of remaining packages are LGPL-2.1-only
2018-10-16docs: add note about compilers with modules (#9520)Mark W. Krentel1-0/+8
compilers.yaml can track a module that is needed for a compiler, but Spack does not fill this in automatically. This adds a note to the documentation informing the user how to do this.
2018-10-15meson: explicitly add --libdir (#9504)Michael Kuhn1-0/+6
If we do not specify libdir explicitly, Meson chooses something like lib/x86_64-linux-gnu, which causes problems when trying to find libraries and pkg-config files.
2018-10-12buildcache install: generate modules (#9509)Peter Scheibel1-0/+1
Spack packages installed using spack buildcache were not running post-install hooks, which create module files and manage licenses (if necessary). This was already occurring for Spack packages installed with spack install --use-cache
2018-10-11permissions: add permission configuration to packages.yaml (#8773)Greg Becker9-8/+335
Spack can now be configured to assign permissions to the files installed by a package. In the `packages.yaml` file under `permissions`, the attributes `read`, `write`, and `group` control the package permissions. These attributes can be set per-package, or for all packages under `all`. If permissions are set under `all` and for a specific package, the package-specific settings take precedence. The `read` and `write` attributes take one of `user`, `group`, and `world`. packages: all: permissions: write: group group: spack my_app: permissions: read: group group: my_team
2018-10-10Make CDash reporting easier to use (#9357)Zack Galbreath2-2/+16
* Better default CLI arguments for CDash reporting --log-format=cdash is now implied if you specify the --cdash-upload-url option to spack install. We also now default to writing CTest XML files to cdash_report/ when using the CDash reporter if no --log-file argument was specified. * Improved documentation on how to use the CDash reporter
2018-10-09Make builtin flag handlers available in package scope (#8668)Greg Becker4-38/+45
* Push default flag handlers into module scope * Preserve backwards compatibility of builtin flag handler names Ensure Spack continues to work for packages using the `Package.env_flags` idiom and equivalent. * update docs and tests to match * Update packages to match new syntax
2018-10-02Update Spack on Cray docs for singularity platform=linux workaround (#9398)Greg Becker1-0/+14
2018-09-27Module path parsing: CRAY_LD_LIBRARY_PATH and PACKAGE_DIR (#9374)Peter Scheibel2-3/+24
Fix two bugs with module file parsing: * Detection of the CRAY_LD_LIBRARY_PATH variable was broken by #9100. This fixes it and adds a test for it. * For module names like "foo-bar/1.0", the associated PACKAGE_DIR environment variable name would be "FOO_BAR_DIR", but Spack was not parsing the components and not converting "-" to "_"
2018-09-25Increase and customize lock timeouts (#9219)Peter Scheibel6-89/+204
Fixes #9166 This is intended to reduce errors related to lock timeouts by making the following changes: * Improves error reporting when acquiring a lock fails (addressing #9166) - there is no longer an attempt to release the lock if an acquire fails * By default locks taken on individual packages no longer have a timeout. This allows multiple spack instances to install overlapping dependency DAGs. For debugging purposes, a timeout can be added by setting 'package_lock_timeout' in config.yaml * Reduces the polling frequency when trying to acquire a lock, to reduce impact in the case where NFS is overtaxed. A simple adaptive strategy is implemented, which starts with a polling interval of .1 seconds and quickly increases to .5 seconds (originally it would poll up to 10^5 times per second). A test is added to check the polling interval generation logic. * The timeout for Spack's whole-database lock (e.g. for managing information about installed packages) is increased from 60s to 120s * Users can configure the whole-database lock timeout using the 'db_lock_timout' setting in config.yaml Generally, Spack locks (those created using spack.llnl.util.lock.Lock) now have no timeout by default This does not address implementations of NFS that do not support file locking, or detect cases where services that may be required (nfslock/statd) aren't running. Users may want to be able to more-aggressively release locks when they know they are the only one using their Spack instance, and they encounter lock errors after a crash (e.g. a remote terminal disconnect mentioned in #8915).
2018-09-24macos: add mojave (#9322)Denis Davydov1-1/+2
2018-09-19report error if failed process captures stderr (#9293)scheibelp1-6/+16
When a Spack Executable was configured to capture stderr and the process failed, the error messages of the process were discarded. This made it difficult to understand why the process failed. The exception is now updated to include the stderr of the process when the Executable captures stderr.
2018-09-18Fix version parsing for cistem package (#9260)Adam J. Stewart2-0/+3
Adds 'code' to the list of suffixes that are excluded from version parsing of URLs, such that if a URL contains the string 'cistem-1.0.0-beta-source-code', a version X will substitute in to produce a URL with cistem-X-source-code ('source' was already excluded). The 'cistem' package version is updated to make use of this (and fix a fetching bug with the cistem package). A unit test is added to check this parsing case.
2018-09-12Bug fix: module file path parsing (#9100)scheibelp2-15/+51
Improve Spack's parsing of module show to eliminate some false positives (e.g. accepting MODULEPATH when it is in fact looking for PATH). This makes the following changes: * Updates the pattern searching for several paths to avoid the case where they are prefixes of unwanted paths * Adds a warning message when an extracted path doesn't exist (which may help catch future module parsing bugs faster) * Adds a test with the content mentioned in #9083
2018-09-06test: Make flag_handlers tests use mock repo (#9132)Michael Kuhn1-9/+9
Without this change, openssl's new perl dependency makes the tests fail.
2018-09-05Spack environment updates take precedence (#9107)becker333-28/+64
Spack originally handled environment modifications in the following order: 1. clear environment variables (unless Spack was invoked with --dirty) 2. apply spack-specific environment variable updates, including variables set by Spack core like CC/PKG_CONFIG_PATH and those set by installed dependencies (e.g. in setup_dependent_environment) 3. load all external/compiler modules 1 and 2 were done together. This splits 1 into its own function and imposes the following order for environment modifications: 1. clear environment variables 2. load all external/compiler modules 3. apply spack-specific environment variable updates As a result, prepend-path actions taken by Spack (or installed Spack dependencies) take precedence over prepend-path actions from compiler and external modules. Additionally, when Spack (or a package dependency) sets/unsets an environment variable, that will override the actions of external/compiler modules.
2018-09-05Extended set of environment modification commands. (#8996)Sergey Kosukhin5-88/+217
2018-09-04Enable testing in parallel when using CMake. (#8484)Chris Green5-15/+44
* Add 'extra_env' argument to Executable.__call__: this will be added to the environment but does not affect whether the current environment is reused. If 'env' is not set, then the current environment is copied and the variables from 'extra_env' are added to it. * MakeExecutable can take a 'jobs_env' parameter that specifies the name of an environment variable used to set the level of parallelism. This is added to 'extra_env' (so does not affect whether the current environment is reused). * CMake-based Spack packages set 'jobs_env' when executing the 'test' target for make and ninja (which does not use -j)
2018-09-03Better warning message when fetching non-existent URLAdam J. Stewart1-7/+7
2018-08-29Update instructions to initialize Spack-global license files (#8991)Michael Sternberg1-44/+55
* Suggest adding contents before the explanatory comment to avoid issue #6534 * Mention that the license file may not need to be edited
2018-08-29Intel prefixes (#7469)Michael Sternberg8-98/+2885
Consolidate prefix calculation logic for intel packages into the IntelPackage class. Add documentation on installing Intel packages with Spack an (alternatively) adding them as external packages in Spack.
2018-08-29spack view suggests -I option for merge conflict (#9035)Andreas Baumbach1-4/+10
2018-08-27import spack load error message (#9088)Andreas Baumbach1-1/+1
It now additionally prints spec hash and options
2018-08-27Moved functions returning default scopes to spack.configMassimiliano Culpo7-34/+39
The functions returning the default scope to be modified or listed have been moved from spack.cmd to spack.config. Lmod now writes the guessed core compiler in the default modify scope instead of the 'site' scope.
2018-08-27Spack can guess lmod core compilers, if none is already presentMassimiliano Culpo2-5/+64
closes #8916 Currently Spack ends with an error if asked to write lmod modules files and the 'core_compilers' entry is not found in `modules.yaml`. After this PR an attempt will be made to guess that entry and the site configuration file will be updated accordingly. This is similar to what Spack already does to guess compilers on first run.
2018-08-25Update to latest version of pytest and dependencies (#9087)Adam J. Stewart1-1/+1
* Update to latest version of pytest and dependencies * Fix bug in installation tests
2018-08-25bugfix: 'submodules' attribute should imply git fetchTodd Gamblin2-1/+8
- 'submodules' wasn't properly declared as an optional attribute for git fetcher - add it and add a test.
2018-08-25commands: add `spack url stats` commandTodd Gamblin2-2/+94
This command prints out stats about all package versions, like so: $ spack url stats ==> 6070 total versions for 2827 packages: ------------------------------------ url 5411 89.1% schemes https 3822 70.6% http 1527 28.2% ftp 27 0.5% file 35 0.6% checksums sha512 6 0.1% sha256 163 3.0% sha1 81 1.5% md5 5161 95.4% ------------------------------------ svn 3 0.0% ------------------------------------ hg 5 0.1% ------------------------------------ go 1 0.0% ------------------------------------ git 650 10.7% security no commit 384 59.1% commit 266 40.9% ------------------------------------
2018-08-24Remove Python 3.3 from testing.Todd Gamblin2-5/+5
- Support for Python 3.3 isn't really needed, as nothing uses it as the default system Python, and nearly everyone will have a newer Python 3 version installed.
2018-08-22More consistent colored messages from activate/deactivate (#9036)Adam J. Stewart1-17/+16
* More consistent colored messages from activate/deactivate * Make 'activating' the default, make 'activated' the debug
2018-08-22Added a new target message to comply with newer versions of GNU Make (#9068)Massimiliano Culpo2-0/+3
#fixes 9067 Unit tests were failing on a system with GNU Make v 4.1
2018-08-21tests: add tests for warnings in build outputTodd Gamblin4-8/+85
- add tests for build warnings and error messages - simplify some code in build environment with plural() function
2018-08-21Display warnings if no errors are found in build logAdam J. Stewart1-1/+10
2018-08-21Fix error caused by missing spack-build.out build logAdam J. Stewart2-3/+3
2018-08-20yaml: use ruamel.yaml instead of pyyamlTodd Gamblin67-8129/+5165
- ruamel.yaml allows round-tripping comments from/to files - ruamel.yaml is single-source, python2/python3 compatible
2018-08-20Even better Makefile target parsing (#8819)Adam J. Stewart30-34/+301
#8223 replaced regex-based makefile target parsing with an invocation of "make -q". #8818 discovered that "make -q" can result in an error for some packages. Also, the "make -q" strategy relied on interpreting the error code, which only worked for GNU Make and not BSD Make (which was deemed acceptable at the time). As an added bonus, this implementation ignores the exit code and instead parses STDERR for any indications that the target does not exist; this works for both GNU Make and BSD Make. #8223 also updated ninja target detection to use "ninja -t targets". This does not change that behavior but makes it more-explicit with "ninja -t targets all" This also adds tests for detection of "make" and "ninja" targets.
2018-08-17install_tree: symlink handling and add 'ignore' option (#9019)scheibelp2-19/+86
Fixes #9001 #8289 added support for install_tree and copy_tree to merge into an existing directory structure. However, it did not properly handle relative symlinks and also removed support for the 'ignore' keyword. Additionally, some of the tests were overly-strict when checking the permissions on the copied files. This updates the install_tree/copy_tree methods and their tests: * copy_tree/install_tree now preserve relative link targets (if the symlink in the source directory structure is relative, the symlink created in the destination will be relative) * Added support for 'ignore' argument back to copy_tree/install_tree (removed in #8289). It is no longer the object output by shutil.ignore_patterns: you pass a function that accepts a path relative to the source and returns whether that path should be copied. * The openfoam packages (currently the only ones making use of the 'ignore' argument) are updated for the new API * When a symlink target is absolute, copy_tree and install_tree now rewrite the source prefix to be the destination prefix * copy_tree tests no longer check permissions: copy_tree doesn't enforce anything about permissions so its tests don't check for that * install_tree tests no longer check for exact permission matching since it can add file permissions
2018-08-16imports: spack uses importlib instead of imp when availableTodd Gamblin7-39/+199
- `imp` is deprecated and seems to have started having some weird issues on certain Linux versions. - In particular, the file argument to `load_source` is ignored on arch linux with Python 3.7. - `imp` is the only way to do imports in 2.6, so we'll keep it around for now and use it if importlib won't work. - `importlib` is the new import system, and it allows us to get lower-level access to the import implementation. - This consolidates all import logic into `spack.util.imp`, and make it use `importlib` if it's avialable.
2018-08-16Fix spack versions behavior when no URL (#8967)Adam J. Stewart3-12/+45
* Fix spack versions behavior when no URL * Unit test packages without URLs or safe versions
2018-08-15install_tree, copy_tree can install into existing directory structures (#8289)Adam J. Stewart3-73/+294
Replace use of `shutil.copytree` with `copy_tree` and `install_tree` functions in `llnl.util.filesystem`. - `copy_tree` copies without setting permissions. It should be used to copy files around in the build directory. - `install_tree` copies files and sets permissions. It should be used to copy files into the installation directory. - `install` and `copy` are analogous single-file functions. - add more extensive tests for these functions - update packages to use these functions.
2018-08-14tests: make the dependency patching test more completeTodd Gamblin2-3/+21
- dependency patching test didn't attempt to apply patches; just to see whether they were on the spec. - it applies the patch now and verifies that that patch was applied.
2018-08-10Branch with the meson build-system (#8467)Tom Merrick9-0/+309
* Branch with the meson build-system * Fix build_environment for dual loads and add create code * Add documentation * Fixed option list * Update build_system_guess for meson * Fixed documentation errors * Added meson to build and configure and updated documentation * fix typos
2018-08-09spec: make full_hash look like dag_hash (#8911)Scott Wittenburg1-2/+7
2018-08-09bugfix: cc handles spaces in flag variables properlyTodd Gamblin2-90/+82
- cc cleanup caused a parsing regression in flag handling - We added proper quoting to array expansions, but flag variables were never actually converted to arrays. Old code relied on this. This commit: - Adds reads to convert flags to arrays. - Makes the cc test check for improper space handling to prevent future regressions.
2018-08-08cc: refactor flag adding so that it's not in reverse orderTodd Gamblin2-29/+56
- flags were prepended in reverse order to args, but this makes it hard to see what order they'll be in on the final command line. - add them in the order they'll appear to make cc easier to maintain. - simplify code for assembling the command line - fix separator used in SPACK_SYSTEM_DIRS test