summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2017-10-24add --all option to `spack flake8`Todd Gamblin1-14/+41
- `-a`/`--all` causes flake8 to run on all files rather than just different ones.
2017-10-23package: remove bare except statements (#5896)scheibelp1-2/+2
2017-10-20getting_started.rst: removed tip suggesting the use of 2 Spack instances (#4061)Massimiliano Culpo1-13/+0
2017-10-20module-type argument defaults to 'tcl' instead of every known type (#5751)Massimiliano Culpo3-3/+6
fixes #5721 This is to solve the fact that lmod needs to be configured by the user (to specify a core compiler).
2017-10-20buildcache command updates (#5860)Patrick Gartung3-37/+159
* Add better spec matching to spack buildcache * skip download of spec.yaml and keys if they exist * autopep8
2017-10-20Update Getting Started docs to clarify that full Xcode suite is required for ↵Adam J. Stewart2-6/+79
qt (#4958) * Update Getting Started docs to clarify that full Xcode suite is required for qt * Better error message when only the command-line tools are installed
2017-10-19spack blame can take a path to a file in the Spack repo (#5793)Todd Gamblin2-7/+28
- Previously `spack blame` only worked for package names; now it works for paths as well, so developers can use it on core spack files.
2017-10-19Allow documentation to build in Python 3 (#5736)Adam J. Stewart1-7/+2
2017-10-17filter_file, don't remove absent backup file (#5733)George Hartzell1-1/+1
I'm tracking down a problem with the perl package that's been generating this error: ``` OSError: OSError: [Errno 2] No such file or directory: '/blah/blah/blah/lib/5.24.1/x86_64-linux/Config.pm~' ``` The real problem is upstream, but it's being masked by an exception raised in `filter_file`s finally block. In my case, `backup` is `False`. The backup is created around line 127, the `re.sub()` calls fails (working on that), the `except` block fires and moves the backup file back, then the finally block tries to remove the non-existent backup file. This change just avoids trying to remove the non-existent file.
2017-10-17Minor typos and corections (#5789)Michael F. Herbst1-2/+2
- The shell script uses arrays and hence only works on sophisticated shells and not the default `sh`. For clarity the shebang `#!/bin/bash` has been used instead.
2017-10-17Exercise more code paths in the git fetcher.Todd Gamblin1-7/+35
- This fakes out GitFetchStrategy to try code paths for different git versions. - This allows us to test code paths for old versions using a newer git version.
2017-10-17Spack tests no longer clutter var/spack/stageTodd Gamblin5-106/+153
- Tests use a session-scoped mock stage directory so as not to interfere with the real install. - Every test is forced to clean up after itself with an additional check. We now automatically assert that no new files have been added to `spack.stage_path` during each test. - This means that tests that fail installs now need to clean up their stages, but in all other cases the check is useful.
2017-10-17``Package.stage`` no longer implicitly makes stage directoryTodd Gamblin1-9/+9
- Be explicit about stage creation during the install process. - This avoids accidental creation of stages - e.g., during `spack install --fake`, stages were erroneously recreated after being destroyed
2017-10-17Spack tests no longer use the var/spack/stage directory.Todd Gamblin1-9/+17
- This prevents the main spack install from being clusttered by invocations of `spack test`. - This uses a session-scoped stage fixture to ensure tests don't interfere.
2017-10-17Spack core and tests no longer use `os.chdir()`Todd Gamblin14-450/+393
- Spack's core package interface was previously overly stateful, in that calling methods like `do_stage()` could change your working directory. - This removes Stage's `chdir` and `chdir_to_source` methods and replaces their usages with `with working_dir(stage.path)` and `with working_dir(stage.source_path)`, respectively. These ensure the original working directory is preserved. - This not only makes the API more usable, it makes the tests more deterministic, as previously a test could leave the current working directory in a bad state and cause subsequent tests to fail mysteriously.
2017-10-14Ensure that FileList makes it into the documentation (#5739)Adam J. Stewart1-0/+1
2017-10-13Clean up logic in Sepc.satisfies_dependencies()Todd Gamblin1-5/+9
- This puts in a fast path when there are no dependencies to satisfy. - Reduces time spent to concretize r-rminer by 2x, down to 5s from 10s
2017-10-13Remove single-root assertion from Spec.rootTodd Gamblin2-11/+6
- Assertion would search for root through all possible paths. - It's also really slow. - This isn't needed anymore. We're pretty good at ensuring single-rooted DAGs, and this assertion has never been thrown. - This shaves another 6 seconds off r-rminer concretization
2017-10-13Use list instead of OrderedDict to find virtual/external candidatesTodd Gamblin1-6/+3
- This reduces concretization time for r-rminer from over 1 minute to only 16 seconds. - OrderedDict ends up taking a *lot* of time to compare larger specs. - An OrderedDict isn't actually needed here. It's actually not possible to find duplicates, and we end up sorting the contents of the OrderedDict anyway.
2017-10-13avoid creating lots of dictionaries in traverse_edges()Todd Gamblin1-15/+14
- This is an optimization to the way traverse_edges iterates over successors. - Previous version called dependencies_dict(), which involved a lot of redundant work (creating dicts and calling caonical_deptype)
2017-10-13Don't redundantly check for providers of non-virtuals.Todd Gamblin1-0/+3
2017-10-13Cache compilers parsed from config filesTodd Gamblin1-29/+39
- Spack ends up constructing compilers frequently from YAML data. - This caches the result of parsing the compiler config - The logic in compilers/__init__.py could use a bigger cleanup, but this makes concretization much faster for now. - on macOS, this also ensures that xcrun is called only twice, as opposed to every time a new compiler object is constructed.
2017-10-13This fixes a bug in creating rpaths relative to $ORIGIN on linux. (#5726)Patrick Gartung3-29/+34
* This fixes a bug in creating rpaths relative to on linux. * fix for macOS as well * found in testing * flake8 * fix testing on macOS * flake8
2017-10-13Docker Workflow (#5582)Axel Huebl1-0/+143
This adds a workflow section on how to use spack on Docker. It provides an example on the best-practices I collected over the last months and circumvents the common pitfalls I tapped in. Works with MPI, CUDA, Modules, execution as root, etc. Background: Developed initially for PIConGPU.
2017-10-12various fixes for macOS high sierra (#5647)Denis Davydov1-1/+10
* various fixes for macOS high sierra * add macOS_version() helper function * flake8 fixes * update oce and trilinos * fix bison
2017-10-12Make --trusted default when running spack gpg list (#5678)David Hows2-1/+2
* Make --trusted default when running spack gpg list Currently running `spack gpg list` with no arguments returns nothing. You must supply either the `--trusted` or the `--signing` options. The idea here is to return some initial data to the user when the command is run. The alternative is to return an error, telling the user to select one of the two options. * Add an extra test case for the empty list command Fixes the issue with code coverage
2017-10-12Speed up concretization (#5716)Todd Gamblin4-17/+23
This isn't a rework of the concretizer but it speeds things up a LOT. The main culprits were: 1. Variant code, `provider_index`, and `concretize.py` were calling `spec.package` when they could use `spec.package_class` - `spec.package` looks up a package instance by `Spec`, which requires a (fast-ish but not that fast) DAG compare. - `spec.package_class` just looks up the package's class by name, and you should use this when all you need is metadata (most of the time). - not really clear that the current way packages are looked up is necessary -- we can consider refactoring that in the future. 2. `Repository.repo_for_pkg` parses a `str` argument into a `Spec` when called with one, via `@_autospec`, but this is not needed. - Add some faster code to handle strings directly and avoid parsing This speeds up concretization 3-9x in my limited tests. Still not super fast but much more bearable: Before: - `spack spec xsdk` took 33.6s - `spack spec dealii` took 1m39s After: - `spack spec xsdk` takes 6.8s - `spack spec dealii` takes 10.8s
2017-10-12Better install output (#5714)Todd Gamblin6-20/+17
* Do not call setup_package for fake installs - setup package could fail if ``setup_dependent_environment`` or other routines expected to use executables from dependencies - xpetsc and boost try to get python config variables in `setup_dependent_package`; this would cause them not to be fake-installable * Remove vestigial deptype_query argument to Spec.traverse() - The `deptype_query` argument isn't used anymore -- it's only passed around and causes confusion when calling traverse. - Get rid of it and just keep the `deptypes` argument * Don't print redundant messages when installing dependencies - `do_install()` was originally depth-first recursive, and printed "<pkg> already installed in ..." multiple times for packages as recursive calls encountered them. - For much cleaner output, use spec.traverse(order='post') to install dependencies instead
2017-10-10compilers/clang: add flang (#5503)Christoph Junghans3-10/+16
* compilers/clang: add flang * Update clang.py
2017-10-09Add package for aspell and ass't dictionaries (#3890)George Hartzell2-0/+59
* Add package for aspell and ass't dictionaries Add a package definition for aspell. Add a handful of dictionaries to convince myself that the support for a bunch of dictionaries works. * Flake8 cleanup * Use six's version of urlparse `urlparse` is not python3 friendly. This works around it (stolen from `.../cmd/md5.py`). * Fix incorrect trimming regexp * Clean up dictionary build - more parsimonious use of `which` (`make()` has already been made) - use `sh` instead of `bash` * Use a helper method to generate info for variants I figured out my issues with static methods. I *think* that it this is pythonic. * Convert aspell to an extendable package Convert aspell to be extendable and rework the dictionaries to be extensions. As it stands, there's a great deal of cut and paste in the dictionaries, I'll abstract that out next. The {de,}activate methods copy a great deal of code out of package.py. Perhaps there's a better way.... * Create AspellDictPackage and use it for the dictionaries Reduce the repeated code, pull it into a base class. I'm confused about why 'from spack import *' wasn't more useful in the base class. * Oops, -de & -es should be AspellDictPackages too * Typo: pakcage -> package * Address some commentary * Update copyright dates, 2016->2017
2017-10-09Fix setup for changes to dirty flag (#5592)becker332-3/+7
* Pass dirty arg through setup to setup_package
2017-10-06Don't check package.installed in _mark_concrete if value=True (#5634)scheibelp2-1/+18
* spec and spec.package.spec can refer to different objects in the database. When these two instances of spec differ in terms of the value of the 'concrete' property, Spec._mark_concrete can fail when checking Spec.package.installed (which requires package.spec to be concrete). This skips the check for spec.package.installed when _mark_concrete is called with 'True' (in other words, when the database is marking all specs as being concrete). * add test to confirm this fixes #5293
2017-10-05Fix formatting typo (#5622)Jimmy Tang1-1/+1
2017-10-05Don't change properties on already-installed packages (#5580)scheibelp2-0/+45
* edits to address issues where spack concretization attempts to set properties on already-installed specs * most added checks only need to check if the spec is concrete; they dont also need to check if the package is installed * add test to ensure that patches are not applied to an installed spec * add test to ensure that an error is detected when a dependent requests a dependency constraint which conflicts with a requested installed dependency
2017-10-04Identify the flag handlers feature as beta (#5609)becker331-0/+5
The flag-handling logic added in #4421 may change semantics, so this commit adds a warning to the documentation for this feature.
2017-10-04Fix module loads (#5599)becker331-10/+16
Fixes #5455 All methods within setup_package use an EnvironmentModifications object to control the environment. Those modifications are applied at the end of setup_package. Module loads for the build environment need to be done after the rest of the environment modifications are applied, as otherwise Spack may unset variables set by those modules (for example LD_LIBRARY_PATH).
2017-10-04module files: restricted token expansion + case sensitivity (#5474)Massimiliano Culpo8-22/+170
closes #2884 closes #4684 In #1848 we decided to use `Spec.format` to expand certain tokens in the module file naming scheme or in the environment variable name. Not all the tokens that are allowed in `Spec.format` make sense in module file generation. This PR restricts the set of tokens that can be used, and adds tests to check that the intended behavior is respected. Additionally, the names of environment variables set/modified by module files were, up to now, always uppercase. There are packages though that require case sensitive variable names to honor certain behaviors (e.g. OpenMPI). This PR restricts the uppercase transformation in variable names to `Spec.format` tokens.
2017-10-04spec.patches: fix dictionary reference (#5608)scheibelp1-2/+3
This fixes a loop that was iterating through the keys of a dictionary when it was intending to use the values.
2017-10-04Hotfix: maintain patch order while fixing hashMassimiliano Culpo2-23/+29
fixes #5587 In trying to preserve patch ordering, #5476 made equality inconsistent for the added 'patches' variant. This commit maintains the original weak ordering of patch applications while preserving consistency of comparisons. The ordering DOES NOT enter the hashing mechanism. It's supposed to be a hotfix, while we think of a cleaner and more-permanent solution.
2017-10-03wrap generator with collection to avoid exhausting it with single iteration ↵scheibelp1-3/+3
(#5586)
2017-10-02patch: add workdir option (#5501)Christoph Junghans3-11/+46
* patch: add working_dir option * added documentation
2017-09-30Port CTest's log scraping logic to Spack (#5561)Todd Gamblin4-62/+387
- This steals the magic regular expressions that CTest uses to parse log files and addds them to Spack. See here: https://github.com/Kitware/CMake/blob/master/Source/CTest/cmCTestBuildHandler.cxx These are BSD licensed, so the port is in `externa/ctest_log_parser.py` - We currently use these to do better filtering of errors from build output. Plan is to use them to generate good CDash output.
2017-09-30Add testing for spack blame; refactor llnl.util testsTodd Gamblin7-23/+106
2017-09-30Add 'spack blame' command: shows contributors to packagesTodd Gamblin6-25/+280
`spack blame` prints out the contributors to a package. By modification time: ``` $ spack blame --time llvm LAST_COMMIT LINES % AUTHOR EMAIL 3 days ago 2 0.6 Andrey Prokopenko <andrey.prok@gmail.com> 3 weeks ago 125 34.7 Massimiliano Culpo <massimiliano.culpo@epfl.ch> 3 weeks ago 3 0.8 Peter Scheibel <scheibel1@llnl.gov> 2 months ago 21 5.8 Adam J. Stewart <ajstewart426@gmail.com> 2 months ago 1 0.3 Gregory Becker <becker33@llnl.gov> 3 months ago 116 32.2 Todd Gamblin <tgamblin@llnl.gov> 5 months ago 2 0.6 Jimmy Tang <jcftang@gmail.com> 5 months ago 6 1.7 Jean-Paul Pelteret <jppelteret@gmail.com> 7 months ago 65 18.1 Tom Scogland <tscogland@llnl.gov> 11 months ago 13 3.6 Kelly (KT) Thompson <kgt@lanl.gov> a year ago 1 0.3 Scott Pakin <pakin@lanl.gov> a year ago 3 0.8 Erik Schnetter <schnetter@gmail.com> 3 years ago 2 0.6 David Beckingsale <davidbeckingsale@gmail.com> 3 days ago 360 100.0 ``` Or by percent contribution: ``` $ spack blame --percent llvm LAST_COMMIT LINES % AUTHOR EMAIL 3 weeks ago 125 34.7 Massimiliano Culpo <massimiliano.culpo@epfl.ch> 3 months ago 116 32.2 Todd Gamblin <tgamblin@llnl.gov> 7 months ago 65 18.1 Tom Scogland <tscogland@llnl.gov> 2 months ago 21 5.8 Adam J. Stewart <ajstewart426@gmail.com> 11 months ago 13 3.6 Kelly (KT) Thompson <kgt@lanl.gov> 5 months ago 6 1.7 Jean-Paul Pelteret <jppelteret@gmail.com> 3 weeks ago 3 0.8 Peter Scheibel <scheibel1@llnl.gov> a year ago 3 0.8 Erik Schnetter <schnetter@gmail.com> 3 years ago 2 0.6 David Beckingsale <davidbeckingsale@gmail.com> 3 days ago 2 0.6 Andrey Prokopenko <andrey.prok@gmail.com> 5 months ago 2 0.6 Jimmy Tang <jcftang@gmail.com> 2 months ago 1 0.3 Gregory Becker <becker33@llnl.gov> a year ago 1 0.3 Scott Pakin <pakin@lanl.gov> 3 days ago 360 100.0 ```
2017-09-30Typo: file -> directory (#5560)George Hartzell1-1/+1
Mirrors are directories (that use `file://` URLS, not files.
2017-09-30Documentation for dependency patching.Todd Gamblin3-23/+189
2017-09-30Patches are hashed with specs, and can be associated with dependencies.Todd Gamblin13-155/+555
- A package can depend on a special patched version of its dependencies. - The `Spec` YAML (and therefore the hash) now includes the sha256 of the patch in the `Spec` YAML, which changes its hash. - The special patched version will be built separately from a "vanilla" version of the same package. - This allows packages to maintain patches on their dependencies without affecting either the dependency package or its dependents. This could previously be accomplished with special variants, but having to add variants means the hash of the dependency changes frequently when it really doesn't need to. This commit allows the hash to change *just* for dependencies that need patches. - Patching dependencies shouldn't be the common case, but some packages (qmcpack, hpctoolkit, openspeedshop) do this kind of thing and it makes the code structure mirror maintenance responsibilities. - Note that this commit means that adding or changing a patch on a package will change its hash. This is probably what *should* happen, but we haven't done it so far. - Only applies to `patch()` directives; `package.py` files (and their `patch()` functions) are not hashed, but we'd like to do that in the future. - The interface looks like this: `depends_on()` can optionally take a patch directive or a list of them: depends_on(<spec>, patches=patch(..., when=<cond>), when=<cond>) # or depends_on(<spec>, patches=[patch(..., when=<cond>), patch(..., when=<cond>)], when=<cond>) - Previously, the `patch()` directive only took an `md5` parameter. Now it only takes a `sha256` parameter. We restrict this because we want to be consistent about which hash is used in the `Spec`. - A side effect of hashing patches is that *compressed* patches fetched from URLs now need *two* checksums: one for the downloaded archive and one for the content of the patch itself. Patches fetched uncompressed only need a checksum for the patch. Rationale: - we include the content of the *patch* in the spec hash, as that is the checksum we can do consistently for patches included in Spack's source and patches fetched remotely, both compressed and uncompressed. - we *still* need the patch of the downloaded archive, because we want to verify the download *before* handing it off to tar, unzip, or another decompressor. Not doing so is a security risk and leaves users exposed to any arbitrary code execution vulnerabilities in compression tools.
2017-09-30add spack flake8 exception for long checksumsTodd Gamblin1-2/+2
2017-09-30Clean up exceptions and function names in directives.Todd Gamblin1-41/+23
- Functions returned by directives were all called `_execute`, which made reading stack traces hard because you couldn't tell what directive a frame came from. - renamed them all to `_execute_<directive>` - Exceptions in directives were only really used in one or two places -- get rid of the boilerplate init functions and let the callsite specify the message.
2017-09-30Consolidate some web-spidering commands in spack.util.webTodd Gamblin6-116/+118
- move `spack.cmd.checksum.get_checksums` to `spack.util.web.spider_checksums` - move `spack.error.NoNetworkError` to `spack.util.web.NoNetworkError` since it is only used there.