summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2018-02-13find has been changed to accept glob expressionsalalazo7-44/+79
Following the discussion with Todd and Adam, find has been modified to accept glob expressions. This should not affect performance as every glob implementation I inspected has 3 cases (no wildcard, wildcard but no directories involved, wildcard and directories involved) and uses fnmatch underneath. Mixins have been changed to do by default a non-recursive search (but a recursive search can still be triggered using the recursive keyword).
2018-02-13filter_compiler_path: added the possibility to narrow search pathalalazo1-10/+33
Following a comment from Todd, the search path for the files listed in `filter_compiler_wrappers` can now be narrowed. Anyhow, the function implementation still makes use of `find`, the rationale being that we have already seen packages that install artifacts in e.g. architecture dependent folders. The possibility to have a relative search path might be a good compromise between the previous approach and the one suggested in the review. Also: 'ignore_absent' and 'backup' keyword arguments can be optionally forwarded to `filter_file`.
2018-02-13mixins: moved debug logs to 'filter_file'. Renamed shadowed variable name.alalazo2-11/+16
Following comments from Todd: - the call to tty.debug has been moved deeper, to log the filtering of each file - the shadowing on the name "kwargs" is avoided
2018-02-13mixins: flush method dictionaries after registration.alalazo1-0/+6
2018-02-13mixins: implemented declarative syntaxalalazo4-54/+144
Implemented a declarative syntax for the additional behavior that can get attached to classes. Implemented a function to filter compiler wrappers that uses the mechanism above.
2018-02-13Improved naming of properties and classes (per Denis comments).alalazo2-9/+11
2018-02-13Added mixins to modularize common behaviors across build-systems.alalazo3-0/+90
Modifications: * added a mixin to filter compiler wrappers from files * modified hdf5, openmpi, mpich, mvapich2 to use it
2018-02-12Rework command reference in docs, add `spack commands` commandTodd Gamblin7-20/+459
- command reference now includes usage for all Spack commands as output by `spack help`. Each command usage links to any related section in the docs. - added `spack commands` command which can list command names, subcommands, and generate RST docs for commands. - added `llnl.util.argparsewriter`, which analyzes an argparse parser and calls hooks for description, usage, options, and subcommands
2018-02-12Fix issues with `spack [cmd] -h` help outputTodd Gamblin2-13/+40
- Shorten Spack command usage for short options. Short options are now shown as [-abc] instead of as [-a] [-b] [-c] - fix bug that mixed long and short options for top-level `spack help`
2018-02-12Fix section labels for some commands in the docsTodd Gamblin2-7/+7
2018-02-12make defaults in command help look like standard argparseTodd Gamblin5-7/+7
2018-02-12Add help for buildcache, reorganize command help categoriesTodd Gamblin5-10/+15
- Add proper help for `spack buildcache` subcommands - Reorganize the help categories of Spack commands so that buildcache is in packaging and diy and setup are now in build.
2018-02-12Clean up help metavar for --scope configuration optionsTodd Gamblin6-14/+32
- previously commands with this argument showed a long list of choices that were platform specific. - use a better metavar: {defaults,system,site,user}[/PLATFORM]
2018-02-07Change location of public key in basic tutorial (#7192)v0.11.2becker331-5/+5
* Change location of public key in basic tutorial * Change tuturial version number to 11.2
2018-02-02Fix ignore-conflicts for extensions in views (#7167)healther3-9/+28
Fixes #7159 When activating extensions in external views, the --ignore-conflicts option was being ignored. In this particular issue the conflict was for the duplicate __init__ file for multiple python packages in the same namespace, but in general any conflict for extensions would cause an error whether or not --ignore-conflicts was set. This also renames the 'force' option of do_activate to 'with_dependencies' and updates views to call do_activate with this set to False (since it traverses the dependency dag anyway). This isn't strictly required, it just avoids redundant calls.
2018-02-01Update tutorial on advanced packaging (#7144)scheibelp1-278/+227
This reorganizes most sections and rewords a significant portion of the content (including all introductions) but keeps all the examples. * Remove section 'What happens at subscript time' from tutorial: it is too detailed for a tutorial * Move the 'Extra query parameters' and 'Attach attributes to other packages' sections into a separate grouping 'Other packaging topics' * move the 'Set variables at build time yourself' section after 'Set environment variables in dependents' section since the latter is more motivating * start the 'set environment variables at build-time for yourself' section with qt as an example * renamed section 'specs build interface' to 'retrieving library information' and updated section introduction * renamed section 'a motivating example' to 'accessing library dependencies'; split out the material which deals with implementing .libs for netlib-lapack into a separate section called 'providing libraries to dependents'. consolidated in material from the section 'single package providing multiple virtual specs' since netlib-lapack is an example of this (this removes the material about intel-parallel studio)
2018-01-31Improve log parsing performance (#7093)Todd Gamblin6-96/+408
* Allow dashes in command names and fix command name handling - Command should allow dashes in their names like the reest of spack, e.g. `spack log-parse` - It might be too late for `spack build-cache` (since it is already called `spack buildcache`), but we should try a bit to avoid inconsistencies in naming conventions - The code was inconsistent about where commands should be called by their python module name (e.g. `log_parse`) and where the actual command name should be used (e.g. `log-parse`). - This made it hard to make a command with a dash in the name, and it made `SpackCommand` fail to recognize commands with dashes. - The code now uses the user-facing name with dashes for function parameters, then converts that the module name when needed. * Improve performance of log parsing - A number of regular expressions from ctest_log_parser have really poor performance, most due to untethered expressions with * or + (i.e., they don't start with ^, so the repetition has to be checked for every position in the string with Python's backtracking regex implementation) - I can't verify that CTest's regexes work with an added ^, so I don't really want to touch them. I tried adding this and found that it caused some tests to break. - Instead of using only "efficient" regular expressions, Added a prefilter() class that allows the parser to quickly check a precondition before evaluating any of the expensive regexes. - Preconditions do things like check whether the string contains "error" or "warning" (linear time things) before evaluating regexes that would require them. It's sad that Python doesn't use Thompson string matching (see https://swtch.com/~rsc/regexp/regexp1.html) - Even with Python's slow implementation, this makes the parser ~200x faster on the input we tried it on. * Add `spack log-parse` command and improve the display of parsed logs - Add better coloring and line wrapping to the log parse output. This makes nasty build output look better with the line numbers. - `spack log-parse` allows the log parsing logic used at the end of builds to be executed on arbitrary files, which is handy even outside of spack. - Also provides a profile option -- we can profile arbitrary files and show which regular expressions in the magic CTest parser take the most time. * Parallelize log parsing - Log parsing now uses multiple threads for long logs - Lines from logs are divided into chnks and farmed out to <ncpus> - Add -j option to `spack log-parse`
2018-01-29Mark slow unit tests (#6994)Massimiliano Culpo14-71/+136
* Marking database tests as slow * Marking url command tests as slow * Marking every test that uses database as slow * Marking tests that import files as slow * Marking gpg tests as slow * Marking all versions and one list tests as slow * Added more markers to unit tests + cli option to skip slow tests Following a discussion with Axel, the generic 'slowtest' marker has been split into 'db', 'network' and 'maybeslow'. A brief description of the meaning of each marker has been added to pytest.ini. A command line option to run only fast tests has been added to 'spack test' * Don't use classes to group tests together Reverted grouping tests under a class, as required in the review * Minor style changes
2018-01-28Remove Package instance caching in Repo (#6367)scheibelp9-60/+27
This attempts to address one of the complaints at #5996 (comment): > Repo currently caches package instances by Spec, and those Package instances have a Spec. > This is unnecessary and causes confusion. I think I thought that we'd need to cache instances > after loading package classes, but really just caching the classes is fine. With this update, Repo's package cache is removed and Specs cache the package reference themselves. One consequence is that Specs which compare as equal will store separate instances of a Package class (not doing this creates issues for #4595 (comment)). There were several references to Spec.package that could be replaced with Spec.package_class without any additional modifications. There are still a couple remaining references to Spec.package in Spec that would require adding functionality before replacing (e.g. calling Package.provides and Package.installed). Note this makes it difficult to mock fetchers for tests which invokes code that reconstructs specs. test_packaging was one example of this where the updates caused a failure (in that case the error was avoided by not making an unnecessary call). Details: * Replace instances of spec.package with spec.package_class where a class method is being called * Remove instances of Repo.get where Spec.package_class can be used in its place * remove Repo.get caching instances of Package class for specs * remove redundant check (which is also incorrect now that each spec stores its own copy of its package) * avoid creating mirror with specs because it copies specs and those copies dont refer to the mocked fetcher (and it is also not required for the test) * remove checks that are no longer necessary since repo doesn't cache specs
2018-01-28Fix gfortran 7 detection (#7017)Adam J. Stewart4-7/+44
2018-01-28Cleaned up JUnit report generation on install (#6977)Massimiliano Culpo3-226/+364
* Cleaned up JUnit report generation on install The generation of a JUnit report was previously part of the install command. This commit factors the logic into its own module, and uses a template for the generation of the report. It also improves report generation, that now can deal with multiple specs installed at once. Finally, extending the list of supported formats is much easier than before, as it entails just writing a new template. * Polished report generation + added tests for failures and errors The generation of a JUnit report has been polished, so that the stacktrace is correctly displayed with Jenkins JUnit plugin. Standard error is still not used. Added unit tests to cover for installation failures and installation errors.
2018-01-26Enable install of multiple specs from yaml files (#6984)Federico Ficarelli2-21/+52
2018-01-25Python Installs: No User Site Dir (#7065)Axel Huebl1-3/+3
Avoid adding an "outside" (local home's) python user site directory during python package installs. Implements #6611 Fixes packages with auto-find side effects such as `py-setuptools` that cause `py-matplotlib` to fail to build #6558
2018-01-23Fix error message typo in 'spack create' cmd (#7042)Alastair Harrison1-1/+1
- Previously would print the spec's namespace twice, rather than printing the spec's namespace, followed by the repo's namespace.
2018-01-19Add --full-path option to module find (#6838)Levi Baber2-4/+16
2018-01-19Added a setter to the Spec.prefix property (#6814)Massimiliano Culpo2-13/+17
This commit removes logic from Spec that was there only to be used in tests, and allows to set a new prefix cleanly in user's code.
2018-01-18Fix type issues with setting flag handlers (#6960)becker332-2/+27
The flag_handlers method was being set as a bound method, but when reset in the package.py file it was being set as an unbound method (all python2 issues). This gets the underlying function information, which is the same in either case. The bug was uncovered for parmetis in #6858. This is a partial fix. Included are changes to the parmetis package.py file to make use of flag_handlers.
2018-01-16Final changes for v0.11.0 (#6318)Todd Gamblin2-2/+1
2018-01-16Fix installing specs from yaml file (#6906)Massimiliano Culpo4-4/+34
The feature added in #4611 is currently broken. This commit fixes the behavior of the command and adds unit tests to ensure the basic semantic is maintained. It also changes slightly the behavior of Spec.concretized to avoid copying caches before the concretization (as this may result in a wrong hash computation for the DAG).
2018-01-16Speed up doc builds with spack list --format=html (#6945)Todd Gamblin5-18/+141
- Generating the HTML from for >2300 packages from RST in Sphinx seems to take forever. - Add an option to `spack list` to generate straight HTML instead. - This reduces the doc build time to about a minute (from 5 minutes on a mac laptop).
2018-01-16Vendor ordereddict for python2.6 only (#6931)Massimiliano Culpo9-281/+181
* Vendor ordereddict for python2.6 only This commit substitutes the custom module 'ordereddict_backport' with the more known 'ordereddict' and vendors it only for python 2.6. Other supported versions of python will use 'collections.OrderedDict'. * Use absolute imports also for python 2.6 See PEP-328 for more information on the subject * Added provenance of vendored ordereddict
2018-01-12Handle test dependencies that are python extensions (#6904)scheibelp1-1/+2
See #6794 This fixes cases where test-only dependencies were omitted from consideration when modifying the environment at build time. This includes an update to the python package definition to add testing-related python extensions to its specialized environment setup.
2018-01-12Relax conflict checks for externals (#6907)scheibelp1-2/+2
This updates the conflict-checking logic to require that the conflict spec matches exactly and that all fields mentioned in the conflict spec are present in the concretized spec in order to report a conflict. This will automatically skip all conflict checks for dependencies of externals (since externals strip dependencies). This will not affect non-external packages since all fields and dependencies are fully specified for such packages.
2018-01-10Keep track of source and versions for external libraries (#6803)Adam J. Stewart1-16/+98
* Keep track of source and versions for external libraries * Note source of more obscure libraries * We aren't upgrading jsonschema after all * Add note on modifications made to pytest
2018-01-10Update to pytest 3.2.5 (#6801)Adam J. Stewart48-2109/+3303
* Update to pytest 3.2.5 * Get pytest to pass Python 2.6 compatibility checks
2018-01-10Add suggestion to Contribution Guide to activate flake8 (#6886)Adam J. Stewart1-9/+19
2018-01-10Update to the latest version of jinja2 (#6790)Adam J. Stewart23-365/+728
2018-01-10Update to py 1.4.34 (#6789)Adam J. Stewart9-79/+25
* Update to the latest version of py * Revert back to py 1.4.34
2018-01-10Add unit tests for spack versions command (#6774)Adam J. Stewart2-1/+47
* Add unit tests for spack versions command * Fix bug in web spidering for Python 2.6
2018-01-10Update to the latest version of six (#6787)Adam J. Stewart1-3/+8
2018-01-10Fix bug in placeholder spack cd command and add unit tests (#6776)Adam J. Stewart3-3/+40
2018-01-10Add unit tests for spack arch command (#6775)Adam J. Stewart1-0/+34
2018-01-10Add unit tests for spack graph command (#6778)Adam J. Stewart1-0/+76
2018-01-10Fix a typo in fetch_strategy error message formatting (#6878)Adam J. Stewart1-1/+1
2018-01-07Add documentation on Codecov (#6842)Adam J. Stewart1-12/+60
2018-01-05Fixes the bug in spack configure spotted in #6833 (#6837)Massimiliano Culpo1-1/+1
2018-01-05Fix spack configure output (#6835)Adam J. Stewart1-1/+1
2018-01-02Fix HTML detection (#6394)Michael Kuhn1-1/+2
Some servers report a content-type header instead of Content-Type.
2018-01-02add OctavePackage (#6746)Denis Davydov4-9/+85
* add OctavePackage 1. remove import CudaPackage which is not needed anymore 2. mention CudaPackage and OctavePackage in packaging guide 3. adjust OctavePackageTemplate 4. add clue file for Octave build 5. sanity check on self.prefix * use setup_environment
2018-01-02Add more unit tests for spack clean command (#6777)Adam J. Stewart1-3/+5