summaryrefslogtreecommitdiff
path: root/lib
AgeCommit message (Collapse)AuthorFilesLines
2022-01-14PythonPackage: install packages with pip (#27798)Adam J. Stewart6-719/+391
* Use pip to bootstrap pip * Bootstrap wheel from source * Update PythonPackage to install using pip * Update several packages * Add wheel as base class dep * Build phase no longer exists * Add py-poetry package, fix py-flit-core bootstrapping * Fix isort build * Clean up many more packages * Remove unused import * Fix unit tests * Don't directly run setup.py * Typo fix * Remove unused imports * Fix issues caught by CI * Remove custom setup.py file handling * Use PythonPackage for installing wheels * Remove custom phases in PythonPackages * Remove <phase>_args methods * Remove unused import * Fix various packages * Try to test Python packages directly in CI * Actually run the pipeline * Fix more packages * Fix mappings, fix packages * Fix dep version * Work around bug in concretizer * Various concretization fixes * Fix gitlab yaml, packages * Fix typo in gitlab yaml * Skip more packages that fail to concretize * Fix? jupyter ecosystem concretization issues * Solve Jupyter concretization issues * Prevent duplicate entries in PYTHONPATH * Skip fenics-dolfinx * Build fewer Python packages * Fix missing npm dep * Specify image * More package fixes * Add backends for every from-source package * Fix version arg * Remove GitLab CI stuff, add py-installer package * Remove test deps, re-add install_options * Function declaration syntax fix * More build fixes * Update spack create template * Update PythonPackage documentation * Fix documentation build * Fix unit tests * Remove pip flag added only in newer pip * flux: add explicit dependency on jsonschema * Update packages that have been added since this was branched off of develop * Move Python 2 deprecation to a separate PR * py-neurolab: add build dep on py-setuptools * Use wheels for pip/wheel * Allow use of pre-installed pip for external Python * pip -> python -m pip * Use python -m pip for all packages * Fix py-wrapt * Add both platlib and purelib to PYTHONPATH * py-pyyaml: setuptools is needed for all versions * py-pyyaml: link flags aren't needed * Appease spack audit packages * Some build backend is required for all versions, distutils -> setuptools * Correctly handle different setup.py filename * Use wheels for py-tomli to avoid circular dep on py-flit-core * Fix busco installation procedure * Clarify things in spack create template * Test other Python build backends * Undo changes to busco * Various fixes * Don't test other backends
2022-01-13Python: improve site_packages_dir handling (#28346)Adam J. Stewart3-33/+27
* Python: improve site_packages_dir handling * Replace all site_packages_dir with purelib/platlib
2022-01-12Do not initialize config on spack compiler list (#28042)Harmen Stoppels4-11/+37
When `spack compiler list` is run without being restricted to a particular scope, and no compilers are found, say that none are available, and hint that the use should run spack compiler find to auto detect compilers. * Improve docs * Check if stdin is a tty * add a test
2022-01-12bootstrap: fix bootstrapping GnuPG from different macOS versions (#28350)Massimiliano Culpo3-8/+27
2022-01-12unparser: handle package-level loops, if statements, and with blocksTodd Gamblin5-74/+1039
Many packages implement logic at the class level to handle complex dependencies and conflicts. Others have started using `with when("@1.0"):` blocks since we added that capability. The loops and other control logic can cause some pure directive logic not to be removed by our package hashing logic -- and in many cases that's a lot of code that will cause unnecessary rebuilds. This commit changes the unparser so that it will descend into these blocks. Specifically: 1. Descend into loops, if statements, and with blocks at the class level. 2. Don't look inside function definitions (in or outside a class). 3. Don't look at nested class definitions (they don't have directives) 4. Add logic to *remove* empty loops/with blocks/if statements if all directives in them were removed. This allows our package hash to ignore a lot of pure metadata that it was not ignoring before, and makes it less sensitive. In addition, we add `maintainers` and `tags` to the list of metadata attributes that Spack should remove from packages when constructing canonoical source for a package hash. - [x] Make unparser handle if/for/while/with at class level. - [x] Add tests for control logic removal. - [x] Add a test to ensure that all packages are not only unparseable, but also that their canonical source is still compilable. This is a test for our control logic removal. - [x] Add another unparse test package that has complex logic.
2022-01-12unparser: add unparser unit testsTodd Gamblin3-1/+528
These are the unit tests from astunparse, converted to pytest, with a few backports from upstream cpython. These should hopefully keep `unparser.py` well covered as we change it.
2022-01-12unparser: rename `t` to `node` to mirror upstreamTodd Gamblin1-359/+363
These refactors have happened in upstream `ast.unparse()`
2022-01-12unparser: rename `_Class()` methods to `visit_Class()` to mirror upstreamTodd Gamblin1-75/+75
These are refactors that have happened in upstream `ast.unparse()`
2022-01-12unparser: do a better job of roundtripping stringsTodd Gamblin1-19/+99
Handle complex f-strings. Backport of: https://github.com/python/cpython/commit/a993e901ebe60c38d46ecb31f771d0b4a206828c#
2022-01-12unparser: treat `print(a, b, c)` and `print((a, b, c))` the sameTodd Gamblin3-18/+453
We can't tell `print(a, b, c)` and `print((a, b, c))` apart -- both of these expressions generate different ASTs in Python 2 and Python 3. However, we can decide that we don't care. This commit treats both of them the same when `py_ver_consistent` is set with `unparse()`. This means that the package hash won't notice changes from printing a tuple to printing multiple values, but we don't care, because this is extremely unlikely to affect the build. More than likely this is just an error message for the user of the package. - [x] treat `print(a, b, c)` and `print((a, b, c))` the same in py2 and py3 - [x] add another package parsing test -- legion -- that exercises this feature
2022-01-12commands: add `spack pkg source` and `spack pkg hash`Todd Gamblin3-7/+128
To make it easier to see how package hashes change and how they are computed, add two commands: * `spack pkg source <spec>`: dumps source code for a package to the terminal * `spack pkg source --canonical <spec>`: dumps canonicalized source code for a package to the terminal. It strips comments, directives, and known-unused multimethods from the package. It is used to generate package hashes. * `spack pkg hash <spec>`: This gives the package hash for a particular spec. It is generated from the canonical source code for the spec. - [x] `add spack pkg source` and `spack pkg hash` - [x] add tests - [x] fix bug in multimethod resolution with boolean `@when` values Co-authored-by: Greg Becker <becker33@llnl.gov>
2022-01-12package_hash: switch to using canonical source instead of AST reprTodd Gamblin4-108/+120
We are planning to switch to using full hashes for Spack specs, which means that the package hash will be included in the deployment descriptor. This means we need a more robust package hash than simply dumping the `repr` of the AST. The AST repr that we previously used for package content is unreliable because it can vary between python versions (Python's AST actually changes fairly frequently). - [x] change `package_hash`, `package_ast`, and `canonical_source` to accept a string for alternate source instead of a filename. - [x] consolidate package hash tests in `test/util/package_hash.py`. - [x] remove old `package_content` method. - [x] make `package_hash` do what `canonical_source_hash` was doing before. - [x] modify `content_hash` in `package.py` to use the new `package_hash` function. Co-authored-by: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
2022-01-12unparser: Don't omit parenthesis when unparsing a sliceTodd Gamblin1-14/+39
Backport of https://github.com/python/cpython/commit/c102a148256b00b7d48c51a1a97df19042e603de Includes support for Python 2.
2022-01-12unparser: Don't put unnecessary parentheses on class declarationsTodd Gamblin1-1/+1
Backport of * https://github.com/python/cpython/commit/25160cdc4775a1ddb4e37c8bf5a6e31ad9c146ed
2022-01-12package_hash: add test to ensure that every package in Spack can be unparsedTodd Gamblin2-6/+26
- [x] add option to canonical source to *not* filter multimethods - [x] add test to unparse every package in builtin
2022-01-12package_hash: add test to ensure consistency across Python versionsTodd Gamblin7-0/+2486
Our package hash is supposed to be consistent from python version to python version. Test this by adding some known unparse inputs and ensuring that they always have the same canonical hash. This test relies on the fact that we run Spack's unit tests across many python versions. We can't compute for several python versions within the same test run so we precompute the hashes and check them in CI.
2022-01-12package_hash: fix handling of multimethods and add testsTodd Gamblin2-35/+261
Package hashing was not properly handling multimethods. In particular, it was removing any functions that had decorators from the output, so we'd miss things like `@run_after("install")`, etc. There were also problems with handling multiple `@when`'s in a single file, and with handling `@when` functions that *had* to be evaluated dynamically. - [x] Rework static `@when` resolution for package hash - [x] Ensure that functions with decorators are not removed from output - [x] Add tests for many different @when scenarios (multiple @when's, combining with other decorators, default/no default, etc.) Co-authored-by: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
2022-01-12package_hash: rework `RemoveDirectives` and add a testTodd Gamblin3-6/+41
Previously we used `directives.__all__` to get directive names, but it wasn't quite right -- it included `DirectiveMeta`, etc. It's not wrong, but it's also not the clearest way to do this. - [x] Refactor `@directive` to track names in `directive_names` global - [x] Rename `_directive_names` to `_directive_dict_names` in `DirectiveMeta` - [x] Add a test for `RemoveDirectives` Co-authored-by: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
2022-01-12package_hash: remove all unassigned strings, not just docstringsTodd Gamblin2-9/+77
Some packages use top-level unassigned strings instead of comments, either just after a docstring on in the body somewhere else. Ignore those strings becasue they have no effect on package behavior. - [x] adjust RemoveDocstrings to remove all free-standing strings. - [x] move tests for util/package_hash.py to test/util/package_hash.py Co-authored-by: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
2022-01-12unparser: handle unicode string literals consistently across Python versionsTodd Gamblin1-2/+16
Python 2 and 3 represent string literals differently in the AST. Python 2 requires '\x' literals, and Python 3 source is always unicode, and allows unicode to be written directly. These also unparse differently by default. - [x] modify unparser to write both out the way `repr` would in Python 2 when `py_ver_consistent` is provided.
2022-01-12unparser: implement operator precedence algorithm for unparserTodd Gamblin6-165/+310
Backport operator precedence algorithm from here: https://github.com/python/cpython/commit/397b96f6d7a89f778ebc0591e32216a8183fe667 This eliminates unnecessary parentheses from our unparsed output and makes Spack's unparser consistent with the one in upstream Python 3.9+, with one exception. Our parser normalizes argument order when `py_ver_consistent` is set, so that star arguments in function calls come last. We have to do this because Python 2's AST doesn't have information about their actual order. If we ever support only Python 3.9 and higher, we can easily switch over to `ast.unparse`, as the unparsing is consistent except for this detail (modulo future changes to `ast.unparse`)
2022-01-12unparser: refactor delimiting with context managers in ast.unparseTodd Gamblin1-190/+177
Backport of https://github.com/python/cpython/commit/4b3b1226e86df6cd45e921c8f2ad23c3639c43b2
2022-01-12unparser: add block() context manager for indentationTodd Gamblin1-63/+63
This is a backport of a refactor from cpython 3.9
2022-01-12unparse: Make unparsing consistent for 2.7 and 3.5-3.10Todd Gamblin2-13/+71
Previously, there were differences in the unparsed code for Python 2.7 and for 3.5-3.10. This makes unparsed code the same across these Python versions by: 1. Ensuring there are no spaces between unary operators and their operands. 2. Ensuring that *args and **kwargs are always the last arguments, regardless of the python version. 3. Always unparsing print as a function. 4. Not putting an extra comma after Python 2 class definitions. Without these changes, the same source can generate different code for different Python versions, depending on subtle AST differences. One place where single source will generate an inconsistent AST is with multi-argument print statements, e.g.: ``` print("foo", "bar", "baz") ``` In Python 2, this prints a tuple; in Python 3, it is the print function with multiple arguments. Use `from __future__ import print_function` to avoid this inconsistency.
2022-01-12externals: add astunparseTodd Gamblin4-11/+1011
Add `astunparse` as `spack_astunparse`. This library unparses Python ASTs and we're adding it under our own name so that we can make modifications to it. Ultimately this will be used to make `package_hash` consistent across Python versions.
2022-01-12Use depends_on over load in lmod module files generated by Spack (#28352)Harmen Stoppels1-15/+3
2022-01-10stage.steal_source: preserve symlinksPeter Scheibel1-1/+1
This avoids dangling symlink errors. ignore_dangling_symlinks option would be more-targeted but is only available for Python >= 3.2 (#28318)
2022-01-10Python: set default config_vars (#28290)Adam J. Stewart1-9/+5
* Python: set default config_vars * Add missing commas * dso_suffix not present for some reason * Remove use of default_site_packages_dir * Use config_vars during bootstrapping too * Catch more errors * Fix unit tests * Catch more errors * Update docstring
2022-01-06Fix double space in prompt after 'spack env activate -p' (#28279)Harmen Stoppels1-1/+1
2022-01-05Add function to determine Linux kernel version (#27855)Andrew W Elble1-0/+20
This reports the kernel version (vs. the distro version) on Linux and returns a valid Version (stripping characters like '+' which may be present for custom-built kernels).
2021-12-29Ensure some version known to Spack can satisfy constraints in `depends_on` ↵Massimiliano Culpo1-0/+40
(#28131) Add a new check to `spack audit` to scan and verify that version constraints may be satisfied Modifications: - [x] Add a new check to `spack audit` to scan and verify that version constraints may be satisfied by some version declared in the built-in repository - [x] Fix issues found by CI Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2021-12-28Fixed typos: 'wtih' instead of 'with' (#28166)Martin Diehl3-4/+4
2021-12-23Merge tag 'v0.17.1' into developMassimiliano Culpo1-1/+1
2021-12-23New subcommand: spack bootstrap status (#28004)Massimiliano Culpo3-0/+200
This command pokes the environment, Python interpreter and bootstrap store to check if dependencies needed by Spack are available. If any are missing, it shows a comprehensible message.
2021-12-23Bump version and update CHANGELOG.mdMassimiliano Culpo1-1/+1
2021-12-23locks: allow locks to work under high contention (#27846)Tom Scogland1-3/+6
* locks: allow locks to work under high contention This is a bug found by Harshitha Menon. The `lock=None` line shouldn't be a release but should be ``` return (lock_type, None) ``` to inform the caller it couldn't get the lock type requested without disturbing the existing lock object in the database. There were also a couple of bugs due to taking write locks at the beginning without any checking or release, and not releasing read locks before requeueing. This version no longer gives me read upgrade to write errors, even running 200 instances on one box. * Change lock in check_deps_status to read, release if not installed, not sure why this was ever write, but read definitely is more appropriate here, and the read lock is only held out of the scope if the package is installed. * Release read lock before requeueing to reduce chance of livelock, the timeout that caused the original issue now happens in roughly 3 of 200 workers instead of 199 on average.
2021-12-23Regenerate views when specs already installed (#28113)Harmen Stoppels1-4/+2
With this commit: ``` $ spack env activate --temp $ spack install zlib ==> All of the packages are already installed ==> Updating view at /tmp/spack-faiirgmt/.spack-env/view $ spack install zlib ==> All of the packages are already installed ``` Before this PR: ``` $ spack env activate --temp $ spack install zlib ==> All of the packages are already installed $ spack install zlib ==> All of the packages are already installed ``` No view was generated
2021-12-23sbang: respect package permissive package permissions for sbang (#25764)Paul Spencer2-12/+96
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov> Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
2021-12-23Add setdefault option to tcl module (#14686)victorusu4-18/+69
This commit introduces the command spack module tcl setdefault <package> similar to the one already available for lmod Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2021-12-23Set backup=False by default in filter_file (#28036)Harmen Stoppels1-1/+1
2021-12-23Fix table formatting (#28037)Harmen Stoppels1-1/+1
2021-12-23Revert "patches: make re-applied patches idempotent (#26784)" (#27625)Harmen Stoppels4-137/+25
This reverts commit c5ca0db27fce5d772dc8a4fcffec3b62bb0bf1f3.
2021-12-23MANPATH needs a trailing ':' to utilize system defaults (#21682)Andrew W Elble2-1/+22
otherwise spack breaks using system man pages by default. Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2021-12-23Added opensuse/leap:15 to spack containerize (#27837)Christian Goll2-0/+21
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2021-12-23Provide meaningful message for empty environment installs (#28031)Tamara Dahlgren2-9/+26
* Provide a meaningful failure message for installation of an empty environment * Allow regenerating view per offline discussion
2021-12-23Add option to minimize full debug cores. include warning message about ↵Greg Becker4-21/+73
performance (#27970) Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2021-12-23Install dir creation message demoted to "debug" level (#27911)Harmen Stoppels1-2/+1
2021-12-23Improve debug info from concretizer (#27707)Greg Becker2-2/+33
2021-12-23spack audit: fix API calls (#27713)Massimiliano Culpo1-4/+4
This broke in #24858
2021-12-23Handle byte sequences which are not encoded as UTF8 while logging. (#21447)Paul Ferrell1-1/+6
Fix builds which produce a lines with non-UTF8 output while logging The alternative is to read in binary mode, and then decode while ignoring errors.
generated by cgit v1.2.3-70-g09d2 (git 2.46.0) at 2024-11-02 20:20:49 +0000