summaryrefslogtreecommitdiff
path: root/bin
AgeCommit message (Collapse)AuthorFilesLines
2020-12-22add mypy to style checks; rename `spack flake8` to `spack style` (#20384)Tom Scogland1-0/+2
I lost my mind a bit after getting the completion stuff working and decided to get Mypy working for spack as well. This adds a `.mypy.ini` that checks all of the spack and llnl modules, though not yet packages, and fixes all of the identified missing types and type issues for the spack library. In addition to these changes, this includes: * rename `spack flake8` to `spack style` Aliases flake8 to style, and just runs flake8 as before, but with a warning. The style command runs both `flake8` and `mypy`, in sequence. Added --no-<tool> options to turn off one or the other, they are on by default. Fixed two issues caught by the tools. * stub typing module for python2.x We don't support typing in Spack for python 2.x. To allow 2.x to support `import typing` and `from typing import ...` without a try/except dance to support old versions, this adds a stub module *just* for python 2.x. Doing it this way means we can only reliably use all type hints in python3.7+, and mypi.ini has been updated to reflect that. * add non-default black check to spack style This is a first step to requiring black. It doesn't enforce it by default, but it will check it if requested. Currently enforcing the line length of 79 since that's what flake8 requires, but it's a bit odd for a black formatted project to be quite that narrow. All settings are in the style command since spack has no pyproject.toml and I don't want to add one until more discussion happens. Also re-format `style.py` since it no longer passed the black style check with the new length. * use style check in github action Update the style and docs action to use `spack style`, adding in mypy and black to the action even if it isn't running black right now.
2020-11-12macos: update build process to use spawn instead of fork (#18205)Peter Scheibel1-2/+4
Spack creates a separate process to do package installation. Different operating systems and Python versions use different methods to create it but up until Python 3.8 both Linux and Mac OS used "fork" (which duplicates process memory, file descriptor table, etc.). Python >= 3.8 on Mac OS prefers creating an entirely new process (referred to as the "spawn" start method) because "fork" was found to cause issues (in other words "spawn" is the default start method used by multiprocessing.Process). Spack was dependent on the particular behavior of fork to replicate process memory and transmit file descriptors. This PR refactors the Spack internals to support starting a child process with the "spawn" method. To achieve this, it makes the following changes: - ensure that the package repository and other global state are transmitted to the child process - ensure that file descriptors are transmitted to the child process in a way that works with multiprocessing and spawn - make all the state needed for the build process and tests picklable (package, stage, etc.) - move a number of locally-defined functions into global scope so that they can be pickled - rework tests where needed to avoid using local functions This PR also reworks sbang tests to work on macOS, where temporary directories are deeper than the Linux sbang limit. We make the limit platform-dependent (macOS supports 512-character shebangs) See: #14102
2020-10-28sbang: vendor sbangTodd Gamblin1-95/+15
`sbang` now lives at https://github.com/spack/sbang, and it has its own test suite that's more extensive than what's in Spack. We'll leave sbang tests to sbang from now on, and just vendor `bin/sbang` directly. Remaining `sbang` tests have to do with patching files, not with `sbang`'s functionality. This update also fixes a bug with `sbang` and multiple command line arguments that was introduced in #19529. See: * https://github.com/spack/sbang/pull/1 * https://github.com/spack/sbang/pull/2 - [x] include latest `sbang` from https://github.com/spack/sbang - [x] remove old `sbang` tests from Spack - [x] update `COPYRIGHT` and `cmd/license.py`
2020-10-27sbang: convert sbang script to POSIX shellTodd Gamblin1-48/+106
`sbang` was previously a bash script but did not need to be. This converts it to a plain old POSIX shell script and adds some options. This also allows us to simplify sbang shebangs to `#!/bin/sh /path/to/sbang` instead of `#!/bin/bash /path/to/sbang`. The new script passes shellcheck (with a few exceptions noted in the file) - [x] `SBANG_DEBUG` env var enables printing what *would* be executed - [x] `sbang` checks whether it has been passed an option and fails gracefully - [x] `sbang` will now fail if it can't find a second shebang line, or if the second line happens to be sbang (avoid infinite loops) - [x] add more rigorous tests for `sbang` behavior using `SBANG_DEBUG`
2020-10-26sbang: add support for php (#18299)Toyohisa Kameyama1-7/+18
PHP supports an initial shebang, but its comment syntax can't handle our 2-line shebangs. So, we need to embed the 2nd-line shebang comment to look like a PHP comment: <?php #!/path/to/php ?> This adds patching support to the sbang hook and support for instrumenting php shebangs. This also patches `phar`, which is a tool used to create php packages. `phar` itself has to add sbangs to those packages (as phar archives apparently contain UTF-8, as well as binary blobs), and `phar` sets a checksum based on the contents of the package. Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2020-03-29spack-python should exec `spack python` (#15738)Todd Gamblin1-1/+1
The current implementation of `spack-python` will leave an extra shell around while it runs. That shell should really replace itself with spack. - [x] add exec to spack-python script
2020-01-24sbang: Fix perl binary detectionMichael Kuhn1-3/+3
The perl binary can also be called `perlX.Y.Z` if using a development build or simply using the versioned binary. We were also dropping all sbang arguments, since `exec $interpreter_v` was only using the first element of the `interpreter_v` array.
2020-01-21pipelines: `spack ci` command with env-based workflow (#12854)Scott Wittenburg3-503/+0
Rework Spack's continuous integration workflow to be environment-based. - Add the `spack ci` command, which replaces the many scripts in `bin/` - `spack ci` decouples the CI workflow from the spack instance: - CI is defined in a spack environment - environment is in its own (single) git repository, separate from Spack - spack instance used to run the pipeline is up to the user - A new `gitlab-ci` section in environments allows users to configure how specs in the environment should be mapped to runners - Compilers can be bootstrapped in the new pipeline workflow - Add extensive documentation on pipelines (see `pipelines.rst` for further details) - Add extensive tests for pipeline code
2019-12-30copyright: update copyright dates for 2020 (#14328)Todd Gamblin5-5/+5
2019-10-22fetching: S3 upload and download (#11117)Omar Padron1-1/+1
This extends Spack functionality so that it can fetch sources and binaries from-, push sources and binaries to-, and index the contents of- mirrors hosted on an S3 bucket. High level to-do list: - [x] Extend mirrors configuration to add support for `file://`, and `s3://` URLs. - [x] Ensure all fetching, pushing, and indexing operations work for `file://` URLs. - [x] Implement S3 source fetching - [x] Implement S3 binary mirror indexing - [x] Implement S3 binary package fetching - [x] Implement S3 source pushing - [x] Implement S3 binary package pushing Important details: * refactor URL handling to handle S3 URLs and mirror URLs more gracefully. - updated parse() to accept already-parsed URL objects. an equivalent object is returned with any extra s3-related attributes intact. Objects created with urllib can also be passed, and the additional s3 handling logic will still be applied. * update mirror schema/parsing (mirror can have separate fetch/push URLs) * implement s3_fetch_strategy/several utility changes * provide more feature-complete S3 fetching * update buildcache create command to support S3 * Move the core logic for reading data from S3 out of the s3 fetch strategy and into the s3 URL handler. The s3 fetch strategy now calls into `read_from_url()` Since read_from_url can now handle S3 URLs, the S3 fetch strategy is redundant. It's not clear whether the ideal design is to have S3 fetching functionality in a fetch strategy, directly implemented in read_from_url, or both. * expanded what can be passed to `spack buildcache` via the -d flag: In addition to a directory on the local filesystem, the name of a configured mirror can be passed, or a push URL can be passed directly.
2019-09-29prefer Python 3 to Python 2 for running SpackTodd Gamblin1-1/+17
The Python landscape is going to be changing in 2020, and Python 2 will be end of life. Spack should *prefer* Python 3 to Python 2 by default, but we still need to run on systems that only have Python2 available. This is trickier than it sounds, as on some systems, the `python` command is `python2`; on others it's `python3`, and RHEL8 doesn't even have the `python` command. Instead, it makes you choose `python3` or `python2`. You can thus no longer make a simple shebang to handle all the cases. This commit makes the `spack` script bilingual. It is still valid Python, but its shebang is `#!/bin/sh`, and it has a tiny bit of shell code at the beginning to pick the right python and execute itself with what it finds. This has a lot of advantages. I think this will help ensure that Spack works well in Python3 -- there are cases where we've missed things because Python2 is still the default `python` on most systems. Also, with this change, you do not lose the ability to execute the `spack` script directly with a python interpreter. This is useful for forcing your own version of python, running coverage tools, and running profiling tools. i.e., these will not break with this change: ```console $ python2 $(which spack) <args> $ coverage run $(which spack) <args> $ pyinstrument $(which spack) <args> ``` These would not work if we split `spack` into a python file and a shell script (see #11783). So, this gives us the best of both worlds. We get to control our interpreter *and* remain a mostly pure python executable.
2019-09-13Fixes identified in ecp facilities hackathon fixes:Scott Wittenburg1-26/+18
- improve mirror git repo management - minio s3 implementation needs endpoint_url - Remove image key from rebuild-index job - Remove image, rely on tags instead
2019-09-13Implement an optional compiler bootstrapping phaseScott Wittenburg2-116/+193
2019-09-13Add pre-ci job generation for a multi-repo setupScott Wittenburg1-0/+93
2019-09-13Refactor release-jobs cmd based on use of environments (no docker either)Scott Wittenburg1-8/+20
2019-05-18Fix typo in spack executable (#11512)Adam J. Stewart1-1/+1
2019-02-21release workflow: Add build scripts for jobs and means to upload pkgsScott Wittenburg2-0/+329
2019-01-01copyright: update license headers for 2013-2019 copyright.Todd Gamblin3-3/+3
2018-11-06bugfix: work around ruamel.yaml vendoring issues (#9725)Massimiliano Culpo1-0/+12
- Delete references to ruamel.yaml at Spack start-up, if they are present - ruamel.yaml generates a .pth file when installed via pip that has the effect of always preferring the version of this package installed at site scope (effectively preventing us from vendoring it). - This mechanism triggers when implicitly importing the 'site' module when the python interpreter is started. In this PR we explicitly delete references to 'ruamel.yaml' and 'ruamel' in sys.modules, if any, after we set 'sys.path' to search from the directory where we store vendored packages. This ensures that the imports after those statements will be done from our vendored version. - See #9206 for further details
2018-10-17relicense: replace LGPL headers with Apache-2.0/MIT SPDX headersTodd Gamblin3-70/+16
- remove the old LGPL license headers from all files in Spack - add SPDX headers to all files - core and most packages are (Apache-2.0 OR MIT) - a very small number of remaining packages are LGPL-2.1-only
2018-08-20yaml: use ruamel.yaml instead of pyyamlTodd Gamblin1-7/+0
- ruamel.yaml allows round-tripping comments from/to files - ruamel.yaml is single-source, python2/python3 compatible
2018-07-14clean up of spack clean (#8610)Andreas Baumbach1-24/+0
* update help of `clean --all` to include `-p` * remove old orphaned `.pyc` removal * restrict removal or orphaned pyc files to `lib/spack` and `var/spack`
2018-03-24Update copyright on LLNL files for 2018. (#7592)Todd Gamblin3-3/+3
2018-01-16Vendor ordereddict for python2.6 only (#6931)Massimiliano Culpo1-0/+4
* Vendor ordereddict for python2.6 only This commit substitutes the custom module 'ordereddict_backport' with the more known 'ordereddict' and vendors it only for python 2.6. Other supported versions of python will use 'collections.OrderedDict'. * Use absolute imports also for python 2.6 See PEP-328 for more information on the subject * Added provenance of vendored ordereddict
2017-11-04Replace github.com/llnl/spack with github.com/spack/spack (#6142)Todd Gamblin3-3/+3
We moved to a new GitHub org! Now make the code and docs reflect that.
2017-09-06Update copyright notices for 2017 (#5295)Michael Kuhn3-3/+3
2017-08-18sbang support: add node-js and fix luahealther1-0/+2
This adds sbang hook support for node-js and fixes the sbang filter for lua (the character class exclusion was swallowing newlines and reporting a false positive if lua was mentioned anywhere in the file).
2017-06-24Make LICENSE recognizable by GitHub. (#4598)Todd Gamblin3-3/+3
2017-06-14fix intltool_sbang issue #4191 (#4192)Luigi Calori1-4/+11
* fix intltool_sbang issue #4191 * add per dep even if already depending on perl-xml-parser * adding more comment, properly spaced
2017-05-08rework spack help (#3033)Todd Gamblin1-201/+21
- Full help is now only generated lazily, when needed. - Executing specific commands doesn't require loading all of them. - All commands are only loaded if we need them for help. - There is now short and long help: - short help (spack help) shows only basic spack options - long help (spack help -a) shows all spack options - Both divide help on commands into high-level sections - Commands now specify attributes from which help is auto-generated: - description: used in help to describe the command. - section: help section - level: short or long - Clean up command descriptions - Add a `spack docs` command to open full documentation in the browser. - move `spack doc` command to `spack pydoc` for clarity - Add a `spack --spec` command to show documentation on the spec syntax.
2017-05-03spack: no stacktrace if not in debug mode + fix emacs variant (#4098)Massimiliano Culpo1-0/+2
* spack: no stacktrace if not in debug mode + fix emacs variant * emacs: removed dead code
2017-05-01Allow user to specify profile sort column on the command line. (#4056)Todd Gamblin1-3/+42
- Add -P <STAT> argument so that caller can specify a sort column for cProfile. Can specify multiple columns with commas. e.g.: spack -P cumtime,module - Add --lines option to Spack spec to control number of profile lines displayed - Sort by time by default (because it works in all Python versions) - Show sort column options in command help. - Do a short profile run in the unit tests.
2017-03-31Spack works with Python 3Todd Gamblin1-2/+2
- Update version guard in spack script to allow python 3 - Update min required version in the docs
2017-03-31Convert Python 2 idioms to Python 2/3-compatible ones.Todd Gamblin1-2/+4
- convert print, StringIO, except as, octals, izip - convert print statement to print function - convert StringIO to six.StringIO - remove usage of csv reader in Spec, in favor of simple regex - csv reader only does byte strings - convert 0755 octal literals to 0o755 - convert `except Foo, e` to `except Foo as e` - fix a few places `str` is used. - may need to switch everything to str later. - convert iteritems usages to use six.iteritems - fix urllib and HTMLParser - port metaclasses to use six.with_metaclass - More octal literal conversions for Python 2/3 - Fix a new octal literal. - Convert `basestring` to `six.string_types` - Convert xrange -> range - Fix various issues with encoding, iteritems, and Python3 semantics. - Convert contextlib.nested to explicitly nexted context managers. - Convert use of filter() to list comprehensions. - Replace reduce() with list comprehensions. - Clean up composite: replace inspect.ismethod() with callable() - Python 3 doesn't have "method" objects; inspect.ismethod returns False. - Need to use callable in Composite to make it work. - Update colify to use future division. - Fix zip() usages that need to be lists. - Python3: Use line-buffered logging instead of unbuffered. - Python3 raises an error with unbuffered I/O - See https://bugs.python.org/issue17404
2017-03-31Update externals to work with Python 3Todd Gamblin1-0/+7
- Update YAML version to support Python 3 - Python 3 support for ordereddict backport - Exclude Python3 YAML from version tests. - Vendor six into Spack. - Make Python version-check tests work with Python 3 - Add ability to add version check exceptions with '# nopyqver' line comments.
2017-03-03Fix unexpected keyword error in Python 2.6 (#2954)Adam J. Stewart1-2/+1
2017-01-26Standardize argparse help messages (#2847)Adam J. Stewart1-7/+7
2016-12-29Fix issues related to the switchover to pytest. (#2685)Todd Gamblin1-8/+0
- Remove stale reference to `import nose` from `bin/spack` script. - Add `py` to externals (missing dependency for `pytest`)
2016-12-29unit tests: replace nose with pytest (#2502)Massimiliano Culpo1-18/+40
* Porting: substitute nose with ytest This huge commit substitutes nose with pytest as a testing system. Things done here: * deleted external/nose as it is no longer used * moved mock resources in their own directory 'test/mock/' * ported two tests (cmd/find, build_system) to pytest native syntax as an example * build_environment, log: used monkeypatch instead of try/catch * moved global mocking of fetch_cache to an auto-used fixture * moved global mocking from test/__init__.py to conftest.py * made `spack test` a wrapper around pytest * run-unit-tests: avoid running python 2.6 tests under coverage to speed them up * use `pytest --cov` instead of coverage run to cut down testing time * mock/packages_test: moved mock yaml configuration to files instead of leaving it in the code as string literals * concretize.py: ported tests to native pytest, reverted multiprocessing in pytest.ini as it was creating the wrong report for coveralls * conftest.py, fixtures: added docstrings * concretize_preferences.py: uses fixtures instead of subclassing MockPackagesTest * directory_layout.py: uses fixtures instead of subclassing MockPackagesTest * install.py: uses fixtures instead of subclassing MockPackagesTest * optional_deps.py: uses fixtures instead of subclassing MockPackagesTest optional_deps.py: uses fixtures instead of subclassing MockPackagesTest * packages.py: uses fixtures instead of subclassing MockPackagesTest * provider_index.py: uses fixtures instead of subclassing MockPackagesTest * spec_yaml.py: uses fixtures instead of subclassing MockPackagesTest * multimethod.py: uses fixtures instead of subclassing MockPackagesTest * install.py: now uses mock_archive_url * git_fetch.py: uses fixtures instead of subclassing MockPackagesTest * hg_fetch.py: uses fixtures instead of subclassing MockPackagesTest * svn_fetch.py, mirror.py: uses fixtures instead of subclassing MockPackagesTest repo.py: deleted * test_compiler_cmd.py: uses fixtures instead of subclassing MockPackagesTest * cmd/module.py, cmd/uninstall.py: uses fixtures instead of subclassing MockDatabase * database.py: uses fixtures instead of subclassing MockDatabase, removed mock/database * pytest: uncluttering fixture implementations * database: changing the scope to 'module' * config.py: uses fixtures instead of subclassing MockPackagesTest * spec_dag.py, spec_semantics.py: uses fixtures instead of subclassing MockPackagesTest * stage.py: uses fixtures instead of subclassing MockPackagesTest. Removed mock directory * pytest: added docstrings to all the fixtures * pytest: final cleanup * build_system_guess.py: fixed naming and docstrings as suggested by @scheibelp * spec_syntax.py: added expected failure on parsing multiple specs closes #1976 * Add pytest and pytest-cov to Spack externals. * Make `spack flake8` ignore externals. * run-unit-tests runs spack test and not pytest. * Remove all the special stuff for `spack test` - Remove `conftest.py` magic and all the special case stuff in `bin/spack` - Spack commands can optionally take unknown arguments, if they want to handle them. - `spack test` is now a command like the others. - `spack test` now just delegates its arguments to `pytest`, but it does it by receiving unknown arguments and NOT taking an explicit help argument. * Fix error in fixtures. * Improve `spack test` command a bit. - Now supports an approximation of the old simple interface - Also supports full pytest options if you want them. * Use external coverage instead of pytest-cov * Make coverage use parallel-mode. * change __init__.py docs to include pytest
2016-12-13Add a test to ensure package names have the right case. (#2562)Todd Gamblin1-2/+2
2016-12-05Use JSON for the database instead of YAML. (#2189)Todd Gamblin1-8/+0
* Use JSON for the database instead of YAML. - JSON is much faster than YAML *and* can preserve ordered keys. - 170x+ faster than Python YAML when using unordered dicts - 55x faster than Python YAML (both using OrderedDicts) - 8x faster than C YAML (with OrderedDicts) - JSON is built into Python, unlike C YAML, so doesn't add a dependency. - Don't need human readability for the package database. - JSON requires no major changes to the code -- same object model as YAML. - add to_json, from_json methods to spec. * Add tests to ensure JSON and YAML don't need to be ordered in DB. * Write index.json first time it's not found instead of requiring reindex. * flake8 bug.
2016-11-16-s now includes the file and line number with infoMatthew Krafczyk1-0/+3
The option -s now causes file and line number information to be printed along with any invocation of msg, info, etc... This will greatly ease debugging.
2016-10-30Bugfix: '::' only worked on top-level key in config.Todd Gamblin1-1/+2
- generalized and fixed to work with any key in YAML file - simplified schema writing, as well - add more unit tests for the config system - Rename test/yaml.py to test/spack_yaml.py - Add test/yaml.pyc to ignored pyc files.
2016-10-12Use cYAML if it is available in Python. (#2010)Todd Gamblin1-0/+10
2016-10-11Make `insecure` option work with curl AND git. (#1786)Todd Gamblin1-1/+1
2016-09-28Prohibit Python3 in Python version check. (#1872)Elizabeth Fischer1-2/+2
2016-09-22Fix sbang for perl (#1802)Eric1-2/+6
* Perform shebang fix for all files * Fix sbang for perl scripts Otherwise perl would look at the #! line and call sbang again, resulting in an infinite loop.
2016-08-30Keep dashes in command names, translate to underscoresAdam J. Stewart1-2/+3
2016-08-30Make subcommands importable, '-' -> '_', fixes #1642Adam J. Stewart1-2/+9
2016-08-29Change spack --profile sort key for 2.6 compatibility. (#1656)Todd Gamblin1-1/+1