summaryrefslogtreecommitdiff
path: root/bin
AgeCommit message (Collapse)AuthorFilesLines
2022-03-17Update tests support for WindowsBetsy McPhail1-5/+5
Fixup common tests * Remove requirement for Python 2.6 * Skip new failing test Windows: Update url util to handle Windows paths (#27959) * update url util to handle windows paths * Update tests to handle fixed url handling * canonicalize path only when the path type matches the host platform * Skip some url tests on Windows Co-authored-by: Omar Padron <omar.padron@kitware.com> Use threading.TIMEOUT_MAX when available (#24246) This value was introduced in Python 3.2. Specifying a timeout greater than this value will raise an OverflowError. Co-authored-by: Lou Lawrence <lou.lawrence@kitware.com> Co-authored-by: John Parent <john.parent@kitware.com> Co-authored-by: Betsy McPhail <betsy.mcphail@kitware.com>
2022-03-17Spack on Windows package portsJohn Parent1-2/+6
CMake - Windows Bootstrap (#25825) Remove hardcoded cmake compiler (#26410) Revert breaking cmake changes Ensure no autotools on Windows Perl on Windows (#26612) Python source build windows (#26313) Reconfigure sysconf for Windows Python2.6 compatibility Fxixup new sbang tests for windows Ruby support (#28287) Add NASM support (#28319) Add mock Ninja package for testing
2022-03-17Add Github Actions for Windows (#24504)John Parent2-3/+9
Setup Installer CI (#25184), (#25191) Co-authored-by: Zack Galbreath <zack.galbreath@kitware.com> Co-authored-by: lou.lawrence@kitware.com <lou.lawrence@kitware.com> Co-authored-by: Betsy McPhail <betsy.mcphail@kitware.com>
2022-03-17Relocate spack_cmd and scripts from installer to root bin (#24651)John W. Parent3-0/+304
Relocate spack_cmd and scripts from installer to root bin Refactor documentation, installer, and launcher to facilitate that change
2022-01-14Update copyright year to 2022Todd Gamblin2-2/+2
2021-11-23Remove support for Python 2.6 (#27256)Massimiliano Culpo1-4/+2
Modifications: - [x] Removed `centos:6` unit test, adjusted vermin checks - [x] Removed backport of `collections.OrderedDict` - [x] Removed backport of `functools.total_ordering` - [x] Removed Python 2.6 specific skip markers in unit tests - [x] Fixed a few minor Python 2.6 related TODOs in code Updating the vendored dependencies will be done in separate PRs
2021-08-17Use a patched argparse only in Python 2.X (#25376)Massimiliano Culpo1-0/+23
Spack is internally using a patched version of `argparse` mainly to backport Python 3 functionality into Python 2. This PR makes it such that for the supported Python 3 versions we use `argparse` from the standard Python library. This PR has been extracted from #25371 where it was needed to be able to use recent versions of `pytest`. * Fixed formatting issues when using a pristine argparse.py * Fix error message for Python 3.X when missing positional arguments * Account for the change of API in Python 3.7 * Layout multi-valued args into columns in error messages * Seamless transition in develop if argparse.pyc is in external * Be more defensive in case we can't remove the file.
2021-05-22Spack can Use RHEL8's `platform-python` if nothing else is available. (#23857)Todd Gamblin1-1/+2
This adds RHEL8's `/usr/libexec/platform-python` to Spack's list of preferred pythons. It will only be used if no other `python` is available in the `PATH`. We have been testing with this python for a while now, and it seems to do all that we need. If Spack one day isn't able to work with it, we'll take it out, but for now it is useful to allow Spack to be used on RHEL8 without a dedicated `python` installation.
2021-04-22Docs: Updated copyrights in files still using 2020 as ending year (#23215)Tamara Dahlgren1-1/+1
2021-03-20notify and error out on more unsupported versions (#22389)Tom Scogland1-9/+14
This is to help debug situations like #22383, where python3.4 is accidentally preferred over python2. It will also help on systems where there is no python2 available or some other issue.
2021-02-12Introduce a SPACK_PYTHON environment variable (#21222)Chuck Atkins1-3/+6
The SPACK_PYTHON environment variable can be set to a python interpreter to be used by the spack command. This allows the spack command itself to use a consistent and separate interpreter from whatever python might be used for package building.
2021-01-02copyrights: update all files with license headers for 2021Todd Gamblin2-2/+2
- [x] add `concretize.lp`, `spack.yaml`, etc. to licensed files - [x] update all licensed files to say 2013-2021 using `spack license update-copyright-year` - [x] appease mypy with some additions to package.py that needed for oneapi.py
2020-12-22add mypy to style checks; rename `spack flake8` to `spack style` (#20384)Tom Scogland1-0/+2
I lost my mind a bit after getting the completion stuff working and decided to get Mypy working for spack as well. This adds a `.mypy.ini` that checks all of the spack and llnl modules, though not yet packages, and fixes all of the identified missing types and type issues for the spack library. In addition to these changes, this includes: * rename `spack flake8` to `spack style` Aliases flake8 to style, and just runs flake8 as before, but with a warning. The style command runs both `flake8` and `mypy`, in sequence. Added --no-<tool> options to turn off one or the other, they are on by default. Fixed two issues caught by the tools. * stub typing module for python2.x We don't support typing in Spack for python 2.x. To allow 2.x to support `import typing` and `from typing import ...` without a try/except dance to support old versions, this adds a stub module *just* for python 2.x. Doing it this way means we can only reliably use all type hints in python3.7+, and mypi.ini has been updated to reflect that. * add non-default black check to spack style This is a first step to requiring black. It doesn't enforce it by default, but it will check it if requested. Currently enforcing the line length of 79 since that's what flake8 requires, but it's a bit odd for a black formatted project to be quite that narrow. All settings are in the style command since spack has no pyproject.toml and I don't want to add one until more discussion happens. Also re-format `style.py` since it no longer passed the black style check with the new length. * use style check in github action Update the style and docs action to use `spack style`, adding in mypy and black to the action even if it isn't running black right now.
2020-11-12macos: update build process to use spawn instead of fork (#18205)Peter Scheibel1-2/+4
Spack creates a separate process to do package installation. Different operating systems and Python versions use different methods to create it but up until Python 3.8 both Linux and Mac OS used "fork" (which duplicates process memory, file descriptor table, etc.). Python >= 3.8 on Mac OS prefers creating an entirely new process (referred to as the "spawn" start method) because "fork" was found to cause issues (in other words "spawn" is the default start method used by multiprocessing.Process). Spack was dependent on the particular behavior of fork to replicate process memory and transmit file descriptors. This PR refactors the Spack internals to support starting a child process with the "spawn" method. To achieve this, it makes the following changes: - ensure that the package repository and other global state are transmitted to the child process - ensure that file descriptors are transmitted to the child process in a way that works with multiprocessing and spawn - make all the state needed for the build process and tests picklable (package, stage, etc.) - move a number of locally-defined functions into global scope so that they can be pickled - rework tests where needed to avoid using local functions This PR also reworks sbang tests to work on macOS, where temporary directories are deeper than the Linux sbang limit. We make the limit platform-dependent (macOS supports 512-character shebangs) See: #14102
2020-10-28sbang: vendor sbangTodd Gamblin1-95/+15
`sbang` now lives at https://github.com/spack/sbang, and it has its own test suite that's more extensive than what's in Spack. We'll leave sbang tests to sbang from now on, and just vendor `bin/sbang` directly. Remaining `sbang` tests have to do with patching files, not with `sbang`'s functionality. This update also fixes a bug with `sbang` and multiple command line arguments that was introduced in #19529. See: * https://github.com/spack/sbang/pull/1 * https://github.com/spack/sbang/pull/2 - [x] include latest `sbang` from https://github.com/spack/sbang - [x] remove old `sbang` tests from Spack - [x] update `COPYRIGHT` and `cmd/license.py`
2020-10-27sbang: convert sbang script to POSIX shellTodd Gamblin1-48/+106
`sbang` was previously a bash script but did not need to be. This converts it to a plain old POSIX shell script and adds some options. This also allows us to simplify sbang shebangs to `#!/bin/sh /path/to/sbang` instead of `#!/bin/bash /path/to/sbang`. The new script passes shellcheck (with a few exceptions noted in the file) - [x] `SBANG_DEBUG` env var enables printing what *would* be executed - [x] `sbang` checks whether it has been passed an option and fails gracefully - [x] `sbang` will now fail if it can't find a second shebang line, or if the second line happens to be sbang (avoid infinite loops) - [x] add more rigorous tests for `sbang` behavior using `SBANG_DEBUG`
2020-10-26sbang: add support for php (#18299)Toyohisa Kameyama1-7/+18
PHP supports an initial shebang, but its comment syntax can't handle our 2-line shebangs. So, we need to embed the 2nd-line shebang comment to look like a PHP comment: <?php #!/path/to/php ?> This adds patching support to the sbang hook and support for instrumenting php shebangs. This also patches `phar`, which is a tool used to create php packages. `phar` itself has to add sbangs to those packages (as phar archives apparently contain UTF-8, as well as binary blobs), and `phar` sets a checksum based on the contents of the package. Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2020-03-29spack-python should exec `spack python` (#15738)Todd Gamblin1-1/+1
The current implementation of `spack-python` will leave an extra shell around while it runs. That shell should really replace itself with spack. - [x] add exec to spack-python script
2020-01-24sbang: Fix perl binary detectionMichael Kuhn1-3/+3
The perl binary can also be called `perlX.Y.Z` if using a development build or simply using the versioned binary. We were also dropping all sbang arguments, since `exec $interpreter_v` was only using the first element of the `interpreter_v` array.
2020-01-21pipelines: `spack ci` command with env-based workflow (#12854)Scott Wittenburg3-503/+0
Rework Spack's continuous integration workflow to be environment-based. - Add the `spack ci` command, which replaces the many scripts in `bin/` - `spack ci` decouples the CI workflow from the spack instance: - CI is defined in a spack environment - environment is in its own (single) git repository, separate from Spack - spack instance used to run the pipeline is up to the user - A new `gitlab-ci` section in environments allows users to configure how specs in the environment should be mapped to runners - Compilers can be bootstrapped in the new pipeline workflow - Add extensive documentation on pipelines (see `pipelines.rst` for further details) - Add extensive tests for pipeline code
2019-12-30copyright: update copyright dates for 2020 (#14328)Todd Gamblin5-5/+5
2019-10-22fetching: S3 upload and download (#11117)Omar Padron1-1/+1
This extends Spack functionality so that it can fetch sources and binaries from-, push sources and binaries to-, and index the contents of- mirrors hosted on an S3 bucket. High level to-do list: - [x] Extend mirrors configuration to add support for `file://`, and `s3://` URLs. - [x] Ensure all fetching, pushing, and indexing operations work for `file://` URLs. - [x] Implement S3 source fetching - [x] Implement S3 binary mirror indexing - [x] Implement S3 binary package fetching - [x] Implement S3 source pushing - [x] Implement S3 binary package pushing Important details: * refactor URL handling to handle S3 URLs and mirror URLs more gracefully. - updated parse() to accept already-parsed URL objects. an equivalent object is returned with any extra s3-related attributes intact. Objects created with urllib can also be passed, and the additional s3 handling logic will still be applied. * update mirror schema/parsing (mirror can have separate fetch/push URLs) * implement s3_fetch_strategy/several utility changes * provide more feature-complete S3 fetching * update buildcache create command to support S3 * Move the core logic for reading data from S3 out of the s3 fetch strategy and into the s3 URL handler. The s3 fetch strategy now calls into `read_from_url()` Since read_from_url can now handle S3 URLs, the S3 fetch strategy is redundant. It's not clear whether the ideal design is to have S3 fetching functionality in a fetch strategy, directly implemented in read_from_url, or both. * expanded what can be passed to `spack buildcache` via the -d flag: In addition to a directory on the local filesystem, the name of a configured mirror can be passed, or a push URL can be passed directly.
2019-09-29prefer Python 3 to Python 2 for running SpackTodd Gamblin1-1/+17
The Python landscape is going to be changing in 2020, and Python 2 will be end of life. Spack should *prefer* Python 3 to Python 2 by default, but we still need to run on systems that only have Python2 available. This is trickier than it sounds, as on some systems, the `python` command is `python2`; on others it's `python3`, and RHEL8 doesn't even have the `python` command. Instead, it makes you choose `python3` or `python2`. You can thus no longer make a simple shebang to handle all the cases. This commit makes the `spack` script bilingual. It is still valid Python, but its shebang is `#!/bin/sh`, and it has a tiny bit of shell code at the beginning to pick the right python and execute itself with what it finds. This has a lot of advantages. I think this will help ensure that Spack works well in Python3 -- there are cases where we've missed things because Python2 is still the default `python` on most systems. Also, with this change, you do not lose the ability to execute the `spack` script directly with a python interpreter. This is useful for forcing your own version of python, running coverage tools, and running profiling tools. i.e., these will not break with this change: ```console $ python2 $(which spack) <args> $ coverage run $(which spack) <args> $ pyinstrument $(which spack) <args> ``` These would not work if we split `spack` into a python file and a shell script (see #11783). So, this gives us the best of both worlds. We get to control our interpreter *and* remain a mostly pure python executable.
2019-09-13Fixes identified in ecp facilities hackathon fixes:Scott Wittenburg1-26/+18
- improve mirror git repo management - minio s3 implementation needs endpoint_url - Remove image key from rebuild-index job - Remove image, rely on tags instead
2019-09-13Implement an optional compiler bootstrapping phaseScott Wittenburg2-116/+193
2019-09-13Add pre-ci job generation for a multi-repo setupScott Wittenburg1-0/+93
2019-09-13Refactor release-jobs cmd based on use of environments (no docker either)Scott Wittenburg1-8/+20
2019-05-18Fix typo in spack executable (#11512)Adam J. Stewart1-1/+1
2019-02-21release workflow: Add build scripts for jobs and means to upload pkgsScott Wittenburg2-0/+329
2019-01-01copyright: update license headers for 2013-2019 copyright.Todd Gamblin3-3/+3
2018-11-06bugfix: work around ruamel.yaml vendoring issues (#9725)Massimiliano Culpo1-0/+12
- Delete references to ruamel.yaml at Spack start-up, if they are present - ruamel.yaml generates a .pth file when installed via pip that has the effect of always preferring the version of this package installed at site scope (effectively preventing us from vendoring it). - This mechanism triggers when implicitly importing the 'site' module when the python interpreter is started. In this PR we explicitly delete references to 'ruamel.yaml' and 'ruamel' in sys.modules, if any, after we set 'sys.path' to search from the directory where we store vendored packages. This ensures that the imports after those statements will be done from our vendored version. - See #9206 for further details
2018-10-17relicense: replace LGPL headers with Apache-2.0/MIT SPDX headersTodd Gamblin3-70/+16
- remove the old LGPL license headers from all files in Spack - add SPDX headers to all files - core and most packages are (Apache-2.0 OR MIT) - a very small number of remaining packages are LGPL-2.1-only
2018-08-20yaml: use ruamel.yaml instead of pyyamlTodd Gamblin1-7/+0
- ruamel.yaml allows round-tripping comments from/to files - ruamel.yaml is single-source, python2/python3 compatible
2018-07-14clean up of spack clean (#8610)Andreas Baumbach1-24/+0
* update help of `clean --all` to include `-p` * remove old orphaned `.pyc` removal * restrict removal or orphaned pyc files to `lib/spack` and `var/spack`
2018-03-24Update copyright on LLNL files for 2018. (#7592)Todd Gamblin3-3/+3
2018-01-16Vendor ordereddict for python2.6 only (#6931)Massimiliano Culpo1-0/+4
* Vendor ordereddict for python2.6 only This commit substitutes the custom module 'ordereddict_backport' with the more known 'ordereddict' and vendors it only for python 2.6. Other supported versions of python will use 'collections.OrderedDict'. * Use absolute imports also for python 2.6 See PEP-328 for more information on the subject * Added provenance of vendored ordereddict
2017-11-04Replace github.com/llnl/spack with github.com/spack/spack (#6142)Todd Gamblin3-3/+3
We moved to a new GitHub org! Now make the code and docs reflect that.
2017-09-06Update copyright notices for 2017 (#5295)Michael Kuhn3-3/+3
2017-08-18sbang support: add node-js and fix luahealther1-0/+2
This adds sbang hook support for node-js and fixes the sbang filter for lua (the character class exclusion was swallowing newlines and reporting a false positive if lua was mentioned anywhere in the file).
2017-06-24Make LICENSE recognizable by GitHub. (#4598)Todd Gamblin3-3/+3
2017-06-14fix intltool_sbang issue #4191 (#4192)Luigi Calori1-4/+11
* fix intltool_sbang issue #4191 * add per dep even if already depending on perl-xml-parser * adding more comment, properly spaced
2017-05-08rework spack help (#3033)Todd Gamblin1-201/+21
- Full help is now only generated lazily, when needed. - Executing specific commands doesn't require loading all of them. - All commands are only loaded if we need them for help. - There is now short and long help: - short help (spack help) shows only basic spack options - long help (spack help -a) shows all spack options - Both divide help on commands into high-level sections - Commands now specify attributes from which help is auto-generated: - description: used in help to describe the command. - section: help section - level: short or long - Clean up command descriptions - Add a `spack docs` command to open full documentation in the browser. - move `spack doc` command to `spack pydoc` for clarity - Add a `spack --spec` command to show documentation on the spec syntax.
2017-05-03spack: no stacktrace if not in debug mode + fix emacs variant (#4098)Massimiliano Culpo1-0/+2
* spack: no stacktrace if not in debug mode + fix emacs variant * emacs: removed dead code
2017-05-01Allow user to specify profile sort column on the command line. (#4056)Todd Gamblin1-3/+42
- Add -P <STAT> argument so that caller can specify a sort column for cProfile. Can specify multiple columns with commas. e.g.: spack -P cumtime,module - Add --lines option to Spack spec to control number of profile lines displayed - Sort by time by default (because it works in all Python versions) - Show sort column options in command help. - Do a short profile run in the unit tests.
2017-03-31Spack works with Python 3Todd Gamblin1-2/+2
- Update version guard in spack script to allow python 3 - Update min required version in the docs
2017-03-31Convert Python 2 idioms to Python 2/3-compatible ones.Todd Gamblin1-2/+4
- convert print, StringIO, except as, octals, izip - convert print statement to print function - convert StringIO to six.StringIO - remove usage of csv reader in Spec, in favor of simple regex - csv reader only does byte strings - convert 0755 octal literals to 0o755 - convert `except Foo, e` to `except Foo as e` - fix a few places `str` is used. - may need to switch everything to str later. - convert iteritems usages to use six.iteritems - fix urllib and HTMLParser - port metaclasses to use six.with_metaclass - More octal literal conversions for Python 2/3 - Fix a new octal literal. - Convert `basestring` to `six.string_types` - Convert xrange -> range - Fix various issues with encoding, iteritems, and Python3 semantics. - Convert contextlib.nested to explicitly nexted context managers. - Convert use of filter() to list comprehensions. - Replace reduce() with list comprehensions. - Clean up composite: replace inspect.ismethod() with callable() - Python 3 doesn't have "method" objects; inspect.ismethod returns False. - Need to use callable in Composite to make it work. - Update colify to use future division. - Fix zip() usages that need to be lists. - Python3: Use line-buffered logging instead of unbuffered. - Python3 raises an error with unbuffered I/O - See https://bugs.python.org/issue17404
2017-03-31Update externals to work with Python 3Todd Gamblin1-0/+7
- Update YAML version to support Python 3 - Python 3 support for ordereddict backport - Exclude Python3 YAML from version tests. - Vendor six into Spack. - Make Python version-check tests work with Python 3 - Add ability to add version check exceptions with '# nopyqver' line comments.
2017-03-03Fix unexpected keyword error in Python 2.6 (#2954)Adam J. Stewart1-2/+1
2017-01-26Standardize argparse help messages (#2847)Adam J. Stewart1-7/+7
2016-12-29Fix issues related to the switchover to pytest. (#2685)Todd Gamblin1-8/+0
- Remove stale reference to `import nose` from `bin/spack` script. - Add `py` to externals (missing dependency for `pytest`)