diff options
author | Tom Scogland <scogland1@llnl.gov> | 2024-03-21 01:32:28 -0700 |
---|---|---|
committer | GitHub <noreply@github.com> | 2024-03-21 01:32:28 -0700 |
commit | 0eb1957999ccd9e54685f0e69c3a99df908c3e12 (patch) | |
tree | dde6125189c2f36d153611f0668c428e80cab4c8 /bin | |
parent | de1f9593c618417b4a3fdd1f7544cb09609408a8 (diff) | |
download | spack-0eb1957999ccd9e54685f0e69c3a99df908c3e12.tar.gz spack-0eb1957999ccd9e54685f0e69c3a99df908c3e12.tar.bz2 spack-0eb1957999ccd9e54685f0e69c3a99df908c3e12.tar.xz spack-0eb1957999ccd9e54685f0e69c3a99df908c3e12.zip |
cmd/python: use runpy to allow multiprocessing in scripts (#41789)
Running a `spack-python` script like this:
```python
import spack
import multiprocessing
def echo(args):
print(args)
if __name__ == "__main__":
pool = multiprocessing.Pool(2)
pool.map(echo, range(10))
```
will fail in `develop` with an error like this:
```console
_pickle.PicklingError: Can't pickle <function echo at 0x104865820>: attribute lookup echo on __main__ failed
```
Python expects to be able to look up the method `echo` in `sys.path["__main__"]` in
subprocesses spawned by `multiprocessing`, but because we use `InteractiveConsole` to
run `spack python`, the executed file isn't considered to be the `__main__` module, and
lookups in subprocesses fail. We tried to fake this by setting `__name__` to `__main__`
in the `spack python` command, but that doesn't fix the fact that no `__main__` module
exists.
Another annoyance with `InteractiveConsole` is that `__file__` is not defined in the
main script scope, so you can't use it in your scripts.
We can use the [runpy.run_path()](https://docs.python.org/3/library/runpy.html#runpy.run_path) function,
which has been around since Python 3.2, to fix this.
- [x] Use `runpy` module to launch non-interactive `spack python` invocations
- [x] Only use `InteractiveConsole` for interactive `spack python`
Diffstat (limited to 'bin')
0 files changed, 0 insertions, 0 deletions