From 0eb1957999ccd9e54685f0e69c3a99df908c3e12 Mon Sep 17 00:00:00 2001 From: Tom Scogland Date: Thu, 21 Mar 2024 01:32:28 -0700 Subject: cmd/python: use runpy to allow multiprocessing in scripts (#41789) Running a `spack-python` script like this: ```python import spack import multiprocessing def echo(args): print(args) if __name__ == "__main__": pool = multiprocessing.Pool(2) pool.map(echo, range(10)) ``` will fail in `develop` with an error like this: ```console _pickle.PicklingError: Can't pickle : attribute lookup echo on __main__ failed ``` Python expects to be able to look up the method `echo` in `sys.path["__main__"]` in subprocesses spawned by `multiprocessing`, but because we use `InteractiveConsole` to run `spack python`, the executed file isn't considered to be the `__main__` module, and lookups in subprocesses fail. We tried to fake this by setting `__name__` to `__main__` in the `spack python` command, but that doesn't fix the fact that no `__main__` module exists. Another annoyance with `InteractiveConsole` is that `__file__` is not defined in the main script scope, so you can't use it in your scripts. We can use the [runpy.run_path()](https://docs.python.org/3/library/runpy.html#runpy.run_path) function, which has been around since Python 3.2, to fix this. - [x] Use `runpy` module to launch non-interactive `spack python` invocations - [x] Only use `InteractiveConsole` for interactive `spack python` --- share/spack/spack-completion.bash | 2 +- share/spack/spack-completion.fish | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) (limited to 'share') diff --git a/share/spack/spack-completion.bash b/share/spack/spack-completion.bash index 90d17c5f48..1319d1685f 100755 --- a/share/spack/spack-completion.bash +++ b/share/spack/spack-completion.bash @@ -1956,7 +1956,7 @@ _spack_uninstall() { _spack_unit_test() { if $list_options then - SPACK_COMPREPLY="-h --help -H --pytest-help -l --list -L --list-long -N --list-names --extension -s -k --showlocals" + SPACK_COMPREPLY="-h --help -H --pytest-help -n --numprocesses -l --list -L --list-long -N --list-names --extension -s -k --showlocals" else _unit_tests fi diff --git a/share/spack/spack-completion.fish b/share/spack/spack-completion.fish index 84e1a4721e..8547de9349 100755 --- a/share/spack/spack-completion.fish +++ b/share/spack/spack-completion.fish @@ -2958,12 +2958,14 @@ complete -c spack -n '__fish_spack_using_command uninstall' -l origin -r -f -a o complete -c spack -n '__fish_spack_using_command uninstall' -l origin -r -d 'only remove DB records with the specified origin' # spack unit-test -set -g __fish_spack_optspecs_spack_unit_test h/help H/pytest-help l/list L/list-long N/list-names extension= s/ k/= showlocals +set -g __fish_spack_optspecs_spack_unit_test h/help H/pytest-help n/numprocesses= l/list L/list-long N/list-names extension= s/ k/= showlocals complete -c spack -n '__fish_spack_using_command_pos_remainder 0 unit-test' -f -a '(__fish_spack_unit_tests)' complete -c spack -n '__fish_spack_using_command unit-test' -s h -l help -f -a help complete -c spack -n '__fish_spack_using_command unit-test' -s h -l help -d 'show this help message and exit' complete -c spack -n '__fish_spack_using_command unit-test' -s H -l pytest-help -f -a pytest_help complete -c spack -n '__fish_spack_using_command unit-test' -s H -l pytest-help -d 'show full pytest help, with advanced options' +complete -c spack -n '__fish_spack_using_command unit-test' -s n -l numprocesses -r -f -a numprocesses +complete -c spack -n '__fish_spack_using_command unit-test' -s n -l numprocesses -r -d 'run tests in parallel up to this wide, default 1 for sequential' complete -c spack -n '__fish_spack_using_command unit-test' -s l -l list -f -a list complete -c spack -n '__fish_spack_using_command unit-test' -s l -l list -d 'list test filenames' complete -c spack -n '__fish_spack_using_command unit-test' -s L -l list-long -f -a list -- cgit v1.2.3-70-g09d2