diff options
| author | Scott Wittenburg <scott.wittenburg@kitware.com> | 2022-11-07 13:31:14 -0700 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2022-11-07 21:31:14 +0100 |
| commit | 28a77c2821d62a2800a63921f116fcf45a8f7342 (patch) | |
| tree | 204a6842e65e37421597f1d4c3a59dcc543e9cd3 /pyproject.toml | |
| parent | 01a578851707281b8e1da736086b41e98bfbe5f4 (diff) | |
| download | spack-28a77c2821d62a2800a63921f116fcf45a8f7342.tar.gz spack-28a77c2821d62a2800a63921f116fcf45a8f7342.tar.bz2 spack-28a77c2821d62a2800a63921f116fcf45a8f7342.tar.xz spack-28a77c2821d62a2800a63921f116fcf45a8f7342.zip | |
binary_distribution: Speed up buildcache update-index (#32796)
This change uses the aws cli, if available, to retrieve spec files
from the mirror to a local temp directory, then parallelizes the
reading of those files from disk using multiprocessing.ThreadPool.
If the aws cli is not available, then a ThreadPool is used to fetch
and read the spec files from the mirror.
Using aws cli results in ~16 times speed up to recreate the binary
mirror index, while just parallelizing the fetching and reading
results in ~3 speed up.
Diffstat (limited to 'pyproject.toml')
0 files changed, 0 insertions, 0 deletions