summaryrefslogtreecommitdiff
path: root/etc
diff options
context:
space:
mode:
authorRobert Blackwell <rblackwell@flatironinstitute.org>2021-10-28 10:39:16 -0400
committerGitHub <noreply@github.com>2021-10-28 07:39:16 -0700
commit8fd94e31142e6eb49cf846e3dcedda17eba87cf1 (patch)
treead940ae95ddfe3c01f2447c94673958c0723dba1 /etc
parentc13f915735636a8285538b458d888742a3d27f4d (diff)
downloadspack-8fd94e31142e6eb49cf846e3dcedda17eba87cf1.tar.gz
spack-8fd94e31142e6eb49cf846e3dcedda17eba87cf1.tar.bz2
spack-8fd94e31142e6eb49cf846e3dcedda17eba87cf1.tar.xz
spack-8fd94e31142e6eb49cf846e3dcedda17eba87cf1.zip
YamlFilesystemView: improve file removal performance via batching (#24355)
* Drastically improve YamlFilesystemView file removal via batching The `remove_file` routine has to check if the file is owned by multiple packages, so it doesn't remove necessary files. This is done by the `get_all_specs` routine, which walks the entire package tree. With large numbers of packages on shared file systems, this can take seconds per file tree traversal, which adds up extremely quickly. For example, a single deactivate of a largish python package in our software stack on GPFS took approximately 40 minutes. This patch simply replaces `remove_file` with a batch `remove_files` routine. This routine removes a list of files rather than a single file, requiring only one traversal per batch. In practice this means a package can be removed in seconds time, rather than potentially hours, essentially a ~100x speedup (ignoring initial deactivation logic, which takes about 3 minutes in our test setup).
Diffstat (limited to 'etc')
0 files changed, 0 insertions, 0 deletions