diff options
author | Robert Blackwell <rblackwell@flatironinstitute.org> | 2021-10-28 10:39:16 -0400 |
---|---|---|
committer | GitHub <noreply@github.com> | 2021-10-28 07:39:16 -0700 |
commit | 8fd94e31142e6eb49cf846e3dcedda17eba87cf1 (patch) | |
tree | ad940ae95ddfe3c01f2447c94673958c0723dba1 /etc | |
parent | c13f915735636a8285538b458d888742a3d27f4d (diff) | |
download | spack-8fd94e31142e6eb49cf846e3dcedda17eba87cf1.tar.gz spack-8fd94e31142e6eb49cf846e3dcedda17eba87cf1.tar.bz2 spack-8fd94e31142e6eb49cf846e3dcedda17eba87cf1.tar.xz spack-8fd94e31142e6eb49cf846e3dcedda17eba87cf1.zip |
YamlFilesystemView: improve file removal performance via batching (#24355)
* Drastically improve YamlFilesystemView file removal via batching
The `remove_file` routine has to check if the file is owned by multiple packages, so it doesn't
remove necessary files. This is done by the `get_all_specs` routine, which walks the entire
package tree. With large numbers of packages on shared file systems, this can take seconds
per file tree traversal, which adds up extremely quickly. For example, a single deactivate
of a largish python package in our software stack on GPFS took approximately 40 minutes.
This patch simply replaces `remove_file` with a batch `remove_files` routine. This routine
removes a list of files rather than a single file, requiring only one traversal per batch. In
practice this means a package can be removed in seconds time, rather than potentially hours,
essentially a ~100x speedup (ignoring initial deactivation logic, which takes about 3 minutes
in our test setup).
Diffstat (limited to 'etc')
0 files changed, 0 insertions, 0 deletions