From 5b272e3ff3f7cac83d4e3db402781f535950d26f Mon Sep 17 00:00:00 2001 From: Massimiliano Culpo Date: Fri, 5 Jun 2020 09:08:32 +0200 Subject: commands: use a single ThreadPool for `spack versions` (#16749) This fixes a fork bomb in `spack versions`. Recursive generation of pools to scrape URLs in `_spider` was creating large numbers of processes. Instead of recursively creating process pools, we now use a single `ThreadPool` with a concurrency limit. More on the issue: having ~10 users running at the same time spack versions on front-end nodes caused kernel lockup due to the high number of sockets opened (sys-admin reports ~210k distributed over 3 nodes). Users were internal, so they had ulimit -n set to ~70k. The forking behavior could be observed by just running: $ spack versions boost and checking the number of processes spawned. Number of processes per se was not the issue, but each one of them opens a socket which can stress `iptables`. In the original issue the kernel watchdog was reporting: Message from syslogd@login03 at May 19 12:01:30 ... kernel:Watchdog CPU:110 Hard LOCKUP Message from syslogd@login03 at May 19 12:01:31 ... kernel:watchdog: BUG: soft lockup - CPU#110 stuck for 23s! [python3:2756] Message from syslogd@login03 at May 19 12:01:31 ... kernel:watchdog: BUG: soft lockup - CPU#94 stuck for 22s! [iptables:5603] --- share/spack/spack-completion.bash | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'share') diff --git a/share/spack/spack-completion.bash b/share/spack/spack-completion.bash index fa1e79c9c7..0a74d14488 100755 --- a/share/spack/spack-completion.bash +++ b/share/spack/spack-completion.bash @@ -1510,7 +1510,7 @@ _spack_verify() { _spack_versions() { if $list_options then - SPACK_COMPREPLY="-h --help -s --safe-only" + SPACK_COMPREPLY="-h --help -s --safe-only -c --concurrency" else _all_packages fi -- cgit v1.2.3-60-g2f50