From 28c0dd9148b5db3e06b38999e234167a872e7dfc Mon Sep 17 00:00:00 2001 From: Peter Scheibel Date: Tue, 25 Sep 2018 18:58:51 -0700 Subject: Increase and customize lock timeouts (#9219) Fixes #9166 This is intended to reduce errors related to lock timeouts by making the following changes: * Improves error reporting when acquiring a lock fails (addressing #9166) - there is no longer an attempt to release the lock if an acquire fails * By default locks taken on individual packages no longer have a timeout. This allows multiple spack instances to install overlapping dependency DAGs. For debugging purposes, a timeout can be added by setting 'package_lock_timeout' in config.yaml * Reduces the polling frequency when trying to acquire a lock, to reduce impact in the case where NFS is overtaxed. A simple adaptive strategy is implemented, which starts with a polling interval of .1 seconds and quickly increases to .5 seconds (originally it would poll up to 10^5 times per second). A test is added to check the polling interval generation logic. * The timeout for Spack's whole-database lock (e.g. for managing information about installed packages) is increased from 60s to 120s * Users can configure the whole-database lock timeout using the 'db_lock_timout' setting in config.yaml Generally, Spack locks (those created using spack.llnl.util.lock.Lock) now have no timeout by default This does not address implementations of NFS that do not support file locking, or detect cases where services that may be required (nfslock/statd) aren't running. Users may want to be able to more-aggressively release locks when they know they are the only one using their Spack instance, and they encounter lock errors after a crash (e.g. a remote terminal disconnect mentioned in #8915). --- etc/spack/defaults/config.yaml | 13 +++++++++++++ 1 file changed, 13 insertions(+) (limited to 'etc') diff --git a/etc/spack/defaults/config.yaml b/etc/spack/defaults/config.yaml index 28628a3d49..4092140e29 100644 --- a/etc/spack/defaults/config.yaml +++ b/etc/spack/defaults/config.yaml @@ -94,3 +94,16 @@ config: # If set to true, spack will use ccache to cache c compiles. ccache: false + + # How long to wait to lock the Spack installation database. This lock is used + # when spack needs to manage its own package metadata and all operations are + # expected to complete within the default time limit. The timeout should + # therefore generally be left untouched. + db_lock_timeout: 120 + + # How long to wait when attempting to modify a package (e.g. to install it). + # This value should typically be 'null' (never time out) unless the Spack + # instance only ever has a single user at a time, and only if the user + # anticipates that a significant delay indicates that the lock attempt will + # never succeed. + package_lock_timeout: null -- cgit v1.2.3-70-g09d2