Install from source if binary cache checksum validation fails (#31696)

* Fix https://github.com/spack/spack/issues/31640

Some packages in the binary cache fail checksum validation. Instead of having to
go back and manually install all failed packages with `--no-cache` option,
requeue those failed packages for installation from source

```shell
$ spack install py-pip
==> Installing py-pip-21.3.1-s2cx4gqrqkdqhashlinqyzkrvuwkl3x7
==> Fetching https://binaries.spack.io/releases/v0.18/build_cache/linux-amzn2-graviton2-gcc-7.3.1-py-pip-21.3.1-s2cx4gqrqkdqhashlinqyzkrvuwkl3x7.spec.json.sig
gpg: Signature made Wed 20 Jul 2022 12:13:43 PM UTC using RSA key ID 3DB0C723
gpg: Good signature from "Spack Project Official Binaries <maintainers@spack.io>"
==> Fetching https://binaries.spack.io/releases/v0.18/build_cache/linux-amzn2-graviton2/gcc-7.3.1/py-pip-21.3.1/linux-amzn2-graviton2-gcc-7.3.1-py-pip-21.3.1-s2cx4gqrqkdqhashlinqyzkrvuwkl3x7.spack
==> Extracting py-pip-21.3.1-s2cx4gqrqkdqhashlinqyzkrvuwkl3x7 from binary cache
==> Error: Failed to install py-pip due to NoChecksumException: Requeue for manual installation.
==> Installing py-pip-21.3.1-s2cx4gqrqkdqhashlinqyzkrvuwkl3x7
==> Using cached archive: /shared/spack/var/spack/cache/_source-cache/archive/de/deaf32dcd9ab821e359cd8330786bcd077604b5c5730c0b096eda46f95c24a2d
==> No patches needed for py-pip
==> py-pip: Executing phase: 'install'
==> py-pip: Successfully installed py-pip-21.3.1-s2cx4gqrqkdqhashlinqyzkrvuwkl3x7
  Fetch: 0.01s.  Build: 2.81s.  Total: 2.82s.
[+] /shared/spack/opt/spack/linux-amzn2-graviton2/gcc-7.3.1/py-pip-21.3.1-s2cx4gqrqkdqhashlinqyzkrvuwkl3x7
```

* Cleanup style

* better wording

Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>

* Update lib/spack/spack/installer.py

Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>

* changes quotes for style checks

* Update lib/spack/spack/installer.py

Co-authored-by: kwryankrattiger <80296582+kwryankrattiger@users.noreply.github.com>

* Addressing @kwryankrattiger comment to use local 'use_cache`

Co-authored-by: Stephen Sachs <stesachs@amazon.com>
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
Co-authored-by: kwryankrattiger <80296582+kwryankrattiger@users.noreply.github.com>

1 files changed, 10 insertions, 0 deletions

diff --git a/lib/spack/spack/installer.py b/lib/spack/spack/installer.py
index a0f505a922..252c799304 100644
--- a/lib/spack/spack/installer.py
+++ b/lib/spack/spack/installer.py
@@ -1773,6 +1773,16 @@ class PackageInstaller(object):
                 spack.hooks.on_install_cancel(task.request.pkg.spec)
                 raise
 
+            except binary_distribution.NoChecksumException as exc:
+                if not task.cache_only:
+                    # Checking hash on downloaded binary failed.
+                    err = "Failed to install {0} from binary cache due to {1}:"
+                    err += " Requeueing to install from source."
+                    tty.error(err.format(pkg.name, str(exc)))
+                    task.use_cache = False
+                    self._requeue_task(task)
+                    continue
+
             except (Exception, SystemExit) as exc:
                 self._update_failed(task, True, exc)
                 spack.hooks.on_install_failure(task.request.pkg.spec)