Bootstrap clingo from binaries (#22720)

* Bootstrap clingo from binaries

* Move information on clingo binaries to a JSON file

* Add support to bootstrap on Cray

Bootstrapping on Cray requires, at the moment, to
swap the platform when looking for binaries - due
to #22800.

* Add SHA256 verification for bootstrapped software

Use sha256 verification for binaries necessary to bootstrap
the concretizer and gpg for signature verification

* patchelf: use Spec._old_concretize() to bootstrap

As noted in #24450 we may happen to need the
concretizer when bootstrapping clingo. In that case
only the old concretizer is available.

* Add a schema for bootstrapping methods

Two fields have been added to bootstrap.yaml:
  "sources" which lists the methods available for
       bootstrapping software
  "trusted" which records if a source is trusted or not

A subcommand has been added to "spack bootstrap" to list
the sources currently available.

* Methods used for bootstrapping are configurable from bootstrap:sources

The function that tries to ensure a given Python module
is importable now tries bootstrapping methods in the same
order as they are defined in `bootstrap.yaml`

* Permit to trust/untrust bootstrapping methods

* Add binary tests for MacOS, Ubuntu

* Add documentation

* Add a note on bash

1 files changed, 139 insertions, 47 deletions

diff --git a/lib/spack/docs/getting_started.rst b/lib/spack/docs/getting_started.rst
index f15277e953..c89b46d441 100644
--- a/lib/spack/docs/getting_started.rst
+++ b/lib/spack/docs/getting_started.rst
@@ -9,22 +9,16 @@
 Getting Started
 ===============
 
--------------
-Prerequisites
--------------
+--------------------
+System Prerequisites
+--------------------
 
-Spack has the following minimum requirements, which must be installed
-before Spack is run:
+Spack has the following minimum system requirements, which are assumed to
+be present on the machine where Spack is run:
 
-#. Python 2 (2.6 or 2.7) or 3 (3.5 - 3.9) to run Spack
-#. A C/C++ compiler for building and the ``bash`` shell for Spack's compiler
-   wrapper
-#. The ``make`` executable for building
-#. The ``tar``, ``gzip``, ``unzip``, ``bzip2``, ``xz`` and optionally ``zstd``
-   executables for extracting source code
-#. The ``patch`` command to apply patches
-#. The ``git`` and ``curl`` commands for fetching
-#. If using the ``gpg`` subcommand, ``gnupg2`` is required
+.. csv-table:: System prerequisites for Spack
+   :file: tables/system_prerequisites.csv
+   :header-rows: 1
 
 These requirements can be easily installed on most modern Linux systems;
 on macOS, XCode is required.  Spack is designed to run on HPC
@@ -90,42 +84,107 @@ sourcing time, ensuring future invocations of the ``spack`` command will
 continue to use the same consistent python version regardless of changes in
 the environment.
 
+^^^^^^^^^^^^^^^^^^^^
+Bootstrapping clingo
+^^^^^^^^^^^^^^^^^^^^
 
-^^^^^^^^^^^^^^^^^^
-Check Installation
-^^^^^^^^^^^^^^^^^^
+Spack supports using ``clingo`` as an external solver to compute which software
+needs to be installed. The default configuration allows Spack to install
+``clingo`` from a public buildcache, created by a Github Action workflow. In this
+case the bootstrapping procedure is transparent to the user, except for a
+slightly long waiting time on the first concretization of a spec:
 
-With Spack installed, you should be able to run some basic Spack
-commands.  For example:
+.. code-block:: console
 
-.. command-output:: spack spec netcdf-c
+   $ spack find -b
+   ==> Showing internal bootstrap store at "/home/spack/.spack/bootstrap/store"
+   ==> 0 installed packages
+
+   $ time spack solve zlib
+   ==> Best of 2 considered solutions.
+   ==> Optimization Criteria:
+     Priority  Criterion                                 Value
+     1         deprecated versions used                      0
+     2         version weight                                0
+     3         number of non-default variants (roots)        0
+     4         multi-valued variants                         0
+     5         preferred providers for roots                 0
+     6         number of non-default variants (non-roots)    0
+     7         preferred providers (non-roots)               0
+     8         compiler mismatches                           0
+     9         version badness                               0
+     10        count of non-root multi-valued variants       0
+     11        non-preferred compilers                       0
+     12        target mismatches                             0
+     13        non-preferred targets                         0
+
+   zlib@1.2.11%gcc@11.1.0+optimize+pic+shared arch=linux-ubuntu18.04-broadwell
+
+   real	0m30,618s
+   user	0m27,278s
+   sys	0m1,549s
+
+After this command you'll see that ``clingo`` has been installed for Spack's own use:
 
-In theory, Spack doesn't need any additional installation; just
-download and run!  But in real life, additional steps are usually
-required before Spack can work in a practical sense.  Read on...
+.. code-block:: console
 
-^^^^^^^^^^^^^^^^^
-Clean Environment
-^^^^^^^^^^^^^^^^^
+   $ spack find -b
+   ==> Showing internal bootstrap store at "/home/spack/.spack/bootstrap/store"
+   ==> 2 installed packages
+   -- linux-rhel5-x86_64 / gcc@9.3.0 -------------------------------
+   clingo-bootstrap@spack  python@3.6
 
-Many packages' installs can be broken by changing environment
-variables.  For example, a package might pick up the wrong build-time
-dependencies (most of them not specified) depending on the setting of
-``PATH``.  ``GCC`` seems to be particularly vulnerable to these issues.
+Subsequent calls to the concretizer will then be much faster:
 
-Therefore, it is recommended that Spack users run with a *clean
-environment*, especially for ``PATH``.  Only software that comes with
-the system, or that you know you wish to use with Spack, should be
-included.  This procedure will avoid many strange build errors.
+.. code-block:: console
 
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Optional: Bootstrapping clingo
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+   $ time spack solve zlib
+   [ ... ]
+   real	0m1,222s
+   user	0m1,146s
+   sys	0m0,059s
+
+If for security or for other reasons you don't want to or can't install precompiled
+binaries, Spack can fall-back to bootstrap ``clingo`` from source files. To forbid
+Spack from retrieving binaries from the bootstrapping buildcache, the following
+command must be given:
+
+.. code-block:: console
+
+   $ spack bootstrap untrust github-actions
+   ==> "github-actions" is now untrusted and will not be used for bootstrapping
+
+since an "untrusted" way of bootstrapping software will not be considered
+by Spack. You can verify the new settings are effective with:
+
+.. code-block:: console
 
-Spack supports using clingo as an external solver to compute which software
-needs to be installed. If you have a default compiler supporting C++14 Spack
-can automatically bootstrap this tool from sources the first time it is
-needed:
+   $ spack bootstrap list
+   Name: github-actions UNTRUSTED
+
+     Type: buildcache
+
+     Info:
+       url: https://mirror.spack.io/bootstrap/github-actions/v0.1
+       homepage: https://github.com/alalazo/spack-bootstrap-mirrors
+       releases: https://github.com/alalazo/spack-bootstrap-mirrors/releases
+
+     Description:
+       Buildcache generated from a public workflow using Github Actions.
+       The sha256 checksum of binaries is checked before installation.
+
+
+   Name: spack-install TRUSTED
+
+     Type: install
+
+     Description:
+       Specs built from sources by Spack. May take a long time.
+
+When bootstrapping from sources, Spack requires a compiler with support
+for C++14 (GCC on ``linux``, Apple Clang on ``darwin``) and static C++
+standard libraries on ``linux``. Spack will build the required software
+on the first request to concretize a spec:
 
 .. code-block:: console
 
@@ -138,14 +197,20 @@ needed:
    ==> Optimization: [0, 0, 0, 0, 0, 1, 0, 0, 0]
    zlib@1.2.11%gcc@10.1.0+optimize+pic+shared arch=linux-ubuntu18.04-broadwell
 
-If you want to speed-up bootstrapping, you may try to search for ``cmake`` and ``bison``
-on your system:
+.. tip::
 
-.. code-block:: console
+   If you want to speed-up bootstrapping ``clingo`` from sources, you may try to
+   search for ``cmake`` and ``bison`` on your system:
+
+   .. code-block:: console
+
+      $ spack external find cmake bison
+      ==> The following specs have been detected on this system and added to /home/spack/.spack/packages.yaml
+      bison@3.0.4  cmake@3.19.4
 
-   $ spack external find cmake bison
-   ==> The following specs have been detected on this system and added to /home/spack/.spack/packages.yaml
-   bison@3.0.4  cmake@3.19.4
+"""""""""""""""""""
+The Bootstrap Store
+"""""""""""""""""""
 
 All the tools Spack needs for its own functioning are installed in a separate store, which lives
 under the ``${HOME}/.spack`` directory. The software installed there can be queried with:
@@ -165,6 +230,33 @@ In case it's needed the bootstrap store can also be cleaned with:
    $ spack clean -b
    ==> Removing software in "/home/spack/.spack/bootstrap/store"
 
+^^^^^^^^^^^^^^^^^^
+Check Installation
+^^^^^^^^^^^^^^^^^^
+
+With Spack installed, you should be able to run some basic Spack
+commands.  For example:
+
+.. command-output:: spack spec netcdf-c
+
+In theory, Spack doesn't need any additional installation; just
+download and run!  But in real life, additional steps are usually
+required before Spack can work in a practical sense.  Read on...
+
+^^^^^^^^^^^^^^^^^
+Clean Environment
+^^^^^^^^^^^^^^^^^
+
+Many packages' installs can be broken by changing environment
+variables.  For example, a package might pick up the wrong build-time
+dependencies (most of them not specified) depending on the setting of
+``PATH``.  ``GCC`` seems to be particularly vulnerable to these issues.
+
+Therefore, it is recommended that Spack users run with a *clean
+environment*, especially for ``PATH``.  Only software that comes with
+the system, or that you know you wish to use with Spack, should be
+included.  This procedure will avoid many strange build errors.
+
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 Optional: Alternate Prefix
 ^^^^^^^^^^^^^^^^^^^^^^^^^^