Bugfix/docs: correct and expand smoke test documentation (#20278)

1 files changed, 151 insertions, 70 deletions

diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst
index d8c2ac9d51..8f38dde1b5 100644
--- a/lib/spack/docs/packaging_guide.rst
+++ b/lib/spack/docs/packaging_guide.rst
@@ -3956,111 +3956,192 @@ Install Tests
 
    The API for adding and running install tests is not yet considered
    stable and may change drastically in future releases. Packages with
-   upstreamed tests will be refactored to match changes to the API.
+   install tests will be refactored to match changes to the API.
 
-While build-tests are integrated with the build system, install tests
-may be added to Spack packages to be run independently of the install
-method.
+While build tests are integrated with the build system, install tests
+are for testing package installations independent of the build process.
 
-Install tests may be added by defining a ``test`` method with the following signature:
+Install tests are executed in a test stage directory that defaults to
+``~/.spack/test``. You can change the location in the high-level ``config``
+by adding the ``test_stage`` path as follows:
+
+.. code-block:: yaml
+
+   config:
+     test_stage: /path/to/stage
+
+""""""""""""""""""""
+Adding install tests
+""""""""""""""""""""
+
+Install tests are added to a package by defining a ``test`` method
+with the following signature:
 
 .. code-block:: python
 
    def test(self):
 
-These tests will be run in an environment set up to provide access to
-this package and all of its dependencies, including ``test``-type
-dependencies. Inside the ``test`` method, standard python ``assert``
-statements and other error reporting mechanisms can be used. Spack
-will report any errors as a test failure.
+These tests run in an environment that provides access to the
+package and all of its dependencies, including ``test``-type
+dependencies.
+
+Standard python ``assert`` statements and other error reporting
+mechanisms can be used in the ``test`` method. Spack will report
+such errors as test failures.
 
-Inside the test method, individual tests can be run separately (and
-continue transparently after a test failure) using the ``run_test``
-method. The signature for the ``run_test`` method is:
+You can implement multiple tests (or test parts) within the ``test``
+method, where each is run separately and testing continues after
+failures by using the ``run_test`` method. The signature for the
+method is:
 
 .. code-block:: python
 
    def run_test(self, exe, options=[], expected=[], status=0, installed=False,
                 purpose='', skip_missing=False, work_dir=None):
 
-This method will operate in ``work_dir`` if one is specified. It will
-search for an executable in the ``PATH`` variable named ``exe``, and
-if ``installed=True`` it will fail if that executable does not come
-from the prefix of the package being tested. If the executable is not
-found, it will fail the test unless ``skip_missing`` is set to
-``True``. The executable will be run with the options specified, and
-the return code will be checked against the ``status`` argument, which
-can be an integer or list of integers. Spack will also check that
-every string in ``expected`` is a regex matching part of the output of
-the executable. The ``purpose`` argument is recorded in the test log
-for debugging purposes.
-
-""""""""""""""""""""""""""""""""""""""
-Install tests that require compilation
-""""""""""""""""""""""""""""""""""""""
+The test fails if there is no executable named ``exe`` found in the
+paths of the ``PATH`` variable **unless** ``skip_missing`` is ``True``.
+The test also fails if the resulting path is not within the prefix of
+the package being tested when ``installed`` is ``True``.
+
+The executable runs in ``work_dir``, when specified, using the provided
+``options``. The return code is checked against the ``status`` argument,
+which can be an integer or list of integers representing status codes
+corresponding to successful execution. Spack also checks that every string
+in ``expected`` is a regex matching part of the output from the test run.
+
+Output from the test is written to its log file. The ``purpose`` argument
+serves as the heading in text logs to highlight the start of each test part.
+
+"""""""""""""""""""""""""
+Enabling test compilation
+"""""""""""""""""""""""""
 
 Some tests may require access to the compiler with which the package
-was built, especially to test library-only packages. To ensure the
-compiler is configured as part of the test environment, set the
-attribute ``tests_require_compiler = True`` on the package. The
-compiler will be available through the canonical environment variables
-(``CC``, ``CXX``, ``FC``, ``F77``) in the test environment.
+was built, especially to test library-only packages. You must enable
+loading the package's compiler configuration by setting the attribute
+``test_requires_compiler`` to ``True``. Doing so makes the compiler
+available in the test environment through the canonical environment
+variables (``CC``, ``CXX``, ``FC``, ``F77``).
 
-""""""""""""""""""""""""""""""""""""""""""""""""
-Install tests that require build-time components
-""""""""""""""""""""""""""""""""""""""""""""""""
+"""""""""""""""""""""""
+Adding build-time files
+"""""""""""""""""""""""
+
+.. note::
+
+    We highly recommend the re-use of build-time tests and input files
+    for testing installed software.  These files are easier to keep
+    synchronized with the software than creating custom install tests.
 
-Some packages cannot be easily tested without components from the
-build-time test suite. For those packages, the
-``cache_extra_test_sources`` method can be used.
+You can use the ``cache_extra_test_sources`` method copy directories
+and or files from the build stage directory to the package's installation
+directory.
+
+For example, a package method for copying everything in the ``tests``
+subdirectory plus the ``foo.c`` and ``bar.c`` files from ``examples``
+can be implemented as follows:
 
 .. code-block:: python
 
    @run_after('install')
    def cache_test_sources(self):
-       srcs = ['./tests/foo.c', './tests/bar.c']
+       srcs = ['tests', 'examples/foo.c', 'examples/bar.c']
        self.cache_extra_test_sources(srcs)
 
-This method will copy the listed methods into the metadata directory
-of the package at the end of the install phase of the build. They will
-be available to the test method in the directory
-``self._extra_tests_path``.
+The use of the ``run_after`` directive ensures the associated files
+are copied **after** the package is installed during the build process.
+
+The method copies files to the package's metadata directory under
+the ``self.install_test_root``. All files in the package source's
+``tests`` directory for the example above will be copied to the
+``join_path(self.install_test_root, 'tests')`` directory. The files
+will be copied to the ``join_path(self.install_test_root, 'examples')``
+directory.
+
+.. note::
+
+    While source and input files are generally recommended, binaries
+    may also be cached by the build process for install testing.
 
-While source files are generally recommended, for many packages
-binaries may also technically be cached in this way for later testing.
+"""""""""""""""""""
+Adding custom files
+"""""""""""""""""""
+
+Some tests may require additional files not available from the build.
+Examples include:
+
+- test source files
+- test input files
+- test build scripts
+- expected test output
+
+These extra files should be added to the ``test`` subdirectory of the
+package in the repository. Spack will automaticaly copy any files in
+that directory to the test staging directory during install testing.
+The ``test`` method can access those files from the
+``self.test_suite.current_test_data_dir`` directory.
 
 .. _cmd-spack-test:
 
-"""""""""""""""""""""
-Running install tests
-"""""""""""""""""""""
+"""""""""""""""""""
+``spack test list``
+"""""""""""""""""""
 
-Install tests can be run using the ``spack test run`` command. The
-``spack test run`` command will create a ``test suite`` out of the
-specs provided to it, or if no specs are provided it will test all
-specs in the active environment, or all specs installed in Spack if no
-environment is active. Test suites can be named using the ``--alias``
-option; test suites not aliased will use the content hash of their
-specs as their name.
+Packages available for install testing can be found using the
+``spack test list`` command. The command outputs all installed
+packages that have defined ``test`` methods.
 
-Packages to install test can be queried using the ``spack test list``
-command, which outputs all installed packages with defined ``test``
-methods.
+""""""""""""""""""
+``spack test run``
+""""""""""""""""""
 
-Test suites can be found using the ``spack test find`` command. It
-will list all test suites that have been run and have not been removed
-using the ``spack test remove`` command. The ``spack test remove``
-command will remove tests to declutter the test stage. The ``spack
-test results`` command will show results for completed test suites.
+Install tests can be run for one or more installed packages using
+the ``spack test run`` command. A ``test suite`` is created from
+the provided specs.  If no specs are provided it will test all specs
+in the active environment or all specs installed in Spack if no
+environment is active.
 
-The test stage is the working directory for all install tests run with
-Spack. By default, Spack uses ``~/.spack/test`` as the test stage. The
-test stage can be set in the high-level config:
+Test suites can be named using the ``--alias`` option. Unaliased
+Test suites will use the content hash of their specs as their name.
 
-.. code-block:: yaml
+Some of the more commonly used debugging options are:
 
-   config:
-     test_stage: /path/to/stage
+- ``--fail-fast`` stops testing each package after the first failure
+- ``--fail-first`` stops testing packages after the first failure
+
+Test output is written to a text log file by default but ``junit``
+and ``cdash`` are outputs are available through the ``--log-format``
+option.
+
+""""""""""""""""""""""
+``spack test results``
+""""""""""""""""""""""
+
+The ``spack test results`` command shows results for all completed
+test suites. Providing the alias or content hash limits reporting
+to the corresponding test suite.
+
+The ``--logs`` option includes the output generated by the associated
+test(s) to facilitate debugging.
+
+The ``--failed`` option limits results shown to that of the failed
+tests, if any, of matching packages.
+
+"""""""""""""""""""
+``spack test find``
+"""""""""""""""""""
+
+The ``spack test find`` command lists the aliases or content hashes
+of all test suites whose results are available.
+
+"""""""""""""""""""""
+``spack test remove``
+"""""""""""""""""""""
+
+The ``spack test remove`` command removes test suites to declutter
+the test results directory. You are prompted to confirm the removal
+of each test suite **unless** you use the ``--yes-to-all`` option.
 
 ---------------------------
 File manipulation functions