Stand-alone testing: make recipe support and processing spack-/pytest-like (#34236)

This is a refactor of Spack's stand-alone test process to be more spack- and pytest-like. 

It is more spack-like in that test parts are no longer "hidden" in a package's run_test()
method and pytest-like in that any package method whose name starts test_ 
(i.e., a "test" method) is a test part. We also support the ability to embed test parts in a
test method when that makes sense.

Test methods are now implicit test parts. The docstring is the purpose for the test part. 
The name of the method is the name of the test part. The working directory is the active
spec's test stage directory. You can embed test parts using the test_part context manager.

Functionality added by this commit:
* Adds support for multiple test_* stand-alone package test methods, each of which is 
   an implicit test_part for execution and reporting purposes;
* Deprecates package use of run_test();
* Exposes some functionality from run_test() as optional helper methods;
* Adds a SkipTest exception that can be used to flag stand-alone tests as being skipped;
* Updates the packaging guide section on stand-alone tests to provide more examples;
* Restores the ability to run tests "inherited" from provided virtual packages;
* Prints the test log path (like we currently do for build log paths);
* Times and reports the post-install process (since it can include post-install tests);
* Corrects context-related error message to distinguish test recipes from build recipes.

2 files changed, 475 insertions, 346 deletions

diff --git a/lib/spack/docs/conf.py b/lib/spack/docs/conf.py
index dafe318275..8ba6717147 100644
--- a/lib/spack/docs/conf.py
+++ b/lib/spack/docs/conf.py
@@ -217,6 +217,7 @@ nitpick_ignore = [
     # Spack classes that intersphinx is unable to resolve
     ("py:class", "spack.version.StandardVersion"),
     ("py:class", "spack.spec.DependencySpec"),
+    ("py:class", "spack.install_test.Pb"),
 ]
 
 # The reST default role (used for this markup: `text`) to use for all documents.
diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst
index 2a4d03bc72..586fdcec6e 100644
--- a/lib/spack/docs/packaging_guide.rst
+++ b/lib/spack/docs/packaging_guide.rst
@@ -18,14 +18,16 @@ have at least some familiarity with Python, and that you've read the
 There are two key parts of Spack:
 
 #. **Specs**: expressions for describing builds of software, and
-#. **Packages**: Python modules that describe how to build
-   software according to a spec.
+#. **Packages**: Python modules that describe how to build and
+   test software according to a spec.
 
 Specs allow a user to describe a *particular* build in a way that a
 package author can understand.  Packages allow the packager to
 encapsulate the build logic for different versions, compilers,
 options, platforms, and dependency combinations in one place.
-Essentially, a package translates a spec into build logic.
+Essentially, a package translates a spec into build logic. It
+also allows the packager to write spec-specific tests of the
+installed software.
 
 Packages in Spack are written in pure Python, so you can do anything
 in Spack that you can do in Python.  Python was chosen as the
@@ -40,7 +42,7 @@ easy.
    The only exception is for proprietary software (e.g., vendor compilers).
 
    If a special build system needs to be added in order to support building
-   a package from source, then the associated code and recipe need to be added
+   a package from source, then the associated code and recipe should be added
    first.
 
 
@@ -1464,6 +1466,8 @@ Go cannot be used to fetch a particular commit or branch, it always
 downloads the head of the repository. This download method is untrusted,
 and is not recommended. Use another fetch strategy whenever possible.
 
+.. _variants:
+
 --------
 Variants
 --------
@@ -4096,6 +4100,8 @@ special parameters to ``configure``, like
 need to supply special compiler flags depending on the compiler.  All
 of this information is available in the spec.
 
+.. _testing-specs:
+
 ^^^^^^^^^^^^^^^^^^^^^^^^
 Testing spec constraints
 ^^^^^^^^^^^^^^^^^^^^^^^^
@@ -4963,106 +4969,294 @@ be left in the build stage directory as illustrated below:
 
 .. _cmd-spack-test:
 
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Stand-alone (or smoke) tests
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^
+Stand-alone tests
+^^^^^^^^^^^^^^^^^
 
-While build-time tests are integrated with the installation process,
-stand-alone tests are independent of that process.  Consequently, such
-tests can be performed days, even weeks, after the software is installed.
+While build-time tests are integrated with the build process, stand-alone
+tests are expected to run days, weeks, even months after the software is
+installed. The goal is to provide a mechanism for gaining confidence that
+packages work as installed **and** *continue* to work as the underlying
+software evolves. Packages can add and inherit stand-alone tests. The
+`spack test`` command is used to manage stand-alone testing.
 
-Stand-alone tests are checks that should run relatively quickly -- as
-in on the order of at most a few minutes -- and ideally execute all
-aspects of the installed software, or at least key functionality.
+.. note::
+
+    Execution speed is important since these tests are intended to quickly
+    assess whether installed specs work on the system. Consequently, they
+    should run relatively quickly -- as in on the order of at most a few
+    minutes -- while ideally executing all, or at least key aspects of the
+    installed software.
 
 .. note::
 
-    Execution speed is important because these tests are intended
-    to quickly assess whether the installed software works on the
-    system.
+    Failing stand-alone tests indicate problems with the installation and,
+    therefore, there is no reason to proceed with more resource-intensive
+    tests until those have been investigated.
 
-    Failing stand-alone tests indicate that there is no reason to
-    proceed with more resource-intensive tests.
+    Passing stand-alone tests indicate that more thorough testing, such
+    as running extensive unit or regression tests, or tests that run at
+    scale can proceed without wasting resources on a problematic installation.
 
-    Passing stand-alone (or smoke) tests can lead to more thorough
-    testing, such as extensive unit or regression tests, or tests
-    that run at scale.
+Tests are defined in the package using methods with names beginning ``test_``.
+This allows Spack to support multiple independent checks, or parts. Files
+needed for testing, such as source, data, and expected outputs, may be saved
+from the build and or stored with the package in the repository. Regardless
+of origin, these files are automatically copied to the spec's test stage
+directory prior to execution of the test method(s). Spack also provides some
+helper functions to facilitate processing.
 
-Stand-alone tests have their own test stage directory, which can be
-configured. These tests can compile or build software with the compiler
-used to build the package. They can use files cached from the build for
-testing the installation. Custom files, such as source, data, or expected
-outputs can be added for use in these tests.
+.. _configure-test-stage:
 
 """"""""""""""""""""""""""""""""""""
 Configuring the test stage directory
 """"""""""""""""""""""""""""""""""""
 
-Stand-alone tests rely on a stage directory for building, running,
-and tracking results.
-The default directory, ``~/.spack/test``, is defined in
-:ref:`etc/spack/defaults/config.yaml <config-yaml>`.
-You can configure the location in the high-level ``config`` by adding
-or changing the ``test_stage`` path in the appropriate ``config.yaml``
-file such that:
+Stand-alone tests utilize a test stage directory for building, running,
+and tracking results in the same way Spack uses a build stage directory.
+The default test stage root directory, ``~/.spack/test``, is defined in
+:ref:`etc/spack/defaults/config.yaml <config-yaml>`. This location is
+customizable by adding or changing the ``test_stage`` path in the high-level
+``config`` of the appropriate ``config.yaml`` file such that:
 
 .. code-block:: yaml
 
    config:
-     test_stage: /path/to/stage
+     test_stage: /path/to/test/stage
 
-The package can access this path **during test processing** using
-`self.test_suite.stage`.
+Packages can use the ``self.test_suite.stage`` property to access this setting.
+Other package properties that provide access to spec-specific subdirectories
+and files are described in :ref:`accessing staged files <accessing-files>`.
 
 .. note::
 
-   The test stage path is established for the entire suite. That
-   means it is the root directory for all specs being installed
-   with the same `spack test run` command. Each spec gets its
-   own stage subdirectory.
+   The test stage path is the root directory for the **entire suite**.
+   In other words, it is the root directory for **all specs** being
+   tested by the ``spack test run`` command. Each spec gets its own
+   stage subdirectory. Use ``self.test_suite.test_dir_for_spec(self.spec)``
+   to access the spec-specific test stage directory.
+
+
+.. _adding-standalone-tests:
+
+""""""""""""""""""""""""
+Adding stand-alone tests
+""""""""""""""""""""""""
+
+Test recipes are defined in the package using methods with names beginning
+``test_``. This allows for the implementation of multiple independent tests.
+Each method has access to the information Spack tracks on the package, such
+as options, compilers, and dependencies, supporting the customization of tests
+to the build. Standard python ``assert`` statements and other error reporting
+mechanisms are available. Such exceptions are automatically caught and reported
+as test failures.
+
+Each test method is an implicit test part named by the method and whose
+purpose is the method's docstring. Providing a purpose gives context for
+aiding debugging. A test method may contain embedded test parts. Spack
+outputs the test name and purpose prior to running each test method and
+any embedded test parts. For example, ``MyPackage`` below provides two basic
+examples of installation tests: ``test_always_fails`` and ``test_example``.
+As the name indicates, the first always fails. The second simply runs the
+installed example.
+
+.. code-block:: python
+
+   class MyPackage(Package):
+       ...
+
+       def test_always_fails(self):
+           """use assert to always fail"""
+           assert False
+
+       def test_example(self):
+           """run installed example"""
+           example = which(self.prefix.bin.example)
+           example()
+
+Output showing the identification of each test part after runnig the tests
+is illustrated below.
+
+.. code-block:: console
+
+   $ spack test run --alias mypackage mypackage@1.0
+   ==> Spack test mypackage
+   ...
+   $ spack test results -l mypackage
+   ==> Results for test suite 'mypackage':
+   ...
+   ==> [2023-03-10-16:03:56.625204] test: test_always_fails: use assert to always fail
+   ...
+   FAILED
+   ==> [2023-03-10-16:03:56.625439] test: test_example: run installed example
+   ...
+   PASSED
+
+
+.. note::
+
+   If ``MyPackage`` were a recipe for a library, the tests should build
+   an example or test program that is then executed.
+
+A test method can include test parts using the ``test_part`` context manager.
+Each part is treated as an independent check to allow subsequent test parts
+to execute even after a test part fails.
+
+.. _test-part:
+
+The signature for ``test_part`` is:
+
+.. code-block:: python
+
+   def test_part(pkg, test_name, purpose, work_dir=".", verbose=False):
+
+where each argument has the following meaning:
+
+* ``pkg`` is an instance of the package for the spec under test.
+
+* ``test_name`` is the name of the test part, which must start with ``test_``.
+
+* ``purpose`` is a brief description used as a heading for the test part.
+
+  Output from the test is written to a test log file allowing the test name
+  and purpose to be searched for test part confirmation and debugging.
+
+* ``work_dir`` is the path to the directory in which the test will run.
+
+  The default of ``None``, or ``"."``, corresponds to the the spec's test
+  stage (i.e., ``self.test_suite.test_dir_for_spec(self.spec)``.
+
+.. admonition:: Tests should **not** run under the installation directory.
+
+   Use of the package spec's installation directory for building and running
+   tests is **strongly** discouraged. Doing so causes permission errors for
+   shared spack instances *and* facilities that install the software in
+   read-only file systems or directories.
+
+Suppose ``MyPackage`` actually installs two examples we want to use for tests.
+These checks can be implemented as separate checks or, as illustrated below,
+embedded test parts.
+
+.. code-block:: python
+
+   class MyPackage(Package):
+       ...
+
+       def test_example(self):
+           """run installed examples"""
+           for example in ["ex1", "ex2"]:
+               with test_part(
+                   self,
+                   "test_example_{0}".format(example),
+                   purpose="run installed {0}".format(example),
+                ):
+                    exe = which(join_path(self.prefix.bin, example))
+                    exe()
+
+In this case, there will be an implicit test part for ``test_example``
+and separate sub-parts for ``ex1`` and ``ex2``. The second sub-part
+will be executed regardless of whether the first passes. The test
+log for a run where the first executable fails and the second passes
+is illustrated below.
+
+.. code-block:: console
+
+   $ spack test run --alias mypackage mypackage@1.0
+   ==> Spack test mypackage
+   ...
+   $ spack test results -l mypackage
+   ==> Results for test suite 'mypackage':
+   ...
+   ==> [2023-03-10-16:03:56.625204] test: test_example: run installed examples
+   ==> [2023-03-10-16:03:56.625439] test: test_example_ex1: run installed ex1
+   ...
+   FAILED
+   ==> [2023-03-10-16:03:56.625555] test: test_example_ex2: run installed ex2
+   ...
+   PASSED
+   ...
+
+.. warning::
+
+   Test results reporting requires that each test method and embedded
+   test part for a package have a unique name.
+
+Stand-alone tests run in an environment that provides access to information
+Spack has on how the software was built, such as build options, dependencies,
+and compilers. Build options and dependencies are accessed with the normal
+spec checks. Examples of checking :ref:`variant settings <variants>` and
+:ref:`spec constraints <testing-specs>` can be found at the provided links.
+Accessing compilers in stand-alone tests that are used by the build requires
+setting a package property as described :ref:`below <test-compilation>`.
+
+
+.. _test-compilation:
 
 """""""""""""""""""""""""
 Enabling test compilation
 """""""""""""""""""""""""
 
-Some stand-alone tests will require access to the compiler with which
-the package was built, especially for library-only packages. You must
-enable loading the package's compiler configuration by setting the
-``test_requires_compiler`` property to ``True`` for your package.
-For example:
+If you want to build and run binaries in tests, then you'll need to tell
+Spack to load the package's compiler configuration. This is accomplished
+by setting the package's ``test_requires_compiler`` property to ``True``.
+
+Setting the property to ``True`` ensures access to the compiler through
+canonical environment variables (e.g., ``CC``, ``CXX``, ``FC``, ``F77``).
+It also gives access to build dependencies like ``cmake`` through their
+``spec objects`` (e.g., ``self.spec["cmake"].prefix.bin.cmake``).
+
+.. note::
+
+   The ``test_requires_compiler`` property should be added at the top of
+   the package near other attributes, such as the ``homepage`` and ``url``.
+
+Below illustrates using this feature to compile an example.
 
 .. code-block:: python
 
-   class MyPackage(Package):
+   class MyLibrary(Package):
        ...
 
        test_requires_compiler = True
+       ...
 
-Setting this property to ``True`` makes the compiler available in the
-test environment through the canonical environment variables (e.g.,
-``CC``, ``CXX``, ``FC``, ``F77``).
-
-.. note::
+       def test_cxx_example(self):
+           """build and run cxx-example"""
+           exe = "cxx-example"
+           ...
+           cxx = which(os.environ["CXX"])
+           cxx(
+               "-L{0}".format(self.prefix.lib),
+               "-I{0}".format(self.prefix.include),
+               "{0}.cpp".format(exe),
+               "-o",
+               exe
+           )
+           cxx_example = which(exe)
+           cxx_example()
 
-   We recommend adding the property at the top of the package with the
-   other attributes, such as ``homepage`` and ``url``.
 
 .. _cache_extra_test_sources:
 
 """""""""""""""""""""""
-Adding build-time files
+Saving build-time files
 """""""""""""""""""""""
 
 .. note::
 
-    We highly recommend re-using build-time tests and input files
-    for testing installed software. These files are easier to keep
-    synchronized since they reside within the software's repository
-    than maintaining custom install test files with the Spack package.
+    We highly recommend re-using build-time test sources and pared down
+    input files for testing installed software. These files are easier
+    to keep synchronized with software capabilities since they reside
+    within the software's repository. 
+    
+    If that is not possible, you can add test-related files to the package
+    repository (see :ref:`adding custom files <cache_custom_files>`). It
+    will be important to maintain them so they work across listed or supported
+    versions of the package.
 
 You can use the ``cache_extra_test_sources`` method to copy directories
-and or files from the build stage directory to the package's installation
-directory.
+and or files from the source build stage directory to the package's
+installation directory.
 
 The signature for ``cache_extra_test_sources`` is:
 
@@ -5070,54 +5264,73 @@ The signature for ``cache_extra_test_sources`` is:
 
    def cache_extra_test_sources(self, srcs):
 
-where ``srcs`` is a string or a list of strings corresponding to
-the paths for the files and or subdirectories, relative to the staged
-source, that are to be copied to the corresponding relative test path
-under the prefix. All of the contents within each subdirectory will
-also be copied.
+where ``srcs`` is a string *or* a list of strings corresponding to the
+paths of subdirectories and or files needed for stand-alone testing. 
+The paths must be relative to the staged source directory. Contents of
+subdirectories and files are copied to a special test cache subdirectory
+of the installation prefix. They are automatically copied to the appropriate
+relative paths under the test stage directory prior to executing stand-alone
+tests.
 
 For example, a package method for copying everything in the ``tests``
 subdirectory plus the ``foo.c`` and ``bar.c`` files from ``examples``
-can be implemented as shown below.
-
-.. note::
-
-   The method name ``copy_test_sources`` here is for illustration
-   purposes. You are free to use a name that is more suited to your
-   package.
-
-   The key to copying the files at build time for stand-alone testing
-   is use of the ``run_after`` directive, which ensures the associated
-   files are copied **after** the provided build stage.
+and using ``foo.c`` in a test method is illustrated below.
 
 .. code-block:: python
 
-   class MyPackage(Package):
+   class MyLibPackage(Package):
        ...
 
        @run_after("install")
-       def copy_test_sources(self):
+       def copy_test_files(self):
            srcs = ["tests",
                    join_path("examples", "foo.c"),
                    join_path("examples", "bar.c")]
            self.cache_extra_test_sources(srcs)
 
+       def test_foo(self):
+           exe = "foo"
+           src_dir = join_path(
+               self.test_suite.current_test_cache_dir, "examples"
+           )
+           with working_dir(src_dir):
+               cc = which(os.environ["CC"])
+               cc(
+                   "-L{0}".format(self.prefix.lib),
+                   "-I{0}".format(self.prefix.include),
+                   "{0}.c".format(exe),
+                   "-o",
+                   exe
+               )
+               foo = which(exe)
+               foo()
+
 In this case, the method copies the associated files from the build
-stage **after** the software is installed to the package's metadata
-directory. The result is the directory and files will be cached in
-a special test subdirectory under the installation prefix.
-
-These paths are **automatically copied** to the test stage directory
-during stand-alone testing. The package's ``test`` method can access
-them using the ``self.test_suite.current_test_cache_dir`` property.
-In our example, the method would use the following paths to reference
+stage, **after** the software is installed, to the package's test
+cache directory. Then ``test_foo`` builds ``foo`` using ``foo.c``
+before running the program.
+
+.. note::
+
+   The method name ``copy_test_files`` here is for illustration purposes.
+   You are free to use a name that is more suited to your package.
+
+   The key to copying files for stand-alone testing at build time is use
+   of the ``run_after`` directive, which ensures the associated files are
+   copied **after** the provided build stage where the files **and**
+   installation prefix are available.
+
+These paths are **automatically copied** from cache to the test stage
+directory prior to the execution of any stand-alone tests. Tests access
+the files using the ``self.test_suite.current_test_cache_dir`` property.
+In our example above, test methods can use the following paths to reference
 the copy of each entry listed in ``srcs``, respectively:
 
 * ``join_path(self.test_suite.current_test_cache_dir, "tests")``
 * ``join_path(self.test_suite.current_test_cache_dir, "examples", "foo.c")``
 * ``join_path(self.test_suite.current_test_cache_dir, "examples", "bar.c")``
 
-.. note::
+.. admonition:: Library packages should build stand-alone tests
 
     Library developers will want to build the associated tests
     against their **installed** libraries before running them.
@@ -5125,10 +5338,22 @@ the copy of each entry listed in ``srcs``, respectively:
 .. note::
 
     While source and input files are generally recommended, binaries
-    **may** also be cached by the build process for install testing.
-    Only you, as the package writer or maintainer, know whether these
-    would be appropriate for ensuring the installed software continues
-    to work as the underlying system evolves.
+    **may** also be cached by the build process. Only you, as the package
+    writer or maintainer, know whether these files would be appropriate
+    for testing the installed software weeks to months later.
+
+.. note::
+
+    If one or more of the copied files needs to be modified to reference
+    the installed software, it is recommended that those changes be made
+    to the cached files **once** in the ``copy_test_sources`` method and
+    ***after** the call to ``self.cache_extra_test_sources()``. This will
+    reduce the amount of unnecessary work in the test method **and** avoid
+    problems testing in shared instances and facility deployments.
+
+    The ``filter_file`` function can be quite useful for such changes.
+    See :ref:`file manipulation <file-manipulation>`.
+
 
 .. _cache_custom_files:
 
@@ -5136,21 +5361,46 @@ the copy of each entry listed in ``srcs``, respectively:
 Adding custom files
 """""""""""""""""""
 
-Some tests may require additional files not available from the build.
-Examples include:
+In some cases it can be useful to have files that can be used to build or
+check the results of tests.  Examples include:
 
 - test source files
 - test input files
 - test build scripts
-- expected test output
+- expected test outputs
+
+While obtaining such files from the software repository is preferred (see
+:ref:`adding build-time files  <cache_extra_test_sources>`), there are
+circumstances where that is not feasible (e.g., the software is not being
+actively maintained). When test files can't be obtained from the repository
+or as a supplement to files that can, Spack supports the inclusion of
+additional files under the ``test`` subdirectory of the package in the
+Spack repository.
+
+Spack **automatically copies** the contents of that directory to the
+test staging directory prior to running stand-alone tests. Test methods
+access those files using the ``self.test_suite.current_test_data_dir``
+property as shown below.
 
-These extra files should be added to the ``test`` subdirectory of the
-package in the Spack repository.
+.. code-block:: python
+
+   class MyLibrary(Package):
+       ...
+
+       test_requires_compiler = True
+       ...
+
+       def test_example(self):
+           """build and run custom-example"""
+           data_dir = self.test_suite.current_test_data_dir
+           exe = "custom-example"
+           src = datadir.join("{0}.cpp".format(exe))
+           ...
+           # TODO: Build custom-example using src and exe
+           ...
+           custom_example = which(exe)
+           custom_example()
 
-Spack will **automatically copy** the contents of that directory to the
-test staging directory for stand-alone testing. The ``test`` method can
-access those files using the ``self.test_suite.current_test_data_dir``
-property.
 
 .. _expected_test_output_from_file:
 
@@ -5160,7 +5410,8 @@ Reading expected output from a file
 
 The helper function ``get_escaped_text_output`` is available for packages
 to retrieve and properly format the text from a file that contains the
-output that is expected when an executable is run using ``self.run_test``.
+expected output from running an executable that may contain special 
+characters.
 
 The signature for ``get_escaped_text_output`` is:
 
@@ -5171,194 +5422,84 @@ The signature for ``get_escaped_text_output`` is:
 where ``filename`` is the path to the file containing the expected output.
 
 The ``filename`` for a :ref:`custom file <cache_custom_files>` can be
-accessed and used as illustrated by a simplified version of an ``sqlite``
-package check:
+accessed by tests using the ``self.test_suite.current_test_data_dir``
+property. The example below illustrates how to read a file that was
+added to the package's ``test`` subdirectory.
 
 .. code-block:: python
 
+   import re
+
    class Sqlite(AutotoolsPackage):
        ...
 
-       def test(self):
+       def test_example(self):
+           """check example table dump"""
            test_data_dir = self.test_suite.current_test_data_dir
            db_filename = test_data_dir.join("packages.db")
            ..
-
            expected = get_escaped_text_output(test_data_dir.join("dump.out"))
-           self.run_test("sqlite3",
-                         [db_filename, ".dump"],
-                         expected,
-                         installed=True,
-                         purpose="test: checking dump output",
-                         skip_missing=False)
-
-Expected outputs do not have to be stored with the Spack package.
-Maintaining them with the source is actually preferable.
+           sqlite3 = which(self.prefix.bin.sqlite3)
+           out = sqlite3(
+               db_filename, ".dump", output=str.split, error=str.split
+           )
+           for exp in expected:
+               assert re.search(exp, out), "Expected '{0}' in output".format(exp)
 
-Suppose a package's source has ``examples/foo.c`` and ``examples/foo.out``
-files that are copied for stand-alone test purposes using
-:ref:`cache_extra_test_sources <cache_extra_test_sources>` and the
-`run_test` method builds the executable ``examples/foo``. The package
-can retrieve the expected output from ``examples/foo.out`` using:
+If the file was instead copied from the ``tests`` subdirectory of the staged
+source code, the path would be obtained as shown below.
 
 .. code-block:: python
 
-   class MyFooPackage(Package):
-       ...
-
-       def test(self):
-           ..
-           filename = join_path(self.test_suite.current_test_cache_dir,
-                                "examples", "foo.out")
-           expected = get_escaped_text_output(filename)
-           ..
+       def test_example(self):
+           """check example table dump"""
+           test_cache_dir = self.test_suite.current_test_cache_dir
+           db_filename = test_cache_dir.join("packages.db")
 
-Alternatively, suppose ``MyFooPackage`` installs tests in ``share/tests``
-and their outputs in ``share/tests/outputs``. The expected output for
-``foo``, assuming it is still called ``foo.out``, can be retrieved as
+Alternatively, if the file was copied to the ``share/tests`` subdirectory
+as part of the installation process, the test could access the path as 
 follows:
 
 .. code-block:: python
 
-   class MyFooPackage(Package):
-       ...
+       def test_example(self):
+           """check example table dump"""
+           db_filename = join_path(self.prefix.share.tests, "packages.db")
 
-       def test(self):
-           ..
-           filename = join_path(self.prefix.share.tests.outputs, "foo.out")
-           expected = get_escaped_text_output(filename)
-           ..
 
+.. _check_outputs:
 
-""""""""""""""""""""""""
-Adding stand-alone tests
-""""""""""""""""""""""""
-
-Stand-alone tests are defined in the package's ``test`` method. The
-default ``test`` method is a no-op so you'll want to override it to
-implement the tests.
-
-.. note::
+""""""""""""""""""""""""""""""""""""
+Comparing expected to actual outputs
+""""""""""""""""""""""""""""""""""""
 
-   Any package method named ``test`` is automatically executed by
-   Spack when the ``spack test run`` command is performed.
+The helper function ``check_outputs`` is available for packages to ensure
+the expected outputs from running an executable are contained within the
+actual outputs.
 
-For example, the ``MyPackage`` package below provides a skeleton for
-the test method.
+The signature for ``check_outputs`` is:
 
 .. code-block:: python
 
-   class MyPackage(Package):
-       ...
-
-       def test(self):
-           # TODO: Add quick checks of the installed software
-           pass
+   def check_outputs(expected, actual):
 
-Stand-alone tests run in an environment that provides access to the
-package and all of its dependencies, including ``test``-type
-dependencies.
+where each argument has the expected type and meaning:
 
-Standard python ``assert`` statements and other error reporting
-mechanisms can be used in the ``test`` method. Spack will report
-such errors as test failures.
+* ``expected`` is a string or list of strings containing the expected (raw)
+  output.
 
-You can implement multiple tests (or test parts) within the ``test``
-method using the ``run_test`` method. Each invocation is run separately
-in a manner that allows testing to continue after failures.
+* ``actual`` is a string containing the actual output from executing the command
 
-The signature for ``run_test`` is:
+Invoking the method is the equivalent of:
 
 .. code-block:: python
 
-   def run_test(self, exe, options=[], expected=[], status=0,
-                installed=False, purpose="", skip_missing=False,
-                work_dir=None):
-
-where each argument has the following meaning:
-
-* ``exe`` is the executable to run.
-
-  If a name, the ``exe`` is required to be found in one of the paths
-  in the ``PATH`` environment variable **unless** ``skip_missing`` is
-  ``True``. Alternatively, a relative (to ``work_dir``) or fully
-  qualified path for the executable can be provided in ``exe``.
-
-  The test will fail if the resulting path is not within the prefix
-  of the package being tested **unless** ``installed`` is ``False``.
-
-* ``options`` is a list of the command line options.
-
-  Options are a list of strings to be passed to the executable when
-  it runs.
-
-  The default is ``[]``, which means no options are provided to the
-  executable.
-
-* ``expected`` is an optional list of expected output strings.
-
-  Spack requires every string in ``expected`` to be a regex matching
-  part of the output from the test run (e.g.,
-  ``expected=["completed successfully", "converged in"]``).  The
-  output can also include expected failure outputs (e.g.,
-  ``expected=["failed to converge"]``).
-
-  The expected output can be :ref:`read from a file
-  <expected_test_output_from_file>`.
-
-  The default is ``expected=[]``, so Spack will not check the output.
-
-* ``status`` is the optional expected return code(s).
-
-  A list of return codes corresponding to successful execution can
-  be provided (e.g., ``status=[0,3,7]``). Support for non-zero return
-  codes allows for basic **expected failure** tests as well as different
-  return codes across versions of the software.
+   for check in expected:
+       if not re.search(check, actual):
+           raise RuntimeError("Expected '{0}' in output '{1}'".format(check, actual))
 
-  The default is ``status=[0]``, which corresponds to **successful**
-  execution in the sense that the executable does not exit with a
-  failure code or raise an exception.
-
-* ``installed`` is used to require ``exe`` to be within the package
-  prefix.
-
-  If ``True``, then the path for ``exe`` is required to be within the
-  package prefix; otherwise, the path is not constrained.
-
-  The default is ``False``, so the fully qualified path for ``exe``
-  does **not** need to be within the installation directory.
-
-* ``purpose`` is an optional heading describing the the test part.
-
-  Output from the test is written to a test log file so this argument
-  serves as a searchable heading in text logs to highlight the start
-  of the test part. Having a description can be helpful when debugging
-  failing tests.
-
-* ``skip_missing`` is used to determine if the test should be skipped.
-
-  If ``True``, then the test part should be skipped if the executable
-  is missing; otherwise, the executable must exist. This option can
-  be useful when test executables are removed or change as the software
-  evolves in subsequent versions.
-
-  The default is ``False``, which means the test executable must be
-  present for any installable version of the software.
-
-* ``work_dir`` is the path to the directory from which the executable
-  will run.
-
-  The default of ``None`` corresponds to the current directory (``"."``).
-  Each call starts with the working directory set to the spec's test stage
-  directory (i.e., ``self.test_suite.test_dir_for_spec(self.spec)``).
-
-.. warning::
-
-   Use of the package spec's installation directory for building and running
-   tests is **strongly** discouraged. Doing so has caused permission errors
-   for shared spack instances *and* for facilities that install the software
-   in read-only file systems or directories.
 
+.. _accessing-files:
 
 """""""""""""""""""""""""""""""""""""""""
 Accessing package- and test-related files
@@ -5366,11 +5507,14 @@ Accessing package- and test-related files
 
 You may need to access files from one or more locations when writing
 stand-alone tests. This can happen if the software's repository does not
-include test source files or includes files but has no way to build the
-executables using the installed headers and libraries. In these cases,
-you may need to reference the files relative to one or more root
-directory. The properties containing package- (or spec-) and test-related
-directory paths are provided in the table below.
+include test source files or includes them but has no way to build the
+executables using the installed headers and libraries. In these cases
+you may need to reference the files relative to one or more root directory.
+The table below lists relevant path properties and provides additional
+examples of their use.
+:ref:`Reading expected output <expected_test_output_from_file>` provides
+examples of accessing files saved from the software repository, package
+repository, and installation.
 
 .. list-table:: Directory-to-property mapping
    :header-rows: 1
@@ -5388,7 +5532,7 @@ directory paths are provided in the table below.
      - ``self.test_suite.stage``
      - ``join_path(self.test_suite.stage, "results.txt")``
    * - Spec's Test Stage
-     - ``self.test_suite.test_dir_for_spec``
+     - ``self.test_suite.test_dir_for_spec(<spec>)``
      - ``self.test_suite.test_dir_for_spec(self.spec)``
    * - Current Spec's Build-time Files
      - ``self.test_suite.current_test_cache_dir``
@@ -5397,15 +5541,20 @@ directory paths are provided in the table below.
      - ``self.test_suite.current_test_data_dir``
      - ``join_path(self.test_suite.current_test_data_dir, "hello.f90")``
 
+
+.. _inheriting-tests:
+
 """"""""""""""""""""""""""""
 Inheriting stand-alone tests
 """"""""""""""""""""""""""""
 
 Stand-alone tests defined in parent (.e.g., :ref:`build-systems`) and
-virtual (e.g., :ref:`virtual-dependencies`) packages are available to
-packages that inherit from or provide interfaces for those packages,
-respectively. The table below summarizes the tests that will be included
-with those provided in the package itself when executing stand-alone tests.
+virtual (e.g., :ref:`virtual-dependencies`) packages are executed by
+packages that inherit from or provide interface implementations for those
+packages, respectively. 
+
+The table below summarizes the stand-alone tests that will be executed along
+with those implemented in the package itself.
 
 .. list-table:: Inherited/provided stand-alone tests
    :header-rows: 1
@@ -5425,86 +5574,64 @@ with those provided in the package itself when executing stand-alone tests.
        <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/mpi>`_
      - Compiles and runs ``mpi_hello`` (``c``, ``fortran``)
    * - :ref:`PythonPackage <pythonpackage>`
-     - Imports installed modules
+     - Imports modules listed in the ``self.import_modules`` property with defaults derived from the tarball
+   * - :ref:`SipPackage <sippackage>`
+     - Imports modules listed in the ``self.import_modules`` property with defaults derived from the tarball
 
-These tests are very generic so it is important that package
-developers and maintainers provide additional stand-alone tests
-customized to the package.
+These tests are very basic so it is important that package developers and
+maintainers provide additional stand-alone tests customized to the package.
 
-One example of a package that adds its own stand-alone (or smoke)
-tests is the `Openmpi package
-<https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/openmpi/package.py>`_.
-The preliminary set of tests for the package performed the
-following checks:
+.. warning::
 
-- installed binaries with the ``--version`` option return the expected
-  version;
-- outputs from (selected) installed binaries match expectations;
-- ``make all`` succeeds when building examples that were copied from the
-  source directory during package installation; and
-- outputs from running the copied and built examples match expectations.
+   Any package that implements a test method with the same name as an
+   inherited method overrides the inherited method. If that is not the
+   goal and you are not explicitly calling and adding functionality to
+   the inherited method for the test, then make sure that all test methods
+   and embedded test parts have unique test names.
 
-Below is an example of running and viewing the stand-alone tests,
-where only the outputs for the first of each set are shown:
+One example of a package that adds its own stand-alone tests to those
+"inherited" by the virtual package it provides an implementation for is
+the `Openmpi package
+<https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/openmpi/package.py>`_.
 
-.. code-block:: console
+Below are snippets from running and viewing the stand-alone test results
+for ``openmpi``:
 
-   $ spack test run --alias openmpi-4.0.5 openmpi@4.0.5
-   ==> Spack test openmpi-4.0.5
-   ==> Testing package openmpi-4.0.5-eygjgve
-   $ spack test results -l openmpi-4.0.5
-   ==> Spack test openmpi-4.0.5
-   ==> Testing package openmpi-4.0.5-eygjgve
-   ==> Results for test suite "openmpi-4.0.5":
-   ==>   openmpi-4.0.5-eygjgve PASSED
-   ==> Testing package openmpi-4.0.5-eygjgve
-   ==> [2021-04-26-17:35:20.259650] test: ensuring version of mpiCC is 8.3.1
-   ==> [2021-04-26-17:35:20.260155] "$SPACK_ROOT/opt/spack/linux-rhel7-broadwell/gcc-8.3.1/openmpi-4.0.5-eygjgvek35awfor2qaljltjind2oa67r/bin/mpiCC" "--version"
-   g++ (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3)
-   Copyright (C) 2018 Free Software Foundation, Inc.
-   This is free software; see the source for copying conditions.  There is NO
-   warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+.. code-block:: console
 
-   PASSED
+   $ spack test run --alias openmpi openmpi@4.1.4
+   ==> Spack test openmpi
+   ==> Testing package openmpi-4.1.4-ubmrigj
+   ============================== 1 passed of 1 spec ==============================
+
+   $ spack test results -l openmpi
+   ==> Results for test suite 'openmpi':
+   ==> test specs:
+   ==>   openmpi-4.1.4-ubmrigj PASSED
+   ==> Testing package openmpi-4.1.4-ubmrigj
+   ==> [2023-03-10-16:03:56.160361] Installing $spack/opt/spack/linux-rhel7-broadwell/gcc-8.3.1/openmpi-4.1.4-ubmrigjrqcafh3hffqcx7yz2nc5jstra/.spack/test to $test_stage/xez37ekynfbi4e7h4zdndfemzufftnym/openmpi-4.1.4-ubmrigj/cache/openmpi
+   ==> [2023-03-10-16:03:56.625204] test: test_bin: test installed binaries
+   ==> [2023-03-10-16:03:56.625439] test: test_bin_mpirun: run and check output of mpirun
+   ==> [2023-03-10-16:03:56.629807] '$spack/opt/spack/linux-rhel7-broadwell/gcc-8.3.1/openmpi-4.1.4-ubmrigjrqcafh3hffqcx7yz2nc5jstra/bin/mpirun' '-n' '1' 'ls' '..'
+   openmpi-4.1.4-ubmrigj            repo
+   openmpi-4.1.4-ubmrigj-test-out.txt  test_suite.lock
+   PASSED: test_bin_mpirun
    ...
-   ==> [2021-04-26-17:35:20.493921] test: checking mpirun output
-   ==> [2021-04-26-17:35:20.494461] "$SPACK_ROOT/opt/spack/linux-rhel7-broadwell/gcc-8.3.1/openmpi-4.0.5-eygjgvek35awfor2qaljltjind2oa67r/bin/mpirun" "-n" "1" "ls" ".."
-   openmpi-4.0.5-eygjgve           repo     test_suite.lock
-   openmpi-4.0.5-eygjgve-test-out.txt  results.txt
-   PASSED
+   ==> [2023-03-10-16:04:01.486977] test: test_version_oshcc: ensure version of oshcc is 8.3.1
+   SKIPPED: test_version_oshcc: oshcc is not installed
    ...
-   ==> [2021-04-26-17:35:20.630452] test: ensuring ability to build the examples
-   ==> [2021-04-26-17:35:20.630943] "/usr/bin/make" "all"
-   mpicc -g  hello_c.c  -o hello_c
-   mpicc -g  ring_c.c  -o ring_c
-   mpicc -g  connectivity_c.c  -o connectivity_c
-   mpicc -g  spc_example.c  -o spc_example
+   ==> [2023-03-10-16:04:02.215227] Completed testing
+   ==> [2023-03-10-16:04:02.215597] 
+   ======================== SUMMARY: openmpi-4.1.4-ubmrigj ========================
+   Openmpi::test_bin_mpirun .. PASSED
+   Openmpi::test_bin_ompi_info .. PASSED
+   Openmpi::test_bin_oshmem_info .. SKIPPED
+   Openmpi::test_bin_oshrun .. SKIPPED
+   Openmpi::test_bin_shmemrun .. SKIPPED
+   Openmpi::test_bin .. PASSED
    ...
-   PASSED
-   ==> [2021-04-26-17:35:23.291214] test: checking hello_c example output and status (0)
-   ==> [2021-04-26-17:35:23.291841] "./hello_c"
-   Hello, world, I am 0 of 1, (Open MPI v4.0.5, package: Open MPI dahlgren@quartz2300 Distribution, ident: 4.0.5, repo rev: v4.0.5, Aug 26, 2020, 114)
-   PASSED
-   ...
-   ==> [2021-04-26-17:35:24.603152] test: ensuring copied examples cleaned up
-   ==> [2021-04-26-17:35:24.603807] "/usr/bin/make" "clean"
-   rm -f hello_c hello_cxx hello_mpifh hello_usempi hello_usempif08 hello_oshmem hello_oshmemcxx hello_oshmemfh Hello.class ring_c ring_cxx ring_mpifh ring_usempi ring_usempif08 ring_oshmem ring_oshmemfh Ring.class connectivity_c oshmem_shmalloc oshmem_circular_shift oshmem_max_reduction oshmem_strided_puts oshmem_symmetric_data spc_example *~ *.o
-   PASSED
-   ==> [2021-04-26-17:35:24.643360] test: mpicc: expect command status in [0]
-   ==> [2021-04-26-17:35:24.643834] "$SPACK_ROOT/opt/spack/linux-rhel7-broadwell/gcc-8.3.1/openmpi-4.0.5-eygjgvek35awfor2qaljltjind2oa67r/bin/mpicc" "-o" "mpi_hello_c" "$HOME/.spack/test/hyzq5eqlqfog6fawlzxwg3prqy5vjhms/openmpi-4.0.5-eygjgve/data/mpi/mpi_hello.c"
-   PASSED
-   ==> [2021-04-26-17:35:24.776765] test: mpirun: expect command status in [0]
-   ==> [2021-04-26-17:35:24.777194] "$SPACK_ROOT/opt/spack/linux-rhel7-broadwell/gcc-8.3.1/openmpi-4.0.5-eygjgvek35awfor2qaljltjind2oa67r/bin/mpirun" "-np" "1" "mpi_hello_c"
-   Hello world! From rank 0 of 1
-   PASSED
-   ...
-
+   ============================== 1 passed of 1 spec ==============================
 
-.. warning::
-
-   The API for adding and running stand-alone tests is not yet considered
-   stable and may change drastically in future releases. Packages with
-   stand-alone tests will be refactored to match changes to the API.
 
 .. _cmd-spack-test-list:
 
@@ -5514,11 +5641,11 @@ where only the outputs for the first of each set are shown:
 
 Packages available for install testing can be found using the
 ``spack test list`` command. The command outputs all installed
-packages that have defined ``test`` methods.
+packages that have defined stand-alone test methods.
 
 Alternatively you can use the ``--all`` option to get a list of
-all packages that have defined ``test`` methods even if they are
-not installed.
+all packages that have stand-alone test methods even if the packages
+are not installed.
 
 For more information, refer to `spack test list
 <https://spack.readthedocs.io/en/latest/command_index.html#spack-test-list>`_.
@@ -5530,22 +5657,22 @@ For more information, refer to `spack test list
 """"""""""""""""""
 
 Install tests can be run for one or more installed packages using
-the ``spack test run`` command. A ``test suite`` is created from
-the provided specs. If no specs are provided it will test all specs
-in the active environment or all specs installed in Spack if no
-environment is active.
+the ``spack test run`` command. A ``test suite`` is created for all
+of the provided specs. The command accepts the same arguments provided
+to ``spack install`` (see :ref:`sec-specs`). If no specs are provided
+the command tests all specs in the active environment or all specs
+installed in the Spack instance if no environment is active.
 
 Test suites can be named using the ``--alias`` option. Unaliased
-Test suites will use the content hash of their specs as their name.
+test suites use the content hash of their specs as their name.
 
 Some of the more commonly used debugging options are:
 
 - ``--fail-fast`` stops testing each package after the first failure
 - ``--fail-first`` stops testing packages after the first failure
 
-Test output is written to a text log file by default but ``junit``
-and ``cdash`` are outputs are available through the ``--log-format``
-option.
+Test output is written to a text log file by default though ``junit``
+and ``cdash`` are outputs available through the ``--log-format`` option.
 
 For more information, refer to `spack test run
 <https://spack.readthedocs.io/en/latest/command_index.html#spack-test-run>`_.
@@ -5558,8 +5685,8 @@ For more information, refer to `spack test run
 """"""""""""""""""""""
 
 The ``spack test results`` command shows results for all completed
-test suites. Providing the alias or content hash limits reporting
-to the corresponding test suite.
+test suites by default. The alias or content hash can be provided to
+limit reporting to the corresponding test suite.
 
 The ``--logs`` option includes the output generated by the associated
 test(s) to facilitate debugging.
@@ -5589,12 +5716,13 @@ For more information, refer to `spack test find
 """""""""""""""""""""
 
 The ``spack test remove`` command removes test suites to declutter
-the test results directory. You are prompted to confirm the removal
+the test stage directory. You are prompted to confirm the removal
 of each test suite **unless** you use the ``--yes-to-all`` option.
 
 For more information, refer to `spack test remove
 <https://spack.readthedocs.io/en/latest/command_index.html#spack-test-remove>`_.
 
+
 .. _file-manipulation:
 
 ---------------------------