Docs: Update stand-alone test information (#44755)

Update and slightly reorganize stand-alone test information to include new and improved examples and more links that can be used in PR feedback.

1 files changed, 387 insertions, 246 deletions

diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst
index ed61bdf81b..4a8112cfb4 100644
--- a/lib/spack/docs/packaging_guide.rst
+++ b/lib/spack/docs/packaging_guide.rst
@@ -5194,12 +5194,6 @@ installed executable. The check is implemented as follows:
                 reframe = Executable(self.prefix.bin.reframe)
                 reframe("-l")
 
-.. warning::
-
-    The API for adding tests is not yet considered stable and may change
-    in future releases.
-
-
 """"""""""""""""""""""""""""""""
 Checking build-time test results
 """"""""""""""""""""""""""""""""
@@ -5237,38 +5231,42 @@ be left in the build stage directory as illustrated below:
 Stand-alone tests
 ^^^^^^^^^^^^^^^^^
 
-While build-time tests are integrated with the build process, stand-alone
+While build-time tests are integrated with the installation process, stand-alone
 tests are expected to run days, weeks, even months after the software is
 installed. The goal is to provide a mechanism for gaining confidence that
 packages work as installed **and** *continue* to work as the underlying
 software evolves. Packages can add and inherit stand-alone tests. The
-`spack test`` command is used to manage stand-alone testing.
+``spack test`` command is used for stand-alone testing.
 
-.. note::
+.. admonition:: Stand-alone test methods should complete within a few minutes.
 
     Execution speed is important since these tests are intended to quickly
-    assess whether installed specs work on the system. Consequently, they
-    should run relatively quickly -- as in on the order of at most a few
-    minutes -- while ideally executing all, or at least key aspects of the
-    installed software.
-
-.. note::
-
-    Failing stand-alone tests indicate problems with the installation and,
-    therefore, there is no reason to proceed with more resource-intensive
-    tests until those have been investigated.
+    assess whether installed specs work on the system. Spack cannot spare
+    resources for more extensive testing of packages included in CI stacks.
 
-    Passing stand-alone tests indicate that more thorough testing, such
-    as running extensive unit or regression tests, or tests that run at
-    scale can proceed without wasting resources on a problematic installation.
+    Consequently, stand-alone tests should run relatively quickly -- as in
+    on the order of at most a few minutes -- while testing at least key aspects
+    of the installed software. Save more extensive testing for other tools.
 
 Tests are defined in the package using methods with names beginning ``test_``.
 This allows Spack to support multiple independent checks, or parts. Files
 needed for testing, such as source, data, and expected outputs, may be saved
 from the build and or stored with the package in the repository. Regardless
 of origin, these files are automatically copied to the spec's test stage
-directory prior to execution of the test method(s). Spack also provides some
-helper functions to facilitate processing.
+directory prior to execution of the test method(s). Spack also provides helper
+functions to facilitate common processing.
+
+.. tip::
+
+    **The status of stand-alone tests can be used to guide follow-up testing efforts.**
+
+    Passing stand-alone tests justify performing more thorough testing, such
+    as running extensive unit or regression tests or tests that run at scale,
+    when available. These tests are outside of the scope of Spack packaging.
+
+    Failing stand-alone tests indicate problems with the installation and,
+    therefore, no reason to proceed with more resource-intensive tests until
+    the failures have been investigated.
 
 .. _configure-test-stage:
 
@@ -5276,30 +5274,26 @@ helper functions to facilitate processing.
 Configuring the test stage directory
 """"""""""""""""""""""""""""""""""""
 
-Stand-alone tests utilize a test stage directory for building, running,
-and tracking results in the same way Spack uses a build stage directory.
-The default test stage root directory, ``~/.spack/test``, is defined in
-:ref:`etc/spack/defaults/config.yaml <config-yaml>`. This location is
-customizable by adding or changing the ``test_stage`` path in the high-level
-``config`` of the appropriate ``config.yaml`` file such that:
+Stand-alone tests utilize a test stage directory to build, run, and track
+tests in the same way Spack uses a build stage directory to install software.
+The default test stage root directory, ``$HOME/.spack/test``, is defined in
+:ref:`config.yaml <config-yaml>`. This location is customizable by adding or
+changing the ``test_stage`` path such that:
 
 .. code-block:: yaml
 
    config:
      test_stage: /path/to/test/stage
 
-Packages can use the ``self.test_suite.stage`` property to access this setting.
-Other package properties that provide access to spec-specific subdirectories
-and files are described in :ref:`accessing staged files <accessing-files>`.
+Packages can use the ``self.test_suite.stage`` property to access the path.
 
-.. note::
+.. admonition:: Each spec being tested has its own test stage directory.
 
-   The test stage path is the root directory for the **entire suite**.
-   In other words, it is the root directory for **all specs** being
-   tested by the ``spack test run`` command. Each spec gets its own
-   stage subdirectory. Use ``self.test_suite.test_dir_for_spec(self.spec)``
-   to access the spec-specific test stage directory.
+   The ``config:test_stage`` option is the path to the root of a
+   **test suite**'s stage directories.
 
+   Other package properties that provide paths to spec-specific subdirectories
+   and files are described in :ref:`accessing-files`.
 
 .. _adding-standalone-tests:
 
@@ -5312,61 +5306,144 @@ Test recipes are defined in the package using methods with names beginning
 Each method has access to the information Spack tracks on the package, such
 as options, compilers, and dependencies, supporting the customization of tests
 to the build. Standard python ``assert`` statements and other error reporting
-mechanisms are available. Such exceptions are automatically caught and reported
+mechanisms can be used. These exceptions are automatically caught and reported
 as test failures.
 
-Each test method is an implicit test part named by the method and whose
-purpose is the method's docstring. Providing a purpose gives context for
-aiding debugging. A test method may contain embedded test parts. Spack
-outputs the test name and purpose prior to running each test method and
-any embedded test parts. For example, ``MyPackage`` below provides two basic
-examples of installation tests: ``test_always_fails`` and ``test_example``.
-As the name indicates, the first always fails. The second simply runs the
-installed example.
+Each test method is an *implicit test part* named by the method. Its purpose
+is the method's docstring. Providing a meaningful purpose for the test gives
+context that can aid debugging. Spack outputs both the name and purpose at the
+start of test execution so it's also important that the docstring/purpose be
+brief.
+
+.. tip::
+
+    We recommend naming test methods so it is clear *what* is being tested.
+    For example, if a test method is building and or running an executable
+    called ``example``, then call the method ``test_example``. This, together
+    with a similarly meaningful test purpose, will aid test comprehension,
+    debugging, and maintainability.
+
+Stand-alone tests run in an environment that provides access to information
+on the installed software, such as build options, dependencies, and compilers.
+Build options and dependencies are accessed using the same spec checks used
+by build recipes. Examples of checking :ref:`variant settings <variants>` and
+:ref:`spec constraints <testing-specs>` can be found at the provided links.
+
+.. admonition:: Spack automatically sets up the test stage directory and environment.
+
+    Spack automatically creates the test stage directory and copies
+    relevant files *prior to* running tests. It can also ensure build
+    dependencies are available **if** necessary.
+
+    The path to the test stage is configurable (see :ref:`configure-test-stage`).
+
+    Files that Spack knows to copy are those saved from the build (see
+    :ref:`cache_extra_test_sources`) and those added to the package repository
+    (see :ref:`cache_custom_files`).
+
+    Spack will use the value of the ``test_requires_compiler`` property to
+    determine whether it needs to also set up build dependencies (see
+    :ref:`test-build-tests`).
+
+The ``MyPackage`` package below provides two basic test examples: 
+``test_example`` and ``test_example2``.  The first runs the installed
+``example`` and ensures its output contains an expected string. The second
+runs ``example2`` without checking output so is only concerned with confirming
+the executable runs successfully. If the installed spec is not expected to have
+``example2``, then the check at the top of the method will raise a special
+``SkipTest`` exception, which is captured to facilitate reporting skipped test
+parts to tools like CDash.
 
 .. code-block:: python
 
    class MyPackage(Package):
        ...
 
-       def test_always_fails(self):
-           """use assert to always fail"""
-           assert False
-
        def test_example(self):
-           """run installed example"""
+           """ensure installed example works"""
+           expected = "Done."
            example = which(self.prefix.bin.example)
-           example()
+
+           # Capture stdout and stderr from running the Executable
+           # and check that the expected output was produced.
+           out = example(output=str.split, error=str.split)
+           assert expected in out, f"Expected '{expected}' in the output"
+
+       def test_example2(self):
+           """run installed example2"""
+           if self.spec.satisfies("@:1.0"):
+               # Raise SkipTest to ensure flagging the test as skipped for
+               # test reporting purposes.
+               raise SkipTest("Test is only available for v1.1 on")
+
+           example2 = which(self.prefix.bin.example2)
+           example2()
 
 Output showing the identification of each test part after running the tests
 is illustrated below.
 
 .. code-block:: console
 
-   $ spack test run --alias mypackage mypackage@1.0
+   $ spack test run --alias mypackage mypackage@2.0
    ==> Spack test mypackage
    ...
    $ spack test results -l mypackage
    ==> Results for test suite 'mypackage':
    ...
-   ==> [2023-03-10-16:03:56.625204] test: test_always_fails: use assert to always fail
+   ==> [2024-03-10-16:03:56.625439] test: test_example: ensure installed example works
    ...
-   FAILED
-   ==> [2023-03-10-16:03:56.625439] test: test_example: run installed example
+   PASSED: MyPackage::test_example
+   ==> [2024-03-10-16:03:56.625439] test: test_example2: run installed example2
    ...
-   PASSED
+   PASSED: MyPackage::test_example2
 
+.. admonition:: Do NOT implement tests that must run in the installation prefix.
 
-.. note::
+   Use of the package spec's installation prefix for building and running
+   tests is **strongly discouraged**. Doing so causes permission errors for
+   shared spack instances *and* facilities that install the software in
+   read-only file systems or directories.
 
-   If ``MyPackage`` were a recipe for a library, the tests should build
-   an example or test program that is then executed.
+   Instead, start these test methods by explicitly copying the needed files
+   from the installation prefix to the test stage directory. Note the test
+   stage directory is the current directory when the test is executed with
+   the ``spack test run`` command.
 
-A test method can include test parts using the ``test_part`` context manager.
-Each part is treated as an independent check to allow subsequent test parts
-to execute even after a test part fails.
+.. admonition:: Test methods for library packages should build test executables.
 
-.. _test-part:
+   Stand-alone tests for library packages *should* build test executables
+   that utilize the *installed* library. Doing so ensures the tests follow
+   a similar build process that users of the library would follow.
+
+   For more information on how to do this, see :ref:`test-build-tests`.
+
+.. tip::
+
+   If you want to see more examples from packages with stand-alone tests, run
+   ``spack pkg grep "def\stest" | sed "s/\/package.py.*//g" | sort -u``
+   from the command line to get a list of the packages.
+
+.. _adding-standalone-test-parts:
+
+"""""""""""""""""""""""""""""
+Adding stand-alone test parts
+"""""""""""""""""""""""""""""
+
+Sometimes dependencies between steps of a test lend themselves to being
+broken into parts. Tracking the pass/fail status of each part may aid
+debugging. Spack provides a ``test_part`` context manager for use within
+test methods.
+
+Each test part is independently run, tracked, and reported. Test parts are
+executed in the order they appear. If one fails, subsequent test parts are
+still performed even if they would also fail. This allows tools like CDash
+to track and report the status of test parts across runs. The pass/fail status
+of the enclosing test is derived from the statuses of the embedded test parts.
+
+.. admonition:: Test method and test part names **must** be unique.
+
+   Test results reporting requires that test methods and embedded test parts
+   within a package have unique names.
 
 The signature for ``test_part`` is:
 
@@ -5388,40 +5465,68 @@ where each argument has the following meaning:
 * ``work_dir`` is the path to the directory in which the test will run.
 
   The default of ``None``, or ``"."``, corresponds to the the spec's test
-  stage (i.e., ``self.test_suite.test_dir_for_spec(self.spec)``.
+  stage (i.e., ``self.test_suite.test_dir_for_spec(self.spec)``).
 
-.. admonition:: Tests should **not** run under the installation directory.
+.. admonition:: Start test part names with the name of the enclosing test.
 
-   Use of the package spec's installation directory for building and running
-   tests is **strongly** discouraged. Doing so causes permission errors for
-   shared spack instances *and* facilities that install the software in
-   read-only file systems or directories.
+   We **highly recommend** starting the names of test parts with the name
+   of the enclosing test. Doing so helps with the comprehension, readability
+   and debugging of test results.
 
-Suppose ``MyPackage`` actually installs two examples we want to use for tests.
-These checks can be implemented as separate checks or, as illustrated below,
-embedded test parts.
+Suppose ``MyPackage`` installs multiple executables that need to run in a
+specific order since the outputs from one are inputs of others. Further suppose
+we want to add an integration test that runs the executables in order. We can
+accomplish this goal by implementing a stand-alone test method consisting of
+test parts for each executable as follows:
 
 .. code-block:: python
 
    class MyPackage(Package):
        ...
 
-       def test_example(self):
-           """run installed examples"""
-           for example in ["ex1", "ex2"]:
-               with test_part(
-                   self,
-                   f"test_example_{example}",
-                   purpose=f"run installed {example}",
-                ):
-                    exe = which(join_path(self.prefix.bin, example))
-                    exe()
-
-In this case, there will be an implicit test part for ``test_example``
-and separate sub-parts for ``ex1`` and ``ex2``. The second sub-part
-will be executed regardless of whether the first passes. The test
-log for a run where the first executable fails and the second passes
-is illustrated below.
+       def test_series(self):
+           """run setup, perform, and report"""
+
+           with test_part(self, "test_series_setup", purpose="setup operation"):
+                exe = which(self.prefix.bin.setup))
+                exe()
+
+           with test_part(self, "test_series_run", purpose="perform operation"):
+                exe = which(self.prefix.bin.run))
+                exe()
+
+           with test_part(self, "test_series_report", purpose="generate report"):
+                exe = which(self.prefix.bin.report))
+                exe()
+
+The result is ``test_series`` runs the following executable in order: ``setup``,
+``run``, and ``report``. In this case no options are passed to any of the
+executables and no outputs from running them are checked. Consequently, the
+implementation could be simplified with a for-loop as follows:
+
+.. code-block:: python
+
+   class MyPackage(Package):
+       ...
+
+       def test_series(self):
+           """execute series setup, run, and report"""
+
+           for exe, reason in [
+               ("setup", "setup operation"),
+               ("run", "perform operation"),
+               ("report", "generate report")
+           ]:
+               with test_part(self, f"test_series_{exe}", purpose=reason):
+                   exe = which(self.prefix.bin.join(exe))
+                   exe()
+
+In both cases, since we're using a context manager, each test part in
+``test_series`` will execute regardless of the status of the other test
+parts.
+
+Now let's look at the output from running the stand-alone tests where
+the second test part, ``test_series_run``, fails.
 
 .. code-block:: console
 
@@ -5431,50 +5536,67 @@ is illustrated below.
    $ spack test results -l mypackage
    ==> Results for test suite 'mypackage':
    ...
-   ==> [2023-03-10-16:03:56.625204] test: test_example: run installed examples
-   ==> [2023-03-10-16:03:56.625439] test: test_example_ex1: run installed ex1
+   ==> [2024-03-10-16:03:56.625204] test: test_series: execute series setup, run, and report
+   ==> [2024-03-10-16:03:56.625439] test: test_series_setup: setup operation
+   ...
+   PASSED: MyPackage::test_series_setup
+   ==> [2024-03-10-16:03:56.625555] test: test_series_run: perform operation
    ...
-   FAILED
-   ==> [2023-03-10-16:03:56.625555] test: test_example_ex2: run installed ex2
+   FAILED: MyPackage::test_series_run
+   ==> [2024-03-10-16:03:57.003456] test: test_series_report: generate report
    ...
-   PASSED
+   FAILED: MyPackage::test_series_report
+   FAILED: MyPackage::test_series
    ...
 
-.. warning::
+Since test parts depended on the success of previous parts, we see that the
+failure of one results in the failure of subsequent checks and the overall
+result of the test method, ``test_series``, is failure.
 
-   Test results reporting requires that each test method and embedded
-   test part for a package have a unique name.
+.. tip::
 
-Stand-alone tests run in an environment that provides access to information
-Spack has on how the software was built, such as build options, dependencies,
-and compilers. Build options and dependencies are accessed with the normal
-spec checks. Examples of checking :ref:`variant settings <variants>` and
-:ref:`spec constraints <testing-specs>` can be found at the provided links.
-Accessing compilers in stand-alone tests that are used by the build requires
-setting a package property as described :ref:`below <test-compilation>`.
+   If you want to see more examples from packages using ``test_part``, run
+   ``spack pkg grep "test_part(" | sed "s/\/package.py.*//g" | sort -u``
+   from the command line to get a list of the packages.
 
+.. _test-build-tests:
 
-.. _test-compilation:
+"""""""""""""""""""""""""""""""""""""
+Building and running test executables
+"""""""""""""""""""""""""""""""""""""
 
-"""""""""""""""""""""""""
-Enabling test compilation
-"""""""""""""""""""""""""
+.. admonition:: Re-use build-time sources and (small) input data sets when possible.
 
-If you want to build and run binaries in tests, then you'll need to tell
-Spack to load the package's compiler configuration. This is accomplished
-by setting the package's ``test_requires_compiler`` property to ``True``.
+    We **highly recommend** re-using build-time test sources and pared down
+    input files for testing installed software. These files are easier
+    to keep synchronized with software capabilities when they reside
+    within the software's repository. More information on saving files from
+    the installation process can be found at :ref:`cache_extra_test_sources`.
 
-Setting the property to ``True`` ensures access to the compiler through
-canonical environment variables (e.g., ``CC``, ``CXX``, ``FC``, ``F77``).
-It also gives access to build dependencies like ``cmake`` through their
-``spec objects`` (e.g., ``self.spec["cmake"].prefix.bin.cmake``).
+    If that is not possible, you can add test-related files to the package
+    repository (see :ref:`cache_custom_files`). It will be important to
+    remember to maintain them so they work across listed or supported versions
+    of the package.
 
-.. note::
+Packages that build libraries are good examples of cases where you'll want
+to build test executables from the installed software before running them.
+Doing so requires you to let Spack know it needs to load the package's
+compiler configuration. This is accomplished by setting the package's
+``test_requires_compiler`` property to ``True``.
+
+.. admonition:: ``test_requires_compiler = True`` is required to build test executables.
 
-   The ``test_requires_compiler`` property should be added at the top of
-   the package near other attributes, such as the ``homepage`` and ``url``.
+   Setting the property to ``True`` ensures access to the compiler through
+   canonical environment variables (e.g., ``CC``, ``CXX``, ``FC``, ``F77``).
+   It also gives access to build dependencies like ``cmake`` through their
+   ``spec objects`` (e.g., ``self.spec["cmake"].prefix.bin.cmake``).
 
-Below illustrates using this feature to compile an example.
+   Be sure to add the property at the top of the package class under other
+   properties like the ``homepage``.
+
+The example below, which ignores how ``cxx-example.cpp`` is acquired,
+illustrates the basic process of compiling a test executable using the
+installed library before running it.
 
 .. code-block:: python
 
@@ -5498,28 +5620,22 @@ Below illustrates using this feature to compile an example.
            cxx_example = which(exe)
            cxx_example()
 
+Typically the files used to build and or run test executables are either
+cached from the installation (see :ref:`cache_extra_test_sources`) or added
+to the package repository (see :ref:`cache_custom_files`). There is nothing
+preventing the use of both.
 
 .. _cache_extra_test_sources:
 
-"""""""""""""""""""""""
-Saving build-time files
-"""""""""""""""""""""""
-
-.. note::
-
-    We highly recommend re-using build-time test sources and pared down
-    input files for testing installed software. These files are easier
-    to keep synchronized with software capabilities since they reside
-    within the software's repository.
-
-    If that is not possible, you can add test-related files to the package
-    repository (see :ref:`adding custom files <cache_custom_files>`). It
-    will be important to maintain them so they work across listed or supported
-    versions of the package.
+""""""""""""""""""""""""""""""""""""
+Saving build- and install-time files
+""""""""""""""""""""""""""""""""""""
 
-You can use the ``cache_extra_test_sources`` helper to copy directories
-and or files from the source build stage directory to the package's
-installation directory.
+You can use the ``cache_extra_test_sources`` helper routine to copy
+directories and or files from the source build stage directory to the
+package's installation directory. Spack will automatically copy these
+files for you when it sets up the test stage directory and before it
+begins running the tests.
 
 The signature for ``cache_extra_test_sources`` is:
 
@@ -5534,46 +5650,69 @@ where each argument has the following meaning:
 * ``srcs`` is a string *or* a list of strings corresponding to the
   paths of subdirectories and or files needed for stand-alone testing.
 
-The paths must be relative to the staged source directory. Contents of
-subdirectories and files are copied to a special test cache subdirectory
-of the installation prefix. They are automatically copied to the appropriate
-relative paths under the test stage directory prior to executing stand-alone
-tests.
+.. warning::
+
+   Paths provided in the ``srcs`` argument **must be relative** to the
+   staged source directory. They will be copied to the equivalent relative
+   location under the test stage directory prior to test execution.
+
+Contents of subdirectories and files are copied to a special test cache
+subdirectory of the installation prefix. They are automatically copied to
+the appropriate relative paths under the test stage directory prior to
+executing stand-alone tests.
 
-For example, a package method for copying everything in the ``tests``
-subdirectory plus the ``foo.c`` and ``bar.c`` files from ``examples``
-and using ``foo.c`` in a test method is illustrated below.
+.. tip:: 
+
+    *Perform test-related conversions once when copying files.*
+
+    If one or more of the copied files needs to be modified to reference
+    the installed software, it is recommended that those changes be made
+    to the cached files **once** in the post-``install`` copy method
+    **after** the call to ``cache_extra_test_sources``. This will reduce
+    the amount of unnecessary work in the test method **and** avoid problems
+    running stand-alone tests in shared instances and facility deployments.
+
+    The ``filter_file`` function can be quite useful for such changes
+    (see :ref:`file-filtering`).
+
+Below is a basic example of a test that relies on files from the installation.
+This package method re-uses the contents of the ``examples`` subdirectory,
+which is assumed to have all of the files implemented to allow ``make`` to
+compile and link ``foo.c`` and ``bar.c`` against the package's installed
+library.
 
 .. code-block:: python
 
-   class MyLibPackage(Package):
+   class MyLibPackage(MakefilePackage):
        ...
 
        @run_after("install")
        def copy_test_files(self):
-           srcs = ["tests",
-                   join_path("examples", "foo.c"),
-                   join_path("examples", "bar.c")]
-           cache_extra_test_sources(self, srcs)
-
-       def test_foo(self):
-           exe = "foo"
-           src_dir = self.test_suite.current_test_cache_dir.examples
-           with working_dir(src_dir):
-               cc = which(os.environ["CC"])
-               cc(
-                   f"-L{self.prefix.lib}",
-                   f"-I{self.prefix.include}",
-                   f"{exe}.c",
-                   "-o", exe
-               )
-               foo = which(exe)
-               foo()
+           cache_extra_test_sources(self, "examples")
 
-In this case, the method copies the associated files from the build
-stage, **after** the software is installed, to the package's test
-cache directory. Then ``test_foo`` builds ``foo`` using ``foo.c``
-before running the program.
+       def test_example(self):
+           """build and run the examples"""
+           examples_dir = self.test_suite.current_test_cache_dir.examples
+           with working_dir(examples_dir):
+               make = which("make")
+               make()
+
+               for program in ["foo", "bar"]:
+                   with test_part(
+                       self,
+                       f"test_example_{program}",
+                       purpose=f"ensure {program} runs"
+                   ):
+                       exe = Executable(program)
+                       exe()
+
+In this case, ``copy_test_files`` copies the associated files from the
+build stage to the package's test cache directory under the installation
+prefix. Running ``spack test run`` for the package results in Spack copying
+the directory and its contents to the the test stage directory. The
+``working_dir`` context manager ensures the commands within it are executed
+from the ``examples_dir``. The test builds the software using ``make`` before
+running each executable, ``foo`` and ``bar``, as independent test parts.
 
 .. note::
 
@@ -5582,43 +5721,18 @@ before running the program.
 
    The key to copying files for stand-alone testing at build time is use
    of the ``run_after`` directive, which ensures the associated files are
-   copied **after** the provided build stage where the files **and**
-   installation prefix are available.
-
-These paths are **automatically copied** from cache to the test stage
-directory prior to the execution of any stand-alone tests. Tests access
-the files using the ``self.test_suite.current_test_cache_dir`` property.
-In our example above, test methods can use the following paths to reference
-the copy of each entry listed in ``srcs``, respectively:
-
-* ``self.test_suite.current_test_cache_dir.tests``
-* ``join_path(self.test_suite.current_test_cache_dir.examples, "foo.c")``
-* ``join_path(self.test_suite.current_test_cache_dir.examples, "bar.c")``
+   copied **after** the provided build stage (``install``) when the installation
+   prefix **and** files are available.
 
-.. admonition:: Library packages should build stand-alone tests
+   The test method uses the path contained in the package's
+   ``self.test_suite.current_test_cache_dir`` property for the root directory
+   of the copied files. In this case, that's the ``examples`` subdirectory.
 
-    Library developers will want to build the associated tests
-    against their **installed** libraries before running them.
-
-.. note::
-
-    While source and input files are generally recommended, binaries
-    **may** also be cached by the build process. Only you, as the package
-    writer or maintainer, know whether these files would be appropriate
-    for testing the installed software weeks to months later.
-
-.. note::
-
-    If one or more of the copied files needs to be modified to reference
-    the installed software, it is recommended that those changes be made
-    to the cached files **once** in the ``copy_test_sources`` method and
-    ***after** the call to ``cache_extra_test_sources()``. This will
-    reduce the amount of unnecessary work in the test method **and** avoid
-    problems testing in shared instances and facility deployments.
-
-    The ``filter_file`` function can be quite useful for such changes.
-    See :ref:`file manipulation <file-manipulation>`.
+.. tip::
 
+   If you want to see more examples from packages that cache build files, run
+   ``spack pkg grep cache_extra_test_sources | sed "s/\/package.py.*//g" | sort -u``
+   from the command line to get a list of the packages.
 
 .. _cache_custom_files:
 
@@ -5626,8 +5740,9 @@ the copy of each entry listed in ``srcs``, respectively:
 Adding custom files
 """""""""""""""""""
 
-In some cases it can be useful to have files that can be used to build or
-check the results of tests.  Examples include:
+Sometimes it is helpful or necessary to include custom files for building and
+or checking the results of tests as part of the package. Examples of the types
+of files that might be useful are:
 
 - test source files
 - test input files
@@ -5635,17 +5750,15 @@ check the results of tests.  Examples include:
 - expected test outputs
 
 While obtaining such files from the software repository is preferred (see
-:ref:`adding build-time files  <cache_extra_test_sources>`), there are
-circumstances where that is not feasible (e.g., the software is not being
-actively maintained). When test files can't be obtained from the repository
-or as a supplement to files that can, Spack supports the inclusion of
-additional files under the ``test`` subdirectory of the package in the
-Spack repository.
+:ref:`cache_extra_test_sources`), there are circumstances where doing so is not
+feasible such as when the software is not being actively maintained. When test
+files cannot be obtained from the repository or there is a need to supplement
+files that can, Spack supports the inclusion of additional files under the
+``test`` subdirectory of the package in the Spack repository.
 
-Spack **automatically copies** the contents of that directory to the
-test staging directory prior to running stand-alone tests. Test methods
-access those files using the ``self.test_suite.current_test_data_dir``
-property as shown below.
+The following example assumes a ``custom-example.c`` is saved in ``MyLibary``
+package's ``test`` subdirectory. It also assumes the program simply needs to
+be compiled and linked against the installed ``MyLibrary`` software.
 
 .. code-block:: python
 
@@ -5655,17 +5768,29 @@ property as shown below.
        test_requires_compiler = True
        ...
 
-       def test_example(self):
+       def test_custom_example(self):
            """build and run custom-example"""
-           data_dir = self.test_suite.current_test_data_dir
+           src_dir = self.test_suite.current_test_data_dir
            exe = "custom-example"
-           src = datadir.join(f"{exe}.cpp")
-           ...
-           # TODO: Build custom-example using src and exe
-           ...
-           custom_example = which(exe)
-           custom_example()
 
+           with working_dir(src_dir):
+               cc = which(os.environ["CC"])
+               cc(
+                   f"-L{self.prefix.lib}",
+                   f"-I{self.prefix.include}",
+                   f"{exe}.cpp",
+                   "-o", exe
+               )
+
+               custom_example = Executable(exe)
+               custom_example()
+
+In this case, ``spack test run`` for the package results in Spack copying
+the contents of the ``test`` subdirectory to the test stage directory path
+in ``self.test_suite.current_test_data_dir`` before calling
+``test_custom_example``. Use of the ``working_dir`` context manager
+ensures the commands to build and run the program are performed from
+within the appropriate subdirectory of the test stage.
 
 .. _expected_test_output_from_file:
 
@@ -5674,9 +5799,8 @@ Reading expected output from a file
 """""""""""""""""""""""""""""""""""
 
 The helper function ``get_escaped_text_output`` is available for packages
-to retrieve and properly format the text from a file that contains the
-expected output from running an executable that may contain special
-characters.
+to retrieve properly formatted text from a file potentially containing
+special characters.
 
 The signature for ``get_escaped_text_output`` is:
 
@@ -5686,10 +5810,13 @@ The signature for ``get_escaped_text_output`` is:
 
 where ``filename`` is the path to the file containing the expected output.
 
-The ``filename`` for a :ref:`custom file <cache_custom_files>` can be
-accessed by tests using the ``self.test_suite.current_test_data_dir``
-property. The example below illustrates how to read a file that was
-added to the package's ``test`` subdirectory.
+The path provided to ``filename`` for one of the copied custom files
+(:ref:`custom file <cache_custom_files>`) is in the path rooted at
+``self.test_suite.current_test_data_dir``.
+
+The example below shows how to reference both the custom database
+(``packages.db``) and expected output (``dump.out``) files Spack copies
+to the test stage:
 
 .. code-block:: python
 
@@ -5711,8 +5838,9 @@ added to the package's ``test`` subdirectory.
            for exp in expected:
                assert re.search(exp, out), f"Expected '{exp}' in output"
 
-If the file was instead copied from the ``tests`` subdirectory of the staged
-source code, the path would be obtained as shown below.
+If the files were instead cached from installing the software, the paths to the
+two files would be found under the ``self.test_suite.current_test_cache_dir``
+directory as shown below:
 
 .. code-block:: python
 
@@ -5720,17 +5848,24 @@ source code, the path would be obtained as shown below.
            """check example table dump"""
            test_cache_dir = self.test_suite.current_test_cache_dir
            db_filename = test_cache_dir.join("packages.db")
+           ..
+           expected = get_escaped_text_output(test_cache_dir.join("dump.out"))
+           ...
 
-Alternatively, if the file was copied to the ``share/tests`` subdirectory
-as part of the installation process, the test could access the path as
-follows:
+Alternatively, if both files had been installed by the software into the
+``share/tests`` subdirectory of the installation prefix, the paths to the
+two files would be referenced as follows:
 
 .. code-block:: python
 
        def test_example(self):
            """check example table dump"""
-           db_filename = join_path(self.prefix.share.tests, "packages.db")
-
+           db_filename = self.prefix.share.tests.join("packages.db")
+           ..
+           expected = get_escaped_text_output(
+               self.prefix.share.tests.join("dump.out")
+           )
+           ...
 
 .. _check_outputs:
 
@@ -5738,9 +5873,9 @@ follows:
 Comparing expected to actual outputs
 """"""""""""""""""""""""""""""""""""
 
-The helper function ``check_outputs`` is available for packages to ensure
-the expected outputs from running an executable are contained within the
-actual outputs.
+The ``check_outputs`` helper routine is available for packages to ensure
+multiple expected outputs from running an executable are contained within
+the actual outputs.
 
 The signature for ``check_outputs`` is:
 
@@ -5766,11 +5901,17 @@ Invoking the method is the equivalent of:
    if errors:
        raise RuntimeError("\n ".join(errors))
 
+.. tip::
+
+   If you want to see more examples from packages that use this helper, run
+   ``spack pkg grep check_outputs | sed "s/\/package.py.*//g" | sort -u``
+   from the command line to get a list of the packages.
+
 
 .. _accessing-files:
 
 """""""""""""""""""""""""""""""""""""""""
-Accessing package- and test-related files
+Finding package- and test-related files
 """""""""""""""""""""""""""""""""""""""""
 
 You may need to access files from one or more locations when writing
@@ -5779,8 +5920,7 @@ include test source files or includes them but has no way to build the
 executables using the installed headers and libraries. In these cases
 you may need to reference the files relative to one or more root directory.
 The table below lists relevant path properties and provides additional
-examples of their use.
-:ref:`Reading expected output <expected_test_output_from_file>` provides
+examples of their use. See :ref:`expected_test_output_from_file` for
 examples of accessing files saved from the software repository, package
 repository, and installation.
 
@@ -5809,7 +5949,6 @@ repository, and installation.
      - ``self.test_suite.current_test_data_dir``
      - ``join_path(self.test_suite.current_test_data_dir, "hello.f90")``
 
-
 .. _inheriting-tests:
 
 """"""""""""""""""""""""""""
@@ -5852,7 +5991,7 @@ maintainers provide additional stand-alone tests customized to the package.
 .. warning::
 
    Any package that implements a test method with the same name as an
-   inherited method overrides the inherited method. If that is not the
+   inherited method will override the inherited method. If that is not the
    goal and you are not explicitly calling and adding functionality to
    the inherited method for the test, then make sure that all test methods
    and embedded test parts have unique test names.
@@ -6017,6 +6156,8 @@ running:
 This is already part of the boilerplate for packages created with
 ``spack create``.
 
+.. _file-filtering:
+
 ^^^^^^^^^^^^^^^^^^^
 Filtering functions
 ^^^^^^^^^^^^^^^^^^^