From 21f30e3074f2a23f4fa945a658cec3fecb979938 Mon Sep 17 00:00:00 2001 From: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com> Date: Tue, 15 Dec 2020 08:38:00 -0800 Subject: Bugfix/docs: correct and expand smoke test documentation (#20278) --- lib/spack/docs/packaging_guide.rst | 221 +++++++++++++++++++++++++------------ 1 file changed, 151 insertions(+), 70 deletions(-) diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst index d8c2ac9d51..8f38dde1b5 100644 --- a/lib/spack/docs/packaging_guide.rst +++ b/lib/spack/docs/packaging_guide.rst @@ -3956,111 +3956,192 @@ Install Tests The API for adding and running install tests is not yet considered stable and may change drastically in future releases. Packages with - upstreamed tests will be refactored to match changes to the API. + install tests will be refactored to match changes to the API. -While build-tests are integrated with the build system, install tests -may be added to Spack packages to be run independently of the install -method. +While build tests are integrated with the build system, install tests +are for testing package installations independent of the build process. -Install tests may be added by defining a ``test`` method with the following signature: +Install tests are executed in a test stage directory that defaults to +``~/.spack/test``. You can change the location in the high-level ``config`` +by adding the ``test_stage`` path as follows: + +.. code-block:: yaml + + config: + test_stage: /path/to/stage + +"""""""""""""""""""" +Adding install tests +"""""""""""""""""""" + +Install tests are added to a package by defining a ``test`` method +with the following signature: .. code-block:: python def test(self): -These tests will be run in an environment set up to provide access to -this package and all of its dependencies, including ``test``-type -dependencies. Inside the ``test`` method, standard python ``assert`` -statements and other error reporting mechanisms can be used. Spack -will report any errors as a test failure. +These tests run in an environment that provides access to the +package and all of its dependencies, including ``test``-type +dependencies. + +Standard python ``assert`` statements and other error reporting +mechanisms can be used in the ``test`` method. Spack will report +such errors as test failures. -Inside the test method, individual tests can be run separately (and -continue transparently after a test failure) using the ``run_test`` -method. The signature for the ``run_test`` method is: +You can implement multiple tests (or test parts) within the ``test`` +method, where each is run separately and testing continues after +failures by using the ``run_test`` method. The signature for the +method is: .. code-block:: python def run_test(self, exe, options=[], expected=[], status=0, installed=False, purpose='', skip_missing=False, work_dir=None): -This method will operate in ``work_dir`` if one is specified. It will -search for an executable in the ``PATH`` variable named ``exe``, and -if ``installed=True`` it will fail if that executable does not come -from the prefix of the package being tested. If the executable is not -found, it will fail the test unless ``skip_missing`` is set to -``True``. The executable will be run with the options specified, and -the return code will be checked against the ``status`` argument, which -can be an integer or list of integers. Spack will also check that -every string in ``expected`` is a regex matching part of the output of -the executable. The ``purpose`` argument is recorded in the test log -for debugging purposes. - -"""""""""""""""""""""""""""""""""""""" -Install tests that require compilation -"""""""""""""""""""""""""""""""""""""" +The test fails if there is no executable named ``exe`` found in the +paths of the ``PATH`` variable **unless** ``skip_missing`` is ``True``. +The test also fails if the resulting path is not within the prefix of +the package being tested when ``installed`` is ``True``. + +The executable runs in ``work_dir``, when specified, using the provided +``options``. The return code is checked against the ``status`` argument, +which can be an integer or list of integers representing status codes +corresponding to successful execution. Spack also checks that every string +in ``expected`` is a regex matching part of the output from the test run. + +Output from the test is written to its log file. The ``purpose`` argument +serves as the heading in text logs to highlight the start of each test part. + +""""""""""""""""""""""""" +Enabling test compilation +""""""""""""""""""""""""" Some tests may require access to the compiler with which the package -was built, especially to test library-only packages. To ensure the -compiler is configured as part of the test environment, set the -attribute ``tests_require_compiler = True`` on the package. The -compiler will be available through the canonical environment variables -(``CC``, ``CXX``, ``FC``, ``F77``) in the test environment. +was built, especially to test library-only packages. You must enable +loading the package's compiler configuration by setting the attribute +``test_requires_compiler`` to ``True``. Doing so makes the compiler +available in the test environment through the canonical environment +variables (``CC``, ``CXX``, ``FC``, ``F77``). -"""""""""""""""""""""""""""""""""""""""""""""""" -Install tests that require build-time components -"""""""""""""""""""""""""""""""""""""""""""""""" +""""""""""""""""""""""" +Adding build-time files +""""""""""""""""""""""" + +.. note:: + + We highly recommend the re-use of build-time tests and input files + for testing installed software. These files are easier to keep + synchronized with the software than creating custom install tests. -Some packages cannot be easily tested without components from the -build-time test suite. For those packages, the -``cache_extra_test_sources`` method can be used. +You can use the ``cache_extra_test_sources`` method copy directories +and or files from the build stage directory to the package's installation +directory. + +For example, a package method for copying everything in the ``tests`` +subdirectory plus the ``foo.c`` and ``bar.c`` files from ``examples`` +can be implemented as follows: .. code-block:: python @run_after('install') def cache_test_sources(self): - srcs = ['./tests/foo.c', './tests/bar.c'] + srcs = ['tests', 'examples/foo.c', 'examples/bar.c'] self.cache_extra_test_sources(srcs) -This method will copy the listed methods into the metadata directory -of the package at the end of the install phase of the build. They will -be available to the test method in the directory -``self._extra_tests_path``. +The use of the ``run_after`` directive ensures the associated files +are copied **after** the package is installed during the build process. + +The method copies files to the package's metadata directory under +the ``self.install_test_root``. All files in the package source's +``tests`` directory for the example above will be copied to the +``join_path(self.install_test_root, 'tests')`` directory. The files +will be copied to the ``join_path(self.install_test_root, 'examples')`` +directory. + +.. note:: + + While source and input files are generally recommended, binaries + may also be cached by the build process for install testing. -While source files are generally recommended, for many packages -binaries may also technically be cached in this way for later testing. +""""""""""""""""""" +Adding custom files +""""""""""""""""""" + +Some tests may require additional files not available from the build. +Examples include: + +- test source files +- test input files +- test build scripts +- expected test output + +These extra files should be added to the ``test`` subdirectory of the +package in the repository. Spack will automaticaly copy any files in +that directory to the test staging directory during install testing. +The ``test`` method can access those files from the +``self.test_suite.current_test_data_dir`` directory. .. _cmd-spack-test: -""""""""""""""""""""" -Running install tests -""""""""""""""""""""" +""""""""""""""""""" +``spack test list`` +""""""""""""""""""" -Install tests can be run using the ``spack test run`` command. The -``spack test run`` command will create a ``test suite`` out of the -specs provided to it, or if no specs are provided it will test all -specs in the active environment, or all specs installed in Spack if no -environment is active. Test suites can be named using the ``--alias`` -option; test suites not aliased will use the content hash of their -specs as their name. +Packages available for install testing can be found using the +``spack test list`` command. The command outputs all installed +packages that have defined ``test`` methods. -Packages to install test can be queried using the ``spack test list`` -command, which outputs all installed packages with defined ``test`` -methods. +"""""""""""""""""" +``spack test run`` +"""""""""""""""""" -Test suites can be found using the ``spack test find`` command. It -will list all test suites that have been run and have not been removed -using the ``spack test remove`` command. The ``spack test remove`` -command will remove tests to declutter the test stage. The ``spack -test results`` command will show results for completed test suites. +Install tests can be run for one or more installed packages using +the ``spack test run`` command. A ``test suite`` is created from +the provided specs. If no specs are provided it will test all specs +in the active environment or all specs installed in Spack if no +environment is active. -The test stage is the working directory for all install tests run with -Spack. By default, Spack uses ``~/.spack/test`` as the test stage. The -test stage can be set in the high-level config: +Test suites can be named using the ``--alias`` option. Unaliased +Test suites will use the content hash of their specs as their name. -.. code-block:: yaml +Some of the more commonly used debugging options are: - config: - test_stage: /path/to/stage +- ``--fail-fast`` stops testing each package after the first failure +- ``--fail-first`` stops testing packages after the first failure + +Test output is written to a text log file by default but ``junit`` +and ``cdash`` are outputs are available through the ``--log-format`` +option. + +"""""""""""""""""""""" +``spack test results`` +"""""""""""""""""""""" + +The ``spack test results`` command shows results for all completed +test suites. Providing the alias or content hash limits reporting +to the corresponding test suite. + +The ``--logs`` option includes the output generated by the associated +test(s) to facilitate debugging. + +The ``--failed`` option limits results shown to that of the failed +tests, if any, of matching packages. + +""""""""""""""""""" +``spack test find`` +""""""""""""""""""" + +The ``spack test find`` command lists the aliases or content hashes +of all test suites whose results are available. + +""""""""""""""""""""" +``spack test remove`` +""""""""""""""""""""" + +The ``spack test remove`` command removes test suites to declutter +the test results directory. You are prompted to confirm the removal +of each test suite **unless** you use the ``--yes-to-all`` option. --------------------------- File manipulation functions -- cgit v1.2.3-70-g09d2