spack test (#15702)

Users can add test() methods to their packages to run smoke tests on
installations with the new `spack test` command (the old `spack test` is
now `spack unit-test`). spack test is environment-aware, so you can
`spack install` an environment and then run `spack test run` to run smoke
tests on all of its packages. Historical test logs can be perused with
`spack test results`. Generic smoke tests for MPI implementations, C,
C++, and Fortran compilers as well as specific smoke tests for 18
packages.

Inside the test method, individual tests can be run separately (and
continue to run best-effort after a test failure) using the `run_test`
method. The `run_test` method encapsulates finding test executables,
running and checking return codes, checking output, and error handling.

This handles the following trickier aspects of testing with direct
support in Spack's package API:

- [x] Caching source or intermediate build files at build time for
      use at test time.
- [x] Test dependencies,
- [x] packages that require a compiler for testing (such as library only
      packages).

See the packaging guide for more details on using Spack testing support.
Included is support for package.py files for virtual packages. This does
not change the Spack interface, but is a major change in internals.

Co-authored-by: Tamara Dahlgren <dahlgren1@llnl.gov>
Co-authored-by: wspear <wjspear@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

48 files changed, 2283 insertions, 620 deletions

diff --git a/lib/spack/docs/contribution_guide.rst b/lib/spack/docs/contribution_guide.rst
index 37cf9091bd..8df8ad65ba 100644
--- a/lib/spack/docs/contribution_guide.rst
+++ b/lib/spack/docs/contribution_guide.rst
@@ -74,7 +74,7 @@ locally to speed up the review process.
 We currently test against Python 2.6, 2.7, and 3.5-3.7 on both macOS and Linux and
 perform 3 types of tests:
 
-.. _cmd-spack-test:
+.. _cmd-spack-unit-test:
 
 ^^^^^^^^^^
 Unit Tests
@@ -96,7 +96,7 @@ To run *all* of the unit tests, use:
 
 .. code-block:: console
 
-   $ spack test
+   $ spack unit-test
 
 These tests may take several minutes to complete. If you know you are
 only modifying a single Spack feature, you can run subsets of tests at a
@@ -105,13 +105,13 @@ time.  For example, this would run all the tests in
 
 .. code-block:: console
 
-   $ spack test lib/spack/spack/test/architecture.py
+   $ spack unit-test lib/spack/spack/test/architecture.py
 
 And this would run the ``test_platform`` test from that file:
 
 .. code-block:: console
 
-   $ spack test lib/spack/spack/test/architecture.py::test_platform
+   $ spack unit-test lib/spack/spack/test/architecture.py::test_platform
 
 This allows you to develop iteratively: make a change, test that change,
 make another change, test that change, etc.  We use `pytest
@@ -121,29 +121,29 @@ pytest docs
 <http://doc.pytest.org/en/latest/usage.html#specifying-tests-selecting-tests>`_
 for more details on test selection syntax.
 
-``spack test`` has a few special options that can help you understand
-what tests are available.  To get a list of all available unit test
-files, run:
+``spack unit-test`` has a few special options that can help you
+understand what tests are available.  To get a list of all available
+unit test files, run:
 
-.. command-output:: spack test --list
+.. command-output:: spack unit-test --list
    :ellipsis: 5
 
-To see a more detailed list of available unit tests, use ``spack test
---list-long``:
+To see a more detailed list of available unit tests, use ``spack
+unit-test --list-long``:
 
-.. command-output:: spack test --list-long
+.. command-output:: spack unit-test --list-long
    :ellipsis: 10
 
 And to see the fully qualified names of all tests, use ``--list-names``:
 
-.. command-output:: spack test --list-names
+.. command-output:: spack unit-test --list-names
    :ellipsis: 5
 
 You can combine these with ``pytest`` arguments to restrict which tests
 you want to know about.  For example, to see just the tests in
 ``architecture.py``:
 
-.. command-output:: spack test --list-long lib/spack/spack/test/architecture.py
+.. command-output:: spack unit-test --list-long lib/spack/spack/test/architecture.py
 
 You can also combine any of these options with a ``pytest`` keyword
 search.  See the `pytest usage docs
@@ -151,7 +151,7 @@ search.  See the `pytest usage docs
 for more details on test selection syntax. For example, to see the names of all tests that have "spec"
 or "concretize" somewhere in their names:
 
-.. command-output:: spack test --list-names -k "spec and concretize"
+.. command-output:: spack unit-test --list-names -k "spec and concretize"
 
 By default, ``pytest`` captures the output of all unit tests, and it will
 print any captured output for failed tests. Sometimes it's helpful to see
@@ -161,7 +161,7 @@ argument to ``pytest``:
 
 .. code-block:: console
 
-   $ spack test -s --list-long lib/spack/spack/test/architecture.py::test_platform
+   $ spack unit-test -s --list-long lib/spack/spack/test/architecture.py::test_platform
 
 Unit tests are crucial to making sure bugs aren't introduced into
 Spack. If you are modifying core Spack libraries or adding new
@@ -176,7 +176,7 @@ how to write tests!
    You may notice the ``share/spack/qa/run-unit-tests`` script in the
    repository.  This script is designed for CI.  It runs the unit
    tests and reports coverage statistics back to Codecov. If you want to
-   run the unit tests yourself, we suggest you use ``spack test``.
+   run the unit tests yourself, we suggest you use ``spack unit-test``.
 
 ^^^^^^^^^^^^
 Flake8 Tests
diff --git a/lib/spack/docs/developer_guide.rst b/lib/spack/docs/developer_guide.rst
index 7fd4d1ec6e..b4b8b5d321 100644
--- a/lib/spack/docs/developer_guide.rst
+++ b/lib/spack/docs/developer_guide.rst
@@ -363,11 +363,12 @@ Developer commands
 ``spack doc``
 ^^^^^^^^^^^^^
 
-^^^^^^^^^^^^^^
-``spack test``
-^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^
+``spack unit-test``
+^^^^^^^^^^^^^^^^^^^
 
-See the :ref:`contributor guide section <cmd-spack-test>` on ``spack test``.
+See the :ref:`contributor guide section <cmd-spack-unit-test>` on
+``spack unit-test``.
 
 .. _cmd-spack-python:
 
diff --git a/lib/spack/docs/extensions.rst b/lib/spack/docs/extensions.rst
index c71a6511ed..15c59c76ef 100644
--- a/lib/spack/docs/extensions.rst
+++ b/lib/spack/docs/extensions.rst
@@ -87,11 +87,12 @@ will be available from the command line:
      --implicit       select specs that are not installed or were installed implicitly
      --output OUTPUT  where to dump the result
 
-The corresponding unit tests can be run giving the appropriate options to ``spack test``:
+The corresponding unit tests can be run giving the appropriate options
+to ``spack unit-test``:
 
 .. code-block:: console
 
-   $ spack test --extension=scripting
+   $ spack unit-test --extension=scripting
 
    ============================================================== test session starts ===============================================================
    platform linux2 -- Python 2.7.15rc1, pytest-3.2.5, py-1.4.34, pluggy-0.4.0
diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst
index 836fc12b83..ec83679a47 100644
--- a/lib/spack/docs/packaging_guide.rst
+++ b/lib/spack/docs/packaging_guide.rst
@@ -3948,6 +3948,118 @@ using the ``run_before`` decorator.
 
 .. _file-manipulation:
 
+^^^^^^^^^^^^^
+Install Tests
+^^^^^^^^^^^^^
+
+.. warning::
+
+   The API for adding and running install tests is not yet considered
+   stable and may change drastically in future releases. Packages with
+   upstreamed tests will be refactored to match changes to the API.
+
+While build-tests are integrated with the build system, install tests
+may be added to Spack packages to be run independently of the install
+method.
+
+Install tests may be added by defining a ``test`` method with the following signature:
+
+.. code-block:: python
+
+   def test(self):
+
+These tests will be run in an environment set up to provide access to
+this package and all of its dependencies, including ``test``-type
+dependencies. Inside the ``test`` method, standard python ``assert``
+statements and other error reporting mechanisms can be used. Spack
+will report any errors as a test failure.
+
+Inside the test method, individual tests can be run separately (and
+continue transparently after a test failure) using the ``run_test``
+method. The signature for the ``run_test`` method is:
+
+.. code-block:: python
+
+   def run_test(self, exe, options=[], expected=[], status=0, installed=False,
+                purpose='', skip_missing=False, work_dir=None):
+
+This method will operate in ``work_dir`` if one is specified. It will
+search for an executable in the ``PATH`` variable named ``exe``, and
+if ``installed=True`` it will fail if that executable does not come
+from the prefix of the package being tested. If the executable is not
+found, it will fail the test unless ``skip_missing`` is set to
+``True``. The executable will be run with the options specified, and
+the return code will be checked against the ``status`` argument, which
+can be an integer or list of integers. Spack will also check that
+every string in ``expected`` is a regex matching part of the output of
+the executable. The ``purpose`` argument is recorded in the test log
+for debugging purposes.
+
+""""""""""""""""""""""""""""""""""""""
+Install tests that require compilation
+""""""""""""""""""""""""""""""""""""""
+
+Some tests may require access to the compiler with which the package
+was built, especially to test library-only packages. To ensure the
+compiler is configured as part of the test environment, set the
+attribute ``tests_require_compiler = True`` on the package. The
+compiler will be available through the canonical environment variables
+(``CC``, ``CXX``, ``FC``, ``F77``) in the test environment.
+
+""""""""""""""""""""""""""""""""""""""""""""""""
+Install tests that require build-time components
+""""""""""""""""""""""""""""""""""""""""""""""""
+
+Some packages cannot be easily tested without components from the
+build-time test suite. For those packages, the
+``cache_extra_test_sources`` method can be used.
+
+.. code-block:: python
+
+   @run_after('install')
+   def cache_test_sources(self):
+       srcs = ['./tests/foo.c', './tests/bar.c']
+       self.cache_extra_test_sources(srcs)
+
+This method will copy the listed methods into the metadata directory
+of the package at the end of the install phase of the build. They will
+be available to the test method in the directory
+``self._extra_tests_path``.
+
+While source files are generally recommended, for many packages
+binaries may also technically be cached in this way for later testing.
+
+"""""""""""""""""""""
+Running install tests
+"""""""""""""""""""""
+
+Install tests can be run using the ``spack test run`` command. The
+``spack test run`` command will create a ``test suite`` out of the
+specs provided to it, or if no specs are provided it will test all
+specs in the active environment, or all specs installed in Spack if no
+environment is active. Test suites can be named using the ``--alias``
+option; test suites not aliased will use the content hash of their
+specs as their name.
+
+Packages to install test can be queried using the ``spack test list``
+command, which outputs all installed packages with defined ``test``
+methods.
+
+Test suites can be found using the ``spack test find`` command. It
+will list all test suites that have been run and have not been removed
+using the ``spack test remove`` command. The ``spack test remove``
+command will remove tests to declutter the test stage. The ``spack
+test results`` command will show results for completed test suites.
+
+The test stage is the working directory for all install tests run with
+Spack. By default, Spack uses ``~/.spack/test`` as the test stage. The
+test stage can be set in the high-level config:
+
+.. code-block:: yaml
+
+   config:
+     test_stage: /path/to/stage
+
 ---------------------------
 File manipulation functions
 ---------------------------
diff --git a/lib/spack/external/ctest_log_parser.py b/lib/spack/external/ctest_log_parser.py
index 0437b6e524..072c10d7a9 100644
--- a/lib/spack/external/ctest_log_parser.py
+++ b/lib/spack/external/ctest_log_parser.py
@@ -118,6 +118,7 @@ _error_matches = [
         "([^:]+): (Error:|error|undefined reference|multiply defined)",
         "([^ :]+) ?: (error|fatal error|catastrophic error)",
         "([^:]+)\\(([^\\)]+)\\) ?: (error|fatal error|catastrophic error)"),
+    "^FAILED",
     "^[Bb]us [Ee]rror",
     "^[Ss]egmentation [Vv]iolation",
     "^[Ss]egmentation [Ff]ault",
diff --git a/lib/spack/llnl/util/filesystem.py b/lib/spack/llnl/util/filesystem.py
index b8d0b4d2f1..d6579555ad 100644
--- a/lib/spack/llnl/util/filesystem.py
+++ b/lib/spack/llnl/util/filesystem.py
@@ -41,6 +41,8 @@ __all__ = [
     'fix_darwin_install_name',
     'force_remove',
     'force_symlink',
+    'chgrp',
+    'chmod_x',
     'copy',
     'install',
     'copy_tree',
@@ -52,6 +54,7 @@ __all__ = [
     'partition_path',
     'prefixes',
     'remove_dead_links',
+    'remove_directory_contents',
     'remove_if_dead_link',
     'remove_linked_tree',
     'set_executable',
@@ -1806,3 +1809,13 @@ def md5sum(file):
     with open(file, "rb") as f:
         md5.update(f.read())
     return md5.digest()
+
+
+def remove_directory_contents(dir):
+    """Remove all contents of a directory."""
+    if os.path.exists(dir):
+        for entry in [os.path.join(dir, entry) for entry in os.listdir(dir)]:
+            if os.path.isfile(entry) or os.path.islink(entry):
+                os.unlink(entry)
+            else:
+                shutil.rmtree(entry)
diff --git a/lib/spack/spack/build_environment.py b/lib/spack/spack/build_environment.py
index a1501034d4..cb1ba21ba5 100644
--- a/lib/spack/spack/build_environment.py
+++ b/lib/spack/spack/build_environment.py
@@ -32,8 +32,8 @@ There are two parts to the build environment:
 Skimming this module is a nice way to get acquainted with the types of
 calls you can make from within the install() function.
 """
-import re
 import inspect
+import re
 import multiprocessing
 import os
 import shutil
@@ -53,10 +53,14 @@ import spack.build_systems.meson
 import spack.config
 import spack.main
 import spack.paths
+import spack.package
+import spack.repo
 import spack.schema.environment
 import spack.store
+import spack.install_test
 import spack.subprocess_context
 import spack.architecture as arch
+import spack.util.path
 from spack.util.string import plural
 from spack.util.environment import (
     env_flag, filter_system_paths, get_path, is_system_path,
@@ -453,7 +457,6 @@ def _set_variables_for_single_module(pkg, module):
 
     jobs = spack.config.get('config:build_jobs', 16) if pkg.parallel else 1
     jobs = min(jobs, multiprocessing.cpu_count())
-    assert jobs is not None, "no default set for config:build_jobs"
 
     m = module
     m.make_jobs = jobs
@@ -713,28 +716,42 @@ def load_external_modules(pkg):
             load_module(external_module)
 
 
-def setup_package(pkg, dirty):
+def setup_package(pkg, dirty, context='build'):
     """Execute all environment setup routines."""
-    build_env = EnvironmentModifications()
+    env = EnvironmentModifications()
 
     if not dirty:
         clean_environment()
 
-    set_compiler_environment_variables(pkg, build_env)
-    set_build_environment_variables(pkg, build_env, dirty)
-    pkg.architecture.platform.setup_platform_environment(pkg, build_env)
+    # setup compilers and build tools for build contexts
+    need_compiler = context == 'build' or (context == 'test' and
+                                           pkg.test_requires_compiler)
+    if need_compiler:
+        set_compiler_environment_variables(pkg, env)
+        set_build_environment_variables(pkg, env, dirty)
 
-    build_env.extend(
-        modifications_from_dependencies(pkg.spec, context='build')
-    )
+    # architecture specific setup
+    pkg.architecture.platform.setup_platform_environment(pkg, env)
 
-    if (not dirty) and (not build_env.is_unset('CPATH')):
-        tty.debug("A dependency has updated CPATH, this may lead pkg-config"
-                  " to assume that the package is part of the system"
-                  " includes and omit it when invoked with '--cflags'.")
+    if context == 'build':
+        # recursive post-order dependency information
+        env.extend(
+            modifications_from_dependencies(pkg.spec, context=context)
+        )
+
+        if (not dirty) and (not env.is_unset('CPATH')):
+            tty.debug("A dependency has updated CPATH, this may lead pkg-"
+                      "config to assume that the package is part of the system"
+                      " includes and omit it when invoked with '--cflags'.")
 
-    set_module_variables_for_package(pkg)
-    pkg.setup_build_environment(build_env)
+        # setup package itself
+        set_module_variables_for_package(pkg)
+        pkg.setup_build_environment(env)
+    elif context == 'test':
+        import spack.user_environment as uenv  # avoid circular import
+        env.extend(uenv.environment_modifications_for_spec(pkg.spec))
+        set_module_variables_for_package(pkg)
+        env.prepend_path('PATH', '.')
 
     # Loading modules, in particular if they are meant to be used outside
     # of Spack, can change environment variables that are relevant to the
@@ -744,15 +761,16 @@ def setup_package(pkg, dirty):
     # unnecessary. Modules affecting these variables will be overwritten anyway
     with preserve_environment('CC', 'CXX', 'FC', 'F77'):
         # All module loads that otherwise would belong in previous
-        # functions have to occur after the build_env object has its
+        # functions have to occur after the env object has its
         # modifications applied. Otherwise the environment modifications
         # could undo module changes, such as unsetting LD_LIBRARY_PATH
         # after a module changes it.
-        for mod in pkg.compiler.modules:
-            # Fixes issue https://github.com/spack/spack/issues/3153
-            if os.environ.get("CRAY_CPU_TARGET") == "mic-knl":
-                load_module("cce")
-            load_module(mod)
+        if need_compiler:
+            for mod in pkg.compiler.modules:
+                # Fixes issue https://github.com/spack/spack/issues/3153
+                if os.environ.get("CRAY_CPU_TARGET") == "mic-knl":
+                    load_module("cce")
+                load_module(mod)
 
         # kludge to handle cray libsci being automatically loaded by PrgEnv
         # modules on cray platform. Module unload does no damage when
@@ -766,12 +784,12 @@ def setup_package(pkg, dirty):
 
     implicit_rpaths = pkg.compiler.implicit_rpaths()
     if implicit_rpaths:
-        build_env.set('SPACK_COMPILER_IMPLICIT_RPATHS',
-                      ':'.join(implicit_rpaths))
+        env.set('SPACK_COMPILER_IMPLICIT_RPATHS',
+                ':'.join(implicit_rpaths))
 
     # Make sure nothing's strange about the Spack environment.
-    validate(build_env, tty.warn)
-    build_env.apply_modifications()
+    validate(env, tty.warn)
+    env.apply_modifications()
 
 
 def modifications_from_dependencies(spec, context):
@@ -791,7 +809,8 @@ def modifications_from_dependencies(spec, context):
     deptype_and_method = {
         'build': (('build', 'link', 'test'),
                   'setup_dependent_build_environment'),
-        'run': (('link', 'run'), 'setup_dependent_run_environment')
+        'run': (('link', 'run'), 'setup_dependent_run_environment'),
+        'test': (('link', 'run', 'test'), 'setup_dependent_run_environment')
     }
     deptype, method = deptype_and_method[context]
 
@@ -808,6 +827,8 @@ def modifications_from_dependencies(spec, context):
 def _setup_pkg_and_run(serialized_pkg, function, kwargs, child_pipe,
                        input_multiprocess_fd):
 
+    context = kwargs.get('context', 'build')
+
     try:
         # We are in the child process. Python sets sys.stdin to
         # open(os.devnull) to prevent our process and its parent from
@@ -821,7 +842,8 @@ def _setup_pkg_and_run(serialized_pkg, function, kwargs, child_pipe,
 
         if not kwargs.get('fake', False):
             kwargs['unmodified_env'] = os.environ.copy()
-            setup_package(pkg, dirty=kwargs.get('dirty', False))
+            setup_package(pkg, dirty=kwargs.get('dirty', False),
+                          context=context)
         return_value = function(pkg, kwargs)
         child_pipe.send(return_value)
 
@@ -841,13 +863,18 @@ def _setup_pkg_and_run(serialized_pkg, function, kwargs, child_pipe,
         # show that, too.
         package_context = get_package_context(tb)
 
-        build_log = None
-        try:
-            if hasattr(pkg, 'log_path'):
-                build_log = pkg.log_path
-        except NameError:
-            # 'pkg' is not defined yet
-            pass
+        logfile = None
+        if context == 'build':
+            try:
+                if hasattr(pkg, 'log_path'):
+                    logfile = pkg.log_path
+            except NameError:
+                # 'pkg' is not defined yet
+                pass
+        elif context == 'test':
+            logfile = os.path.join(
+                pkg.test_suite.stage,
+                spack.install_test.TestSuite.test_log_name(pkg.spec))
 
         # make a pickleable exception to send to parent.
         msg = "%s: %s" % (exc_type.__name__, str(exc))
@@ -855,7 +882,7 @@ def _setup_pkg_and_run(serialized_pkg, function, kwargs, child_pipe,
         ce = ChildError(msg,
                         exc_type.__module__,
                         exc_type.__name__,
-                        tb_string, build_log, package_context)
+                        tb_string, logfile, context, package_context)
         child_pipe.send(ce)
 
     finally:
@@ -873,9 +900,6 @@ def start_build_process(pkg, function, kwargs):
             child process for.
         function (callable): argless function to run in the child
             process.
-        dirty (bool): If True, do NOT clean the environment before
-            building.
-        fake (bool): If True, skip package setup b/c it's not a real build
 
     Usage::
 
@@ -961,6 +985,7 @@ def get_package_context(traceback, context=3):
     Args:
         traceback (traceback): A traceback from some exception raised during
             install
+
         context (int): Lines of context to show before and after the line
             where the error happened
 
@@ -1067,13 +1092,14 @@ class ChildError(InstallError):
     # context instead of Python context.
     build_errors = [('spack.util.executable', 'ProcessError')]
 
-    def __init__(self, msg, module, classname, traceback_string, build_log,
-                 context):
+    def __init__(self, msg, module, classname, traceback_string, log_name,
+                 log_type, context):
         super(ChildError, self).__init__(msg)
         self.module = module
         self.name = classname
         self.traceback = traceback_string
-        self.build_log = build_log
+        self.log_name = log_name
+        self.log_type = log_type
         self.context = context
 
     @property
@@ -1081,26 +1107,16 @@ class ChildError(InstallError):
         out = StringIO()
         out.write(self._long_message if self._long_message else '')
 
+        have_log = self.log_name and os.path.exists(self.log_name)
+
         if (self.module, self.name) in ChildError.build_errors:
             # The error happened in some external executed process. Show
-            # the build log with errors or warnings highlighted.
-            if self.build_log and os.path.exists(self.build_log):
-                errors, warnings = parse_log_events(self.build_log)
-                nerr = len(errors)
-                nwar = len(warnings)
-                if nerr > 0:
-                    # If errors are found, only display errors
-                    out.write(
-                        "\n%s found in build log:\n" % plural(nerr, 'error'))
-                    out.write(make_log_context(errors))
-                elif nwar > 0:
-                    # If no errors are found but warnings are, display warnings
-                    out.write(
-                        "\n%s found in build log:\n" % plural(nwar, 'warning'))
-                    out.write(make_log_context(warnings))
+            # the log with errors or warnings highlighted.
+            if have_log:
+                write_log_summary(out, self.log_type, self.log_name)
 
         else:
-            # The error happened in in the Python code, so try to show
+            # The error happened in the Python code, so try to show
             # some context from the Package itself.
             if self.context:
                 out.write('\n')
@@ -1110,14 +1126,14 @@ class ChildError(InstallError):
         if out.getvalue():
             out.write('\n')
 
-        if self.build_log and os.path.exists(self.build_log):
-            out.write('See build log for details:\n')
-            out.write('  %s\n' % self.build_log)
+        if have_log:
+            out.write('See {0} log for details:\n'.format(self.log_type))
+            out.write('  {0}\n'.format(self.log_name))
 
         return out.getvalue()
 
     def __str__(self):
-        return self.message + self.long_message + self.traceback
+        return self.message
 
     def __reduce__(self):
         """__reduce__ is used to serialize (pickle) ChildErrors.
@@ -1130,13 +1146,14 @@ class ChildError(InstallError):
             self.module,
             self.name,
             self.traceback,
-            self.build_log,
+            self.log_name,
+            self.log_type,
             self.context)
 
 
-def _make_child_error(msg, module, name, traceback, build_log, context):
+def _make_child_error(msg, module, name, traceback, log, log_type, context):
     """Used by __reduce__ in ChildError to reconstruct pickled errors."""
-    return ChildError(msg, module, name, traceback, build_log, context)
+    return ChildError(msg, module, name, traceback, log, log_type, context)
 
 
 class StopPhase(spack.error.SpackError):
@@ -1147,3 +1164,30 @@ class StopPhase(spack.error.SpackError):
 
 def _make_stop_phase(msg, long_msg):
     return StopPhase(msg, long_msg)
+
+
+def write_log_summary(out, log_type, log, last=None):
+    errors, warnings = parse_log_events(log)
+    nerr = len(errors)
+    nwar = len(warnings)
+
+    if nerr > 0:
+        if last and nerr > last:
+            errors = errors[-last:]
+            nerr = last
+
+        # If errors are found, only display errors
+        out.write(
+            "\n%s found in %s log:\n" %
+            (plural(nerr, 'error'), log_type))
+        out.write(make_log_context(errors))
+    elif nwar > 0:
+        if last and nwar > last:
+            warnings = warnings[-last:]
+            nwar = last
+
+        # If no errors are found but warnings are, display warnings
+        out.write(
+            "\n%s found in %s log:\n" %
+            (plural(nwar, 'warning'), log_type))
+        out.write(make_log_context(warnings))
diff --git a/lib/spack/spack/build_systems/cmake.py b/lib/spack/spack/build_systems/cmake.py
index 4b679b358a..1336069846 100644
--- a/lib/spack/spack/build_systems/cmake.py
+++ b/lib/spack/spack/build_systems/cmake.py
@@ -325,13 +325,20 @@ class CMakePackage(PackageBase):
                                                                libs_flags))
 
     @property
+    def build_dirname(self):
+        """Returns the directory name to use when building the package
+
+        :return: name of the subdirectory for building the package
+        """
+        return 'spack-build-%s' % self.spec.dag_hash(7)
+
+    @property
     def build_directory(self):
         """Returns the directory to use when building the package
 
         :return: directory where to build the package
         """
-        dirname = 'spack-build-%s' % self.spec.dag_hash(7)
-        return os.path.join(self.stage.path, dirname)
+        return os.path.join(self.stage.path, self.build_dirname)
 
     def cmake_args(self):
         """Produces a list containing all the arguments that must be passed to
diff --git a/lib/spack/spack/build_systems/intel.py b/lib/spack/spack/build_systems/intel.py
index a74d5e9613..0e0bb9378b 100644
--- a/lib/spack/spack/build_systems/intel.py
+++ b/lib/spack/spack/build_systems/intel.py
@@ -1017,6 +1017,15 @@ class IntelPackage(PackageBase):
 
         env.extend(EnvironmentModifications.from_sourcing_file(f, *args))
 
+        if self.spec.name in ('intel', 'intel-parallel-studio'):
+            # this package provides compilers
+            # TODO: fix check above when compilers are dependencies
+            env.set('CC', self.prefix.bin.icc)
+            env.set('CXX', self.prefix.bin.icpc)
+            env.set('FC', self.prefix.bin.ifort)
+            env.set('F77', self.prefix.bin.ifort)
+            env.set('F90', self.prefix.bin.ifort)
+
     def setup_dependent_build_environment(self, env, dependent_spec):
         # NB: This function is overwritten by 'mpi' provider packages:
         #
diff --git a/lib/spack/spack/build_systems/python.py b/lib/spack/spack/build_systems/python.py
index 99dc4ce0dc..76159d88a1 100644
--- a/lib/spack/spack/build_systems/python.py
+++ b/lib/spack/spack/build_systems/python.py
@@ -89,7 +89,7 @@ class PythonPackage(PackageBase):
     build_system_class = 'PythonPackage'
 
     #: Callback names for build-time test
-    build_time_test_callbacks = ['test']
+    build_time_test_callbacks = ['build_test']
 
     #: Callback names for install-time test
     install_time_test_callbacks = ['import_module_test']
@@ -359,7 +359,7 @@ class PythonPackage(PackageBase):
 
     # Testing
 
-    def test(self):
+    def build_test(self):
         """Run unit tests after in-place build.
 
         These tests are only run if the package actually has a 'test' command.
diff --git a/lib/spack/spack/build_systems/scons.py b/lib/spack/spack/build_systems/scons.py
index ffa4390249..5e17666b71 100644
--- a/lib/spack/spack/build_systems/scons.py
+++ b/lib/spack/spack/build_systems/scons.py
@@ -33,7 +33,7 @@ class SConsPackage(PackageBase):
     build_system_class = 'SConsPackage'
 
     #: Callback names for build-time test
-    build_time_test_callbacks = ['test']
+    build_time_test_callbacks = ['build_test']
 
     depends_on('scons', type='build')
 
@@ -59,7 +59,7 @@ class SConsPackage(PackageBase):
 
     # Testing
 
-    def test(self):
+    def build_test(self):
         """Run unit tests after build.
 
         By default, does nothing. Override this if you want to
diff --git a/lib/spack/spack/build_systems/waf.py b/lib/spack/spack/build_systems/waf.py
index a1581660f2..a6dbbbdb35 100644
--- a/lib/spack/spack/build_systems/waf.py
+++ b/lib/spack/spack/build_systems/waf.py
@@ -47,10 +47,10 @@ class WafPackage(PackageBase):
     build_system_class = 'WafPackage'
 
     # Callback names for build-time test
-    build_time_test_callbacks = ['test']
+    build_time_test_callbacks = ['build_test']
 
     # Callback names for install-time test
-    install_time_test_callbacks = ['installtest']
+    install_time_test_callbacks = ['install_test']
 
     # Much like AutotoolsPackage does not require automake and autoconf
     # to build, WafPackage does not require waf to build. It only requires
@@ -106,7 +106,7 @@ class WafPackage(PackageBase):
 
     # Testing
 
-    def test(self):
+    def build_test(self):
         """Run unit tests after build.
 
         By default, does nothing. Override this if you want to
@@ -116,7 +116,7 @@ class WafPackage(PackageBase):
 
     run_after('build')(PackageBase._run_default_build_time_test_callbacks)
 
-    def installtest(self):
+    def install_test(self):
         """Run unit tests after install.
 
         By default, does nothing. Override this if you want to
diff --git a/lib/spack/spack/cmd/build_env.py b/lib/spack/spack/cmd/build_env.py
index 128d167a29..ef8b1f6e6b 100644
--- a/lib/spack/spack/cmd/build_env.py
+++ b/lib/spack/spack/cmd/build_env.py
@@ -2,86 +2,15 @@
 # Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
-
-from __future__ import print_function
-
-import argparse
-import os
-
-import llnl.util.tty as tty
-import spack.build_environment as build_environment
-import spack.cmd
-import spack.cmd.common.arguments as arguments
-from spack.util.environment import dump_environment, pickle_environment
+import spack.cmd.common.env_utility as env_utility
 
 description = "run a command in a spec's install environment, " \
               "or dump its environment to screen or file"
 section = "build"
 level = "long"
 
-
-def setup_parser(subparser):
-    arguments.add_common_arguments(subparser, ['clean', 'dirty'])
-    subparser.add_argument(
-        '--dump', metavar="FILE",
-        help="dump a source-able environment to FILE"
-    )
-    subparser.add_argument(
-        '--pickle', metavar="FILE",
-        help="dump a pickled source-able environment to FILE"
-    )
-    subparser.add_argument(
-        'spec', nargs=argparse.REMAINDER,
-        metavar='spec [--] [cmd]...',
-        help="spec of package environment to emulate")
-    subparser.epilog\
-        = 'If a command is not specified, the environment will be printed ' \
-        'to standard output (cf /usr/bin/env) unless --dump and/or --pickle ' \
-        'are specified.\n\nIf a command is specified and spec is ' \
-        'multi-word, then the -- separator is obligatory.'
+setup_parser = env_utility.setup_parser
 
 
 def build_env(parser, args):
-    if not args.spec:
-        tty.die("spack build-env requires a spec.")
-
-    # Specs may have spaces in them, so if they do, require that the
-    # caller put a '--' between the spec and the command to be
-    # executed.  If there is no '--', assume that the spec is the
-    # first argument.
-    sep = '--'
-    if sep in args.spec:
-        s = args.spec.index(sep)
-        spec = args.spec[:s]
-        cmd = args.spec[s + 1:]
-    else:
-        spec = args.spec[0]
-        cmd = args.spec[1:]
-
-    specs = spack.cmd.parse_specs(spec, concretize=True)
-    if len(specs) > 1:
-        tty.die("spack build-env only takes one spec.")
-    spec = specs[0]
-
-    build_environment.setup_package(spec.package, args.dirty)
-
-    if args.dump:
-        # Dump a source-able environment to a text file.
-        tty.msg("Dumping a source-able environment to {0}".format(args.dump))
-        dump_environment(args.dump)
-
-    if args.pickle:
-        # Dump a source-able environment to a pickle file.
-        tty.msg(
-            "Pickling a source-able environment to {0}".format(args.pickle))
-        pickle_environment(args.pickle)
-
-    if cmd:
-        # Execute the command with the new environment
-        os.execvp(cmd[0], cmd)
-
-    elif not bool(args.pickle or args.dump):
-        # If no command or dump/pickle option act like the "env" command
-        # and print out env vars.
-        for key, val in os.environ.items():
-            print("%s=%s" % (key, val))
+    env_utility.emulate_env_utility('build-env', 'build', args)
diff --git a/lib/spack/spack/cmd/clean.py b/lib/spack/spack/cmd/clean.py
index d847e7a7c0..f69b959293 100644
--- a/lib/spack/spack/cmd/clean.py
+++ b/lib/spack/spack/cmd/clean.py
@@ -10,10 +10,11 @@ import shutil
 import llnl.util.tty as tty
 
 import spack.caches
-import spack.cmd
+import spack.cmd.test
 import spack.cmd.common.arguments as arguments
 import spack.repo
 import spack.stage
+import spack.config
 from spack.paths import lib_path, var_path
 
 
diff --git a/lib/spack/spack/cmd/common/arguments.py b/lib/spack/spack/cmd/common/arguments.py
index e5945bda9c..e5c4c0dde8 100644
--- a/lib/spack/spack/cmd/common/arguments.py
+++ b/lib/spack/spack/cmd/common/arguments.py
@@ -275,3 +275,53 @@ def no_checksum():
     return Args(
         '-n', '--no-checksum', action='store_true', default=False,
         help="do not use checksums to verify downloaded files (unsafe)")
+
+
+def add_cdash_args(subparser, add_help):
+    cdash_help = {}
+    if add_help:
+        cdash_help['upload-url'] = "CDash URL where reports will be uploaded"
+        cdash_help['build'] = """The name of the build that will be reported to CDash.
+Defaults to spec of the package to operate on."""
+        cdash_help['site'] = """The site name that will be reported to CDash.
+Defaults to current system hostname."""
+        cdash_help['track'] = """Results will be reported to this group on CDash.
+Defaults to Experimental."""
+        cdash_help['buildstamp'] = """Instead of letting the CDash reporter prepare the
+buildstamp which, when combined with build name, site and project,
+uniquely identifies the build, provide this argument to identify
+the build yourself.  Format: %%Y%%m%%d-%%H%%M-[cdash-track]"""
+    else:
+        cdash_help['upload-url'] = argparse.SUPPRESS
+        cdash_help['build'] = argparse.SUPPRESS
+        cdash_help['site'] = argparse.SUPPRESS
+        cdash_help['track'] = argparse.SUPPRESS
+        cdash_help['buildstamp'] = argparse.SUPPRESS
+
+    subparser.add_argument(
+        '--cdash-upload-url',
+        default=None,
+        help=cdash_help['upload-url']
+    )
+    subparser.add_argument(
+        '--cdash-build',
+        default=None,
+        help=cdash_help['build']
+    )
+    subparser.add_argument(
+        '--cdash-site',
+        default=None,
+        help=cdash_help['site']
+    )
+
+    cdash_subgroup = subparser.add_mutually_exclusive_group()
+    cdash_subgroup.add_argument(
+        '--cdash-track',
+        default='Experimental',
+        help=cdash_help['track']
+    )
+    cdash_subgroup.add_argument(
+        '--cdash-buildstamp',
+        default=None,
+        help=cdash_help['buildstamp']
+    )
diff --git a/lib/spack/spack/cmd/common/env_utility.py b/lib/spack/spack/cmd/common/env_utility.py
new file mode 100644
index 0000000000..e3f32737b4
--- /dev/null
+++ b/lib/spack/spack/cmd/common/env_utility.py
@@ -0,0 +1,82 @@
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+from __future__ import print_function
+
+import argparse
+import os
+
+import llnl.util.tty as tty
+import spack.build_environment as build_environment
+import spack.paths
+import spack.cmd
+import spack.cmd.common.arguments as arguments
+from spack.util.environment import dump_environment, pickle_environment
+
+
+def setup_parser(subparser):
+    arguments.add_common_arguments(subparser, ['clean', 'dirty'])
+    subparser.add_argument(
+        '--dump', metavar="FILE",
+        help="dump a source-able environment to FILE"
+    )
+    subparser.add_argument(
+        '--pickle', metavar="FILE",
+        help="dump a pickled source-able environment to FILE"
+    )
+    subparser.add_argument(
+        'spec', nargs=argparse.REMAINDER,
+        metavar='spec [--] [cmd]...',
+        help="specs of package environment to emulate")
+    subparser.epilog\
+        = 'If a command is not specified, the environment will be printed ' \
+        'to standard output (cf /usr/bin/env) unless --dump and/or --pickle ' \
+        'are specified.\n\nIf a command is specified and spec is ' \
+        'multi-word, then the -- separator is obligatory.'
+
+
+def emulate_env_utility(cmd_name, context, args):
+    if not args.spec:
+        tty.die("spack %s requires a spec." % cmd_name)
+
+    # Specs may have spaces in them, so if they do, require that the
+    # caller put a '--' between the spec and the command to be
+    # executed.  If there is no '--', assume that the spec is the
+    # first argument.
+    sep = '--'
+    if sep in args.spec:
+        s = args.spec.index(sep)
+        spec = args.spec[:s]
+        cmd = args.spec[s + 1:]
+    else:
+        spec = args.spec[0]
+        cmd = args.spec[1:]
+
+    specs = spack.cmd.parse_specs(spec, concretize=True)
+    if len(specs) > 1:
+        tty.die("spack %s only takes one spec." % cmd_name)
+    spec = specs[0]
+
+    build_environment.setup_package(spec.package, args.dirty, context)
+
+    if args.dump:
+        # Dump a source-able environment to a text file.
+        tty.msg("Dumping a source-able environment to {0}".format(args.dump))
+        dump_environment(args.dump)
+
+    if args.pickle:
+        # Dump a source-able environment to a pickle file.
+        tty.msg(
+            "Pickling a source-able environment to {0}".format(args.pickle))
+        pickle_environment(args.pickle)
+
+    if cmd:
+        # Execute the command with the new environment
+        os.execvp(cmd[0], cmd)
+
+    elif not bool(args.pickle or args.dump):
+        # If no command or dump/pickle option then act like the "env" command
+        # and print out env vars.
+        for key, val in os.environ.items():
+            print("%s=%s" % (key, val))
diff --git a/lib/spack/spack/cmd/install.py b/lib/spack/spack/cmd/install.py
index c8673b5330..3b5954b1ad 100644
--- a/lib/spack/spack/cmd/install.py
+++ b/lib/spack/spack/cmd/install.py
@@ -167,65 +167,8 @@ packages. If neither are chosen, don't run tests for any packages."""
         action='store_true',
         help="Show usage instructions for CDash reporting"
     )
-    subparser.add_argument(
-        '-y', '--yes-to-all',
-        action='store_true',
-        dest='yes_to_all',
-        help="""assume "yes" is the answer to every confirmation request.
-To run completely non-interactively, also specify '--no-checksum'."""
-    )
-    add_cdash_args(subparser, False)
-    arguments.add_common_arguments(subparser, ['spec'])
-
-
-def add_cdash_args(subparser, add_help):
-    cdash_help = {}
-    if add_help:
-        cdash_help['upload-url'] = "CDash URL where reports will be uploaded"
-        cdash_help['build'] = """The name of the build that will be reported to CDash.
-Defaults to spec of the package to install."""
-        cdash_help['site'] = """The site name that will be reported to CDash.
-Defaults to current system hostname."""
-        cdash_help['track'] = """Results will be reported to this group on CDash.
-Defaults to Experimental."""
-        cdash_help['buildstamp'] = """Instead of letting the CDash reporter prepare the
-buildstamp which, when combined with build name, site and project,
-uniquely identifies the build, provide this argument to identify
-the build yourself.  Format: %%Y%%m%%d-%%H%%M-[cdash-track]"""
-    else:
-        cdash_help['upload-url'] = argparse.SUPPRESS
-        cdash_help['build'] = argparse.SUPPRESS
-        cdash_help['site'] = argparse.SUPPRESS
-        cdash_help['track'] = argparse.SUPPRESS
-        cdash_help['buildstamp'] = argparse.SUPPRESS
-
-    subparser.add_argument(
-        '--cdash-upload-url',
-        default=None,
-        help=cdash_help['upload-url']
-    )
-    subparser.add_argument(
-        '--cdash-build',
-        default=None,
-        help=cdash_help['build']
-    )
-    subparser.add_argument(
-        '--cdash-site',
-        default=None,
-        help=cdash_help['site']
-    )
-
-    cdash_subgroup = subparser.add_mutually_exclusive_group()
-    cdash_subgroup.add_argument(
-        '--cdash-track',
-        default='Experimental',
-        help=cdash_help['track']
-    )
-    cdash_subgroup.add_argument(
-        '--cdash-buildstamp',
-        default=None,
-        help=cdash_help['buildstamp']
-    )
+    arguments.add_cdash_args(subparser, False)
+    arguments.add_common_arguments(subparser, ['yes_to_all', 'spec'])
 
 
 def default_log_file(spec):
@@ -283,11 +226,12 @@ environment variables:
   SPACK_CDASH_AUTH_TOKEN
                         authentication token to present to CDash
                         '''))
-        add_cdash_args(parser, True)
+        arguments.add_cdash_args(parser, True)
         parser.print_help()
         return
 
-    reporter = spack.report.collect_info(args.log_format, args)
+    reporter = spack.report.collect_info(
+        spack.package.PackageInstaller, '_install_task', args.log_format, args)
     if args.log_file:
         reporter.filename = args.log_file
 
@@ -383,7 +327,7 @@ environment variables:
     if not args.log_file and not reporter.filename:
         reporter.filename = default_log_file(specs[0])
     reporter.specs = specs
-    with reporter:
+    with reporter('build'):
         if args.overwrite:
 
             installed = list(filter(lambda x: x,
diff --git a/lib/spack/spack/cmd/list.py b/lib/spack/spack/cmd/list.py
index 656282599d..a7d4cb0ac3 100644
--- a/lib/spack/spack/cmd/list.py
+++ b/lib/spack/spack/cmd/list.py
@@ -54,6 +54,9 @@ def setup_parser(subparser):
     subparser.add_argument(
         '--update', metavar='FILE', default=None, action='store',
         help='write output to the specified file, if any package is newer')
+    subparser.add_argument(
+        '-v', '--virtuals', action='store_true', default=False,
+        help='include virtual packages in list')
 
     arguments.add_common_arguments(subparser, ['tags'])
 
@@ -267,7 +270,7 @@ def list(parser, args):
     formatter = formatters[args.format]
 
     # Retrieve the names of all the packages
-    pkgs = set(spack.repo.all_package_names())
+    pkgs = set(spack.repo.all_package_names(args.virtuals))
     # Filter the set appropriately
     sorted_packages = filter_by_name(pkgs, args)
 
diff --git a/lib/spack/spack/cmd/test.py b/lib/spack/spack/cmd/test.py
index 8cbc0fbccf..3362b8a109 100644
--- a/lib/spack/spack/cmd/test.py
+++ b/lib/spack/spack/cmd/test.py
@@ -4,166 +4,381 @@
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
 from __future__ import print_function
-from __future__ import division
-
-import collections
-import sys
-import re
+import os
 import argparse
-import pytest
-from six import StringIO
+import textwrap
+import fnmatch
+import re
+import shutil
 
-import llnl.util.tty.color as color
-from llnl.util.filesystem import working_dir
-from llnl.util.tty.colify import colify
+import llnl.util.tty as tty
 
-import spack.paths
+import spack.install_test
+import spack.environment as ev
+import spack.cmd
+import spack.cmd.common.arguments as arguments
+import spack.report
+import spack.package
 
-description = "run spack's unit tests (wrapper around pytest)"
-section = "developer"
+description = "run spack's tests for an install"
+section = "administrator"
 level = "long"
 
 
+def first_line(docstring):
+    """Return the first line of the docstring."""
+    return docstring.split('\n')[0]
+
+
 def setup_parser(subparser):
-    subparser.add_argument(
-        '-H', '--pytest-help', action='store_true', default=False,
-        help="show full pytest help, with advanced options")
-
-    # extra spack arguments to list tests
-    list_group = subparser.add_argument_group("listing tests")
-    list_mutex = list_group.add_mutually_exclusive_group()
-    list_mutex.add_argument(
-        '-l', '--list', action='store_const', default=None,
-        dest='list', const='list', help="list test filenames")
-    list_mutex.add_argument(
-        '-L', '--list-long', action='store_const', default=None,
-        dest='list', const='long', help="list all test functions")
-    list_mutex.add_argument(
-        '-N', '--list-names', action='store_const', default=None,
-        dest='list', const='names', help="list full names of all tests")
-
-    # use tests for extension
-    subparser.add_argument(
-        '--extension', default=None,
-        help="run test for a given spack extension")
-
-    # spell out some common pytest arguments, so they'll show up in help
-    pytest_group = subparser.add_argument_group(
-        "common pytest arguments (spack test --pytest-help for more details)")
-    pytest_group.add_argument(
-        "-s", action='append_const', dest='parsed_args', const='-s',
-        help="print output while tests run (disable capture)")
-    pytest_group.add_argument(
-        "-k", action='store', metavar="EXPRESSION", dest='expression',
-        help="filter tests by keyword (can also use w/list options)")
-    pytest_group.add_argument(
-        "--showlocals", action='append_const', dest='parsed_args',
-        const='--showlocals', help="show local variable values in tracebacks")
-
-    # remainder is just passed to pytest
-    subparser.add_argument(
-        'pytest_args', nargs=argparse.REMAINDER, help="arguments for pytest")
-
-
-def do_list(args, extra_args):
-    """Print a lists of tests than what pytest offers."""
-    # Run test collection and get the tree out.
-    old_output = sys.stdout
-    try:
-        sys.stdout = output = StringIO()
-        pytest.main(['--collect-only'] + extra_args)
-    finally:
-        sys.stdout = old_output
-
-    lines = output.getvalue().split('\n')
-    tests = collections.defaultdict(lambda: set())
-    prefix = []
-
-    # collect tests into sections
-    for line in lines:
-        match = re.match(r"(\s*)<([^ ]*) '([^']*)'", line)
-        if not match:
-            continue
-        indent, nodetype, name = match.groups()
-
-        # strip parametrized tests
-        if "[" in name:
-            name = name[:name.index("[")]
-
-        depth = len(indent) // 2
-
-        if nodetype.endswith("Function"):
-            key = tuple(prefix)
-            tests[key].add(name)
-        else:
-            prefix = prefix[:depth]
-            prefix.append(name)
-
-    def colorize(c, prefix):
-        if isinstance(prefix, tuple):
-            return "::".join(
-                color.colorize("@%s{%s}" % (c, p))
-                for p in prefix if p != "()"
-            )
-        return color.colorize("@%s{%s}" % (c, prefix))
-
-    if args.list == "list":
-        files = set(prefix[0] for prefix in tests)
-        color_files = [colorize("B", file) for file in sorted(files)]
-        colify(color_files)
-
-    elif args.list == "long":
-        for prefix, functions in sorted(tests.items()):
-            path = colorize("*B", prefix) + "::"
-            functions = [colorize("c", f) for f in sorted(functions)]
-            color.cprint(path)
-            colify(functions, indent=4)
-            print()
-
-    else:  # args.list == "names"
-        all_functions = [
-            colorize("*B", prefix) + "::" + colorize("c", f)
-            for prefix, functions in sorted(tests.items())
-            for f in sorted(functions)
-        ]
-        colify(all_functions)
-
-
-def add_back_pytest_args(args, unknown_args):
-    """Add parsed pytest args, unknown args, and remainder together.
-
-    We add some basic pytest arguments to the Spack parser to ensure that
-    they show up in the short help, so we have to reassemble things here.
+    sp = subparser.add_subparsers(metavar='SUBCOMMAND', dest='test_command')
+
+    # Run
+    run_parser = sp.add_parser('run', description=test_run.__doc__,
+                               help=first_line(test_run.__doc__))
+
+    alias_help_msg = "Provide an alias for this test-suite"
+    alias_help_msg += " for subsequent access."
+    run_parser.add_argument('--alias', help=alias_help_msg)
+
+    run_parser.add_argument(
+        '--fail-fast', action='store_true',
+        help="Stop tests for each package after the first failure."
+    )
+    run_parser.add_argument(
+        '--fail-first', action='store_true',
+        help="Stop after the first failed package."
+    )
+    run_parser.add_argument(
+        '--keep-stage',
+        action='store_true',
+        help='Keep testing directory for debugging'
+    )
+    run_parser.add_argument(
+        '--log-format',
+        default=None,
+        choices=spack.report.valid_formats,
+        help="format to be used for log files"
+    )
+    run_parser.add_argument(
+        '--log-file',
+        default=None,
+        help="filename for the log file. if not passed a default will be used"
+    )
+    arguments.add_cdash_args(run_parser, False)
+    run_parser.add_argument(
+        '--help-cdash',
+        action='store_true',
+        help="Show usage instructions for CDash reporting"
+    )
+
+    cd_group = run_parser.add_mutually_exclusive_group()
+    arguments.add_common_arguments(cd_group, ['clean', 'dirty'])
+
+    arguments.add_common_arguments(run_parser, ['installed_specs'])
+
+    # List
+    sp.add_parser('list', description=test_list.__doc__,
+                  help=first_line(test_list.__doc__))
+
+    # Find
+    find_parser = sp.add_parser('find', description=test_find.__doc__,
+                                help=first_line(test_find.__doc__))
+    find_parser.add_argument(
+        'filter', nargs=argparse.REMAINDER,
+        help='optional case-insensitive glob patterns to filter results.')
+
+    # Status
+    status_parser = sp.add_parser('status', description=test_status.__doc__,
+                                  help=first_line(test_status.__doc__))
+    status_parser.add_argument(
+        'names', nargs=argparse.REMAINDER,
+        help="Test suites for which to print status")
+
+    # Results
+    results_parser = sp.add_parser('results', description=test_results.__doc__,
+                                   help=first_line(test_results.__doc__))
+    results_parser.add_argument(
+        '-l', '--logs', action='store_true',
+        help="print the test log for each matching package")
+    results_parser.add_argument(
+        '-f', '--failed', action='store_true',
+        help="only show results for failed tests of matching packages")
+    results_parser.add_argument(
+        'names', nargs=argparse.REMAINDER,
+        metavar='[name(s)] [-- installed_specs]...',
+        help="suite names and installed package constraints")
+    results_parser.epilog = 'Test results will be filtered by space-' \
+        'separated suite name(s) and installed\nspecs when provided.  '\
+        'If names are provided, then only results for those test\nsuites '\
+        'will be shown.  If installed specs are provided, then ony results'\
+        '\nmatching those specs will be shown.'
+
+    # Remove
+    remove_parser = sp.add_parser('remove', description=test_remove.__doc__,
+                                  help=first_line(test_remove.__doc__))
+    arguments.add_common_arguments(remove_parser, ['yes_to_all'])
+    remove_parser.add_argument(
+        'names', nargs=argparse.REMAINDER,
+        help="Test suites to remove from test stage")
+
+
+def test_run(args):
+    """Run tests for the specified installed packages.
+
+    If no specs are listed, run tests for all packages in the current
+    environment or all installed packages if there is no active environment.
     """
-    result = args.parsed_args or []
-    result += unknown_args or []
-    result += args.pytest_args or []
-    if args.expression:
-        result += ["-k", args.expression]
-    return result
-
-
-def test(parser, args, unknown_args):
-    if args.pytest_help:
-        # make the pytest.main help output more accurate
-        sys.argv[0] = 'spack test'
-        return pytest.main(['-h'])
-
-    # add back any parsed pytest args we need to pass to pytest
-    pytest_args = add_back_pytest_args(args, unknown_args)
-
-    # The default is to test the core of Spack. If the option `--extension`
-    # has been used, then test that extension.
-    pytest_root = spack.paths.spack_root
-    if args.extension:
-        target = args.extension
-        extensions = spack.config.get('config:extensions')
-        pytest_root = spack.extensions.path_for_extension(target, *extensions)
-
-    # pytest.ini lives in the root of the spack repository.
-    with working_dir(pytest_root):
-        if args.list:
-            do_list(args, pytest_args)
+    # cdash help option
+    if args.help_cdash:
+        parser = argparse.ArgumentParser(
+            formatter_class=argparse.RawDescriptionHelpFormatter,
+            epilog=textwrap.dedent('''\
+environment variables:
+  SPACK_CDASH_AUTH_TOKEN
+                        authentication token to present to CDash
+                        '''))
+        arguments.add_cdash_args(parser, True)
+        parser.print_help()
+        return
+
+    # set config option for fail-fast
+    if args.fail_fast:
+        spack.config.set('config:fail_fast', True, scope='command_line')
+
+    # Get specs to test
+    env = ev.get_env(args, 'test')
+    hashes = env.all_hashes() if env else None
+
+    specs = spack.cmd.parse_specs(args.specs) if args.specs else [None]
+    specs_to_test = []
+    for spec in specs:
+        matching = spack.store.db.query_local(spec, hashes=hashes)
+        if spec and not matching:
+            tty.warn("No installed packages match spec %s" % spec)
+        specs_to_test.extend(matching)
+
+    # test_stage_dir
+    test_suite = spack.install_test.TestSuite(specs_to_test, args.alias)
+    test_suite.ensure_stage()
+    tty.msg("Spack test %s" % test_suite.name)
+
+    # Set up reporter
+    setattr(args, 'package', [s.format() for s in test_suite.specs])
+    reporter = spack.report.collect_info(
+        spack.package.PackageBase, 'do_test', args.log_format, args)
+    if not reporter.filename:
+        if args.log_file:
+            if os.path.isabs(args.log_file):
+                log_file = args.log_file
+            else:
+                log_dir = os.getcwd()
+                log_file = os.path.join(log_dir, args.log_file)
+        else:
+            log_file = os.path.join(
+                os.getcwd(),
+                'test-%s' % test_suite.name)
+        reporter.filename = log_file
+    reporter.specs = specs_to_test
+
+    with reporter('test', test_suite.stage):
+        test_suite(remove_directory=not args.keep_stage,
+                   dirty=args.dirty,
+                   fail_first=args.fail_first)
+
+
+def has_test_method(pkg):
+    return pkg.test.__func__ != spack.package.PackageBase.test
+
+
+def test_list(args):
+    """List all installed packages with available tests."""
+    # TODO: This can be extended to have all of the output formatting options
+    # from `spack find`.
+    env = ev.get_env(args, 'test')
+    hashes = env.all_hashes() if env else None
+
+    specs = spack.store.db.query(hashes=hashes)
+    specs = list(filter(lambda s: has_test_method(s.package), specs))
+
+    spack.cmd.display_specs(specs, long=True)
+
+
+def test_find(args):  # TODO: merge with status (noargs)
+    """Find tests that are running or have available results.
+
+    Displays aliases for tests that have them, otherwise test suite content
+    hashes."""
+    test_suites = spack.install_test.get_all_test_suites()
+
+    # Filter tests by filter argument
+    if args.filter:
+        def create_filter(f):
+            raw = fnmatch.translate('f' if '*' in f or '?' in f
+                                    else '*' + f + '*')
+            return re.compile(raw, flags=re.IGNORECASE)
+        filters = [create_filter(f) for f in args.filter]
+
+        def match(t, f):
+            return f.match(t)
+        test_suites = [t for t in test_suites
+                       if any(match(t.alias, f) for f in filters) and
+                       os.path.isdir(t.stage)]
+
+    names = [t.name for t in test_suites]
+
+    if names:
+        # TODO: Make these specify results vs active
+        msg = "Spack test results available for the following tests:\n"
+        msg += "        %s\n" % ' '.join(names)
+        msg += "    Run `spack test remove` to remove all tests"
+        tty.msg(msg)
+    else:
+        msg = "No test results match the query\n"
+        msg += "        Tests may have been removed using `spack test remove`"
+        tty.msg(msg)
+
+
+def test_status(args):
+    """Get the current status for the specified Spack test suite(s)."""
+    if args.names:
+        test_suites = []
+        for name in args.names:
+            test_suite = spack.install_test.get_test_suite(name)
+            if test_suite:
+                test_suites.append(test_suite)
+            else:
+                tty.msg("No test suite %s found in test stage" % name)
+    else:
+        test_suites = spack.install_test.get_all_test_suites()
+        if not test_suites:
+            tty.msg("No test suites with status to report")
+
+    for test_suite in test_suites:
+        # TODO: Make this handle capability tests too
+        # TODO: Make this handle tests running in another process
+        tty.msg("Test suite %s completed" % test_suite.name)
+
+
+def _report_suite_results(test_suite, args, constraints):
+    """Report the relevant test suite results."""
+
+    # TODO: Make this handle capability tests too
+    # The results file may turn out to be a placeholder for future work
+
+    if constraints:
+        # TBD: Should I be refactoring or re-using ConstraintAction?
+        qspecs = spack.cmd.parse_specs(constraints)
+        specs = {}
+        for spec in qspecs:
+            for s in spack.store.db.query(spec, installed=True):
+                specs[s.dag_hash()] = s
+        specs = sorted(specs.values())
+        test_specs = dict((test_suite.test_pkg_id(s), s) for s in
+                          test_suite.specs if s in specs)
+    else:
+        test_specs = dict((test_suite.test_pkg_id(s), s) for s in
+                          test_suite.specs)
+
+    if not test_specs:
+        return
+
+    if os.path.exists(test_suite.results_file):
+        results_desc = 'Failing results' if args.failed else 'Results'
+        matching = ", spec matching '{0}'".format(' '.join(constraints)) \
+            if constraints else ''
+        tty.msg("{0} for test suite '{1}'{2}:"
+                .format(results_desc, test_suite.name, matching))
+
+        results = {}
+        with open(test_suite.results_file, 'r') as f:
+            for line in f:
+                pkg_id, status = line.split()
+                results[pkg_id] = status
+
+        for pkg_id in test_specs:
+            if pkg_id in results:
+                status = results[pkg_id]
+                if args.failed and status != 'FAILED':
+                    continue
+
+                msg = "  {0} {1}".format(pkg_id, status)
+                if args.logs:
+                    spec = test_specs[pkg_id]
+                    log_file = test_suite.log_file_for_spec(spec)
+                    if os.path.isfile(log_file):
+                        with open(log_file, 'r') as f:
+                            msg += '\n{0}'.format(''.join(f.readlines()))
+                tty.msg(msg)
+    else:
+        msg = "Test %s has no results.\n" % test_suite.name
+        msg += "        Check if it is running with "
+        msg += "`spack test status %s`" % test_suite.name
+        tty.msg(msg)
+
+
+def test_results(args):
+    """Get the results from Spack test suite(s) (default all)."""
+    if args.names:
+        try:
+            sep_index = args.names.index('--')
+            names = args.names[:sep_index]
+            constraints = args.names[sep_index + 1:]
+        except ValueError:
+            names = args.names
+            constraints = None
+    else:
+        names, constraints = None, None
+
+    if names:
+        test_suites = [spack.install_test.get_test_suite(name) for name
+                       in names]
+        if not test_suites:
+            tty.msg('No test suite(s) found in test stage: {0}'
+                    .format(', '.join(names)))
+    else:
+        test_suites = spack.install_test.get_all_test_suites()
+        if not test_suites:
+            tty.msg("No test suites with results to report")
+
+    for test_suite in test_suites:
+        _report_suite_results(test_suite, args, constraints)
+
+
+def test_remove(args):
+    """Remove results from Spack test suite(s) (default all).
+
+    If no test suite is listed, remove results for all suites.
+
+    Removed tests can no longer be accessed for results or status, and will not
+    appear in `spack test list` results."""
+    if args.names:
+        test_suites = []
+        for name in args.names:
+            test_suite = spack.install_test.get_test_suite(name)
+            if test_suite:
+                test_suites.append(test_suite)
+            else:
+                tty.msg("No test suite %s found in test stage" % name)
+    else:
+        test_suites = spack.install_test.get_all_test_suites()
+
+    if not test_suites:
+        tty.msg("No test suites to remove")
+        return
+
+    if not args.yes_to_all:
+        msg = 'The following test suites will be removed:\n\n'
+        msg += '    ' + '   '.join(test.name for test in test_suites) + '\n'
+        tty.msg(msg)
+        answer = tty.get_yes_or_no('Do you want to proceed?', default=False)
+        if not answer:
+            tty.msg('Aborting removal of test suites')
             return
 
-        return pytest.main(pytest_args)
+    for test_suite in test_suites:
+        shutil.rmtree(test_suite.stage)
+
+
+def test(parser, args):
+    globals()['test_%s' % args.test_command](args)
diff --git a/lib/spack/spack/cmd/test_env.py b/lib/spack/spack/cmd/test_env.py
new file mode 100644
index 0000000000..61e85046c1
--- /dev/null
+++ b/lib/spack/spack/cmd/test_env.py
@@ -0,0 +1,16 @@
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+import spack.cmd.common.env_utility as env_utility
+
+description = "run a command in a spec's test environment, " \
+              "or dump its environment to screen or file"
+section = "administration"
+level = "long"
+
+setup_parser = env_utility.setup_parser
+
+
+def test_env(parser, args):
+    env_utility.emulate_env_utility('test-env', 'test', args)
diff --git a/lib/spack/spack/cmd/unit_test.py b/lib/spack/spack/cmd/unit_test.py
new file mode 100644
index 0000000000..509211de04
--- /dev/null
+++ b/lib/spack/spack/cmd/unit_test.py
@@ -0,0 +1,169 @@
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+from __future__ import print_function
+from __future__ import division
+
+import collections
+import sys
+import re
+import argparse
+import pytest
+from six import StringIO
+
+import llnl.util.tty.color as color
+from llnl.util.filesystem import working_dir
+from llnl.util.tty.colify import colify
+
+import spack.paths
+
+description = "run spack's unit tests (wrapper around pytest)"
+section = "developer"
+level = "long"
+
+
+def setup_parser(subparser):
+    subparser.add_argument(
+        '-H', '--pytest-help', action='store_true', default=False,
+        help="show full pytest help, with advanced options")
+
+    # extra spack arguments to list tests
+    list_group = subparser.add_argument_group("listing tests")
+    list_mutex = list_group.add_mutually_exclusive_group()
+    list_mutex.add_argument(
+        '-l', '--list', action='store_const', default=None,
+        dest='list', const='list', help="list test filenames")
+    list_mutex.add_argument(
+        '-L', '--list-long', action='store_const', default=None,
+        dest='list', const='long', help="list all test functions")
+    list_mutex.add_argument(
+        '-N', '--list-names', action='store_const', default=None,
+        dest='list', const='names', help="list full names of all tests")
+
+    # use tests for extension
+    subparser.add_argument(
+        '--extension', default=None,
+        help="run test for a given spack extension")
+
+    # spell out some common pytest arguments, so they'll show up in help
+    pytest_group = subparser.add_argument_group(
+        "common pytest arguments (spack unit-test --pytest-help for more)")
+    pytest_group.add_argument(
+        "-s", action='append_const', dest='parsed_args', const='-s',
+        help="print output while tests run (disable capture)")
+    pytest_group.add_argument(
+        "-k", action='store', metavar="EXPRESSION", dest='expression',
+        help="filter tests by keyword (can also use w/list options)")
+    pytest_group.add_argument(
+        "--showlocals", action='append_const', dest='parsed_args',
+        const='--showlocals', help="show local variable values in tracebacks")
+
+    # remainder is just passed to pytest
+    subparser.add_argument(
+        'pytest_args', nargs=argparse.REMAINDER, help="arguments for pytest")
+
+
+def do_list(args, extra_args):
+    """Print a lists of tests than what pytest offers."""
+    # Run test collection and get the tree out.
+    old_output = sys.stdout
+    try:
+        sys.stdout = output = StringIO()
+        pytest.main(['--collect-only'] + extra_args)
+    finally:
+        sys.stdout = old_output
+
+    lines = output.getvalue().split('\n')
+    tests = collections.defaultdict(lambda: set())
+    prefix = []
+
+    # collect tests into sections
+    for line in lines:
+        match = re.match(r"(\s*)<([^ ]*) '([^']*)'", line)
+        if not match:
+            continue
+        indent, nodetype, name = match.groups()
+
+        # strip parametrized tests
+        if "[" in name:
+            name = name[:name.index("[")]
+
+        depth = len(indent) // 2
+
+        if nodetype.endswith("Function"):
+            key = tuple(prefix)
+            tests[key].add(name)
+        else:
+            prefix = prefix[:depth]
+            prefix.append(name)
+
+    def colorize(c, prefix):
+        if isinstance(prefix, tuple):
+            return "::".join(
+                color.colorize("@%s{%s}" % (c, p))
+                for p in prefix if p != "()"
+            )
+        return color.colorize("@%s{%s}" % (c, prefix))
+
+    if args.list == "list":
+        files = set(prefix[0] for prefix in tests)
+        color_files = [colorize("B", file) for file in sorted(files)]
+        colify(color_files)
+
+    elif args.list == "long":
+        for prefix, functions in sorted(tests.items()):
+            path = colorize("*B", prefix) + "::"
+            functions = [colorize("c", f) for f in sorted(functions)]
+            color.cprint(path)
+            colify(functions, indent=4)
+            print()
+
+    else:  # args.list == "names"
+        all_functions = [
+            colorize("*B", prefix) + "::" + colorize("c", f)
+            for prefix, functions in sorted(tests.items())
+            for f in sorted(functions)
+        ]
+        colify(all_functions)
+
+
+def add_back_pytest_args(args, unknown_args):
+    """Add parsed pytest args, unknown args, and remainder together.
+
+    We add some basic pytest arguments to the Spack parser to ensure that
+    they show up in the short help, so we have to reassemble things here.
+    """
+    result = args.parsed_args or []
+    result += unknown_args or []
+    result += args.pytest_args or []
+    if args.expression:
+        result += ["-k", args.expression]
+    return result
+
+
+def unit_test(parser, args, unknown_args):
+    if args.pytest_help:
+        # make the pytest.main help output more accurate
+        sys.argv[0] = 'spack test'
+        return pytest.main(['-h'])
+
+    # add back any parsed pytest args we need to pass to pytest
+    pytest_args = add_back_pytest_args(args, unknown_args)
+
+    # The default is to test the core of Spack. If the option `--extension`
+    # has been used, then test that extension.
+    pytest_root = spack.paths.spack_root
+    if args.extension:
+        target = args.extension
+        extensions = spack.config.get('config:extensions')
+        pytest_root = spack.extensions.path_for_extension(target, *extensions)
+
+    # pytest.ini lives in the root of the spack repository.
+    with working_dir(pytest_root):
+        if args.list:
+            do_list(args, pytest_args)
+            return
+
+        return pytest.main(pytest_args)
diff --git a/lib/spack/spack/directives.py b/lib/spack/spack/directives.py
index 7b2084d229..41276b5b48 100644
--- a/lib/spack/spack/directives.py
+++ b/lib/spack/spack/directives.py
@@ -299,8 +299,18 @@ def _depends_on(pkg, spec, when=None, type=default_deptype, patches=None):
 
     # call this patches here for clarity -- we want patch to be a list,
     # but the caller doesn't have to make it one.
-    if patches and dep_spec.virtual:
-        raise DependencyPatchError("Cannot patch a virtual dependency.")
+
+    # Note: we cannot check whether a package is virtual in a directive
+    # because directives are run as part of class instantiation, and specs
+    # instantiate the package class as part of the `virtual` check.
+    # To be technical, specs only instantiate the package class as part of the
+    # virtual check if the provider index hasn't been created yet.
+    # TODO: There could be a cache warming strategy that would allow us to
+    # ensure `Spec.virtual` is a valid thing to call in a directive.
+    # For now, we comment out the following check to allow for virtual packages
+    # with package files.
+    # if patches and dep_spec.virtual:
+    #     raise DependencyPatchError("Cannot patch a virtual dependency.")
 
     # ensure patches is a list
     if patches is None:
diff --git a/lib/spack/spack/install_test.py b/lib/spack/spack/install_test.py
new file mode 100644
index 0000000000..6c2c095a2e
--- /dev/null
+++ b/lib/spack/spack/install_test.py
@@ -0,0 +1,266 @@
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+import base64
+import hashlib
+import os
+import re
+import shutil
+import sys
+import tty
+
+import llnl.util.filesystem as fs
+
+from spack.spec import Spec
+
+import spack.error
+import spack.util.prefix
+import spack.util.spack_json as sjson
+
+
+test_suite_filename = 'test_suite.lock'
+results_filename = 'results.txt'
+
+
+def get_escaped_text_output(filename):
+    """Retrieve and escape the expected text output from the file
+
+    Args:
+        filename (str): path to the file
+
+    Returns:
+        (list of str): escaped text lines read from the file
+    """
+    with open(filename, 'r') as f:
+        # Ensure special characters are escaped as needed
+        expected = f.read()
+
+    # Split the lines to make it easier to debug failures when there is
+    # a lot of output
+    return [re.escape(ln) for ln in expected.split('\n')]
+
+
+def get_test_stage_dir():
+    return spack.util.path.canonicalize_path(
+        spack.config.get('config:test_stage', '~/.spack/test'))
+
+
+def get_all_test_suites():
+    stage_root = get_test_stage_dir()
+    if not os.path.isdir(stage_root):
+        return []
+
+    def valid_stage(d):
+        dirpath = os.path.join(stage_root, d)
+        return (os.path.isdir(dirpath) and
+                test_suite_filename in os.listdir(dirpath))
+
+    candidates = [
+        os.path.join(stage_root, d, test_suite_filename)
+        for d in os.listdir(stage_root)
+        if valid_stage(d)
+    ]
+
+    test_suites = [TestSuite.from_file(c) for c in candidates]
+    return test_suites
+
+
+def get_test_suite(name):
+    assert name, "Cannot search for empty test name or 'None'"
+    test_suites = get_all_test_suites()
+    names = [ts for ts in test_suites
+             if ts.name == name]
+    assert len(names) < 2, "alias shadows test suite hash"
+
+    if not names:
+        return None
+    return names[0]
+
+
+class TestSuite(object):
+    def __init__(self, specs, alias=None):
+        # copy so that different test suites have different package objects
+        # even if they contain the same spec
+        self.specs = [spec.copy() for spec in specs]
+        self.current_test_spec = None  # spec currently tested, can be virtual
+        self.current_base_spec = None  # spec currently running do_test
+
+        self.alias = alias
+        self._hash = None
+
+    @property
+    def name(self):
+        return self.alias if self.alias else self.content_hash
+
+    @property
+    def content_hash(self):
+        if not self._hash:
+            json_text = sjson.dump(self.to_dict())
+            sha = hashlib.sha1(json_text.encode('utf-8'))
+            b32_hash = base64.b32encode(sha.digest()).lower()
+            if sys.version_info[0] >= 3:
+                b32_hash = b32_hash.decode('utf-8')
+            self._hash = b32_hash
+        return self._hash
+
+    def __call__(self, *args, **kwargs):
+        self.write_reproducibility_data()
+
+        remove_directory = kwargs.get('remove_directory', True)
+        dirty = kwargs.get('dirty', False)
+        fail_first = kwargs.get('fail_first', False)
+
+        for spec in self.specs:
+            try:
+                msg = "A package object cannot run in two test suites at once"
+                assert not spec.package.test_suite, msg
+
+                # Set up the test suite to know which test is running
+                spec.package.test_suite = self
+                self.current_base_spec = spec
+                self.current_test_spec = spec
+
+                # setup per-test directory in the stage dir
+                test_dir = self.test_dir_for_spec(spec)
+                if os.path.exists(test_dir):
+                    shutil.rmtree(test_dir)
+                fs.mkdirp(test_dir)
+
+                # run the package tests
+                spec.package.do_test(
+                    dirty=dirty
+                )
+
+                # Clean up on success and log passed test
+                if remove_directory:
+                    shutil.rmtree(test_dir)
+                self.write_test_result(spec, 'PASSED')
+            except BaseException as exc:
+                if isinstance(exc, SyntaxError):
+                    # Create the test log file and report the error.
+                    self.ensure_stage()
+                    msg = 'Testing package {0}\n{1}'\
+                        .format(self.test_pkg_id(spec), str(exc))
+                    _add_msg_to_file(self.log_file_for_spec(spec), msg)
+
+                self.write_test_result(spec, 'FAILED')
+                if fail_first:
+                    break
+            finally:
+                spec.package.test_suite = None
+                self.current_test_spec = None
+                self.current_base_spec = None
+
+    def ensure_stage(self):
+        if not os.path.exists(self.stage):
+            fs.mkdirp(self.stage)
+
+    @property
+    def stage(self):
+        return spack.util.prefix.Prefix(
+            os.path.join(get_test_stage_dir(), self.content_hash))
+
+    @property
+    def results_file(self):
+        return self.stage.join(results_filename)
+
+    @classmethod
+    def test_pkg_id(cls, spec):
+        """Build the standard install test package identifier
+
+        Args:
+        spec (Spec): instance of the spec under test
+
+        Returns:
+        (str): the install test package identifier
+        """
+        return spec.format('{name}-{version}-{hash:7}')
+
+    @classmethod
+    def test_log_name(cls, spec):
+        return '%s-test-out.txt' % cls.test_pkg_id(spec)
+
+    def log_file_for_spec(self, spec):
+        return self.stage.join(self.test_log_name(spec))
+
+    def test_dir_for_spec(self, spec):
+        return self.stage.join(self.test_pkg_id(spec))
+
+    @property
+    def current_test_data_dir(self):
+        assert self.current_test_spec and self.current_base_spec
+        test_spec = self.current_test_spec
+        base_spec = self.current_base_spec
+        return self.test_dir_for_spec(base_spec).data.join(test_spec.name)
+
+    def add_failure(self, exc, msg):
+        current_hash = self.current_base_spec.dag_hash()
+        current_failures = self.failures.get(current_hash, [])
+        current_failures.append((exc, msg))
+        self.failures[current_hash] = current_failures
+
+    def write_test_result(self, spec, result):
+        msg = "{0} {1}".format(self.test_pkg_id(spec), result)
+        _add_msg_to_file(self.results_file, msg)
+
+    def write_reproducibility_data(self):
+        for spec in self.specs:
+            repo_cache_path = self.stage.repo.join(spec.name)
+            spack.repo.path.dump_provenance(spec, repo_cache_path)
+            for vspec in spec.package.virtuals_provided:
+                repo_cache_path = self.stage.repo.join(vspec.name)
+                if not os.path.exists(repo_cache_path):
+                    try:
+                        spack.repo.path.dump_provenance(vspec, repo_cache_path)
+                    except spack.repo.UnknownPackageError:
+                        pass  # not all virtuals have package files
+
+        with open(self.stage.join(test_suite_filename), 'w') as f:
+            sjson.dump(self.to_dict(), stream=f)
+
+    def to_dict(self):
+        specs = [s.to_dict() for s in self.specs]
+        d = {'specs': specs}
+        if self.alias:
+            d['alias'] = self.alias
+        return d
+
+    @staticmethod
+    def from_dict(d):
+        specs = [Spec.from_dict(spec_dict) for spec_dict in d['specs']]
+        alias = d.get('alias', None)
+        return TestSuite(specs, alias)
+
+    @staticmethod
+    def from_file(filename):
+        try:
+            with open(filename, 'r') as f:
+                data = sjson.load(f)
+                return TestSuite.from_dict(data)
+        except Exception as e:
+            tty.debug(e)
+            raise sjson.SpackJSONError("error parsing JSON TestSuite:", str(e))
+
+
+def _add_msg_to_file(filename, msg):
+    """Add the message to the specified file
+
+    Args:
+        filename (str): path to the file
+        msg (str): message to be appended to the file
+    """
+    with open(filename, 'a+') as f:
+        f.write('{0}\n'.format(msg))
+
+
+class TestFailure(spack.error.SpackError):
+    """Raised when package tests have failed for an installation."""
+    def __init__(self, failures):
+        # Failures are all exceptions
+        msg = "%d tests failed.\n" % len(failures)
+        for failure, message in failures:
+            msg += '\n\n%s\n' % str(failure)
+            msg += '\n%s\n' % message
+
+        super(TestFailure, self).__init__(msg)
diff --git a/lib/spack/spack/installer.py b/lib/spack/spack/installer.py
index cf7b5bfb99..dd35db0839 100644
--- a/lib/spack/spack/installer.py
+++ b/lib/spack/spack/installer.py
@@ -1610,12 +1610,12 @@ def build_process(pkg, kwargs):
 
     This function's return value is returned to the parent process.
     """
-    keep_stage = kwargs.get('keep_stage', False)
+    fake = kwargs.get('fake', False)
     install_source = kwargs.get('install_source', False)
+    keep_stage = kwargs.get('keep_stage', False)
     skip_patch = kwargs.get('skip_patch', False)
-    verbose = kwargs.get('verbose', False)
-    fake = kwargs.get('fake', False)
     unmodified_env = kwargs.get('unmodified_env', {})
+    verbose = kwargs.get('verbose', False)
 
     start_time = time.time()
     if not fake:
@@ -1958,6 +1958,7 @@ class BuildRequest(object):
     def _add_default_args(self):
         """Ensure standard install options are set to at least the default."""
         for arg, default in [('cache_only', False),
+                             ('context', 'build'),  # installs *always* build
                              ('dirty', False),
                              ('fail_fast', False),
                              ('fake', False),
diff --git a/lib/spack/spack/modules/lmod.py b/lib/spack/spack/modules/lmod.py
index 018edb35ad..80f6933063 100644
--- a/lib/spack/spack/modules/lmod.py
+++ b/lib/spack/spack/modules/lmod.py
@@ -12,6 +12,7 @@ import collections
 import spack.config
 import spack.compilers
 import spack.spec
+import spack.repo
 import spack.error
 import spack.tengine as tengine
 
@@ -125,7 +126,9 @@ class LmodConfiguration(BaseConfiguration):
 
         # Check if all the tokens in the hierarchy are virtual specs.
         # If not warn the user and raise an error.
-        not_virtual = [t for t in tokens if not spack.spec.Spec.is_virtual(t)]
+        not_virtual = [t for t in tokens
+                       if t != 'compiler' and
+                       not spack.repo.path.is_virtual(t)]
         if not_virtual:
             msg = "Non-virtual specs in 'hierarchy' list for lmod: {0}\n"
             msg += "Please check the 'modules.yaml' configuration files"
diff --git a/lib/spack/spack/package.py b/lib/spack/spack/package.py
index b8dca6e55a..de394e2d45 100644
--- a/lib/spack/spack/package.py
+++ b/lib/spack/spack/package.py
@@ -24,10 +24,12 @@ import sys
 import textwrap
 import time
 import traceback
-
 import six
+import types
 
+import llnl.util.filesystem as fsys
 import llnl.util.tty as tty
+
 import spack.compilers
 import spack.config
 import spack.dependency
@@ -45,15 +47,14 @@ import spack.store
 import spack.url
 import spack.util.environment
 import spack.util.web
-from llnl.util.filesystem import mkdirp, touch, working_dir
 from llnl.util.lang import memoized
 from llnl.util.link_tree import LinkTree
 from ordereddict_backport import OrderedDict
-from six import StringIO
-from six import string_types
-from six import with_metaclass
 from spack.filesystem_view import YamlFilesystemView
 from spack.installer import PackageInstaller, InstallError
+from spack.install_test import TestFailure, TestSuite
+from spack.util.executable import which, ProcessError
+from spack.util.prefix import Prefix
 from spack.stage import stage_prefix, Stage, ResourceStage, StageComposite
 from spack.util.package_hash import package_hash
 from spack.version import Version
@@ -452,7 +453,21 @@ class PackageViewMixin(object):
             view.remove_file(src, dst)
 
 
-class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)):
+def test_log_pathname(test_stage, spec):
+    """Build the pathname of the test log file
+
+    Args:
+        test_stage (str): path to the test stage directory
+        spec (Spec): instance of the spec under test
+
+    Returns:
+        (str): the pathname of the test log file
+    """
+    return os.path.join(test_stage,
+                        'test-{0}-out.txt'.format(TestSuite.test_pkg_id(spec)))
+
+
+class PackageBase(six.with_metaclass(PackageMeta, PackageViewMixin, object)):
     """This is the superclass for all spack packages.
 
     ***The Package class***
@@ -542,6 +557,10 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)):
     #: are executed or 'None' if there are no such test functions.
     build_time_test_callbacks = None
 
+    #: By default, packages are not virtual
+    #: Virtual packages override this attribute
+    virtual = False
+
     #: Most Spack packages are used to install source or binary code while
     #: those that do not can be used to install a set of other Spack packages.
     has_code = True
@@ -633,6 +652,18 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)):
     metadata_attrs = ['homepage', 'url', 'urls', 'list_url', 'extendable',
                       'parallel', 'make_jobs']
 
+    #: Boolean. If set to ``True``, the smoke/install test requires a compiler.
+    #: This is currently used by smoke tests to ensure a compiler is available
+    #: to build a custom test code.
+    test_requires_compiler = False
+
+    #: List of test failures encountered during a smoke/install test run.
+    test_failures = None
+
+    #: TestSuite instance used to manage smoke/install tests for one or more
+    #: specs.
+    test_suite = None
+
     def __init__(self, spec):
         # this determines how the package should be built.
         self.spec = spec
@@ -1002,19 +1033,22 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)):
             return os.path.join(self.stage.path, _spack_build_envfile)
 
     @property
+    def metadata_dir(self):
+        """Return the install metadata directory."""
+        return spack.store.layout.metadata_path(self.spec)
+
+    @property
     def install_env_path(self):
         """
         Return the build environment file path on successful installation.
         """
-        install_path = spack.store.layout.metadata_path(self.spec)
-
         # Backward compatibility: Return the name of an existing log path;
         # otherwise, return the current install env path name.
-        old_filename = os.path.join(install_path, 'build.env')
+        old_filename = os.path.join(self.metadata_dir, 'build.env')
         if os.path.exists(old_filename):
             return old_filename
         else:
-            return os.path.join(install_path, _spack_build_envfile)
+            return os.path.join(self.metadata_dir, _spack_build_envfile)
 
     @property
     def log_path(self):
@@ -1031,16 +1065,14 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)):
     @property
     def install_log_path(self):
         """Return the build log file path on successful installation."""
-        install_path = spack.store.layout.metadata_path(self.spec)
-
         # Backward compatibility: Return the name of an existing install log.
         for filename in ['build.out', 'build.txt']:
-            old_log = os.path.join(install_path, filename)
+            old_log = os.path.join(self.metadata_dir, filename)
             if os.path.exists(old_log):
                 return old_log
 
         # Otherwise, return the current install log path name.
-        return os.path.join(install_path, _spack_build_logfile)
+        return os.path.join(self.metadata_dir, _spack_build_logfile)
 
     @property
     def configure_args_path(self):
@@ -1050,9 +1082,12 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)):
     @property
     def install_configure_args_path(self):
         """Return the configure args file path on successful installation."""
-        install_path = spack.store.layout.metadata_path(self.spec)
+        return os.path.join(self.metadata_dir, _spack_configure_argsfile)
 
-        return os.path.join(install_path, _spack_configure_argsfile)
+    @property
+    def install_test_root(self):
+        """Return the install test root directory."""
+        return os.path.join(self.metadata_dir, 'test')
 
     def _make_fetcher(self):
         # Construct a composite fetcher that always contains at least
@@ -1322,7 +1357,7 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)):
                 raise FetchError("Archive was empty for %s" % self.name)
         else:
             # Support for post-install hooks requires a stage.source_path
-            mkdirp(self.stage.source_path)
+            fsys.mkdirp(self.stage.source_path)
 
     def do_patch(self):
         """Applies patches if they haven't been applied already."""
@@ -1368,7 +1403,7 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)):
         patched = False
         for patch in patches:
             try:
-                with working_dir(self.stage.source_path):
+                with fsys.working_dir(self.stage.source_path):
                     patch.apply(self.stage)
                 tty.debug('Applied patch {0}'.format(patch.path_or_url))
                 patched = True
@@ -1377,12 +1412,12 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)):
 
                 # Touch bad file if anything goes wrong.
                 tty.msg('Patch %s failed.' % patch.path_or_url)
-                touch(bad_file)
+                fsys.touch(bad_file)
                 raise
 
         if has_patch_fun:
             try:
-                with working_dir(self.stage.source_path):
+                with fsys.working_dir(self.stage.source_path):
                     self.patch()
                 tty.debug('Ran patch() for {0}'.format(self.name))
                 patched = True
@@ -1400,7 +1435,7 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)):
 
                 # Touch bad file if anything goes wrong.
                 tty.msg('patch() function failed for {0}'.format(self.name))
-                touch(bad_file)
+                fsys.touch(bad_file)
                 raise
 
         # Get rid of any old failed file -- patches have either succeeded
@@ -1411,9 +1446,9 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)):
 
         # touch good or no patches file so that we skip next time.
         if patched:
-            touch(good_file)
+            fsys.touch(good_file)
         else:
-            touch(no_patches_file)
+            fsys.touch(no_patches_file)
 
     @classmethod
     def all_patches(cls):
@@ -1657,6 +1692,175 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)):
         builder = PackageInstaller([(self, kwargs)])
         builder.install()
 
+    def cache_extra_test_sources(self, srcs):
+        """Copy relative source paths to the corresponding install test subdir
+
+        This method is intended as an optional install test setup helper for
+        grabbing source files/directories during the installation process and
+        copying them to the installation test subdirectory for subsequent use
+        during install testing.
+
+        Args:
+            srcs (str or list of str): relative path for files and or
+                subdirectories located in the staged source path that are to
+                be copied to the corresponding location(s) under the install
+                testing directory.
+        """
+        paths = [srcs] if isinstance(srcs, six.string_types) else srcs
+
+        for path in paths:
+            src_path = os.path.join(self.stage.source_path, path)
+            dest_path = os.path.join(self.install_test_root, path)
+            if os.path.isdir(src_path):
+                fsys.install_tree(src_path, dest_path)
+            else:
+                fsys.mkdirp(os.path.dirname(dest_path))
+                fsys.copy(src_path, dest_path)
+
+    def do_test(self, dirty=False):
+        if self.test_requires_compiler:
+            compilers = spack.compilers.compilers_for_spec(
+                self.spec.compiler, arch_spec=self.spec.architecture)
+            if not compilers:
+                tty.error('Skipping tests for package %s\n' %
+                          self.spec.format('{name}-{version}-{hash:7}') +
+                          'Package test requires missing compiler %s' %
+                          self.spec.compiler)
+                return
+
+        # Clear test failures
+        self.test_failures = []
+        self.test_log_file = self.test_suite.log_file_for_spec(self.spec)
+        fsys.touch(self.test_log_file)  # Otherwise log_parse complains
+
+        kwargs = {'dirty': dirty, 'fake': False, 'context': 'test'}
+        spack.build_environment.start_build_process(self, test_process, kwargs)
+
+    def test(self):
+        pass
+
+    def run_test(self, exe, options=[], expected=[], status=0,
+                 installed=False, purpose='', skip_missing=False,
+                 work_dir=None):
+        """Run the test and confirm the expected results are obtained
+
+        Log any failures and continue, they will be re-raised later
+
+        Args:
+            exe (str): the name of the executable
+            options (str or list of str): list of options to pass to the runner
+            expected (str or list of str): list of expected output strings.
+                Each string is a regex expected to match part of the output.
+            status (int or list of int): possible passing status values
+                with 0 meaning the test is expected to succeed
+            installed (bool): if ``True``, the executable must be in the
+                install prefix
+            purpose (str): message to display before running test
+            skip_missing (bool): skip the test if the executable is not
+                in the install prefix bin directory or the provided work_dir
+            work_dir (str or None): path to the smoke test directory
+        """
+        wdir = '.' if work_dir is None else work_dir
+        with fsys.working_dir(wdir):
+            try:
+                runner = which(exe)
+                if runner is None and skip_missing:
+                    return
+                assert runner is not None, \
+                    "Failed to find executable '{0}'".format(exe)
+
+                self._run_test_helper(
+                    runner, options, expected, status, installed, purpose)
+                print("PASSED")
+                return True
+            except BaseException as e:
+                # print a summary of the error to the log file
+                # so that cdash and junit reporters know about it
+                exc_type, _, tb = sys.exc_info()
+                print('FAILED: {0}'.format(e))
+                import traceback
+                # remove the current call frame to exclude the extract_stack
+                # call from the error
+                stack = traceback.extract_stack()[:-1]
+
+                # Package files have a line added at import time, so we re-read
+                # the file to make line numbers match. We have to subtract two
+                # from the line number because the original line number is
+                # inflated once by the import statement and the lines are
+                # displaced one by the import statement.
+                for i, entry in enumerate(stack):
+                    filename, lineno, function, text = entry
+                    if spack.repo.is_package_file(filename):
+                        with open(filename, 'r') as f:
+                            lines = f.readlines()
+                        new_lineno = lineno - 2
+                        text = lines[new_lineno]
+                        stack[i] = (filename, new_lineno, function, text)
+
+                # Format the stack to print and print it
+                out = traceback.format_list(stack)
+                for line in out:
+                    print(line.rstrip('\n'))
+
+                if exc_type is spack.util.executable.ProcessError:
+                    out = six.StringIO()
+                    spack.build_environment.write_log_summary(
+                        out, 'test', self.test_log_file, last=1)
+                    m = out.getvalue()
+                else:
+                    # We're below the package context, so get context from
+                    # stack instead of from traceback.
+                    # The traceback is truncated here, so we can't use it to
+                    # traverse the stack.
+                    m = '\n'.join(
+                        spack.build_environment.get_package_context(tb)
+                    )
+
+                exc = e  # e is deleted after this block
+
+                # If we fail fast, raise another error
+                if spack.config.get('config:fail_fast', False):
+                    raise TestFailure([(exc, m)])
+                else:
+                    self.test_failures.append((exc, m))
+                return False
+
+    def _run_test_helper(self, runner, options, expected, status, installed,
+                         purpose):
+        status = [status] if isinstance(status, six.integer_types) else status
+        expected = [expected] if isinstance(expected, six.string_types) else \
+            expected
+        options = [options] if isinstance(options, six.string_types) else \
+            options
+
+        if purpose:
+            tty.msg(purpose)
+        else:
+            tty.debug('test: {0}: expect command status in {1}'
+                      .format(runner.name, status))
+
+        if installed:
+            msg = "Executable '{0}' expected in prefix".format(runner.name)
+            msg += ", found in {0} instead".format(runner.path)
+            assert runner.path.startswith(self.spec.prefix), msg
+
+        try:
+            output = runner(*options, output=str.split, error=str.split)
+
+            assert 0 in status, \
+                'Expected {0} execution to fail'.format(runner.name)
+        except ProcessError as err:
+            output = str(err)
+            match = re.search(r'exited with status ([0-9]+)', output)
+            if not (match and int(match.group(1)) in status):
+                raise
+
+        for check in expected:
+            cmd = ' '.join([runner.name] + options)
+            msg = "Expected '{0}' to match output of `{1}`".format(check, cmd)
+            msg += '\n\nOutput: {0}'.format(output)
+            assert re.search(check, output), msg
+
     def unit_test_check(self):
         """Hook for unit tests to assert things about package internals.
 
@@ -1678,7 +1882,7 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)):
         """This function checks whether install succeeded."""
 
         def check_paths(path_list, filetype, predicate):
-            if isinstance(path_list, string_types):
+            if isinstance(path_list, six.string_types):
                 path_list = [path_list]
 
             for path in path_list:
@@ -2031,7 +2235,7 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)):
         # copy spec metadata to "deprecated" dir of deprecator
         depr_yaml = spack.store.layout.deprecated_file_path(spec,
                                                             deprecator)
-        fs.mkdirp(os.path.dirname(depr_yaml))
+        fsys.mkdirp(os.path.dirname(depr_yaml))
         shutil.copy2(self_yaml, depr_yaml)
 
         # Any specs deprecated in favor of this spec are re-deprecated in
@@ -2210,7 +2414,7 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)):
 
         doc = re.sub(r'\s+', ' ', self.__doc__)
         lines = textwrap.wrap(doc, 72)
-        results = StringIO()
+        results = six.StringIO()
         for line in lines:
             results.write((" " * indent) + line + "\n")
         return results.getvalue()
@@ -2313,6 +2517,71 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)):
                 tty.warn(msg.format(name))
 
 
+def test_process(pkg, kwargs):
+    with tty.log.log_output(pkg.test_log_file) as logger:
+        with logger.force_echo():
+            tty.msg('Testing package {0}'
+                    .format(pkg.test_suite.test_pkg_id(pkg.spec)))
+
+        # use debug print levels for log file to record commands
+        old_debug = tty.is_debug()
+        tty.set_debug(True)
+
+        # run test methods from the package and all virtuals it
+        # provides virtuals have to be deduped by name
+        v_names = list(set([vspec.name
+                            for vspec in pkg.virtuals_provided]))
+
+        # hack for compilers that are not dependencies (yet)
+        # TODO: this all eventually goes away
+        c_names = ('gcc', 'intel', 'intel-parallel-studio', 'pgi')
+        if pkg.name in c_names:
+            v_names.extend(['c', 'cxx', 'fortran'])
+        if pkg.spec.satisfies('llvm+clang'):
+            v_names.extend(['c', 'cxx'])
+
+        test_specs = [pkg.spec] + [spack.spec.Spec(v_name)
+                                   for v_name in sorted(v_names)]
+
+        try:
+            with fsys.working_dir(
+                    pkg.test_suite.test_dir_for_spec(pkg.spec)):
+                for spec in test_specs:
+                    pkg.test_suite.current_test_spec = spec
+                    # Fail gracefully if a virtual has no package/tests
+                    try:
+                        spec_pkg = spec.package
+                    except spack.repo.UnknownPackageError:
+                        continue
+
+                    # copy test data into test data dir
+                    data_source = Prefix(spec_pkg.package_dir).test
+                    data_dir = pkg.test_suite.current_test_data_dir
+                    if (os.path.isdir(data_source) and
+                            not os.path.exists(data_dir)):
+                        # We assume data dir is used read-only
+                        # maybe enforce this later
+                        shutil.copytree(data_source, data_dir)
+
+                    # grab the function for each method so we can call
+                    # it with the package
+                    test_fn = spec_pkg.__class__.test
+                    if not isinstance(test_fn, types.FunctionType):
+                        test_fn = test_fn.__func__
+
+                    # Run the tests
+                    test_fn(pkg)
+
+            # If fail-fast was on, we error out above
+            # If we collect errors, raise them in batch here
+            if pkg.test_failures:
+                raise TestFailure(pkg.test_failures)
+
+        finally:
+            # reset debug level
+            tty.set_debug(old_debug)
+
+
 inject_flags = PackageBase.inject_flags
 env_flags = PackageBase.env_flags
 build_system_flags = PackageBase.build_system_flags
diff --git a/lib/spack/spack/paths.py b/lib/spack/spack/paths.py
index cb2240359c..9c803cba7e 100644
--- a/lib/spack/spack/paths.py
+++ b/lib/spack/spack/paths.py
@@ -12,7 +12,6 @@ dependencies.
 import os
 from llnl.util.filesystem import ancestor
 
-
 #: This file lives in $prefix/lib/spack/spack/__file__
 prefix = ancestor(__file__, 4)
 
@@ -42,6 +41,7 @@ test_path             = os.path.join(module_path, "test")
 hooks_path            = os.path.join(module_path, "hooks")
 var_path              = os.path.join(prefix, "var", "spack")
 repos_path            = os.path.join(var_path, "repos")
+tests_path            = os.path.join(var_path, "tests")
 share_path            = os.path.join(prefix, "share", "spack")
 
 # Paths to built-in Spack repositories.
diff --git a/lib/spack/spack/pkgkit.py b/lib/spack/spack/pkgkit.py
index ac0a7eee0d..4f25d41dfb 100644
--- a/lib/spack/spack/pkgkit.py
+++ b/lib/spack/spack/pkgkit.py
@@ -59,6 +59,7 @@ from spack.package import \
 
 from spack.installer import \
     ExternalPackageError, InstallError, InstallLockError, UpstreamPackageError
+from spack.install_test import get_escaped_text_output
 
 from spack.variant import any_combination_of, auto_or_any_combination_of
 from spack.variant import disjoint_sets
diff --git a/lib/spack/spack/repo.py b/lib/spack/spack/repo.py
index 5e1457c165..4af7b382f0 100644
--- a/lib/spack/spack/repo.py
+++ b/lib/spack/spack/repo.py
@@ -91,6 +91,19 @@ def autospec(function):
     return converter
 
 
+def is_package_file(filename):
+    """Determine whether we are in a package file from a repo."""
+    # Package files are named `package.py` and are not in lib/spack/spack
+    # We have to remove the file extension because it can be .py and can be
+    # .pyc depending on context, and can differ between the files
+    import spack.package  # break cycle
+    filename_noext = os.path.splitext(filename)[0]
+    packagebase_filename_noext = os.path.splitext(
+        inspect.getfile(spack.package.PackageBase))[0]
+    return (filename_noext != packagebase_filename_noext and
+            os.path.basename(filename_noext) == 'package')
+
+
 class SpackNamespace(types.ModuleType):
     """ Allow lazy loading of modules."""
 
@@ -131,6 +144,11 @@ class FastPackageChecker(Mapping):
         #: Reference to the appropriate entry in the global cache
         self._packages_to_stats = self._paths_cache[packages_path]
 
+    def invalidate(self):
+        """Regenerate cache for this checker."""
+        self._paths_cache[self.packages_path] = self._create_new_cache()
+        self._packages_to_stats = self._paths_cache[self.packages_path]
+
     def _create_new_cache(self):
         """Create a new cache for packages in a repo.
 
@@ -308,6 +326,9 @@ class ProviderIndexer(Indexer):
         self.index = spack.provider_index.ProviderIndex.from_json(stream)
 
     def update(self, pkg_fullname):
+        name = pkg_fullname.split('.')[-1]
+        if spack.repo.path.is_virtual(name, use_index=False):
+            return
         self.index.remove_provider(pkg_fullname)
         self.index.update(pkg_fullname)
 
@@ -517,12 +538,12 @@ class RepoPath(object):
         """Get the first repo in precedence order."""
         return self.repos[0] if self.repos else None
 
-    def all_package_names(self):
+    def all_package_names(self, include_virtuals=False):
         """Return all unique package names in all repositories."""
         if self._all_package_names is None:
             all_pkgs = set()
             for repo in self.repos:
-                for name in repo.all_package_names():
+                for name in repo.all_package_names(include_virtuals):
                     all_pkgs.add(name)
             self._all_package_names = sorted(all_pkgs, key=lambda n: n.lower())
         return self._all_package_names
@@ -679,12 +700,20 @@ class RepoPath(object):
         """
         return any(repo.exists(pkg_name) for repo in self.repos)
 
-    def is_virtual(self, pkg_name):
-        """True if the package with this name is virtual, False otherwise."""
-        if not isinstance(pkg_name, str):
+    def is_virtual(self, pkg_name, use_index=True):
+        """True if the package with this name is virtual, False otherwise.
+
+        Set `use_index` False when calling from a code block that could
+        be run during the computation of the provider index."""
+        have_name = pkg_name is not None
+        if have_name and not isinstance(pkg_name, str):
             raise ValueError(
                 "is_virtual(): expected package name, got %s" % type(pkg_name))
-        return pkg_name in self.provider_index
+        if use_index:
+            return have_name and pkg_name in self.provider_index
+        else:
+            return have_name and (not self.exists(pkg_name) or
+                                  self.get_pkg_class(pkg_name).virtual)
 
     def __contains__(self, pkg_name):
         return self.exists(pkg_name)
@@ -913,10 +942,6 @@ class Repo(object):
            This dumps the package file and any associated patch files.
            Raises UnknownPackageError if not found.
         """
-        # Some preliminary checks.
-        if spec.virtual:
-            raise UnknownPackageError(spec.name)
-
         if spec.namespace and spec.namespace != self.namespace:
             raise UnknownPackageError(
                 "Repository %s does not contain package %s."
@@ -999,9 +1024,12 @@ class Repo(object):
             self._fast_package_checker = FastPackageChecker(self.packages_path)
         return self._fast_package_checker
 
-    def all_package_names(self):
+    def all_package_names(self, include_virtuals=False):
         """Returns a sorted list of all package names in the Repo."""
-        return sorted(self._pkg_checker.keys())
+        names = sorted(self._pkg_checker.keys())
+        if include_virtuals:
+            return names
+        return [x for x in names if not self.is_virtual(x)]
 
     def packages_with_tags(self, *tags):
         v = set(self.all_package_names())
@@ -1040,7 +1068,7 @@ class Repo(object):
 
     def is_virtual(self, pkg_name):
         """True if the package with this name is virtual, False otherwise."""
-        return self.provider_index.contains(pkg_name)
+        return pkg_name in self.provider_index
 
     def _get_pkg_module(self, pkg_name):
         """Create a module for a particular package.
@@ -1074,7 +1102,8 @@ class Repo(object):
                 # manually construct the error message in order to give the
                 # user the correct package.py where the syntax error is located
                 raise SyntaxError('invalid syntax in {0:}, line {1:}'
-                                  ''.format(file_path, e.lineno))
+                                  .format(file_path, e.lineno))
+
             module.__package__ = self.full_namespace
             module.__loader__ = self
             self._modules[pkg_name] = module
@@ -1205,9 +1234,9 @@ def get(spec):
     return path.get(spec)
 
 
-def all_package_names():
+def all_package_names(include_virtuals=False):
     """Convenience wrapper around ``spack.repo.all_package_names()``."""
-    return path.all_package_names()
+    return path.all_package_names(include_virtuals)
 
 
 def set_path(repo):
diff --git a/lib/spack/spack/report.py b/lib/spack/spack/report.py
index 4e5bc9c993..ebae2d0adc 100644
--- a/lib/spack/spack/report.py
+++ b/lib/spack/spack/report.py
@@ -9,11 +9,13 @@ import collections
 import functools
 import time
 import traceback
+import os
 
 import llnl.util.lang
 import spack.build_environment
 import spack.fetch_strategy
 import spack.package
+from spack.install_test import TestSuite
 from spack.reporter import Reporter
 from spack.reporters.cdash import CDash
 from spack.reporters.junit import JUnit
@@ -33,12 +35,16 @@ __all__ = [
 ]
 
 
-def fetch_package_log(pkg):
+def fetch_log(pkg, do_fn, dir):
+    log_files = {
+        '_install_task': pkg.build_log_path,
+        'do_test': os.path.join(dir, TestSuite.test_log_name(pkg.spec)),
+    }
     try:
-        with codecs.open(pkg.build_log_path, 'r', 'utf-8') as f:
+        with codecs.open(log_files[do_fn.__name__], 'r', 'utf-8') as f:
             return ''.join(f.readlines())
     except Exception:
-        return 'Cannot open build log for {0}'.format(
+        return 'Cannot open log for {0}'.format(
             pkg.spec.cshort_spec
         )
 
@@ -58,15 +64,20 @@ class InfoCollector(object):
         specs (list of Spec): specs whose install information will
            be recorded
     """
-    #: Backup of PackageInstaller._install_task
-    _backup__install_task = spack.package.PackageInstaller._install_task
-
-    def __init__(self, specs):
-        #: Specs that will be installed
+    def __init__(self, wrap_class, do_fn, specs, dir):
+        #: Class for which to wrap a function
+        self.wrap_class = wrap_class
+        #: Action to be reported on
+        self.do_fn = do_fn
+        #: Backup of PackageBase function
+        self._backup_do_fn = getattr(self.wrap_class, do_fn)
+        #: Specs that will be acted on
         self.input_specs = specs
         #: This is where we record the data that will be included
         #: in our report.
         self.specs = []
+        #: Record directory for test log paths
+        self.dir = dir
 
     def __enter__(self):
         # Initialize the spec report with the data that is available upfront.
@@ -98,30 +109,37 @@ class InfoCollector(object):
                 Property('compiler', input_spec.compiler))
 
             # Check which specs are already installed and mark them as skipped
-            for dep in filter(lambda x: x.package.installed,
-                              input_spec.traverse()):
-                package = {
-                    'name': dep.name,
-                    'id': dep.dag_hash(),
-                    'elapsed_time': '0.0',
-                    'result': 'skipped',
-                    'message': 'Spec already installed'
-                }
-                spec['packages'].append(package)
-
-        def gather_info(_install_task):
-            """Decorates PackageInstaller._install_task to gather useful
-            information on PackageBase.do_install for a CI report.
+            # only for install_task
+            if self.do_fn == '_install_task':
+                for dep in filter(lambda x: x.package.installed,
+                                  input_spec.traverse()):
+                    package = {
+                        'name': dep.name,
+                        'id': dep.dag_hash(),
+                        'elapsed_time': '0.0',
+                        'result': 'skipped',
+                        'message': 'Spec already installed'
+                    }
+                    spec['packages'].append(package)
+
+        def gather_info(do_fn):
+            """Decorates do_fn to gather useful information for
+            a CI report.
 
             It's defined here to capture the environment and build
             this context as the installations proceed.
             """
-            @functools.wraps(_install_task)
-            def wrapper(installer, task, *args, **kwargs):
-                pkg = task.pkg
+            @functools.wraps(do_fn)
+            def wrapper(instance, *args, **kwargs):
+                if isinstance(instance, spack.package.PackageBase):
+                    pkg = instance
+                elif hasattr(args[0], 'pkg'):
+                    pkg = args[0].pkg
+                else:
+                    raise Exception
 
                 # We accounted before for what is already installed
-                installed_on_entry = pkg.installed
+                installed_already = pkg.installed
 
                 package = {
                     'name': pkg.name,
@@ -135,13 +153,12 @@ class InfoCollector(object):
                 start_time = time.time()
                 value = None
                 try:
-
-                    value = _install_task(installer, task, *args, **kwargs)
+                    value = do_fn(instance, *args, **kwargs)
                     package['result'] = 'success'
-                    package['stdout'] = fetch_package_log(pkg)
+                    package['stdout'] = fetch_log(pkg, do_fn, self.dir)
                     package['installed_from_binary_cache'] = \
                         pkg.installed_from_binary_cache
-                    if installed_on_entry:
+                    if do_fn.__name__ == '_install_task' and installed_already:
                         return
 
                 except spack.build_environment.InstallError as e:
@@ -149,7 +166,7 @@ class InfoCollector(object):
                     # didn't work correctly)
                     package['result'] = 'failure'
                     package['message'] = e.message or 'Installation failure'
-                    package['stdout'] = fetch_package_log(pkg)
+                    package['stdout'] = fetch_log(pkg, do_fn, self.dir)
                     package['stdout'] += package['message']
                     package['exception'] = e.traceback
 
@@ -157,7 +174,7 @@ class InfoCollector(object):
                     # Everything else is an error (the installation
                     # failed outside of the child process)
                     package['result'] = 'error'
-                    package['stdout'] = fetch_package_log(pkg)
+                    package['stdout'] = fetch_log(pkg, do_fn, self.dir)
                     package['message'] = str(e) or 'Unknown error'
                     package['exception'] = traceback.format_exc()
 
@@ -184,15 +201,14 @@ class InfoCollector(object):
 
             return wrapper
 
-        spack.package.PackageInstaller._install_task = gather_info(
-            spack.package.PackageInstaller._install_task
-        )
+        setattr(self.wrap_class, self.do_fn, gather_info(
+            getattr(self.wrap_class, self.do_fn)
+        ))
 
     def __exit__(self, exc_type, exc_val, exc_tb):
 
-        # Restore the original method in PackageInstaller
-        spack.package.PackageInstaller._install_task = \
-            InfoCollector._backup__install_task
+        # Restore the original method in PackageBase
+        setattr(self.wrap_class, self.do_fn, self._backup_do_fn)
 
         for spec in self.specs:
             spec['npackages'] = len(spec['packages'])
@@ -225,22 +241,26 @@ class collect_info(object):
 
             # The file 'junit.xml' is written when exiting
             # the context
-            specs = [Spec('hdf5').concretized()]
-            with collect_info(specs, 'junit', 'junit.xml'):
+            s = [Spec('hdf5').concretized()]
+            with collect_info(PackageBase, do_install, s, 'junit', 'a.xml'):
                 # A report will be generated for these specs...
-                for spec in specs:
-                    spec.do_install()
+                for spec in s:
+                    getattr(class, function)(spec)
                 # ...but not for this one
                 Spec('zlib').concretized().do_install()
 
     Args:
+        class: class on which to wrap a function
+        function: function to wrap
         format_name (str or None): one of the supported formats
-        args (dict): args passed to spack install
+        args (dict): args passed to function
 
     Raises:
         ValueError: when ``format_name`` is not in ``valid_formats``
     """
-    def __init__(self, format_name, args):
+    def __init__(self, cls, function, format_name, args):
+        self.cls = cls
+        self.function = function
         self.filename = None
         if args.cdash_upload_url:
             self.format_name = 'cdash'
@@ -253,13 +273,19 @@ class collect_info(object):
                              .format(self.format_name))
         self.report_writer = report_writers[self.format_name](args)
 
+    def __call__(self, type, dir=os.getcwd()):
+        self.type = type
+        self.dir = dir
+        return self
+
     def concretization_report(self, msg):
         self.report_writer.concretization_report(self.filename, msg)
 
     def __enter__(self):
         if self.format_name:
-            # Start the collector and patch PackageInstaller._install_task
-            self.collector = InfoCollector(self.specs)
+            # Start the collector and patch self.function on appropriate class
+            self.collector = InfoCollector(
+                self.cls, self.function, self.specs, self.dir)
             self.collector.__enter__()
 
     def __exit__(self, exc_type, exc_val, exc_tb):
@@ -269,4 +295,5 @@ class collect_info(object):
             self.collector.__exit__(exc_type, exc_val, exc_tb)
 
             report_data = {'specs': self.collector.specs}
-            self.report_writer.build_report(self.filename, report_data)
+            report_fn = getattr(self.report_writer, '%s_report' % self.type)
+            report_fn(self.filename, report_data)
diff --git a/lib/spack/spack/reporter.py b/lib/spack/spack/reporter.py
index 25d4042a5e..6314054139 100644
--- a/lib/spack/spack/reporter.py
+++ b/lib/spack/spack/reporter.py
@@ -16,5 +16,8 @@ class Reporter(object):
     def build_report(self, filename, report_data):
         pass
 
+    def test_report(self, filename, report_data):
+        pass
+
     def concretization_report(self, filename, msg):
         pass
diff --git a/lib/spack/spack/reporters/cdash.py b/lib/spack/spack/reporters/cdash.py
index 580df7866f..c1a220963e 100644
--- a/lib/spack/spack/reporters/cdash.py
+++ b/lib/spack/spack/reporters/cdash.py
@@ -72,8 +72,10 @@ class CDash(Reporter):
             tty.verbose("Using CDash auth token from environment")
             self.authtoken = os.environ.get('SPACK_CDASH_AUTH_TOKEN')
 
-        if args.spec:
+        if getattr(args, 'spec', ''):
             packages = args.spec
+        elif getattr(args, 'specs', ''):
+            packages = args.specs
         else:
             packages = []
             for file in args.specfiles:
@@ -98,7 +100,7 @@ class CDash(Reporter):
             self.revision = git('rev-parse', 'HEAD', output=str).strip()
         self.multiple_packages = False
 
-    def report_for_package(self, directory_name, package, duration):
+    def build_report_for_package(self, directory_name, package, duration):
         if 'stdout' not in package:
             # Skip reporting on packages that did not generate any output.
             return
@@ -158,8 +160,8 @@ class CDash(Reporter):
                 '\n'.join(report_data[phase]['loglines'])
             errors, warnings = parse_log_events(report_data[phase]['loglines'])
             # Cap the number of errors and warnings at 50 each.
-            errors = errors[0:49]
-            warnings = warnings[0:49]
+            errors = errors[:50]
+            warnings = warnings[:50]
             nerrors = len(errors)
 
             if phase == 'configure' and nerrors > 0:
@@ -250,7 +252,114 @@ class CDash(Reporter):
             if 'time' in spec:
                 duration = int(spec['time'])
             for package in spec['packages']:
-                self.report_for_package(directory_name, package, duration)
+                self.build_report_for_package(
+                    directory_name, package, duration)
+        self.print_cdash_link()
+
+    def test_report_for_package(self, directory_name, package, duration):
+        if 'stdout' not in package:
+            # Skip reporting on packages that did not generate any output.
+            return
+
+        self.current_package_name = package['name']
+        self.buildname = "{0} - {1}".format(
+            self.base_buildname, package['name'])
+
+        report_data = self.initialize_report(directory_name)
+
+        for phase in ('test', 'update'):
+            report_data[phase] = {}
+            report_data[phase]['loglines'] = []
+            report_data[phase]['status'] = 0
+            report_data[phase]['endtime'] = self.endtime
+
+        # Track the phases we perform so we know what reports to create.
+        # We always report the update step because this is how we tell CDash
+        # what revision of Spack we are using.
+        phases_encountered = ['test', 'update']
+
+        # Generate a report for this package.
+        # The first line just says "Testing package name-hash"
+        report_data['test']['loglines'].append(
+            text_type("{0} output for {1}:".format(
+                'test', package['name'])))
+        for line in package['stdout'].splitlines()[1:]:
+            report_data['test']['loglines'].append(
+                xml.sax.saxutils.escape(line))
+
+        self.starttime = self.endtime - duration
+        for phase in phases_encountered:
+            report_data[phase]['starttime'] = self.starttime
+            report_data[phase]['log'] = \
+                '\n'.join(report_data[phase]['loglines'])
+            errors, warnings = parse_log_events(report_data[phase]['loglines'])
+            # Cap the number of errors and warnings at 50 each.
+            errors = errors[0:49]
+            warnings = warnings[0:49]
+
+            if phase == 'test':
+                # Convert log output from ASCII to Unicode and escape for XML.
+                def clean_log_event(event):
+                    event = vars(event)
+                    event['text'] = xml.sax.saxutils.escape(event['text'])
+                    event['pre_context'] = xml.sax.saxutils.escape(
+                        '\n'.join(event['pre_context']))
+                    event['post_context'] = xml.sax.saxutils.escape(
+                        '\n'.join(event['post_context']))
+                    # source_file and source_line_no are either strings or
+                    # the tuple (None,).  Distinguish between these two cases.
+                    if event['source_file'][0] is None:
+                        event['source_file'] = ''
+                        event['source_line_no'] = ''
+                    else:
+                        event['source_file'] = xml.sax.saxutils.escape(
+                            event['source_file'])
+                    return event
+
+                # Convert errors to warnings if the package reported success.
+                if package['result'] == 'success':
+                    warnings = errors + warnings
+                    errors = []
+
+                report_data[phase]['errors'] = []
+                report_data[phase]['warnings'] = []
+                for error in errors:
+                    report_data[phase]['errors'].append(clean_log_event(error))
+                for warning in warnings:
+                    report_data[phase]['warnings'].append(
+                        clean_log_event(warning))
+
+            if phase == 'update':
+                report_data[phase]['revision'] = self.revision
+
+            # Write the report.
+            report_name = phase.capitalize() + ".xml"
+            report_file_name = package['name'] + "_" + report_name
+            phase_report = os.path.join(directory_name, report_file_name)
+
+            with codecs.open(phase_report, 'w', 'utf-8') as f:
+                env = spack.tengine.make_environment()
+                if phase != 'update':
+                    # Update.xml stores site information differently
+                    # than the rest of the CTest XML files.
+                    site_template = os.path.join(self.template_dir, 'Site.xml')
+                    t = env.get_template(site_template)
+                    f.write(t.render(report_data))
+
+                phase_template = os.path.join(self.template_dir, report_name)
+                t = env.get_template(phase_template)
+                f.write(t.render(report_data))
+            self.upload(phase_report)
+
+    def test_report(self, directory_name, input_data):
+        # Generate reports for each package in each spec.
+        for spec in input_data['specs']:
+            duration = 0
+            if 'time' in spec:
+                duration = int(spec['time'])
+            for package in spec['packages']:
+                self.test_report_for_package(
+                    directory_name, package, duration)
         self.print_cdash_link()
 
     def concretization_report(self, directory_name, msg):
diff --git a/lib/spack/spack/reporters/junit.py b/lib/spack/spack/reporters/junit.py
index 6c54c45b42..598b308934 100644
--- a/lib/spack/spack/reporters/junit.py
+++ b/lib/spack/spack/reporters/junit.py
@@ -27,3 +27,6 @@ class JUnit(Reporter):
             env = spack.tengine.make_environment()
             t = env.get_template(self.template_file)
             f.write(t.render(report_data))
+
+    def test_report(self, filename, report_data):
+        self.build_report(filename, report_data)
diff --git a/lib/spack/spack/schema/config.py b/lib/spack/spack/schema/config.py
index b1d3332a3f..0f83eb86f4 100644
--- a/lib/spack/spack/schema/config.py
+++ b/lib/spack/spack/schema/config.py
@@ -45,6 +45,7 @@ properties = {
                     {'type': 'array',
                      'items': {'type': 'string'}}],
             },
+            'test_stage': {'type': 'string'},
             'extensions': {
                 'type': 'array',
                 'items': {'type': 'string'}
diff --git a/lib/spack/spack/spec.py b/lib/spack/spack/spec.py
index 8a4fc12861..743f84c3c8 100644
--- a/lib/spack/spack/spec.py
+++ b/lib/spack/spack/spec.py
@@ -951,7 +951,7 @@ class SpecBuildInterface(lang.ObjectWrapper):
     def __init__(self, spec, name, query_parameters):
         super(SpecBuildInterface, self).__init__(spec)
 
-        is_virtual = Spec.is_virtual(name)
+        is_virtual = spack.repo.path.is_virtual(name)
         self.last_query = QueryState(
             name=name,
             extra_parameters=query_parameters,
@@ -1227,12 +1227,9 @@ class Spec(object):
            Possible idea: just use conventin and make virtual deps all
            caps, e.g., MPI vs mpi.
         """
-        return Spec.is_virtual(self.name)
-
-    @staticmethod
-    def is_virtual(name):
-        """Test if a name is virtual without requiring a Spec."""
-        return (name is not None) and (not spack.repo.path.exists(name))
+        # This method can be called while regenerating the provider index
+        # So we turn off using the index to detect virtuals
+        return spack.repo.path.is_virtual(self.name, use_index=False)
 
     @property
     def concrete(self):
diff --git a/lib/spack/spack/tengine.py b/lib/spack/spack/tengine.py
index 8581c8b41e..15268e682d 100644
--- a/lib/spack/spack/tengine.py
+++ b/lib/spack/spack/tengine.py
@@ -68,7 +68,8 @@ def make_environment(dirs=None):
     """Returns an configured environment for template rendering."""
     if dirs is None:
         # Default directories where to search for templates
-        builtins = spack.config.get('config:template_dirs')
+        builtins = spack.config.get('config:template_dirs',
+                                    ['$spack/share/spack/templates'])
         extensions = spack.extensions.get_template_dirs()
         dirs = [canonicalize_path(d)
                 for d in itertools.chain(builtins, extensions)]
diff --git a/lib/spack/spack/test/cmd/clean.py b/lib/spack/spack/test/cmd/clean.py
index 4df708cf50..dcaf0c916c 100644
--- a/lib/spack/spack/test/cmd/clean.py
+++ b/lib/spack/spack/test/cmd/clean.py
@@ -15,44 +15,51 @@ clean = spack.main.SpackCommand('clean')
 @pytest.fixture()
 def mock_calls_for_clean(monkeypatch):
 
+    counts = {}
+
     class Counter(object):
-        def __init__(self):
-            self.call_count = 0
+        def __init__(self, name):
+            self.name = name
+            counts[name] = 0
 
         def __call__(self, *args, **kwargs):
-            self.call_count += 1
+            counts[self.name] += 1
 
-    monkeypatch.setattr(spack.package.PackageBase, 'do_clean', Counter())
-    monkeypatch.setattr(spack.stage, 'purge', Counter())
+    monkeypatch.setattr(spack.package.PackageBase, 'do_clean',
+                        Counter('package'))
+    monkeypatch.setattr(spack.stage, 'purge', Counter('stages'))
     monkeypatch.setattr(
-        spack.caches.fetch_cache, 'destroy', Counter(), raising=False)
+        spack.caches.fetch_cache, 'destroy', Counter('downloads'),
+        raising=False)
     monkeypatch.setattr(
-        spack.caches.misc_cache, 'destroy', Counter())
+        spack.caches.misc_cache, 'destroy', Counter('caches'))
     monkeypatch.setattr(
-        spack.installer, 'clear_failures', Counter())
+        spack.installer, 'clear_failures', Counter('failures'))
+
+    yield counts
+
+
+all_effects = ['stages', 'downloads', 'caches', 'failures']
 
 
 @pytest.mark.usefixtures(
-    'mock_packages', 'config', 'mock_calls_for_clean'
+    'mock_packages', 'config'
 )
-@pytest.mark.parametrize('command_line,counters', [
-    ('mpileaks', [1, 0, 0, 0, 0]),
-    ('-s',       [0, 1, 0, 0, 0]),
-    ('-sd',      [0, 1, 1, 0, 0]),
-    ('-m',       [0, 0, 0, 1, 0]),
-    ('-f',       [0, 0, 0, 0, 1]),
-    ('-a',       [0, 1, 1, 1, 1]),
-    ('',         [0, 0, 0, 0, 0]),
+@pytest.mark.parametrize('command_line,effects', [
+    ('mpileaks', ['package']),
+    ('-s',       ['stages']),
+    ('-sd',      ['stages', 'downloads']),
+    ('-m',       ['caches']),
+    ('-f',       ['failures']),
+    ('-a',       all_effects),
+    ('',         []),
 ])
-def test_function_calls(command_line, counters):
+def test_function_calls(command_line, effects, mock_calls_for_clean):
 
     # Call the command with the supplied command line
     clean(command_line)
 
     # Assert that we called the expected functions the correct
     # number of times
-    assert spack.package.PackageBase.do_clean.call_count == counters[0]
-    assert spack.stage.purge.call_count == counters[1]
-    assert spack.caches.fetch_cache.destroy.call_count == counters[2]
-    assert spack.caches.misc_cache.destroy.call_count == counters[3]
-    assert spack.installer.clear_failures.call_count == counters[4]
+    for name in ['package'] + all_effects:
+        assert mock_calls_for_clean[name] == (1 if name in effects else 0)
diff --git a/lib/spack/spack/test/cmd/mirror.py b/lib/spack/spack/test/cmd/mirror.py
index f6fe0b24dd..0957624cba 100644
--- a/lib/spack/spack/test/cmd/mirror.py
+++ b/lib/spack/spack/test/cmd/mirror.py
@@ -102,7 +102,7 @@ class MockMirrorArgs(object):
         self.exclude_specs = exclude_specs
 
 
-def test_exclude_specs(mock_packages):
+def test_exclude_specs(mock_packages, config):
     args = MockMirrorArgs(
         specs=['mpich'],
         versions_per_spec='all',
@@ -117,7 +117,7 @@ def test_exclude_specs(mock_packages):
     assert (not expected_exclude & set(mirror_specs))
 
 
-def test_exclude_file(mock_packages, tmpdir):
+def test_exclude_file(mock_packages, tmpdir, config):
     exclude_path = os.path.join(str(tmpdir), 'test-exclude.txt')
     with open(exclude_path, 'w') as exclude_file:
         exclude_file.write("""\
diff --git a/lib/spack/spack/test/cmd/pkg.py b/lib/spack/spack/test/cmd/pkg.py
index af634beffc..ca9e3a1a3e 100644
--- a/lib/spack/spack/test/cmd/pkg.py
+++ b/lib/spack/spack/test/cmd/pkg.py
@@ -62,9 +62,9 @@ def mock_pkg_git_repo(tmpdir_factory):
         mkdirp('pkg-a', 'pkg-b', 'pkg-c')
         with open('pkg-a/package.py', 'w') as f:
             f.write(pkg_template.format(name='PkgA'))
-        with open('pkg-c/package.py', 'w') as f:
-            f.write(pkg_template.format(name='PkgB'))
         with open('pkg-b/package.py', 'w') as f:
+            f.write(pkg_template.format(name='PkgB'))
+        with open('pkg-c/package.py', 'w') as f:
             f.write(pkg_template.format(name='PkgC'))
         git('add', 'pkg-a', 'pkg-b', 'pkg-c')
         git('-c', 'commit.gpgsign=false', 'commit',
@@ -128,6 +128,8 @@ def test_pkg_add(mock_pkg_git_repo):
                     git('status', '--short', output=str))
         finally:
             shutil.rmtree('pkg-e')
+            # Removing a package mid-run disrupts Spack's caching
+            spack.repo.path.repos[0]._fast_package_checker.invalidate()
 
     with pytest.raises(spack.main.SpackCommandError):
         pkg('add', 'does-not-exist')
diff --git a/lib/spack/spack/test/cmd/test.py b/lib/spack/spack/test/cmd/test.py
index a9ef735afe..4163853274 100644
--- a/lib/spack/spack/test/cmd/test.py
+++ b/lib/spack/spack/test/cmd/test.py
@@ -3,93 +3,181 @@
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
+import argparse
+import os
+
+import pytest
+
+import spack.config
+import spack.package
+import spack.cmd.install
 from spack.main import SpackCommand
 
+install = SpackCommand('install')
 spack_test = SpackCommand('test')
-cmd_test_py = 'lib/spack/spack/test/cmd/test.py'
-
-
-def test_list():
-    output = spack_test('--list')
-    assert "test.py" in output
-    assert "spec_semantics.py" in output
-    assert "test_list" not in output
-
-
-def test_list_with_pytest_arg():
-    output = spack_test('--list', cmd_test_py)
-    assert output.strip() == cmd_test_py
-
-
-def test_list_with_keywords():
-    output = spack_test('--list', '-k', 'cmd/test.py')
-    assert output.strip() == cmd_test_py
-
-
-def test_list_long(capsys):
-    with capsys.disabled():
-        output = spack_test('--list-long')
-    assert "test.py::\n" in output
-    assert "test_list" in output
-    assert "test_list_with_pytest_arg" in output
-    assert "test_list_with_keywords" in output
-    assert "test_list_long" in output
-    assert "test_list_long_with_pytest_arg" in output
-    assert "test_list_names" in output
-    assert "test_list_names_with_pytest_arg" in output
-
-    assert "spec_dag.py::\n" in output
-    assert 'test_installed_deps' in output
-    assert 'test_test_deptype' in output
-
-
-def test_list_long_with_pytest_arg(capsys):
-    with capsys.disabled():
-        output = spack_test('--list-long', cmd_test_py)
-    assert "test.py::\n" in output
-    assert "test_list" in output
-    assert "test_list_with_pytest_arg" in output
-    assert "test_list_with_keywords" in output
-    assert "test_list_long" in output
-    assert "test_list_long_with_pytest_arg" in output
-    assert "test_list_names" in output
-    assert "test_list_names_with_pytest_arg" in output
-
-    assert "spec_dag.py::\n" not in output
-    assert 'test_installed_deps' not in output
-    assert 'test_test_deptype' not in output
-
-
-def test_list_names():
-    output = spack_test('--list-names')
-    assert "test.py::test_list\n" in output
-    assert "test.py::test_list_with_pytest_arg\n" in output
-    assert "test.py::test_list_with_keywords\n" in output
-    assert "test.py::test_list_long\n" in output
-    assert "test.py::test_list_long_with_pytest_arg\n" in output
-    assert "test.py::test_list_names\n" in output
-    assert "test.py::test_list_names_with_pytest_arg\n" in output
-
-    assert "spec_dag.py::test_installed_deps\n" in output
-    assert 'spec_dag.py::test_test_deptype\n' in output
-
-
-def test_list_names_with_pytest_arg():
-    output = spack_test('--list-names', cmd_test_py)
-    assert "test.py::test_list\n" in output
-    assert "test.py::test_list_with_pytest_arg\n" in output
-    assert "test.py::test_list_with_keywords\n" in output
-    assert "test.py::test_list_long\n" in output
-    assert "test.py::test_list_long_with_pytest_arg\n" in output
-    assert "test.py::test_list_names\n" in output
-    assert "test.py::test_list_names_with_pytest_arg\n" in output
-
-    assert "spec_dag.py::test_installed_deps\n" not in output
-    assert 'spec_dag.py::test_test_deptype\n' not in output
-
-
-def test_pytest_help():
-    output = spack_test('--pytest-help')
-    assert "-k EXPRESSION" in output
-    assert "pytest-warnings:" in output
-    assert "--collect-only" in output
+
+
+def test_test_package_not_installed(
+        tmpdir, mock_packages, mock_archive, mock_fetch, config,
+        install_mockery_mutable_config, mock_test_stage):
+
+    output = spack_test('run', 'libdwarf')
+
+    assert "No installed packages match spec libdwarf" in output
+
+
+@pytest.mark.parametrize('arguments,expected', [
+    (['run'], spack.config.get('config:dirty')),  # default from config file
+    (['run', '--clean'], False),
+    (['run', '--dirty'], True),
+])
+def test_test_dirty_flag(arguments, expected):
+    parser = argparse.ArgumentParser()
+    spack.cmd.test.setup_parser(parser)
+    args = parser.parse_args(arguments)
+    assert args.dirty == expected
+
+
+def test_test_output(mock_test_stage, mock_packages, mock_archive, mock_fetch,
+                     install_mockery_mutable_config):
+    """Ensure output printed from pkgs is captured by output redirection."""
+    install('printing-package')
+    spack_test('run', 'printing-package')
+
+    stage_files = os.listdir(mock_test_stage)
+    assert len(stage_files) == 1
+
+    # Grab test stage directory contents
+    testdir = os.path.join(mock_test_stage, stage_files[0])
+    testdir_files = os.listdir(testdir)
+
+    # Grab the output from the test log
+    testlog = list(filter(lambda x: x.endswith('out.txt') and
+                          x != 'results.txt', testdir_files))
+    outfile = os.path.join(testdir, testlog[0])
+    with open(outfile, 'r') as f:
+        output = f.read()
+    assert "BEFORE TEST" in output
+    assert "true: expect command status in [" in output
+    assert "AFTER TEST" in output
+    assert "FAILED" not in output
+
+
+def test_test_output_on_error(
+    mock_packages, mock_archive, mock_fetch, install_mockery_mutable_config,
+    capfd, mock_test_stage
+):
+    install('test-error')
+    # capfd interferes with Spack's capturing
+    with capfd.disabled():
+        out = spack_test('run', 'test-error', fail_on_error=False)
+
+    assert "TestFailure" in out
+    assert "Command exited with status 1" in out
+
+
+def test_test_output_on_failure(
+    mock_packages, mock_archive, mock_fetch, install_mockery_mutable_config,
+    capfd, mock_test_stage
+):
+    install('test-fail')
+    with capfd.disabled():
+        out = spack_test('run', 'test-fail', fail_on_error=False)
+
+    assert "Expected 'not in the output' to match output of `true`" in out
+    assert "TestFailure" in out
+
+
+def test_show_log_on_error(
+    mock_packages, mock_archive, mock_fetch,
+    install_mockery_mutable_config, capfd, mock_test_stage
+):
+    """Make sure spack prints location of test log on failure."""
+    install('test-error')
+    with capfd.disabled():
+        out = spack_test('run', 'test-error', fail_on_error=False)
+
+    assert 'See test log' in out
+    assert mock_test_stage in out
+
+
+@pytest.mark.usefixtures(
+    'mock_packages', 'mock_archive', 'mock_fetch',
+    'install_mockery_mutable_config'
+)
+@pytest.mark.parametrize('pkg_name,msgs', [
+    ('test-error', ['FAILED: Command exited', 'TestFailure']),
+    ('test-fail', ['FAILED: Expected', 'TestFailure'])
+])
+def test_junit_output_with_failures(tmpdir, mock_test_stage, pkg_name, msgs):
+    install(pkg_name)
+    with tmpdir.as_cwd():
+        spack_test('run',
+                   '--log-format=junit', '--log-file=test.xml',
+                   pkg_name)
+
+    files = tmpdir.listdir()
+    filename = tmpdir.join('test.xml')
+    assert filename in files
+
+    content = filename.open().read()
+
+    # Count failures and errors correctly
+    assert 'tests="1"' in content
+    assert 'failures="1"' in content
+    assert 'errors="0"' in content
+
+    # We want to have both stdout and stderr
+    assert '<system-out>' in content
+    for msg in msgs:
+        assert msg in content
+
+
+def test_cdash_output_test_error(
+        tmpdir, mock_fetch, install_mockery_mutable_config, mock_packages,
+        mock_archive, mock_test_stage, capfd):
+    install('test-error')
+    with tmpdir.as_cwd():
+        spack_test('run',
+                   '--log-format=cdash',
+                   '--log-file=cdash_reports',
+                   'test-error')
+        report_dir = tmpdir.join('cdash_reports')
+        print(tmpdir.listdir())
+        assert report_dir in tmpdir.listdir()
+        report_file = report_dir.join('test-error_Test.xml')
+        assert report_file in report_dir.listdir()
+        content = report_file.open().read()
+        assert 'FAILED: Command exited with status 1' in content
+
+
+def test_cdash_upload_clean_test(
+        tmpdir, mock_fetch, install_mockery_mutable_config, mock_packages,
+        mock_archive, mock_test_stage):
+    install('printing-package')
+    with tmpdir.as_cwd():
+        spack_test('run',
+                   '--log-file=cdash_reports',
+                   '--log-format=cdash',
+                   'printing-package')
+        report_dir = tmpdir.join('cdash_reports')
+        assert report_dir in tmpdir.listdir()
+        report_file = report_dir.join('printing-package_Test.xml')
+        assert report_file in report_dir.listdir()
+        content = report_file.open().read()
+        assert '</Test>' in content
+        assert '<Text>' not in content
+
+
+def test_test_help_does_not_show_cdash_options(mock_test_stage, capsys):
+    """Make sure `spack test --help` does not describe CDash arguments"""
+    with pytest.raises(SystemExit):
+        spack_test('run', '--help')
+        captured = capsys.readouterr()
+        assert 'CDash URL' not in captured.out
+
+
+def test_test_help_cdash(mock_test_stage):
+    """Make sure `spack test --help-cdash` describes CDash arguments"""
+    out = spack_test('run', '--help-cdash')
+    assert 'CDash URL' in out
diff --git a/lib/spack/spack/test/cmd/unit_test.py b/lib/spack/spack/test/cmd/unit_test.py
new file mode 100644
index 0000000000..c5b8eb765e
--- /dev/null
+++ b/lib/spack/spack/test/cmd/unit_test.py
@@ -0,0 +1,96 @@
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+from spack.main import SpackCommand
+
+spack_test = SpackCommand('unit-test')
+cmd_test_py = 'lib/spack/spack/test/cmd/unit_test.py'
+
+
+def test_list():
+    output = spack_test('--list')
+    assert "unit_test.py" in output
+    assert "spec_semantics.py" in output
+    assert "test_list" not in output
+
+
+def test_list_with_pytest_arg():
+    output = spack_test('--list', cmd_test_py)
+    assert output.strip() == cmd_test_py
+
+
+def test_list_with_keywords():
+    output = spack_test('--list', '-k', 'cmd/unit_test.py')
+    assert output.strip() == cmd_test_py
+
+
+def test_list_long(capsys):
+    with capsys.disabled():
+        output = spack_test('--list-long')
+    assert "unit_test.py::\n" in output
+    assert "test_list" in output
+    assert "test_list_with_pytest_arg" in output
+    assert "test_list_with_keywords" in output
+    assert "test_list_long" in output
+    assert "test_list_long_with_pytest_arg" in output
+    assert "test_list_names" in output
+    assert "test_list_names_with_pytest_arg" in output
+
+    assert "spec_dag.py::\n" in output
+    assert 'test_installed_deps' in output
+    assert 'test_test_deptype' in output
+
+
+def test_list_long_with_pytest_arg(capsys):
+    with capsys.disabled():
+        output = spack_test('--list-long', cmd_test_py)
+    print(output)
+    assert "unit_test.py::\n" in output
+    assert "test_list" in output
+    assert "test_list_with_pytest_arg" in output
+    assert "test_list_with_keywords" in output
+    assert "test_list_long" in output
+    assert "test_list_long_with_pytest_arg" in output
+    assert "test_list_names" in output
+    assert "test_list_names_with_pytest_arg" in output
+
+    assert "spec_dag.py::\n" not in output
+    assert 'test_installed_deps' not in output
+    assert 'test_test_deptype' not in output
+
+
+def test_list_names():
+    output = spack_test('--list-names')
+    assert "unit_test.py::test_list\n" in output
+    assert "unit_test.py::test_list_with_pytest_arg\n" in output
+    assert "unit_test.py::test_list_with_keywords\n" in output
+    assert "unit_test.py::test_list_long\n" in output
+    assert "unit_test.py::test_list_long_with_pytest_arg\n" in output
+    assert "unit_test.py::test_list_names\n" in output
+    assert "unit_test.py::test_list_names_with_pytest_arg\n" in output
+
+    assert "spec_dag.py::test_installed_deps\n" in output
+    assert 'spec_dag.py::test_test_deptype\n' in output
+
+
+def test_list_names_with_pytest_arg():
+    output = spack_test('--list-names', cmd_test_py)
+    assert "unit_test.py::test_list\n" in output
+    assert "unit_test.py::test_list_with_pytest_arg\n" in output
+    assert "unit_test.py::test_list_with_keywords\n" in output
+    assert "unit_test.py::test_list_long\n" in output
+    assert "unit_test.py::test_list_long_with_pytest_arg\n" in output
+    assert "unit_test.py::test_list_names\n" in output
+    assert "unit_test.py::test_list_names_with_pytest_arg\n" in output
+
+    assert "spec_dag.py::test_installed_deps\n" not in output
+    assert 'spec_dag.py::test_test_deptype\n' not in output
+
+
+def test_pytest_help():
+    output = spack_test('--pytest-help')
+    assert "-k EXPRESSION" in output
+    assert "pytest-warnings:" in output
+    assert "--collect-only" in output
diff --git a/lib/spack/spack/test/conftest.py b/lib/spack/spack/test/conftest.py
index 527d6a3380..8615b14abe 100644
--- a/lib/spack/spack/test/conftest.py
+++ b/lib/spack/spack/test/conftest.py
@@ -1273,3 +1273,14 @@ def mock_executable(tmpdir):
         return str(f)
 
     return _factory
+
+
+@pytest.fixture()
+def mock_test_stage(mutable_config, tmpdir):
+    # NOTE: This fixture MUST be applied after any fixture that uses
+    # the config fixture under the hood
+    # No need to unset because we use mutable_config
+    tmp_stage = str(tmpdir.join('test_stage'))
+    mutable_config.set('config:test_stage', tmp_stage)
+
+    yield tmp_stage
diff --git a/lib/spack/spack/test/llnl/util/tty/__init__.py b/lib/spack/spack/test/llnl/util/tty/__init__.py
new file mode 100644
index 0000000000..9f87532b85
--- /dev/null
+++ b/lib/spack/spack/test/llnl/util/tty/__init__.py
@@ -0,0 +1,4 @@
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
diff --git a/lib/spack/spack/test/mirror.py b/lib/spack/spack/test/mirror.py
index 05cf46dc20..8d3ec2d07b 100644
--- a/lib/spack/spack/test/mirror.py
+++ b/lib/spack/spack/test/mirror.py
@@ -16,7 +16,7 @@ from spack.util.executable import which
 
 from llnl.util.filesystem import resolve_link_target_relative_to_the_link
 
-pytestmark = pytest.mark.usefixtures('config', 'mutable_mock_repo')
+pytestmark = pytest.mark.usefixtures('mutable_config', 'mutable_mock_repo')
 
 # paths in repos that shouldn't be in the mirror tarballs.
 exclude = ['.hg', '.git', '.svn']
@@ -97,7 +97,7 @@ def check_mirror():
                         # tarball
                         assert not dcmp.right_only
                         # and that all original files are present.
-                        assert all(l in exclude for l in dcmp.left_only)
+                        assert all(left in exclude for left in dcmp.left_only)
 
 
 def test_url_mirror(mock_archive):
diff --git a/lib/spack/spack/test/package_class.py b/lib/spack/spack/test/package_class.py
index d540ac663e..33e5eb1c0a 100644
--- a/lib/spack/spack/test/package_class.py
+++ b/lib/spack/spack/test/package_class.py
@@ -10,7 +10,12 @@ etc.).  Only methods like ``possible_dependencies()`` that deal with the
 static DSL metadata for packages.
 """
 
+import os
 import pytest
+import shutil
+
+import llnl.util.filesystem as fs
+
 import spack.package
 import spack.repo
 
@@ -119,3 +124,72 @@ def test_possible_dependencies_with_multiple_classes(
     })
 
     assert expected == spack.package.possible_dependencies(*pkgs)
+
+
+def setup_install_test(source_paths, install_test_root):
+    """
+    Set up the install test by creating sources and install test roots.
+
+    The convention used here is to create an empty file if the path name
+    ends with an extension otherwise, a directory is created.
+    """
+    fs.mkdirp(install_test_root)
+    for path in source_paths:
+        if os.path.splitext(path)[1]:
+            fs.touchp(path)
+        else:
+            fs.mkdirp(path)
+
+
+@pytest.mark.parametrize('spec,sources,extras,expect', [
+    ('a',
+     ['example/a.c'],   # Source(s)
+     ['example/a.c'],   # Extra test source
+     ['example/a.c']),  # Test install dir source(s)
+    ('b',
+     ['test/b.cpp', 'test/b.hpp', 'example/b.txt'],   # Source(s)
+     ['test'],                                        # Extra test source
+     ['test/b.cpp', 'test/b.hpp']),                   # Test install dir source
+    ('c',
+     ['examples/a.py', 'examples/b.py', 'examples/c.py', 'tests/d.py'],
+     ['examples/b.py', 'tests'],
+     ['examples/b.py', 'tests/d.py']),
+])
+def test_cache_extra_sources(install_mockery, spec, sources, extras, expect):
+    """Test the package's cache extra test sources helper function."""
+
+    pkg = spack.repo.get(spec)
+    pkg.spec.concretize()
+    source_path = pkg.stage.source_path
+
+    srcs = [fs.join_path(source_path, s) for s in sources]
+    setup_install_test(srcs, pkg.install_test_root)
+
+    emsg_dir = 'Expected {0} to be a directory'
+    emsg_file = 'Expected {0} to be a file'
+    for s in srcs:
+        assert os.path.exists(s), 'Expected {0} to exist'.format(s)
+        if os.path.splitext(s)[1]:
+            assert os.path.isfile(s), emsg_file.format(s)
+        else:
+            assert os.path.isdir(s), emsg_dir.format(s)
+
+    pkg.cache_extra_test_sources(extras)
+
+    src_dests = [fs.join_path(pkg.install_test_root, s) for s in sources]
+    exp_dests = [fs.join_path(pkg.install_test_root, e) for e in expect]
+    poss_dests = set(src_dests) | set(exp_dests)
+
+    msg = 'Expected {0} to{1} exist'
+    for pd in poss_dests:
+        if pd in exp_dests:
+            assert os.path.exists(pd), msg.format(pd, '')
+            if os.path.splitext(pd)[1]:
+                assert os.path.isfile(pd), emsg_file.format(pd)
+            else:
+                assert os.path.isdir(pd), emsg_dir.format(pd)
+        else:
+            assert not os.path.exists(pd), msg.format(pd, ' not')
+
+    # Perform a little cleanup
+    shutil.rmtree(os.path.dirname(source_path))
diff --git a/lib/spack/spack/test/test_suite.py b/lib/spack/spack/test/test_suite.py
new file mode 100644
index 0000000000..1ec5106182
--- /dev/null
+++ b/lib/spack/spack/test/test_suite.py
@@ -0,0 +1,53 @@
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+import os
+import spack.install_test
+import spack.spec
+
+
+def test_test_log_pathname(mock_packages, config):
+    """Ensure test log path is reasonable."""
+    spec = spack.spec.Spec('libdwarf').concretized()
+
+    test_name = 'test_name'
+
+    test_suite = spack.install_test.TestSuite([spec], test_name)
+    logfile = test_suite.log_file_for_spec(spec)
+
+    assert test_suite.stage in logfile
+    assert test_suite.test_log_name(spec) in logfile
+
+
+def test_test_ensure_stage(mock_test_stage):
+    """Make sure test stage directory is properly set up."""
+    spec = spack.spec.Spec('libdwarf').concretized()
+
+    test_name = 'test_name'
+
+    test_suite = spack.install_test.TestSuite([spec], test_name)
+    test_suite.ensure_stage()
+
+    assert os.path.isdir(test_suite.stage)
+    assert mock_test_stage in test_suite.stage
+
+
+def test_write_test_result(mock_packages, mock_test_stage):
+    """Ensure test results written to a results file."""
+    spec = spack.spec.Spec('libdwarf').concretized()
+    result = 'TEST'
+    test_name = 'write-test'
+
+    test_suite = spack.install_test.TestSuite([spec], test_name)
+    test_suite.ensure_stage()
+    results_file = test_suite.results_file
+    test_suite.write_test_result(spec, result)
+
+    with open(results_file, 'r') as f:
+        lines = f.readlines()
+        assert len(lines) == 1
+
+        msg = lines[0]
+        assert result in msg
+        assert spec.name in msg
diff --git a/lib/spack/spack/util/executable.py b/lib/spack/spack/util/executable.py
index 097da3337e..614bc1725a 100644
--- a/lib/spack/spack/util/executable.py
+++ b/lib/spack/spack/util/executable.py
@@ -2,7 +2,7 @@
 # Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
-
+import sys
 import os
 import re
 import shlex
@@ -98,6 +98,9 @@ class Executable(object):
           If both ``output`` and ``error`` are set to ``str``, then one string
           is returned containing output concatenated with error. Not valid
           for ``input``
+        * ``str.split``, as in the ``split`` method of the Python string type.
+          Behaves the same as ``str``, except that value is also written to
+          ``stdout`` or ``stderr``.
 
         By default, the subprocess inherits the parent's file descriptors.
 
@@ -132,7 +135,7 @@ class Executable(object):
         def streamify(arg, mode):
             if isinstance(arg, string_types):
                 return open(arg, mode), True
-            elif arg is str:
+            elif arg in (str, str.split):
                 return subprocess.PIPE, False
             else:
                 return arg, False
@@ -168,12 +171,18 @@ class Executable(object):
             out, err = proc.communicate()
 
             result = None
-            if output is str or error is str:
+            if output in (str, str.split) or error in (str, str.split):
                 result = ''
-                if output is str:
-                    result += text_type(out.decode('utf-8'))
-                if error is str:
-                    result += text_type(err.decode('utf-8'))
+                if output in (str, str.split):
+                    outstr = text_type(out.decode('utf-8'))
+                    result += outstr
+                    if output is str.split:
+                        sys.stdout.write(outstr)
+                if error in (str, str.split):
+                    errstr = text_type(err.decode('utf-8'))
+                    result += errstr
+                    if error is str.split:
+                        sys.stderr.write(errstr)
 
             rc = self.returncode = proc.returncode
             if fail_on_error and rc != 0 and (rc not in ignore_errors):
diff --git a/lib/spack/spack/util/mock_package.py b/lib/spack/spack/util/mock_package.py
index e855aae015..4751f5af7e 100644
--- a/lib/spack/spack/util/mock_package.py
+++ b/lib/spack/spack/util/mock_package.py
@@ -21,6 +21,8 @@ class MockPackageBase(object):
     Use ``MockPackageMultiRepo.add_package()`` to create new instances.
 
     """
+    virtual = False
+
     def __init__(self, dependencies, dependency_types,
                  conditions=None, versions=None):
         """Instantiate a new MockPackageBase.
@@ -92,7 +94,7 @@ class MockPackageMultiRepo(object):
     def exists(self, name):
         return name in self.spec_to_pkg
 
-    def is_virtual(self, name):
+    def is_virtual(self, name, use_index=True):
         return False
 
     def repo_for_pkg(self, name):