diff options
Diffstat (limited to 'lib')
48 files changed, 2283 insertions, 620 deletions
diff --git a/lib/spack/docs/contribution_guide.rst b/lib/spack/docs/contribution_guide.rst index 37cf9091bd..8df8ad65ba 100644 --- a/lib/spack/docs/contribution_guide.rst +++ b/lib/spack/docs/contribution_guide.rst @@ -74,7 +74,7 @@ locally to speed up the review process. We currently test against Python 2.6, 2.7, and 3.5-3.7 on both macOS and Linux and perform 3 types of tests: -.. _cmd-spack-test: +.. _cmd-spack-unit-test: ^^^^^^^^^^ Unit Tests @@ -96,7 +96,7 @@ To run *all* of the unit tests, use: .. code-block:: console - $ spack test + $ spack unit-test These tests may take several minutes to complete. If you know you are only modifying a single Spack feature, you can run subsets of tests at a @@ -105,13 +105,13 @@ time. For example, this would run all the tests in .. code-block:: console - $ spack test lib/spack/spack/test/architecture.py + $ spack unit-test lib/spack/spack/test/architecture.py And this would run the ``test_platform`` test from that file: .. code-block:: console - $ spack test lib/spack/spack/test/architecture.py::test_platform + $ spack unit-test lib/spack/spack/test/architecture.py::test_platform This allows you to develop iteratively: make a change, test that change, make another change, test that change, etc. We use `pytest @@ -121,29 +121,29 @@ pytest docs <http://doc.pytest.org/en/latest/usage.html#specifying-tests-selecting-tests>`_ for more details on test selection syntax. -``spack test`` has a few special options that can help you understand -what tests are available. To get a list of all available unit test -files, run: +``spack unit-test`` has a few special options that can help you +understand what tests are available. To get a list of all available +unit test files, run: -.. command-output:: spack test --list +.. command-output:: spack unit-test --list :ellipsis: 5 -To see a more detailed list of available unit tests, use ``spack test ---list-long``: +To see a more detailed list of available unit tests, use ``spack +unit-test --list-long``: -.. command-output:: spack test --list-long +.. command-output:: spack unit-test --list-long :ellipsis: 10 And to see the fully qualified names of all tests, use ``--list-names``: -.. command-output:: spack test --list-names +.. command-output:: spack unit-test --list-names :ellipsis: 5 You can combine these with ``pytest`` arguments to restrict which tests you want to know about. For example, to see just the tests in ``architecture.py``: -.. command-output:: spack test --list-long lib/spack/spack/test/architecture.py +.. command-output:: spack unit-test --list-long lib/spack/spack/test/architecture.py You can also combine any of these options with a ``pytest`` keyword search. See the `pytest usage docs @@ -151,7 +151,7 @@ search. See the `pytest usage docs for more details on test selection syntax. For example, to see the names of all tests that have "spec" or "concretize" somewhere in their names: -.. command-output:: spack test --list-names -k "spec and concretize" +.. command-output:: spack unit-test --list-names -k "spec and concretize" By default, ``pytest`` captures the output of all unit tests, and it will print any captured output for failed tests. Sometimes it's helpful to see @@ -161,7 +161,7 @@ argument to ``pytest``: .. code-block:: console - $ spack test -s --list-long lib/spack/spack/test/architecture.py::test_platform + $ spack unit-test -s --list-long lib/spack/spack/test/architecture.py::test_platform Unit tests are crucial to making sure bugs aren't introduced into Spack. If you are modifying core Spack libraries or adding new @@ -176,7 +176,7 @@ how to write tests! You may notice the ``share/spack/qa/run-unit-tests`` script in the repository. This script is designed for CI. It runs the unit tests and reports coverage statistics back to Codecov. If you want to - run the unit tests yourself, we suggest you use ``spack test``. + run the unit tests yourself, we suggest you use ``spack unit-test``. ^^^^^^^^^^^^ Flake8 Tests diff --git a/lib/spack/docs/developer_guide.rst b/lib/spack/docs/developer_guide.rst index 7fd4d1ec6e..b4b8b5d321 100644 --- a/lib/spack/docs/developer_guide.rst +++ b/lib/spack/docs/developer_guide.rst @@ -363,11 +363,12 @@ Developer commands ``spack doc`` ^^^^^^^^^^^^^ -^^^^^^^^^^^^^^ -``spack test`` -^^^^^^^^^^^^^^ +^^^^^^^^^^^^^^^^^^^ +``spack unit-test`` +^^^^^^^^^^^^^^^^^^^ -See the :ref:`contributor guide section <cmd-spack-test>` on ``spack test``. +See the :ref:`contributor guide section <cmd-spack-unit-test>` on +``spack unit-test``. .. _cmd-spack-python: diff --git a/lib/spack/docs/extensions.rst b/lib/spack/docs/extensions.rst index c71a6511ed..15c59c76ef 100644 --- a/lib/spack/docs/extensions.rst +++ b/lib/spack/docs/extensions.rst @@ -87,11 +87,12 @@ will be available from the command line: --implicit select specs that are not installed or were installed implicitly --output OUTPUT where to dump the result -The corresponding unit tests can be run giving the appropriate options to ``spack test``: +The corresponding unit tests can be run giving the appropriate options +to ``spack unit-test``: .. code-block:: console - $ spack test --extension=scripting + $ spack unit-test --extension=scripting ============================================================== test session starts =============================================================== platform linux2 -- Python 2.7.15rc1, pytest-3.2.5, py-1.4.34, pluggy-0.4.0 diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst index 836fc12b83..ec83679a47 100644 --- a/lib/spack/docs/packaging_guide.rst +++ b/lib/spack/docs/packaging_guide.rst @@ -3948,6 +3948,118 @@ using the ``run_before`` decorator. .. _file-manipulation: +^^^^^^^^^^^^^ +Install Tests +^^^^^^^^^^^^^ + +.. warning:: + + The API for adding and running install tests is not yet considered + stable and may change drastically in future releases. Packages with + upstreamed tests will be refactored to match changes to the API. + +While build-tests are integrated with the build system, install tests +may be added to Spack packages to be run independently of the install +method. + +Install tests may be added by defining a ``test`` method with the following signature: + +.. code-block:: python + + def test(self): + +These tests will be run in an environment set up to provide access to +this package and all of its dependencies, including ``test``-type +dependencies. Inside the ``test`` method, standard python ``assert`` +statements and other error reporting mechanisms can be used. Spack +will report any errors as a test failure. + +Inside the test method, individual tests can be run separately (and +continue transparently after a test failure) using the ``run_test`` +method. The signature for the ``run_test`` method is: + +.. code-block:: python + + def run_test(self, exe, options=[], expected=[], status=0, installed=False, + purpose='', skip_missing=False, work_dir=None): + +This method will operate in ``work_dir`` if one is specified. It will +search for an executable in the ``PATH`` variable named ``exe``, and +if ``installed=True`` it will fail if that executable does not come +from the prefix of the package being tested. If the executable is not +found, it will fail the test unless ``skip_missing`` is set to +``True``. The executable will be run with the options specified, and +the return code will be checked against the ``status`` argument, which +can be an integer or list of integers. Spack will also check that +every string in ``expected`` is a regex matching part of the output of +the executable. The ``purpose`` argument is recorded in the test log +for debugging purposes. + +"""""""""""""""""""""""""""""""""""""" +Install tests that require compilation +"""""""""""""""""""""""""""""""""""""" + +Some tests may require access to the compiler with which the package +was built, especially to test library-only packages. To ensure the +compiler is configured as part of the test environment, set the +attribute ``tests_require_compiler = True`` on the package. The +compiler will be available through the canonical environment variables +(``CC``, ``CXX``, ``FC``, ``F77``) in the test environment. + +"""""""""""""""""""""""""""""""""""""""""""""""" +Install tests that require build-time components +"""""""""""""""""""""""""""""""""""""""""""""""" + +Some packages cannot be easily tested without components from the +build-time test suite. For those packages, the +``cache_extra_test_sources`` method can be used. + +.. code-block:: python + + @run_after('install') + def cache_test_sources(self): + srcs = ['./tests/foo.c', './tests/bar.c'] + self.cache_extra_test_sources(srcs) + +This method will copy the listed methods into the metadata directory +of the package at the end of the install phase of the build. They will +be available to the test method in the directory +``self._extra_tests_path``. + +While source files are generally recommended, for many packages +binaries may also technically be cached in this way for later testing. + +""""""""""""""""""""" +Running install tests +""""""""""""""""""""" + +Install tests can be run using the ``spack test run`` command. The +``spack test run`` command will create a ``test suite`` out of the +specs provided to it, or if no specs are provided it will test all +specs in the active environment, or all specs installed in Spack if no +environment is active. Test suites can be named using the ``--alias`` +option; test suites not aliased will use the content hash of their +specs as their name. + +Packages to install test can be queried using the ``spack test list`` +command, which outputs all installed packages with defined ``test`` +methods. + +Test suites can be found using the ``spack test find`` command. It +will list all test suites that have been run and have not been removed +using the ``spack test remove`` command. The ``spack test remove`` +command will remove tests to declutter the test stage. The ``spack +test results`` command will show results for completed test suites. + +The test stage is the working directory for all install tests run with +Spack. By default, Spack uses ``~/.spack/test`` as the test stage. The +test stage can be set in the high-level config: + +.. code-block:: yaml + + config: + test_stage: /path/to/stage + --------------------------- File manipulation functions --------------------------- diff --git a/lib/spack/external/ctest_log_parser.py b/lib/spack/external/ctest_log_parser.py index 0437b6e524..072c10d7a9 100644 --- a/lib/spack/external/ctest_log_parser.py +++ b/lib/spack/external/ctest_log_parser.py @@ -118,6 +118,7 @@ _error_matches = [ "([^:]+): (Error:|error|undefined reference|multiply defined)", "([^ :]+) ?: (error|fatal error|catastrophic error)", "([^:]+)\\(([^\\)]+)\\) ?: (error|fatal error|catastrophic error)"), + "^FAILED", "^[Bb]us [Ee]rror", "^[Ss]egmentation [Vv]iolation", "^[Ss]egmentation [Ff]ault", diff --git a/lib/spack/llnl/util/filesystem.py b/lib/spack/llnl/util/filesystem.py index b8d0b4d2f1..d6579555ad 100644 --- a/lib/spack/llnl/util/filesystem.py +++ b/lib/spack/llnl/util/filesystem.py @@ -41,6 +41,8 @@ __all__ = [ 'fix_darwin_install_name', 'force_remove', 'force_symlink', + 'chgrp', + 'chmod_x', 'copy', 'install', 'copy_tree', @@ -52,6 +54,7 @@ __all__ = [ 'partition_path', 'prefixes', 'remove_dead_links', + 'remove_directory_contents', 'remove_if_dead_link', 'remove_linked_tree', 'set_executable', @@ -1806,3 +1809,13 @@ def md5sum(file): with open(file, "rb") as f: md5.update(f.read()) return md5.digest() + + +def remove_directory_contents(dir): + """Remove all contents of a directory.""" + if os.path.exists(dir): + for entry in [os.path.join(dir, entry) for entry in os.listdir(dir)]: + if os.path.isfile(entry) or os.path.islink(entry): + os.unlink(entry) + else: + shutil.rmtree(entry) diff --git a/lib/spack/spack/build_environment.py b/lib/spack/spack/build_environment.py index a1501034d4..cb1ba21ba5 100644 --- a/lib/spack/spack/build_environment.py +++ b/lib/spack/spack/build_environment.py @@ -32,8 +32,8 @@ There are two parts to the build environment: Skimming this module is a nice way to get acquainted with the types of calls you can make from within the install() function. """ -import re import inspect +import re import multiprocessing import os import shutil @@ -53,10 +53,14 @@ import spack.build_systems.meson import spack.config import spack.main import spack.paths +import spack.package +import spack.repo import spack.schema.environment import spack.store +import spack.install_test import spack.subprocess_context import spack.architecture as arch +import spack.util.path from spack.util.string import plural from spack.util.environment import ( env_flag, filter_system_paths, get_path, is_system_path, @@ -453,7 +457,6 @@ def _set_variables_for_single_module(pkg, module): jobs = spack.config.get('config:build_jobs', 16) if pkg.parallel else 1 jobs = min(jobs, multiprocessing.cpu_count()) - assert jobs is not None, "no default set for config:build_jobs" m = module m.make_jobs = jobs @@ -713,28 +716,42 @@ def load_external_modules(pkg): load_module(external_module) -def setup_package(pkg, dirty): +def setup_package(pkg, dirty, context='build'): """Execute all environment setup routines.""" - build_env = EnvironmentModifications() + env = EnvironmentModifications() if not dirty: clean_environment() - set_compiler_environment_variables(pkg, build_env) - set_build_environment_variables(pkg, build_env, dirty) - pkg.architecture.platform.setup_platform_environment(pkg, build_env) + # setup compilers and build tools for build contexts + need_compiler = context == 'build' or (context == 'test' and + pkg.test_requires_compiler) + if need_compiler: + set_compiler_environment_variables(pkg, env) + set_build_environment_variables(pkg, env, dirty) - build_env.extend( - modifications_from_dependencies(pkg.spec, context='build') - ) + # architecture specific setup + pkg.architecture.platform.setup_platform_environment(pkg, env) - if (not dirty) and (not build_env.is_unset('CPATH')): - tty.debug("A dependency has updated CPATH, this may lead pkg-config" - " to assume that the package is part of the system" - " includes and omit it when invoked with '--cflags'.") + if context == 'build': + # recursive post-order dependency information + env.extend( + modifications_from_dependencies(pkg.spec, context=context) + ) + + if (not dirty) and (not env.is_unset('CPATH')): + tty.debug("A dependency has updated CPATH, this may lead pkg-" + "config to assume that the package is part of the system" + " includes and omit it when invoked with '--cflags'.") - set_module_variables_for_package(pkg) - pkg.setup_build_environment(build_env) + # setup package itself + set_module_variables_for_package(pkg) + pkg.setup_build_environment(env) + elif context == 'test': + import spack.user_environment as uenv # avoid circular import + env.extend(uenv.environment_modifications_for_spec(pkg.spec)) + set_module_variables_for_package(pkg) + env.prepend_path('PATH', '.') # Loading modules, in particular if they are meant to be used outside # of Spack, can change environment variables that are relevant to the @@ -744,15 +761,16 @@ def setup_package(pkg, dirty): # unnecessary. Modules affecting these variables will be overwritten anyway with preserve_environment('CC', 'CXX', 'FC', 'F77'): # All module loads that otherwise would belong in previous - # functions have to occur after the build_env object has its + # functions have to occur after the env object has its # modifications applied. Otherwise the environment modifications # could undo module changes, such as unsetting LD_LIBRARY_PATH # after a module changes it. - for mod in pkg.compiler.modules: - # Fixes issue https://github.com/spack/spack/issues/3153 - if os.environ.get("CRAY_CPU_TARGET") == "mic-knl": - load_module("cce") - load_module(mod) + if need_compiler: + for mod in pkg.compiler.modules: + # Fixes issue https://github.com/spack/spack/issues/3153 + if os.environ.get("CRAY_CPU_TARGET") == "mic-knl": + load_module("cce") + load_module(mod) # kludge to handle cray libsci being automatically loaded by PrgEnv # modules on cray platform. Module unload does no damage when @@ -766,12 +784,12 @@ def setup_package(pkg, dirty): implicit_rpaths = pkg.compiler.implicit_rpaths() if implicit_rpaths: - build_env.set('SPACK_COMPILER_IMPLICIT_RPATHS', - ':'.join(implicit_rpaths)) + env.set('SPACK_COMPILER_IMPLICIT_RPATHS', + ':'.join(implicit_rpaths)) # Make sure nothing's strange about the Spack environment. - validate(build_env, tty.warn) - build_env.apply_modifications() + validate(env, tty.warn) + env.apply_modifications() def modifications_from_dependencies(spec, context): @@ -791,7 +809,8 @@ def modifications_from_dependencies(spec, context): deptype_and_method = { 'build': (('build', 'link', 'test'), 'setup_dependent_build_environment'), - 'run': (('link', 'run'), 'setup_dependent_run_environment') + 'run': (('link', 'run'), 'setup_dependent_run_environment'), + 'test': (('link', 'run', 'test'), 'setup_dependent_run_environment') } deptype, method = deptype_and_method[context] @@ -808,6 +827,8 @@ def modifications_from_dependencies(spec, context): def _setup_pkg_and_run(serialized_pkg, function, kwargs, child_pipe, input_multiprocess_fd): + context = kwargs.get('context', 'build') + try: # We are in the child process. Python sets sys.stdin to # open(os.devnull) to prevent our process and its parent from @@ -821,7 +842,8 @@ def _setup_pkg_and_run(serialized_pkg, function, kwargs, child_pipe, if not kwargs.get('fake', False): kwargs['unmodified_env'] = os.environ.copy() - setup_package(pkg, dirty=kwargs.get('dirty', False)) + setup_package(pkg, dirty=kwargs.get('dirty', False), + context=context) return_value = function(pkg, kwargs) child_pipe.send(return_value) @@ -841,13 +863,18 @@ def _setup_pkg_and_run(serialized_pkg, function, kwargs, child_pipe, # show that, too. package_context = get_package_context(tb) - build_log = None - try: - if hasattr(pkg, 'log_path'): - build_log = pkg.log_path - except NameError: - # 'pkg' is not defined yet - pass + logfile = None + if context == 'build': + try: + if hasattr(pkg, 'log_path'): + logfile = pkg.log_path + except NameError: + # 'pkg' is not defined yet + pass + elif context == 'test': + logfile = os.path.join( + pkg.test_suite.stage, + spack.install_test.TestSuite.test_log_name(pkg.spec)) # make a pickleable exception to send to parent. msg = "%s: %s" % (exc_type.__name__, str(exc)) @@ -855,7 +882,7 @@ def _setup_pkg_and_run(serialized_pkg, function, kwargs, child_pipe, ce = ChildError(msg, exc_type.__module__, exc_type.__name__, - tb_string, build_log, package_context) + tb_string, logfile, context, package_context) child_pipe.send(ce) finally: @@ -873,9 +900,6 @@ def start_build_process(pkg, function, kwargs): child process for. function (callable): argless function to run in the child process. - dirty (bool): If True, do NOT clean the environment before - building. - fake (bool): If True, skip package setup b/c it's not a real build Usage:: @@ -961,6 +985,7 @@ def get_package_context(traceback, context=3): Args: traceback (traceback): A traceback from some exception raised during install + context (int): Lines of context to show before and after the line where the error happened @@ -1067,13 +1092,14 @@ class ChildError(InstallError): # context instead of Python context. build_errors = [('spack.util.executable', 'ProcessError')] - def __init__(self, msg, module, classname, traceback_string, build_log, - context): + def __init__(self, msg, module, classname, traceback_string, log_name, + log_type, context): super(ChildError, self).__init__(msg) self.module = module self.name = classname self.traceback = traceback_string - self.build_log = build_log + self.log_name = log_name + self.log_type = log_type self.context = context @property @@ -1081,26 +1107,16 @@ class ChildError(InstallError): out = StringIO() out.write(self._long_message if self._long_message else '') + have_log = self.log_name and os.path.exists(self.log_name) + if (self.module, self.name) in ChildError.build_errors: # The error happened in some external executed process. Show - # the build log with errors or warnings highlighted. - if self.build_log and os.path.exists(self.build_log): - errors, warnings = parse_log_events(self.build_log) - nerr = len(errors) - nwar = len(warnings) - if nerr > 0: - # If errors are found, only display errors - out.write( - "\n%s found in build log:\n" % plural(nerr, 'error')) - out.write(make_log_context(errors)) - elif nwar > 0: - # If no errors are found but warnings are, display warnings - out.write( - "\n%s found in build log:\n" % plural(nwar, 'warning')) - out.write(make_log_context(warnings)) + # the log with errors or warnings highlighted. + if have_log: + write_log_summary(out, self.log_type, self.log_name) else: - # The error happened in in the Python code, so try to show + # The error happened in the Python code, so try to show # some context from the Package itself. if self.context: out.write('\n') @@ -1110,14 +1126,14 @@ class ChildError(InstallError): if out.getvalue(): out.write('\n') - if self.build_log and os.path.exists(self.build_log): - out.write('See build log for details:\n') - out.write(' %s\n' % self.build_log) + if have_log: + out.write('See {0} log for details:\n'.format(self.log_type)) + out.write(' {0}\n'.format(self.log_name)) return out.getvalue() def __str__(self): - return self.message + self.long_message + self.traceback + return self.message def __reduce__(self): """__reduce__ is used to serialize (pickle) ChildErrors. @@ -1130,13 +1146,14 @@ class ChildError(InstallError): self.module, self.name, self.traceback, - self.build_log, + self.log_name, + self.log_type, self.context) -def _make_child_error(msg, module, name, traceback, build_log, context): +def _make_child_error(msg, module, name, traceback, log, log_type, context): """Used by __reduce__ in ChildError to reconstruct pickled errors.""" - return ChildError(msg, module, name, traceback, build_log, context) + return ChildError(msg, module, name, traceback, log, log_type, context) class StopPhase(spack.error.SpackError): @@ -1147,3 +1164,30 @@ class StopPhase(spack.error.SpackError): def _make_stop_phase(msg, long_msg): return StopPhase(msg, long_msg) + + +def write_log_summary(out, log_type, log, last=None): + errors, warnings = parse_log_events(log) + nerr = len(errors) + nwar = len(warnings) + + if nerr > 0: + if last and nerr > last: + errors = errors[-last:] + nerr = last + + # If errors are found, only display errors + out.write( + "\n%s found in %s log:\n" % + (plural(nerr, 'error'), log_type)) + out.write(make_log_context(errors)) + elif nwar > 0: + if last and nwar > last: + warnings = warnings[-last:] + nwar = last + + # If no errors are found but warnings are, display warnings + out.write( + "\n%s found in %s log:\n" % + (plural(nwar, 'warning'), log_type)) + out.write(make_log_context(warnings)) diff --git a/lib/spack/spack/build_systems/cmake.py b/lib/spack/spack/build_systems/cmake.py index 4b679b358a..1336069846 100644 --- a/lib/spack/spack/build_systems/cmake.py +++ b/lib/spack/spack/build_systems/cmake.py @@ -325,13 +325,20 @@ class CMakePackage(PackageBase): libs_flags)) @property + def build_dirname(self): + """Returns the directory name to use when building the package + + :return: name of the subdirectory for building the package + """ + return 'spack-build-%s' % self.spec.dag_hash(7) + + @property def build_directory(self): """Returns the directory to use when building the package :return: directory where to build the package """ - dirname = 'spack-build-%s' % self.spec.dag_hash(7) - return os.path.join(self.stage.path, dirname) + return os.path.join(self.stage.path, self.build_dirname) def cmake_args(self): """Produces a list containing all the arguments that must be passed to diff --git a/lib/spack/spack/build_systems/intel.py b/lib/spack/spack/build_systems/intel.py index a74d5e9613..0e0bb9378b 100644 --- a/lib/spack/spack/build_systems/intel.py +++ b/lib/spack/spack/build_systems/intel.py @@ -1017,6 +1017,15 @@ class IntelPackage(PackageBase): env.extend(EnvironmentModifications.from_sourcing_file(f, *args)) + if self.spec.name in ('intel', 'intel-parallel-studio'): + # this package provides compilers + # TODO: fix check above when compilers are dependencies + env.set('CC', self.prefix.bin.icc) + env.set('CXX', self.prefix.bin.icpc) + env.set('FC', self.prefix.bin.ifort) + env.set('F77', self.prefix.bin.ifort) + env.set('F90', self.prefix.bin.ifort) + def setup_dependent_build_environment(self, env, dependent_spec): # NB: This function is overwritten by 'mpi' provider packages: # diff --git a/lib/spack/spack/build_systems/python.py b/lib/spack/spack/build_systems/python.py index 99dc4ce0dc..76159d88a1 100644 --- a/lib/spack/spack/build_systems/python.py +++ b/lib/spack/spack/build_systems/python.py @@ -89,7 +89,7 @@ class PythonPackage(PackageBase): build_system_class = 'PythonPackage' #: Callback names for build-time test - build_time_test_callbacks = ['test'] + build_time_test_callbacks = ['build_test'] #: Callback names for install-time test install_time_test_callbacks = ['import_module_test'] @@ -359,7 +359,7 @@ class PythonPackage(PackageBase): # Testing - def test(self): + def build_test(self): """Run unit tests after in-place build. These tests are only run if the package actually has a 'test' command. diff --git a/lib/spack/spack/build_systems/scons.py b/lib/spack/spack/build_systems/scons.py index ffa4390249..5e17666b71 100644 --- a/lib/spack/spack/build_systems/scons.py +++ b/lib/spack/spack/build_systems/scons.py @@ -33,7 +33,7 @@ class SConsPackage(PackageBase): build_system_class = 'SConsPackage' #: Callback names for build-time test - build_time_test_callbacks = ['test'] + build_time_test_callbacks = ['build_test'] depends_on('scons', type='build') @@ -59,7 +59,7 @@ class SConsPackage(PackageBase): # Testing - def test(self): + def build_test(self): """Run unit tests after build. By default, does nothing. Override this if you want to diff --git a/lib/spack/spack/build_systems/waf.py b/lib/spack/spack/build_systems/waf.py index a1581660f2..a6dbbbdb35 100644 --- a/lib/spack/spack/build_systems/waf.py +++ b/lib/spack/spack/build_systems/waf.py @@ -47,10 +47,10 @@ class WafPackage(PackageBase): build_system_class = 'WafPackage' # Callback names for build-time test - build_time_test_callbacks = ['test'] + build_time_test_callbacks = ['build_test'] # Callback names for install-time test - install_time_test_callbacks = ['installtest'] + install_time_test_callbacks = ['install_test'] # Much like AutotoolsPackage does not require automake and autoconf # to build, WafPackage does not require waf to build. It only requires @@ -106,7 +106,7 @@ class WafPackage(PackageBase): # Testing - def test(self): + def build_test(self): """Run unit tests after build. By default, does nothing. Override this if you want to @@ -116,7 +116,7 @@ class WafPackage(PackageBase): run_after('build')(PackageBase._run_default_build_time_test_callbacks) - def installtest(self): + def install_test(self): """Run unit tests after install. By default, does nothing. Override this if you want to diff --git a/lib/spack/spack/cmd/build_env.py b/lib/spack/spack/cmd/build_env.py index 128d167a29..ef8b1f6e6b 100644 --- a/lib/spack/spack/cmd/build_env.py +++ b/lib/spack/spack/cmd/build_env.py @@ -2,86 +2,15 @@ # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) - -from __future__ import print_function - -import argparse -import os - -import llnl.util.tty as tty -import spack.build_environment as build_environment -import spack.cmd -import spack.cmd.common.arguments as arguments -from spack.util.environment import dump_environment, pickle_environment +import spack.cmd.common.env_utility as env_utility description = "run a command in a spec's install environment, " \ "or dump its environment to screen or file" section = "build" level = "long" - -def setup_parser(subparser): - arguments.add_common_arguments(subparser, ['clean', 'dirty']) - subparser.add_argument( - '--dump', metavar="FILE", - help="dump a source-able environment to FILE" - ) - subparser.add_argument( - '--pickle', metavar="FILE", - help="dump a pickled source-able environment to FILE" - ) - subparser.add_argument( - 'spec', nargs=argparse.REMAINDER, - metavar='spec [--] [cmd]...', - help="spec of package environment to emulate") - subparser.epilog\ - = 'If a command is not specified, the environment will be printed ' \ - 'to standard output (cf /usr/bin/env) unless --dump and/or --pickle ' \ - 'are specified.\n\nIf a command is specified and spec is ' \ - 'multi-word, then the -- separator is obligatory.' +setup_parser = env_utility.setup_parser def build_env(parser, args): - if not args.spec: - tty.die("spack build-env requires a spec.") - - # Specs may have spaces in them, so if they do, require that the - # caller put a '--' between the spec and the command to be - # executed. If there is no '--', assume that the spec is the - # first argument. - sep = '--' - if sep in args.spec: - s = args.spec.index(sep) - spec = args.spec[:s] - cmd = args.spec[s + 1:] - else: - spec = args.spec[0] - cmd = args.spec[1:] - - specs = spack.cmd.parse_specs(spec, concretize=True) - if len(specs) > 1: - tty.die("spack build-env only takes one spec.") - spec = specs[0] - - build_environment.setup_package(spec.package, args.dirty) - - if args.dump: - # Dump a source-able environment to a text file. - tty.msg("Dumping a source-able environment to {0}".format(args.dump)) - dump_environment(args.dump) - - if args.pickle: - # Dump a source-able environment to a pickle file. - tty.msg( - "Pickling a source-able environment to {0}".format(args.pickle)) - pickle_environment(args.pickle) - - if cmd: - # Execute the command with the new environment - os.execvp(cmd[0], cmd) - - elif not bool(args.pickle or args.dump): - # If no command or dump/pickle option act like the "env" command - # and print out env vars. - for key, val in os.environ.items(): - print("%s=%s" % (key, val)) + env_utility.emulate_env_utility('build-env', 'build', args) diff --git a/lib/spack/spack/cmd/clean.py b/lib/spack/spack/cmd/clean.py index d847e7a7c0..f69b959293 100644 --- a/lib/spack/spack/cmd/clean.py +++ b/lib/spack/spack/cmd/clean.py @@ -10,10 +10,11 @@ import shutil import llnl.util.tty as tty import spack.caches -import spack.cmd +import spack.cmd.test import spack.cmd.common.arguments as arguments import spack.repo import spack.stage +import spack.config from spack.paths import lib_path, var_path diff --git a/lib/spack/spack/cmd/common/arguments.py b/lib/spack/spack/cmd/common/arguments.py index e5945bda9c..e5c4c0dde8 100644 --- a/lib/spack/spack/cmd/common/arguments.py +++ b/lib/spack/spack/cmd/common/arguments.py @@ -275,3 +275,53 @@ def no_checksum(): return Args( '-n', '--no-checksum', action='store_true', default=False, help="do not use checksums to verify downloaded files (unsafe)") + + +def add_cdash_args(subparser, add_help): + cdash_help = {} + if add_help: + cdash_help['upload-url'] = "CDash URL where reports will be uploaded" + cdash_help['build'] = """The name of the build that will be reported to CDash. +Defaults to spec of the package to operate on.""" + cdash_help['site'] = """The site name that will be reported to CDash. +Defaults to current system hostname.""" + cdash_help['track'] = """Results will be reported to this group on CDash. +Defaults to Experimental.""" + cdash_help['buildstamp'] = """Instead of letting the CDash reporter prepare the +buildstamp which, when combined with build name, site and project, +uniquely identifies the build, provide this argument to identify +the build yourself. Format: %%Y%%m%%d-%%H%%M-[cdash-track]""" + else: + cdash_help['upload-url'] = argparse.SUPPRESS + cdash_help['build'] = argparse.SUPPRESS + cdash_help['site'] = argparse.SUPPRESS + cdash_help['track'] = argparse.SUPPRESS + cdash_help['buildstamp'] = argparse.SUPPRESS + + subparser.add_argument( + '--cdash-upload-url', + default=None, + help=cdash_help['upload-url'] + ) + subparser.add_argument( + '--cdash-build', + default=None, + help=cdash_help['build'] + ) + subparser.add_argument( + '--cdash-site', + default=None, + help=cdash_help['site'] + ) + + cdash_subgroup = subparser.add_mutually_exclusive_group() + cdash_subgroup.add_argument( + '--cdash-track', + default='Experimental', + help=cdash_help['track'] + ) + cdash_subgroup.add_argument( + '--cdash-buildstamp', + default=None, + help=cdash_help['buildstamp'] + ) diff --git a/lib/spack/spack/cmd/common/env_utility.py b/lib/spack/spack/cmd/common/env_utility.py new file mode 100644 index 0000000000..e3f32737b4 --- /dev/null +++ b/lib/spack/spack/cmd/common/env_utility.py @@ -0,0 +1,82 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) +from __future__ import print_function + +import argparse +import os + +import llnl.util.tty as tty +import spack.build_environment as build_environment +import spack.paths +import spack.cmd +import spack.cmd.common.arguments as arguments +from spack.util.environment import dump_environment, pickle_environment + + +def setup_parser(subparser): + arguments.add_common_arguments(subparser, ['clean', 'dirty']) + subparser.add_argument( + '--dump', metavar="FILE", + help="dump a source-able environment to FILE" + ) + subparser.add_argument( + '--pickle', metavar="FILE", + help="dump a pickled source-able environment to FILE" + ) + subparser.add_argument( + 'spec', nargs=argparse.REMAINDER, + metavar='spec [--] [cmd]...', + help="specs of package environment to emulate") + subparser.epilog\ + = 'If a command is not specified, the environment will be printed ' \ + 'to standard output (cf /usr/bin/env) unless --dump and/or --pickle ' \ + 'are specified.\n\nIf a command is specified and spec is ' \ + 'multi-word, then the -- separator is obligatory.' + + +def emulate_env_utility(cmd_name, context, args): + if not args.spec: + tty.die("spack %s requires a spec." % cmd_name) + + # Specs may have spaces in them, so if they do, require that the + # caller put a '--' between the spec and the command to be + # executed. If there is no '--', assume that the spec is the + # first argument. + sep = '--' + if sep in args.spec: + s = args.spec.index(sep) + spec = args.spec[:s] + cmd = args.spec[s + 1:] + else: + spec = args.spec[0] + cmd = args.spec[1:] + + specs = spack.cmd.parse_specs(spec, concretize=True) + if len(specs) > 1: + tty.die("spack %s only takes one spec." % cmd_name) + spec = specs[0] + + build_environment.setup_package(spec.package, args.dirty, context) + + if args.dump: + # Dump a source-able environment to a text file. + tty.msg("Dumping a source-able environment to {0}".format(args.dump)) + dump_environment(args.dump) + + if args.pickle: + # Dump a source-able environment to a pickle file. + tty.msg( + "Pickling a source-able environment to {0}".format(args.pickle)) + pickle_environment(args.pickle) + + if cmd: + # Execute the command with the new environment + os.execvp(cmd[0], cmd) + + elif not bool(args.pickle or args.dump): + # If no command or dump/pickle option then act like the "env" command + # and print out env vars. + for key, val in os.environ.items(): + print("%s=%s" % (key, val)) diff --git a/lib/spack/spack/cmd/install.py b/lib/spack/spack/cmd/install.py index c8673b5330..3b5954b1ad 100644 --- a/lib/spack/spack/cmd/install.py +++ b/lib/spack/spack/cmd/install.py @@ -167,65 +167,8 @@ packages. If neither are chosen, don't run tests for any packages.""" action='store_true', help="Show usage instructions for CDash reporting" ) - subparser.add_argument( - '-y', '--yes-to-all', - action='store_true', - dest='yes_to_all', - help="""assume "yes" is the answer to every confirmation request. -To run completely non-interactively, also specify '--no-checksum'.""" - ) - add_cdash_args(subparser, False) - arguments.add_common_arguments(subparser, ['spec']) - - -def add_cdash_args(subparser, add_help): - cdash_help = {} - if add_help: - cdash_help['upload-url'] = "CDash URL where reports will be uploaded" - cdash_help['build'] = """The name of the build that will be reported to CDash. -Defaults to spec of the package to install.""" - cdash_help['site'] = """The site name that will be reported to CDash. -Defaults to current system hostname.""" - cdash_help['track'] = """Results will be reported to this group on CDash. -Defaults to Experimental.""" - cdash_help['buildstamp'] = """Instead of letting the CDash reporter prepare the -buildstamp which, when combined with build name, site and project, -uniquely identifies the build, provide this argument to identify -the build yourself. Format: %%Y%%m%%d-%%H%%M-[cdash-track]""" - else: - cdash_help['upload-url'] = argparse.SUPPRESS - cdash_help['build'] = argparse.SUPPRESS - cdash_help['site'] = argparse.SUPPRESS - cdash_help['track'] = argparse.SUPPRESS - cdash_help['buildstamp'] = argparse.SUPPRESS - - subparser.add_argument( - '--cdash-upload-url', - default=None, - help=cdash_help['upload-url'] - ) - subparser.add_argument( - '--cdash-build', - default=None, - help=cdash_help['build'] - ) - subparser.add_argument( - '--cdash-site', - default=None, - help=cdash_help['site'] - ) - - cdash_subgroup = subparser.add_mutually_exclusive_group() - cdash_subgroup.add_argument( - '--cdash-track', - default='Experimental', - help=cdash_help['track'] - ) - cdash_subgroup.add_argument( - '--cdash-buildstamp', - default=None, - help=cdash_help['buildstamp'] - ) + arguments.add_cdash_args(subparser, False) + arguments.add_common_arguments(subparser, ['yes_to_all', 'spec']) def default_log_file(spec): @@ -283,11 +226,12 @@ environment variables: SPACK_CDASH_AUTH_TOKEN authentication token to present to CDash ''')) - add_cdash_args(parser, True) + arguments.add_cdash_args(parser, True) parser.print_help() return - reporter = spack.report.collect_info(args.log_format, args) + reporter = spack.report.collect_info( + spack.package.PackageInstaller, '_install_task', args.log_format, args) if args.log_file: reporter.filename = args.log_file @@ -383,7 +327,7 @@ environment variables: if not args.log_file and not reporter.filename: reporter.filename = default_log_file(specs[0]) reporter.specs = specs - with reporter: + with reporter('build'): if args.overwrite: installed = list(filter(lambda x: x, diff --git a/lib/spack/spack/cmd/list.py b/lib/spack/spack/cmd/list.py index 656282599d..a7d4cb0ac3 100644 --- a/lib/spack/spack/cmd/list.py +++ b/lib/spack/spack/cmd/list.py @@ -54,6 +54,9 @@ def setup_parser(subparser): subparser.add_argument( '--update', metavar='FILE', default=None, action='store', help='write output to the specified file, if any package is newer') + subparser.add_argument( + '-v', '--virtuals', action='store_true', default=False, + help='include virtual packages in list') arguments.add_common_arguments(subparser, ['tags']) @@ -267,7 +270,7 @@ def list(parser, args): formatter = formatters[args.format] # Retrieve the names of all the packages - pkgs = set(spack.repo.all_package_names()) + pkgs = set(spack.repo.all_package_names(args.virtuals)) # Filter the set appropriately sorted_packages = filter_by_name(pkgs, args) diff --git a/lib/spack/spack/cmd/test.py b/lib/spack/spack/cmd/test.py index 8cbc0fbccf..3362b8a109 100644 --- a/lib/spack/spack/cmd/test.py +++ b/lib/spack/spack/cmd/test.py @@ -4,166 +4,381 @@ # SPDX-License-Identifier: (Apache-2.0 OR MIT) from __future__ import print_function -from __future__ import division - -import collections -import sys -import re +import os import argparse -import pytest -from six import StringIO +import textwrap +import fnmatch +import re +import shutil -import llnl.util.tty.color as color -from llnl.util.filesystem import working_dir -from llnl.util.tty.colify import colify +import llnl.util.tty as tty -import spack.paths +import spack.install_test +import spack.environment as ev +import spack.cmd +import spack.cmd.common.arguments as arguments +import spack.report +import spack.package -description = "run spack's unit tests (wrapper around pytest)" -section = "developer" +description = "run spack's tests for an install" +section = "administrator" level = "long" +def first_line(docstring): + """Return the first line of the docstring.""" + return docstring.split('\n')[0] + + def setup_parser(subparser): - subparser.add_argument( - '-H', '--pytest-help', action='store_true', default=False, - help="show full pytest help, with advanced options") - - # extra spack arguments to list tests - list_group = subparser.add_argument_group("listing tests") - list_mutex = list_group.add_mutually_exclusive_group() - list_mutex.add_argument( - '-l', '--list', action='store_const', default=None, - dest='list', const='list', help="list test filenames") - list_mutex.add_argument( - '-L', '--list-long', action='store_const', default=None, - dest='list', const='long', help="list all test functions") - list_mutex.add_argument( - '-N', '--list-names', action='store_const', default=None, - dest='list', const='names', help="list full names of all tests") - - # use tests for extension - subparser.add_argument( - '--extension', default=None, - help="run test for a given spack extension") - - # spell out some common pytest arguments, so they'll show up in help - pytest_group = subparser.add_argument_group( - "common pytest arguments (spack test --pytest-help for more details)") - pytest_group.add_argument( - "-s", action='append_const', dest='parsed_args', const='-s', - help="print output while tests run (disable capture)") - pytest_group.add_argument( - "-k", action='store', metavar="EXPRESSION", dest='expression', - help="filter tests by keyword (can also use w/list options)") - pytest_group.add_argument( - "--showlocals", action='append_const', dest='parsed_args', - const='--showlocals', help="show local variable values in tracebacks") - - # remainder is just passed to pytest - subparser.add_argument( - 'pytest_args', nargs=argparse.REMAINDER, help="arguments for pytest") - - -def do_list(args, extra_args): - """Print a lists of tests than what pytest offers.""" - # Run test collection and get the tree out. - old_output = sys.stdout - try: - sys.stdout = output = StringIO() - pytest.main(['--collect-only'] + extra_args) - finally: - sys.stdout = old_output - - lines = output.getvalue().split('\n') - tests = collections.defaultdict(lambda: set()) - prefix = [] - - # collect tests into sections - for line in lines: - match = re.match(r"(\s*)<([^ ]*) '([^']*)'", line) - if not match: - continue - indent, nodetype, name = match.groups() - - # strip parametrized tests - if "[" in name: - name = name[:name.index("[")] - - depth = len(indent) // 2 - - if nodetype.endswith("Function"): - key = tuple(prefix) - tests[key].add(name) - else: - prefix = prefix[:depth] - prefix.append(name) - - def colorize(c, prefix): - if isinstance(prefix, tuple): - return "::".join( - color.colorize("@%s{%s}" % (c, p)) - for p in prefix if p != "()" - ) - return color.colorize("@%s{%s}" % (c, prefix)) - - if args.list == "list": - files = set(prefix[0] for prefix in tests) - color_files = [colorize("B", file) for file in sorted(files)] - colify(color_files) - - elif args.list == "long": - for prefix, functions in sorted(tests.items()): - path = colorize("*B", prefix) + "::" - functions = [colorize("c", f) for f in sorted(functions)] - color.cprint(path) - colify(functions, indent=4) - print() - - else: # args.list == "names" - all_functions = [ - colorize("*B", prefix) + "::" + colorize("c", f) - for prefix, functions in sorted(tests.items()) - for f in sorted(functions) - ] - colify(all_functions) - - -def add_back_pytest_args(args, unknown_args): - """Add parsed pytest args, unknown args, and remainder together. - - We add some basic pytest arguments to the Spack parser to ensure that - they show up in the short help, so we have to reassemble things here. + sp = subparser.add_subparsers(metavar='SUBCOMMAND', dest='test_command') + + # Run + run_parser = sp.add_parser('run', description=test_run.__doc__, + help=first_line(test_run.__doc__)) + + alias_help_msg = "Provide an alias for this test-suite" + alias_help_msg += " for subsequent access." + run_parser.add_argument('--alias', help=alias_help_msg) + + run_parser.add_argument( + '--fail-fast', action='store_true', + help="Stop tests for each package after the first failure." + ) + run_parser.add_argument( + '--fail-first', action='store_true', + help="Stop after the first failed package." + ) + run_parser.add_argument( + '--keep-stage', + action='store_true', + help='Keep testing directory for debugging' + ) + run_parser.add_argument( + '--log-format', + default=None, + choices=spack.report.valid_formats, + help="format to be used for log files" + ) + run_parser.add_argument( + '--log-file', + default=None, + help="filename for the log file. if not passed a default will be used" + ) + arguments.add_cdash_args(run_parser, False) + run_parser.add_argument( + '--help-cdash', + action='store_true', + help="Show usage instructions for CDash reporting" + ) + + cd_group = run_parser.add_mutually_exclusive_group() + arguments.add_common_arguments(cd_group, ['clean', 'dirty']) + + arguments.add_common_arguments(run_parser, ['installed_specs']) + + # List + sp.add_parser('list', description=test_list.__doc__, + help=first_line(test_list.__doc__)) + + # Find + find_parser = sp.add_parser('find', description=test_find.__doc__, + help=first_line(test_find.__doc__)) + find_parser.add_argument( + 'filter', nargs=argparse.REMAINDER, + help='optional case-insensitive glob patterns to filter results.') + + # Status + status_parser = sp.add_parser('status', description=test_status.__doc__, + help=first_line(test_status.__doc__)) + status_parser.add_argument( + 'names', nargs=argparse.REMAINDER, + help="Test suites for which to print status") + + # Results + results_parser = sp.add_parser('results', description=test_results.__doc__, + help=first_line(test_results.__doc__)) + results_parser.add_argument( + '-l', '--logs', action='store_true', + help="print the test log for each matching package") + results_parser.add_argument( + '-f', '--failed', action='store_true', + help="only show results for failed tests of matching packages") + results_parser.add_argument( + 'names', nargs=argparse.REMAINDER, + metavar='[name(s)] [-- installed_specs]...', + help="suite names and installed package constraints") + results_parser.epilog = 'Test results will be filtered by space-' \ + 'separated suite name(s) and installed\nspecs when provided. '\ + 'If names are provided, then only results for those test\nsuites '\ + 'will be shown. If installed specs are provided, then ony results'\ + '\nmatching those specs will be shown.' + + # Remove + remove_parser = sp.add_parser('remove', description=test_remove.__doc__, + help=first_line(test_remove.__doc__)) + arguments.add_common_arguments(remove_parser, ['yes_to_all']) + remove_parser.add_argument( + 'names', nargs=argparse.REMAINDER, + help="Test suites to remove from test stage") + + +def test_run(args): + """Run tests for the specified installed packages. + + If no specs are listed, run tests for all packages in the current + environment or all installed packages if there is no active environment. """ - result = args.parsed_args or [] - result += unknown_args or [] - result += args.pytest_args or [] - if args.expression: - result += ["-k", args.expression] - return result - - -def test(parser, args, unknown_args): - if args.pytest_help: - # make the pytest.main help output more accurate - sys.argv[0] = 'spack test' - return pytest.main(['-h']) - - # add back any parsed pytest args we need to pass to pytest - pytest_args = add_back_pytest_args(args, unknown_args) - - # The default is to test the core of Spack. If the option `--extension` - # has been used, then test that extension. - pytest_root = spack.paths.spack_root - if args.extension: - target = args.extension - extensions = spack.config.get('config:extensions') - pytest_root = spack.extensions.path_for_extension(target, *extensions) - - # pytest.ini lives in the root of the spack repository. - with working_dir(pytest_root): - if args.list: - do_list(args, pytest_args) + # cdash help option + if args.help_cdash: + parser = argparse.ArgumentParser( + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=textwrap.dedent('''\ +environment variables: + SPACK_CDASH_AUTH_TOKEN + authentication token to present to CDash + ''')) + arguments.add_cdash_args(parser, True) + parser.print_help() + return + + # set config option for fail-fast + if args.fail_fast: + spack.config.set('config:fail_fast', True, scope='command_line') + + # Get specs to test + env = ev.get_env(args, 'test') + hashes = env.all_hashes() if env else None + + specs = spack.cmd.parse_specs(args.specs) if args.specs else [None] + specs_to_test = [] + for spec in specs: + matching = spack.store.db.query_local(spec, hashes=hashes) + if spec and not matching: + tty.warn("No installed packages match spec %s" % spec) + specs_to_test.extend(matching) + + # test_stage_dir + test_suite = spack.install_test.TestSuite(specs_to_test, args.alias) + test_suite.ensure_stage() + tty.msg("Spack test %s" % test_suite.name) + + # Set up reporter + setattr(args, 'package', [s.format() for s in test_suite.specs]) + reporter = spack.report.collect_info( + spack.package.PackageBase, 'do_test', args.log_format, args) + if not reporter.filename: + if args.log_file: + if os.path.isabs(args.log_file): + log_file = args.log_file + else: + log_dir = os.getcwd() + log_file = os.path.join(log_dir, args.log_file) + else: + log_file = os.path.join( + os.getcwd(), + 'test-%s' % test_suite.name) + reporter.filename = log_file + reporter.specs = specs_to_test + + with reporter('test', test_suite.stage): + test_suite(remove_directory=not args.keep_stage, + dirty=args.dirty, + fail_first=args.fail_first) + + +def has_test_method(pkg): + return pkg.test.__func__ != spack.package.PackageBase.test + + +def test_list(args): + """List all installed packages with available tests.""" + # TODO: This can be extended to have all of the output formatting options + # from `spack find`. + env = ev.get_env(args, 'test') + hashes = env.all_hashes() if env else None + + specs = spack.store.db.query(hashes=hashes) + specs = list(filter(lambda s: has_test_method(s.package), specs)) + + spack.cmd.display_specs(specs, long=True) + + +def test_find(args): # TODO: merge with status (noargs) + """Find tests that are running or have available results. + + Displays aliases for tests that have them, otherwise test suite content + hashes.""" + test_suites = spack.install_test.get_all_test_suites() + + # Filter tests by filter argument + if args.filter: + def create_filter(f): + raw = fnmatch.translate('f' if '*' in f or '?' in f + else '*' + f + '*') + return re.compile(raw, flags=re.IGNORECASE) + filters = [create_filter(f) for f in args.filter] + + def match(t, f): + return f.match(t) + test_suites = [t for t in test_suites + if any(match(t.alias, f) for f in filters) and + os.path.isdir(t.stage)] + + names = [t.name for t in test_suites] + + if names: + # TODO: Make these specify results vs active + msg = "Spack test results available for the following tests:\n" + msg += " %s\n" % ' '.join(names) + msg += " Run `spack test remove` to remove all tests" + tty.msg(msg) + else: + msg = "No test results match the query\n" + msg += " Tests may have been removed using `spack test remove`" + tty.msg(msg) + + +def test_status(args): + """Get the current status for the specified Spack test suite(s).""" + if args.names: + test_suites = [] + for name in args.names: + test_suite = spack.install_test.get_test_suite(name) + if test_suite: + test_suites.append(test_suite) + else: + tty.msg("No test suite %s found in test stage" % name) + else: + test_suites = spack.install_test.get_all_test_suites() + if not test_suites: + tty.msg("No test suites with status to report") + + for test_suite in test_suites: + # TODO: Make this handle capability tests too + # TODO: Make this handle tests running in another process + tty.msg("Test suite %s completed" % test_suite.name) + + +def _report_suite_results(test_suite, args, constraints): + """Report the relevant test suite results.""" + + # TODO: Make this handle capability tests too + # The results file may turn out to be a placeholder for future work + + if constraints: + # TBD: Should I be refactoring or re-using ConstraintAction? + qspecs = spack.cmd.parse_specs(constraints) + specs = {} + for spec in qspecs: + for s in spack.store.db.query(spec, installed=True): + specs[s.dag_hash()] = s + specs = sorted(specs.values()) + test_specs = dict((test_suite.test_pkg_id(s), s) for s in + test_suite.specs if s in specs) + else: + test_specs = dict((test_suite.test_pkg_id(s), s) for s in + test_suite.specs) + + if not test_specs: + return + + if os.path.exists(test_suite.results_file): + results_desc = 'Failing results' if args.failed else 'Results' + matching = ", spec matching '{0}'".format(' '.join(constraints)) \ + if constraints else '' + tty.msg("{0} for test suite '{1}'{2}:" + .format(results_desc, test_suite.name, matching)) + + results = {} + with open(test_suite.results_file, 'r') as f: + for line in f: + pkg_id, status = line.split() + results[pkg_id] = status + + for pkg_id in test_specs: + if pkg_id in results: + status = results[pkg_id] + if args.failed and status != 'FAILED': + continue + + msg = " {0} {1}".format(pkg_id, status) + if args.logs: + spec = test_specs[pkg_id] + log_file = test_suite.log_file_for_spec(spec) + if os.path.isfile(log_file): + with open(log_file, 'r') as f: + msg += '\n{0}'.format(''.join(f.readlines())) + tty.msg(msg) + else: + msg = "Test %s has no results.\n" % test_suite.name + msg += " Check if it is running with " + msg += "`spack test status %s`" % test_suite.name + tty.msg(msg) + + +def test_results(args): + """Get the results from Spack test suite(s) (default all).""" + if args.names: + try: + sep_index = args.names.index('--') + names = args.names[:sep_index] + constraints = args.names[sep_index + 1:] + except ValueError: + names = args.names + constraints = None + else: + names, constraints = None, None + + if names: + test_suites = [spack.install_test.get_test_suite(name) for name + in names] + if not test_suites: + tty.msg('No test suite(s) found in test stage: {0}' + .format(', '.join(names))) + else: + test_suites = spack.install_test.get_all_test_suites() + if not test_suites: + tty.msg("No test suites with results to report") + + for test_suite in test_suites: + _report_suite_results(test_suite, args, constraints) + + +def test_remove(args): + """Remove results from Spack test suite(s) (default all). + + If no test suite is listed, remove results for all suites. + + Removed tests can no longer be accessed for results or status, and will not + appear in `spack test list` results.""" + if args.names: + test_suites = [] + for name in args.names: + test_suite = spack.install_test.get_test_suite(name) + if test_suite: + test_suites.append(test_suite) + else: + tty.msg("No test suite %s found in test stage" % name) + else: + test_suites = spack.install_test.get_all_test_suites() + + if not test_suites: + tty.msg("No test suites to remove") + return + + if not args.yes_to_all: + msg = 'The following test suites will be removed:\n\n' + msg += ' ' + ' '.join(test.name for test in test_suites) + '\n' + tty.msg(msg) + answer = tty.get_yes_or_no('Do you want to proceed?', default=False) + if not answer: + tty.msg('Aborting removal of test suites') return - return pytest.main(pytest_args) + for test_suite in test_suites: + shutil.rmtree(test_suite.stage) + + +def test(parser, args): + globals()['test_%s' % args.test_command](args) diff --git a/lib/spack/spack/cmd/test_env.py b/lib/spack/spack/cmd/test_env.py new file mode 100644 index 0000000000..61e85046c1 --- /dev/null +++ b/lib/spack/spack/cmd/test_env.py @@ -0,0 +1,16 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) +import spack.cmd.common.env_utility as env_utility + +description = "run a command in a spec's test environment, " \ + "or dump its environment to screen or file" +section = "administration" +level = "long" + +setup_parser = env_utility.setup_parser + + +def test_env(parser, args): + env_utility.emulate_env_utility('test-env', 'test', args) diff --git a/lib/spack/spack/cmd/unit_test.py b/lib/spack/spack/cmd/unit_test.py new file mode 100644 index 0000000000..509211de04 --- /dev/null +++ b/lib/spack/spack/cmd/unit_test.py @@ -0,0 +1,169 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +from __future__ import print_function +from __future__ import division + +import collections +import sys +import re +import argparse +import pytest +from six import StringIO + +import llnl.util.tty.color as color +from llnl.util.filesystem import working_dir +from llnl.util.tty.colify import colify + +import spack.paths + +description = "run spack's unit tests (wrapper around pytest)" +section = "developer" +level = "long" + + +def setup_parser(subparser): + subparser.add_argument( + '-H', '--pytest-help', action='store_true', default=False, + help="show full pytest help, with advanced options") + + # extra spack arguments to list tests + list_group = subparser.add_argument_group("listing tests") + list_mutex = list_group.add_mutually_exclusive_group() + list_mutex.add_argument( + '-l', '--list', action='store_const', default=None, + dest='list', const='list', help="list test filenames") + list_mutex.add_argument( + '-L', '--list-long', action='store_const', default=None, + dest='list', const='long', help="list all test functions") + list_mutex.add_argument( + '-N', '--list-names', action='store_const', default=None, + dest='list', const='names', help="list full names of all tests") + + # use tests for extension + subparser.add_argument( + '--extension', default=None, + help="run test for a given spack extension") + + # spell out some common pytest arguments, so they'll show up in help + pytest_group = subparser.add_argument_group( + "common pytest arguments (spack unit-test --pytest-help for more)") + pytest_group.add_argument( + "-s", action='append_const', dest='parsed_args', const='-s', + help="print output while tests run (disable capture)") + pytest_group.add_argument( + "-k", action='store', metavar="EXPRESSION", dest='expression', + help="filter tests by keyword (can also use w/list options)") + pytest_group.add_argument( + "--showlocals", action='append_const', dest='parsed_args', + const='--showlocals', help="show local variable values in tracebacks") + + # remainder is just passed to pytest + subparser.add_argument( + 'pytest_args', nargs=argparse.REMAINDER, help="arguments for pytest") + + +def do_list(args, extra_args): + """Print a lists of tests than what pytest offers.""" + # Run test collection and get the tree out. + old_output = sys.stdout + try: + sys.stdout = output = StringIO() + pytest.main(['--collect-only'] + extra_args) + finally: + sys.stdout = old_output + + lines = output.getvalue().split('\n') + tests = collections.defaultdict(lambda: set()) + prefix = [] + + # collect tests into sections + for line in lines: + match = re.match(r"(\s*)<([^ ]*) '([^']*)'", line) + if not match: + continue + indent, nodetype, name = match.groups() + + # strip parametrized tests + if "[" in name: + name = name[:name.index("[")] + + depth = len(indent) // 2 + + if nodetype.endswith("Function"): + key = tuple(prefix) + tests[key].add(name) + else: + prefix = prefix[:depth] + prefix.append(name) + + def colorize(c, prefix): + if isinstance(prefix, tuple): + return "::".join( + color.colorize("@%s{%s}" % (c, p)) + for p in prefix if p != "()" + ) + return color.colorize("@%s{%s}" % (c, prefix)) + + if args.list == "list": + files = set(prefix[0] for prefix in tests) + color_files = [colorize("B", file) for file in sorted(files)] + colify(color_files) + + elif args.list == "long": + for prefix, functions in sorted(tests.items()): + path = colorize("*B", prefix) + "::" + functions = [colorize("c", f) for f in sorted(functions)] + color.cprint(path) + colify(functions, indent=4) + print() + + else: # args.list == "names" + all_functions = [ + colorize("*B", prefix) + "::" + colorize("c", f) + for prefix, functions in sorted(tests.items()) + for f in sorted(functions) + ] + colify(all_functions) + + +def add_back_pytest_args(args, unknown_args): + """Add parsed pytest args, unknown args, and remainder together. + + We add some basic pytest arguments to the Spack parser to ensure that + they show up in the short help, so we have to reassemble things here. + """ + result = args.parsed_args or [] + result += unknown_args or [] + result += args.pytest_args or [] + if args.expression: + result += ["-k", args.expression] + return result + + +def unit_test(parser, args, unknown_args): + if args.pytest_help: + # make the pytest.main help output more accurate + sys.argv[0] = 'spack test' + return pytest.main(['-h']) + + # add back any parsed pytest args we need to pass to pytest + pytest_args = add_back_pytest_args(args, unknown_args) + + # The default is to test the core of Spack. If the option `--extension` + # has been used, then test that extension. + pytest_root = spack.paths.spack_root + if args.extension: + target = args.extension + extensions = spack.config.get('config:extensions') + pytest_root = spack.extensions.path_for_extension(target, *extensions) + + # pytest.ini lives in the root of the spack repository. + with working_dir(pytest_root): + if args.list: + do_list(args, pytest_args) + return + + return pytest.main(pytest_args) diff --git a/lib/spack/spack/directives.py b/lib/spack/spack/directives.py index 7b2084d229..41276b5b48 100644 --- a/lib/spack/spack/directives.py +++ b/lib/spack/spack/directives.py @@ -299,8 +299,18 @@ def _depends_on(pkg, spec, when=None, type=default_deptype, patches=None): # call this patches here for clarity -- we want patch to be a list, # but the caller doesn't have to make it one. - if patches and dep_spec.virtual: - raise DependencyPatchError("Cannot patch a virtual dependency.") + + # Note: we cannot check whether a package is virtual in a directive + # because directives are run as part of class instantiation, and specs + # instantiate the package class as part of the `virtual` check. + # To be technical, specs only instantiate the package class as part of the + # virtual check if the provider index hasn't been created yet. + # TODO: There could be a cache warming strategy that would allow us to + # ensure `Spec.virtual` is a valid thing to call in a directive. + # For now, we comment out the following check to allow for virtual packages + # with package files. + # if patches and dep_spec.virtual: + # raise DependencyPatchError("Cannot patch a virtual dependency.") # ensure patches is a list if patches is None: diff --git a/lib/spack/spack/install_test.py b/lib/spack/spack/install_test.py new file mode 100644 index 0000000000..6c2c095a2e --- /dev/null +++ b/lib/spack/spack/install_test.py @@ -0,0 +1,266 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) +import base64 +import hashlib +import os +import re +import shutil +import sys +import tty + +import llnl.util.filesystem as fs + +from spack.spec import Spec + +import spack.error +import spack.util.prefix +import spack.util.spack_json as sjson + + +test_suite_filename = 'test_suite.lock' +results_filename = 'results.txt' + + +def get_escaped_text_output(filename): + """Retrieve and escape the expected text output from the file + + Args: + filename (str): path to the file + + Returns: + (list of str): escaped text lines read from the file + """ + with open(filename, 'r') as f: + # Ensure special characters are escaped as needed + expected = f.read() + + # Split the lines to make it easier to debug failures when there is + # a lot of output + return [re.escape(ln) for ln in expected.split('\n')] + + +def get_test_stage_dir(): + return spack.util.path.canonicalize_path( + spack.config.get('config:test_stage', '~/.spack/test')) + + +def get_all_test_suites(): + stage_root = get_test_stage_dir() + if not os.path.isdir(stage_root): + return [] + + def valid_stage(d): + dirpath = os.path.join(stage_root, d) + return (os.path.isdir(dirpath) and + test_suite_filename in os.listdir(dirpath)) + + candidates = [ + os.path.join(stage_root, d, test_suite_filename) + for d in os.listdir(stage_root) + if valid_stage(d) + ] + + test_suites = [TestSuite.from_file(c) for c in candidates] + return test_suites + + +def get_test_suite(name): + assert name, "Cannot search for empty test name or 'None'" + test_suites = get_all_test_suites() + names = [ts for ts in test_suites + if ts.name == name] + assert len(names) < 2, "alias shadows test suite hash" + + if not names: + return None + return names[0] + + +class TestSuite(object): + def __init__(self, specs, alias=None): + # copy so that different test suites have different package objects + # even if they contain the same spec + self.specs = [spec.copy() for spec in specs] + self.current_test_spec = None # spec currently tested, can be virtual + self.current_base_spec = None # spec currently running do_test + + self.alias = alias + self._hash = None + + @property + def name(self): + return self.alias if self.alias else self.content_hash + + @property + def content_hash(self): + if not self._hash: + json_text = sjson.dump(self.to_dict()) + sha = hashlib.sha1(json_text.encode('utf-8')) + b32_hash = base64.b32encode(sha.digest()).lower() + if sys.version_info[0] >= 3: + b32_hash = b32_hash.decode('utf-8') + self._hash = b32_hash + return self._hash + + def __call__(self, *args, **kwargs): + self.write_reproducibility_data() + + remove_directory = kwargs.get('remove_directory', True) + dirty = kwargs.get('dirty', False) + fail_first = kwargs.get('fail_first', False) + + for spec in self.specs: + try: + msg = "A package object cannot run in two test suites at once" + assert not spec.package.test_suite, msg + + # Set up the test suite to know which test is running + spec.package.test_suite = self + self.current_base_spec = spec + self.current_test_spec = spec + + # setup per-test directory in the stage dir + test_dir = self.test_dir_for_spec(spec) + if os.path.exists(test_dir): + shutil.rmtree(test_dir) + fs.mkdirp(test_dir) + + # run the package tests + spec.package.do_test( + dirty=dirty + ) + + # Clean up on success and log passed test + if remove_directory: + shutil.rmtree(test_dir) + self.write_test_result(spec, 'PASSED') + except BaseException as exc: + if isinstance(exc, SyntaxError): + # Create the test log file and report the error. + self.ensure_stage() + msg = 'Testing package {0}\n{1}'\ + .format(self.test_pkg_id(spec), str(exc)) + _add_msg_to_file(self.log_file_for_spec(spec), msg) + + self.write_test_result(spec, 'FAILED') + if fail_first: + break + finally: + spec.package.test_suite = None + self.current_test_spec = None + self.current_base_spec = None + + def ensure_stage(self): + if not os.path.exists(self.stage): + fs.mkdirp(self.stage) + + @property + def stage(self): + return spack.util.prefix.Prefix( + os.path.join(get_test_stage_dir(), self.content_hash)) + + @property + def results_file(self): + return self.stage.join(results_filename) + + @classmethod + def test_pkg_id(cls, spec): + """Build the standard install test package identifier + + Args: + spec (Spec): instance of the spec under test + + Returns: + (str): the install test package identifier + """ + return spec.format('{name}-{version}-{hash:7}') + + @classmethod + def test_log_name(cls, spec): + return '%s-test-out.txt' % cls.test_pkg_id(spec) + + def log_file_for_spec(self, spec): + return self.stage.join(self.test_log_name(spec)) + + def test_dir_for_spec(self, spec): + return self.stage.join(self.test_pkg_id(spec)) + + @property + def current_test_data_dir(self): + assert self.current_test_spec and self.current_base_spec + test_spec = self.current_test_spec + base_spec = self.current_base_spec + return self.test_dir_for_spec(base_spec).data.join(test_spec.name) + + def add_failure(self, exc, msg): + current_hash = self.current_base_spec.dag_hash() + current_failures = self.failures.get(current_hash, []) + current_failures.append((exc, msg)) + self.failures[current_hash] = current_failures + + def write_test_result(self, spec, result): + msg = "{0} {1}".format(self.test_pkg_id(spec), result) + _add_msg_to_file(self.results_file, msg) + + def write_reproducibility_data(self): + for spec in self.specs: + repo_cache_path = self.stage.repo.join(spec.name) + spack.repo.path.dump_provenance(spec, repo_cache_path) + for vspec in spec.package.virtuals_provided: + repo_cache_path = self.stage.repo.join(vspec.name) + if not os.path.exists(repo_cache_path): + try: + spack.repo.path.dump_provenance(vspec, repo_cache_path) + except spack.repo.UnknownPackageError: + pass # not all virtuals have package files + + with open(self.stage.join(test_suite_filename), 'w') as f: + sjson.dump(self.to_dict(), stream=f) + + def to_dict(self): + specs = [s.to_dict() for s in self.specs] + d = {'specs': specs} + if self.alias: + d['alias'] = self.alias + return d + + @staticmethod + def from_dict(d): + specs = [Spec.from_dict(spec_dict) for spec_dict in d['specs']] + alias = d.get('alias', None) + return TestSuite(specs, alias) + + @staticmethod + def from_file(filename): + try: + with open(filename, 'r') as f: + data = sjson.load(f) + return TestSuite.from_dict(data) + except Exception as e: + tty.debug(e) + raise sjson.SpackJSONError("error parsing JSON TestSuite:", str(e)) + + +def _add_msg_to_file(filename, msg): + """Add the message to the specified file + + Args: + filename (str): path to the file + msg (str): message to be appended to the file + """ + with open(filename, 'a+') as f: + f.write('{0}\n'.format(msg)) + + +class TestFailure(spack.error.SpackError): + """Raised when package tests have failed for an installation.""" + def __init__(self, failures): + # Failures are all exceptions + msg = "%d tests failed.\n" % len(failures) + for failure, message in failures: + msg += '\n\n%s\n' % str(failure) + msg += '\n%s\n' % message + + super(TestFailure, self).__init__(msg) diff --git a/lib/spack/spack/installer.py b/lib/spack/spack/installer.py index cf7b5bfb99..dd35db0839 100644 --- a/lib/spack/spack/installer.py +++ b/lib/spack/spack/installer.py @@ -1610,12 +1610,12 @@ def build_process(pkg, kwargs): This function's return value is returned to the parent process. """ - keep_stage = kwargs.get('keep_stage', False) + fake = kwargs.get('fake', False) install_source = kwargs.get('install_source', False) + keep_stage = kwargs.get('keep_stage', False) skip_patch = kwargs.get('skip_patch', False) - verbose = kwargs.get('verbose', False) - fake = kwargs.get('fake', False) unmodified_env = kwargs.get('unmodified_env', {}) + verbose = kwargs.get('verbose', False) start_time = time.time() if not fake: @@ -1958,6 +1958,7 @@ class BuildRequest(object): def _add_default_args(self): """Ensure standard install options are set to at least the default.""" for arg, default in [('cache_only', False), + ('context', 'build'), # installs *always* build ('dirty', False), ('fail_fast', False), ('fake', False), diff --git a/lib/spack/spack/modules/lmod.py b/lib/spack/spack/modules/lmod.py index 018edb35ad..80f6933063 100644 --- a/lib/spack/spack/modules/lmod.py +++ b/lib/spack/spack/modules/lmod.py @@ -12,6 +12,7 @@ import collections import spack.config import spack.compilers import spack.spec +import spack.repo import spack.error import spack.tengine as tengine @@ -125,7 +126,9 @@ class LmodConfiguration(BaseConfiguration): # Check if all the tokens in the hierarchy are virtual specs. # If not warn the user and raise an error. - not_virtual = [t for t in tokens if not spack.spec.Spec.is_virtual(t)] + not_virtual = [t for t in tokens + if t != 'compiler' and + not spack.repo.path.is_virtual(t)] if not_virtual: msg = "Non-virtual specs in 'hierarchy' list for lmod: {0}\n" msg += "Please check the 'modules.yaml' configuration files" diff --git a/lib/spack/spack/package.py b/lib/spack/spack/package.py index b8dca6e55a..de394e2d45 100644 --- a/lib/spack/spack/package.py +++ b/lib/spack/spack/package.py @@ -24,10 +24,12 @@ import sys import textwrap import time import traceback - import six +import types +import llnl.util.filesystem as fsys import llnl.util.tty as tty + import spack.compilers import spack.config import spack.dependency @@ -45,15 +47,14 @@ import spack.store import spack.url import spack.util.environment import spack.util.web -from llnl.util.filesystem import mkdirp, touch, working_dir from llnl.util.lang import memoized from llnl.util.link_tree import LinkTree from ordereddict_backport import OrderedDict -from six import StringIO -from six import string_types -from six import with_metaclass from spack.filesystem_view import YamlFilesystemView from spack.installer import PackageInstaller, InstallError +from spack.install_test import TestFailure, TestSuite +from spack.util.executable import which, ProcessError +from spack.util.prefix import Prefix from spack.stage import stage_prefix, Stage, ResourceStage, StageComposite from spack.util.package_hash import package_hash from spack.version import Version @@ -452,7 +453,21 @@ class PackageViewMixin(object): view.remove_file(src, dst) -class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): +def test_log_pathname(test_stage, spec): + """Build the pathname of the test log file + + Args: + test_stage (str): path to the test stage directory + spec (Spec): instance of the spec under test + + Returns: + (str): the pathname of the test log file + """ + return os.path.join(test_stage, + 'test-{0}-out.txt'.format(TestSuite.test_pkg_id(spec))) + + +class PackageBase(six.with_metaclass(PackageMeta, PackageViewMixin, object)): """This is the superclass for all spack packages. ***The Package class*** @@ -542,6 +557,10 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): #: are executed or 'None' if there are no such test functions. build_time_test_callbacks = None + #: By default, packages are not virtual + #: Virtual packages override this attribute + virtual = False + #: Most Spack packages are used to install source or binary code while #: those that do not can be used to install a set of other Spack packages. has_code = True @@ -633,6 +652,18 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): metadata_attrs = ['homepage', 'url', 'urls', 'list_url', 'extendable', 'parallel', 'make_jobs'] + #: Boolean. If set to ``True``, the smoke/install test requires a compiler. + #: This is currently used by smoke tests to ensure a compiler is available + #: to build a custom test code. + test_requires_compiler = False + + #: List of test failures encountered during a smoke/install test run. + test_failures = None + + #: TestSuite instance used to manage smoke/install tests for one or more + #: specs. + test_suite = None + def __init__(self, spec): # this determines how the package should be built. self.spec = spec @@ -1002,19 +1033,22 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): return os.path.join(self.stage.path, _spack_build_envfile) @property + def metadata_dir(self): + """Return the install metadata directory.""" + return spack.store.layout.metadata_path(self.spec) + + @property def install_env_path(self): """ Return the build environment file path on successful installation. """ - install_path = spack.store.layout.metadata_path(self.spec) - # Backward compatibility: Return the name of an existing log path; # otherwise, return the current install env path name. - old_filename = os.path.join(install_path, 'build.env') + old_filename = os.path.join(self.metadata_dir, 'build.env') if os.path.exists(old_filename): return old_filename else: - return os.path.join(install_path, _spack_build_envfile) + return os.path.join(self.metadata_dir, _spack_build_envfile) @property def log_path(self): @@ -1031,16 +1065,14 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): @property def install_log_path(self): """Return the build log file path on successful installation.""" - install_path = spack.store.layout.metadata_path(self.spec) - # Backward compatibility: Return the name of an existing install log. for filename in ['build.out', 'build.txt']: - old_log = os.path.join(install_path, filename) + old_log = os.path.join(self.metadata_dir, filename) if os.path.exists(old_log): return old_log # Otherwise, return the current install log path name. - return os.path.join(install_path, _spack_build_logfile) + return os.path.join(self.metadata_dir, _spack_build_logfile) @property def configure_args_path(self): @@ -1050,9 +1082,12 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): @property def install_configure_args_path(self): """Return the configure args file path on successful installation.""" - install_path = spack.store.layout.metadata_path(self.spec) + return os.path.join(self.metadata_dir, _spack_configure_argsfile) - return os.path.join(install_path, _spack_configure_argsfile) + @property + def install_test_root(self): + """Return the install test root directory.""" + return os.path.join(self.metadata_dir, 'test') def _make_fetcher(self): # Construct a composite fetcher that always contains at least @@ -1322,7 +1357,7 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): raise FetchError("Archive was empty for %s" % self.name) else: # Support for post-install hooks requires a stage.source_path - mkdirp(self.stage.source_path) + fsys.mkdirp(self.stage.source_path) def do_patch(self): """Applies patches if they haven't been applied already.""" @@ -1368,7 +1403,7 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): patched = False for patch in patches: try: - with working_dir(self.stage.source_path): + with fsys.working_dir(self.stage.source_path): patch.apply(self.stage) tty.debug('Applied patch {0}'.format(patch.path_or_url)) patched = True @@ -1377,12 +1412,12 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): # Touch bad file if anything goes wrong. tty.msg('Patch %s failed.' % patch.path_or_url) - touch(bad_file) + fsys.touch(bad_file) raise if has_patch_fun: try: - with working_dir(self.stage.source_path): + with fsys.working_dir(self.stage.source_path): self.patch() tty.debug('Ran patch() for {0}'.format(self.name)) patched = True @@ -1400,7 +1435,7 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): # Touch bad file if anything goes wrong. tty.msg('patch() function failed for {0}'.format(self.name)) - touch(bad_file) + fsys.touch(bad_file) raise # Get rid of any old failed file -- patches have either succeeded @@ -1411,9 +1446,9 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): # touch good or no patches file so that we skip next time. if patched: - touch(good_file) + fsys.touch(good_file) else: - touch(no_patches_file) + fsys.touch(no_patches_file) @classmethod def all_patches(cls): @@ -1657,6 +1692,175 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): builder = PackageInstaller([(self, kwargs)]) builder.install() + def cache_extra_test_sources(self, srcs): + """Copy relative source paths to the corresponding install test subdir + + This method is intended as an optional install test setup helper for + grabbing source files/directories during the installation process and + copying them to the installation test subdirectory for subsequent use + during install testing. + + Args: + srcs (str or list of str): relative path for files and or + subdirectories located in the staged source path that are to + be copied to the corresponding location(s) under the install + testing directory. + """ + paths = [srcs] if isinstance(srcs, six.string_types) else srcs + + for path in paths: + src_path = os.path.join(self.stage.source_path, path) + dest_path = os.path.join(self.install_test_root, path) + if os.path.isdir(src_path): + fsys.install_tree(src_path, dest_path) + else: + fsys.mkdirp(os.path.dirname(dest_path)) + fsys.copy(src_path, dest_path) + + def do_test(self, dirty=False): + if self.test_requires_compiler: + compilers = spack.compilers.compilers_for_spec( + self.spec.compiler, arch_spec=self.spec.architecture) + if not compilers: + tty.error('Skipping tests for package %s\n' % + self.spec.format('{name}-{version}-{hash:7}') + + 'Package test requires missing compiler %s' % + self.spec.compiler) + return + + # Clear test failures + self.test_failures = [] + self.test_log_file = self.test_suite.log_file_for_spec(self.spec) + fsys.touch(self.test_log_file) # Otherwise log_parse complains + + kwargs = {'dirty': dirty, 'fake': False, 'context': 'test'} + spack.build_environment.start_build_process(self, test_process, kwargs) + + def test(self): + pass + + def run_test(self, exe, options=[], expected=[], status=0, + installed=False, purpose='', skip_missing=False, + work_dir=None): + """Run the test and confirm the expected results are obtained + + Log any failures and continue, they will be re-raised later + + Args: + exe (str): the name of the executable + options (str or list of str): list of options to pass to the runner + expected (str or list of str): list of expected output strings. + Each string is a regex expected to match part of the output. + status (int or list of int): possible passing status values + with 0 meaning the test is expected to succeed + installed (bool): if ``True``, the executable must be in the + install prefix + purpose (str): message to display before running test + skip_missing (bool): skip the test if the executable is not + in the install prefix bin directory or the provided work_dir + work_dir (str or None): path to the smoke test directory + """ + wdir = '.' if work_dir is None else work_dir + with fsys.working_dir(wdir): + try: + runner = which(exe) + if runner is None and skip_missing: + return + assert runner is not None, \ + "Failed to find executable '{0}'".format(exe) + + self._run_test_helper( + runner, options, expected, status, installed, purpose) + print("PASSED") + return True + except BaseException as e: + # print a summary of the error to the log file + # so that cdash and junit reporters know about it + exc_type, _, tb = sys.exc_info() + print('FAILED: {0}'.format(e)) + import traceback + # remove the current call frame to exclude the extract_stack + # call from the error + stack = traceback.extract_stack()[:-1] + + # Package files have a line added at import time, so we re-read + # the file to make line numbers match. We have to subtract two + # from the line number because the original line number is + # inflated once by the import statement and the lines are + # displaced one by the import statement. + for i, entry in enumerate(stack): + filename, lineno, function, text = entry + if spack.repo.is_package_file(filename): + with open(filename, 'r') as f: + lines = f.readlines() + new_lineno = lineno - 2 + text = lines[new_lineno] + stack[i] = (filename, new_lineno, function, text) + + # Format the stack to print and print it + out = traceback.format_list(stack) + for line in out: + print(line.rstrip('\n')) + + if exc_type is spack.util.executable.ProcessError: + out = six.StringIO() + spack.build_environment.write_log_summary( + out, 'test', self.test_log_file, last=1) + m = out.getvalue() + else: + # We're below the package context, so get context from + # stack instead of from traceback. + # The traceback is truncated here, so we can't use it to + # traverse the stack. + m = '\n'.join( + spack.build_environment.get_package_context(tb) + ) + + exc = e # e is deleted after this block + + # If we fail fast, raise another error + if spack.config.get('config:fail_fast', False): + raise TestFailure([(exc, m)]) + else: + self.test_failures.append((exc, m)) + return False + + def _run_test_helper(self, runner, options, expected, status, installed, + purpose): + status = [status] if isinstance(status, six.integer_types) else status + expected = [expected] if isinstance(expected, six.string_types) else \ + expected + options = [options] if isinstance(options, six.string_types) else \ + options + + if purpose: + tty.msg(purpose) + else: + tty.debug('test: {0}: expect command status in {1}' + .format(runner.name, status)) + + if installed: + msg = "Executable '{0}' expected in prefix".format(runner.name) + msg += ", found in {0} instead".format(runner.path) + assert runner.path.startswith(self.spec.prefix), msg + + try: + output = runner(*options, output=str.split, error=str.split) + + assert 0 in status, \ + 'Expected {0} execution to fail'.format(runner.name) + except ProcessError as err: + output = str(err) + match = re.search(r'exited with status ([0-9]+)', output) + if not (match and int(match.group(1)) in status): + raise + + for check in expected: + cmd = ' '.join([runner.name] + options) + msg = "Expected '{0}' to match output of `{1}`".format(check, cmd) + msg += '\n\nOutput: {0}'.format(output) + assert re.search(check, output), msg + def unit_test_check(self): """Hook for unit tests to assert things about package internals. @@ -1678,7 +1882,7 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): """This function checks whether install succeeded.""" def check_paths(path_list, filetype, predicate): - if isinstance(path_list, string_types): + if isinstance(path_list, six.string_types): path_list = [path_list] for path in path_list: @@ -2031,7 +2235,7 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): # copy spec metadata to "deprecated" dir of deprecator depr_yaml = spack.store.layout.deprecated_file_path(spec, deprecator) - fs.mkdirp(os.path.dirname(depr_yaml)) + fsys.mkdirp(os.path.dirname(depr_yaml)) shutil.copy2(self_yaml, depr_yaml) # Any specs deprecated in favor of this spec are re-deprecated in @@ -2210,7 +2414,7 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): doc = re.sub(r'\s+', ' ', self.__doc__) lines = textwrap.wrap(doc, 72) - results = StringIO() + results = six.StringIO() for line in lines: results.write((" " * indent) + line + "\n") return results.getvalue() @@ -2313,6 +2517,71 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): tty.warn(msg.format(name)) +def test_process(pkg, kwargs): + with tty.log.log_output(pkg.test_log_file) as logger: + with logger.force_echo(): + tty.msg('Testing package {0}' + .format(pkg.test_suite.test_pkg_id(pkg.spec))) + + # use debug print levels for log file to record commands + old_debug = tty.is_debug() + tty.set_debug(True) + + # run test methods from the package and all virtuals it + # provides virtuals have to be deduped by name + v_names = list(set([vspec.name + for vspec in pkg.virtuals_provided])) + + # hack for compilers that are not dependencies (yet) + # TODO: this all eventually goes away + c_names = ('gcc', 'intel', 'intel-parallel-studio', 'pgi') + if pkg.name in c_names: + v_names.extend(['c', 'cxx', 'fortran']) + if pkg.spec.satisfies('llvm+clang'): + v_names.extend(['c', 'cxx']) + + test_specs = [pkg.spec] + [spack.spec.Spec(v_name) + for v_name in sorted(v_names)] + + try: + with fsys.working_dir( + pkg.test_suite.test_dir_for_spec(pkg.spec)): + for spec in test_specs: + pkg.test_suite.current_test_spec = spec + # Fail gracefully if a virtual has no package/tests + try: + spec_pkg = spec.package + except spack.repo.UnknownPackageError: + continue + + # copy test data into test data dir + data_source = Prefix(spec_pkg.package_dir).test + data_dir = pkg.test_suite.current_test_data_dir + if (os.path.isdir(data_source) and + not os.path.exists(data_dir)): + # We assume data dir is used read-only + # maybe enforce this later + shutil.copytree(data_source, data_dir) + + # grab the function for each method so we can call + # it with the package + test_fn = spec_pkg.__class__.test + if not isinstance(test_fn, types.FunctionType): + test_fn = test_fn.__func__ + + # Run the tests + test_fn(pkg) + + # If fail-fast was on, we error out above + # If we collect errors, raise them in batch here + if pkg.test_failures: + raise TestFailure(pkg.test_failures) + + finally: + # reset debug level + tty.set_debug(old_debug) + + inject_flags = PackageBase.inject_flags env_flags = PackageBase.env_flags build_system_flags = PackageBase.build_system_flags diff --git a/lib/spack/spack/paths.py b/lib/spack/spack/paths.py index cb2240359c..9c803cba7e 100644 --- a/lib/spack/spack/paths.py +++ b/lib/spack/spack/paths.py @@ -12,7 +12,6 @@ dependencies. import os from llnl.util.filesystem import ancestor - #: This file lives in $prefix/lib/spack/spack/__file__ prefix = ancestor(__file__, 4) @@ -42,6 +41,7 @@ test_path = os.path.join(module_path, "test") hooks_path = os.path.join(module_path, "hooks") var_path = os.path.join(prefix, "var", "spack") repos_path = os.path.join(var_path, "repos") +tests_path = os.path.join(var_path, "tests") share_path = os.path.join(prefix, "share", "spack") # Paths to built-in Spack repositories. diff --git a/lib/spack/spack/pkgkit.py b/lib/spack/spack/pkgkit.py index ac0a7eee0d..4f25d41dfb 100644 --- a/lib/spack/spack/pkgkit.py +++ b/lib/spack/spack/pkgkit.py @@ -59,6 +59,7 @@ from spack.package import \ from spack.installer import \ ExternalPackageError, InstallError, InstallLockError, UpstreamPackageError +from spack.install_test import get_escaped_text_output from spack.variant import any_combination_of, auto_or_any_combination_of from spack.variant import disjoint_sets diff --git a/lib/spack/spack/repo.py b/lib/spack/spack/repo.py index 5e1457c165..4af7b382f0 100644 --- a/lib/spack/spack/repo.py +++ b/lib/spack/spack/repo.py @@ -91,6 +91,19 @@ def autospec(function): return converter +def is_package_file(filename): + """Determine whether we are in a package file from a repo.""" + # Package files are named `package.py` and are not in lib/spack/spack + # We have to remove the file extension because it can be .py and can be + # .pyc depending on context, and can differ between the files + import spack.package # break cycle + filename_noext = os.path.splitext(filename)[0] + packagebase_filename_noext = os.path.splitext( + inspect.getfile(spack.package.PackageBase))[0] + return (filename_noext != packagebase_filename_noext and + os.path.basename(filename_noext) == 'package') + + class SpackNamespace(types.ModuleType): """ Allow lazy loading of modules.""" @@ -131,6 +144,11 @@ class FastPackageChecker(Mapping): #: Reference to the appropriate entry in the global cache self._packages_to_stats = self._paths_cache[packages_path] + def invalidate(self): + """Regenerate cache for this checker.""" + self._paths_cache[self.packages_path] = self._create_new_cache() + self._packages_to_stats = self._paths_cache[self.packages_path] + def _create_new_cache(self): """Create a new cache for packages in a repo. @@ -308,6 +326,9 @@ class ProviderIndexer(Indexer): self.index = spack.provider_index.ProviderIndex.from_json(stream) def update(self, pkg_fullname): + name = pkg_fullname.split('.')[-1] + if spack.repo.path.is_virtual(name, use_index=False): + return self.index.remove_provider(pkg_fullname) self.index.update(pkg_fullname) @@ -517,12 +538,12 @@ class RepoPath(object): """Get the first repo in precedence order.""" return self.repos[0] if self.repos else None - def all_package_names(self): + def all_package_names(self, include_virtuals=False): """Return all unique package names in all repositories.""" if self._all_package_names is None: all_pkgs = set() for repo in self.repos: - for name in repo.all_package_names(): + for name in repo.all_package_names(include_virtuals): all_pkgs.add(name) self._all_package_names = sorted(all_pkgs, key=lambda n: n.lower()) return self._all_package_names @@ -679,12 +700,20 @@ class RepoPath(object): """ return any(repo.exists(pkg_name) for repo in self.repos) - def is_virtual(self, pkg_name): - """True if the package with this name is virtual, False otherwise.""" - if not isinstance(pkg_name, str): + def is_virtual(self, pkg_name, use_index=True): + """True if the package with this name is virtual, False otherwise. + + Set `use_index` False when calling from a code block that could + be run during the computation of the provider index.""" + have_name = pkg_name is not None + if have_name and not isinstance(pkg_name, str): raise ValueError( "is_virtual(): expected package name, got %s" % type(pkg_name)) - return pkg_name in self.provider_index + if use_index: + return have_name and pkg_name in self.provider_index + else: + return have_name and (not self.exists(pkg_name) or + self.get_pkg_class(pkg_name).virtual) def __contains__(self, pkg_name): return self.exists(pkg_name) @@ -913,10 +942,6 @@ class Repo(object): This dumps the package file and any associated patch files. Raises UnknownPackageError if not found. """ - # Some preliminary checks. - if spec.virtual: - raise UnknownPackageError(spec.name) - if spec.namespace and spec.namespace != self.namespace: raise UnknownPackageError( "Repository %s does not contain package %s." @@ -999,9 +1024,12 @@ class Repo(object): self._fast_package_checker = FastPackageChecker(self.packages_path) return self._fast_package_checker - def all_package_names(self): + def all_package_names(self, include_virtuals=False): """Returns a sorted list of all package names in the Repo.""" - return sorted(self._pkg_checker.keys()) + names = sorted(self._pkg_checker.keys()) + if include_virtuals: + return names + return [x for x in names if not self.is_virtual(x)] def packages_with_tags(self, *tags): v = set(self.all_package_names()) @@ -1040,7 +1068,7 @@ class Repo(object): def is_virtual(self, pkg_name): """True if the package with this name is virtual, False otherwise.""" - return self.provider_index.contains(pkg_name) + return pkg_name in self.provider_index def _get_pkg_module(self, pkg_name): """Create a module for a particular package. @@ -1074,7 +1102,8 @@ class Repo(object): # manually construct the error message in order to give the # user the correct package.py where the syntax error is located raise SyntaxError('invalid syntax in {0:}, line {1:}' - ''.format(file_path, e.lineno)) + .format(file_path, e.lineno)) + module.__package__ = self.full_namespace module.__loader__ = self self._modules[pkg_name] = module @@ -1205,9 +1234,9 @@ def get(spec): return path.get(spec) -def all_package_names(): +def all_package_names(include_virtuals=False): """Convenience wrapper around ``spack.repo.all_package_names()``.""" - return path.all_package_names() + return path.all_package_names(include_virtuals) def set_path(repo): diff --git a/lib/spack/spack/report.py b/lib/spack/spack/report.py index 4e5bc9c993..ebae2d0adc 100644 --- a/lib/spack/spack/report.py +++ b/lib/spack/spack/report.py @@ -9,11 +9,13 @@ import collections import functools import time import traceback +import os import llnl.util.lang import spack.build_environment import spack.fetch_strategy import spack.package +from spack.install_test import TestSuite from spack.reporter import Reporter from spack.reporters.cdash import CDash from spack.reporters.junit import JUnit @@ -33,12 +35,16 @@ __all__ = [ ] -def fetch_package_log(pkg): +def fetch_log(pkg, do_fn, dir): + log_files = { + '_install_task': pkg.build_log_path, + 'do_test': os.path.join(dir, TestSuite.test_log_name(pkg.spec)), + } try: - with codecs.open(pkg.build_log_path, 'r', 'utf-8') as f: + with codecs.open(log_files[do_fn.__name__], 'r', 'utf-8') as f: return ''.join(f.readlines()) except Exception: - return 'Cannot open build log for {0}'.format( + return 'Cannot open log for {0}'.format( pkg.spec.cshort_spec ) @@ -58,15 +64,20 @@ class InfoCollector(object): specs (list of Spec): specs whose install information will be recorded """ - #: Backup of PackageInstaller._install_task - _backup__install_task = spack.package.PackageInstaller._install_task - - def __init__(self, specs): - #: Specs that will be installed + def __init__(self, wrap_class, do_fn, specs, dir): + #: Class for which to wrap a function + self.wrap_class = wrap_class + #: Action to be reported on + self.do_fn = do_fn + #: Backup of PackageBase function + self._backup_do_fn = getattr(self.wrap_class, do_fn) + #: Specs that will be acted on self.input_specs = specs #: This is where we record the data that will be included #: in our report. self.specs = [] + #: Record directory for test log paths + self.dir = dir def __enter__(self): # Initialize the spec report with the data that is available upfront. @@ -98,30 +109,37 @@ class InfoCollector(object): Property('compiler', input_spec.compiler)) # Check which specs are already installed and mark them as skipped - for dep in filter(lambda x: x.package.installed, - input_spec.traverse()): - package = { - 'name': dep.name, - 'id': dep.dag_hash(), - 'elapsed_time': '0.0', - 'result': 'skipped', - 'message': 'Spec already installed' - } - spec['packages'].append(package) - - def gather_info(_install_task): - """Decorates PackageInstaller._install_task to gather useful - information on PackageBase.do_install for a CI report. + # only for install_task + if self.do_fn == '_install_task': + for dep in filter(lambda x: x.package.installed, + input_spec.traverse()): + package = { + 'name': dep.name, + 'id': dep.dag_hash(), + 'elapsed_time': '0.0', + 'result': 'skipped', + 'message': 'Spec already installed' + } + spec['packages'].append(package) + + def gather_info(do_fn): + """Decorates do_fn to gather useful information for + a CI report. It's defined here to capture the environment and build this context as the installations proceed. """ - @functools.wraps(_install_task) - def wrapper(installer, task, *args, **kwargs): - pkg = task.pkg + @functools.wraps(do_fn) + def wrapper(instance, *args, **kwargs): + if isinstance(instance, spack.package.PackageBase): + pkg = instance + elif hasattr(args[0], 'pkg'): + pkg = args[0].pkg + else: + raise Exception # We accounted before for what is already installed - installed_on_entry = pkg.installed + installed_already = pkg.installed package = { 'name': pkg.name, @@ -135,13 +153,12 @@ class InfoCollector(object): start_time = time.time() value = None try: - - value = _install_task(installer, task, *args, **kwargs) + value = do_fn(instance, *args, **kwargs) package['result'] = 'success' - package['stdout'] = fetch_package_log(pkg) + package['stdout'] = fetch_log(pkg, do_fn, self.dir) package['installed_from_binary_cache'] = \ pkg.installed_from_binary_cache - if installed_on_entry: + if do_fn.__name__ == '_install_task' and installed_already: return except spack.build_environment.InstallError as e: @@ -149,7 +166,7 @@ class InfoCollector(object): # didn't work correctly) package['result'] = 'failure' package['message'] = e.message or 'Installation failure' - package['stdout'] = fetch_package_log(pkg) + package['stdout'] = fetch_log(pkg, do_fn, self.dir) package['stdout'] += package['message'] package['exception'] = e.traceback @@ -157,7 +174,7 @@ class InfoCollector(object): # Everything else is an error (the installation # failed outside of the child process) package['result'] = 'error' - package['stdout'] = fetch_package_log(pkg) + package['stdout'] = fetch_log(pkg, do_fn, self.dir) package['message'] = str(e) or 'Unknown error' package['exception'] = traceback.format_exc() @@ -184,15 +201,14 @@ class InfoCollector(object): return wrapper - spack.package.PackageInstaller._install_task = gather_info( - spack.package.PackageInstaller._install_task - ) + setattr(self.wrap_class, self.do_fn, gather_info( + getattr(self.wrap_class, self.do_fn) + )) def __exit__(self, exc_type, exc_val, exc_tb): - # Restore the original method in PackageInstaller - spack.package.PackageInstaller._install_task = \ - InfoCollector._backup__install_task + # Restore the original method in PackageBase + setattr(self.wrap_class, self.do_fn, self._backup_do_fn) for spec in self.specs: spec['npackages'] = len(spec['packages']) @@ -225,22 +241,26 @@ class collect_info(object): # The file 'junit.xml' is written when exiting # the context - specs = [Spec('hdf5').concretized()] - with collect_info(specs, 'junit', 'junit.xml'): + s = [Spec('hdf5').concretized()] + with collect_info(PackageBase, do_install, s, 'junit', 'a.xml'): # A report will be generated for these specs... - for spec in specs: - spec.do_install() + for spec in s: + getattr(class, function)(spec) # ...but not for this one Spec('zlib').concretized().do_install() Args: + class: class on which to wrap a function + function: function to wrap format_name (str or None): one of the supported formats - args (dict): args passed to spack install + args (dict): args passed to function Raises: ValueError: when ``format_name`` is not in ``valid_formats`` """ - def __init__(self, format_name, args): + def __init__(self, cls, function, format_name, args): + self.cls = cls + self.function = function self.filename = None if args.cdash_upload_url: self.format_name = 'cdash' @@ -253,13 +273,19 @@ class collect_info(object): .format(self.format_name)) self.report_writer = report_writers[self.format_name](args) + def __call__(self, type, dir=os.getcwd()): + self.type = type + self.dir = dir + return self + def concretization_report(self, msg): self.report_writer.concretization_report(self.filename, msg) def __enter__(self): if self.format_name: - # Start the collector and patch PackageInstaller._install_task - self.collector = InfoCollector(self.specs) + # Start the collector and patch self.function on appropriate class + self.collector = InfoCollector( + self.cls, self.function, self.specs, self.dir) self.collector.__enter__() def __exit__(self, exc_type, exc_val, exc_tb): @@ -269,4 +295,5 @@ class collect_info(object): self.collector.__exit__(exc_type, exc_val, exc_tb) report_data = {'specs': self.collector.specs} - self.report_writer.build_report(self.filename, report_data) + report_fn = getattr(self.report_writer, '%s_report' % self.type) + report_fn(self.filename, report_data) diff --git a/lib/spack/spack/reporter.py b/lib/spack/spack/reporter.py index 25d4042a5e..6314054139 100644 --- a/lib/spack/spack/reporter.py +++ b/lib/spack/spack/reporter.py @@ -16,5 +16,8 @@ class Reporter(object): def build_report(self, filename, report_data): pass + def test_report(self, filename, report_data): + pass + def concretization_report(self, filename, msg): pass diff --git a/lib/spack/spack/reporters/cdash.py b/lib/spack/spack/reporters/cdash.py index 580df7866f..c1a220963e 100644 --- a/lib/spack/spack/reporters/cdash.py +++ b/lib/spack/spack/reporters/cdash.py @@ -72,8 +72,10 @@ class CDash(Reporter): tty.verbose("Using CDash auth token from environment") self.authtoken = os.environ.get('SPACK_CDASH_AUTH_TOKEN') - if args.spec: + if getattr(args, 'spec', ''): packages = args.spec + elif getattr(args, 'specs', ''): + packages = args.specs else: packages = [] for file in args.specfiles: @@ -98,7 +100,7 @@ class CDash(Reporter): self.revision = git('rev-parse', 'HEAD', output=str).strip() self.multiple_packages = False - def report_for_package(self, directory_name, package, duration): + def build_report_for_package(self, directory_name, package, duration): if 'stdout' not in package: # Skip reporting on packages that did not generate any output. return @@ -158,8 +160,8 @@ class CDash(Reporter): '\n'.join(report_data[phase]['loglines']) errors, warnings = parse_log_events(report_data[phase]['loglines']) # Cap the number of errors and warnings at 50 each. - errors = errors[0:49] - warnings = warnings[0:49] + errors = errors[:50] + warnings = warnings[:50] nerrors = len(errors) if phase == 'configure' and nerrors > 0: @@ -250,7 +252,114 @@ class CDash(Reporter): if 'time' in spec: duration = int(spec['time']) for package in spec['packages']: - self.report_for_package(directory_name, package, duration) + self.build_report_for_package( + directory_name, package, duration) + self.print_cdash_link() + + def test_report_for_package(self, directory_name, package, duration): + if 'stdout' not in package: + # Skip reporting on packages that did not generate any output. + return + + self.current_package_name = package['name'] + self.buildname = "{0} - {1}".format( + self.base_buildname, package['name']) + + report_data = self.initialize_report(directory_name) + + for phase in ('test', 'update'): + report_data[phase] = {} + report_data[phase]['loglines'] = [] + report_data[phase]['status'] = 0 + report_data[phase]['endtime'] = self.endtime + + # Track the phases we perform so we know what reports to create. + # We always report the update step because this is how we tell CDash + # what revision of Spack we are using. + phases_encountered = ['test', 'update'] + + # Generate a report for this package. + # The first line just says "Testing package name-hash" + report_data['test']['loglines'].append( + text_type("{0} output for {1}:".format( + 'test', package['name']))) + for line in package['stdout'].splitlines()[1:]: + report_data['test']['loglines'].append( + xml.sax.saxutils.escape(line)) + + self.starttime = self.endtime - duration + for phase in phases_encountered: + report_data[phase]['starttime'] = self.starttime + report_data[phase]['log'] = \ + '\n'.join(report_data[phase]['loglines']) + errors, warnings = parse_log_events(report_data[phase]['loglines']) + # Cap the number of errors and warnings at 50 each. + errors = errors[0:49] + warnings = warnings[0:49] + + if phase == 'test': + # Convert log output from ASCII to Unicode and escape for XML. + def clean_log_event(event): + event = vars(event) + event['text'] = xml.sax.saxutils.escape(event['text']) + event['pre_context'] = xml.sax.saxutils.escape( + '\n'.join(event['pre_context'])) + event['post_context'] = xml.sax.saxutils.escape( + '\n'.join(event['post_context'])) + # source_file and source_line_no are either strings or + # the tuple (None,). Distinguish between these two cases. + if event['source_file'][0] is None: + event['source_file'] = '' + event['source_line_no'] = '' + else: + event['source_file'] = xml.sax.saxutils.escape( + event['source_file']) + return event + + # Convert errors to warnings if the package reported success. + if package['result'] == 'success': + warnings = errors + warnings + errors = [] + + report_data[phase]['errors'] = [] + report_data[phase]['warnings'] = [] + for error in errors: + report_data[phase]['errors'].append(clean_log_event(error)) + for warning in warnings: + report_data[phase]['warnings'].append( + clean_log_event(warning)) + + if phase == 'update': + report_data[phase]['revision'] = self.revision + + # Write the report. + report_name = phase.capitalize() + ".xml" + report_file_name = package['name'] + "_" + report_name + phase_report = os.path.join(directory_name, report_file_name) + + with codecs.open(phase_report, 'w', 'utf-8') as f: + env = spack.tengine.make_environment() + if phase != 'update': + # Update.xml stores site information differently + # than the rest of the CTest XML files. + site_template = os.path.join(self.template_dir, 'Site.xml') + t = env.get_template(site_template) + f.write(t.render(report_data)) + + phase_template = os.path.join(self.template_dir, report_name) + t = env.get_template(phase_template) + f.write(t.render(report_data)) + self.upload(phase_report) + + def test_report(self, directory_name, input_data): + # Generate reports for each package in each spec. + for spec in input_data['specs']: + duration = 0 + if 'time' in spec: + duration = int(spec['time']) + for package in spec['packages']: + self.test_report_for_package( + directory_name, package, duration) self.print_cdash_link() def concretization_report(self, directory_name, msg): diff --git a/lib/spack/spack/reporters/junit.py b/lib/spack/spack/reporters/junit.py index 6c54c45b42..598b308934 100644 --- a/lib/spack/spack/reporters/junit.py +++ b/lib/spack/spack/reporters/junit.py @@ -27,3 +27,6 @@ class JUnit(Reporter): env = spack.tengine.make_environment() t = env.get_template(self.template_file) f.write(t.render(report_data)) + + def test_report(self, filename, report_data): + self.build_report(filename, report_data) diff --git a/lib/spack/spack/schema/config.py b/lib/spack/spack/schema/config.py index b1d3332a3f..0f83eb86f4 100644 --- a/lib/spack/spack/schema/config.py +++ b/lib/spack/spack/schema/config.py @@ -45,6 +45,7 @@ properties = { {'type': 'array', 'items': {'type': 'string'}}], }, + 'test_stage': {'type': 'string'}, 'extensions': { 'type': 'array', 'items': {'type': 'string'} diff --git a/lib/spack/spack/spec.py b/lib/spack/spack/spec.py index 8a4fc12861..743f84c3c8 100644 --- a/lib/spack/spack/spec.py +++ b/lib/spack/spack/spec.py @@ -951,7 +951,7 @@ class SpecBuildInterface(lang.ObjectWrapper): def __init__(self, spec, name, query_parameters): super(SpecBuildInterface, self).__init__(spec) - is_virtual = Spec.is_virtual(name) + is_virtual = spack.repo.path.is_virtual(name) self.last_query = QueryState( name=name, extra_parameters=query_parameters, @@ -1227,12 +1227,9 @@ class Spec(object): Possible idea: just use conventin and make virtual deps all caps, e.g., MPI vs mpi. """ - return Spec.is_virtual(self.name) - - @staticmethod - def is_virtual(name): - """Test if a name is virtual without requiring a Spec.""" - return (name is not None) and (not spack.repo.path.exists(name)) + # This method can be called while regenerating the provider index + # So we turn off using the index to detect virtuals + return spack.repo.path.is_virtual(self.name, use_index=False) @property def concrete(self): diff --git a/lib/spack/spack/tengine.py b/lib/spack/spack/tengine.py index 8581c8b41e..15268e682d 100644 --- a/lib/spack/spack/tengine.py +++ b/lib/spack/spack/tengine.py @@ -68,7 +68,8 @@ def make_environment(dirs=None): """Returns an configured environment for template rendering.""" if dirs is None: # Default directories where to search for templates - builtins = spack.config.get('config:template_dirs') + builtins = spack.config.get('config:template_dirs', + ['$spack/share/spack/templates']) extensions = spack.extensions.get_template_dirs() dirs = [canonicalize_path(d) for d in itertools.chain(builtins, extensions)] diff --git a/lib/spack/spack/test/cmd/clean.py b/lib/spack/spack/test/cmd/clean.py index 4df708cf50..dcaf0c916c 100644 --- a/lib/spack/spack/test/cmd/clean.py +++ b/lib/spack/spack/test/cmd/clean.py @@ -15,44 +15,51 @@ clean = spack.main.SpackCommand('clean') @pytest.fixture() def mock_calls_for_clean(monkeypatch): + counts = {} + class Counter(object): - def __init__(self): - self.call_count = 0 + def __init__(self, name): + self.name = name + counts[name] = 0 def __call__(self, *args, **kwargs): - self.call_count += 1 + counts[self.name] += 1 - monkeypatch.setattr(spack.package.PackageBase, 'do_clean', Counter()) - monkeypatch.setattr(spack.stage, 'purge', Counter()) + monkeypatch.setattr(spack.package.PackageBase, 'do_clean', + Counter('package')) + monkeypatch.setattr(spack.stage, 'purge', Counter('stages')) monkeypatch.setattr( - spack.caches.fetch_cache, 'destroy', Counter(), raising=False) + spack.caches.fetch_cache, 'destroy', Counter('downloads'), + raising=False) monkeypatch.setattr( - spack.caches.misc_cache, 'destroy', Counter()) + spack.caches.misc_cache, 'destroy', Counter('caches')) monkeypatch.setattr( - spack.installer, 'clear_failures', Counter()) + spack.installer, 'clear_failures', Counter('failures')) + + yield counts + + +all_effects = ['stages', 'downloads', 'caches', 'failures'] @pytest.mark.usefixtures( - 'mock_packages', 'config', 'mock_calls_for_clean' + 'mock_packages', 'config' ) -@pytest.mark.parametrize('command_line,counters', [ - ('mpileaks', [1, 0, 0, 0, 0]), - ('-s', [0, 1, 0, 0, 0]), - ('-sd', [0, 1, 1, 0, 0]), - ('-m', [0, 0, 0, 1, 0]), - ('-f', [0, 0, 0, 0, 1]), - ('-a', [0, 1, 1, 1, 1]), - ('', [0, 0, 0, 0, 0]), +@pytest.mark.parametrize('command_line,effects', [ + ('mpileaks', ['package']), + ('-s', ['stages']), + ('-sd', ['stages', 'downloads']), + ('-m', ['caches']), + ('-f', ['failures']), + ('-a', all_effects), + ('', []), ]) -def test_function_calls(command_line, counters): +def test_function_calls(command_line, effects, mock_calls_for_clean): # Call the command with the supplied command line clean(command_line) # Assert that we called the expected functions the correct # number of times - assert spack.package.PackageBase.do_clean.call_count == counters[0] - assert spack.stage.purge.call_count == counters[1] - assert spack.caches.fetch_cache.destroy.call_count == counters[2] - assert spack.caches.misc_cache.destroy.call_count == counters[3] - assert spack.installer.clear_failures.call_count == counters[4] + for name in ['package'] + all_effects: + assert mock_calls_for_clean[name] == (1 if name in effects else 0) diff --git a/lib/spack/spack/test/cmd/mirror.py b/lib/spack/spack/test/cmd/mirror.py index f6fe0b24dd..0957624cba 100644 --- a/lib/spack/spack/test/cmd/mirror.py +++ b/lib/spack/spack/test/cmd/mirror.py @@ -102,7 +102,7 @@ class MockMirrorArgs(object): self.exclude_specs = exclude_specs -def test_exclude_specs(mock_packages): +def test_exclude_specs(mock_packages, config): args = MockMirrorArgs( specs=['mpich'], versions_per_spec='all', @@ -117,7 +117,7 @@ def test_exclude_specs(mock_packages): assert (not expected_exclude & set(mirror_specs)) -def test_exclude_file(mock_packages, tmpdir): +def test_exclude_file(mock_packages, tmpdir, config): exclude_path = os.path.join(str(tmpdir), 'test-exclude.txt') with open(exclude_path, 'w') as exclude_file: exclude_file.write("""\ diff --git a/lib/spack/spack/test/cmd/pkg.py b/lib/spack/spack/test/cmd/pkg.py index af634beffc..ca9e3a1a3e 100644 --- a/lib/spack/spack/test/cmd/pkg.py +++ b/lib/spack/spack/test/cmd/pkg.py @@ -62,9 +62,9 @@ def mock_pkg_git_repo(tmpdir_factory): mkdirp('pkg-a', 'pkg-b', 'pkg-c') with open('pkg-a/package.py', 'w') as f: f.write(pkg_template.format(name='PkgA')) - with open('pkg-c/package.py', 'w') as f: - f.write(pkg_template.format(name='PkgB')) with open('pkg-b/package.py', 'w') as f: + f.write(pkg_template.format(name='PkgB')) + with open('pkg-c/package.py', 'w') as f: f.write(pkg_template.format(name='PkgC')) git('add', 'pkg-a', 'pkg-b', 'pkg-c') git('-c', 'commit.gpgsign=false', 'commit', @@ -128,6 +128,8 @@ def test_pkg_add(mock_pkg_git_repo): git('status', '--short', output=str)) finally: shutil.rmtree('pkg-e') + # Removing a package mid-run disrupts Spack's caching + spack.repo.path.repos[0]._fast_package_checker.invalidate() with pytest.raises(spack.main.SpackCommandError): pkg('add', 'does-not-exist') diff --git a/lib/spack/spack/test/cmd/test.py b/lib/spack/spack/test/cmd/test.py index a9ef735afe..4163853274 100644 --- a/lib/spack/spack/test/cmd/test.py +++ b/lib/spack/spack/test/cmd/test.py @@ -3,93 +3,181 @@ # # SPDX-License-Identifier: (Apache-2.0 OR MIT) +import argparse +import os + +import pytest + +import spack.config +import spack.package +import spack.cmd.install from spack.main import SpackCommand +install = SpackCommand('install') spack_test = SpackCommand('test') -cmd_test_py = 'lib/spack/spack/test/cmd/test.py' - - -def test_list(): - output = spack_test('--list') - assert "test.py" in output - assert "spec_semantics.py" in output - assert "test_list" not in output - - -def test_list_with_pytest_arg(): - output = spack_test('--list', cmd_test_py) - assert output.strip() == cmd_test_py - - -def test_list_with_keywords(): - output = spack_test('--list', '-k', 'cmd/test.py') - assert output.strip() == cmd_test_py - - -def test_list_long(capsys): - with capsys.disabled(): - output = spack_test('--list-long') - assert "test.py::\n" in output - assert "test_list" in output - assert "test_list_with_pytest_arg" in output - assert "test_list_with_keywords" in output - assert "test_list_long" in output - assert "test_list_long_with_pytest_arg" in output - assert "test_list_names" in output - assert "test_list_names_with_pytest_arg" in output - - assert "spec_dag.py::\n" in output - assert 'test_installed_deps' in output - assert 'test_test_deptype' in output - - -def test_list_long_with_pytest_arg(capsys): - with capsys.disabled(): - output = spack_test('--list-long', cmd_test_py) - assert "test.py::\n" in output - assert "test_list" in output - assert "test_list_with_pytest_arg" in output - assert "test_list_with_keywords" in output - assert "test_list_long" in output - assert "test_list_long_with_pytest_arg" in output - assert "test_list_names" in output - assert "test_list_names_with_pytest_arg" in output - - assert "spec_dag.py::\n" not in output - assert 'test_installed_deps' not in output - assert 'test_test_deptype' not in output - - -def test_list_names(): - output = spack_test('--list-names') - assert "test.py::test_list\n" in output - assert "test.py::test_list_with_pytest_arg\n" in output - assert "test.py::test_list_with_keywords\n" in output - assert "test.py::test_list_long\n" in output - assert "test.py::test_list_long_with_pytest_arg\n" in output - assert "test.py::test_list_names\n" in output - assert "test.py::test_list_names_with_pytest_arg\n" in output - - assert "spec_dag.py::test_installed_deps\n" in output - assert 'spec_dag.py::test_test_deptype\n' in output - - -def test_list_names_with_pytest_arg(): - output = spack_test('--list-names', cmd_test_py) - assert "test.py::test_list\n" in output - assert "test.py::test_list_with_pytest_arg\n" in output - assert "test.py::test_list_with_keywords\n" in output - assert "test.py::test_list_long\n" in output - assert "test.py::test_list_long_with_pytest_arg\n" in output - assert "test.py::test_list_names\n" in output - assert "test.py::test_list_names_with_pytest_arg\n" in output - - assert "spec_dag.py::test_installed_deps\n" not in output - assert 'spec_dag.py::test_test_deptype\n' not in output - - -def test_pytest_help(): - output = spack_test('--pytest-help') - assert "-k EXPRESSION" in output - assert "pytest-warnings:" in output - assert "--collect-only" in output + + +def test_test_package_not_installed( + tmpdir, mock_packages, mock_archive, mock_fetch, config, + install_mockery_mutable_config, mock_test_stage): + + output = spack_test('run', 'libdwarf') + + assert "No installed packages match spec libdwarf" in output + + +@pytest.mark.parametrize('arguments,expected', [ + (['run'], spack.config.get('config:dirty')), # default from config file + (['run', '--clean'], False), + (['run', '--dirty'], True), +]) +def test_test_dirty_flag(arguments, expected): + parser = argparse.ArgumentParser() + spack.cmd.test.setup_parser(parser) + args = parser.parse_args(arguments) + assert args.dirty == expected + + +def test_test_output(mock_test_stage, mock_packages, mock_archive, mock_fetch, + install_mockery_mutable_config): + """Ensure output printed from pkgs is captured by output redirection.""" + install('printing-package') + spack_test('run', 'printing-package') + + stage_files = os.listdir(mock_test_stage) + assert len(stage_files) == 1 + + # Grab test stage directory contents + testdir = os.path.join(mock_test_stage, stage_files[0]) + testdir_files = os.listdir(testdir) + + # Grab the output from the test log + testlog = list(filter(lambda x: x.endswith('out.txt') and + x != 'results.txt', testdir_files)) + outfile = os.path.join(testdir, testlog[0]) + with open(outfile, 'r') as f: + output = f.read() + assert "BEFORE TEST" in output + assert "true: expect command status in [" in output + assert "AFTER TEST" in output + assert "FAILED" not in output + + +def test_test_output_on_error( + mock_packages, mock_archive, mock_fetch, install_mockery_mutable_config, + capfd, mock_test_stage +): + install('test-error') + # capfd interferes with Spack's capturing + with capfd.disabled(): + out = spack_test('run', 'test-error', fail_on_error=False) + + assert "TestFailure" in out + assert "Command exited with status 1" in out + + +def test_test_output_on_failure( + mock_packages, mock_archive, mock_fetch, install_mockery_mutable_config, + capfd, mock_test_stage +): + install('test-fail') + with capfd.disabled(): + out = spack_test('run', 'test-fail', fail_on_error=False) + + assert "Expected 'not in the output' to match output of `true`" in out + assert "TestFailure" in out + + +def test_show_log_on_error( + mock_packages, mock_archive, mock_fetch, + install_mockery_mutable_config, capfd, mock_test_stage +): + """Make sure spack prints location of test log on failure.""" + install('test-error') + with capfd.disabled(): + out = spack_test('run', 'test-error', fail_on_error=False) + + assert 'See test log' in out + assert mock_test_stage in out + + +@pytest.mark.usefixtures( + 'mock_packages', 'mock_archive', 'mock_fetch', + 'install_mockery_mutable_config' +) +@pytest.mark.parametrize('pkg_name,msgs', [ + ('test-error', ['FAILED: Command exited', 'TestFailure']), + ('test-fail', ['FAILED: Expected', 'TestFailure']) +]) +def test_junit_output_with_failures(tmpdir, mock_test_stage, pkg_name, msgs): + install(pkg_name) + with tmpdir.as_cwd(): + spack_test('run', + '--log-format=junit', '--log-file=test.xml', + pkg_name) + + files = tmpdir.listdir() + filename = tmpdir.join('test.xml') + assert filename in files + + content = filename.open().read() + + # Count failures and errors correctly + assert 'tests="1"' in content + assert 'failures="1"' in content + assert 'errors="0"' in content + + # We want to have both stdout and stderr + assert '<system-out>' in content + for msg in msgs: + assert msg in content + + +def test_cdash_output_test_error( + tmpdir, mock_fetch, install_mockery_mutable_config, mock_packages, + mock_archive, mock_test_stage, capfd): + install('test-error') + with tmpdir.as_cwd(): + spack_test('run', + '--log-format=cdash', + '--log-file=cdash_reports', + 'test-error') + report_dir = tmpdir.join('cdash_reports') + print(tmpdir.listdir()) + assert report_dir in tmpdir.listdir() + report_file = report_dir.join('test-error_Test.xml') + assert report_file in report_dir.listdir() + content = report_file.open().read() + assert 'FAILED: Command exited with status 1' in content + + +def test_cdash_upload_clean_test( + tmpdir, mock_fetch, install_mockery_mutable_config, mock_packages, + mock_archive, mock_test_stage): + install('printing-package') + with tmpdir.as_cwd(): + spack_test('run', + '--log-file=cdash_reports', + '--log-format=cdash', + 'printing-package') + report_dir = tmpdir.join('cdash_reports') + assert report_dir in tmpdir.listdir() + report_file = report_dir.join('printing-package_Test.xml') + assert report_file in report_dir.listdir() + content = report_file.open().read() + assert '</Test>' in content + assert '<Text>' not in content + + +def test_test_help_does_not_show_cdash_options(mock_test_stage, capsys): + """Make sure `spack test --help` does not describe CDash arguments""" + with pytest.raises(SystemExit): + spack_test('run', '--help') + captured = capsys.readouterr() + assert 'CDash URL' not in captured.out + + +def test_test_help_cdash(mock_test_stage): + """Make sure `spack test --help-cdash` describes CDash arguments""" + out = spack_test('run', '--help-cdash') + assert 'CDash URL' in out diff --git a/lib/spack/spack/test/cmd/unit_test.py b/lib/spack/spack/test/cmd/unit_test.py new file mode 100644 index 0000000000..c5b8eb765e --- /dev/null +++ b/lib/spack/spack/test/cmd/unit_test.py @@ -0,0 +1,96 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +from spack.main import SpackCommand + +spack_test = SpackCommand('unit-test') +cmd_test_py = 'lib/spack/spack/test/cmd/unit_test.py' + + +def test_list(): + output = spack_test('--list') + assert "unit_test.py" in output + assert "spec_semantics.py" in output + assert "test_list" not in output + + +def test_list_with_pytest_arg(): + output = spack_test('--list', cmd_test_py) + assert output.strip() == cmd_test_py + + +def test_list_with_keywords(): + output = spack_test('--list', '-k', 'cmd/unit_test.py') + assert output.strip() == cmd_test_py + + +def test_list_long(capsys): + with capsys.disabled(): + output = spack_test('--list-long') + assert "unit_test.py::\n" in output + assert "test_list" in output + assert "test_list_with_pytest_arg" in output + assert "test_list_with_keywords" in output + assert "test_list_long" in output + assert "test_list_long_with_pytest_arg" in output + assert "test_list_names" in output + assert "test_list_names_with_pytest_arg" in output + + assert "spec_dag.py::\n" in output + assert 'test_installed_deps' in output + assert 'test_test_deptype' in output + + +def test_list_long_with_pytest_arg(capsys): + with capsys.disabled(): + output = spack_test('--list-long', cmd_test_py) + print(output) + assert "unit_test.py::\n" in output + assert "test_list" in output + assert "test_list_with_pytest_arg" in output + assert "test_list_with_keywords" in output + assert "test_list_long" in output + assert "test_list_long_with_pytest_arg" in output + assert "test_list_names" in output + assert "test_list_names_with_pytest_arg" in output + + assert "spec_dag.py::\n" not in output + assert 'test_installed_deps' not in output + assert 'test_test_deptype' not in output + + +def test_list_names(): + output = spack_test('--list-names') + assert "unit_test.py::test_list\n" in output + assert "unit_test.py::test_list_with_pytest_arg\n" in output + assert "unit_test.py::test_list_with_keywords\n" in output + assert "unit_test.py::test_list_long\n" in output + assert "unit_test.py::test_list_long_with_pytest_arg\n" in output + assert "unit_test.py::test_list_names\n" in output + assert "unit_test.py::test_list_names_with_pytest_arg\n" in output + + assert "spec_dag.py::test_installed_deps\n" in output + assert 'spec_dag.py::test_test_deptype\n' in output + + +def test_list_names_with_pytest_arg(): + output = spack_test('--list-names', cmd_test_py) + assert "unit_test.py::test_list\n" in output + assert "unit_test.py::test_list_with_pytest_arg\n" in output + assert "unit_test.py::test_list_with_keywords\n" in output + assert "unit_test.py::test_list_long\n" in output + assert "unit_test.py::test_list_long_with_pytest_arg\n" in output + assert "unit_test.py::test_list_names\n" in output + assert "unit_test.py::test_list_names_with_pytest_arg\n" in output + + assert "spec_dag.py::test_installed_deps\n" not in output + assert 'spec_dag.py::test_test_deptype\n' not in output + + +def test_pytest_help(): + output = spack_test('--pytest-help') + assert "-k EXPRESSION" in output + assert "pytest-warnings:" in output + assert "--collect-only" in output diff --git a/lib/spack/spack/test/conftest.py b/lib/spack/spack/test/conftest.py index 527d6a3380..8615b14abe 100644 --- a/lib/spack/spack/test/conftest.py +++ b/lib/spack/spack/test/conftest.py @@ -1273,3 +1273,14 @@ def mock_executable(tmpdir): return str(f) return _factory + + +@pytest.fixture() +def mock_test_stage(mutable_config, tmpdir): + # NOTE: This fixture MUST be applied after any fixture that uses + # the config fixture under the hood + # No need to unset because we use mutable_config + tmp_stage = str(tmpdir.join('test_stage')) + mutable_config.set('config:test_stage', tmp_stage) + + yield tmp_stage diff --git a/lib/spack/spack/test/llnl/util/tty/__init__.py b/lib/spack/spack/test/llnl/util/tty/__init__.py new file mode 100644 index 0000000000..9f87532b85 --- /dev/null +++ b/lib/spack/spack/test/llnl/util/tty/__init__.py @@ -0,0 +1,4 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/mirror.py b/lib/spack/spack/test/mirror.py index 05cf46dc20..8d3ec2d07b 100644 --- a/lib/spack/spack/test/mirror.py +++ b/lib/spack/spack/test/mirror.py @@ -16,7 +16,7 @@ from spack.util.executable import which from llnl.util.filesystem import resolve_link_target_relative_to_the_link -pytestmark = pytest.mark.usefixtures('config', 'mutable_mock_repo') +pytestmark = pytest.mark.usefixtures('mutable_config', 'mutable_mock_repo') # paths in repos that shouldn't be in the mirror tarballs. exclude = ['.hg', '.git', '.svn'] @@ -97,7 +97,7 @@ def check_mirror(): # tarball assert not dcmp.right_only # and that all original files are present. - assert all(l in exclude for l in dcmp.left_only) + assert all(left in exclude for left in dcmp.left_only) def test_url_mirror(mock_archive): diff --git a/lib/spack/spack/test/package_class.py b/lib/spack/spack/test/package_class.py index d540ac663e..33e5eb1c0a 100644 --- a/lib/spack/spack/test/package_class.py +++ b/lib/spack/spack/test/package_class.py @@ -10,7 +10,12 @@ etc.). Only methods like ``possible_dependencies()`` that deal with the static DSL metadata for packages. """ +import os import pytest +import shutil + +import llnl.util.filesystem as fs + import spack.package import spack.repo @@ -119,3 +124,72 @@ def test_possible_dependencies_with_multiple_classes( }) assert expected == spack.package.possible_dependencies(*pkgs) + + +def setup_install_test(source_paths, install_test_root): + """ + Set up the install test by creating sources and install test roots. + + The convention used here is to create an empty file if the path name + ends with an extension otherwise, a directory is created. + """ + fs.mkdirp(install_test_root) + for path in source_paths: + if os.path.splitext(path)[1]: + fs.touchp(path) + else: + fs.mkdirp(path) + + +@pytest.mark.parametrize('spec,sources,extras,expect', [ + ('a', + ['example/a.c'], # Source(s) + ['example/a.c'], # Extra test source + ['example/a.c']), # Test install dir source(s) + ('b', + ['test/b.cpp', 'test/b.hpp', 'example/b.txt'], # Source(s) + ['test'], # Extra test source + ['test/b.cpp', 'test/b.hpp']), # Test install dir source + ('c', + ['examples/a.py', 'examples/b.py', 'examples/c.py', 'tests/d.py'], + ['examples/b.py', 'tests'], + ['examples/b.py', 'tests/d.py']), +]) +def test_cache_extra_sources(install_mockery, spec, sources, extras, expect): + """Test the package's cache extra test sources helper function.""" + + pkg = spack.repo.get(spec) + pkg.spec.concretize() + source_path = pkg.stage.source_path + + srcs = [fs.join_path(source_path, s) for s in sources] + setup_install_test(srcs, pkg.install_test_root) + + emsg_dir = 'Expected {0} to be a directory' + emsg_file = 'Expected {0} to be a file' + for s in srcs: + assert os.path.exists(s), 'Expected {0} to exist'.format(s) + if os.path.splitext(s)[1]: + assert os.path.isfile(s), emsg_file.format(s) + else: + assert os.path.isdir(s), emsg_dir.format(s) + + pkg.cache_extra_test_sources(extras) + + src_dests = [fs.join_path(pkg.install_test_root, s) for s in sources] + exp_dests = [fs.join_path(pkg.install_test_root, e) for e in expect] + poss_dests = set(src_dests) | set(exp_dests) + + msg = 'Expected {0} to{1} exist' + for pd in poss_dests: + if pd in exp_dests: + assert os.path.exists(pd), msg.format(pd, '') + if os.path.splitext(pd)[1]: + assert os.path.isfile(pd), emsg_file.format(pd) + else: + assert os.path.isdir(pd), emsg_dir.format(pd) + else: + assert not os.path.exists(pd), msg.format(pd, ' not') + + # Perform a little cleanup + shutil.rmtree(os.path.dirname(source_path)) diff --git a/lib/spack/spack/test/test_suite.py b/lib/spack/spack/test/test_suite.py new file mode 100644 index 0000000000..1ec5106182 --- /dev/null +++ b/lib/spack/spack/test/test_suite.py @@ -0,0 +1,53 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) +import os +import spack.install_test +import spack.spec + + +def test_test_log_pathname(mock_packages, config): + """Ensure test log path is reasonable.""" + spec = spack.spec.Spec('libdwarf').concretized() + + test_name = 'test_name' + + test_suite = spack.install_test.TestSuite([spec], test_name) + logfile = test_suite.log_file_for_spec(spec) + + assert test_suite.stage in logfile + assert test_suite.test_log_name(spec) in logfile + + +def test_test_ensure_stage(mock_test_stage): + """Make sure test stage directory is properly set up.""" + spec = spack.spec.Spec('libdwarf').concretized() + + test_name = 'test_name' + + test_suite = spack.install_test.TestSuite([spec], test_name) + test_suite.ensure_stage() + + assert os.path.isdir(test_suite.stage) + assert mock_test_stage in test_suite.stage + + +def test_write_test_result(mock_packages, mock_test_stage): + """Ensure test results written to a results file.""" + spec = spack.spec.Spec('libdwarf').concretized() + result = 'TEST' + test_name = 'write-test' + + test_suite = spack.install_test.TestSuite([spec], test_name) + test_suite.ensure_stage() + results_file = test_suite.results_file + test_suite.write_test_result(spec, result) + + with open(results_file, 'r') as f: + lines = f.readlines() + assert len(lines) == 1 + + msg = lines[0] + assert result in msg + assert spec.name in msg diff --git a/lib/spack/spack/util/executable.py b/lib/spack/spack/util/executable.py index 097da3337e..614bc1725a 100644 --- a/lib/spack/spack/util/executable.py +++ b/lib/spack/spack/util/executable.py @@ -2,7 +2,7 @@ # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) - +import sys import os import re import shlex @@ -98,6 +98,9 @@ class Executable(object): If both ``output`` and ``error`` are set to ``str``, then one string is returned containing output concatenated with error. Not valid for ``input`` + * ``str.split``, as in the ``split`` method of the Python string type. + Behaves the same as ``str``, except that value is also written to + ``stdout`` or ``stderr``. By default, the subprocess inherits the parent's file descriptors. @@ -132,7 +135,7 @@ class Executable(object): def streamify(arg, mode): if isinstance(arg, string_types): return open(arg, mode), True - elif arg is str: + elif arg in (str, str.split): return subprocess.PIPE, False else: return arg, False @@ -168,12 +171,18 @@ class Executable(object): out, err = proc.communicate() result = None - if output is str or error is str: + if output in (str, str.split) or error in (str, str.split): result = '' - if output is str: - result += text_type(out.decode('utf-8')) - if error is str: - result += text_type(err.decode('utf-8')) + if output in (str, str.split): + outstr = text_type(out.decode('utf-8')) + result += outstr + if output is str.split: + sys.stdout.write(outstr) + if error in (str, str.split): + errstr = text_type(err.decode('utf-8')) + result += errstr + if error is str.split: + sys.stderr.write(errstr) rc = self.returncode = proc.returncode if fail_on_error and rc != 0 and (rc not in ignore_errors): diff --git a/lib/spack/spack/util/mock_package.py b/lib/spack/spack/util/mock_package.py index e855aae015..4751f5af7e 100644 --- a/lib/spack/spack/util/mock_package.py +++ b/lib/spack/spack/util/mock_package.py @@ -21,6 +21,8 @@ class MockPackageBase(object): Use ``MockPackageMultiRepo.add_package()`` to create new instances. """ + virtual = False + def __init__(self, dependencies, dependency_types, conditions=None, versions=None): """Instantiate a new MockPackageBase. @@ -92,7 +94,7 @@ class MockPackageMultiRepo(object): def exists(self, name): return name in self.spec_to_pkg - def is_virtual(self, name): + def is_virtual(self, name, use_index=True): return False def repo_for_pkg(self, name): |