diff options
131 files changed, 3599 insertions, 676 deletions
diff --git a/.github/workflows/linux_unit_tests.yaml b/.github/workflows/linux_unit_tests.yaml index c66d58c284..c87ea6e07a 100644 --- a/.github/workflows/linux_unit_tests.yaml +++ b/.github/workflows/linux_unit_tests.yaml @@ -132,7 +132,7 @@ jobs: . share/spack/setup-env.sh spack compiler find spack solve mpileaks%gcc - coverage run $(which spack) test -v + coverage run $(which spack) unit-test -v coverage combine coverage xml - uses: codecov/codecov-action@v1 diff --git a/.github/workflows/macos_unit_tests.yaml b/.github/workflows/macos_unit_tests.yaml index 2de92394f8..29caaa2e08 100644 --- a/.github/workflows/macos_unit_tests.yaml +++ b/.github/workflows/macos_unit_tests.yaml @@ -35,7 +35,7 @@ jobs: git --version . .github/workflows/setup_git.sh . share/spack/setup-env.sh - coverage run $(which spack) test + coverage run $(which spack) unit-test coverage combine coverage xml - uses: codecov/codecov-action@v1 diff --git a/etc/spack/defaults/config.yaml b/etc/spack/defaults/config.yaml index 15ce68c68f..d1a7f35a6d 100644 --- a/etc/spack/defaults/config.yaml +++ b/etc/spack/defaults/config.yaml @@ -70,6 +70,10 @@ config: - ~/.spack/stage # - $spack/var/spack/stage + # Directory in which to run tests and store test results. + # Tests will be stored in directories named by date/time and package + # name/hash. + test_stage: ~/.spack/test # Cache directory for already downloaded source tarballs and archived # repositories. This can be purged with `spack clean --downloads`. diff --git a/lib/spack/docs/contribution_guide.rst b/lib/spack/docs/contribution_guide.rst index 37cf9091bd..8df8ad65ba 100644 --- a/lib/spack/docs/contribution_guide.rst +++ b/lib/spack/docs/contribution_guide.rst @@ -74,7 +74,7 @@ locally to speed up the review process. We currently test against Python 2.6, 2.7, and 3.5-3.7 on both macOS and Linux and perform 3 types of tests: -.. _cmd-spack-test: +.. _cmd-spack-unit-test: ^^^^^^^^^^ Unit Tests @@ -96,7 +96,7 @@ To run *all* of the unit tests, use: .. code-block:: console - $ spack test + $ spack unit-test These tests may take several minutes to complete. If you know you are only modifying a single Spack feature, you can run subsets of tests at a @@ -105,13 +105,13 @@ time. For example, this would run all the tests in .. code-block:: console - $ spack test lib/spack/spack/test/architecture.py + $ spack unit-test lib/spack/spack/test/architecture.py And this would run the ``test_platform`` test from that file: .. code-block:: console - $ spack test lib/spack/spack/test/architecture.py::test_platform + $ spack unit-test lib/spack/spack/test/architecture.py::test_platform This allows you to develop iteratively: make a change, test that change, make another change, test that change, etc. We use `pytest @@ -121,29 +121,29 @@ pytest docs <http://doc.pytest.org/en/latest/usage.html#specifying-tests-selecting-tests>`_ for more details on test selection syntax. -``spack test`` has a few special options that can help you understand -what tests are available. To get a list of all available unit test -files, run: +``spack unit-test`` has a few special options that can help you +understand what tests are available. To get a list of all available +unit test files, run: -.. command-output:: spack test --list +.. command-output:: spack unit-test --list :ellipsis: 5 -To see a more detailed list of available unit tests, use ``spack test ---list-long``: +To see a more detailed list of available unit tests, use ``spack +unit-test --list-long``: -.. command-output:: spack test --list-long +.. command-output:: spack unit-test --list-long :ellipsis: 10 And to see the fully qualified names of all tests, use ``--list-names``: -.. command-output:: spack test --list-names +.. command-output:: spack unit-test --list-names :ellipsis: 5 You can combine these with ``pytest`` arguments to restrict which tests you want to know about. For example, to see just the tests in ``architecture.py``: -.. command-output:: spack test --list-long lib/spack/spack/test/architecture.py +.. command-output:: spack unit-test --list-long lib/spack/spack/test/architecture.py You can also combine any of these options with a ``pytest`` keyword search. See the `pytest usage docs @@ -151,7 +151,7 @@ search. See the `pytest usage docs for more details on test selection syntax. For example, to see the names of all tests that have "spec" or "concretize" somewhere in their names: -.. command-output:: spack test --list-names -k "spec and concretize" +.. command-output:: spack unit-test --list-names -k "spec and concretize" By default, ``pytest`` captures the output of all unit tests, and it will print any captured output for failed tests. Sometimes it's helpful to see @@ -161,7 +161,7 @@ argument to ``pytest``: .. code-block:: console - $ spack test -s --list-long lib/spack/spack/test/architecture.py::test_platform + $ spack unit-test -s --list-long lib/spack/spack/test/architecture.py::test_platform Unit tests are crucial to making sure bugs aren't introduced into Spack. If you are modifying core Spack libraries or adding new @@ -176,7 +176,7 @@ how to write tests! You may notice the ``share/spack/qa/run-unit-tests`` script in the repository. This script is designed for CI. It runs the unit tests and reports coverage statistics back to Codecov. If you want to - run the unit tests yourself, we suggest you use ``spack test``. + run the unit tests yourself, we suggest you use ``spack unit-test``. ^^^^^^^^^^^^ Flake8 Tests diff --git a/lib/spack/docs/developer_guide.rst b/lib/spack/docs/developer_guide.rst index 7fd4d1ec6e..b4b8b5d321 100644 --- a/lib/spack/docs/developer_guide.rst +++ b/lib/spack/docs/developer_guide.rst @@ -363,11 +363,12 @@ Developer commands ``spack doc`` ^^^^^^^^^^^^^ -^^^^^^^^^^^^^^ -``spack test`` -^^^^^^^^^^^^^^ +^^^^^^^^^^^^^^^^^^^ +``spack unit-test`` +^^^^^^^^^^^^^^^^^^^ -See the :ref:`contributor guide section <cmd-spack-test>` on ``spack test``. +See the :ref:`contributor guide section <cmd-spack-unit-test>` on +``spack unit-test``. .. _cmd-spack-python: diff --git a/lib/spack/docs/extensions.rst b/lib/spack/docs/extensions.rst index c71a6511ed..15c59c76ef 100644 --- a/lib/spack/docs/extensions.rst +++ b/lib/spack/docs/extensions.rst @@ -87,11 +87,12 @@ will be available from the command line: --implicit select specs that are not installed or were installed implicitly --output OUTPUT where to dump the result -The corresponding unit tests can be run giving the appropriate options to ``spack test``: +The corresponding unit tests can be run giving the appropriate options +to ``spack unit-test``: .. code-block:: console - $ spack test --extension=scripting + $ spack unit-test --extension=scripting ============================================================== test session starts =============================================================== platform linux2 -- Python 2.7.15rc1, pytest-3.2.5, py-1.4.34, pluggy-0.4.0 diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst index 836fc12b83..ec83679a47 100644 --- a/lib/spack/docs/packaging_guide.rst +++ b/lib/spack/docs/packaging_guide.rst @@ -3948,6 +3948,118 @@ using the ``run_before`` decorator. .. _file-manipulation: +^^^^^^^^^^^^^ +Install Tests +^^^^^^^^^^^^^ + +.. warning:: + + The API for adding and running install tests is not yet considered + stable and may change drastically in future releases. Packages with + upstreamed tests will be refactored to match changes to the API. + +While build-tests are integrated with the build system, install tests +may be added to Spack packages to be run independently of the install +method. + +Install tests may be added by defining a ``test`` method with the following signature: + +.. code-block:: python + + def test(self): + +These tests will be run in an environment set up to provide access to +this package and all of its dependencies, including ``test``-type +dependencies. Inside the ``test`` method, standard python ``assert`` +statements and other error reporting mechanisms can be used. Spack +will report any errors as a test failure. + +Inside the test method, individual tests can be run separately (and +continue transparently after a test failure) using the ``run_test`` +method. The signature for the ``run_test`` method is: + +.. code-block:: python + + def run_test(self, exe, options=[], expected=[], status=0, installed=False, + purpose='', skip_missing=False, work_dir=None): + +This method will operate in ``work_dir`` if one is specified. It will +search for an executable in the ``PATH`` variable named ``exe``, and +if ``installed=True`` it will fail if that executable does not come +from the prefix of the package being tested. If the executable is not +found, it will fail the test unless ``skip_missing`` is set to +``True``. The executable will be run with the options specified, and +the return code will be checked against the ``status`` argument, which +can be an integer or list of integers. Spack will also check that +every string in ``expected`` is a regex matching part of the output of +the executable. The ``purpose`` argument is recorded in the test log +for debugging purposes. + +"""""""""""""""""""""""""""""""""""""" +Install tests that require compilation +"""""""""""""""""""""""""""""""""""""" + +Some tests may require access to the compiler with which the package +was built, especially to test library-only packages. To ensure the +compiler is configured as part of the test environment, set the +attribute ``tests_require_compiler = True`` on the package. The +compiler will be available through the canonical environment variables +(``CC``, ``CXX``, ``FC``, ``F77``) in the test environment. + +"""""""""""""""""""""""""""""""""""""""""""""""" +Install tests that require build-time components +"""""""""""""""""""""""""""""""""""""""""""""""" + +Some packages cannot be easily tested without components from the +build-time test suite. For those packages, the +``cache_extra_test_sources`` method can be used. + +.. code-block:: python + + @run_after('install') + def cache_test_sources(self): + srcs = ['./tests/foo.c', './tests/bar.c'] + self.cache_extra_test_sources(srcs) + +This method will copy the listed methods into the metadata directory +of the package at the end of the install phase of the build. They will +be available to the test method in the directory +``self._extra_tests_path``. + +While source files are generally recommended, for many packages +binaries may also technically be cached in this way for later testing. + +""""""""""""""""""""" +Running install tests +""""""""""""""""""""" + +Install tests can be run using the ``spack test run`` command. The +``spack test run`` command will create a ``test suite`` out of the +specs provided to it, or if no specs are provided it will test all +specs in the active environment, or all specs installed in Spack if no +environment is active. Test suites can be named using the ``--alias`` +option; test suites not aliased will use the content hash of their +specs as their name. + +Packages to install test can be queried using the ``spack test list`` +command, which outputs all installed packages with defined ``test`` +methods. + +Test suites can be found using the ``spack test find`` command. It +will list all test suites that have been run and have not been removed +using the ``spack test remove`` command. The ``spack test remove`` +command will remove tests to declutter the test stage. The ``spack +test results`` command will show results for completed test suites. + +The test stage is the working directory for all install tests run with +Spack. By default, Spack uses ``~/.spack/test`` as the test stage. The +test stage can be set in the high-level config: + +.. code-block:: yaml + + config: + test_stage: /path/to/stage + --------------------------- File manipulation functions --------------------------- diff --git a/lib/spack/external/ctest_log_parser.py b/lib/spack/external/ctest_log_parser.py index 0437b6e524..072c10d7a9 100644 --- a/lib/spack/external/ctest_log_parser.py +++ b/lib/spack/external/ctest_log_parser.py @@ -118,6 +118,7 @@ _error_matches = [ "([^:]+): (Error:|error|undefined reference|multiply defined)", "([^ :]+) ?: (error|fatal error|catastrophic error)", "([^:]+)\\(([^\\)]+)\\) ?: (error|fatal error|catastrophic error)"), + "^FAILED", "^[Bb]us [Ee]rror", "^[Ss]egmentation [Vv]iolation", "^[Ss]egmentation [Ff]ault", diff --git a/lib/spack/llnl/util/filesystem.py b/lib/spack/llnl/util/filesystem.py index b8d0b4d2f1..d6579555ad 100644 --- a/lib/spack/llnl/util/filesystem.py +++ b/lib/spack/llnl/util/filesystem.py @@ -41,6 +41,8 @@ __all__ = [ 'fix_darwin_install_name', 'force_remove', 'force_symlink', + 'chgrp', + 'chmod_x', 'copy', 'install', 'copy_tree', @@ -52,6 +54,7 @@ __all__ = [ 'partition_path', 'prefixes', 'remove_dead_links', + 'remove_directory_contents', 'remove_if_dead_link', 'remove_linked_tree', 'set_executable', @@ -1806,3 +1809,13 @@ def md5sum(file): with open(file, "rb") as f: md5.update(f.read()) return md5.digest() + + +def remove_directory_contents(dir): + """Remove all contents of a directory.""" + if os.path.exists(dir): + for entry in [os.path.join(dir, entry) for entry in os.listdir(dir)]: + if os.path.isfile(entry) or os.path.islink(entry): + os.unlink(entry) + else: + shutil.rmtree(entry) diff --git a/lib/spack/spack/build_environment.py b/lib/spack/spack/build_environment.py index a1501034d4..cb1ba21ba5 100644 --- a/lib/spack/spack/build_environment.py +++ b/lib/spack/spack/build_environment.py @@ -32,8 +32,8 @@ There are two parts to the build environment: Skimming this module is a nice way to get acquainted with the types of calls you can make from within the install() function. """ -import re import inspect +import re import multiprocessing import os import shutil @@ -53,10 +53,14 @@ import spack.build_systems.meson import spack.config import spack.main import spack.paths +import spack.package +import spack.repo import spack.schema.environment import spack.store +import spack.install_test import spack.subprocess_context import spack.architecture as arch +import spack.util.path from spack.util.string import plural from spack.util.environment import ( env_flag, filter_system_paths, get_path, is_system_path, @@ -453,7 +457,6 @@ def _set_variables_for_single_module(pkg, module): jobs = spack.config.get('config:build_jobs', 16) if pkg.parallel else 1 jobs = min(jobs, multiprocessing.cpu_count()) - assert jobs is not None, "no default set for config:build_jobs" m = module m.make_jobs = jobs @@ -713,28 +716,42 @@ def load_external_modules(pkg): load_module(external_module) -def setup_package(pkg, dirty): +def setup_package(pkg, dirty, context='build'): """Execute all environment setup routines.""" - build_env = EnvironmentModifications() + env = EnvironmentModifications() if not dirty: clean_environment() - set_compiler_environment_variables(pkg, build_env) - set_build_environment_variables(pkg, build_env, dirty) - pkg.architecture.platform.setup_platform_environment(pkg, build_env) + # setup compilers and build tools for build contexts + need_compiler = context == 'build' or (context == 'test' and + pkg.test_requires_compiler) + if need_compiler: + set_compiler_environment_variables(pkg, env) + set_build_environment_variables(pkg, env, dirty) - build_env.extend( - modifications_from_dependencies(pkg.spec, context='build') - ) + # architecture specific setup + pkg.architecture.platform.setup_platform_environment(pkg, env) - if (not dirty) and (not build_env.is_unset('CPATH')): - tty.debug("A dependency has updated CPATH, this may lead pkg-config" - " to assume that the package is part of the system" - " includes and omit it when invoked with '--cflags'.") + if context == 'build': + # recursive post-order dependency information + env.extend( + modifications_from_dependencies(pkg.spec, context=context) + ) + + if (not dirty) and (not env.is_unset('CPATH')): + tty.debug("A dependency has updated CPATH, this may lead pkg-" + "config to assume that the package is part of the system" + " includes and omit it when invoked with '--cflags'.") - set_module_variables_for_package(pkg) - pkg.setup_build_environment(build_env) + # setup package itself + set_module_variables_for_package(pkg) + pkg.setup_build_environment(env) + elif context == 'test': + import spack.user_environment as uenv # avoid circular import + env.extend(uenv.environment_modifications_for_spec(pkg.spec)) + set_module_variables_for_package(pkg) + env.prepend_path('PATH', '.') # Loading modules, in particular if they are meant to be used outside # of Spack, can change environment variables that are relevant to the @@ -744,15 +761,16 @@ def setup_package(pkg, dirty): # unnecessary. Modules affecting these variables will be overwritten anyway with preserve_environment('CC', 'CXX', 'FC', 'F77'): # All module loads that otherwise would belong in previous - # functions have to occur after the build_env object has its + # functions have to occur after the env object has its # modifications applied. Otherwise the environment modifications # could undo module changes, such as unsetting LD_LIBRARY_PATH # after a module changes it. - for mod in pkg.compiler.modules: - # Fixes issue https://github.com/spack/spack/issues/3153 - if os.environ.get("CRAY_CPU_TARGET") == "mic-knl": - load_module("cce") - load_module(mod) + if need_compiler: + for mod in pkg.compiler.modules: + # Fixes issue https://github.com/spack/spack/issues/3153 + if os.environ.get("CRAY_CPU_TARGET") == "mic-knl": + load_module("cce") + load_module(mod) # kludge to handle cray libsci being automatically loaded by PrgEnv # modules on cray platform. Module unload does no damage when @@ -766,12 +784,12 @@ def setup_package(pkg, dirty): implicit_rpaths = pkg.compiler.implicit_rpaths() if implicit_rpaths: - build_env.set('SPACK_COMPILER_IMPLICIT_RPATHS', - ':'.join(implicit_rpaths)) + env.set('SPACK_COMPILER_IMPLICIT_RPATHS', + ':'.join(implicit_rpaths)) # Make sure nothing's strange about the Spack environment. - validate(build_env, tty.warn) - build_env.apply_modifications() + validate(env, tty.warn) + env.apply_modifications() def modifications_from_dependencies(spec, context): @@ -791,7 +809,8 @@ def modifications_from_dependencies(spec, context): deptype_and_method = { 'build': (('build', 'link', 'test'), 'setup_dependent_build_environment'), - 'run': (('link', 'run'), 'setup_dependent_run_environment') + 'run': (('link', 'run'), 'setup_dependent_run_environment'), + 'test': (('link', 'run', 'test'), 'setup_dependent_run_environment') } deptype, method = deptype_and_method[context] @@ -808,6 +827,8 @@ def modifications_from_dependencies(spec, context): def _setup_pkg_and_run(serialized_pkg, function, kwargs, child_pipe, input_multiprocess_fd): + context = kwargs.get('context', 'build') + try: # We are in the child process. Python sets sys.stdin to # open(os.devnull) to prevent our process and its parent from @@ -821,7 +842,8 @@ def _setup_pkg_and_run(serialized_pkg, function, kwargs, child_pipe, if not kwargs.get('fake', False): kwargs['unmodified_env'] = os.environ.copy() - setup_package(pkg, dirty=kwargs.get('dirty', False)) + setup_package(pkg, dirty=kwargs.get('dirty', False), + context=context) return_value = function(pkg, kwargs) child_pipe.send(return_value) @@ -841,13 +863,18 @@ def _setup_pkg_and_run(serialized_pkg, function, kwargs, child_pipe, # show that, too. package_context = get_package_context(tb) - build_log = None - try: - if hasattr(pkg, 'log_path'): - build_log = pkg.log_path - except NameError: - # 'pkg' is not defined yet - pass + logfile = None + if context == 'build': + try: + if hasattr(pkg, 'log_path'): + logfile = pkg.log_path + except NameError: + # 'pkg' is not defined yet + pass + elif context == 'test': + logfile = os.path.join( + pkg.test_suite.stage, + spack.install_test.TestSuite.test_log_name(pkg.spec)) # make a pickleable exception to send to parent. msg = "%s: %s" % (exc_type.__name__, str(exc)) @@ -855,7 +882,7 @@ def _setup_pkg_and_run(serialized_pkg, function, kwargs, child_pipe, ce = ChildError(msg, exc_type.__module__, exc_type.__name__, - tb_string, build_log, package_context) + tb_string, logfile, context, package_context) child_pipe.send(ce) finally: @@ -873,9 +900,6 @@ def start_build_process(pkg, function, kwargs): child process for. function (callable): argless function to run in the child process. - dirty (bool): If True, do NOT clean the environment before - building. - fake (bool): If True, skip package setup b/c it's not a real build Usage:: @@ -961,6 +985,7 @@ def get_package_context(traceback, context=3): Args: traceback (traceback): A traceback from some exception raised during install + context (int): Lines of context to show before and after the line where the error happened @@ -1067,13 +1092,14 @@ class ChildError(InstallError): # context instead of Python context. build_errors = [('spack.util.executable', 'ProcessError')] - def __init__(self, msg, module, classname, traceback_string, build_log, - context): + def __init__(self, msg, module, classname, traceback_string, log_name, + log_type, context): super(ChildError, self).__init__(msg) self.module = module self.name = classname self.traceback = traceback_string - self.build_log = build_log + self.log_name = log_name + self.log_type = log_type self.context = context @property @@ -1081,26 +1107,16 @@ class ChildError(InstallError): out = StringIO() out.write(self._long_message if self._long_message else '') + have_log = self.log_name and os.path.exists(self.log_name) + if (self.module, self.name) in ChildError.build_errors: # The error happened in some external executed process. Show - # the build log with errors or warnings highlighted. - if self.build_log and os.path.exists(self.build_log): - errors, warnings = parse_log_events(self.build_log) - nerr = len(errors) - nwar = len(warnings) - if nerr > 0: - # If errors are found, only display errors - out.write( - "\n%s found in build log:\n" % plural(nerr, 'error')) - out.write(make_log_context(errors)) - elif nwar > 0: - # If no errors are found but warnings are, display warnings - out.write( - "\n%s found in build log:\n" % plural(nwar, 'warning')) - out.write(make_log_context(warnings)) + # the log with errors or warnings highlighted. + if have_log: + write_log_summary(out, self.log_type, self.log_name) else: - # The error happened in in the Python code, so try to show + # The error happened in the Python code, so try to show # some context from the Package itself. if self.context: out.write('\n') @@ -1110,14 +1126,14 @@ class ChildError(InstallError): if out.getvalue(): out.write('\n') - if self.build_log and os.path.exists(self.build_log): - out.write('See build log for details:\n') - out.write(' %s\n' % self.build_log) + if have_log: + out.write('See {0} log for details:\n'.format(self.log_type)) + out.write(' {0}\n'.format(self.log_name)) return out.getvalue() def __str__(self): - return self.message + self.long_message + self.traceback + return self.message def __reduce__(self): """__reduce__ is used to serialize (pickle) ChildErrors. @@ -1130,13 +1146,14 @@ class ChildError(InstallError): self.module, self.name, self.traceback, - self.build_log, + self.log_name, + self.log_type, self.context) -def _make_child_error(msg, module, name, traceback, build_log, context): +def _make_child_error(msg, module, name, traceback, log, log_type, context): """Used by __reduce__ in ChildError to reconstruct pickled errors.""" - return ChildError(msg, module, name, traceback, build_log, context) + return ChildError(msg, module, name, traceback, log, log_type, context) class StopPhase(spack.error.SpackError): @@ -1147,3 +1164,30 @@ class StopPhase(spack.error.SpackError): def _make_stop_phase(msg, long_msg): return StopPhase(msg, long_msg) + + +def write_log_summary(out, log_type, log, last=None): + errors, warnings = parse_log_events(log) + nerr = len(errors) + nwar = len(warnings) + + if nerr > 0: + if last and nerr > last: + errors = errors[-last:] + nerr = last + + # If errors are found, only display errors + out.write( + "\n%s found in %s log:\n" % + (plural(nerr, 'error'), log_type)) + out.write(make_log_context(errors)) + elif nwar > 0: + if last and nwar > last: + warnings = warnings[-last:] + nwar = last + + # If no errors are found but warnings are, display warnings + out.write( + "\n%s found in %s log:\n" % + (plural(nwar, 'warning'), log_type)) + out.write(make_log_context(warnings)) diff --git a/lib/spack/spack/build_systems/cmake.py b/lib/spack/spack/build_systems/cmake.py index 4b679b358a..1336069846 100644 --- a/lib/spack/spack/build_systems/cmake.py +++ b/lib/spack/spack/build_systems/cmake.py @@ -325,13 +325,20 @@ class CMakePackage(PackageBase): libs_flags)) @property + def build_dirname(self): + """Returns the directory name to use when building the package + + :return: name of the subdirectory for building the package + """ + return 'spack-build-%s' % self.spec.dag_hash(7) + + @property def build_directory(self): """Returns the directory to use when building the package :return: directory where to build the package """ - dirname = 'spack-build-%s' % self.spec.dag_hash(7) - return os.path.join(self.stage.path, dirname) + return os.path.join(self.stage.path, self.build_dirname) def cmake_args(self): """Produces a list containing all the arguments that must be passed to diff --git a/lib/spack/spack/build_systems/intel.py b/lib/spack/spack/build_systems/intel.py index a74d5e9613..0e0bb9378b 100644 --- a/lib/spack/spack/build_systems/intel.py +++ b/lib/spack/spack/build_systems/intel.py @@ -1017,6 +1017,15 @@ class IntelPackage(PackageBase): env.extend(EnvironmentModifications.from_sourcing_file(f, *args)) + if self.spec.name in ('intel', 'intel-parallel-studio'): + # this package provides compilers + # TODO: fix check above when compilers are dependencies + env.set('CC', self.prefix.bin.icc) + env.set('CXX', self.prefix.bin.icpc) + env.set('FC', self.prefix.bin.ifort) + env.set('F77', self.prefix.bin.ifort) + env.set('F90', self.prefix.bin.ifort) + def setup_dependent_build_environment(self, env, dependent_spec): # NB: This function is overwritten by 'mpi' provider packages: # diff --git a/lib/spack/spack/build_systems/python.py b/lib/spack/spack/build_systems/python.py index 99dc4ce0dc..76159d88a1 100644 --- a/lib/spack/spack/build_systems/python.py +++ b/lib/spack/spack/build_systems/python.py @@ -89,7 +89,7 @@ class PythonPackage(PackageBase): build_system_class = 'PythonPackage' #: Callback names for build-time test - build_time_test_callbacks = ['test'] + build_time_test_callbacks = ['build_test'] #: Callback names for install-time test install_time_test_callbacks = ['import_module_test'] @@ -359,7 +359,7 @@ class PythonPackage(PackageBase): # Testing - def test(self): + def build_test(self): """Run unit tests after in-place build. These tests are only run if the package actually has a 'test' command. diff --git a/lib/spack/spack/build_systems/scons.py b/lib/spack/spack/build_systems/scons.py index ffa4390249..5e17666b71 100644 --- a/lib/spack/spack/build_systems/scons.py +++ b/lib/spack/spack/build_systems/scons.py @@ -33,7 +33,7 @@ class SConsPackage(PackageBase): build_system_class = 'SConsPackage' #: Callback names for build-time test - build_time_test_callbacks = ['test'] + build_time_test_callbacks = ['build_test'] depends_on('scons', type='build') @@ -59,7 +59,7 @@ class SConsPackage(PackageBase): # Testing - def test(self): + def build_test(self): """Run unit tests after build. By default, does nothing. Override this if you want to diff --git a/lib/spack/spack/build_systems/waf.py b/lib/spack/spack/build_systems/waf.py index a1581660f2..a6dbbbdb35 100644 --- a/lib/spack/spack/build_systems/waf.py +++ b/lib/spack/spack/build_systems/waf.py @@ -47,10 +47,10 @@ class WafPackage(PackageBase): build_system_class = 'WafPackage' # Callback names for build-time test - build_time_test_callbacks = ['test'] + build_time_test_callbacks = ['build_test'] # Callback names for install-time test - install_time_test_callbacks = ['installtest'] + install_time_test_callbacks = ['install_test'] # Much like AutotoolsPackage does not require automake and autoconf # to build, WafPackage does not require waf to build. It only requires @@ -106,7 +106,7 @@ class WafPackage(PackageBase): # Testing - def test(self): + def build_test(self): """Run unit tests after build. By default, does nothing. Override this if you want to @@ -116,7 +116,7 @@ class WafPackage(PackageBase): run_after('build')(PackageBase._run_default_build_time_test_callbacks) - def installtest(self): + def install_test(self): """Run unit tests after install. By default, does nothing. Override this if you want to diff --git a/lib/spack/spack/cmd/build_env.py b/lib/spack/spack/cmd/build_env.py index 128d167a29..ef8b1f6e6b 100644 --- a/lib/spack/spack/cmd/build_env.py +++ b/lib/spack/spack/cmd/build_env.py @@ -2,86 +2,15 @@ # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) - -from __future__ import print_function - -import argparse -import os - -import llnl.util.tty as tty -import spack.build_environment as build_environment -import spack.cmd -import spack.cmd.common.arguments as arguments -from spack.util.environment import dump_environment, pickle_environment +import spack.cmd.common.env_utility as env_utility description = "run a command in a spec's install environment, " \ "or dump its environment to screen or file" section = "build" level = "long" - -def setup_parser(subparser): - arguments.add_common_arguments(subparser, ['clean', 'dirty']) - subparser.add_argument( - '--dump', metavar="FILE", - help="dump a source-able environment to FILE" - ) - subparser.add_argument( - '--pickle', metavar="FILE", - help="dump a pickled source-able environment to FILE" - ) - subparser.add_argument( - 'spec', nargs=argparse.REMAINDER, - metavar='spec [--] [cmd]...', - help="spec of package environment to emulate") - subparser.epilog\ - = 'If a command is not specified, the environment will be printed ' \ - 'to standard output (cf /usr/bin/env) unless --dump and/or --pickle ' \ - 'are specified.\n\nIf a command is specified and spec is ' \ - 'multi-word, then the -- separator is obligatory.' +setup_parser = env_utility.setup_parser def build_env(parser, args): - if not args.spec: - tty.die("spack build-env requires a spec.") - - # Specs may have spaces in them, so if they do, require that the - # caller put a '--' between the spec and the command to be - # executed. If there is no '--', assume that the spec is the - # first argument. - sep = '--' - if sep in args.spec: - s = args.spec.index(sep) - spec = args.spec[:s] - cmd = args.spec[s + 1:] - else: - spec = args.spec[0] - cmd = args.spec[1:] - - specs = spack.cmd.parse_specs(spec, concretize=True) - if len(specs) > 1: - tty.die("spack build-env only takes one spec.") - spec = specs[0] - - build_environment.setup_package(spec.package, args.dirty) - - if args.dump: - # Dump a source-able environment to a text file. - tty.msg("Dumping a source-able environment to {0}".format(args.dump)) - dump_environment(args.dump) - - if args.pickle: - # Dump a source-able environment to a pickle file. - tty.msg( - "Pickling a source-able environment to {0}".format(args.pickle)) - pickle_environment(args.pickle) - - if cmd: - # Execute the command with the new environment - os.execvp(cmd[0], cmd) - - elif not bool(args.pickle or args.dump): - # If no command or dump/pickle option act like the "env" command - # and print out env vars. - for key, val in os.environ.items(): - print("%s=%s" % (key, val)) + env_utility.emulate_env_utility('build-env', 'build', args) diff --git a/lib/spack/spack/cmd/clean.py b/lib/spack/spack/cmd/clean.py index d847e7a7c0..f69b959293 100644 --- a/lib/spack/spack/cmd/clean.py +++ b/lib/spack/spack/cmd/clean.py @@ -10,10 +10,11 @@ import shutil import llnl.util.tty as tty import spack.caches -import spack.cmd +import spack.cmd.test import spack.cmd.common.arguments as arguments import spack.repo import spack.stage +import spack.config from spack.paths import lib_path, var_path diff --git a/lib/spack/spack/cmd/common/arguments.py b/lib/spack/spack/cmd/common/arguments.py index e5945bda9c..e5c4c0dde8 100644 --- a/lib/spack/spack/cmd/common/arguments.py +++ b/lib/spack/spack/cmd/common/arguments.py @@ -275,3 +275,53 @@ def no_checksum(): return Args( '-n', '--no-checksum', action='store_true', default=False, help="do not use checksums to verify downloaded files (unsafe)") + + +def add_cdash_args(subparser, add_help): + cdash_help = {} + if add_help: + cdash_help['upload-url'] = "CDash URL where reports will be uploaded" + cdash_help['build'] = """The name of the build that will be reported to CDash. +Defaults to spec of the package to operate on.""" + cdash_help['site'] = """The site name that will be reported to CDash. +Defaults to current system hostname.""" + cdash_help['track'] = """Results will be reported to this group on CDash. +Defaults to Experimental.""" + cdash_help['buildstamp'] = """Instead of letting the CDash reporter prepare the +buildstamp which, when combined with build name, site and project, +uniquely identifies the build, provide this argument to identify +the build yourself. Format: %%Y%%m%%d-%%H%%M-[cdash-track]""" + else: + cdash_help['upload-url'] = argparse.SUPPRESS + cdash_help['build'] = argparse.SUPPRESS + cdash_help['site'] = argparse.SUPPRESS + cdash_help['track'] = argparse.SUPPRESS + cdash_help['buildstamp'] = argparse.SUPPRESS + + subparser.add_argument( + '--cdash-upload-url', + default=None, + help=cdash_help['upload-url'] + ) + subparser.add_argument( + '--cdash-build', + default=None, + help=cdash_help['build'] + ) + subparser.add_argument( + '--cdash-site', + default=None, + help=cdash_help['site'] + ) + + cdash_subgroup = subparser.add_mutually_exclusive_group() + cdash_subgroup.add_argument( + '--cdash-track', + default='Experimental', + help=cdash_help['track'] + ) + cdash_subgroup.add_argument( + '--cdash-buildstamp', + default=None, + help=cdash_help['buildstamp'] + ) diff --git a/lib/spack/spack/cmd/common/env_utility.py b/lib/spack/spack/cmd/common/env_utility.py new file mode 100644 index 0000000000..e3f32737b4 --- /dev/null +++ b/lib/spack/spack/cmd/common/env_utility.py @@ -0,0 +1,82 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) +from __future__ import print_function + +import argparse +import os + +import llnl.util.tty as tty +import spack.build_environment as build_environment +import spack.paths +import spack.cmd +import spack.cmd.common.arguments as arguments +from spack.util.environment import dump_environment, pickle_environment + + +def setup_parser(subparser): + arguments.add_common_arguments(subparser, ['clean', 'dirty']) + subparser.add_argument( + '--dump', metavar="FILE", + help="dump a source-able environment to FILE" + ) + subparser.add_argument( + '--pickle', metavar="FILE", + help="dump a pickled source-able environment to FILE" + ) + subparser.add_argument( + 'spec', nargs=argparse.REMAINDER, + metavar='spec [--] [cmd]...', + help="specs of package environment to emulate") + subparser.epilog\ + = 'If a command is not specified, the environment will be printed ' \ + 'to standard output (cf /usr/bin/env) unless --dump and/or --pickle ' \ + 'are specified.\n\nIf a command is specified and spec is ' \ + 'multi-word, then the -- separator is obligatory.' + + +def emulate_env_utility(cmd_name, context, args): + if not args.spec: + tty.die("spack %s requires a spec." % cmd_name) + + # Specs may have spaces in them, so if they do, require that the + # caller put a '--' between the spec and the command to be + # executed. If there is no '--', assume that the spec is the + # first argument. + sep = '--' + if sep in args.spec: + s = args.spec.index(sep) + spec = args.spec[:s] + cmd = args.spec[s + 1:] + else: + spec = args.spec[0] + cmd = args.spec[1:] + + specs = spack.cmd.parse_specs(spec, concretize=True) + if len(specs) > 1: + tty.die("spack %s only takes one spec." % cmd_name) + spec = specs[0] + + build_environment.setup_package(spec.package, args.dirty, context) + + if args.dump: + # Dump a source-able environment to a text file. + tty.msg("Dumping a source-able environment to {0}".format(args.dump)) + dump_environment(args.dump) + + if args.pickle: + # Dump a source-able environment to a pickle file. + tty.msg( + "Pickling a source-able environment to {0}".format(args.pickle)) + pickle_environment(args.pickle) + + if cmd: + # Execute the command with the new environment + os.execvp(cmd[0], cmd) + + elif not bool(args.pickle or args.dump): + # If no command or dump/pickle option then act like the "env" command + # and print out env vars. + for key, val in os.environ.items(): + print("%s=%s" % (key, val)) diff --git a/lib/spack/spack/cmd/install.py b/lib/spack/spack/cmd/install.py index c8673b5330..3b5954b1ad 100644 --- a/lib/spack/spack/cmd/install.py +++ b/lib/spack/spack/cmd/install.py @@ -167,65 +167,8 @@ packages. If neither are chosen, don't run tests for any packages.""" action='store_true', help="Show usage instructions for CDash reporting" ) - subparser.add_argument( - '-y', '--yes-to-all', - action='store_true', - dest='yes_to_all', - help="""assume "yes" is the answer to every confirmation request. -To run completely non-interactively, also specify '--no-checksum'.""" - ) - add_cdash_args(subparser, False) - arguments.add_common_arguments(subparser, ['spec']) - - -def add_cdash_args(subparser, add_help): - cdash_help = {} - if add_help: - cdash_help['upload-url'] = "CDash URL where reports will be uploaded" - cdash_help['build'] = """The name of the build that will be reported to CDash. -Defaults to spec of the package to install.""" - cdash_help['site'] = """The site name that will be reported to CDash. -Defaults to current system hostname.""" - cdash_help['track'] = """Results will be reported to this group on CDash. -Defaults to Experimental.""" - cdash_help['buildstamp'] = """Instead of letting the CDash reporter prepare the -buildstamp which, when combined with build name, site and project, -uniquely identifies the build, provide this argument to identify -the build yourself. Format: %%Y%%m%%d-%%H%%M-[cdash-track]""" - else: - cdash_help['upload-url'] = argparse.SUPPRESS - cdash_help['build'] = argparse.SUPPRESS - cdash_help['site'] = argparse.SUPPRESS - cdash_help['track'] = argparse.SUPPRESS - cdash_help['buildstamp'] = argparse.SUPPRESS - - subparser.add_argument( - '--cdash-upload-url', - default=None, - help=cdash_help['upload-url'] - ) - subparser.add_argument( - '--cdash-build', - default=None, - help=cdash_help['build'] - ) - subparser.add_argument( - '--cdash-site', - default=None, - help=cdash_help['site'] - ) - - cdash_subgroup = subparser.add_mutually_exclusive_group() - cdash_subgroup.add_argument( - '--cdash-track', - default='Experimental', - help=cdash_help['track'] - ) - cdash_subgroup.add_argument( - '--cdash-buildstamp', - default=None, - help=cdash_help['buildstamp'] - ) + arguments.add_cdash_args(subparser, False) + arguments.add_common_arguments(subparser, ['yes_to_all', 'spec']) def default_log_file(spec): @@ -283,11 +226,12 @@ environment variables: SPACK_CDASH_AUTH_TOKEN authentication token to present to CDash ''')) - add_cdash_args(parser, True) + arguments.add_cdash_args(parser, True) parser.print_help() return - reporter = spack.report.collect_info(args.log_format, args) + reporter = spack.report.collect_info( + spack.package.PackageInstaller, '_install_task', args.log_format, args) if args.log_file: reporter.filename = args.log_file @@ -383,7 +327,7 @@ environment variables: if not args.log_file and not reporter.filename: reporter.filename = default_log_file(specs[0]) reporter.specs = specs - with reporter: + with reporter('build'): if args.overwrite: installed = list(filter(lambda x: x, diff --git a/lib/spack/spack/cmd/list.py b/lib/spack/spack/cmd/list.py index 656282599d..a7d4cb0ac3 100644 --- a/lib/spack/spack/cmd/list.py +++ b/lib/spack/spack/cmd/list.py @@ -54,6 +54,9 @@ def setup_parser(subparser): subparser.add_argument( '--update', metavar='FILE', default=None, action='store', help='write output to the specified file, if any package is newer') + subparser.add_argument( + '-v', '--virtuals', action='store_true', default=False, + help='include virtual packages in list') arguments.add_common_arguments(subparser, ['tags']) @@ -267,7 +270,7 @@ def list(parser, args): formatter = formatters[args.format] # Retrieve the names of all the packages - pkgs = set(spack.repo.all_package_names()) + pkgs = set(spack.repo.all_package_names(args.virtuals)) # Filter the set appropriately sorted_packages = filter_by_name(pkgs, args) diff --git a/lib/spack/spack/cmd/test.py b/lib/spack/spack/cmd/test.py index 8cbc0fbccf..3362b8a109 100644 --- a/lib/spack/spack/cmd/test.py +++ b/lib/spack/spack/cmd/test.py @@ -4,166 +4,381 @@ # SPDX-License-Identifier: (Apache-2.0 OR MIT) from __future__ import print_function -from __future__ import division - -import collections -import sys -import re +import os import argparse -import pytest -from six import StringIO +import textwrap +import fnmatch +import re +import shutil -import llnl.util.tty.color as color -from llnl.util.filesystem import working_dir -from llnl.util.tty.colify import colify +import llnl.util.tty as tty -import spack.paths +import spack.install_test +import spack.environment as ev +import spack.cmd +import spack.cmd.common.arguments as arguments +import spack.report +import spack.package -description = "run spack's unit tests (wrapper around pytest)" -section = "developer" +description = "run spack's tests for an install" +section = "administrator" level = "long" +def first_line(docstring): + """Return the first line of the docstring.""" + return docstring.split('\n')[0] + + def setup_parser(subparser): - subparser.add_argument( - '-H', '--pytest-help', action='store_true', default=False, - help="show full pytest help, with advanced options") - - # extra spack arguments to list tests - list_group = subparser.add_argument_group("listing tests") - list_mutex = list_group.add_mutually_exclusive_group() - list_mutex.add_argument( - '-l', '--list', action='store_const', default=None, - dest='list', const='list', help="list test filenames") - list_mutex.add_argument( - '-L', '--list-long', action='store_const', default=None, - dest='list', const='long', help="list all test functions") - list_mutex.add_argument( - '-N', '--list-names', action='store_const', default=None, - dest='list', const='names', help="list full names of all tests") - - # use tests for extension - subparser.add_argument( - '--extension', default=None, - help="run test for a given spack extension") - - # spell out some common pytest arguments, so they'll show up in help - pytest_group = subparser.add_argument_group( - "common pytest arguments (spack test --pytest-help for more details)") - pytest_group.add_argument( - "-s", action='append_const', dest='parsed_args', const='-s', - help="print output while tests run (disable capture)") - pytest_group.add_argument( - "-k", action='store', metavar="EXPRESSION", dest='expression', - help="filter tests by keyword (can also use w/list options)") - pytest_group.add_argument( - "--showlocals", action='append_const', dest='parsed_args', - const='--showlocals', help="show local variable values in tracebacks") - - # remainder is just passed to pytest - subparser.add_argument( - 'pytest_args', nargs=argparse.REMAINDER, help="arguments for pytest") - - -def do_list(args, extra_args): - """Print a lists of tests than what pytest offers.""" - # Run test collection and get the tree out. - old_output = sys.stdout - try: - sys.stdout = output = StringIO() - pytest.main(['--collect-only'] + extra_args) - finally: - sys.stdout = old_output - - lines = output.getvalue().split('\n') - tests = collections.defaultdict(lambda: set()) - prefix = [] - - # collect tests into sections - for line in lines: - match = re.match(r"(\s*)<([^ ]*) '([^']*)'", line) - if not match: - continue - indent, nodetype, name = match.groups() - - # strip parametrized tests - if "[" in name: - name = name[:name.index("[")] - - depth = len(indent) // 2 - - if nodetype.endswith("Function"): - key = tuple(prefix) - tests[key].add(name) - else: - prefix = prefix[:depth] - prefix.append(name) - - def colorize(c, prefix): - if isinstance(prefix, tuple): - return "::".join( - color.colorize("@%s{%s}" % (c, p)) - for p in prefix if p != "()" - ) - return color.colorize("@%s{%s}" % (c, prefix)) - - if args.list == "list": - files = set(prefix[0] for prefix in tests) - color_files = [colorize("B", file) for file in sorted(files)] - colify(color_files) - - elif args.list == "long": - for prefix, functions in sorted(tests.items()): - path = colorize("*B", prefix) + "::" - functions = [colorize("c", f) for f in sorted(functions)] - color.cprint(path) - colify(functions, indent=4) - print() - - else: # args.list == "names" - all_functions = [ - colorize("*B", prefix) + "::" + colorize("c", f) - for prefix, functions in sorted(tests.items()) - for f in sorted(functions) - ] - colify(all_functions) - - -def add_back_pytest_args(args, unknown_args): - """Add parsed pytest args, unknown args, and remainder together. - - We add some basic pytest arguments to the Spack parser to ensure that - they show up in the short help, so we have to reassemble things here. + sp = subparser.add_subparsers(metavar='SUBCOMMAND', dest='test_command') + + # Run + run_parser = sp.add_parser('run', description=test_run.__doc__, + help=first_line(test_run.__doc__)) + + alias_help_msg = "Provide an alias for this test-suite" + alias_help_msg += " for subsequent access." + run_parser.add_argument('--alias', help=alias_help_msg) + + run_parser.add_argument( + '--fail-fast', action='store_true', + help="Stop tests for each package after the first failure." + ) + run_parser.add_argument( + '--fail-first', action='store_true', + help="Stop after the first failed package." + ) + run_parser.add_argument( + '--keep-stage', + action='store_true', + help='Keep testing directory for debugging' + ) + run_parser.add_argument( + '--log-format', + default=None, + choices=spack.report.valid_formats, + help="format to be used for log files" + ) + run_parser.add_argument( + '--log-file', + default=None, + help="filename for the log file. if not passed a default will be used" + ) + arguments.add_cdash_args(run_parser, False) + run_parser.add_argument( + '--help-cdash', + action='store_true', + help="Show usage instructions for CDash reporting" + ) + + cd_group = run_parser.add_mutually_exclusive_group() + arguments.add_common_arguments(cd_group, ['clean', 'dirty']) + + arguments.add_common_arguments(run_parser, ['installed_specs']) + + # List + sp.add_parser('list', description=test_list.__doc__, + help=first_line(test_list.__doc__)) + + # Find + find_parser = sp.add_parser('find', description=test_find.__doc__, + help=first_line(test_find.__doc__)) + find_parser.add_argument( + 'filter', nargs=argparse.REMAINDER, + help='optional case-insensitive glob patterns to filter results.') + + # Status + status_parser = sp.add_parser('status', description=test_status.__doc__, + help=first_line(test_status.__doc__)) + status_parser.add_argument( + 'names', nargs=argparse.REMAINDER, + help="Test suites for which to print status") + + # Results + results_parser = sp.add_parser('results', description=test_results.__doc__, + help=first_line(test_results.__doc__)) + results_parser.add_argument( + '-l', '--logs', action='store_true', + help="print the test log for each matching package") + results_parser.add_argument( + '-f', '--failed', action='store_true', + help="only show results for failed tests of matching packages") + results_parser.add_argument( + 'names', nargs=argparse.REMAINDER, + metavar='[name(s)] [-- installed_specs]...', + help="suite names and installed package constraints") + results_parser.epilog = 'Test results will be filtered by space-' \ + 'separated suite name(s) and installed\nspecs when provided. '\ + 'If names are provided, then only results for those test\nsuites '\ + 'will be shown. If installed specs are provided, then ony results'\ + '\nmatching those specs will be shown.' + + # Remove + remove_parser = sp.add_parser('remove', description=test_remove.__doc__, + help=first_line(test_remove.__doc__)) + arguments.add_common_arguments(remove_parser, ['yes_to_all']) + remove_parser.add_argument( + 'names', nargs=argparse.REMAINDER, + help="Test suites to remove from test stage") + + +def test_run(args): + """Run tests for the specified installed packages. + + If no specs are listed, run tests for all packages in the current + environment or all installed packages if there is no active environment. """ - result = args.parsed_args or [] - result += unknown_args or [] - result += args.pytest_args or [] - if args.expression: - result += ["-k", args.expression] - return result - - -def test(parser, args, unknown_args): - if args.pytest_help: - # make the pytest.main help output more accurate - sys.argv[0] = 'spack test' - return pytest.main(['-h']) - - # add back any parsed pytest args we need to pass to pytest - pytest_args = add_back_pytest_args(args, unknown_args) - - # The default is to test the core of Spack. If the option `--extension` - # has been used, then test that extension. - pytest_root = spack.paths.spack_root - if args.extension: - target = args.extension - extensions = spack.config.get('config:extensions') - pytest_root = spack.extensions.path_for_extension(target, *extensions) - - # pytest.ini lives in the root of the spack repository. - with working_dir(pytest_root): - if args.list: - do_list(args, pytest_args) + # cdash help option + if args.help_cdash: + parser = argparse.ArgumentParser( + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=textwrap.dedent('''\ +environment variables: + SPACK_CDASH_AUTH_TOKEN + authentication token to present to CDash + ''')) + arguments.add_cdash_args(parser, True) + parser.print_help() + return + + # set config option for fail-fast + if args.fail_fast: + spack.config.set('config:fail_fast', True, scope='command_line') + + # Get specs to test + env = ev.get_env(args, 'test') + hashes = env.all_hashes() if env else None + + specs = spack.cmd.parse_specs(args.specs) if args.specs else [None] + specs_to_test = [] + for spec in specs: + matching = spack.store.db.query_local(spec, hashes=hashes) + if spec and not matching: + tty.warn("No installed packages match spec %s" % spec) + specs_to_test.extend(matching) + + # test_stage_dir + test_suite = spack.install_test.TestSuite(specs_to_test, args.alias) + test_suite.ensure_stage() + tty.msg("Spack test %s" % test_suite.name) + + # Set up reporter + setattr(args, 'package', [s.format() for s in test_suite.specs]) + reporter = spack.report.collect_info( + spack.package.PackageBase, 'do_test', args.log_format, args) + if not reporter.filename: + if args.log_file: + if os.path.isabs(args.log_file): + log_file = args.log_file + else: + log_dir = os.getcwd() + log_file = os.path.join(log_dir, args.log_file) + else: + log_file = os.path.join( + os.getcwd(), + 'test-%s' % test_suite.name) + reporter.filename = log_file + reporter.specs = specs_to_test + + with reporter('test', test_suite.stage): + test_suite(remove_directory=not args.keep_stage, + dirty=args.dirty, + fail_first=args.fail_first) + + +def has_test_method(pkg): + return pkg.test.__func__ != spack.package.PackageBase.test + + +def test_list(args): + """List all installed packages with available tests.""" + # TODO: This can be extended to have all of the output formatting options + # from `spack find`. + env = ev.get_env(args, 'test') + hashes = env.all_hashes() if env else None + + specs = spack.store.db.query(hashes=hashes) + specs = list(filter(lambda s: has_test_method(s.package), specs)) + + spack.cmd.display_specs(specs, long=True) + + +def test_find(args): # TODO: merge with status (noargs) + """Find tests that are running or have available results. + + Displays aliases for tests that have them, otherwise test suite content + hashes.""" + test_suites = spack.install_test.get_all_test_suites() + + # Filter tests by filter argument + if args.filter: + def create_filter(f): + raw = fnmatch.translate('f' if '*' in f or '?' in f + else '*' + f + '*') + return re.compile(raw, flags=re.IGNORECASE) + filters = [create_filter(f) for f in args.filter] + + def match(t, f): + return f.match(t) + test_suites = [t for t in test_suites + if any(match(t.alias, f) for f in filters) and + os.path.isdir(t.stage)] + + names = [t.name for t in test_suites] + + if names: + # TODO: Make these specify results vs active + msg = "Spack test results available for the following tests:\n" + msg += " %s\n" % ' '.join(names) + msg += " Run `spack test remove` to remove all tests" + tty.msg(msg) + else: + msg = "No test results match the query\n" + msg += " Tests may have been removed using `spack test remove`" + tty.msg(msg) + + +def test_status(args): + """Get the current status for the specified Spack test suite(s).""" + if args.names: + test_suites = [] + for name in args.names: + test_suite = spack.install_test.get_test_suite(name) + if test_suite: + test_suites.append(test_suite) + else: + tty.msg("No test suite %s found in test stage" % name) + else: + test_suites = spack.install_test.get_all_test_suites() + if not test_suites: + tty.msg("No test suites with status to report") + + for test_suite in test_suites: + # TODO: Make this handle capability tests too + # TODO: Make this handle tests running in another process + tty.msg("Test suite %s completed" % test_suite.name) + + +def _report_suite_results(test_suite, args, constraints): + """Report the relevant test suite results.""" + + # TODO: Make this handle capability tests too + # The results file may turn out to be a placeholder for future work + + if constraints: + # TBD: Should I be refactoring or re-using ConstraintAction? + qspecs = spack.cmd.parse_specs(constraints) + specs = {} + for spec in qspecs: + for s in spack.store.db.query(spec, installed=True): + specs[s.dag_hash()] = s + specs = sorted(specs.values()) + test_specs = dict((test_suite.test_pkg_id(s), s) for s in + test_suite.specs if s in specs) + else: + test_specs = dict((test_suite.test_pkg_id(s), s) for s in + test_suite.specs) + + if not test_specs: + return + + if os.path.exists(test_suite.results_file): + results_desc = 'Failing results' if args.failed else 'Results' + matching = ", spec matching '{0}'".format(' '.join(constraints)) \ + if constraints else '' + tty.msg("{0} for test suite '{1}'{2}:" + .format(results_desc, test_suite.name, matching)) + + results = {} + with open(test_suite.results_file, 'r') as f: + for line in f: + pkg_id, status = line.split() + results[pkg_id] = status + + for pkg_id in test_specs: + if pkg_id in results: + status = results[pkg_id] + if args.failed and status != 'FAILED': + continue + + msg = " {0} {1}".format(pkg_id, status) + if args.logs: + spec = test_specs[pkg_id] + log_file = test_suite.log_file_for_spec(spec) + if os.path.isfile(log_file): + with open(log_file, 'r') as f: + msg += '\n{0}'.format(''.join(f.readlines())) + tty.msg(msg) + else: + msg = "Test %s has no results.\n" % test_suite.name + msg += " Check if it is running with " + msg += "`spack test status %s`" % test_suite.name + tty.msg(msg) + + +def test_results(args): + """Get the results from Spack test suite(s) (default all).""" + if args.names: + try: + sep_index = args.names.index('--') + names = args.names[:sep_index] + constraints = args.names[sep_index + 1:] + except ValueError: + names = args.names + constraints = None + else: + names, constraints = None, None + + if names: + test_suites = [spack.install_test.get_test_suite(name) for name + in names] + if not test_suites: + tty.msg('No test suite(s) found in test stage: {0}' + .format(', '.join(names))) + else: + test_suites = spack.install_test.get_all_test_suites() + if not test_suites: + tty.msg("No test suites with results to report") + + for test_suite in test_suites: + _report_suite_results(test_suite, args, constraints) + + +def test_remove(args): + """Remove results from Spack test suite(s) (default all). + + If no test suite is listed, remove results for all suites. + + Removed tests can no longer be accessed for results or status, and will not + appear in `spack test list` results.""" + if args.names: + test_suites = [] + for name in args.names: + test_suite = spack.install_test.get_test_suite(name) + if test_suite: + test_suites.append(test_suite) + else: + tty.msg("No test suite %s found in test stage" % name) + else: + test_suites = spack.install_test.get_all_test_suites() + + if not test_suites: + tty.msg("No test suites to remove") + return + + if not args.yes_to_all: + msg = 'The following test suites will be removed:\n\n' + msg += ' ' + ' '.join(test.name for test in test_suites) + '\n' + tty.msg(msg) + answer = tty.get_yes_or_no('Do you want to proceed?', default=False) + if not answer: + tty.msg('Aborting removal of test suites') return - return pytest.main(pytest_args) + for test_suite in test_suites: + shutil.rmtree(test_suite.stage) + + +def test(parser, args): + globals()['test_%s' % args.test_command](args) diff --git a/lib/spack/spack/cmd/test_env.py b/lib/spack/spack/cmd/test_env.py new file mode 100644 index 0000000000..61e85046c1 --- /dev/null +++ b/lib/spack/spack/cmd/test_env.py @@ -0,0 +1,16 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) +import spack.cmd.common.env_utility as env_utility + +description = "run a command in a spec's test environment, " \ + "or dump its environment to screen or file" +section = "administration" +level = "long" + +setup_parser = env_utility.setup_parser + + +def test_env(parser, args): + env_utility.emulate_env_utility('test-env', 'test', args) diff --git a/lib/spack/spack/cmd/unit_test.py b/lib/spack/spack/cmd/unit_test.py new file mode 100644 index 0000000000..509211de04 --- /dev/null +++ b/lib/spack/spack/cmd/unit_test.py @@ -0,0 +1,169 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +from __future__ import print_function +from __future__ import division + +import collections +import sys +import re +import argparse +import pytest +from six import StringIO + +import llnl.util.tty.color as color +from llnl.util.filesystem import working_dir +from llnl.util.tty.colify import colify + +import spack.paths + +description = "run spack's unit tests (wrapper around pytest)" +section = "developer" +level = "long" + + +def setup_parser(subparser): + subparser.add_argument( + '-H', '--pytest-help', action='store_true', default=False, + help="show full pytest help, with advanced options") + + # extra spack arguments to list tests + list_group = subparser.add_argument_group("listing tests") + list_mutex = list_group.add_mutually_exclusive_group() + list_mutex.add_argument( + '-l', '--list', action='store_const', default=None, + dest='list', const='list', help="list test filenames") + list_mutex.add_argument( + '-L', '--list-long', action='store_const', default=None, + dest='list', const='long', help="list all test functions") + list_mutex.add_argument( + '-N', '--list-names', action='store_const', default=None, + dest='list', const='names', help="list full names of all tests") + + # use tests for extension + subparser.add_argument( + '--extension', default=None, + help="run test for a given spack extension") + + # spell out some common pytest arguments, so they'll show up in help + pytest_group = subparser.add_argument_group( + "common pytest arguments (spack unit-test --pytest-help for more)") + pytest_group.add_argument( + "-s", action='append_const', dest='parsed_args', const='-s', + help="print output while tests run (disable capture)") + pytest_group.add_argument( + "-k", action='store', metavar="EXPRESSION", dest='expression', + help="filter tests by keyword (can also use w/list options)") + pytest_group.add_argument( + "--showlocals", action='append_const', dest='parsed_args', + const='--showlocals', help="show local variable values in tracebacks") + + # remainder is just passed to pytest + subparser.add_argument( + 'pytest_args', nargs=argparse.REMAINDER, help="arguments for pytest") + + +def do_list(args, extra_args): + """Print a lists of tests than what pytest offers.""" + # Run test collection and get the tree out. + old_output = sys.stdout + try: + sys.stdout = output = StringIO() + pytest.main(['--collect-only'] + extra_args) + finally: + sys.stdout = old_output + + lines = output.getvalue().split('\n') + tests = collections.defaultdict(lambda: set()) + prefix = [] + + # collect tests into sections + for line in lines: + match = re.match(r"(\s*)<([^ ]*) '([^']*)'", line) + if not match: + continue + indent, nodetype, name = match.groups() + + # strip parametrized tests + if "[" in name: + name = name[:name.index("[")] + + depth = len(indent) // 2 + + if nodetype.endswith("Function"): + key = tuple(prefix) + tests[key].add(name) + else: + prefix = prefix[:depth] + prefix.append(name) + + def colorize(c, prefix): + if isinstance(prefix, tuple): + return "::".join( + color.colorize("@%s{%s}" % (c, p)) + for p in prefix if p != "()" + ) + return color.colorize("@%s{%s}" % (c, prefix)) + + if args.list == "list": + files = set(prefix[0] for prefix in tests) + color_files = [colorize("B", file) for file in sorted(files)] + colify(color_files) + + elif args.list == "long": + for prefix, functions in sorted(tests.items()): + path = colorize("*B", prefix) + "::" + functions = [colorize("c", f) for f in sorted(functions)] + color.cprint(path) + colify(functions, indent=4) + print() + + else: # args.list == "names" + all_functions = [ + colorize("*B", prefix) + "::" + colorize("c", f) + for prefix, functions in sorted(tests.items()) + for f in sorted(functions) + ] + colify(all_functions) + + +def add_back_pytest_args(args, unknown_args): + """Add parsed pytest args, unknown args, and remainder together. + + We add some basic pytest arguments to the Spack parser to ensure that + they show up in the short help, so we have to reassemble things here. + """ + result = args.parsed_args or [] + result += unknown_args or [] + result += args.pytest_args or [] + if args.expression: + result += ["-k", args.expression] + return result + + +def unit_test(parser, args, unknown_args): + if args.pytest_help: + # make the pytest.main help output more accurate + sys.argv[0] = 'spack test' + return pytest.main(['-h']) + + # add back any parsed pytest args we need to pass to pytest + pytest_args = add_back_pytest_args(args, unknown_args) + + # The default is to test the core of Spack. If the option `--extension` + # has been used, then test that extension. + pytest_root = spack.paths.spack_root + if args.extension: + target = args.extension + extensions = spack.config.get('config:extensions') + pytest_root = spack.extensions.path_for_extension(target, *extensions) + + # pytest.ini lives in the root of the spack repository. + with working_dir(pytest_root): + if args.list: + do_list(args, pytest_args) + return + + return pytest.main(pytest_args) diff --git a/lib/spack/spack/directives.py b/lib/spack/spack/directives.py index 7b2084d229..41276b5b48 100644 --- a/lib/spack/spack/directives.py +++ b/lib/spack/spack/directives.py @@ -299,8 +299,18 @@ def _depends_on(pkg, spec, when=None, type=default_deptype, patches=None): # call this patches here for clarity -- we want patch to be a list, # but the caller doesn't have to make it one. - if patches and dep_spec.virtual: - raise DependencyPatchError("Cannot patch a virtual dependency.") + + # Note: we cannot check whether a package is virtual in a directive + # because directives are run as part of class instantiation, and specs + # instantiate the package class as part of the `virtual` check. + # To be technical, specs only instantiate the package class as part of the + # virtual check if the provider index hasn't been created yet. + # TODO: There could be a cache warming strategy that would allow us to + # ensure `Spec.virtual` is a valid thing to call in a directive. + # For now, we comment out the following check to allow for virtual packages + # with package files. + # if patches and dep_spec.virtual: + # raise DependencyPatchError("Cannot patch a virtual dependency.") # ensure patches is a list if patches is None: diff --git a/lib/spack/spack/install_test.py b/lib/spack/spack/install_test.py new file mode 100644 index 0000000000..6c2c095a2e --- /dev/null +++ b/lib/spack/spack/install_test.py @@ -0,0 +1,266 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) +import base64 +import hashlib +import os +import re +import shutil +import sys +import tty + +import llnl.util.filesystem as fs + +from spack.spec import Spec + +import spack.error +import spack.util.prefix +import spack.util.spack_json as sjson + + +test_suite_filename = 'test_suite.lock' +results_filename = 'results.txt' + + +def get_escaped_text_output(filename): + """Retrieve and escape the expected text output from the file + + Args: + filename (str): path to the file + + Returns: + (list of str): escaped text lines read from the file + """ + with open(filename, 'r') as f: + # Ensure special characters are escaped as needed + expected = f.read() + + # Split the lines to make it easier to debug failures when there is + # a lot of output + return [re.escape(ln) for ln in expected.split('\n')] + + +def get_test_stage_dir(): + return spack.util.path.canonicalize_path( + spack.config.get('config:test_stage', '~/.spack/test')) + + +def get_all_test_suites(): + stage_root = get_test_stage_dir() + if not os.path.isdir(stage_root): + return [] + + def valid_stage(d): + dirpath = os.path.join(stage_root, d) + return (os.path.isdir(dirpath) and + test_suite_filename in os.listdir(dirpath)) + + candidates = [ + os.path.join(stage_root, d, test_suite_filename) + for d in os.listdir(stage_root) + if valid_stage(d) + ] + + test_suites = [TestSuite.from_file(c) for c in candidates] + return test_suites + + +def get_test_suite(name): + assert name, "Cannot search for empty test name or 'None'" + test_suites = get_all_test_suites() + names = [ts for ts in test_suites + if ts.name == name] + assert len(names) < 2, "alias shadows test suite hash" + + if not names: + return None + return names[0] + + +class TestSuite(object): + def __init__(self, specs, alias=None): + # copy so that different test suites have different package objects + # even if they contain the same spec + self.specs = [spec.copy() for spec in specs] + self.current_test_spec = None # spec currently tested, can be virtual + self.current_base_spec = None # spec currently running do_test + + self.alias = alias + self._hash = None + + @property + def name(self): + return self.alias if self.alias else self.content_hash + + @property + def content_hash(self): + if not self._hash: + json_text = sjson.dump(self.to_dict()) + sha = hashlib.sha1(json_text.encode('utf-8')) + b32_hash = base64.b32encode(sha.digest()).lower() + if sys.version_info[0] >= 3: + b32_hash = b32_hash.decode('utf-8') + self._hash = b32_hash + return self._hash + + def __call__(self, *args, **kwargs): + self.write_reproducibility_data() + + remove_directory = kwargs.get('remove_directory', True) + dirty = kwargs.get('dirty', False) + fail_first = kwargs.get('fail_first', False) + + for spec in self.specs: + try: + msg = "A package object cannot run in two test suites at once" + assert not spec.package.test_suite, msg + + # Set up the test suite to know which test is running + spec.package.test_suite = self + self.current_base_spec = spec + self.current_test_spec = spec + + # setup per-test directory in the stage dir + test_dir = self.test_dir_for_spec(spec) + if os.path.exists(test_dir): + shutil.rmtree(test_dir) + fs.mkdirp(test_dir) + + # run the package tests + spec.package.do_test( + dirty=dirty + ) + + # Clean up on success and log passed test + if remove_directory: + shutil.rmtree(test_dir) + self.write_test_result(spec, 'PASSED') + except BaseException as exc: + if isinstance(exc, SyntaxError): + # Create the test log file and report the error. + self.ensure_stage() + msg = 'Testing package {0}\n{1}'\ + .format(self.test_pkg_id(spec), str(exc)) + _add_msg_to_file(self.log_file_for_spec(spec), msg) + + self.write_test_result(spec, 'FAILED') + if fail_first: + break + finally: + spec.package.test_suite = None + self.current_test_spec = None + self.current_base_spec = None + + def ensure_stage(self): + if not os.path.exists(self.stage): + fs.mkdirp(self.stage) + + @property + def stage(self): + return spack.util.prefix.Prefix( + os.path.join(get_test_stage_dir(), self.content_hash)) + + @property + def results_file(self): + return self.stage.join(results_filename) + + @classmethod + def test_pkg_id(cls, spec): + """Build the standard install test package identifier + + Args: + spec (Spec): instance of the spec under test + + Returns: + (str): the install test package identifier + """ + return spec.format('{name}-{version}-{hash:7}') + + @classmethod + def test_log_name(cls, spec): + return '%s-test-out.txt' % cls.test_pkg_id(spec) + + def log_file_for_spec(self, spec): + return self.stage.join(self.test_log_name(spec)) + + def test_dir_for_spec(self, spec): + return self.stage.join(self.test_pkg_id(spec)) + + @property + def current_test_data_dir(self): + assert self.current_test_spec and self.current_base_spec + test_spec = self.current_test_spec + base_spec = self.current_base_spec + return self.test_dir_for_spec(base_spec).data.join(test_spec.name) + + def add_failure(self, exc, msg): + current_hash = self.current_base_spec.dag_hash() + current_failures = self.failures.get(current_hash, []) + current_failures.append((exc, msg)) + self.failures[current_hash] = current_failures + + def write_test_result(self, spec, result): + msg = "{0} {1}".format(self.test_pkg_id(spec), result) + _add_msg_to_file(self.results_file, msg) + + def write_reproducibility_data(self): + for spec in self.specs: + repo_cache_path = self.stage.repo.join(spec.name) + spack.repo.path.dump_provenance(spec, repo_cache_path) + for vspec in spec.package.virtuals_provided: + repo_cache_path = self.stage.repo.join(vspec.name) + if not os.path.exists(repo_cache_path): + try: + spack.repo.path.dump_provenance(vspec, repo_cache_path) + except spack.repo.UnknownPackageError: + pass # not all virtuals have package files + + with open(self.stage.join(test_suite_filename), 'w') as f: + sjson.dump(self.to_dict(), stream=f) + + def to_dict(self): + specs = [s.to_dict() for s in self.specs] + d = {'specs': specs} + if self.alias: + d['alias'] = self.alias + return d + + @staticmethod + def from_dict(d): + specs = [Spec.from_dict(spec_dict) for spec_dict in d['specs']] + alias = d.get('alias', None) + return TestSuite(specs, alias) + + @staticmethod + def from_file(filename): + try: + with open(filename, 'r') as f: + data = sjson.load(f) + return TestSuite.from_dict(data) + except Exception as e: + tty.debug(e) + raise sjson.SpackJSONError("error parsing JSON TestSuite:", str(e)) + + +def _add_msg_to_file(filename, msg): + """Add the message to the specified file + + Args: + filename (str): path to the file + msg (str): message to be appended to the file + """ + with open(filename, 'a+') as f: + f.write('{0}\n'.format(msg)) + + +class TestFailure(spack.error.SpackError): + """Raised when package tests have failed for an installation.""" + def __init__(self, failures): + # Failures are all exceptions + msg = "%d tests failed.\n" % len(failures) + for failure, message in failures: + msg += '\n\n%s\n' % str(failure) + msg += '\n%s\n' % message + + super(TestFailure, self).__init__(msg) diff --git a/lib/spack/spack/installer.py b/lib/spack/spack/installer.py index cf7b5bfb99..dd35db0839 100644 --- a/lib/spack/spack/installer.py +++ b/lib/spack/spack/installer.py @@ -1610,12 +1610,12 @@ def build_process(pkg, kwargs): This function's return value is returned to the parent process. """ - keep_stage = kwargs.get('keep_stage', False) + fake = kwargs.get('fake', False) install_source = kwargs.get('install_source', False) + keep_stage = kwargs.get('keep_stage', False) skip_patch = kwargs.get('skip_patch', False) - verbose = kwargs.get('verbose', False) - fake = kwargs.get('fake', False) unmodified_env = kwargs.get('unmodified_env', {}) + verbose = kwargs.get('verbose', False) start_time = time.time() if not fake: @@ -1958,6 +1958,7 @@ class BuildRequest(object): def _add_default_args(self): """Ensure standard install options are set to at least the default.""" for arg, default in [('cache_only', False), + ('context', 'build'), # installs *always* build ('dirty', False), ('fail_fast', False), ('fake', False), diff --git a/lib/spack/spack/modules/lmod.py b/lib/spack/spack/modules/lmod.py index 018edb35ad..80f6933063 100644 --- a/lib/spack/spack/modules/lmod.py +++ b/lib/spack/spack/modules/lmod.py @@ -12,6 +12,7 @@ import collections import spack.config import spack.compilers import spack.spec +import spack.repo import spack.error import spack.tengine as tengine @@ -125,7 +126,9 @@ class LmodConfiguration(BaseConfiguration): # Check if all the tokens in the hierarchy are virtual specs. # If not warn the user and raise an error. - not_virtual = [t for t in tokens if not spack.spec.Spec.is_virtual(t)] + not_virtual = [t for t in tokens + if t != 'compiler' and + not spack.repo.path.is_virtual(t)] if not_virtual: msg = "Non-virtual specs in 'hierarchy' list for lmod: {0}\n" msg += "Please check the 'modules.yaml' configuration files" diff --git a/lib/spack/spack/package.py b/lib/spack/spack/package.py index b8dca6e55a..de394e2d45 100644 --- a/lib/spack/spack/package.py +++ b/lib/spack/spack/package.py @@ -24,10 +24,12 @@ import sys import textwrap import time import traceback - import six +import types +import llnl.util.filesystem as fsys import llnl.util.tty as tty + import spack.compilers import spack.config import spack.dependency @@ -45,15 +47,14 @@ import spack.store import spack.url import spack.util.environment import spack.util.web -from llnl.util.filesystem import mkdirp, touch, working_dir from llnl.util.lang import memoized from llnl.util.link_tree import LinkTree from ordereddict_backport import OrderedDict -from six import StringIO -from six import string_types -from six import with_metaclass from spack.filesystem_view import YamlFilesystemView from spack.installer import PackageInstaller, InstallError +from spack.install_test import TestFailure, TestSuite +from spack.util.executable import which, ProcessError +from spack.util.prefix import Prefix from spack.stage import stage_prefix, Stage, ResourceStage, StageComposite from spack.util.package_hash import package_hash from spack.version import Version @@ -452,7 +453,21 @@ class PackageViewMixin(object): view.remove_file(src, dst) -class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): +def test_log_pathname(test_stage, spec): + """Build the pathname of the test log file + + Args: + test_stage (str): path to the test stage directory + spec (Spec): instance of the spec under test + + Returns: + (str): the pathname of the test log file + """ + return os.path.join(test_stage, + 'test-{0}-out.txt'.format(TestSuite.test_pkg_id(spec))) + + +class PackageBase(six.with_metaclass(PackageMeta, PackageViewMixin, object)): """This is the superclass for all spack packages. ***The Package class*** @@ -542,6 +557,10 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): #: are executed or 'None' if there are no such test functions. build_time_test_callbacks = None + #: By default, packages are not virtual + #: Virtual packages override this attribute + virtual = False + #: Most Spack packages are used to install source or binary code while #: those that do not can be used to install a set of other Spack packages. has_code = True @@ -633,6 +652,18 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): metadata_attrs = ['homepage', 'url', 'urls', 'list_url', 'extendable', 'parallel', 'make_jobs'] + #: Boolean. If set to ``True``, the smoke/install test requires a compiler. + #: This is currently used by smoke tests to ensure a compiler is available + #: to build a custom test code. + test_requires_compiler = False + + #: List of test failures encountered during a smoke/install test run. + test_failures = None + + #: TestSuite instance used to manage smoke/install tests for one or more + #: specs. + test_suite = None + def __init__(self, spec): # this determines how the package should be built. self.spec = spec @@ -1002,19 +1033,22 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): return os.path.join(self.stage.path, _spack_build_envfile) @property + def metadata_dir(self): + """Return the install metadata directory.""" + return spack.store.layout.metadata_path(self.spec) + + @property def install_env_path(self): """ Return the build environment file path on successful installation. """ - install_path = spack.store.layout.metadata_path(self.spec) - # Backward compatibility: Return the name of an existing log path; # otherwise, return the current install env path name. - old_filename = os.path.join(install_path, 'build.env') + old_filename = os.path.join(self.metadata_dir, 'build.env') if os.path.exists(old_filename): return old_filename else: - return os.path.join(install_path, _spack_build_envfile) + return os.path.join(self.metadata_dir, _spack_build_envfile) @property def log_path(self): @@ -1031,16 +1065,14 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): @property def install_log_path(self): """Return the build log file path on successful installation.""" - install_path = spack.store.layout.metadata_path(self.spec) - # Backward compatibility: Return the name of an existing install log. for filename in ['build.out', 'build.txt']: - old_log = os.path.join(install_path, filename) + old_log = os.path.join(self.metadata_dir, filename) if os.path.exists(old_log): return old_log # Otherwise, return the current install log path name. - return os.path.join(install_path, _spack_build_logfile) + return os.path.join(self.metadata_dir, _spack_build_logfile) @property def configure_args_path(self): @@ -1050,9 +1082,12 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): @property def install_configure_args_path(self): """Return the configure args file path on successful installation.""" - install_path = spack.store.layout.metadata_path(self.spec) + return os.path.join(self.metadata_dir, _spack_configure_argsfile) - return os.path.join(install_path, _spack_configure_argsfile) + @property + def install_test_root(self): + """Return the install test root directory.""" + return os.path.join(self.metadata_dir, 'test') def _make_fetcher(self): # Construct a composite fetcher that always contains at least @@ -1322,7 +1357,7 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): raise FetchError("Archive was empty for %s" % self.name) else: # Support for post-install hooks requires a stage.source_path - mkdirp(self.stage.source_path) + fsys.mkdirp(self.stage.source_path) def do_patch(self): """Applies patches if they haven't been applied already.""" @@ -1368,7 +1403,7 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): patched = False for patch in patches: try: - with working_dir(self.stage.source_path): + with fsys.working_dir(self.stage.source_path): patch.apply(self.stage) tty.debug('Applied patch {0}'.format(patch.path_or_url)) patched = True @@ -1377,12 +1412,12 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): # Touch bad file if anything goes wrong. tty.msg('Patch %s failed.' % patch.path_or_url) - touch(bad_file) + fsys.touch(bad_file) raise if has_patch_fun: try: - with working_dir(self.stage.source_path): + with fsys.working_dir(self.stage.source_path): self.patch() tty.debug('Ran patch() for {0}'.format(self.name)) patched = True @@ -1400,7 +1435,7 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): # Touch bad file if anything goes wrong. tty.msg('patch() function failed for {0}'.format(self.name)) - touch(bad_file) + fsys.touch(bad_file) raise # Get rid of any old failed file -- patches have either succeeded @@ -1411,9 +1446,9 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): # touch good or no patches file so that we skip next time. if patched: - touch(good_file) + fsys.touch(good_file) else: - touch(no_patches_file) + fsys.touch(no_patches_file) @classmethod def all_patches(cls): @@ -1657,6 +1692,175 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): builder = PackageInstaller([(self, kwargs)]) builder.install() + def cache_extra_test_sources(self, srcs): + """Copy relative source paths to the corresponding install test subdir + + This method is intended as an optional install test setup helper for + grabbing source files/directories during the installation process and + copying them to the installation test subdirectory for subsequent use + during install testing. + + Args: + srcs (str or list of str): relative path for files and or + subdirectories located in the staged source path that are to + be copied to the corresponding location(s) under the install + testing directory. + """ + paths = [srcs] if isinstance(srcs, six.string_types) else srcs + + for path in paths: + src_path = os.path.join(self.stage.source_path, path) + dest_path = os.path.join(self.install_test_root, path) + if os.path.isdir(src_path): + fsys.install_tree(src_path, dest_path) + else: + fsys.mkdirp(os.path.dirname(dest_path)) + fsys.copy(src_path, dest_path) + + def do_test(self, dirty=False): + if self.test_requires_compiler: + compilers = spack.compilers.compilers_for_spec( + self.spec.compiler, arch_spec=self.spec.architecture) + if not compilers: + tty.error('Skipping tests for package %s\n' % + self.spec.format('{name}-{version}-{hash:7}') + + 'Package test requires missing compiler %s' % + self.spec.compiler) + return + + # Clear test failures + self.test_failures = [] + self.test_log_file = self.test_suite.log_file_for_spec(self.spec) + fsys.touch(self.test_log_file) # Otherwise log_parse complains + + kwargs = {'dirty': dirty, 'fake': False, 'context': 'test'} + spack.build_environment.start_build_process(self, test_process, kwargs) + + def test(self): + pass + + def run_test(self, exe, options=[], expected=[], status=0, + installed=False, purpose='', skip_missing=False, + work_dir=None): + """Run the test and confirm the expected results are obtained + + Log any failures and continue, they will be re-raised later + + Args: + exe (str): the name of the executable + options (str or list of str): list of options to pass to the runner + expected (str or list of str): list of expected output strings. + Each string is a regex expected to match part of the output. + status (int or list of int): possible passing status values + with 0 meaning the test is expected to succeed + installed (bool): if ``True``, the executable must be in the + install prefix + purpose (str): message to display before running test + skip_missing (bool): skip the test if the executable is not + in the install prefix bin directory or the provided work_dir + work_dir (str or None): path to the smoke test directory + """ + wdir = '.' if work_dir is None else work_dir + with fsys.working_dir(wdir): + try: + runner = which(exe) + if runner is None and skip_missing: + return + assert runner is not None, \ + "Failed to find executable '{0}'".format(exe) + + self._run_test_helper( + runner, options, expected, status, installed, purpose) + print("PASSED") + return True + except BaseException as e: + # print a summary of the error to the log file + # so that cdash and junit reporters know about it + exc_type, _, tb = sys.exc_info() + print('FAILED: {0}'.format(e)) + import traceback + # remove the current call frame to exclude the extract_stack + # call from the error + stack = traceback.extract_stack()[:-1] + + # Package files have a line added at import time, so we re-read + # the file to make line numbers match. We have to subtract two + # from the line number because the original line number is + # inflated once by the import statement and the lines are + # displaced one by the import statement. + for i, entry in enumerate(stack): + filename, lineno, function, text = entry + if spack.repo.is_package_file(filename): + with open(filename, 'r') as f: + lines = f.readlines() + new_lineno = lineno - 2 + text = lines[new_lineno] + stack[i] = (filename, new_lineno, function, text) + + # Format the stack to print and print it + out = traceback.format_list(stack) + for line in out: + print(line.rstrip('\n')) + + if exc_type is spack.util.executable.ProcessError: + out = six.StringIO() + spack.build_environment.write_log_summary( + out, 'test', self.test_log_file, last=1) + m = out.getvalue() + else: + # We're below the package context, so get context from + # stack instead of from traceback. + # The traceback is truncated here, so we can't use it to + # traverse the stack. + m = '\n'.join( + spack.build_environment.get_package_context(tb) + ) + + exc = e # e is deleted after this block + + # If we fail fast, raise another error + if spack.config.get('config:fail_fast', False): + raise TestFailure([(exc, m)]) + else: + self.test_failures.append((exc, m)) + return False + + def _run_test_helper(self, runner, options, expected, status, installed, + purpose): + status = [status] if isinstance(status, six.integer_types) else status + expected = [expected] if isinstance(expected, six.string_types) else \ + expected + options = [options] if isinstance(options, six.string_types) else \ + options + + if purpose: + tty.msg(purpose) + else: + tty.debug('test: {0}: expect command status in {1}' + .format(runner.name, status)) + + if installed: + msg = "Executable '{0}' expected in prefix".format(runner.name) + msg += ", found in {0} instead".format(runner.path) + assert runner.path.startswith(self.spec.prefix), msg + + try: + output = runner(*options, output=str.split, error=str.split) + + assert 0 in status, \ + 'Expected {0} execution to fail'.format(runner.name) + except ProcessError as err: + output = str(err) + match = re.search(r'exited with status ([0-9]+)', output) + if not (match and int(match.group(1)) in status): + raise + + for check in expected: + cmd = ' '.join([runner.name] + options) + msg = "Expected '{0}' to match output of `{1}`".format(check, cmd) + msg += '\n\nOutput: {0}'.format(output) + assert re.search(check, output), msg + def unit_test_check(self): """Hook for unit tests to assert things about package internals. @@ -1678,7 +1882,7 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): """This function checks whether install succeeded.""" def check_paths(path_list, filetype, predicate): - if isinstance(path_list, string_types): + if isinstance(path_list, six.string_types): path_list = [path_list] for path in path_list: @@ -2031,7 +2235,7 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): # copy spec metadata to "deprecated" dir of deprecator depr_yaml = spack.store.layout.deprecated_file_path(spec, deprecator) - fs.mkdirp(os.path.dirname(depr_yaml)) + fsys.mkdirp(os.path.dirname(depr_yaml)) shutil.copy2(self_yaml, depr_yaml) # Any specs deprecated in favor of this spec are re-deprecated in @@ -2210,7 +2414,7 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): doc = re.sub(r'\s+', ' ', self.__doc__) lines = textwrap.wrap(doc, 72) - results = StringIO() + results = six.StringIO() for line in lines: results.write((" " * indent) + line + "\n") return results.getvalue() @@ -2313,6 +2517,71 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): tty.warn(msg.format(name)) +def test_process(pkg, kwargs): + with tty.log.log_output(pkg.test_log_file) as logger: + with logger.force_echo(): + tty.msg('Testing package {0}' + .format(pkg.test_suite.test_pkg_id(pkg.spec))) + + # use debug print levels for log file to record commands + old_debug = tty.is_debug() + tty.set_debug(True) + + # run test methods from the package and all virtuals it + # provides virtuals have to be deduped by name + v_names = list(set([vspec.name + for vspec in pkg.virtuals_provided])) + + # hack for compilers that are not dependencies (yet) + # TODO: this all eventually goes away + c_names = ('gcc', 'intel', 'intel-parallel-studio', 'pgi') + if pkg.name in c_names: + v_names.extend(['c', 'cxx', 'fortran']) + if pkg.spec.satisfies('llvm+clang'): + v_names.extend(['c', 'cxx']) + + test_specs = [pkg.spec] + [spack.spec.Spec(v_name) + for v_name in sorted(v_names)] + + try: + with fsys.working_dir( + pkg.test_suite.test_dir_for_spec(pkg.spec)): + for spec in test_specs: + pkg.test_suite.current_test_spec = spec + # Fail gracefully if a virtual has no package/tests + try: + spec_pkg = spec.package + except spack.repo.UnknownPackageError: + continue + + # copy test data into test data dir + data_source = Prefix(spec_pkg.package_dir).test + data_dir = pkg.test_suite.current_test_data_dir + if (os.path.isdir(data_source) and + not os.path.exists(data_dir)): + # We assume data dir is used read-only + # maybe enforce this later + shutil.copytree(data_source, data_dir) + + # grab the function for each method so we can call + # it with the package + test_fn = spec_pkg.__class__.test + if not isinstance(test_fn, types.FunctionType): + test_fn = test_fn.__func__ + + # Run the tests + test_fn(pkg) + + # If fail-fast was on, we error out above + # If we collect errors, raise them in batch here + if pkg.test_failures: + raise TestFailure(pkg.test_failures) + + finally: + # reset debug level + tty.set_debug(old_debug) + + inject_flags = PackageBase.inject_flags env_flags = PackageBase.env_flags build_system_flags = PackageBase.build_system_flags diff --git a/lib/spack/spack/paths.py b/lib/spack/spack/paths.py index cb2240359c..9c803cba7e 100644 --- a/lib/spack/spack/paths.py +++ b/lib/spack/spack/paths.py @@ -12,7 +12,6 @@ dependencies. import os from llnl.util.filesystem import ancestor - #: This file lives in $prefix/lib/spack/spack/__file__ prefix = ancestor(__file__, 4) @@ -42,6 +41,7 @@ test_path = os.path.join(module_path, "test") hooks_path = os.path.join(module_path, "hooks") var_path = os.path.join(prefix, "var", "spack") repos_path = os.path.join(var_path, "repos") +tests_path = os.path.join(var_path, "tests") share_path = os.path.join(prefix, "share", "spack") # Paths to built-in Spack repositories. diff --git a/lib/spack/spack/pkgkit.py b/lib/spack/spack/pkgkit.py index ac0a7eee0d..4f25d41dfb 100644 --- a/lib/spack/spack/pkgkit.py +++ b/lib/spack/spack/pkgkit.py @@ -59,6 +59,7 @@ from spack.package import \ from spack.installer import \ ExternalPackageError, InstallError, InstallLockError, UpstreamPackageError +from spack.install_test import get_escaped_text_output from spack.variant import any_combination_of, auto_or_any_combination_of from spack.variant import disjoint_sets diff --git a/lib/spack/spack/repo.py b/lib/spack/spack/repo.py index 5e1457c165..4af7b382f0 100644 --- a/lib/spack/spack/repo.py +++ b/lib/spack/spack/repo.py @@ -91,6 +91,19 @@ def autospec(function): return converter +def is_package_file(filename): + """Determine whether we are in a package file from a repo.""" + # Package files are named `package.py` and are not in lib/spack/spack + # We have to remove the file extension because it can be .py and can be + # .pyc depending on context, and can differ between the files + import spack.package # break cycle + filename_noext = os.path.splitext(filename)[0] + packagebase_filename_noext = os.path.splitext( + inspect.getfile(spack.package.PackageBase))[0] + return (filename_noext != packagebase_filename_noext and + os.path.basename(filename_noext) == 'package') + + class SpackNamespace(types.ModuleType): """ Allow lazy loading of modules.""" @@ -131,6 +144,11 @@ class FastPackageChecker(Mapping): #: Reference to the appropriate entry in the global cache self._packages_to_stats = self._paths_cache[packages_path] + def invalidate(self): + """Regenerate cache for this checker.""" + self._paths_cache[self.packages_path] = self._create_new_cache() + self._packages_to_stats = self._paths_cache[self.packages_path] + def _create_new_cache(self): """Create a new cache for packages in a repo. @@ -308,6 +326,9 @@ class ProviderIndexer(Indexer): self.index = spack.provider_index.ProviderIndex.from_json(stream) def update(self, pkg_fullname): + name = pkg_fullname.split('.')[-1] + if spack.repo.path.is_virtual(name, use_index=False): + return self.index.remove_provider(pkg_fullname) self.index.update(pkg_fullname) @@ -517,12 +538,12 @@ class RepoPath(object): """Get the first repo in precedence order.""" return self.repos[0] if self.repos else None - def all_package_names(self): + def all_package_names(self, include_virtuals=False): """Return all unique package names in all repositories.""" if self._all_package_names is None: all_pkgs = set() for repo in self.repos: - for name in repo.all_package_names(): + for name in repo.all_package_names(include_virtuals): all_pkgs.add(name) self._all_package_names = sorted(all_pkgs, key=lambda n: n.lower()) return self._all_package_names @@ -679,12 +700,20 @@ class RepoPath(object): """ return any(repo.exists(pkg_name) for repo in self.repos) - def is_virtual(self, pkg_name): - """True if the package with this name is virtual, False otherwise.""" - if not isinstance(pkg_name, str): + def is_virtual(self, pkg_name, use_index=True): + """True if the package with this name is virtual, False otherwise. + + Set `use_index` False when calling from a code block that could + be run during the computation of the provider index.""" + have_name = pkg_name is not None + if have_name and not isinstance(pkg_name, str): raise ValueError( "is_virtual(): expected package name, got %s" % type(pkg_name)) - return pkg_name in self.provider_index + if use_index: + return have_name and pkg_name in self.provider_index + else: + return have_name and (not self.exists(pkg_name) or + self.get_pkg_class(pkg_name).virtual) def __contains__(self, pkg_name): return self.exists(pkg_name) @@ -913,10 +942,6 @@ class Repo(object): This dumps the package file and any associated patch files. Raises UnknownPackageError if not found. """ - # Some preliminary checks. - if spec.virtual: - raise UnknownPackageError(spec.name) - if spec.namespace and spec.namespace != self.namespace: raise UnknownPackageError( "Repository %s does not contain package %s." @@ -999,9 +1024,12 @@ class Repo(object): self._fast_package_checker = FastPackageChecker(self.packages_path) return self._fast_package_checker - def all_package_names(self): + def all_package_names(self, include_virtuals=False): """Returns a sorted list of all package names in the Repo.""" - return sorted(self._pkg_checker.keys()) + names = sorted(self._pkg_checker.keys()) + if include_virtuals: + return names + return [x for x in names if not self.is_virtual(x)] def packages_with_tags(self, *tags): v = set(self.all_package_names()) @@ -1040,7 +1068,7 @@ class Repo(object): def is_virtual(self, pkg_name): """True if the package with this name is virtual, False otherwise.""" - return self.provider_index.contains(pkg_name) + return pkg_name in self.provider_index def _get_pkg_module(self, pkg_name): """Create a module for a particular package. @@ -1074,7 +1102,8 @@ class Repo(object): # manually construct the error message in order to give the # user the correct package.py where the syntax error is located raise SyntaxError('invalid syntax in {0:}, line {1:}' - ''.format(file_path, e.lineno)) + .format(file_path, e.lineno)) + module.__package__ = self.full_namespace module.__loader__ = self self._modules[pkg_name] = module @@ -1205,9 +1234,9 @@ def get(spec): return path.get(spec) -def all_package_names(): +def all_package_names(include_virtuals=False): """Convenience wrapper around ``spack.repo.all_package_names()``.""" - return path.all_package_names() + return path.all_package_names(include_virtuals) def set_path(repo): diff --git a/lib/spack/spack/report.py b/lib/spack/spack/report.py index 4e5bc9c993..ebae2d0adc 100644 --- a/lib/spack/spack/report.py +++ b/lib/spack/spack/report.py @@ -9,11 +9,13 @@ import collections import functools import time import traceback +import os import llnl.util.lang import spack.build_environment import spack.fetch_strategy import spack.package +from spack.install_test import TestSuite from spack.reporter import Reporter from spack.reporters.cdash import CDash from spack.reporters.junit import JUnit @@ -33,12 +35,16 @@ __all__ = [ ] -def fetch_package_log(pkg): +def fetch_log(pkg, do_fn, dir): + log_files = { + '_install_task': pkg.build_log_path, + 'do_test': os.path.join(dir, TestSuite.test_log_name(pkg.spec)), + } try: - with codecs.open(pkg.build_log_path, 'r', 'utf-8') as f: + with codecs.open(log_files[do_fn.__name__], 'r', 'utf-8') as f: return ''.join(f.readlines()) except Exception: - return 'Cannot open build log for {0}'.format( + return 'Cannot open log for {0}'.format( pkg.spec.cshort_spec ) @@ -58,15 +64,20 @@ class InfoCollector(object): specs (list of Spec): specs whose install information will be recorded """ - #: Backup of PackageInstaller._install_task - _backup__install_task = spack.package.PackageInstaller._install_task - - def __init__(self, specs): - #: Specs that will be installed + def __init__(self, wrap_class, do_fn, specs, dir): + #: Class for which to wrap a function + self.wrap_class = wrap_class + #: Action to be reported on + self.do_fn = do_fn + #: Backup of PackageBase function + self._backup_do_fn = getattr(self.wrap_class, do_fn) + #: Specs that will be acted on self.input_specs = specs #: This is where we record the data that will be included #: in our report. self.specs = [] + #: Record directory for test log paths + self.dir = dir def __enter__(self): # Initialize the spec report with the data that is available upfront. @@ -98,30 +109,37 @@ class InfoCollector(object): Property('compiler', input_spec.compiler)) # Check which specs are already installed and mark them as skipped - for dep in filter(lambda x: x.package.installed, - input_spec.traverse()): - package = { - 'name': dep.name, - 'id': dep.dag_hash(), - 'elapsed_time': '0.0', - 'result': 'skipped', - 'message': 'Spec already installed' - } - spec['packages'].append(package) - - def gather_info(_install_task): - """Decorates PackageInstaller._install_task to gather useful - information on PackageBase.do_install for a CI report. + # only for install_task + if self.do_fn == '_install_task': + for dep in filter(lambda x: x.package.installed, + input_spec.traverse()): + package = { + 'name': dep.name, + 'id': dep.dag_hash(), + 'elapsed_time': '0.0', + 'result': 'skipped', + 'message': 'Spec already installed' + } + spec['packages'].append(package) + + def gather_info(do_fn): + """Decorates do_fn to gather useful information for + a CI report. It's defined here to capture the environment and build this context as the installations proceed. """ - @functools.wraps(_install_task) - def wrapper(installer, task, *args, **kwargs): - pkg = task.pkg + @functools.wraps(do_fn) + def wrapper(instance, *args, **kwargs): + if isinstance(instance, spack.package.PackageBase): + pkg = instance + elif hasattr(args[0], 'pkg'): + pkg = args[0].pkg + else: + raise Exception # We accounted before for what is already installed - installed_on_entry = pkg.installed + installed_already = pkg.installed package = { 'name': pkg.name, @@ -135,13 +153,12 @@ class InfoCollector(object): start_time = time.time() value = None try: - - value = _install_task(installer, task, *args, **kwargs) + value = do_fn(instance, *args, **kwargs) package['result'] = 'success' - package['stdout'] = fetch_package_log(pkg) + package['stdout'] = fetch_log(pkg, do_fn, self.dir) package['installed_from_binary_cache'] = \ pkg.installed_from_binary_cache - if installed_on_entry: + if do_fn.__name__ == '_install_task' and installed_already: return except spack.build_environment.InstallError as e: @@ -149,7 +166,7 @@ class InfoCollector(object): # didn't work correctly) package['result'] = 'failure' package['message'] = e.message or 'Installation failure' - package['stdout'] = fetch_package_log(pkg) + package['stdout'] = fetch_log(pkg, do_fn, self.dir) package['stdout'] += package['message'] package['exception'] = e.traceback @@ -157,7 +174,7 @@ class InfoCollector(object): # Everything else is an error (the installation # failed outside of the child process) package['result'] = 'error' - package['stdout'] = fetch_package_log(pkg) + package['stdout'] = fetch_log(pkg, do_fn, self.dir) package['message'] = str(e) or 'Unknown error' package['exception'] = traceback.format_exc() @@ -184,15 +201,14 @@ class InfoCollector(object): return wrapper - spack.package.PackageInstaller._install_task = gather_info( - spack.package.PackageInstaller._install_task - ) + setattr(self.wrap_class, self.do_fn, gather_info( + getattr(self.wrap_class, self.do_fn) + )) def __exit__(self, exc_type, exc_val, exc_tb): - # Restore the original method in PackageInstaller - spack.package.PackageInstaller._install_task = \ - InfoCollector._backup__install_task + # Restore the original method in PackageBase + setattr(self.wrap_class, self.do_fn, self._backup_do_fn) for spec in self.specs: spec['npackages'] = len(spec['packages']) @@ -225,22 +241,26 @@ class collect_info(object): # The file 'junit.xml' is written when exiting # the context - specs = [Spec('hdf5').concretized()] - with collect_info(specs, 'junit', 'junit.xml'): + s = [Spec('hdf5').concretized()] + with collect_info(PackageBase, do_install, s, 'junit', 'a.xml'): # A report will be generated for these specs... - for spec in specs: - spec.do_install() + for spec in s: + getattr(class, function)(spec) # ...but not for this one Spec('zlib').concretized().do_install() Args: + class: class on which to wrap a function + function: function to wrap format_name (str or None): one of the supported formats - args (dict): args passed to spack install + args (dict): args passed to function Raises: ValueError: when ``format_name`` is not in ``valid_formats`` """ - def __init__(self, format_name, args): + def __init__(self, cls, function, format_name, args): + self.cls = cls + self.function = function self.filename = None if args.cdash_upload_url: self.format_name = 'cdash' @@ -253,13 +273,19 @@ class collect_info(object): .format(self.format_name)) self.report_writer = report_writers[self.format_name](args) + def __call__(self, type, dir=os.getcwd()): + self.type = type + self.dir = dir + return self + def concretization_report(self, msg): self.report_writer.concretization_report(self.filename, msg) def __enter__(self): if self.format_name: - # Start the collector and patch PackageInstaller._install_task - self.collector = InfoCollector(self.specs) + # Start the collector and patch self.function on appropriate class + self.collector = InfoCollector( + self.cls, self.function, self.specs, self.dir) self.collector.__enter__() def __exit__(self, exc_type, exc_val, exc_tb): @@ -269,4 +295,5 @@ class collect_info(object): self.collector.__exit__(exc_type, exc_val, exc_tb) report_data = {'specs': self.collector.specs} - self.report_writer.build_report(self.filename, report_data) + report_fn = getattr(self.report_writer, '%s_report' % self.type) + report_fn(self.filename, report_data) diff --git a/lib/spack/spack/reporter.py b/lib/spack/spack/reporter.py index 25d4042a5e..6314054139 100644 --- a/lib/spack/spack/reporter.py +++ b/lib/spack/spack/reporter.py @@ -16,5 +16,8 @@ class Reporter(object): def build_report(self, filename, report_data): pass + def test_report(self, filename, report_data): + pass + def concretization_report(self, filename, msg): pass diff --git a/lib/spack/spack/reporters/cdash.py b/lib/spack/spack/reporters/cdash.py index 580df7866f..c1a220963e 100644 --- a/lib/spack/spack/reporters/cdash.py +++ b/lib/spack/spack/reporters/cdash.py @@ -72,8 +72,10 @@ class CDash(Reporter): tty.verbose("Using CDash auth token from environment") self.authtoken = os.environ.get('SPACK_CDASH_AUTH_TOKEN') - if args.spec: + if getattr(args, 'spec', ''): packages = args.spec + elif getattr(args, 'specs', ''): + packages = args.specs else: packages = [] for file in args.specfiles: @@ -98,7 +100,7 @@ class CDash(Reporter): self.revision = git('rev-parse', 'HEAD', output=str).strip() self.multiple_packages = False - def report_for_package(self, directory_name, package, duration): + def build_report_for_package(self, directory_name, package, duration): if 'stdout' not in package: # Skip reporting on packages that did not generate any output. return @@ -158,8 +160,8 @@ class CDash(Reporter): '\n'.join(report_data[phase]['loglines']) errors, warnings = parse_log_events(report_data[phase]['loglines']) # Cap the number of errors and warnings at 50 each. - errors = errors[0:49] - warnings = warnings[0:49] + errors = errors[:50] + warnings = warnings[:50] nerrors = len(errors) if phase == 'configure' and nerrors > 0: @@ -250,7 +252,114 @@ class CDash(Reporter): if 'time' in spec: duration = int(spec['time']) for package in spec['packages']: - self.report_for_package(directory_name, package, duration) + self.build_report_for_package( + directory_name, package, duration) + self.print_cdash_link() + + def test_report_for_package(self, directory_name, package, duration): + if 'stdout' not in package: + # Skip reporting on packages that did not generate any output. + return + + self.current_package_name = package['name'] + self.buildname = "{0} - {1}".format( + self.base_buildname, package['name']) + + report_data = self.initialize_report(directory_name) + + for phase in ('test', 'update'): + report_data[phase] = {} + report_data[phase]['loglines'] = [] + report_data[phase]['status'] = 0 + report_data[phase]['endtime'] = self.endtime + + # Track the phases we perform so we know what reports to create. + # We always report the update step because this is how we tell CDash + # what revision of Spack we are using. + phases_encountered = ['test', 'update'] + + # Generate a report for this package. + # The first line just says "Testing package name-hash" + report_data['test']['loglines'].append( + text_type("{0} output for {1}:".format( + 'test', package['name']))) + for line in package['stdout'].splitlines()[1:]: + report_data['test']['loglines'].append( + xml.sax.saxutils.escape(line)) + + self.starttime = self.endtime - duration + for phase in phases_encountered: + report_data[phase]['starttime'] = self.starttime + report_data[phase]['log'] = \ + '\n'.join(report_data[phase]['loglines']) + errors, warnings = parse_log_events(report_data[phase]['loglines']) + # Cap the number of errors and warnings at 50 each. + errors = errors[0:49] + warnings = warnings[0:49] + + if phase == 'test': + # Convert log output from ASCII to Unicode and escape for XML. + def clean_log_event(event): + event = vars(event) + event['text'] = xml.sax.saxutils.escape(event['text']) + event['pre_context'] = xml.sax.saxutils.escape( + '\n'.join(event['pre_context'])) + event['post_context'] = xml.sax.saxutils.escape( + '\n'.join(event['post_context'])) + # source_file and source_line_no are either strings or + # the tuple (None,). Distinguish between these two cases. + if event['source_file'][0] is None: + event['source_file'] = '' + event['source_line_no'] = '' + else: + event['source_file'] = xml.sax.saxutils.escape( + event['source_file']) + return event + + # Convert errors to warnings if the package reported success. + if package['result'] == 'success': + warnings = errors + warnings + errors = [] + + report_data[phase]['errors'] = [] + report_data[phase]['warnings'] = [] + for error in errors: + report_data[phase]['errors'].append(clean_log_event(error)) + for warning in warnings: + report_data[phase]['warnings'].append( + clean_log_event(warning)) + + if phase == 'update': + report_data[phase]['revision'] = self.revision + + # Write the report. + report_name = phase.capitalize() + ".xml" + report_file_name = package['name'] + "_" + report_name + phase_report = os.path.join(directory_name, report_file_name) + + with codecs.open(phase_report, 'w', 'utf-8') as f: + env = spack.tengine.make_environment() + if phase != 'update': + # Update.xml stores site information differently + # than the rest of the CTest XML files. + site_template = os.path.join(self.template_dir, 'Site.xml') + t = env.get_template(site_template) + f.write(t.render(report_data)) + + phase_template = os.path.join(self.template_dir, report_name) + t = env.get_template(phase_template) + f.write(t.render(report_data)) + self.upload(phase_report) + + def test_report(self, directory_name, input_data): + # Generate reports for each package in each spec. + for spec in input_data['specs']: + duration = 0 + if 'time' in spec: + duration = int(spec['time']) + for package in spec['packages']: + self.test_report_for_package( + directory_name, package, duration) self.print_cdash_link() def concretization_report(self, directory_name, msg): diff --git a/lib/spack/spack/reporters/junit.py b/lib/spack/spack/reporters/junit.py index 6c54c45b42..598b308934 100644 --- a/lib/spack/spack/reporters/junit.py +++ b/lib/spack/spack/reporters/junit.py @@ -27,3 +27,6 @@ class JUnit(Reporter): env = spack.tengine.make_environment() t = env.get_template(self.template_file) f.write(t.render(report_data)) + + def test_report(self, filename, report_data): + self.build_report(filename, report_data) diff --git a/lib/spack/spack/schema/config.py b/lib/spack/spack/schema/config.py index b1d3332a3f..0f83eb86f4 100644 --- a/lib/spack/spack/schema/config.py +++ b/lib/spack/spack/schema/config.py @@ -45,6 +45,7 @@ properties = { {'type': 'array', 'items': {'type': 'string'}}], }, + 'test_stage': {'type': 'string'}, 'extensions': { 'type': 'array', 'items': {'type': 'string'} diff --git a/lib/spack/spack/spec.py b/lib/spack/spack/spec.py index 8a4fc12861..743f84c3c8 100644 --- a/lib/spack/spack/spec.py +++ b/lib/spack/spack/spec.py @@ -951,7 +951,7 @@ class SpecBuildInterface(lang.ObjectWrapper): def __init__(self, spec, name, query_parameters): super(SpecBuildInterface, self).__init__(spec) - is_virtual = Spec.is_virtual(name) + is_virtual = spack.repo.path.is_virtual(name) self.last_query = QueryState( name=name, extra_parameters=query_parameters, @@ -1227,12 +1227,9 @@ class Spec(object): Possible idea: just use conventin and make virtual deps all caps, e.g., MPI vs mpi. """ - return Spec.is_virtual(self.name) - - @staticmethod - def is_virtual(name): - """Test if a name is virtual without requiring a Spec.""" - return (name is not None) and (not spack.repo.path.exists(name)) + # This method can be called while regenerating the provider index + # So we turn off using the index to detect virtuals + return spack.repo.path.is_virtual(self.name, use_index=False) @property def concrete(self): diff --git a/lib/spack/spack/tengine.py b/lib/spack/spack/tengine.py index 8581c8b41e..15268e682d 100644 --- a/lib/spack/spack/tengine.py +++ b/lib/spack/spack/tengine.py @@ -68,7 +68,8 @@ def make_environment(dirs=None): """Returns an configured environment for template rendering.""" if dirs is None: # Default directories where to search for templates - builtins = spack.config.get('config:template_dirs') + builtins = spack.config.get('config:template_dirs', + ['$spack/share/spack/templates']) extensions = spack.extensions.get_template_dirs() dirs = [canonicalize_path(d) for d in itertools.chain(builtins, extensions)] diff --git a/lib/spack/spack/test/cmd/clean.py b/lib/spack/spack/test/cmd/clean.py index 4df708cf50..dcaf0c916c 100644 --- a/lib/spack/spack/test/cmd/clean.py +++ b/lib/spack/spack/test/cmd/clean.py @@ -15,44 +15,51 @@ clean = spack.main.SpackCommand('clean') @pytest.fixture() def mock_calls_for_clean(monkeypatch): + counts = {} + class Counter(object): - def __init__(self): - self.call_count = 0 + def __init__(self, name): + self.name = name + counts[name] = 0 def __call__(self, *args, **kwargs): - self.call_count += 1 + counts[self.name] += 1 - monkeypatch.setattr(spack.package.PackageBase, 'do_clean', Counter()) - monkeypatch.setattr(spack.stage, 'purge', Counter()) + monkeypatch.setattr(spack.package.PackageBase, 'do_clean', + Counter('package')) + monkeypatch.setattr(spack.stage, 'purge', Counter('stages')) monkeypatch.setattr( - spack.caches.fetch_cache, 'destroy', Counter(), raising=False) + spack.caches.fetch_cache, 'destroy', Counter('downloads'), + raising=False) monkeypatch.setattr( - spack.caches.misc_cache, 'destroy', Counter()) + spack.caches.misc_cache, 'destroy', Counter('caches')) monkeypatch.setattr( - spack.installer, 'clear_failures', Counter()) + spack.installer, 'clear_failures', Counter('failures')) + + yield counts + + +all_effects = ['stages', 'downloads', 'caches', 'failures'] @pytest.mark.usefixtures( - 'mock_packages', 'config', 'mock_calls_for_clean' + 'mock_packages', 'config' ) -@pytest.mark.parametrize('command_line,counters', [ - ('mpileaks', [1, 0, 0, 0, 0]), - ('-s', [0, 1, 0, 0, 0]), - ('-sd', [0, 1, 1, 0, 0]), - ('-m', [0, 0, 0, 1, 0]), - ('-f', [0, 0, 0, 0, 1]), - ('-a', [0, 1, 1, 1, 1]), - ('', [0, 0, 0, 0, 0]), +@pytest.mark.parametrize('command_line,effects', [ + ('mpileaks', ['package']), + ('-s', ['stages']), + ('-sd', ['stages', 'downloads']), + ('-m', ['caches']), + ('-f', ['failures']), + ('-a', all_effects), + ('', []), ]) -def test_function_calls(command_line, counters): +def test_function_calls(command_line, effects, mock_calls_for_clean): # Call the command with the supplied command line clean(command_line) # Assert that we called the expected functions the correct # number of times - assert spack.package.PackageBase.do_clean.call_count == counters[0] - assert spack.stage.purge.call_count == counters[1] - assert spack.caches.fetch_cache.destroy.call_count == counters[2] - assert spack.caches.misc_cache.destroy.call_count == counters[3] - assert spack.installer.clear_failures.call_count == counters[4] + for name in ['package'] + all_effects: + assert mock_calls_for_clean[name] == (1 if name in effects else 0) diff --git a/lib/spack/spack/test/cmd/mirror.py b/lib/spack/spack/test/cmd/mirror.py index f6fe0b24dd..0957624cba 100644 --- a/lib/spack/spack/test/cmd/mirror.py +++ b/lib/spack/spack/test/cmd/mirror.py @@ -102,7 +102,7 @@ class MockMirrorArgs(object): self.exclude_specs = exclude_specs -def test_exclude_specs(mock_packages): +def test_exclude_specs(mock_packages, config): args = MockMirrorArgs( specs=['mpich'], versions_per_spec='all', @@ -117,7 +117,7 @@ def test_exclude_specs(mock_packages): assert (not expected_exclude & set(mirror_specs)) -def test_exclude_file(mock_packages, tmpdir): +def test_exclude_file(mock_packages, tmpdir, config): exclude_path = os.path.join(str(tmpdir), 'test-exclude.txt') with open(exclude_path, 'w') as exclude_file: exclude_file.write("""\ diff --git a/lib/spack/spack/test/cmd/pkg.py b/lib/spack/spack/test/cmd/pkg.py index af634beffc..ca9e3a1a3e 100644 --- a/lib/spack/spack/test/cmd/pkg.py +++ b/lib/spack/spack/test/cmd/pkg.py @@ -62,9 +62,9 @@ def mock_pkg_git_repo(tmpdir_factory): mkdirp('pkg-a', 'pkg-b', 'pkg-c') with open('pkg-a/package.py', 'w') as f: f.write(pkg_template.format(name='PkgA')) - with open('pkg-c/package.py', 'w') as f: - f.write(pkg_template.format(name='PkgB')) with open('pkg-b/package.py', 'w') as f: + f.write(pkg_template.format(name='PkgB')) + with open('pkg-c/package.py', 'w') as f: f.write(pkg_template.format(name='PkgC')) git('add', 'pkg-a', 'pkg-b', 'pkg-c') git('-c', 'commit.gpgsign=false', 'commit', @@ -128,6 +128,8 @@ def test_pkg_add(mock_pkg_git_repo): git('status', '--short', output=str)) finally: shutil.rmtree('pkg-e') + # Removing a package mid-run disrupts Spack's caching + spack.repo.path.repos[0]._fast_package_checker.invalidate() with pytest.raises(spack.main.SpackCommandError): pkg('add', 'does-not-exist') diff --git a/lib/spack/spack/test/cmd/test.py b/lib/spack/spack/test/cmd/test.py index a9ef735afe..4163853274 100644 --- a/lib/spack/spack/test/cmd/test.py +++ b/lib/spack/spack/test/cmd/test.py @@ -3,93 +3,181 @@ # # SPDX-License-Identifier: (Apache-2.0 OR MIT) +import argparse +import os + +import pytest + +import spack.config +import spack.package +import spack.cmd.install from spack.main import SpackCommand +install = SpackCommand('install') spack_test = SpackCommand('test') -cmd_test_py = 'lib/spack/spack/test/cmd/test.py' - - -def test_list(): - output = spack_test('--list') - assert "test.py" in output - assert "spec_semantics.py" in output - assert "test_list" not in output - - -def test_list_with_pytest_arg(): - output = spack_test('--list', cmd_test_py) - assert output.strip() == cmd_test_py - - -def test_list_with_keywords(): - output = spack_test('--list', '-k', 'cmd/test.py') - assert output.strip() == cmd_test_py - - -def test_list_long(capsys): - with capsys.disabled(): - output = spack_test('--list-long') - assert "test.py::\n" in output - assert "test_list" in output - assert "test_list_with_pytest_arg" in output - assert "test_list_with_keywords" in output - assert "test_list_long" in output - assert "test_list_long_with_pytest_arg" in output - assert "test_list_names" in output - assert "test_list_names_with_pytest_arg" in output - - assert "spec_dag.py::\n" in output - assert 'test_installed_deps' in output - assert 'test_test_deptype' in output - - -def test_list_long_with_pytest_arg(capsys): - with capsys.disabled(): - output = spack_test('--list-long', cmd_test_py) - assert "test.py::\n" in output - assert "test_list" in output - assert "test_list_with_pytest_arg" in output - assert "test_list_with_keywords" in output - assert "test_list_long" in output - assert "test_list_long_with_pytest_arg" in output - assert "test_list_names" in output - assert "test_list_names_with_pytest_arg" in output - - assert "spec_dag.py::\n" not in output - assert 'test_installed_deps' not in output - assert 'test_test_deptype' not in output - - -def test_list_names(): - output = spack_test('--list-names') - assert "test.py::test_list\n" in output - assert "test.py::test_list_with_pytest_arg\n" in output - assert "test.py::test_list_with_keywords\n" in output - assert "test.py::test_list_long\n" in output - assert "test.py::test_list_long_with_pytest_arg\n" in output - assert "test.py::test_list_names\n" in output - assert "test.py::test_list_names_with_pytest_arg\n" in output - - assert "spec_dag.py::test_installed_deps\n" in output - assert 'spec_dag.py::test_test_deptype\n' in output - - -def test_list_names_with_pytest_arg(): - output = spack_test('--list-names', cmd_test_py) - assert "test.py::test_list\n" in output - assert "test.py::test_list_with_pytest_arg\n" in output - assert "test.py::test_list_with_keywords\n" in output - assert "test.py::test_list_long\n" in output - assert "test.py::test_list_long_with_pytest_arg\n" in output - assert "test.py::test_list_names\n" in output - assert "test.py::test_list_names_with_pytest_arg\n" in output - - assert "spec_dag.py::test_installed_deps\n" not in output - assert 'spec_dag.py::test_test_deptype\n' not in output - - -def test_pytest_help(): - output = spack_test('--pytest-help') - assert "-k EXPRESSION" in output - assert "pytest-warnings:" in output - assert "--collect-only" in output + + +def test_test_package_not_installed( + tmpdir, mock_packages, mock_archive, mock_fetch, config, + install_mockery_mutable_config, mock_test_stage): + + output = spack_test('run', 'libdwarf') + + assert "No installed packages match spec libdwarf" in output + + +@pytest.mark.parametrize('arguments,expected', [ + (['run'], spack.config.get('config:dirty')), # default from config file + (['run', '--clean'], False), + (['run', '--dirty'], True), +]) +def test_test_dirty_flag(arguments, expected): + parser = argparse.ArgumentParser() + spack.cmd.test.setup_parser(parser) + args = parser.parse_args(arguments) + assert args.dirty == expected + + +def test_test_output(mock_test_stage, mock_packages, mock_archive, mock_fetch, + install_mockery_mutable_config): + """Ensure output printed from pkgs is captured by output redirection.""" + install('printing-package') + spack_test('run', 'printing-package') + + stage_files = os.listdir(mock_test_stage) + assert len(stage_files) == 1 + + # Grab test stage directory contents + testdir = os.path.join(mock_test_stage, stage_files[0]) + testdir_files = os.listdir(testdir) + + # Grab the output from the test log + testlog = list(filter(lambda x: x.endswith('out.txt') and + x != 'results.txt', testdir_files)) + outfile = os.path.join(testdir, testlog[0]) + with open(outfile, 'r') as f: + output = f.read() + assert "BEFORE TEST" in output + assert "true: expect command status in [" in output + assert "AFTER TEST" in output + assert "FAILED" not in output + + +def test_test_output_on_error( + mock_packages, mock_archive, mock_fetch, install_mockery_mutable_config, + capfd, mock_test_stage +): + install('test-error') + # capfd interferes with Spack's capturing + with capfd.disabled(): + out = spack_test('run', 'test-error', fail_on_error=False) + + assert "TestFailure" in out + assert "Command exited with status 1" in out + + +def test_test_output_on_failure( + mock_packages, mock_archive, mock_fetch, install_mockery_mutable_config, + capfd, mock_test_stage +): + install('test-fail') + with capfd.disabled(): + out = spack_test('run', 'test-fail', fail_on_error=False) + + assert "Expected 'not in the output' to match output of `true`" in out + assert "TestFailure" in out + + +def test_show_log_on_error( + mock_packages, mock_archive, mock_fetch, + install_mockery_mutable_config, capfd, mock_test_stage +): + """Make sure spack prints location of test log on failure.""" + install('test-error') + with capfd.disabled(): + out = spack_test('run', 'test-error', fail_on_error=False) + + assert 'See test log' in out + assert mock_test_stage in out + + +@pytest.mark.usefixtures( + 'mock_packages', 'mock_archive', 'mock_fetch', + 'install_mockery_mutable_config' +) +@pytest.mark.parametrize('pkg_name,msgs', [ + ('test-error', ['FAILED: Command exited', 'TestFailure']), + ('test-fail', ['FAILED: Expected', 'TestFailure']) +]) +def test_junit_output_with_failures(tmpdir, mock_test_stage, pkg_name, msgs): + install(pkg_name) + with tmpdir.as_cwd(): + spack_test('run', + '--log-format=junit', '--log-file=test.xml', + pkg_name) + + files = tmpdir.listdir() + filename = tmpdir.join('test.xml') + assert filename in files + + content = filename.open().read() + + # Count failures and errors correctly + assert 'tests="1"' in content + assert 'failures="1"' in content + assert 'errors="0"' in content + + # We want to have both stdout and stderr + assert '<system-out>' in content + for msg in msgs: + assert msg in content + + +def test_cdash_output_test_error( + tmpdir, mock_fetch, install_mockery_mutable_config, mock_packages, + mock_archive, mock_test_stage, capfd): + install('test-error') + with tmpdir.as_cwd(): + spack_test('run', + '--log-format=cdash', + '--log-file=cdash_reports', + 'test-error') + report_dir = tmpdir.join('cdash_reports') + print(tmpdir.listdir()) + assert report_dir in tmpdir.listdir() + report_file = report_dir.join('test-error_Test.xml') + assert report_file in report_dir.listdir() + content = report_file.open().read() + assert 'FAILED: Command exited with status 1' in content + + +def test_cdash_upload_clean_test( + tmpdir, mock_fetch, install_mockery_mutable_config, mock_packages, + mock_archive, mock_test_stage): + install('printing-package') + with tmpdir.as_cwd(): + spack_test('run', + '--log-file=cdash_reports', + '--log-format=cdash', + 'printing-package') + report_dir = tmpdir.join('cdash_reports') + assert report_dir in tmpdir.listdir() + report_file = report_dir.join('printing-package_Test.xml') + assert report_file in report_dir.listdir() + content = report_file.open().read() + assert '</Test>' in content + assert '<Text>' not in content + + +def test_test_help_does_not_show_cdash_options(mock_test_stage, capsys): + """Make sure `spack test --help` does not describe CDash arguments""" + with pytest.raises(SystemExit): + spack_test('run', '--help') + captured = capsys.readouterr() + assert 'CDash URL' not in captured.out + + +def test_test_help_cdash(mock_test_stage): + """Make sure `spack test --help-cdash` describes CDash arguments""" + out = spack_test('run', '--help-cdash') + assert 'CDash URL' in out diff --git a/lib/spack/spack/test/cmd/unit_test.py b/lib/spack/spack/test/cmd/unit_test.py new file mode 100644 index 0000000000..c5b8eb765e --- /dev/null +++ b/lib/spack/spack/test/cmd/unit_test.py @@ -0,0 +1,96 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +from spack.main import SpackCommand + +spack_test = SpackCommand('unit-test') +cmd_test_py = 'lib/spack/spack/test/cmd/unit_test.py' + + +def test_list(): + output = spack_test('--list') + assert "unit_test.py" in output + assert "spec_semantics.py" in output + assert "test_list" not in output + + +def test_list_with_pytest_arg(): + output = spack_test('--list', cmd_test_py) + assert output.strip() == cmd_test_py + + +def test_list_with_keywords(): + output = spack_test('--list', '-k', 'cmd/unit_test.py') + assert output.strip() == cmd_test_py + + +def test_list_long(capsys): + with capsys.disabled(): + output = spack_test('--list-long') + assert "unit_test.py::\n" in output + assert "test_list" in output + assert "test_list_with_pytest_arg" in output + assert "test_list_with_keywords" in output + assert "test_list_long" in output + assert "test_list_long_with_pytest_arg" in output + assert "test_list_names" in output + assert "test_list_names_with_pytest_arg" in output + + assert "spec_dag.py::\n" in output + assert 'test_installed_deps' in output + assert 'test_test_deptype' in output + + +def test_list_long_with_pytest_arg(capsys): + with capsys.disabled(): + output = spack_test('--list-long', cmd_test_py) + print(output) + assert "unit_test.py::\n" in output + assert "test_list" in output + assert "test_list_with_pytest_arg" in output + assert "test_list_with_keywords" in output + assert "test_list_long" in output + assert "test_list_long_with_pytest_arg" in output + assert "test_list_names" in output + assert "test_list_names_with_pytest_arg" in output + + assert "spec_dag.py::\n" not in output + assert 'test_installed_deps' not in output + assert 'test_test_deptype' not in output + + +def test_list_names(): + output = spack_test('--list-names') + assert "unit_test.py::test_list\n" in output + assert "unit_test.py::test_list_with_pytest_arg\n" in output + assert "unit_test.py::test_list_with_keywords\n" in output + assert "unit_test.py::test_list_long\n" in output + assert "unit_test.py::test_list_long_with_pytest_arg\n" in output + assert "unit_test.py::test_list_names\n" in output + assert "unit_test.py::test_list_names_with_pytest_arg\n" in output + + assert "spec_dag.py::test_installed_deps\n" in output + assert 'spec_dag.py::test_test_deptype\n' in output + + +def test_list_names_with_pytest_arg(): + output = spack_test('--list-names', cmd_test_py) + assert "unit_test.py::test_list\n" in output + assert "unit_test.py::test_list_with_pytest_arg\n" in output + assert "unit_test.py::test_list_with_keywords\n" in output + assert "unit_test.py::test_list_long\n" in output + assert "unit_test.py::test_list_long_with_pytest_arg\n" in output + assert "unit_test.py::test_list_names\n" in output + assert "unit_test.py::test_list_names_with_pytest_arg\n" in output + + assert "spec_dag.py::test_installed_deps\n" not in output + assert 'spec_dag.py::test_test_deptype\n' not in output + + +def test_pytest_help(): + output = spack_test('--pytest-help') + assert "-k EXPRESSION" in output + assert "pytest-warnings:" in output + assert "--collect-only" in output diff --git a/lib/spack/spack/test/conftest.py b/lib/spack/spack/test/conftest.py index 527d6a3380..8615b14abe 100644 --- a/lib/spack/spack/test/conftest.py +++ b/lib/spack/spack/test/conftest.py @@ -1273,3 +1273,14 @@ def mock_executable(tmpdir): return str(f) return _factory + + +@pytest.fixture() +def mock_test_stage(mutable_config, tmpdir): + # NOTE: This fixture MUST be applied after any fixture that uses + # the config fixture under the hood + # No need to unset because we use mutable_config + tmp_stage = str(tmpdir.join('test_stage')) + mutable_config.set('config:test_stage', tmp_stage) + + yield tmp_stage diff --git a/lib/spack/spack/test/llnl/util/tty/__init__.py b/lib/spack/spack/test/llnl/util/tty/__init__.py new file mode 100644 index 0000000000..9f87532b85 --- /dev/null +++ b/lib/spack/spack/test/llnl/util/tty/__init__.py @@ -0,0 +1,4 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/mirror.py b/lib/spack/spack/test/mirror.py index 05cf46dc20..8d3ec2d07b 100644 --- a/lib/spack/spack/test/mirror.py +++ b/lib/spack/spack/test/mirror.py @@ -16,7 +16,7 @@ from spack.util.executable import which from llnl.util.filesystem import resolve_link_target_relative_to_the_link -pytestmark = pytest.mark.usefixtures('config', 'mutable_mock_repo') +pytestmark = pytest.mark.usefixtures('mutable_config', 'mutable_mock_repo') # paths in repos that shouldn't be in the mirror tarballs. exclude = ['.hg', '.git', '.svn'] @@ -97,7 +97,7 @@ def check_mirror(): # tarball assert not dcmp.right_only # and that all original files are present. - assert all(l in exclude for l in dcmp.left_only) + assert all(left in exclude for left in dcmp.left_only) def test_url_mirror(mock_archive): diff --git a/lib/spack/spack/test/package_class.py b/lib/spack/spack/test/package_class.py index d540ac663e..33e5eb1c0a 100644 --- a/lib/spack/spack/test/package_class.py +++ b/lib/spack/spack/test/package_class.py @@ -10,7 +10,12 @@ etc.). Only methods like ``possible_dependencies()`` that deal with the static DSL metadata for packages. """ +import os import pytest +import shutil + +import llnl.util.filesystem as fs + import spack.package import spack.repo @@ -119,3 +124,72 @@ def test_possible_dependencies_with_multiple_classes( }) assert expected == spack.package.possible_dependencies(*pkgs) + + +def setup_install_test(source_paths, install_test_root): + """ + Set up the install test by creating sources and install test roots. + + The convention used here is to create an empty file if the path name + ends with an extension otherwise, a directory is created. + """ + fs.mkdirp(install_test_root) + for path in source_paths: + if os.path.splitext(path)[1]: + fs.touchp(path) + else: + fs.mkdirp(path) + + +@pytest.mark.parametrize('spec,sources,extras,expect', [ + ('a', + ['example/a.c'], # Source(s) + ['example/a.c'], # Extra test source + ['example/a.c']), # Test install dir source(s) + ('b', + ['test/b.cpp', 'test/b.hpp', 'example/b.txt'], # Source(s) + ['test'], # Extra test source + ['test/b.cpp', 'test/b.hpp']), # Test install dir source + ('c', + ['examples/a.py', 'examples/b.py', 'examples/c.py', 'tests/d.py'], + ['examples/b.py', 'tests'], + ['examples/b.py', 'tests/d.py']), +]) +def test_cache_extra_sources(install_mockery, spec, sources, extras, expect): + """Test the package's cache extra test sources helper function.""" + + pkg = spack.repo.get(spec) + pkg.spec.concretize() + source_path = pkg.stage.source_path + + srcs = [fs.join_path(source_path, s) for s in sources] + setup_install_test(srcs, pkg.install_test_root) + + emsg_dir = 'Expected {0} to be a directory' + emsg_file = 'Expected {0} to be a file' + for s in srcs: + assert os.path.exists(s), 'Expected {0} to exist'.format(s) + if os.path.splitext(s)[1]: + assert os.path.isfile(s), emsg_file.format(s) + else: + assert os.path.isdir(s), emsg_dir.format(s) + + pkg.cache_extra_test_sources(extras) + + src_dests = [fs.join_path(pkg.install_test_root, s) for s in sources] + exp_dests = [fs.join_path(pkg.install_test_root, e) for e in expect] + poss_dests = set(src_dests) | set(exp_dests) + + msg = 'Expected {0} to{1} exist' + for pd in poss_dests: + if pd in exp_dests: + assert os.path.exists(pd), msg.format(pd, '') + if os.path.splitext(pd)[1]: + assert os.path.isfile(pd), emsg_file.format(pd) + else: + assert os.path.isdir(pd), emsg_dir.format(pd) + else: + assert not os.path.exists(pd), msg.format(pd, ' not') + + # Perform a little cleanup + shutil.rmtree(os.path.dirname(source_path)) diff --git a/lib/spack/spack/test/test_suite.py b/lib/spack/spack/test/test_suite.py new file mode 100644 index 0000000000..1ec5106182 --- /dev/null +++ b/lib/spack/spack/test/test_suite.py @@ -0,0 +1,53 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) +import os +import spack.install_test +import spack.spec + + +def test_test_log_pathname(mock_packages, config): + """Ensure test log path is reasonable.""" + spec = spack.spec.Spec('libdwarf').concretized() + + test_name = 'test_name' + + test_suite = spack.install_test.TestSuite([spec], test_name) + logfile = test_suite.log_file_for_spec(spec) + + assert test_suite.stage in logfile + assert test_suite.test_log_name(spec) in logfile + + +def test_test_ensure_stage(mock_test_stage): + """Make sure test stage directory is properly set up.""" + spec = spack.spec.Spec('libdwarf').concretized() + + test_name = 'test_name' + + test_suite = spack.install_test.TestSuite([spec], test_name) + test_suite.ensure_stage() + + assert os.path.isdir(test_suite.stage) + assert mock_test_stage in test_suite.stage + + +def test_write_test_result(mock_packages, mock_test_stage): + """Ensure test results written to a results file.""" + spec = spack.spec.Spec('libdwarf').concretized() + result = 'TEST' + test_name = 'write-test' + + test_suite = spack.install_test.TestSuite([spec], test_name) + test_suite.ensure_stage() + results_file = test_suite.results_file + test_suite.write_test_result(spec, result) + + with open(results_file, 'r') as f: + lines = f.readlines() + assert len(lines) == 1 + + msg = lines[0] + assert result in msg + assert spec.name in msg diff --git a/lib/spack/spack/util/executable.py b/lib/spack/spack/util/executable.py index 097da3337e..614bc1725a 100644 --- a/lib/spack/spack/util/executable.py +++ b/lib/spack/spack/util/executable.py @@ -2,7 +2,7 @@ # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) - +import sys import os import re import shlex @@ -98,6 +98,9 @@ class Executable(object): If both ``output`` and ``error`` are set to ``str``, then one string is returned containing output concatenated with error. Not valid for ``input`` + * ``str.split``, as in the ``split`` method of the Python string type. + Behaves the same as ``str``, except that value is also written to + ``stdout`` or ``stderr``. By default, the subprocess inherits the parent's file descriptors. @@ -132,7 +135,7 @@ class Executable(object): def streamify(arg, mode): if isinstance(arg, string_types): return open(arg, mode), True - elif arg is str: + elif arg in (str, str.split): return subprocess.PIPE, False else: return arg, False @@ -168,12 +171,18 @@ class Executable(object): out, err = proc.communicate() result = None - if output is str or error is str: + if output in (str, str.split) or error in (str, str.split): result = '' - if output is str: - result += text_type(out.decode('utf-8')) - if error is str: - result += text_type(err.decode('utf-8')) + if output in (str, str.split): + outstr = text_type(out.decode('utf-8')) + result += outstr + if output is str.split: + sys.stdout.write(outstr) + if error in (str, str.split): + errstr = text_type(err.decode('utf-8')) + result += errstr + if error is str.split: + sys.stderr.write(errstr) rc = self.returncode = proc.returncode if fail_on_error and rc != 0 and (rc not in ignore_errors): diff --git a/lib/spack/spack/util/mock_package.py b/lib/spack/spack/util/mock_package.py index e855aae015..4751f5af7e 100644 --- a/lib/spack/spack/util/mock_package.py +++ b/lib/spack/spack/util/mock_package.py @@ -21,6 +21,8 @@ class MockPackageBase(object): Use ``MockPackageMultiRepo.add_package()`` to create new instances. """ + virtual = False + def __init__(self, dependencies, dependency_types, conditions=None, versions=None): """Instantiate a new MockPackageBase. @@ -92,7 +94,7 @@ class MockPackageMultiRepo(object): def exists(self, name): return name in self.spec_to_pkg - def is_virtual(self, name): + def is_virtual(self, name, use_index=True): return False def repo_for_pkg(self, name): diff --git a/share/spack/qa/completion-test.sh b/share/spack/qa/completion-test.sh index 5b326b4a6d..59a5181b98 100755 --- a/share/spack/qa/completion-test.sh +++ b/share/spack/qa/completion-test.sh @@ -56,7 +56,7 @@ contains 'hdf5' _spack_completions spack -d install --jobs 8 '' contains 'hdf5' _spack_completions spack install -v '' # XFAIL: Fails for Python 2.6 because pkg_resources not found? -#contains 'compilers.py' _spack_completions spack test '' +#contains 'compilers.py' _spack_completions spack unit-test '' title 'Testing debugging functions' diff --git a/share/spack/qa/run-unit-tests b/share/spack/qa/run-unit-tests index c529f8297e..ec8aaf76b9 100755 --- a/share/spack/qa/run-unit-tests +++ b/share/spack/qa/run-unit-tests @@ -42,4 +42,4 @@ spack -p --lines 20 spec mpileaks%gcc ^elfutils@0.170 #----------------------------------------------------------- # Run unit tests with code coverage #----------------------------------------------------------- -$coverage_run $(which spack) test -x --verbose +$coverage_run $(which spack) unit-test -x --verbose diff --git a/share/spack/spack-completion.bash b/share/spack/spack-completion.bash index 7d54414397..969a0898fe 100755 --- a/share/spack/spack-completion.bash +++ b/share/spack/spack-completion.bash @@ -320,7 +320,7 @@ _spack() { then SPACK_COMPREPLY="-h --help -H --all-help --color -C --config-scope -d --debug --timestamp --pdb -e --env -D --env-dir -E --no-env --use-env-repo -k --insecure -l --enable-locks -L --disable-locks -m --mock -p --profile --sorted-profile --lines -v --verbose --stacktrace -V --version --print-shell-vars" else - SPACK_COMPREPLY="activate add arch blame build-env buildcache cd checksum ci clean clone commands compiler compilers concretize config containerize create deactivate debug dependencies dependents deprecate dev-build develop docs edit env extensions external fetch find flake8 gc gpg graph help info install license list load location log-parse maintainers mirror module patch pkg providers pydoc python reindex remove rm repo resource restage setup solve spec stage test tutorial undevelop uninstall unload url verify versions view" + SPACK_COMPREPLY="activate add arch blame build-env buildcache cd checksum ci clean clone commands compiler compilers concretize config containerize create deactivate debug dependencies dependents deprecate dev-build develop docs edit env extensions external fetch find flake8 gc gpg graph help info install license list load location log-parse maintainers mirror module patch pkg providers pydoc python reindex remove rm repo resource restage setup solve spec stage test test-env tutorial undevelop uninstall unit-test unload url verify versions view" fi } @@ -1020,7 +1020,7 @@ _spack_info() { _spack_install() { if $list_options then - SPACK_COMPREPLY="-h --help --only -u --until -j --jobs --overwrite --fail-fast --keep-prefix --keep-stage --dont-restage --use-cache --no-cache --cache-only --no-check-signature --require-full-hash-match --show-log-on-error --source -n --no-checksum -v --verbose --fake --only-concrete -f --file --clean --dirty --test --run-tests --log-format --log-file --help-cdash -y --yes-to-all --cdash-upload-url --cdash-build --cdash-site --cdash-track --cdash-buildstamp" + SPACK_COMPREPLY="-h --help --only -u --until -j --jobs --overwrite --fail-fast --keep-prefix --keep-stage --dont-restage --use-cache --no-cache --cache-only --no-check-signature --require-full-hash-match --show-log-on-error --source -n --no-checksum -v --verbose --fake --only-concrete -f --file --clean --dirty --test --run-tests --log-format --log-file --help-cdash --cdash-upload-url --cdash-build --cdash-site --cdash-track --cdash-buildstamp -y --yes-to-all" else _all_packages fi @@ -1046,7 +1046,7 @@ _spack_license_verify() { _spack_list() { if $list_options then - SPACK_COMPREPLY="-h --help -d --search-description --format --update -t --tags" + SPACK_COMPREPLY="-h --help -d --search-description --format --update -v --virtuals -t --tags" else _all_packages fi @@ -1494,9 +1494,67 @@ _spack_stage() { _spack_test() { if $list_options then - SPACK_COMPREPLY="-h --help -H --pytest-help -l --list -L --list-long -N --list-names --extension -s -k --showlocals" + SPACK_COMPREPLY="-h --help" else - _tests + SPACK_COMPREPLY="run list find status results remove" + fi +} + +_spack_test_run() { + if $list_options + then + SPACK_COMPREPLY="-h --help --alias --fail-fast --fail-first --keep-stage --log-format --log-file --cdash-upload-url --cdash-build --cdash-site --cdash-track --cdash-buildstamp --help-cdash --clean --dirty" + else + _installed_packages + fi +} + +_spack_test_list() { + SPACK_COMPREPLY="-h --help" +} + +_spack_test_find() { + if $list_options + then + SPACK_COMPREPLY="-h --help" + else + _all_packages + fi +} + +_spack_test_status() { + if $list_options + then + SPACK_COMPREPLY="-h --help" + else + SPACK_COMPREPLY="" + fi +} + +_spack_test_results() { + if $list_options + then + SPACK_COMPREPLY="-h --help -l --logs -f --failed" + else + SPACK_COMPREPLY="" + fi +} + +_spack_test_remove() { + if $list_options + then + SPACK_COMPREPLY="-h --help -y --yes-to-all" + else + SPACK_COMPREPLY="" + fi +} + +_spack_test_env() { + if $list_options + then + SPACK_COMPREPLY="-h --help --clean --dirty --dump --pickle" + else + _all_packages fi } @@ -1522,6 +1580,15 @@ _spack_uninstall() { fi } +_spack_unit_test() { + if $list_options + then + SPACK_COMPREPLY="-h --help -H --pytest-help -l --list -L --list-long -N --list-names --extension -s -k --showlocals" + else + _tests + fi +} + _spack_unload() { if $list_options then diff --git a/share/spack/templates/reports/cdash/Test.xml b/share/spack/templates/reports/cdash/Test.xml new file mode 100644 index 0000000000..6aeed4e263 --- /dev/null +++ b/share/spack/templates/reports/cdash/Test.xml @@ -0,0 +1,27 @@ + <Test> + <StartTestTime>{{ test.starttime }}</StartTestTime> + <TestCommand>{{ install_command }}</TestCommand> +{% for warning in test.warnings %} + <Warning> + <TestLogLine>{{ warning.line_no }}</TestLogLine> + <Text>{{ warning.text }}</Text> + <SourceFile>{{ warning.source_file }}</SourceFile> + <SourceLineNumber>{{ warning.source_line_no }}</SourceLineNumber> + <PreContext>{{ warning.pre_context }}</PreContext> + <PostContext>{{ warning.post_context }}</PostContext> + </Warning> +{% endfor %} +{% for error in test.errors %} + <Error> + <TestLogLine>{{ error.line_no }}</TestLogLine> + <Text>{{ error.text }}</Text> + <SourceFile>{{ error.source_file }}</SourceFile> + <SourceLineNumber>{{ error.source_line_no }}</SourceLineNumber> + <PreContext>{{ error.pre_context }}</PreContext> + <PostContext>{{ error.post_context }}</PostContext> + </Error> +{% endfor %} + <EndTestTime>{{ test.endtime }}</EndTestTime> + <ElapsedMinutes>0</ElapsedMinutes> + </Test> +</Site> diff --git a/var/spack/repos/builtin.mock/packages/printing-package/package.py b/var/spack/repos/builtin.mock/packages/printing-package/package.py index 1d4d32d54d..096a49d211 100644 --- a/var/spack/repos/builtin.mock/packages/printing-package/package.py +++ b/var/spack/repos/builtin.mock/packages/printing-package/package.py @@ -24,3 +24,8 @@ class PrintingPackage(Package): make('install') print("AFTER INSTALL") + + def test(self): + print("BEFORE TEST") + self.run_test('true') # run /bin/true + print("AFTER TEST") diff --git a/var/spack/repos/builtin.mock/packages/test-error/package.py b/var/spack/repos/builtin.mock/packages/test-error/package.py new file mode 100644 index 0000000000..ce36ee7ca3 --- /dev/null +++ b/var/spack/repos/builtin.mock/packages/test-error/package.py @@ -0,0 +1,21 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +from spack import * + + +class TestError(Package): + """This package has a test method that fails in a subprocess.""" + + homepage = "http://www.example.com/test-failure" + url = "http://www.test-failure.test/test-failure-1.0.tar.gz" + + version('1.0', 'foobarbaz') + + def install(self, spec, prefix): + mkdirp(prefix.bin) + + def test(self): + self.run_test('false') diff --git a/var/spack/repos/builtin.mock/packages/test-fail/package.py b/var/spack/repos/builtin.mock/packages/test-fail/package.py new file mode 100644 index 0000000000..6587ef2bb9 --- /dev/null +++ b/var/spack/repos/builtin.mock/packages/test-fail/package.py @@ -0,0 +1,21 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +from spack import * + + +class TestFail(Package): + """This package has a test method that fails in a subprocess.""" + + homepage = "http://www.example.com/test-failure" + url = "http://www.test-failure.test/test-failure-1.0.tar.gz" + + version('1.0', 'foobarbaz') + + def install(self, spec, prefix): + mkdirp(prefix.bin) + + def test(self): + self.run_test('true', expected=['not in the output']) diff --git a/var/spack/repos/builtin/packages/bazel/package.py b/var/spack/repos/builtin/packages/bazel/package.py index 66c039cafb..ad1239529d 100644 --- a/var/spack/repos/builtin/packages/bazel/package.py +++ b/var/spack/repos/builtin/packages/bazel/package.py @@ -184,7 +184,7 @@ class Bazel(Package): @run_after('install') @on_package_attributes(run_tests=True) - def test(self): + def install_test(self): # https://github.com/Homebrew/homebrew-core/blob/master/Formula/bazel.rb # Bazel does not work properly on NFS, switch to /tmp diff --git a/var/spack/repos/builtin/packages/berkeley-db/package.py b/var/spack/repos/builtin/packages/berkeley-db/package.py index e72d823cb5..0385de81a1 100644 --- a/var/spack/repos/builtin/packages/berkeley-db/package.py +++ b/var/spack/repos/builtin/packages/berkeley-db/package.py @@ -3,8 +3,6 @@ # # SPDX-License-Identifier: (Apache-2.0 OR MIT) -from spack import * - class BerkeleyDb(AutotoolsPackage): """Oracle Berkeley DB""" @@ -47,3 +45,15 @@ class BerkeleyDb(AutotoolsPackage): config_args.append('--disable-atomicsupport') return config_args + + def test(self): + """Perform smoke tests on the installed package binaries.""" + exes = [ + 'db_checkpoint', 'db_deadlock', 'db_dump', 'db_load', + 'db_printlog', 'db_stat', 'db_upgrade', 'db_verify' + ] + for exe in exes: + reason = 'test version of {0} is {1}'.format(exe, + self.spec.version) + self.run_test(exe, ['-V'], [self.spec.version.string], + installed=True, purpose=reason, skip_missing=True) diff --git a/var/spack/repos/builtin/packages/binutils/package.py b/var/spack/repos/builtin/packages/binutils/package.py index e2ddfe8c1e..f79015cf6b 100644 --- a/var/spack/repos/builtin/packages/binutils/package.py +++ b/var/spack/repos/builtin/packages/binutils/package.py @@ -129,3 +129,29 @@ class Binutils(AutotoolsPackage, GNUMirrorPackage): if self.spec.satisfies('@:2.34 %gcc@10:'): flags.append('-fcommon') return (flags, None, None) + + def test(self): + spec_vers = str(self.spec.version) + + checks = { + 'ar': spec_vers, + 'c++filt': spec_vers, + 'coffdump': spec_vers, + 'dlltool': spec_vers, + 'elfedit': spec_vers, + 'gprof': spec_vers, + 'ld': spec_vers, + 'nm': spec_vers, + 'objdump': spec_vers, + 'ranlib': spec_vers, + 'readelf': spec_vers, + 'size': spec_vers, + 'strings': spec_vers, + } + + for exe in checks: + expected = checks[exe] + reason = 'test: ensuring version of {0} is {1}' \ + .format(exe, expected) + self.run_test(exe, '--version', expected, installed=True, + purpose=reason, skip_missing=True) diff --git a/var/spack/repos/builtin/packages/c/package.py b/var/spack/repos/builtin/packages/c/package.py new file mode 100644 index 0000000000..72a3343aa1 --- /dev/null +++ b/var/spack/repos/builtin/packages/c/package.py @@ -0,0 +1,27 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +import os + + +class C(Package): + """Virtual package for C compilers.""" + homepage = 'http://open-std.org/JTC1/SC22/WG14/www/standards' + virtual = True + + def test(self): + test_source = self.test_suite.current_test_data_dir + + for test in os.listdir(test_source): + filepath = test_source.join(test) + exe_name = '%s.exe' % test + + cc_exe = os.environ['CC'] + cc_opts = ['-o', exe_name, filepath] + compiled = self.run_test(cc_exe, options=cc_opts, installed=True) + + if compiled: + expected = ['Hello world', 'YES!'] + self.run_test(exe_name, expected=expected) diff --git a/var/spack/repos/builtin/packages/c/test/hello.c b/var/spack/repos/builtin/packages/c/test/hello.c new file mode 100644 index 0000000000..de950e1e88 --- /dev/null +++ b/var/spack/repos/builtin/packages/c/test/hello.c @@ -0,0 +1,7 @@ +#include <stdio.h> +int main() +{ + printf ("Hello world from C!\n"); + printf ("YES!"); + return 0; +} diff --git a/var/spack/repos/builtin/packages/cantera/package.py b/var/spack/repos/builtin/packages/cantera/package.py index 773cc6d8cf..9846fff48d 100644 --- a/var/spack/repos/builtin/packages/cantera/package.py +++ b/var/spack/repos/builtin/packages/cantera/package.py @@ -146,7 +146,7 @@ class Cantera(SConsPackage): return args - def test(self): + def build_test(self): if '+python' in self.spec: # Tests will always fail if Python dependencies aren't built # In addition, 3 of the tests fail when run in parallel diff --git a/var/spack/repos/builtin/packages/cmake/package.py b/var/spack/repos/builtin/packages/cmake/package.py index 97168c05f1..a3bad699e1 100644 --- a/var/spack/repos/builtin/packages/cmake/package.py +++ b/var/spack/repos/builtin/packages/cmake/package.py @@ -2,6 +2,7 @@ # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) + import re @@ -250,7 +251,7 @@ class Cmake(Package): @run_after('build') @on_package_attributes(run_tests=True) - def test(self): + def build_test(self): # Some tests fail, takes forever make('test') @@ -262,3 +263,12 @@ class Cmake(Package): filter_file('mpcc_r)', 'mpcc_r mpifcc)', f, string=True) filter_file('mpc++_r)', 'mpc++_r mpiFCC)', f, string=True) filter_file('mpifc)', 'mpifc mpifrt)', f, string=True) + + def test(self): + """Perform smoke tests on the installed package.""" + spec_vers_str = 'version {0}'.format(self.spec.version) + + for exe in ['ccmake', 'cmake', 'cpack', 'ctest']: + reason = 'test version of {0} is {1}'.format(exe, spec_vers_str) + self.run_test(exe, ['--version'], [spec_vers_str], + installed=True, purpose=reason, skip_missing=True) diff --git a/var/spack/repos/builtin/packages/conduit/package.py b/var/spack/repos/builtin/packages/conduit/package.py index 7b6edf21f9..57d49da70e 100644 --- a/var/spack/repos/builtin/packages/conduit/package.py +++ b/var/spack/repos/builtin/packages/conduit/package.py @@ -217,7 +217,7 @@ class Conduit(Package): @run_after('build') @on_package_attributes(run_tests=True) - def test(self): + def build_test(self): with working_dir('spack-build'): print("Running Conduit Unit Tests...") make("test") diff --git a/var/spack/repos/builtin/packages/cxx/package.py b/var/spack/repos/builtin/packages/cxx/package.py new file mode 100644 index 0000000000..0be36c3ae5 --- /dev/null +++ b/var/spack/repos/builtin/packages/cxx/package.py @@ -0,0 +1,38 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +import os + + +class Cxx(Package): + """Virtual package for the C++ language.""" + homepage = 'https://isocpp.org/std/the-standard' + virtual = True + + def test(self): + test_source = self.test_suite.current_test_data_dir + + for test in os.listdir(test_source): + filepath = os.path.join(test_source, test) + exe_name = '%s.exe' % test + + cxx_exe = os.environ['CXX'] + + # standard options + # Hack to get compiler attributes + # TODO: remove this when compilers are dependencies + c_name = clang if self.spec.satisfies('llvm+clang') else self.name + c_spec = spack.spec.CompilerSpec(c_name, self.spec.version) + c_cls = spack.compilers.class_for_compiler_name(c_name) + compiler = c_cls(c_spec, None, None, ['fakecc', 'fakecxx']) + + cxx_opts = [compiler.cxx11_flag] if 'c++11' in test else [] + + cxx_opts += ['-o', exe_name, filepath] + compiled = self.run_test(cxx_exe, options=cxx_opts, installed=True) + + if compiled: + expected = ['Hello world', 'YES!'] + self.run_test(exe_name, expected=expected) diff --git a/var/spack/repos/builtin/packages/cxx/test/hello.c++ b/var/spack/repos/builtin/packages/cxx/test/hello.c++ new file mode 100644 index 0000000000..f0ad7caffb --- /dev/null +++ b/var/spack/repos/builtin/packages/cxx/test/hello.c++ @@ -0,0 +1,9 @@ +#include <stdio.h> + +int main() +{ + printf ("Hello world from C++\n"); + printf ("YES!"); + + return 0; +} diff --git a/var/spack/repos/builtin/packages/cxx/test/hello.cc b/var/spack/repos/builtin/packages/cxx/test/hello.cc new file mode 100644 index 0000000000..2a85869996 --- /dev/null +++ b/var/spack/repos/builtin/packages/cxx/test/hello.cc @@ -0,0 +1,9 @@ +#include <iostream> +using namespace std; + +int main() +{ + cout << "Hello world from C++!" << endl; + cout << "YES!" << endl; + return (0); +} diff --git a/var/spack/repos/builtin/packages/cxx/test/hello.cpp b/var/spack/repos/builtin/packages/cxx/test/hello.cpp new file mode 100644 index 0000000000..b49db59f4a --- /dev/null +++ b/var/spack/repos/builtin/packages/cxx/test/hello.cpp @@ -0,0 +1,9 @@ +#include <iostream> +using namespace std; + +int main() +{ + cout << "Hello world from C++!" << endl; + cout << "YES!" << endl; + return (0); +} diff --git a/var/spack/repos/builtin/packages/cxx/test/hello_c++11.cc b/var/spack/repos/builtin/packages/cxx/test/hello_c++11.cc new file mode 100644 index 0000000000..10f57c3f75 --- /dev/null +++ b/var/spack/repos/builtin/packages/cxx/test/hello_c++11.cc @@ -0,0 +1,17 @@ +#include <iostream> +#include <regex> + +using namespace std; + +int main() +{ + auto func = [] () { cout << "Hello world from C++11" << endl; }; + func(); // now call the function + + std::regex r("st|mt|tr"); + std::cout << "std::regex r(\"st|mt|tr\")" << " match tr? "; + if (std::regex_match("tr", r) == 0) + std::cout << "NO!\n ==> Using pre g++ 4.9.2 libstdc++ which doesn't implement regex properly" << std::endl; + else + std::cout << "YES!\n ==> Correct libstdc++11 implementation of regex (4.9.2 or later)" << std::endl; +} diff --git a/var/spack/repos/builtin/packages/emacs/package.py b/var/spack/repos/builtin/packages/emacs/package.py index 393def5cc5..0759fd28d5 100644 --- a/var/spack/repos/builtin/packages/emacs/package.py +++ b/var/spack/repos/builtin/packages/emacs/package.py @@ -80,3 +80,18 @@ class Emacs(AutotoolsPackage, GNUMirrorPackage): args.append('--without-gnutls') return args + + def _test_check_versions(self): + """Perform version checks on installed package binaries.""" + checks = ['ctags', 'ebrowse', 'emacs', 'emacsclient', 'etags'] + + for exe in checks: + expected = str(self.spec.version) + reason = 'test version of {0} is {1}'.format(exe, expected) + self.run_test(exe, ['--version'], expected, installed=True, + purpose=reason, skip_missing=True) + + def test(self): + """Perform smoke tests on the installed package.""" + # Simple version check tests on known binaries + self._test_check_versions() diff --git a/var/spack/repos/builtin/packages/fortran/package.py b/var/spack/repos/builtin/packages/fortran/package.py new file mode 100644 index 0000000000..6383ff856b --- /dev/null +++ b/var/spack/repos/builtin/packages/fortran/package.py @@ -0,0 +1,28 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +import os + + +class Fortran(Package): + """Virtual package for the Fortran language.""" + homepage = 'https://wg5-fortran.org/' + virtual = True + + def test(self): + test_source = self.test_suite.current_test_data_dir + + for test in os.listdir(test_source): + filepath = os.path.join(test_source, test) + exe_name = '%s.exe' % test + + fc_exe = os.environ['FC'] + fc_opts = ['-o', exe_name, filepath] + + compiled = self.run_test(fc_exe, options=fc_opts, installed=True) + + if compiled: + expected = ['Hello world', 'YES!'] + self.run_test(exe_name, expected=expected) diff --git a/var/spack/repos/builtin/packages/fortran/test/hello.F b/var/spack/repos/builtin/packages/fortran/test/hello.F new file mode 100644 index 0000000000..886046eaed --- /dev/null +++ b/var/spack/repos/builtin/packages/fortran/test/hello.F @@ -0,0 +1,6 @@ + program line + + write (*,*) "Hello world from FORTRAN" + write (*,*) "YES!" + + end diff --git a/var/spack/repos/builtin/packages/fortran/test/hello.f90 b/var/spack/repos/builtin/packages/fortran/test/hello.f90 new file mode 100644 index 0000000000..21717d11dd --- /dev/null +++ b/var/spack/repos/builtin/packages/fortran/test/hello.f90 @@ -0,0 +1,6 @@ +program line + + write (*,*) "Hello world from FORTRAN" + write (*,*) "YES!" + +end program line diff --git a/var/spack/repos/builtin/packages/gdal/package.py b/var/spack/repos/builtin/packages/gdal/package.py index 1162ffe9ae..a6f1dfd909 100644 --- a/var/spack/repos/builtin/packages/gdal/package.py +++ b/var/spack/repos/builtin/packages/gdal/package.py @@ -124,7 +124,7 @@ class Gdal(AutotoolsPackage): depends_on('hdf5', when='+hdf5') depends_on('kealib', when='+kea @2:') depends_on('netcdf-c', when='+netcdf') - depends_on('jasper@1.900.1', patches='uuid.patch', when='+jasper') + depends_on('jasper@1.900.1', patches=[patch('uuid.patch')], when='+jasper') depends_on('openjpeg', when='+openjpeg') depends_on('xerces-c', when='+xerces') depends_on('expat', when='+expat') diff --git a/var/spack/repos/builtin/packages/hdf/package.py b/var/spack/repos/builtin/packages/hdf/package.py index d40a0c21fe..76c2205f27 100644 --- a/var/spack/repos/builtin/packages/hdf/package.py +++ b/var/spack/repos/builtin/packages/hdf/package.py @@ -4,6 +4,7 @@ # SPDX-License-Identifier: (Apache-2.0 OR MIT) import sys +import os class Hdf(AutotoolsPackage): @@ -151,3 +152,67 @@ class Hdf(AutotoolsPackage): def check(self): with working_dir(self.build_directory): make('check', parallel=False) + + extra_install_tests = 'hdf/util/testfiles' + + @run_after('install') + def setup_build_tests(self): + """Copy the build test files after the package is installed to an + install test subdirectory for use during `spack test run`.""" + self.cache_extra_test_sources(self.extra_install_tests) + + def _test_check_versions(self): + """Perform version checks on selected installed package binaries.""" + spec_vers_str = 'Version {0}'.format(self.spec.version.up_to(2)) + + exes = ['hdfimport', 'hrepack', 'ncdump', 'ncgen'] + for exe in exes: + reason = 'test: ensuring version of {0} is {1}' \ + .format(exe, spec_vers_str) + self.run_test(exe, ['-V'], spec_vers_str, installed=True, + purpose=reason, skip_missing=True) + + def _test_gif_converters(self): + """This test performs an image conversion sequence and diff.""" + work_dir = '.' + storm_fn = os.path.join(self.install_test_root, + self.extra_install_tests, 'storm110.hdf') + gif_fn = 'storm110.gif' + new_hdf_fn = 'storm110gif.hdf' + + # Convert a test HDF file to a gif + self.run_test('hdf2gif', [storm_fn, gif_fn], '', installed=True, + purpose="test: hdf-to-gif", work_dir=work_dir) + + # Convert the gif to an HDF file + self.run_test('gif2hdf', [gif_fn, new_hdf_fn], '', installed=True, + purpose="test: gif-to-hdf", work_dir=work_dir) + + # Compare the original and new HDF files + self.run_test('hdiff', [new_hdf_fn, storm_fn], '', installed=True, + purpose="test: compare orig to new hdf", + work_dir=work_dir) + + def _test_list(self): + """This test compares low-level HDF file information to expected.""" + storm_fn = os.path.join(self.install_test_root, + self.extra_install_tests, 'storm110.hdf') + test_data_dir = self.test_suite.current_test_data_dir + work_dir = '.' + + reason = 'test: checking hdfls output' + details_file = os.path.join(test_data_dir, 'storm110.out') + expected = get_escaped_text_output(details_file) + self.run_test('hdfls', [storm_fn], expected, installed=True, + purpose=reason, skip_missing=True, work_dir=work_dir) + + def test(self): + """Perform smoke tests on the installed package.""" + # Simple version check tests on subset of known binaries that respond + self._test_check_versions() + + # Run gif converter sequence test + self._test_gif_converters() + + # Run hdfls output + self._test_list() diff --git a/var/spack/repos/builtin/packages/hdf/test/storm110.out b/var/spack/repos/builtin/packages/hdf/test/storm110.out new file mode 100644 index 0000000000..f17e4ce2b3 --- /dev/null +++ b/var/spack/repos/builtin/packages/hdf/test/storm110.out @@ -0,0 +1,17 @@ +File library version: Major= 0, Minor=0, Release=0 +String= + +Number type : (tag 106) + Ref nos: 110 +Machine type : (tag 107) + Ref nos: 4369 +Image Dimensions-8 : (tag 200) + Ref nos: 110 +Raster Image-8 : (tag 202) + Ref nos: 110 +Image Dimensions : (tag 300) + Ref nos: 110 +Raster Image Data : (tag 302) + Ref nos: 110 +Raster Image Group : (tag 306) + Ref nos: 110 diff --git a/var/spack/repos/builtin/packages/hdf5/package.py b/var/spack/repos/builtin/packages/hdf5/package.py index e592ab5e9f..b37c7ede65 100644 --- a/var/spack/repos/builtin/packages/hdf5/package.py +++ b/var/spack/repos/builtin/packages/hdf5/package.py @@ -6,8 +6,6 @@ import shutil import sys -from spack import * - class Hdf5(AutotoolsPackage): """HDF5 is a data model, library, and file format for storing and managing @@ -327,6 +325,9 @@ class Hdf5(AutotoolsPackage): @run_after('install') @on_package_attributes(run_tests=True) def check_install(self): + self._check_install() + + def _check_install(self): # Build and run a small program to test the installed HDF5 library spec = self.spec print("Checking HDF5 installation...") @@ -375,3 +376,55 @@ HDF5 version {version} {version} print('-' * 80) raise RuntimeError("HDF5 install check failed") shutil.rmtree(checkdir) + + def _test_check_versions(self): + """Perform version checks on selected installed package binaries.""" + spec_vers_str = 'Version {0}'.format(self.spec.version) + + exes = [ + 'h5copy', 'h5diff', 'h5dump', 'h5format_convert', 'h5ls', + 'h5mkgrp', 'h5repack', 'h5stat', 'h5unjam', + ] + use_short_opt = ['h52gif', 'h5repart', 'h5unjam'] + for exe in exes: + reason = 'test: ensuring version of {0} is {1}' \ + .format(exe, spec_vers_str) + option = '-V' if exe in use_short_opt else '--version' + self.run_test(exe, option, spec_vers_str, installed=True, + purpose=reason, skip_missing=True) + + def _test_example(self): + """This test performs copy, dump, and diff on an example hdf5 file.""" + test_data_dir = self.test_suite.current_test_data_dir + + filename = 'spack.h5' + h5_file = test_data_dir.join(filename) + + reason = 'test: ensuring h5dump produces expected output' + expected = get_escaped_text_output(test_data_dir.join('dump.out')) + self.run_test('h5dump', filename, expected, installed=True, + purpose=reason, skip_missing=True, + work_dir=test_data_dir) + + reason = 'test: ensuring h5copy runs' + options = ['-i', h5_file, '-s', 'Spack', '-o', 'test.h5', '-d', + 'Spack'] + self.run_test('h5copy', options, [], installed=True, + purpose=reason, skip_missing=True, work_dir='.') + + reason = ('test: ensuring h5diff shows no differences between orig and' + ' copy') + self.run_test('h5diff', [h5_file, 'test.h5'], [], installed=True, + purpose=reason, skip_missing=True, work_dir='.') + + def test(self): + """Perform smoke tests on the installed package.""" + # Simple version check tests on known binaries + self._test_check_versions() + + # Run sequence of commands on an hdf5 file + self._test_example() + + # Run existing install check + # TODO: Restore once address built vs. installed state + # self._check_install() diff --git a/var/spack/repos/builtin/packages/hdf5/test/dump.out b/var/spack/repos/builtin/packages/hdf5/test/dump.out new file mode 100644 index 0000000000..58decefc12 --- /dev/null +++ b/var/spack/repos/builtin/packages/hdf5/test/dump.out @@ -0,0 +1,45 @@ +HDF5 "spack.h5" { +GROUP "/" { + GROUP "Spack" { + GROUP "Software" { + ATTRIBUTE "Distribution" { + DATATYPE H5T_STRING { + STRSIZE H5T_VARIABLE; + STRPAD H5T_STR_NULLTERM; + CSET H5T_CSET_UTF8; + CTYPE H5T_C_S1; + } + DATASPACE SCALAR + DATA { + (0): "Open Source" + } + } + DATASET "data" { + DATATYPE H5T_IEEE_F64LE + DATASPACE SIMPLE { ( 7, 11 ) / ( 7, 11 ) } + DATA { + (0,0): 0.371141, 0.508482, 0.585975, 0.0944911, 0.684849, + (0,5): 0.580396, 0.720271, 0.693561, 0.340432, 0.217145, + (0,10): 0.636083, + (1,0): 0.686996, 0.773501, 0.656767, 0.617543, 0.226132, + (1,5): 0.768632, 0.0548711, 0.54572, 0.355544, 0.591548, + (1,10): 0.233007, + (2,0): 0.230032, 0.192087, 0.293845, 0.0369338, 0.038727, + (2,5): 0.0977931, 0.966522, 0.0821391, 0.857921, 0.495703, + (2,10): 0.746006, + (3,0): 0.598494, 0.990266, 0.993009, 0.187481, 0.746391, + (3,5): 0.140095, 0.122661, 0.929242, 0.542415, 0.802758, + (3,10): 0.757941, + (4,0): 0.372124, 0.411982, 0.270479, 0.950033, 0.329948, + (4,5): 0.936704, 0.105097, 0.742285, 0.556565, 0.18988, 0.72797, + (5,0): 0.801669, 0.271807, 0.910649, 0.186251, 0.868865, + (5,5): 0.191484, 0.788371, 0.920173, 0.582249, 0.682022, + (5,10): 0.146883, + (6,0): 0.826824, 0.0886705, 0.402606, 0.0532444, 0.72509, + (6,5): 0.964683, 0.330362, 0.833284, 0.630456, 0.411489, 0.247806 + } + } + } + } +} +} diff --git a/var/spack/repos/builtin/packages/hdf5/test/spack.h5 b/var/spack/repos/builtin/packages/hdf5/test/spack.h5 Binary files differnew file mode 100644 index 0000000000..c2f3a6f39d --- /dev/null +++ b/var/spack/repos/builtin/packages/hdf5/test/spack.h5 diff --git a/var/spack/repos/builtin/packages/jq/package.py b/var/spack/repos/builtin/packages/jq/package.py index 13d3d939a2..9f67ce5bbe 100644 --- a/var/spack/repos/builtin/packages/jq/package.py +++ b/var/spack/repos/builtin/packages/jq/package.py @@ -21,7 +21,7 @@ class Jq(AutotoolsPackage): @run_after('install') @on_package_attributes(run_tests=True) - def installtest(self): + def install_test(self): jq = self.spec['jq'].command f = os.path.join(os.path.dirname(__file__), 'input.json') diff --git a/var/spack/repos/builtin/packages/kcov/package.py b/var/spack/repos/builtin/packages/kcov/package.py index 8f01ffc985..5e7bf48bd5 100644 --- a/var/spack/repos/builtin/packages/kcov/package.py +++ b/var/spack/repos/builtin/packages/kcov/package.py @@ -27,7 +27,7 @@ class Kcov(CMakePackage): @run_after('install') @on_package_attributes(run_tests=True) - def test(self): + def test_install(self): # The help message exits with an exit code of 1 kcov = Executable(self.prefix.bin.kcov) kcov('-h', ignore_errors=1) diff --git a/var/spack/repos/builtin/packages/libsigsegv/package.py b/var/spack/repos/builtin/packages/libsigsegv/package.py index 7aab695b76..119778f018 100644 --- a/var/spack/repos/builtin/packages/libsigsegv/package.py +++ b/var/spack/repos/builtin/packages/libsigsegv/package.py @@ -3,8 +3,6 @@ # # SPDX-License-Identifier: (Apache-2.0 OR MIT) -from spack import * - class Libsigsegv(AutotoolsPackage, GNUMirrorPackage): """GNU libsigsegv is a library for handling page faults in user mode.""" @@ -18,5 +16,60 @@ class Libsigsegv(AutotoolsPackage, GNUMirrorPackage): patch('patch.new_config_guess', when='@2.10') + test_requires_compiler = True + def configure_args(self): return ['--enable-shared'] + + extra_install_tests = 'tests/.libs' + + @run_after('install') + def setup_build_tests(self): + """Copy the build test files after the package is installed to an + install test subdirectory for use during `spack test run`.""" + self.cache_extra_test_sources(self.extra_install_tests) + + def _run_smoke_tests(self): + """Build and run the added smoke (install) test.""" + data_dir = self.test_suite.current_test_data_dir + prog = 'smoke_test' + src = data_dir.join('{0}.c'.format(prog)) + + options = [ + '-I{0}'.format(self.prefix.include), + src, + '-o', + prog, + '-L{0}'.format(self.prefix.lib), + '-lsigsegv', + '{0}{1}'.format(self.compiler.cc_rpath_arg, self.prefix.lib)] + reason = 'test: checking ability to link to the library' + self.run_test('cc', options, [], installed=False, purpose=reason) + + # Now run the program and confirm the output matches expectations + expected = get_escaped_text_output(data_dir.join('smoke_test.out')) + reason = 'test: checking ability to use the library' + self.run_test(prog, [], expected, purpose=reason) + + def _run_build_tests(self): + """Run selected build tests.""" + passed = 'Test passed' + checks = { + 'sigsegv1': [passed], + 'sigsegv2': [passed], + 'sigsegv3': ['caught', passed], + 'stackoverflow1': ['recursion', 'Stack overflow', passed], + 'stackoverflow2': ['recursion', 'overflow', 'violation', passed], + } + + for exe, expected in checks.items(): + reason = 'test: checking {0} output'.format(exe) + self.run_test(exe, [], expected, installed=True, purpose=reason, + skip_missing=True) + + def test(self): + # Run the simple built-in smoke test + self._run_smoke_tests() + + # Run test programs pulled from the build + self._run_build_tests() diff --git a/var/spack/repos/builtin/packages/libsigsegv/test/smoke_test.c b/var/spack/repos/builtin/packages/libsigsegv/test/smoke_test.c new file mode 100644 index 0000000000..f1ab68cd53 --- /dev/null +++ b/var/spack/repos/builtin/packages/libsigsegv/test/smoke_test.c @@ -0,0 +1,70 @@ +/* Simple "Hello World" test set up to handle a single page fault + * + * Inspired by libsigsegv's test cases with argument names for handlers + * taken from the header files. + */ + +#include "sigsegv.h" +#include <stdio.h> +#include <stdlib.h> /* for exit */ +# include <stddef.h> /* for NULL on SunOS4 (per libsigsegv examples) */ +#include <setjmp.h> /* for controlling handler-related flow */ + + +/* Calling environment */ +jmp_buf calling_env; + +char *message = "Hello, World!"; + +/* Track the number of times the handler is called */ +volatile int times_called = 0; + + +/* Continuation function, which relies on the latest libsigsegv API */ +static void +resume(void *cont_arg1, void *cont_arg2, void *cont_arg3) +{ + /* Go to calling environment and restore state. */ + longjmp(calling_env, times_called); +} + +/* sigsegv handler */ +int +handle_sigsegv(void *fault_address, int serious) +{ + times_called++; + + /* Generate handler output for the test. */ + printf("Caught sigsegv #%d\n", times_called); + + return sigsegv_leave_handler(resume, NULL, NULL, NULL); +} + +/* "Buggy" function used to demonstrate non-local goto */ +void printit(char *m) +{ + if (times_called < 1) { + /* Force SIGSEGV only on the first call. */ + volatile int *fail_ptr = 0; + int failure = *fail_ptr; + printf("%s\n", m); + } else { + /* Print it correctly. */ + printf("%s\n", m); + } +} + +int +main(void) +{ + /* Install the global SIGSEGV handler */ + sigsegv_install_handler(&handle_sigsegv); + + char *msg = "Hello World!"; + int calls = setjmp(calling_env); /* Resume here after detecting sigsegv */ + + /* Call the function that will trigger the page fault. */ + printit(msg); + + return 0; +} diff --git a/var/spack/repos/builtin/packages/libsigsegv/test/smoke_test.out b/var/spack/repos/builtin/packages/libsigsegv/test/smoke_test.out new file mode 100644 index 0000000000..31071777e2 --- /dev/null +++ b/var/spack/repos/builtin/packages/libsigsegv/test/smoke_test.out @@ -0,0 +1,2 @@ +Caught sigsegv #1 +Hello World! diff --git a/var/spack/repos/builtin/packages/libxml2/package.py b/var/spack/repos/builtin/packages/libxml2/package.py index 2602378f89..9cbc8a6817 100644 --- a/var/spack/repos/builtin/packages/libxml2/package.py +++ b/var/spack/repos/builtin/packages/libxml2/package.py @@ -2,6 +2,8 @@ # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) +import llnl.util.filesystem as fs +import llnl.util.tty as tty from spack import * @@ -82,3 +84,34 @@ class Libxml2(AutotoolsPackage): if '+python' in self.spec: with working_dir('spack-test', create=True): python('-c', 'import libxml2') + + def test(self): + """Perform smoke tests on the installed package""" + # Start with what we already have post-install + tty.msg('test: Performing simple import test') + self.import_module_test() + + data_dir = self.test_suite.current_test_data_dir + + # Now run defined tests based on expected executables + dtd_path = data_dir.join('info.dtd') + test_filename = 'test.xml' + exec_checks = { + 'xml2-config': [ + ('--version', [str(self.spec.version)], 0)], + 'xmllint': [ + (['--auto', '-o', test_filename], [], 0), + (['--postvalid', test_filename], + ['validity error', 'no DTD found', 'does not validate'], 3), + (['--dtdvalid', dtd_path, test_filename], + ['validity error', 'does not follow the DTD'], 3), + (['--dtdvalid', dtd_path, data_dir.join('info.xml')], [], 0)], + 'xmlcatalog': [ + ('--create', ['<catalog xmlns', 'catalog"/>'], 0)], + } + for exe in exec_checks: + for options, expected, status in exec_checks[exe]: + self.run_test(exe, options, expected, status) + + # Perform some cleanup + fs.force_remove(test_filename) diff --git a/var/spack/repos/builtin/packages/libxml2/test/info.dtd b/var/spack/repos/builtin/packages/libxml2/test/info.dtd new file mode 100644 index 0000000000..aec2dbe705 --- /dev/null +++ b/var/spack/repos/builtin/packages/libxml2/test/info.dtd @@ -0,0 +1,2 @@ +<!ELEMENT info (data)> +<!ELEMENT data (#PCDATA)> diff --git a/var/spack/repos/builtin/packages/libxml2/test/info.xml b/var/spack/repos/builtin/packages/libxml2/test/info.xml new file mode 100644 index 0000000000..23803694a7 --- /dev/null +++ b/var/spack/repos/builtin/packages/libxml2/test/info.xml @@ -0,0 +1,4 @@ +<?xml version="1.0"?> +<info> +<data>abc</data> +</info> diff --git a/var/spack/repos/builtin/packages/m4/package.py b/var/spack/repos/builtin/packages/m4/package.py index 6695cdf862..b0d037b3f2 100644 --- a/var/spack/repos/builtin/packages/m4/package.py +++ b/var/spack/repos/builtin/packages/m4/package.py @@ -74,3 +74,16 @@ class M4(AutotoolsPackage, GNUMirrorPackage): args.append('ac_cv_type_struct_sched_param=yes') return args + + def test(self): + spec_vers = str(self.spec.version) + reason = 'test: ensuring m4 version is {0}'.format(spec_vers) + self.run_test('m4', '--version', spec_vers, installed=True, + purpose=reason, skip_missing=False) + + reason = 'test: ensuring m4 example succeeds' + test_data_dir = self.test_suite.current_test_data_dir + hello_file = test_data_dir.join('hello.m4') + expected = get_escaped_text_output(test_data_dir.join('hello.out')) + self.run_test('m4', hello_file, expected, installed=True, + purpose=reason, skip_missing=False) diff --git a/var/spack/repos/builtin/packages/m4/test/hello.m4 b/var/spack/repos/builtin/packages/m4/test/hello.m4 new file mode 100644 index 0000000000..6132c41093 --- /dev/null +++ b/var/spack/repos/builtin/packages/m4/test/hello.m4 @@ -0,0 +1,4 @@ +define(NAME, World) +dnl This line should not show up +// macro is ifdef(`NAME', , not)defined +Hello, NAME! diff --git a/var/spack/repos/builtin/packages/m4/test/hello.out b/var/spack/repos/builtin/packages/m4/test/hello.out new file mode 100644 index 0000000000..c8d3be7e16 --- /dev/null +++ b/var/spack/repos/builtin/packages/m4/test/hello.out @@ -0,0 +1,3 @@ + +// macro is defined +Hello, World! diff --git a/var/spack/repos/builtin/packages/mpi/package.py b/var/spack/repos/builtin/packages/mpi/package.py new file mode 100644 index 0000000000..731a5ac731 --- /dev/null +++ b/var/spack/repos/builtin/packages/mpi/package.py @@ -0,0 +1,31 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +import os + + +class Mpi(Package): + """Virtual package for the Message Passing Interface.""" + homepage = 'https://www.mpi-forum.org/' + virtual = True + + def test(self): + for lang in ('c', 'f'): + filename = self.test_suite.current_test_data_dir.join( + 'mpi_hello.' + lang) + + compiler_var = 'MPICC' if lang == 'c' else 'MPIF90' + compiler = os.environ[compiler_var] + + exe_name = 'mpi_hello_%s' % lang + mpirun = join_path(self.prefix.bin, 'mpirun') + + compiled = self.run_test(compiler, + options=['-o', exe_name, filename]) + if compiled: + self.run_test(mpirun, + options=['-np', '1', exe_name], + expected=[r'Hello world! From rank \s*0 of \s*1'] + ) diff --git a/var/spack/repos/builtin/packages/mpi/test/mpi_hello.c b/var/spack/repos/builtin/packages/mpi/test/mpi_hello.c new file mode 100644 index 0000000000..9db7c5a436 --- /dev/null +++ b/var/spack/repos/builtin/packages/mpi/test/mpi_hello.c @@ -0,0 +1,16 @@ +#include <stdio.h> +#include <mpi.h> + +int main(int argc, char** argv) { + MPI_Init(&argc, &argv); + + int rank; + int num_ranks; + MPI_Comm_rank(MPI_COMM_WORLD, &rank); + MPI_Comm_size(MPI_COMM_WORLD, &num_ranks); + + printf("Hello world! From rank %d of %d\n", rank, num_ranks); + + MPI_Finalize(); + return(0); +} diff --git a/var/spack/repos/builtin/packages/mpi/test/mpi_hello.f b/var/spack/repos/builtin/packages/mpi/test/mpi_hello.f new file mode 100644 index 0000000000..ecc7005d00 --- /dev/null +++ b/var/spack/repos/builtin/packages/mpi/test/mpi_hello.f @@ -0,0 +1,11 @@ +c Fortran example + program hello + include 'mpif.h' + integer rank, num_ranks, err_flag + + call MPI_INIT(err_flag) + call MPI_COMM_SIZE(MPI_COMM_WORLD, num_ranks, err_flag) + call MPI_COMM_RANK(MPI_COMM_WORLD, rank, err_flag) + print*, 'Hello world! From rank', rank, 'of ', num_ranks + call MPI_FINALIZE(err_flag) + end diff --git a/var/spack/repos/builtin/packages/ninja-fortran/package.py b/var/spack/repos/builtin/packages/ninja-fortran/package.py index 8e9fcb9851..d85a5b4542 100644 --- a/var/spack/repos/builtin/packages/ninja-fortran/package.py +++ b/var/spack/repos/builtin/packages/ninja-fortran/package.py @@ -51,7 +51,7 @@ class NinjaFortran(Package): @run_after('configure') @on_package_attributes(run_tests=True) - def test(self): + def configure_test(self): ninja = Executable('./ninja') ninja('-j{0}'.format(make_jobs), 'ninja_test') ninja_test = Executable('./ninja_test') diff --git a/var/spack/repos/builtin/packages/ninja/package.py b/var/spack/repos/builtin/packages/ninja/package.py index 40890c212e..96cd0252a2 100644 --- a/var/spack/repos/builtin/packages/ninja/package.py +++ b/var/spack/repos/builtin/packages/ninja/package.py @@ -2,7 +2,6 @@ # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) -import re class Ninja(Package): @@ -40,7 +39,7 @@ class Ninja(Package): @run_after('configure') @on_package_attributes(run_tests=True) - def test(self): + def configure_test(self): ninja = Executable('./ninja') ninja('-j{0}'.format(make_jobs), 'ninja_test') ninja_test = Executable('./ninja_test') diff --git a/var/spack/repos/builtin/packages/node-js/package.py b/var/spack/repos/builtin/packages/node-js/package.py index b35e1d0def..28def78333 100644 --- a/var/spack/repos/builtin/packages/node-js/package.py +++ b/var/spack/repos/builtin/packages/node-js/package.py @@ -127,7 +127,7 @@ class NodeJs(Package): @run_after('build') @on_package_attributes(run_tests=True) - def test(self): + def build_test(self): make('test') make('test-addons') diff --git a/var/spack/repos/builtin/packages/openmpi/package.py b/var/spack/repos/builtin/packages/openmpi/package.py index b56c10fb67..124973bbc1 100644 --- a/var/spack/repos/builtin/packages/openmpi/package.py +++ b/var/spack/repos/builtin/packages/openmpi/package.py @@ -336,6 +336,8 @@ class Openmpi(AutotoolsPackage): filter_compiler_wrappers('openmpi/*-wrapper-data*', relative_root='share') + extra_install_tests = 'examples' + @classmethod def determine_version(cls, exe): output = Executable(exe)(output=str, error=str) @@ -846,6 +848,149 @@ class Openmpi(AutotoolsPackage): else: copy(script_stub, exe) + @run_after('install') + def setup_install_tests(self): + """ + Copy the example files after the package is installed to an + install test subdirectory for use during `spack test run`. + """ + self.cache_extra_test_sources(self.extra_install_tests) + + def _test_bin_ops(self): + info = ([], ['Ident string: {0}'.format(self.spec.version), 'MCA'], + 0) + + ls = (['-n', '1', 'ls', '..'], + ['openmpi-{0}'.format(self.spec.version)], 0) + + checks = { + 'mpirun': ls, + 'ompi_info': info, + 'oshmem_info': info, + 'oshrun': ls, + 'shmemrun': ls, + } + + for exe in checks: + options, expected, status = checks[exe] + reason = 'test: checking {0} output'.format(exe) + self.run_test(exe, options, expected, status, installed=True, + purpose=reason, skip_missing=True) + + def _test_check_versions(self): + comp_vers = str(self.spec.compiler.version) + spec_vers = str(self.spec.version) + checks = { + # Binaries available in at least versions 2.0.0 through 4.0.3 + 'mpiCC': comp_vers, + 'mpic++': comp_vers, + 'mpicc': comp_vers, + 'mpicxx': comp_vers, + 'mpiexec': spec_vers, + 'mpif77': comp_vers, + 'mpif90': comp_vers, + 'mpifort': comp_vers, + 'mpirun': spec_vers, + 'ompi_info': spec_vers, + 'ortecc': comp_vers, + 'orterun': spec_vers, + + # Binaries available in versions 2.0.0 through 2.1.6 + 'ompi-submit': spec_vers, + 'orte-submit': spec_vers, + + # Binaries available in versions 2.0.0 through 3.1.5 + 'ompi-dvm': spec_vers, + 'orte-dvm': spec_vers, + 'oshcc': comp_vers, + 'oshfort': comp_vers, + 'oshmem_info': spec_vers, + 'oshrun': spec_vers, + 'shmemcc': comp_vers, + 'shmemfort': comp_vers, + 'shmemrun': spec_vers, + + # Binary available in version 3.1.0 through 3.1.5 + 'prun': spec_vers, + + # Binaries available in versions 3.0.0 through 3.1.5 + 'oshCC': comp_vers, + 'oshc++': comp_vers, + 'oshcxx': comp_vers, + 'shmemCC': comp_vers, + 'shmemc++': comp_vers, + 'shmemcxx': comp_vers, + } + + for exe in checks: + expected = checks[exe] + purpose = 'test: ensuring version of {0} is {1}' \ + .format(exe, expected) + self.run_test(exe, '--version', expected, installed=True, + purpose=purpose, skip_missing=True) + + def _test_examples(self): + # First build the examples + self.run_test('make', ['all'], [], + purpose='test: ensuring ability to build the examples', + work_dir=join_path(self.install_test_root, + self.extra_install_tests)) + + # Now run those with known results + have_spml = self.spec.satisfies('@2.0.0:2.1.6') + + hello_world = (['Hello, world', 'I am', '0 of', '1'], 0) + + max_red = (['0/1 dst = 0 1 2'], 0) + + missing_spml = (['No available spml components'], 1) + + no_out = ([''], 0) + + ring_out = (['1 processes in ring', '0 exiting'], 0) + + strided = (['not in valid range'], 255) + + checks = { + 'hello_c': hello_world, + 'hello_cxx': hello_world, + 'hello_mpifh': hello_world, + 'hello_oshmem': hello_world if have_spml else missing_spml, + 'hello_oshmemcxx': hello_world if have_spml else missing_spml, + 'hello_oshmemfh': hello_world if have_spml else missing_spml, + 'hello_usempi': hello_world, + 'hello_usempif08': hello_world, + 'oshmem_circular_shift': ring_out if have_spml else missing_spml, + 'oshmem_max_reduction': max_red if have_spml else missing_spml, + 'oshmem_shmalloc': no_out if have_spml else missing_spml, + 'oshmem_strided_puts': strided if have_spml else missing_spml, + 'oshmem_symmetric_data': no_out if have_spml else missing_spml, + 'ring_c': ring_out, + 'ring_cxx': ring_out, + 'ring_mpifh': ring_out, + 'ring_oshmem': ring_out if have_spml else missing_spml, + 'ring_oshmemfh': ring_out if have_spml else missing_spml, + 'ring_usempi': ring_out, + 'ring_usempif08': ring_out, + } + + for exe in checks: + expected = checks[exe] + reason = 'test: checking example {0} output'.format(exe) + self.run_test(exe, [], expected, 0, installed=True, + purpose=reason, skip_missing=True) + + def test(self): + """Perform smoke tests on the installed package.""" + # Simple version check tests on known packages + self._test_check_versions() + + # Test the operation of selected executables + self._test_bin_ops() + + # Test example programs pulled from the build + self._test_examples() + def get_spack_compiler_spec(path): spack_compilers = spack.compilers.find_compilers([path]) diff --git a/var/spack/repos/builtin/packages/patchelf/package.py b/var/spack/repos/builtin/packages/patchelf/package.py index d17bb3bea1..796fd533eb 100644 --- a/var/spack/repos/builtin/packages/patchelf/package.py +++ b/var/spack/repos/builtin/packages/patchelf/package.py @@ -2,8 +2,8 @@ # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) - from spack import * +import os class Patchelf(AutotoolsPackage): @@ -18,3 +18,24 @@ class Patchelf(AutotoolsPackage): version('0.10', sha256='b2deabce05c34ce98558c0efb965f209de592197b2c88e930298d740ead09019') version('0.9', sha256='f2aa40a6148cb3b0ca807a1bf836b081793e55ec9e5540a5356d800132be7e0a') version('0.8', sha256='14af06a2da688d577d64ff8dac065bb8903bbffbe01d30c62df7af9bf4ce72fe') + + def test(self): + # Check patchelf in prefix and reports the correct version + reason = 'test: ensuring patchelf version is {0}' \ + .format(self.spec.version) + self.run_test('patchelf', + options='--version', + expected=['patchelf %s' % self.spec.version], + installed=True, + purpose=reason) + + # Check the rpath is changed + currdir = os.getcwd() + hello_file = self.test_suite.current_test_data_dir.join('hello') + self.run_test('patchelf', ['--set-rpath', currdir, hello_file], + purpose='test: ensuring that patchelf can change rpath') + + self.run_test('patchelf', + options=['--print-rpath', hello_file], + expected=[currdir], + purpose='test: ensuring that patchelf changed rpath') diff --git a/var/spack/repos/builtin/packages/patchelf/test/hello b/var/spack/repos/builtin/packages/patchelf/test/hello Binary files differnew file mode 100755 index 0000000000..8767836f8e --- /dev/null +++ b/var/spack/repos/builtin/packages/patchelf/test/hello diff --git a/var/spack/repos/builtin/packages/perl/package.py b/var/spack/repos/builtin/packages/perl/package.py index 744b55495a..2532f97160 100644 --- a/var/spack/repos/builtin/packages/perl/package.py +++ b/var/spack/repos/builtin/packages/perl/package.py @@ -204,7 +204,7 @@ class Perl(Package): # Perl doesn't use Autotools, it should subclass Package @run_after('build') @on_package_attributes(run_tests=True) - def test(self): + def build_test(self): make('test') def install(self, spec, prefix): @@ -364,3 +364,16 @@ class Perl(Package): # Perl doesn't use Autotools, it should subclass Package else: msg = 'Unable to locate {0} command in {1}' raise RuntimeError(msg.format(self.spec.name, self.prefix.bin)) + + def test(self): + """Smoke tests""" + exe = self.spec['perl'].command.name + + reason = 'test: checking version is {0}'.format(self.spec.version) + self.run_test(exe, '--version', ['perl', str(self.spec.version)], + installed=True, purpose=reason) + + reason = 'test: ensuring perl runs' + msg = 'Hello, World!' + options = ['-e', 'use warnings; use strict;\nprint("%s\n");' % msg] + self.run_test(exe, options, msg, installed=True, purpose=reason) diff --git a/var/spack/repos/builtin/packages/py-cloudpickle/package.py b/var/spack/repos/builtin/packages/py-cloudpickle/package.py index ff1e459e09..0f7e525290 100644 --- a/var/spack/repos/builtin/packages/py-cloudpickle/package.py +++ b/var/spack/repos/builtin/packages/py-cloudpickle/package.py @@ -19,6 +19,6 @@ class PyCloudpickle(PythonPackage): depends_on('py-setuptools', type='build') - def test(self): + def build_test(self): # PyPI tarball does not come with unit tests pass diff --git a/var/spack/repos/builtin/packages/py-cython/package.py b/var/spack/repos/builtin/packages/py-cython/package.py index aa30f56237..2dd532be2f 100644 --- a/var/spack/repos/builtin/packages/py-cython/package.py +++ b/var/spack/repos/builtin/packages/py-cython/package.py @@ -47,6 +47,6 @@ class PyCython(PythonPackage): """Returns the Cython command""" return Executable(self.prefix.bin.cython) - def test(self): + def build_test(self): # Warning: full suite of unit tests takes a very long time python('runtests.py', '-j', str(make_jobs)) diff --git a/var/spack/repos/builtin/packages/py-fiona/package.py b/var/spack/repos/builtin/packages/py-fiona/package.py index b6ad22a9a7..b4213d5f61 100644 --- a/var/spack/repos/builtin/packages/py-fiona/package.py +++ b/var/spack/repos/builtin/packages/py-fiona/package.py @@ -34,6 +34,6 @@ class PyFiona(PythonPackage): depends_on('py-ordereddict', type=('build', 'run'), when='^python@:2.6') depends_on('py-enum34', type=('build', 'run'), when='^python@:3.3') - def test(self): + def build_test(self): # PyPI tarball does not come with unit tests pass diff --git a/var/spack/repos/builtin/packages/py-matplotlib/package.py b/var/spack/repos/builtin/packages/py-matplotlib/package.py index c2ce697ec2..ebb95033cf 100644 --- a/var/spack/repos/builtin/packages/py-matplotlib/package.py +++ b/var/spack/repos/builtin/packages/py-matplotlib/package.py @@ -183,6 +183,6 @@ class PyMatplotlib(PythonPackage): setup.write('system_freetype = True\n') setup.write('system_qhull = True\n') - def test(self): + def build_test(self): pytest = which('pytest') pytest() diff --git a/var/spack/repos/builtin/packages/py-numpy/package.py b/var/spack/repos/builtin/packages/py-numpy/package.py index 53b1f759a6..fd09019469 100644 --- a/var/spack/repos/builtin/packages/py-numpy/package.py +++ b/var/spack/repos/builtin/packages/py-numpy/package.py @@ -306,7 +306,7 @@ class PyNumpy(PythonPackage): return args - def test(self): + def build_test(self): # `setup.py test` is not supported. Use one of the following # instead: # diff --git a/var/spack/repos/builtin/packages/py-py/package.py b/var/spack/repos/builtin/packages/py-py/package.py index 2cb000a846..995612d20a 100644 --- a/var/spack/repos/builtin/packages/py-py/package.py +++ b/var/spack/repos/builtin/packages/py-py/package.py @@ -28,6 +28,6 @@ class PyPy(PythonPackage): depends_on('py-setuptools', type='build') depends_on('py-setuptools-scm', type='build') - def test(self): + def build_test(self): # Tests require pytest, creating a circular dependency pass diff --git a/var/spack/repos/builtin/packages/py-py2cairo/package.py b/var/spack/repos/builtin/packages/py-py2cairo/package.py index 1427c7e45c..5c492a9167 100644 --- a/var/spack/repos/builtin/packages/py-py2cairo/package.py +++ b/var/spack/repos/builtin/packages/py-py2cairo/package.py @@ -23,7 +23,7 @@ class PyPy2cairo(WafPackage): depends_on('py-pytest', type='test') - def installtest(self): + def install_test(self): with working_dir('test'): pytest = which('py.test') pytest() diff --git a/var/spack/repos/builtin/packages/py-pybind11/package.py b/var/spack/repos/builtin/packages/py-pybind11/package.py index 3fe7402a0f..1c07734e6f 100644 --- a/var/spack/repos/builtin/packages/py-pybind11/package.py +++ b/var/spack/repos/builtin/packages/py-pybind11/package.py @@ -74,7 +74,7 @@ class PyPybind11(CMakePackage): @run_after('install') @on_package_attributes(run_tests=True) - def test(self): + def install_test(self): with working_dir('spack-test', create=True): # test include helper points to right location python = self.spec['python'].command diff --git a/var/spack/repos/builtin/packages/py-pygments/package.py b/var/spack/repos/builtin/packages/py-pygments/package.py index 87476c64d6..d7559b36d8 100644 --- a/var/spack/repos/builtin/packages/py-pygments/package.py +++ b/var/spack/repos/builtin/packages/py-pygments/package.py @@ -29,6 +29,6 @@ class PyPygments(PythonPackage): depends_on('python@3.5:', type=('build', 'run'), when='@2.6:') depends_on('py-setuptools', type=('build', 'run')) - def test(self): + def build_test(self): # Unit tests require sphinx, but that creates a circular dependency pass diff --git a/var/spack/repos/builtin/packages/py-python-dateutil/package.py b/var/spack/repos/builtin/packages/py-python-dateutil/package.py index 13e04b4e6e..16bff6858d 100644 --- a/var/spack/repos/builtin/packages/py-python-dateutil/package.py +++ b/var/spack/repos/builtin/packages/py-python-dateutil/package.py @@ -31,7 +31,7 @@ class PyPythonDateutil(PythonPackage): # depends_on('py-hypothesis', type='test') # depends_on('py-freezegun', type='test') - def test(self): + def build_test(self): # Tests require freezegun, which depends on python-dateutil, # creating circular dependency # pytest = which('pytest') diff --git a/var/spack/repos/builtin/packages/py-scipy/package.py b/var/spack/repos/builtin/packages/py-scipy/package.py index 533d404fa6..afeae0dd1c 100644 --- a/var/spack/repos/builtin/packages/py-scipy/package.py +++ b/var/spack/repos/builtin/packages/py-scipy/package.py @@ -99,7 +99,7 @@ class PyScipy(PythonPackage): return args - def test(self): + def build_test(self): # `setup.py test` is not supported. Use one of the following # instead: # diff --git a/var/spack/repos/builtin/packages/py-setuptools/package.py b/var/spack/repos/builtin/packages/py-setuptools/package.py index 27786cd27e..1a2930498b 100644 --- a/var/spack/repos/builtin/packages/py-setuptools/package.py +++ b/var/spack/repos/builtin/packages/py-setuptools/package.py @@ -71,6 +71,6 @@ class PySetuptools(PythonPackage): return url - def test(self): + def build_test(self): # Unit tests require pytest, creating a circular dependency pass diff --git a/var/spack/repos/builtin/packages/py-shapely/package.py b/var/spack/repos/builtin/packages/py-shapely/package.py index 6dc62888ab..c03106a6be 100644 --- a/var/spack/repos/builtin/packages/py-shapely/package.py +++ b/var/spack/repos/builtin/packages/py-shapely/package.py @@ -64,5 +64,5 @@ class PyShapely(PythonPackage): else: env.prepend_path('LD_LIBRARY_PATH', libs) - def test(self): + def test_install(self): python('-m', 'pytest') diff --git a/var/spack/repos/builtin/packages/py-sphinxcontrib-applehelp/package.py b/var/spack/repos/builtin/packages/py-sphinxcontrib-applehelp/package.py index 25f1a7ce5f..67ba38134c 100644 --- a/var/spack/repos/builtin/packages/py-sphinxcontrib-applehelp/package.py +++ b/var/spack/repos/builtin/packages/py-sphinxcontrib-applehelp/package.py @@ -18,6 +18,6 @@ class PySphinxcontribApplehelp(PythonPackage): depends_on('python@3.5:', type=('build', 'run')) depends_on('py-setuptools', type='build') - def test(self): + def build_test(self): # Requires sphinx, creating a circular dependency pass diff --git a/var/spack/repos/builtin/packages/py-sphinxcontrib-devhelp/package.py b/var/spack/repos/builtin/packages/py-sphinxcontrib-devhelp/package.py index ff90a9a5d4..1954fc9677 100644 --- a/var/spack/repos/builtin/packages/py-sphinxcontrib-devhelp/package.py +++ b/var/spack/repos/builtin/packages/py-sphinxcontrib-devhelp/package.py @@ -18,6 +18,6 @@ class PySphinxcontribDevhelp(PythonPackage): depends_on('python@3.5:', type=('build', 'run')) depends_on('py-setuptools', type='build') - def test(self): + def build_test(self): # Requires sphinx, creating a circular dependency pass diff --git a/var/spack/repos/builtin/packages/py-sphinxcontrib-htmlhelp/package.py b/var/spack/repos/builtin/packages/py-sphinxcontrib-htmlhelp/package.py index 96a51d3113..95f6819d59 100644 --- a/var/spack/repos/builtin/packages/py-sphinxcontrib-htmlhelp/package.py +++ b/var/spack/repos/builtin/packages/py-sphinxcontrib-htmlhelp/package.py @@ -18,6 +18,6 @@ class PySphinxcontribHtmlhelp(PythonPackage): depends_on('python@3.5:', type=('build', 'run')) depends_on('py-setuptools', type='build') - def test(self): + def build_test(self): # Requires sphinx, creating a circular dependency pass diff --git a/var/spack/repos/builtin/packages/py-sphinxcontrib-jsmath/package.py b/var/spack/repos/builtin/packages/py-sphinxcontrib-jsmath/package.py index 45ee46bc67..add0160ac8 100644 --- a/var/spack/repos/builtin/packages/py-sphinxcontrib-jsmath/package.py +++ b/var/spack/repos/builtin/packages/py-sphinxcontrib-jsmath/package.py @@ -17,6 +17,6 @@ class PySphinxcontribJsmath(PythonPackage): depends_on('python@3.5:', type=('build', 'run')) depends_on('py-setuptools', type='build') - def test(self): + def build_test(self): # Requires sphinx, creating a circular dependency pass diff --git a/var/spack/repos/builtin/packages/py-sphinxcontrib-qthelp/package.py b/var/spack/repos/builtin/packages/py-sphinxcontrib-qthelp/package.py index 86a58d456f..19fd328f37 100644 --- a/var/spack/repos/builtin/packages/py-sphinxcontrib-qthelp/package.py +++ b/var/spack/repos/builtin/packages/py-sphinxcontrib-qthelp/package.py @@ -18,6 +18,6 @@ class PySphinxcontribQthelp(PythonPackage): depends_on('python@3.5:', type=('build', 'run')) depends_on('py-setuptools', type='build') - def test(self): + def build_test(self): # Requires sphinx, creating a circular dependency pass diff --git a/var/spack/repos/builtin/packages/py-sphinxcontrib-serializinghtml/package.py b/var/spack/repos/builtin/packages/py-sphinxcontrib-serializinghtml/package.py index 97b79a8012..3bad6d661a 100644 --- a/var/spack/repos/builtin/packages/py-sphinxcontrib-serializinghtml/package.py +++ b/var/spack/repos/builtin/packages/py-sphinxcontrib-serializinghtml/package.py @@ -18,6 +18,6 @@ class PySphinxcontribSerializinghtml(PythonPackage): depends_on('python@3.5:', type=('build', 'run')) depends_on('py-setuptools', type='build') - def test(self): + def build_test(self): # Requires sphinx, creating a circular dependency pass diff --git a/var/spack/repos/builtin/packages/py-sphinxcontrib-websupport/package.py b/var/spack/repos/builtin/packages/py-sphinxcontrib-websupport/package.py index c5a8f80a5b..1e4d1051a3 100644 --- a/var/spack/repos/builtin/packages/py-sphinxcontrib-websupport/package.py +++ b/var/spack/repos/builtin/packages/py-sphinxcontrib-websupport/package.py @@ -26,6 +26,6 @@ class PySphinxcontribWebsupport(PythonPackage): depends_on('python@2.7:2.8,3.4:', type=('build', 'run')) depends_on('py-setuptools', type='build') - def test(self): + def build_test(self): # Unit tests require sphinx, creating a circular dependency pass diff --git a/var/spack/repos/builtin/packages/py-statsmodels/package.py b/var/spack/repos/builtin/packages/py-statsmodels/package.py index 26c006bccf..b00e51730b 100644 --- a/var/spack/repos/builtin/packages/py-statsmodels/package.py +++ b/var/spack/repos/builtin/packages/py-statsmodels/package.py @@ -42,7 +42,7 @@ class PyStatsmodels(PythonPackage): depends_on('py-pytest', type='test') - def test(self): + def build_test(self): dirs = glob.glob("build/lib*") # There can be only one... with working_dir(dirs[0]): pytest = which('pytest') diff --git a/var/spack/repos/builtin/packages/python/package.py b/var/spack/repos/builtin/packages/python/package.py index 53924a85b0..4d0df211e6 100644 --- a/var/spack/repos/builtin/packages/python/package.py +++ b/var/spack/repos/builtin/packages/python/package.py @@ -1127,3 +1127,21 @@ class Python(AutotoolsPackage): view.remove_file(src, dst) else: os.remove(dst) + + def test(self): + # do not use self.command because we are also testing the run env + exe = self.spec['python'].command.name + + # test hello world + msg = 'hello world!' + reason = 'test: running {0}'.format(msg) + options = ['-c', 'print("{0}")'.format(msg)] + self.run_test(exe, options=options, expected=[msg], installed=True, + purpose=reason) + + # checks import works and executable comes from the spec prefix + reason = 'test: checking import and executable' + print_str = self.print_string('sys.executable') + options = ['-c', 'import sys; {0}'.format(print_str)] + self.run_test(exe, options=options, expected=[self.spec.prefix], + installed=True, purpose=reason) diff --git a/var/spack/repos/builtin/packages/raja/package.py b/var/spack/repos/builtin/packages/raja/package.py index 7da9c6c4fd..8d1db659cc 100644 --- a/var/spack/repos/builtin/packages/raja/package.py +++ b/var/spack/repos/builtin/packages/raja/package.py @@ -3,8 +3,6 @@ # # SPDX-License-Identifier: (Apache-2.0 OR MIT) -from spack import * - class Raja(CMakePackage, CudaPackage): """RAJA Parallel Framework.""" @@ -74,3 +72,52 @@ class Raja(CMakePackage, CudaPackage): options.append('-DENABLE_TESTS=ON') return options + + @property + def build_relpath(self): + """Relative path to the cmake build subdirectory.""" + return join_path('..', self.build_dirname) + + @run_after('install') + def setup_build_tests(self): + """Copy the build test files after the package is installed to a + relative install test subdirectory for use during `spack test run`.""" + # Now copy the relative files + self.cache_extra_test_sources(self.build_relpath) + + # Ensure the path exists since relying on a relative path at the + # same level as the normal stage source path. + mkdirp(self.install_test_root) + + @property + def _extra_tests_path(self): + # TODO: The tests should be converted to re-build and run examples + # TODO: using the installed libraries. + return join_path(self.install_test_root, self.build_relpath, 'bin') + + def _test_examples(self): + """Perform very basic checks on a subset of copied examples.""" + checks = [ + ('ex5_line-of-sight_solution', + [r'RAJA sequential', r'RAJA OpenMP', r'result -- PASS']), + ('ex6_stencil-offset-layout_solution', + [r'RAJA Views \(permuted\)', r'result -- PASS']), + ('ex8_tiled-matrix-transpose_solution', + [r'parallel top inner loop', + r'collapsed inner loops', r'result -- PASS']), + ('kernel-dynamic-tile', [r'Running index', r'(24,24)']), + ('plugin-example', + [r'Launching host kernel for the 10 time']), + ('tut_batched-matrix-multiply', [r'result -- PASS']), + ('wave-eqn', [r'Max Error = 2', r'Evolved solution to time']) + ] + for exe, expected in checks: + reason = 'test: checking output of {0} for {1}' \ + .format(exe, expected) + self.run_test(exe, [], expected, installed=False, + purpose=reason, skip_missing=True, + work_dir=self._extra_tests_path) + + def test(self): + """Perform smoke tests.""" + self._test_examples() diff --git a/var/spack/repos/builtin/packages/serf/package.py b/var/spack/repos/builtin/packages/serf/package.py index 4c762bf9a5..96df6f7579 100644 --- a/var/spack/repos/builtin/packages/serf/package.py +++ b/var/spack/repos/builtin/packages/serf/package.py @@ -63,7 +63,7 @@ class Serf(SConsPackage): return args - def test(self): + def build_test(self): # FIXME: Several test failures: # # There were 14 failures: diff --git a/var/spack/repos/builtin/packages/sqlite/package.py b/var/spack/repos/builtin/packages/sqlite/package.py index b400965b3f..db456cb0d7 100644 --- a/var/spack/repos/builtin/packages/sqlite/package.py +++ b/var/spack/repos/builtin/packages/sqlite/package.py @@ -3,7 +3,6 @@ # # SPDX-License-Identifier: (Apache-2.0 OR MIT) -from spack import * from spack import architecture @@ -129,3 +128,41 @@ class Sqlite(AutotoolsPackage): cc(self.compiler.cc_pic_flag, '-lm', '-shared', 'extension-functions.c', '-o', libraryname) install(libraryname, self.prefix.lib) + + def _test_example(self): + """Ensure a sequence of commands on example db are successful.""" + + test_data_dir = self.test_suite.current_test_data_dir + db_filename = test_data_dir.join('packages.db') + exe = 'sqlite3' + + # Ensure the database only contains one table + expected = 'packages' + reason = 'test: ensuring only table is "{0}"'.format(expected) + self.run_test(exe, [db_filename, '.tables'], expected, installed=True, + purpose=reason, skip_missing=False) + + # Ensure the database dump matches expectations, where special + # characters are replaced with spaces in the expected and actual + # output to avoid pattern errors. + reason = 'test: checking dump output' + expected = get_escaped_text_output(test_data_dir.join('dump.out')) + self.run_test(exe, [db_filename, '.dump'], expected, installed=True, + purpose=reason, skip_missing=False) + + def _test_version(self): + """Perform version check on the installed package.""" + exe = 'sqlite3' + vers_str = str(self.spec.version) + + reason = 'test: ensuring version of {0} is {1}'.format(exe, vers_str) + self.run_test(exe, '-version', vers_str, installed=True, + purpose=reason, skip_missing=False) + + def test(self): + """Perform smoke tests on the installed package.""" + # Perform a simple version check + self._test_version() + + # Run a sequence of operations + self._test_example() diff --git a/var/spack/repos/builtin/packages/sqlite/test/dump.out b/var/spack/repos/builtin/packages/sqlite/test/dump.out new file mode 100644 index 0000000000..3dda19d1c5 --- /dev/null +++ b/var/spack/repos/builtin/packages/sqlite/test/dump.out @@ -0,0 +1,10 @@ +PRAGMA foreign_keys=OFF; +BEGIN TRANSACTION; +CREATE TABLE packages ( +name varchar(80) primary key, +has_code integer, +url varchar(160)); +INSERT INTO packages VALUES('sqlite',1,'https://www.sqlite.org'); +INSERT INTO packages VALUES('readline',1,'https://tiswww.case.edu/php/chet/readline/rltop.html'); +INSERT INTO packages VALUES('xsdk',0,'http://xsdk.info'); +COMMIT; diff --git a/var/spack/repos/builtin/packages/sqlite/test/packages.db b/var/spack/repos/builtin/packages/sqlite/test/packages.db Binary files differnew file mode 100644 index 0000000000..252962235c --- /dev/null +++ b/var/spack/repos/builtin/packages/sqlite/test/packages.db diff --git a/var/spack/repos/builtin/packages/subversion/package.py b/var/spack/repos/builtin/packages/subversion/package.py index 1f5f65215c..98874ee2bd 100644 --- a/var/spack/repos/builtin/packages/subversion/package.py +++ b/var/spack/repos/builtin/packages/subversion/package.py @@ -98,7 +98,7 @@ class Subversion(AutotoolsPackage): perl = spec['perl'].command perl('Makefile.PL', 'INSTALL_BASE={0}'.format(prefix)) - def test(self): + def check(self): make('check') if '+perl' in self.spec: make('check-swig-pl') diff --git a/var/spack/repos/builtin/packages/umpire/package.py b/var/spack/repos/builtin/packages/umpire/package.py index 08b6d487ff..6b39aad5f6 100644 --- a/var/spack/repos/builtin/packages/umpire/package.py +++ b/var/spack/repos/builtin/packages/umpire/package.py @@ -3,8 +3,8 @@ # # SPDX-License-Identifier: (Apache-2.0 OR MIT) - -from spack import * +import llnl.util.lang as lang +import llnl.util.tty as tty class Umpire(CMakePackage, CudaPackage): @@ -115,3 +115,189 @@ class Umpire(CMakePackage, CudaPackage): 'Off' if 'tests=none' in spec else 'On')) return options + + @property + def build_relpath(self): + """Relative path to the cmake build subdirectory.""" + return join_path('..', self.build_dirname) + + @run_after('install') + def setup_build_tests(self): + """Copy the build test files after the package is installed to an + install test subdirectory for use during `spack test run`.""" + # Now copy the relative files + self.cache_extra_test_sources(self.build_relpath) + + # Ensure the path exists since relying on a relative path at the + # same level as the normal stage source path. + mkdirp(self.install_test_root) + + @property + @lang.memoized + def _extra_tests_path(self): + # TODO: The tests should be converted to re-build and run examples + # TODO: using the installed libraries. + return join_path(self.install_test_root, self.build_relpath) + + @property + @lang.memoized + def _has_bad_strategy(self): + return self.spec.satisfies('@0.2.0:0.2.3') + + def _run_checks(self, dirs, checks): + """Run the specified checks in the provided directories.""" + + if not dirs or not checks: + return + + for exe in checks: + if exe == 'strategy_example' and self._has_bad_strategy: + # Skip this test until install testing can properly capture + # the abort associated with this version. + # (An umpire::util::Exception is thrown; status value is -6.) + tty.warn('Skipping {0} test until Spack can handle core dump' + .format(exe)) + continue + + expected, status = checks[exe] + for work_dir in dirs: + src = 'from build ' if 'spack-build' in work_dir else '' + reason = 'test {0} {1}output'.format(exe, src) + self.run_test(exe, [], expected, status, installed=False, + purpose=reason, skip_missing=True, + work_dir=work_dir) + + def _run_bench_checks(self): + """Run the benchmark smoke test checks.""" + tty.info('Running benchmark checks') + + dirs = [] + if self.spec.satisfies('@0.3.3:1.0.1'): + dirs.append(join_path(self._extra_tests_path, 'benchmarks')) + elif self.spec.satisfies('@1.1.0:'): + dirs.append(self.prefix.bin) + + checks = { + # Versions 0.3.3:1.0.1 (spack-build/bin/benchmarks) + # Versions 1.1.0:2.1.0 (spack-build/bin) + 'allocator_benchmarks': ( + ['Malloc/malloc', 'Malloc/free', 'ns', + 'Host/allocate', 'Host/deallocate', + 'FixedPoolHost/allocate', + 'FixedPoolHost/deallocate'], 0), + 'copy_benchmarks': (['benchmark_copy/host_host', 'ns'], 0), + 'debuglog_benchmarks': (['benchmark_DebugLogger', 'ns'], 0), + } + self._run_checks(dirs, checks) + + def _run_cookbook_checks(self): + """Run the cookbook smoke test checks.""" + tty.info('Running cookbook checks') + + dirs = [] + cb_subdir = join_path('examples', 'cookbook') + if self.spec.satisfies('@0.3.3:1.0.1'): + dirs.append(join_path(self._extra_tests_path, cb_subdir)) + elif self.spec.satisfies('@1.1.0'): + dirs.append(join_path(self.prefix.bin, cb_subdir)) + elif self.spec.satisfies('@2.0.0:'): + dirs.append(self.prefix.bin) + + checks = { + # Versions 0.3.3:1.0.1 (spack-build/bin/examples/cookbook) + # Versions 2.0.0:2.1.0 (spack-build/bin) + # Versions 1.1.0 (prefix.bin/examples/cookbook) + # Versions 2.0.0:2.1.0 (prefix.bin) + 'recipe_dynamic_pool_heuristic': (['in the pool', 'releas'], 0), + 'recipe_no_introspection': (['has allocated', 'used'], 0), + } + self._run_checks(dirs, checks) + + def _run_example_checks(self): + """Run the example smoke test checks.""" + tty.info('Running example checks') + + dirs = [] + if self.spec.satisfies('@0.1.3:0.3.1'): + dirs.append(self._extra_tests_path) + elif self.spec.satisfies('@0.3.3:1.0.1'): + dirs.append(join_path(self._extra_tests_path, 'examples')) + elif self.spec.satisfies('@1.1.0'): + dirs.append(join_path(self.prefix.bin, 'examples')) + elif self.spec.satisfies('@2.0.0:'): + dirs.append(self.prefix.bin) + + # Check the results from a subset of the (potentially) available + # executables + checks = { + # Versions 0.1.3:0.3.1 (spack-build/bin) + # Versions 0.3.3:1.0.1 (spack-build/bin/examples) + # Versions 2.0.0:2.1.0 (spack-build/bin) + # Version 1.1.0 (prefix.bin/examples) + # Versions 2.0.0:2.1.0 (prefix.bin) + 'malloc': (['99 should be 99'], 0), + 'strategy_example': (['Available allocators', 'HOST'], 0), + 'vector_allocator': ([''], 0), + } + self._run_checks(dirs, checks) + + def _run_plots_checks(self): + """Run the plots smoke test checks.""" + tty.info('Running plots checks') + + dirs = [self.prefix.bin] if self.spec.satisfies('@0.3.3:0.3.5') else [] + checks = { + # Versions 0.3.3:0.3.5 (prefix.bin) + 'plot_allocations': ([''], 0), + } + self._run_checks(dirs, checks) + + def _run_tools_checks(self): + """Run the tools smoke test checks.""" + tty.info('Running tools checks') + + dirs = [self.prefix.bin] if self.spec.satisfies('@0.3.3:0.3.5') else [] + checks = { + # Versions 0.3.3:0.3.5 (spack-build/bin/tools) + 'replay': (['No input file'], 0), + } + self._run_checks(dirs, checks) + + def _run_tut_checks(self): + """Run the tutorial smoke test checks.""" + tty.info('Running tutorials checks') + + dirs = [] + tut_subdir = join_path('examples', 'tutorial') + if self.spec.satisfies('@0.2.4:0.3.1'): + dirs.append(self._extra_tests_path) + elif self.spec.satisfies('@0.3.3:1.0.1'): + dirs.append(join_path(self._extra_tests_path, tut_subdir)) + elif self.spec.satisfies('@1.1.0'): + dirs.append(join_path(self.prefix.bin, tut_subdir)) + elif self.spec.satisfies('@2.0.0:'): + dirs.append(self.prefix.bin) + + checks = { + # Versions 0.2.4:0.3.1 (spack-build/bin) + # Versions 0.3.3:1.0.1 (spack-build/bin/examples/tutorial) + # Versions 2.0.0:2.1.0 (spack-build/bin) + # Version 1.1.0 (prefix.bin/examples/tutorial) + # Versions 2.0.0:2.1.0 (prefix.bin) + 'tut_copy': (['Copied source data'], 0), + 'tut_introspection': ( + ['Allocator used is HOST', 'size of the allocation'], 0), + 'tut_memset': (['Set data from HOST'], 0), + 'tut_move': (['Moved source data', 'HOST'], 0), + 'tut_reallocate': (['Reallocated data'], 0), + } + self._run_checks(dirs, checks) + + def test(self): + """Perform smoke tests on the installed package.""" + self._run_bench_checks() + self._run_cookbook_checks() + self._run_example_checks() + self._run_plots_checks() + self._run_tools_checks() + self._run_tut_checks() |