diff options
| author | Greg Becker <becker33@llnl.gov> | 2020-11-18 02:39:02 -0800 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2020-11-18 02:39:02 -0800 |
| commit | 77b2e578ec47f7713cae965fede1ab6e60aa69c4 (patch) | |
| tree | a03377956a08356bc3b73dc23b4d466c4f2f5aa6 | |
| parent | b81bbfb6e9691036fc284f034e749a3b9361bf5a (diff) | |
| download | spack-77b2e578ec47f7713cae965fede1ab6e60aa69c4.tar.gz spack-77b2e578ec47f7713cae965fede1ab6e60aa69c4.tar.bz2 spack-77b2e578ec47f7713cae965fede1ab6e60aa69c4.tar.xz spack-77b2e578ec47f7713cae965fede1ab6e60aa69c4.zip | |
spack test (#15702)
Users can add test() methods to their packages to run smoke tests on
installations with the new `spack test` command (the old `spack test` is
now `spack unit-test`). spack test is environment-aware, so you can
`spack install` an environment and then run `spack test run` to run smoke
tests on all of its packages. Historical test logs can be perused with
`spack test results`. Generic smoke tests for MPI implementations, C,
C++, and Fortran compilers as well as specific smoke tests for 18
packages.
Inside the test method, individual tests can be run separately (and
continue to run best-effort after a test failure) using the `run_test`
method. The `run_test` method encapsulates finding test executables,
running and checking return codes, checking output, and error handling.
This handles the following trickier aspects of testing with direct
support in Spack's package API:
- [x] Caching source or intermediate build files at build time for
use at test time.
- [x] Test dependencies,
- [x] packages that require a compiler for testing (such as library only
packages).
See the packaging guide for more details on using Spack testing support.
Included is support for package.py files for virtual packages. This does
not change the Spack interface, but is a major change in internals.
Co-authored-by: Tamara Dahlgren <dahlgren1@llnl.gov>
Co-authored-by: wspear <wjspear@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
131 files changed, 3599 insertions, 676 deletions
diff --git a/.github/workflows/linux_unit_tests.yaml b/.github/workflows/linux_unit_tests.yaml index c66d58c284..c87ea6e07a 100644 --- a/.github/workflows/linux_unit_tests.yaml +++ b/.github/workflows/linux_unit_tests.yaml @@ -132,7 +132,7 @@ jobs: . share/spack/setup-env.sh spack compiler find spack solve mpileaks%gcc - coverage run $(which spack) test -v + coverage run $(which spack) unit-test -v coverage combine coverage xml - uses: codecov/codecov-action@v1 diff --git a/.github/workflows/macos_unit_tests.yaml b/.github/workflows/macos_unit_tests.yaml index 2de92394f8..29caaa2e08 100644 --- a/.github/workflows/macos_unit_tests.yaml +++ b/.github/workflows/macos_unit_tests.yaml @@ -35,7 +35,7 @@ jobs: git --version . .github/workflows/setup_git.sh . share/spack/setup-env.sh - coverage run $(which spack) test + coverage run $(which spack) unit-test coverage combine coverage xml - uses: codecov/codecov-action@v1 diff --git a/etc/spack/defaults/config.yaml b/etc/spack/defaults/config.yaml index 15ce68c68f..d1a7f35a6d 100644 --- a/etc/spack/defaults/config.yaml +++ b/etc/spack/defaults/config.yaml @@ -70,6 +70,10 @@ config: - ~/.spack/stage # - $spack/var/spack/stage + # Directory in which to run tests and store test results. + # Tests will be stored in directories named by date/time and package + # name/hash. + test_stage: ~/.spack/test # Cache directory for already downloaded source tarballs and archived # repositories. This can be purged with `spack clean --downloads`. diff --git a/lib/spack/docs/contribution_guide.rst b/lib/spack/docs/contribution_guide.rst index 37cf9091bd..8df8ad65ba 100644 --- a/lib/spack/docs/contribution_guide.rst +++ b/lib/spack/docs/contribution_guide.rst @@ -74,7 +74,7 @@ locally to speed up the review process. We currently test against Python 2.6, 2.7, and 3.5-3.7 on both macOS and Linux and perform 3 types of tests: -.. _cmd-spack-test: +.. _cmd-spack-unit-test: ^^^^^^^^^^ Unit Tests @@ -96,7 +96,7 @@ To run *all* of the unit tests, use: .. code-block:: console - $ spack test + $ spack unit-test These tests may take several minutes to complete. If you know you are only modifying a single Spack feature, you can run subsets of tests at a @@ -105,13 +105,13 @@ time. For example, this would run all the tests in .. code-block:: console - $ spack test lib/spack/spack/test/architecture.py + $ spack unit-test lib/spack/spack/test/architecture.py And this would run the ``test_platform`` test from that file: .. code-block:: console - $ spack test lib/spack/spack/test/architecture.py::test_platform + $ spack unit-test lib/spack/spack/test/architecture.py::test_platform This allows you to develop iteratively: make a change, test that change, make another change, test that change, etc. We use `pytest @@ -121,29 +121,29 @@ pytest docs <http://doc.pytest.org/en/latest/usage.html#specifying-tests-selecting-tests>`_ for more details on test selection syntax. -``spack test`` has a few special options that can help you understand -what tests are available. To get a list of all available unit test -files, run: +``spack unit-test`` has a few special options that can help you +understand what tests are available. To get a list of all available +unit test files, run: -.. command-output:: spack test --list +.. command-output:: spack unit-test --list :ellipsis: 5 -To see a more detailed list of available unit tests, use ``spack test ---list-long``: +To see a more detailed list of available unit tests, use ``spack +unit-test --list-long``: -.. command-output:: spack test --list-long +.. command-output:: spack unit-test --list-long :ellipsis: 10 And to see the fully qualified names of all tests, use ``--list-names``: -.. command-output:: spack test --list-names +.. command-output:: spack unit-test --list-names :ellipsis: 5 You can combine these with ``pytest`` arguments to restrict which tests you want to know about. For example, to see just the tests in ``architecture.py``: -.. command-output:: spack test --list-long lib/spack/spack/test/architecture.py +.. command-output:: spack unit-test --list-long lib/spack/spack/test/architecture.py You can also combine any of these options with a ``pytest`` keyword search. See the `pytest usage docs @@ -151,7 +151,7 @@ search. See the `pytest usage docs for more details on test selection syntax. For example, to see the names of all tests that have "spec" or "concretize" somewhere in their names: -.. command-output:: spack test --list-names -k "spec and concretize" +.. command-output:: spack unit-test --list-names -k "spec and concretize" By default, ``pytest`` captures the output of all unit tests, and it will print any captured output for failed tests. Sometimes it's helpful to see @@ -161,7 +161,7 @@ argument to ``pytest``: .. code-block:: console - $ spack test -s --list-long lib/spack/spack/test/architecture.py::test_platform + $ spack unit-test -s --list-long lib/spack/spack/test/architecture.py::test_platform Unit tests are crucial to making sure bugs aren't introduced into Spack. If you are modifying core Spack libraries or adding new @@ -176,7 +176,7 @@ how to write tests! You may notice the ``share/spack/qa/run-unit-tests`` script in the repository. This script is designed for CI. It runs the unit tests and reports coverage statistics back to Codecov. If you want to - run the unit tests yourself, we suggest you use ``spack test``. + run the unit tests yourself, we suggest you use ``spack unit-test``. ^^^^^^^^^^^^ Flake8 Tests diff --git a/lib/spack/docs/developer_guide.rst b/lib/spack/docs/developer_guide.rst index 7fd4d1ec6e..b4b8b5d321 100644 --- a/lib/spack/docs/developer_guide.rst +++ b/lib/spack/docs/developer_guide.rst @@ -363,11 +363,12 @@ Developer commands ``spack doc`` ^^^^^^^^^^^^^ -^^^^^^^^^^^^^^ -``spack test`` -^^^^^^^^^^^^^^ +^^^^^^^^^^^^^^^^^^^ +``spack unit-test`` +^^^^^^^^^^^^^^^^^^^ -See the :ref:`contributor guide section <cmd-spack-test>` on ``spack test``. +See the :ref:`contributor guide section <cmd-spack-unit-test>` on +``spack unit-test``. .. _cmd-spack-python: diff --git a/lib/spack/docs/extensions.rst b/lib/spack/docs/extensions.rst index c71a6511ed..15c59c76ef 100644 --- a/lib/spack/docs/extensions.rst +++ b/lib/spack/docs/extensions.rst @@ -87,11 +87,12 @@ will be available from the command line: --implicit select specs that are not installed or were installed implicitly --output OUTPUT where to dump the result -The corresponding unit tests can be run giving the appropriate options to ``spack test``: +The corresponding unit tests can be run giving the appropriate options +to ``spack unit-test``: .. code-block:: console - $ spack test --extension=scripting + $ spack unit-test --extension=scripting ============================================================== test session starts =============================================================== platform linux2 -- Python 2.7.15rc1, pytest-3.2.5, py-1.4.34, pluggy-0.4.0 diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst index 836fc12b83..ec83679a47 100644 --- a/lib/spack/docs/packaging_guide.rst +++ b/lib/spack/docs/packaging_guide.rst @@ -3948,6 +3948,118 @@ using the ``run_before`` decorator. .. _file-manipulation: +^^^^^^^^^^^^^ +Install Tests +^^^^^^^^^^^^^ + +.. warning:: + + The API for adding and running install tests is not yet considered + stable and may change drastically in future releases. Packages with + upstreamed tests will be refactored to match changes to the API. + +While build-tests are integrated with the build system, install tests +may be added to Spack packages to be run independently of the install +method. + +Install tests may be added by defining a ``test`` method with the following signature: + +.. code-block:: python + + def test(self): + +These tests will be run in an environment set up to provide access to +this package and all of its dependencies, including ``test``-type +dependencies. Inside the ``test`` method, standard python ``assert`` +statements and other error reporting mechanisms can be used. Spack +will report any errors as a test failure. + +Inside the test method, individual tests can be run separately (and +continue transparently after a test failure) using the ``run_test`` +method. The signature for the ``run_test`` method is: + +.. code-block:: python + + def run_test(self, exe, options=[], expected=[], status=0, installed=False, + purpose='', skip_missing=False, work_dir=None): + +This method will operate in ``work_dir`` if one is specified. It will +search for an executable in the ``PATH`` variable named ``exe``, and +if ``installed=True`` it will fail if that executable does not come +from the prefix of the package being tested. If the executable is not +found, it will fail the test unless ``skip_missing`` is set to +``True``. The executable will be run with the options specified, and +the return code will be checked against the ``status`` argument, which +can be an integer or list of integers. Spack will also check that +every string in ``expected`` is a regex matching part of the output of +the executable. The ``purpose`` argument is recorded in the test log +for debugging purposes. + +"""""""""""""""""""""""""""""""""""""" +Install tests that require compilation +"""""""""""""""""""""""""""""""""""""" + +Some tests may require access to the compiler with which the package +was built, especially to test library-only packages. To ensure the +compiler is configured as part of the test environment, set the +attribute ``tests_require_compiler = True`` on the package. The +compiler will be available through the canonical environment variables +(``CC``, ``CXX``, ``FC``, ``F77``) in the test environment. + +"""""""""""""""""""""""""""""""""""""""""""""""" +Install tests that require build-time components +"""""""""""""""""""""""""""""""""""""""""""""""" + +Some packages cannot be easily tested without components from the +build-time test suite. For those packages, the +``cache_extra_test_sources`` method can be used. + +.. code-block:: python + + @run_after('install') + def cache_test_sources(self): + srcs = ['./tests/foo.c', './tests/bar.c'] + self.cache_extra_test_sources(srcs) + +This method will copy the listed methods into the metadata directory +of the package at the end of the install phase of the build. They will +be available to the test method in the directory +``self._extra_tests_path``. + +While source files are generally recommended, for many packages +binaries may also technically be cached in this way for later testing. + +""""""""""""""""""""" +Running install tests +""""""""""""""""""""" + +Install tests can be run using the ``spack test run`` command. The +``spack test run`` command will create a ``test suite`` out of the +specs provided to it, or if no specs are provided it will test all +specs in the active environment, or all specs installed in Spack if no +environment is active. Test suites can be named using the ``--alias`` +option; test suites not aliased will use the content hash of their +specs as their name. + +Packages to install test can be queried using the ``spack test list`` +command, which outputs all installed packages with defined ``test`` +methods. + +Test suites can be found using the ``spack test find`` command. It +will list all test suites that have been run and have not been removed +using the ``spack test remove`` command. The ``spack test remove`` +command will remove tests to declutter the test stage. The ``spack +test results`` command will show results for completed test suites. + +The test stage is the working directory for all install tests run with +Spack. By default, Spack uses ``~/.spack/test`` as the test stage. The +test stage can be set in the high-level config: + +.. code-block:: yaml + + config: + test_stage: /path/to/stage + --------------------------- File manipulation functions --------------------------- diff --git a/lib/spack/external/ctest_log_parser.py b/lib/spack/external/ctest_log_parser.py index 0437b6e524..072c10d7a9 100644 --- a/lib/spack/external/ctest_log_parser.py +++ b/lib/spack/external/ctest_log_parser.py @@ -118,6 +118,7 @@ _error_matches = [ "([^:]+): (Error:|error|undefined reference|multiply defined)", "([^ :]+) ?: (error|fatal error|catastrophic error)", "([^:]+)\\(([^\\)]+)\\) ?: (error|fatal error|catastrophic error)"), + "^FAILED", "^[Bb]us [Ee]rror", "^[Ss]egmentation [Vv]iolation", "^[Ss]egmentation [Ff]ault", diff --git a/lib/spack/llnl/util/filesystem.py b/lib/spack/llnl/util/filesystem.py index b8d0b4d2f1..d6579555ad 100644 --- a/lib/spack/llnl/util/filesystem.py +++ b/lib/spack/llnl/util/filesystem.py @@ -41,6 +41,8 @@ __all__ = [ 'fix_darwin_install_name', 'force_remove', 'force_symlink', + 'chgrp', + 'chmod_x', 'copy', 'install', 'copy_tree', @@ -52,6 +54,7 @@ __all__ = [ 'partition_path', 'prefixes', 'remove_dead_links', + 'remove_directory_contents', 'remove_if_dead_link', 'remove_linked_tree', 'set_executable', @@ -1806,3 +1809,13 @@ def md5sum(file): with open(file, "rb") as f: md5.update(f.read()) return md5.digest() + + +def remove_directory_contents(dir): + """Remove all contents of a directory.""" + if os.path.exists(dir): + for entry in [os.path.join(dir, entry) for entry in os.listdir(dir)]: + if os.path.isfile(entry) or os.path.islink(entry): + os.unlink(entry) + else: + shutil.rmtree(entry) diff --git a/lib/spack/spack/build_environment.py b/lib/spack/spack/build_environment.py index a1501034d4..cb1ba21ba5 100644 --- a/lib/spack/spack/build_environment.py +++ b/lib/spack/spack/build_environment.py @@ -32,8 +32,8 @@ There are two parts to the build environment: Skimming this module is a nice way to get acquainted with the types of calls you can make from within the install() function. """ -import re import inspect +import re import multiprocessing import os import shutil @@ -53,10 +53,14 @@ import spack.build_systems.meson import spack.config import spack.main import spack.paths +import spack.package +import spack.repo import spack.schema.environment import spack.store +import spack.install_test import spack.subprocess_context import spack.architecture as arch +import spack.util.path from spack.util.string import plural from spack.util.environment import ( env_flag, filter_system_paths, get_path, is_system_path, @@ -453,7 +457,6 @@ def _set_variables_for_single_module(pkg, module): jobs = spack.config.get('config:build_jobs', 16) if pkg.parallel else 1 jobs = min(jobs, multiprocessing.cpu_count()) - assert jobs is not None, "no default set for config:build_jobs" m = module m.make_jobs = jobs @@ -713,28 +716,42 @@ def load_external_modules(pkg): load_module(external_module) -def setup_package(pkg, dirty): +def setup_package(pkg, dirty, context='build'): """Execute all environment setup routines.""" - build_env = EnvironmentModifications() + env = EnvironmentModifications() if not dirty: clean_environment() - set_compiler_environment_variables(pkg, build_env) - set_build_environment_variables(pkg, build_env, dirty) - pkg.architecture.platform.setup_platform_environment(pkg, build_env) + # setup compilers and build tools for build contexts + need_compiler = context == 'build' or (context == 'test' and + pkg.test_requires_compiler) + if need_compiler: + set_compiler_environment_variables(pkg, env) + set_build_environment_variables(pkg, env, dirty) - build_env.extend( - modifications_from_dependencies(pkg.spec, context='build') - ) + # architecture specific setup + pkg.architecture.platform.setup_platform_environment(pkg, env) - if (not dirty) and (not build_env.is_unset('CPATH')): - tty.debug("A dependency has updated CPATH, this may lead pkg-config" - " to assume that the package is part of the system" - " includes and omit it when invoked with '--cflags'.") + if context == 'build': + # recursive post-order dependency information + env.extend( + modifications_from_dependencies(pkg.spec, context=context) + ) + + if (not dirty) and (not env.is_unset('CPATH')): + tty.debug("A dependency has updated CPATH, this may lead pkg-" + "config to assume that the package is part of the system" + " includes and omit it when invoked with '--cflags'.") - set_module_variables_for_package(pkg) - pkg.setup_build_environment(build_env) + # setup package itself + set_module_variables_for_package(pkg) + pkg.setup_build_environment(env) + elif context == 'test': + import spack.user_environment as uenv # avoid circular import + env.extend(uenv.environment_modifications_for_spec(pkg.spec)) + set_module_variables_for_package(pkg) + env.prepend_path('PATH', '.') # Loading modules, in particular if they are meant to be used outside # of Spack, can change environment variables that are relevant to the @@ -744,15 +761,16 @@ def setup_package(pkg, dirty): # unnecessary. Modules affecting these variables will be overwritten anyway with preserve_environment('CC', 'CXX', 'FC', 'F77'): # All module loads that otherwise would belong in previous - # functions have to occur after the build_env object has its + # functions have to occur after the env object has its # modifications applied. Otherwise the environment modifications # could undo module changes, such as unsetting LD_LIBRARY_PATH # after a module changes it. - for mod in pkg.compiler.modules: - # Fixes issue https://github.com/spack/spack/issues/3153 - if os.environ.get("CRAY_CPU_TARGET") == "mic-knl": - load_module("cce") - load_module(mod) + if need_compiler: + for mod in pkg.compiler.modules: + # Fixes issue https://github.com/spack/spack/issues/3153 + if os.environ.get("CRAY_CPU_TARGET") == "mic-knl": + load_module("cce") + load_module(mod) # kludge to handle cray libsci being automatically loaded by PrgEnv # modules on cray platform. Module unload does no damage when @@ -766,12 +784,12 @@ def setup_package(pkg, dirty): implicit_rpaths = pkg.compiler.implicit_rpaths() if implicit_rpaths: - build_env.set('SPACK_COMPILER_IMPLICIT_RPATHS', - ':'.join(implicit_rpaths)) + env.set('SPACK_COMPILER_IMPLICIT_RPATHS', + ':'.join(implicit_rpaths)) # Make sure nothing's strange about the Spack environment. - validate(build_env, tty.warn) - build_env.apply_modifications() + validate(env, tty.warn) + env.apply_modifications() def modifications_from_dependencies(spec, context): @@ -791,7 +809,8 @@ def modifications_from_dependencies(spec, context): deptype_and_method = { 'build': (('build', 'link', 'test'), 'setup_dependent_build_environment'), - 'run': (('link', 'run'), 'setup_dependent_run_environment') + 'run': (('link', 'run'), 'setup_dependent_run_environment'), + 'test': (('link', 'run', 'test'), 'setup_dependent_run_environment') } deptype, method = deptype_and_method[context] @@ -808,6 +827,8 @@ def modifications_from_dependencies(spec, context): def _setup_pkg_and_run(serialized_pkg, function, kwargs, child_pipe, input_multiprocess_fd): + context = kwargs.get('context', 'build') + try: # We are in the child process. Python sets sys.stdin to # open(os.devnull) to prevent our process and its parent from @@ -821,7 +842,8 @@ def _setup_pkg_and_run(serialized_pkg, function, kwargs, child_pipe, if not kwargs.get('fake', False): kwargs['unmodified_env'] = os.environ.copy() - setup_package(pkg, dirty=kwargs.get('dirty', False)) + setup_package(pkg, dirty=kwargs.get('dirty', False), + context=context) return_value = function(pkg, kwargs) child_pipe.send(return_value) @@ -841,13 +863,18 @@ def _setup_pkg_and_run(serialized_pkg, function, kwargs, child_pipe, # show that, too. package_context = get_package_context(tb) - build_log = None - try: - if hasattr(pkg, 'log_path'): - build_log = pkg.log_path - except NameError: - # 'pkg' is not defined yet - pass + logfile = None + if context == 'build': + try: + if hasattr(pkg, 'log_path'): + logfile = pkg.log_path + except NameError: + # 'pkg' is not defined yet + pass + elif context == 'test': + logfile = os.path.join( + pkg.test_suite.stage, + spack.install_test.TestSuite.test_log_name(pkg.spec)) # make a pickleable exception to send to parent. msg = "%s: %s" % (exc_type.__name__, str(exc)) @@ -855,7 +882,7 @@ def _setup_pkg_and_run(serialized_pkg, function, kwargs, child_pipe, ce = ChildError(msg, exc_type.__module__, exc_type.__name__, - tb_string, build_log, package_context) + tb_string, logfile, context, package_context) child_pipe.send(ce) finally: @@ -873,9 +900,6 @@ def start_build_process(pkg, function, kwargs): child process for. function (callable): argless function to run in the child process. - dirty (bool): If True, do NOT clean the environment before - building. - fake (bool): If True, skip package setup b/c it's not a real build Usage:: @@ -961,6 +985,7 @@ def get_package_context(traceback, context=3): Args: traceback (traceback): A traceback from some exception raised during install + context (int): Lines of context to show before and after the line where the error happened @@ -1067,13 +1092,14 @@ class ChildError(InstallError): # context instead of Python context. build_errors = [('spack.util.executable', 'ProcessError')] - def __init__(self, msg, module, classname, traceback_string, build_log, - context): + def __init__(self, msg, module, classname, traceback_string, log_name, + log_type, context): super(ChildError, self).__init__(msg) self.module = module self.name = classname self.traceback = traceback_string - self.build_log = build_log + self.log_name = log_name + self.log_type = log_type self.context = context @property @@ -1081,26 +1107,16 @@ class ChildError(InstallError): out = StringIO() out.write(self._long_message if self._long_message else '') + have_log = self.log_name and os.path.exists(self.log_name) + if (self.module, self.name) in ChildError.build_errors: # The error happened in some external executed process. Show - # the build log with errors or warnings highlighted. - if self.build_log and os.path.exists(self.build_log): - errors, warnings = parse_log_events(self.build_log) - nerr = len(errors) - nwar = len(warnings) - if nerr > 0: - # If errors are found, only display errors - out.write( - "\n%s found in build log:\n" % plural(nerr, 'error')) - out.write(make_log_context(errors)) - elif nwar > 0: - # If no errors are found but warnings are, display warnings - out.write( - "\n%s found in build log:\n" % plural(nwar, 'warning')) - out.write(make_log_context(warnings)) + # the log with errors or warnings highlighted. + if have_log: + write_log_summary(out, self.log_type, self.log_name) else: - # The error happened in in the Python code, so try to show + # The error happened in the Python code, so try to show # some context from the Package itself. if self.context: out.write('\n') @@ -1110,14 +1126,14 @@ class ChildError(InstallError): if out.getvalue(): out.write('\n') - if self.build_log and os.path.exists(self.build_log): - out.write('See build log for details:\n') - out.write(' %s\n' % self.build_log) + if have_log: + out.write('See {0} log for details:\n'.format(self.log_type)) + out.write(' {0}\n'.format(self.log_name)) return out.getvalue() def __str__(self): - return self.message + self.long_message + self.traceback + return self.message def __reduce__(self): """__reduce__ is used to serialize (pickle) ChildErrors. @@ -1130,13 +1146,14 @@ class ChildError(InstallError): self.module, self.name, self.traceback, - self.build_log, + self.log_name, + self.log_type, self.context) -def _make_child_error(msg, module, name, traceback, build_log, context): +def _make_child_error(msg, module, name, traceback, log, log_type, context): """Used by __reduce__ in ChildError to reconstruct pickled errors.""" - return ChildError(msg, module, name, traceback, build_log, context) + return ChildError(msg, module, name, traceback, log, log_type, context) class StopPhase(spack.error.SpackError): @@ -1147,3 +1164,30 @@ class StopPhase(spack.error.SpackError): def _make_stop_phase(msg, long_msg): return StopPhase(msg, long_msg) + + +def write_log_summary(out, log_type, log, last=None): + errors, warnings = parse_log_events(log) + nerr = len(errors) + nwar = len(warnings) + + if nerr > 0: + if last and nerr > last: + errors = errors[-last:] + nerr = last + + # If errors are found, only display errors + out.write( + "\n%s found in %s log:\n" % + (plural(nerr, 'error'), log_type)) + out.write(make_log_context(errors)) + elif nwar > 0: + if last and nwar > last: + warnings = warnings[-last:] + nwar = last + + # If no errors are found but warnings are, display warnings + out.write( + "\n%s found in %s log:\n" % + (plural(nwar, 'warning'), log_type)) + out.write(make_log_context(warnings)) diff --git a/lib/spack/spack/build_systems/cmake.py b/lib/spack/spack/build_systems/cmake.py index 4b679b358a..1336069846 100644 --- a/lib/spack/spack/build_systems/cmake.py +++ b/lib/spack/spack/build_systems/cmake.py @@ -325,13 +325,20 @@ class CMakePackage(PackageBase): libs_flags)) @property + def build_dirname(self): + """Returns the directory name to use when building the package + + :return: name of the subdirectory for building the package + """ + return 'spack-build-%s' % self.spec.dag_hash(7) + + @property def build_directory(self): """Returns the directory to use when building the package :return: directory where to build the package """ - dirname = 'spack-build-%s' % self.spec.dag_hash(7) - return os.path.join(self.stage.path, dirname) + return os.path.join(self.stage.path, self.build_dirname) def cmake_args(self): """Produces a list containing all the arguments that must be passed to diff --git a/lib/spack/spack/build_systems/intel.py b/lib/spack/spack/build_systems/intel.py index a74d5e9613..0e0bb9378b 100644 --- a/lib/spack/spack/build_systems/intel.py +++ b/lib/spack/spack/build_systems/intel.py @@ -1017,6 +1017,15 @@ class IntelPackage(PackageBase): env.extend(EnvironmentModifications.from_sourcing_file(f, *args)) + if self.spec.name in ('intel', 'intel-parallel-studio'): + # this package provides compilers + # TODO: fix check above when compilers are dependencies + env.set('CC', self.prefix.bin.icc) + env.set('CXX', self.prefix.bin.icpc) + env.set('FC', self.prefix.bin.ifort) + env.set('F77', self.prefix.bin.ifort) + env.set('F90', self.prefix.bin.ifort) + def setup_dependent_build_environment(self, env, dependent_spec): # NB: This function is overwritten by 'mpi' provider packages: # diff --git a/lib/spack/spack/build_systems/python.py b/lib/spack/spack/build_systems/python.py index 99dc4ce0dc..76159d88a1 100644 --- a/lib/spack/spack/build_systems/python.py +++ b/lib/spack/spack/build_systems/python.py @@ -89,7 +89,7 @@ class PythonPackage(PackageBase): build_system_class = 'PythonPackage' #: Callback names for build-time test - build_time_test_callbacks = ['test'] + build_time_test_callbacks = ['build_test'] #: Callback names for install-time test install_time_test_callbacks = ['import_module_test'] @@ -359,7 +359,7 @@ class PythonPackage(PackageBase): # Testing - def test(self): + def build_test(self): """Run unit tests after in-place build. These tests are only run if the package actually has a 'test' command. diff --git a/lib/spack/spack/build_systems/scons.py b/lib/spack/spack/build_systems/scons.py index ffa4390249..5e17666b71 100644 --- a/lib/spack/spack/build_systems/scons.py +++ b/lib/spack/spack/build_systems/scons.py @@ -33,7 +33,7 @@ class SConsPackage(PackageBase): build_system_class = 'SConsPackage' #: Callback names for build-time test - build_time_test_callbacks = ['test'] + build_time_test_callbacks = ['build_test'] depends_on('scons', type='build') @@ -59,7 +59,7 @@ class SConsPackage(PackageBase): # Testing - def test(self): + def build_test(self): """Run unit tests after build. By default, does nothing. Override this if you want to diff --git a/lib/spack/spack/build_systems/waf.py b/lib/spack/spack/build_systems/waf.py index a1581660f2..a6dbbbdb35 100644 --- a/lib/spack/spack/build_systems/waf.py +++ b/lib/spack/spack/build_systems/waf.py @@ -47,10 +47,10 @@ class WafPackage(PackageBase): build_system_class = 'WafPackage' # Callback names for build-time test - build_time_test_callbacks = ['test'] + build_time_test_callbacks = ['build_test'] # Callback names for install-time test - install_time_test_callbacks = ['installtest'] + install_time_test_callbacks = ['install_test'] # Much like AutotoolsPackage does not require automake and autoconf # to build, WafPackage does not require waf to build. It only requires @@ -106,7 +106,7 @@ class WafPackage(PackageBase): # Testing - def test(self): + def build_test(self): """Run unit tests after build. By default, does nothing. Override this if you want to @@ -116,7 +116,7 @@ class WafPackage(PackageBase): run_after('build')(PackageBase._run_default_build_time_test_callbacks) - def installtest(self): + def install_test(self): """Run unit tests after install. By default, does nothing. Override this if you want to diff --git a/lib/spack/spack/cmd/build_env.py b/lib/spack/spack/cmd/build_env.py index 128d167a29..ef8b1f6e6b 100644 --- a/lib/spack/spack/cmd/build_env.py +++ b/lib/spack/spack/cmd/build_env.py @@ -2,86 +2,15 @@ # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) - -from __future__ import print_function - -import argparse -import os - -import llnl.util.tty as tty -import spack.build_environment as build_environment -import spack.cmd -import spack.cmd.common.arguments as arguments -from spack.util.environment import dump_environment, pickle_environment +import spack.cmd.common.env_utility as env_utility description = "run a command in a spec's install environment, " \ "or dump its environment to screen or file" section = "build" level = "long" - -def setup_parser(subparser): - arguments.add_common_arguments(subparser, ['clean', 'dirty']) - subparser.add_argument( - '--dump', metavar="FILE", - help="dump a source-able environment to FILE" - ) - subparser.add_argument( - '--pickle', metavar="FILE", - help="dump a pickled source-able environment to FILE" - ) - subparser.add_argument( - 'spec', nargs=argparse.REMAINDER, - metavar='spec [--] [cmd]...', - help="spec of package environment to emulate") - subparser.epilog\ - = 'If a command is not specified, the environment will be printed ' \ - 'to standard output (cf /usr/bin/env) unless --dump and/or --pickle ' \ - 'are specified.\n\nIf a command is specified and spec is ' \ - 'multi-word, then the -- separator is obligatory.' +setup_parser = env_utility.setup_parser def build_env(parser, args): - if not args.spec: - tty.die("spack build-env requires a spec.") - - # Specs may have spaces in them, so if they do, require that the - # caller put a '--' between the spec and the command to be - # executed. If there is no '--', assume that the spec is the - # first argument. - sep = '--' - if sep in args.spec: - s = args.spec.index(sep) - spec = args.spec[:s] - cmd = args.spec[s + 1:] - else: - spec = args.spec[0] - cmd = args.spec[1:] - - specs = spack.cmd.parse_specs(spec, concretize=True) - if len(specs) > 1: - tty.die("spack build-env only takes one spec.") - spec = specs[0] - - build_environment.setup_package(spec.package, args.dirty) - - if args.dump: - # Dump a source-able environment to a text file. - tty.msg("Dumping a source-able environment to {0}".format(args.dump)) - dump_environment(args.dump) - - if args.pickle: - # Dump a source-able environment to a pickle file. - tty.msg( - "Pickling a source-able environment to {0}".format(args.pickle)) - pickle_environment(args.pickle) - - if cmd: - # Execute the command with the new environment - os.execvp(cmd[0], cmd) - - elif not bool(args.pickle or args.dump): - # If no command or dump/pickle option act like the "env" command - # and print out env vars. - for key, val in os.environ.items(): - print("%s=%s" % (key, val)) + env_utility.emulate_env_utility('build-env', 'build', args) diff --git a/lib/spack/spack/cmd/clean.py b/lib/spack/spack/cmd/clean.py index d847e7a7c0..f69b959293 100644 --- a/lib/spack/spack/cmd/clean.py +++ b/lib/spack/spack/cmd/clean.py @@ -10,10 +10,11 @@ import shutil import llnl.util.tty as tty import spack.caches -import spack.cmd +import spack.cmd.test import spack.cmd.common.arguments as arguments import spack.repo import spack.stage +import spack.config from spack.paths import lib_path, var_path diff --git a/lib/spack/spack/cmd/common/arguments.py b/lib/spack/spack/cmd/common/arguments.py index e5945bda9c..e5c4c0dde8 100644 --- a/lib/spack/spack/cmd/common/arguments.py +++ b/lib/spack/spack/cmd/common/arguments.py @@ -275,3 +275,53 @@ def no_checksum(): return Args( '-n', '--no-checksum', action='store_true', default=False, help="do not use checksums to verify downloaded files (unsafe)") + + +def add_cdash_args(subparser, add_help): + cdash_help = {} + if add_help: + cdash_help['upload-url'] = "CDash URL where reports will be uploaded" + cdash_help['build'] = """The name of the build that will be reported to CDash. +Defaults to spec of the package to operate on.""" + cdash_help['site'] = """The site name that will be reported to CDash. +Defaults to current system hostname.""" + cdash_help['track'] = """Results will be reported to this group on CDash. +Defaults to Experimental.""" + cdash_help['buildstamp'] = """Instead of letting the CDash reporter prepare the +buildstamp which, when combined with build name, site and project, +uniquely identifies the build, provide this argument to identify +the build yourself. Format: %%Y%%m%%d-%%H%%M-[cdash-track]""" + else: + cdash_help['upload-url'] = argparse.SUPPRESS + cdash_help['build'] = argparse.SUPPRESS + cdash_help['site'] = argparse.SUPPRESS + cdash_help['track'] = argparse.SUPPRESS + cdash_help['buildstamp'] = argparse.SUPPRESS + + subparser.add_argument( + '--cdash-upload-url', + default=None, + help=cdash_help['upload-url'] + ) + subparser.add_argument( + '--cdash-build', + default=None, + help=cdash_help['build'] + ) + subparser.add_argument( + '--cdash-site', + default=None, + help=cdash_help['site'] + ) + + cdash_subgroup = subparser.add_mutually_exclusive_group() + cdash_subgroup.add_argument( + '--cdash-track', + default='Experimental', + help=cdash_help['track'] + ) + cdash_subgroup.add_argument( + '--cdash-buildstamp', + default=None, + help=cdash_help['buildstamp'] + ) diff --git a/lib/spack/spack/cmd/common/env_utility.py b/lib/spack/spack/cmd/common/env_utility.py new file mode 100644 index 0000000000..e3f32737b4 --- /dev/null +++ b/lib/spack/spack/cmd/common/env_utility.py @@ -0,0 +1,82 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) +from __future__ import print_function + +import argparse +import os + +import llnl.util.tty as tty +import spack.build_environment as build_environment +import spack.paths +import spack.cmd +import spack.cmd.common.arguments as arguments +from spack.util.environment import dump_environment, pickle_environment + + +def setup_parser(subparser): + arguments.add_common_arguments(subparser, ['clean', 'dirty']) + subparser.add_argument( + '--dump', metavar="FILE", + help="dump a source-able environment to FILE" + ) + subparser.add_argument( + '--pickle', metavar="FILE", + help="dump a pickled source-able environment to FILE" + ) + subparser.add_argument( + 'spec', nargs=argparse.REMAINDER, + metavar='spec [--] [cmd]...', + help="specs of package environment to emulate") + subparser.epilog\ + = 'If a command is not specified, the environment will be printed ' \ + 'to standard output (cf /usr/bin/env) unless --dump and/or --pickle ' \ + 'are specified.\n\nIf a command is specified and spec is ' \ + 'multi-word, then the -- separator is obligatory.' + + +def emulate_env_utility(cmd_name, context, args): + if not args.spec: + tty.die("spack %s requires a spec." % cmd_name) + + # Specs may have spaces in them, so if they do, require that the + # caller put a '--' between the spec and the command to be + # executed. If there is no '--', assume that the spec is the + # first argument. + sep = '--' + if sep in args.spec: + s = args.spec.index(sep) + spec = args.spec[:s] + cmd = args.spec[s + 1:] + else: + spec = args.spec[0] + cmd = args.spec[1:] + + specs = spack.cmd.parse_specs(spec, concretize=True) + if len(specs) > 1: + tty.die("spack %s only takes one spec." % cmd_name) + spec = specs[0] + + build_environment.setup_package(spec.package, args.dirty, context) + + if args.dump: + # Dump a source-able environment to a text file. + tty.msg("Dumping a source-able environment to {0}".format(args.dump)) + dump_environment(args.dump) + + if args.pickle: + # Dump a source-able environment to a pickle file. + tty.msg( + "Pickling a source-able environment to {0}".format(args.pickle)) + pickle_environment(args.pickle) + + if cmd: + # Execute the command with the new environment + os.execvp(cmd[0], cmd) + + elif not bool(args.pickle or args.dump): + # If no command or dump/pickle option then act like the "env" command + # and print out env vars. + for key, val in os.environ.items(): + print("%s=%s" % (key, val)) diff --git a/lib/spack/spack/cmd/install.py b/lib/spack/spack/cmd/install.py index c8673b5330..3b5954b1ad 100644 --- a/lib/spack/spack/cmd/install.py +++ b/lib/spack/spack/cmd/install.py @@ -167,65 +167,8 @@ packages. If neither are chosen, don't run tests for any packages.""" action='store_true', help="Show usage instructions for CDash reporting" ) - subparser.add_argument( - '-y', '--yes-to-all', - action='store_true', - dest='yes_to_all', - help="""assume "yes" is the answer to every confirmation request. -To run completely non-interactively, also specify '--no-checksum'.""" - ) - add_cdash_args(subparser, False) - arguments.add_common_arguments(subparser, ['spec']) - - -def add_cdash_args(subparser, add_help): - cdash_help = {} - if add_help: - cdash_help['upload-url'] = "CDash URL where reports will be uploaded" - cdash_help['build'] = """The name of the build that will be reported to CDash. -Defaults to spec of the package to install.""" - cdash_help['site'] = """The site name that will be reported to CDash. -Defaults to current system hostname.""" - cdash_help['track'] = """Results will be reported to this group on CDash. -Defaults to Experimental.""" - cdash_help['buildstamp'] = """Instead of letting the CDash reporter prepare the -buildstamp which, when combined with build name, site and project, -uniquely identifies the build, provide this argument to identify -the build yourself. Format: %%Y%%m%%d-%%H%%M-[cdash-track]""" - else: - cdash_help['upload-url'] = argparse.SUPPRESS - cdash_help['build'] = argparse.SUPPRESS - cdash_help['site'] = argparse.SUPPRESS - cdash_help['track'] = argparse.SUPPRESS - cdash_help['buildstamp'] = argparse.SUPPRESS - - subparser.add_argument( - '--cdash-upload-url', - default=None, - help=cdash_help['upload-url'] - ) - subparser.add_argument( - '--cdash-build', - default=None, - help=cdash_help['build'] - ) - subparser.add_argument( - '--cdash-site', - default=None, - help=cdash_help['site'] - ) - - cdash_subgroup = subparser.add_mutually_exclusive_group() - cdash_subgroup.add_argument( - '--cdash-track', - default='Experimental', - help=cdash_help['track'] - ) - cdash_subgroup.add_argument( - '--cdash-buildstamp', - default=None, - help=cdash_help['buildstamp'] - ) + arguments.add_cdash_args(subparser, False) + arguments.add_common_arguments(subparser, ['yes_to_all', 'spec']) def default_log_file(spec): @@ -283,11 +226,12 @@ environment variables: SPACK_CDASH_AUTH_TOKEN authentication token to present to CDash ''')) - add_cdash_args(parser, True) + arguments.add_cdash_args(parser, True) parser.print_help() return - reporter = spack.report.collect_info(args.log_format, args) + reporter = spack.report.collect_info( + spack.package.PackageInstaller, '_install_task', args.log_format, args) if args.log_file: reporter.filename = args.log_file @@ -383,7 +327,7 @@ environment variables: if not args.log_file and not reporter.filename: reporter.filename = default_log_file(specs[0]) reporter.specs = specs - with reporter: + with reporter('build'): if args.overwrite: installed = list(filter(lambda x: x, diff --git a/lib/spack/spack/cmd/list.py b/lib/spack/spack/cmd/list.py index 656282599d..a7d4cb0ac3 100644 --- a/lib/spack/spack/cmd/list.py +++ b/lib/spack/spack/cmd/list.py @@ -54,6 +54,9 @@ def setup_parser(subparser): subparser.add_argument( '--update', metavar='FILE', default=None, action='store', help='write output to the specified file, if any package is newer') + subparser.add_argument( + '-v', '--virtuals', action='store_true', default=False, + help='include virtual packages in list') arguments.add_common_arguments(subparser, ['tags']) @@ -267,7 +270,7 @@ def list(parser, args): formatter = formatters[args.format] # Retrieve the names of all the packages - pkgs = set(spack.repo.all_package_names()) + pkgs = set(spack.repo.all_package_names(args.virtuals)) # Filter the set appropriately sorted_packages = filter_by_name(pkgs, args) diff --git a/lib/spack/spack/cmd/test.py b/lib/spack/spack/cmd/test.py index 8cbc0fbccf..3362b8a109 100644 --- a/lib/spack/spack/cmd/test.py +++ b/lib/spack/spack/cmd/test.py @@ -4,166 +4,381 @@ # SPDX-License-Identifier: (Apache-2.0 OR MIT) from __future__ import print_function -from __future__ import division - -import collections -import sys -import re +import os import argparse -import pytest -from six import StringIO +import textwrap +import fnmatch +import re +import shutil -import llnl.util.tty.color as color -from llnl.util.filesystem import working_dir -from llnl.util.tty.colify import colify +import llnl.util.tty as tty -import spack.paths +import spack.install_test +import spack.environment as ev +import spack.cmd +import spack.cmd.common.arguments as arguments +import spack.report +import spack.package -description = "run spack's unit tests (wrapper around pytest)" -section = "developer" +description = "run spack's tests for an install" +section = "administrator" level = "long" +def first_line(docstring): + """Return the first line of the docstring.""" + return docstring.split('\n')[0] + + def setup_parser(subparser): - subparser.add_argument( - '-H', '--pytest-help', action='store_true', default=False, - help="show full pytest help, with advanced options") - - # extra spack arguments to list tests - list_group = subparser.add_argument_group("listing tests") - list_mutex = list_group.add_mutually_exclusive_group() - list_mutex.add_argument( - '-l', '--list', action='store_const', default=None, - dest='list', const='list', help="list test filenames") - list_mutex.add_argument( - '-L', '--list-long', action='store_const', default=None, - dest='list', const='long', help="list all test functions") - list_mutex.add_argument( - '-N', '--list-names', action='store_const', default=None, - dest='list', const='names', help="list full names of all tests") - - # use tests for extension - subparser.add_argument( - '--extension', default=None, - help="run test for a given spack extension") - - # spell out some common pytest arguments, so they'll show up in help - pytest_group = subparser.add_argument_group( - "common pytest arguments (spack test --pytest-help for more details)") - pytest_group.add_argument( - "-s", action='append_const', dest='parsed_args', const='-s', - help="print output while tests run (disable capture)") - pytest_group.add_argument( - "-k", action='store', metavar="EXPRESSION", dest='expression', - help="filter tests by keyword (can also use w/list options)") - pytest_group.add_argument( - "--showlocals", action='append_const', dest='parsed_args', - const='--showlocals', help="show local variable values in tracebacks") - - # remainder is just passed to pytest - subparser.add_argument( - 'pytest_args', nargs=argparse.REMAINDER, help="arguments for pytest") - - -def do_list(args, extra_args): - """Print a lists of tests than what pytest offers.""" - # Run test collection and get the tree out. - old_output = sys.stdout - try: - sys.stdout = output = StringIO() - pytest.main(['--collect-only'] + extra_args) - finally: - sys.stdout = old_output - - lines = output.getvalue().split('\n') - tests = collections.defaultdict(lambda: set()) - prefix = [] - - # collect tests into sections - for line in lines: - match = re.match(r"(\s*)<([^ ]*) '([^']*)'", line) - if not match: - continue - indent, nodetype, name = match.groups() - - # strip parametrized tests - if "[" in name: - name = name[:name.index("[")] - - depth = len(indent) // 2 - - if nodetype.endswith("Function"): - key = tuple(prefix) - tests[key].add(name) - else: - prefix = prefix[:depth] - prefix.append(name) - - def colorize(c, prefix): - if isinstance(prefix, tuple): - return "::".join( - color.colorize("@%s{%s}" % (c, p)) - for p in prefix if p != "()" - ) - return color.colorize("@%s{%s}" % (c, prefix)) - - if args.list == "list": - files = set(prefix[0] for prefix in tests) - color_files = [colorize("B", file) for file in sorted(files)] - colify(color_files) - - elif args.list == "long": - for prefix, functions in sorted(tests.items()): - path = colorize("*B", prefix) + "::" - functions = [colorize("c", f) for f in sorted(functions)] - color.cprint(path) - colify(functions, indent=4) - print() - - else: # args.list == "names" - all_functions = [ - colorize("*B", prefix) + "::" + colorize("c", f) - for prefix, functions in sorted(tests.items()) - for f in sorted(functions) - ] - colify(all_functions) - - -def add_back_pytest_args(args, unknown_args): - """Add parsed pytest args, unknown args, and remainder together. - - We add some basic pytest arguments to the Spack parser to ensure that - they show up in the short help, so we have to reassemble things here. + sp = subparser.add_subparsers(metavar='SUBCOMMAND', dest='test_command') + + # Run + run_parser = sp.add_parser('run', description=test_run.__doc__, + help=first_line(test_run.__doc__)) + + alias_help_msg = "Provide an alias for this test-suite" + alias_help_msg += " for subsequent access." + run_parser.add_argument('--alias', help=alias_help_msg) + + run_parser.add_argument( + '--fail-fast', action='store_true', + help="Stop tests for each package after the first failure." + ) + run_parser.add_argument( + '--fail-first', action='store_true', + help="Stop after the first failed package." + ) + run_parser.add_argument( + '--keep-stage', + action='store_true', + help='Keep testing directory for debugging' + ) + run_parser.add_argument( + '--log-format', + default=None, + choices=spack.report.valid_formats, + help="format to be used for log files" + ) + run_parser.add_argument( + '--log-file', + default=None, + help="filename for the log file. if not passed a default will be used" + ) + arguments.add_cdash_args(run_parser, False) + run_parser.add_argument( + '--help-cdash', + action='store_true', + help="Show usage instructions for CDash reporting" + ) + + cd_group = run_parser.add_mutually_exclusive_group() + arguments.add_common_arguments(cd_group, ['clean', 'dirty']) + + arguments.add_common_arguments(run_parser, ['installed_specs']) + + # List + sp.add_parser('list', description=test_list.__doc__, + help=first_line(test_list.__doc__)) + + # Find + find_parser = sp.add_parser('find', description=test_find.__doc__, + help=first_line(test_find.__doc__)) + find_parser.add_argument( + 'filter', nargs=argparse.REMAINDER, + help='optional case-insensitive glob patterns to filter results.') + + # Status + status_parser = sp.add_parser('status', description=test_status.__doc__, + help=first_line(test_status.__doc__)) + status_parser.add_argument( + 'names', nargs=argparse.REMAINDER, + help="Test suites for which to print status") + + # Results + results_parser = sp.add_parser('results', description=test_results.__doc__, + help=first_line(test_results.__doc__)) + results_parser.add_argument( + '-l', '--logs', action='store_true', + help="print the test log for each matching package") + results_parser.add_argument( + '-f', '--failed', action='store_true', + help="only show results for failed tests of matching packages") + results_parser.add_argument( + 'names', nargs=argparse.REMAINDER, + metavar='[name(s)] [-- installed_specs]...', + help="suite names and installed package constraints") + results_parser.epilog = 'Test results will be filtered by space-' \ + 'separated suite name(s) and installed\nspecs when provided. '\ + 'If names are provided, then only results for those test\nsuites '\ + 'will be shown. If installed specs are provided, then ony results'\ + '\nmatching those specs will be shown.' + + # Remove + remove_parser = sp.add_parser('remove', description=test_remove.__doc__, + help=first_line(test_remove.__doc__)) + arguments.add_common_arguments(remove_parser, ['yes_to_all']) + remove_parser.add_argument( + 'names', nargs=argparse.REMAINDER, + help="Test suites to remove from test stage") + + +def test_run(args): + """Run tests for the specified installed packages. + + If no specs are listed, run tests for all packages in the current + environment or all installed packages if there is no active environment. """ - result = args.parsed_args or [] - result += unknown_args or [] - result += args.pytest_args or [] - if args.expression: - result += ["-k", args.expression] - return result - - -def test(parser, args, unknown_args): - if args.pytest_help: - # make the pytest.main help output more accurate - sys.argv[0] = 'spack test' - return pytest.main(['-h']) - - # add back any parsed pytest args we need to pass to pytest - pytest_args = add_back_pytest_args(args, unknown_args) - - # The default is to test the core of Spack. If the option `--extension` - # has been used, then test that extension. - pytest_root = spack.paths.spack_root - if args.extension: - target = args.extension - extensions = spack.config.get('config:extensions') - pytest_root = spack.extensions.path_for_extension(target, *extensions) - - # pytest.ini lives in the root of the spack repository. - with working_dir(pytest_root): - if args.list: - do_list(args, pytest_args) + # cdash help option + if args.help_cdash: + parser = argparse.ArgumentParser( + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=textwrap.dedent('''\ +environment variables: + SPACK_CDASH_AUTH_TOKEN + authentication token to present to CDash + ''')) + arguments.add_cdash_args(parser, True) + parser.print_help() + return + + # set config option for fail-fast + if args.fail_fast: + spack.config.set('config:fail_fast', True, scope='command_line') + + # Get specs to test + env = ev.get_env(args, 'test') + hashes = env.all_hashes() if env else None + + specs = spack.cmd.parse_specs(args.specs) if args.specs else [None] + specs_to_test = [] + for spec in specs: + matching = spack.store.db.query_local(spec, hashes=hashes) + if spec and not matching: + tty.warn("No installed packages match spec %s" % spec) + specs_to_test.extend(matching) + + # test_stage_dir + test_suite = spack.install_test.TestSuite(specs_to_test, args.alias) + test_suite.ensure_stage() + tty.msg("Spack test %s" % test_suite.name) + + # Set up reporter + setattr(args, 'package', [s.format() for s in test_suite.specs]) + reporter = spack.report.collect_info( + spack.package.PackageBase, 'do_test', args.log_format, args) + if not reporter.filename: + if args.log_file: + if os.path.isabs(args.log_file): + log_file = args.log_file + else: + log_dir = os.getcwd() + log_file = os.path.join(log_dir, args.log_file) + else: + log_file = os.path.join( + os.getcwd(), + 'test-%s' % test_suite.name) + reporter.filename = log_file + reporter.specs = specs_to_test + + with reporter('test', test_suite.stage): + test_suite(remove_directory=not args.keep_stage, + dirty=args.dirty, + fail_first=args.fail_first) + + +def has_test_method(pkg): + return pkg.test.__func__ != spack.package.PackageBase.test + + +def test_list(args): + """List all installed packages with available tests.""" + # TODO: This can be extended to have all of the output formatting options + # from `spack find`. + env = ev.get_env(args, 'test') + hashes = env.all_hashes() if env else None + + specs = spack.store.db.query(hashes=hashes) + specs = list(filter(lambda s: has_test_method(s.package), specs)) + + spack.cmd.display_specs(specs, long=True) + + +def test_find(args): # TODO: merge with status (noargs) + """Find tests that are running or have available results. + + Displays aliases for tests that have them, otherwise test suite content + hashes.""" + test_suites = spack.install_test.get_all_test_suites() + + # Filter tests by filter argument + if args.filter: + def create_filter(f): + raw = fnmatch.translate('f' if '*' in f or '?' in f + else '*' + f + '*') + return re.compile(raw, flags=re.IGNORECASE) + filters = [create_filter(f) for f in args.filter] + + def match(t, f): + return f.match(t) + test_suites = [t for t in test_suites + if any(match(t.alias, f) for f in filters) and + os.path.isdir(t.stage)] + + names = [t.name for t in test_suites] + + if names: + # TODO: Make these specify results vs active + msg = "Spack test results available for the following tests:\n" + msg += " %s\n" % ' '.join(names) + msg += " Run `spack test remove` to remove all tests" + tty.msg(msg) + else: + msg = "No test results match the query\n" + msg += " Tests may have been removed using `spack test remove`" + tty.msg(msg) + + +def test_status(args): + """Get the current status for the specified Spack test suite(s).""" + if args.names: + test_suites = [] + for name in args.names: + test_suite = spack.install_test.get_test_suite(name) + if test_suite: + test_suites.append(test_suite) + else: + tty.msg("No test suite %s found in test stage" % name) + else: + test_suites = spack.install_test.get_all_test_suites() + if not test_suites: + tty.msg("No test suites with status to report") + + for test_suite in test_suites: + # TODO: Make this handle capability tests too + # TODO: Make this handle tests running in another process + tty.msg("Test suite %s completed" % test_suite.name) + + +def _report_suite_results(test_suite, args, constraints): + """Report the relevant test suite results.""" + + # TODO: Make this handle capability tests too + # The results file may turn out to be a placeholder for future work + + if constraints: + # TBD: Should I be refactoring or re-using ConstraintAction? + qspecs = spack.cmd.parse_specs(constraints) + specs = {} + for spec in qspecs: + for s in spack.store.db.query(spec, installed=True): + specs[s.dag_hash()] = s + specs = sorted(specs.values()) + test_specs = dict((test_suite.test_pkg_id(s), s) for s in + test_suite.specs if s in specs) + else: + test_specs = dict((test_suite.test_pkg_id(s), s) for s in + test_suite.specs) + + if not test_specs: + return + + if os.path.exists(test_suite.results_file): + results_desc = 'Failing results' if args.failed else 'Results' + matching = ", spec matching '{0}'".format(' '.join(constraints)) \ + if constraints else '' + tty.msg("{0} for test suite '{1}'{2}:" + .format(results_desc, test_suite.name, matching)) + + results = {} + with open(test_suite.results_file, 'r') as f: + for line in f: + pkg_id, status = line.split() + results[pkg_id] = status + + for pkg_id in test_specs: + if pkg_id in results: + status = results[pkg_id] + if args.failed and status != 'FAILED': + continue + + msg = " {0} {1}".format(pkg_id, status) + if args.logs: + spec = test_specs[pkg_id] + log_file = test_suite.log_file_for_spec(spec) + if os.path.isfile(log_file): + with open(log_file, 'r') as f: + msg += '\n{0}'.format(''.join(f.readlines())) + tty.msg(msg) + else: + msg = "Test %s has no results.\n" % test_suite.name + msg += " Check if it is running with " + msg += "`spack test status %s`" % test_suite.name + tty.msg(msg) + + +def test_results(args): + """Get the results from Spack test suite(s) (default all).""" + if args.names: + try: + sep_index = args.names.index('--') + names = args.names[:sep_index] + constraints = args.names[sep_index + 1:] + except ValueError: + names = args.names + constraints = None + else: + names, constraints = None, None + + if names: + test_suites = [spack.install_test.get_test_suite(name) for name + in names] + if not test_suites: + tty.msg('No test suite(s) found in test stage: {0}' + .format(', '.join(names))) + else: + test_suites = spack.install_test.get_all_test_suites() + if not test_suites: + tty.msg("No test suites with results to report") + + for test_suite in test_suites: + _report_suite_results(test_suite, args, constraints) + + +def test_remove(args): + """Remove results from Spack test suite(s) (default all). + + If no test suite is listed, remove results for all suites. + + Removed tests can no longer be accessed for results or status, and will not + appear in `spack test list` results.""" + if args.names: + test_suites = [] + for name in args.names: + test_suite = spack.install_test.get_test_suite(name) + if test_suite: + test_suites.append(test_suite) + else: + tty.msg("No test suite %s found in test stage" % name) + else: + test_suites = spack.install_test.get_all_test_suites() + + if not test_suites: + tty.msg("No test suites to remove") + return + + if not args.yes_to_all: + msg = 'The following test suites will be removed:\n\n' + msg += ' ' + ' '.join(test.name for test in test_suites) + '\n' + tty.msg(msg) + answer = tty.get_yes_or_no('Do you want to proceed?', default=False) + if not answer: + tty.msg('Aborting removal of test suites') return - return pytest.main(pytest_args) + for test_suite in test_suites: + shutil.rmtree(test_suite.stage) + + +def test(parser, args): + globals()['test_%s' % args.test_command](args) diff --git a/lib/spack/spack/cmd/test_env.py b/lib/spack/spack/cmd/test_env.py new file mode 100644 index 0000000000..61e85046c1 --- /dev/null +++ b/lib/spack/spack/cmd/test_env.py @@ -0,0 +1,16 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) +import spack.cmd.common.env_utility as env_utility + +description = "run a command in a spec's test environment, " \ + "or dump its environment to screen or file" +section = "administration" +level = "long" + +setup_parser = env_utility.setup_parser + + +def test_env(parser, args): + env_utility.emulate_env_utility('test-env', 'test', args) diff --git a/lib/spack/spack/cmd/unit_test.py b/lib/spack/spack/cmd/unit_test.py new file mode 100644 index 0000000000..509211de04 --- /dev/null +++ b/lib/spack/spack/cmd/unit_test.py @@ -0,0 +1,169 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +from __future__ import print_function +from __future__ import division + +import collections +import sys +import re +import argparse +import pytest +from six import StringIO + +import llnl.util.tty.color as color +from llnl.util.filesystem import working_dir +from llnl.util.tty.colify import colify + +import spack.paths + +description = "run spack's unit tests (wrapper around pytest)" +section = "developer" +level = "long" + + +def setup_parser(subparser): + subparser.add_argument( + '-H', '--pytest-help', action='store_true', default=False, + help="show full pytest help, with advanced options") + + # extra spack arguments to list tests + list_group = subparser.add_argument_group("listing tests") + list_mutex = list_group.add_mutually_exclusive_group() + list_mutex.add_argument( + '-l', '--list', action='store_const', default=None, + dest='list', const='list', help="list test filenames") + list_mutex.add_argument( + '-L', '--list-long', action='store_const', default=None, + dest='list', const='long', help="list all test functions") + list_mutex.add_argument( + '-N', '--list-names', action='store_const', default=None, + dest='list', const='names', help="list full names of all tests") + + # use tests for extension + subparser.add_argument( + '--extension', default=None, + help="run test for a given spack extension") + + # spell out some common pytest arguments, so they'll show up in help + pytest_group = subparser.add_argument_group( + "common pytest arguments (spack unit-test --pytest-help for more)") + pytest_group.add_argument( + "-s", action='append_const', dest='parsed_args', const='-s', + help="print output while tests run (disable capture)") + pytest_group.add_argument( + "-k", action='store', metavar="EXPRESSION", dest='expression', + help="filter tests by keyword (can also use w/list options)") + pytest_group.add_argument( + "--showlocals", action='append_const', dest='parsed_args', + const='--showlocals', help="show local variable values in tracebacks") + + # remainder is just passed to pytest + subparser.add_argument( + 'pytest_args', nargs=argparse.REMAINDER, help="arguments for pytest") + + +def do_list(args, extra_args): + """Print a lists of tests than what pytest offers.""" + # Run test collection and get the tree out. + old_output = sys.stdout + try: + sys.stdout = output = StringIO() + pytest.main(['--collect-only'] + extra_args) + finally: + sys.stdout = old_output + + lines = output.getvalue().split('\n') + tests = collections.defaultdict(lambda: set()) + prefix = [] + + # collect tests into sections + for line in lines: + match = re.match(r"(\s*)<([^ ]*) '([^']*)'", line) + if not match: + continue + indent, nodetype, name = match.groups() + + # strip parametrized tests + if "[" in name: + name = name[:name.index("[")] + + depth = len(indent) // 2 + + if nodetype.endswith("Function"): + key = tuple(prefix) + tests[key].add(name) + else: + prefix = prefix[:depth] + prefix.append(name) + + def colorize(c, prefix): + if isinstance(prefix, tuple): + return "::".join( + color.colorize("@%s{%s}" % (c, p)) + for p in prefix if p != "()" + ) + return color.colorize("@%s{%s}" % (c, prefix)) + + if args.list == "list": + files = set(prefix[0] for prefix in tests) + color_files = [colorize("B", file) for file in sorted(files)] + colify(color_files) + + elif args.list == "long": + for prefix, functions in sorted(tests.items()): + path = colorize("*B", prefix) + "::" + functions = [colorize("c", f) for f in sorted(functions)] + color.cprint(path) + colify(functions, indent=4) + print() + + else: # args.list == "names" + all_functions = [ + colorize("*B", prefix) + "::" + colorize("c", f) + for prefix, functions in sorted(tests.items()) + for f in sorted(functions) + ] + colify(all_functions) + + +def add_back_pytest_args(args, unknown_args): + """Add parsed pytest args, unknown args, and remainder together. + + We add some basic pytest arguments to the Spack parser to ensure that + they show up in the short help, so we have to reassemble things here. + """ + result = args.parsed_args or [] + result += unknown_args or [] + result += args.pytest_args or [] + if args.expression: + result += ["-k", args.expression] + return result + + +def unit_test(parser, args, unknown_args): + if args.pytest_help: + # make the pytest.main help output more accurate + sys.argv[0] = 'spack test' + return pytest.main(['-h']) + + # add back any parsed pytest args we need to pass to pytest + pytest_args = add_back_pytest_args(args, unknown_args) + + # The default is to test the core of Spack. If the option `--extension` + # has been used, then test that extension. + pytest_root = spack.paths.spack_root + if args.extension: + target = args.extension + extensions = spack.config.get('config:extensions') + pytest_root = spack.extensions.path_for_extension(target, *extensions) + + # pytest.ini lives in the root of the spack repository. + with working_dir(pytest_root): + if args.list: + do_list(args, pytest_args) + return + + return pytest.main(pytest_args) diff --git a/lib/spack/spack/directives.py b/lib/spack/spack/directives.py index 7b2084d229..41276b5b48 100644 --- a/lib/spack/spack/directives.py +++ b/lib/spack/spack/directives.py @@ -299,8 +299,18 @@ def _depends_on(pkg, spec, when=None, type=default_deptype, patches=None): # call this patches here for clarity -- we want patch to be a list, # but the caller doesn't have to make it one. - if patches and dep_spec.virtual: - raise DependencyPatchError("Cannot patch a virtual dependency.") + + # Note: we cannot check whether a package is virtual in a directive + # because directives are run as part of class instantiation, and specs + # instantiate the package class as part of the `virtual` check. + # To be technical, specs only instantiate the package class as part of the + # virtual check if the provider index hasn't been created yet. + # TODO: There could be a cache warming strategy that would allow us to + # ensure `Spec.virtual` is a valid thing to call in a directive. + # For now, we comment out the following check to allow for virtual packages + # with package files. + # if patches and dep_spec.virtual: + # raise DependencyPatchError("Cannot patch a virtual dependency.") # ensure patches is a list if patches is None: diff --git a/lib/spack/spack/install_test.py b/lib/spack/spack/install_test.py new file mode 100644 index 0000000000..6c2c095a2e --- /dev/null +++ b/lib/spack/spack/install_test.py @@ -0,0 +1,266 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) +import base64 +import hashlib +import os +import re +import shutil +import sys +import tty + +import llnl.util.filesystem as fs + +from spack.spec import Spec + +import spack.error +import spack.util.prefix +import spack.util.spack_json as sjson + + +test_suite_filename = 'test_suite.lock' +results_filename = 'results.txt' + + +def get_escaped_text_output(filename): + """Retrieve and escape the expected text output from the file + + Args: + filename (str): path to the file + + Returns: + (list of str): escaped text lines read from the file + """ + with open(filename, 'r') as f: + # Ensure special characters are escaped as needed + expected = f.read() + + # Split the lines to make it easier to debug failures when there is + # a lot of output + return [re.escape(ln) for ln in expected.split('\n')] + + +def get_test_stage_dir(): + return spack.util.path.canonicalize_path( + spack.config.get('config:test_stage', '~/.spack/test')) + + +def get_all_test_suites(): + stage_root = get_test_stage_dir() + if not os.path.isdir(stage_root): + return [] + + def valid_stage(d): + dirpath = os.path.join(stage_root, d) + return (os.path.isdir(dirpath) and + test_suite_filename in os.listdir(dirpath)) + + candidates = [ + os.path.join(stage_root, d, test_suite_filename) + for d in os.listdir(stage_root) + if valid_stage(d) + ] + + test_suites = [TestSuite.from_file(c) for c in candidates] + return test_suites + + +def get_test_suite(name): + assert name, "Cannot search for empty test name or 'None'" + test_suites = get_all_test_suites() + names = [ts for ts in test_suites + if ts.name == name] + assert len(names) < 2, "alias shadows test suite hash" + + if not names: + return None + return names[0] + + +class TestSuite(object): + def __init__(self, specs, alias=None): + # copy so that different test suites have different package objects + # even if they contain the same spec + self.specs = [spec.copy() for spec in specs] + self.current_test_spec = None # spec currently tested, can be virtual + self.current_base_spec = None # spec currently running do_test + + self.alias = alias + self._hash = None + + @property + def name(self): + return self.alias if self.alias else self.content_hash + + @property + def content_hash(self): + if not self._hash: + json_text = sjson.dump(self.to_dict()) + sha = hashlib.sha1(json_text.encode('utf-8')) + b32_hash = base64.b32encode(sha.digest()).lower() + if sys.version_info[0] >= 3: + b32_hash = b32_hash.decode('utf-8') + self._hash = b32_hash + return self._hash + + def __call__(self, *args, **kwargs): + self.write_reproducibility_data() + + remove_directory = kwargs.get('remove_directory', True) + dirty = kwargs.get('dirty', False) + fail_first = kwargs.get('fail_first', False) + + for spec in self.specs: + try: + msg = "A package object cannot run in two test suites at once" + assert not spec.package.test_suite, msg + + # Set up the test suite to know which test is running + spec.package.test_suite = self + self.current_base_spec = spec + self.current_test_spec = spec + + # setup per-test directory in the stage dir + test_dir = self.test_dir_for_spec(spec) + if os.path.exists(test_dir): + shutil.rmtree(test_dir) + fs.mkdirp(test_dir) + + # run the package tests + spec.package.do_test( + dirty=dirty + ) + + # Clean up on success and log passed test + if remove_directory: + shutil.rmtree(test_dir) + self.write_test_result(spec, 'PASSED') + except BaseException as exc: + if isinstance(exc, SyntaxError): + # Create the test log file and report the error. + self.ensure_stage() + msg = 'Testing package {0}\n{1}'\ + .format(self.test_pkg_id(spec), str(exc)) + _add_msg_to_file(self.log_file_for_spec(spec), msg) + + self.write_test_result(spec, 'FAILED') + if fail_first: + break + finally: + spec.package.test_suite = None + self.current_test_spec = None + self.current_base_spec = None + + def ensure_stage(self): + if not os.path.exists(self.stage): + fs.mkdirp(self.stage) + + @property + def stage(self): + return spack.util.prefix.Prefix( + os.path.join(get_test_stage_dir(), self.content_hash)) + + @property + def results_file(self): + return self.stage.join(results_filename) + + @classmethod + def test_pkg_id(cls, spec): + """Build the standard install test package identifier + + Args: + spec (Spec): instance of the spec under test + + Returns: + (str): the install test package identifier + """ + return spec.format('{name}-{version}-{hash:7}') + + @classmethod + def test_log_name(cls, spec): + return '%s-test-out.txt' % cls.test_pkg_id(spec) + + def log_file_for_spec(self, spec): + return self.stage.join(self.test_log_name(spec)) + + def test_dir_for_spec(self, spec): + return self.stage.join(self.test_pkg_id(spec)) + + @property + def current_test_data_dir(self): + assert self.current_test_spec and self.current_base_spec + test_spec = self.current_test_spec + base_spec = self.current_base_spec + return self.test_dir_for_spec(base_spec).data.join(test_spec.name) + + def add_failure(self, exc, msg): + current_hash = self.current_base_spec.dag_hash() + current_failures = self.failures.get(current_hash, []) + current_failures.append((exc, msg)) + self.failures[current_hash] = current_failures + + def write_test_result(self, spec, result): + msg = "{0} {1}".format(self.test_pkg_id(spec), result) + _add_msg_to_file(self.results_file, msg) + + def write_reproducibility_data(self): + for spec in self.specs: + repo_cache_path = self.stage.repo.join(spec.name) + spack.repo.path.dump_provenance(spec, repo_cache_path) + for vspec in spec.package.virtuals_provided: + repo_cache_path = self.stage.repo.join(vspec.name) + if not os.path.exists(repo_cache_path): + try: + spack.repo.path.dump_provenance(vspec, repo_cache_path) + except spack.repo.UnknownPackageError: + pass # not all virtuals have package files + + with open(self.stage.join(test_suite_filename), 'w') as f: + sjson.dump(self.to_dict(), stream=f) + + def to_dict(self): + specs = [s.to_dict() for s in self.specs] + d = {'specs': specs} + if self.alias: + d['alias'] = self.alias + return d + + @staticmethod + def from_dict(d): + specs = [Spec.from_dict(spec_dict) for spec_dict in d['specs']] + alias = d.get('alias', None) + return TestSuite(specs, alias) + + @staticmethod + def from_file(filename): + try: + with open(filename, 'r') as f: + data = sjson.load(f) + return TestSuite.from_dict(data) + except Exception as e: + tty.debug(e) + raise sjson.SpackJSONError("error parsing JSON TestSuite:", str(e)) + + +def _add_msg_to_file(filename, msg): + """Add the message to the specified file + + Args: + filename (str): path to the file + msg (str): message to be appended to the file + """ + with open(filename, 'a+') as f: + f.write('{0}\n'.format(msg)) + + +class TestFailure(spack.error.SpackError): + """Raised when package tests have failed for an installation.""" + def __init__(self, failures): + # Failures are all exceptions + msg = "%d tests failed.\n" % len(failures) + for failure, message in failures: + msg += '\n\n%s\n' % str(failure) + msg += '\n%s\n' % message + + super(TestFailure, self).__init__(msg) diff --git a/lib/spack/spack/installer.py b/lib/spack/spack/installer.py index cf7b5bfb99..dd35db0839 100644 --- a/lib/spack/spack/installer.py +++ b/lib/spack/spack/installer.py @@ -1610,12 +1610,12 @@ def build_process(pkg, kwargs): This function's return value is returned to the parent process. """ - keep_stage = kwargs.get('keep_stage', False) + fake = kwargs.get('fake', False) install_source = kwargs.get('install_source', False) + keep_stage = kwargs.get('keep_stage', False) skip_patch = kwargs.get('skip_patch', False) - verbose = kwargs.get('verbose', False) - fake = kwargs.get('fake', False) unmodified_env = kwargs.get('unmodified_env', {}) + verbose = kwargs.get('verbose', False) start_time = time.time() if not fake: @@ -1958,6 +1958,7 @@ class BuildRequest(object): def _add_default_args(self): """Ensure standard install options are set to at least the default.""" for arg, default in [('cache_only', False), + ('context', 'build'), # installs *always* build ('dirty', False), ('fail_fast', False), ('fake', False), diff --git a/lib/spack/spack/modules/lmod.py b/lib/spack/spack/modules/lmod.py index 018edb35ad..80f6933063 100644 --- a/lib/spack/spack/modules/lmod.py +++ b/lib/spack/spack/modules/lmod.py @@ -12,6 +12,7 @@ import collections import spack.config import spack.compilers import spack.spec +import spack.repo import spack.error import spack.tengine as tengine @@ -125,7 +126,9 @@ class LmodConfiguration(BaseConfiguration): # Check if all the tokens in the hierarchy are virtual specs. # If not warn the user and raise an error. - not_virtual = [t for t in tokens if not spack.spec.Spec.is_virtual(t)] + not_virtual = [t for t in tokens + if t != 'compiler' and + not spack.repo.path.is_virtual(t)] if not_virtual: msg = "Non-virtual specs in 'hierarchy' list for lmod: {0}\n" msg += "Please check the 'modules.yaml' configuration files" diff --git a/lib/spack/spack/package.py b/lib/spack/spack/package.py index b8dca6e55a..de394e2d45 100644 --- a/lib/spack/spack/package.py +++ b/lib/spack/spack/package.py @@ -24,10 +24,12 @@ import sys import textwrap import time import traceback - import six +import types +import llnl.util.filesystem as fsys import llnl.util.tty as tty + import spack.compilers import spack.config import spack.dependency @@ -45,15 +47,14 @@ import spack.store import spack.url import spack.util.environment import spack.util.web -from llnl.util.filesystem import mkdirp, touch, working_dir from llnl.util.lang import memoized from llnl.util.link_tree import LinkTree from ordereddict_backport import OrderedDict -from six import StringIO -from six import string_types -from six import with_metaclass from spack.filesystem_view import YamlFilesystemView from spack.installer import PackageInstaller, InstallError +from spack.install_test import TestFailure, TestSuite +from spack.util.executable import which, ProcessError +from spack.util.prefix import Prefix from spack.stage import stage_prefix, Stage, ResourceStage, StageComposite from spack.util.package_hash import package_hash from spack.version import Version @@ -452,7 +453,21 @@ class PackageViewMixin(object): view.remove_file(src, dst) -class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): +def test_log_pathname(test_stage, spec): + """Build the pathname of the test log file + + Args: + test_stage (str): path to the test stage directory + spec (Spec): instance of the spec under test + + Returns: + (str): the pathname of the test log file + """ + return os.path.join(test_stage, + 'test-{0}-out.txt'.format(TestSuite.test_pkg_id(spec))) + + +class PackageBase(six.with_metaclass(PackageMeta, PackageViewMixin, object)): """This is the superclass for all spack packages. ***The Package class*** @@ -542,6 +557,10 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): #: are executed or 'None' if there are no such test functions. build_time_test_callbacks = None + #: By default, packages are not virtual + #: Virtual packages override this attribute + virtual = False + #: Most Spack packages are used to install source or binary code while #: those that do not can be used to install a set of other Spack packages. has_code = True @@ -633,6 +652,18 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): metadata_attrs = ['homepage', 'url', 'urls', 'list_url', 'extendable', 'parallel', 'make_jobs'] + #: Boolean. If set to ``True``, the smoke/install test requires a compiler. + #: This is currently used by smoke tests to ensure a compiler is available + #: to build a custom test code. + test_requires_compiler = False + + #: List of test failures encountered during a smoke/install test run. + test_failures = None + + #: TestSuite instance used to manage smoke/install tests for one or more + #: specs. + test_suite = None + def __init__(self, spec): # this determines how the package should be built. self.spec = spec @@ -1002,19 +1033,22 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): return os.path.join(self.stage.path, _spack_build_envfile) @property + def metadata_dir(self): + """Return the install metadata directory.""" + return spack.store.layout.metadata_path(self.spec) + + @property def install_env_path(self): """ Return the build environment file path on successful installation. """ - install_path = spack.store.layout.metadata_path(self.spec) - # Backward compatibility: Return the name of an existing log path; # otherwise, return the current install env path name. - old_filename = os.path.join(install_path, 'build.env') + old_filename = os.path.join(self.metadata_dir, 'build.env') if os.path.exists(old_filename): return old_filename else: - return os.path.join(install_path, _spack_build_envfile) + return os.path.join(self.metadata_dir, _spack_build_envfile) @property def log_path(self): @@ -1031,16 +1065,14 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): @property def install_log_path(self): """Return the build log file path on successful installation.""" - install_path = spack.store.layout.metadata_path(self.spec) - # Backward compatibility: Return the name of an existing install log. for filename in ['build.out', 'build.txt']: - old_log = os.path.join(install_path, filename) + old_log = os.path.join(self.metadata_dir, filename) if os.path.exists(old_log): return old_log # Otherwise, return the current install log path name. - return os.path.join(install_path, _spack_build_logfile) + return os.path.join(self.metadata_dir, _spack_build_logfile) @property def configure_args_path(self): @@ -1050,9 +1082,12 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): @property def install_configure_args_path(self): """Return the configure args file path on successful installation.""" - install_path = spack.store.layout.metadata_path(self.spec) + return os.path.join(self.metadata_dir, _spack_configure_argsfile) - return os.path.join(install_path, _spack_configure_argsfile) + @property + def install_test_root(self): + """Return the install test root directory.""" + return os.path.join(self.metadata_dir, 'test') def _make_fetcher(self): # Construct a composite fetcher that always contains at least @@ -1322,7 +1357,7 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): raise FetchError("Archive was empty for %s" % self.name) else: # Support for post-install hooks requires a stage.source_path - mkdirp(self.stage.source_path) + fsys.mkdirp(self.stage.source_path) def do_patch(self): """Applies patches if they haven't been applied already.""" @@ -1368,7 +1403,7 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): patched = False for patch in patches: try: - with working_dir(self.stage.source_path): + with fsys.working_dir(self.stage.source_path): patch.apply(self.stage) tty.debug('Applied patch {0}'.format(patch.path_or_url)) patched = True @@ -1377,12 +1412,12 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): # Touch bad file if anything goes wrong. tty.msg('Patch %s failed.' % patch.path_or_url) - touch(bad_file) + fsys.touch(bad_file) raise if has_patch_fun: try: - with working_dir(self.stage.source_path): + with fsys.working_dir(self.stage.source_path): self.patch() tty.debug('Ran patch() for {0}'.format(self.name)) patched = True @@ -1400,7 +1435,7 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): # Touch bad file if anything goes wrong. tty.msg('patch() function failed for {0}'.format(self.name)) - touch(bad_file) + fsys.touch(bad_file) raise # Get rid of any old failed file -- patches have either succeeded @@ -1411,9 +1446,9 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): # touch good or no patches file so that we skip next time. if patched: - touch(good_file) + fsys.touch(good_file) else: - touch(no_patches_file) + fsys.touch(no_patches_file) @classmethod def all_patches(cls): @@ -1657,6 +1692,175 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): builder = PackageInstaller([(self, kwargs)]) builder.install() + def cache_extra_test_sources(self, srcs): + """Copy relative source paths to the corresponding install test subdir + + This method is intended as an optional install test setup helper for + grabbing source files/directories during the installation process and + copying them to the installation test subdirectory for subsequent use + during install testing. + + Args: + srcs (str or list of str): relative path for files and or + subdirectories located in the staged source path that are to + be copied to the corresponding location(s) under the install + testing directory. + """ + paths = [srcs] if isinstance(srcs, six.string_types) else srcs + + for path in paths: + src_path = os.path.join(self.stage.source_path, path) + dest_path = os.path.join(self.install_test_root, path) + if os.path.isdir(src_path): + fsys.install_tree(src_path, dest_path) + else: + fsys.mkdirp(os.path.dirname(dest_path)) + fsys.copy(src_path, dest_path) + + def do_test(self, dirty=False): + if self.test_requires_compiler: + compilers = spack.compilers.compilers_for_spec( + self.spec.compiler, arch_spec=self.spec.architecture) + if not compilers: + tty.error('Skipping tests for package %s\n' % + self.spec.format('{name}-{version}-{hash:7}') + + 'Package test requires missing compiler %s' % + self.spec.compiler) + return + + # Clear test failures + self.test_failures = [] + self.test_log_file = self.test_suite.log_file_for_spec(self.spec) + fsys.touch(self.test_log_file) # Otherwise log_parse complains + + kwargs = {'dirty': dirty, 'fake': False, 'context': 'test'} + spack.build_environment.start_build_process(self, test_process, kwargs) + + def test(self): + pass + + def run_test(self, exe, options=[], expected=[], status=0, + installed=False, purpose='', skip_missing=False, + work_dir=None): + """Run the test and confirm the expected results are obtained + + Log any failures and continue, they will be re-raised later + + Args: + exe (str): the name of the executable + options (str or list of str): list of options to pass to the runner + expected (str or list of str): list of expected output strings. + Each string is a regex expected to match part of the output. + status (int or list of int): possible passing status values + with 0 meaning the test is expected to succeed + installed (bool): if ``True``, the executable must be in the + install prefix + purpose (str): message to display before running test + skip_missing (bool): skip the test if the executable is not + in the install prefix bin directory or the provided work_dir + work_dir (str or None): path to the smoke test directory + """ + wdir = '.' if work_dir is None else work_dir + with fsys.working_dir(wdir): + try: + runner = which(exe) + if runner is None and skip_missing: + return + assert runner is not None, \ + "Failed to find executable '{0}'".format(exe) + + self._run_test_helper( + runner, options, expected, status, installed, purpose) + print("PASSED") + return True + except BaseException as e: + # print a summary of the error to the log file + # so that cdash and junit reporters know about it + exc_type, _, tb = sys.exc_info() + print('FAILED: {0}'.format(e)) + import traceback + # remove the current call frame to exclude the extract_stack + # call from the error + stack = traceback.extract_stack()[:-1] + + # Package files have a line added at import time, so we re-read + # the file to make line numbers match. We have to subtract two + # from the line number because the original line number is + # inflated once by the import statement and the lines are + # displaced one by the import statement. + for i, entry in enumerate(stack): + filename, lineno, function, text = entry + if spack.repo.is_package_file(filename): + with open(filename, 'r') as f: + lines = f.readlines() + new_lineno = lineno - 2 + text = lines[new_lineno] + stack[i] = (filename, new_lineno, function, text) + + # Format the stack to print and print it + out = traceback.format_list(stack) + for line in out: + print(line.rstrip('\n')) + + if exc_type is spack.util.executable.ProcessError: + out = six.StringIO() + spack.build_environment.write_log_summary( + out, 'test', self.test_log_file, last=1) + m = out.getvalue() + else: + # We're below the package context, so get context from + # stack instead of from traceback. + # The traceback is truncated here, so we can't use it to + # traverse the stack. + m = '\n'.join( + spack.build_environment.get_package_context(tb) + ) + + exc = e # e is deleted after this block + + # If we fail fast, raise another error + if spack.config.get('config:fail_fast', False): + raise TestFailure([(exc, m)]) + else: + self.test_failures.append((exc, m)) + return False + + def _run_test_helper(self, runner, options, expected, status, installed, + purpose): + status = [status] if isinstance(status, six.integer_types) else status + expected = [expected] if isinstance(expected, six.string_types) else \ + expected + options = [options] if isinstance(options, six.string_types) else \ + options + + if purpose: + tty.msg(purpose) + else: + tty.debug('test: {0}: expect command status in {1}' + .format(runner.name, status)) + + if installed: + msg = "Executable '{0}' expected in prefix".format(runner.name) + msg += ", found in {0} instead".format(runner.path) + assert runner.path.startswith(self.spec.prefix), msg + + try: + output = runner(*options, output=str.split, error=str.split) + + assert 0 in status, \ + 'Expected {0} execution to fail'.format(runner.name) + except ProcessError as err: + output = str(err) + match = re.search(r'exited with status ([0-9]+)', output) + if not (match and int(match.group(1)) in status): + raise + + for check in expected: + cmd = ' '.join([runner.name] + options) + msg = "Expected '{0}' to match output of `{1}`".format(check, cmd) + msg += '\n\nOutput: {0}'.format(output) + assert re.search(check, output), msg + def unit_test_check(self): """Hook for unit tests to assert things about package internals. @@ -1678,7 +1882,7 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): """This function checks whether install succeeded.""" def check_paths(path_list, filetype, predicate): - if isinstance(path_list, string_types): + if isinstance(path_list, six.string_types): path_list = [path_list] for path in path_list: @@ -2031,7 +2235,7 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): # copy spec metadata to "deprecated" dir of deprecator depr_yaml = spack.store.layout.deprecated_file_path(spec, deprecator) - fs.mkdirp(os.path.dirname(depr_yaml)) + fsys.mkdirp(os.path.dirname(depr_yaml)) shutil.copy2(self_yaml, depr_yaml) # Any specs deprecated in favor of this spec are re-deprecated in @@ -2210,7 +2414,7 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): doc = re.sub(r'\s+', ' ', self.__doc__) lines = textwrap.wrap(doc, 72) - results = StringIO() + results = six.StringIO() for line in lines: results.write((" " * indent) + line + "\n") return results.getvalue() @@ -2313,6 +2517,71 @@ class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)): tty.warn(msg.format(name)) +def test_process(pkg, kwargs): + with tty.log.log_output(pkg.test_log_file) as logger: + with logger.force_echo(): + tty.msg('Testing package {0}' + .format(pkg.test_suite.test_pkg_id(pkg.spec))) + + # use debug print levels for log file to record commands + old_debug = tty.is_debug() + tty.set_debug(True) + + # run test methods from the package and all virtuals it + # provides virtuals have to be deduped by name + v_names = list(set([vspec.name + for vspec in pkg.virtuals_provided])) + + # hack for compilers that are not dependencies (yet) + # TODO: this all eventually goes away + c_names = ('gcc', 'intel', 'intel-parallel-studio', 'pgi') + if pkg.name in c_names: + v_names.extend(['c', 'cxx', 'fortran']) + if pkg.spec.satisfies('llvm+clang'): + v_names.extend(['c', 'cxx']) + + test_specs = [pkg.spec] + [spack.spec.Spec(v_name) + for v_name in sorted(v_names)] + + try: + with fsys.working_dir( + pkg.test_suite.test_dir_for_spec(pkg.spec)): + for spec in test_specs: + pkg.test_suite.current_test_spec = spec + # Fail gracefully if a virtual has no package/tests + try: + spec_pkg = spec.package + except spack.repo.UnknownPackageError: + continue + + # copy test data into test data dir + data_source = Prefix(spec_pkg.package_dir).test + data_dir = pkg.test_suite.current_test_data_dir + if (os.path.isdir(data_source) and + not os.path.exists(data_dir)): + # We assume data dir is used read-only + # maybe enforce this later + shutil.copytree(data_source, data_dir) + + # grab the function for each method so we can call + # it with the package + test_fn = spec_pkg.__class__.test + if not isinstance(test_fn, types.FunctionType): + test_fn = test_fn.__func__ + + # Run the tests + test_fn(pkg) + + # If fail-fast was on, we error out above + # If we collect errors, raise them in batch here + if pkg.test_failures: + raise TestFailure(pkg.test_failures) + + finally: + # reset debug level + tty.set_debug(old_debug) + + inject_flags = PackageBase.inject_flags env_flags = PackageBase.env_flags build_system_flags = PackageBase.build_system_flags diff --git a/lib/spack/spack/paths.py b/lib/spack/spack/paths.py index cb2240359c..9c803cba7e 100644 --- a/lib/spack/spack/paths.py +++ b/lib/spack/spack/paths.py @@ -12,7 +12,6 @@ dependencies. import os from llnl.util.filesystem import ancestor - #: This file lives in $prefix/lib/spack/spack/__file__ prefix = ancestor(__file__, 4) @@ -42,6 +41,7 @@ test_path = os.path.join(module_path, "test") hooks_path = os.path.join(module_path, "hooks") var_path = os.path.join(prefix, "var", "spack") repos_path = os.path.join(var_path, "repos") +tests_path = os.path.join(var_path, "tests") share_path = os.path.join(prefix, "share", "spack") # Paths to built-in Spack repositories. diff --git a/lib/spack/spack/pkgkit.py b/lib/spack/spack/pkgkit.py index ac0a7eee0d..4f25d41dfb 100644 --- a/lib/spack/spack/pkgkit.py +++ b/lib/spack/spack/pkgkit.py @@ -59,6 +59,7 @@ from spack.package import \ from spack.installer import \ ExternalPackageError, InstallError, InstallLockError, UpstreamPackageError +from spack.install_test import get_escaped_text_output from spack.variant import any_combination_of, auto_or_any_combination_of from spack.variant import disjoint_sets diff --git a/lib/spack/spack/repo.py b/lib/spack/spack/repo.py index 5e1457c165..4af7b382f0 100644 --- a/lib/spack/spack/repo.py +++ b/lib/spack/spack/repo.py @@ -91,6 +91,19 @@ def autospec(function): return converter +def is_package_file(filename): + """Determine whether we are in a package file from a repo.""" + # Package files are named `package.py` and are not in lib/spack/spack + # We have to remove the file extension because it can be .py and can be + # .pyc depending on context, and can differ between the files + import spack.package # break cycle + filename_noext = os.path.splitext(filename)[0] + packagebase_filename_noext = os.path.splitext( + inspect.getfile(spack.package.PackageBase))[0] + return (filename_noext != packagebase_filename_noext and + os.path.basename(filename_noext) == 'package') + + class SpackNamespace(types.ModuleType): """ Allow lazy loading of modules.""" @@ -131,6 +144,11 @@ class FastPackageChecker(Mapping): #: Reference to the appropriate entry in the global cache self._packages_to_stats = self._paths_cache[packages_path] + def invalidate(self): + """Regenerate cache for this checker.""" + self._paths_cache[self.packages_path] = self._create_new_cache() + self._packages_to_stats = self._paths_cache[self.packages_path] + def _create_new_cache(self): """Create a new cache for packages in a repo. @@ -308,6 +326,9 @@ class ProviderIndexer(Indexer): self.index = spack.provider_index.ProviderIndex.from_json(stream) def update(self, pkg_fullname): + name = pkg_fullname.split('.')[-1] + if spack.repo.path.is_virtual(name, use_index=False): + return self.index.remove_provider(pkg_fullname) self.index.update(pkg_fullname) @@ -517,12 +538,12 @@ class RepoPath(object): """Get the first repo in precedence order.""" return self.repos[0] if self.repos else None - def all_package_names(self): + def all_package_names(self, include_virtuals=False): """Return all unique package names in all repositories.""" if self._all_package_names is None: all_pkgs = set() for repo in self.repos: - for name in repo.all_package_names(): + for name in repo.all_package_names(include_virtuals): all_pkgs.add(name) self._all_package_names = sorted(all_pkgs, key=lambda n: n.lower()) return self._all_package_names @@ -679,12 +700,20 @@ class RepoPath(object): """ return any(repo.exists(pkg_name) for repo in self.repos) - def is_virtual(self, pkg_name): - """True if the package with this name is virtual, False otherwise.""" - if not isinstance(pkg_name, str): + def is_virtual(self, pkg_name, use_index=True): + """True if the package with this name is virtual, False otherwise. + + Set `use_index` False when calling from a code block that could + be run during the computation of the provider index.""" + have_name = pkg_name is not None + if have_name and not isinstance(pkg_name, str): raise ValueError( "is_virtual(): expected package name, got %s" % type(pkg_name)) - return pkg_name in self.provider_index + if use_index: + return have_name and pkg_name in self.provider_index + else: + return have_name and (not self.exists(pkg_name) or + self.get_pkg_class(pkg_name).virtual) def __contains__(self, pkg_name): return self.exists(pkg_name) @@ -913,10 +942,6 @@ class Repo(object): This dumps the package file and any associated patch files. Raises UnknownPackageError if not found. """ - # Some preliminary checks. - if spec.virtual: - raise UnknownPackageError(spec.name) - if spec.namespace and spec.namespace != self.namespace: raise UnknownPackageError( "Repository %s does not contain package %s." @@ -999,9 +1024,12 @@ class Repo(object): self._fast_package_checker = FastPackageChecker(self.packages_path) return self._fast_package_checker - def all_package_names(self): + def all_package_names(self, include_virtuals=False): """Returns a sorted list of all package names in the Repo.""" - return sorted(self._pkg_checker.keys()) + names = sorted(self._pkg_checker.keys()) + if include_virtuals: + return names + return [x for x in names if not self.is_virtual(x)] def packages_with_tags(self, *tags): v = set(self.all_package_names()) @@ -1040,7 +1068,7 @@ class Repo(object): def is_virtual(self, pkg_name): """True if the package with this name is virtual, False otherwise.""" - return self.provider_index.contains(pkg_name) + return pkg_name in self.provider_index def _get_pkg_module(self, pkg_name): """Create a module for a particular package. @@ -1074,7 +1102,8 @@ class Repo(object): # manually construct the error message in order to give the # user the correct package.py where the syntax error is located raise SyntaxError('invalid syntax in {0:}, line {1:}' - ''.format(file_path, e.lineno)) + .format(file_path, e.lineno)) + module.__package__ = self.full_namespace module.__loader__ = self self._modules[pkg_name] = module @@ -1205,9 +1234,9 @@ def get(spec): return path.get(spec) -def all_package_names(): +def all_package_names(include_virtuals=False): """Convenience wrapper around ``spack.repo.all_package_names()``.""" - return path.all_package_names() + return path.all_package_names(include_virtuals) def set_path(repo): diff --git a/lib/spack/spack/report.py b/lib/spack/spack/report.py index 4e5bc9c993..ebae2d0adc 100644 --- a/lib/spack/spack/report.py +++ b/lib/spack/spack/report.py @@ -9,11 +9,13 @@ import collections import functools import time import traceback +import os import llnl.util.lang import spack.build_environment import spack.fetch_strategy import spack.package +from spack.install_test import TestSuite from spack.reporter import Reporter from spack.reporters.cdash import CDash from spack.reporters.junit import JUnit @@ -33,12 +35,16 @@ __all__ = [ ] -def fetch_package_log(pkg): +def fetch_log(pkg, do_fn, dir): + log_files = { + '_install_task': pkg.build_log_path, + 'do_test': os.path.join(dir, TestSuite.test_log_name(pkg.spec)), + } try: - with codecs.open(pkg.build_log_path, 'r', 'utf-8') as f: + with codecs.open(log_files[do_fn.__name__], 'r', 'utf-8') as f: return ''.join(f.readlines()) except Exception: - return 'Cannot open build log for {0}'.format( + return 'Cannot open log for {0}'.format( pkg.spec.cshort_spec ) @@ -58,15 +64,20 @@ class InfoCollector(object): specs (list of Spec): specs whose install information will be recorded """ - #: Backup of PackageInstaller._install_task - _backup__install_task = spack.package.PackageInstaller._install_task - - def __init__(self, specs): - #: Specs that will be installed + def __init__(self, wrap_class, do_fn, specs, dir): + #: Class for which to wrap a function + self.wrap_class = wrap_class + #: Action to be reported on + self.do_fn = do_fn + #: Backup of PackageBase function + self._backup_do_fn = getattr(self.wrap_class, do_fn) + #: Specs that will be acted on self.input_specs = specs #: This is where we record the data that will be included #: in our report. self.specs = [] + #: Record directory for test log paths + self.dir = dir def __enter__(self): # Initialize the spec report with the data that is available upfront. @@ -98,30 +109,37 @@ class InfoCollector(object): Property('compiler', input_spec.compiler)) # Check which specs are already installed and mark them as skipped - for dep in filter(lambda x: x.package.installed, - input_spec.traverse()): - package = { - 'name': dep.name, - 'id': dep.dag_hash(), - 'elapsed_time': '0.0', - 'result': 'skipped', - 'message': 'Spec already installed' - } - spec['packages'].append(package) - - def gather_info(_install_task): - """Decorates PackageInstaller._install_task to gather useful - information on PackageBase.do_install for a CI report. + # only for install_task + if self.do_fn == '_install_task': + for dep in filter(lambda x: x.package.installed, + input_spec.traverse()): + package = { + 'name': dep.name, + 'id': dep.dag_hash(), + 'elapsed_time': '0.0', + 'result': 'skipped', + 'message': 'Spec already installed' + } + spec['packages'].append(package) + + def gather_info(do_fn): + """Decorates do_fn to gather useful information for + a CI report. It's defined here to capture the environment and build this context as the installations proceed. """ - @functools.wraps(_install_task) - def wrapper(installer, task, *args, **kwargs): - pkg = task.pkg + @functools.wraps(do_fn) + def wrapper(instance, *args, **kwargs): + if isinstance(instance, spack.package.PackageBase): + pkg = instance + elif hasattr(args[0], 'pkg'): + pkg = args[0].pkg + else: + raise Exception # We accounted before for what is already installed - installed_on_entry = pkg.installed + installed_already = pkg.installed package = { 'name': pkg.name, @@ -135,13 +153,12 @@ class InfoCollector(object): start_time = time.time() value = None try: - - value = _install_task(installer, task, *args, **kwargs) + value = do_fn(instance, *args, **kwargs) package['result'] = 'success' - package['stdout'] = fetch_package_log(pkg) + package['stdout'] = fetch_log(pkg, do_fn, self.dir) package['installed_from_binary_cache'] = \ pkg.installed_from_binary_cache - if installed_on_entry: + if do_fn.__name__ == '_install_task' and installed_already: return except spack.build_environment.InstallError as e: @@ -149,7 +166,7 @@ class InfoCollector(object): # didn't work correctly) package['result'] = 'failure' package['message'] = e.message or 'Installation failure' - package['stdout'] = fetch_package_log(pkg) + package['stdout'] = fetch_log(pkg, do_fn, self.dir) package['stdout'] += package['message'] package['exception'] = e.traceback @@ -157,7 +174,7 @@ class InfoCollector(object): # Everything else is an error (the installation # failed outside of the child process) package['result'] = 'error' - package['stdout'] = fetch_package_log(pkg) + package['stdout'] = fetch_log(pkg, do_fn, self.dir) package['message'] = str(e) or 'Unknown error' package['exception'] = traceback.format_exc() @@ -184,15 +201,14 @@ class InfoCollector(object): return wrapper - spack.package.PackageInstaller._install_task = gather_info( - spack.package.PackageInstaller._install_task - ) + setattr(self.wrap_class, self.do_fn, gather_info( + getattr(self.wrap_class, self.do_fn) + )) def __exit__(self, exc_type, exc_val, exc_tb): - # Restore the original method in PackageInstaller - spack.package.PackageInstaller._install_task = \ - InfoCollector._backup__install_task + # Restore the original method in PackageBase + setattr(self.wrap_class, self.do_fn, self._backup_do_fn) for spec in self.specs: spec['npackages'] = len(spec['packages']) @@ -225,22 +241,26 @@ class collect_info(object): # The file 'junit.xml' is written when exiting # the context - specs = [Spec('hdf5').concretized()] - with collect_info(specs, 'junit', 'junit.xml'): + s = [Spec('hdf5').concretized()] + with collect_info(PackageBase, do_install, s, 'junit', 'a.xml'): # A report will be generated for these specs... - for spec in specs: - spec.do_install() + for spec in s: + getattr(class, function)(spec) # ...but not for this one Spec('zlib').concretized().do_install() Args: + class: class on which to wrap a function + function: function to wrap format_name (str or None): one of the supported formats - args (dict): args passed to spack install + args (dict): args passed to function Raises: ValueError: when ``format_name`` is not in ``valid_formats`` """ - def __init__(self, format_name, args): + def __init__(self, cls, function, format_name, args): + self.cls = cls + self.function = function self.filename = None if args.cdash_upload_url: self.format_name = 'cdash' @@ -253,13 +273,19 @@ class collect_info(object): .format(self.format_name)) self.report_writer = report_writers[self.format_name](args) + def __call__(self, type, dir=os.getcwd()): + self.type = type + self.dir = dir + return self + def concretization_report(self, msg): self.report_writer.concretization_report(self.filename, msg) def __enter__(self): if self.format_name: - # Start the collector and patch PackageInstaller._install_task - self.collector = InfoCollector(self.specs) + # Start the collector and patch self.function on appropriate class + self.collector = InfoCollector( + self.cls, self.function, self.specs, self.dir) self.collector.__enter__() def __exit__(self, exc_type, exc_val, exc_tb): @@ -269,4 +295,5 @@ class collect_info(object): self.collector.__exit__(exc_type, exc_val, exc_tb) report_data = {'specs': self.collector.specs} - self.report_writer.build_report(self.filename, report_data) + report_fn = getattr(self.report_writer, '%s_report' % self.type) + report_fn(self.filename, report_data) diff --git a/lib/spack/spack/reporter.py b/lib/spack/spack/reporter.py index 25d4042a5e..6314054139 100644 --- a/lib/spack/spack/reporter.py +++ b/lib/spack/spack/reporter.py @@ -16,5 +16,8 @@ class Reporter(object): def build_report(self, filename, report_data): pass + def test_report(self, filename, report_data): + pass + def concretization_report(self, filename, msg): pass diff --git a/lib/spack/spack/reporters/cdash.py b/lib/spack/spack/reporters/cdash.py index 580df7866f..c1a220963e 100644 --- a/lib/spack/spack/reporters/cdash.py +++ b/lib/spack/spack/reporters/cdash.py @@ -72,8 +72,10 @@ class CDash(Reporter): tty.verbose("Using CDash auth token from environment") self.authtoken = os.environ.get('SPACK_CDASH_AUTH_TOKEN') - if args.spec: + if getattr(args, 'spec', ''): packages = args.spec + elif getattr(args, 'specs', ''): + packages = args.specs else: packages = [] for file in args.specfiles: @@ -98,7 +100,7 @@ class CDash(Reporter): self.revision = git('rev-parse', 'HEAD', output=str).strip() self.multiple_packages = False - def report_for_package(self, directory_name, package, duration): + def build_report_for_package(self, directory_name, package, duration): if 'stdout' not in package: # Skip reporting on packages that did not generate any output. return @@ -158,8 +160,8 @@ class CDash(Reporter): '\n'.join(report_data[phase]['loglines']) errors, warnings = parse_log_events(report_data[phase]['loglines']) # Cap the number of errors and warnings at 50 each. - errors = errors[0:49] - warnings = warnings[0:49] + errors = errors[:50] + warnings = warnings[:50] nerrors = len(errors) if phase == 'configure' and nerrors > 0: @@ -250,7 +252,114 @@ class CDash(Reporter): if 'time' in spec: duration = int(spec['time']) for package in spec['packages']: - self.report_for_package(directory_name, package, duration) + self.build_report_for_package( + directory_name, package, duration) + self.print_cdash_link() + + def test_report_for_package(self, directory_name, package, duration): + if 'stdout' not in package: + # Skip reporting on packages that did not generate any output. + return + + self.current_package_name = package['name'] + self.buildname = "{0} - {1}".format( + self.base_buildname, package['name']) + + report_data = self.initialize_report(directory_name) + + for phase in ('test', 'update'): + report_data[phase] = {} + report_data[phase]['loglines'] = [] + report_data[phase]['status'] = 0 + report_data[phase]['endtime'] = self.endtime + + # Track the phases we perform so we know what reports to create. + # We always report the update step because this is how we tell CDash + # what revision of Spack we are using. + phases_encountered = ['test', 'update'] + + # Generate a report for this package. + # The first line just says "Testing package name-hash" + report_data['test']['loglines'].append( + text_type("{0} output for {1}:".format( + 'test', package['name']))) + for line in package['stdout'].splitlines()[1:]: + report_data['test']['loglines'].append( + xml.sax.saxutils.escape(line)) + + self.starttime = self.endtime - duration + for phase in phases_encountered: + report_data[phase]['starttime'] = self.starttime + report_data[phase]['log'] = \ + '\n'.join(report_data[phase]['loglines']) + errors, warnings = parse_log_events(report_data[phase]['loglines']) + # Cap the number of errors and warnings at 50 each. + errors = errors[0:49] + warnings = warnings[0:49] + + if phase == 'test': + # Convert log output from ASCII to Unicode and escape for XML. + def clean_log_event(event): + event = vars(event) + event['text'] = xml.sax.saxutils.escape(event['text']) + event['pre_context'] = xml.sax.saxutils.escape( + '\n'.join(event['pre_context'])) + event['post_context'] = xml.sax.saxutils.escape( + '\n'.join(event['post_context'])) + # source_file and source_line_no are either strings or + # the tuple (None,). Distinguish between these two cases. + if event['source_file'][0] is None: + event['source_file'] = '' + event['source_line_no'] = '' + else: + event['source_file'] = xml.sax.saxutils.escape( + event['source_file']) + return event + + # Convert errors to warnings if the package reported success. + if package['result'] == 'success': + warnings = errors + warnings + errors = [] + + report_data[phase]['errors'] = [] + report_data[phase]['warnings'] = [] + for error in errors: + report_data[phase]['errors'].append(clean_log_event(error)) + for warning in warnings: + report_data[phase]['warnings'].append( + clean_log_event(warning)) + + if phase == 'update': + report_data[phase]['revision'] = self.revision + + # Write the report. + report_name = phase.capitalize() + ".xml" + report_file_name = package['name'] + "_" + report_name + phase_report = os.path.join(directory_name, report_file_name) + + with codecs.open(phase_report, 'w', 'utf-8') as f: + env = spack.tengine.make_environment() + if phase != 'update': + # Update.xml stores site information differently + # than the rest of the CTest XML files. + site_template = os.path.join(self.template_dir, 'Site.xml') + t = env.get_template(site_template) + f.write(t.render(report_data)) + + phase_template = os.path.join(self.template_dir, report_name) + t = env.get_template(phase_template) + f.write(t.render(report_data)) + self.upload(phase_report) + + def test_report(self, directory_name, input_data): + # Generate reports for each package in each spec. + for spec in input_data['specs']: + duration = 0 + if 'time' in spec: + duration = int(spec['time']) + for package in spec['packages']: + self.test_report_for_package( + directory_name, package, duration) self.print_cdash_link() def concretization_report(self, directory_name, msg): diff --git a/lib/spack/spack/reporters/junit.py b/lib/spack/spack/reporters/junit.py index 6c54c45b42..598b308934 100644 --- a/lib/spack/spack/reporters/junit.py +++ b/lib/spack/spack/reporters/junit.py @@ -27,3 +27,6 @@ class JUnit(Reporter): env = spack.tengine.make_environment() t = env.get_template(self.template_file) f.write(t.render(report_data)) + + def test_report(self, filename, report_data): + self.build_report(filename, report_data) diff --git a/lib/spack/spack/schema/config.py b/lib/spack/spack/schema/config.py index b1d3332a3f..0f83eb86f4 100644 --- a/lib/spack/spack/schema/config.py +++ b/lib/spack/spack/schema/config.py @@ -45,6 +45,7 @@ properties = { {'type': 'array', 'items': {'type': 'string'}}], }, + 'test_stage': {'type': 'string'}, 'extensions': { 'type': 'array', 'items': {'type': 'string'} diff --git a/lib/spack/spack/spec.py b/lib/spack/spack/spec.py index 8a4fc12861..743f84c3c8 100644 --- a/lib/spack/spack/spec.py +++ b/lib/spack/spack/spec.py @@ -951,7 +951,7 @@ class SpecBuildInterface(lang.ObjectWrapper): def __init__(self, spec, name, query_parameters): super(SpecBuildInterface, self).__init__(spec) - is_virtual = Spec.is_virtual(name) + is_virtual = spack.repo.path.is_virtual(name) self.last_query = QueryState( name=name, extra_parameters=query_parameters, @@ -1227,12 +1227,9 @@ class Spec(object): Possible idea: just use conventin and make virtual deps all caps, e.g., MPI vs mpi. """ - return Spec.is_virtual(self.name) - - @staticmethod - def is_virtual(name): - """Test if a name is virtual without requiring a Spec.""" - return (name is not None) and (not spack.repo.path.exists(name)) + # This method can be called while regenerating the provider index + # So we turn off using the index to detect virtuals + return spack.repo.path.is_virtual(self.name, use_index=False) @property def concrete(self): diff --git a/lib/spack/spack/tengine.py b/lib/spack/spack/tengine.py index 8581c8b41e..15268e682d 100644 --- a/lib/spack/spack/tengine.py +++ b/lib/spack/spack/tengine.py @@ -68,7 +68,8 @@ def make_environment(dirs=None): """Returns an configured environment for template rendering.""" if dirs is None: # Default directories where to search for templates - builtins = spack.config.get('config:template_dirs') + builtins = spack.config.get('config:template_dirs', + ['$spack/share/spack/templates']) extensions = spack.extensions.get_template_dirs() dirs = [canonicalize_path(d) for d in itertools.chain(builtins, extensions)] diff --git a/lib/spack/spack/test/cmd/clean.py b/lib/spack/spack/test/cmd/clean.py index 4df708cf50..dcaf0c916c 100644 --- a/lib/spack/spack/test/cmd/clean.py +++ b/lib/spack/spack/test/cmd/clean.py @@ -15,44 +15,51 @@ clean = spack.main.SpackCommand('clean') @pytest.fixture() def mock_calls_for_clean(monkeypatch): + counts = {} + class Counter(object): - def __init__(self): - self.call_count = 0 + def __init__(self, name): + self.name = name + counts[name] = 0 def __call__(self, *args, **kwargs): - self.call_count += 1 + counts[self.name] += 1 - monkeypatch.setattr(spack.package.PackageBase, 'do_clean', Counter()) - monkeypatch.setattr(spack.stage, 'purge', Counter()) + monkeypatch.setattr(spack.package.PackageBase, 'do_clean', + Counter('package')) + monkeypatch.setattr(spack.stage, 'purge', Counter('stages')) monkeypatch.setattr( - spack.caches.fetch_cache, 'destroy', Counter(), raising=False) + spack.caches.fetch_cache, 'destroy', Counter('downloads'), + raising=False) monkeypatch.setattr( - spack.caches.misc_cache, 'destroy', Counter()) + spack.caches.misc_cache, 'destroy', Counter('caches')) monkeypatch.setattr( - spack.installer, 'clear_failures', Counter()) + spack.installer, 'clear_failures', Counter('failures')) + + yield counts + + +all_effects = ['stages', 'downloads', 'caches', 'failures'] @pytest.mark.usefixtures( - 'mock_packages', 'config', 'mock_calls_for_clean' + 'mock_packages', 'config' ) -@pytest.mark.parametrize('command_line,counters', [ - ('mpileaks', [1, 0, 0, 0, 0]), - ('-s', [0, 1, 0, 0, 0]), - ('-sd', [0, 1, 1, 0, 0]), - ('-m', [0, 0, 0, 1, 0]), - ('-f', [0, 0, 0, 0, 1]), - ('-a', [0, 1, 1, 1, 1]), - ('', [0, 0, 0, 0, 0]), +@pytest.mark.parametrize('command_line,effects', [ + ('mpileaks', ['package']), + ('-s', ['stages']), + ('-sd', ['stages', 'downloads']), + ('-m', ['caches']), + ('-f', ['failures']), + ('-a', all_effects), + ('', []), ]) -def test_function_calls(command_line, counters): +def test_function_calls(command_line, effects, mock_calls_for_clean): # Call the command with the supplied command line clean(command_line) # Assert that we called the expected functions the correct # number of times - assert spack.package.PackageBase.do_clean.call_count == counters[0] - assert spack.stage.purge.call_count == counters[1] - assert spack.caches.fetch_cache.destroy.call_count == counters[2] - assert spack.caches.misc_cache.destroy.call_count == counters[3] - assert spack.installer.clear_failures.call_count == counters[4] + for name in ['package'] + all_effects: + assert mock_calls_for_clean[name] == (1 if name in effects else 0) diff --git a/lib/spack/spack/test/cmd/mirror.py b/lib/spack/spack/test/cmd/mirror.py index f6fe0b24dd..0957624cba 100644 --- a/lib/spack/spack/test/cmd/mirror.py +++ b/lib/spack/spack/test/cmd/mirror.py @@ -102,7 +102,7 @@ class MockMirrorArgs(object): self.exclude_specs = exclude_specs -def test_exclude_specs(mock_packages): +def test_exclude_specs(mock_packages, config): args = MockMirrorArgs( specs=['mpich'], versions_per_spec='all', @@ -117,7 +117,7 @@ def test_exclude_specs(mock_packages): assert (not expected_exclude & set(mirror_specs)) -def test_exclude_file(mock_packages, tmpdir): +def test_exclude_file(mock_packages, tmpdir, config): exclude_path = os.path.join(str(tmpdir), 'test-exclude.txt') with open(exclude_path, 'w') as exclude_file: exclude_file.write("""\ diff --git a/lib/spack/spack/test/cmd/pkg.py b/lib/spack/spack/test/cmd/pkg.py index af634beffc..ca9e3a1a3e 100644 --- a/lib/spack/spack/test/cmd/pkg.py +++ b/lib/spack/spack/test/cmd/pkg.py @@ -62,9 +62,9 @@ def mock_pkg_git_repo(tmpdir_factory): mkdirp('pkg-a', 'pkg-b', 'pkg-c') with open('pkg-a/package.py', 'w') as f: f.write(pkg_template.format(name='PkgA')) - with open('pkg-c/package.py', 'w') as f: - f.write(pkg_template.format(name='PkgB')) with open('pkg-b/package.py', 'w') as f: + f.write(pkg_template.format(name='PkgB')) + with open('pkg-c/package.py', 'w') as f: f.write(pkg_template.format(name='PkgC')) git('add', 'pkg-a', 'pkg-b', 'pkg-c') git('-c', 'commit.gpgsign=false', 'commit', @@ -128,6 +128,8 @@ def test_pkg_add(mock_pkg_git_repo): git('status', '--short', output=str)) finally: shutil.rmtree('pkg-e') + # Removing a package mid-run disrupts Spack's caching + spack.repo.path.repos[0]._fast_package_checker.invalidate() with pytest.raises(spack.main.SpackCommandError): pkg('add', 'does-not-exist') diff --git a/lib/spack/spack/test/cmd/test.py b/lib/spack/spack/test/cmd/test.py index a9ef735afe..4163853274 100644 --- a/lib/spack/spack/test/cmd/test.py +++ b/lib/spack/spack/test/cmd/test.py @@ -3,93 +3,181 @@ # # SPDX-License-Identifier: (Apache-2.0 OR MIT) +import argparse +import os + +import pytest + +import spack.config +import spack.package +import spack.cmd.install from spack.main import SpackCommand +install = SpackCommand('install') spack_test = SpackCommand('test') -cmd_test_py = 'lib/spack/spack/test/cmd/test.py' - - -def test_list(): - output = spack_test('--list') - assert "test.py" in output - assert "spec_semantics.py" in output - assert "test_list" not in output - - -def test_list_with_pytest_arg(): - output = spack_test('--list', cmd_test_py) - assert output.strip() == cmd_test_py - - -def test_list_with_keywords(): - output = spack_test('--list', '-k', 'cmd/test.py') - assert output.strip() == cmd_test_py - - -def test_list_long(capsys): - with capsys.disabled(): - output = spack_test('--list-long') - assert "test.py::\n" in output - assert "test_list" in output - assert "test_list_with_pytest_arg" in output - assert "test_list_with_keywords" in output - assert "test_list_long" in output - assert "test_list_long_with_pytest_arg" in output - assert "test_list_names" in output - assert "test_list_names_with_pytest_arg" in output - - assert "spec_dag.py::\n" in output - assert 'test_installed_deps' in output - assert 'test_test_deptype' in output - - -def test_list_long_with_pytest_arg(capsys): - with capsys.disabled(): - output = spack_test('--list-long', cmd_test_py) - assert "test.py::\n" in output - assert "test_list" in output - assert "test_list_with_pytest_arg" in output - assert "test_list_with_keywords" in output - assert "test_list_long" in output - assert "test_list_long_with_pytest_arg" in output - assert "test_list_names" in output - assert "test_list_names_with_pytest_arg" in output - - assert "spec_dag.py::\n" not in output - assert 'test_installed_deps' not in output - assert 'test_test_deptype' not in output - - -def test_list_names(): - output = spack_test('--list-names') - assert "test.py::test_list\n" in output - assert "test.py::test_list_with_pytest_arg\n" in output - assert "test.py::test_list_with_keywords\n" in output - assert "test.py::test_list_long\n" in output - assert "test.py::test_list_long_with_pytest_arg\n" in output - assert "test.py::test_list_names\n" in output - assert "test.py::test_list_names_with_pytest_arg\n" in output - - assert "spec_dag.py::test_installed_deps\n" in output - assert 'spec_dag.py::test_test_deptype\n' in output - - -def test_list_names_with_pytest_arg(): - output = spack_test('--list-names', cmd_test_py) - assert "test.py::test_list\n" in output - assert "test.py::test_list_with_pytest_arg\n" in output - assert "test.py::test_list_with_keywords\n" in output - assert "test.py::test_list_long\n" in output - assert "test.py::test_list_long_with_pytest_arg\n" in output - assert "test.py::test_list_names\n" in output - assert "test.py::test_list_names_with_pytest_arg\n" in output - - assert "spec_dag.py::test_installed_deps\n" not in output - assert 'spec_dag.py::test_test_deptype\n' not in output - - -def test_pytest_help(): - output = spack_test('--pytest-help') - assert "-k EXPRESSION" in output - assert "pytest-warnings:" in output - assert "--collect-only" in output + + +def test_test_package_not_installed( + tmpdir, mock_packages, mock_archive, mock_fetch, config, + install_mockery_mutable_config, mock_test_stage): + + output = spack_test('run', 'libdwarf') + + assert "No installed packages match spec libdwarf" in output + + +@pytest.mark.parametrize('arguments,expected', [ + (['run'], spack.config.get('config:dirty')), # default from config file + (['run', '--clean'], False), + (['run', '--dirty'], True), +]) +def test_test_dirty_flag(arguments, expected): + parser = argparse.ArgumentParser() + spack.cmd.test.setup_parser(parser) + args = parser.parse_args(arguments) + assert args.dirty == expected + + +def test_test_output(mock_test_stage, mock_packages, mock_archive, mock_fetch, + install_mockery_mutable_config): + """Ensure output printed from pkgs is captured by output redirection.""" + install('printing-package') + spack_test('run', 'printing-package') + + stage_files = os.listdir(mock_test_stage) + assert len(stage_files) == 1 + + # Grab test stage directory contents + testdir = os.path.join(mock_test_stage, stage_files[0]) + testdir_files = os.listdir(testdir) + + # Grab the output from the test log + testlog = list(filter(lambda x: x.endswith('out.txt') and + x != 'results.txt', testdir_files)) + outfile = os.path.join(testdir, testlog[0]) + with open(outfile, 'r') as f: + output = f.read() + assert "BEFORE TEST" in output + assert "true: expect command status in [" in output + assert "AFTER TEST" in output + assert "FAILED" not in output + + +def test_test_output_on_error( + mock_packages, mock_archive, mock_fetch, install_mockery_mutable_config, + capfd, mock_test_stage +): + install('test-error') + # capfd interferes with Spack's capturing + with capfd.disabled(): + out = spack_test('run', 'test-error', fail_on_error=False) + + assert "TestFailure" in out + assert "Command exited with status 1" in out + + +def test_test_output_on_failure( + mock_packages, mock_archive, mock_fetch, install_mockery_mutable_config, + capfd, mock_test_stage +): + install('test-fail') + with capfd.disabled(): + out = spack_test('run', 'test-fail', fail_on_error=False) + + assert "Expected 'not in the output' to match output of `true`" in out + assert "TestFailure" in out + + +def test_show_log_on_error( + mock_packages, mock_archive, mock_fetch, + install_mockery_mutable_config, capfd, mock_test_stage +): + """Make sure spack prints location of test log on failure.""" + install('test-error') + with capfd.disabled(): + out = spack_test('run', 'test-error', fail_on_error=False) + + assert 'See test log' in out + assert mock_test_stage in out + + +@pytest.mark.usefixtures( + 'mock_packages', 'mock_archive', 'mock_fetch', + 'install_mockery_mutable_config' +) +@pytest.mark.parametrize('pkg_name,msgs', [ + ('test-error', ['FAILED: Command exited', 'TestFailure']), + ('test-fail', ['FAILED: Expected', 'TestFailure']) +]) +def test_junit_output_with_failures(tmpdir, mock_test_stage, pkg_name, msgs): + install(pkg_name) + with tmpdir.as_cwd(): + spack_test('run', + '--log-format=junit', '--log-file=test.xml', + pkg_name) + + files = tmpdir.listdir() + filename = tmpdir.join('test.xml') + assert filename in files + + content = filename.open().read() + + # Count failures and errors correctly + assert 'tests="1"' in content + assert 'failures="1"' in content + assert 'errors="0"' in content + + # We want to have both stdout and stderr + assert '<system-out>' in content + for msg in msgs: + assert msg in content + + +def test_cdash_output_test_error( + tmpdir, mock_fetch, install_mockery_mutable_config, mock_packages, + mock_archive, mock_test_stage, capfd): + install('test-error') + with tmpdir.as_cwd(): + spack_test('run', + '--log-format=cdash', + '--log-file=cdash_reports', + 'test-error') + report_dir = tmpdir.join('cdash_reports') + print(tmpdir.listdir()) + assert report_dir in tmpdir.listdir() + report_file = report_dir.join('test-error_Test.xml') + assert report_file in report_dir.listdir() + content = report_file.open().read() + assert 'FAILED: Command exited with status 1' in content + + +def test_cdash_upload_clean_test( + tmpdir, mock_fetch, install_mockery_mutable_config, mock_packages, + mock_archive, mock_test_stage): + install('printing-package') + with tmpdir.as_cwd(): + spack_test('run', + '--log-file=cdash_reports', + '--log-format=cdash', + 'printing-package') + report_dir = tmpdir.join('cdash_reports') + assert report_dir in tmpdir.listdir() + report_file = report_dir.join('printing-package_Test.xml') + assert report_file in report_dir.listdir() + content = report_file.open().read() + assert '</Test>' in content + assert '<Text>' not in content + + +def test_test_help_does_not_show_cdash_options(mock_test_stage, capsys): + """Make sure `spack test --help` does not describe CDash arguments""" + with pytest.raises(SystemExit): + spack_test('run', '--help') + captured = capsys.readouterr() + assert 'CDash URL' not in captured.out + + +def test_test_help_cdash(mock_test_stage): + """Make sure `spack test --help-cdash` describes CDash arguments""" + out = spack_test('run', '--help-cdash') + assert 'CDash URL' in out diff --git a/lib/spack/spack/test/cmd/unit_test.py b/lib/spack/spack/test/cmd/unit_test.py new file mode 100644 index 0000000000..c5b8eb765e --- /dev/null +++ b/lib/spack/spack/test/cmd/unit_test.py @@ -0,0 +1,96 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +from spack.main import SpackCommand + +spack_test = SpackCommand('unit-test') +cmd_test_py = 'lib/spack/spack/test/cmd/unit_test.py' + + +def test_list(): + output = spack_test('--list') + assert "unit_test.py" in output + assert "spec_semantics.py" in output + assert "test_list" not in output + + +def test_list_with_pytest_arg(): + output = spack_test('--list', cmd_test_py) + assert output.strip() == cmd_test_py + + +def test_list_with_keywords(): + output = spack_test('--list', '-k', 'cmd/unit_test.py') + assert output.strip() == cmd_test_py + + +def test_list_long(capsys): + with capsys.disabled(): + output = spack_test('--list-long') + assert "unit_test.py::\n" in output + assert "test_list" in output + assert "test_list_with_pytest_arg" in output + assert "test_list_with_keywords" in output + assert "test_list_long" in output + assert "test_list_long_with_pytest_arg" in output + assert "test_list_names" in output + assert "test_list_names_with_pytest_arg" in output + + assert "spec_dag.py::\n" in output + assert 'test_installed_deps' in output + assert 'test_test_deptype' in output + + +def test_list_long_with_pytest_arg(capsys): + with capsys.disabled(): + output = spack_test('--list-long', cmd_test_py) + print(output) + assert "unit_test.py::\n" in output + assert "test_list" in output + assert "test_list_with_pytest_arg" in output + assert "test_list_with_keywords" in output + assert "test_list_long" in output + assert "test_list_long_with_pytest_arg" in output + assert "test_list_names" in output + assert "test_list_names_with_pytest_arg" in output + + assert "spec_dag.py::\n" not in output + assert 'test_installed_deps' not in output + assert 'test_test_deptype' not in output + + +def test_list_names(): + output = spack_test('--list-names') + assert "unit_test.py::test_list\n" in output + assert "unit_test.py::test_list_with_pytest_arg\n" in output + assert "unit_test.py::test_list_with_keywords\n" in output + assert "unit_test.py::test_list_long\n" in output + assert "unit_test.py::test_list_long_with_pytest_arg\n" in output + assert "unit_test.py::test_list_names\n" in output + assert "unit_test.py::test_list_names_with_pytest_arg\n" in output + + assert "spec_dag.py::test_installed_deps\n" in output + assert 'spec_dag.py::test_test_deptype\n' in output + + +def test_list_names_with_pytest_arg(): + output = spack_test('--list-names', cmd_test_py) + assert "unit_test.py::test_list\n" in output + assert "unit_test.py::test_list_with_pytest_arg\n" in output + assert "unit_test.py::test_list_with_keywords\n" in output + assert "unit_test.py::test_list_long\n" in output + assert "unit_test.py::test_list_long_with_pytest_arg\n" in output + assert "unit_test.py::test_list_names\n" in output + assert "unit_test.py::test_list_names_with_pytest_arg\n" in output + + assert "spec_dag.py::test_installed_deps\n" not in output + assert 'spec_dag.py::test_test_deptype\n' not in output + + +def test_pytest_help(): + output = spack_test('--pytest-help') + assert "-k EXPRESSION" in output + assert "pytest-warnings:" in output + assert "--collect-only" in output diff --git a/lib/spack/spack/test/conftest.py b/lib/spack/spack/test/conftest.py index 527d6a3380..8615b14abe 100644 --- a/lib/spack/spack/test/conftest.py +++ b/lib/spack/spack/test/conftest.py @@ -1273,3 +1273,14 @@ def mock_executable(tmpdir): return str(f) return _factory + + +@pytest.fixture() +def mock_test_stage(mutable_config, tmpdir): + # NOTE: This fixture MUST be applied after any fixture that uses + # the config fixture under the hood + # No need to unset because we use mutable_config + tmp_stage = str(tmpdir.join('test_stage')) + mutable_config.set('config:test_stage', tmp_stage) + + yield tmp_stage diff --git a/lib/spack/spack/test/llnl/util/tty/__init__.py b/lib/spack/spack/test/llnl/util/tty/__init__.py new file mode 100644 index 0000000000..9f87532b85 --- /dev/null +++ b/lib/spack/spack/test/llnl/util/tty/__init__.py @@ -0,0 +1,4 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/mirror.py b/lib/spack/spack/test/mirror.py index 05cf46dc20..8d3ec2d07b 100644 --- a/lib/spack/spack/test/mirror.py +++ b/lib/spack/spack/test/mirror.py @@ -16,7 +16,7 @@ from spack.util.executable import which from llnl.util.filesystem import resolve_link_target_relative_to_the_link -pytestmark = pytest.mark.usefixtures('config', 'mutable_mock_repo') +pytestmark = pytest.mark.usefixtures('mutable_config', 'mutable_mock_repo') # paths in repos that shouldn't be in the mirror tarballs. exclude = ['.hg', '.git', '.svn'] @@ -97,7 +97,7 @@ def check_mirror(): # tarball assert not dcmp.right_only # and that all original files are present. - assert all(l in exclude for l in dcmp.left_only) + assert all(left in exclude for left in dcmp.left_only) def test_url_mirror(mock_archive): diff --git a/lib/spack/spack/test/package_class.py b/lib/spack/spack/test/package_class.py index d540ac663e..33e5eb1c0a 100644 --- a/lib/spack/spack/test/package_class.py +++ b/lib/spack/spack/test/package_class.py @@ -10,7 +10,12 @@ etc.). Only methods like ``possible_dependencies()`` that deal with the static DSL metadata for packages. """ +import os import pytest +import shutil + +import llnl.util.filesystem as fs + import spack.package import spack.repo @@ -119,3 +124,72 @@ def test_possible_dependencies_with_multiple_classes( }) assert expected == spack.package.possible_dependencies(*pkgs) + + +def setup_install_test(source_paths, install_test_root): + """ + Set up the install test by creating sources and install test roots. + + The convention used here is to create an empty file if the path name + ends with an extension otherwise, a directory is created. + """ + fs.mkdirp(install_test_root) + for path in source_paths: + if os.path.splitext(path)[1]: + fs.touchp(path) + else: + fs.mkdirp(path) + + +@pytest.mark.parametrize('spec,sources,extras,expect', [ + ('a', + ['example/a.c'], # Source(s) + ['example/a.c'], # Extra test source + ['example/a.c']), # Test install dir source(s) + ('b', + ['test/b.cpp', 'test/b.hpp', 'example/b.txt'], # Source(s) + ['test'], # Extra test source + ['test/b.cpp', 'test/b.hpp']), # Test install dir source + ('c', + ['examples/a.py', 'examples/b.py', 'examples/c.py', 'tests/d.py'], + ['examples/b.py', 'tests'], + ['examples/b.py', 'tests/d.py']), +]) +def test_cache_extra_sources(install_mockery, spec, sources, extras, expect): + """Test the package's cache extra test sources helper function.""" + + pkg = spack.repo.get(spec) + pkg.spec.concretize() + source_path = pkg.stage.source_path + + srcs = [fs.join_path(source_path, s) for s in sources] + setup_install_test(srcs, pkg.install_test_root) + + emsg_dir = 'Expected {0} to be a directory' + emsg_file = 'Expected {0} to be a file' + for s in srcs: + assert os.path.exists(s), 'Expected {0} to exist'.format(s) + if os.path.splitext(s)[1]: + assert os.path.isfile(s), emsg_file.format(s) + else: + assert os.path.isdir(s), emsg_dir.format(s) + + pkg.cache_extra_test_sources(extras) + + src_dests = [fs.join_path(pkg.install_test_root, s) for s in sources] + exp_dests = [fs.join_path(pkg.install_test_root, e) for e in expect] + poss_dests = set(src_dests) | set(exp_dests) + + msg = 'Expected {0} to{1} exist' + for pd in poss_dests: + if pd in exp_dests: + assert os.path.exists(pd), msg.format(pd, '') + if os.path.splitext(pd)[1]: + assert os.path.isfile(pd), emsg_file.format(pd) + else: + assert os.path.isdir(pd), emsg_dir.format(pd) + else: + assert not os.path.exists(pd), msg.format(pd, ' not') + + # Perform a little cleanup + shutil.rmtree(os.path.dirname(source_path)) diff --git a/lib/spack/spack/test/test_suite.py b/lib/spack/spack/test/test_suite.py new file mode 100644 index 0000000000..1ec5106182 --- /dev/null +++ b/lib/spack/spack/test/test_suite.py @@ -0,0 +1,53 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) +import os +import spack.install_test +import spack.spec + + +def test_test_log_pathname(mock_packages, config): + """Ensure test log path is reasonable.""" + spec = spack.spec.Spec('libdwarf').concretized() + + test_name = 'test_name' + + test_suite = spack.install_test.TestSuite([spec], test_name) + logfile = test_suite.log_file_for_spec(spec) + + assert test_suite.stage in logfile + assert test_suite.test_log_name(spec) in logfile + + +def test_test_ensure_stage(mock_test_stage): + """Make sure test stage directory is properly set up.""" + spec = spack.spec.Spec('libdwarf').concretized() + + test_name = 'test_name' + + test_suite = spack.install_test.TestSuite([spec], test_name) + test_suite.ensure_stage() + + assert os.path.isdir(test_suite.stage) + assert mock_test_stage in test_suite.stage + + +def test_write_test_result(mock_packages, mock_test_stage): + """Ensure test results written to a results file.""" + spec = spack.spec.Spec('libdwarf').concretized() + result = 'TEST' + test_name = 'write-test' + + test_suite = spack.install_test.TestSuite([spec], test_name) + test_suite.ensure_stage() + results_file = test_suite.results_file + test_suite.write_test_result(spec, result) + + with open(results_file, 'r') as f: + lines = f.readlines() + assert len(lines) == 1 + + msg = lines[0] + assert result in msg + assert spec.name in msg diff --git a/lib/spack/spack/util/executable.py b/lib/spack/spack/util/executable.py index 097da3337e..614bc1725a 100644 --- a/lib/spack/spack/util/executable.py +++ b/lib/spack/spack/util/executable.py @@ -2,7 +2,7 @@ # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) - +import sys import os import re import shlex @@ -98,6 +98,9 @@ class Executable(object): If both ``output`` and ``error`` are set to ``str``, then one string is returned containing output concatenated with error. Not valid for ``input`` + * ``str.split``, as in the ``split`` method of the Python string type. + Behaves the same as ``str``, except that value is also written to + ``stdout`` or ``stderr``. By default, the subprocess inherits the parent's file descriptors. @@ -132,7 +135,7 @@ class Executable(object): def streamify(arg, mode): if isinstance(arg, string_types): return open(arg, mode), True - elif arg is str: + elif arg in (str, str.split): return subprocess.PIPE, False else: return arg, False @@ -168,12 +171,18 @@ class Executable(object): out, err = proc.communicate() result = None - if output is str or error is str: + if output in (str, str.split) or error in (str, str.split): result = '' - if output is str: - result += text_type(out.decode('utf-8')) - if error is str: - result += text_type(err.decode('utf-8')) + if output in (str, str.split): + outstr = text_type(out.decode('utf-8')) + result += outstr + if output is str.split: + sys.stdout.write(outstr) + if error in (str, str.split): + errstr = text_type(err.decode('utf-8')) + result += errstr + if error is str.split: + sys.stderr.write(errstr) rc = self.returncode = proc.returncode if fail_on_error and rc != 0 and (rc not in ignore_errors): diff --git a/lib/spack/spack/util/mock_package.py b/lib/spack/spack/util/mock_package.py index e855aae015..4751f5af7e 100644 --- a/lib/spack/spack/util/mock_package.py +++ b/lib/spack/spack/util/mock_package.py @@ -21,6 +21,8 @@ class MockPackageBase(object): Use ``MockPackageMultiRepo.add_package()`` to create new instances. """ + virtual = False + def __init__(self, dependencies, dependency_types, conditions=None, versions=None): """Instantiate a new MockPackageBase. @@ -92,7 +94,7 @@ class MockPackageMultiRepo(object): def exists(self, name): return name in self.spec_to_pkg - def is_virtual(self, name): + def is_virtual(self, name, use_index=True): return False def repo_for_pkg(self, name): diff --git a/share/spack/qa/completion-test.sh b/share/spack/qa/completion-test.sh index 5b326b4a6d..59a5181b98 100755 --- a/share/spack/qa/completion-test.sh +++ b/share/spack/qa/completion-test.sh @@ -56,7 +56,7 @@ contains 'hdf5' _spack_completions spack -d install --jobs 8 '' contains 'hdf5' _spack_completions spack install -v '' # XFAIL: Fails for Python 2.6 because pkg_resources not found? -#contains 'compilers.py' _spack_completions spack test '' +#contains 'compilers.py' _spack_completions spack unit-test '' title 'Testing debugging functions' diff --git a/share/spack/qa/run-unit-tests b/share/spack/qa/run-unit-tests index c529f8297e..ec8aaf76b9 100755 --- a/share/spack/qa/run-unit-tests +++ b/share/spack/qa/run-unit-tests @@ -42,4 +42,4 @@ spack -p --lines 20 spec mpileaks%gcc ^elfutils@0.170 #----------------------------------------------------------- # Run unit tests with code coverage #----------------------------------------------------------- -$coverage_run $(which spack) test -x --verbose +$coverage_run $(which spack) unit-test -x --verbose diff --git a/share/spack/spack-completion.bash b/share/spack/spack-completion.bash index 7d54414397..969a0898fe 100755 --- a/share/spack/spack-completion.bash +++ b/share/spack/spack-completion.bash @@ -320,7 +320,7 @@ _spack() { then SPACK_COMPREPLY="-h --help -H --all-help --color -C --config-scope -d --debug --timestamp --pdb -e --env -D --env-dir -E --no-env --use-env-repo -k --insecure -l --enable-locks -L --disable-locks -m --mock -p --profile --sorted-profile --lines -v --verbose --stacktrace -V --version --print-shell-vars" else - SPACK_COMPREPLY="activate add arch blame build-env buildcache cd checksum ci clean clone commands compiler compilers concretize config containerize create deactivate debug dependencies dependents deprecate dev-build develop docs edit env extensions external fetch find flake8 gc gpg graph help info install license list load location log-parse maintainers mirror module patch pkg providers pydoc python reindex remove rm repo resource restage setup solve spec stage test tutorial undevelop uninstall unload url verify versions view" + SPACK_COMPREPLY="activate add arch blame build-env buildcache cd checksum ci clean clone commands compiler compilers concretize config containerize create deactivate debug dependencies dependents deprecate dev-build develop docs edit env extensions external fetch find flake8 gc gpg graph help info install license list load location log-parse maintainers mirror module patch pkg providers pydoc python reindex remove rm repo resource restage setup solve spec stage test test-env tutorial undevelop uninstall unit-test unload url verify versions view" fi } @@ -1020,7 +1020,7 @@ _spack_info() { _spack_install() { if $list_options then - SPACK_COMPREPLY="-h --help --only -u --until -j --jobs --overwrite --fail-fast --keep-prefix --keep-stage --dont-restage --use-cache --no-cache --cache-only --no-check-signature --require-full-hash-match --show-log-on-error --source -n --no-checksum -v --verbose --fake --only-concrete -f --file --clean --dirty --test --run-tests --log-format --log-file --help-cdash -y --yes-to-all --cdash-upload-url --cdash-build --cdash-site --cdash-track --cdash-buildstamp" + SPACK_COMPREPLY="-h --help --only -u --until -j --jobs --overwrite --fail-fast --keep-prefix --keep-stage --dont-restage --use-cache --no-cache --cache-only --no-check-signature --require-full-hash-match --show-log-on-error --source -n --no-checksum -v --verbose --fake --only-concrete -f --file --clean --dirty --test --run-tests --log-format --log-file --help-cdash --cdash-upload-url --cdash-build --cdash-site --cdash-track --cdash-buildstamp -y --yes-to-all" else _all_packages fi @@ -1046,7 +1046,7 @@ _spack_license_verify() { _spack_list() { if $list_options then - SPACK_COMPREPLY="-h --help -d --search-description --format --update -t --tags" + SPACK_COMPREPLY="-h --help -d --search-description --format --update -v --virtuals -t --tags" else _all_packages fi @@ -1494,9 +1494,67 @@ _spack_stage() { _spack_test() { if $list_options then - SPACK_COMPREPLY="-h --help -H --pytest-help -l --list -L --list-long -N --list-names --extension -s -k --showlocals" + SPACK_COMPREPLY="-h --help" else - _tests + SPACK_COMPREPLY="run list find status results remove" + fi +} + +_spack_test_run() { + if $list_options + then + SPACK_COMPREPLY="-h --help --alias --fail-fast --fail-first --keep-stage --log-format --log-file --cdash-upload-url --cdash-build --cdash-site --cdash-track --cdash-buildstamp --help-cdash --clean --dirty" + else + _installed_packages + fi +} + +_spack_test_list() { + SPACK_COMPREPLY="-h --help" +} + +_spack_test_find() { + if $list_options + then + SPACK_COMPREPLY="-h --help" + else + _all_packages + fi +} + +_spack_test_status() { + if $list_options + then + SPACK_COMPREPLY="-h --help" + else + SPACK_COMPREPLY="" + fi +} + +_spack_test_results() { + if $list_options + then + SPACK_COMPREPLY="-h --help -l --logs -f --failed" + else + SPACK_COMPREPLY="" + fi +} + +_spack_test_remove() { + if $list_options + then + SPACK_COMPREPLY="-h --help -y --yes-to-all" + else + SPACK_COMPREPLY="" + fi +} + +_spack_test_env() { + if $list_options + then + SPACK_COMPREPLY="-h --help --clean --dirty --dump --pickle" + else + _all_packages fi } @@ -1522,6 +1580,15 @@ _spack_uninstall() { fi } +_spack_unit_test() { + if $list_options + then + SPACK_COMPREPLY="-h --help -H --pytest-help -l --list -L --list-long -N --list-names --extension -s -k --showlocals" + else + _tests + fi +} + _spack_unload() { if $list_options then diff --git a/share/spack/templates/reports/cdash/Test.xml b/share/spack/templates/reports/cdash/Test.xml new file mode 100644 index 0000000000..6aeed4e263 --- /dev/null +++ b/share/spack/templates/reports/cdash/Test.xml @@ -0,0 +1,27 @@ + <Test> + <StartTestTime>{{ test.starttime }}</StartTestTime> + <TestCommand>{{ install_command }}</TestCommand> +{% for warning in test.warnings %} + <Warning> + <TestLogLine>{{ warning.line_no }}</TestLogLine> + <Text>{{ warning.text }}</Text> + <SourceFile>{{ warning.source_file }}</SourceFile> + <SourceLineNumber>{{ warning.source_line_no }}</SourceLineNumber> + <PreContext>{{ warning.pre_context }}</PreContext> + <PostContext>{{ warning.post_context }}</PostContext> + </Warning> +{% endfor %} +{% for error in test.errors %} + <Error> + <TestLogLine>{{ error.line_no }}</TestLogLine> + <Text>{{ error.text }}</Text> + <SourceFile>{{ error.source_file }}</SourceFile> + <SourceLineNumber>{{ error.source_line_no }}</SourceLineNumber> + <PreContext>{{ error.pre_context }}</PreContext> + <PostContext>{{ error.post_context }}</PostContext> + </Error> +{% endfor %} + <EndTestTime>{{ test.endtime }}</EndTestTime> + <ElapsedMinutes>0</ElapsedMinutes> + </Test> +</Site> diff --git a/var/spack/repos/builtin.mock/packages/printing-package/package.py b/var/spack/repos/builtin.mock/packages/printing-package/package.py index 1d4d32d54d..096a49d211 100644 --- a/var/spack/repos/builtin.mock/packages/printing-package/package.py +++ b/var/spack/repos/builtin.mock/packages/printing-package/package.py @@ -24,3 +24,8 @@ class PrintingPackage(Package): make('install') print("AFTER INSTALL") + + def test(self): + print("BEFORE TEST") + self.run_test('true') # run /bin/true + print("AFTER TEST") diff --git a/var/spack/repos/builtin.mock/packages/test-error/package.py b/var/spack/repos/builtin.mock/packages/test-error/package.py new file mode 100644 index 0000000000..ce36ee7ca3 --- /dev/null +++ b/var/spack/repos/builtin.mock/packages/test-error/package.py @@ -0,0 +1,21 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +from spack import * + + +class TestError(Package): + """This package has a test method that fails in a subprocess.""" + + homepage = "http://www.example.com/test-failure" + url = "http://www.test-failure.test/test-failure-1.0.tar.gz" + + version('1.0', 'foobarbaz') + + def install(self, spec, prefix): + mkdirp(prefix.bin) + + def test(self): + self.run_test('false') diff --git a/var/spack/repos/builtin.mock/packages/test-fail/package.py b/var/spack/repos/builtin.mock/packages/test-fail/package.py new file mode 100644 index 0000000000..6587ef2bb9 --- /dev/null +++ b/var/spack/repos/builtin.mock/packages/test-fail/package.py @@ -0,0 +1,21 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +from spack import * + + +class TestFail(Package): + """This package has a test method that fails in a subprocess.""" + + homepage = "http://www.example.com/test-failure" + url = "http://www.test-failure.test/test-failure-1.0.tar.gz" + + version('1.0', 'foobarbaz') + + def install(self, spec, prefix): + mkdirp(prefix.bin) + + def test(self): + self.run_test('true', expected=['not in the output']) diff --git a/var/spack/repos/builtin/packages/bazel/package.py b/var/spack/repos/builtin/packages/bazel/package.py index 66c039cafb..ad1239529d 100644 --- a/var/spack/repos/builtin/packages/bazel/package.py +++ b/var/spack/repos/builtin/packages/bazel/package.py @@ -184,7 +184,7 @@ class Bazel(Package): @run_after('install') @on_package_attributes(run_tests=True) - def test(self): + def install_test(self): # https://github.com/Homebrew/homebrew-core/blob/master/Formula/bazel.rb # Bazel does not work properly on NFS, switch to /tmp diff --git a/var/spack/repos/builtin/packages/berkeley-db/package.py b/var/spack/repos/builtin/packages/berkeley-db/package.py index e72d823cb5..0385de81a1 100644 --- a/var/spack/repos/builtin/packages/berkeley-db/package.py +++ b/var/spack/repos/builtin/packages/berkeley-db/package.py @@ -3,8 +3,6 @@ # # SPDX-License-Identifier: (Apache-2.0 OR MIT) -from spack import * - class BerkeleyDb(AutotoolsPackage): """Oracle Berkeley DB""" @@ -47,3 +45,15 @@ class BerkeleyDb(AutotoolsPackage): config_args.append('--disable-atomicsupport') return config_args + + def test(self): + """Perform smoke tests on the installed package binaries.""" + exes = [ + 'db_checkpoint', 'db_deadlock', 'db_dump', 'db_load', + 'db_printlog', 'db_stat', 'db_upgrade', 'db_verify' + ] + for exe in exes: + reason = 'test version of {0} is {1}'.format(exe, + self.spec.version) + self.run_test(exe, ['-V'], [self.spec.version.string], + installed=True, purpose=reason, skip_missing=True) diff --git a/var/spack/repos/builtin/packages/binutils/package.py b/var/spack/repos/builtin/packages/binutils/package.py index e2ddfe8c1e..f79015cf6b 100644 --- a/var/spack/repos/builtin/packages/binutils/package.py +++ b/var/spack/repos/builtin/packages/binutils/package.py @@ -129,3 +129,29 @@ class Binutils(AutotoolsPackage, GNUMirrorPackage): if self.spec.satisfies('@:2.34 %gcc@10:'): flags.append('-fcommon') return (flags, None, None) + + def test(self): + spec_vers = str(self.spec.version) + + checks = { + 'ar': spec_vers, + 'c++filt': spec_vers, + 'coffdump': spec_vers, + 'dlltool': spec_vers, + 'elfedit': spec_vers, + 'gprof': spec_vers, + 'ld': spec_vers, + 'nm': spec_vers, + 'objdump': spec_vers, + 'ranlib': spec_vers, + 'readelf': spec_vers, + 'size': spec_vers, + 'strings': spec_vers, + } + + for exe in checks: + expected = checks[exe] + reason = 'test: ensuring version of {0} is {1}' \ + .format(exe, expected) + self.run_test(exe, '--version', expected, installed=True, + purpose=reason, skip_missing=True) diff --git a/var/spack/repos/builtin/packages/c/package.py b/var/spack/repos/builtin/packages/c/package.py new file mode 100644 index 0000000000..72a3343aa1 --- /dev/null +++ b/var/spack/repos/builtin/packages/c/package.py @@ -0,0 +1,27 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +import os + + +class C(Package): + """Virtual package for C compilers.""" + homepage = 'http://open-std.org/JTC1/SC22/WG14/www/standards' + virtual = True + + def test(self): + test_source = self.test_suite.current_test_data_dir + + for test in os.listdir(test_source): + filepath = test_source.join(test) + exe_name = '%s.exe' % test + + cc_exe = os.environ['CC'] + cc_opts = ['-o', exe_name, filepath] + compiled = self.run_test(cc_exe, options=cc_opts, installed=True) + + if compiled: + expected = ['Hello world', 'YES!'] + self.run_test(exe_name, expected=expected) diff --git a/var/spack/repos/builtin/packages/c/test/hello.c b/var/spack/repos/builtin/packages/c/test/hello.c new file mode 100644 index 0000000000..de950e1e88 --- /dev/null +++ b/var/spack/repos/builtin/packages/c/test/hello.c @@ -0,0 +1,7 @@ +#include <stdio.h> +int main() +{ + printf ("Hello world from C!\n"); + printf ("YES!"); + return 0; +} diff --git a/var/spack/repos/builtin/packages/cantera/package.py b/var/spack/repos/builtin/packages/cantera/package.py index 773cc6d8cf..9846fff48d 100644 --- a/var/spack/repos/builtin/packages/cantera/package.py +++ b/var/spack/repos/builtin/packages/cantera/package.py @@ -146,7 +146,7 @@ class Cantera(SConsPackage): return args - def test(self): + def build_test(self): if '+python' in self.spec: # Tests will always fail if Python dependencies aren't built # In addition, 3 of the tests fail when run in parallel diff --git a/var/spack/repos/builtin/packages/cmake/package.py b/var/spack/repos/builtin/packages/cmake/package.py index 97168c05f1..a3bad699e1 100644 --- a/var/spack/repos/builtin/packages/cmake/package.py +++ b/var/spack/repos/builtin/packages/cmake/package.py @@ -2,6 +2,7 @@ # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) + import re @@ -250,7 +251,7 @@ class Cmake(Package): @run_after('build') @on_package_attributes(run_tests=True) - def test(self): + def build_test(self): # Some tests fail, takes forever make('test') @@ -262,3 +263,12 @@ class Cmake(Package): filter_file('mpcc_r)', 'mpcc_r mpifcc)', f, string=True) filter_file('mpc++_r)', 'mpc++_r mpiFCC)', f, string=True) filter_file('mpifc)', 'mpifc mpifrt)', f, string=True) + + def test(self): + """Perform smoke tests on the installed package.""" + spec_vers_str = 'version {0}'.format(self.spec.version) + + for exe in ['ccmake', 'cmake', 'cpack', 'ctest']: + reason = 'test version of {0} is {1}'.format(exe, spec_vers_str) + self.run_test(exe, ['--version'], [spec_vers_str], + installed=True, purpose=reason, skip_missing=True) diff --git a/var/spack/repos/builtin/packages/conduit/package.py b/var/spack/repos/builtin/packages/conduit/package.py index 7b6edf21f9..57d49da70e 100644 --- a/var/spack/repos/builtin/packages/conduit/package.py +++ b/var/spack/repos/builtin/packages/conduit/package.py @@ -217,7 +217,7 @@ class Conduit(Package): @run_after('build') @on_package_attributes(run_tests=True) - def test(self): + def build_test(self): with working_dir('spack-build'): print("Running Conduit Unit Tests...") make("test") diff --git a/var/spack/repos/builtin/packages/cxx/package.py b/var/spack/repos/builtin/packages/cxx/package.py new file mode 100644 index 0000000000..0be36c3ae5 --- /dev/null +++ b/var/spack/repos/builtin/packages/cxx/package.py @@ -0,0 +1,38 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +import os + + +class Cxx(Package): + """Virtual package for the C++ language.""" + homepage = 'https://isocpp.org/std/the-standard' + virtual = True + + def test(self): + test_source = self.test_suite.current_test_data_dir + + for test in os.listdir(test_source): + filepath = os.path.join(test_source, test) + exe_name = '%s.exe' % test + + cxx_exe = os.environ['CXX'] + + # standard options + # Hack to get compiler attributes + # TODO: remove this when compilers are dependencies + c_name = clang if self.spec.satisfies('llvm+clang') else self.name + c_spec = spack.spec.CompilerSpec(c_name, self.spec.version) + c_cls = spack.compilers.class_for_compiler_name(c_name) + compiler = c_cls(c_spec, None, None, ['fakecc', 'fakecxx']) + + cxx_opts = [compiler.cxx11_flag] if 'c++11' in test else [] + + cxx_opts += ['-o', exe_name, filepath] + compiled = self.run_test(cxx_exe, options=cxx_opts, installed=True) + + if compiled: + expected = ['Hello world', 'YES!'] + self.run_test(exe_name, expected=expected) diff --git a/var/spack/repos/builtin/packages/cxx/test/hello.c++ b/var/spack/repos/builtin/packages/cxx/test/hello.c++ new file mode 100644 index 0000000000..f0ad7caffb --- /dev/null +++ b/var/spack/repos/builtin/packages/cxx/test/hello.c++ @@ -0,0 +1,9 @@ +#include <stdio.h> + +int main() +{ + printf ("Hello world from C++\n"); + printf ("YES!"); + + return 0; +} diff --git a/var/spack/repos/builtin/packages/cxx/test/hello.cc b/var/spack/repos/builtin/packages/cxx/test/hello.cc new file mode 100644 index 0000000000..2a85869996 --- /dev/null +++ b/var/spack/repos/builtin/packages/cxx/test/hello.cc @@ -0,0 +1,9 @@ +#include <iostream> +using namespace std; + +int main() +{ + cout << "Hello world from C++!" << endl; + cout << "YES!" << endl; + return (0); +} diff --git a/var/spack/repos/builtin/packages/cxx/test/hello.cpp b/var/spack/repos/builtin/packages/cxx/test/hello.cpp new file mode 100644 index 0000000000..b49db59f4a --- /dev/null +++ b/var/spack/repos/builtin/packages/cxx/test/hello.cpp @@ -0,0 +1,9 @@ +#include <iostream> +using namespace std; + +int main() +{ + cout << "Hello world from C++!" << endl; + cout << "YES!" << endl; + return (0); +} diff --git a/var/spack/repos/builtin/packages/cxx/test/hello_c++11.cc b/var/spack/repos/builtin/packages/cxx/test/hello_c++11.cc new file mode 100644 index 0000000000..10f57c3f75 --- /dev/null +++ b/var/spack/repos/builtin/packages/cxx/test/hello_c++11.cc @@ -0,0 +1,17 @@ +#include <iostream> +#include <regex> + +using namespace std; + +int main() +{ + auto func = [] () { cout << "Hello world from C++11" << endl; }; + func(); // now call the function + + std::regex r("st|mt|tr"); + std::cout << "std::regex r(\"st|mt|tr\")" << " match tr? "; + if (std::regex_match("tr", r) == 0) + std::cout << "NO!\n ==> Using pre g++ 4.9.2 libstdc++ which doesn't implement regex properly" << std::endl; + else + std::cout << "YES!\n ==> Correct libstdc++11 implementation of regex (4.9.2 or later)" << std::endl; +} diff --git a/var/spack/repos/builtin/packages/emacs/package.py b/var/spack/repos/builtin/packages/emacs/package.py index 393def5cc5..0759fd28d5 100644 --- a/var/spack/repos/builtin/packages/emacs/package.py +++ b/var/spack/repos/builtin/packages/emacs/package.py @@ -80,3 +80,18 @@ class Emacs(AutotoolsPackage, GNUMirrorPackage): args.append('--without-gnutls') return args + + def _test_check_versions(self): + """Perform version checks on installed package binaries.""" + checks = ['ctags', 'ebrowse', 'emacs', 'emacsclient', 'etags'] + + for exe in checks: + expected = str(self.spec.version) + reason = 'test version of {0} is {1}'.format(exe, expected) + self.run_test(exe, ['--version'], expected, installed=True, + purpose=reason, skip_missing=True) + + def test(self): + """Perform smoke tests on the installed package.""" + # Simple version check tests on known binaries + self._test_check_versions() diff --git a/var/spack/repos/builtin/packages/fortran/package.py b/var/spack/repos/builtin/packages/fortran/package.py new file mode 100644 index 0000000000..6383ff856b --- /dev/null +++ b/var/spack/repos/builtin/packages/fortran/package.py @@ -0,0 +1,28 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +import os + + +class Fortran(Package): + """Virtual package for the Fortran language.""" + homepage = 'https://wg5-fortran.org/' + virtual = True + + def test(self): + test_source = self.test_suite.current_test_data_dir + + for test in os.listdir(test_source): + filepath = os.path.join(test_source, test) + exe_name = '%s.exe' % test + + fc_exe = os.environ['FC'] + fc_opts = ['-o', exe_name, filepath] + + compiled = self.run_test(fc_exe, options=fc_opts, installed=True) + + if compiled: + expected = ['Hello world', 'YES!'] + self.run_test(exe_name, expected=expected) diff --git a/var/spack/repos/builtin/packages/fortran/test/hello.F b/var/spack/repos/builtin/packages/fortran/test/hello.F new file mode 100644 index 0000000000..886046eaed --- /dev/null +++ b/var/spack/repos/builtin/packages/fortran/test/hello.F @@ -0,0 +1,6 @@ + program line + + write (*,*) "Hello world from FORTRAN" + write (*,*) "YES!" + + end diff --git a/var/spack/repos/builtin/packages/fortran/test/hello.f90 b/var/spack/repos/builtin/packages/fortran/test/hello.f90 new file mode 100644 index 0000000000..21717d11dd --- /dev/null +++ b/var/spack/repos/builtin/packages/fortran/test/hello.f90 @@ -0,0 +1,6 @@ +program line + + write (*,*) "Hello world from FORTRAN" + write (*,*) "YES!" + +end program line diff --git a/var/spack/repos/builtin/packages/gdal/package.py b/var/spack/repos/builtin/packages/gdal/package.py index 1162ffe9ae..a6f1dfd909 100644 --- a/var/spack/repos/builtin/packages/gdal/package.py +++ b/var/spack/repos/builtin/packages/gdal/package.py @@ -124,7 +124,7 @@ class Gdal(AutotoolsPackage): depends_on('hdf5', when='+hdf5') depends_on('kealib', when='+kea @2:') depends_on('netcdf-c', when='+netcdf') - depends_on('jasper@1.900.1', patches='uuid.patch', when='+jasper') + depends_on('jasper@1.900.1', patches=[patch('uuid.patch')], when='+jasper') depends_on('openjpeg', when='+openjpeg') depends_on('xerces-c', when='+xerces') depends_on('expat', when='+expat') diff --git a/var/spack/repos/builtin/packages/hdf/package.py b/var/spack/repos/builtin/packages/hdf/package.py index d40a0c21fe..76c2205f27 100644 --- a/var/spack/repos/builtin/packages/hdf/package.py +++ b/var/spack/repos/builtin/packages/hdf/package.py @@ -4,6 +4,7 @@ # SPDX-License-Identifier: (Apache-2.0 OR MIT) import sys +import os class Hdf(AutotoolsPackage): @@ -151,3 +152,67 @@ class Hdf(AutotoolsPackage): def check(self): with working_dir(self.build_directory): make('check', parallel=False) + + extra_install_tests = 'hdf/util/testfiles' + + @run_after('install') + def setup_build_tests(self): + """Copy the build test files after the package is installed to an + install test subdirectory for use during `spack test run`.""" + self.cache_extra_test_sources(self.extra_install_tests) + + def _test_check_versions(self): + """Perform version checks on selected installed package binaries.""" + spec_vers_str = 'Version {0}'.format(self.spec.version.up_to(2)) + + exes = ['hdfimport', 'hrepack', 'ncdump', 'ncgen'] + for exe in exes: + reason = 'test: ensuring version of {0} is {1}' \ + .format(exe, spec_vers_str) + self.run_test(exe, ['-V'], spec_vers_str, installed=True, + purpose=reason, skip_missing=True) + + def _test_gif_converters(self): + """This test performs an image conversion sequence and diff.""" + work_dir = '.' + storm_fn = os.path.join(self.install_test_root, + self.extra_install_tests, 'storm110.hdf') + gif_fn = 'storm110.gif' + new_hdf_fn = 'storm110gif.hdf' + + # Convert a test HDF file to a gif + self.run_test('hdf2gif', [storm_fn, gif_fn], '', installed=True, + purpose="test: hdf-to-gif", work_dir=work_dir) + + # Convert the gif to an HDF file + self.run_test('gif2hdf', [gif_fn, new_hdf_fn], '', installed=True, + purpose="test: gif-to-hdf", work_dir=work_dir) + + # Compare the original and new HDF files + self.run_test('hdiff', [new_hdf_fn, storm_fn], '', installed=True, + purpose="test: compare orig to new hdf", + work_dir=work_dir) + + def _test_list(self): + """This test compares low-level HDF file information to expected.""" + storm_fn = os.path.join(self.install_test_root, + self.extra_install_tests, 'storm110.hdf') + test_data_dir = self.test_suite.current_test_data_dir + work_dir = '.' + + reason = 'test: checking hdfls output' + details_file = os.path.join(test_data_dir, 'storm110.out') + expected = get_escaped_text_output(details_file) + self.run_test('hdfls', [storm_fn], expected, installed=True, + purpose=reason, skip_missing=True, work_dir=work_dir) + + def test(self): + """Perform smoke tests on the installed package.""" + # Simple version check tests on subset of known binaries that respond + self._test_check_versions() + + # Run gif converter sequence test + self._test_gif_converters() + + # Run hdfls output + self._test_list() diff --git a/var/spack/repos/builtin/packages/hdf/test/storm110.out b/var/spack/repos/builtin/packages/hdf/test/storm110.out new file mode 100644 index 0000000000..f17e4ce2b3 --- /dev/null +++ b/var/spack/repos/builtin/packages/hdf/test/storm110.out @@ -0,0 +1,17 @@ +File library version: Major= 0, Minor=0, Release=0 +String= + +Number type : (tag 106) + Ref nos: 110 +Machine type : (tag 107) + Ref nos: 4369 +Image Dimensions-8 : (tag 200) + Ref nos: 110 +Raster Image-8 : (tag 202) + Ref nos: 110 +Image Dimensions : (tag 300) + Ref nos: 110 +Raster Image Data : (tag 302) + Ref nos: 110 +Raster Image Group : (tag 306) + Ref nos: 110 diff --git a/var/spack/repos/builtin/packages/hdf5/package.py b/var/spack/repos/builtin/packages/hdf5/package.py index e592ab5e9f..b37c7ede65 100644 --- a/var/spack/repos/builtin/packages/hdf5/package.py +++ b/var/spack/repos/builtin/packages/hdf5/package.py @@ -6,8 +6,6 @@ import shutil import sys -from spack import * - class Hdf5(AutotoolsPackage): """HDF5 is a data model, library, and file format for storing and managing @@ -327,6 +325,9 @@ class Hdf5(AutotoolsPackage): @run_after('install') @on_package_attributes(run_tests=True) def check_install(self): + self._check_install() + + def _check_install(self): # Build and run a small program to test the installed HDF5 library spec = self.spec print("Checking HDF5 installation...") @@ -375,3 +376,55 @@ HDF5 version {version} {version} print('-' * 80) raise RuntimeError("HDF5 install check failed") shutil.rmtree(checkdir) + + def _test_check_versions(self): + """Perform version checks on selected installed package binaries.""" + spec_vers_str = 'Version {0}'.format(self.spec.version) + + exes = [ + 'h5copy', 'h5diff', 'h5dump', 'h5format_convert', 'h5ls', + 'h5mkgrp', 'h5repack', 'h5stat', 'h5unjam', + ] + use_short_opt = ['h52gif', 'h5repart', 'h5unjam'] + for exe in exes: + reason = 'test: ensuring version of {0} is {1}' \ + .format(exe, spec_vers_str) + option = '-V' if exe in use_short_opt else '--version' + self.run_test(exe, option, spec_vers_str, installed=True, + purpose=reason, skip_missing=True) + + def _test_example(self): + """This test performs copy, dump, and diff on an example hdf5 file.""" + test_data_dir = self.test_suite.current_test_data_dir + + filename = 'spack.h5' + h5_file = test_data_dir.join(filename) + + reason = 'test: ensuring h5dump produces expected output' + expected = get_escaped_text_output(test_data_dir.join('dump.out')) + self.run_test('h5dump', filename, expected, installed=True, + purpose=reason, skip_missing=True, + work_dir=test_data_dir) + + reason = 'test: ensuring h5copy runs' + options = ['-i', h5_file, '-s', 'Spack', '-o', 'test.h5', '-d', + 'Spack'] + self.run_test('h5copy', options, [], installed=True, + purpose=reason, skip_missing=True, work_dir='.') + + reason = ('test: ensuring h5diff shows no differences between orig and' + ' copy') + self.run_test('h5diff', [h5_file, 'test.h5'], [], installed=True, + purpose=reason, skip_missing=True, work_dir='.') + + def test(self): + """Perform smoke tests on the installed package.""" + # Simple version check tests on known binaries + self._test_check_versions() + + # Run sequence of commands on an hdf5 file + self._test_example() + + # Run existing install check + # TODO: Restore once address built vs. installed state + # self._check_install() diff --git a/var/spack/repos/builtin/packages/hdf5/test/dump.out b/var/spack/repos/builtin/packages/hdf5/test/dump.out new file mode 100644 index 0000000000..58decefc12 --- /dev/null +++ b/var/spack/repos/builtin/packages/hdf5/test/dump.out @@ -0,0 +1,45 @@ +HDF5 "spack.h5" { +GROUP "/" { + GROUP "Spack" { + GROUP "Software" { + ATTRIBUTE "Distribution" { + DATATYPE H5T_STRING { + STRSIZE H5T_VARIABLE; + STRPAD H5T_STR_NULLTERM; + CSET H5T_CSET_UTF8; + CTYPE H5T_C_S1; + } + DATASPACE SCALAR + DATA { + (0): "Open Source" + } + } + DATASET "data" { + DATATYPE H5T_IEEE_F64LE + DATASPACE SIMPLE { ( 7, 11 ) / ( 7, 11 ) } + DATA { + (0,0): 0.371141, 0.508482, 0.585975, 0.0944911, 0.684849, + (0,5): 0.580396, 0.720271, 0.693561, 0.340432, 0.217145, + (0,10): 0.636083, + (1,0): 0.686996, 0.773501, 0.656767, 0.617543, 0.226132, + (1,5): 0.768632, 0.0548711, 0.54572, 0.355544, 0.591548, + (1,10): 0.233007, + (2,0): 0.230032, 0.192087, 0.293845, 0.0369338, 0.038727, + (2,5): 0.0977931, 0.966522, 0.0821391, 0.857921, 0.495703, + (2,10): 0.746006, + (3,0): 0.598494, 0.990266, 0.993009, 0.187481, 0.746391, + (3,5): 0.140095, 0.122661, 0.929242, 0.542415, 0.802758, + (3,10): 0.757941, + (4,0): 0.372124, 0.411982, 0.270479, 0.950033, 0.329948, + (4,5): 0.936704, 0.105097, 0.742285, 0.556565, 0.18988, 0.72797, + (5,0): 0.801669, 0.271807, 0.910649, 0.186251, 0.868865, + (5,5): 0.191484, 0.788371, 0.920173, 0.582249, 0.682022, + (5,10): 0.146883, + (6,0): 0.826824, 0.0886705, 0.402606, 0.0532444, 0.72509, + (6,5): 0.964683, 0.330362, 0.833284, 0.630456, 0.411489, 0.247806 + } + } + } + } +} +} diff --git a/var/spack/repos/builtin/packages/hdf5/test/spack.h5 b/var/spack/repos/builtin/packages/hdf5/test/spack.h5 Binary files differnew file mode 100644 index 0000000000..c2f3a6f39d --- /dev/null +++ b/var/spack/repos/builtin/packages/hdf5/test/spack.h5 diff --git a/var/spack/repos/builtin/packages/jq/package.py b/var/spack/repos/builtin/packages/jq/package.py index 13d3d939a2..9f67ce5bbe 100644 --- a/var/spack/repos/builtin/packages/jq/package.py +++ b/var/spack/repos/builtin/packages/jq/package.py @@ -21,7 +21,7 @@ class Jq(AutotoolsPackage): @run_after('install') @on_package_attributes(run_tests=True) - def installtest(self): + def install_test(self): jq = self.spec['jq'].command f = os.path.join(os.path.dirname(__file__), 'input.json') diff --git a/var/spack/repos/builtin/packages/kcov/package.py b/var/spack/repos/builtin/packages/kcov/package.py index 8f01ffc985..5e7bf48bd5 100644 --- a/var/spack/repos/builtin/packages/kcov/package.py +++ b/var/spack/repos/builtin/packages/kcov/package.py @@ -27,7 +27,7 @@ class Kcov(CMakePackage): @run_after('install') @on_package_attributes(run_tests=True) - def test(self): + def test_install(self): # The help message exits with an exit code of 1 kcov = Executable(self.prefix.bin.kcov) kcov('-h', ignore_errors=1) diff --git a/var/spack/repos/builtin/packages/libsigsegv/package.py b/var/spack/repos/builtin/packages/libsigsegv/package.py index 7aab695b76..119778f018 100644 --- a/var/spack/repos/builtin/packages/libsigsegv/package.py +++ b/var/spack/repos/builtin/packages/libsigsegv/package.py @@ -3,8 +3,6 @@ # # SPDX-License-Identifier: (Apache-2.0 OR MIT) -from spack import * - class Libsigsegv(AutotoolsPackage, GNUMirrorPackage): """GNU libsigsegv is a library for handling page faults in user mode.""" @@ -18,5 +16,60 @@ class Libsigsegv(AutotoolsPackage, GNUMirrorPackage): patch('patch.new_config_guess', when='@2.10') + test_requires_compiler = True + def configure_args(self): return ['--enable-shared'] + + extra_install_tests = 'tests/.libs' + + @run_after('install') + def setup_build_tests(self): + """Copy the build test files after the package is installed to an + install test subdirectory for use during `spack test run`.""" + self.cache_extra_test_sources(self.extra_install_tests) + + def _run_smoke_tests(self): + """Build and run the added smoke (install) test.""" + data_dir = self.test_suite.current_test_data_dir + prog = 'smoke_test' + src = data_dir.join('{0}.c'.format(prog)) + + options = [ + '-I{0}'.format(self.prefix.include), + src, + '-o', + prog, + '-L{0}'.format(self.prefix.lib), + '-lsigsegv', + '{0}{1}'.format(self.compiler.cc_rpath_arg, self.prefix.lib)] + reason = 'test: checking ability to link to the library' + self.run_test('cc', options, [], installed=False, purpose=reason) + + # Now run the program and confirm the output matches expectations + expected = get_escaped_text_output(data_dir.join('smoke_test.out')) + reason = 'test: checking ability to use the library' + self.run_test(prog, [], expected, purpose=reason) + + def _run_build_tests(self): + """Run selected build tests.""" + passed = 'Test passed' + checks = { + 'sigsegv1': [passed], + 'sigsegv2': [passed], + 'sigsegv3': ['caught', passed], + 'stackoverflow1': ['recursion', 'Stack overflow', passed], + 'stackoverflow2': ['recursion', 'overflow', 'violation', passed], + } + + for exe, expected in checks.items(): + reason = 'test: checking {0} output'.format(exe) + self.run_test(exe, [], expected, installed=True, purpose=reason, + skip_missing=True) + + def test(self): + # Run the simple built-in smoke test + self._run_smoke_tests() + + # Run test programs pulled from the build + self._run_build_tests() diff --git a/var/spack/repos/builtin/packages/libsigsegv/test/smoke_test.c b/var/spack/repos/builtin/packages/libsigsegv/test/smoke_test.c new file mode 100644 index 0000000000..f1ab68cd53 --- /dev/null +++ b/var/spack/repos/builtin/packages/libsigsegv/test/smoke_test.c @@ -0,0 +1,70 @@ +/* Simple "Hello World" test set up to handle a single page fault + * + * Inspired by libsigsegv's test cases with argument names for handlers + * taken from the header files. + */ + +#include "sigsegv.h" +#include <stdio.h> +#include <stdlib.h> /* for exit */ +# include <stddef.h> /* for NULL on SunOS4 (per libsigsegv examples) */ +#include <setjmp.h> /* for controlling handler-related flow */ + + +/* Calling environment */ +jmp_buf calling_env; + +char *message = "Hello, World!"; + +/* Track the number of times the handler is called */ +volatile int times_called = 0; + + +/* Continuation function, which relies on the latest libsigsegv API */ +static void +resume(void *cont_arg1, void *cont_arg2, void *cont_arg3) +{ + /* Go to calling environment and restore state. */ + longjmp(calling_env, times_called); +} + +/* sigsegv handler */ +int +handle_sigsegv(void *fault_address, int serious) +{ + times_called++; + + /* Generate handler output for the test. */ + printf("Caught sigsegv #%d\n", times_called); + + return sigsegv_leave_handler(resume, NULL, NULL, NULL); +} + +/* "Buggy" function used to demonstrate non-local goto */ +void printit(char *m) +{ + if (times_called < 1) { + /* Force SIGSEGV only on the first call. */ + volatile int *fail_ptr = 0; + int failure = *fail_ptr; + printf("%s\n", m); + } else { + /* Print it correctly. */ + printf("%s\n", m); + } +} + +int +main(void) +{ + /* Install the global SIGSEGV handler */ + sigsegv_install_handler(&handle_sigsegv); + + char *msg = "Hello World!"; + int calls = setjmp(calling_env); /* Resume here after detecting sigsegv */ + + /* Call the function that will trigger the page fault. */ + printit(msg); + + return 0; +} diff --git a/var/spack/repos/builtin/packages/libsigsegv/test/smoke_test.out b/var/spack/repos/builtin/packages/libsigsegv/test/smoke_test.out new file mode 100644 index 0000000000..31071777e2 --- /dev/null +++ b/var/spack/repos/builtin/packages/libsigsegv/test/smoke_test.out @@ -0,0 +1,2 @@ +Caught sigsegv #1 +Hello World! diff --git a/var/spack/repos/builtin/packages/libxml2/package.py b/var/spack/repos/builtin/packages/libxml2/package.py index 2602378f89..9cbc8a6817 100644 --- a/var/spack/repos/builtin/packages/libxml2/package.py +++ b/var/spack/repos/builtin/packages/libxml2/package.py @@ -2,6 +2,8 @@ # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) +import llnl.util.filesystem as fs +import llnl.util.tty as tty from spack import * @@ -82,3 +84,34 @@ class Libxml2(AutotoolsPackage): if '+python' in self.spec: with working_dir('spack-test', create=True): python('-c', 'import libxml2') + + def test(self): + """Perform smoke tests on the installed package""" + # Start with what we already have post-install + tty.msg('test: Performing simple import test') + self.import_module_test() + + data_dir = self.test_suite.current_test_data_dir + + # Now run defined tests based on expected executables + dtd_path = data_dir.join('info.dtd') + test_filename = 'test.xml' + exec_checks = { + 'xml2-config': [ + ('--version', [str(self.spec.version)], 0)], + 'xmllint': [ + (['--auto', '-o', test_filename], [], 0), + (['--postvalid', test_filename], + ['validity error', 'no DTD found', 'does not validate'], 3), + (['--dtdvalid', dtd_path, test_filename], + ['validity error', 'does not follow the DTD'], 3), + (['--dtdvalid', dtd_path, data_dir.join('info.xml')], [], 0)], + 'xmlcatalog': [ + ('--create', ['<catalog xmlns', 'catalog"/>'], 0)], + } + for exe in exec_checks: + for options, expected, status in exec_checks[exe]: + self.run_test(exe, options, expected, status) + + # Perform some cleanup + fs.force_remove(test_filename) diff --git a/var/spack/repos/builtin/packages/libxml2/test/info.dtd b/var/spack/repos/builtin/packages/libxml2/test/info.dtd new file mode 100644 index 0000000000..aec2dbe705 --- /dev/null +++ b/var/spack/repos/builtin/packages/libxml2/test/info.dtd @@ -0,0 +1,2 @@ +<!ELEMENT info (data)> +<!ELEMENT data (#PCDATA)> diff --git a/var/spack/repos/builtin/packages/libxml2/test/info.xml b/var/spack/repos/builtin/packages/libxml2/test/info.xml new file mode 100644 index 0000000000..23803694a7 --- /dev/null +++ b/var/spack/repos/builtin/packages/libxml2/test/info.xml @@ -0,0 +1,4 @@ +<?xml version="1.0"?> +<info> +<data>abc</data> +</info> diff --git a/var/spack/repos/builtin/packages/m4/package.py b/var/spack/repos/builtin/packages/m4/package.py index 6695cdf862..b0d037b3f2 100644 --- a/var/spack/repos/builtin/packages/m4/package.py +++ b/var/spack/repos/builtin/packages/m4/package.py @@ -74,3 +74,16 @@ class M4(AutotoolsPackage, GNUMirrorPackage): args.append('ac_cv_type_struct_sched_param=yes') return args + + def test(self): + spec_vers = str(self.spec.version) + reason = 'test: ensuring m4 version is {0}'.format(spec_vers) + self.run_test('m4', '--version', spec_vers, installed=True, + purpose=reason, skip_missing=False) + + reason = 'test: ensuring m4 example succeeds' + test_data_dir = self.test_suite.current_test_data_dir + hello_file = test_data_dir.join('hello.m4') + expected = get_escaped_text_output(test_data_dir.join('hello.out')) + self.run_test('m4', hello_file, expected, installed=True, + purpose=reason, skip_missing=False) diff --git a/var/spack/repos/builtin/packages/m4/test/hello.m4 b/var/spack/repos/builtin/packages/m4/test/hello.m4 new file mode 100644 index 0000000000..6132c41093 --- /dev/null +++ b/var/spack/repos/builtin/packages/m4/test/hello.m4 @@ -0,0 +1,4 @@ +define(NAME, World) +dnl This line should not show up +// macro is ifdef(`NAME', , not)defined +Hello, NAME! diff --git a/var/spack/repos/builtin/packages/m4/test/hello.out b/var/spack/repos/builtin/packages/m4/test/hello.out new file mode 100644 index 0000000000..c8d3be7e16 --- /dev/null +++ b/var/spack/repos/builtin/packages/m4/test/hello.out @@ -0,0 +1,3 @@ + +// macro is defined +Hello, World! diff --git a/var/spack/repos/builtin/packages/mpi/package.py b/var/spack/repos/builtin/packages/mpi/package.py new file mode 100644 index 0000000000..731a5ac731 --- /dev/null +++ b/var/spack/repos/builtin/packages/mpi/package.py @@ -0,0 +1,31 @@ +# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +import os + + +class Mpi(Package): + """Virtual package for the Message Passing Interface.""" + homepage = 'https://www.mpi-forum.org/' + virtual = True + + def test(self): + for lang in ('c', 'f'): + filename = self.test_suite.current_test_data_dir.join( + 'mpi_hello.' + lang) + + compiler_var = 'MPICC' if lang == 'c' else 'MPIF90' + compiler = os.environ[compiler_var] + + exe_name = 'mpi_hello_%s' % lang + mpirun = join_path(self.prefix.bin, 'mpirun') + + compiled = self.run_test(compiler, + options=['-o', exe_name, filename]) + if compiled: + self.run_test(mpirun, + options=['-np', '1', exe_name], + expected=[r'Hello world! From rank \s*0 of \s*1'] + ) diff --git a/var/spack/repos/builtin/packages/mpi/test/mpi_hello.c b/var/spack/repos/builtin/packages/mpi/test/mpi_hello.c new file mode 100644 index 0000000000..9db7c5a436 --- /dev/null +++ b/var/spack/repos/builtin/packages/mpi/test/mpi_hello.c @@ -0,0 +1,16 @@ +#include <stdio.h> +#include <mpi.h> + +int main(int argc, char** argv) { + MPI_Init(&argc, &argv); + + int rank; + int num_ranks; + MPI_Comm_rank(MPI_COMM_WORLD, &rank); + MPI_Comm_size(MPI_COMM_WORLD, &num_ranks); + + printf("Hello world! From rank %d of %d\n", rank, num_ranks); + + MPI_Finalize(); + return(0); +} diff --git a/var/spack/repos/builtin/packages/mpi/test/mpi_hello.f b/var/spack/repos/builtin/packages/mpi/test/mpi_hello.f new file mode 100644 index 0000000000..ecc7005d00 --- /dev/null +++ b/var/spack/repos/builtin/packages/mpi/test/mpi_hello.f @@ -0,0 +1,11 @@ +c Fortran example + program hello + include 'mpif.h' + integer rank, num_ranks, err_flag + + call MPI_INIT(err_flag) + call MPI_COMM_SIZE(MPI_COMM_WORLD, num_ranks, err_flag) + call MPI_COMM_RANK(MPI_COMM_WORLD, rank, err_flag) + print*, 'Hello world! From rank', rank, 'of ', num_ranks + call MPI_FINALIZE(err_flag) + end diff --git a/var/spack/repos/builtin/packages/ninja-fortran/package.py b/var/spack/repos/builtin/packages/ninja-fortran/package.py index 8e9fcb9851..d85a5b4542 100644 --- a/var/spack/repos/builtin/packages/ninja-fortran/package.py +++ b/var/spack/repos/builtin/packages/ninja-fortran/package.py @@ -51,7 +51,7 @@ class NinjaFortran(Package): @run_after('configure') @on_package_attributes(run_tests=True) - def test(self): + def configure_test(self): ninja = Executable('./ninja') ninja('-j{0}'.format(make_jobs), 'ninja_test') ninja_test = Executable('./ninja_test') diff --git a/var/spack/repos/builtin/packages/ninja/package.py b/var/spack/repos/builtin/packages/ninja/package.py index 40890c212e..96cd0252a2 100644 --- a/var/spack/repos/builtin/packages/ninja/package.py +++ b/var/spack/repos/builtin/packages/ninja/package.py @@ -2,7 +2,6 @@ # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) -import re class Ninja(Package): @@ -40,7 +39,7 @@ class Ninja(Package): @run_after('configure') @on_package_attributes(run_tests=True) - def test(self): + def configure_test(self): ninja = Executable('./ninja') ninja('-j{0}'.format(make_jobs), 'ninja_test') ninja_test = Executable('./ninja_test') diff --git a/var/spack/repos/builtin/packages/node-js/package.py b/var/spack/repos/builtin/packages/node-js/package.py index b35e1d0def..28def78333 100644 --- a/var/spack/repos/builtin/packages/node-js/package.py +++ b/var/spack/repos/builtin/packages/node-js/package.py @@ -127,7 +127,7 @@ class NodeJs(Package): @run_after('build') @on_package_attributes(run_tests=True) - def test(self): + def build_test(self): make('test') make('test-addons') diff --git a/var/spack/repos/builtin/packages/openmpi/package.py b/var/spack/repos/builtin/packages/openmpi/package.py index b56c10fb67..124973bbc1 100644 --- a/var/spack/repos/builtin/packages/openmpi/package.py +++ b/var/spack/repos/builtin/packages/openmpi/package.py @@ -336,6 +336,8 @@ class Openmpi(AutotoolsPackage): filter_compiler_wrappers('openmpi/*-wrapper-data*', relative_root='share') + extra_install_tests = 'examples' + @classmethod def determine_version(cls, exe): output = Executable(exe)(output=str, error=str) @@ -846,6 +848,149 @@ class Openmpi(AutotoolsPackage): else: copy(script_stub, exe) + @run_after('install') + def setup_install_tests(self): + """ + Copy the example files after the package is installed to an + install test subdirectory for use during `spack test run`. + """ + self.cache_extra_test_sources(self.extra_install_tests) + + def _test_bin_ops(self): + info = ([], ['Ident string: {0}'.format(self.spec.version), 'MCA'], + 0) + + ls = (['-n', '1', 'ls', '..'], + ['openmpi-{0}'.format(self.spec.version)], 0) + + checks = { + 'mpirun': ls, + 'ompi_info': info, + 'oshmem_info': info, + 'oshrun': ls, + 'shmemrun': ls, + } + + for exe in checks: + options, expected, status = checks[exe] + reason = 'test: checking {0} output'.format(exe) + self.run_test(exe, options, expected, status, installed=True, + purpose=reason, skip_missing=True) + + def _test_check_versions(self): + comp_vers = str(self.spec.compiler.version) + spec_vers = str(self.spec.version) + checks = { + # Binaries available in at least versions 2.0.0 through 4.0.3 + 'mpiCC': comp_vers, + 'mpic++': comp_vers, + 'mpicc': comp_vers, + 'mpicxx': comp_vers, + 'mpiexec': spec_vers, + 'mpif77': comp_vers, + 'mpif90': comp_vers, + 'mpifort': comp_vers, + 'mpirun': spec_vers, + 'ompi_info': spec_vers, + 'ortecc': comp_vers, + 'orterun': spec_vers, + + # Binaries available in versions 2.0.0 through 2.1.6 + 'ompi-submit': spec_vers, + 'orte-submit': spec_vers, + + # Binaries available in versions 2.0.0 through 3.1.5 + 'ompi-dvm': spec_vers, + 'orte-dvm': spec_vers, + 'oshcc': comp_vers, + 'oshfort': comp_vers, + 'oshmem_info': spec_vers, + 'oshrun': spec_vers, + 'shmemcc': comp_vers, + 'shmemfort': comp_vers, + 'shmemrun': spec_vers, + + # Binary available in version 3.1.0 through 3.1.5 + 'prun': spec_vers, + + # Binaries available in versions 3.0.0 through 3.1.5 + 'oshCC': comp_vers, + 'oshc++': comp_vers, + 'oshcxx': comp_vers, + 'shmemCC': comp_vers, + 'shmemc++': comp_vers, + 'shmemcxx': comp_vers, + } + + for exe in checks: + expected = checks[exe] + purpose = 'test: ensuring version of {0} is {1}' \ + .format(exe, expected) + self.run_test(exe, '--version', expected, installed=True, + purpose=purpose, skip_missing=True) + + def _test_examples(self): + # First build the examples + self.run_test('make', ['all'], [], + purpose='test: ensuring ability to build the examples', + work_dir=join_path(self.install_test_root, + self.extra_install_tests)) + + # Now run those with known results + have_spml = self.spec.satisfies('@2.0.0:2.1.6') + + hello_world = (['Hello, world', 'I am', '0 of', '1'], 0) + + max_red = (['0/1 dst = 0 1 2'], 0) + + missing_spml = (['No available spml components'], 1) + + no_out = ([''], 0) + + ring_out = (['1 processes in ring', '0 exiting'], 0) + + strided = (['not in valid range'], 255) + + checks = { + 'hello_c': hello_world, + 'hello_cxx': hello_world, + 'hello_mpifh': hello_world, + 'hello_oshmem': hello_world if have_spml else missing_spml, + 'hello_oshmemcxx': hello_world if have_spml else missing_spml, + 'hello_oshmemfh': hello_world if have_spml else missing_spml, + 'hello_usempi': hello_world, + 'hello_usempif08': hello_world, + 'oshmem_circular_shift': ring_out if have_spml else missing_spml, + 'oshmem_max_reduction': max_red if have_spml else missing_spml, + 'oshmem_shmalloc': no_out if have_spml else missing_spml, + 'oshmem_strided_puts': strided if have_spml else missing_spml, + 'oshmem_symmetric_data': no_out if have_spml else missing_spml, + 'ring_c': ring_out, + 'ring_cxx': ring_out, + 'ring_mpifh': ring_out, + 'ring_oshmem': ring_out if have_spml else missing_spml, + 'ring_oshmemfh': ring_out if have_spml else missing_spml, + 'ring_usempi': ring_out, + 'ring_usempif08': ring_out, + } + + for exe in checks: + expected = checks[exe] + reason = 'test: checking example {0} output'.format(exe) + self.run_test(exe, [], expected, 0, installed=True, + purpose=reason, skip_missing=True) + + def test(self): + """Perform smoke tests on the installed package.""" + # Simple version check tests on known packages + self._test_check_versions() + + # Test the operation of selected executables + self._test_bin_ops() + + # Test example programs pulled from the build + self._test_examples() + def get_spack_compiler_spec(path): spack_compilers = spack.compilers.find_compilers([path]) diff --git a/var/spack/repos/builtin/packages/patchelf/package.py b/var/spack/repos/builtin/packages/patchelf/package.py index d17bb3bea1..796fd533eb 100644 --- a/var/spack/repos/builtin/packages/patchelf/package.py +++ b/var/spack/repos/builtin/packages/patchelf/package.py @@ -2,8 +2,8 @@ # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) - from spack import * +import os class Patchelf(AutotoolsPackage): @@ -18,3 +18,24 @@ class Patchelf(AutotoolsPackage): version('0.10', sha256='b2deabce05c34ce98558c0efb965f209de592197b2c88e930298d740ead09019') version('0.9', sha256='f2aa40a6148cb3b0ca807a1bf836b081793e55ec9e5540a5356d800132be7e0a') version('0.8', sha256='14af06a2da688d577d64ff8dac065bb8903bbffbe01d30c62df7af9bf4ce72fe') + + def test(self): + # Check patchelf in prefix and reports the correct version + reason = 'test: ensuring patchelf version is {0}' \ + .format(self.spec.version) + self.run_test('patchelf', + options='--version', + expected=['patchelf %s' % self.spec.version], + installed=True, + purpose=reason) + + # Check the rpath is changed + currdir = os.getcwd() + hello_file = self.test_suite.current_test_data_dir.join('hello') + self.run_test('patchelf', ['--set-rpath', currdir, hello_file], + purpose='test: ensuring that patchelf can change rpath') + + self.run_test('patchelf', + options=['--print-rpath', hello_file], + expected=[currdir], + purpose='test: ensuring that patchelf changed rpath') diff --git a/var/spack/repos/builtin/packages/patchelf/test/hello b/var/spack/repos/builtin/packages/patchelf/test/hello Binary files differnew file mode 100755 index 0000000000..8767836f8e --- /dev/null +++ b/var/spack/repos/builtin/packages/patchelf/test/hello diff --git a/var/spack/repos/builtin/packages/perl/package.py b/var/spack/repos/builtin/packages/perl/package.py index 744b55495a..2532f97160 100644 --- a/var/spack/repos/builtin/packages/perl/package.py +++ b/var/spack/repos/builtin/packages/perl/package.py @@ -204,7 +204,7 @@ class Perl(Package): # Perl doesn't use Autotools, it should subclass Package @run_after('build') @on_package_attributes(run_tests=True) - def test(self): + def build_test(self): make('test') def install(self, spec, prefix): @@ -364,3 +364,16 @@ class Perl(Package): # Perl doesn't use Autotools, it should subclass Package else: msg = 'Unable to locate {0} command in {1}' raise RuntimeError(msg.format(self.spec.name, self.prefix.bin)) + + def test(self): + """Smoke tests""" + exe = self.spec['perl'].command.name + + reason = 'test: checking version is {0}'.format(self.spec.version) + self.run_test(exe, '--version', ['perl', str(self.spec.version)], + installed=True, purpose=reason) + + reason = 'test: ensuring perl runs' + msg = 'Hello, World!' + options = ['-e', 'use warnings; use strict;\nprint("%s\n");' % msg] + self.run_test(exe, options, msg, installed=True, purpose=reason) diff --git a/var/spack/repos/builtin/packages/py-cloudpickle/package.py b/var/spack/repos/builtin/packages/py-cloudpickle/package.py index ff1e459e09..0f7e525290 100644 --- a/var/spack/repos/builtin/packages/py-cloudpickle/package.py +++ b/var/spack/repos/builtin/packages/py-cloudpickle/package.py @@ -19,6 +19,6 @@ class PyCloudpickle(PythonPackage): depends_on('py-setuptools', type='build') - def test(self): + def build_test(self): # PyPI tarball does not come with unit tests pass diff --git a/var/spack/repos/builtin/packages/py-cython/package.py b/var/spack/repos/builtin/packages/py-cython/package.py index aa30f56237..2dd532be2f 100644 --- a/var/spack/repos/builtin/packages/py-cython/package.py +++ b/var/spack/repos/builtin/packages/py-cython/package.py @@ -47,6 +47,6 @@ class PyCython(PythonPackage): """Returns the Cython command""" return Executable(self.prefix.bin.cython) - def test(self): + def build_test(self): # Warning: full suite of unit tests takes a very long time python('runtests.py', '-j', str(make_jobs)) diff --git a/var/spack/repos/builtin/packages/py-fiona/package.py b/var/spack/repos/builtin/packages/py-fiona/package.py index b6ad22a9a7..b4213d5f61 100644 --- a/var/spack/repos/builtin/packages/py-fiona/package.py +++ b/var/spack/repos/builtin/packages/py-fiona/package.py @@ -34,6 +34,6 @@ class PyFiona(PythonPackage): depends_on('py-ordereddict', type=('build', 'run'), when='^python@:2.6') depends_on('py-enum34', type=('build', 'run'), when='^python@:3.3') - def test(self): + def build_test(self): # PyPI tarball does not come with unit tests pass diff --git a/var/spack/repos/builtin/packages/py-matplotlib/package.py b/var/spack/repos/builtin/packages/py-matplotlib/package.py index c2ce697ec2..ebb95033cf 100644 --- a/var/spack/repos/builtin/packages/py-matplotlib/package.py +++ b/var/spack/repos/builtin/packages/py-matplotlib/package.py @@ -183,6 +183,6 @@ class PyMatplotlib(PythonPackage): setup.write('system_freetype = True\n') setup.write('system_qhull = True\n') - def test(self): + def build_test(self): pytest = which('pytest') pytest() diff --git a/var/spack/repos/builtin/packages/py-numpy/package.py b/var/spack/repos/builtin/packages/py-numpy/package.py index 53b1f759a6..fd09019469 100644 --- a/var/spack/repos/builtin/packages/py-numpy/package.py +++ b/var/spack/repos/builtin/packages/py-numpy/package.py @@ -306,7 +306,7 @@ class PyNumpy(PythonPackage): return args - def test(self): + def build_test(self): # `setup.py test` is not supported. Use one of the following # instead: # diff --git a/var/spack/repos/builtin/packages/py-py/package.py b/var/spack/repos/builtin/packages/py-py/package.py index 2cb000a846..995612d20a 100644 --- a/var/spack/repos/builtin/packages/py-py/package.py +++ b/var/spack/repos/builtin/packages/py-py/package.py @@ -28,6 +28,6 @@ class PyPy(PythonPackage): depends_on('py-setuptools', type='build') depends_on('py-setuptools-scm', type='build') - def test(self): + def build_test(self): # Tests require pytest, creating a circular dependency pass diff --git a/var/spack/repos/builtin/packages/py-py2cairo/package.py b/var/spack/repos/builtin/packages/py-py2cairo/package.py index 1427c7e45c..5c492a9167 100644 --- a/var/spack/repos/builtin/packages/py-py2cairo/package.py +++ b/var/spack/repos/builtin/packages/py-py2cairo/package.py @@ -23,7 +23,7 @@ class PyPy2cairo(WafPackage): depends_on('py-pytest', type='test') - def installtest(self): + def install_test(self): with working_dir('test'): pytest = which('py.test') pytest() diff --git a/var/spack/repos/builtin/packages/py-pybind11/package.py b/var/spack/repos/builtin/packages/py-pybind11/package.py index 3fe7402a0f..1c07734e6f 100644 --- a/var/spack/repos/builtin/packages/py-pybind11/package.py +++ b/var/spack/repos/builtin/packages/py-pybind11/package.py @@ -74,7 +74,7 @@ class PyPybind11(CMakePackage): @run_after('install') @on_package_attributes(run_tests=True) - def test(self): + def install_test(self): with working_dir('spack-test', create=True): # test include helper points to right location python = self.spec['python'].command diff --git a/var/spack/repos/builtin/packages/py-pygments/package.py b/var/spack/repos/builtin/packages/py-pygments/package.py index 87476c64d6..d7559b36d8 100644 --- a/var/spack/repos/builtin/packages/py-pygments/package.py +++ b/var/spack/repos/builtin/packages/py-pygments/package.py @@ -29,6 +29,6 @@ class PyPygments(PythonPackage): depends_on('python@3.5:', type=('build', 'run'), when='@2.6:') depends_on('py-setuptools', type=('build', 'run')) - def test(self): + def build_test(self): # Unit tests require sphinx, but that creates a circular dependency pass diff --git a/var/spack/repos/builtin/packages/py-python-dateutil/package.py b/var/spack/repos/builtin/packages/py-python-dateutil/package.py index 13e04b4e6e..16bff6858d 100644 --- a/var/spack/repos/builtin/packages/py-python-dateutil/package.py +++ b/var/spack/repos/builtin/packages/py-python-dateutil/package.py @@ -31,7 +31,7 @@ class PyPythonDateutil(PythonPackage): # depends_on('py-hypothesis', type='test') # depends_on('py-freezegun', type='test') - def test(self): + def build_test(self): # Tests require freezegun, which depends on python-dateutil, # creating circular dependency # pytest = which('pytest') diff --git a/var/spack/repos/builtin/packages/py-scipy/package.py b/var/spack/repos/builtin/packages/py-scipy/package.py index 533d404fa6..afeae0dd1c 100644 --- a/var/spack/repos/builtin/packages/py-scipy/package.py +++ b/var/spack/repos/builtin/packages/py-scipy/package.py @@ -99,7 +99,7 @@ class PyScipy(PythonPackage): return args - def test(self): + def build_test(self): # `setup.py test` is not supported. Use one of the following # instead: # diff --git a/var/spack/repos/builtin/packages/py-setuptools/package.py b/var/spack/repos/builtin/packages/py-setuptools/package.py index 27786cd27e..1a2930498b 100644 --- a/var/spack/repos/builtin/packages/py-setuptools/package.py +++ b/var/spack/repos/builtin/packages/py-setuptools/package.py @@ -71,6 +71,6 @@ class PySetuptools(PythonPackage): return url - def test(self): + def build_test(self): # Unit tests require pytest, creating a circular dependency pass diff --git a/var/spack/repos/builtin/packages/py-shapely/package.py b/var/spack/repos/builtin/packages/py-shapely/package.py index 6dc62888ab..c03106a6be 100644 --- a/var/spack/repos/builtin/packages/py-shapely/package.py +++ b/var/spack/repos/builtin/packages/py-shapely/package.py @@ -64,5 +64,5 @@ class PyShapely(PythonPackage): else: env.prepend_path('LD_LIBRARY_PATH', libs) - def test(self): + def test_install(self): python('-m', 'pytest') diff --git a/var/spack/repos/builtin/packages/py-sphinxcontrib-applehelp/package.py b/var/spack/repos/builtin/packages/py-sphinxcontrib-applehelp/package.py index 25f1a7ce5f..67ba38134c 100644 --- a/var/spack/repos/builtin/packages/py-sphinxcontrib-applehelp/package.py +++ b/var/spack/repos/builtin/packages/py-sphinxcontrib-applehelp/package.py @@ -18,6 +18,6 @@ class PySphinxcontribApplehelp(PythonPackage): depends_on('python@3.5:', type=('build', 'run')) depends_on('py-setuptools', type='build') - def test(self): + def build_test(self): # Requires sphinx, creating a circular dependency pass diff --git a/var/spack/repos/builtin/packages/py-sphinxcontrib-devhelp/package.py b/var/spack/repos/builtin/packages/py-sphinxcontrib-devhelp/package.py index ff90a9a5d4..1954fc9677 100644 --- a/var/spack/repos/builtin/packages/py-sphinxcontrib-devhelp/package.py +++ b/var/spack/repos/builtin/packages/py-sphinxcontrib-devhelp/package.py @@ -18,6 +18,6 @@ class PySphinxcontribDevhelp(PythonPackage): depends_on('python@3.5:', type=('build', 'run')) depends_on('py-setuptools', type='build') - def test(self): + def build_test(self): # Requires sphinx, creating a circular dependency pass diff --git a/var/spack/repos/builtin/packages/py-sphinxcontrib-htmlhelp/package.py b/var/spack/repos/builtin/packages/py-sphinxcontrib-htmlhelp/package.py index 96a51d3113..95f6819d59 100644 --- a/var/spack/repos/builtin/packages/py-sphinxcontrib-htmlhelp/package.py +++ b/var/spack/repos/builtin/packages/py-sphinxcontrib-htmlhelp/package.py @@ -18,6 +18,6 @@ class PySphinxcontribHtmlhelp(PythonPackage): depends_on('python@3.5:', type=('build', 'run')) depends_on('py-setuptools', type='build') - def test(self): + def build_test(self): # Requires sphinx, creating a circular dependency pass diff --git a/var/spack/repos/builtin/packages/py-sphinxcontrib-jsmath/package.py b/var/spack/repos/builtin/packages/py-sphinxcontrib-jsmath/package.py index 45ee46bc67..add0160ac8 100644 --- a/var/spack/repos/builtin/packages/py-sphinxcontrib-jsmath/package.py +++ b/var/spack/repos/builtin/packages/py-sphinxcontrib-jsmath/package.py @@ -17,6 +17,6 @@ class PySphinxcontribJsmath(PythonPackage): depends_on('python@3.5:', type=('build', 'run')) depends_on('py-setuptools', type='build') - def test(self): + def build_test(self): # Requires sphinx, creating a circular dependency pass diff --git a/var/spack/repos/builtin/packages/py-sphinxcontrib-qthelp/package.py b/var/spack/repos/builtin/packages/py-sphinxcontrib-qthelp/package.py index 86a58d456f..19fd328f37 100644 --- a/var/spack/repos/builtin/packages/py-sphinxcontrib-qthelp/package.py +++ b/var/spack/repos/builtin/packages/py-sphinxcontrib-qthelp/package.py @@ -18,6 +18,6 @@ class PySphinxcontribQthelp(PythonPackage): depends_on('python@3.5:', type=('build', 'run')) depends_on('py-setuptools', type='build') - def test(self): + def build_test(self): # Requires sphinx, creating a circular dependency pass diff --git a/var/spack/repos/builtin/packages/py-sphinxcontrib-serializinghtml/package.py b/var/spack/repos/builtin/packages/py-sphinxcontrib-serializinghtml/package.py index 97b79a8012..3bad6d661a 100644 --- a/var/spack/repos/builtin/packages/py-sphinxcontrib-serializinghtml/package.py +++ b/var/spack/repos/builtin/packages/py-sphinxcontrib-serializinghtml/package.py @@ -18,6 +18,6 @@ class PySphinxcontribSerializinghtml(PythonPackage): depends_on('python@3.5:', type=('build', 'run')) depends_on('py-setuptools', type='build') - def test(self): + def build_test(self): # Requires sphinx, creating a circular dependency pass diff --git a/var/spack/repos/builtin/packages/py-sphinxcontrib-websupport/package.py b/var/spack/repos/builtin/packages/py-sphinxcontrib-websupport/package.py index c5a8f80a5b..1e4d1051a3 100644 --- a/var/spack/repos/builtin/packages/py-sphinxcontrib-websupport/package.py +++ b/var/spack/repos/builtin/packages/py-sphinxcontrib-websupport/package.py @@ -26,6 +26,6 @@ class PySphinxcontribWebsupport(PythonPackage): depends_on('python@2.7:2.8,3.4:', type=('build', 'run')) depends_on('py-setuptools', type='build') - def test(self): + def build_test(self): # Unit tests require sphinx, creating a circular dependency pass diff --git a/var/spack/repos/builtin/packages/py-statsmodels/package.py b/var/spack/repos/builtin/packages/py-statsmodels/package.py index 26c006bccf..b00e51730b 100644 --- a/var/spack/repos/builtin/packages/py-statsmodels/package.py +++ b/var/spack/repos/builtin/packages/py-statsmodels/package.py @@ -42,7 +42,7 @@ class PyStatsmodels(PythonPackage): depends_on('py-pytest', type='test') - def test(self): + def build_test(self): dirs = glob.glob("build/lib*") # There can be only one... with working_dir(dirs[0]): pytest = which('pytest') diff --git a/var/spack/repos/builtin/packages/python/package.py b/var/spack/repos/builtin/packages/python/package.py index 53924a85b0..4d0df211e6 100644 --- a/var/spack/repos/builtin/packages/python/package.py +++ b/var/spack/repos/builtin/packages/python/package.py @@ -1127,3 +1127,21 @@ class Python(AutotoolsPackage): view.remove_file(src, dst) else: os.remove(dst) + + def test(self): + # do not use self.command because we are also testing the run env + exe = self.spec['python'].command.name + + # test hello world + msg = 'hello world!' + reason = 'test: running {0}'.format(msg) + options = ['-c', 'print("{0}")'.format(msg)] + self.run_test(exe, options=options, expected=[msg], installed=True, + purpose=reason) + + # checks import works and executable comes from the spec prefix + reason = 'test: checking import and executable' + print_str = self.print_string('sys.executable') + options = ['-c', 'import sys; {0}'.format(print_str)] + self.run_test(exe, options=options, expected=[self.spec.prefix], + installed=True, purpose=reason) diff --git a/var/spack/repos/builtin/packages/raja/package.py b/var/spack/repos/builtin/packages/raja/package.py index 7da9c6c4fd..8d1db659cc 100644 --- a/var/spack/repos/builtin/packages/raja/package.py +++ b/var/spack/repos/builtin/packages/raja/package.py @@ -3,8 +3,6 @@ # # SPDX-License-Identifier: (Apache-2.0 OR MIT) -from spack import * - class Raja(CMakePackage, CudaPackage): """RAJA Parallel Framework.""" @@ -74,3 +72,52 @@ class Raja(CMakePackage, CudaPackage): options.append('-DENABLE_TESTS=ON') return options + + @property + def build_relpath(self): + """Relative path to the cmake build subdirectory.""" + return join_path('..', self.build_dirname) + + @run_after('install') + def setup_build_tests(self): + """Copy the build test files after the package is installed to a + relative install test subdirectory for use during `spack test run`.""" + # Now copy the relative files + self.cache_extra_test_sources(self.build_relpath) + + # Ensure the path exists since relying on a relative path at the + # same level as the normal stage source path. + mkdirp(self.install_test_root) + + @property + def _extra_tests_path(self): + # TODO: The tests should be converted to re-build and run examples + # TODO: using the installed libraries. + return join_path(self.install_test_root, self.build_relpath, 'bin') + + def _test_examples(self): + """Perform very basic checks on a subset of copied examples.""" + checks = [ + ('ex5_line-of-sight_solution', + [r'RAJA sequential', r'RAJA OpenMP', r'result -- PASS']), + ('ex6_stencil-offset-layout_solution', + [r'RAJA Views \(permuted\)', r'result -- PASS']), + ('ex8_tiled-matrix-transpose_solution', + [r'parallel top inner loop', + r'collapsed inner loops', r'result -- PASS']), + ('kernel-dynamic-tile', [r'Running index', r'(24,24)']), + ('plugin-example', + [r'Launching host kernel for the 10 time']), + ('tut_batched-matrix-multiply', [r'result -- PASS']), + ('wave-eqn', [r'Max Error = 2', r'Evolved solution to time']) + ] + for exe, expected in checks: + reason = 'test: checking output of {0} for {1}' \ + .format(exe, expected) + self.run_test(exe, [], expected, installed=False, + purpose=reason, skip_missing=True, + work_dir=self._extra_tests_path) + + def test(self): + """Perform smoke tests.""" + self._test_examples() diff --git a/var/spack/repos/builtin/packages/serf/package.py b/var/spack/repos/builtin/packages/serf/package.py index 4c762bf9a5..96df6f7579 100644 --- a/var/spack/repos/builtin/packages/serf/package.py +++ b/var/spack/repos/builtin/packages/serf/package.py @@ -63,7 +63,7 @@ class Serf(SConsPackage): return args - def test(self): + def build_test(self): # FIXME: Several test failures: # # There were 14 failures: diff --git a/var/spack/repos/builtin/packages/sqlite/package.py b/var/spack/repos/builtin/packages/sqlite/package.py index b400965b3f..db456cb0d7 100644 --- a/var/spack/repos/builtin/packages/sqlite/package.py +++ b/var/spack/repos/builtin/packages/sqlite/package.py @@ -3,7 +3,6 @@ # # SPDX-License-Identifier: (Apache-2.0 OR MIT) -from spack import * from spack import architecture @@ -129,3 +128,41 @@ class Sqlite(AutotoolsPackage): cc(self.compiler.cc_pic_flag, '-lm', '-shared', 'extension-functions.c', '-o', libraryname) install(libraryname, self.prefix.lib) + + def _test_example(self): + """Ensure a sequence of commands on example db are successful.""" + + test_data_dir = self.test_suite.current_test_data_dir + db_filename = test_data_dir.join('packages.db') + exe = 'sqlite3' + + # Ensure the database only contains one table + expected = 'packages' + reason = 'test: ensuring only table is "{0}"'.format(expected) + self.run_test(exe, [db_filename, '.tables'], expected, installed=True, + purpose=reason, skip_missing=False) + + # Ensure the database dump matches expectations, where special + # characters are replaced with spaces in the expected and actual + # output to avoid pattern errors. + reason = 'test: checking dump output' + expected = get_escaped_text_output(test_data_dir.join('dump.out')) + self.run_test(exe, [db_filename, '.dump'], expected, installed=True, + purpose=reason, skip_missing=False) + + def _test_version(self): + """Perform version check on the installed package.""" + exe = 'sqlite3' + vers_str = str(self.spec.version) + + reason = 'test: ensuring version of {0} is {1}'.format(exe, vers_str) + self.run_test(exe, '-version', vers_str, installed=True, + purpose=reason, skip_missing=False) + + def test(self): + """Perform smoke tests on the installed package.""" + # Perform a simple version check + self._test_version() + + # Run a sequence of operations + self._test_example() diff --git a/var/spack/repos/builtin/packages/sqlite/test/dump.out b/var/spack/repos/builtin/packages/sqlite/test/dump.out new file mode 100644 index 0000000000..3dda19d1c5 --- /dev/null +++ b/var/spack/repos/builtin/packages/sqlite/test/dump.out @@ -0,0 +1,10 @@ +PRAGMA foreign_keys=OFF; +BEGIN TRANSACTION; +CREATE TABLE packages ( +name varchar(80) primary key, +has_code integer, +url varchar(160)); +INSERT INTO packages VALUES('sqlite',1,'https://www.sqlite.org'); +INSERT INTO packages VALUES('readline',1,'https://tiswww.case.edu/php/chet/readline/rltop.html'); +INSERT INTO packages VALUES('xsdk',0,'http://xsdk.info'); +COMMIT; diff --git a/var/spack/repos/builtin/packages/sqlite/test/packages.db b/var/spack/repos/builtin/packages/sqlite/test/packages.db Binary files differnew file mode 100644 index 0000000000..252962235c --- /dev/null +++ b/var/spack/repos/builtin/packages/sqlite/test/packages.db diff --git a/var/spack/repos/builtin/packages/subversion/package.py b/var/spack/repos/builtin/packages/subversion/package.py index 1f5f65215c..98874ee2bd 100644 --- a/var/spack/repos/builtin/packages/subversion/package.py +++ b/var/spack/repos/builtin/packages/subversion/package.py @@ -98,7 +98,7 @@ class Subversion(AutotoolsPackage): perl = spec['perl'].command perl('Makefile.PL', 'INSTALL_BASE={0}'.format(prefix)) - def test(self): + def check(self): make('check') if '+perl' in self.spec: make('check-swig-pl') diff --git a/var/spack/repos/builtin/packages/umpire/package.py b/var/spack/repos/builtin/packages/umpire/package.py index 08b6d487ff..6b39aad5f6 100644 --- a/var/spack/repos/builtin/packages/umpire/package.py +++ b/var/spack/repos/builtin/packages/umpire/package.py @@ -3,8 +3,8 @@ # # SPDX-License-Identifier: (Apache-2.0 OR MIT) - -from spack import * +import llnl.util.lang as lang +import llnl.util.tty as tty class Umpire(CMakePackage, CudaPackage): @@ -115,3 +115,189 @@ class Umpire(CMakePackage, CudaPackage): 'Off' if 'tests=none' in spec else 'On')) return options + + @property + def build_relpath(self): + """Relative path to the cmake build subdirectory.""" + return join_path('..', self.build_dirname) + + @run_after('install') + def setup_build_tests(self): + """Copy the build test files after the package is installed to an + install test subdirectory for use during `spack test run`.""" + # Now copy the relative files + self.cache_extra_test_sources(self.build_relpath) + + # Ensure the path exists since relying on a relative path at the + # same level as the normal stage source path. + mkdirp(self.install_test_root) + + @property + @lang.memoized + def _extra_tests_path(self): + # TODO: The tests should be converted to re-build and run examples + # TODO: using the installed libraries. + return join_path(self.install_test_root, self.build_relpath) + + @property + @lang.memoized + def _has_bad_strategy(self): + return self.spec.satisfies('@0.2.0:0.2.3') + + def _run_checks(self, dirs, checks): + """Run the specified checks in the provided directories.""" + + if not dirs or not checks: + return + + for exe in checks: + if exe == 'strategy_example' and self._has_bad_strategy: + # Skip this test until install testing can properly capture + # the abort associated with this version. + # (An umpire::util::Exception is thrown; status value is -6.) + tty.warn('Skipping {0} test until Spack can handle core dump' + .format(exe)) + continue + + expected, status = checks[exe] + for work_dir in dirs: + src = 'from build ' if 'spack-build' in work_dir else '' + reason = 'test {0} {1}output'.format(exe, src) + self.run_test(exe, [], expected, status, installed=False, + purpose=reason, skip_missing=True, + work_dir=work_dir) + + def _run_bench_checks(self): + """Run the benchmark smoke test checks.""" + tty.info('Running benchmark checks') + + dirs = [] + if self.spec.satisfies('@0.3.3:1.0.1'): + dirs.append(join_path(self._extra_tests_path, 'benchmarks')) + elif self.spec.satisfies('@1.1.0:'): + dirs.append(self.prefix.bin) + + checks = { + # Versions 0.3.3:1.0.1 (spack-build/bin/benchmarks) + # Versions 1.1.0:2.1.0 (spack-build/bin) + 'allocator_benchmarks': ( + ['Malloc/malloc', 'Malloc/free', 'ns', + 'Host/allocate', 'Host/deallocate', + 'FixedPoolHost/allocate', + 'FixedPoolHost/deallocate'], 0), + 'copy_benchmarks': (['benchmark_copy/host_host', 'ns'], 0), + 'debuglog_benchmarks': (['benchmark_DebugLogger', 'ns'], 0), + } + self._run_checks(dirs, checks) + + def _run_cookbook_checks(self): + """Run the cookbook smoke test checks.""" + tty.info('Running cookbook checks') + + dirs = [] + cb_subdir = join_path('examples', 'cookbook') + if self.spec.satisfies('@0.3.3:1.0.1'): + dirs.append(join_path(self._extra_tests_path, cb_subdir)) + elif self.spec.satisfies('@1.1.0'): + dirs.append(join_path(self.prefix.bin, cb_subdir)) + elif self.spec.satisfies('@2.0.0:'): + dirs.append(self.prefix.bin) + + checks = { + # Versions 0.3.3:1.0.1 (spack-build/bin/examples/cookbook) + # Versions 2.0.0:2.1.0 (spack-build/bin) + # Versions 1.1.0 (prefix.bin/examples/cookbook) + # Versions 2.0.0:2.1.0 (prefix.bin) + 'recipe_dynamic_pool_heuristic': (['in the pool', 'releas'], 0), + 'recipe_no_introspection': (['has allocated', 'used'], 0), + } + self._run_checks(dirs, checks) + + def _run_example_checks(self): + """Run the example smoke test checks.""" + tty.info('Running example checks') + + dirs = [] + if self.spec.satisfies('@0.1.3:0.3.1'): + dirs.append(self._extra_tests_path) + elif self.spec.satisfies('@0.3.3:1.0.1'): + dirs.append(join_path(self._extra_tests_path, 'examples')) + elif self.spec.satisfies('@1.1.0'): + dirs.append(join_path(self.prefix.bin, 'examples')) + elif self.spec.satisfies('@2.0.0:'): + dirs.append(self.prefix.bin) + + # Check the results from a subset of the (potentially) available + # executables + checks = { + # Versions 0.1.3:0.3.1 (spack-build/bin) + # Versions 0.3.3:1.0.1 (spack-build/bin/examples) + # Versions 2.0.0:2.1.0 (spack-build/bin) + # Version 1.1.0 (prefix.bin/examples) + # Versions 2.0.0:2.1.0 (prefix.bin) + 'malloc': (['99 should be 99'], 0), + 'strategy_example': (['Available allocators', 'HOST'], 0), + 'vector_allocator': ([''], 0), + } + self._run_checks(dirs, checks) + + def _run_plots_checks(self): + """Run the plots smoke test checks.""" + tty.info('Running plots checks') + + dirs = [self.prefix.bin] if self.spec.satisfies('@0.3.3:0.3.5') else [] + checks = { + # Versions 0.3.3:0.3.5 (prefix.bin) + 'plot_allocations': ([''], 0), + } + self._run_checks(dirs, checks) + + def _run_tools_checks(self): + """Run the tools smoke test checks.""" + tty.info('Running tools checks') + + dirs = [self.prefix.bin] if self.spec.satisfies('@0.3.3:0.3.5') else [] + checks = { + # Versions 0.3.3:0.3.5 (spack-build/bin/tools) + 'replay': (['No input file'], 0), + } + self._run_checks(dirs, checks) + + def _run_tut_checks(self): + """Run the tutorial smoke test checks.""" + tty.info('Running tutorials checks') + + dirs = [] + tut_subdir = join_path('examples', 'tutorial') + if self.spec.satisfies('@0.2.4:0.3.1'): + dirs.append(self._extra_tests_path) + elif self.spec.satisfies('@0.3.3:1.0.1'): + dirs.append(join_path(self._extra_tests_path, tut_subdir)) + elif self.spec.satisfies('@1.1.0'): + dirs.append(join_path(self.prefix.bin, tut_subdir)) + elif self.spec.satisfies('@2.0.0:'): + dirs.append(self.prefix.bin) + + checks = { + # Versions 0.2.4:0.3.1 (spack-build/bin) + # Versions 0.3.3:1.0.1 (spack-build/bin/examples/tutorial) + # Versions 2.0.0:2.1.0 (spack-build/bin) + # Version 1.1.0 (prefix.bin/examples/tutorial) + # Versions 2.0.0:2.1.0 (prefix.bin) + 'tut_copy': (['Copied source data'], 0), + 'tut_introspection': ( + ['Allocator used is HOST', 'size of the allocation'], 0), + 'tut_memset': (['Set data from HOST'], 0), + 'tut_move': (['Moved source data', 'HOST'], 0), + 'tut_reallocate': (['Reallocated data'], 0), + } + self._run_checks(dirs, checks) + + def test(self): + """Perform smoke tests on the installed package.""" + self._run_bench_checks() + self._run_cookbook_checks() + self._run_example_checks() + self._run_plots_checks() + self._run_tools_checks() + self._run_tut_checks() |