diff options
Diffstat (limited to 'lib')
513 files changed, 11068 insertions, 3685 deletions
diff --git a/lib/spack/docs/analyze.rst b/lib/spack/docs/analyze.rst new file mode 100644 index 0000000000..38af77cd7f --- /dev/null +++ b/lib/spack/docs/analyze.rst @@ -0,0 +1,162 @@ +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other + Spack Project Developers. See the top-level COPYRIGHT file for details. + + SPDX-License-Identifier: (Apache-2.0 OR MIT) + +.. _analyze: + +======= +Analyze +======= + + +The analyze command is a front-end to various tools that let us analyze +package installations. Each analyzer is a module for a different kind +of analysis that can be done on a package installation, including (but not +limited to) binary, log, or text analysis. Thus, the analyze command group +allows you to take an existing package install, choose an analyzer, +and extract some output for the package using it. + + +----------------- +Analyzer Metadata +----------------- + +For all analyzers, we write to an ``analyzers`` folder in ``~/.spack``, or the +value that you specify in your spack config at ``config:analyzers_dir``. +For example, here we see the results of running an analysis on zlib: + +.. code-block:: console + + $ tree ~/.spack/analyzers/ + └── linux-ubuntu20.04-skylake + └── gcc-9.3.0 + └── zlib-1.2.11-sl7m27mzkbejtkrajigj3a3m37ygv4u2 + ├── environment_variables + │  └── spack-analyzer-environment-variables.json + ├── install_files + │  └── spack-analyzer-install-files.json + └── libabigail + └── spack-analyzer-libabigail-libz.so.1.2.11.xml + + +This means that you can always find analyzer output in this folder, and it +is organized with the same logic as the package install it was run for. +If you want to customize this top level folder, simply provide the ``--path`` +argument to ``spack analyze run``. The nested organization will be maintained +within your custom root. + +----------------- +Listing Analyzers +----------------- + +If you aren't familiar with Spack's analyzers, you can quickly list those that +are available: + +.. code-block:: console + + $ spack analyze list-analyzers + install_files : install file listing read from install_manifest.json + environment_variables : environment variables parsed from spack-build-env.txt + config_args : config args loaded from spack-configure-args.txt + abigail : Application Binary Interface (ABI) features for objects + + +In the above, the first three are fairly simple - parsing metadata files from +a package install directory to save + +------------------- +Analyzing a Package +------------------- + +The analyze command, akin to install, will accept a package spec to perform +an analysis for. The package must be installed. Let's walk through an example +with zlib. We first ask to analyze it. However, since we have more than one +install, we are asked to disambiguate: + +.. code-block:: console + + $ spack analyze run zlib + ==> Error: zlib matches multiple packages. + Matching packages: + fz2bs56 zlib@1.2.11%gcc@7.5.0 arch=linux-ubuntu18.04-skylake + sl7m27m zlib@1.2.11%gcc@9.3.0 arch=linux-ubuntu20.04-skylake + Use a more specific spec. + + +We can then specify the spec version that we want to analyze: + +.. code-block:: console + + $ spack analyze run zlib/fz2bs56 + +If you don't provide any specific analyzer names, by default all analyzers +(shown in the ``list-analyzers`` subcommand list) will be run. If an analyzer does not +have any result, it will be skipped. For example, here is a result running for +zlib: + +.. code-block:: console + + $ ls ~/.spack/analyzers/linux-ubuntu20.04-skylake/gcc-9.3.0/zlib-1.2.11-sl7m27mzkbejtkrajigj3a3m37ygv4u2/ + spack-analyzer-environment-variables.json + spack-analyzer-install-files.json + spack-analyzer-libabigail-libz.so.1.2.11.xml + +If you want to run a specific analyzer, ask for it with `--analyzer`. Here we run +spack analyze on libabigail (already installed) _using_ libabigail1 + +.. code-block:: console + + $ spack analyze run --analyzer abigail libabigail + + +.. _analyze_monitoring: + +---------------------- +Monitoring An Analysis +---------------------- + +For any kind of analysis, you can +use a `spack monitor <https://github.com/spack/spack-monitor>`_ "Spackmon" +as a server to upload the same run metadata to. You can +follow the instructions in the `spack monitor documentation <https://spack-monitor.readthedocs.org>`_ +to first create a server along with a username and token for yourself. +You can then use this guide to interact with the server. + +You should first export our spack monitor token and username to the environment: + +.. code-block:: console + + $ export SPACKMON_TOKEN=50445263afd8f67e59bd79bff597836ee6c05438 + $ export SPACKMON_USER=spacky + + +By default, the host for your server is expected to be at ``http://127.0.0.1`` +with a prefix of ``ms1``, and if this is the case, you can simply add the +``--monitor`` flag to the install command: + +.. code-block:: console + + $ spack analyze run --monitor wget + +If you need to customize the host or the prefix, you can do that as well: + +.. code-block:: console + + $ spack analyze run --monitor --monitor-prefix monitor --monitor-host https://monitor-service.io wget + +If your server doesn't have authentication, you can skip it: + +.. code-block:: console + + $ spack analyze run --monitor --monitor-disable-auth wget + +Regardless of your choice, when you run analyze on an installed package (whether +it was installed with ``--monitor`` or not, you'll see the results generating as they did +before, and a message that the monitor server was pinged: + +.. code-block:: console + + $ spack analyze --monitor wget + ... + ==> Sending result for wget bin/wget to monitor. diff --git a/lib/spack/docs/basic_usage.rst b/lib/spack/docs/basic_usage.rst index f7def9d439..db69ba42e9 100644 --- a/lib/spack/docs/basic_usage.rst +++ b/lib/spack/docs/basic_usage.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -27,12 +27,18 @@ It is recommended that the following be put in your ``.bashrc`` file: If you do not see colorized output when using ``less -R`` it is because color is being disabled in the piped output. In this case, tell spack to force -colorized output. +colorized output with a flag .. code-block:: console $ spack --color always | less -R +or an environment variable + +.. code-block:: console + + $ SPACK_COLOR=always spack | less -R + -------------------------- Listing available packages -------------------------- @@ -133,8 +139,7 @@ dependencies first. It then fetches the ``mpileaks`` tarball, expands it, verifies that it was downloaded without errors, builds it, and installs it in its own directory under ``$SPACK_ROOT/opt``. You'll see a number of messages from Spack, a lot of build output, and a message -that the package is installed. Add one or more debug options (``-d``) -to get increasingly detailed output. +that the package is installed. .. code-block:: console @@ -151,8 +156,8 @@ to get increasingly detailed output. The last line, with the ``[+]``, indicates where the package is installed. -Add the debug option -- ``spack install -d mpileaks`` -- to get additional -output. +Add the Spack debug option (one or more times) -- ``spack -d install +mpileaks`` -- to get additional (and even more verbose) output. ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Building a specific version @@ -964,7 +969,7 @@ Variants are named options associated with a particular package. They are optional, as each package must provide default values for each variant it makes available. Variants can be specified using a flexible parameter syntax ``name=<value>``. For example, -``spack install libelf debug=True`` will install libelf built with debug +``spack install mercury debug=True`` will install mercury built with debug flags. The names of particular variants available for a package depend on what was provided by the package author. ``spack info <package>`` will provide information on what build variants are available. @@ -972,11 +977,11 @@ provide information on what build variants are available. For compatibility with earlier versions, variants which happen to be boolean in nature can be specified by a syntax that represents turning options on and off. For example, in the previous spec we could have -supplied ``libelf +debug`` with the same effect of enabling the debug +supplied ``mercury +debug`` with the same effect of enabling the debug compile time option for the libelf package. Depending on the package a variant may have any default value. For -``libelf`` here, ``debug`` is ``False`` by default, and we turned it on +``mercury`` here, ``debug`` is ``False`` by default, and we turned it on with ``debug=True`` or ``+debug``. If a variant is ``True`` by default you can turn it off by either adding ``-name`` or ``~name`` to the spec. diff --git a/lib/spack/docs/binary_caches.rst b/lib/spack/docs/binary_caches.rst index 6e8c91c943..17617b1491 100644 --- a/lib/spack/docs/binary_caches.rst +++ b/lib/spack/docs/binary_caches.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/docs/build_settings.rst b/lib/spack/docs/build_settings.rst index 3e7a21c3e8..a0d4445d3f 100644 --- a/lib/spack/docs/build_settings.rst +++ b/lib/spack/docs/build_settings.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/docs/build_systems.rst b/lib/spack/docs/build_systems.rst index 84c608077f..59b9bb643a 100644 --- a/lib/spack/docs/build_systems.rst +++ b/lib/spack/docs/build_systems.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -59,7 +59,9 @@ on these ideas for each distinct build system that Spack supports: build_systems/bundlepackage build_systems/cudapackage + build_systems/inteloneapipackage build_systems/intelpackage + build_systems/rocmpackage build_systems/custompackage For reference, the :py:mod:`Build System API docs <spack.build_systems>` diff --git a/lib/spack/docs/build_systems/autotoolspackage.rst b/lib/spack/docs/build_systems/autotoolspackage.rst index a4c15e5155..479a11b00d 100644 --- a/lib/spack/docs/build_systems/autotoolspackage.rst +++ b/lib/spack/docs/build_systems/autotoolspackage.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -155,7 +155,7 @@ version, this can be done like so: @property def force_autoreconf(self): - return self.version == Version('1.2.3'): + return self.version == Version('1.2.3') ^^^^^^^^^^^^^^^^^^^^^^^ Finding configure flags diff --git a/lib/spack/docs/build_systems/bundlepackage.rst b/lib/spack/docs/build_systems/bundlepackage.rst index e22e4b26be..d6429c2519 100644 --- a/lib/spack/docs/build_systems/bundlepackage.rst +++ b/lib/spack/docs/build_systems/bundlepackage.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/docs/build_systems/cmakepackage.rst b/lib/spack/docs/build_systems/cmakepackage.rst index 76e89c80b1..a3282514e8 100644 --- a/lib/spack/docs/build_systems/cmakepackage.rst +++ b/lib/spack/docs/build_systems/cmakepackage.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/docs/build_systems/cudapackage.rst b/lib/spack/docs/build_systems/cudapackage.rst index 15150599d3..2d148d0483 100644 --- a/lib/spack/docs/build_systems/cudapackage.rst +++ b/lib/spack/docs/build_systems/cudapackage.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -9,35 +9,120 @@ CudaPackage ----------- -Different from other packages, ``CudaPackage`` does not represent a build -system. Instead its goal is to simplify and unify usage of ``CUDA`` in other -packages. +Different from other packages, ``CudaPackage`` does not represent a build system. +Instead its goal is to simplify and unify usage of ``CUDA`` in other packages by providing a ` mixin-class <https://en.wikipedia.org/wiki/Mixin>`__. -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Provided variants and dependencies -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +You can find source for the package at +`<https://github.com/spack/spack/blob/develop/lib/spack/spack/build_systems/cuda.py>`__. -``CudaPackage`` provides ``cuda`` variant (default to ``off``) to enable/disable -``CUDA``, and ``cuda_arch`` variant to optionally specify the architecture. -It also declares dependencies on the ``CUDA`` package ``depends_on('cuda@...')`` -based on the architecture as well as specifies conflicts for certain compiler versions. +^^^^^^^^ +Variants +^^^^^^^^ + +This package provides the following variants: + +* **cuda** + + This variant is used to enable/disable building with ``CUDA``. The default + is disabled (or ``False``). + +* **cuda_arch** + + This variant supports the optional specification of the architecture. + Valid values are maintained in the ``cuda_arch_values`` property and + are the numeric character equivalent of the compute capability version + (e.g., '10' for version 1.0). Each provided value affects associated + ``CUDA`` dependencies and compiler conflicts. + + GPUs and their compute capability versions are listed at + https://developer.nvidia.com/cuda-gpus . + +^^^^^^^^^ +Conflicts +^^^^^^^^^ + +Conflicts are used to prevent builds with known bugs or issues. While +base ``CUDA`` conflicts have been included with this package, you may +want to add more for your software. + +For example, if your package requires ``cuda_arch`` to be specified when +``cuda`` is enabled, you can add the following conflict to your package +to terminate such build attempts with a suitable message: + +.. code-block:: python + + conflicts('cuda_arch=none', when='+cuda', + msg='CUDA architecture is required') + +Similarly, if your software does not support all versions of the property, +you could add ``conflicts`` to your package for those versions. For example, +suppose your software does not work with CUDA compute capability versions +prior to SM 5.0 (``50``). You can add the following code to display a +custom message should a user attempt such a build: + +.. code-block:: python + + unsupported_cuda_archs = [ + '10', '11', '12', '13', + '20', '21', + '30', '32', '35', '37' + ] + for value in unsupported_cuda_archs: + conflicts('cuda_arch={0}'.format(value), when='+cuda', + msg='CUDA architecture {0} is not supported'.format(value)) + +^^^^^^^ +Methods +^^^^^^^ + +This package provides one custom helper method, which is used to build +standard CUDA compiler flags. + +**cuda_flags** + + This built-in static method returns a list of command line flags + for the chosen ``cuda_arch`` value(s). The flags are intended to + be passed to the CUDA compiler driver (i.e., ``nvcc``). + + This method must be explicitly called when you are creating the + arguments for your build in order to use the values. ^^^^^ Usage ^^^^^ -In order to use it, just add another base class to your package, for example: +This helper package can be added to your package by adding it as a base +class of your package. For example, you can add it to your +:ref:`CMakePackage <cmakepackage>`-based package as follows: .. code-block:: python + :emphasize-lines: 1,7-16 - class MyPackage(CMakePackage, CudaPackage): + class MyCudaPackage(CMakePackage, CudaPackage): ... def cmake_args(self): spec = self.spec + args = [] + ... if '+cuda' in spec: - options.append('-DWITH_CUDA=ON') - cuda_arch = spec.variants['cuda_arch'].value + # Set up the cuda macros needed by the build + args.append('-DWITH_CUDA=ON') + cuda_arch_list = spec.variants['cuda_arch'].value + cuda_arch = cuda_arch_list[0] if cuda_arch != 'none': - options.append('-DCUDA_FLAGS=-arch=sm_{0}'.format(cuda_arch[0])) + args.append('-DCUDA_FLAGS=-arch=sm_{0}'.format(cuda_arch)) else: - options.append('-DWITH_CUDA=OFF') + # Ensure build with cuda is disabled + args.append('-DWITH_CUDA=OFF') + ... + return args + +assuming only the ``WITH_CUDA`` and ``CUDA_FLAGS`` flags are required. +You will need to customize options as needed for your build. + +This example also illustrates how to check for the ``cuda`` variant using +``self.spec`` and how to retrieve the ``cuda_arch`` variant's value, which +is a list, using ``self.spec.variants['cuda_arch'].value``. + +With over 70 packages using ``CudaPackage`` as of January 2021 there are +lots of examples to choose from to get more ideas for using this package. diff --git a/lib/spack/docs/build_systems/custompackage.rst b/lib/spack/docs/build_systems/custompackage.rst index 8a20dada92..4cb506acd7 100644 --- a/lib/spack/docs/build_systems/custompackage.rst +++ b/lib/spack/docs/build_systems/custompackage.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -9,7 +9,7 @@ Custom Build Systems -------------------- -While the build systems listed above should meet your needs for the +While the built-in build systems should meet your needs for the vast majority of packages, some packages provide custom build scripts. This guide is intended for the following use cases: @@ -31,7 +31,7 @@ installation. Both of these packages require custom build systems. Base class ^^^^^^^^^^ -If your package does not belong to any of the aforementioned build +If your package does not belong to any of the built-in build systems that Spack already supports, you should inherit from the ``Package`` base class. ``Package`` is a simple base class with a single phase: ``install``. If your package is simple, you may be able @@ -168,7 +168,8 @@ if and only if this flag is set, we would use the following line: Testing ^^^^^^^ -Let's put everything together and add unit tests to our package. +Let's put everything together and add unit tests to be optionally run +during the installation of our package. In the ``perl`` package, we can see: .. code-block:: python @@ -182,12 +183,6 @@ As you can guess, this runs ``make test`` *after* building the package, if and only if testing is requested. Again, this is not specific to custom build systems, it can be added to existing build systems as well. -Ideally, every package in Spack will have some sort of test to ensure -that it was built correctly. It is up to the package authors to make -sure this happens. If you are adding a package for some software and -the developers list commands to test the installation, please add these -tests to your ``package.py``. - .. warning:: The order of decorators matters. The following ordering: @@ -207,3 +202,12 @@ tests to your ``package.py``. the tests will always be run regardless of whether or not ``--test=root`` is requested. See https://github.com/spack/spack/issues/3833 for more information + +Ideally, every package in Spack will have some sort of test to ensure +that it was built correctly. It is up to the package authors to make +sure this happens. If you are adding a package for some software and +the developers list commands to test the installation, please add these +tests to your ``package.py``. + +For more information on other forms of package testing, refer to +:ref:`Checking an installation <checking_an_installation>`. diff --git a/lib/spack/docs/build_systems/inteloneapipackage.rst b/lib/spack/docs/build_systems/inteloneapipackage.rst new file mode 100644 index 0000000000..1e535775e2 --- /dev/null +++ b/lib/spack/docs/build_systems/inteloneapipackage.rst @@ -0,0 +1,155 @@ +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other + Spack Project Developers. See the top-level COPYRIGHT file for details. + + SPDX-License-Identifier: (Apache-2.0 OR MIT) + +.. _inteloneapipackage: + + +==================== + IntelOneapiPackage +==================== + + +.. contents:: + + +oneAPI packages in Spack +======================== + +Spack can install and use the Intel oneAPI products. You may either +use spack to install the oneAPI tools or use the `Intel +installers`_. After installation, you may use the tools directly, or +use Spack to build packages with the tools. + +The Spack Python class ``IntelOneapiPackage`` is a base class that is +used by ``IntelOneapiCompilers``, ``IntelOneapiMkl``, +``IntelOneapiTbb`` and other classes to implement the oneAPI +packages. See the :ref:<package-list> for the full list of available +oneAPI packages or use:: + + spack list -d oneAPI + +For more information on a specific package, do:: + + spack info <package-name> + +Intel no longer releases new versions of Parallel Studio, which can be +used in Spack via the :ref:<intelpackage>. All of its components can +now be found in oneAPI. + +Examples +======== + +Building a Package With icx +--------------------------- + +In this example, we build patchelf with ``icc`` and ``icx``. The +compilers are installed with spack. + +Install the oneAPI compilers:: + + spack install intel-oneapi-compilers + +Add the compilers to your ``compilers.yaml`` so spack can use them:: + + spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/linux/bin/intel64 + spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/linux/bin + +Verify that the compilers are available:: + + spack compiler list + +The ``intel-oneapi-compilers`` package includes 2 families of +compilers: + +* ``intel``: ``icc``, ``icpc``, ``ifort``. Intel's *classic* + compilers. +* ``oneapi``: ``icx``, ``icpx``, ``ifx``. Intel's new generation of + compilers based on LLVM. + +To build the ``patchelf`` Spack package with ``icc``, do:: + + spack install patchelf%intel + +To build with with ``icx``, do :: + + spack install patchelf%oneapi + +Using oneAPI MPI to Satisfy a Virtual Dependence +------------------------------------------------------ + +The ``hdf5`` package works with any compatible MPI implementation. To +build ``hdf5`` with Intel oneAPI MPI do:: + + spack install hdf5 +mpi ^intel-oneapi-mpi + +Using an Externally Installed oneAPI +==================================== + +Spack can also use oneAPI tools that are manually installed with +`Intel Installers`_. The procedures for configuring Spack to use +external compilers and libraries are different. + +Compilers +--------- + +To use the compilers, add some information about the installation to +``compilers.yaml``. For most users, it is sufficient to do:: + + spack compiler add /opt/intel/oneapi/compiler/latest/linux/bin/intel64 + spack compiler add /opt/intel/oneapi/compiler/latest/linux/bin + +Adapt the paths above if you did not install the tools in the default +location. After adding the compilers, using them is the same +as if you had installed the ``intel-oneapi-compilers`` package. +Another option is to manually add the configuration to +``compilers.yaml`` as described in :ref:`Compiler configuration +<compiler-config>`. + +Libraries +--------- + +If you want Spack to use MKL that you have installed without Spack in +the default location, then add the following to +``~/.spack/packages.yaml``, adjusting the version as appropriate:: + + intel-oneapi-mkl: + externals: + - spec: intel-oneapi-mkl@2021.1.1 + prefix: /opt/intel/oneapi/ + + +Using oneAPI Tools Installed by Spack +===================================== + +Spack can be a convenient way to install and configure compilers and +libaries, even if you do not intend to build a Spack package. If you +want to build a Makefile project using Spack-installed oneAPI compilers, +then use spack to configure your environment:: + + spack load intel-oneapi-compilers + +And then you can build with:: + + CXX=icpx make + +You can also use Spack-installed libraries. For example:: + + spack load intel-oneapi-mkl + +Will update your environment CPATH, LIBRARY_PATH, and other +environment variables for building an application with MKL. + +More information +================ + +This section describes basic use of oneAPI, especially if it has +changed compared to Parallel Studio. See :ref:<intelpackage> for more +information on :ref:<intel-virtual-packages>, +:ref:<intel-unrelated-packages>, +:ref:<intel-integrating-external-libraries>, and +:ref:<using-mkl-tips>. + + +.. _`Intel installers`: https://software.intel.com/content/www/us/en/develop/documentation/installation-guide-for-intel-oneapi-toolkits-linux/top.html diff --git a/lib/spack/docs/build_systems/intelpackage.rst b/lib/spack/docs/build_systems/intelpackage.rst index 8594c8d425..9a157c3bb3 100644 --- a/lib/spack/docs/build_systems/intelpackage.rst +++ b/lib/spack/docs/build_systems/intelpackage.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -137,6 +137,7 @@ If you need to save disk space or installation time, you could install the ``intel`` compilers-only subset (0.6 GB) and just the library packages you need, for example ``intel-mpi`` (0.5 GB) and ``intel-mkl`` (2.5 GB). +.. _intel-unrelated-packages: """""""""""""""""""" Unrelated packages @@ -358,6 +359,8 @@ affected by an advanced third method: Next, visit section `Selecting Intel Compilers`_ to learn how to tell Spack to use the newly configured compilers. +.. _intel-integrating-external-libraries: + """""""""""""""""""""""""""""""""" Integrating external libraries """""""""""""""""""""""""""""""""" @@ -558,43 +561,29 @@ follow `the next section <intel-install-libs_>`_ instead. modules: [] spec: intel@18.0.3 paths: - cc: stub - cxx: stub - f77: stub - fc: stub + cc: /usr/bin/true + cxx: /usr/bin/true + f77: /usr/bin/true + fc: /usr/bin/true Replace ``18.0.3`` with the version that you determined in the preceding - step. The contents under ``paths:`` do not matter yet. - - You are right to ask: "Why on earth is that necessary?" [fn8]_. - The answer lies in Spack striving for strict compiler consistency. - Consider what happens without such a pre-declared compiler stub: - Say, you ask Spack to install a particular version - ``intel-parallel-studio@edition.V``. Spack will apply an unrelated compiler - spec to concretize and install your request, resulting in - ``intel-parallel-studio@edition.V %X``. That compiler ``%X`` is not going to - be the version that this new package itself provides. Rather, it would - typically be ``%gcc@...`` in a default Spack installation or possibly indeed - ``%intel@...``, but at a version that precedes ``V``. - - The problem comes to the fore as soon as you try to use any virtual ``mkl`` - or ``mpi`` packages that you would expect to now be provided by - ``intel-parallel-studio@edition.V``. Spack will indeed see those virtual - packages, but only as being tied to the compiler that the package - ``intel-parallel-studio@edition.V`` was concretized with *at installation*. - If you were to install a client package with the new compilers now available - to you, you would naturally run ``spack install foo +mkl %intel@V``, yet - Spack will either complain about ``mkl%intel@V`` being missing (because it - only knows about ``mkl%X``) or it will go and attempt to install *another - instance* of ``intel-parallel-studio@edition.V %intel@V`` so as to match the - compiler spec ``%intel@V`` that you gave for your client package ``foo``. - This will be unexpected and will quickly get annoying because each - reinstallation takes up time and extra disk space. - - To escape this trap, put the compiler stub declaration shown here in place, - then use that pre-declared compiler spec to install the actual package, as - shown next. This approach works because during installation only the - package's own self-sufficient installer will be used, not any compiler. + step. The exact contents under ``paths:`` do not matter yet, but the paths must exist. + + This temporary stub is required such that the ``intel-parallel-studio`` package + can be installed for the ``intel`` compiler (which the package itself is going + to provide after the installation) rather than an arbitrary system compiler. + The paths given in ``cc``, ``cxx``, ``f77``, ``fc`` must exist, but will + never be used to build anything during the installation of ``intel-parallel-studio``. + + The reason for this stub is that ``intel-parallel-studio`` also provides the + ``mpi`` and ``mkl`` packages and when concretizing a spec, Spack ensures + strong consistency of the used compiler across all dependencies: [fn8]_. + Installing a package ``foo +mkl %intel`` will make Spack look for a package + ``mkl %intel``, which can be provided by ``intel-parallel-studio+mkl %intel``, + but not by ``intel-parallel-studio+mkl %gcc``. + + Failure to do so may result in additional installations of ``mkl``, ``intel-mpi`` or + even ``intel-parallel-studio`` as dependencies for other packages. .. _`verify-compiler-anticipated`: @@ -645,11 +634,25 @@ follow `the next section <intel-install-libs_>`_ instead. want to use the ``intel64`` variant. The ``icpc`` and ``ifort`` compilers will be located in the same directory as ``icc``. - * Use the ``modules:`` and/or ``cflags:`` tokens to specify a suitable accompanying + * Make sure to specify ``modules: ['intel-parallel-studio-cluster2018.3-intel-18.0.3-HASH']`` + (with ``HASH`` being the short hash as displayed when running + ``spack find -l intel-parallel-studio@cluster.2018.3`` and the versions adapted accordingly) + to ensure that the correct and complete environment for the Intel compilers gets + loaded when running them. With modern versions of the Intel compiler you may otherwise see + issues about missing libraries. Please also note that module name must exactly match + the name as returned by ``module avail`` (and shown in the example above). + + * Use the ``modules:`` and/or ``cflags:`` tokens to further specify a suitable accompanying ``gcc`` version to help pacify picky client packages that ask for C++ standards more recent than supported by your system-provided ``gcc`` and its ``libstdc++.so``. + * If you specified a custom variant (for example ``+vtune``) you may want to add this as your + preferred variant in the packages configuration for the ``intel-parallel-studio`` package + as described in :ref:`concretization-preferences`. Otherwise you will have to specify + the variant everytime ``intel-parallel-studio`` is being used as ``mkl``, ``fftw`` or ``mpi`` + implementation to avoid pulling in a different variant. + * To set the Intel compilers for default use in Spack, instead of the usual ``%gcc``, follow section `Selecting Intel compilers`_. @@ -834,6 +837,7 @@ for example: compiler: [ intel@18, intel@17, gcc@4.4.7, gcc@4.9.3, gcc@7.3.0, ] +.. _intel-virtual-packages: """""""""""""""""""""""""""""""""""""""""""""""" Selecting libraries to satisfy virtual packages @@ -907,6 +911,7 @@ With the proper installation as detailed above, no special steps should be required when a client package specifically (and thus deliberately) requests an Intel package as dependency, this being one of the target use cases for Spack. +.. _using-mkl-tips: """"""""""""""""""""""""""""""""""""""""""""""" Tips for configuring client packages to use MKL diff --git a/lib/spack/docs/build_systems/makefilepackage.rst b/lib/spack/docs/build_systems/makefilepackage.rst index 406ec8ce89..e78c558a5a 100644 --- a/lib/spack/docs/build_systems/makefilepackage.rst +++ b/lib/spack/docs/build_systems/makefilepackage.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -147,8 +147,10 @@ and a ``filter_file`` method to help with this. For example: def edit(self, spec, prefix): makefile = FileFilter('Makefile') - makefile.filter('CC = gcc', 'CC = cc') - makefile.filter('CXX = g++', 'CC = c++') + makefile.filter(r'^\s*CC\s*=.*', 'CC = ' + spack_cc) + makefile.filter(r'^\s*CXX\s*=.*', 'CXX = ' + spack_cxx) + makefile.filter(r'^\s*F77\s*=.*', 'F77 = ' + spack_f77) + makefile.filter(r'^\s*FC\s*=.*', 'FC = ' + spack_fc) `stream <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/stream/package.py>`_ diff --git a/lib/spack/docs/build_systems/mavenpackage.rst b/lib/spack/docs/build_systems/mavenpackage.rst index 6f833e8bec..7c8c0e3741 100644 --- a/lib/spack/docs/build_systems/mavenpackage.rst +++ b/lib/spack/docs/build_systems/mavenpackage.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/docs/build_systems/mesonpackage.rst b/lib/spack/docs/build_systems/mesonpackage.rst index 75b0e1caae..a7dd7a76fb 100644 --- a/lib/spack/docs/build_systems/mesonpackage.rst +++ b/lib/spack/docs/build_systems/mesonpackage.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -121,11 +121,15 @@ override the ``meson_args`` method like so: .. code-block:: python def meson_args(self): - return ['--default-library=both'] + return ['--warnlevel=3'] This method can be used to pass flags as well as variables. +Note that the ``MesonPackage`` base class already defines variants for +``buildtype``, ``default_library`` and ``strip``, which are mapped to default +Meson arguments, meaning that you don't have to specify these. + ^^^^^^^^^^^^^^^^^^^^^^ External documentation ^^^^^^^^^^^^^^^^^^^^^^ diff --git a/lib/spack/docs/build_systems/octavepackage.rst b/lib/spack/docs/build_systems/octavepackage.rst index 90e582e36c..baccd0c0e7 100644 --- a/lib/spack/docs/build_systems/octavepackage.rst +++ b/lib/spack/docs/build_systems/octavepackage.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/docs/build_systems/perlpackage.rst b/lib/spack/docs/build_systems/perlpackage.rst index 11968e1560..9fc57f6337 100644 --- a/lib/spack/docs/build_systems/perlpackage.rst +++ b/lib/spack/docs/build_systems/perlpackage.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -120,8 +120,6 @@ so ``PerlPackage`` contains: extends('perl') - depends_on('perl', type=('build', 'run')) - If your package requires a specific version of Perl, you should specify this. diff --git a/lib/spack/docs/build_systems/pythonpackage.rst b/lib/spack/docs/build_systems/pythonpackage.rst index ef06725e01..931c58724a 100644 --- a/lib/spack/docs/build_systems/pythonpackage.rst +++ b/lib/spack/docs/build_systems/pythonpackage.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -23,20 +23,11 @@ can be overridden: * ``build_ext`` * ``build_clib`` * ``build_scripts`` -* ``clean`` * ``install`` * ``install_lib`` * ``install_headers`` * ``install_scripts`` * ``install_data`` -* ``sdist`` -* ``register`` -* ``bdist`` -* ``bdist_dumb`` -* ``bdist_rpm`` -* ``bdist_wininst`` -* ``upload`` -* ``check`` These are all standard ``setup.py`` commands and can be found by running: @@ -55,7 +46,7 @@ If for whatever reason you need to run more phases, simply modify your .. code-block:: python - phases = ['build_ext', 'install', 'bdist'] + phases = ['build_ext', 'install'] Each phase provides a function ``<phase>`` that runs: @@ -90,7 +81,7 @@ Instead of using the ``PythonPackage`` base class, you should extend the ``Package`` base class and implement the following custom installation procedure: -.. code-block:: +.. code-block:: python def install(self, spec, prefix): pip = which('pip') @@ -134,9 +125,9 @@ The zip file will not contain a ``setup.py``, but it will contain a ``METADATA`` file which contains all the information you need to write a ``package.py`` build recipe. -^^^^^^^^^^^^^^^^^^^^^^^ -Finding Python packages -^^^^^^^^^^^^^^^^^^^^^^^ +^^^^ +PyPI +^^^^ The vast majority of Python packages are hosted on PyPI - The Python Package Index. ``pip`` only supports packages hosted on PyPI, making @@ -148,6 +139,26 @@ if a newer version is available. The download page is usually at:: https://pypi.org/project/<package-name> + +Since PyPI is so common, the ``PythonPackage`` base class has a +``pypi`` attribute that can be set. Once set, ``pypi`` will be used +to define the ``homepage``, ``url``, and ``list_url``. For example, +the following: + +.. code-block:: python + + homepage = 'https://pypi.org/project/setuptools/' + url = 'https://pypi.org/packages/source/s/setuptools/setuptools-49.2.0.zip' + list_url = 'https://pypi.org/simple/setuptools/' + + +is equivalent to: + +.. code-block:: python + + pypi = 'setuptools/setuptools-49.2.0.zip' + + ^^^^^^^^^^^ Description ^^^^^^^^^^^ @@ -184,50 +195,11 @@ also get the homepage on the command-line by running: URL ^^^ -You may have noticed that Spack allows you to add multiple versions of -the same package without adding multiple versions of the download URL. -It does this by guessing what the version string in the URL is and -replacing this with the requested version. Obviously, if Spack cannot -guess the version correctly, or if non-version-related things change -in the URL, Spack cannot substitute the version properly. - -Once upon a time, PyPI offered nice, simple download URLs like:: - - https://pypi.python.org/packages/source/n/numpy/numpy-1.13.1.zip - - -As you can see, the version is 1.13.1. It probably isn't hard to guess -what URL to use to download version 1.12.0, and Spack was perfectly -capable of performing this calculation. - -However, PyPI switched to a new download URL format:: - - https://pypi.python.org/packages/c0/3a/40967d9f5675fbb097ffec170f59c2ba19fc96373e73ad47c2cae9a30aed/numpy-1.13.1.zip#md5=2c3c0f4edf720c3a7b525dacc825b9ae - - -and more recently:: - - https://files.pythonhosted.org/packages/b0/2b/497c2bb7c660b2606d4a96e2035e92554429e139c6c71cdff67af66b58d2/numpy-1.14.3.zip - - -As you can imagine, it is impossible for Spack to guess what URL to -use to download version 1.12.0 given this URL. There is a solution, -however. PyPI offers a new hidden interface for downloading -Python packages that does not include a hash in the URL:: - - https://pypi.io/packages/source/n/numpy/numpy-1.13.1.zip - - -This URL redirects to the https://files.pythonhosted.org URL. The general -syntax for this https://pypi.io URL is:: - - https://pypi.io/packages/<type>/<first-letter-of-name>/<name>/<name>-<version>.<extension> - - -Please use the https://pypi.io URL instead of the https://pypi.python.org -URL. If both ``.tar.gz`` and ``.zip`` versions are available, ``.tar.gz`` -is preferred. If some releases offer both ``.tar.gz`` and ``.zip`` versions, -but some only offer ``.zip`` versions, use ``.zip``. +If ``pypi`` is set as mentioned above, ``url`` and ``list_url`` will +be automatically set for you. If both ``.tar.gz`` and ``.zip`` versions +are available, ``.tar.gz`` is preferred. If some releases offer both +``.tar.gz`` and ``.zip`` versions, but some only offer ``.zip`` versions, +use ``.zip``. Some Python packages are closed-source and do not ship ``.tar.gz`` or ``.zip`` files on either PyPI or GitHub. If this is the case, you can still download @@ -237,10 +209,9 @@ and can be downloaded from:: https://pypi.io/packages/py3/a/azureml_sdk/azureml_sdk-1.11.0-py3-none-any.whl -Note that instead of ``<type>`` being ``source``, it is now ``py3`` since this -wheel will work for any generic version of Python 3. You may see Python-specific -or OS-specific URLs. Note that when you add a ``.whl`` URL, you should add -``expand=False`` to ensure that Spack doesn't try to extract the wheel: +You may see Python-specific or OS-specific URLs. Note that when you add a +``.whl`` URL, you should add ``expand=False`` to ensure that Spack doesn't +try to extract the wheel: .. code-block:: python @@ -255,7 +226,7 @@ Many packages are hosted on PyPI, but are developed on GitHub or another version control systems. The tarball can be downloaded from either location, but PyPI is preferred for the following reasons: -#. PyPI contains the bare minimum of files to install the package. +#. PyPI contains the bare minimum number of files needed to install the package. You may notice that the tarball you download from PyPI does not have the same checksum as the tarball you download from GitHub. @@ -292,19 +263,6 @@ location, but PyPI is preferred for the following reasons: PyPI is nice because it makes it physically impossible to re-release the same version of a package with a different checksum. -There are some reasons to prefer downloading from GitHub: - -#. The GitHub tarball may contain unit tests. - - As previously mentioned, the PyPI tarball contains the bare minimum - of files to install the package. Unless explicitly specified by the - developers, it will not contain development files like unit tests. - If you desire to run the unit tests during installation, you should - use the GitHub tarball instead. - -If you really want to run these unit tests, no one will stop you from -submitting a PR for a new package that downloads from GitHub. - ^^^^^^^^^^^^^^^^^^^^^^^^^ Build system dependencies ^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -569,7 +527,8 @@ If the package uses ``setuptools``, check for the following clues: These are packages that are required to run the unit tests for the package. These dependencies can be specified using the - ``type='test'`` dependency type. + ``type='test'`` dependency type. However, the PyPI tarballs rarely + contain unit tests, so there is usually no reason to add these. In the root directory of the package, you may notice a ``requirements.txt`` file. It may look like this file contains a list @@ -625,7 +584,8 @@ add run-time dependencies if they aren't needed, so you need to determine whether or not setuptools is needed. Grep the installation directory for any files containing a reference to ``setuptools`` or ``pkg_resources``. Both modules come from ``py-setuptools``. -``pkg_resources`` is particularly common in scripts in ``prefix/bin``. +``pkg_resources`` is particularly common in scripts found in +``prefix/bin``. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Passing arguments to setup.py @@ -667,7 +627,8 @@ adds: Testing ^^^^^^^ -``PythonPackage`` provides a couple of options for testing packages. +``PythonPackage`` provides a couple of options for testing packages +both during and after the installation process. """""""""""" Import tests @@ -699,48 +660,74 @@ a "package" is a directory containing files like: foo/baz.py -whereas a "module" is a single Python file. Since ``find_packages`` -only returns packages, you'll have to determine the correct module -names yourself. You can now add these packages and modules to the -package like so: +whereas a "module" is a single Python file. + +The ``PythonPackage`` base class automatically detects these module +names for you. If, for whatever reason, the module names detected +are wrong, you can provide the names yourself by overriding +``import_modules`` like so: .. code-block:: python import_modules = ['six'] -When you run ``spack install --test=root py-six``, Spack will attempt -to import the ``six`` module after installation. +Sometimes the list of module names to import depends on how the +package was built. For example, the ``py-pyyaml`` package has a +``+libyaml`` variant that enables the build of a faster optimized +version of the library. If the user chooses ``~libyaml``, only the +``yaml`` library will be importable. If the user chooses ``+libyaml``, +both the ``yaml`` and ``yaml.cyaml`` libraries will be available. +This can be expressed like so: + +.. code-block:: python -These tests most often catch missing dependencies and non-RPATHed + @property + def import_modules(self): + modules = ['yaml'] + + if '+libyaml' in self.spec: + modules.append('yaml.cyaml') + + return modules + + +These tests often catch missing dependencies and non-RPATHed libraries. Make sure not to add modules/packages containing the word -"test", as these likely won't end up in installation directory. +"test", as these likely won't end up in the installation directory, +or may require test dependencies like pytest to be installed. + +Import tests can be run during the installation using ``spack install +--test=root`` or at any time after the installation using +``spack test run``. """""""""" Unit tests """""""""" -The package you want to install may come with additional unit tests. -By default, Spack runs: +The package may have its own unit or regression tests. Spack can +run these tests during the installation by adding phase-appropriate +test methods. -.. code-block:: console +For example, ``py-numpy`` adds the following as a check to run +after the ``install`` phase: - $ python setup.py test +.. code-block:: python + @run_after('install') + @on_package_attributes(run_tests=True) + def install_test(self): + with working_dir('spack-test', create=True): + python('-c', 'import numpy; numpy.test("full", verbose=2)') -if it detects that the ``setup.py`` file supports a ``test`` phase. -You can add additional build-time or install-time tests by overriding -``test`` or adding a custom install-time test function. For example, -``py-numpy`` adds: -.. code-block:: python +when testing is enabled during the installation (i.e., ``spack install +--test=root``). - install_time_test_callbacks = ['install_test', 'import_module_test'] - - def install_test(self): - with working_dir('..'): - python('-c', 'import numpy; numpy.test("full", verbose=2)') +.. note:: + Additional information is available on :ref:`install phase tests + <install_phase-tests>`. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Setup file in a sub-directory @@ -781,7 +768,7 @@ PythonPackage vs. packages that use Python ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ There are many packages that make use of Python, but packages that depend -on Python are not necessarily ``PythonPackages``. +on Python are not necessarily ``PythonPackage``'s. """"""""""""""""""""""" Choosing a build system @@ -878,8 +865,8 @@ and ``pip`` may be a perfectly valid alternative to using Spack. The main advantage of Spack over ``pip`` is its ability to compile non-Python dependencies. It can also build cythonized versions of a package or link to an optimized BLAS/LAPACK library like MKL, -resulting in calculations that run orders of magnitude faster. -Spack does not offer a significant advantage to other python-management +resulting in calculations that run orders of magnitudes faster. +Spack does not offer a significant advantage over other python-management systems for installing and using tools like flake8 and sphinx. But if you need packages with non-Python dependencies like numpy and scipy, Spack will be very valuable to you. diff --git a/lib/spack/docs/build_systems/qmakepackage.rst b/lib/spack/docs/build_systems/qmakepackage.rst index 81d0f7f798..6e85d2bbcd 100644 --- a/lib/spack/docs/build_systems/qmakepackage.rst +++ b/lib/spack/docs/build_systems/qmakepackage.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -108,6 +108,19 @@ override the ``qmake_args`` method like so: This method can be used to pass flags as well as variables. +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +``*.pro`` file in a sub-directory +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If the ``*.pro`` file used to tell QMake how to build the package is +found in a sub-directory, you can tell Spack to run all phases in this +sub-directory by adding the following to the package: + +.. code-block:: python + + build_directory = 'src' + + ^^^^^^^^^^^^^^^^^^^^^^ External documentation ^^^^^^^^^^^^^^^^^^^^^^ diff --git a/lib/spack/docs/build_systems/rocmpackage.rst b/lib/spack/docs/build_systems/rocmpackage.rst new file mode 100644 index 0000000000..e322194adf --- /dev/null +++ b/lib/spack/docs/build_systems/rocmpackage.rst @@ -0,0 +1,122 @@ +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other + Spack Project Developers. See the top-level COPYRIGHT file for details. + + SPDX-License-Identifier: (Apache-2.0 OR MIT) + +.. _rocmpackage: + +----------- +ROCmPackage +----------- + +The ``ROCmPackage`` is not a build system but a helper package. Like ``CudaPackage``, +it provides standard variants, dependencies, and conflicts to facilitate building +packages using GPUs though for AMD in this case. + +You can find the source for this package (and suggestions for setting up your +``compilers.yaml`` and ``packages.yaml`` files) at +`<https://github.com/spack/spack/blob/develop/lib/spack/spack/build_systems/rocm.py>`__. + +^^^^^^^^ +Variants +^^^^^^^^ + +This package provides the following variants: + +* **rocm** + + This variant is used to enable/disable building with ``rocm``. + The default is disabled (or ``False``). + +* **amdgpu_target** + + This variant supports the optional specification of the AMD GPU architecture. + Valid values are the names of the GPUs (e.g., ``gfx701``), which are maintained + in the ``amdgpu_targets`` property. + +^^^^^^^^^^^^ +Dependencies +^^^^^^^^^^^^ + +This package defines basic ``rocm`` dependencies, including ``llvm`` and ``hip``. + +^^^^^^^^^ +Conflicts +^^^^^^^^^ + +Conflicts are used to prevent builds with known bugs or issues. This package +already requires that the ``amdgpu_target`` always be specified for ``rocm`` +builds. It also defines a conflict that prevents builds with an ``amdgpu_target`` +when ``rocm`` is disabled. + +Refer to `Conflicts <https://spack.readthedocs.io/en/latest/packaging_guide.html?highlight=conflicts#conflicts>`__ +for more information on package conflicts. + +^^^^^^^ +Methods +^^^^^^^ + +This package provides one custom helper method, which is used to build +standard AMD hip compiler flags. + +**hip_flags** + + This built-in static method returns the appropriately formatted + ``--amdgpu-target`` build option for ``hipcc``. + + This method must be explicitly called when you are creating the + arguments for your build in order to use the values. + +^^^^^ +Usage +^^^^^ + +This helper package can be added to your package by adding it as a base +class of your package. For example, you can add it to your +:ref:`CMakePackage <cmakepackage>`-based package as follows: + +.. code-block:: python + :emphasize-lines: 1,3-7,14-25 + + class MyRocmPackage(CMakePackage, ROCmPackage): + ... + # Ensure +rocm and amdgpu_targets are passed to dependencies + depends_on('mydeppackage', when='+rocm') + for val in ROCmPackage.amdgpu_targets: + depends_on('mydeppackage amdgpu_target={0}'.format(val), + when='amdgpu_target={0}'.format(val)) + ... + + def cmake_args(self): + spec = self.spec + args = [] + ... + if '+rocm' in spec: + # Set up the hip macros needed by the build + args.extend([ + '-DENABLE_HIP=ON', + '-DHIP_ROOT_DIR={0}'.format(spec['hip'].prefix]) + rocm_archs = spec.variants['amdgpu_target'].value + if 'none' not in rocm_archs: + args.append('-DHIP_HIPCC_FLAGS=--amdgpu-target={0}' + .format(",".join(rocm_archs))) + else: + # Ensure build with hip is disabled + args.append('-DENABLE_HIP=OFF') + ... + return args + ... + +assuming only on the ``ENABLE_HIP``, ``HIP_ROOT_DIR``, and ``HIP_HIPCC_FLAGS`` +macros are required to be set and the only dependency needing rocm options +is ``mydeppackage``. You will need to customize the flags as needed for your +build. + +This example also illustrates how to check for the ``rocm`` variant using +``self.spec`` and how to retrieve the ``amdgpu_target`` variant's value +using ``self.spec.variants['amdgpu_target'].value``. + +All five packages using ``ROCmPackage`` as of January 2021 also use the +:ref:`CudaPackage <cudapackage>`. So it is worth looking at those packages +to get ideas for creating a package that can support both ``cuda`` and +``rocm``. diff --git a/lib/spack/docs/build_systems/rpackage.rst b/lib/spack/docs/build_systems/rpackage.rst index 60c76816c5..a375b5328c 100644 --- a/lib/spack/docs/build_systems/rpackage.rst +++ b/lib/spack/docs/build_systems/rpackage.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -79,12 +79,14 @@ Description The first thing you'll need to add to your new package is a description. The top of the homepage for ``caret`` lists the following description: - caret: Classification and Regression Training + Classification and Regression Training Misc functions for training and plotting classification and regression models. -You can either use the short description (first line), long description -(second line), or both depending on what you feel is most appropriate. +The first line is a short description (title) and the second line is a long +description. In this case the description is only one line but often the +description is several lines. Spack makes use of both short and long +descriptions and convention is to use both when creating an R package. ^^^^^^^^ Homepage @@ -124,6 +126,67 @@ If you only specify the URL for the latest release, your package will no longer be able to fetch that version as soon as a new release comes out. To get around this, add the archive directory as a ``list_url``. +^^^^^^^^^^^^^^^^^^^^^ +Bioconductor packages +^^^^^^^^^^^^^^^^^^^^^ + +Bioconductor packages are set up in a similar way to CRAN packages, but there +are some very important distinctions. Bioconductor packages can be found at: +https://bioconductor.org/. Bioconductor packages are R packages and so follow +the same packaging scheme as CRAN packages. What is different is that +Bioconductor itself is versioned and released. This scheme, using the +Bioconductor package installer, allows further specification of the minimum +version of R as well as further restrictions on the dependencies between +packages than what is possible with the native R packaging system. Spack can +not replicate these extra features and thus Bioconductor packages in Spack need +to be managed as a group during updates in order to maintain package +consistency with Bioconductor itself. + +Another key difference is that, while previous versions of packages are +available, they are not available from a site that can be programmatically set, +thus a ``list_url`` attribute can not be used. However, each package is also +available in a git repository, with branches corresponding to each Bioconductor +release. Thus, it is always possible to retrieve the version of any package +corresponding to a Bioconductor release simply by fetching the branch that +corresponds to the Bioconductor release of the package repository. For this +reason, spack Bioconductor R packages use the git repository, with the commit +of the respective branch used in the ``version()`` attribute of the package. + +^^^^^^^^^^^^^^^^^^^^^^^^ +cran and bioc attributes +^^^^^^^^^^^^^^^^^^^^^^^^ + +Much like the ``pypi`` attribute for python packages, due to the fact that R +packages are obtained from specific repositories, it is possible to set up shortcut +attributes that can be used to set ``homepage``, ``url``, ``list_url``, and +``git``. For example, the following ``cran`` attribute: + +.. code-block:: python + + cran = 'caret' + +is equivalent to: + +.. code-block:: python + + homepage = 'https://cloud.r-project.org/package=caret' + url = 'https://cloud.r-project.org/src/contrib/caret_6.0-86.tar.gz' + list_url = 'https://cloud.r-project.org/src/contrib/Archive/caret' + +Likewise, the following ``bioc`` attribute: + +.. code-block:: python + + bioc = 'BiocVersion' + +is equivalent to: + +.. code-block:: python + + homepage = 'https://bioconductor.org/packages/BiocVersion/' + git = 'https://git.bioconductor.org/packages/BiocVersion' + + ^^^^^^^^^^^^^^^^^^^^^^^^^ Build system dependencies ^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -138,7 +201,6 @@ every R package needs this, the ``RPackage`` base class contains: .. code-block:: python extends('r') - depends_on('r', type=('build', 'run')) Take a close look at the homepage for ``caret``. If you look at the @@ -157,7 +219,7 @@ R dependencies R packages are often small and follow the classic Unix philosophy of doing one thing well. They are modular and usually depend on several other packages. You may find a single package with over a -hundred dependencies. Luckily, CRAN packages are well-documented +hundred dependencies. Luckily, R packages are well-documented and list all of their dependencies in the following sections: * Depends @@ -298,8 +360,8 @@ like so: .. code-block:: python - def configure_args(self, spec, prefix): - mpi_name = spec['mpi'].name + def configure_args(self): + mpi_name = self.spec['mpi'].name # The type of MPI. Supported values are: # OPENMPI, LAM, MPICH, MPICH2, or CRAY diff --git a/lib/spack/docs/build_systems/rubypackage.rst b/lib/spack/docs/build_systems/rubypackage.rst index d1deaf6fa7..226d0bc4a2 100644 --- a/lib/spack/docs/build_systems/rubypackage.rst +++ b/lib/spack/docs/build_systems/rubypackage.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -113,7 +113,6 @@ the base class contains: .. code-block:: python extends('ruby') - depends_on('ruby', type=('build', 'run')) The ``*.gemspec`` file may contain something like: diff --git a/lib/spack/docs/build_systems/sconspackage.rst b/lib/spack/docs/build_systems/sconspackage.rst index f29d6c5f45..f1f67b76b4 100644 --- a/lib/spack/docs/build_systems/sconspackage.rst +++ b/lib/spack/docs/build_systems/sconspackage.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/docs/build_systems/sippackage.rst b/lib/spack/docs/build_systems/sippackage.rst index ddf9a26ab9..f72d83ca30 100644 --- a/lib/spack/docs/build_systems/sippackage.rst +++ b/lib/spack/docs/build_systems/sippackage.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -93,10 +93,17 @@ in the site-packages directory: $ python >>> import setuptools >>> setuptools.find_packages() - ['QtPy5'] + [ + 'PyQt5', 'PyQt5.QtCore', 'PyQt5.QtGui', 'PyQt5.QtHelp', + 'PyQt5.QtMultimedia', 'PyQt5.QtMultimediaWidgets', 'PyQt5.QtNetwork', + 'PyQt5.QtOpenGL', 'PyQt5.QtPrintSupport', 'PyQt5.QtQml', + 'PyQt5.QtQuick', 'PyQt5.QtSvg', 'PyQt5.QtTest', 'PyQt5.QtWebChannel', + 'PyQt5.QtWebSockets', 'PyQt5.QtWidgets', 'PyQt5.QtXml', + 'PyQt5.QtXmlPatterns' + ] -Large, complex packages like ``QtPy5`` will return a long list of +Large, complex packages like ``py-pyqt5`` will return a long list of packages, while other packages may return an empty list. These packages only install a single ``foo.py`` file. In Python packaging lingo, a "package" is a directory containing files like: @@ -108,21 +115,25 @@ a "package" is a directory containing files like: foo/baz.py -whereas a "module" is a single Python file. Since ``find_packages`` -only returns packages, you'll have to determine the correct module -names yourself. You can now add these packages and modules to the -package like so: +whereas a "module" is a single Python file. + +The ``SIPPackage`` base class automatically detects these module +names for you. If, for whatever reason, the module names detected +are wrong, you can provide the names yourself by overriding +``import_modules`` like so: .. code-block:: python import_modules = ['PyQt5'] -When you run ``spack install --test=root py-pyqt5``, Spack will attempt -to import the ``PyQt5`` module after installation. +These tests often catch missing dependencies and non-RPATHed +libraries. Make sure not to add modules/packages containing the word +"test", as these likely won't end up in the installation directory, +or may require test dependencies like pytest to be installed. -These tests most often catch missing dependencies and non-RPATHed -libraries. +These tests can be triggered by running ``spack install --test=root`` +or by running ``spack test run`` after the installation has finished. ^^^^^^^^^^^^^^^^^^^^^^ External documentation diff --git a/lib/spack/docs/build_systems/wafpackage.rst b/lib/spack/docs/build_systems/wafpackage.rst index 36fc21a772..1bb879e4f2 100644 --- a/lib/spack/docs/build_systems/wafpackage.rst +++ b/lib/spack/docs/build_systems/wafpackage.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/docs/chain.rst b/lib/spack/docs/chain.rst index d84db87ff3..ddebbbeec3 100644 --- a/lib/spack/docs/chain.rst +++ b/lib/spack/docs/chain.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/docs/conf.py b/lib/spack/docs/conf.py index e19dd05511..2de12a5a99 100644 --- a/lib/spack/docs/conf.py +++ b/lib/spack/docs/conf.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -128,7 +128,7 @@ master_doc = 'index' # General information about the project. project = u'Spack' -copyright = u'2013-2019, Lawrence Livermore National Laboratory.' +copyright = u'2013-2021, Lawrence Livermore National Laboratory.' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the diff --git a/lib/spack/docs/config_yaml.rst b/lib/spack/docs/config_yaml.rst index bcb41527a9..631b8f6b1b 100644 --- a/lib/spack/docs/config_yaml.rst +++ b/lib/spack/docs/config_yaml.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -202,21 +202,23 @@ of builds. Unless overridden in a package or on the command line, Spack builds all packages in parallel. The default parallelism is equal to the number of -cores on your machine, up to 16. Parallelism cannot exceed the number of -cores available on the host. For a build system that uses Makefiles, this -means running: +cores available to the process, up to 16 (the default of ``build_jobs``). +For a build system that uses Makefiles, this ``spack install`` runs: - ``make -j<build_jobs>``, when ``build_jobs`` is less than the number of - cores on the machine + cores available - ``make -j<ncores>``, when ``build_jobs`` is greater or equal to the - number of cores on the machine + number of cores available If you work on a shared login node or have a strict ulimit, it may be necessary to set the default to a lower value. By setting ``build_jobs`` to 4, for example, commands like ``spack install`` will run ``make -j4`` -instead of hogging every core. +instead of hogging every core. To build all software in serial, +set ``build_jobs`` to 1. -To build all software in serial, set ``build_jobs`` to 1. +Note that specifying the number of jobs on the command line always takes +priority, so that ``spack install -j<n>`` always runs `make -j<n>`, even +when that exceeds the number of cores available. -------------------- ``ccache`` diff --git a/lib/spack/docs/configuration.rst b/lib/spack/docs/configuration.rst index 41c9de79f9..10b442ab74 100644 --- a/lib/spack/docs/configuration.rst +++ b/lib/spack/docs/configuration.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -78,6 +78,13 @@ are six configuration scopes. From lowest to highest: If multiple scopes are listed on the command line, they are ordered from lowest to highest precedence. +#. **environment**: When using Spack :ref:`environments`, Spack reads + additional configuration from the environment file. See + :ref:`environment-configuration` for further details on these + scopes. Environment scopes can be referenced from the command line + as ``env:name`` (to reference environment ``foo``, use + ``env:foo``). + #. **command line**: Build settings specified on the command line take precedence over all other scopes. @@ -192,10 +199,11 @@ with MPICH. You can create different configuration scopes for use with Platform-specific Scopes ------------------------ -For each scope above, there can also be platform-specific settings. -For example, on most platforms, GCC is the preferred compiler. -However, on macOS (darwin), Clang often works for more packages, -and is set as the default compiler. This configuration is set in +For each scope above (excluding environment scopes), there can also be +platform-specific settings. For example, on most platforms, GCC is +the preferred compiler. However, on macOS (darwin), Clang often works +for more packages, and is set as the default compiler. This +configuration is set in ``$(prefix)/etc/spack/defaults/darwin/packages.yaml``. It will take precedence over settings in the ``defaults`` scope, but can still be overridden by settings in ``system``, ``system/darwin``, ``site``, diff --git a/lib/spack/docs/containers.rst b/lib/spack/docs/containers.rst index f5bd5602bc..e88b55a226 100644 --- a/lib/spack/docs/containers.rst +++ b/lib/spack/docs/containers.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -227,7 +227,7 @@ following ``spack.yaml``: container: images: - os: centos/7 + os: centos:7 spack: 0.15.4 uses ``spack/centos7:0.15.4`` and ``centos:7`` for the stages where the diff --git a/lib/spack/docs/contribution_guide.rst b/lib/spack/docs/contribution_guide.rst index 8df8ad65ba..f4527355c4 100644 --- a/lib/spack/docs/contribution_guide.rst +++ b/lib/spack/docs/contribution_guide.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -179,24 +179,26 @@ how to write tests! run the unit tests yourself, we suggest you use ``spack unit-test``. ^^^^^^^^^^^^ -Flake8 Tests +Style Tests ^^^^^^^^^^^^ Spack uses `Flake8 <http://flake8.pycqa.org/en/latest/>`_ to test for -`PEP 8 <https://www.python.org/dev/peps/pep-0008/>`_ conformance. PEP 8 is +`PEP 8 <https://www.python.org/dev/peps/pep-0008/>`_ conformance and +`mypy <https://mypy.readthedocs.io/en/stable/>` for type checking. PEP 8 is a series of style guides for Python that provide suggestions for everything from variable naming to indentation. In order to limit the number of PRs that were mostly style changes, we decided to enforce PEP 8 conformance. Your PR -needs to comply with PEP 8 in order to be accepted. +needs to comply with PEP 8 in order to be accepted, and if it modifies the +spack library it needs to successfully type-check with mypy as well. -Testing for PEP 8 compliance is easy. Simply run the ``spack flake8`` +Testing for compliance with spack's style is easy. Simply run the ``spack style`` command: .. code-block:: console - $ spack flake8 + $ spack style -``spack flake8`` has a couple advantages over running ``flake8`` by hand: +``spack style`` has a couple advantages over running the tools by hand: #. It only tests files that you have modified since branching off of ``develop``. @@ -207,7 +209,9 @@ command: checks. For example, URLs are often longer than 80 characters, so we exempt them from line length checks. We also exempt lines that start with "homepage", "url", "version", "variant", "depends_on", and - "extends" in ``package.py`` files. + "extends" in ``package.py`` files. This is now also possible when directly + running flake8 if you can use the ``spack`` formatter plugin included with + spack. More approved flake8 exemptions can be found `here <https://github.com/spack/spack/blob/develop/.flake8>`_. @@ -240,13 +244,13 @@ However, if you aren't compliant with PEP 8, flake8 will complain: Most of the error messages are straightforward, but if you don't understand what they mean, just ask questions about them when you submit your PR. The line numbers -will change if you add or delete lines, so simply run ``spack flake8`` again +will change if you add or delete lines, so simply run ``spack style`` again to update them. .. tip:: Try fixing flake8 errors in reverse order. This eliminates the need for - multiple runs of ``spack flake8`` just to re-compute line numbers and + multiple runs of ``spack style`` just to re-compute line numbers and makes it much easier to fix errors directly off of the CI output. .. warning:: diff --git a/lib/spack/docs/developer_guide.rst b/lib/spack/docs/developer_guide.rst index b4b8b5d321..7ae6938eda 100644 --- a/lib/spack/docs/developer_guide.rst +++ b/lib/spack/docs/developer_guide.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -106,11 +106,21 @@ with a high level view of Spack's directory structure: external/ <- external libs included in Spack distro llnl/ <- some general-use libraries - spack/ <- spack module; contains Python code - cmd/ <- each file in here is a spack subcommand - compilers/ <- compiler description files - test/ <- unit test modules - util/ <- common code + spack/ <- spack module; contains Python code + analyzers/ <- modules to run analysis on installed packages + build_systems/ <- modules for different build systems + cmd/ <- each file in here is a spack subcommand + compilers/ <- compiler description files + container/ <- module for spack containerize + hooks/ <- hook modules to run at different points + modules/ <- modules for lmod, tcl, etc. + operating_systems/ <- operating system modules + platforms/ <- different spack platforms + reporters/ <- reporters like cdash, junit + schema/ <- schemas to validate data structures + solver/ <- the spack solver + test/ <- unit test modules + util/ <- common code Spack is designed so that it could live within a `standard UNIX directory hierarchy <http://linux.die.net/man/7/hier>`_, so ``lib``, @@ -251,6 +261,22 @@ Unit tests This is a fake package hierarchy used to mock up packages for Spack's test suite. +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Research and Monitoring Modules +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +:mod:`spack.monitor` + Contains :class:`SpackMonitor <spack.monitor.SpackMonitor>`. This is accessed + from the ``spack install`` and ``spack analyze`` commands to send build + and package metadada up to a `Spack Monitor <https://github.com/spack/spack-monitor>`_ server. + + +:mod:`spack.analyzers` + A module folder with a :class:`AnalyzerBase <spack.analyzers.analyzer_base.AnalyzerBase>` + that provides base functions to run, save, and (optionally) upload analysis + results to a `Spack Monitor <https://github.com/spack/spack-monitor>`_ server. + + ^^^^^^^^^^^^^ Other Modules ^^^^^^^^^^^^^ @@ -299,6 +325,235 @@ Conceptually, packages are overloaded. They contain: Stage objects ------------- + +.. _writing-analyzers: + +----------------- +Writing analyzers +----------------- + +To write an analyzer, you should add a new python file to the +analyzers module directory at ``lib/spack/spack/analyzers`` . +Your analyzer should be a subclass of the :class:`AnalyzerBase <spack.analyzers.analyzer_base.AnalyzerBase>`. For example, if you want +to add an analyzer class ``Myanalyzer`` you woul write to +``spack/analyzers/myanalyzer.py`` and import and +use the base as follows: + +.. code-block:: python + + from .analyzer_base import AnalyzerBase + + class Myanalyzer(AnalyzerBase): + + +Note that the class name is your module file name, all lowercase +except for the first capital letter. You can look at other analyzers in +that analyzer directory for examples. The guide here will tell you about the basic functions needed. + +^^^^^^^^^^^^^^^^^^^^^^^^^ +Analyzer Output Directory +^^^^^^^^^^^^^^^^^^^^^^^^^ + +By default, when you run ``spack analyze run`` an analyzer output directory will +be created in your spack user directory in your ``$HOME``. The reason we output here +is because the install directory might not always be writable. + +.. code-block:: console + + ~/.spack/ + analyzers + +Result files will be written here, organized in subfolders in the same structure +as the package, with each analyzer owning it's own subfolder. for example: + + +.. code-block:: console + + $ tree ~/.spack/analyzers/ + /home/spackuser/.spack/analyzers/ + └── linux-ubuntu20.04-skylake + └── gcc-9.3.0 + └── zlib-1.2.11-sl7m27mzkbejtkrajigj3a3m37ygv4u2 + ├── environment_variables + │  └── spack-analyzer-environment-variables.json + ├── install_files + │  └── spack-analyzer-install-files.json + └── libabigail + └── lib + └── spack-analyzer-libabigail-libz.so.1.2.11.xml + + +Notice that for the libabigail analyzer, since results are generated per object, +we honor the object's folder in case there are equivalently named files in +different folders. The result files are typically written as json so they can be easily read and uploaded in a future interaction with a monitor. + + +^^^^^^^^^^^^^^^^^ +Analyzer Metadata +^^^^^^^^^^^^^^^^^ + +Your analyzer is required to have the class attributes ``name``, ``outfile``, +and ``description``. These are printed to the user with they use the subcommand +``spack analyze list-analyzers``. Here is an example. +As we mentioned above, note that this analyzer would live in a module named +``libabigail.py`` in the analyzers folder so that the class can be discovered. + + +.. code-block:: python + + class Libabigail(AnalyzerBase): + + name = "libabigail" + outfile = "spack-analyzer-libabigail.json" + description = "Application Binary Interface (ABI) features for objects" + + +This means that the name and output file should be unique for your analyzer. +Note that "all" cannot be the name of an analyzer, as this key is used to indicate +that the user wants to run all analyzers. + +.. _analyzer_run_function: + + +^^^^^^^^^^^^^^^^^^^^^^^^ +An analyzer run Function +^^^^^^^^^^^^^^^^^^^^^^^^ + +The core of an analyzer is its ``run()`` function, which should accept no +arguments. You can assume your analyzer has the package spec of interest at ``self.spec`` +and it's up to the run function to generate whatever analysis data you need, +and then return the object with a key as the analyzer name. The result data +should be a list of objects, each with a name, ``analyzer_name``, ``install_file``, +and one of ``value`` or ``binary_value``. The install file should be for a relative +path, and not the absolute path. For example, let's say we extract a metric called +``metric`` for ``bin/wget`` using our analyzer ``thebest-analyzer``. +We might have data that looks like this: + +.. code-block:: python + + result = {"name": "metric", "analyzer_name": "thebest-analyzer", "value": "1", "install_file": "bin/wget"} + + +We'd then return it as follows - note that they key is the analyzer name at ``self.name``. + +.. code-block:: python + + return {self.name: result} + +This will save the complete result to the analyzer metadata folder, as described +previously. If you want support for adding a different kind of metadata (e.g., +not associated with an install file) then the monitor server would need to be updated +to support this first. + + +^^^^^^^^^^^^^^^^^^^^^^^^^ +An analyzer init Function +^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you don't need any extra dependencies or checks, you can skip defining an analyzer +init function, as the base class will handle it. Typically, it will accept +a spec, and an optional output directory (if the user does not want the default +metadata folder for analyzer results). The analyzer init function should call +it's parent init, and then do any extra checks or validation that are required to +work. For example: + +.. code-block:: python + + def __init__(self, spec, dirname=None): + super(Myanalyzer, self).__init__(spec, dirname) + + # install extra dependencies, do extra preparation and checks here + + +At the end of the init, you will have available to you: + + - **self.spec**: the spec object + - **self.dirname**: an optional directory name the user as provided at init to save + - **self.output_dir**: the analyzer metadata directory, where we save by default + - **self.meta_dir**: the path to the package metadata directory (.spack) if you need it + +And can proceed to write your analyzer. + + +^^^^^^^^^^^^^^^^^^^^^^^ +Saving Analyzer Results +^^^^^^^^^^^^^^^^^^^^^^^ + +The analyzer will have ``save_result`` called, with the result object generated +to save it to the filesystem, and if the user has added the ``--monitor`` flag +to upload it to a monitor server. If your result follows an accepted result +format and you don't need to parse it further, you don't need to add this +function to your class. However, if your result data is large or otherwise +needs additional parsing, you can define it. If you define the function, it +is useful to know about the ``output_dir`` property, which you can join +with your output file relative path of choice: + +.. code-block:: python + + outfile = os.path.join(self.output_dir, "my-output-file.txt") + + +The directory will be provided by the ``output_dir`` property but it won't exist, +so you should create it: + + +.. code::block:: python + + # Create the output directory + if not os.path.exists(self._output_dir): + os.makedirs(self._output_dir) + + +If you are generating results that match to specific files in the package +install directory, you should try to maintain those paths in the case that +there are equivalently named files in different directories that would +overwrite one another. As an example of an analyzer with a custom save, +the Libabigail analyzer saves ``*.xml`` files to the analyzer metadata +folder in ``run()``, as they are either binaries, or as xml (text) would +usually be too big to pass in one request. For this reason, the files +are saved during ``run()`` and the filenames added to the result object, +and then when the result object is passed back into ``save_result()``, +we skip saving to the filesystem, and instead read the file and send +each one (separately) to the monitor: + + +.. code-block:: python + + def save_result(self, result, monitor=None, overwrite=False): + """ABI results are saved to individual files, so each one needs to be + read and uploaded. Result here should be the lookup generated in run(), + the key is the analyzer name, and each value is the result file. + We currently upload the entire xml as text because libabigail can't + easily read gzipped xml, but this will be updated when it can. + """ + if not monitor: + return + + name = self.spec.package.name + + for obj, filename in result.get(self.name, {}).items(): + + # Don't include the prefix + rel_path = obj.replace(self.spec.prefix + os.path.sep, "") + + # We've already saved the results to file during run + content = spack.monitor.read_file(filename) + + # A result needs an analyzer, value or binary_value, and name + data = {"value": content, "install_file": rel_path, "name": "abidw-xml"} + tty.info("Sending result for %s %s to monitor." % (name, rel_path)) + monitor.send_analyze_metadata(self.spec.package, {"libabigail": [data]}) + + + +Notice that this function, if you define it, requires a result object (generated by +``run()``, a monitor (if you want to send), and a boolean ``overwrite`` to be used +to check if a result exists first, and not write to it if the result exists and +overwrite is False. Also notice that since we already saved these files to the analyzer metadata folder, we return early if a monitor isn't defined, because this function serves to send results to the monitor. If you haven't saved anything to the analyzer metadata folder +yet, you might want to do that here. You should also use ``tty.info`` to give +the user a message of "Writing result to $DIRNAME." + + .. _writing-commands: ---------------- @@ -345,6 +600,183 @@ Whenever you add/remove/rename a command or flags for an existing command, make sure to update Spack's `Bash tab completion script <https://github.com/adamjstewart/spack/blob/develop/share/spack/spack-completion.bash>`_. + +------------- +Writing Hooks +------------- + +A hook is a callback that makes it easy to design functions that run +for different events. We do this by way of defining hook types, and then +inserting them at different places in the spack code base. Whenever a hook +type triggers by way of a function call, we find all the hooks of that type, +and run them. + +Spack defines hooks by way of a module at ``lib/spack/spack/hooks`` where we can define +types of hooks in the ``__init__.py``, and then python files in that folder +can use hook functions. The files are automatically parsed, so if you write +a new file for some integration (e.g., ``lib/spack/spack/hooks/myintegration.py`` +you can then write hook functions in that file that will be automatically detected, +and run whenever your hook is called. This section will cover the basic kind +of hooks, and how to write them. + +^^^^^^^^^^^^^^ +Types of Hooks +^^^^^^^^^^^^^^ + +The following hooks are currently implemented to make it easy for you, +the developer, to add hooks at different stages of a spack install or similar. +If there is a hook that you would like and is missing, you can propose to add a new one. + +""""""""""""""""""""" +``pre_install(spec)`` +""""""""""""""""""""" + +A ``pre_install`` hook is run within an install subprocess, directly before +the install starts. It expects a single argument of a spec, and is run in +a multiprocessing subprocess. Note that if you see ``pre_install`` functions associated with packages these are not hooks +as we have defined them here, but rather callback functions associated with +a package install. + + +"""""""""""""""""""""" +``post_install(spec)`` +"""""""""""""""""""""" + +A ``post_install`` hook is run within an install subprocess, directly after +the install finishes, but before the build stage is removed. If you +write one of these hooks, you should expect it to accept a spec as the only +argument. This is run in a multiprocessing subprocess. This ``post_install`` is +also seen in packages, but in this context not related to the hooks described +here. + + +"""""""""""""""""""""""""" +``on_install_start(spec)`` +"""""""""""""""""""""""""" + +This hook is run at the beginning of ``lib/spack/spack/installer.py``, +in the install function of a ``PackageInstaller``, +and importantly is not part of a build process, but before it. This is when +we have just newly grabbed the task, and are preparing to install. If you +write a hook of this type, you should provide the spec to it. + +.. code-block:: python + + def on_install_start(spec): + """On start of an install, we want to... + """ + print('on_install_start') + + +"""""""""""""""""""""""""""" +``on_install_success(spec)`` +"""""""""""""""""""""""""""" + +This hook is run on a successful install, and is also run inside the build +process, akin to ``post_install``. The main difference is that this hook +is run outside of the context of the stage directory, meaning after the +build stage has been removed and the user is alerted that the install was +successful. If you need to write a hook that is run on success of a particular +phase, you should use ``on_phase_success``. + +"""""""""""""""""""""""""""" +``on_install_failure(spec)`` +"""""""""""""""""""""""""""" + +This hook is run given an install failure that happens outside of the build +subprocess, but somewhere in ``installer.py`` when something else goes wrong. +If you need to write a hook that is relevant to a failure within a build +process, you would want to instead use ``on_phase_failure``. + + +""""""""""""""""""""""""""""""""""""""""""""""" +``on_phase_success(pkg, phase_name, log_file)`` +""""""""""""""""""""""""""""""""""""""""""""""" + +This hook is run within the install subprocess, and specifically when a phase +successfully finishes. Since we are interested in the package, the name of +the phase, and any output from it, we require: + + - **pkg**: the package variable, which also has the attached spec at ``pkg.spec`` + - **phase_name**: the name of the phase that was successful (e.g., configure) + - **log_file**: the path to the file with output, in case you need to inspect or otherwise interact with it. + +""""""""""""""""""""""""""""""""""""""""""""" +``on_phase_error(pkg, phase_name, log_file)`` +""""""""""""""""""""""""""""""""""""""""""""" + +In the case of an error during a phase, we might want to trigger some event +with a hook, and this is the purpose of this particular hook. Akin to +``on_phase_success`` we require the same variables - the package that failed, +the name of the phase, and the log file where we might find errors. + +""""""""""""""""""""""""""""""""" +``on_analyzer_save(pkg, result)`` +""""""""""""""""""""""""""""""""" + +After an analyzer has saved some result for a package, this hook is called, +and it provides the package that we just ran the analysis for, along with +the loaded result. Typically, a result is structured to have the name +of the analyzer as key, and the result object that is defined in detail in +:ref:`analyzer_run_function`. + +.. code-block:: python + + def on_analyzer_save(pkg, result): + """given a package and a result... + """ + print('Do something extra with a package analysis result here') + + +^^^^^^^^^^^^^^^^^^^^^^ +Adding a New Hook Type +^^^^^^^^^^^^^^^^^^^^^^ + +Adding a new hook type is very simple! In ``lib/spack/spack/hooks/__init__.py`` +you can simply create a new ``HookRunner`` that is named to match your new hook. +For example, let's say you want to add a new hook called ``post_log_write`` +to trigger after anything is written to a logger. You would add it as follows: + +.. code-block:: python + + # pre/post install and run by the install subprocess + pre_install = HookRunner('pre_install') + post_install = HookRunner('post_install') + + # hooks related to logging + post_log_write = HookRunner('post_log_write') # <- here is my new hook! + + +You then need to decide what arguments my hook would expect. Since this is +related to logging, let's say that you want a message and level. That means +that when you add a python file to the ``lib/spack/spack/hooks`` +folder with one or more callbacks intended to be triggered by this hook. You might +use my new hook as follows: + +.. code-block:: python + + def post_log_write(message, level): + """Do something custom with the messsage and level every time we write + to the log + """ + print('running post_log_write!') + + +To use the hook, we would call it as follows somewhere in the logic to do logging. +In this example, we use it outside of a logger that is already defined: + +.. code-block:: python + + import spack.hooks + + # We do something here to generate a logger and message + spack.hooks.post_log_write(message, logger.level) + + +This is not to say that this would be the best way to implement an integration +with the logger (you'd probably want to write a custom logger, or you could +have the hook defined within the logger) but serves as an example of writing a hook. + ---------- Unit tests ---------- @@ -396,21 +828,43 @@ other Spack modules: True >>> -You can also run a single command: +If you prefer using an IPython interpreter, given that IPython is installed +you can specify the interpreter with ``-i``: + +.. code-block:: console + + $ spack python -i ipython + Python 3.8.3 (default, May 19 2020, 18:47:26) + Type 'copyright', 'credits' or 'license' for more information + IPython 7.17.0 -- An enhanced Interactive Python. Type '?' for help. + + + Spack version 0.16.0 + Python 3.8.3, Linux x86_64 + + In [1]: + + +With either interpreter you can run a single command: .. code-block:: console $ spack python -c 'import distro; distro.linux_distribution()' - ('Fedora', '25', 'Workstation Edition') + ('Ubuntu', '18.04', 'Bionic Beaver') + + $ spack python -i ipython -c 'import distro; distro.linux_distribution()' + Out[1]: ('Ubuntu', '18.04', 'Bionic Beaver') or a file: .. code-block:: console $ spack python ~/test_fetching.py + $ spack python -i ipython ~/test_fetching.py just like you would with the normal ``python`` command. + .. _cmd-spack-url: ^^^^^^^^^^^^^ @@ -553,8 +1007,10 @@ develop onto release branches. This is typically done by cherry-picking bugfix commits off of ``develop``. To avoid version churn for users of a release series, minor releases -should **not** make changes that would change the concretization of +**should not** make changes that would change the concretization of packages. They should generally only contain fixes to the Spack core. +However, sometimes priorities are such that new functionality needs to +be added to a minor release. Both major and minor releases are tagged. After each release, we merge the release branch back into ``develop`` so that the version bump and any @@ -563,50 +1019,51 @@ convenience, we also tag the latest release as ``releases/latest``, so that users can easily check it out to get the latest stable version. See :ref:`merging-releases` for more details. - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Scheduling work for releases ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ We schedule work for releases by creating `GitHub projects <https://github.com/spack/spack/projects>`_. At any time, there may be -several open release projects. For example, here are two releases (from +several open release projects. For example, below are two releases (from some past version of the page linked above): .. image:: images/projects.png -Here, there's one release in progress for ``0.15.1`` and another for +This image shows one release in progress for ``0.15.1`` and another for ``0.16.0``. Each of these releases has a project board containing issues and pull requests. GitHub shows a status bar with completed work in green, work in progress in purple, and work not started yet in gray, so it's fairly easy to see progress. -Spack's project boards are not firm commitments, and we move work between +Spack's project boards are not firm commitments so we move work between releases frequently. If we need to make a release and some tasks are not -yet done, we will simply move them to next minor or major release, rather +yet done, we will simply move them to the next minor or major release, rather than delaying the release to complete them. For more on using GitHub project boards, see `GitHub's documentation <https://docs.github.com/en/github/managing-your-work-on-github/about-project-boards>`_. + .. _major-releases: ^^^^^^^^^^^^^^^^^^^^^ -Making Major Releases +Making major releases ^^^^^^^^^^^^^^^^^^^^^ -Assuming you've already created a project board and completed the work -for a major release, the steps to make the release are as follows: +Assuming a project board has already been created and all required work +completed, the steps to make the major release are: #. Create two new project boards: * One for the next major release * One for the next point release -#. Move any tasks that aren't done yet to one of the new project boards. - Small bugfixes should go to the next point release. Major features, - refactors, and changes that could affect concretization should go in - the next major release. +#. Move any optional tasks that are not done to one of the new project boards. + + In general, small bugfixes should go to the next point release. Major + features, refactors, and changes that could affect concretization should + go in the next major release. #. Create a branch for the release, based on ``develop``: @@ -618,11 +1075,14 @@ for a major release, the steps to make the release are as follows: ``releases/vX.Y``. That is, you should create a ``releases/vX.Y`` branch if you are preparing the ``X.Y.0`` release. -#. Bump the version in ``lib/spack/spack/__init__.py``. See `this example from 0.13.0 +#. Bump the version in ``lib/spack/spack/__init__.py``. + + See `this example from 0.13.0 <https://github.com/spack/spack/commit/8eeb64096c98b8a43d1c587f13ece743c864fba9>`_ -#. Update ``CHANGELOG.md`` with major highlights in bullet form. Use - proper markdown formatting, like `this example from 0.15.0 +#. Update ``CHANGELOG.md`` with major highlights in bullet form. + + Use proper markdown formatting, like `this example from 0.15.0 <https://github.com/spack/spack/commit/d4bf70d9882fcfe88507e9cb444331d7dd7ba71c>`_. #. Push the release branch to GitHub. @@ -646,33 +1106,33 @@ for a major release, the steps to make the release are as follows: .. _point-releases: ^^^^^^^^^^^^^^^^^^^^^ -Making Point Releases +Making point releases ^^^^^^^^^^^^^^^^^^^^^ -This assumes you've already created a project board for a point release -and completed the work to be done for the release. To make a point -release: +Assuming a project board has already been created and all required work +completed, the steps to make the point release are: -#. Create one new project board for the next point release. +#. Create a new project board for the next point release. -#. Move any cards that aren't done yet to the next project board. +#. Move any optional tasks that are not done to the next project board. -#. Check out the release branch (it should already exist). For the - ``X.Y.Z`` release, the release branch is called ``releases/vX.Y``. For - ``v0.15.1``, you would check out ``releases/v0.15``: +#. Check out the release branch (it should already exist). + + For the ``X.Y.Z`` release, the release branch is called ``releases/vX.Y``. + For ``v0.15.1``, you would check out ``releases/v0.15``: .. code-block:: console $ git checkout releases/v0.15 #. Cherry-pick each pull request in the ``Done`` column of the release - project onto the release branch. + project board onto the release branch. This is **usually** fairly simple since we squash the commits from the - vast majority of pull requests, which means there is only one commit + vast majority of pull requests. That means there is only one commit per pull request to cherry-pick. For example, `this pull request <https://github.com/spack/spack/pull/15777>`_ has three commits, but - the were squashed into a single commit on merge. You can see the + they were squashed into a single commit on merge. You can see the commit that was created here: .. image:: images/pr-commit.png @@ -684,9 +1144,8 @@ release: $ git cherry-pick 7e46da7 - For pull requests that were rebased, you'll need to cherry-pick each - rebased commit individually. There have not been any rebased PRs like - this in recent point releases. + For pull requests that were rebased (or not squashed), you'll need to + cherry-pick each associated commit individually. .. warning:: @@ -699,30 +1158,35 @@ release: cherry-picked all the commits in order. This generally means there is some other intervening pull request that the one you're trying to pick depends on. In these cases, you'll need to make a judgment - call: + call regarding those pull requests. Consider the number of affected + files and or the resulting differences. - 1. If the dependency is small, you might just cherry-pick it, too. - If you do this, add it to the release board. + 1. If the dependency changes are small, you might just cherry-pick it, + too. If you do this, add the task to the release board. - 2. If it is large, then you may decide that this fix is not worth - including in a point release, in which case you should remove it - from the release project. + 2. If the changes are large, then you may decide that this fix is not + worth including in a point release, in which case you should remove + the task from the release project. 3. You can always decide to manually back-port the fix to the release branch if neither of the above options makes sense, but this can require a lot of work. It's seldom the right choice. -#. Bump the version in ``lib/spack/spack/__init__.py``. See `this example from 0.14.1 +#. Bump the version in ``lib/spack/spack/__init__.py``. + + See `this example from 0.14.1 <https://github.com/spack/spack/commit/ff0abb9838121522321df2a054d18e54b566b44a>`_. -#. Update ``CHANGELOG.md`` with a list of bugfixes. This is typically just a - summary of the commits you cherry-picked onto the release branch. See - `the changelog from 0.14.1 +#. Update ``CHANGELOG.md`` with a list of the changes. + + This is typically a summary of the commits you cherry-picked onto the + release branch. See `the changelog from 0.14.1 <https://github.com/spack/spack/commit/ff0abb9838121522321df2a054d18e54b566b44a>`_. #. Push the release branch to GitHub. #. Make sure CI passes on the release branch, including: + * Regular unit tests * Build tests * The E4S pipeline at `gitlab.spack.io <https://gitlab.spack.io>`_ @@ -745,23 +1209,26 @@ release: Publishing a release on GitHub ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -#. Go to `github.com/spack/spack/releases - <https://github.com/spack/spack/releases>`_ and click ``Draft a new - release``. Set the following: +#. Create the release in GitHub. + + * Go to + `github.com/spack/spack/releases <https://github.com/spack/spack/releases>`_ + and click ``Draft a new release``. - * ``Tag version`` should start with ``v`` and contain *all three* - parts of the version, .g. ``v0.15.1``. This is the name of the tag - that will be created. + * Set ``Tag version`` to the name of the tag that will be created. - * ``Target`` should be the ``releases/vX.Y`` branch (e.g., ``releases/v0.15``). + The name should start with ``v`` and contain *all three* + parts of the version (e.g. ``v0.15.0`` or ``v0.15.1``). - * ``Release title`` should be ``vX.Y.Z`` (To match the tag, e.g., ``v0.15.1``). + * Set ``Target`` to the ``releases/vX.Y`` branch (e.g., ``releases/v0.15``). - * For the text, paste the latest release markdown from your ``CHANGELOG.md``. + * Set ``Release title`` to ``vX.Y.Z`` to match the tag (e.g., ``v0.15.1``). - You can save the draft and keep coming back to this as you prepare the release. + * Paste the latest release markdown from your ``CHANGELOG.md`` file as the text. -#. When you are done, click ``Publish release``. + * Save the draft so you can keep coming back to it as you prepare the release. + +#. When you are ready to finalize the release, click ``Publish release``. #. Immediately after publishing, go back to `github.com/spack/spack/releases @@ -769,22 +1236,26 @@ Publishing a release on GitHub auto-generated ``.tar.gz`` file for the release. It's the ``Source code (tar.gz)`` link. -#. Click ``Edit`` on the release you just did and attach the downloaded +#. Click ``Edit`` on the release you just made and attach the downloaded release tarball as a binary. This does two things: - #. Makes sure that the hash of our releases doesn't change over time. - GitHub sometimes annoyingly changes they way they generate - tarballs, and then hashes can change if you rely on the + #. Makes sure that the hash of our releases does not change over time. + + GitHub sometimes annoyingly changes the way they generate tarballs + that can result in the hashes changing if you rely on the auto-generated tarball links. - #. Gets us download counts on releases visible through the GitHub - API. GitHub tracks downloads of artifacts, but *not* the source + #. Gets download counts on releases visible through the GitHub API. + + GitHub tracks downloads of artifacts, but *not* the source links. See the `releases page <https://api.github.com/repos/spack/spack/releases>`_ and search for ``download_count`` to see this. -#. Go to `readthedocs.org <https://readthedocs.org/projects/spack>`_ and activate - the release tag. This builds the documentation and makes the released version +#. Go to `readthedocs.org <https://readthedocs.org/projects/spack>`_ and + activate the release tag. + + This builds the documentation and makes the released version selectable in the versions menu. @@ -798,23 +1269,23 @@ If the new release is the **highest** Spack release yet, you should also tag it as ``releases/latest``. For example, suppose the highest release is currently ``0.15.3``: - * If you are releasing ``0.15.4`` or ``0.16.0``, then you should tag - it with ``releases/latest``, as these are higher than ``0.15.3``. +* If you are releasing ``0.15.4`` or ``0.16.0``, then you should tag + it with ``releases/latest``, as these are higher than ``0.15.3``. - * If you are making a new release of an **older** major version of - Spack, e.g. ``0.14.4``, then you should not tag it as - ``releases/latest`` (as there are newer major versions). +* If you are making a new release of an **older** major version of + Spack, e.g. ``0.14.4``, then you should not tag it as + ``releases/latest`` (as there are newer major versions). - To tag ``releases/latest``, do this: +To tag ``releases/latest``, do this: - .. code-block:: console +.. code-block:: console - $ git checkout releases/vX.Y # vX.Y is the new release's branch - $ git tag --force releases/latest - $ git push --tags + $ git checkout releases/vX.Y # vX.Y is the new release's branch + $ git tag --force releases/latest + $ git push --force --tags - The ``--force`` argument makes ``git`` overwrite the existing - ``releases/latest`` tag with the new one. +The ``--force`` argument to ``git tag`` makes ``git`` overwrite the existing +``releases/latest`` tag with the new one. We also merge each release that we tag as ``releases/latest`` into ``develop``. Make sure to do this with a merge commit: @@ -822,17 +1293,17 @@ Make sure to do this with a merge commit: .. code-block:: console $ git checkout develop - $ git merge --no-ff vX.Y.Z # vX.Y.Z is the new release's tag + $ git merge --no-ff -s ours vX.Y.Z # vX.Y.Z is the new release's tag $ git push We merge back to ``develop`` because it: - * updates the version and ``CHANGELOG.md`` on ``develop``. + * updates the version and ``CHANGELOG.md`` on ``develop``; and * ensures that your release tag is reachable from the head of - ``develop`` + ``develop``. -We *must* use a real merge commit (via the ``--no-ff`` option) because it -ensures that the release tag is reachable from the tip of ``develop``. +We *must* use a real merge commit (via the ``--no-ff`` option) to +ensure that the release tag is reachable from the tip of ``develop``. This is necessary for ``spack -V`` to work properly -- it uses ``git describe --tags`` to find the last reachable tag in the repository and reports how far we are from it. For example: @@ -850,6 +1321,7 @@ the release is complete and tagged. If you do it before you've tagged the release and later decide you want to tag some later commit, you'll need to merge again. + .. _announcing-releases: ^^^^^^^^^^^^^^^^^^^^ @@ -860,20 +1332,40 @@ We announce releases in all of the major Spack communication channels. Publishing the release takes care of GitHub. The remaining channels are Twitter, Slack, and the mailing list. Here are the steps: -#. Make a tweet to announce the release. It should link to the release's - page on GitHub. You can base it on `this example tweet - <https://twitter.com/spackpm/status/1231761858182307840>`_. +#. Announce the release on Twitter. + + * Compose the tweet on the ``@spackpm`` account per the + ``spack-twitter`` slack channel. + + * Be sure to include a link to the release's page on GitHub. + + You can base the tweet on `this + example <https://twitter.com/spackpm/status/1231761858182307840>`_. + +#. Announce the release on Slack. + + * Compose a message in the ``#general`` Slack channel + (`spackpm.slack.com <https://spackpm.slack.com>`_). + + * Preface the message with ``@channel`` to notify even those + people not currently logged in. + + * Be sure to include a link to the tweet above. + + The tweet will be shown inline so that you do not have to retype + your release announcement. + +#. Announce the release on the Spack mailing list. + + * Compose an email to the Spack mailing list. + + * Be sure to include a link to the release's page on GitHub. -#. Ping ``@channel`` in ``#general`` on Slack (`spackpm.slack.com - <https://spackpm.slack.com>`_) with a link to the tweet. The tweet - will be shown inline so that you do not have to retype your release - announcement. + * It is also helpful to include some information directly in the + email. -#. Email the Spack mailing list to let them know about the release. As - with the tweet, you likely want to link to the release's page on - GitHub. It's also helpful to include some information directly in the - email. You can base yours on this `example email - <https://groups.google.com/forum/#!topic/spack/WT4CT9i_X4s>`_. + You can base your announcement on this `example + email <https://groups.google.com/forum/#!topic/spack/WT4CT9i_X4s>`_. -Once you've announced the release, congratulations, you're done! You've -finished making the release! +Once you've completed the above steps, congratulations, you're done! +You've finished making the release! diff --git a/lib/spack/docs/environments.rst b/lib/spack/docs/environments.rst index 1710ec6b5a..12f6f4121c 100644 --- a/lib/spack/docs/environments.rst +++ b/lib/spack/docs/environments.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -248,9 +248,9 @@ Users can add abstract specs to an Environment using the ``spack add`` command. The most important component of an Environment is a list of abstract specs. -Adding a spec adds to the manifest (the ``spack.yaml`` file) and to -the roots of the Environment, but does not affect the concrete specs -in the lockfile, nor does it install the spec. +Adding a spec adds to the manifest (the ``spack.yaml`` file), which is +used to define the roots of the Environment, but does not affect the +concrete specs in the lockfile, nor does it install the spec. The ``spack add`` command is environment aware. It adds to the currently active environment. All environment aware commands can also @@ -356,6 +356,18 @@ command also stores a Spack repo containing the ``package.py`` file used at install time for each package in the ``repos/`` directory in the Environment. +The ``--no-add`` option can be used in a concrete environment to tell +spack to install specs already present in the environment but not to +add any new root specs to the environment. For root specs provided +to ``spack install`` on the command line, ``--no-add`` is the default, +while for dependency specs on the other hand, it is optional. In other +words, if there is an unambiguous match in the active concrete environment +for a root spec provided to ``spack install`` on the command line, spack +does not require you to specify the ``--no-add` option to prevent the spec +from being added again. At the same time, a spec that already exists in the +environment, but only as a dependency, will be added to the environment as a +root spec without the ``--no-add`` option. + ^^^^^^^ Loading ^^^^^^^ @@ -399,6 +411,12 @@ There are two ways to include configuration information in a Spack Environment: #. Included in the ``spack.yaml`` file from another file. +Many Spack commands also affect configuration information in files +automatically. Those commands take a ``--scope`` argument, and the +environment can be specified by ``env:NAME`` (to affect environment +``foo``, set ``--scope env:foo``). These commands will automatically +manipulate configuration inline in the ``spack.yaml`` file. + """"""""""""""""""""" Inline configurations """"""""""""""""""""" @@ -441,8 +459,8 @@ Environments can include files with either relative or absolute paths. Inline configurations take precedence over included configurations, so you don't have to change shared configuration files to make small changes to an individual Environment. Included configs -listed later will have higher precedence, as the included configs are -applied in order. +listed earlier will have higher precedence, as the included configs are +applied in reverse order. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Manually Editing the Specs List diff --git a/lib/spack/docs/extensions.rst b/lib/spack/docs/extensions.rst index 15c59c76ef..b0cb38d00b 100644 --- a/lib/spack/docs/extensions.rst +++ b/lib/spack/docs/extensions.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/docs/features.rst b/lib/spack/docs/features.rst index df212c11fc..7edfba01cc 100644 --- a/lib/spack/docs/features.rst +++ b/lib/spack/docs/features.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/docs/getting_started.rst b/lib/spack/docs/getting_started.rst index 09cc2c488a..805149af90 100644 --- a/lib/spack/docs/getting_started.rst +++ b/lib/spack/docs/getting_started.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -19,7 +19,7 @@ before Spack is run: #. Python 2 (2.6 or 2.7) or 3 (3.5 - 3.9) to run Spack #. A C/C++ compiler for building #. The ``make`` executable for building -#. The ``tar``, ``gzip``, ``bzip2``, ``xz`` and optionally ``zstd`` +#. The ``tar``, ``gzip``, ``unzip``, ``bzip2``, ``xz`` and optionally ``zstd`` executables for extracting source code #. The ``patch`` command to apply patches #. The ``git`` and ``curl`` commands for fetching @@ -75,6 +75,14 @@ shell integration for :ref:`certain commands <packaging-shell-support>`, If you do not want to use Spack's shell support, you can always just run the ``spack`` command directly from ``spack/bin/spack``. +When the ``spack`` command is executed it searches for an appropriate +Python interpreter to use, which can be explicitly overridden by setting +the ``SPACK_PYTHON`` environment variable. When sourcing the appropriate shell +setup script, ``SPACK_PYTHON`` will be set to the interpreter found at +sourcing time, ensuring future invocations of the ``spack`` command will +continue to use the same consistent python version regardless of changes in +the environment. + ^^^^^^^^^^^^^^^^^^ Check Installation diff --git a/lib/spack/docs/index.rst b/lib/spack/docs/index.rst index 39fee81387..fb9c669c12 100644 --- a/lib/spack/docs/index.rst +++ b/lib/spack/docs/index.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -67,6 +67,7 @@ or refer to the full manual below. build_settings environments containers + monitoring mirrors module_file_support repositories @@ -79,6 +80,12 @@ or refer to the full manual below. .. toctree:: :maxdepth: 2 + :caption: Research + + analyze + +.. toctree:: + :maxdepth: 2 :caption: Contributing contribution_guide diff --git a/lib/spack/docs/known_issues.rst b/lib/spack/docs/known_issues.rst index accec2422b..b9c4dd088f 100644 --- a/lib/spack/docs/known_issues.rst +++ b/lib/spack/docs/known_issues.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/docs/mirrors.rst b/lib/spack/docs/mirrors.rst index 5268c87067..83d4fd9512 100644 --- a/lib/spack/docs/mirrors.rst +++ b/lib/spack/docs/mirrors.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -159,6 +159,27 @@ can supply a file with specs in it, one per line: This is useful if there is a specific suite of software managed by your site. +^^^^^^^^^^^^^^^^^^ +Mirror environment +^^^^^^^^^^^^^^^^^^ + +To create a mirror of all packages required by a concerte environment, activate the environment and call ``spack mirror create -a``. +This is especially useful to create a mirror of an environment concretized on another machine. + +.. code-block:: console + + [remote] $ spack env create myenv + [remote] $ spack env activate myenv + [remote] $ spack add ... + [remote] $ spack concretize + + $ sftp remote:/spack/var/environment/myenv/spack.lock + $ spack env create myenv spack.lock + $ spack env activate myenv + $ spack mirror create -a + + + .. _cmd-spack-mirror-add: -------------------- diff --git a/lib/spack/docs/module_file_support.rst b/lib/spack/docs/module_file_support.rst index 721b18d5fc..d46ec3143d 100644 --- a/lib/spack/docs/module_file_support.rst +++ b/lib/spack/docs/module_file_support.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -130,9 +130,8 @@ list of environment modifications. to the corresponding environment variables: ================== ================================= - LIBRARY_PATH ``self.prefix/rlib/R/lib`` LD_LIBRARY_PATH ``self.prefix/rlib/R/lib`` - CPATH ``self.prefix/rlib/R/include`` + PKG_CONFIG_PATH ``self.prefix/rlib/pkgconfig`` ================== ================================= with the following snippet: diff --git a/lib/spack/docs/monitoring.rst b/lib/spack/docs/monitoring.rst new file mode 100644 index 0000000000..b1d491c563 --- /dev/null +++ b/lib/spack/docs/monitoring.rst @@ -0,0 +1,104 @@ +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other + Spack Project Developers. See the top-level COPYRIGHT file for details. + + SPDX-License-Identifier: (Apache-2.0 OR MIT) + +.. _monitoring: + +========== +Monitoring +========== + +You can use a `spack monitor <https://github.com/spack/spack-monitor>`_ "Spackmon" +server to store a database of your packages, builds, and associated metadata +for provenance, research, or some other kind of development. You should +follow the instructions in the `spack monitor documentation <https://spack-monitor.readthedocs.org>`_ +to first create a server along with a username and token for yourself. +You can then use this guide to interact with the server. + +------------------- +Analysis Monitoring +------------------- + +To read about how to monitor an analysis (meaning you want to send analysis results +to a server) see :ref:`analyze_monitoring`. + +--------------------- +Monitoring An Install +--------------------- + +Since an install is typically when you build packages, we logically want +to tell spack to monitor during this step. Let's start with an example +where we want to monitor the install of hdf5. Unless you have disabled authentication +for the server, we first want to export our spack monitor token and username to the environment: + +.. code-block:: console + + $ export SPACKMON_TOKEN=50445263afd8f67e59bd79bff597836ee6c05438 + $ export SPACKMON_USER=spacky + + +By default, the host for your server is expected to be at ``http://127.0.0.1`` +with a prefix of ``ms1``, and if this is the case, you can simply add the +``--monitor`` flag to the install command: + +.. code-block:: console + + $ spack install --monitor hdf5 + + +If you need to customize the host or the prefix, you can do that as well: + +.. code-block:: console + + $ spack install --monitor --monitor-prefix monitor --monitor-host https://monitor-service.io hdf5 + + +As a precaution, we cut out early in the spack client if you have not provided +authentication credentials. For example, if you run the command above without +exporting your username or token, you'll see: + +.. code-block:: console + + ==> Error: You are required to export SPACKMON_TOKEN and SPACKMON_USER + +This extra check is to ensure that we don't start any builds, +and then discover that you forgot to export your token. However, if +your monitoring server has authentication disabled, you can tell this to +the client to skip this step: + +.. code-block:: console + + $ spack install --monitor --monitor-disable-auth hdf5 + +If the service is not running, you'll cleanly exit early - the install will +not continue if you've asked it to monitor and there is no service. +For example, here is what you'll see if the monitoring service is not running: + +.. code-block:: console + + [Errno 111] Connection refused + + +If you want to continue builds (and stop monitoring) you can set the ``--monitor-keep-going`` +flag. + +.. code-block:: console + + $ spack install --monitor --monitor-keep-going hdf5 + +This could mean that if a request fails, you only have partial or no data +added to your monitoring database. This setting will not be applied to the +first request to check if the server is running, but to subsequent requests. +If you don't have a monitor server running and you want to build, simply +don't provide the ``--monitor`` flag! Finally, if you want to provide one or +more tags to your build, you can do: + +.. code-block:: console + + # Add one tag, "pizza" + $ spack install --monitor --monitor-tags pizza hdf5 + + # Add two tags, "pizza" and "pasta" + $ spack install --monitor --monitor-tags pizza,pasta hdf5 + diff --git a/lib/spack/docs/package_list.rst b/lib/spack/docs/package_list.rst index 67f78e202d..b0607edb86 100644 --- a/lib/spack/docs/package_list.rst +++ b/lib/spack/docs/package_list.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst index ec83679a47..5d00a64fac 100644 --- a/lib/spack/docs/packaging_guide.rst +++ b/lib/spack/docs/packaging_guide.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -612,6 +612,62 @@ it executable, then runs it with some arguments. installer = Executable(self.stage.archive_file) installer('--prefix=%s' % prefix, 'arg1', 'arg2', 'etc.') + +^^^^^^^^^^^^^^^^^^^^^^^^ +Deprecating old versions +^^^^^^^^^^^^^^^^^^^^^^^^ + +There are many reasons to remove old versions of software: + +#. Security vulnerabilities (most serious reason) +#. Changing build systems that increase package complexity +#. Changing dependencies/patches/resources/flags that increase package complexity +#. Maintainer/developer inability/unwillingness to support old versions +#. No longer available for download (right to be forgotten) +#. Package or version rename + +At the same time, there are many reasons to keep old versions of software: + +#. Reproducibility +#. Requirements for older packages (e.g. some packages still rely on Qt 3) + +In general, you should not remove old versions from a ``package.py``. Instead, +you should first deprecate them using the following syntax: + +.. code-block:: python + + version('1.2.3', sha256='...', deprecated=True) + + +This has two effects. First, ``spack info`` will no longer advertise that +version. Second, commands like ``spack install`` that fetch the package will +require user approval: + +.. code-block:: console + + $ spack install openssl@1.0.1e + ==> Warning: openssl@1.0.1e is deprecated and may be removed in a future Spack release. + ==> Fetch anyway? [y/N] + + +If you use ``spack install --deprecated``, this check can be skipped. + +This also applies to package recipes that are renamed or removed. You should +first deprecate all versions before removing a package. If you need to rename +it, you can deprecate the old package and create a new package at the same +time. + +Version deprecations should always last at least one Spack minor release cycle +before the version is completely removed. For example, if a version is +deprecated in Spack 0.16.0, it should not be removed until Spack 0.17.0. No +version should be removed without such a deprecation process. This gives users +a chance to complain about the deprecation in case the old version is needed +for some application. If you require a deprecated version of a package, simply +submit a PR to remove ``deprecated=True`` from the package. However, you may be +asked to help maintain this version of the package if the current maintainers +are unwilling to support this older version. + + ^^^^^^^^^^^^^^^^ Download caching ^^^^^^^^^^^^^^^^ @@ -1150,7 +1206,7 @@ Variants Many software packages can be configured to enable optional features, which often come at the expense of additional dependencies or -longer build-times. To be flexible enough and support a wide variety of +longer build times. To be flexible enough and support a wide variety of use cases, Spack permits to expose to the end-user the ability to choose which features should be activated in a package at the time it is installed. The mechanism to be employed is the :py:func:`spack.directives.variant` directive. @@ -1864,14 +1920,21 @@ of ``mpich`` using the following command: $ srun -N 2 -n 8 spack install -j 4 mpich@3.3.2 -This will create eight concurrent four-job installation on two different +This will create eight concurrent, four-job installs on two different nodes. +Alternatively, you could run the same installs on one node by entering +the following at the command line of a bash shell: + +.. code-block:: console + + $ for i in {1..12}; do nohup spack install -j 4 mpich@3.3.2 >> mpich_install.txt 2>&1 &; done + .. note:: - The effective parallelism will be based on the maximum number of - packages that can be installed at the same time, which will limited - by the number of packages with no (remaining) uninstalled dependencies. + The effective parallelism is based on the maximum number of packages + that can be installed at the same time, which is limited by the + number of packages with no (remaining) uninstalled dependencies. .. _dependencies: @@ -2737,12 +2800,14 @@ The classes that are currently provided by Spack are: | | built using CMake | +-------------------------------+----------------------------------+ | :py:class:`.CudaPackage` | A helper class for packages that | - | | use CUDA. It is intended to be | - | | used in combination with others | + | | use CUDA | +-------------------------------+----------------------------------+ | :py:class:`.QMakePackage` | Specialized class for packages | | | build using QMake | +-------------------------------+----------------------------------+ + | :py:class:`.ROCmPackage` | A helper class for packages that | + | | use ROCm | + +-------------------------------+----------------------------------+ | :py:class:`.SConsPackage` | Specialized class for packages | | | built using SCons | +-------------------------------+----------------------------------+ @@ -3846,219 +3911,851 @@ using MPI wrappers will work, even on even on a Cray: This is because on Cray, ``spec['mpi'].mpicc`` is just ``spack_cc``. -.. _sanity-checks: +.. _checking_an_installation: ------------------------ Checking an installation ------------------------ -By default, Spack assumes that a build has failed if nothing is -written to the install prefix, and that it has succeeded if anything -(a file, a directory, etc.) is written to the install prefix after -``install()`` completes. +A package that *appears* to install successfully does not mean +it is actually installed correctly or will continue to work indefinitely. +There are a number of possible points of failure so Spack provides +features for checking the software along the way. + +Failures can occur during and after the installation process. The +build may start but the software not end up fully installed. The +installed software may not work at all or as expected. The software +may work after being installed but, due to changes on the system, +may stop working days, weeks, or months after being installed. + +This section describes Spack's support for checks that can be performed +during and after its installation. The former checks are referred to as +``build-time tests`` and the latter as ``stand-alone (or smoke) tests``. + +.. _build_time-tests: + +^^^^^^^^^^^^^^^^ +Build-time tests +^^^^^^^^^^^^^^^^ + +Spack infers the status of a build based on the contents of the install +prefix. Success is assumed if anything (e.g., a file, directory) is +written after ``install()`` completes. Otherwise, the build is assumed +to have failed. However, the presence of install prefix contents +is not a sufficient indicator of success. -Consider a simple autotools build like this: +Consider a simple autotools build using the following commands: + +.. code-block:: console + + $ ./configure --prefix=/path/to/installation/prefix + $ make + $ make install + +Standard Autotools and CMake do not write anything to the prefix from +the ``configure`` and ``make`` commands. Files are only written from +the ``make install`` after the build completes. + +.. note:: + + If you want to learn more about ``Autotools`` and ``CMake`` packages + in Spack, refer to :ref:`AutotoolsPackage <autotoolspackage>` and + :ref:`CMakePackage <cmakepackage>`, respectively. + +What can you do to check that the build is progressing satisfactorily? +If there are specific files and or directories expected of a successful +installation, you can add basic, fast ``sanity checks``. You can also add +checks to be performed after one or more installation phases. + +.. _sanity-checks: + +"""""""""""""""""""" +Adding sanity checks +"""""""""""""""""""" + +Unfortunately, many builds of scientific software modify the installation +prefix **before** ``make install``. Builds like this can falsely report +success when an error occurs before the installation is complete. Simple +sanity checks can be used to identify files and or directories that are +required of a successful installation. Spack checks for the presence of +the files and directories after ``install()`` runs. + +If any of the listed files or directories are missing, then the build will +fail and the install prefix will be removed. If they all exist, then Spack +considers the build successful from a sanity check perspective and keeps +the prefix in place. + +For example, the sanity checks for the ``reframe`` package below specify +that eight paths must exist within the installation prefix after the +``install`` method completes. .. code-block:: python - def install(self, spec, prefix): - configure("--prefix={0}".format(prefix)) - make() - make("install") + class Reframe(Package): + ... + + # sanity check + sanity_check_is_file = [join_path('bin', 'reframe')] + sanity_check_is_dir = ['bin', 'config', 'docs', 'reframe', 'tutorials', + 'unittests', 'cscs-checks'] -If you are using using standard autotools or CMake, ``configure`` and -``make`` will not write anything to the install prefix. Only ``make -install`` writes the files, and only once the build is already -complete. +Spack will then ensure the installation created the **file**: -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -``sanity_check_is_file`` and ``sanity_check_is_dir`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +* ``self.prefix/bin/reframe`` -Unfortunately, many builds of scientific -software modify the install prefix *before* ``make install``. Builds -like this can falsely report that they were successfully installed if -an error occurs before the install is complete but after files have -been written to the ``prefix``. +It will also check for the existence of the following **directories**: -You can optionally specify *sanity checks* to deal with this problem. -Add properties like this to your package: +* ``self.prefix/bin`` +* ``self.prefix/config`` +* ``self.prefix/docs`` +* ``self.prefix/reframe`` +* ``self.prefix/tutorials`` +* ``self.prefix/unittests`` +* ``self.prefix/cscs-checks`` + +.. note:: + + You **MUST** use ``sanity_check_is_file`` to specify required + files and ``sanity_check_is_dir`` for required directories. + +.. _install_phase-tests: + +""""""""""""""""""""""""""""""" +Adding installation phase tests +""""""""""""""""""""""""""""""" + +Sometimes packages appear to build "correctly" only to have run-time +behavior issues discovered at a later stage, such as after a full +software stack relying on them has been built. Checks can be performed +at different phases of the package installation to possibly avoid +these types of problems. Some checks are built-in to different build +systems, while others will need to be added to the package. + +Built-in installation phase tests are provided by packages inheriting +from select :ref:`build systems <build-systems>`, where naming conventions +are used to identify typical test identifiers for those systems. In +general, you won't need to add anything to your package to take advantage +of these tests if your software's build system complies with the convention; +otherwise, you'll want or need to override the post-phase method to perform +other checks. + +.. list-table:: Built-in installation phase tests + :header-rows: 1 + + * - Build System Class + - Post-Build Phase Method (Runs) + - Post-Install Phase Method (Runs) + * - `AutotoolsPackage <build_systems/autotoolspackage>` + - ``check`` (``make test``, ``make check``) + - ``installcheck`` (``make installcheck``) + * - `CMakePackage <build_systems/cmakepackage>` + - ``check`` (``make check``, ``make test``) + - Not applicable + * - `MakefilePackage <build_systems/makefilepackage>` + - ``check`` (``make test``, ``make check``) + - ``installcheck`` (``make installcheck``) + * - `MesonPackage <build_systems/mesonpackage>` + - ``check`` (``make test``, ``make check``) + - Not applicable + * - `PerlPackage <build_systems/perlpackage>` + - ``check`` (``make test``) + - Not applicable + * - `PythonPackage <build_systems/pythonpackage>` + - Not applicable + - ``test`` (module imports) + * - `QMakePackage <build_systems/qmakepackage>` + - ``check`` (``make check``) + - Not applicable + * - `SConsPackage <build_systems/sconspackage>` + - ``build_test`` (must be overridden) + - Not applicable + * - `SIPPackage <build_systems/sippackage>` + - Not applicable + - ``test`` (module imports) + +For example, the ``Libelf`` package inherits from ``AutotoolsPackage`` +and its ``Makefile`` has a standard ``check`` target. So Spack will +automatically run ``make check`` after the ``build`` phase when it +is installed using the ``--test`` option, such as: + +.. code-block:: console + + $ spack install --test=root libelf + +In addition to overriding any built-in build system installation +phase tests, you can write your own install phase tests. You will +need to use two decorators for each phase test method: + +* ``run_after`` +* ``on_package_attributes`` + +The first decorator tells Spack when in the installation process to +run your test method installation process; namely *after* the provided +installation phase. The second decorator tells Spack to only run the +checks when the ``--test`` option is provided on the command line. + +.. note:: + + Be sure to place the directives above your test method in the order + ``run_after`` *then* ``on_package_attributes``. + +.. note:: + + You also want to be sure the package supports the phase you use + in the ``run_after`` directive. For example, ``PackageBase`` only + supports the ``install`` phase while the ``AutotoolsPackage`` and + ``MakefilePackage`` support both ``install`` and ``build`` phases. + +Assuming both ``build`` and ``install`` phases are available to you, +you could add additional checks to be performed after each of those +phases based on the skeleton provided below. + +.. code-block:: python + + class YourMakefilePackage(MakefilePackage): + ... + + @run_after('build') + @on_package_attributes(run_tests=True) + def check_build(self): + # Add your custom post-build phase tests + pass + + @run_after('install') + @on_package_attributes(run_tests=True) + def check_install(self): + # Add your custom post-install phase tests + pass + +.. note:: + + You could also schedule work to be done **before** a given phase + using the ``run_before`` decorator. + +By way of a concrete example, the ``reframe`` package mentioned +previously has a simple installation phase check that runs the +installed executable. The check is implemented as follows: + +.. code-block:: python + + class Reframe(Package): + ... + + # check if we can run reframe + @run_after('install') + @on_package_attributes(run_tests=True) + def check_list(self): + with working_dir(self.stage.source_path): + reframe = Executable(join_path(self.prefix, 'bin', 'reframe')) + reframe('-l') + +.. warning:: + + The API for adding tests is not yet considered stable and may change + in future releases. + +.. _cmd-spack-test: + +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Stand-alone (or smoke) tests +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +While build-time tests are integrated with the installation process, +stand-alone tests are independent of that process. Consequently, such +tests can be performed days, even weeks, after the software is installed. + +Stand-alone tests are checks that should run relatively quickly -- as +in on the order of at most a few minutes -- and ideally execute all +aspects of the installed software, or at least key functionality. + +.. note:: + + Execution speed is important because these tests are intended + to quickly assess whether the installed software works on the + system. + + Failing stand-alone tests indicate that there is no reason to + proceed with more resource-intensive tests. + + Passing stand-alone (or smoke) tests can lead to more thorough + testing, such as extensive unit or regression tests, or tests + that run at scale. Spack support for more thorough testing is + a work in progress. + +Stand-alone tests have their own test stage directory, which can be +configured. These tests can compile or build software with the compiler +used to build the package. They can use files cached from the build for +testing the installation. Custom files, such as source, data, or expected +outputs can be added for use in these tests. + +"""""""""""""""""""""""""""""""""""" +Configuring the test stage directory +"""""""""""""""""""""""""""""""""""" + +Stand-alone tests rely on a stage directory for building, running, +and tracking results. +The default directory, ``~/.spack/test``, is defined in +:ref:`etc/spack/defaults/config.yaml <config-yaml>`. +You can configure the location in the high-level ``config`` by adding +or changing the ``test_stage`` path in the appropriate ``config.yaml`` +file such that: + +.. code-block:: yaml + + config: + test_stage: /path/to/stage + +The package can access this path **during test processing** using +`self.test_suite.stage`. + +.. note:: + + The test stage path is established for the entire suite. That + means it is the root directory for all specs being installed + with the same `spack test run` command. Each spec gets its + own stage subdirectory. + +""""""""""""""""""""""""" +Enabling test compilation +""""""""""""""""""""""""" + +Some stand-alone tests will require access to the compiler with which +the package was built, especially for library-only packages. You must +enable loading the package's compiler configuration by setting the +``test_requires_compiler`` property to ``True`` for your package. +For example: .. code-block:: python class MyPackage(Package): ... - sanity_check_is_file = ['include/libelf.h'] - sanity_check_is_dir = [lib] + test_requires_compiler = True - def install(self, spec, prefix): - configure("--prefix=" + prefix) - make() - make("install") +Setting this property to ``True`` makes the compiler available in the +test environment through the canonical environment variables (e.g., +``CC``, ``CXX``, ``FC``, ``F77``). -Now, after ``install()`` runs, Spack will check whether -``$prefix/include/libelf.h`` exists and is a file, and whether -``$prefix/lib`` exists and is a directory. If the checks fail, then -the build will fail and the install prefix will be removed. If they -succeed, Spack considers the build successful and keeps the prefix in -place. +.. note:: -^^^^^^^^^^^^^^^^ -Build-time tests -^^^^^^^^^^^^^^^^ + We recommend adding the property at the top of the package with the + other attributes, such as ``homepage`` and ``url``. + +.. _cache_extra_test_sources: + +""""""""""""""""""""""" +Adding build-time files +""""""""""""""""""""""" + +.. note:: -Sometimes packages finish to build "correctly" and issues with their run-time -behavior are discovered only at a later stage, maybe after a full software stack -relying on them has already been built. To avoid situations of that kind it's possible -to write build-time tests that will be executed only if the option ``--run-tests`` -of ``spack install`` has been activated. + We highly recommend re-using build-time tests and input files + for testing installed software. These files are easier to keep + synchronized since they reside within the software's repository + than maintaining custom install test files with the Spack package. -The proper way to write these tests is relying on two decorators that come with -any base class listed in :ref:`installation_procedure`. +You can use the ``cache_extra_test_sources`` method to copy directories +and or files from the build stage directory to the package's installation +directory. + +The signature for ``cache_extra_test_sources`` is: .. code-block:: python - @run_after('build') - @on_package_attributes(run_tests=True) - def check_build(self): - # Custom implementation goes here - pass + def cache_extra_test_sources(self, srcs): + +where ``srcs`` is a string or a list of strings corresponding to +the paths for the files and or subdirectories, relative to the staged +source, that are to be copied to the corresponding path relative to +``self.install_test_root``. All of the contents within each subdirectory +will be also be copied. -The first decorator ``run_after('build')`` schedules this -function to be invoked after the ``build`` phase has been executed, while the -second one makes the invocation conditional on the fact that ``self.run_tests == True``. -It is also possible to schedule a function to be invoked *before* a given phase -using the ``run_before`` decorator. +For example, a package method for copying everything in the ``tests`` +subdirectory plus the ``foo.c`` and ``bar.c`` files from ``examples`` +can be implemented as shown below. .. note:: - Default implementations for build-time tests + The ``run_after`` directive ensures associated files are copied + **after** the package is installed by the build process. - Packages that are built using specific build systems may already have a - default implementation for build-time tests. For instance :py:class:`~.AutotoolsPackage` - based packages will try to invoke ``make test`` and ``make check`` if - Spack is asked to run tests. - More information on each class is available in the the :py:mod:`~.spack.build_systems` - documentation. +.. code-block:: python -.. warning:: + class MyPackage(Package): + ... - The API for adding tests is not yet considered stable and may change drastically in future releases. + @run_after('install') + def copy_test_sources(self): + srcs = ['tests', + join_path('examples', 'foo.c'), + join_path('examples', 'bar.c')] + self.cache_extra_test_sources(srcs) -.. _file-manipulation: +In this case, the method copies the associated files from the build +stage **after** the software is installed to the package's metadata +directory. The result is the following directory and files will be +available for use in stand-alone tests: -^^^^^^^^^^^^^ -Install Tests -^^^^^^^^^^^^^ +* ``join_path(self.install_test_root, 'tests')`` along with its files and subdirectories +* ``join_path(self.install_test_root, 'examples', 'foo.c')`` +* ``join_path(self.install_test_root, 'examples', 'bar.c')`` -.. warning:: +.. note:: - The API for adding and running install tests is not yet considered - stable and may change drastically in future releases. Packages with - upstreamed tests will be refactored to match changes to the API. + While source and input files are generally recommended, binaries + **may** also be cached by the build process for install testing. + Only you, as the package writer or maintainer, know whether these + would be appropriate stand-alone tests. + +.. note:: + + You are free to use a method name that is more suitable for + your package. + +.. _cache_custom_files: -While build-tests are integrated with the build system, install tests -may be added to Spack packages to be run independently of the install -method. +""""""""""""""""""" +Adding custom files +""""""""""""""""""" -Install tests may be added by defining a ``test`` method with the following signature: +Some tests may require additional files not available from the build. +Examples include: + +- test source files +- test input files +- test build scripts +- expected test output + +These extra files should be added to the ``test`` subdirectory of the +package in the Spack repository. Spack will automatically copy any files +in that directory to the test staging directory during stand-alone testing. + +The ``test`` method can access those files from the +``self.test_suite.current_test_data_dir`` directory. + +.. _expected_test_output_from_file: + +""""""""""""""""""""""""""""""""""" +Reading expected output from a file +""""""""""""""""""""""""""""""""""" + +The helper function ``get_escaped_text_output`` is available for packages +to retrieve and properly format the text from a file that contains the +output that is expected when an executable is run using ``self.run_test``. + +The signature for ``get_escaped_text_output`` is: .. code-block:: python - def test(self): + def get_escaped_text_output(filename): -These tests will be run in an environment set up to provide access to -this package and all of its dependencies, including ``test``-type -dependencies. Inside the ``test`` method, standard python ``assert`` -statements and other error reporting mechanisms can be used. Spack -will report any errors as a test failure. +where ``filename`` is the path to the file containing the expected output. -Inside the test method, individual tests can be run separately (and -continue transparently after a test failure) using the ``run_test`` -method. The signature for the ``run_test`` method is: +The ``filename`` for a :ref:`custom file <cache_custom_files>` can be +accessed and used as illustrated by a simplified version of an ``sqlite`` +package check: .. code-block:: python - def run_test(self, exe, options=[], expected=[], status=0, installed=False, - purpose='', skip_missing=False, work_dir=None): - -This method will operate in ``work_dir`` if one is specified. It will -search for an executable in the ``PATH`` variable named ``exe``, and -if ``installed=True`` it will fail if that executable does not come -from the prefix of the package being tested. If the executable is not -found, it will fail the test unless ``skip_missing`` is set to -``True``. The executable will be run with the options specified, and -the return code will be checked against the ``status`` argument, which -can be an integer or list of integers. Spack will also check that -every string in ``expected`` is a regex matching part of the output of -the executable. The ``purpose`` argument is recorded in the test log -for debugging purposes. - -"""""""""""""""""""""""""""""""""""""" -Install tests that require compilation -"""""""""""""""""""""""""""""""""""""" - -Some tests may require access to the compiler with which the package -was built, especially to test library-only packages. To ensure the -compiler is configured as part of the test environment, set the -attribute ``tests_require_compiler = True`` on the package. The -compiler will be available through the canonical environment variables -(``CC``, ``CXX``, ``FC``, ``F77``) in the test environment. - -"""""""""""""""""""""""""""""""""""""""""""""""" -Install tests that require build-time components -"""""""""""""""""""""""""""""""""""""""""""""""" - -Some packages cannot be easily tested without components from the -build-time test suite. For those packages, the -``cache_extra_test_sources`` method can be used. + class Sqlite(AutotoolsPackage): + ... + + def test(self): + test_data_dir = self.test_suite.current_test_data_dir + db_filename = test_data_dir.join('packages.db') + .. + + expected = get_escaped_text_output(test_data_dir.join('dump.out')) + self.run_test('sqlite3', + [db_filename, '.dump'], + expected, + installed=True, + purpose='test: checking dump output', + skip_missing=False) + +Expected outputs do not have to be stored with the Spack package. +Maintaining them with the source is actually preferable. + +Suppose a package's source has ``examples/foo.c`` and ``examples/foo.out`` +files that are copied for stand-alone test purposes using +:ref:`cache_extra_test_sources <cache_extra_test_sources>` and the +`run_test` method builds the executable ``examples/foo``. The package +can retrieve the expected output from ``examples/foo.out`` using: .. code-block:: python - @run_after('install') - def cache_test_sources(self): - srcs = ['./tests/foo.c', './tests/bar.c'] - self.cache_extra_test_sources(srcs) + class MyFooPackage(Package): + ... + + def test(self): + .. + filename = join_path(self.install_test_root, 'examples', 'foo.out') + expected = get_escaped_text_output(filename) + .. -This method will copy the listed methods into the metadata directory -of the package at the end of the install phase of the build. They will -be available to the test method in the directory -``self._extra_tests_path``. +Alternatively, suppose ``MyFooPackage`` installs tests in ``share/tests`` +and their outputs in ``share/tests/outputs``. The expected output for +``foo``, assuming it is still called ``foo.out``, can be retrieved as +follows: -While source files are generally recommended, for many packages -binaries may also technically be cached in this way for later testing. +.. code-block:: python -""""""""""""""""""""" -Running install tests -""""""""""""""""""""" + class MyFooPackage(Package): + ... -Install tests can be run using the ``spack test run`` command. The -``spack test run`` command will create a ``test suite`` out of the -specs provided to it, or if no specs are provided it will test all -specs in the active environment, or all specs installed in Spack if no -environment is active. Test suites can be named using the ``--alias`` -option; test suites not aliased will use the content hash of their -specs as their name. + def test(self): + .. + filename = join_path(self.prefix.share.tests.outputs, 'foo.out') + expected = get_escaped_text_output(filename) + .. -Packages to install test can be queried using the ``spack test list`` -command, which outputs all installed packages with defined ``test`` -methods. -Test suites can be found using the ``spack test find`` command. It -will list all test suites that have been run and have not been removed -using the ``spack test remove`` command. The ``spack test remove`` -command will remove tests to declutter the test stage. The ``spack -test results`` command will show results for completed test suites. +"""""""""""""""""""""""" +Adding stand-alone tests +"""""""""""""""""""""""" -The test stage is the working directory for all install tests run with -Spack. By default, Spack uses ``~/.spack/test`` as the test stage. The -test stage can be set in the high-level config: +Stand-alone tests are defined in the package's ``test`` method. The +default ``test`` method is a no-op so you'll want to override it to +implement the tests. -.. code-block:: yaml +.. note:: - config: - test_stage: /path/to/stage + Any package method named ``test`` is automatically executed by + Spack when the ``spack test run`` command is performed. + +For example, the ``MyPackage`` package below provides a skeleton for +the test method. + +.. code-block:: python + + class MyPackage(Package): + ... + + def test(self): + # TODO: Add quick checks of the installed software + pass + +Stand-alone tests run in an environment that provides access to the +package and all of its dependencies, including ``test``-type +dependencies. + +Standard python ``assert`` statements and other error reporting +mechanisms can be used in the ``test`` method. Spack will report +such errors as test failures. + +You can implement multiple tests (or test parts) within the ``test`` +method using the ``run_test`` method. Each invocation is run separately +in a manner that allows testing to continue after failures. + +The signature for ``run_test`` is: + +.. code-block:: python + + def run_test(self, exe, options=[], expected=[], status=0, + installed=False, purpose='', skip_missing=False, + work_dir=None): + +where each argument has the following meaning: + +* ``exe`` is the executable to run. + + If a name, the ``exe`` is required to be found in one of the paths + in the ``PATH`` environment variable **unless** ``skip_missing`` is + ``True``. Alternatively, a relative (to ``work_dir``) or fully + qualified path for the executable can be provided in ``exe``. + + The test will fail if the resulting path is not within the prefix + of the package being tested **unless** ``installed`` is ``False``. + +* ``options`` is a list of the command line options. + + Options are a list of strings to be passed to the executable when + it runs. + + The default is ``[]``, which means no options are provided to the + executable. + +* ``expected`` is an optional list of expected output strings. + + Spack requires every string in ``expected`` to be a regex matching + part of the output from the test run (e.g., + ``expected=['completed successfully', 'converged in']``). The + output can also include expected failure outputs (e.g., + ``expected=['failed to converge']``). + + The expected output can be :ref:`read from a file + <expected_test_output_from_file>`. + + The default is ``expected=[]``, so Spack will not check the output. + +* ``status`` is the optional expected return code(s). + + A list of return codes corresponding to successful execution can + be provided (e.g., ``status=[0,3,7]``). Support for non-zero return + codes allows for basic **expected failure** tests as well as different + return codes across versions of the software. + + The default is ``status=[0]``, which corresponds to **successful** + execution in the sense that the executable does not exit with a + failure code or raise an exception. + +* ``installed`` is used to require ``exe`` to be within the package + prefix. + + If ``True``, then the path for ``exe`` is required to be within the + package prefix; otherwise, the path is not constrained. + + The default is ``False``, so the fully qualified path for ``exe`` + does **not** need to be within the installation directory. + +* ``purpose`` is an optional heading describing the the test part. + + Output from the test is written to a test log file so this argument + serves as a searchable heading in text logs to highlight the start + of the test part. Having a description can be helpful when debugging + failing tests. + +* ``skip_missing`` is used to determine if the test should be skipped. + + If ``True``, then the test part should be skipped if the executable + is missing; otherwise, the executable must exist. This option can + be useful when test executables are removed or change as the software + evolves in subsequent versions. + + The default is ``False``, which means the test executable must be + present for any installable version of the software. + +* ``work_dir`` is the path to the directory from which the executable + will run. + + The default of ``None`` corresponds to the current directory (``'.'``). + +You may need to access files from one or more locations when writing +the tests. This can happen if the software's repository does not +include test source files or includes files but no way to build the +executables using the installed headers and libraries. In these +cases, you may need to reference the files relative to one or more +root directory and associated package property. These are given in +the table below. + +.. list-table:: Directory-to-property mapping + :header-rows: 1 + + * - Root Directory + - Package Property + - Example(s) + * - Package Installation Files + - ``self.prefix`` + - ``self.prefix.include``, ``self.prefix.lib`` + * - Package Dependency's Files + - ``self.spec['<dependency-package>'].prefix`` + - ``self.spec['trilinos'].prefix.include`` + * - Copied Build-time Files + - ``self.install_test_root`` + - ``join_path(self.install_test_root, 'examples', 'foo.c')`` + * - Custom Package Files + - ``self.test_suite.current_test_data_dir`` + - ``join_path(self.test_suite.current_test_data_dir, 'hello.f90')`` + +"""""""""""""""""""""""""""" +Inheriting stand-alone tests +"""""""""""""""""""""""""""" + +Stand-alone tests defined in parent (.e.g., :ref:`build-systems`) and +virtual (e.g., :ref:`virtual-dependencies`) packages are available to +packages that inherit from or provide interfaces for those packages, +respectively. The table below summarizes the tests that will be included +with those provided in the package itself when executing stand-alone tests. + +.. list-table:: Inherited/provided stand-alone tests + :header-rows: 1 + + * - Parent/Provider Package + - Stand-alone Tests + * - `C + <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/c>`_ + - Compiles ``hello.c`` and runs it + * - `Cxx + <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/cxx>`_ + - Compiles and runs several ``hello`` programs + * - `Fortan + <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/fortran>`_ + - Compiles and runs ``hello`` programs (``F`` and ``f90``) + * - `Mpi + <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/mpi>`_ + - Compiles and runs ``mpi_hello`` (``c``, ``fortran``) + * - `PythonPackage <build_systems/pythonpackage>` + - Imports installed modules + +These tests are very generic so it is important that package +developers and maintainers provide additional stand-alone tests +customized to the package. + +One example of a package that adds its own stand-alone (or smoke) +tests is the `Openmpi package +<https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/openmpi/package.py>`_. +The preliminary set of tests for the package performed the +following checks: + +- installed binaries with the ``--version`` option return the expected + version; +- outputs from (selected) installed binaries match expectations; +- ``make all`` succeeds when building examples that were copied from the + source directory during package installation; and +- outputs from running the copied and built examples match expectations. + +Below is an example of running and viewing the stand-alone tests, +where only the outputs for the first of each set are shown: + +.. code-block:: console + + $ spack test run --alias openmpi-4.0.5 openmpi@4.0.5 + ==> Spack test openmpi-4.0.5 + ==> Testing package openmpi-4.0.5-eygjgve + $ spack test results -l openmpi-4.0.5 + ==> Spack test openmpi-4.0.5 + ==> Testing package openmpi-4.0.5-eygjgve + ==> Results for test suite 'openmpi-4.0.5': + ==> openmpi-4.0.5-eygjgve PASSED + ==> Testing package openmpi-4.0.5-eygjgve + ==> [2021-04-26-17:35:20.259650] test: ensuring version of mpiCC is 8.3.1 + ==> [2021-04-26-17:35:20.260155] '$SPACK_ROOT/opt/spack/linux-rhel7-broadwell/gcc-8.3.1/openmpi-4.0.5-eygjgvek35awfor2qaljltjind2oa67r/bin/mpiCC' '--version' + g++ (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3) + Copyright (C) 2018 Free Software Foundation, Inc. + This is free software; see the source for copying conditions. There is NO + warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. + + PASSED + ... + ==> [2021-04-26-17:35:20.493921] test: checking mpirun output + ==> [2021-04-26-17:35:20.494461] '$SPACK_ROOT/opt/spack/linux-rhel7-broadwell/gcc-8.3.1/openmpi-4.0.5-eygjgvek35awfor2qaljltjind2oa67r/bin/mpirun' '-n' '1' 'ls' '..' + openmpi-4.0.5-eygjgve repo test_suite.lock + openmpi-4.0.5-eygjgve-test-out.txt results.txt + PASSED + ... + ==> [2021-04-26-17:35:20.630452] test: ensuring ability to build the examples + ==> [2021-04-26-17:35:20.630943] '/usr/bin/make' 'all' + mpicc -g hello_c.c -o hello_c + mpicc -g ring_c.c -o ring_c + mpicc -g connectivity_c.c -o connectivity_c + mpicc -g spc_example.c -o spc_example + ... + PASSED + ==> [2021-04-26-17:35:23.291214] test: checking hello_c example output and status (0) + ==> [2021-04-26-17:35:23.291841] './hello_c' + Hello, world, I am 0 of 1, (Open MPI v4.0.5, package: Open MPI dahlgren@quartz2300 Distribution, ident: 4.0.5, repo rev: v4.0.5, Aug 26, 2020, 114) + PASSED + ... + ==> [2021-04-26-17:35:24.603152] test: ensuring copied examples cleaned up + ==> [2021-04-26-17:35:24.603807] '/usr/bin/make' 'clean' + rm -f hello_c hello_cxx hello_mpifh hello_usempi hello_usempif08 hello_oshmem hello_oshmemcxx hello_oshmemfh Hello.class ring_c ring_cxx ring_mpifh ring_usempi ring_usempif08 ring_oshmem ring_oshmemfh Ring.class connectivity_c oshmem_shmalloc oshmem_circular_shift oshmem_max_reduction oshmem_strided_puts oshmem_symmetric_data spc_example *~ *.o + PASSED + ==> [2021-04-26-17:35:24.643360] test: mpicc: expect command status in [0] + ==> [2021-04-26-17:35:24.643834] '$SPACK_ROOT/opt/spack/linux-rhel7-broadwell/gcc-8.3.1/openmpi-4.0.5-eygjgvek35awfor2qaljltjind2oa67r/bin/mpicc' '-o' 'mpi_hello_c' '$HOME/.spack/test/hyzq5eqlqfog6fawlzxwg3prqy5vjhms/openmpi-4.0.5-eygjgve/data/mpi/mpi_hello.c' + PASSED + ==> [2021-04-26-17:35:24.776765] test: mpirun: expect command status in [0] + ==> [2021-04-26-17:35:24.777194] '$SPACK_ROOT/opt/spack/linux-rhel7-broadwell/gcc-8.3.1/openmpi-4.0.5-eygjgvek35awfor2qaljltjind2oa67r/bin/mpirun' '-np' '1' 'mpi_hello_c' + Hello world! From rank 0 of 1 + PASSED + ... + + +.. warning:: + + The API for adding and running stand-alone tests is not yet considered + stable and may change drastically in future releases. Packages with + stand-alone tests will be refactored to match changes to the API. + +.. _cmd-spack-test-list: + +""""""""""""""""""" +``spack test list`` +""""""""""""""""""" + +Packages available for install testing can be found using the +``spack test list`` command. The command outputs all installed +packages that have defined ``test`` methods. + +Alternatively you can use the ``--all`` option to get a list of +all packages that have defined ``test`` methods even if they are +not installed. + +For more information, refer to `spack test list +<https://spack.readthedocs.io/en/latest/command_index.html#spack-test-list>`_. + +.. _cmd-spack-test-run: + +"""""""""""""""""" +``spack test run`` +"""""""""""""""""" + +Install tests can be run for one or more installed packages using +the ``spack test run`` command. A ``test suite`` is created from +the provided specs. If no specs are provided it will test all specs +in the active environment or all specs installed in Spack if no +environment is active. + +Test suites can be named using the ``--alias`` option. Unaliased +Test suites will use the content hash of their specs as their name. + +Some of the more commonly used debugging options are: + +- ``--fail-fast`` stops testing each package after the first failure +- ``--fail-first`` stops testing packages after the first failure + +Test output is written to a text log file by default but ``junit`` +and ``cdash`` are outputs are available through the ``--log-format`` +option. + +For more information, refer to `spack test run +<https://spack.readthedocs.io/en/latest/command_index.html#spack-test-run>`_. + + +.. _cmd-spack-test-results: + +"""""""""""""""""""""" +``spack test results`` +"""""""""""""""""""""" + +The ``spack test results`` command shows results for all completed +test suites. Providing the alias or content hash limits reporting +to the corresponding test suite. + +The ``--logs`` option includes the output generated by the associated +test(s) to facilitate debugging. + +The ``--failed`` option limits results shown to that of the failed +tests, if any, of matching packages. + +For more information, refer to `spack test results +<https://spack.readthedocs.io/en/latest/command_index.html#spack-test-results>`_. + +.. _cmd-spack-test-find: + +""""""""""""""""""" +``spack test find`` +""""""""""""""""""" + +The ``spack test find`` command lists the aliases or content hashes +of all test suites whose results are available. + +For more information, refer to `spack test find +<https://spack.readthedocs.io/en/latest/command_index.html#spack-test-find>`_. + +.. _cmd-spack-test-remove: + +""""""""""""""""""""" +``spack test remove`` +""""""""""""""""""""" + +The ``spack test remove`` command removes test suites to declutter +the test results directory. You are prompted to confirm the removal +of each test suite **unless** you use the ``--yes-to-all`` option. + +For more information, refer to `spack test remove +<https://spack.readthedocs.io/en/latest/command_index.html#spack-test-remove>`_. + +.. _file-manipulation: --------------------------- File manipulation functions @@ -4070,9 +4767,9 @@ ways of setting compilers and options, you may need to edit files or install some files yourself to get them working with Spack. You can do this with standard Python code, and Python has rich -libraries with functions for file manipulation and filtering. Spack +libraries with functions for file manipulation and filtering. Spack also provides a number of convenience functions of its own to make -your life even easier. These functions are described in this section. +your life even easier. These functions are described in this section. All of the functions in this section can be included by simply running: @@ -4103,10 +4800,10 @@ Filtering functions .. code-block:: python - filter_file(r'^CC\s*=.*', spack_cc, 'Makefile') - filter_file(r'^CXX\s*=.*', spack_cxx, 'Makefile') - filter_file(r'^F77\s*=.*', spack_f77, 'Makefile') - filter_file(r'^FC\s*=.*', spack_fc, 'Makefile') + filter_file(r'^\s*CC\s*=.*', 'CC = ' + spack_cc, 'Makefile') + filter_file(r'^\s*CXX\s*=.*', 'CXX = ' + spack_cxx, 'Makefile') + filter_file(r'^\s*F77\s*=.*', 'F77 = ' + spack_f77, 'Makefile') + filter_file(r'^\s*FC\s*=.*', 'FC = ' + spack_fc, 'Makefile') #. Replacing ``#!/usr/bin/perl`` with ``#!/usr/bin/env perl`` in ``bib2xhtml``: diff --git a/lib/spack/docs/pipelines.rst b/lib/spack/docs/pipelines.rst index d0cb6fa670..973ec2182c 100644 --- a/lib/spack/docs/pipelines.rst +++ b/lib/spack/docs/pipelines.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -122,9 +122,26 @@ pipeline jobs. Concretizes the specs in the active environment, stages them (as described in :ref:`staging_algorithm`), and writes the resulting ``.gitlab-ci.yml`` to disk. -This sub-command takes two arguments, but the most useful is ``--output-file``, -which should be an absolute path (including file name) to the generated -pipeline, if the default (``./.gitlab-ci.yml``) is not desired. +Using ``--prune-dag`` or ``--no-prune-dag`` configures whether or not jobs are +generated for specs that are already up to date on the mirror. If enabling +DAG pruning using ``--prune-dag``, more information may be required in your +``spack.yaml`` file, see the :ref:`noop_jobs` section below regarding +``service-job-attributes``. + +The ``--optimize`` argument is experimental and runs the generated pipeline +document through a series of optimization passes designed to reduce the size +of the generated file. + +The ``--dependencies`` is also experimental and disables what in Gitlab is +referred to as DAG scheduling, internally using the ``dependencies`` keyword +rather than ``needs`` to list dependency jobs. The drawback of using this option +is that before any job can begin, all jobs in previous stages must first +complete. The benefit is that Gitlab allows more dependencies to be listed +when using ``dependencies`` instead of ``needs``. + +The optional ``--output-file`` argument should be an absolute path (including +file name) to the generated pipeline, and if not given, the default is +``./.gitlab-ci.yml``. .. _cmd-spack-ci-rebuild: @@ -223,20 +240,12 @@ takes a boolean and determines whether the pipeline uses artifacts to store and pass along the buildcaches from one stage to the next (the default if you don't provide this option is ``False``). -The -``final-stage-rebuild-index`` section controls whether an extra job is added to the -end of your pipeline (in a stage by itself) which will regenerate the mirror's -buildcache index. Under normal operation, each pipeline job that rebuilds a package -will re-generate the mirror's buildcache index after the buildcache entry for that -job has been created and pushed to the mirror. Since jobs in the same stage can run in -parallel, there is the possibility that at the end of some stage, the index may not -reflect all the binaries in the buildcache. Adding the ``final-stage-rebuild-index`` -section ensures that at the end of the pipeline, the index will be in sync with the -binaries on the mirror. If the mirror lives in an S3 bucket, this job will need to -run on a machine with the Python ``boto3`` module installed, and consequently the -``final-stage-rebuild-index`` needs to specify a list of ``tags`` to pick a runner -satisfying that condition. It can also take an ``image`` key so Docker executor type -runners can pick the right image for the index regeneration job. +The optional ``broken-specs-url`` key tells Spack to check against a list of +specs that are known to be currently broken in ``develop``. If any such specs +are found, the ``spack ci generate`` command will fail with an error message +informing the user what broken specs were encountered. This allows the pipeline +to fail early and avoid wasting compute resources attempting to build packages +that will not succeed. The optional ``cdash`` section provides information that will be used by the ``spack ci generate`` command (invoked by ``spack ci start``) for reporting @@ -251,6 +260,76 @@ Take a look at the for the gitlab-ci section of the spack environment file, to see precisely what syntax is allowed there. +.. _rebuild_index: + +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Note about rebuilding buildcache index +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +By default, while a pipeline job may rebuild a package, create a buildcache +entry, and push it to the mirror, it does not automatically re-generate the +mirror's buildcache index afterward. Because the index is not needed by the +default rebuild jobs in the pipeline, not updating the index at the end of +each job avoids possible race conditions between simultaneous jobs, and it +avoids the computational expense of regenerating the index. This potentially +saves minutes per job, depending on the number of binary packages in the +mirror. As a result, the default is that the mirror's buildcache index may +not correctly reflect the mirror's contents at the end of a pipeline. + +To make sure the buildcache index is up to date at the end of your pipeline, +spack generates a job to update the buildcache index of the target mirror +at the end of each pipeline by default. You can disable this behavior by +adding ``rebuild-index: False`` inside the ``gitlab-ci`` section of your +spack environment. Spack will assign the job any runner attributes found +on the ``service-job-attributes``, if you have provided that in your +``spack.yaml``. + +.. _noop_jobs: + +^^^^^^^^^^^^^^^^^^^^^^^ +Note about "no-op" jobs +^^^^^^^^^^^^^^^^^^^^^^^ + +If no specs in an environment need to be rebuilt during a given pipeline run +(meaning all are already up to date on the mirror), a single succesful job +(a NO-OP) is still generated to avoid an empty pipeline (which GitLab +considers to be an error). An optional ``service-job-attributes`` section +can be added to your ``spack.yaml`` where you can provide ``tags`` and +``image`` or ``variables`` for the generated NO-OP job. This section also +supports providing ``before_script``, ``script``, and ``after_script``, in +case you want to take some custom actions in the case of any empty pipeline. + +Following is an example of this section added to a ``spack.yaml``: + +.. code-block:: yaml + + spack: + specs: + - openmpi + mirrors: + cloud_gitlab: https://mirror.spack.io + gitlab-ci: + mappings: + - match: + - os=centos8 + runner-attributes: + tags: + - custom + - tag + image: spack/centos7 + service-job-attributes: + tags: ['custom', 'tag'] + image: + name: 'some.image.registry/custom-image:latest' + entrypoint: ['/bin/bash'] + script: + - echo "Custom message in a custom script" + +The example above illustrates how you can provide the attributes used to run +the NO-OP job in the case of an empty pipeline. The only field for the NO-OP +job that might be generated for you is ``script``, but that will only happen +if you do not provide one yourself. + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Assignment of specs to runners ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -482,7 +561,7 @@ provision of a custom ``script`` section. The reason for this is to run Now imagine you have long pipelines with many specs to be built, and you are pointing to a spack repository and branch that has a tendency to change -frequently, such as the main repo and it's ``develop`` branch. If each child +frequently, such as the main repo and its ``develop`` branch. If each child job checks out the ``develop`` branch, that could result in some jobs running with one SHA of spack, while later jobs run with another. To help avoid this issue, the pipeline generation process saves global variables called diff --git a/lib/spack/docs/repositories.rst b/lib/spack/docs/repositories.rst index 4a2c163886..3368c26a28 100644 --- a/lib/spack/docs/repositories.rst +++ b/lib/spack/docs/repositories.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -9,7 +9,7 @@ Package Repositories ============================= -Spack comes with over 1,000 built-in package recipes in +Spack comes with thousands of built-in package recipes in ``var/spack/repos/builtin/``. This is a **package repository** -- a directory that Spack searches when it needs to find a package by name. You may need to maintain packages for restricted, proprietary or diff --git a/lib/spack/docs/spack.yaml b/lib/spack/docs/spack.yaml index 3c8cd14d10..69d0e301fd 100644 --- a/lib/spack/docs/spack.yaml +++ b/lib/spack/docs/spack.yaml @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/docs/workflows.rst b/lib/spack/docs/workflows.rst index 4d3a97012c..c327c137ac 100644 --- a/lib/spack/docs/workflows.rst +++ b/lib/spack/docs/workflows.rst @@ -1,4 +1,4 @@ -.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details. SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/env/cc b/lib/spack/env/cc index 5efe015c9e..bd479f05ec 100755 --- a/lib/spack/env/cc +++ b/lib/spack/env/cc @@ -1,6 +1,6 @@ #!/bin/bash # -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -125,7 +125,7 @@ case "$command" in comp="FC" lang_flags=F ;; - f77|xlf|xlf_r|pgf77|frt|flang) + f77|xlf|xlf_r|pgf77) command="$SPACK_F77" language="Fortran 77" comp="F77" @@ -277,14 +277,22 @@ other_args=() isystem_system_includes=() isystem_includes=() -while [ -n "$1" ]; do +while [ $# -ne 0 ]; do + # an RPATH to be added after the case statement. rp="" + # Multiple consecutive spaces in the command line can + # result in blank arguments + if [ -z "$1" ]; then + shift + continue + fi + case "$1" in -isystem*) arg="${1#-isystem}" - isystem_was_used=true + isystem_was_used=true if [ -z "$arg" ]; then shift; arg="$1"; fi if system_dir "$arg"; then isystem_system_includes+=("$arg") @@ -311,6 +319,16 @@ while [ -n "$1" ]; do fi ;; -l*) + # -loopopt=0 is generated erroneously in autoconf <= 2.69, + # and passed by ifx to the linker, which confuses it with a + # library. Filter it out. + # TODO: generalize filtering of args with an env var, so that + # TODO: we do not have to special case this here. + if { [ "$mode" = "ccld" ] || [ $mode = "ld" ]; } \ + && [ "$1" != "${1#-loopopt}" ]; then + shift + continue + fi arg="${1#-l}" if [ -z "$arg" ]; then shift; arg="$1"; fi other_args+=("-l$arg") @@ -320,9 +338,13 @@ while [ -n "$1" ]; do if [ -z "$arg" ]; then shift; arg="$1"; fi if [[ "$arg" = -rpath=* ]]; then rp="${arg#-rpath=}" + elif [[ "$arg" = --rpath=* ]]; then + rp="${arg#--rpath=}" elif [[ "$arg" = -rpath,* ]]; then rp="${arg#-rpath,}" - elif [[ "$arg" = -rpath ]]; then + elif [[ "$arg" = --rpath,* ]]; then + rp="${arg#--rpath,}" + elif [[ "$arg" =~ ^-?-rpath$ ]]; then shift; arg="$1" if [[ "$arg" != -Wl,* ]]; then die "-Wl,-rpath was not followed by -Wl,*" @@ -339,7 +361,9 @@ while [ -n "$1" ]; do if [ -z "$arg" ]; then shift; arg="$1"; fi if [[ "$arg" = -rpath=* ]]; then rp="${arg#-rpath=}" - elif [[ "$arg" = -rpath ]]; then + elif [[ "$arg" = --rpath=* ]]; then + rp="${arg#--rpath=}" + elif [[ "$arg" = -rpath ]] || [[ "$arg" = --rpath ]]; then shift; arg="$1" if [[ "$arg" != -Xlinker,* ]]; then die "-Xlinker,-rpath was not followed by -Xlinker,*" @@ -434,7 +458,7 @@ then ld) flags=("${flags[@]}" -headerpad_max_install_names) ;; ccld) - flags=("${flags[@]}" -Wl,-headerpad_max_install_names) ;; + flags=("${flags[@]}" "-Wl,-headerpad_max_install_names") ;; esac fi @@ -491,19 +515,19 @@ args+=("${flags[@]}") # Insert include directories just prior to any system include directories for dir in "${includes[@]}"; do args+=("-I$dir"); done -for dir in "${isystem_includes[@]}"; do args+=("-isystem$dir"); done +for dir in "${isystem_includes[@]}"; do args+=("-isystem" "$dir"); done IFS=':' read -ra spack_include_dirs <<< "$SPACK_INCLUDE_DIRS" if [[ $mode == cpp || $mode == cc || $mode == as || $mode == ccld ]]; then if [[ "$isystem_was_used" == "true" ]] ; then - for dir in "${spack_include_dirs[@]}"; do args+=("-isystem$dir"); done + for dir in "${spack_include_dirs[@]}"; do args+=("-isystem" "$dir"); done else - for dir in "${spack_include_dirs[@]}"; do args+=("-I$dir"); done + for dir in "${spack_include_dirs[@]}"; do args+=("-I$dir"); done fi fi for dir in "${system_includes[@]}"; do args+=("-I$dir"); done -for dir in "${isystem_system_includes[@]}"; do args+=("-isystem$dir"); done +for dir in "${isystem_system_includes[@]}"; do args+=("-isystem" "$dir"); done # Library search paths for dir in "${libdirs[@]}"; do args+=("-L$dir"); done @@ -512,12 +536,12 @@ for dir in "${system_libdirs[@]}"; do args+=("-L$dir"); done # RPATHs arguments case "$mode" in ccld) - if [ ! -z "$dtags_to_add" ] ; then args+=("$linker_arg$dtags_to_add") ; fi + if [ -n "$dtags_to_add" ] ; then args+=("$linker_arg$dtags_to_add") ; fi for dir in "${rpaths[@]}"; do args+=("$rpath$dir"); done for dir in "${system_rpaths[@]}"; do args+=("$rpath$dir"); done ;; ld) - if [ ! -z "$dtags_to_add" ] ; then args+=("$dtags_to_add") ; fi + if [ -n "$dtags_to_add" ] ; then args+=("$dtags_to_add") ; fi for dir in "${rpaths[@]}"; do args+=("-rpath" "$dir"); done for dir in "${system_rpaths[@]}"; do args+=("-rpath" "$dir"); done ;; diff --git a/lib/spack/external/__init__.py b/lib/spack/external/__init__.py index 1c98852eb6..1d707a6d9a 100644 --- a/lib/spack/external/__init__.py +++ b/lib/spack/external/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -11,7 +11,7 @@ archspec * Homepage: https://pypi.python.org/pypi/archspec * Usage: Labeling, comparison and detection of microarchitectures -* Version: 0.1.2 (commit 2846749dc5b12ae2b30ff1d3f0270a4a5954710d) +* Version: 0.1.2 (commit 130607c373fd88cd3c43da94c0d3afd3a44084b0) argparse -------- diff --git a/lib/spack/external/archspec/__init__.py b/lib/spack/external/archspec/__init__.py index 1349425634..1188c6cecc 100644 --- a/lib/spack/external/archspec/__init__.py +++ b/lib/spack/external/archspec/__init__.py @@ -1,2 +1,2 @@ """Init file to avoid namespace packages""" -__version__ = "0.1.1" +__version__ = "0.1.2" diff --git a/lib/spack/external/archspec/cpu/detect.py b/lib/spack/external/archspec/cpu/detect.py index e65076c788..eb75cdfb0e 100644 --- a/lib/spack/external/archspec/cpu/detect.py +++ b/lib/spack/external/archspec/cpu/detect.py @@ -99,17 +99,29 @@ def sysctl_info_dict(): def sysctl(*args): return _check_output(["sysctl"] + list(args), env=child_environment).strip() - flags = ( - sysctl("-n", "machdep.cpu.features").lower() - + " " - + sysctl("-n", "machdep.cpu.leaf7_features").lower() - ) - info = { - "vendor_id": sysctl("-n", "machdep.cpu.vendor"), - "flags": flags, - "model": sysctl("-n", "machdep.cpu.model"), - "model name": sysctl("-n", "machdep.cpu.brand_string"), - } + if platform.machine() == "x86_64": + flags = ( + sysctl("-n", "machdep.cpu.features").lower() + + " " + + sysctl("-n", "machdep.cpu.leaf7_features").lower() + ) + info = { + "vendor_id": sysctl("-n", "machdep.cpu.vendor"), + "flags": flags, + "model": sysctl("-n", "machdep.cpu.model"), + "model name": sysctl("-n", "machdep.cpu.brand_string"), + } + else: + model = ( + "m1" if "Apple" in sysctl("-n", "machdep.cpu.brand_string") else "unknown" + ) + info = { + "vendor_id": "Apple", + "flags": [], + "model": model, + "CPU implementer": "Apple", + "model name": sysctl("-n", "machdep.cpu.brand_string"), + } return info @@ -173,6 +185,11 @@ def compatible_microarchitectures(info): info (dict): dictionary containing information on the host cpu """ architecture_family = platform.machine() + # On Apple M1 platform.machine() returns "arm64" instead of "aarch64" + # so we should normalize the name here + if architecture_family == "arm64": + architecture_family = "aarch64" + # If a tester is not registered, be conservative and assume no known # target is compatible with the host tester = COMPATIBILITY_CHECKS.get(architecture_family, lambda x, y: False) diff --git a/lib/spack/external/archspec/json/cpu/microarchitectures.json b/lib/spack/external/archspec/json/cpu/microarchitectures.json index 889848c8ec..0eacadfbaa 100644 --- a/lib/spack/external/archspec/json/cpu/microarchitectures.json +++ b/lib/spack/external/archspec/json/cpu/microarchitectures.json @@ -75,6 +75,13 @@ "flags": "-march={name} -mtune=generic" } ], + "aocc": [ + { + "versions": "2.2:", + "name": "x86-64", + "flags": "-march={name} -mtune=generic" + } + ], "intel": [ { "versions": ":", @@ -84,6 +91,166 @@ ] } }, + "x86_64_v2": { + "from": ["x86_64"], + "vendor": "generic", + "features": [ + "cx16", + "lahf_lm", + "mmx", + "sse", + "sse2", + "ssse3", + "sse4_1", + "sse4_2", + "popcnt" + ], + "compilers": { + "gcc": [ + { + "versions": "11.1:", + "name": "x86-64-v2", + "flags": "-march={name} -mtune=generic" + }, + { + "versions": "4.6:11.0", + "name": "x86-64", + "flags": "-march={name} -mtune=generic -mcx16 -msahf -mpopcnt -msse3 -msse4.1 -msse4.2 -mssse3" + } + ], + "clang": [ + { + "versions": "12.0:", + "name": "x86-64-v2", + "flags": "-march={name} -mtune=generic" + }, + { + "versions": "3.9:11.1", + "name": "x86-64", + "flags": "-march={name} -mtune=generic -mcx16 -msahf -mpopcnt -msse3 -msse4.1 -msse4.2 -mssse3" + } + ] + } + }, + "x86_64_v3": { + "from": ["x86_64_v2"], + "vendor": "generic", + "features": [ + "cx16", + "lahf_lm", + "mmx", + "sse", + "sse2", + "ssse3", + "sse4_1", + "sse4_2", + "popcnt", + "avx", + "avx2", + "bmi1", + "bmi2", + "f16c", + "fma", + "abm", + "movbe", + "xsave" + ], + "compilers": { + "gcc": [ + { + "versions": "11.1:", + "name": "x86-64-v3", + "flags": "-march={name} -mtune=generic" + }, + { + "versions": "4.8:11.0", + "name": "x86-64", + "flags": "-march={name} -mtune=generic -mcx16 -msahf -mpopcnt -msse3 -msse4.1 -msse4.2 -mssse3 -mavx -mavx2 -mbmi -mbmi2 -mf16c -mfma -mlzcnt -mmovbe -mxsave" + } + ], + "clang": [ + { + "versions": "12.0:", + "name": "x86-64-v3", + "flags": "-march={name} -mtune=generic" + }, + { + "versions": "3.9:11.1", + "name": "x86-64", + "flags": "-march={name} -mtune=generic -mcx16 -msahf -mpopcnt -msse3 -msse4.1 -msse4.2 -mssse3 -mavx -mavx2 -mbmi -mbmi2 -mf16c -mfma -mlzcnt -mmovbe -mxsave" + } + ], + "apple-clang": [ + { + "versions": "8.0:", + "name": "x86-64", + "flags": "-march={name} -mtune=generic -mcx16 -msahf -mpopcnt -msse3 -msse4.1 -msse4.2 -mssse3 -mavx -mavx2 -mbmi -mbmi2 -mf16c -mfma -mlzcnt -mmovbe -mxsave" + } + ] + } + }, + "x86_64_v4": { + "from": ["x86_64_v3"], + "vendor": "generic", + "features": [ + "cx16", + "lahf_lm", + "mmx", + "sse", + "sse2", + "ssse3", + "sse4_1", + "sse4_2", + "popcnt", + "avx", + "avx2", + "bmi1", + "bmi2", + "f16c", + "fma", + "abm", + "movbe", + "xsave", + "avx512f", + "avx512bw", + "avx512cd", + "avx512dq", + "avx512vl" + ], + "compilers": { + "gcc": [ + { + "versions": "11.1:", + "name": "x86-64-v4", + "flags": "-march={name} -mtune=generic" + }, + { + "versions": "6.0:11.0", + "name": "x86-64", + "flags": "-march={name} -mtune=generic -mcx16 -msahf -mpopcnt -msse3 -msse4.1 -msse4.2 -mssse3 -mavx -mavx2 -mbmi -mbmi2 -mf16c -mfma -mlzcnt -mmovbe -mxsave -mavx512f -mavx512bw -mavx512cd -mavx512dq -mavx512vl" + } + ], + "clang": [ + { + "versions": "12.0:", + "name": "x86-64-v4", + "flags": "-march={name} -mtune=generic" + }, + { + "versions": "3.9:11.1", + "name": "x86-64", + "flags": "-march={name} -mtune=generic -mcx16 -msahf -mpopcnt -msse3 -msse4.1 -msse4.2 -mssse3 -mavx -mavx2 -mbmi -mbmi2 -mf16c -mfma -mlzcnt -mmovbe -mxsave -mavx512f -mavx512bw -mavx512cd -mavx512dq -mavx512vl" + } + ], + "apple-clang": [ + { + "versions": "8.0:", + "name": "x86-64", + "flags": "-march={name} -mtune=generic -mcx16 -msahf -mpopcnt -msse3 -msse4.1 -msse4.2 -mssse3 -mavx -mavx2 -mbmi -mbmi2 -mf16c -mfma -mlzcnt -mmovbe -mxsave -mavx512f -mavx512bw -mavx512cd -mavx512dq -mavx512vl" + } + ] + } + }, "nocona": { "from": ["x86_64"], "vendor": "GenuineIntel", @@ -106,6 +273,12 @@ "flags": "-march={name} -mtune={name}" } ], + "aocc": [ + { + "versions": "2.2:", + "flags": "-march={name} -mtune=generic" + } + ], "apple-clang": [ { "versions": "8.0:", @@ -143,6 +316,12 @@ "flags": "-march={name} -mtune={name}" } ], + "aocc": [ + { + "versions": "2.2:", + "flags": "-march={name} -mtune=generic" + } + ], "apple-clang": [ { "versions": "8.0:", @@ -152,13 +331,13 @@ "intel": [ { "versions": "16.0:", - "flags": "-march={name} -mtune={name}}" + "flags": "-march={name} -mtune={name}" } ] } }, "nehalem": { - "from": ["core2"], + "from": ["core2", "x86_64_v2"], "vendor": "GenuineIntel", "features": [ "mmx", @@ -187,6 +366,12 @@ "flags": "-march={name} -mtune={name}" } ], + "aocc": [ + { + "versions": "2.2:", + "flags": "-march={name} -mtune=generic" + } + ], "apple-clang": [ { "versions": "8.0:", @@ -229,6 +414,12 @@ "flags": "-march={name} -mtune={name}" } ], + "aocc": [ + { + "versions": "2.2:", + "flags": "-march={name} -mtune=generic" + } + ], "apple-clang": [ { "versions": "8.0:", @@ -277,6 +468,12 @@ "flags": "-march={name} -mtune={name}" } ], + "aocc": [ + { + "versions": "2.2:", + "flags": "-march={name} -mtune={name}" + } + ], "apple-clang": [ { "versions": "8.0:", @@ -331,6 +528,12 @@ "flags": "-march={name} -mtune={name}" } ], + "aocc": [ + { + "versions": "2.2:", + "flags": "-march={name} -mtune={name}" + } + ], "apple-clang": [ { "versions": "8.0:", @@ -351,7 +554,7 @@ } }, "haswell": { - "from": ["ivybridge"], + "from": ["ivybridge", "x86_64_v3"], "vendor": "GenuineIntel", "features": [ "mmx", @@ -390,6 +593,12 @@ "flags": "-march={name} -mtune={name}" } ], + "aocc": [ + { + "versions": "2.2:", + "flags": "-march={name} -mtune={name}" + } + ], "apple-clang": [ { "versions": "8.0:", @@ -446,6 +655,12 @@ "flags": "-march={name} -mtune={name}" } ], + "aocc": [ + { + "versions": "2.2:", + "flags": "-march={name} -mtune={name}" + } + ], "apple-clang": [ { "versions": "8.0:", @@ -500,6 +715,12 @@ "flags": "-march={name} -mtune={name}" } ], + "aocc": [ + { + "versions": "2.2:", + "flags": "-march={name} -mtune={name}" + } + ], "apple-clang": [ { "versions": "8.0:", @@ -558,6 +779,13 @@ "flags": "-march={name} -mtune={name}" } ], + "aocc": [ + { + "versions": "2.2:", + "name": "knl", + "flags": "-march={name} -mtune=generic" + } + ], "apple-clang": [ { "versions": "8.0:", @@ -574,7 +802,7 @@ } }, "skylake_avx512": { - "from": ["skylake"], + "from": ["skylake", "x86_64_v4"], "vendor": "GenuineIntel", "features": [ "mmx", @@ -621,6 +849,13 @@ "flags": "-march={name} -mtune={name}" } ], + "aocc": [ + { + "versions": "2.2:", + "name": "skylake-avx512", + "flags": "-march={name} -mtune=generic" + } + ], "apple-clang": [ { "versions": "8.0:", @@ -685,6 +920,12 @@ "flags": "-march={name} -mtune={name}" } ], + "aocc": [ + { + "versions": "2.2:", + "flags": "-march={name} -mtune={name}" + } + ], "apple-clang": [ { "versions": "8.0:", @@ -746,6 +987,12 @@ "flags": "-march={name} -mtune={name}" } ], + "aocc": [ + { + "versions": "2.2:", + "flags": "-march={name} -mtune={name}" + } + ], "apple-clang": [ { "versions": "11.0:", @@ -754,7 +1001,7 @@ ], "intel": [ { - "versions": "19.0:", + "versions": "19.0.1:", "flags": "-march={name} -mtune={name}" } ] @@ -827,6 +1074,13 @@ "flags": "-march={name} -mtune={name}" } ], + "aocc": [ + { + "versions": "2.2:", + "name": "icelake-client", + "flags": "-march={name} -mtune={name}" + } + ], "apple-clang": [ { "versions": "10.0.1:", @@ -875,6 +1129,13 @@ "flags": "-march={name} -mtune={name}" } ], + "aocc": [ + { + "versions": "2.2:", + "name": "amdfam10", + "flags": "-march={name} -mtune={name}" + } + ], "intel": [ { "versions": "16.0:", @@ -885,7 +1146,7 @@ } }, "bulldozer": { - "from": ["x86_64"], + "from": ["x86_64_v2"], "vendor": "AuthenticAMD", "features": [ "mmx", @@ -918,6 +1179,13 @@ "flags": "-march={name} -mtune={name}" } ], + "aocc": [ + { + "versions": "2.2:", + "name": "bdver1", + "flags": "-march={name} -mtune={name}" + } + ], "intel": [ { "versions": "16.0:", @@ -965,6 +1233,13 @@ "flags": "-march={name} -mtune={name}" } ], + "aocc": [ + { + "versions": "2.2:", + "name": "bdver2", + "flags": "-march={name} -mtune={name}" + } + ], "intel": [ { "versions": "16.0:", @@ -1013,6 +1288,13 @@ "flags": "-march={name} -mtune={name}" } ], + "aocc": [ + { + "versions": "2.2:", + "name": "bdver3", + "flags": "-march={name} -mtune={name}" + } + ], "intel": [ { "versions": "16.0:", @@ -1023,7 +1305,7 @@ } }, "excavator": { - "from": ["steamroller"], + "from": ["steamroller", "x86_64_v3"], "vendor": "AuthenticAMD", "features": [ "mmx", @@ -1064,6 +1346,13 @@ "flags": "-march={name} -mtune={name}" } ], + "aocc": [ + { + "versions": "2.2:", + "name": "bdver4", + "flags": "-march={name} -mtune={name}" + } + ], "intel": [ { "versions": "16.0:", @@ -1075,7 +1364,7 @@ } }, "zen": { - "from": ["x86_64"], + "from": ["x86_64_v3"], "vendor": "AuthenticAMD", "features": [ "bmi1", @@ -1119,6 +1408,13 @@ "flags": "-march={name} -mtune={name}" } ], + "aocc": [ + { + "versions": "2.2:", + "name": "znver1", + "flags": "-march={name} -mtune={name}" + } + ], "intel": [ { "versions": "16.0:", @@ -1175,6 +1471,13 @@ "flags": "-march={name} -mtune={name}" } ], + "aocc": [ + { + "versions": "2.2:", + "name": "znver2", + "flags": "-march={name} -mtune={name}" + } + ], "intel": [ { "versions": "16.0:", @@ -1185,6 +1488,64 @@ ] } }, + "zen3": { + "from": ["zen2"], + "vendor": "AuthenticAMD", + "features": [ + "bmi1", + "bmi2", + "f16c", + "fma", + "fsgsbase", + "avx", + "avx2", + "rdseed", + "clzero", + "aes", + "pclmulqdq", + "cx16", + "movbe", + "mmx", + "sse", + "sse2", + "sse4a", + "ssse3", + "sse4_1", + "sse4_2", + "abm", + "xsavec", + "xsaveopt", + "clflushopt", + "popcnt", + "clwb", + "vaes", + "vpclmulqdq", + "pku" + ], + "compilers": { + "gcc": [ + { + "versions": "10.3:", + "name": "znver3", + "flags": "-march={name} -mtune={name}" + } + ], + "clang": [ + { + "versions": "12.0:", + "name": "znver3", + "flags": "-march={name} -mtune={name}" + } + ], + "aocc": [ + { + "versions": "3.0:", + "name": "znver3", + "flags": "-march={name} -mtune={name}" + } + ] + } + }, "ppc64": { "from": [], "vendor": "generic", @@ -1358,6 +1719,12 @@ "versions": ":", "flags": "-march=armv8-a -mtune=generic" } + ], + "apple-clang": [ + { + "versions": ":", + "flags": "-march=armv8-a -mtune=generic" + } ] } }, @@ -1566,6 +1933,31 @@ ] } }, + "m1": { + "from": ["aarch64"], + "vendor": "Apple", + "features": [], + "compilers": { + "gcc": [ + { + "versions": "8.0:", + "flags" : "-march=armv8.4-a -mtune=generic" + } + ], + "clang" : [ + { + "versions": "9.0:", + "flags" : "-march=armv8.4-a" + } + ], + "apple-clang": [ + { + "versions": "11.0:", + "flags" : "-march=armv8.4-a" + } + ] + } + }, "arm": { "from": [], "vendor": "generic", diff --git a/lib/spack/external/ordereddict_backport.py b/lib/spack/external/ordereddict_backport.py index 5ec8493cc9..3c7f012e9e 100644 --- a/lib/spack/external/ordereddict_backport.py +++ b/lib/spack/external/ordereddict_backport.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/external/py2/typing.py b/lib/spack/external/py2/typing.py new file mode 100644 index 0000000000..cc32de8844 --- /dev/null +++ b/lib/spack/external/py2/typing.py @@ -0,0 +1,84 @@ +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) +""" +This is a fake set of symbols to allow spack to import typing in python +versions where we do not support type checking (<3) +""" +Annotated = None +Any = None +Callable = None +ForwardRef = None +Generic = None +Literal = None +Optional = None +Tuple = None +TypeVar = None +Union = None +AbstractSet = None +ByteString = None +Container = None +Hashable = None +ItemsView = None +Iterable = None +Iterator = None +KeysView = None +Mapping = None +MappingView = None +MutableMapping = None +MutableSequence = None +MutableSet = None +Sequence = None +Sized = None +ValuesView = None +Awaitable = None +AsyncIterator = None +AsyncIterable = None +Coroutine = None +Collection = None +AsyncGenerator = None +AsyncContextManager = None +Reversible = None +SupportsAbs = None +SupportsBytes = None +SupportsComplex = None +SupportsFloat = None +SupportsInt = None +SupportsRound = None +ChainMap = None +Dict = None +List = None +OrderedDict = None +Set = None +FrozenSet = None +NamedTuple = None +Generator = None +AnyStr = None +cast = None +get_args = None +get_origin = None +get_type_hints = None +no_type_check = None +no_type_check_decorator = None +NoReturn = None + +# these are the typing extension symbols +ClassVar = None +Final = None +Protocol = None +Type = None +TypedDict = None +ContextManager = None +Counter = None +Deque = None +DefaultDict = None +SupportsIndex = None +final = None +IntVar = None +Literal = None +NewType = None +overload = None +runtime_checkable = None +Text = None +TYPE_CHECKING = None diff --git a/lib/spack/llnl/__init__.py b/lib/spack/llnl/__init__.py index 9f87532b85..103eae6134 100644 --- a/lib/spack/llnl/__init__.py +++ b/lib/spack/llnl/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/llnl/util/__init__.py b/lib/spack/llnl/util/__init__.py index 9f87532b85..103eae6134 100644 --- a/lib/spack/llnl/util/__init__.py +++ b/lib/spack/llnl/util/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/llnl/util/argparsewriter.py b/lib/spack/llnl/util/argparsewriter.py index 8ecf6acc88..9b90096460 100644 --- a/lib/spack/llnl/util/argparsewriter.py +++ b/lib/spack/llnl/util/argparsewriter.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/llnl/util/filesystem.py b/lib/spack/llnl/util/filesystem.py index bc55d29be0..c7ecf09af2 100644 --- a/lib/spack/llnl/util/filesystem.py +++ b/lib/spack/llnl/util/filesystem.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -69,7 +69,8 @@ __all__ = [ 'touchp', 'traverse_tree', 'unset_executable_mode', - 'working_dir' + 'working_dir', + 'keep_modification_time' ] @@ -1825,3 +1826,24 @@ def remove_directory_contents(dir): os.unlink(entry) else: shutil.rmtree(entry) + + +@contextmanager +def keep_modification_time(*filenames): + """ + Context manager to keep the modification timestamps of the input files. + Tolerates and has no effect on non-existent files and files that are + deleted by the nested code. + + Parameters: + *filenames: one or more files that must have their modification + timestamps unchanged + """ + mtimes = {} + for f in filenames: + if os.path.exists(f): + mtimes[f] = os.path.getmtime(f) + yield + for f, mtime in mtimes.items(): + if os.path.exists(f): + os.utime(f, (os.path.getatime(f), mtime)) diff --git a/lib/spack/llnl/util/lang.py b/lib/spack/llnl/util/lang.py index 471f90490f..b20cd91db4 100644 --- a/lib/spack/llnl/util/lang.py +++ b/lib/spack/llnl/util/lang.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -14,6 +14,11 @@ from datetime import datetime, timedelta from six import string_types import sys +if sys.version_info < (3, 0): + from itertools import izip_longest # novm + zip_longest = izip_longest +else: + from itertools import zip_longest # novm if sys.version_info >= (3, 3): from collections.abc import Hashable, MutableMapping # novm @@ -227,48 +232,222 @@ def list_modules(directory, **kwargs): yield re.sub('.py$', '', name) -def key_ordering(cls): - """Decorates a class with extra methods that implement rich comparison - operations and ``__hash__``. The decorator assumes that the class - implements a function called ``_cmp_key()``. The rich comparison - operations will compare objects using this key, and the ``__hash__`` - function will return the hash of this key. +def decorator_with_or_without_args(decorator): + """Allows a decorator to be used with or without arguments, e.g.:: + + # Calls the decorator function some args + @decorator(with, arguments, and=kwargs) + + or:: + + # Calls the decorator function with zero arguments + @decorator + + """ + # See https://stackoverflow.com/questions/653368 for more on this + @functools.wraps(decorator) + def new_dec(*args, **kwargs): + if len(args) == 1 and len(kwargs) == 0 and callable(args[0]): + # actual decorated function + return decorator(args[0]) + else: + # decorator arguments + return lambda realf: decorator(realf, *args, **kwargs) + + return new_dec + + +#: sentinel for testing that iterators are done in lazy_lexicographic_ordering +done = object() + + +def tuplify(seq): + """Helper for lazy_lexicographic_ordering().""" + return tuple((tuplify(x) if callable(x) else x) for x in seq()) + + +def lazy_eq(lseq, rseq): + """Equality comparison for two lazily generated sequences. + + See ``lazy_lexicographic_ordering``. + """ + liter = lseq() # call generators + riter = rseq() + + # zip_longest is implemented in native code, so use it for speed. + # use zip_longest instead of zip because it allows us to tell + # which iterator was longer. + for left, right in zip_longest(liter, riter, fillvalue=done): + if (left is done) or (right is done): + return False + + # recursively enumerate any generators, otherwise compare + equal = lazy_eq(left, right) if callable(left) else left == right + if not equal: + return False + + return True + + +def lazy_lt(lseq, rseq): + """Less-than comparison for two lazily generated sequences. + + See ``lazy_lexicographic_ordering``. + """ + liter = lseq() + riter = rseq() + + for left, right in zip_longest(liter, riter, fillvalue=done): + if (left is done) or (right is done): + return left is done # left was shorter than right + + sequence = callable(left) + equal = lazy_eq(left, right) if sequence else left == right + if equal: + continue + + if sequence: + return lazy_lt(left, right) + if left is None: + return True + if right is None: + return False + + return left < right + + return False # if equal, return False + - If a class already has ``__eq__``, ``__ne__``, ``__lt__``, ``__le__``, - ``__gt__``, or ``__ge__`` defined, this decorator will overwrite them. +@decorator_with_or_without_args +def lazy_lexicographic_ordering(cls, set_hash=True): + """Decorates a class with extra methods that implement rich comparison. + + This is a lazy version of the tuple comparison used frequently to + implement comparison in Python. Given some objects with fields, you + might use tuple keys to implement comparison, e.g.:: + + class Widget: + def _cmp_key(self): + return ( + self.a, + self.b, + (self.c, self.d), + self.e + ) + + def __eq__(self, other): + return self._cmp_key() == other._cmp_key() + + def __lt__(self): + return self._cmp_key() < other._cmp_key() + + # etc. + + Python would compare ``Widgets`` lexicographically based on their + tuples. The issue there for simple comparators is that we have to + bulid the tuples *and* we have to generate all the values in them up + front. When implementing comparisons for large data structures, this + can be costly. + + Lazy lexicographic comparison maps the tuple comparison shown above + to generator functions. Instead of comparing based on pre-constructed + tuple keys, users of this decorator can compare using elements from a + generator. So, you'd write:: + + @lazy_lexicographic_ordering + class Widget: + def _cmp_iter(self): + yield a + yield b + def cd_fun(): + yield c + yield d + yield cd_fun + yield e + + # operators are added by decorator + + There are no tuples preconstructed, and the generator does not have + to complete. Instead of tuples, we simply make functions that lazily + yield what would've been in the tuple. The + ``@lazy_lexicographic_ordering`` decorator handles the details of + implementing comparison operators, and the ``Widget`` implementor + only has to worry about writing ``_cmp_iter``, and making sure the + elements in it are also comparable. + + Some things to note: + + * If a class already has ``__eq__``, ``__ne__``, ``__lt__``, + ``__le__``, ``__gt__``, ``__ge__``, or ``__hash__`` defined, this + decorator will overwrite them. + + * If ``set_hash`` is ``False``, this will not overwrite + ``__hash__``. + + * This class uses Python 2 None-comparison semantics. If you yield + None and it is compared to a non-None type, None will always be + less than the other object. + + Raises: + TypeError: If the class does not have a ``_cmp_iter`` method - Raises: - TypeError: If the class does not have a ``_cmp_key`` method """ - def setter(name, value): - value.__name__ = name - setattr(cls, name, value) - - if not has_method(cls, '_cmp_key'): - raise TypeError("'%s' doesn't define _cmp_key()." % cls.__name__) - - setter('__eq__', - lambda s, o: - (s is o) or (o is not None and s._cmp_key() == o._cmp_key())) - setter('__lt__', - lambda s, o: o is not None and s._cmp_key() < o._cmp_key()) - setter('__le__', - lambda s, o: o is not None and s._cmp_key() <= o._cmp_key()) - - setter('__ne__', - lambda s, o: - (s is not o) and (o is None or s._cmp_key() != o._cmp_key())) - setter('__gt__', - lambda s, o: o is None or s._cmp_key() > o._cmp_key()) - setter('__ge__', - lambda s, o: o is None or s._cmp_key() >= o._cmp_key()) - - setter('__hash__', lambda self: hash(self._cmp_key())) + if not has_method(cls, "_cmp_iter"): + raise TypeError("'%s' doesn't define _cmp_iter()." % cls.__name__) + + # comparison operators are implemented in terms of lazy_eq and lazy_lt + def eq(self, other): + if self is other: + return True + return (other is not None) and lazy_eq(self._cmp_iter, other._cmp_iter) + + def lt(self, other): + if self is other: + return False + return (other is not None) and lazy_lt(self._cmp_iter, other._cmp_iter) + + def ne(self, other): + if self is other: + return False + return (other is None) or not lazy_eq(self._cmp_iter, other._cmp_iter) + + def gt(self, other): + if self is other: + return False + return (other is None) or lazy_lt(other._cmp_iter, self._cmp_iter) + + def le(self, other): + if self is other: + return True + return (other is not None) and not lazy_lt(other._cmp_iter, + self._cmp_iter) + + def ge(self, other): + if self is other: + return True + return (other is None) or not lazy_lt(self._cmp_iter, other._cmp_iter) + + def h(self): + return hash(tuplify(self._cmp_iter)) + + def add_func_to_class(name, func): + """Add a function to a class with a particular name.""" + func.__name__ = name + setattr(cls, name, func) + + add_func_to_class("__eq__", eq) + add_func_to_class("__ne__", ne) + add_func_to_class("__lt__", lt) + add_func_to_class("__le__", le) + add_func_to_class("__gt__", gt) + add_func_to_class("__ge__", ge) + if set_hash: + add_func_to_class("__hash__", h) return cls -@key_ordering +@lazy_lexicographic_ordering class HashableMap(MutableMapping): """This is a hashable, comparable dictionary. Hash is performed on a tuple of the values in the dictionary.""" @@ -291,8 +470,9 @@ class HashableMap(MutableMapping): def __delitem__(self, key): del self.dict[key] - def _cmp_key(self): - return tuple(sorted(self.values())) + def _cmp_iter(self): + for _, v in sorted(self.items()): + yield v def copy(self): """Type-agnostic clone method. Preserves subclass type.""" @@ -624,6 +804,9 @@ class LazyReference(object): def load_module_from_file(module_name, module_path): """Loads a python module from the path of the corresponding file. + If the module is already in ``sys.modules`` it will be returned as + is and not reloaded. + Args: module_name (str): namespace where the python module will be loaded, e.g. ``foo.bar`` @@ -636,12 +819,28 @@ def load_module_from_file(module_name, module_path): ImportError: when the module can't be loaded FileNotFoundError: when module_path doesn't exist """ + if module_name in sys.modules: + return sys.modules[module_name] + + # This recipe is adapted from https://stackoverflow.com/a/67692/771663 if sys.version_info[0] == 3 and sys.version_info[1] >= 5: import importlib.util spec = importlib.util.spec_from_file_location( # novm module_name, module_path) module = importlib.util.module_from_spec(spec) # novm - spec.loader.exec_module(module) + # The module object needs to exist in sys.modules before the + # loader executes the module code. + # + # See https://docs.python.org/3/reference/import.html#loading + sys.modules[spec.name] = module + try: + spec.loader.exec_module(module) + except BaseException: + try: + del sys.modules[spec.name] + except KeyError: + pass + raise elif sys.version_info[0] == 3 and sys.version_info[1] < 5: import importlib.machinery loader = importlib.machinery.SourceFileLoader( # novm diff --git a/lib/spack/llnl/util/link_tree.py b/lib/spack/llnl/util/link_tree.py index d29b1289e3..c286671401 100644 --- a/lib/spack/llnl/util/link_tree.py +++ b/lib/spack/llnl/util/link_tree.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/llnl/util/lock.py b/lib/spack/llnl/util/lock.py index 5fd7163e2e..bd3308a57f 100644 --- a/lib/spack/llnl/util/lock.py +++ b/lib/spack/llnl/util/lock.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -264,7 +264,7 @@ class Lock(object): self.old_host = self.host self.pid = os.getpid() - self.host = socket.getfqdn() + self.host = socket.gethostname() # write pid, host to disk to sync over FS self._file.seek(0) diff --git a/lib/spack/llnl/util/multiproc.py b/lib/spack/llnl/util/multiproc.py index f87b32c307..c73ebaed20 100644 --- a/lib/spack/llnl/util/multiproc.py +++ b/lib/spack/llnl/util/multiproc.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/llnl/util/tty/__init__.py b/lib/spack/llnl/util/tty/__init__.py index 79ca5a9929..cf70834e8d 100644 --- a/lib/spack/llnl/util/tty/__init__.py +++ b/lib/spack/llnl/util/tty/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/llnl/util/tty/colify.py b/lib/spack/llnl/util/tty/colify.py index e0b4a5fa05..56baeb5bfb 100644 --- a/lib/spack/llnl/util/tty/colify.py +++ b/lib/spack/llnl/util/tty/colify.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/llnl/util/tty/color.py b/lib/spack/llnl/util/tty/color.py index b6f5ec782f..ffd1b8c497 100644 --- a/lib/spack/llnl/util/tty/color.py +++ b/lib/spack/llnl/util/tty/color.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/llnl/util/tty/log.py b/lib/spack/llnl/util/tty/log.py index 658de5c596..337eb0ac68 100644 --- a/lib/spack/llnl/util/tty/log.py +++ b/lib/spack/llnl/util/tty/log.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -20,12 +20,17 @@ from contextlib import contextmanager from six import string_types from six import StringIO +from typing import Optional # novm +from types import ModuleType # novm + import llnl.util.tty as tty +termios = None # type: Optional[ModuleType] try: - import termios + import termios as term_mod + termios = term_mod except ImportError: - termios = None + pass # Use this to strip escape sequences @@ -316,7 +321,10 @@ class FileWrapper(object): def unwrap(self): if self.open: if self.file_like: - self.file = open(self.file_like, 'w') + if sys.version_info < (3,): + self.file = open(self.file_like, 'w') + else: + self.file = open(self.file_like, 'w', encoding='utf-8') else: self.file = StringIO() return self.file @@ -717,7 +725,11 @@ def _writer_daemon(stdin_multiprocess_fd, read_multiprocess_fd, write_fd, echo, # Use line buffering (3rd param = 1) since Python 3 has a bug # that prevents unbuffered text I/O. - in_pipe = os.fdopen(read_multiprocess_fd.fd, 'r', 1) + if sys.version_info < (3,): + in_pipe = os.fdopen(read_multiprocess_fd.fd, 'r', 1) + else: + # Python 3.x before 3.7 does not open with UTF-8 encoding by default + in_pipe = os.fdopen(read_multiprocess_fd.fd, 'r', 1, encoding='utf-8') if stdin_multiprocess_fd: stdin = os.fdopen(stdin_multiprocess_fd.fd) diff --git a/lib/spack/llnl/util/tty/pty.py b/lib/spack/llnl/util/tty/pty.py index 84c272a6e2..2a5fb7afa4 100644 --- a/lib/spack/llnl/util/tty/pty.py +++ b/lib/spack/llnl/util/tty/pty.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/__init__.py b/lib/spack/spack/__init__.py index cefee1203f..a17fc1dfc1 100644 --- a/lib/spack/spack/__init__.py +++ b/lib/spack/spack/__init__.py @@ -1,11 +1,11 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) #: major, minor, patch version for Spack, in a tuple -spack_version_info = (0, 16, 2) +spack_version_info = (0, 16, 1) #: String containing Spack version joined with .'s spack_version = '.'.join(str(v) for v in spack_version_info) diff --git a/lib/spack/spack/abi.py b/lib/spack/spack/abi.py index 986583d270..da796e2100 100644 --- a/lib/spack/spack/abi.py +++ b/lib/spack/spack/abi.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/analyzers/__init__.py b/lib/spack/spack/analyzers/__init__.py new file mode 100644 index 0000000000..9e36ed7b3f --- /dev/null +++ b/lib/spack/spack/analyzers/__init__.py @@ -0,0 +1,43 @@ +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +"""This package contains code for creating analyzers to extract Application +Binary Interface (ABI) information, along with simple analyses that just load +existing metadata. +""" + +from __future__ import absolute_import + +import spack.util.classes +import spack.paths + +import llnl.util.tty as tty + + +mod_path = spack.paths.analyzers_path +analyzers = spack.util.classes.list_classes("spack.analyzers", mod_path) + +# The base analyzer does not have a name, and cannot do dict comprehension +analyzer_types = {} +for a in analyzers: + if not hasattr(a, "name"): + continue + analyzer_types[a.name] = a + + +def list_all(): + """A helper function to list all analyzers and their descriptions + """ + for name, analyzer in analyzer_types.items(): + print("%-25s: %-35s" % (name, analyzer.description)) + + +def get_analyzer(name): + """Courtesy function to retrieve an analyzer, and exit on error if it + does not exist. + """ + if name in analyzer_types: + return analyzer_types[name] + tty.die("Analyzer %s does not exist" % name) diff --git a/lib/spack/spack/analyzers/analyzer_base.py b/lib/spack/spack/analyzers/analyzer_base.py new file mode 100644 index 0000000000..41f456b71a --- /dev/null +++ b/lib/spack/spack/analyzers/analyzer_base.py @@ -0,0 +1,115 @@ +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +"""An analyzer base provides basic functions to run the analysis, save results, +and (optionally) interact with a Spack Monitor +""" + +import spack.monitor +import spack.hooks +import llnl.util.tty as tty +import spack.util.path +import spack.config + +import os + + +def get_analyzer_dir(spec, analyzer_dir=None): + """ + Given a spec, return the directory to save analyzer results. + + We create the directory if it does not exist. We also check that the + spec has an associated package. An analyzer cannot be run if the spec isn't + associated with a package. If the user provides a custom analyzer_dir, + we use it over checking the config and the default at ~/.spack/analyzers + """ + # An analyzer cannot be run if the spec isn't associated with a package + if not hasattr(spec, "package") or not spec.package: + tty.die("A spec can only be analyzed with an associated package.") + + # The top level directory is in the user home, or a custom location + if not analyzer_dir: + analyzer_dir = spack.util.path.canonicalize_path( + spack.config.get('config:analyzers_dir', '~/.spack/analyzers')) + + # We follow the same convention as the spec install (this could be better) + package_prefix = os.sep.join(spec.package.prefix.split('/')[-3:]) + meta_dir = os.path.join(analyzer_dir, package_prefix) + return meta_dir + + +class AnalyzerBase(object): + + def __init__(self, spec, dirname=None): + """ + Verify that the analyzer has correct metadata. + + An Analyzer is intended to run on one spec install, so the spec + with its associated package is required on init. The child analyzer + class should define an init function that super's the init here, and + also check that the analyzer has all dependencies that it + needs. If an analyzer subclass does not have dependencies, it does not + need to define an init. An Analyzer should not be allowed to proceed + if one or more dependencies are missing. The dirname, if defined, + is an optional directory name to save to (instead of the default meta + spack directory). + """ + self.spec = spec + self.dirname = dirname + self.meta_dir = os.path.dirname(spec.package.install_log_path) + + for required in ["name", "outfile", "description"]: + if not hasattr(self, required): + tty.die("Please add a %s attribute on the analyzer." % required) + + def run(self): + """ + Given a spec with an installed package, run the analyzer on it. + """ + raise NotImplementedError + + @property + def output_dir(self): + """ + The full path to the output directory. + + This includes the nested analyzer directory structure. This function + does not create anything. + """ + if not hasattr(self, "_output_dir"): + output_dir = get_analyzer_dir(self.spec, self.dirname) + self._output_dir = os.path.join(output_dir, self.name) + + return self._output_dir + + def save_result(self, result, overwrite=False): + """ + Save a result to the associated spack monitor, if defined. + + This function is on the level of the analyzer because it might be + the case that the result is large (appropriate for a single request) + or that the data is organized differently (e.g., more than one + request per result). If an analyzer subclass needs to over-write + this function with a custom save, that is appropriate to do (see abi). + """ + # We maintain the structure in json with the analyzer as key so + # that in the future, we could upload to a monitor server + if result[self.name]: + + outfile = os.path.join(self.output_dir, self.outfile) + + # Only try to create the results directory if we have a result + if not os.path.exists(self._output_dir): + os.makedirs(self._output_dir) + + # Don't overwrite an existing result if overwrite is False + if os.path.exists(outfile) and not overwrite: + tty.info("%s exists and overwrite is False, skipping." % outfile) + else: + tty.info("Writing result to %s" % outfile) + spack.monitor.write_json(result[self.name], outfile) + + # This hook runs after a save result + spack.hooks.on_analyzer_save(self.spec.package, result) diff --git a/lib/spack/spack/analyzers/config_args.py b/lib/spack/spack/analyzers/config_args.py new file mode 100644 index 0000000000..f29993dcb3 --- /dev/null +++ b/lib/spack/spack/analyzers/config_args.py @@ -0,0 +1,32 @@ +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +"""A configargs analyzer is a class of analyzer that typically just uploads +already existing metadata about config args from a package spec install +directory.""" + + +import spack.monitor +from .analyzer_base import AnalyzerBase + +import os + + +class ConfigArgs(AnalyzerBase): + + name = "config_args" + outfile = "spack-analyzer-config-args.json" + description = "config args loaded from spack-configure-args.txt" + + def run(self): + """ + Load the configure-args.txt and save in json. + + The run function will find the spack-config-args.txt file in the + package install directory, and read it into a json structure that has + the name of the analyzer as the key. + """ + config_file = os.path.join(self.meta_dir, "spack-configure-args.txt") + return {self.name: spack.monitor.read_file(config_file)} diff --git a/lib/spack/spack/analyzers/environment_variables.py b/lib/spack/spack/analyzers/environment_variables.py new file mode 100644 index 0000000000..33c4034e31 --- /dev/null +++ b/lib/spack/spack/analyzers/environment_variables.py @@ -0,0 +1,51 @@ +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +"""An environment analyzer will read and parse the environment variables +file in the installed package directory, generating a json file that has +an index of key, value pairs for environment variables.""" + + +from .analyzer_base import AnalyzerBase +from spack.util.environment import EnvironmentModifications + + +import os + + +class EnvironmentVariables(AnalyzerBase): + + name = "environment_variables" + outfile = "spack-analyzer-environment-variables.json" + description = "environment variables parsed from spack-build-env.txt" + + def run(self): + """ + Load, parse, and save spack-build-env.txt to analyzers. + + Read in the spack-build-env.txt file from the package install + directory and parse the environment variables into key value pairs. + The result should have the key for the analyzer, the name. + """ + env_file = os.path.join(self.meta_dir, "spack-build-env.txt") + return {self.name: self._read_environment_file(env_file)} + + def _read_environment_file(self, filename): + """ + Read and parse the environment file. + + Given an environment file, we want to read it, split by semicolons + and new lines, and then parse down to the subset of SPACK_* variables. + We assume that all spack prefix variables are not secrets, and unlike + the install_manifest.json, we don't (at least to start) parse the values + to remove path prefixes specific to user systems. + """ + if not os.path.exists(filename): + return + + mods = EnvironmentModifications.from_sourcing_file(filename) + env = {} + mods.apply_modifications(env) + return env diff --git a/lib/spack/spack/analyzers/install_files.py b/lib/spack/spack/analyzers/install_files.py new file mode 100644 index 0000000000..bdebac50fc --- /dev/null +++ b/lib/spack/spack/analyzers/install_files.py @@ -0,0 +1,30 @@ +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +"""The install files json file (install_manifest.json) already exists in +the package install folder, so this analyzer simply moves it to the user +analyzer folder for further processing.""" + + +import spack.monitor +from .analyzer_base import AnalyzerBase + +import os + + +class InstallFiles(AnalyzerBase): + + name = "install_files" + outfile = "spack-analyzer-install-files.json" + description = "install file listing read from install_manifest.json" + + def run(self): + """ + Load in the install_manifest.json and save to analyzers. + + We write it out to the analyzers folder, with key as the analyzer name. + """ + manifest_file = os.path.join(self.meta_dir, "install_manifest.json") + return {self.name: spack.monitor.read_json(manifest_file)} diff --git a/lib/spack/spack/analyzers/libabigail.py b/lib/spack/spack/analyzers/libabigail.py new file mode 100644 index 0000000000..b53369b75f --- /dev/null +++ b/lib/spack/spack/analyzers/libabigail.py @@ -0,0 +1,116 @@ +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + + +import spack +import spack.error +import spack.bootstrap +import spack.hooks +import spack.monitor +import spack.binary_distribution +import spack.package +import spack.repo + +import llnl.util.tty as tty + +from .analyzer_base import AnalyzerBase + +import os + + +class Libabigail(AnalyzerBase): + + name = "libabigail" + outfile = "spack-analyzer-libabigail.json" + description = "Application Binary Interface (ABI) features for objects" + + def __init__(self, spec, dirname=None): + """ + init for an analyzer ensures we have all needed dependencies. + + For the libabigail analyzer, this means Libabigail. + Since the output for libabigail is one file per object, we communicate + with the monitor multiple times. + """ + super(Libabigail, self).__init__(spec, dirname) + + # This doesn't seem to work to import on the module level + tty.debug("Preparing to use Libabigail, will install if missing.") + + with spack.bootstrap.ensure_bootstrap_configuration(): + + # libabigail won't install lib/bin/share without docs + spec = spack.spec.Spec("libabigail+docs") + spec.concretize() + + self.abidw = spack.bootstrap.get_executable( + "abidw", spec=spec, install=True) + + def run(self): + """ + Run libabigail, and save results to filename. + + This run function differs in that we write as we generate and then + return a dict with the analyzer name as the key, and the value of a + dict of results, where the key is the object name, and the value is + the output file written to. + """ + manifest = spack.binary_distribution.get_buildfile_manifest(self.spec) + + # This result will store a path to each file + result = {} + + # Generate an output file for each binary or object + for obj in manifest.get("binary_to_relocate_fullpath", []): + + # We want to preserve the path in the install directory in case + # a library has an equivalenly named lib or executable, for example + outdir = os.path.dirname(obj.replace(self.spec.package.prefix, + '').strip(os.path.sep)) + outfile = "spack-analyzer-libabigail-%s.xml" % os.path.basename(obj) + outfile = os.path.join(self.output_dir, outdir, outfile) + outdir = os.path.dirname(outfile) + + # Create the output directory + if not os.path.exists(outdir): + os.makedirs(outdir) + + # Sometimes libabigail segfaults and dumps + try: + self.abidw(obj, "--out-file", outfile) + result[obj] = outfile + tty.info("Writing result to %s" % outfile) + except spack.error.SpackError: + tty.warn("Issue running abidw for %s" % obj) + + return {self.name: result} + + def save_result(self, result, overwrite=False): + """ + Read saved ABI results and upload to monitor server. + + ABI results are saved to individual files, so each one needs to be + read and uploaded. Result here should be the lookup generated in run(), + the key is the analyzer name, and each value is the result file. + We currently upload the entire xml as text because libabigail can't + easily read gzipped xml, but this will be updated when it can. + """ + if not spack.monitor.cli: + return + + name = self.spec.package.name + + for obj, filename in result.get(self.name, {}).items(): + + # Don't include the prefix + rel_path = obj.replace(self.spec.prefix + os.path.sep, "") + + # We've already saved the results to file during run + content = spack.monitor.read_file(filename) + + # A result needs an analyzer, value or binary_value, and name + data = {"value": content, "install_file": rel_path, "name": "abidw-xml"} + tty.info("Sending result for %s %s to monitor." % (name, rel_path)) + spack.hooks.on_analyzer_save(self.spec.package, {"libabigail": [data]}) diff --git a/lib/spack/spack/architecture.py b/lib/spack/spack/architecture.py index ece9040a7d..04cd85be49 100644 --- a/lib/spack/spack/architecture.py +++ b/lib/spack/spack/architecture.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -58,14 +58,13 @@ will be responsible for compiler detection. """ import contextlib import functools -import inspect import warnings import archspec.cpu import six import llnl.util.tty as tty -from llnl.util.lang import memoized, list_modules, key_ordering +import llnl.util.lang as lang import spack.compiler import spack.compilers @@ -74,7 +73,7 @@ import spack.paths import spack.error as serr import spack.util.executable import spack.version -from spack.util.naming import mod_to_class +import spack.util.classes from spack.util.spack_yaml import syaml_dict @@ -232,24 +231,25 @@ class Target(object): ) -@key_ordering +@lang.lazy_lexicographic_ordering class Platform(object): """ Abstract class that each type of Platform will subclass. Will return a instance of it once it is returned. """ - priority = None # Subclass sets number. Controls detection order + # Subclass sets number. Controls detection order + priority = None # type: int #: binary formats used on this platform; used by relocation logic binary_formats = ['elf'] - front_end = None - back_end = None - default = None # The default back end target. + front_end = None # type: str + back_end = None # type: str + default = None # type: str # The default back end target. - front_os = None - back_os = None - default_os = None + front_os = None # type: str + back_os = None # type: str + default_os = None # type: str reserved_targets = ['default_target', 'frontend', 'fe', 'backend', 'be'] reserved_oss = ['default_os', 'frontend', 'fe', 'backend', 'be'] @@ -328,23 +328,27 @@ class Platform(object): def __str__(self): return self.name - def _cmp_key(self): - t_keys = ''.join(str(t._cmp_key()) for t in - sorted(self.targets.values())) - o_keys = ''.join(str(o._cmp_key()) for o in - sorted(self.operating_sys.values())) - return (self.name, - self.default, - self.front_end, - self.back_end, - self.default_os, - self.front_os, - self.back_os, - t_keys, - o_keys) - - -@key_ordering + def _cmp_iter(self): + yield self.name + yield self.default + yield self.front_end + yield self.back_end + yield self.default_os + yield self.front_os + yield self.back_os + + def targets(): + for t in sorted(self.targets.values()): + yield t._cmp_iter + yield targets + + def oses(): + for o in sorted(self.operating_sys.values()): + yield o._cmp_iter + yield oses + + +@lang.lazy_lexicographic_ordering class OperatingSystem(object): """ Operating System will be like a class similar to platform extended by subclasses for the specifics. Operating System will contain the @@ -362,8 +366,9 @@ class OperatingSystem(object): def __repr__(self): return self.__str__() - def _cmp_key(self): - return (self.name, self.version) + def _cmp_iter(self): + yield self.name + yield self.version def to_dict(self): return syaml_dict([ @@ -372,7 +377,7 @@ class OperatingSystem(object): ]) -@key_ordering +@lang.lazy_lexicographic_ordering class Arch(object): """Architecture is now a class to help with setting attributes. @@ -422,20 +427,21 @@ class Arch(object): self.target is not None) __bool__ = __nonzero__ - def _cmp_key(self): + def _cmp_iter(self): if isinstance(self.platform, Platform): - platform = self.platform.name + yield self.platform.name else: - platform = self.platform + yield self.platform + if isinstance(self.os, OperatingSystem): - os = self.os.name + yield self.os.name else: - os = self.os + yield self.os + if isinstance(self.target, Target): - target = self.target.microarchitecture + yield self.target.microarchitecture else: - target = self.target - return (platform, os, target) + yield self.target def to_dict(self): str_or_none = lambda v: str(v) if v else None @@ -457,7 +463,7 @@ class Arch(object): return arch_for_spec(spec) -@memoized +@lang.memoized def get_platform(platform_name): """Returns a platform object that corresponds to the given name.""" platform_list = all_platforms() @@ -493,28 +499,13 @@ def arch_for_spec(arch_spec): return Arch(arch_plat, arch_spec.os, arch_spec.target) -@memoized +@lang.memoized def _all_platforms(): - classes = [] mod_path = spack.paths.platform_path - parent_module = "spack.platforms" - - for name in list_modules(mod_path): - mod_name = '%s.%s' % (parent_module, name) - class_name = mod_to_class(name) - mod = __import__(mod_name, fromlist=[class_name]) - if not hasattr(mod, class_name): - tty.die('No class %s defined in %s' % (class_name, mod_name)) - cls = getattr(mod, class_name) - if not inspect.isclass(cls): - tty.die('%s.%s is not a class' % (mod_name, class_name)) - - classes.append(cls) - - return classes + return spack.util.classes.list_classes("spack.platforms", mod_path) -@memoized +@lang.memoized def _platform(): """Detects the platform for this machine. @@ -545,7 +536,7 @@ platform = _platform all_platforms = _all_platforms -@memoized +@lang.memoized def default_arch(): """Default ``Arch`` object for this machine. @@ -569,7 +560,7 @@ def sys_type(): return str(default_arch()) -@memoized +@lang.memoized def compatible_sys_types(): """Returns a list of all the systypes compatible with the current host.""" compatible_archs = [] diff --git a/lib/spack/spack/binary_distribution.py b/lib/spack/spack/binary_distribution.py index 00fffdfe51..1ac89b615c 100644 --- a/lib/spack/spack/binary_distribution.py +++ b/lib/spack/spack/binary_distribution.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -37,13 +37,11 @@ import spack.util.spack_yaml as syaml import spack.mirror import spack.util.url as url_util import spack.util.web as web_util +from spack.caches import misc_cache_location from spack.spec import Spec from spack.stage import Stage -#: default root, relative to the Spack install path -default_binary_index_root = os.path.join(spack.paths.opt_path, 'spack') - _build_cache_relative_path = 'build_cache' _build_cache_keys_relative_path = '_pgp' @@ -67,9 +65,8 @@ class BinaryCacheIndex(object): mean we should have paid the price to update the cache earlier? """ - def __init__(self, cache_root=None): - self._cache_root = cache_root or default_binary_index_root - self._index_cache_root = os.path.join(self._cache_root, 'indices') + def __init__(self, cache_root): + self._index_cache_root = cache_root # the key associated with the serialized _local_index_cache self._index_contents_key = 'contents.json' @@ -440,13 +437,15 @@ class BinaryCacheIndex(object): return True +def binary_index_location(): + """Set up a BinaryCacheIndex for remote buildcache dbs in the user's homedir.""" + cache_root = os.path.join(misc_cache_location(), 'indices') + return spack.util.path.canonicalize_path(cache_root) + + def _binary_index(): """Get the singleton store instance.""" - cache_root = spack.config.get( - 'config:binary_index_root', default_binary_index_root) - cache_root = spack.util.path.canonicalize_path(cache_root) - - return BinaryCacheIndex(cache_root) + return BinaryCacheIndex(binary_index_location()) #: Singleton binary_index instance @@ -551,40 +550,38 @@ def read_buildinfo_file(prefix): return buildinfo -def write_buildinfo_file(spec, workdir, rel=False): +def get_buildfile_manifest(spec): """ - Create a cache file containing information - required for the relocation + Return a data structure with information about a build, including + text_to_relocate, binary_to_relocate, binary_to_relocate_fullpath + link_to_relocate, and other, which means it doesn't fit any of previous + checks (and should not be relocated). We blacklist docs (man) and + metadata (.spack). This can be used to find a particular kind of file + in spack, or to generate the build metadata. """ - prefix = spec.prefix - text_to_relocate = [] - binary_to_relocate = [] - link_to_relocate = [] + data = {"text_to_relocate": [], "binary_to_relocate": [], + "link_to_relocate": [], "other": [], + "binary_to_relocate_fullpath": []} + blacklist = (".spack", "man") - prefix_to_hash = dict() - prefix_to_hash[str(spec.package.prefix)] = spec.dag_hash() - deps = spack.build_environment.get_rpath_deps(spec.package) - for d in deps: - prefix_to_hash[str(d.prefix)] = d.dag_hash() + # Do this at during tarball creation to save time when tarball unpacked. # Used by make_package_relative to determine binaries to change. - for root, dirs, files in os.walk(prefix, topdown=True): + for root, dirs, files in os.walk(spec.prefix, topdown=True): dirs[:] = [d for d in dirs if d not in blacklist] for filename in files: path_name = os.path.join(root, filename) m_type, m_subtype = relocate.mime_type(path_name) + rel_path_name = os.path.relpath(path_name, spec.prefix) + added = False + if os.path.islink(path_name): link = os.readlink(path_name) if os.path.isabs(link): # Relocate absolute links into the spack tree if link.startswith(spack.store.layout.root): - rel_path_name = os.path.relpath(path_name, prefix) - link_to_relocate.append(rel_path_name) - else: - msg = 'Absolute link %s to %s ' % (path_name, link) - msg += 'outside of prefix %s ' % prefix - msg += 'should not be relocated.' - tty.warn(msg) + data['link_to_relocate'].append(rel_path_name) + added = True if relocate.needs_binary_relocation(m_type, m_subtype): if ((m_subtype in ('x-executable', 'x-sharedlib') @@ -592,11 +589,31 @@ def write_buildinfo_file(spec, workdir, rel=False): (m_subtype in ('x-mach-binary') and sys.platform == 'darwin') or (not filename.endswith('.o'))): - rel_path_name = os.path.relpath(path_name, prefix) - binary_to_relocate.append(rel_path_name) + data['binary_to_relocate'].append(rel_path_name) + data['binary_to_relocate_fullpath'].append(path_name) + added = True + if relocate.needs_text_relocation(m_type, m_subtype): - rel_path_name = os.path.relpath(path_name, prefix) - text_to_relocate.append(rel_path_name) + data['text_to_relocate'].append(rel_path_name) + added = True + + if not added: + data['other'].append(path_name) + return data + + +def write_buildinfo_file(spec, workdir, rel=False): + """ + Create a cache file containing information + required for the relocation + """ + manifest = get_buildfile_manifest(spec) + + prefix_to_hash = dict() + prefix_to_hash[str(spec.package.prefix)] = spec.dag_hash() + deps = spack.build_environment.get_rpath_deps(spec.package) + for d in deps: + prefix_to_hash[str(d.prefix)] = d.dag_hash() # Create buildinfo data and write it to disk import spack.hooks.sbang as sbang @@ -606,10 +623,10 @@ def write_buildinfo_file(spec, workdir, rel=False): buildinfo['buildpath'] = spack.store.layout.root buildinfo['spackprefix'] = spack.paths.prefix buildinfo['relative_prefix'] = os.path.relpath( - prefix, spack.store.layout.root) - buildinfo['relocate_textfiles'] = text_to_relocate - buildinfo['relocate_binaries'] = binary_to_relocate - buildinfo['relocate_links'] = link_to_relocate + spec.prefix, spack.store.layout.root) + buildinfo['relocate_textfiles'] = manifest['text_to_relocate'] + buildinfo['relocate_binaries'] = manifest['binary_to_relocate'] + buildinfo['relocate_links'] = manifest['link_to_relocate'] buildinfo['prefix_to_hash'] = prefix_to_hash filename = buildinfo_file_name(workdir) with open(filename, 'w') as outfile: @@ -763,6 +780,10 @@ def generate_package_index(cache_prefix): url_util.join(cache_prefix, 'index.json.hash'), keep_original=False, extra_args={'ContentType': 'text/plain'}) + except Exception as err: + msg = 'Encountered problem pushing package index to {0}: {1}'.format( + cache_prefix, err) + tty.warn(msg) finally: shutil.rmtree(tmpdir) @@ -823,6 +844,10 @@ def generate_key_index(key_prefix, tmpdir=None): url_util.join(key_prefix, 'index.json'), keep_original=False, extra_args={'ContentType': 'application/json'}) + except Exception as err: + msg = 'Encountered problem pushing key index to {0}: {1}'.format( + key_prefix, err) + tty.warn(msg) finally: if remove_tmpdir: shutil.rmtree(tmpdir) @@ -1081,6 +1106,8 @@ def relocate_package(spec, allow_root): """ Relocate the given package """ + import spack.hooks.sbang as sbang + workdir = str(spec.prefix) buildinfo = read_buildinfo_file(workdir) new_layout_root = str(spack.store.layout.root) @@ -1119,9 +1146,7 @@ def relocate_package(spec, allow_root): prefix_to_prefix_bin = OrderedDict({}) if old_sbang_install_path: - import spack.hooks.sbang as sbang - prefix_to_prefix_text[old_sbang_install_path] = \ - sbang.sbang_install_path() + prefix_to_prefix_text[old_sbang_install_path] = sbang.sbang_install_path() prefix_to_prefix_text[old_prefix] = new_prefix prefix_to_prefix_bin[old_prefix] = new_prefix @@ -1134,7 +1159,6 @@ def relocate_package(spec, allow_root): # sbang was a bash script, and it lived in the spack prefix. It is # now a POSIX script that lives in the install prefix. Old packages # will have the old sbang location in their shebangs. - import spack.hooks.sbang as sbang orig_sbang = '#!/bin/bash {0}/bin/sbang'.format(old_spack_prefix) new_sbang = sbang.sbang_shebang_line() prefix_to_prefix_text[orig_sbang] = new_sbang @@ -1153,8 +1177,8 @@ def relocate_package(spec, allow_root): if not is_backup_file(text_name): text_names.append(text_name) -# If we are not installing back to the same install tree do the relocation - if old_layout_root != new_layout_root: + # If we are not installing back to the same install tree do the relocation + if old_prefix != new_prefix: files_to_relocate = [os.path.join(workdir, filename) for filename in buildinfo.get('relocate_binaries') ] @@ -1323,7 +1347,7 @@ def extract_tarball(spec, filename, allow_root=False, unsigned=False, os.remove(filename) -def try_direct_fetch(spec, force=False, full_hash_match=False, mirrors=None): +def try_direct_fetch(spec, full_hash_match=False, mirrors=None): """ Try to find the spec directly on the configured mirrors """ @@ -1361,11 +1385,26 @@ def try_direct_fetch(spec, force=False, full_hash_match=False, mirrors=None): return found_specs -def get_mirrors_for_spec(spec=None, force=False, full_hash_match=False, - mirrors_to_check=None): +def get_mirrors_for_spec(spec=None, full_hash_match=False, + mirrors_to_check=None, index_only=False): """ Check if concrete spec exists on mirrors and return a list indicating the mirrors on which it can be found + + Args: + spec (Spec): The spec to look for in binary mirrors + full_hash_match (bool): If True, only includes mirrors where the spec + full hash matches the locally computed full hash of the ``spec`` + argument. If False, any mirror which has a matching DAG hash + is included in the results. + mirrors_to_check (dict): Optionally override the configured mirrors + with the mirrors in this dictionary. + index_only (bool): Do not attempt direct fetching of ``spec.yaml`` + files from remote mirrors, only consider the indices. + + Return: + A list of objects, each containing a ``mirror_url`` and ``spec`` key + indicating all mirrors where the spec can be found. """ if spec is None: return [] @@ -1391,10 +1430,9 @@ def get_mirrors_for_spec(spec=None, force=False, full_hash_match=False, results = filter_candidates(candidates) # Maybe we just didn't have the latest information from the mirror, so - # try to fetch directly. - if not results: + # try to fetch directly, unless we are only considering the indices. + if not results and not index_only: results = try_direct_fetch(spec, - force=force, full_hash_match=full_hash_match, mirrors=mirrors_to_check) diff --git a/lib/spack/spack/bootstrap.py b/lib/spack/spack/bootstrap.py index f8d0703174..6a536ce6a7 100644 --- a/lib/spack/spack/bootstrap.py +++ b/lib/spack/spack/bootstrap.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -198,9 +198,7 @@ def get_executable(exe, spec=None, install=False): def _bootstrap_config_scopes(): tty.debug('[BOOTSTRAP CONFIG SCOPE] name=_builtin') config_scopes = [ - spack.config.InternalConfigScope( - '_builtin', spack.config.config_defaults - ) + spack.config.InternalConfigScope('_builtin', spack.config.config_defaults) ] for name, path in spack.config.configuration_paths: platform = spack.architecture.platform().name diff --git a/lib/spack/spack/build_environment.py b/lib/spack/spack/build_environment.py index 3cf02dcbb4..b9fcece027 100644 --- a/lib/spack/spack/build_environment.py +++ b/lib/spack/spack/build_environment.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -70,8 +70,7 @@ from spack.error import NoLibrariesError, NoHeadersError from spack.util.executable import Executable from spack.util.module_cmd import load_module, path_from_modules, module from spack.util.log_parse import parse_log_events, make_log_context - - +from spack.util.cpus import cpus_available # # This can be set by the user to globally disable parallel builds. # @@ -149,12 +148,17 @@ def clean_environment(): # can affect how some packages find libraries. We want to make # sure that builds never pull in unintended external dependencies. env.unset('LD_LIBRARY_PATH') - env.unset('LIBRARY_PATH') - env.unset('CPATH') env.unset('LD_RUN_PATH') env.unset('DYLD_LIBRARY_PATH') env.unset('DYLD_FALLBACK_LIBRARY_PATH') + # These vars affect how the compiler finds libraries and include dirs. + env.unset('LIBRARY_PATH') + env.unset('CPATH') + env.unset('C_INCLUDE_PATH') + env.unset('CPLUS_INCLUDE_PATH') + env.unset('OBJC_INCLUDE_PATH') + # On Cray "cluster" systems, unset CRAY_LD_LIBRARY_PATH to avoid # interference with Spack dependencies. # CNL requires these variables to be set (or at least some of them, @@ -440,8 +444,7 @@ def set_build_environment_variables(pkg, env, dirty): # directory. Add that to the path too. env_paths = [] compiler_specific = os.path.join( - spack.paths.build_env_path, - os.path.dirname(pkg.compiler.link_paths['cc'])) + spack.paths.build_env_path, os.path.dirname(pkg.compiler.link_paths['cc'])) for item in [spack.paths.build_env_path, compiler_specific]: env_paths.append(item) ci = os.path.join(item, 'case-insensitive') @@ -476,6 +479,38 @@ def set_build_environment_variables(pkg, env, dirty): return env +def determine_number_of_jobs( + parallel=False, command_line=None, config_default=None, max_cpus=None): + """ + Packages that require sequential builds need 1 job. Otherwise we use the + number of jobs set on the command line. If not set, then we use the config + defaults (which is usually set through the builtin config scope), but we + cap to the number of CPUs available to avoid oversubscription. + + Parameters: + parallel (bool): true when package supports parallel builds + command_line (int/None): command line override + config_default (int/None): config default number of jobs + max_cpus (int/None): maximum number of CPUs available. When None, this + value is automatically determined. + """ + if not parallel: + return 1 + + if command_line is None and 'command_line' in spack.config.scopes(): + command_line = spack.config.get('config:build_jobs', scope='command_line') + + if command_line is not None: + return command_line + + max_cpus = max_cpus or cpus_available() + + # in some rare cases _builtin config may not be set, so default to max 16 + config_default = config_default or spack.config.get('config:build_jobs', 16) + + return min(max_cpus, config_default) + + def _set_variables_for_single_module(pkg, module): """Helper function to set module variables for single module.""" # Put a marker on this module so that it won't execute the body of this @@ -484,8 +519,7 @@ def _set_variables_for_single_module(pkg, module): if getattr(module, marker, False): return - jobs = spack.config.get('config:build_jobs', 16) if pkg.parallel else 1 - jobs = min(jobs, multiprocessing.cpu_count()) + jobs = determine_number_of_jobs(parallel=pkg.parallel) m = module m.make_jobs = jobs diff --git a/lib/spack/spack/build_systems/__init__.py b/lib/spack/spack/build_systems/__init__.py index 9f87532b85..103eae6134 100644 --- a/lib/spack/spack/build_systems/__init__.py +++ b/lib/spack/spack/build_systems/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/build_systems/aspell_dict.py b/lib/spack/spack/build_systems/aspell_dict.py index 7da4d2f10c..55ea78145a 100644 --- a/lib/spack/spack/build_systems/aspell_dict.py +++ b/lib/spack/spack/build_systems/aspell_dict.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/build_systems/autotools.py b/lib/spack/spack/build_systems/autotools.py index ee9fb6884e..bce9afebf7 100644 --- a/lib/spack/spack/build_systems/autotools.py +++ b/lib/spack/spack/build_systems/autotools.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -6,8 +6,10 @@ import inspect import itertools import os import os.path +import stat from subprocess import PIPE from subprocess import check_call +from typing import List # novm import llnl.util.tty as tty import llnl.util.filesystem as fs @@ -61,7 +63,7 @@ class AutotoolsPackage(PackageBase): #: Targets for ``make`` during the :py:meth:`~.AutotoolsPackage.build` #: phase - build_targets = [] + build_targets = [] # type: List[str] #: Targets for ``make`` during the :py:meth:`~.AutotoolsPackage.install` #: phase install_targets = ['install'] @@ -75,7 +77,7 @@ class AutotoolsPackage(PackageBase): #: Set to true to force the autoreconf step even if configure is present force_autoreconf = False #: Options to be passed to autoreconf when using the default implementation - autoreconf_extra_args = [] + autoreconf_extra_args = [] # type: List[str] #: If False deletes all the .la files in the prefix folder #: after the installation. If True instead it installs them. @@ -173,7 +175,10 @@ class AutotoolsPackage(PackageBase): # Copy the good files over the bad ones for abs_path in to_be_patched: name = os.path.basename(abs_path) + mode = os.stat(abs_path).st_mode + os.chmod(abs_path, stat.S_IWUSR) fs.copy(substitutes[name], abs_path) + os.chmod(abs_path, mode) @run_before('configure') def _set_autotools_environment_variables(self): diff --git a/lib/spack/spack/build_systems/cached_cmake.py b/lib/spack/spack/build_systems/cached_cmake.py new file mode 100644 index 0000000000..2c3e150528 --- /dev/null +++ b/lib/spack/spack/build_systems/cached_cmake.py @@ -0,0 +1,249 @@ +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) +import os + +from llnl.util.filesystem import install, mkdirp +import llnl.util.tty as tty + +from spack.build_systems.cmake import CMakePackage +from spack.package import run_after + + +def cmake_cache_path(name, value, comment=""): + """Generate a string for a cmake cache variable""" + return 'set({0} "{1}" CACHE PATH "{2}")\n'.format(name, value, comment) + + +def cmake_cache_string(name, value, comment=""): + """Generate a string for a cmake cache variable""" + return 'set({0} "{1}" CACHE STRING "{2}")\n'.format(name, value, comment) + + +def cmake_cache_option(name, boolean_value, comment=""): + """Generate a string for a cmake configuration option""" + + value = "ON" if boolean_value else "OFF" + return 'set({0} {1} CACHE BOOL "{2}")\n'.format(name, value, comment) + + +class CachedCMakePackage(CMakePackage): + """Specialized class for packages built using CMake initial cache. + + This feature of CMake allows packages to increase reproducibility, + especially between Spack- and manual builds. It also allows packages to + sidestep certain parsing bugs in extremely long ``cmake`` commands, and to + avoid system limits on the length of the command line.""" + + phases = ['initconfig', 'cmake', 'build', 'install'] + + @property + def cache_name(self): + return "{0}-{1}-{2}@{3}.cmake".format( + self.name, + self.spec.architecture, + self.spec.compiler.name, + self.spec.compiler.version, + ) + + @property + def cache_path(self): + return os.path.join(self.stage.source_path, self.cache_name) + + def flag_handler(self, name, flags): + if name in ('cflags', 'cxxflags', 'cppflags', 'fflags'): + return (None, None, None) # handled in the cmake cache + return (flags, None, None) + + def initconfig_compiler_entries(self): + # This will tell cmake to use the Spack compiler wrappers when run + # through Spack, but use the underlying compiler when run outside of + # Spack + spec = self.spec + + # Fortran compiler is optional + if "FC" in os.environ: + spack_fc_entry = cmake_cache_path( + "CMAKE_Fortran_COMPILER", os.environ['FC']) + system_fc_entry = cmake_cache_path( + "CMAKE_Fortran_COMPILER", self.compiler.fc) + else: + spack_fc_entry = "# No Fortran compiler defined in spec" + system_fc_entry = "# No Fortran compiler defined in spec" + + entries = [ + "#------------------{0}".format("-" * 60), + "# Compilers", + "#------------------{0}".format("-" * 60), + "# Compiler Spec: {0}".format(spec.compiler), + "#------------------{0}".format("-" * 60), + 'if(DEFINED ENV{SPACK_CC})\n', + ' ' + cmake_cache_path( + "CMAKE_C_COMPILER", os.environ['CC']), + ' ' + cmake_cache_path( + "CMAKE_CXX_COMPILER", os.environ['CXX']), + ' ' + spack_fc_entry, + 'else()\n', + ' ' + cmake_cache_path( + "CMAKE_C_COMPILER", self.compiler.cc), + ' ' + cmake_cache_path( + "CMAKE_CXX_COMPILER", self.compiler.cxx), + ' ' + system_fc_entry, + 'endif()\n' + ] + + # use global spack compiler flags + cppflags = ' '.join(spec.compiler_flags['cppflags']) + if cppflags: + # avoid always ending up with ' ' with no flags defined + cppflags += ' ' + cflags = cppflags + ' '.join(spec.compiler_flags['cflags']) + if cflags: + entries.append(cmake_cache_string("CMAKE_C_FLAGS", cflags)) + cxxflags = cppflags + ' '.join(spec.compiler_flags['cxxflags']) + if cxxflags: + entries.append(cmake_cache_string("CMAKE_CXX_FLAGS", cxxflags)) + fflags = ' '.join(spec.compiler_flags['fflags']) + if fflags: + entries.append(cmake_cache_string("CMAKE_Fortran_FLAGS", fflags)) + + # Override XL compiler family + familymsg = ("Override to proper compiler family for XL") + if "xlf" in (self.compiler.fc or ''): # noqa: F821 + entries.append(cmake_cache_string( + "CMAKE_Fortran_COMPILER_ID", "XL", + familymsg)) + if "xlc" in self.compiler.cc: # noqa: F821 + entries.append(cmake_cache_string( + "CMAKE_C_COMPILER_ID", "XL", + familymsg)) + if "xlC" in self.compiler.cxx: # noqa: F821 + entries.append(cmake_cache_string( + "CMAKE_CXX_COMPILER_ID", "XL", + familymsg)) + + return entries + + def initconfig_mpi_entries(self): + spec = self.spec + + if not spec.satisfies('^mpi'): + return [] + + entries = [ + "#------------------{0}".format("-" * 60), + "# MPI", + "#------------------{0}\n".format("-" * 60), + ] + + entries.append(cmake_cache_path("MPI_C_COMPILER", + spec['mpi'].mpicc)) + entries.append(cmake_cache_path("MPI_CXX_COMPILER", + spec['mpi'].mpicxx)) + entries.append(cmake_cache_path("MPI_Fortran_COMPILER", + spec['mpi'].mpifc)) + + # Check for slurm + using_slurm = False + slurm_checks = ['+slurm', + 'schedulers=slurm', + 'process_managers=slurm'] + if any(spec['mpi'].satisfies(variant) for variant in slurm_checks): + using_slurm = True + + # Determine MPIEXEC + if using_slurm: + if spec['mpi'].external: + # Heuristic until we have dependents on externals + mpiexec = '/usr/bin/srun' + else: + mpiexec = os.path.join(spec['slurm'].prefix.bin, 'srun') + else: + mpiexec = os.path.join(spec['mpi'].prefix.bin, 'mpirun') + if not os.path.exists(mpiexec): + mpiexec = os.path.join(spec['mpi'].prefix.bin, 'mpiexec') + + if not os.path.exists(mpiexec): + msg = "Unable to determine MPIEXEC, %s tests may fail" % self.name + entries.append("# {0}\n".format(msg)) + tty.warn(msg) + else: + # starting with cmake 3.10, FindMPI expects MPIEXEC_EXECUTABLE + # vs the older versions which expect MPIEXEC + if self.spec["cmake"].satisfies('@3.10:'): + entries.append(cmake_cache_path("MPIEXEC_EXECUTABLE", + mpiexec)) + else: + entries.append(cmake_cache_path("MPIEXEC", mpiexec)) + + # Determine MPIEXEC_NUMPROC_FLAG + if using_slurm: + entries.append(cmake_cache_string("MPIEXEC_NUMPROC_FLAG", "-n")) + else: + entries.append(cmake_cache_string("MPIEXEC_NUMPROC_FLAG", "-np")) + + return entries + + def initconfig_hardware_entries(self): + spec = self.spec + + entries = [ + "#------------------{0}".format("-" * 60), + "# Hardware", + "#------------------{0}\n".format("-" * 60), + ] + + if spec.satisfies('^cuda'): + entries.append("#------------------{0}".format("-" * 30)) + entries.append("# Cuda") + entries.append("#------------------{0}\n".format("-" * 30)) + + cudatoolkitdir = spec['cuda'].prefix + entries.append(cmake_cache_path("CUDA_TOOLKIT_ROOT_DIR", + cudatoolkitdir)) + cudacompiler = "${CUDA_TOOLKIT_ROOT_DIR}/bin/nvcc" + entries.append(cmake_cache_path("CMAKE_CUDA_COMPILER", + cudacompiler)) + + if spec.satisfies('^mpi'): + entries.append(cmake_cache_path("CMAKE_CUDA_HOST_COMPILER", + "${MPI_CXX_COMPILER}")) + else: + entries.append(cmake_cache_path("CMAKE_CUDA_HOST_COMPILER", + "${CMAKE_CXX_COMPILER}")) + + return entries + + def std_initconfig_entries(self): + return [ + "#------------------{0}".format("-" * 60), + "# !!!! This is a generated file, edit at own risk !!!!", + "#------------------{0}".format("-" * 60), + "# CMake executable path: {0}".format( + self.spec['cmake'].command.path), + "#------------------{0}\n".format("-" * 60), + ] + + def initconfig(self, spec, prefix): + cache_entries = (self.std_initconfig_entries() + + self.initconfig_compiler_entries() + + self.initconfig_mpi_entries() + + self.initconfig_hardware_entries() + + self.initconfig_package_entries()) + + with open(self.cache_name, 'w') as f: + for entry in cache_entries: + f.write('%s\n' % entry) + f.write('\n') + + @property + def std_cmake_args(self): + args = super(CachedCMakePackage, self).std_cmake_args + args.extend(['-C', self.cache_path]) + return args + + @run_after('install') + def install_cmake_cache(self): + mkdirp(self.spec.prefix.share.cmake) + install(self.cache_path, self.spec.prefix.share.cmake) diff --git a/lib/spack/spack/build_systems/cmake.py b/lib/spack/spack/build_systems/cmake.py index 1336069846..61b3a3cb57 100644 --- a/lib/spack/spack/build_systems/cmake.py +++ b/lib/spack/spack/build_systems/cmake.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -8,6 +8,7 @@ import inspect import os import platform import re +from typing import List # novm import spack.build_environment from llnl.util.filesystem import working_dir @@ -74,7 +75,7 @@ class CMakePackage(PackageBase): #: system base class build_system_class = 'CMakePackage' - build_targets = [] + build_targets = [] # type: List[str] install_targets = ['install'] build_time_test_callbacks = ['check'] diff --git a/lib/spack/spack/build_systems/cuda.py b/lib/spack/spack/build_systems/cuda.py index 61007431a4..25b5381900 100644 --- a/lib/spack/spack/build_systems/cuda.py +++ b/lib/spack/spack/build_systems/cuda.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/build_systems/gnu.py b/lib/spack/spack/build_systems/gnu.py index c5b5e74bb9..6f5771214e 100644 --- a/lib/spack/spack/build_systems/gnu.py +++ b/lib/spack/spack/build_systems/gnu.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/build_systems/intel.py b/lib/spack/spack/build_systems/intel.py index 0e0bb9378b..16a7e50084 100644 --- a/lib/spack/spack/build_systems/intel.py +++ b/lib/spack/spack/build_systems/intel.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -151,7 +151,7 @@ class IntelPackage(PackageBase): '+advisor': 'advisor', '+inspector': 'inspector', '+itac': 'itac', - '+vtune': 'vtune_amplifier', + '+vtune': 'vtune_profiler', }.items(): if variant in self.spec: dirs.append(self.normalize_path( @@ -202,7 +202,8 @@ class IntelPackage(PackageBase): '+itac': ' intel-itac intel-ta intel-tc' ' intel-trace-analyzer intel-trace-collector', # Trace Analyzer and Collector - '+vtune': ' intel-vtune-amplifier', # VTune + '+vtune': ' intel-vtune' + # VTune, ..-profiler since 2020, ..-amplifier before }.items(): if variant in self.spec: c += components_to_add @@ -535,8 +536,9 @@ class IntelPackage(PackageBase): [None, '2016:', 'compilers_and_libraries'], ['advisor', ':2016', 'advisor_xe'], ['inspector', ':2016', 'inspector_xe'], - ['vtune_amplifier', ':2017', 'vtune_amplifier_xe'], + ['vtune_profiler', ':2017', 'vtune_amplifier_xe'], ['vtune', ':2017', 'vtune_amplifier_xe'], # alt. + ['vtune_profiler', ':2019', 'vtune_amplifier'], ['itac', ':', 'itac', [os.sep + standalone_glob]], ]: if cs == rename_rule[0] and v.satisfies(ver(rename_rule[1])): @@ -815,6 +817,7 @@ class IntelPackage(PackageBase): # Was supported only up to 2015. blacs_lib = 'libmkl_blacs' elif ('^mpich@2:' in spec_root or + '^cray-mpich' in spec_root or '^mvapich2' in spec_root or '^intel-mpi' in spec_root or '^intel-parallel-studio' in spec_root): diff --git a/lib/spack/spack/build_systems/makefile.py b/lib/spack/spack/build_systems/makefile.py index 6cd05c7ad9..fbc415e0ed 100644 --- a/lib/spack/spack/build_systems/makefile.py +++ b/lib/spack/spack/build_systems/makefile.py @@ -1,10 +1,11 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) import inspect +from typing import List # novm import llnl.util.tty as tty from llnl.util.filesystem import working_dir @@ -48,7 +49,7 @@ class MakefilePackage(PackageBase): #: Targets for ``make`` during the :py:meth:`~.MakefilePackage.build` #: phase - build_targets = [] + build_targets = [] # type: List[str] #: Targets for ``make`` during the :py:meth:`~.MakefilePackage.install` #: phase install_targets = ['install'] diff --git a/lib/spack/spack/build_systems/maven.py b/lib/spack/spack/build_systems/maven.py index e9df34e791..7a5caabd29 100644 --- a/lib/spack/spack/build_systems/maven.py +++ b/lib/spack/spack/build_systems/maven.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/build_systems/meson.py b/lib/spack/spack/build_systems/meson.py index 825b4c98c3..8f9972ba2e 100644 --- a/lib/spack/spack/build_systems/meson.py +++ b/lib/spack/spack/build_systems/meson.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -6,6 +6,7 @@ import inspect import os +from typing import List # novm from llnl.util.filesystem import working_dir from spack.directives import depends_on, variant @@ -46,14 +47,18 @@ class MesonPackage(PackageBase): #: system base class build_system_class = 'MesonPackage' - build_targets = [] + build_targets = [] # type: List[str] install_targets = ['install'] build_time_test_callbacks = ['check'] - variant('buildtype', default='release', + variant('buildtype', default='debugoptimized', description='Meson build type', values=('plain', 'debug', 'debugoptimized', 'release', 'minsize')) + variant('default_library', default='shared', + description=' Default library type', + values=('shared', 'static', 'both')) + variant('strip', default=False, description='Strip targets on install') depends_on('meson', type='build') depends_on('ninja', type='build') @@ -95,6 +100,13 @@ class MesonPackage(PackageBase): except KeyError: build_type = 'release' + strip = 'true' if '+strip' in pkg.spec else 'false' + + try: + default_library = pkg.spec.variants['default_library'].value + except KeyError: + default_library = 'shared' + args = [ '--prefix={0}'.format(pkg.prefix), # If we do not specify libdir explicitly, Meson chooses something @@ -102,8 +114,9 @@ class MesonPackage(PackageBase): # find libraries and pkg-config files. # See https://github.com/mesonbuild/meson/issues/2197 '--libdir={0}'.format(pkg.prefix.lib), - '--buildtype={0}'.format(build_type), - '--strip', + '-Dbuildtype={0}'.format(build_type), + '-Dstrip={0}'.format(strip), + '-Ddefault_library={0}'.format(default_library) ] return args @@ -130,6 +143,7 @@ class MesonPackage(PackageBase): * ``--libdir`` * ``--buildtype`` * ``--strip`` + * ``--default_library`` which will be set automatically. diff --git a/lib/spack/spack/build_systems/octave.py b/lib/spack/spack/build_systems/octave.py index 16d6a0a1b8..bedf717811 100644 --- a/lib/spack/spack/build_systems/octave.py +++ b/lib/spack/spack/build_systems/octave.py @@ -1,11 +1,11 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) import inspect -from spack.directives import depends_on, extends +from spack.directives import extends from spack.package import PackageBase, run_after @@ -27,7 +27,6 @@ class OctavePackage(PackageBase): build_system_class = 'OctavePackage' extends('octave') - depends_on('octave', type=('build', 'run')) def setup_build_environment(self, env): # octave does not like those environment variables to be set: diff --git a/lib/spack/spack/build_systems/oneapi.py b/lib/spack/spack/build_systems/oneapi.py index ec8732bbd6..0844ff5d82 100644 --- a/lib/spack/spack/build_systems/oneapi.py +++ b/lib/spack/spack/build_systems/oneapi.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -7,12 +7,16 @@ """ -from os.path import dirname, isdir +import getpass +import shutil +from sys import platform +from os.path import basename, dirname, isdir from spack.package import Package +from spack.util.environment import EnvironmentModifications from spack.util.executable import Executable -from llnl.util.filesystem import find_headers, find_libraries +from llnl.util.filesystem import find_headers, find_libraries, join_path class IntelOneApiPackage(Package): @@ -22,46 +26,73 @@ class IntelOneApiPackage(Package): phases = ['install'] - def component_info(self, - dir_name, - components, - releases, - url_name): - self._dir_name = dir_name - self._components = components - self._releases = releases - self._url_name = url_name + # oneAPI license does not allow mirroring outside of the + # organization (e.g. University/Company). + redistribute_source = False - def url_for_version(self, version): - release = self._release(version) - return 'https://registrationcenter-download.intel.com/akdlm/irc_nas/%s/%s' % ( - release['irc_id'], self._oneapi_file(version, release)) + @property + def component_dir(self): + """Subdirectory for this component in the install prefix.""" + raise NotImplementedError + + @property + def component_path(self): + """Path to component <prefix>/<component>/<version>.""" + return join_path(self.prefix, self.component_dir, str(self.spec.version)) + + def install(self, spec, prefix, installer_path=None): + """Shared install method for all oneapi packages.""" + + # intel-oneapi-compilers overrides the installer_path when + # installing fortran, which comes from a spack resource + if installer_path is None: + installer_path = basename(self.url_for_version(spec.version)) + + if platform == 'linux': + # Intel installer assumes and enforces that all components + # are installed into a single prefix. Spack wants to + # install each component in a separate prefix. The + # installer mechanism is implemented by saving install + # information in a directory called installercache for + # future runs. The location of the installercache depends + # on the userid. For root it is always in /var/intel. For + # non-root it is in $HOME/intel. + # + # The method for preventing this install from interfering + # with other install depends on the userid. For root, we + # delete the installercache before and after install. For + # non root we redefine the HOME environment variable. + if getpass.getuser() == 'root': + shutil.rmtree('/var/intel/installercache', ignore_errors=True) + + bash = Executable('bash') + + # Installer writes files in ~/intel set HOME so it goes to prefix + bash.add_default_env('HOME', prefix) + + bash(installer_path, + '-s', '-a', '-s', '--action', 'install', + '--eula', 'accept', + '--install-dir', prefix) - def install(self, spec, prefix): - bash = Executable('bash') + if getpass.getuser() == 'root': + shutil.rmtree('/var/intel/installercache', ignore_errors=True) - # Installer writes files in ~/intel set HOME so it goes to prefix - bash.add_default_env('HOME', prefix) + # Some installers have a bug and do not return an error code when failing + if not isdir(join_path(prefix, self.component_dir)): + raise RuntimeError('install failed') - version = spec.versions.lowest() - release = self._release(version) - bash('./%s' % self._oneapi_file(version, release), - '-s', '-a', '-s', '--action', 'install', - '--eula', 'accept', - '--components', - self._components, - '--install-dir', prefix) + def setup_run_environment(self, env): + """Adds environment variables to the generated module file. - # - # Helper functions - # + These environment variables come from running: - def _release(self, version): - return self._releases[str(version)] + .. code-block:: console - def _oneapi_file(self, version, release): - return 'l_%s_p_%s.%s_offline.sh' % ( - self._url_name, version, release['build']) + $ source {prefix}/{component}/{version}/env/vars.sh + """ + env.extend(EnvironmentModifications.from_sourcing_file( + join_path(self.component_path, 'env', 'vars.sh'))) class IntelOneApiLibraryPackage(IntelOneApiPackage): @@ -69,12 +100,11 @@ class IntelOneApiLibraryPackage(IntelOneApiPackage): @property def headers(self): - include_path = '%s/%s/latest/include' % ( - self.prefix, self._dir_name) + include_path = join_path(self.component_path, 'include') return find_headers('*', include_path, recursive=True) @property def libs(self): - lib_path = '%s/%s/latest/lib/intel64' % (self.prefix, self._dir_name) + lib_path = join_path(self.component_path, 'lib', 'intel64') lib_path = lib_path if isdir(lib_path) else dirname(lib_path) return find_libraries('*', root=lib_path, shared=True, recursive=True) diff --git a/lib/spack/spack/build_systems/perl.py b/lib/spack/spack/build_systems/perl.py index d134dc06f2..a805c51176 100644 --- a/lib/spack/spack/build_systems/perl.py +++ b/lib/spack/spack/build_systems/perl.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -7,7 +7,7 @@ import inspect import os -from spack.directives import depends_on, extends +from spack.directives import extends from spack.package import PackageBase, run_after from spack.util.executable import Executable from llnl.util.filesystem import filter_file @@ -45,8 +45,6 @@ class PerlPackage(PackageBase): extends('perl') - depends_on('perl', type=('build', 'run')) - def configure_args(self): """Produces a list containing the arguments that must be passed to :py:meth:`~.PerlPackage.configure`. Arguments should not include diff --git a/lib/spack/spack/build_systems/python.py b/lib/spack/spack/build_systems/python.py index 2616642732..696fb4aec9 100644 --- a/lib/spack/spack/build_systems/python.py +++ b/lib/spack/spack/build_systems/python.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -6,12 +6,13 @@ import inspect import os import shutil -from spack.directives import depends_on, extends +from spack.directives import extends from spack.package import PackageBase, run_after from llnl.util.filesystem import (working_dir, get_filetype, filter_file, - path_contains_subdirectory, same_path) + path_contains_subdirectory, same_path, find) from llnl.util.lang import match_predicate +import llnl.util.tty as tty class PythonPackage(PackageBase): @@ -25,20 +26,11 @@ class PythonPackage(PackageBase): * build_ext * build_clib * build_scripts - * clean * install * install_lib * install_headers * install_scripts * install_data - * sdist - * register - * bdist - * bdist_dumb - * bdist_rpm - * bdist_wininst - * upload - * check These are all standard setup.py commands and can be found by running: @@ -71,35 +63,84 @@ class PythonPackage(PackageBase): def configure(self, spec, prefix): self.setup_py('configure') """ + #: Package name, version, and extension on PyPI + pypi = None + + maintainers = ['adamjstewart'] + # Default phases phases = ['build', 'install'] - # Name of modules that the Python package provides - # This is used to test whether or not the installation succeeded - # These names generally come from running: - # - # >>> import setuptools - # >>> setuptools.find_packages() - # - # in the source tarball directory - import_modules = [] - # To be used in UI queries that require to know which # build-system class we are using build_system_class = 'PythonPackage' - #: Callback names for build-time test - build_time_test_callbacks = ['build_test'] - #: Callback names for install-time test - install_time_test_callbacks = ['import_module_test'] + install_time_test_callbacks = ['test'] extends('python') - depends_on('python', type=('build', 'run')) - py_namespace = None + @property + def homepage(self): + if self.pypi: + name = self.pypi.split('/')[0] + return 'https://pypi.org/project/' + name + '/' + + @property + def url(self): + if self.pypi: + return ( + 'https://files.pythonhosted.org/packages/source/' + + self.pypi[0] + '/' + self.pypi + ) + + @property + def list_url(self): + if self.pypi: + name = self.pypi.split('/')[0] + return 'https://pypi.org/simple/' + name + '/' + + @property + def import_modules(self): + """Names of modules that the Python package provides. + + These are used to test whether or not the installation succeeded. + These names generally come from running: + + .. code-block:: python + + >> import setuptools + >> setuptools.find_packages() + + in the source tarball directory. If the module names are incorrectly + detected, this property can be overridden by the package. + + Returns: + list: list of strings of module names + """ + modules = [] + + # Python libraries may be installed in lib or lib64 + # See issues #18520 and #17126 + for lib in ['lib', 'lib64']: + root = os.path.join(self.prefix, lib, 'python{0}'.format( + self.spec['python'].version.up_to(2)), 'site-packages') + # Some Python libraries are packages: collections of modules + # distributed in directories containing __init__.py files + for path in find(root, '__init__.py', recursive=True): + modules.append(path.replace(root + os.sep, '', 1).replace( + os.sep + '__init__.py', '').replace('/', '.')) + # Some Python libraries are modules: individual *.py files + # found in the site-packages directory + for path in find(root, '*.py', recursive=False): + modules.append(path.replace(root + os.sep, '', 1).replace( + '.py', '').replace('/', '.')) + + tty.debug('Detected the following modules: {0}'.format(modules)) + return modules + def setup_file(self): """Returns the name of the setup file to use.""" return 'setup.py' @@ -118,27 +159,6 @@ class PythonPackage(PackageBase): with working_dir(self.build_directory): self.python('-s', setup, '--no-user-cfg', *args, **kwargs) - def _setup_command_available(self, command): - """Determines whether or not a setup.py command exists. - - Args: - command (str): The command to look for - - Returns: - bool: True if the command is found, else False - """ - kwargs = { - 'output': os.devnull, - 'error': os.devnull, - 'fail_on_error': False - } - - python = inspect.getmodule(self).python - setup = self.setup_file() - - python('-s', setup, '--no-user-cfg', command, '--help', **kwargs) - return python.returncode == 0 - # The following phases and their descriptions come from: # $ python setup.py --help-commands @@ -194,16 +214,6 @@ class PythonPackage(PackageBase): """Arguments to pass to build_scripts.""" return [] - def clean(self, spec, prefix): - """Clean up temporary files from 'build' command.""" - args = self.clean_args(spec, prefix) - - self.setup_py('clean', *args) - - def clean_args(self, spec, prefix): - """Arguments to pass to clean.""" - return [] - def install(self, spec, prefix): """Install everything from build directory.""" args = self.install_args(spec, prefix) @@ -252,7 +262,7 @@ class PythonPackage(PackageBase): '--install-purelib=%s' % pure_site_packages_dir, '--install-platlib=%s' % plat_site_packages_dir, '--install-scripts=bin', - '--install-data=""', + '--install-data=', '--install-headers=%s' % inc_dir ] @@ -298,115 +308,18 @@ class PythonPackage(PackageBase): """Arguments to pass to install_data.""" return [] - def sdist(self, spec, prefix): - """Create a source distribution (tarball, zip file, etc.).""" - args = self.sdist_args(spec, prefix) - - self.setup_py('sdist', *args) - - def sdist_args(self, spec, prefix): - """Arguments to pass to sdist.""" - return [] - - def register(self, spec, prefix): - """Register the distribution with the Python package index.""" - args = self.register_args(spec, prefix) - - self.setup_py('register', *args) - - def register_args(self, spec, prefix): - """Arguments to pass to register.""" - return [] - - def bdist(self, spec, prefix): - """Create a built (binary) distribution.""" - args = self.bdist_args(spec, prefix) - - self.setup_py('bdist', *args) - - def bdist_args(self, spec, prefix): - """Arguments to pass to bdist.""" - return [] - - def bdist_dumb(self, spec, prefix): - '''Create a "dumb" built distribution.''' - args = self.bdist_dumb_args(spec, prefix) - - self.setup_py('bdist_dumb', *args) - - def bdist_dumb_args(self, spec, prefix): - """Arguments to pass to bdist_dumb.""" - return [] - - def bdist_rpm(self, spec, prefix): - """Create an RPM distribution.""" - args = self.bdist_rpm(spec, prefix) - - self.setup_py('bdist_rpm', *args) - - def bdist_rpm_args(self, spec, prefix): - """Arguments to pass to bdist_rpm.""" - return [] - - def bdist_wininst(self, spec, prefix): - """Create an executable installer for MS Windows.""" - args = self.bdist_wininst_args(spec, prefix) - - self.setup_py('bdist_wininst', *args) - - def bdist_wininst_args(self, spec, prefix): - """Arguments to pass to bdist_wininst.""" - return [] - - def upload(self, spec, prefix): - """Upload binary package to PyPI.""" - args = self.upload_args(spec, prefix) - - self.setup_py('upload', *args) - - def upload_args(self, spec, prefix): - """Arguments to pass to upload.""" - return [] - - def check(self, spec, prefix): - """Perform some checks on the package.""" - args = self.check_args(spec, prefix) - - self.setup_py('check', *args) - - def check_args(self, spec, prefix): - """Arguments to pass to check.""" - return [] - # Testing - def build_test(self): - """Run unit tests after in-place build. - - These tests are only run if the package actually has a 'test' command. - """ - if self._setup_command_available('test'): - args = self.test_args(self.spec, self.prefix) - - self.setup_py('test', *args) - - def test_args(self, spec, prefix): - """Arguments to pass to test.""" - return [] - - run_after('build')(PackageBase._run_default_build_time_test_callbacks) - - def import_module_test(self): - """Attempts to import the module that was just installed. - - This test is only run if the package overrides - :py:attr:`import_modules` with a list of module names.""" + def test(self): + """Attempts to import modules of the installed package.""" # Make sure we are importing the installed modules, - # not the ones in the current directory - with working_dir('spack-test', create=True): - for module in self.import_modules: - self.python('-c', 'import {0}'.format(module)) + # not the ones in the source directory + for module in self.import_modules: + self.run_test(inspect.getmodule(self).python.path, + ['-c', 'import {0}'.format(module)], + purpose='checking import of {0}'.format(module), + work_dir='spack-test') run_after('install')(PackageBase._run_default_install_time_test_callbacks) diff --git a/lib/spack/spack/build_systems/qmake.py b/lib/spack/spack/build_systems/qmake.py index 22914d4d3a..5ddb8d7fa3 100644 --- a/lib/spack/spack/build_systems/qmake.py +++ b/lib/spack/spack/build_systems/qmake.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -6,6 +6,7 @@ import inspect +from llnl.util.filesystem import working_dir from spack.directives import depends_on from spack.package import PackageBase, run_after @@ -37,6 +38,11 @@ class QMakePackage(PackageBase): depends_on('qt', type='build') + @property + def build_directory(self): + """The directory containing the ``*.pro`` file.""" + return self.stage.source_path + def qmake_args(self): """Produces a list containing all the arguments that must be passed to qmake @@ -45,22 +51,30 @@ class QMakePackage(PackageBase): def qmake(self, spec, prefix): """Run ``qmake`` to configure the project and generate a Makefile.""" - inspect.getmodule(self).qmake(*self.qmake_args()) + + with working_dir(self.build_directory): + inspect.getmodule(self).qmake(*self.qmake_args()) def build(self, spec, prefix): """Make the build targets""" - inspect.getmodule(self).make() + + with working_dir(self.build_directory): + inspect.getmodule(self).make() def install(self, spec, prefix): """Make the install targets""" - inspect.getmodule(self).make('install') + + with working_dir(self.build_directory): + inspect.getmodule(self).make('install') # Tests def check(self): """Searches the Makefile for a ``check:`` target and runs it if found. """ - self._if_make_target_execute('check') + + with working_dir(self.build_directory): + self._if_make_target_execute('check') run_after('build')(PackageBase._run_default_build_time_test_callbacks) diff --git a/lib/spack/spack/build_systems/r.py b/lib/spack/spack/build_systems/r.py index 6ee2668833..ac57672a39 100644 --- a/lib/spack/spack/build_systems/r.py +++ b/lib/spack/spack/build_systems/r.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -6,7 +6,7 @@ import inspect -from spack.directives import depends_on, extends +from spack.directives import extends from spack.package import PackageBase, run_after @@ -25,13 +25,49 @@ class RPackage(PackageBase): """ phases = ['install'] + # package attributes that can be expanded to set the homepage, url, + # list_url, and git values + # For CRAN packages + cran = None + + # For Bioconductor packages + bioc = None + + maintainers = ['glennpj'] + #: This attribute is used in UI queries that need to know the build #: system base class build_system_class = 'RPackage' extends('r') - depends_on('r', type=('build', 'run')) + @property + def homepage(self): + if self.cran: + return 'https://cloud.r-project.org/package=' + self.cran + elif self.bioc: + return 'https://bioconductor.org/packages/' + self.bioc + + @property + def url(self): + if self.cran: + return ( + 'https://cloud.r-project.org/src/contrib/' + + self.cran + '_' + str(list(self.versions)[0]) + '.tar.gz' + ) + + @property + def list_url(self): + if self.cran: + return ( + 'https://cloud.r-project.org/src/contrib/Archive/' + + self.cran + '/' + ) + + @property + def git(self): + if self.bioc: + return 'https://git.bioconductor.org/packages/' + self.bioc def configure_args(self): """Arguments to pass to install via ``--configure-args``.""" @@ -48,6 +84,7 @@ class RPackage(PackageBase): config_vars = self.configure_vars() args = [ + '--vanilla', 'CMD', 'INSTALL' ] diff --git a/lib/spack/spack/build_systems/rocm.py b/lib/spack/spack/build_systems/rocm.py index 0107c6376b..21a200e159 100644 --- a/lib/spack/spack/build_systems/rocm.py +++ b/lib/spack/spack/build_systems/rocm.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -30,6 +30,8 @@ # environment: {} # extra_rpaths: [] # +# It is advisable to replace /rocm/ in the paths above with /rocm-version/ +# and introduce spec version numbers to ensure reproducible results. # # 2. hip and its dependencies are currently NOT picked up by spack # automatically, and should therefore be added to packages.yaml by hand: @@ -37,7 +39,7 @@ # in packages.yaml: # hip: # externals: -# - spec: hip@3.8.20371-d1886b0b +# - spec: hip # prefix: /opt/rocm/hip # extra_attributes: # compilers: @@ -64,6 +66,9 @@ # cxx: /opt/rocm/llvm/bin/clang++ # buildable: false # +# It is advisable to replace /rocm/ in the paths above with /rocm-version/ +# and introduce spec version numbers to ensure reproducible results. +# # 3. In part 2, DO NOT list the path to hsa as /opt/rocm/hsa ! You want spack # to find hsa in /opt/rocm/include/hsa/hsa.h . The directory of # /opt/rocm/hsa also has an hsa.h file, but it won't be found because spack @@ -102,6 +107,8 @@ class ROCmPackage(PackageBase): depends_on('hsa-rocr-dev', when='+rocm') depends_on('hip', when='+rocm') + conflicts('^blt@:0.3.6', when='+rocm') + # need amd gpu type for rocm builds conflicts('amdgpu_target=none', when='+rocm') diff --git a/lib/spack/spack/build_systems/ruby.py b/lib/spack/spack/build_systems/ruby.py index 32d6633164..5736d0913a 100644 --- a/lib/spack/spack/build_systems/ruby.py +++ b/lib/spack/spack/build_systems/ruby.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -6,7 +6,7 @@ import glob import inspect -from spack.directives import depends_on, extends +from spack.directives import extends from spack.package import PackageBase, run_after @@ -27,8 +27,6 @@ class RubyPackage(PackageBase): extends('ruby') - depends_on('ruby', type=('build', 'run')) - def build(self, spec, prefix): """Build a Ruby gem.""" diff --git a/lib/spack/spack/build_systems/scons.py b/lib/spack/spack/build_systems/scons.py index 5e17666b71..56ee08a113 100644 --- a/lib/spack/spack/build_systems/scons.py +++ b/lib/spack/spack/build_systems/scons.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/build_systems/sip.py b/lib/spack/spack/build_systems/sip.py index f814ef1837..47713a100a 100644 --- a/lib/spack/spack/build_systems/sip.py +++ b/lib/spack/spack/build_systems/sip.py @@ -1,14 +1,15 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) import inspect +import os -from llnl.util.filesystem import working_dir, join_path +from llnl.util.filesystem import find, working_dir, join_path from spack.directives import depends_on, extends from spack.package import PackageBase, run_after -import os +import llnl.util.tty as tty class SIPPackage(PackageBase): @@ -36,13 +37,52 @@ class SIPPackage(PackageBase): sip_module = 'sip' #: Callback names for install-time test - install_time_test_callbacks = ['import_module_test'] + install_time_test_callbacks = ['test'] extends('python') depends_on('qt') depends_on('py-sip') + @property + def import_modules(self): + """Names of modules that the Python package provides. + + These are used to test whether or not the installation succeeded. + These names generally come from running: + + .. code-block:: python + + >> import setuptools + >> setuptools.find_packages() + + in the source tarball directory. If the module names are incorrectly + detected, this property can be overridden by the package. + + Returns: + list: list of strings of module names + """ + modules = [] + + # Python libraries may be installed in lib or lib64 + # See issues #18520 and #17126 + for lib in ['lib', 'lib64']: + root = os.path.join(self.prefix, lib, 'python{0}'.format( + self.spec['python'].version.up_to(2)), 'site-packages') + # Some Python libraries are packages: collections of modules + # distributed in directories containing __init__.py files + for path in find(root, '__init__.py', recursive=True): + modules.append(path.replace(root + os.sep, '', 1).replace( + os.sep + '__init__.py', '').replace('/', '.')) + # Some Python libraries are modules: individual *.py files + # found in the site-packages directory + for path in find(root, '*.py', recursive=False): + modules.append(path.replace(root + os.sep, '', 1).replace( + '.py', '').replace('/', '.')) + + tty.debug('Detected the following modules: {0}'.format(modules)) + return modules + def python(self, *args, **kwargs): """The python ``Executable``.""" inspect.getmodule(self).python(*args, **kwargs) @@ -98,17 +138,16 @@ class SIPPackage(PackageBase): # Testing - def import_module_test(self): - """Attempts to import the module that was just installed. - - This test is only run if the package overrides - :py:attr:`import_modules` with a list of module names.""" + def test(self): + """Attempts to import modules of the installed package.""" # Make sure we are importing the installed modules, - # not the ones in the current directory - with working_dir('spack-test', create=True): - for module in self.import_modules: - self.python('-c', 'import {0}'.format(module)) + # not the ones in the source directory + for module in self.import_modules: + self.run_test(inspect.getmodule(self).python.path, + ['-c', 'import {0}'.format(module)], + purpose='checking import of {0}'.format(module), + work_dir='spack-test') run_after('install')(PackageBase._run_default_install_time_test_callbacks) diff --git a/lib/spack/spack/build_systems/sourceforge.py b/lib/spack/spack/build_systems/sourceforge.py index 602a099c17..cfea664f51 100644 --- a/lib/spack/spack/build_systems/sourceforge.py +++ b/lib/spack/spack/build_systems/sourceforge.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/build_systems/sourceware.py b/lib/spack/spack/build_systems/sourceware.py index b779b530dc..89a157c22c 100644 --- a/lib/spack/spack/build_systems/sourceware.py +++ b/lib/spack/spack/build_systems/sourceware.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/build_systems/waf.py b/lib/spack/spack/build_systems/waf.py index a6dbbbdb35..99679ba64a 100644 --- a/lib/spack/spack/build_systems/waf.py +++ b/lib/spack/spack/build_systems/waf.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/build_systems/xorg.py b/lib/spack/spack/build_systems/xorg.py index ae28b30929..801ad37eeb 100644 --- a/lib/spack/spack/build_systems/xorg.py +++ b/lib/spack/spack/build_systems/xorg.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/caches.py b/lib/spack/spack/caches.py index 49624c06b2..9f99ab75ed 100644 --- a/lib/spack/spack/caches.py +++ b/lib/spack/spack/caches.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -17,7 +17,7 @@ import spack.util.file_cache import spack.util.path -def _misc_cache(): +def misc_cache_location(): """The ``misc_cache`` is Spack's cache for small data. Currently the ``misc_cache`` stores indexes for virtual dependency @@ -27,7 +27,11 @@ def _misc_cache(): if not path: path = os.path.join(spack.paths.user_config_path, 'cache') path = spack.util.path.canonicalize_path(path) + return path + +def _misc_cache(): + path = misc_cache_location() return spack.util.file_cache.FileCache(path) @@ -35,7 +39,7 @@ def _misc_cache(): misc_cache = llnl.util.lang.Singleton(_misc_cache) -def _fetch_cache(): +def fetch_cache_location(): """Filesystem cache of downloaded archives. This prevents Spack from repeatedly fetch the same files when @@ -45,7 +49,11 @@ def _fetch_cache(): if not path: path = os.path.join(spack.paths.var_path, "cache") path = spack.util.path.canonicalize_path(path) + return path + +def _fetch_cache(): + path = fetch_cache_location() return spack.fetch_strategy.FsCache(path) diff --git a/lib/spack/spack/ci.py b/lib/spack/spack/ci.py index f38ab95d96..4daffac44c 100644 --- a/lib/spack/spack/ci.py +++ b/lib/spack/spack/ci.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -22,7 +22,7 @@ import llnl.util.tty as tty import spack import spack.binary_distribution as bindist -import spack.cmd.buildcache as buildcache +import spack.cmd import spack.compilers as compilers import spack.config as cfg import spack.environment as ev @@ -41,7 +41,8 @@ JOB_RETRY_CONDITIONS = [ 'always', ] -SPACK_PR_MIRRORS_ROOT_URL = 's3://spack-pr-mirrors' +SPACK_PR_MIRRORS_ROOT_URL = 's3://spack-binaries-prs' +TEMP_STORAGE_MIRROR_NAME = 'ci_temporary_mirror' spack_gpg = spack.main.SpackCommand('gpg') spack_compiler = spack.main.SpackCommand('compiler') @@ -202,8 +203,8 @@ def format_root_spec(spec, main_phase, strip_compiler): # spec.name, spec.version, spec.compiler, spec.architecture) -def spec_deps_key_label(s): - return s.dag_hash(), "%s/%s" % (s.name, s.dag_hash(7)) +def spec_deps_key(s): + return '{0}/{1}'.format(s.name, s.dag_hash(7)) def _add_dependency(spec_label, dep_label, deps): @@ -214,8 +215,8 @@ def _add_dependency(spec_label, dep_label, deps): deps[spec_label].add(dep_label) -def get_spec_dependencies(specs, deps, spec_labels): - spec_deps_obj = compute_spec_deps(specs) +def get_spec_dependencies(specs, deps, spec_labels, check_index_only=False): + spec_deps_obj = compute_spec_deps(specs, check_index_only=check_index_only) if spec_deps_obj: dependencies = spec_deps_obj['dependencies'] @@ -225,19 +226,25 @@ def get_spec_dependencies(specs, deps, spec_labels): spec_labels[entry['label']] = { 'spec': Spec(entry['spec']), 'rootSpec': entry['root_spec'], + 'needs_rebuild': entry['needs_rebuild'], } for entry in dependencies: _add_dependency(entry['spec'], entry['depends'], deps) -def stage_spec_jobs(specs): +def stage_spec_jobs(specs, check_index_only=False): """Take a set of release specs and generate a list of "stages", where the jobs in any stage are dependent only on jobs in previous stages. This allows us to maximize build parallelism within the gitlab-ci framework. Arguments: specs (Iterable): Specs to build + check_index_only (bool): Regardless of whether DAG pruning is enabled, + all configured mirrors are searched to see if binaries for specs + are up to date on those mirrors. This flag limits that search to + the binary cache indices on those mirrors to speed the process up, + even though there is no garantee the index is up to date. Returns: A tuple of information objects describing the specs, dependencies and stages: @@ -274,7 +281,8 @@ def stage_spec_jobs(specs): deps = {} spec_labels = {} - get_spec_dependencies(specs, deps, spec_labels) + get_spec_dependencies( + specs, deps, spec_labels, check_index_only=check_index_only) # Save the original deps, as we need to return them at the end of the # function. In the while loop below, the "dependencies" variable is @@ -311,12 +319,15 @@ def print_staging_summary(spec_labels, dependencies, stages): for job in sorted(stage): s = spec_labels[job]['spec'] - tty.msg(' {0} -> {1}'.format(job, get_spec_string(s))) + tty.msg(' [{1}] {0} -> {2}'.format( + job, + 'x' if spec_labels[job]['needs_rebuild'] else ' ', + get_spec_string(s))) stage_index += 1 -def compute_spec_deps(spec_list): +def compute_spec_deps(spec_list, check_index_only=False): """ Computes all the dependencies for the spec(s) and generates a JSON object which provides both a list of unique spec names as well as a @@ -386,33 +397,35 @@ def compute_spec_deps(spec_list): # root_spec = get_spec_string(spec) root_spec = spec - rkey, rlabel = spec_deps_key_label(spec) - for s in spec.traverse(deptype=all): if s.external: tty.msg('Will not stage external pkg: {0}'.format(s)) continue - skey, slabel = spec_deps_key_label(s) - spec_labels[slabel] = { + up_to_date_mirrors = bindist.get_mirrors_for_spec( + spec=s, full_hash_match=True, index_only=check_index_only) + + skey = spec_deps_key(s) + spec_labels[skey] = { 'spec': get_spec_string(s), 'root': root_spec, + 'needs_rebuild': not up_to_date_mirrors, } - append_dep(rlabel, slabel) for d in s.dependencies(deptype=all): - dkey, dlabel = spec_deps_key_label(d) + dkey = spec_deps_key(d) if d.external: tty.msg('Will not stage external dep: {0}'.format(d)) continue - append_dep(slabel, dlabel) + append_dep(skey, dkey) for spec_label, spec_holder in spec_labels.items(): specs.append({ 'label': spec_label, 'spec': spec_holder['spec'], 'root_spec': spec_holder['root'], + 'needs_rebuild': spec_holder['needs_rebuild'], }) deps_json_obj = { @@ -481,22 +494,43 @@ def pkg_name_from_spec_label(spec_label): def format_job_needs(phase_name, strip_compilers, dep_jobs, - osname, build_group, enable_artifacts_buildcache): + osname, build_group, prune_dag, stage_spec_dict, + enable_artifacts_buildcache): needs_list = [] for dep_job in dep_jobs: - needs_list.append({ - 'job': get_job_name(phase_name, - strip_compilers, - dep_job, - osname, - build_group), - 'artifacts': enable_artifacts_buildcache, - }) + dep_spec_key = spec_deps_key(dep_job) + dep_spec_info = stage_spec_dict[dep_spec_key] + + if not prune_dag or dep_spec_info['needs_rebuild']: + needs_list.append({ + 'job': get_job_name(phase_name, + strip_compilers, + dep_job, + osname, + build_group), + 'artifacts': enable_artifacts_buildcache, + }) return needs_list -def generate_gitlab_ci_yaml(env, print_summary, output_file, - run_optimizer=False, use_dependencies=False): +def add_pr_mirror(url): + cfg_scope = cfg.default_modify_scope() + mirrors = cfg.get('mirrors', scope=cfg_scope) + items = [(n, u) for n, u in mirrors.items()] + items.insert(0, ('ci_pr_mirror', url)) + cfg.set('mirrors', syaml.syaml_dict(items), scope=cfg_scope) + + +def remove_pr_mirror(): + cfg_scope = cfg.default_modify_scope() + mirrors = cfg.get('mirrors', scope=cfg_scope) + mirrors.pop('ci_pr_mirror') + cfg.set('mirrors', mirrors, scope=cfg_scope) + + +def generate_gitlab_ci_yaml(env, print_summary, output_file, prune_dag=False, + check_index_only=False, run_optimizer=False, + use_dependencies=False): # FIXME: What's the difference between one that opens with 'spack' # and one that opens with 'env'? This will only handle the former. with spack.concretize.disable_compiler_existence_check(): @@ -509,10 +543,6 @@ def generate_gitlab_ci_yaml(env, print_summary, output_file, gitlab_ci = yaml_root['gitlab-ci'] - final_job_config = None - if 'final-stage-rebuild-index' in gitlab_ci: - final_job_config = gitlab_ci['final-stage-rebuild-index'] - build_group = None enable_cdash_reporting = False cdash_auth_token = None @@ -539,13 +569,31 @@ def generate_gitlab_ci_yaml(env, print_summary, output_file, pr_mirror_url = url_util.join(SPACK_PR_MIRRORS_ROOT_URL, spack_pr_branch) + if 'mirrors' not in yaml_root or len(yaml_root['mirrors'].values()) < 1: + tty.die('spack ci generate requires an env containing a mirror') + ci_mirrors = yaml_root['mirrors'] mirror_urls = [url for url in ci_mirrors.values()] + # Check for a list of "known broken" specs that we should not bother + # trying to build. + broken_specs_url = '' + known_broken_specs_encountered = [] + if 'broken-specs-url' in gitlab_ci: + broken_specs_url = gitlab_ci['broken-specs-url'] + enable_artifacts_buildcache = False if 'enable-artifacts-buildcache' in gitlab_ci: enable_artifacts_buildcache = gitlab_ci['enable-artifacts-buildcache'] + rebuild_index_enabled = True + if 'rebuild-index' in gitlab_ci and gitlab_ci['rebuild-index'] is False: + rebuild_index_enabled = False + + temp_storage_url_prefix = None + if 'temporary-storage-url-prefix' in gitlab_ci: + temp_storage_url_prefix = gitlab_ci['temporary-storage-url-prefix'] + bootstrap_specs = [] phases = [] if 'bootstrap' in gitlab_ci: @@ -573,19 +621,27 @@ def generate_gitlab_ci_yaml(env, print_summary, output_file, 'strip-compilers': False, }) - staged_phases = {} - for phase in phases: - phase_name = phase['name'] - with spack.concretize.disable_compiler_existence_check(): - staged_phases[phase_name] = stage_spec_jobs( - env.spec_lists[phase_name]) + # Add this mirror if it's enabled, as some specs might be up to date + # here and thus not need to be rebuilt. + if pr_mirror_url: + add_pr_mirror(pr_mirror_url) - if print_summary: + # Speed up staging by first fetching binary indices from all mirrors + # (including the per-PR mirror we may have just added above). + bindist.binary_index.update() + + staged_phases = {} + try: for phase in phases: phase_name = phase['name'] - tty.msg('Stages for phase "{0}"'.format(phase_name)) - phase_stages = staged_phases[phase_name] - print_staging_summary(*phase_stages) + with spack.concretize.disable_compiler_existence_check(): + staged_phases[phase_name] = stage_spec_jobs( + env.spec_lists[phase_name], + check_index_only=check_index_only) + finally: + # Clean up PR mirror if enabled + if pr_mirror_url: + remove_pr_mirror() all_job_names = [] output_object = {} @@ -611,10 +667,19 @@ def generate_gitlab_ci_yaml(env, print_summary, output_file, stage_id += 1 for spec_label in stage_jobs: - root_spec = spec_labels[spec_label]['rootSpec'] + spec_record = spec_labels[spec_label] + root_spec = spec_record['rootSpec'] pkg_name = pkg_name_from_spec_label(spec_label) release_spec = root_spec[pkg_name] + # Check if this spec is in our list of known failures. + if broken_specs_url: + full_hash = release_spec.full_hash() + broken_spec_path = url_util.join(broken_specs_url, full_hash) + if web_util.url_exists(broken_spec_path): + known_broken_specs_encountered.append('{0} ({1})'.format( + release_spec, full_hash)) + runner_attribs = find_matching_config( release_spec, gitlab_ci) @@ -678,11 +743,15 @@ def generate_gitlab_ci_yaml(env, print_summary, output_file, job_dependencies = [] if spec_label in dependencies: if enable_artifacts_buildcache: + # Get dependencies transitively, so they're all + # available in the artifacts buildcache. dep_jobs = [ d for d in release_spec.traverse(deptype=all, root=False) ] else: + # In this case, "needs" is only used for scheduling + # purposes, so we only get the direct dependencies. dep_jobs = [] for dep_label in dependencies[spec_label]: dep_pkg = pkg_name_from_spec_label(dep_label) @@ -690,10 +759,13 @@ def generate_gitlab_ci_yaml(env, print_summary, output_file, dep_jobs.append(dep_root[dep_pkg]) job_dependencies.extend( - format_job_needs(phase_name, strip_compilers, dep_jobs, - osname, build_group, + format_job_needs(phase_name, strip_compilers, + dep_jobs, osname, build_group, + prune_dag, spec_labels, enable_artifacts_buildcache)) + rebuild_spec = spec_record['needs_rebuild'] + # This next section helps gitlab make sure the right # bootstrapped compiler exists in the artifacts buildcache by # creating an artificial dependency between this spec and its @@ -709,11 +781,12 @@ def generate_gitlab_ci_yaml(env, print_summary, output_file, compiler_pkg_spec = compilers.pkg_spec_for_compiler( release_spec.compiler) for bs in bootstrap_specs: - bs_arch = bs['spec'].architecture + c_spec = bs['spec'] + bs_arch = c_spec.architecture bs_arch_family = (bs_arch.target .microarchitecture .family) - if (bs['spec'].satisfies(compiler_pkg_spec) and + if (c_spec.satisfies(compiler_pkg_spec) and bs_arch_family == spec_arch_family): # We found the bootstrap compiler this release spec # should be built with, so for DAG scheduling @@ -721,10 +794,24 @@ def generate_gitlab_ci_yaml(env, print_summary, output_file, # to the jobs "needs". But if artifact buildcache # is enabled, we'll have to add all transtive deps # of the compiler as well. - dep_jobs = [bs['spec']] + + # Here we check whether the bootstrapped compiler + # needs to be rebuilt. Until compilers are proper + # dependencies, we artificially force the spec to + # be rebuilt if the compiler targeted to build it + # needs to be rebuilt. + bs_specs, _, _ = staged_phases[bs['phase-name']] + c_spec_key = spec_deps_key(c_spec) + rbld_comp = bs_specs[c_spec_key]['needs_rebuild'] + rebuild_spec = rebuild_spec or rbld_comp + # Also update record so dependents do not fail to + # add this spec to their "needs" + spec_record['needs_rebuild'] = rebuild_spec + + dep_jobs = [c_spec] if enable_artifacts_buildcache: dep_jobs = [ - d for d in bs['spec'].traverse(deptype=all) + d for d in c_spec.traverse(deptype=all) ] job_dependencies.extend( @@ -733,6 +820,8 @@ def generate_gitlab_ci_yaml(env, print_summary, output_file, dep_jobs, str(bs_arch), build_group, + prune_dag, + bs_specs, enable_artifacts_buildcache)) else: debug_msg = ''.join([ @@ -741,9 +830,14 @@ def generate_gitlab_ci_yaml(env, print_summary, output_file, 'not the compiler required by the spec, or ', 'because the target arch families of the ', 'spec and the compiler did not match' - ]).format(bs['spec'], release_spec) + ]).format(c_spec, release_spec) tty.debug(debug_msg) + if prune_dag and not rebuild_spec: + continue + + job_vars['SPACK_SPEC_NEEDS_REBUILD'] = str(rebuild_spec) + if enable_cdash_reporting: cdash_build_name = get_cdash_build_name( release_spec, build_group) @@ -787,7 +881,8 @@ def generate_gitlab_ci_yaml(env, print_summary, output_file, 'retry': { 'max': 2, 'when': JOB_RETRY_CONDITIONS, - } + }, + 'interruptible': True } length_needs = len(job_dependencies) @@ -812,11 +907,19 @@ def generate_gitlab_ci_yaml(env, print_summary, output_file, output_object[job_name] = job_object job_id += 1 + if print_summary: + for phase in phases: + phase_name = phase['name'] + tty.msg('Stages for phase "{0}"'.format(phase_name)) + phase_stages = staged_phases[phase_name] + print_staging_summary(*phase_stages) + tty.debug('{0} build jobs generated in {1} stages'.format( job_id, stage_id)) - tty.debug('The max_needs_job is {0}, with {1} needs'.format( - max_needs_job, max_length_needs)) + if job_id > 0: + tty.debug('The max_needs_job is {0}, with {1} needs'.format( + max_needs_job, max_length_needs)) # Use "all_job_names" to populate the build group for this set if enable_cdash_reporting and cdash_auth_token: @@ -828,63 +931,126 @@ def generate_gitlab_ci_yaml(env, print_summary, output_file, else: tty.warn('Unable to populate buildgroup without CDash credentials') - if final_job_config and not is_pr_pipeline: - # Add an extra, final job to regenerate the index - final_stage = 'stage-rebuild-index' - final_job = { - 'stage': final_stage, - 'script': 'spack buildcache update-index --keys -d {0}'.format( - mirror_urls[0]), - 'tags': final_job_config['tags'], - 'when': 'always' - } - if 'image' in final_job_config: - final_job['image'] = final_job_config['image'] - if before_script: - final_job['before_script'] = before_script - if after_script: - final_job['after_script'] = after_script - output_object['rebuild-index'] = final_job - stage_names.append(final_stage) - - output_object['stages'] = stage_names - - # Capture the version of spack used to generate the pipeline, transform it - # into a value that can be passed to "git checkout", and save it in a - # global yaml variable - spack_version = spack.main.get_version() - version_to_clone = None - v_match = re.match(r"^\d+\.\d+\.\d+$", spack_version) - if v_match: - version_to_clone = 'v{0}'.format(v_match.group(0)) - else: - v_match = re.match(r"^[^-]+-[^-]+-([a-f\d]+)$", spack_version) + service_job_config = None + if 'service-job-attributes' in gitlab_ci: + service_job_config = gitlab_ci['service-job-attributes'] + + default_attrs = [ + 'image', + 'tags', + 'variables', + 'before_script', + # 'script', + 'after_script', + ] + + if job_id > 0: + if temp_storage_url_prefix: + # There were some rebuild jobs scheduled, so we will need to + # schedule a job to clean up the temporary storage location + # associated with this pipeline. + stage_names.append('cleanup-temp-storage') + cleanup_job = {} + + if service_job_config: + copy_attributes(default_attrs, + service_job_config, + cleanup_job) + + cleanup_job['stage'] = 'cleanup-temp-storage' + cleanup_job['script'] = [ + 'spack -d mirror destroy --mirror-url {0}/$CI_PIPELINE_ID'.format( + temp_storage_url_prefix) + ] + cleanup_job['when'] = 'always' + + output_object['cleanup'] = cleanup_job + + if rebuild_index_enabled: + # Add a final job to regenerate the index + stage_names.append('stage-rebuild-index') + final_job = {} + + if service_job_config: + copy_attributes(default_attrs, + service_job_config, + final_job) + + index_target_mirror = mirror_urls[0] + if is_pr_pipeline: + index_target_mirror = pr_mirror_url + + final_job['stage'] = 'stage-rebuild-index' + final_job['script'] = [ + 'spack buildcache update-index --keys -d {0}'.format( + index_target_mirror) + ] + final_job['when'] = 'always' + + output_object['rebuild-index'] = final_job + + output_object['stages'] = stage_names + + # Capture the version of spack used to generate the pipeline, transform it + # into a value that can be passed to "git checkout", and save it in a + # global yaml variable + spack_version = spack.main.get_version() + version_to_clone = None + v_match = re.match(r"^\d+\.\d+\.\d+$", spack_version) if v_match: - version_to_clone = v_match.group(1) + version_to_clone = 'v{0}'.format(v_match.group(0)) else: - version_to_clone = spack_version + v_match = re.match(r"^[^-]+-[^-]+-([a-f\d]+)$", spack_version) + if v_match: + version_to_clone = v_match.group(1) + else: + version_to_clone = spack_version - output_object['variables'] = { - 'SPACK_VERSION': spack_version, - 'SPACK_CHECKOUT_VERSION': version_to_clone, - } + output_object['variables'] = { + 'SPACK_VERSION': spack_version, + 'SPACK_CHECKOUT_VERSION': version_to_clone, + } - if pr_mirror_url: - output_object['variables']['SPACK_PR_MIRROR_URL'] = pr_mirror_url + if pr_mirror_url: + output_object['variables']['SPACK_PR_MIRROR_URL'] = pr_mirror_url - sorted_output = {} - for output_key, output_value in sorted(output_object.items()): - sorted_output[output_key] = output_value + sorted_output = {} + for output_key, output_value in sorted(output_object.items()): + sorted_output[output_key] = output_value - # TODO(opadron): remove this or refactor - if run_optimizer: - import spack.ci_optimization as ci_opt - sorted_output = ci_opt.optimizer(sorted_output) + # TODO(opadron): remove this or refactor + if run_optimizer: + import spack.ci_optimization as ci_opt + sorted_output = ci_opt.optimizer(sorted_output) - # TODO(opadron): remove this or refactor - if use_dependencies: - import spack.ci_needs_workaround as cinw - sorted_output = cinw.needs_to_dependencies(sorted_output) + # TODO(opadron): remove this or refactor + if use_dependencies: + import spack.ci_needs_workaround as cinw + sorted_output = cinw.needs_to_dependencies(sorted_output) + else: + # No jobs were generated + tty.debug('No specs to rebuild, generating no-op job') + noop_job = {} + + if service_job_config: + copy_attributes(default_attrs, + service_job_config, + noop_job) + + if 'script' not in noop_job: + noop_job['script'] = [ + 'echo "All specs already up to date, nothing to rebuild."', + ] + + sorted_output = {'no-specs-to-rebuild': noop_job} + + if known_broken_specs_encountered: + error_msg = ( + 'Pipeline generation failed due to the presence of the ' + 'following specs that are known to be broken in develop:\n') + for broken_spec in known_broken_specs_encountered: + error_msg += '* {0}\n'.format(broken_spec) + tty.die(error_msg) with open(output_file, 'w') as outf: outf.write(syaml.dump_config(sorted_output, default_flow_style=True)) @@ -1120,16 +1286,37 @@ def read_cdashid_from_mirror(spec, mirror_url): def push_mirror_contents(env, spec, yaml_path, mirror_url, build_id, sign_binaries): if mirror_url: - unsigned = not sign_binaries - tty.debug('Creating buildcache ({0})'.format( - 'unsigned' if unsigned else 'signed')) - buildcache._createtarball(env, spec_yaml=yaml_path, add_deps=False, - output_location=mirror_url, force=True, - allow_root=True, unsigned=unsigned) - if build_id: - tty.debug('Writing cdashid ({0}) to remote mirror: {1}'.format( - build_id, mirror_url)) - write_cdashid_to_mirror(build_id, spec, mirror_url) + try: + unsigned = not sign_binaries + tty.debug('Creating buildcache ({0})'.format( + 'unsigned' if unsigned else 'signed')) + spack.cmd.buildcache._createtarball( + env, spec_yaml=yaml_path, add_deps=False, + output_location=mirror_url, force=True, allow_root=True, + unsigned=unsigned) + if build_id: + tty.debug('Writing cdashid ({0}) to remote mirror: {1}'.format( + build_id, mirror_url)) + write_cdashid_to_mirror(build_id, spec, mirror_url) + except Exception as inst: + # If the mirror we're pushing to is on S3 and there's some + # permissions problem, for example, we can't just target + # that exception type here, since users of the + # `spack ci rebuild' may not need or want any dependency + # on boto3. So we use the first non-boto exception type + # in the heirarchy: + # boto3.exceptions.S3UploadFailedError + # boto3.exceptions.Boto3Error + # Exception + # BaseException + # object + err_msg = 'Error msg: {0}'.format(inst) + if 'Access Denied' in err_msg: + tty.msg('Permission problem writing to {0}'.format( + mirror_url)) + tty.msg(err_msg) + else: + raise inst def copy_stage_logs_to_artifacts(job_spec, job_log_dir): diff --git a/lib/spack/spack/ci_needs_workaround.py b/lib/spack/spack/ci_needs_workaround.py index ba9f7c62af..ff3c621f97 100644 --- a/lib/spack/spack/ci_needs_workaround.py +++ b/lib/spack/spack/ci_needs_workaround.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/ci_optimization.py b/lib/spack/spack/ci_optimization.py index c5e69c649c..94528c856a 100644 --- a/lib/spack/spack/ci_optimization.py +++ b/lib/spack/spack/ci_optimization.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/__init__.py b/lib/spack/spack/cmd/__init__.py index d728325699..e9b4b60299 100644 --- a/lib/spack/spack/cmd/__init__.py +++ b/lib/spack/spack/cmd/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/activate.py b/lib/spack/spack/cmd/activate.py index bfca9f5604..a6c27b2f1d 100644 --- a/lib/spack/spack/cmd/activate.py +++ b/lib/spack/spack/cmd/activate.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/add.py b/lib/spack/spack/cmd/add.py index 94c8620dbb..ae1a9c9b7e 100644 --- a/lib/spack/spack/cmd/add.py +++ b/lib/spack/spack/cmd/add.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/analyze.py b/lib/spack/spack/cmd/analyze.py new file mode 100644 index 0000000000..e32dbcd7b1 --- /dev/null +++ b/lib/spack/spack/cmd/analyze.py @@ -0,0 +1,118 @@ +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +import sys + +import llnl.util.tty as tty + +import spack.analyzers +import spack.build_environment +import spack.cmd +import spack.cmd.common.arguments as arguments +import spack.environment as ev +import spack.fetch_strategy +import spack.monitor +import spack.paths +import spack.report + + +description = "run analyzers on installed packages" +section = "analysis" +level = "long" + + +def setup_parser(subparser): + sp = subparser.add_subparsers(metavar='SUBCOMMAND', dest='analyze_command') + + sp.add_parser('list-analyzers', + description="list available analyzers", + help="show list of analyzers that are available to run.") + + # This adds the monitor group to the subparser + spack.monitor.get_monitor_group(subparser) + + # Run Parser + run_parser = sp.add_parser('run', description="run an analyzer", + help="provide the name of the analyzer to run.") + + run_parser.add_argument( + '--overwrite', action='store_true', + help="re-analyze even if the output file already exists.") + run_parser.add_argument( + '-p', '--path', default=None, + dest='path', + help="write output to a different directory than ~/.spack/analyzers") + run_parser.add_argument( + '-a', '--analyzers', default=None, + dest="analyzers", action="append", + help="add an analyzer (defaults to all available)") + arguments.add_common_arguments(run_parser, ['spec']) + + +def analyze_spec(spec, analyzers=None, outdir=None, monitor=None, overwrite=False): + """ + Do an analysis for a spec, optionally adding monitoring. + + We also allow the user to specify a custom output directory. + analyze_spec(spec, args.analyzers, args.outdir, monitor) + + Args: + spec (Spec): spec object of installed package + analyzers (list): list of analyzer (keys) to run + monitor (monitor.SpackMonitorClient): a monitor client + overwrite (bool): overwrite result if already exists + """ + analyzers = analyzers or list(spack.analyzers.analyzer_types.keys()) + + # Load the build environment from the spec install directory, and send + # the spec to the monitor if it's not known + if monitor: + monitor.load_build_environment(spec) + monitor.new_configuration([spec]) + + for name in analyzers: + + # Instantiate the analyzer with the spec and outdir + analyzer = spack.analyzers.get_analyzer(name)(spec, outdir) + + # Run the analyzer to get a json result - results are returned as + # a dictionary with a key corresponding to the analyzer type, so + # we can just update the data + result = analyzer.run() + + # Send the result. We do them separately because: + # 1. each analyzer might have differently organized output + # 2. the size of a result can be large + analyzer.save_result(result, overwrite) + + +def analyze(parser, args, **kwargs): + + # If the user wants to list analyzers, do so and exit + if args.analyze_command == "list-analyzers": + spack.analyzers.list_all() + sys.exit(0) + + # handle active environment, if any + env = ev.get_env(args, 'analyze') + + # Get an disambiguate spec (we should only have one) + specs = spack.cmd.parse_specs(args.spec) + if not specs: + tty.die("You must provide one or more specs to analyze.") + spec = spack.cmd.disambiguate_spec(specs[0], env) + + # The user wants to monitor builds using github.com/spack/spack-monitor + # It is instantianted once here, and then available at spack.monitor.cli + monitor = None + if args.use_monitor: + monitor = spack.monitor.get_client( + host=args.monitor_host, + prefix=args.monitor_prefix, + disable_auth=args.monitor_disable_auth, + ) + + # Run the analysis + analyze_spec(spec, args.analyzers, args.path, monitor, args.overwrite) diff --git a/lib/spack/spack/cmd/arch.py b/lib/spack/spack/cmd/arch.py index 9751599b10..7435fb1246 100644 --- a/lib/spack/spack/cmd/arch.py +++ b/lib/spack/spack/cmd/arch.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/blame.py b/lib/spack/spack/cmd/blame.py index b806058aec..f61ed03b4a 100644 --- a/lib/spack/spack/cmd/blame.py +++ b/lib/spack/spack/cmd/blame.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/build_env.py b/lib/spack/spack/cmd/build_env.py index ef8b1f6e6b..fc9f95f5ba 100644 --- a/lib/spack/spack/cmd/build_env.py +++ b/lib/spack/spack/cmd/build_env.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/buildcache.py b/lib/spack/spack/cmd/buildcache.py index 025de653c5..f5cb94e7c7 100644 --- a/lib/spack/spack/cmd/buildcache.py +++ b/lib/spack/spack/cmd/buildcache.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -769,19 +769,14 @@ def buildcache_copy(args): shutil.copyfile(cdashid_src_path, cdashid_dest_path) -def buildcache_update_index(args): - """Update a buildcache index.""" - outdir = '.' - if args.mirror_url: - outdir = args.mirror_url - - mirror = spack.mirror.MirrorCollection().lookup(outdir) +def update_index(mirror_url, update_keys=False): + mirror = spack.mirror.MirrorCollection().lookup(mirror_url) outdir = url_util.format(mirror.push_url) bindist.generate_package_index( url_util.join(outdir, bindist.build_cache_relative_path())) - if args.keys: + if update_keys: keys_url = url_util.join(outdir, bindist.build_cache_relative_path(), bindist.build_cache_keys_relative_path()) @@ -789,6 +784,15 @@ def buildcache_update_index(args): bindist.generate_key_index(keys_url) +def buildcache_update_index(args): + """Update a buildcache index.""" + outdir = '.' + if args.mirror_url: + outdir = args.mirror_url + + update_index(outdir, update_keys=args.keys) + + def buildcache(parser, args): if args.func: args.func(args) diff --git a/lib/spack/spack/cmd/cd.py b/lib/spack/spack/cmd/cd.py index cbbe9db04d..f0f3ba9eb3 100644 --- a/lib/spack/spack/cmd/cd.py +++ b/lib/spack/spack/cmd/cd.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/checksum.py b/lib/spack/spack/cmd/checksum.py index 188deb1149..937b054c27 100644 --- a/lib/spack/spack/cmd/checksum.py +++ b/lib/spack/spack/cmd/checksum.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/ci.py b/lib/spack/spack/cmd/ci.py index 3dd97e53b0..bd42e10238 100644 --- a/lib/spack/spack/cmd/ci.py +++ b/lib/spack/spack/cmd/ci.py @@ -1,4 +1,4 @@ -# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -17,6 +17,7 @@ import spack.cmd.buildcache as buildcache import spack.environment as ev import spack.hash_types as ht import spack.util.executable as exe +import spack.util.url as url_util description = "manage continuous integration pipelines" @@ -54,6 +55,26 @@ def setup_parser(subparser): '--dependencies', action='store_true', default=False, help="(Experimental) disable DAG scheduling; use " ' "plain" dependencies.') + prune_group = generate.add_mutually_exclusive_group() + prune_group.add_argument( + '--prune-dag', action='store_true', dest='prune_dag', + default=True, help="""Do not generate jobs for specs already up to +date on the mirror""") + prune_group.add_argument( + '--no-prune-dag', action='store_false', dest='prune_dag', + default=True, help="""Generate jobs for specs already up to date +on the mirror""") + generate.add_argument( + '--check-index-only', action='store_true', dest='index_only', + default=False, help="""Spack always check specs against configured +binary mirrors when generating the pipeline, regardless of whether or not +DAG pruning is enabled. This flag controls whether it might attempt to +fetch remote spec.yaml files directly (ensuring no spec is rebuilt if it is +present on the mirror), or whether it should reduce pipeline generation time +by assuming all remote buildcache indices are up to date and only use those +to determine whether a given spec is up to date on mirrors. In the latter +case, specs might be needlessly rebuilt if remote buildcache indices are out +of date.""") generate.set_defaults(func=ci_generate) # Check a spec against mirror. Rebuild, create buildcache and push to @@ -61,6 +82,11 @@ def setup_parser(subparser): rebuild = subparsers.add_parser('rebuild', help=ci_rebuild.__doc__) rebuild.set_defaults(func=ci_rebuild) + # Rebuild the buildcache index associated with the mirror in the + # active, gitlab-enabled environment. + index = subparsers.add_parser('rebuild-index', help=ci_reindex.__doc__) + index.set_defaults(func=ci_reindex) + def ci_generate(args): """Generate jobs file from a spack environment file containing CI info. @@ -75,6 +101,8 @@ def ci_generate(args): copy_yaml_to = args.copy_to run_optimizer = args.optimize use_dependencies = args.dependencies + prune_dag = args.prune_dag + index_only = args.index_only if not output_file: output_file = os.path.abspath(".gitlab-ci.yml") @@ -86,7 +114,8 @@ def ci_generate(args): # Generate the jobs spack_ci.generate_gitlab_ci_yaml( - env, True, output_file, run_optimizer=run_optimizer, + env, True, output_file, prune_dag=prune_dag, + check_index_only=index_only, run_optimizer=run_optimizer, use_dependencies=use_dependencies) if copy_yaml_to: @@ -131,6 +160,7 @@ def ci_rebuild(args): # SPACK_SIGNING_KEY ci_artifact_dir = get_env_var('CI_PROJECT_DIR') + ci_pipeline_id = get_env_var('CI_PIPELINE_ID') signing_key = get_env_var('SPACK_SIGNING_KEY') root_spec = get_env_var('SPACK_ROOT_SPEC') job_spec_pkg_name = get_env_var('SPACK_JOB_SPEC_PKG_NAME') @@ -191,19 +221,28 @@ def ci_rebuild(args): spack_is_pr_pipeline = True if pr_env_var == 'True' else False + pipeline_mirror_url = None + temp_storage_url_prefix = None + if 'temporary-storage-url-prefix' in gitlab_ci: + temp_storage_url_prefix = gitlab_ci['temporary-storage-url-prefix'] + pipeline_mirror_url = url_util.join( + temp_storage_url_prefix, ci_pipeline_id) + enable_artifacts_mirror = False - artifact_mirror_url = None if 'enable-artifacts-buildcache' in gitlab_ci: enable_artifacts_mirror = gitlab_ci['enable-artifacts-buildcache'] - if enable_artifacts_mirror or spack_is_pr_pipeline: - # If this is a PR pipeline, we will override the setting to - # make sure that artifacts buildcache is enabled. Otherwise - # jobs will not have binary deps available since we do not - # allow pushing binaries to remote mirror during PR pipelines + if (enable_artifacts_mirror or (spack_is_pr_pipeline and + not enable_artifacts_mirror and not temp_storage_url_prefix)): + # If you explicitly enabled the artifacts buildcache feature, or + # if this is a PR pipeline but you did not enable either of the + # per-pipeline temporary storage features, we force the use of + # artifacts buildcache. Otherwise jobs will not have binary + # dependencies from previous stages available since we do not + # allow pushing binaries to the remote mirror during PR pipelines. enable_artifacts_mirror = True - artifact_mirror_url = 'file://' + local_mirror_dir + pipeline_mirror_url = 'file://' + local_mirror_dir mirror_msg = 'artifact buildcache enabled, mirror url: {0}'.format( - artifact_mirror_url) + pipeline_mirror_url) tty.debug(mirror_msg) # Clean out scratch directory from last stage @@ -297,8 +336,8 @@ def ci_rebuild(args): if pr_mirror_url: add_mirror('ci_pr_mirror', pr_mirror_url) - if enable_artifacts_mirror: - add_mirror('ci_artifact_mirror', artifact_mirror_url) + if pipeline_mirror_url: + add_mirror(spack_ci.TEMP_STORAGE_MIRROR_NAME, pipeline_mirror_url) tty.debug('listing spack mirrors:') spack_cmd('mirror', 'list') @@ -306,8 +345,8 @@ def ci_rebuild(args): # Checks all mirrors for a built spec with a matching full hash matches = bindist.get_mirrors_for_spec( - job_spec, force=False, full_hash_match=True, - mirrors_to_check=mirrors_to_check) + job_spec, full_hash_match=True, mirrors_to_check=mirrors_to_check, + index_only=False) if matches: # Got at full hash match on at least one configured mirror. All @@ -374,38 +413,40 @@ def ci_rebuild(args): else: buildcache_mirror_url = remote_mirror_url - try: - spack_ci.push_mirror_contents( - env, job_spec, job_spec_yaml_path, buildcache_mirror_url, - cdash_build_id, sign_binaries) - except Exception as inst: - # If the mirror we're pushing to is on S3 and there's some - # permissions problem, for example, we can't just target - # that exception type here, since users of the - # `spack ci rebuild' may not need or want any dependency - # on boto3. So we use the first non-boto exception type - # in the heirarchy: - # boto3.exceptions.S3UploadFailedError - # boto3.exceptions.Boto3Error - # Exception - # BaseException - # object - err_msg = 'Error msg: {0}'.format(inst) - if 'Access Denied' in err_msg: - tty.msg('Permission problem writing to mirror') - tty.msg(err_msg) - - # Create another copy of that buildcache on "local artifact - # mirror" (only done if artifacts buildcache is enabled) - spack_ci.push_mirror_contents(env, job_spec, job_spec_yaml_path, - artifact_mirror_url, cdash_build_id, - sign_binaries) + # Create buildcache in either the main remote mirror, or in the + # per-PR mirror, if this is a PR pipeline + spack_ci.push_mirror_contents( + env, job_spec, job_spec_yaml_path, buildcache_mirror_url, + cdash_build_id, sign_binaries) + + # Create another copy of that buildcache in the per-pipeline + # temporary storage mirror (this is only done if either artifacts + # buildcache is enabled or a temporary storage url prefix is set) + spack_ci.push_mirror_contents( + env, job_spec, job_spec_yaml_path, pipeline_mirror_url, + cdash_build_id, sign_binaries) # Relate this build to its dependencies on CDash (if enabled) if enable_cdash: spack_ci.relate_cdash_builds( spec_map, cdash_base_url, cdash_build_id, cdash_project, - artifact_mirror_url or pr_mirror_url or remote_mirror_url) + pipeline_mirror_url or pr_mirror_url or remote_mirror_url) + + +def ci_reindex(args): + """Rebuild the buildcache index associated with the mirror in the + active, gitlab-enabled environment. """ + env = ev.get_env(args, 'ci rebuild-index', required=True) + yaml_root = ev.config_dict(env.yaml) + + if 'mirrors' not in yaml_root or len(yaml_root['mirrors'].values()) < 1: + tty.die('spack ci rebuild-index requires an env containing a mirror') + + ci_mirrors = yaml_root['mirrors'] + mirror_urls = [url for url in ci_mirrors.values()] + remote_mirror_url = mirror_urls[0] + + buildcache.update_index(remote_mirror_url, update_keys=True) def ci(parser, args): diff --git a/lib/spack/spack/cmd/clean.py b/lib/spack/spack/cmd/clean.py index 2e1ec1255f..f82198abbe 100644 --- a/lib/spack/spack/cmd/clean.py +++ b/lib/spack/spack/cmd/clean.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/clone.py b/lib/spack/spack/cmd/clone.py index ecf5855dab..0ea9cf7ef1 100644 --- a/lib/spack/spack/cmd/clone.py +++ b/lib/spack/spack/cmd/clone.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/commands.py b/lib/spack/spack/cmd/commands.py index be934e0048..e63e162c3f 100644 --- a/lib/spack/spack/cmd/commands.py +++ b/lib/spack/spack/cmd/commands.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -262,20 +262,7 @@ def _commands(parser, args): if args.header and not os.path.exists(args.header): tty.die("No such file: '%s'" % args.header) - # if we're updating an existing file, only write output if a command - # or the header is newer than the file. if args.update: - if os.path.exists(args.update): - files = [ - spack.cmd.get_module(command).__file__.rstrip('c') # pyc -> py - for command in spack.cmd.all_commands()] - if args.header: - files.append(args.header) - last_update = os.path.getmtime(args.update) - if not any(os.path.getmtime(f) > last_update for f in files): - tty.msg('File is up to date: %s' % args.update) - return - tty.msg('Updating file: %s' % args.update) with open(args.update, 'w') as f: prepend_header(args, f) diff --git a/lib/spack/spack/cmd/common/__init__.py b/lib/spack/spack/cmd/common/__init__.py index 35b802db48..6ff4faba7c 100644 --- a/lib/spack/spack/cmd/common/__init__.py +++ b/lib/spack/spack/cmd/common/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -22,7 +22,7 @@ def shell_init_instructions(cmd, equivalent): shell_specific = "{sh_arg}" in equivalent msg = [ - "`%s` requires spack's shell support." % cmd, + "`%s` requires Spack's shell support." % cmd, "", "To set up shell support, run the command below for your shell.", "", @@ -49,5 +49,12 @@ def shell_init_instructions(cmd, equivalent): else: msg += [" " + equivalent] + msg += [ + "", + "If you have already set up Spack's shell support but still receive", + "this message, please make sure to call Spack via the `spack` command", + "without any path components (such as `bin/spack`).", + ] + msg += [''] tty.error(*msg) diff --git a/lib/spack/spack/cmd/common/arguments.py b/lib/spack/spack/cmd/common/arguments.py index e5c4c0dde8..40a30a2618 100644 --- a/lib/spack/spack/cmd/common/arguments.py +++ b/lib/spack/spack/cmd/common/arguments.py @@ -1,11 +1,10 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) import argparse -import multiprocessing import spack.cmd import spack.config @@ -102,22 +101,10 @@ class SetParallelJobs(argparse.Action): '[expected a positive integer, got "{1}"]' raise ValueError(msg.format(option_string, jobs)) - jobs = min(jobs, multiprocessing.cpu_count()) spack.config.set('config:build_jobs', jobs, scope='command_line') setattr(namespace, 'jobs', jobs) - @property - def default(self): - # This default is coded as a property so that look-up - # of this value is done only on demand - return min(spack.config.get('config:build_jobs', 16), - multiprocessing.cpu_count()) - - @default.setter - def default(self, value): - pass - class DeptypeAction(argparse.Action): """Creates a tuple of valid dependency types from a deptype argument.""" @@ -250,8 +237,8 @@ def very_long(): @arg def tags(): return Args( - '-t', '--tags', action='append', - help='filter a package query by tags') + '-t', '--tag', action='append', dest='tags', metavar='TAG', + help='filter a package query by tag (multiple use allowed)') @arg @@ -267,6 +254,7 @@ def install_status(): '-I', '--install-status', action='store_true', default=False, help='show install status of packages. packages can be: ' 'installed [+], missing and needed by an installed package [-], ' + 'installed in and upstream instance [^], ' 'or not installed (no annotation)') @@ -277,6 +265,13 @@ def no_checksum(): help="do not use checksums to verify downloaded files (unsafe)") +@arg +def deprecated(): + return Args( + '--deprecated', action='store_true', default=False, + help='fetch deprecated versions without warning') + + def add_cdash_args(subparser, add_help): cdash_help = {} if add_help: diff --git a/lib/spack/spack/cmd/common/env_utility.py b/lib/spack/spack/cmd/common/env_utility.py index a16cc3ba0d..b556d3edbc 100644 --- a/lib/spack/spack/cmd/common/env_utility.py +++ b/lib/spack/spack/cmd/common/env_utility.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -53,6 +53,9 @@ def emulate_env_utility(cmd_name, context, args): spec = args.spec[0] cmd = args.spec[1:] + if not spec: + tty.die("spack %s requires a spec." % cmd_name) + specs = spack.cmd.parse_specs(spec, concretize=False) if len(specs) > 1: tty.die("spack %s only takes one spec." % cmd_name) diff --git a/lib/spack/spack/cmd/compiler.py b/lib/spack/spack/cmd/compiler.py index fb9e24ea76..08232022da 100644 --- a/lib/spack/spack/cmd/compiler.py +++ b/lib/spack/spack/cmd/compiler.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/compilers.py b/lib/spack/spack/cmd/compilers.py index c29e2257b7..b30b27233f 100644 --- a/lib/spack/spack/cmd/compilers.py +++ b/lib/spack/spack/cmd/compilers.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/concretize.py b/lib/spack/spack/cmd/concretize.py index d28f7b4a5d..e6a8ecf6f5 100644 --- a/lib/spack/spack/cmd/concretize.py +++ b/lib/spack/spack/cmd/concretize.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -14,11 +14,25 @@ def setup_parser(subparser): subparser.add_argument( '-f', '--force', action='store_true', help="Re-concretize even if already concretized.") + subparser.add_argument( + '--test', default=None, + choices=['root', 'all'], + help="""Concretize with test dependencies. When 'root' is chosen, test +dependencies are only added for the environment's root specs. When 'all' is +chosen, test dependencies are enabled for all packages in the environment.""") def concretize(parser, args): env = ev.get_env(args, 'concretize', required=True) + + if args.test == 'all': + tests = True + elif args.test == 'root': + tests = [spec.name for spec in env.user_specs] + else: + tests = False + with env.write_transaction(): - concretized_specs = env.concretize(force=args.force) + concretized_specs = env.concretize(force=args.force, tests=tests) ev.display_specs(concretized_specs) env.write() diff --git a/lib/spack/spack/cmd/config.py b/lib/spack/spack/cmd/config.py index 2cd914ef89..68ce8362fd 100644 --- a/lib/spack/spack/cmd/config.py +++ b/lib/spack/spack/cmd/config.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -17,6 +17,8 @@ import spack.environment as ev import spack.schema.packages import spack.util.spack_yaml as syaml from spack.util.editor import editor +import spack.store +import spack.repo description = "get and set configuration options" section = "config" @@ -73,6 +75,16 @@ def setup_parser(subparser): help="file from which to set all config values" ) + prefer_upstream_parser = sp.add_parser( + 'prefer-upstream', + help='set package preferences from upstream') + + prefer_upstream_parser.add_argument( + '--local', action='store_true', default=False, + help="Set packages preferences based on local installs, rather " + "than upstream." + ) + remove_parser = sp.add_parser('remove', aliases=['rm'], help='remove configuration parameters') remove_parser.add_argument( @@ -188,71 +200,11 @@ def config_add(args): scope, section = _get_scope_and_section(args) - # Updates from file if args.file: - # Get file as config dict - data = spack.config.read_config_file(args.file) - if any(k in data for k in spack.schema.env.keys): - data = ev.config_dict(data) - - # update all sections from config dict - # We have to iterate on keys to keep overrides from the file - for section in data.keys(): - if section in spack.config.section_schemas.keys(): - # Special handling for compiler scope difference - # Has to be handled after we choose a section - if scope is None: - scope = spack.config.default_modify_scope(section) - - value = data[section] - existing = spack.config.get(section, scope=scope) - new = spack.config.merge_yaml(existing, value) - - spack.config.set(section, new, scope) + spack.config.add_from_file(args.file, scope=scope) if args.path: - components = spack.config.process_config_path(args.path) - - has_existing_value = True - path = '' - override = False - for idx, name in enumerate(components[:-1]): - # First handle double colons in constructing path - colon = '::' if override else ':' if path else '' - path += colon + name - if getattr(name, 'override', False): - override = True - else: - override = False - - # Test whether there is an existing value at this level - existing = spack.config.get(path, scope=scope) - - if existing is None: - has_existing_value = False - # We've nested further than existing config, so we need the - # type information for validation to know how to handle bare - # values appended to lists. - existing = spack.config.get_valid_type(path) - - # construct value from this point down - value = syaml.load_config(components[-1]) - for component in reversed(components[idx + 1:-1]): - value = {component: value} - break - - if has_existing_value: - path, _, value = args.path.rpartition(':') - value = syaml.load_config(value) - existing = spack.config.get(path, scope=scope) - - # append values to lists - if isinstance(existing, list) and not isinstance(value, list): - value = [value] - - # merge value into existing - new = spack.config.merge_yaml(existing, value) - spack.config.set(path, new, scope) + spack.config.add(args.path, scope=scope) def config_remove(args): @@ -431,6 +383,79 @@ def config_revert(args): tty.msg(msg.format(cfg_file)) +def config_prefer_upstream(args): + """Generate a packages config based on the configuration of all upstream + installs.""" + + scope = args.scope + if scope is None: + scope = spack.config.default_modify_scope('packages') + + all_specs = set(spack.store.db.query(installed=True)) + local_specs = set(spack.store.db.query_local(installed=True)) + pref_specs = local_specs if args.local else all_specs - local_specs + + conflicting_variants = set() + + pkgs = {} + for spec in pref_specs: + # Collect all the upstream compilers and versions for this package. + pkg = pkgs.get(spec.name, { + 'version': [], + 'compiler': [], + }) + pkgs[spec.name] = pkg + + # We have no existing variant if this is our first added version. + existing_variants = pkg.get('variants', + None if not pkg['version'] else '') + + version = spec.version.string + if version not in pkg['version']: + pkg['version'].append(version) + + compiler = str(spec.compiler) + if compiler not in pkg['compiler']: + pkg['compiler'].append(compiler) + + # Get and list all the variants that differ from the default. + variants = [] + for var_name, variant in spec.variants.items(): + if (var_name in ['patches'] + or var_name not in spec.package.variants): + continue + + if variant.value != spec.package.variants[var_name].default: + variants.append(str(variant)) + variants.sort() + variants = ' '.join(variants) + + if spec.name not in conflicting_variants: + # Only specify the variants if there's a single variant + # set across all versions/compilers. + if existing_variants is not None and existing_variants != variants: + conflicting_variants.add(spec.name) + pkg.pop('variants', None) + elif variants: + pkg['variants'] = variants + + if conflicting_variants: + tty.warn( + "The following packages have multiple conflicting upstream " + "specs. You may have to specify, by " + "concretized hash, which spec you want when building " + "packages that depend on them:\n - {0}" + .format("\n - ".join(sorted(conflicting_variants)))) + + # Simply write the config to the specified file. + existing = spack.config.get('packages', scope=scope) + new = spack.config.merge_yaml(existing, pkgs) + spack.config.set('packages', new, scope) + config_file = spack.config.config.get_config_filename(scope, section) + + tty.msg("Updated config at {0}".format(config_file)) + + def config(parser, args): action = { 'get': config_get, @@ -441,6 +466,7 @@ def config(parser, args): 'rm': config_remove, 'remove': config_remove, 'update': config_update, - 'revert': config_revert + 'revert': config_revert, + 'prefer-upstream': config_prefer_upstream, } action[args.config_command](args) diff --git a/lib/spack/spack/cmd/containerize.py b/lib/spack/spack/cmd/containerize.py index cc2c001560..27ef988f69 100644 --- a/lib/spack/spack/cmd/containerize.py +++ b/lib/spack/spack/cmd/containerize.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/create.py b/lib/spack/spack/cmd/create.py index a31b9537b6..85fea9d98a 100644 --- a/lib/spack/spack/cmd/create.py +++ b/lib/spack/spack/cmd/create.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -28,7 +28,7 @@ level = "short" package_template = '''\ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -117,7 +117,7 @@ class PackageTemplate(BundlePackageTemplate): make() make('install')""" - url_line = """ url = \"{url}\"""" + url_line = ' url = "{url}"' def __init__(self, name, url, versions): super(PackageTemplate, self).__init__(name, versions) @@ -270,14 +270,47 @@ class PythonPackageTemplate(PackageTemplate): args = [] return args""" - def __init__(self, name, *args, **kwargs): + def __init__(self, name, url, *args, **kwargs): # If the user provided `--name py-numpy`, don't rename it py-py-numpy if not name.startswith('py-'): # Make it more obvious that we are renaming the package tty.msg("Changing package name from {0} to py-{0}".format(name)) name = 'py-{0}'.format(name) - super(PythonPackageTemplate, self).__init__(name, *args, **kwargs) + # Simple PyPI URLs: + # https://<hostname>/packages/<type>/<first character of project>/<project>/<download file> + # e.g. https://pypi.io/packages/source/n/numpy/numpy-1.19.4.zip + # e.g. https://www.pypi.io/packages/source/n/numpy/numpy-1.19.4.zip + # e.g. https://pypi.org/packages/source/n/numpy/numpy-1.19.4.zip + # e.g. https://pypi.python.org/packages/source/n/numpy/numpy-1.19.4.zip + # e.g. https://files.pythonhosted.org/packages/source/n/numpy/numpy-1.19.4.zip + + # PyPI URLs containing hash: + # https://<hostname>/packages/<two character hash>/<two character hash>/<longer hash>/<download file> + # e.g. https://pypi.io/packages/c5/63/a48648ebc57711348420670bb074998f79828291f68aebfff1642be212ec/numpy-1.19.4.zip + # e.g. https://files.pythonhosted.org/packages/c5/63/a48648ebc57711348420670bb074998f79828291f68aebfff1642be212ec/numpy-1.19.4.zip + # e.g. https://files.pythonhosted.org/packages/c5/63/a48648ebc57711348420670bb074998f79828291f68aebfff1642be212ec/numpy-1.19.4.zip#sha256=141ec3a3300ab89c7f2b0775289954d193cc8edb621ea05f99db9cb181530512 + + # PyPI URLs for wheels are too complicated, ignore them for now + + match = re.search( + r'(?:pypi|pythonhosted)[^/]+/packages' + '/([^/#]+)' * 4, + url + ) + if match: + if len(match.group(2)) == 1: + # Simple PyPI URL + url = '/'.join(match.group(3, 4)) + else: + # PyPI URL containing hash + # Project name doesn't necessarily match download name, but it + # usually does, so this is the best we can do + project = parse_name(url) + url = '/'.join([project, match.group(4)]) + + self.url_line = ' pypi = "{url}"' + + super(PythonPackageTemplate, self).__init__(name, url, *args, **kwargs) class RPackageTemplate(PackageTemplate): @@ -289,20 +322,40 @@ class RPackageTemplate(PackageTemplate): # depends_on('r-foo', type=('build', 'run'))""" body_def = """\ - def configure_args(self, spec, prefix): + def configure_args(self): # FIXME: Add arguments to pass to install via --configure-args # FIXME: If not needed delete this function args = [] return args""" - def __init__(self, name, *args, **kwargs): + def __init__(self, name, url, *args, **kwargs): # If the user provided `--name r-rcpp`, don't rename it r-r-rcpp if not name.startswith('r-'): # Make it more obvious that we are renaming the package tty.msg("Changing package name from {0} to r-{0}".format(name)) name = 'r-{0}'.format(name) - super(RPackageTemplate, self).__init__(name, *args, **kwargs) + r_name = parse_name(url) + + cran = re.search( + r'(?:r-project|rstudio)[^/]+/src' + '/([^/]+)' * 2, + url + ) + + if cran: + url = r_name + self.url_line = ' cran = "{url}"' + + bioc = re.search( + r'(?:bioconductor)[^/]+/packages' + '/([^/]+)' * 5, + url + ) + + if bioc: + self.url_line = ' url = "{0}"\n'\ + ' bioc = "{1}"'.format(url, r_name) + + super(RPackageTemplate, self).__init__(name, url, *args, **kwargs) class PerlmakePackageTemplate(PackageTemplate): @@ -545,7 +598,8 @@ class BuildSystemGuesser: ] # Peek inside the compressed file. - if stage.archive_file.endswith('.zip'): + if (stage.archive_file.endswith('.zip') or + '.zip#' in stage.archive_file): try: unzip = which('unzip') output = unzip('-lq', stage.archive_file, output=str) diff --git a/lib/spack/spack/cmd/deactivate.py b/lib/spack/spack/cmd/deactivate.py index 3c72531a9c..0afa4af82b 100644 --- a/lib/spack/spack/cmd/deactivate.py +++ b/lib/spack/spack/cmd/deactivate.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/debug.py b/lib/spack/spack/cmd/debug.py index 074e95209a..8580416cba 100644 --- a/lib/spack/spack/cmd/debug.py +++ b/lib/spack/spack/cmd/debug.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -15,6 +15,7 @@ import llnl.util.tty as tty from llnl.util.filesystem import working_dir import spack.architecture as architecture +import spack.config import spack.paths from spack.main import get_version from spack.util.executable import which @@ -89,6 +90,7 @@ def report(args): print('* **Python:**', platform.python_version()) print('* **Platform:**', architecture.Arch( architecture.platform(), 'frontend', 'frontend')) + print('* **Concretizer:**', spack.config.get('config:concretizer')) def debug(parser, args): diff --git a/lib/spack/spack/cmd/dependencies.py b/lib/spack/spack/cmd/dependencies.py index bbccbe23ee..2f923fece0 100644 --- a/lib/spack/spack/cmd/dependencies.py +++ b/lib/spack/spack/cmd/dependencies.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/dependents.py b/lib/spack/spack/cmd/dependents.py index 5563701801..242a07da36 100644 --- a/lib/spack/spack/cmd/dependents.py +++ b/lib/spack/spack/cmd/dependents.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/deprecate.py b/lib/spack/spack/cmd/deprecate.py index 6172033fa7..2928a5b264 100644 --- a/lib/spack/spack/cmd/deprecate.py +++ b/lib/spack/spack/cmd/deprecate.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/dev_build.py b/lib/spack/spack/cmd/dev_build.py index 429ba07246..2e7c51da42 100644 --- a/lib/spack/spack/cmd/dev_build.py +++ b/lib/spack/spack/cmd/dev_build.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -26,7 +26,7 @@ def setup_parser(subparser): subparser.add_argument( '-i', '--ignore-dependencies', action='store_true', dest='ignore_deps', help="don't try to install dependencies of requested packages") - arguments.add_common_arguments(subparser, ['no_checksum']) + arguments.add_common_arguments(subparser, ['no_checksum', 'deprecated']) subparser.add_argument( '--keep-prefix', action='store_true', help="do not remove the install prefix if installation fails") @@ -98,6 +98,9 @@ def dev_build(self, args): if args.no_checksum: spack.config.set('config:checksum', False, scope='command_line') + if args.deprecated: + spack.config.set('config:deprecated', True, scope='command_line') + tests = False if args.test == 'all': tests = True diff --git a/lib/spack/spack/cmd/develop.py b/lib/spack/spack/cmd/develop.py index 957060536d..d2dd8d0049 100644 --- a/lib/spack/spack/cmd/develop.py +++ b/lib/spack/spack/cmd/develop.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/docs.py b/lib/spack/spack/cmd/docs.py index a7f2a641ee..ac3a91721e 100644 --- a/lib/spack/spack/cmd/docs.py +++ b/lib/spack/spack/cmd/docs.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/edit.py b/lib/spack/spack/cmd/edit.py index e01d50dd03..a277ed519e 100644 --- a/lib/spack/spack/cmd/edit.py +++ b/lib/spack/spack/cmd/edit.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/env.py b/lib/spack/spack/cmd/env.py index 4699efd26b..11b422a9bd 100644 --- a/lib/spack/spack/cmd/env.py +++ b/lib/spack/spack/cmd/env.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -157,6 +157,10 @@ def env_create_setup_parser(subparser): subparser.add_argument( '-d', '--dir', action='store_true', help='create an environment in a specific directory') + subparser.add_argument( + '--keep-relative', action='store_true', + help='copy relative develop paths verbatim into the new environment' + ' when initializing from envfile') view_opts = subparser.add_mutually_exclusive_group() view_opts.add_argument( '--without-view', action='store_true', @@ -184,13 +188,14 @@ def env_create(args): if args.envfile: with open(args.envfile) as f: _env_create(args.create_env, f, args.dir, - with_view=with_view) + with_view=with_view, keep_relative=args.keep_relative) else: _env_create(args.create_env, None, args.dir, with_view=with_view) -def _env_create(name_or_path, init_file=None, dir=False, with_view=None): +def _env_create(name_or_path, init_file=None, dir=False, with_view=None, + keep_relative=False): """Create a new environment, with an optional yaml description. Arguments: @@ -199,15 +204,18 @@ def _env_create(name_or_path, init_file=None, dir=False, with_view=None): spack.yaml or spack.lock dir (bool): if True, create an environment in a directory instead of a named environment + keep_relative (bool): if True, develop paths are copied verbatim into + the new environment file, otherwise they may be made absolute if the + new environment is in a different location """ if dir: - env = ev.Environment(name_or_path, init_file, with_view) + env = ev.Environment(name_or_path, init_file, with_view, keep_relative) env.write() tty.msg("Created environment in %s" % env.path) tty.msg("You can activate this environment with:") tty.msg(" spack env activate %s" % env.path) else: - env = ev.create(name_or_path, init_file, with_view) + env = ev.create(name_or_path, init_file, with_view, keep_relative) env.write() tty.msg("Created environment '%s' in %s" % (name_or_path, env.path)) tty.msg("You can activate this environment with:") diff --git a/lib/spack/spack/cmd/extensions.py b/lib/spack/spack/cmd/extensions.py index b4fb7faa1f..a57aa031ff 100644 --- a/lib/spack/spack/cmd/extensions.py +++ b/lib/spack/spack/cmd/extensions.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/external.py b/lib/spack/spack/cmd/external.py index 52af72ed8f..c7d045e66d 100644 --- a/lib/spack/spack/cmd/external.py +++ b/lib/spack/spack/cmd/external.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -16,6 +16,7 @@ import llnl.util.tty.colify as colify import six import spack import spack.cmd +import spack.cmd.common.arguments import spack.error import spack.util.environment import spack.util.spack_yaml as syaml @@ -42,6 +43,7 @@ def setup_parser(subparser): '--scope', choices=scopes, metavar=scopes_metavar, default=spack.config.default_modify_scope('packages'), help="configuration scope to modify") + spack.cmd.common.arguments.add_common_arguments(find_parser, ['tags']) find_parser.add_argument('packages', nargs=argparse.REMAINDER) sp.add_parser( @@ -148,9 +150,28 @@ def _spec_is_valid(spec): def external_find(args): + # Construct the list of possible packages to be detected + packages_to_check = [] + + # Add the packages that have been required explicitly if args.packages: packages_to_check = list(spack.repo.get(pkg) for pkg in args.packages) - else: + if args.tags: + allowed = set(spack.repo.path.packages_with_tags(*args.tags)) + packages_to_check = [x for x in packages_to_check if x in allowed] + + if args.tags and not packages_to_check: + # If we arrived here we didn't have any explicit package passed + # as argument, which means to search all packages. + # Since tags are cached it's much faster to construct what we need + # to search directly, rather than filtering after the fact + packages_to_check = [ + spack.repo.get(pkg) for pkg in + spack.repo.path.packages_with_tags(*args.tags) + ] + + # If the list of packages is empty, search for every possible package + if not args.tags and not packages_to_check: packages_to_check = spack.repo.path.all_packages() pkg_to_entries = _get_external_packages(packages_to_check) diff --git a/lib/spack/spack/cmd/fetch.py b/lib/spack/spack/cmd/fetch.py index d2cd53c69f..bb4f535180 100644 --- a/lib/spack/spack/cmd/fetch.py +++ b/lib/spack/spack/cmd/fetch.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -17,7 +17,7 @@ level = "long" def setup_parser(subparser): - arguments.add_common_arguments(subparser, ['no_checksum']) + arguments.add_common_arguments(subparser, ['no_checksum', 'deprecated']) subparser.add_argument( "-m", "--missing", @@ -64,6 +64,9 @@ def fetch(parser, args): if args.no_checksum: spack.config.set("config:checksum", False, scope="command_line") + if args.deprecated: + spack.config.set('config:deprecated', True, scope='command_line') + for spec in specs: if args.missing or args.dependencies: for s in spec.traverse(): diff --git a/lib/spack/spack/cmd/find.py b/lib/spack/spack/cmd/find.py index f770a02b25..21d4d01ce8 100644 --- a/lib/spack/spack/cmd/find.py +++ b/lib/spack/spack/cmd/find.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/flake8.py b/lib/spack/spack/cmd/flake8.py index cdfce2cab2..7da059ba69 100644 --- a/lib/spack/spack/cmd/flake8.py +++ b/lib/spack/spack/cmd/flake8.py @@ -1,321 +1,26 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) from __future__ import print_function -import re -import os -import sys -import shutil -import tempfile -import argparse +import llnl.util.tty as tty -from llnl.util.filesystem import working_dir, mkdirp +import spack.cmd.style -import spack.paths -from spack.util.executable import which - -description = "runs source code style checks on Spack. requires flake8" -section = "developer" -level = "long" - - -def is_package(f): - """Whether flake8 should consider a file as a core file or a package. - - We run flake8 with different exceptions for the core and for - packages, since we allow `from spack import *` and poking globals - into packages. - """ - return f.startswith('var/spack/repos/') or 'docs/tutorial/examples' in f - - -#: List of directories to exclude from checks. -exclude_directories = [spack.paths.external_path] - -#: max line length we're enforcing (note: this duplicates what's in .flake8) -max_line_length = 79 - -#: This is a dict that maps: -#: filename pattern -> -#: flake8 exemption code -> -#: list of patterns, for which matching lines should have codes applied. -#: -#: For each file, if the filename pattern matches, we'll add per-line -#: exemptions if any patterns in the sub-dict match. -pattern_exemptions = { - # exemptions applied only to package.py files. - r'package.py$': { - # Allow 'from spack import *' in packages, but no other wildcards - 'F403': [ - r'^from spack import \*$' - ], - # Exempt lines with urls and descriptions from overlong line errors. - 'E501': [ - r'^\s*homepage\s*=', - r'^\s*url\s*=', - r'^\s*git\s*=', - r'^\s*svn\s*=', - r'^\s*hg\s*=', - r'^\s*list_url\s*=', - r'^\s*version\(', - r'^\s*variant\(', - r'^\s*provides\(', - r'^\s*extends\(', - r'^\s*depends_on\(', - r'^\s*conflicts\(', - r'^\s*resource\(', - r'^\s*patch\(', - ], - # Exempt '@when' decorated functions from redefinition errors. - 'F811': [ - r'^\s*@when\(.*\)', - ], - }, - - # exemptions applied to all files. - r'.py$': { - 'E501': [ - r'(https?|ftp|file)\:', # URLs - r'([\'"])[0-9a-fA-F]{32,}\1', # long hex checksums - ] - }, -} - -# compile all regular expressions. -pattern_exemptions = dict( - (re.compile(file_pattern), - dict((code, [re.compile(p) for p in patterns]) - for code, patterns in error_dict.items())) - for file_pattern, error_dict in pattern_exemptions.items()) - - -def changed_files(base=None, untracked=True, all_files=False): - """Get list of changed files in the Spack repository.""" - - git = which('git', required=True) - - if base is None: - base = os.environ.get('TRAVIS_BRANCH', 'develop') - - range = "{0}...".format(base) - - git_args = [ - # Add changed files committed since branching off of develop - ['diff', '--name-only', '--diff-filter=ACMR', range], - # Add changed files that have been staged but not yet committed - ['diff', '--name-only', '--diff-filter=ACMR', '--cached'], - # Add changed files that are unstaged - ['diff', '--name-only', '--diff-filter=ACMR'], - ] - - # Add new files that are untracked - if untracked: - git_args.append(['ls-files', '--exclude-standard', '--other']) - - # add everything if the user asked for it - if all_files: - git_args.append(['ls-files', '--exclude-standard']) - - excludes = [os.path.realpath(f) for f in exclude_directories] - changed = set() - - for arg_list in git_args: - files = git(*arg_list, output=str).split('\n') - - for f in files: - # Ignore non-Python files - if not (f.endswith('.py') or f == 'bin/spack'): - continue - - # Ignore files in the exclude locations - if any(os.path.realpath(f).startswith(e) for e in excludes): - continue - - changed.add(f) - - return sorted(changed) - - -def add_pattern_exemptions(line, codes): - """Add a flake8 exemption to a line.""" - if line.startswith('#'): - return line - - line = line.rstrip('\n') - - # Line is already ignored - if line.endswith('# noqa'): - return line + '\n' - - orig_len = len(line) - codes = set(codes) - - # don't add E501 unless the line is actually too long, as it can mask - # other errors like trailing whitespace - if orig_len <= max_line_length and "E501" in codes: - codes.remove("E501") - if not codes: - return line + "\n" - - exemptions = ','.join(sorted(codes)) - - # append exemption to line - if '# noqa: ' in line: - line += ',{0}'.format(exemptions) - elif line: # ignore noqa on empty lines - line += ' # noqa: {0}'.format(exemptions) - - # if THIS made the line too long, add an exemption for that - if len(line) > max_line_length and orig_len <= max_line_length: - line += ',E501' - - return line + '\n' - - -def filter_file(source, dest, output=False): - """Filter a single file through all the patterns in pattern_exemptions.""" - - # Prior to Python 3.8, `noqa: F811` needed to be placed on the `@when` line - # Starting with Python 3.8, it must be placed on the `def` line - # https://gitlab.com/pycqa/flake8/issues/583 - ignore_f811_on_previous_line = False - - with open(source) as infile: - parent = os.path.dirname(dest) - mkdirp(parent) - - with open(dest, 'w') as outfile: - for line in infile: - line_errors = [] - - # pattern exemptions - for file_pattern, errors in pattern_exemptions.items(): - if not file_pattern.search(source): - continue - - for code, patterns in errors.items(): - for pattern in patterns: - if pattern.search(line): - line_errors.append(code) - break - - if 'F811' in line_errors: - ignore_f811_on_previous_line = True - elif ignore_f811_on_previous_line: - line_errors.append('F811') - ignore_f811_on_previous_line = False - - if line_errors: - line = add_pattern_exemptions(line, line_errors) - - outfile.write(line) - if output: - sys.stdout.write(line) +description = "alias for spack style (deprecated)" +section = spack.cmd.style.section +level = spack.cmd.style.level def setup_parser(subparser): - subparser.add_argument( - '-b', '--base', action='store', default=None, - help="select base branch for collecting list of modified files") - subparser.add_argument( - '-k', '--keep-temp', action='store_true', - help="do not delete temporary directory where flake8 runs. " - "use for debugging, to see filtered files") - subparser.add_argument( - '-a', '--all', action='store_true', - help="check all files, not just changed files") - subparser.add_argument( - '-o', '--output', action='store_true', - help="send filtered files to stdout as well as temp files") - subparser.add_argument( - '-r', '--root-relative', action='store_true', default=False, - help="print root-relative paths (default: cwd-relative)") - subparser.add_argument( - '-U', '--no-untracked', dest='untracked', action='store_false', - default=True, help="exclude untracked files from checks") - subparser.add_argument( - 'files', nargs=argparse.REMAINDER, help="specific files to check") + spack.cmd.style.setup_parser(subparser) def flake8(parser, args): - flake8 = which('flake8', required=True) - - temp = tempfile.mkdtemp() - try: - file_list = args.files - if file_list: - def prefix_relative(path): - return os.path.relpath( - os.path.abspath(os.path.realpath(path)), - spack.paths.prefix) - - file_list = [prefix_relative(p) for p in file_list] - - with working_dir(spack.paths.prefix): - if not file_list: - file_list = changed_files(args.base, args.untracked, args.all) - - print('=======================================================') - print('flake8: running flake8 code checks on spack.') - print() - print('Modified files:') - for filename in file_list: - print(' {0}'.format(filename.strip())) - print('=======================================================') - - # filter files into a temporary directory with exemptions added. - for filename in file_list: - src_path = os.path.join(spack.paths.prefix, filename) - dest_path = os.path.join(temp, filename) - filter_file(src_path, dest_path, args.output) - - # run flake8 on the temporary tree, once for core, once for pkgs - package_file_list = [f for f in file_list if is_package(f)] - file_list = [f for f in file_list if not is_package(f)] - - returncode = 0 - with working_dir(temp): - output = '' - if file_list: - output += flake8( - '--format', 'pylint', - '--config=%s' % os.path.join(spack.paths.prefix, - '.flake8'), - *file_list, fail_on_error=False, output=str) - returncode |= flake8.returncode - if package_file_list: - output += flake8( - '--format', 'pylint', - '--config=%s' % os.path.join(spack.paths.prefix, - '.flake8_packages'), - *package_file_list, fail_on_error=False, output=str) - returncode |= flake8.returncode - - if args.root_relative: - # print results relative to repo root. - print(output) - else: - # print results relative to current working directory - def cwd_relative(path): - return '{0}: ['.format(os.path.relpath( - os.path.join( - spack.paths.prefix, path.group(1)), os.getcwd())) - - for line in output.split('\n'): - print(re.sub(r'^(.*): \[', cwd_relative, line)) - - if returncode != 0: - print('Flake8 found errors.') - sys.exit(1) - else: - print('Flake8 checks were clean.') - - finally: - if args.keep_temp: - print('Temporary files are in: ', temp) - else: - shutil.rmtree(temp, ignore_errors=True) + tty.warn( + "spack flake8 is deprecated", "please use `spack style` to run style checks" + ) + return spack.cmd.style.style(parser, args) diff --git a/lib/spack/spack/cmd/gc.py b/lib/spack/spack/cmd/gc.py index ca9d88169a..f85f27bfca 100644 --- a/lib/spack/spack/cmd/gc.py +++ b/lib/spack/spack/cmd/gc.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/gpg.py b/lib/spack/spack/cmd/gpg.py index 59fba4a4df..e34c2c4c84 100644 --- a/lib/spack/spack/cmd/gpg.py +++ b/lib/spack/spack/cmd/gpg.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/graph.py b/lib/spack/spack/cmd/graph.py index d0fbf8e6c6..3d962a52f8 100644 --- a/lib/spack/spack/cmd/graph.py +++ b/lib/spack/spack/cmd/graph.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/help.py b/lib/spack/spack/cmd/help.py index d730d4f0fe..bc289c0363 100644 --- a/lib/spack/spack/cmd/help.py +++ b/lib/spack/spack/cmd/help.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/info.py b/lib/spack/spack/cmd/info.py index fa674317ab..6b316f3cc4 100644 --- a/lib/spack/spack/cmd/info.py +++ b/lib/spack/spack/cmd/info.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -111,7 +111,7 @@ class VariantFormatter(object): yield ' None' else: yield ' ' + self.fmt % self.headers - underline = tuple([l * "=" for l in self.column_widths]) + underline = tuple([w * "=" for w in self.column_widths]) yield ' ' + self.fmt % underline yield '' for k, v in sorted(self.variants.items()): @@ -189,10 +189,11 @@ def print_text_info(pkg): color.cprint(section_title('Safe versions: ')) for v in reversed(sorted(pkg.versions)): - if pkg.has_code: - url = fs.for_package_version(pkg, v) - line = version(' {0}'.format(pad(v))) + color.cescape(url) - color.cprint(line) + if not pkg.versions[v].get('deprecated', False): + if pkg.has_code: + url = fs.for_package_version(pkg, v) + line = version(' {0}'.format(pad(v))) + color.cescape(url) + color.cprint(line) color.cprint('') color.cprint(section_title('Variants:')) diff --git a/lib/spack/spack/cmd/install.py b/lib/spack/spack/cmd/install.py index 8f3902d71c..587f039656 100644 --- a/lib/spack/spack/cmd/install.py +++ b/lib/spack/spack/cmd/install.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -17,6 +17,7 @@ import spack.cmd import spack.cmd.common.arguments as arguments import spack.environment as ev import spack.fetch_strategy +import spack.monitor import spack.paths import spack.report from spack.error import SpackError @@ -43,6 +44,7 @@ def update_kwargs_from_args(args, kwargs): 'dirty': args.dirty, 'use_cache': args.use_cache, 'cache_only': args.cache_only, + 'include_build_deps': args.include_build_deps, 'explicit': True, # Always true for install command 'stop_at': args.until, 'unsigned': args.unsigned, @@ -105,6 +107,13 @@ the dependencies""" '--cache-only', action='store_true', dest='cache_only', default=False, help="only install package from binary mirrors") + monitor_group = spack.monitor.get_monitor_group(subparser) # noqa + + subparser.add_argument( + '--include-build-deps', action='store_true', dest='include_build_deps', + default=False, help="""include build deps when installing from cache, +which is useful for CI pipeline troubleshooting""") + subparser.add_argument( '--no-check-signature', action='store_true', dest='unsigned', default=False, @@ -120,7 +129,7 @@ remote spec matches that of the local spec""") subparser.add_argument( '--source', action='store_true', dest='install_source', help="install source files in prefix") - arguments.add_common_arguments(subparser, ['no_checksum']) + arguments.add_common_arguments(subparser, ['no_checksum', 'deprecated']) subparser.add_argument( '-v', '--verbose', action='store_true', help="display verbose build output while installing") @@ -131,6 +140,10 @@ remote spec matches that of the local spec""") '--only-concrete', action='store_true', default=False, help='(with environment) only install already concretized specs') subparser.add_argument( + '--no-add', action='store_true', default=False, + help="""(with environment) only install specs provided as argument +if they are already in the concretized environment""") + subparser.add_argument( '-f', '--file', action='append', default=[], dest='specfiles', metavar='SPEC_YAML_FILE', help="install from file. Read specs to install from .yaml files") @@ -177,7 +190,7 @@ def default_log_file(spec): """ fmt = 'test-{x.name}-{x.version}-{hash}.xml' basename = fmt.format(x=spec, hash=spec.dag_hash()) - dirname = fs.os.path.join(spack.paths.var_path, 'junit-report') + dirname = fs.os.path.join(spack.paths.reports_path, 'junit') fs.mkdirp(dirname) return fs.os.path.join(dirname, basename) @@ -196,11 +209,66 @@ def install_specs(cli_args, kwargs, specs): try: if env: + specs_to_install = [] + specs_to_add = [] for abstract, concrete in specs: - with env.write_transaction(): - concrete = env.concretize_and_add(abstract, concrete) - env.write(regenerate_views=False) - env.install_all(cli_args, **kwargs) + # This won't find specs added to the env since last + # concretize, therefore should we consider enforcing + # concretization of the env before allowing to install + # specs? + m_spec = env.matching_spec(abstract) + + # If there is any ambiguity in the above call to matching_spec + # (i.e. if more than one spec in the environment matches), then + # SpackEnvironmentError is rasied, with a message listing the + # the matches. Getting to this point means there were either + # no matches or exactly one match. + + if not m_spec: + tty.debug('{0} matched nothing in the env'.format( + abstract.name)) + # no matches in the env + if cli_args.no_add: + msg = ('You asked to install {0} without adding it ' + + '(--no-add), but no such spec exists in ' + + 'environment').format(abstract.name) + tty.die(msg) + else: + tty.debug('adding {0} as a root'.format(abstract.name)) + specs_to_add.append((abstract, concrete)) + + continue + + tty.debug('exactly one match for {0} in env -> {1}'.format( + m_spec.name, m_spec.dag_hash())) + + if m_spec in env.roots() or cli_args.no_add: + # either the single match is a root spec (and --no-add is + # the default for roots) or --no-add was stated explictly + tty.debug('just install {0}'.format(m_spec.name)) + specs_to_install.append(m_spec) + else: + # the single match is not a root (i.e. it's a dependency), + # and --no-add was not specified, so we'll add it as a + # root before installing + tty.debug('add {0} then install it'.format(m_spec.name)) + specs_to_add.append((abstract, concrete)) + + if specs_to_add: + tty.debug('Adding the following specs as roots:') + for abstract, concrete in specs_to_add: + tty.debug(' {0}'.format(abstract.name)) + with env.write_transaction(): + specs_to_install.append( + env.concretize_and_add(abstract, concrete)) + env.write(regenerate_views=False) + + # Install the validated list of cli specs + if specs_to_install: + tty.debug('Installing the following cli specs:') + for s in specs_to_install: + tty.debug(' {0}'.format(s.name)) + env.install_specs(specs_to_install, args=cli_args, **kwargs) else: installs = [(concrete.package, kwargs) for _, concrete in specs] builder = PackageInstaller(installs) @@ -218,6 +286,7 @@ def install_specs(cli_args, kwargs, specs): def install(parser, args, **kwargs): + if args.help_cdash: parser = argparse.ArgumentParser( formatter_class=argparse.RawDescriptionHelpFormatter, @@ -230,19 +299,42 @@ environment variables: parser.print_help() return + # The user wants to monitor builds using github.com/spack/spack-monitor + if args.use_monitor: + monitor = spack.monitor.get_client( + host=args.monitor_host, + prefix=args.monitor_prefix, + disable_auth=args.monitor_disable_auth, + tags=args.monitor_tags, + ) + reporter = spack.report.collect_info( spack.package.PackageInstaller, '_install_task', args.log_format, args) if args.log_file: reporter.filename = args.log_file + if args.run_tests: + tty.warn("Deprecated option: --run-tests: use --test=all instead") + + def get_tests(specs): + if args.test == 'all' or args.run_tests: + return True + elif args.test == 'root': + return [spec.name for spec in specs] + else: + return False + if not args.spec and not args.specfiles: # if there are no args but an active environment # then install the packages from it. env = ev.get_env(args, 'install') if env: + tests = get_tests(env.user_specs) + kwargs['tests'] = tests + if not args.only_concrete: with env.write_transaction(): - concretized_specs = env.concretize() + concretized_specs = env.concretize(tests=tests) ev.display_specs(concretized_specs) # save view regeneration for later, so that we only do it @@ -282,20 +374,16 @@ environment variables: if args.no_checksum: spack.config.set('config:checksum', False, scope='command_line') + if args.deprecated: + spack.config.set('config:deprecated', True, scope='command_line') + # Parse cli arguments and construct a dictionary # that will be passed to the package installer update_kwargs_from_args(args, kwargs) - if args.run_tests: - tty.warn("Deprecated option: --run-tests: use --test=all instead") - # 1. Abstract specs from cli abstract_specs = spack.cmd.parse_specs(args.spec) - tests = False - if args.test == 'all' or args.run_tests: - tests = True - elif args.test == 'root': - tests = [spec.name for spec in abstract_specs] + tests = get_tests(abstract_specs) kwargs['tests'] = tests try: @@ -362,4 +450,17 @@ environment variables: # overwrite all concrete explicit specs from this build kwargs['overwrite'] = [spec.dag_hash() for spec in specs] + # Update install_args with the monitor args, needed for build task + kwargs.update({ + "monitor_disable_auth": args.monitor_disable_auth, + "monitor_keep_going": args.monitor_keep_going, + "monitor_host": args.monitor_host, + "use_monitor": args.use_monitor, + "monitor_prefix": args.monitor_prefix, + }) + + # If we are using the monitor, we send configs. and create build + # The full_hash is the main package id, the build_hash for others + if args.use_monitor and specs: + monitor.new_configuration(specs) install_specs(args, kwargs, zip(abstract_specs, specs)) diff --git a/lib/spack/spack/cmd/license.py b/lib/spack/spack/cmd/license.py index 84a4a02282..8af0f4fa26 100644 --- a/lib/spack/spack/cmd/license.py +++ b/lib/spack/spack/cmd/license.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -9,6 +9,7 @@ import os import re from collections import defaultdict +import llnl.util.filesystem as fs import llnl.util.tty as tty import spack.paths @@ -36,12 +37,14 @@ licensed_files = [ # all of spack core r'^lib/spack/spack/.*\.py$', r'^lib/spack/spack/.*\.sh$', + r'^lib/spack/spack/.*\.lp$', r'^lib/spack/llnl/.*\.py$', r'^lib/spack/env/cc$', # rst files in documentation r'^lib/spack/docs/(?!command_index|spack|llnl).*\.rst$', r'^lib/spack/docs/.*\.py$', + r'^lib/spack/docs/spack.yaml$', # 2 files in external r'^lib/spack/external/__init__.py$', @@ -51,7 +54,13 @@ licensed_files = [ r'^share/spack/.*\.sh$', r'^share/spack/.*\.bash$', r'^share/spack/.*\.csh$', + r'^share/spack/.*\.fish$', r'^share/spack/qa/run-[^/]*$', + r'^share/spack/bash/spack-completion.in$', + r'^share/spack/templates/misc/coconcretization.pyt$', + + # action workflows + r'^.github/actions/.*\.py$', # all packages r'^var/spack/repos/.*/package.py$' @@ -77,15 +86,15 @@ def _all_spack_files(root=spack.paths.prefix): visited.add(path) -def _licensed_files(root=spack.paths.prefix): - for relpath in _all_spack_files(root): +def _licensed_files(args): + for relpath in _all_spack_files(args.root): if any(regex.match(relpath) for regex in licensed_files): yield relpath def list_files(args): """list files in spack that should have license headers""" - for relpath in sorted(_licensed_files()): + for relpath in sorted(_licensed_files(args)): print(os.path.join(spack.paths.spack_root, relpath)) @@ -93,6 +102,8 @@ def list_files(args): # bool(value) evaluates to True OLD_LICENSE, SPDX_MISMATCH, GENERAL_MISMATCH = range(1, 4) +strict_date = r'Copyright 2013-2021' + class LicenseError(object): def __init__(self): @@ -118,17 +129,15 @@ class LicenseError(object): def _check_license(lines, path): license_lines = [ - r'Copyright 2013-(?:201[789]|202\d) Lawrence Livermore National Security, LLC and other', # noqa: E501 + r'Copyright 2013-(?:202[01]) Lawrence Livermore National Security, LLC and other', # noqa: E501 r'Spack Project Developers\. See the top-level COPYRIGHT file for details.', # noqa: E501 r'SPDX-License-Identifier: \(Apache-2\.0 OR MIT\)' ] - strict_date = r'Copyright 2013-2020' - found = [] for line in lines: - line = re.sub(r'^[\s#\.]*', '', line) + line = re.sub(r'^[\s#\%\.]*', '', line) line = line.rstrip() for i, license_line in enumerate(license_lines): if re.match(license_line, line): @@ -175,7 +184,7 @@ def verify(args): license_errors = LicenseError() - for relpath in _licensed_files(args.root): + for relpath in _licensed_files(args): path = os.path.join(args.root, relpath) with open(path) as f: lines = [line for line in f][:license_lines] @@ -190,15 +199,28 @@ def verify(args): tty.msg('No license issues found.') -def setup_parser(subparser): - sp = subparser.add_subparsers(metavar='SUBCOMMAND', dest='license_command') - sp.add_parser('list-files', help=list_files.__doc__) +def update_copyright_year(args): + """update copyright for the current year in all licensed files""" - verify_parser = sp.add_parser('verify', help=verify.__doc__) - verify_parser.add_argument( + llns_and_other = ' Lawrence Livermore National Security, LLC and other' + for filename in _licensed_files(args): + fs.filter_file( + r'Copyright \d{4}-\d{4}' + llns_and_other, + strict_date + llns_and_other, + os.path.join(args.root, filename) + ) + + +def setup_parser(subparser): + subparser.add_argument( '--root', action='store', default=spack.paths.prefix, help='scan a different prefix for license issues') + sp = subparser.add_subparsers(metavar='SUBCOMMAND', dest='license_command') + sp.add_parser('list-files', help=list_files.__doc__) + sp.add_parser('verify', help=verify.__doc__) + sp.add_parser('update-copyright-year', help=update_copyright_year.__doc__) + def license(parser, args): if not git: @@ -209,5 +231,6 @@ def license(parser, args): commands = { 'list-files': list_files, 'verify': verify, + 'update-copyright-year': update_copyright_year, } return commands[args.license_command](args) diff --git a/lib/spack/spack/cmd/list.py b/lib/spack/spack/cmd/list.py index a7d4cb0ac3..c00e7a903d 100644 --- a/lib/spack/spack/cmd/list.py +++ b/lib/spack/spack/cmd/list.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/load.py b/lib/spack/spack/cmd/load.py index a8f59fef2f..df0d56c444 100644 --- a/lib/spack/spack/cmd/load.py +++ b/lib/spack/spack/cmd/load.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -63,7 +63,7 @@ def load(parser, args): specs_str = ' '.join(args.specs) or "SPECS" spack.cmd.common.shell_init_instructions( "spack load", - " eval `spack load {sh_arg}` %s" % specs_str, + " eval `spack load {sh_arg} %s`" % specs_str, ) return 1 diff --git a/lib/spack/spack/cmd/location.py b/lib/spack/spack/cmd/location.py index 650e8a477a..9c63881be3 100644 --- a/lib/spack/spack/cmd/location.py +++ b/lib/spack/spack/cmd/location.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -56,7 +56,7 @@ def setup_parser(subparser): help="build directory for a spec " "(requires it to be staged first)") directories.add_argument( - '-e', '--env', action='store', + '-e', '--env', action='store', dest='location_env', help="location of an environment managed by spack") arguments.add_common_arguments(subparser, ['spec']) @@ -71,10 +71,10 @@ def location(parser, args): print(spack.paths.prefix) return - if args.env: - path = spack.environment.root(args.env) + if args.location_env: + path = spack.environment.root(args.location_env) if not os.path.isdir(path): - tty.die("no such environment: '%s'" % args.env) + tty.die("no such environment: '%s'" % args.location_env) print(path) return @@ -119,8 +119,8 @@ def location(parser, args): if args.build_dir: # Out of source builds have build_directory defined if hasattr(pkg, 'build_directory'): - # build_directory can be either absolute or relative to the - # stage path in either case os.path.join makes it absolute + # build_directory can be either absolute or relative to the stage path + # in either case os.path.join makes it absolute print(os.path.normpath(os.path.join( pkg.stage.path, pkg.build_directory diff --git a/lib/spack/spack/cmd/log_parse.py b/lib/spack/spack/cmd/log_parse.py index faa44a4ef0..4f7fd119c0 100644 --- a/lib/spack/spack/cmd/log_parse.py +++ b/lib/spack/spack/cmd/log_parse.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/maintainers.py b/lib/spack/spack/cmd/maintainers.py index a1cf477146..9813103704 100644 --- a/lib/spack/spack/cmd/maintainers.py +++ b/lib/spack/spack/cmd/maintainers.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/mark.py b/lib/spack/spack/cmd/mark.py index 85a22b9741..96c041ba72 100644 --- a/lib/spack/spack/cmd/mark.py +++ b/lib/spack/spack/cmd/mark.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/mirror.py b/lib/spack/spack/cmd/mirror.py index 2d338204d3..45aba2441a 100644 --- a/lib/spack/spack/cmd/mirror.py +++ b/lib/spack/spack/cmd/mirror.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -28,7 +28,7 @@ level = "long" def setup_parser(subparser): - arguments.add_common_arguments(subparser, ['no_checksum']) + arguments.add_common_arguments(subparser, ['no_checksum', 'deprecated']) sp = subparser.add_subparsers( metavar='SUBCOMMAND', dest='mirror_command') @@ -67,6 +67,19 @@ def setup_parser(subparser): " retrieve all versions of each package") arguments.add_common_arguments(create_parser, ['specs']) + # Destroy + destroy_parser = sp.add_parser('destroy', help=mirror_destroy.__doc__) + + destroy_target = destroy_parser.add_mutually_exclusive_group(required=True) + destroy_target.add_argument('-m', '--mirror-name', + metavar='mirror_name', + type=str, + help="find mirror to destroy by name") + destroy_target.add_argument('--mirror-url', + metavar='mirror_url', + type=str, + help="find mirror to destroy by url") + # used to construct scope arguments below scopes = spack.config.scopes() scopes_metavar = spack.config.scopes_metavar @@ -360,8 +373,22 @@ def mirror_create(args): sys.exit(1) +def mirror_destroy(args): + """Given a url, recursively delete everything under it.""" + mirror_url = None + + if args.mirror_name: + result = spack.mirror.MirrorCollection().lookup(args.mirror_name) + mirror_url = result.push_url + elif args.mirror_url: + mirror_url = args.mirror_url + + web_util.remove_url(mirror_url, recursive=True) + + def mirror(parser, args): action = {'create': mirror_create, + 'destroy': mirror_destroy, 'add': mirror_add, 'remove': mirror_remove, 'rm': mirror_remove, @@ -371,4 +398,7 @@ def mirror(parser, args): if args.no_checksum: spack.config.set('config:checksum', False, scope='command_line') + if args.deprecated: + spack.config.set('config:deprecated', True, scope='command_line') + action[args.mirror_command](args) diff --git a/lib/spack/spack/cmd/module.py b/lib/spack/spack/cmd/module.py index f86f3e5f25..e0fd0ab44f 100644 --- a/lib/spack/spack/cmd/module.py +++ b/lib/spack/spack/cmd/module.py @@ -1,21 +1,22 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) import argparse +from typing import Dict, Callable # novm import llnl.util.tty as tty import spack.cmd.modules.lmod import spack.cmd.modules.tcl -description = "manipulate module files" +description = "generate/manage module files" section = "user environment" level = "short" -_subcommands = {} +_subcommands = {} # type: Dict[str, Callable] _deprecated_commands = ('refresh', 'find', 'rm', 'loads') diff --git a/lib/spack/spack/cmd/modules/__init__.py b/lib/spack/spack/cmd/modules/__init__.py index 7f3dd29ef1..7fbce3bb9d 100644 --- a/lib/spack/spack/cmd/modules/__init__.py +++ b/lib/spack/spack/cmd/modules/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/modules/lmod.py b/lib/spack/spack/cmd/modules/lmod.py index d3610bf333..61f2fc28d8 100644 --- a/lib/spack/spack/cmd/modules/lmod.py +++ b/lib/spack/spack/cmd/modules/lmod.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/modules/tcl.py b/lib/spack/spack/cmd/modules/tcl.py index cdde21e821..a9486b9de4 100644 --- a/lib/spack/spack/cmd/modules/tcl.py +++ b/lib/spack/spack/cmd/modules/tcl.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/patch.py b/lib/spack/spack/cmd/patch.py index 8f91edb8f1..14739b8d7d 100644 --- a/lib/spack/spack/cmd/patch.py +++ b/lib/spack/spack/cmd/patch.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -16,7 +16,8 @@ level = "long" def setup_parser(subparser): - arguments.add_common_arguments(subparser, ['no_checksum', 'specs']) + arguments.add_common_arguments( + subparser, ['no_checksum', 'deprecated', 'specs']) def patch(parser, args): @@ -26,6 +27,9 @@ def patch(parser, args): if args.no_checksum: spack.config.set('config:checksum', False, scope='command_line') + if args.deprecated: + spack.config.set('config:deprecated', True, scope='command_line') + specs = spack.cmd.parse_specs(args.specs, concretize=True) for spec in specs: package = spack.repo.get(spec) diff --git a/lib/spack/spack/cmd/pkg.py b/lib/spack/spack/cmd/pkg.py index b988d6a848..f7e3523ff4 100644 --- a/lib/spack/spack/cmd/pkg.py +++ b/lib/spack/spack/cmd/pkg.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/providers.py b/lib/spack/spack/cmd/providers.py index 8ca34542eb..5effdacf8c 100644 --- a/lib/spack/spack/cmd/providers.py +++ b/lib/spack/spack/cmd/providers.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/pydoc.py b/lib/spack/spack/cmd/pydoc.py index 1f81718e38..6e834932d0 100644 --- a/lib/spack/spack/cmd/pydoc.py +++ b/lib/spack/spack/cmd/pydoc.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/python.py b/lib/spack/spack/cmd/python.py index 019e0cf27e..5cee319ff5 100644 --- a/lib/spack/spack/cmd/python.py +++ b/lib/spack/spack/cmd/python.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -28,9 +28,15 @@ def setup_parser(subparser): subparser.add_argument( '-c', dest='python_command', help='command to execute') subparser.add_argument( + '-i', dest='python_interpreter', help='python interpreter', + choices=['python', 'ipython'], default='python') + subparser.add_argument( '-m', dest='module', action='store', help='run library module as a script') subparser.add_argument( + '--path', action='store_true', dest='show_path', + help='show path to python interpreter that spack uses') + subparser.add_argument( 'python_args', nargs=argparse.REMAINDER, help="file to run plus arguments") @@ -40,6 +46,10 @@ def python(parser, args, unknown_args): print('Python', platform.python_version()) return + if args.show_path: + print(sys.executable) + return + if args.module: sys.argv = ['spack-python'] + unknown_args + args.python_args runpy.run_module(args.module, run_name="__main__", alter_sys=True) @@ -48,24 +58,63 @@ def python(parser, args, unknown_args): if unknown_args: tty.die("Unknown arguments:", " ".join(unknown_args)) + # Unexpected behavior from supplying both + if args.python_command and args.python_args: + tty.die("You can only specify a command OR script, but not both.") + + # Run user choice of interpreter + if args.python_interpreter == "ipython": + return spack.cmd.python.ipython_interpreter(args) + return spack.cmd.python.python_interpreter(args) + + +def ipython_interpreter(args): + """An ipython interpreter is intended to be interactive, so it doesn't + support running a script or arguments + """ + try: + import IPython + except ImportError: + tty.die("ipython is not installed, install and try again.") + + if "PYTHONSTARTUP" in os.environ: + startup_file = os.environ["PYTHONSTARTUP"] + if os.path.isfile(startup_file): + with open(startup_file) as startup: + exec(startup.read()) + + # IPython can also support running a script OR command, not both + if args.python_args: + IPython.start_ipython(argv=args.python_args) + elif args.python_command: + IPython.start_ipython(argv=['-c', args.python_command]) + else: + header = ("Spack version %s\nPython %s, %s %s" + % (spack.spack_version, platform.python_version(), + platform.system(), platform.machine())) + + __name__ = "__main__" # noqa + IPython.embed(module="__main__", header=header) + + +def python_interpreter(args): + """A python interpreter is the default interpreter + """ # Fake a main python shell by setting __name__ to __main__. console = code.InteractiveConsole({'__name__': '__main__', 'spack': spack}) - if "PYTHONSTARTUP" in os.environ: startup_file = os.environ["PYTHONSTARTUP"] if os.path.isfile(startup_file): with open(startup_file) as startup: console.runsource(startup.read(), startup_file, 'exec') - python_args = args.python_args - python_command = args.python_command - if python_command: - console.runsource(python_command) - elif python_args: - sys.argv = python_args - with open(python_args[0]) as file: - console.runsource(file.read(), python_args[0], 'exec') + if args.python_command: + console.runsource(args.python_command) + elif args.python_args: + sys.argv = args.python_args + with open(args.python_args[0]) as file: + console.runsource(file.read(), args.python_args[0], 'exec') else: # Provides readline support, allowing user to use arrow keys console.push('import readline') diff --git a/lib/spack/spack/cmd/reindex.py b/lib/spack/spack/cmd/reindex.py index b9f3a2ac3b..1ff5488f5b 100644 --- a/lib/spack/spack/cmd/reindex.py +++ b/lib/spack/spack/cmd/reindex.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/remove.py b/lib/spack/spack/cmd/remove.py index ef01052c29..376d73b687 100644 --- a/lib/spack/spack/cmd/remove.py +++ b/lib/spack/spack/cmd/remove.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/repo.py b/lib/spack/spack/cmd/repo.py index f271790994..3930a03d0f 100644 --- a/lib/spack/spack/cmd/repo.py +++ b/lib/spack/spack/cmd/repo.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/resource.py b/lib/spack/spack/cmd/resource.py index 0d26c39ab8..ed9e7982fa 100644 --- a/lib/spack/spack/cmd/resource.py +++ b/lib/spack/spack/cmd/resource.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/restage.py b/lib/spack/spack/cmd/restage.py index 0f55884bfe..b0582498d6 100644 --- a/lib/spack/spack/cmd/restage.py +++ b/lib/spack/spack/cmd/restage.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/setup.py b/lib/spack/spack/cmd/setup.py deleted file mode 100644 index 45919893cc..0000000000 --- a/lib/spack/spack/cmd/setup.py +++ /dev/null @@ -1,163 +0,0 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other -# Spack Project Developers. See the top-level COPYRIGHT file for details. -# -# SPDX-License-Identifier: (Apache-2.0 OR MIT) - -import argparse -import copy -import os -import sys - -import llnl.util.tty as tty -from llnl.util.filesystem import set_executable - -import spack.repo -import spack.store -import spack.build_systems.cmake -import spack.cmd -import spack.cmd.install as install -import spack.cmd.common.arguments as arguments -from spack.util.executable import which - -from spack.stage import DIYStage - -description = "create a configuration script and module, but don't build" -section = "build" -level = "long" - - -def setup_parser(subparser): - subparser.add_argument( - '-i', '--ignore-dependencies', action='store_true', dest='ignore_deps', - help="do not try to install dependencies of requested packages") - arguments.add_common_arguments(subparser, ['no_checksum', 'spec']) - subparser.add_argument( - '-v', '--verbose', action='store_true', dest='verbose', - help="display verbose build output while installing") - - cd_group = subparser.add_mutually_exclusive_group() - arguments.add_common_arguments(cd_group, ['clean', 'dirty']) - subparser.epilog = 'DEPRECATED: use `spack dev-build` instead' - - -def write_spconfig(package, dirty): - # Set-up the environment - spack.build_environment.setup_package(package, dirty) - - cmd = [str(which('cmake'))] + package.std_cmake_args + package.cmake_args() - - env = dict() - - paths = os.environ['PATH'].split(':') - paths = [item for item in paths if 'spack/env' not in item] - env['PATH'] = ':'.join(paths) - env['CMAKE_PREFIX_PATH'] = os.environ['CMAKE_PREFIX_PATH'] - env['SPACK_INCLUDE_DIRS'] = os.environ['SPACK_INCLUDE_DIRS'] - env['CC'] = os.environ['SPACK_CC'] - env['CXX'] = os.environ['SPACK_CXX'] - env['FC'] = os.environ['SPACK_FC'] - - setup_fname = 'spconfig.py' - with open(setup_fname, 'w') as fout: - fout.write( - r"""#!%s -# - -import sys -import os -import subprocess - -def cmdlist(str): - return list(x.strip().replace("'",'') for x in str.split('\n') if x) -env = dict(os.environ) -""" % sys.executable) - - env_vars = sorted(list(env.keys())) - for name in env_vars: - val = env[name] - if name.find('PATH') < 0: - fout.write('env[%s] = %s\n' % (repr(name), repr(val))) - else: - if name == 'SPACK_INCLUDE_DIRS': - sep = ';' - else: - sep = ':' - - fout.write( - 'env[%s] = "%s".join(cmdlist("""\n' % (repr(name), sep)) - for part in val.split(sep): - fout.write(' %s\n' % part) - fout.write('"""))\n') - - fout.write('\ncmd = cmdlist("""\n') - fout.write('%s\n' % cmd[0]) - for arg in cmd[1:]: - fout.write(' %s\n' % arg) - fout.write('""") + sys.argv[1:]\n') - fout.write('\nproc = subprocess.Popen(cmd, env=env)\nproc.wait()\n') - set_executable(setup_fname) - - -def setup(self, args): - tty.warn('DEPRECATED: use `spack dev-build` instead') - - if not args.spec: - tty.die("spack setup requires a package spec argument.") - - specs = spack.cmd.parse_specs(args.spec) - if len(specs) > 1: - tty.die("spack setup only takes one spec.") - - # Take a write lock before checking for existence. - with spack.store.db.write_transaction(): - spec = specs[0] - if not spack.repo.path.exists(spec.name): - tty.die("No package for '{0}' was found.".format(spec.name), - " Use `spack create` to create a new package") - if not spec.versions.concrete: - tty.die( - "spack setup spec must have a single, concrete version. " - "Did you forget a package version number?") - - spec.concretize() - package = spack.repo.get(spec) - if not isinstance(package, spack.build_systems.cmake.CMakePackage): - tty.die( - 'Support for {0} derived packages not yet implemented'.format( - package.build_system_class)) - - # It's OK if the package is already installed. - - # Forces the build to run out of the current directory. - package.stage = DIYStage(os.getcwd()) - - # disable checksumming if requested - if args.no_checksum: - spack.config.set('config:checksum', False, scope='command_line') - - # Install dependencies if requested to do so - if not args.ignore_deps: - parser = argparse.ArgumentParser() - install.setup_parser(parser) - inst_args = copy.deepcopy(args) - inst_args = parser.parse_args( - ['--only=dependencies'] + args.spec, - namespace=inst_args - ) - install.install(parser, inst_args) - - # Generate spconfig.py - tty.msg( - 'Generating spconfig.py [{0}]'.format(package.spec.cshort_spec) - ) - dirty = args.dirty - write_spconfig(package, dirty) - - # Install this package to register it in the DB and permit - # module file regeneration - inst_args = copy.deepcopy(args) - inst_args = parser.parse_args( - ['--only=package', '--fake'] + args.spec, - namespace=inst_args - ) - install.install(parser, inst_args) diff --git a/lib/spack/spack/cmd/solve.py b/lib/spack/spack/cmd/solve.py index ff6ab4e676..e81161de90 100644 --- a/lib/spack/spack/cmd/solve.py +++ b/lib/spack/spack/cmd/solve.py @@ -1,4 +1,4 @@ -# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -10,6 +10,7 @@ import re import sys import llnl.util.tty as tty +import llnl.util.tty.color as color import spack import spack.cmd @@ -23,15 +24,20 @@ section = 'developer' level = 'long' #: output options -show_options = ('asp', 'output', 'solutions') +show_options = ('asp', 'opt', 'output', 'solutions') def setup_parser(subparser): # Solver arguments subparser.add_argument( - '--show', action='store', default=('solutions'), - help="outputs: a list with any of: " - "%s (default), all" % ', '.join(show_options)) + '--show', action='store', default='opt,solutions', + help="select outputs: comma-separated list of: \n" + " asp asp program text\n" + " opt optimization criteria for best model\n" + " output raw clingo output\n" + " solutions models found by asp program\n" + " all all of the above" + ) subparser.add_argument( '--models', action='store', type=int, default=0, help="number of solutions to search (default 0 for all)") @@ -41,10 +47,10 @@ def setup_parser(subparser): subparser, ['long', 'very_long', 'install_status']) subparser.add_argument( '-y', '--yaml', action='store_const', dest='format', default=None, - const='yaml', help='print concrete spec as YAML') + const='yaml', help='print concrete spec as yaml') subparser.add_argument( '-j', '--json', action='store_const', dest='format', default=None, - const='json', help='print concrete spec as YAML') + const='json', help='print concrete spec as json') subparser.add_argument( '-c', '--cover', action='store', default='nodes', choices=['nodes', 'edges', 'paths'], @@ -113,9 +119,18 @@ def solve(parser, args): best = min(result.answers) opt, _, answer = best - if not args.format: - tty.msg("Best of %d answers." % result.nmodels) - tty.msg("Optimization: %s" % opt) + if ("opt" in dump) and (not args.format): + tty.msg("Best of %d considered solutions." % result.nmodels) + tty.msg("Optimization Criteria:") + + maxlen = max(len(s) for s in result.criteria) + color.cprint( + "@*{ Priority Criterion %sValue}" % ((maxlen - 10) * " ") + ) + for i, (name, val) in enumerate(zip(result.criteria, opt)): + fmt = " @K{%%-8d} %%-%ds%%5d" % maxlen + color.cprint(fmt % (i + 1, name, val)) + print() # iterate over roots from command line for input_spec in specs: diff --git a/lib/spack/spack/cmd/spec.py b/lib/spack/spack/cmd/spec.py index cbfd58ec1a..3a66a05f0b 100644 --- a/lib/spack/spack/cmd/spec.py +++ b/lib/spack/spack/cmd/spec.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -43,6 +43,10 @@ for further documentation regarding the spec syntax, see: '-N', '--namespaces', action='store_true', default=False, help='show fully qualified package names') subparser.add_argument( + '--hash-type', default="build_hash", + choices=['build_hash', 'full_hash', 'dag_hash'], + help='generate spec with a particular hash type.') + subparser.add_argument( '-t', '--types', action='store_true', default=False, help='show dependency types') arguments.add_common_arguments(subparser, ['specs']) @@ -83,11 +87,14 @@ def spec(parser, args): if spec.name in spack.repo.path or spec.virtual: spec.concretize() + # The user can specify the hash type to use + hash_type = getattr(ht, args.hash_type) + if args.format == 'yaml': # use write because to_yaml already has a newline. - sys.stdout.write(spec.to_yaml(hash=ht.build_hash)) + sys.stdout.write(spec.to_yaml(hash=hash_type)) else: - print(spec.to_json(hash=ht.build_hash)) + print(spec.to_json(hash=hash_type)) continue with tree_context(): diff --git a/lib/spack/spack/cmd/stage.py b/lib/spack/spack/cmd/stage.py index 1acefb723c..639f32f28a 100644 --- a/lib/spack/spack/cmd/stage.py +++ b/lib/spack/spack/cmd/stage.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -16,7 +16,8 @@ level = "long" def setup_parser(subparser): - arguments.add_common_arguments(subparser, ['no_checksum', 'specs']) + arguments.add_common_arguments( + subparser, ['no_checksum', 'deprecated', 'specs']) subparser.add_argument( '-p', '--path', dest='path', help="path to stage package, does not add to spack tree") @@ -37,8 +38,17 @@ def stage(parser, args): if args.no_checksum: spack.config.set('config:checksum', False, scope='command_line') - specs = spack.cmd.parse_specs(args.specs, concretize=True) + if args.deprecated: + spack.config.set('config:deprecated', True, scope='command_line') + + specs = spack.cmd.parse_specs(args.specs, concretize=False) + + # prevent multiple specs from extracting in the same folder + if len(specs) > 1 and args.path: + tty.die("`--path` requires a single spec, but multiple were provided") + for spec in specs: + spec = spack.cmd.matching_spec_from_env(spec) package = spack.repo.get(spec) if args.path: package.path = args.path diff --git a/lib/spack/spack/cmd/style.py b/lib/spack/spack/cmd/style.py new file mode 100644 index 0000000000..899ba5b6d7 --- /dev/null +++ b/lib/spack/spack/cmd/style.py @@ -0,0 +1,317 @@ +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +from __future__ import print_function + +import re +import os +import sys +import argparse + +from llnl.util.filesystem import working_dir +import llnl.util.tty as tty + +import spack.paths +from spack.util.executable import which + +if sys.version_info < (3, 0): + from itertools import izip_longest # novm + + zip_longest = izip_longest +else: + from itertools import zip_longest # novm + + +description = "runs source code style checks on spack" +section = "developer" +level = "long" + + +def grouper(iterable, n, fillvalue=None): + "Collect data into fixed-length chunks or blocks" + # grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx" + args = [iter(iterable)] * n + return zip_longest(*args, fillvalue=fillvalue) + + +#: List of directories to exclude from checks. +exclude_directories = [spack.paths.external_path] + +#: max line length we're enforcing (note: this duplicates what's in .flake8) +max_line_length = 79 + + +def is_package(f): + """Whether flake8 should consider a file as a core file or a package. + + We run flake8 with different exceptions for the core and for + packages, since we allow `from spack import *` and poking globals + into packages. + """ + return f.startswith("var/spack/repos/") or "docs/tutorial/examples" in f + + +def changed_files(base=None, untracked=True, all_files=False): + """Get list of changed files in the Spack repository.""" + + git = which("git", required=True) + + if base is None: + base = os.environ.get("TRAVIS_BRANCH", "develop") + + range = "{0}...".format(base) + + git_args = [ + # Add changed files committed since branching off of develop + ["diff", "--name-only", "--diff-filter=ACMR", range], + # Add changed files that have been staged but not yet committed + ["diff", "--name-only", "--diff-filter=ACMR", "--cached"], + # Add changed files that are unstaged + ["diff", "--name-only", "--diff-filter=ACMR"], + ] + + # Add new files that are untracked + if untracked: + git_args.append(["ls-files", "--exclude-standard", "--other"]) + + # add everything if the user asked for it + if all_files: + git_args.append(["ls-files", "--exclude-standard"]) + + excludes = [os.path.realpath(f) for f in exclude_directories] + changed = set() + + for arg_list in git_args: + files = git(*arg_list, output=str).split("\n") + + for f in files: + # Ignore non-Python files + if not (f.endswith(".py") or f == "bin/spack"): + continue + + # Ignore files in the exclude locations + if any(os.path.realpath(f).startswith(e) for e in excludes): + continue + + changed.add(f) + + return sorted(changed) + + +def setup_parser(subparser): + subparser.add_argument( + "-b", + "--base", + action="store", + default=None, + help="select base branch for collecting list of modified files", + ) + subparser.add_argument( + "-a", + "--all", + action="store_true", + help="check all files, not just changed files", + ) + subparser.add_argument( + "-o", + "--output", + action="store_true", + help="send filtered files to stdout as well as temp files", + ) + subparser.add_argument( + "-r", + "--root-relative", + action="store_true", + default=False, + help="print root-relative paths (default: cwd-relative)", + ) + subparser.add_argument( + "-U", + "--no-untracked", + dest="untracked", + action="store_false", + default=True, + help="exclude untracked files from checks", + ) + subparser.add_argument( + "--no-flake8", + dest="flake8", + action="store_false", + help="Do not run flake8, default is run flake8", + ) + subparser.add_argument( + "--no-mypy", + dest="mypy", + action="store_false", + help="Do not run mypy, default is run mypy if available", + ) + subparser.add_argument( + "--black", + dest="black", + action="store_true", + help="Run black checks, default is skip", + ) + subparser.add_argument( + "files", nargs=argparse.REMAINDER, help="specific files to check" + ) + + +def rewrite_and_print_output( + output, args, re_obj=re.compile(r"^(.+):([0-9]+):"), replacement=r"{0}:{1}:" +): + """rewrite ouput with <file>:<line>: format to respect path args""" + if args.root_relative or re_obj is None: + # print results relative to repo root. + print(output) + else: + # print results relative to current working directory + def cwd_relative(path): + return replacement.format( + os.path.relpath( + os.path.join(spack.paths.prefix, path.group(1)), os.getcwd() + ), + *list(path.groups()[1:]) + ) + + for line in output.split("\n"): + if not line: + continue + print(re_obj.sub(cwd_relative, line)) + + +def print_style_header(file_list, args): + tty.msg("style: running code checks on spack.") + tools = [] + if args.flake8: + tools.append("flake8") + if args.mypy: + tools.append("mypy") + if args.black: + tools.append("black") + tty.msg("style: tools selected: " + ", ".join(tools)) + tty.msg("Modified files:", *[filename.strip() for filename in file_list]) + sys.stdout.flush() + + +def print_tool_header(tool): + sys.stdout.flush() + tty.msg("style: running %s checks on spack." % tool) + sys.stdout.flush() + + +def run_flake8(file_list, args): + returncode = 0 + print_tool_header("flake8") + flake8_cmd = which("flake8", required=True) + + output = "" + # run in chunks of 100 at a time to avoid line length limit + # filename parameter in config *does not work* for this reliably + for chunk in grouper(file_list, 100): + chunk = filter(lambda e: e is not None, chunk) + + output = flake8_cmd( + # use .flake8 implicitly to work around bug in flake8 upstream + # append-config is ignored if `--config` is explicitly listed + # see: https://gitlab.com/pycqa/flake8/-/issues/455 + # "--config=.flake8", + *chunk, + fail_on_error=False, + output=str + ) + returncode |= flake8_cmd.returncode + + rewrite_and_print_output(output, args) + + if returncode == 0: + tty.msg("Flake8 style checks were clean") + else: + tty.error("Flake8 style checks found errors") + return returncode + + +def run_mypy(file_list, args): + mypy_cmd = which("mypy") + if mypy_cmd is None: + tty.error("style: mypy is not available in path, skipping") + return 1 + + print_tool_header("mypy") + mpy_args = ["--package", "spack", "--package", "llnl"] + # not yet, need other updates to enable this + # if any([is_package(f) for f in file_list]): + # mpy_args.extend(["--package", "packages"]) + + output = mypy_cmd(*mpy_args, fail_on_error=False, output=str) + returncode = mypy_cmd.returncode + + rewrite_and_print_output(output, args) + + if returncode == 0: + tty.msg("mypy checks were clean") + else: + tty.error("mypy checks found errors") + return returncode + + +def run_black(file_list, args): + black_cmd = which("black") + if black_cmd is None: + tty.error("style: black is not available in path, skipping") + return 1 + + print_tool_header("black") + + pat = re.compile("would reformat +(.*)") + replacement = "would reformat {0}" + returncode = 0 + output = "" + # run in chunks of 100 at a time to avoid line length limit + # filename parameter in config *does not work* for this reliably + for chunk in grouper(file_list, 100): + chunk = filter(lambda e: e is not None, chunk) + + output = black_cmd( + "--check", "--diff", *chunk, fail_on_error=False, output=str, error=str + ) + returncode |= black_cmd.returncode + + rewrite_and_print_output(output, args, pat, replacement) + + if returncode == 0: + tty.msg("black style checks were clean") + else: + tty.error("black checks found errors") + return returncode + + +def style(parser, args): + file_list = args.files + if file_list: + + def prefix_relative(path): + return os.path.relpath( + os.path.abspath(os.path.realpath(path)), spack.paths.prefix + ) + + file_list = [prefix_relative(p) for p in file_list] + + returncode = 0 + with working_dir(spack.paths.prefix): + if not file_list: + file_list = changed_files(args.base, args.untracked, args.all) + print_style_header(file_list, args) + if args.flake8: + returncode = run_flake8(file_list, args) + if args.mypy: + returncode |= run_mypy(file_list, args) + if args.black: + returncode |= run_black(file_list, args) + + if returncode != 0: + print("spack style found errors.") + sys.exit(1) + else: + print("spack style checks were clean.") diff --git a/lib/spack/spack/cmd/test.py b/lib/spack/spack/cmd/test.py index d164e26334..343c019bce 100644 --- a/lib/spack/spack/cmd/test.py +++ b/lib/spack/spack/cmd/test.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -7,18 +7,22 @@ from __future__ import print_function import os import argparse import textwrap +import inspect import fnmatch import re import shutil +import sys import llnl.util.tty as tty +import llnl.util.tty.colify as colify import spack.install_test import spack.environment as ev import spack.cmd import spack.cmd.common.arguments as arguments -import spack.report import spack.package +import spack.repo +import spack.report description = "run spack's tests for an install" section = "admin" @@ -78,8 +82,11 @@ def setup_parser(subparser): arguments.add_common_arguments(run_parser, ['installed_specs']) # List - sp.add_parser('list', description=test_list.__doc__, - help=first_line(test_list.__doc__)) + list_parser = sp.add_parser('list', description=test_list.__doc__, + help=first_line(test_list.__doc__)) + list_parser.add_argument( + "-a", "--all", action="store_true", dest="list_all", + help="list all packages with tests (not just installed)") # Find find_parser = sp.add_parser('find', description=test_find.__doc__, @@ -188,18 +195,36 @@ environment variables: def has_test_method(pkg): - return pkg.test.__func__ != spack.package.PackageBase.test + if not inspect.isclass(pkg): + tty.die('{0}: is not a class, it is {1}'.format(pkg, type(pkg))) + + pkg_base = spack.package.PackageBase + return ( + (issubclass(pkg, pkg_base) and pkg.test != pkg_base.test) or + (isinstance(pkg, pkg_base) and pkg.test.__func__ != pkg_base.test) + ) def test_list(args): - """List all installed packages with available tests.""" + """List installed packages with available tests.""" + if args.list_all: + all_packages_with_tests = [ + pkg_class.name + for pkg_class in spack.repo.path.all_package_classes() + if has_test_method(pkg_class) + ] + if sys.stdout.isatty(): + tty.msg("%d packages with tests." % len(all_packages_with_tests)) + colify.colify(all_packages_with_tests) + return + # TODO: This can be extended to have all of the output formatting options # from `spack find`. env = ev.get_env(args, 'test') hashes = env.all_hashes() if env else None specs = spack.store.db.query(hashes=hashes) - specs = list(filter(lambda s: has_test_method(s.package), specs)) + specs = list(filter(lambda s: has_test_method(s.package_class), specs)) spack.cmd.display_specs(specs, long=True) diff --git a/lib/spack/spack/cmd/test_env.py b/lib/spack/spack/cmd/test_env.py index a0f54d482f..3d3827a7da 100644 --- a/lib/spack/spack/cmd/test_env.py +++ b/lib/spack/spack/cmd/test_env.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/tutorial.py b/lib/spack/spack/cmd/tutorial.py index 6c0aa40a4a..a8bc1f1ddd 100644 --- a/lib/spack/spack/cmd/tutorial.py +++ b/lib/spack/spack/cmd/tutorial.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/undevelop.py b/lib/spack/spack/cmd/undevelop.py index 6319b88e06..09fe558d56 100644 --- a/lib/spack/spack/cmd/undevelop.py +++ b/lib/spack/spack/cmd/undevelop.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/uninstall.py b/lib/spack/spack/cmd/uninstall.py index e541eaf91b..2a9e6b00b7 100644 --- a/lib/spack/spack/cmd/uninstall.py +++ b/lib/spack/spack/cmd/uninstall.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/unit_test.py b/lib/spack/spack/cmd/unit_test.py index 509211de04..c290bc8572 100644 --- a/lib/spack/spack/cmd/unit_test.py +++ b/lib/spack/spack/cmd/unit_test.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -146,7 +146,7 @@ def add_back_pytest_args(args, unknown_args): def unit_test(parser, args, unknown_args): if args.pytest_help: # make the pytest.main help output more accurate - sys.argv[0] = 'spack test' + sys.argv[0] = 'spack unit-test' return pytest.main(['-h']) # add back any parsed pytest args we need to pass to pytest diff --git a/lib/spack/spack/cmd/unload.py b/lib/spack/spack/cmd/unload.py index 8494453489..c854466e05 100644 --- a/lib/spack/spack/cmd/unload.py +++ b/lib/spack/spack/cmd/unload.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/url.py b/lib/spack/spack/cmd/url.py index db8f358887..e5dde5e9c7 100644 --- a/lib/spack/spack/cmd/url.py +++ b/lib/spack/spack/cmd/url.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/verify.py b/lib/spack/spack/cmd/verify.py index b20d795ce5..6e7ff0f3bd 100644 --- a/lib/spack/spack/cmd/verify.py +++ b/lib/spack/spack/cmd/verify.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/cmd/versions.py b/lib/spack/spack/cmd/versions.py index 366307f0b2..ec808e0e89 100644 --- a/lib/spack/spack/cmd/versions.py +++ b/lib/spack/spack/cmd/versions.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -12,6 +12,7 @@ import llnl.util.tty as tty import spack.cmd.common.arguments as arguments import spack.repo +from spack.version import ver, infinity_versions description = "list available versions of a package" section = "packaging" @@ -19,8 +20,17 @@ level = "long" def setup_parser(subparser): - subparser.add_argument('-s', '--safe-only', action='store_true', - help='only list safe versions of the package') + output = subparser.add_mutually_exclusive_group() + output.add_argument('-s', '--safe', action='store_true', + help='only list safe versions of the package') + output.add_argument('--safe-only', action='store_true', + help='[deprecated] only list safe versions ' + 'of the package') + output.add_argument('-r', '--remote', action='store_true', + help='only list remote versions of the package') + output.add_argument('-n', '--new', action='store_true', + help='only list remote versions newer than ' + 'the latest checksummed version') subparser.add_argument( '-c', '--concurrency', default=32, type=int, help='number of concurrent requests' @@ -31,26 +41,41 @@ def setup_parser(subparser): def versions(parser, args): pkg = spack.repo.get(args.package) - if sys.stdout.isatty(): - tty.msg('Safe versions (already checksummed):') - safe_versions = pkg.versions - if not safe_versions: + if args.safe_only: + tty.warn('"--safe-only" is deprecated. Use "--safe" instead.') + args.safe = args.safe_only + + if not (args.remote or args.new): if sys.stdout.isatty(): - tty.warn('Found no versions for {0}'.format(pkg.name)) - tty.debug('Manually add versions to the package.') - else: - colify(sorted(safe_versions, reverse=True), indent=2) + tty.msg('Safe versions (already checksummed):') - if args.safe_only: - return + if not safe_versions: + if sys.stdout.isatty(): + tty.warn('Found no versions for {0}'.format(pkg.name)) + tty.debug('Manually add versions to the package.') + else: + colify(sorted(safe_versions, reverse=True), indent=2) - if sys.stdout.isatty(): - tty.msg('Remote versions (not yet checksummed):') + if args.safe: + return fetched_versions = pkg.fetch_remote_versions(args.concurrency) - remote_versions = set(fetched_versions).difference(safe_versions) + + if args.new: + if sys.stdout.isatty(): + tty.msg('New remote versions (not yet checksummed):') + numeric_safe_versions = list(filter( + lambda v: str(v) not in infinity_versions, + safe_versions)) + highest_safe_version = max(numeric_safe_versions) + remote_versions = set([ver(v) for v in set(fetched_versions) + if v > highest_safe_version]) + else: + if sys.stdout.isatty(): + tty.msg('Remote versions (not yet checksummed):') + remote_versions = set(fetched_versions).difference(safe_versions) if not remote_versions: if sys.stdout.isatty(): diff --git a/lib/spack/spack/cmd/view.py b/lib/spack/spack/cmd/view.py index 151f6c1564..ec8003bbf7 100644 --- a/lib/spack/spack/cmd/view.py +++ b/lib/spack/spack/cmd/view.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/compiler.py b/lib/spack/spack/compiler.py index d498803633..f6c7e98ff0 100644 --- a/lib/spack/spack/compiler.py +++ b/lib/spack/spack/compiler.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -10,6 +10,7 @@ import re import itertools import shutil import tempfile +from typing import Sequence, List # novm import llnl.util.lang from llnl.util.filesystem import ( @@ -190,20 +191,20 @@ class Compiler(object): and how to identify the particular type of compiler.""" # Subclasses use possible names of C compiler - cc_names = [] + cc_names = [] # type: List[str] # Subclasses use possible names of C++ compiler - cxx_names = [] + cxx_names = [] # type: List[str] # Subclasses use possible names of Fortran 77 compiler - f77_names = [] + f77_names = [] # type: List[str] # Subclasses use possible names of Fortran 90 compiler - fc_names = [] + fc_names = [] # type: List[str] # Optional prefix regexes for searching for this type of compiler. # Prefixes are sometimes used for toolchains - prefixes = [] + prefixes = [] # type: List[str] # Optional suffix regexes for searching for this type of compiler. # Suffixes are used by some frameworks, e.g. macports uses an '-mp-X.Y' @@ -214,7 +215,7 @@ class Compiler(object): version_argument = '-dumpversion' #: Return values to ignore when invoking the compiler to get its version - ignore_version_errors = () + ignore_version_errors = () # type: Sequence[int] #: Regex used to extract version from compiler's output version_regex = '(.*)' @@ -266,9 +267,9 @@ class Compiler(object): return ['-O', '-O0', '-O1', '-O2', '-O3'] # Cray PrgEnv name that can be used to load this compiler - PrgEnv = None + PrgEnv = None # type: str # Name of module used to switch versions of this compiler - PrgEnv_compiler = None + PrgEnv_compiler = None # type: str def __init__(self, cspec, operating_system, target, paths, modules=None, alias=None, environment=None, diff --git a/lib/spack/spack/compilers/__init__.py b/lib/spack/spack/compilers/__init__.py index 85e0275a29..817d21add6 100644 --- a/lib/spack/spack/compilers/__init__.py +++ b/lib/spack/spack/compilers/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -11,6 +11,7 @@ import itertools import multiprocessing.pool import os import six +from typing import Dict # novm import llnl.util.lang import llnl.util.filesystem as fs @@ -21,6 +22,7 @@ import spack.paths import spack.error import spack.spec import spack.config +import spack.compiler import spack.architecture from spack.util.environment import get_path @@ -35,7 +37,7 @@ _cache_config_file = [] # TODO: Caches at module level make it difficult to mock configurations in # TODO: unit tests. It might be worth reworking their implementation. #: cache of compilers constructed from config data, keyed by config entry id. -_compiler_cache = {} +_compiler_cache = {} # type: Dict[str, spack.compiler.Compiler] _compiler_to_pkg = { 'clang': 'llvm+clang', diff --git a/lib/spack/spack/compilers/aocc.py b/lib/spack/spack/compilers/aocc.py index 5c9e885490..dcb6589861 100644 --- a/lib/spack/spack/compilers/aocc.py +++ b/lib/spack/spack/compilers/aocc.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -9,6 +9,7 @@ import sys import llnl.util.lang from spack.compiler import Compiler +from spack.version import ver class Aocc(Compiler): @@ -24,6 +25,9 @@ class Aocc(Compiler): # Subclasses use possible names of Fortran 90 compiler fc_names = ['flang'] + PrgEnv = 'PrgEnv-aocc' + PrgEnv_compiler = 'aocc' + version_argument = '--version' @property @@ -114,3 +118,26 @@ class Aocc(Compiler): @classmethod def f77_version(cls, f77): return cls.fc_version(f77) + + @property + def stdcxx_libs(self): + return ('-lstdc++', ) + + @property + def cflags(self): + return self._handle_default_flag_addtions() + + @property + def cxxflags(self): + return self._handle_default_flag_addtions() + + @property + def fflags(self): + return self._handle_default_flag_addtions() + + def _handle_default_flag_addtions(self): + # This is a known issue for AOCC 3.0 see: + # https://developer.amd.com/wp-content/resources/AOCC-3.0-Install-Guide.pdf + if self.real_version == ver('3.0.0'): + return ("-Wno-unused-command-line-argument " + "-mllvm -eliminate-similar-expr=false") diff --git a/lib/spack/spack/compilers/apple_clang.py b/lib/spack/spack/compilers/apple_clang.py index e03117ae05..372a677a47 100644 --- a/lib/spack/spack/compilers/apple_clang.py +++ b/lib/spack/spack/compilers/apple_clang.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/compilers/arm.py b/lib/spack/spack/compilers/arm.py index 59eb1714b8..4c59d89210 100644 --- a/lib/spack/spack/compilers/arm.py +++ b/lib/spack/spack/compilers/arm.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/compilers/cce.py b/lib/spack/spack/compilers/cce.py index a325bf3a45..64bf62543d 100644 --- a/lib/spack/spack/compilers/cce.py +++ b/lib/spack/spack/compilers/cce.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -112,3 +112,9 @@ class Cce(Compiler): if self.is_clang_based: return "-fPIC" return "-h PIC" + + @property + def stdcxx_libs(self): + # Cray compiler wrappers link to the standard C++ library + # without additional flags. + return () diff --git a/lib/spack/spack/compilers/clang.py b/lib/spack/spack/compilers/clang.py index 5eb08cbf0a..d2dc3db192 100644 --- a/lib/spack/spack/compilers/clang.py +++ b/lib/spack/spack/compilers/clang.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -159,11 +159,11 @@ class Clang(Compiler): match = re.search( # Normal clang compiler versions are left as-is - r'clang version ([^ )]+)-svn[~.\w\d-]*|' + r'clang version ([^ )\n]+)-svn[~.\w\d-]*|' # Don't include hyphenated patch numbers in the version # (see https://github.com/spack/spack/pull/14365 for details) - r'clang version ([^ )]+?)-[~.\w\d-]*|' - r'clang version ([^ )]+)', + r'clang version ([^ )\n]+?)-[~.\w\d-]*|' + r'clang version ([^ )\n]+)', output ) if match: diff --git a/lib/spack/spack/compilers/fj.py b/lib/spack/spack/compilers/fj.py index c2cc66e110..d53fa02fba 100644 --- a/lib/spack/spack/compilers/fj.py +++ b/lib/spack/spack/compilers/fj.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -35,8 +35,12 @@ class Fj(spack.compiler.Compiler): return "-v" @property + def debug_flags(self): + return "-g" + + @property def opt_flags(self): - return ['-O', '-O0', '-O1', '-O2', '-O3', '-O4'] + return ['-O0', '-O1', '-O2', '-O3', '-Ofast'] @property def openmp_flag(self): @@ -55,6 +59,10 @@ class Fj(spack.compiler.Compiler): return "-std=c++14" @property + def cxx17_flag(self): + return "-std=c++17" + + @property def c99_flag(self): return "-std=c99" diff --git a/lib/spack/spack/compilers/gcc.py b/lib/spack/spack/compilers/gcc.py index 02ee3e5db9..ff8675324e 100644 --- a/lib/spack/spack/compilers/gcc.py +++ b/lib/spack/spack/compilers/gcc.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/compilers/intel.py b/lib/spack/spack/compilers/intel.py index 5e64a1439e..a05b992a10 100644 --- a/lib/spack/spack/compilers/intel.py +++ b/lib/spack/spack/compilers/intel.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/compilers/nag.py b/lib/spack/spack/compilers/nag.py index 503a31e404..c72a980327 100644 --- a/lib/spack/spack/compilers/nag.py +++ b/lib/spack/spack/compilers/nag.py @@ -1,17 +1,19 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) +from typing import List # novm + import spack.compiler class Nag(spack.compiler.Compiler): # Subclasses use possible names of C compiler - cc_names = [] + cc_names = [] # type: List[str] # Subclasses use possible names of C++ compiler - cxx_names = [] + cxx_names = [] # type: List[str] # Subclasses use possible names of Fortran 77 compiler f77_names = ['nagfor'] @@ -100,3 +102,13 @@ class Nag(spack.compiler.Compiler): @property def linker_arg(self): return '-Wl,-Wl,,' + + @property + def disable_new_dtags(self): + # Disable RPATH/RUNPATH forcing for NAG/GCC mixed toolchains: + return '' + + @property + def enable_new_dtags(self): + # Disable RPATH/RUNPATH forcing for NAG/GCC mixed toolchains: + return '' diff --git a/lib/spack/spack/compilers/nvhpc.py b/lib/spack/spack/compilers/nvhpc.py index c102320132..1929d50613 100644 --- a/lib/spack/spack/compilers/nvhpc.py +++ b/lib/spack/spack/compilers/nvhpc.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/compilers/oneapi.py b/lib/spack/spack/compilers/oneapi.py index 1b029699b5..0289d266b8 100644 --- a/lib/spack/spack/compilers/oneapi.py +++ b/lib/spack/spack/compilers/oneapi.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -29,14 +29,15 @@ class Oneapi(Compiler): PrgEnv_compiler = 'oneapi' version_argument = '--version' - version_regex = r'(?:(?:oneAPI DPC\+\+ Compiler)|(?:ifx \(IFORT\))) (\S+)' + version_regex = r'(?:(?:oneAPI DPC\+\+ Compiler)|(?:\(IFORT\))) (\S+)' @property def verbose_flag(self): return "-v" required_libs = ['libirc', 'libifcore', 'libifcoremt', 'libirng', - 'libsvml', 'libintlc', 'libimf'] + 'libsvml', 'libintlc', 'libimf', 'libsycl', + 'libOpenCL'] @property def debug_flags(self): diff --git a/lib/spack/spack/compilers/pgi.py b/lib/spack/spack/compilers/pgi.py index 383281a9f4..3de7b708c7 100644 --- a/lib/spack/spack/compilers/pgi.py +++ b/lib/spack/spack/compilers/pgi.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -88,3 +88,7 @@ class Pgi(Compiler): 'the C11 standard', 'c11_flag', '< 15.3') + + @property + def stdcxx_libs(self): + return ('-pgc++libs',) diff --git a/lib/spack/spack/compilers/xl.py b/lib/spack/spack/compilers/xl.py index 66009646e3..7e39d1c530 100644 --- a/lib/spack/spack/compilers/xl.py +++ b/lib/spack/spack/compilers/xl.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/compilers/xl_r.py b/lib/spack/spack/compilers/xl_r.py index c9a785d5d8..81100e782a 100644 --- a/lib/spack/spack/compilers/xl_r.py +++ b/lib/spack/spack/compilers/xl_r.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/concretize.py b/lib/spack/spack/concretize.py index 48210f0959..900009e283 100644 --- a/lib/spack/spack/concretize.py +++ b/lib/spack/spack/concretize.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -73,9 +73,7 @@ class Concretizer(object): if not dev_info: return False - path = dev_info['path'] - path = path if os.path.isabs(path) else os.path.join( - env.path, path) + path = os.path.normpath(os.path.join(env.path, dev_info['path'])) if 'dev_path' in spec.variants: assert spec.variants['dev_path'].value == path @@ -718,6 +716,8 @@ def concretize_specs_together(*abstract_specs, **kwargs): """Given a number of specs as input, tries to concretize them together. Args: + tests (bool or list or set): False to run no tests, True to test + all packages, or a list of package names to run tests for some *abstract_specs: abstract specs to be concretized, given either as Specs or strings @@ -775,7 +775,7 @@ def _concretize_specs_together_original(*abstract_specs, **kwargs): with spack.repo.additional_repository(concretization_repository): # Spec from a helper package that depends on all the abstract_specs concretization_root = spack.spec.Spec('concretizationroot') - concretization_root.concretize() + concretization_root.concretize(tests=kwargs.get('tests', False)) # Retrieve the direct dependencies concrete_specs = [ concretization_root[spec.name].copy() for spec in abstract_specs diff --git a/lib/spack/spack/config.py b/lib/spack/spack/config.py index 8794072e96..574902f508 100644 --- a/lib/spack/spack/config.py +++ b/lib/spack/spack/config.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -35,10 +35,10 @@ import functools import os import re import sys -import multiprocessing from contextlib import contextmanager from six import iteritems from ordereddict_backport import OrderedDict +from typing import List # novm import ruamel.yaml as yaml from ruamel.yaml.error import MarkedYAMLError @@ -60,6 +60,7 @@ import spack.schema.config import spack.schema.upstreams import spack.schema.env from spack.error import SpackError +from spack.util.cpus import cpus_available # Hacked yaml for configuration files preserves line numbers. import spack.util.spack_yaml as syaml @@ -109,7 +110,7 @@ config_defaults = { 'verify_ssl': True, 'checksum': True, 'dirty': False, - 'build_jobs': min(16, multiprocessing.cpu_count()), + 'build_jobs': min(16, cpus_available()), 'build_stage': '$tempdir/spack-stage', 'concretizer': 'original', } @@ -128,7 +129,7 @@ def first_existing(dictionary, keys): try: return next(k for k in keys if k in dictionary) except StopIteration: - raise KeyError("None of %s is in dict!" % keys) + raise KeyError("None of %s is in dict!" % str(keys)) class ConfigScope(object): @@ -564,7 +565,7 @@ class Configuration(object): If ``scope`` is ``None`` or not provided, return the merged contents of all of Spack's configuration scopes. If ``scope`` is provided, - return only the confiugration as specified in that scope. + return only the configuration as specified in that scope. This off the top-level name from the YAML section. That is, for a YAML config file that looks like this:: @@ -749,7 +750,7 @@ def override(path_or_scope, value=None): #: configuration scopes added on the command line #: set by ``spack.main.main()``. -command_line_scopes = [] +command_line_scopes = [] # type: List[str] def _add_platform_scope(cfg, scope_type, name, path): @@ -817,6 +818,81 @@ def _config(): config = llnl.util.lang.Singleton(_config) +def add_from_file(filename, scope=None): + """Add updates to a config from a filename + """ + import spack.environment as ev + + # Get file as config dict + data = read_config_file(filename) + if any(k in data for k in spack.schema.env.keys): + data = ev.config_dict(data) + + # update all sections from config dict + # We have to iterate on keys to keep overrides from the file + for section in data.keys(): + if section in section_schemas.keys(): + # Special handling for compiler scope difference + # Has to be handled after we choose a section + if scope is None: + scope = default_modify_scope(section) + + value = data[section] + existing = get(section, scope=scope) + new = merge_yaml(existing, value) + + # We cannot call config.set directly (set is a type) + config.set(section, new, scope) + + +def add(fullpath, scope=None): + """Add the given configuration to the specified config scope. + Add accepts a path. If you want to add from a filename, use add_from_file""" + + components = process_config_path(fullpath) + + has_existing_value = True + path = '' + override = False + for idx, name in enumerate(components[:-1]): + # First handle double colons in constructing path + colon = '::' if override else ':' if path else '' + path += colon + name + if getattr(name, 'override', False): + override = True + else: + override = False + + # Test whether there is an existing value at this level + existing = get(path, scope=scope) + + if existing is None: + has_existing_value = False + # We've nested further than existing config, so we need the + # type information for validation to know how to handle bare + # values appended to lists. + existing = get_valid_type(path) + + # construct value from this point down + value = syaml.load_config(components[-1]) + for component in reversed(components[idx + 1:-1]): + value = {component: value} + break + + if has_existing_value: + path, _, value = fullpath.rpartition(':') + value = syaml.load_config(value) + existing = get(path, scope=scope) + + # append values to lists + if isinstance(existing, list) and not isinstance(value, list): + value = [value] + + # merge value into existing + new = merge_yaml(existing, value) + config.set(path, new, scope) + + def get(path, default=None, scope=None): """Module-level wrapper for ``Configuration.get()``.""" return config.get(path, default, scope) diff --git a/lib/spack/spack/container/__init__.py b/lib/spack/spack/container/__init__.py index 8206efef01..7211f94ff1 100644 --- a/lib/spack/spack/container/__init__.py +++ b/lib/spack/spack/container/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/container/images.py b/lib/spack/spack/container/images.py index 32fcff2770..9d2e15f195 100644 --- a/lib/spack/spack/container/images.py +++ b/lib/spack/spack/container/images.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/container/writers/__init__.py b/lib/spack/spack/container/writers/__init__.py index 176dd7a50c..b1c82a7bdf 100644 --- a/lib/spack/spack/container/writers/__init__.py +++ b/lib/spack/spack/container/writers/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/container/writers/docker.py b/lib/spack/spack/container/writers/docker.py index 557d22c803..7c9372fb86 100644 --- a/lib/spack/spack/container/writers/docker.py +++ b/lib/spack/spack/container/writers/docker.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/container/writers/singularity.py b/lib/spack/spack/container/writers/singularity.py index 32f29eb83d..e3eaaea008 100644 --- a/lib/spack/spack/container/writers/singularity.py +++ b/lib/spack/spack/container/writers/singularity.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/database.py b/lib/spack/spack/database.py index db1e6a636b..650ab48492 100644 --- a/lib/spack/spack/database.py +++ b/lib/spack/spack/database.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -27,6 +27,8 @@ import six import socket import sys import time +from typing import Dict # novm + try: import uuid _use_uuid = True @@ -289,10 +291,10 @@ _query_docstring = """ class Database(object): """Per-process lock objects for each install prefix.""" - _prefix_locks = {} + _prefix_locks = {} # type: Dict[str, lk.Lock] """Per-process failure (lock) objects for each install prefix.""" - _prefix_failures = {} + _prefix_failures = {} # type: Dict[str, lk.Lock] def __init__(self, root, db_dir=None, upstream_dbs=None, is_upstream=False, enable_transaction_locking=True, @@ -793,7 +795,7 @@ class Database(object): # do it *while* we're constructing specs,it causes hashes to be # cached prematurely. for hash_key, rec in data.items(): - rec.spec._mark_concrete() + rec.spec._mark_root_concrete() self._data = data @@ -1452,11 +1454,12 @@ class Database(object): rec.spec.name) != known: continue - inst_date = datetime.datetime.fromtimestamp( - rec.installation_time - ) - if not (start_date < inst_date < end_date): - continue + if start_date or end_date: + inst_date = datetime.datetime.fromtimestamp( + rec.installation_time + ) + if not (start_date < inst_date < end_date): + continue if (query_spec is any or rec.spec.satisfies(query_spec, strict=True)): @@ -1464,6 +1467,8 @@ class Database(object): return results + if _query.__doc__ is None: + _query.__doc__ = "" _query.__doc__ += _query_docstring def query_local(self, *args, **kwargs): @@ -1471,6 +1476,8 @@ class Database(object): with self.read_transaction(): return sorted(self._query(*args, **kwargs)) + if query_local.__doc__ is None: + query_local.__doc__ = "" query_local.__doc__ += _query_docstring def query(self, *args, **kwargs): @@ -1489,6 +1496,8 @@ class Database(object): return sorted(results) + if query.__doc__ is None: + query.__doc__ = "" query.__doc__ += _query_docstring def query_one(self, query_spec, known=any, installed=True): diff --git a/lib/spack/spack/dependency.py b/lib/spack/spack/dependency.py index fe7d6b5983..0773ce0763 100644 --- a/lib/spack/spack/dependency.py +++ b/lib/spack/spack/dependency.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/directives.py b/lib/spack/spack/directives.py index b7793a893f..30a45f1b06 100644 --- a/lib/spack/spack/directives.py +++ b/lib/spack/spack/directives.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -34,6 +34,7 @@ import re import sys from six import string_types +from typing import Set, List # novm import llnl.util.lang import llnl.util.tty.color @@ -111,8 +112,8 @@ class DirectiveMeta(type): """ # Set of all known directives - _directive_names = set() - _directives_to_be_executed = [] + _directive_names = set() # type: Set[str] + _directives_to_be_executed = [] # type: List[str] def __new__(cls, name, bases, attr_dict): # Initialize the attribute containing the list of directives @@ -277,6 +278,9 @@ def version(ver, checksum=None, **kwargs): The ``dict`` of arguments is turned into a valid fetch strategy for code packages later. See ``spack.fetch_strategy.for_package_version()``. + + Keyword Arguments: + deprecated (bool): whether or not this version is deprecated """ def _execute_version(pkg): if checksum is not None: @@ -400,7 +404,7 @@ def depends_on(spec, when=None, type=default_deptype, patches=None): @directive(('extendees', 'dependencies')) -def extends(spec, **kwargs): +def extends(spec, type=('build', 'run'), **kwargs): """Same as depends_on, but allows symlinking into dependency's prefix tree. @@ -421,7 +425,7 @@ def extends(spec, **kwargs): if not when_spec: return - _depends_on(pkg, spec, when=when) + _depends_on(pkg, spec, when=when, type=type) pkg.extendees[spec] = (spack.spec.Spec(spec), kwargs) return _execute_extends diff --git a/lib/spack/spack/directory_layout.py b/lib/spack/spack/directory_layout.py index 06a652f450..8a429a850f 100644 --- a/lib/spack/spack/directory_layout.py +++ b/lib/spack/spack/directory_layout.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -7,6 +7,7 @@ import os import shutil import glob import tempfile +import errno from contextlib import contextmanager import ruamel.yaml as yaml @@ -14,7 +15,9 @@ import ruamel.yaml as yaml from llnl.util.filesystem import mkdirp import spack.config +import spack.hash_types as ht import spack.spec +import spack.util.spack_json as sjson from spack.error import SpackError @@ -119,9 +122,17 @@ class DirectoryLayout(object): path = os.path.dirname(path) while path != self.root: if os.path.isdir(path): - if os.listdir(path): - return - os.rmdir(path) + try: + os.rmdir(path) + except OSError as e: + if e.errno == errno.ENOENT: + # already deleted, continue with parent + pass + elif e.errno == errno.ENOTEMPTY: + # directory wasn't empty, done + return + else: + raise e path = os.path.dirname(path) @@ -233,7 +244,20 @@ class YamlDirectoryLayout(DirectoryLayout): """Write a spec out to a file.""" _check_concrete(spec) with open(path, 'w') as f: - spec.to_yaml(f) + # The hash the the projection is the DAG hash but we write out the + # full provenance by full hash so it's availabe if we want it later + spec.to_yaml(f, hash=ht.full_hash) + + def write_host_environment(self, spec): + """The host environment is a json file with os, kernel, and spack + versioning. We use it in the case that an analysis later needs to + easily access this information. + """ + from spack.util.environment import get_host_environment_metadata + env_file = self.env_metadata_path(spec) + environ = get_host_environment_metadata() + with open(env_file, 'w') as fd: + sjson.dump(environ, fd) def read_spec(self, path): """Read the contents of a file and parse them as a spec""" @@ -288,6 +312,9 @@ class YamlDirectoryLayout(DirectoryLayout): def metadata_path(self, spec): return os.path.join(spec.prefix, self.metadata_dir) + def env_metadata_path(self, spec): + return os.path.join(self.metadata_path(spec), "install_environment.json") + def build_packages_path(self, spec): return os.path.join(self.metadata_path(spec), self.packages_dir) @@ -335,7 +362,13 @@ class YamlDirectoryLayout(DirectoryLayout): # # TODO: remove this when we do better concretization and don't # ignore build-only deps in hashes. - elif installed_spec == spec.copy(deps=('link', 'run')): + elif (installed_spec.copy(deps=('link', 'run')) == + spec.copy(deps=('link', 'run'))): + # The directory layout prefix is based on the dag hash, so among + # specs with differing full-hash but matching dag-hash, only one + # may be installed. This means for example that for two instances + # that differ only in CMake version used to build, only one will + # be installed. return path if spec.dag_hash() == installed_spec.dag_hash(): @@ -411,8 +444,8 @@ class YamlViewExtensionsLayout(ExtensionsLayout): def check_extension_conflict(self, spec, ext_spec): exts = self._extension_map(spec) if ext_spec.name in exts: - installed_spec = exts[ext_spec.name] - if ext_spec == installed_spec: + installed_spec = exts[ext_spec.name].copy(deps=('link', 'run')) + if ext_spec.copy(deps=('link', 'run')) == installed_spec: raise ExtensionAlreadyInstalledError(spec, ext_spec) else: raise ExtensionConflictError(spec, ext_spec, installed_spec) diff --git a/lib/spack/spack/environment.py b/lib/spack/spack/environment.py index 7f7625f2e6..647352a10e 100644 --- a/lib/spack/spack/environment.py +++ b/lib/spack/spack/environment.py @@ -1,17 +1,15 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) - import collections import os import re import sys import shutil import copy -import socket - import six +import ruamel.yaml as yaml from ordereddict_backport import OrderedDict @@ -33,13 +31,13 @@ import spack.config import spack.user_environment as uenv from spack.filesystem_view import YamlFilesystemView import spack.util.environment -import spack.architecture as architecture from spack.spec import Spec from spack.spec_list import SpecList, InvalidSpecConstraintError from spack.variant import UnknownVariantError +import spack.util.hash import spack.util.lock as lk from spack.util.path import substitute_path_variables -from spack.installer import PackageInstaller +import spack.util.path #: environment variable used to indicate the active environment spack_env_var = 'SPACK_ENV' @@ -131,7 +129,7 @@ def activate( if use_env_repo: spack.repo.path.put_first(_active_environment.repo) - tty.debug("Using environmennt '%s'" % _active_environment.name) + tty.debug("Using environment '%s'" % _active_environment.name) # Construct the commands to run cmds = '' @@ -392,12 +390,12 @@ def read(name): return Environment(root(name)) -def create(name, init_file=None, with_view=None): +def create(name, init_file=None, with_view=None, keep_relative=False): """Create a named environment in Spack.""" validate_env_name(name) if exists(name): raise SpackEnvironmentError("'%s': environment already exists" % name) - return Environment(root(name), init_file, with_view) + return Environment(root(name), init_file, with_view, keep_relative) def config_dict(yaml_data): @@ -446,21 +444,11 @@ def _write_yaml(data, str_or_file): def _eval_conditional(string): """Evaluate conditional definitions using restricted variable scope.""" - arch = architecture.Arch( - architecture.platform(), 'default_os', 'default_target') - arch_spec = spack.spec.Spec('arch=%s' % arch) - valid_variables = { - 'target': str(arch.target), - 'os': str(arch.os), - 'platform': str(arch.platform), - 'arch': arch_spec, - 'architecture': arch_spec, - 'arch_str': str(arch), + valid_variables = spack.util.environment.get_host_environment() + valid_variables.update({ 're': re, 'env': os.environ, - 'hostname': socket.gethostname() - } - + }) return eval(string, valid_variables) @@ -468,7 +456,7 @@ class ViewDescriptor(object): def __init__(self, base_path, root, projections={}, select=[], exclude=[], link=default_view_link): self.base = base_path - self.root = root + self.root = spack.util.path.canonicalize_path(root) self.projections = projections self.select = select self.select_fn = lambda x: any(x.satisfies(s) for s in self.select) @@ -485,8 +473,13 @@ class ViewDescriptor(object): self.link == other.link]) def to_dict(self): - ret = {'root': self.root} + ret = syaml.syaml_dict([('root', self.root)]) if self.projections: + # projections guaranteed to be ordered dict if true-ish + # for python2.6, may be syaml or ruamel.yaml implementation + # so we have to check for both + types = (OrderedDict, syaml.syaml_dict, yaml.comments.CommentedMap) + assert isinstance(self.projections, types) ret['projections'] = self.projections if self.select: ret['select'] = self.select @@ -505,10 +498,66 @@ class ViewDescriptor(object): d.get('exclude', []), d.get('link', default_view_link)) - def view(self): - root = self.root - if not os.path.isabs(root): - root = os.path.normpath(os.path.join(self.base, self.root)) + @property + def _current_root(self): + if not os.path.exists(self.root): + return None + + root = os.readlink(self.root) + if os.path.isabs(root): + return root + + root_dir = os.path.dirname(self.root) + return os.path.join(root_dir, root) + + def _next_root(self, specs): + content_hash = self.content_hash(specs) + root_dir = os.path.dirname(self.root) + root_name = os.path.basename(self.root) + return os.path.join(root_dir, '._%s' % root_name, content_hash) + + def content_hash(self, specs): + d = syaml.syaml_dict([ + ('descriptor', self.to_dict()), + ('specs', [(spec.full_hash(), spec.prefix) for spec in sorted(specs)]) + ]) + contents = sjson.dump(d) + return spack.util.hash.b32_hash(contents) + + def get_projection_for_spec(self, spec): + """Get projection for spec relative to view root + + Getting the projection from the underlying root will get the temporary + projection. This gives the permanent projection relative to the root + symlink. + """ + view = self.view() + view_path = view.get_projection_for_spec(spec) + rel_path = os.path.relpath(view_path, self._current_root) + return os.path.join(self.root, rel_path) + + def view(self, new=None): + """ + Generate the FilesystemView object for this ViewDescriptor + + By default, this method returns a FilesystemView object rooted at the + current underlying root of this ViewDescriptor (self._current_root) + + Raise if new is None and there is no current view + + Arguments: + new (string or None): If a string, create a FilesystemView + rooted at that path. Default None. This should only be used to + regenerate the view, and cannot be used to access specs. + """ + root = self._current_root + if new: + root = new + if not root: + # This can only be hit if we write a future bug + msg = ("Attempting to get nonexistent view from environment. " + "View root is at %s" % self.root) + raise SpackEnvironmentViewError(msg) return YamlFilesystemView(root, spack.store.layout, ignore_conflicts=True, projections=self.projections) @@ -530,9 +579,10 @@ class ViewDescriptor(object): return True - def regenerate(self, all_specs, roots): + def specs_for_view(self, all_specs, roots): specs_for_view = [] specs = all_specs if self.link == 'all' else roots + for spec in specs: # The view does not store build deps, so if we want it to # recognize environment specs (which do store build deps), @@ -544,6 +594,10 @@ class ViewDescriptor(object): spec_copy._hash = spec._hash spec_copy._normal = spec._normal specs_for_view.append(spec_copy) + return specs_for_view + + def regenerate(self, all_specs, roots): + specs_for_view = self.specs_for_view(all_specs, roots) # regeneration queries the database quite a bit; this read # transaction ensures that we don't repeatedly lock/unlock. @@ -553,31 +607,56 @@ class ViewDescriptor(object): # To ensure there are no conflicts with packages being installed # that cannot be resolved or have repos that have been removed - # we always regenerate the view from scratch. We must first make - # sure the root directory exists for the very first time though. - root = self.root - if not os.path.isabs(root): - root = os.path.normpath(os.path.join(self.base, self.root)) - fs.mkdirp(root) - with fs.replace_directory_transaction(root): - view = self.view() - - view.clean() - specs_in_view = set(view.get_all_specs()) - tty.msg("Updating view at {0}".format(self.root)) - - rm_specs = specs_in_view - installed_specs_for_view - add_specs = installed_specs_for_view - specs_in_view - - # pass all_specs in, as it's expensive to read all the - # spec.yaml files twice. - view.remove_specs(*rm_specs, with_dependents=False, - all_specs=specs_in_view) - view.add_specs(*add_specs, with_dependencies=False) + # we always regenerate the view from scratch. + # We will do this by hashing the view contents and putting the view + # in a directory by hash, and then having a symlink to the real + # view in the root. The real root for a view at /dirname/basename + # will be /dirname/._basename_<hash>. + # This allows for atomic swaps when we update the view + + # cache the roots because the way we determine which is which does + # not work while we are updating + new_root = self._next_root(installed_specs_for_view) + old_root = self._current_root + + if new_root == old_root: + tty.debug("View at %s does not need regeneration." % self.root) + return + + # construct view at new_root + tty.msg("Updating view at {0}".format(self.root)) + + view = self.view(new=new_root) + fs.mkdirp(new_root) + view.add_specs(*installed_specs_for_view, + with_dependencies=False) + + # create symlink from tmpname to new_root + root_dirname = os.path.dirname(self.root) + tmp_symlink_name = os.path.join(root_dirname, '._view_link') + if os.path.exists(tmp_symlink_name): + os.unlink(tmp_symlink_name) + os.symlink(new_root, tmp_symlink_name) + + # mv symlink atomically over root symlink to old_root + if os.path.exists(self.root) and not os.path.islink(self.root): + msg = "Cannot create view: " + msg += "file already exists and is not a link: %s" % self.root + raise SpackEnvironmentViewError(msg) + os.rename(tmp_symlink_name, self.root) + + # remove old_root + if old_root and os.path.exists(old_root): + try: + shutil.rmtree(old_root) + except (IOError, OSError) as e: + msg = "Failed to remove old view at %s\n" % old_root + msg += str(e) + tty.warn(msg) class Environment(object): - def __init__(self, path, init_file=None, with_view=None): + def __init__(self, path, init_file=None, with_view=None, keep_relative=False): """Create a new environment. The environment can be optionally initialized with either a @@ -590,6 +669,10 @@ class Environment(object): with_view (str or bool): whether a view should be maintained for the environment. If the value is a string, it specifies the path to the view. + keep_relative (bool): if True, develop paths are copied verbatim + into the new environment file, otherwise they are made absolute + when the environment path is different from init_file's + directory. """ self.path = os.path.abspath(path) @@ -611,6 +694,13 @@ class Environment(object): self._set_user_specs_from_lockfile() else: self._read_manifest(f, raw_yaml=default_manifest_yaml) + + # Rewrite relative develop paths when initializing a new + # environment in a different location from the spack.yaml file. + if not keep_relative and hasattr(f, 'name') and \ + f.name.endswith('.yaml'): + init_file_dir = os.path.abspath(os.path.dirname(f.name)) + self._rewrite_relative_paths_on_relocation(init_file_dir) else: with lk.ReadTransaction(self.txlock): self._read() @@ -627,6 +717,27 @@ class Environment(object): # If with_view is None, then defer to the view settings determined by # the manifest file + def _rewrite_relative_paths_on_relocation(self, init_file_dir): + """When initializing the environment from a manifest file and we plan + to store the environment in a different directory, we have to rewrite + relative paths to absolute ones.""" + if init_file_dir == self.path: + return + + for name, entry in self.dev_specs.items(): + dev_path = entry['path'] + expanded_path = os.path.normpath(os.path.join( + init_file_dir, entry['path'])) + + # Skip if the expanded path is the same (e.g. when absolute) + if dev_path == expanded_path: + continue + + tty.debug("Expanding develop path for {0} to {1}".format( + name, expanded_path)) + + self.dev_specs[name]['path'] = expanded_path + def _re_read(self): """Reinitialize the environment object if it has been written (this may not be true if the environment was just created in this running @@ -1034,8 +1145,7 @@ class Environment(object): if clone: # "steal" the source code via staging API - abspath = path if os.path.isabs(path) else os.path.join( - self.path, path) + abspath = os.path.normpath(os.path.join(self.path, path)) stage = spec.package.stage stage.steal_source(abspath) @@ -1054,7 +1164,7 @@ class Environment(object): return True return False - def concretize(self, force=False): + def concretize(self, force=False, tests=False): """Concretize user_specs in this environment. Only concretizes specs that haven't been concretized yet unless @@ -1066,6 +1176,8 @@ class Environment(object): Arguments: force (bool): re-concretize ALL specs, even those that were already concretized + tests (bool or list or set): False to run no tests, True to test + all packages, or a list of package names to run tests for some Returns: List of specs that have been concretized. Each entry is a tuple of @@ -1079,14 +1191,14 @@ class Environment(object): # Pick the right concretization strategy if self.concretization == 'together': - return self._concretize_together() + return self._concretize_together(tests=tests) if self.concretization == 'separately': - return self._concretize_separately() + return self._concretize_separately(tests=tests) msg = 'concretization strategy not implemented [{0}]' raise SpackEnvironmentError(msg.format(self.concretization)) - def _concretize_together(self): + def _concretize_together(self, tests=False): """Concretization strategy that concretizes all the specs in the same DAG. """ @@ -1119,14 +1231,13 @@ class Environment(object): self.specs_by_hash = {} concrete_specs = spack.concretize.concretize_specs_together( - *self.user_specs - ) + *self.user_specs, tests=tests) concretized_specs = [x for x in zip(self.user_specs, concrete_specs)] for abstract, concrete in concretized_specs: self._add_concrete_spec(abstract, concrete) return concretized_specs - def _concretize_separately(self): + def _concretize_separately(self, tests=False): """Concretization strategy that concretizes separately one user spec after the other. """ @@ -1149,12 +1260,12 @@ class Environment(object): for uspec, uspec_constraints in zip( self.user_specs, self.user_specs.specs_as_constraints): if uspec not in old_concretized_user_specs: - concrete = _concretize_from_constraints(uspec_constraints) + concrete = _concretize_from_constraints(uspec_constraints, tests=tests) self._add_concrete_spec(uspec, concrete) concretized_specs.append((uspec, concrete)) return concretized_specs - def concretize_and_add(self, user_spec, concrete_spec=None): + def concretize_and_add(self, user_spec, concrete_spec=None, tests=False): """Concretize and add a single spec to the environment. Concretize the provided ``user_spec`` and add it along with the @@ -1177,7 +1288,7 @@ class Environment(object): spec = Spec(user_spec) if self.add(spec): - concrete = concrete_spec or spec.concretized() + concrete = concrete_spec or spec.concretized(tests=tests) self._add_concrete_spec(spec, concrete) else: # spec might be in the user_specs, but not installed. @@ -1187,7 +1298,7 @@ class Environment(object): ) concrete = self.specs_by_hash.get(spec.build_hash()) if not concrete: - concrete = spec.concretized() + concrete = spec.concretized(tests=tests) self._add_concrete_spec(spec, concrete) return concrete @@ -1248,19 +1359,36 @@ class Environment(object): def _env_modifications_for_default_view(self, reverse=False): all_mods = spack.util.environment.EnvironmentModifications() + visited = set() + errors = [] - for _, spec in self.concretized_specs(): - if spec in self.default_view and spec.package.installed: - try: - mods = uenv.environment_modifications_for_spec( - spec, self.default_view) - except Exception as e: - msg = ("couldn't get environment settings for %s" - % spec.format("{name}@{version} /{hash:7}")) - errors.append((msg, str(e))) - continue + for _, root_spec in self.concretized_specs(): + if root_spec in self.default_view and root_spec.package.installed: + for spec in root_spec.traverse(deptype='run', root=True): + if spec.name in visited: + # It is expected that only one instance of the package + # can be added to the environment - do not attempt to + # add multiple. + tty.debug( + "Not adding {0} to shell modifications: " + "this package has already been added".format( + spec.format("{name}/{hash:7}") + ) + ) + continue + else: + visited.add(spec.name) - all_mods.extend(mods.reversed() if reverse else mods) + try: + mods = uenv.environment_modifications_for_spec( + spec, self.default_view) + except Exception as e: + msg = ("couldn't get environment settings for %s" + % spec.format("{name}@{version} /{hash:7}")) + errors.append((msg, str(e))) + continue + + all_mods.extend(mods.reversed() if reverse else mods) return all_mods, errors @@ -1412,20 +1540,31 @@ class Environment(object): args (Namespace): argparse namespace with command arguments install_args (dict): keyword install arguments """ + self.install_specs(None, args=args, **install_args) + + def install_specs(self, specs=None, args=None, **install_args): + from spack.installer import PackageInstaller + tty.debug('Assessing installation status of environment packages') # If "spack install" is invoked repeatedly for a large environment # where all specs are already installed, the operation can take # a large amount of time due to repeatedly acquiring and releasing # locks, this does an initial check across all specs within a single - # DB read transaction to reduce time spent in this case. - specs_to_install = self.uninstalled_specs() + # DB read transaction to reduce time spent in this case. In the next + # three lines we remove any already-installed root specs from the list + # to install. However, uninstalled_specs() only considers root specs, + # so this will allow dep specs to be unnecessarily re-installed. + uninstalled_roots = self.uninstalled_specs() + specs_to_install = specs or uninstalled_roots + specs_to_install = [s for s in specs_to_install + if s not in self.roots() or s in uninstalled_roots] if not specs_to_install: tty.msg('All of the packages are already installed') return - tty.debug('Processing {0} uninstalled specs' - .format(len(specs_to_install))) + tty.debug('Processing {0} uninstalled specs'.format( + len(specs_to_install))) install_args['overwrite'] = install_args.get( 'overwrite', []) + self._get_overwrite_specs() @@ -1545,7 +1684,7 @@ class Environment(object): if abstract) if len(root_matches) == 1: - return root_matches[0][1] + return list(root_matches.items())[0][0] # More than one spec matched, and either multiple roots matched or # none of the matches were roots @@ -1894,7 +2033,7 @@ def display_specs(concretized_specs): print('') -def _concretize_from_constraints(spec_constraints): +def _concretize_from_constraints(spec_constraints, tests=False): # Accept only valid constraints from list and concretize spec # Get the named spec even if out of order root_spec = [s for s in spec_constraints if s.name] @@ -1913,7 +2052,7 @@ def _concretize_from_constraints(spec_constraints): if c not in invalid_constraints: s.constrain(c) try: - return s.concretized() + return s.concretized(tests=tests) except spack.spec.InvalidDependencyError as e: invalid_deps_string = ['^' + d for d in e.invalid_deps] invalid_deps = [c for c in spec_constraints @@ -2050,3 +2189,7 @@ def is_latest_format(manifest): class SpackEnvironmentError(spack.error.SpackError): """Superclass for all errors to do with Spack environments.""" + + +class SpackEnvironmentViewError(SpackEnvironmentError): + """Class for errors regarding view generation.""" diff --git a/lib/spack/spack/error.py b/lib/spack/spack/error.py index f79744f6a5..fa66a35b0e 100644 --- a/lib/spack/spack/error.py +++ b/lib/spack/spack/error.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/extensions.py b/lib/spack/spack/extensions.py index 4358dcd52f..9ea7373b8c 100644 --- a/lib/spack/spack/extensions.py +++ b/lib/spack/spack/extensions.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/fetch_strategy.py b/lib/spack/spack/fetch_strategy.py index 1b94206fbf..f885980ccd 100644 --- a/lib/spack/spack/fetch_strategy.py +++ b/lib/spack/spack/fetch_strategy.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -29,6 +29,7 @@ import os.path import re import shutil import sys +from typing import Optional, List # novm import llnl.util.tty as tty import six @@ -92,11 +93,12 @@ class FetchStrategy(object): #: The URL attribute must be specified either at the package class #: level, or as a keyword argument to ``version()``. It is used to #: distinguish fetchers for different versions in the package DSL. - url_attr = None + url_attr = None # type: Optional[str] #: Optional attributes can be used to distinguish fetchers when : #: classes have multiple ``url_attrs`` at the top-level. - optional_attrs = [] # optional attributes in version() args. + # optional attributes in version() args. + optional_attrs = [] # type: List[str] def __init__(self, **kwargs): # The stage is initialized late, so that fetch strategies can be @@ -290,7 +292,15 @@ class URLFetchStrategy(FetchStrategy): @property def candidate_urls(self): - return [self.url] + (self.mirrors or []) + urls = [] + + for url in [self.url] + (self.mirrors or []): + if url.startswith('file://'): + path = urllib_parse.quote(url[len('file://'):]) + url = 'file://' + path + urls.append(url) + + return urls @_needs_stage def fetch(self): @@ -422,7 +432,7 @@ class URLFetchStrategy(FetchStrategy): warn_content_type_mismatch(self.archive_file or "the archive") return partial_file, save_file - @property + @property # type: ignore # decorated properties unsupported in mypy @_needs_stage def archive_file(self): """Path to the source archive within this stage directory.""" @@ -465,6 +475,8 @@ class URLFetchStrategy(FetchStrategy): tarball_container = os.path.join(self.stage.path, "spack-expanded-archive") + # Below we assume that the command to decompress expand the + # archive in the current working directory mkdirp(tarball_container) with working_dir(tarball_container): decompress(self.archive_file) @@ -776,6 +788,12 @@ class GitFetchStrategy(VCSFetchStrategy): if not self._git: self._git = which('git', required=True) + # Disable advice for a quieter fetch + # https://github.com/git/git/blob/master/Documentation/RelNotes/1.7.2.txt + if self.git_version >= Version('1.7.2'): + self._git.add_default_arg('-c') + self._git.add_default_arg('advice.detachedHead=false') + # If the user asked for insecure fetching, make that work # with git as well. if not spack.config.get('config:verify_ssl'): diff --git a/lib/spack/spack/filesystem_view.py b/lib/spack/spack/filesystem_view.py index b6501064b9..232223b69f 100644 --- a/lib/spack/spack/filesystem_view.py +++ b/lib/spack/spack/filesystem_view.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/graph.py b/lib/spack/spack/graph.py index b41c923a16..87ad51e030 100644 --- a/lib/spack/spack/graph.py +++ b/lib/spack/spack/graph.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -62,17 +62,16 @@ def topological_sort(spec, reverse=False, deptype='all'): """ deptype = canonical_deptype(deptype) - if not reverse: - parents = lambda s: s.dependents() - children = lambda s: s.dependencies() - else: - parents = lambda s: s.dependencies() - children = lambda s: s.dependents() - # Work on a copy so this is nondestructive. spec = spec.copy(deps=deptype) nodes = spec.index(deptype=deptype) + parents = lambda s: [p for p in s.dependents() if p.name in nodes] + children = lambda s: s.dependencies() + + if reverse: + parents, children = children, parents + topo_order = [] par = dict((name, parents(nodes[name])) for name in nodes.keys()) remaining = [name for name in nodes.keys() if not parents(nodes[name])] diff --git a/lib/spack/spack/hash_types.py b/lib/spack/spack/hash_types.py index 9404076e81..554665e24a 100644 --- a/lib/spack/spack/hash_types.py +++ b/lib/spack/spack/hash_types.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -18,6 +18,9 @@ class SpecHashDescriptor(object): We currently use different hashes for different use cases. """ + + hash_types = ('_dag_hash', '_build_hash', '_full_hash') + def __init__(self, deptype=('link', 'run'), package_hash=False, attr=None): self.deptype = dp.canonical_deptype(deptype) self.package_hash = package_hash @@ -35,5 +38,5 @@ build_hash = SpecHashDescriptor( #: Full hash used in build pipelines to determine when to rebuild packages. -full_hash = SpecHashDescriptor(deptype=('link', 'run'), package_hash=True, - attr='_full_hash') +full_hash = SpecHashDescriptor( + deptype=('build', 'link', 'run'), package_hash=True, attr='_full_hash') diff --git a/lib/spack/spack/hooks/__init__.py b/lib/spack/spack/hooks/__init__.py index 5a5a04b49f..3c15b978d3 100644 --- a/lib/spack/spack/hooks/__init__.py +++ b/lib/spack/spack/hooks/__init__.py @@ -1,25 +1,31 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) """This package contains modules with hooks for various stages in the - - Spack install process. You can add modules here and they'll be - executed by package at various times during the package lifecycle. - - Each hook is just a function that takes a package as a parameter. - Hooks are not executed in any particular order. - - Currently the following hooks are supported: - - * pre_install(spec) - * post_install(spec) - * pre_uninstall(spec) - * post_uninstall(spec) - - This can be used to implement support for things like module - systems (e.g. modules, lmod, etc.) or to add other custom - features. +Spack install process. You can add modules here and they'll be +executed by package at various times during the package lifecycle. + +Each hook is just a function that takes a package as a parameter. +Hooks are not executed in any particular order. + +Currently the following hooks are supported: + + * pre_install(spec) + * post_install(spec) + * pre_uninstall(spec) + * post_uninstall(spec) + * on_install_start(spec) + * on_install_success(spec) + * on_install_failure(spec) + * on_phase_success(pkg, phase_name, log_file) + * on_phase_error(pkg, phase_name, log_file) + * on_phase_error(pkg, phase_name, log_file) + * on_analyzer_save(pkg, result) + +This can be used to implement support for things like module +systems (e.g. modules, lmod, etc.) or to add other custom +features. """ import llnl.util.lang import spack.paths @@ -68,8 +74,20 @@ class _HookRunner(object): hook(*args, **kwargs) +# pre/post install and run by the install subprocess pre_install = _HookRunner('pre_install') post_install = _HookRunner('post_install') +# These hooks are run within an install subprocess pre_uninstall = _HookRunner('pre_uninstall') post_uninstall = _HookRunner('post_uninstall') +on_phase_success = _HookRunner('on_phase_success') +on_phase_error = _HookRunner('on_phase_error') + +# These are hooks in installer.py, before starting install subprocess +on_install_start = _HookRunner('on_install_start') +on_install_success = _HookRunner('on_install_success') +on_install_failure = _HookRunner('on_install_failure') + +# Analyzer hooks +on_analyzer_save = _HookRunner('on_analyzer_save') diff --git a/lib/spack/spack/hooks/extensions.py b/lib/spack/spack/hooks/extensions.py index 05fce302ac..a639df6181 100644 --- a/lib/spack/spack/hooks/extensions.py +++ b/lib/spack/spack/hooks/extensions.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/hooks/licensing.py b/lib/spack/spack/hooks/licensing.py index d4e5376cda..8e789d9635 100644 --- a/lib/spack/spack/hooks/licensing.py +++ b/lib/spack/spack/hooks/licensing.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/hooks/module_file_generation.py b/lib/spack/spack/hooks/module_file_generation.py index ba30561d86..363654efc4 100644 --- a/lib/spack/spack/hooks/module_file_generation.py +++ b/lib/spack/spack/hooks/module_file_generation.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/hooks/monitor.py b/lib/spack/spack/hooks/monitor.py new file mode 100644 index 0000000000..8d161b72fa --- /dev/null +++ b/lib/spack/spack/hooks/monitor.py @@ -0,0 +1,73 @@ +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +import llnl.util.tty as tty +import spack.monitor + + +def on_install_start(spec): + """On start of an install, we want to ping the server if it exists + """ + if not spack.monitor.cli: + return + + tty.debug("Running on_install_start for %s" % spec) + build_id = spack.monitor.cli.new_build(spec) + tty.verbose("Build created with id %s" % build_id) + + +def on_install_success(spec): + """On the success of an install (after everything is complete) + """ + if not spack.monitor.cli: + return + + tty.debug("Running on_install_success for %s" % spec) + result = spack.monitor.cli.update_build(spec, status="SUCCESS") + tty.verbose(result.get('message')) + + +def on_install_failure(spec): + """Triggered on failure of an install + """ + if not spack.monitor.cli: + return + + tty.debug("Running on_install_failure for %s" % spec) + result = spack.monitor.cli.fail_task(spec) + tty.verbose(result.get('message')) + + +def on_phase_success(pkg, phase_name, log_file): + """Triggered on a phase success + """ + if not spack.monitor.cli: + return + + tty.debug("Running on_phase_success %s, phase %s" % (pkg.name, phase_name)) + result = spack.monitor.cli.send_phase(pkg, phase_name, log_file, "SUCCESS") + tty.verbose(result.get('message')) + + +def on_phase_error(pkg, phase_name, log_file): + """Triggered on a phase error + """ + if not spack.monitor.cli: + return + + tty.debug("Running on_phase_error %s, phase %s" % (pkg.name, phase_name)) + result = spack.monitor.cli.send_phase(pkg, phase_name, log_file, "ERROR") + tty.verbose(result.get('message')) + + +def on_analyzer_save(pkg, result): + """given a package and a result, if we have a spack monitor, upload + the result to it. + """ + if not spack.monitor.cli: + return + + # This hook runs after a save result + spack.monitor.cli.send_analyze_metadata(pkg, result) diff --git a/lib/spack/spack/hooks/permissions_setters.py b/lib/spack/spack/hooks/permissions_setters.py index 0a3c2acf21..d640018515 100644 --- a/lib/spack/spack/hooks/permissions_setters.py +++ b/lib/spack/spack/hooks/permissions_setters.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/hooks/sbang.py b/lib/spack/spack/hooks/sbang.py index 687cf651f4..7dc00165d5 100644 --- a/lib/spack/spack/hooks/sbang.py +++ b/lib/spack/spack/hooks/sbang.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/hooks/write_install_manifest.py b/lib/spack/spack/hooks/write_install_manifest.py index 5b678fed88..0fb6acdf35 100644 --- a/lib/spack/spack/hooks/write_install_manifest.py +++ b/lib/spack/spack/hooks/write_install_manifest.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/install_test.py b/lib/spack/spack/install_test.py index 6c2c095a2e..ae50a49215 100644 --- a/lib/spack/spack/install_test.py +++ b/lib/spack/spack/install_test.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/installer.py b/lib/spack/spack/installer.py index 15bc23738b..4ee56db26d 100644 --- a/lib/spack/spack/installer.py +++ b/lib/spack/spack/installer.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -46,6 +46,7 @@ import spack.binary_distribution as binary_distribution import spack.compilers import spack.error import spack.hooks +import spack.monitor import spack.package import spack.package_prefs as prefs import spack.repo @@ -392,7 +393,7 @@ def _try_install_from_binary_cache(pkg, explicit, unsigned=False, pkg_id = package_id(pkg) tty.debug('Searching for binary cache of {0}'.format(pkg_id)) matches = binary_distribution.get_mirrors_for_spec( - pkg.spec, force=False, full_hash_match=full_hash_match) + pkg.spec, full_hash_match=full_hash_match) if not matches: return False @@ -412,6 +413,25 @@ def clear_failures(): spack.store.db.clear_all_failures() +def combine_phase_logs(phase_log_files, log_path): + """ + Read set or list of logs and combine them into one file. + + Each phase will produce it's own log, so this function aims to cat all the + separate phase log output files into the pkg.log_path. It is written + generally to accept some list of files, and a log path to combine them to. + + Args: + phase_log_files (list): a list or iterator of logs to combine + log_path (path): the path to combine them to + """ + + with open(log_path, 'w') as log_file: + for phase_log_file in phase_log_files: + with open(phase_log_file, 'r') as phase_log: + log_file.write(phase_log.read()) + + def dump_packages(spec, path): """ Dump all package information for a spec and its dependencies. @@ -521,6 +541,12 @@ def log(pkg): # Archive the whole stdout + stderr for the package fs.install(pkg.log_path, pkg.install_log_path) + # Archive all phase log paths + for phase_log in pkg.phase_log_files: + log_file = os.path.basename(phase_log) + log_file = os.path.join(os.path.dirname(packages_dir), log_file) + fs.install(phase_log, log_file) + # Archive the environment used for the build fs.install(pkg.env_path, pkg.install_env_path) @@ -1250,8 +1276,8 @@ class PackageInstaller(object): task (BuildTask): the installation build task for a package """ if task.status not in [STATUS_INSTALLED, STATUS_INSTALLING]: - tty.msg('{0} {1}'.format(install_msg(task.pkg_id, self.pid), - 'in progress by another process')) + tty.debug('{0} {1}'.format(install_msg(task.pkg_id, self.pid), + 'in progress by another process')) new_task = task.next_attempt(self.installed) new_task.status = STATUS_INSTALLING @@ -1260,6 +1286,7 @@ class PackageInstaller(object): def _setup_install_dir(self, pkg): """ Create and ensure proper access controls for the install directory. + Write a small metadata file with the current spack environment. Args: pkg (Package): the package to be built and installed @@ -1285,6 +1312,9 @@ class PackageInstaller(object): # Ensure the metadata path exists as well fs.mkdirp(spack.store.layout.metadata_path(pkg.spec), mode=perms) + # Always write host environment - we assume this can change + spack.store.layout.write_host_environment(pkg.spec) + def _update_failed(self, task, mark=False, exc=None): """ Update the task and transitive dependents as failed; optionally mark @@ -1388,8 +1418,8 @@ class PackageInstaller(object): Args: pkg (Package): the package to be built and installed""" - self._init_queue() + self._init_queue() fail_fast_err = 'Terminating after first install failure' single_explicit_spec = len(self.build_requests) == 1 failed_explicits = [] @@ -1400,12 +1430,12 @@ class PackageInstaller(object): if task is None: continue + spack.hooks.on_install_start(task.request.pkg.spec) install_args = task.request.install_args keep_prefix = install_args.get('keep_prefix') pkg, pkg_id, spec = task.pkg, task.pkg_id, task.pkg.spec tty.verbose('Processing {0}: task={1}'.format(pkg_id, task)) - # Ensure that the current spec has NO uninstalled dependencies, # which is assumed to be reflected directly in its priority. # @@ -1423,6 +1453,10 @@ class PackageInstaller(object): tty.warn('{0} does NOT actually have any uninstalled deps' ' left'.format(pkg_id)) dep_str = 'dependencies' if task.priority > 1 else 'dependency' + + # Hook to indicate task failure, but without an exception + spack.hooks.on_install_failure(task.request.pkg.spec) + raise InstallError( 'Cannot proceed with {0}: {1} uninstalled {2}: {3}' .format(pkg_id, task.priority, dep_str, @@ -1442,6 +1476,11 @@ class PackageInstaller(object): tty.warn('{0} failed to install'.format(pkg_id)) self._update_failed(task) + # Mark that the package failed + # TODO: this should also be for the task.pkg, but we don't + # model transitive yet. + spack.hooks.on_install_failure(task.request.pkg.spec) + if self.fail_fast: raise InstallError(fail_fast_err) @@ -1543,7 +1582,7 @@ class PackageInstaller(object): (stop_before_phase is None and last_phase is None) except spack.directory_layout.InstallDirectoryAlreadyExistsError \ - as err: + as exc: tty.debug('Install prefix for {0} exists, keeping {1} in ' 'place.'.format(pkg.name, pkg.prefix)) self._update_installed(task) @@ -1551,10 +1590,11 @@ class PackageInstaller(object): # Only terminate at this point if a single build request was # made. if task.explicit and single_explicit_spec: + spack.hooks.on_install_failure(task.request.pkg.spec) raise if task.explicit: - exists_errors.append((pkg_id, str(err))) + exists_errors.append((pkg_id, str(exc))) except KeyboardInterrupt as exc: # The build has been terminated with a Ctrl-C so terminate @@ -1562,10 +1602,12 @@ class PackageInstaller(object): err = 'Failed to install {0} due to {1}: {2}' tty.error(err.format(pkg.name, exc.__class__.__name__, str(exc))) + spack.hooks.on_install_failure(task.request.pkg.spec) raise except (Exception, SystemExit) as exc: self._update_failed(task, True, exc) + spack.hooks.on_install_failure(task.request.pkg.spec) # Best effort installs suppress the exception and mark the # package as a failure. @@ -1610,15 +1652,22 @@ class PackageInstaller(object): self._cleanup_all_tasks() # Ensure we properly report if one or more explicit specs failed - if exists_errors or failed_explicits: + # or were not installed when should have been. + missing = [request.pkg_id for request in self.build_requests if + request.install_args.get('install_package') and + request.pkg_id not in self.installed] + if exists_errors or failed_explicits or missing: for pkg_id, err in exists_errors: tty.error('{0}: {1}'.format(pkg_id, err)) for pkg_id, err in failed_explicits: tty.error('{0}: {1}'.format(pkg_id, err)) + for pkg_id in missing: + tty.error('{0}: Package was not installed'.format(pkg_id)) + raise InstallError('Installation request failed. Refer to ' - 'recent errors for specific package(s).') + 'reported errors for failing package(s).') def build_process(pkg, kwargs): @@ -1656,6 +1705,7 @@ def build_process(pkg, kwargs): echo = spack.package.PackageBase._verbose pkg.stage.keep = keep_stage + with pkg.stage: # Run the pre-install hook in the child process after # the directory is created. @@ -1673,6 +1723,7 @@ def build_process(pkg, kwargs): # Do the real install in the source directory. with fs.working_dir(pkg.stage.source_path): + # Save the build environment in a file before building. dump_environment(pkg.env_path) @@ -1693,25 +1744,48 @@ def build_process(pkg, kwargs): debug_level = tty.debug_level() # Spawn a daemon that reads from a pipe and redirects - # everything to log_path - with log_output(pkg.log_path, echo, True, - env=unmodified_env) as logger: + # everything to log_path, and provide the phase for logging + for i, (phase_name, phase_attr) in enumerate(zip( + pkg.phases, pkg._InstallPhase_phases)): - for phase_name, phase_attr in zip( - pkg.phases, pkg._InstallPhase_phases): + # Keep a log file for each phase + log_dir = os.path.dirname(pkg.log_path) + log_file = "spack-build-%02d-%s-out.txt" % ( + i + 1, phase_name.lower() + ) + log_file = os.path.join(log_dir, log_file) - with logger.force_echo(): - inner_debug_level = tty.debug_level() - tty.set_debug(debug_level) - tty.msg("{0} Executing phase: '{1}'" - .format(pre, phase_name)) - tty.set_debug(inner_debug_level) + try: + # DEBUGGING TIP - to debug this section, insert an IPython + # embed here, and run the sections below without log capture + with log_output(log_file, echo, True, + env=unmodified_env) as logger: - # Redirect stdout and stderr to daemon pipe - phase = getattr(pkg, phase_attr) - phase(pkg.spec, pkg.prefix) + with logger.force_echo(): + inner_debug_level = tty.debug_level() + tty.set_debug(debug_level) + tty.msg("{0} Executing phase: '{1}'" + .format(pre, phase_name)) + tty.set_debug(inner_debug_level) - echo = logger.echo + # Redirect stdout and stderr to daemon pipe + phase = getattr(pkg, phase_attr) + + # Catch any errors to report to logging + + phase(pkg.spec, pkg.prefix) + spack.hooks.on_phase_success(pkg, phase_name, log_file) + + except BaseException: + combine_phase_logs(pkg.phase_log_files, pkg.log_path) + spack.hooks.on_phase_error(pkg, phase_name, log_file) + raise + + # We assume loggers share echo True/False + echo = logger.echo + + # After log, we can get all output/error files from the package stage + combine_phase_logs(pkg.phase_log_files, pkg.log_path) log(pkg) # Run post install hooks before build stage is removed. @@ -1721,13 +1795,15 @@ def build_process(pkg, kwargs): pkg._total_time = time.time() - start_time build_time = pkg._total_time - pkg._fetch_time - tty.debug('{0} Successfully installed {1}' - .format(pre, pkg_id), - 'Fetch: {0}. Build: {1}. Total: {2}.' - .format(_hms(pkg._fetch_time), _hms(build_time), - _hms(pkg._total_time))) + tty.msg('{0} Successfully installed {1}'.format(pre, pkg_id), + 'Fetch: {0}. Build: {1}. Total: {2}.' + .format(_hms(pkg._fetch_time), _hms(build_time), + _hms(pkg._total_time))) _print_installed_pkg(pkg.prefix) + # Send final status that install is successful + spack.hooks.on_install_success(pkg.spec) + # preserve verbosity across runs return echo @@ -2013,7 +2089,8 @@ class BuildRequest(object): (tuple) required dependency type(s) for the package """ deptypes = ['link', 'run'] - if not self.install_args.get('cache_only'): + include_build_deps = self.install_args.get('include_build_deps') + if not self.install_args.get('cache_only') or include_build_deps: deptypes.append('build') if self.run_tests(pkg): deptypes.append('test') diff --git a/lib/spack/spack/main.py b/lib/spack/spack/main.py index f4265a5d68..6359d6dacf 100644 --- a/lib/spack/spack/main.py +++ b/lib/spack/spack/main.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -40,7 +40,6 @@ import spack.util.path import spack.util.executable as exe from spack.error import SpackError - #: names of profile statistics stat_names = pstats.Stats.sort_arg_dict_default @@ -355,10 +354,14 @@ def make_argument_parser(**kwargs): dest='help', action='store_const', const='long', default=None, help="show help for all commands (same as spack help --all)") parser.add_argument( - '--color', action='store', default='auto', + '--color', action='store', + default=os.environ.get('SPACK_COLOR', 'auto'), choices=('always', 'never', 'auto'), help="when to colorize output (default: auto)") parser.add_argument( + '-c', '--config', default=None, action="append", dest="config_vars", + help="add one or more custom, one off config settings.") + parser.add_argument( '-C', '--config-scope', dest='config_scopes', action='append', metavar='DIR', help="add a custom configuration scope") parser.add_argument( @@ -413,6 +416,7 @@ def make_argument_parser(**kwargs): help="print additional output during builds") parser.add_argument( '--stacktrace', action='store_true', + default='SPACK_STACKTRACE' in os.environ, help="add stacktraces to all printed statements") parser.add_argument( '-V', '--version', action='store_true', @@ -463,6 +467,10 @@ def setup_main_options(args): tty.warn("You asked for --insecure. Will NOT check SSL certificates.") spack.config.set('config:verify_ssl', False, scope='command_line') + # Use the spack config command to handle parsing the config strings + for config_var in (args.config_vars or []): + spack.config.add(fullpath=config_var, scope="command_line") + # when to use color (takes always, auto, or never) color.set_color_when(args.color) @@ -522,6 +530,8 @@ class SpackCommand(object): Keyword Args: fail_on_error (optional bool): Don't raise an exception on error + global_args (optional list): List of global spack arguments: + simulates ``spack [global_args] [command] [*argv]`` Returns: (str): combined output and error as a string @@ -534,8 +544,10 @@ class SpackCommand(object): self.returncode = None self.error = None + prepend = kwargs['global_args'] if 'global_args' in kwargs else [] + args, unknown = self.parser.parse_known_args( - [self.command_name] + list(argv)) + prepend + [self.command_name] + list(argv)) fail_on_error = kwargs.get('fail_on_error', True) diff --git a/lib/spack/spack/mirror.py b/lib/spack/spack/mirror.py index 045ca5ffec..cdebc7fd4e 100644 --- a/lib/spack/spack/mirror.py +++ b/lib/spack/spack/mirror.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -23,9 +23,9 @@ import ruamel.yaml.error as yaml_error from ordereddict_backport import OrderedDict -try: +if sys.version_info >= (3, 5): from collections.abc import Mapping # novm -except ImportError: +else: from collections import Mapping import llnl.util.tty as tty diff --git a/lib/spack/spack/mixins.py b/lib/spack/spack/mixins.py index 6da12e4dce..be13e3ea3a 100644 --- a/lib/spack/spack/mixins.py +++ b/lib/spack/spack/mixins.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -8,6 +8,12 @@ package. """ import collections import os +import sys +from typing import Callable, DefaultDict, Dict, List # novm +if sys.version_info >= (3, 5): + CallbackDict = DefaultDict[str, List[Callable]] +else: + CallbackDict = None import llnl.util.filesystem @@ -26,12 +32,12 @@ class PackageMixinsMeta(type): gets implicitly attached to the package class by calling the mixin. """ - _methods_to_be_added = {} - _add_method_before = collections.defaultdict(list) - _add_method_after = collections.defaultdict(list) + _methods_to_be_added = {} # type: Dict[str, Callable] + _add_method_before = collections.defaultdict(list) # type: CallbackDict + _add_method_after = collections.defaultdict(list) # type: CallbackDict @staticmethod - def register_method_before(fn, phase): + def register_method_before(fn, phase): # type: (Callable, str) -> None """Registers a method to be run before a certain phase. Args: @@ -42,7 +48,7 @@ class PackageMixinsMeta(type): PackageMixinsMeta._add_method_before[phase].append(fn) @staticmethod - def register_method_after(fn, phase): + def register_method_after(fn, phase): # type: (Callable, str) -> None """Registers a method to be run after a certain phase. Args: diff --git a/lib/spack/spack/modules/__init__.py b/lib/spack/spack/modules/__init__.py index 2fb4d94b05..b65260277f 100644 --- a/lib/spack/spack/modules/__init__.py +++ b/lib/spack/spack/modules/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/modules/common.py b/lib/spack/spack/modules/common.py index cbc064520c..ed843e03af 100644 --- a/lib/spack/spack/modules/common.py +++ b/lib/spack/spack/modules/common.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -34,6 +34,7 @@ import datetime import inspect import os.path import re +from typing import Optional # novm import llnl.util.filesystem from llnl.util.lang import dedupe @@ -540,7 +541,7 @@ class BaseFileLayout(object): """ #: This needs to be redefined - extension = None + extension = None # type: Optional[str] def __init__(self, configuration): self.conf = configuration diff --git a/lib/spack/spack/modules/lmod.py b/lib/spack/spack/modules/lmod.py index 80f6933063..bb4476a7b0 100644 --- a/lib/spack/spack/modules/lmod.py +++ b/lib/spack/spack/modules/lmod.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -8,6 +8,7 @@ import os.path import llnl.util.lang as lang import itertools import collections +from typing import Dict, Any # novm import spack.config import spack.compilers @@ -26,7 +27,7 @@ def configuration(): #: Caches the configuration {spec_hash: configuration} -configuration_registry = {} +configuration_registry = {} # type: Dict[str, Any] def make_configuration(spec): diff --git a/lib/spack/spack/modules/tcl.py b/lib/spack/spack/modules/tcl.py index 0efc6332fe..e1d2ac7fe3 100644 --- a/lib/spack/spack/modules/tcl.py +++ b/lib/spack/spack/modules/tcl.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -8,6 +8,7 @@ non-hierarchical modules. """ import os.path import string +from typing import Dict, Any # novm import llnl.util.tty as tty @@ -24,7 +25,7 @@ def configuration(): #: Caches the configuration {spec_hash: configuration} -configuration_registry = {} +configuration_registry = {} # type: Dict[str, Any] def make_configuration(spec): diff --git a/lib/spack/spack/monitor.py b/lib/spack/spack/monitor.py new file mode 100644 index 0000000000..5d6246ebf1 --- /dev/null +++ b/lib/spack/spack/monitor.py @@ -0,0 +1,522 @@ +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +"""Interact with a Spack Monitor Service. Derived from +https://github.com/spack/spack-monitor/blob/main/script/spackmoncli.py +""" + +import base64 +import os +import re + +try: + from urllib.request import Request, urlopen + from urllib.error import URLError +except ImportError: + from urllib2 import urlopen, Request, URLError # type: ignore # novm + +import spack +import spack.hash_types as ht +import spack.main +import spack.store +import spack.util.spack_json as sjson +import spack.util.spack_yaml as syaml +import llnl.util.tty as tty +from copy import deepcopy + + +# A global client to instantiate once +cli = None + + +def get_client(host, prefix="ms1", disable_auth=False, allow_fail=False, tags=None): + """ + Get a monitor client for a particular host and prefix. + + If the client is not running, we exit early, unless allow_fail is set + to true, indicating that we should continue the build even if the + server is not present. Note that this client is defined globally as "cli" + so we can istantiate it once (checking for credentials, etc.) and then + always have access to it via spack.monitor.cli. Also note that + typically, we call the monitor by way of hooks in spack.hooks.monitor. + So if you want the monitor to have a new interaction with some part of + the codebase, it's recommended to write a hook first, and then have + the monitor use it. + """ + global cli + cli = SpackMonitorClient(host=host, prefix=prefix, allow_fail=allow_fail, + tags=tags) + + # If we don't disable auth, environment credentials are required + if not disable_auth: + cli.require_auth() + + # We will exit early if the monitoring service is not running + info = cli.service_info() + + # If we allow failure, the response will be done + if info: + tty.debug("%s v.%s has status %s" % ( + info['id'], + info['version'], + info['status']) + ) + return cli + + else: + tty.debug("spack-monitor server not found, continuing as allow_fail is True.") + + +def get_monitor_group(subparser): + """ + Retrieve the monitor group for the argument parser. + + Since the monitor group is shared between commands, we provide a common + function to generate the group for it. The user can pass the subparser, and + the group is added, and returned. + """ + # Monitoring via https://github.com/spack/spack-monitor + monitor_group = subparser.add_argument_group() + monitor_group.add_argument( + '--monitor', action='store_true', dest='use_monitor', default=False, + help="interact with a montor server during builds.") + monitor_group.add_argument( + '--monitor-no-auth', action='store_true', dest='monitor_disable_auth', + default=False, help="the monitoring server does not require auth.") + monitor_group.add_argument( + '--monitor-tags', dest='monitor_tags', default=None, + help="One or more (comma separated) tags for a build.") + monitor_group.add_argument( + '--monitor-keep-going', action='store_true', dest='monitor_keep_going', + default=False, help="continue the build if a request to monitor fails.") + monitor_group.add_argument( + '--monitor-host', dest='monitor_host', default="http://127.0.0.1", + help="If using a monitor, customize the host.") + monitor_group.add_argument( + '--monitor-prefix', dest='monitor_prefix', default="ms1", + help="The API prefix for the monitor service.") + return monitor_group + + +class SpackMonitorClient: + """Client to interact with a spack monitor server. + + We require the host url, along with the prefix to discover the + service_info endpoint. If allow_fail is set to True, we will not exit + on error with tty.die given that a request is not successful. The spack + version is one of the fields to uniquely identify a spec, so we add it + to the client on init. + """ + + def __init__(self, host=None, prefix="ms1", allow_fail=False, tags=None): + self.host = host or "http://127.0.0.1" + self.baseurl = "%s/%s" % (self.host, prefix.strip("/")) + self.token = os.environ.get("SPACKMON_TOKEN") + self.username = os.environ.get("SPACKMON_USER") + self.headers = {} + self.allow_fail = allow_fail + self.spack_version = spack.main.get_version() + self.capture_build_environment() + self.tags = tags + + # We keey lookup of build_id by full_hash + self.build_ids = {} + + def load_build_environment(self, spec): + """ + Load a build environment from install_environment.json. + + If we are running an analyze command, we will need to load previously + used build environment metadata from install_environment.json to capture + what was done during the build. + """ + if not hasattr(spec, "package") or not spec.package: + tty.die("A spec must have a package to load the environment.") + + pkg_dir = os.path.dirname(spec.package.install_log_path) + env_file = os.path.join(pkg_dir, "install_environment.json") + build_environment = read_json(env_file) + if not build_environment: + tty.warning( + "install_environment.json not found in package folder. " + " This means that the current environment metadata will be used." + ) + else: + self.build_environment = build_environment + + def capture_build_environment(self): + """ + Capture the environment for the build. + + This uses spack.util.environment.get_host_environment_metadata to do so. + This is important because it's a unique identifier, along with the spec, + for a Build. It should look something like this: + + {'host_os': 'ubuntu20.04', + 'platform': 'linux', + 'host_target': 'skylake', + 'hostname': 'vanessa-ThinkPad-T490s', + 'spack_version': '0.16.1-1455-52d5b55b65', + 'kernel_version': '#73-Ubuntu SMP Mon Jan 18 17:25:17 UTC 2021'} + + This is saved to a package install's metadata folder as + install_environment.json, and can be loaded by the monitor for uploading + data relevant to a later analysis. + """ + from spack.util.environment import get_host_environment_metadata + self.build_environment = get_host_environment_metadata() + + def require_auth(self): + """ + Require authentication. + + The token and username must not be unset + """ + if not self.token or not self.username: + tty.die("You are required to export SPACKMON_TOKEN and SPACKMON_USER") + + def set_header(self, name, value): + self.headers.update({name: value}) + + def set_basic_auth(self, username, password): + """ + A wrapper to adding basic authentication to the Request + """ + auth_str = "%s:%s" % (username, password) + auth_header = base64.b64encode(auth_str.encode("utf-8")) + self.set_header("Authorization", "Basic %s" % auth_header.decode("utf-8")) + + def reset(self): + """ + Reset and prepare for a new request. + """ + if "Authorization" in self.headers: + self.headers = {"Authorization": self.headers['Authorization']} + else: + self.headers = {} + + def prepare_request(self, endpoint, data, headers): + """ + Prepare a request given an endpoint, data, and headers. + + If data is provided, urllib makes the request a POST + """ + # Always reset headers for new request. + self.reset() + + # Preserve previously used auth token + headers = headers or self.headers + + # The calling function can provide a full or partial url + if not endpoint.startswith("http"): + endpoint = "%s/%s" % (self.baseurl, endpoint) + + # If we have data, the request will be POST + if data: + if not isinstance(data, str): + data = sjson.dump(data) + data = data.encode('ascii') + + return Request(endpoint, data=data, headers=headers) + + def issue_request(self, request, retry=True): + """ + Given a prepared request, issue it. + + If we get an error, die. If + there are times when we don't want to exit on error (but instead + disable using the monitoring service) we could add that here. + """ + try: + response = urlopen(request) + except URLError as e: + + # If we have an authorization request, retry once with auth + if hasattr(e, "code") and e.code == 401 and retry: + if self.authenticate_request(e): + request = self.prepare_request( + e.url, + sjson.load(request.data.decode('utf-8')), + self.headers + ) + return self.issue_request(request, False) + + # Otherwise, relay the message and exit on error + msg = "" + if hasattr(e, 'reason'): + msg = e.reason + elif hasattr(e, 'code'): + msg = e.code + + if self.allow_fail: + tty.warning("Request to %s was not successful, but continuing." % e.url) + return + + tty.die(msg) + + return response + + def do_request(self, endpoint, data=None, headers=None, url=None): + """ + Do the actual request. + + If data is provided, it is POST, otherwise GET. + If an entire URL is provided, don't use the endpoint + """ + request = self.prepare_request(endpoint, data, headers) + + # If we have an authorization error, we retry with + response = self.issue_request(request) + + # A 200/201 response incidates success + if response.code in [200, 201]: + return sjson.load(response.read().decode('utf-8')) + + return response + + def authenticate_request(self, originalResponse): + """ + Authenticate the request. + + Given a response (an HTTPError 401), look for a Www-Authenticate + header to parse. We return True/False to indicate if the request + should be retried. + """ + authHeaderRaw = originalResponse.headers.get("Www-Authenticate") + if not authHeaderRaw: + return False + + # If we have a username and password, set basic auth automatically + if self.token and self.username: + self.set_basic_auth(self.username, self.token) + + headers = deepcopy(self.headers) + if "Authorization" not in headers: + tty.error( + "This endpoint requires a token. Please set " + "client.set_basic_auth(username, password) first " + "or export them to the environment." + ) + return False + + # Prepare request to retry + h = parse_auth_header(authHeaderRaw) + headers.update({ + "service": h.Service, + "Accept": "application/json", + "User-Agent": "spackmoncli"} + ) + + # Currently we don't set a scope (it defaults to build) + authResponse = self.do_request(h.Realm, headers=headers) + + # Request the token + token = authResponse.get("token") + if not token: + return False + + # Set the token to the original request and retry + self.headers.update({"Authorization": "Bearer %s" % token}) + return True + + # Functions correspond to endpoints + def service_info(self): + """ + Get the service information endpoint + """ + # Base endpoint provides service info + return self.do_request("") + + def new_configuration(self, specs): + """ + Given a list of specs, generate a new configuration for each. + + We return a lookup of specs with their package names. This assumes + that we are only installing one version of each package. We aren't + starting or creating any builds, so we don't need a build environment. + """ + configs = {} + + # There should only be one spec generally (what cases would have >1?) + for spec in specs: + # Not sure if this is needed here, but I see it elsewhere + if spec.name in spack.repo.path or spec.virtual: + spec.concretize() + as_dict = {"spec": spec.to_dict(hash=ht.full_hash), + "spack_version": self.spack_version} + response = self.do_request("specs/new/", data=sjson.dump(as_dict)) + configs[spec.package.name] = response.get('data', {}) + return configs + + def new_build(self, spec): + """ + Create a new build. + + This means sending the hash of the spec to be built, + along with the build environment. These two sets of data uniquely can + identify the build, and we will add objects (the binaries produced) to + it. We return the build id to the calling client. + """ + return self.get_build_id(spec, return_response=True) + + def get_build_id(self, spec, return_response=False, spec_exists=True): + """ + Retrieve a build id, either in the local cache, or query the server. + """ + full_hash = spec.full_hash() + if full_hash in self.build_ids: + return self.build_ids[full_hash] + + # Prepare build environment data (including spack version) + data = self.build_environment.copy() + data['full_hash'] = full_hash + + # If the build should be tagged, add it + if self.tags: + data['tags'] = self.tags + + # If we allow the spec to not exist (meaning we create it) we need to + # include the full spec.yaml here + if not spec_exists: + meta_dir = os.path.dirname(spec.package.install_log_path) + spec_file = os.path.join(meta_dir, "spec.yaml") + data['spec'] = syaml.load(read_file(spec_file)) + + response = self.do_request("builds/new/", data=sjson.dump(data)) + + # Add the build id to the lookup + bid = self.build_ids[full_hash] = response['data']['build']['build_id'] + self.build_ids[full_hash] = bid + + # If the function is called directly, the user might want output + if return_response: + return response + return bid + + def update_build(self, spec, status="SUCCESS"): + """ + Update a build with a new status. + + This typically updates the relevant package to indicate a + successful install. This endpoint can take a general status to update. + """ + data = {"build_id": self.get_build_id(spec), "status": status} + return self.do_request("builds/update/", data=sjson.dump(data)) + + def fail_task(self, spec): + """Given a spec, mark it as failed. This means that Spack Monitor + marks all dependencies as cancelled, unless they are already successful + """ + return self.update_build(spec, status="FAILED") + + def send_analyze_metadata(self, pkg, metadata): + """ + Send spack analyzer metadata to the spack monitor server. + + Given a dictionary of analyzers (with key as analyzer type, and + value as the data) upload the analyzer output to Spack Monitor. + Spack Monitor should either have a known understanding of the analyzer, + or if not (the key is not recognized), it's assumed to be a dictionary + of objects/files, each with attributes to be updated. E.g., + + {"analyzer-name": {"object-file-path": {"feature1": "value1"}}} + """ + # Prepare build environment data (including spack version) + # Since the build might not have been generated, we include the spec + data = {"build_id": self.get_build_id(pkg.spec, spec_exists=False), + "metadata": metadata} + return self.do_request("analyze/builds/", data=sjson.dump(data)) + + def send_phase(self, pkg, phase_name, phase_output_file, status): + """ + Send the result of a phase during install. + + Given a package, phase name, and status, update the monitor endpoint + to alert of the status of the stage. This includes parsing the package + metadata folder for phase output and error files + """ + data = {"build_id": self.get_build_id(pkg.spec)} + + # Send output specific to the phase (does this include error?) + data.update({"status": status, + "output": read_file(phase_output_file), + "phase_name": phase_name}) + + return self.do_request("builds/phases/update/", data=sjson.dump(data)) + + def upload_specfile(self, filename): + """ + Upload a spec file to the spack monitor server. + + Given a spec file (must be json) upload to the UploadSpec endpoint. + This function is not used in the spack to server workflow, but could + be useful is Spack Monitor is intended to send an already generated + file in some kind of separate analysis. For the environment file, we + parse out SPACK_* variables to include. + """ + # We load as json just to validate it + spec = read_json(filename) + data = {"spec": spec, "spack_verison": self.spack_version} + return self.do_request("specs/new/", data=sjson.dump(data)) + + +# Helper functions + +def parse_auth_header(authHeaderRaw): + """ + Parse an authentication header into relevant pieces + """ + regex = re.compile('([a-zA-z]+)="(.+?)"') + matches = regex.findall(authHeaderRaw) + lookup = dict() + for match in matches: + lookup[match[0]] = match[1] + return authHeader(lookup) + + +class authHeader: + def __init__(self, lookup): + """Given a dictionary of values, match them to class attributes""" + for key in lookup: + if key in ["realm", "service", "scope"]: + setattr(self, key.capitalize(), lookup[key]) + + +def read_file(filename): + """ + Read a file, if it exists. Otherwise return None + """ + if not os.path.exists(filename): + return + with open(filename, 'r') as fd: + content = fd.read() + return content + + +def write_file(content, filename): + """ + Write content to file + """ + with open(filename, 'w') as fd: + fd.writelines(content) + return content + + +def write_json(obj, filename): + """ + Write a json file, if the output directory exists. + """ + if not os.path.exists(os.path.dirname(filename)): + return + return write_file(sjson.dump(obj), filename) + + +def read_json(filename): + """ + Read a file and load into json, if it exists. Otherwise return None. + """ + if not os.path.exists(filename): + return + return sjson.load(read_file(filename)) diff --git a/lib/spack/spack/multimethod.py b/lib/spack/spack/multimethod.py index 73cf342b5b..d9fe6882b9 100644 --- a/lib/spack/spack/multimethod.py +++ b/lib/spack/spack/multimethod.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/operating_systems/__init__.py b/lib/spack/spack/operating_systems/__init__.py index 9f87532b85..103eae6134 100644 --- a/lib/spack/spack/operating_systems/__init__.py +++ b/lib/spack/spack/operating_systems/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/operating_systems/cray_backend.py b/lib/spack/spack/operating_systems/cray_backend.py index 5f113eba0b..0e59361d70 100644 --- a/lib/spack/spack/operating_systems/cray_backend.py +++ b/lib/spack/spack/operating_systems/cray_backend.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/operating_systems/cray_frontend.py b/lib/spack/spack/operating_systems/cray_frontend.py index d06920af43..820db506cb 100644 --- a/lib/spack/spack/operating_systems/cray_frontend.py +++ b/lib/spack/spack/operating_systems/cray_frontend.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/operating_systems/linux_distro.py b/lib/spack/spack/operating_systems/linux_distro.py index a6608e6d3b..b1901fc41c 100644 --- a/lib/spack/spack/operating_systems/linux_distro.py +++ b/lib/spack/spack/operating_systems/linux_distro.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/operating_systems/mac_os.py b/lib/spack/spack/operating_systems/mac_os.py index 0efc298338..de9bcf659d 100644 --- a/lib/spack/spack/operating_systems/mac_os.py +++ b/lib/spack/spack/operating_systems/mac_os.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/package.py b/lib/spack/spack/package.py index 8aa86098e2..b24f6bf113 100644 --- a/lib/spack/spack/package.py +++ b/lib/spack/spack/package.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -15,6 +15,7 @@ import collections import contextlib import copy import functools +import glob import hashlib import inspect import os @@ -26,6 +27,7 @@ import time import traceback import six import types +from typing import Optional, List, Dict, Any, Callable # novm import llnl.util.filesystem as fsys import llnl.util.tty as tty @@ -253,8 +255,9 @@ class PackageMeta( """ phase_fmt = '_InstallPhase_{0}' - _InstallPhase_run_before = {} - _InstallPhase_run_after = {} + # These are accessed only through getattr, by name + _InstallPhase_run_before = {} # type: Dict[str, List[Callable]] + _InstallPhase_run_after = {} # type: Dict[str, List[Callable]] def __new__(cls, name, bases, attr_dict): """ @@ -555,7 +558,7 @@ class PackageBase(six.with_metaclass(PackageMeta, PackageViewMixin, object)): #: A list or set of build time test functions to be called when tests #: are executed or 'None' if there are no such test functions. - build_time_test_callbacks = None + build_time_test_callbacks = None # type: Optional[List[str]] #: By default, packages are not virtual #: Virtual packages override this attribute @@ -567,7 +570,7 @@ class PackageBase(six.with_metaclass(PackageMeta, PackageViewMixin, object)): #: A list or set of install time test functions to be called when tests #: are executed or 'None' if there are no such test functions. - install_time_test_callbacks = None + install_time_test_callbacks = None # type: Optional[List[str]] #: By default we build in parallel. Subclasses can override this. parallel = True @@ -589,19 +592,19 @@ class PackageBase(six.with_metaclass(PackageMeta, PackageViewMixin, object)): #: List of prefix-relative file paths (or a single path). If these do #: not exist after install, or if they exist but are not files, #: sanity checks fail. - sanity_check_is_file = [] + sanity_check_is_file = [] # type: List[str] #: List of prefix-relative directory paths (or a single path). If #: these do not exist after install, or if they exist but are not #: directories, sanity checks will fail. - sanity_check_is_dir = [] + sanity_check_is_dir = [] # type: List[str] #: List of glob expressions. Each expression must either be #: absolute or relative to the package source path. #: Matching artifacts found at the end of the build process will be #: copied in the same directory tree as _spack_build_logfile and #: _spack_build_envfile. - archive_files = [] + archive_files = [] # type: List[str] #: Boolean. Set to ``True`` for packages that require a manual download. #: This is currently used by package sanity tests and generation of a @@ -609,7 +612,7 @@ class PackageBase(six.with_metaclass(PackageMeta, PackageViewMixin, object)): manual_download = False #: Set of additional options used when fetching package versions. - fetch_options = {} + fetch_options = {} # type: Dict[str, Any] # # Set default licensing information @@ -627,12 +630,12 @@ class PackageBase(six.with_metaclass(PackageMeta, PackageViewMixin, object)): #: looking for a license. All file paths must be relative to the #: installation directory. More complex packages like Intel may require #: multiple licenses for individual components. Defaults to the empty list. - license_files = [] + license_files = [] # type: List[str] #: List of strings. Environment variables that can be set to tell the #: software where to look for a license if it is not in the usual location. #: Defaults to the empty list. - license_vars = [] + license_vars = [] # type: List[str] #: String. A URL pointing to license setup instructions for the software. #: Defaults to the empty string. @@ -644,9 +647,18 @@ class PackageBase(six.with_metaclass(PackageMeta, PackageViewMixin, object)): #: index of patches by sha256 sum, built lazily _patches_by_hash = None + #: Package homepage where users can find more information about the package + homepage = None # type: str + + #: Default list URL (place to find available versions) + list_url = None # type: str + + #: Link depth to which list_url should be searched for new versions + list_depth = 0 + #: List of strings which contains GitHub usernames of package maintainers. #: Do not include @ here in order not to unnecessarily ping the users. - maintainers = [] + maintainers = [] # type: List[str] #: List of attributes to be excluded from a package's hash. metadata_attrs = ['homepage', 'url', 'urls', 'list_url', 'extendable', @@ -681,13 +693,6 @@ class PackageBase(six.with_metaclass(PackageMeta, PackageViewMixin, object)): msg += " [package '{0.name}' defines both]" raise ValueError(msg.format(self)) - # Set a default list URL (place to find available versions) - if not hasattr(self, 'list_url'): - self.list_url = None - - if not hasattr(self, 'list_depth'): - self.list_depth = 0 - # init internal variables self._stage = None self._fetcher = None @@ -1063,6 +1068,14 @@ class PackageBase(six.with_metaclass(PackageMeta, PackageViewMixin, object)): return os.path.join(self.stage.path, _spack_build_logfile) @property + def phase_log_files(self): + """Find sorted phase log files written to the staging directory""" + logs_dir = os.path.join(self.stage.path, "spack-build-*-out.txt") + log_files = glob.glob(logs_dir) + log_files.sort() + return log_files + + @property def install_log_path(self): """Return the build log file path on successful installation.""" # Backward compatibility: Return the name of an existing install log. @@ -1305,7 +1318,6 @@ class PackageBase(six.with_metaclass(PackageMeta, PackageViewMixin, object)): tty.debug('No fetch required for {0}: package has no code.' .format(self.name)) - start_time = time.time() checksum = spack.config.get('config:checksum') fetch = self.stage.managed_by_spack if checksum and fetch and self.version not in self.versions: @@ -1327,8 +1339,34 @@ class PackageBase(six.with_metaclass(PackageMeta, PackageViewMixin, object)): raise FetchError("Will not fetch %s" % self.spec.format('{name}{@version}'), ck_msg) + deprecated = spack.config.get('config:deprecated') + if not deprecated and self.versions.get( + self.version, {}).get('deprecated', False): + tty.warn("{0} is deprecated and may be removed in a future Spack " + "release.".format( + self.spec.format('{name}{@version}'))) + + # Ask the user whether to install deprecated version if we're + # interactive, but just fail if non-interactive. + dp_msg = ("If you are willing to be a maintainer for this version " + "of the package, submit a PR to remove `deprecated=False" + "`, or use `--deprecated` to skip this check.") + ignore_deprecation = False + if sys.stdout.isatty(): + ignore_deprecation = tty.get_yes_or_no(" Fetch anyway?", + default=False) + + if ignore_deprecation: + tty.debug("Fetching deprecated version. {0}".format( + dp_msg)) + + if not ignore_deprecation: + raise FetchError("Will not fetch {0}".format( + self.spec.format('{name}{@version}')), dp_msg) + self.stage.create() err_msg = None if not self.manual_download else self.download_instr + start_time = time.time() self.stage.fetch(mirror_only, err_msg=err_msg) self._fetch_time = time.time() - start_time @@ -1375,7 +1413,7 @@ class PackageBase(six.with_metaclass(PackageMeta, PackageViewMixin, object)): # If there are no patches, note it. if not patches and not has_patch_fun: - tty.debug('No patches needed for {0}'.format(self.name)) + tty.msg('No patches needed for {0}'.format(self.name)) return # Construct paths to special files in the archive dir used to @@ -1393,10 +1431,10 @@ class PackageBase(six.with_metaclass(PackageMeta, PackageViewMixin, object)): # If this file exists, then we already applied all the patches. if os.path.isfile(good_file): - tty.debug('Already patched {0}'.format(self.name)) + tty.msg('Already patched {0}'.format(self.name)) return elif os.path.isfile(no_patches_file): - tty.debug('No patches needed for {0}'.format(self.name)) + tty.msg('No patches needed for {0}'.format(self.name)) return # Apply all the patches for specs that match this one @@ -1405,7 +1443,7 @@ class PackageBase(six.with_metaclass(PackageMeta, PackageViewMixin, object)): try: with fsys.working_dir(self.stage.source_path): patch.apply(self.stage) - tty.debug('Applied patch {0}'.format(patch.path_or_url)) + tty.msg('Applied patch {0}'.format(patch.path_or_url)) patched = True except spack.error.SpackError as e: tty.debug(e) @@ -1419,7 +1457,7 @@ class PackageBase(six.with_metaclass(PackageMeta, PackageViewMixin, object)): try: with fsys.working_dir(self.stage.source_path): self.patch() - tty.debug('Ran patch() for {0}'.format(self.name)) + tty.msg('Ran patch() for {0}'.format(self.name)) patched = True except spack.multimethod.NoSuchMethodError: # We are running a multimethod without a default case. @@ -1429,7 +1467,7 @@ class PackageBase(six.with_metaclass(PackageMeta, PackageViewMixin, object)): # directive, AND the patch function didn't apply, say # no patches are needed. Otherwise, we already # printed a message for each patch. - tty.debug('No patches needed for {0}'.format(self.name)) + tty.msg('No patches needed for {0}'.format(self.name)) except spack.error.SpackError as e: tty.debug(e) @@ -1497,10 +1535,16 @@ class PackageBase(six.with_metaclass(PackageMeta, PackageViewMixin, object)): hash_content.extend(':'.join((p.sha256, str(p.level))).encode('utf-8') for p in self.spec.patches) hash_content.append(package_hash(self.spec, content)) - return base64.b32encode( + b32_hash = base64.b32encode( hashlib.sha256(bytes().join( sorted(hash_content))).digest()).lower() + # convert from bytes if running python 3 + if sys.version_info[0] >= 3: + b32_hash = b32_hash.decode('utf-8') + + return b32_hash + def _has_make_target(self, target): """Checks to see if 'target' is a valid target in a Makefile. @@ -1761,7 +1805,7 @@ class PackageBase(six.with_metaclass(PackageMeta, PackageViewMixin, object)): work_dir (str or None): path to the smoke test directory """ wdir = '.' if work_dir is None else work_dir - with fsys.working_dir(wdir): + with fsys.working_dir(wdir, create=True): try: runner = which(exe) if runner is None and skip_missing: @@ -2596,7 +2640,7 @@ class BundlePackage(PackageBase): """General purpose bundle, or no-code, package class.""" #: There are no phases by default but the property is required to support #: post-install hooks (e.g., for module generation). - phases = [] + phases = [] # type: List[str] #: This attribute is used in UI queries that require to know which #: build-system class we are using build_system_class = 'BundlePackage' diff --git a/lib/spack/spack/package_prefs.py b/lib/spack/spack/package_prefs.py index 149e10ea26..cf74c7ee52 100644 --- a/lib/spack/spack/package_prefs.py +++ b/lib/spack/spack/package_prefs.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/package_test.py b/lib/spack/spack/package_test.py index 6e9da9a985..6a4df096fc 100644 --- a/lib/spack/spack/package_test.py +++ b/lib/spack/spack/package_test.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/parse.py b/lib/spack/spack/parse.py index 12bbcee588..1fc853bee4 100644 --- a/lib/spack/spack/parse.py +++ b/lib/spack/spack/parse.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/patch.py b/lib/spack/spack/patch.py index 3a839c5b0f..d38b2ef767 100644 --- a/lib/spack/spack/patch.py +++ b/lib/spack/spack/patch.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/paths.py b/lib/spack/spack/paths.py index 068a7a6fdb..b0ff031dd1 100644 --- a/lib/spack/spack/paths.py +++ b/lib/spack/spack/paths.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -33,6 +33,7 @@ external_path = os.path.join(lib_path, "external") build_env_path = os.path.join(lib_path, "env") module_path = os.path.join(lib_path, "spack") command_path = os.path.join(module_path, "cmd") +analyzers_path = os.path.join(module_path, "analyzers") platform_path = os.path.join(module_path, 'platforms') compilers_path = os.path.join(module_path, "compilers") build_systems_path = os.path.join(module_path, 'build_systems') @@ -52,6 +53,8 @@ mock_packages_path = os.path.join(repos_path, "builtin.mock") user_config_path = os.path.expanduser('~/.spack') user_bootstrap_path = os.path.join(user_config_path, 'bootstrap') user_bootstrap_store = os.path.join(user_bootstrap_path, 'store') +reports_path = os.path.join(user_config_path, "reports") + opt_path = os.path.join(prefix, "opt") etc_path = os.path.join(prefix, "etc") diff --git a/lib/spack/spack/pkgkit.py b/lib/spack/spack/pkgkit.py index 423c0fb05f..bc44d0b7b2 100644 --- a/lib/spack/spack/pkgkit.py +++ b/lib/spack/spack/pkgkit.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -19,6 +19,10 @@ from spack.build_systems.makefile import MakefilePackage from spack.build_systems.aspell_dict import AspellDictPackage from spack.build_systems.autotools import AutotoolsPackage from spack.build_systems.cmake import CMakePackage +from spack.build_systems.cached_cmake import ( + CachedCMakePackage, cmake_cache_option, cmake_cache_path, + cmake_cache_string +) from spack.build_systems.cuda import CudaPackage from spack.build_systems.oneapi import IntelOneApiPackage from spack.build_systems.oneapi import IntelOneApiLibraryPackage diff --git a/lib/spack/spack/platforms/__init__.py b/lib/spack/spack/platforms/__init__.py index 9f87532b85..103eae6134 100644 --- a/lib/spack/spack/platforms/__init__.py +++ b/lib/spack/spack/platforms/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/platforms/cray.py b/lib/spack/spack/platforms/cray.py index 091f8d3ffb..5eb0ee9310 100644 --- a/lib/spack/spack/platforms/cray.py +++ b/lib/spack/spack/platforms/cray.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -169,6 +169,7 @@ class Cray(Platform): for mod in modules: if 'craype-' in mod: name = mod[7:] + name = name.split()[0] _n = name.replace('-', '_') # test for mic-knl/mic_knl is_target_name = (name in archspec.cpu.TARGETS or _n in archspec.cpu.TARGETS) diff --git a/lib/spack/spack/platforms/darwin.py b/lib/spack/spack/platforms/darwin.py index e36722723b..3969cc9ec8 100644 --- a/lib/spack/spack/platforms/darwin.py +++ b/lib/spack/spack/platforms/darwin.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/platforms/linux.py b/lib/spack/spack/platforms/linux.py index c7e1def2d9..46906229ba 100644 --- a/lib/spack/spack/platforms/linux.py +++ b/lib/spack/spack/platforms/linux.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/platforms/test.py b/lib/spack/spack/platforms/test.py index f938eb749d..80d9431879 100644 --- a/lib/spack/spack/platforms/test.py +++ b/lib/spack/spack/platforms/test.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/projections.py b/lib/spack/spack/projections.py index b91d321436..9f78e24409 100644 --- a/lib/spack/spack/projections.py +++ b/lib/spack/spack/projections.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/provider_index.py b/lib/spack/spack/provider_index.py index 326f6aa8f1..6a84d8f40a 100644 --- a/lib/spack/spack/provider_index.py +++ b/lib/spack/spack/provider_index.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/relocate.py b/lib/spack/spack/relocate.py index e1726b060e..e167472b75 100644 --- a/lib/spack/spack/relocate.py +++ b/lib/spack/spack/relocate.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/repo.py b/lib/spack/spack/repo.py index 1534054075..61f5e1d28c 100644 --- a/lib/spack/spack/repo.py +++ b/lib/spack/spack/repo.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -13,18 +13,18 @@ import itertools import os import re import shutil +import six import stat import sys import traceback import types +from typing import Dict # novm -try: +if sys.version_info >= (3, 5): from collections.abc import Mapping # novm -except ImportError: +else: from collections import Mapping -import six - import ruamel.yaml as yaml import llnl.util.lang @@ -118,7 +118,11 @@ class SpackNamespace(types.ModuleType): def __getattr__(self, name): """Getattr lazily loads modules if they're not already loaded.""" submodule = self.__package__ + '.' + name - setattr(self, name, __import__(submodule)) + try: + setattr(self, name, __import__(submodule)) + except ImportError: + msg = "'{0}' object has no attribute {1}" + raise AttributeError(msg.format(type(self), name)) return getattr(self, name) @@ -131,7 +135,7 @@ class FastPackageChecker(Mapping): during instance initialization. """ #: Global cache, reused by every instance - _paths_cache = {} + _paths_cache = {} # type: Dict[str, Dict[str, os.stat_result]] def __init__(self, packages_path): # The path of the repository managed by this instance @@ -149,7 +153,7 @@ class FastPackageChecker(Mapping): self._paths_cache[self.packages_path] = self._create_new_cache() self._packages_to_stats = self._paths_cache[self.packages_path] - def _create_new_cache(self): + def _create_new_cache(self): # type: () -> Dict[str, os.stat_result] """Create a new cache for packages in a repo. The implementation here should try to minimize filesystem @@ -159,7 +163,7 @@ class FastPackageChecker(Mapping): """ # Create a dictionary that will store the mapping between a # package name and its stat info - cache = {} + cache = {} # type: Dict[str, os.stat_result] for pkg_name in os.listdir(self.packages_path): # Skip non-directories in the package root. pkg_dir = os.path.join(self.packages_path, pkg_name) @@ -341,7 +345,7 @@ class PatchIndexer(Indexer): def _create(self): return spack.patch.PatchCache() - def needs_update(): + def needs_update(self): # TODO: patches can change under a package and we should handle # TODO: it, but we currently punt. This should be refactored to # TODO: check whether patches changed each time a package loads, diff --git a/lib/spack/spack/report.py b/lib/spack/spack/report.py index ebae2d0adc..8e97d036c0 100644 --- a/lib/spack/spack/report.py +++ b/lib/spack/spack/report.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/reporter.py b/lib/spack/spack/reporter.py index 6314054139..629051e223 100644 --- a/lib/spack/spack/reporter.py +++ b/lib/spack/spack/reporter.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/reporters/__init__.py b/lib/spack/spack/reporters/__init__.py index 9f87532b85..103eae6134 100644 --- a/lib/spack/spack/reporters/__init__.py +++ b/lib/spack/spack/reporters/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/reporters/cdash.py b/lib/spack/spack/reporters/cdash.py index c1a220963e..36660fa890 100644 --- a/lib/spack/spack/reporters/cdash.py +++ b/lib/spack/spack/reporters/cdash.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -22,6 +22,7 @@ from ordereddict_backport import OrderedDict import spack.build_environment import spack.fetch_strategy import spack.package +from spack.error import SpackError from spack.reporter import Reporter from spack.util.crypto import checksum from spack.util.executable import which @@ -60,6 +61,7 @@ class CDash(Reporter): def __init__(self, args): Reporter.__init__(self, args) tty.set_verbose(args.verbose) + self.success = True self.template_dir = os.path.join('reports', 'cdash') self.cdash_upload_url = args.cdash_upload_url @@ -159,13 +161,21 @@ class CDash(Reporter): report_data[phase]['log'] = \ '\n'.join(report_data[phase]['loglines']) errors, warnings = parse_log_events(report_data[phase]['loglines']) + + # Convert errors to warnings if the package reported success. + if package['result'] == 'success': + warnings = errors + warnings + errors = [] + # Cap the number of errors and warnings at 50 each. errors = errors[:50] warnings = warnings[:50] nerrors = len(errors) - if phase == 'configure' and nerrors > 0: - report_data[phase]['status'] = 1 + if nerrors > 0: + self.success = False + if phase == 'configure': + report_data[phase]['status'] = 1 if phase == 'build': # Convert log output from ASCII to Unicode and escape for XML. @@ -186,11 +196,6 @@ class CDash(Reporter): event['source_file']) return event - # Convert errors to warnings if the package reported success. - if package['result'] == 'success': - warnings = errors + warnings - errors = [] - report_data[phase]['errors'] = [] report_data[phase]['warnings'] = [] for error in errors: @@ -254,7 +259,7 @@ class CDash(Reporter): for package in spec['packages']: self.build_report_for_package( directory_name, package, duration) - self.print_cdash_link() + self.finalize_report() def test_report_for_package(self, directory_name, package, duration): if 'stdout' not in package: @@ -360,7 +365,7 @@ class CDash(Reporter): for package in spec['packages']: self.test_report_for_package( directory_name, package, duration) - self.print_cdash_link() + self.finalize_report() def concretization_report(self, directory_name, msg): self.buildname = self.base_buildname @@ -381,7 +386,8 @@ class CDash(Reporter): # errors so refer to this report with the base buildname instead. self.current_package_name = self.base_buildname self.upload(output_filename) - self.print_cdash_link() + self.success = False + self.finalize_report() def initialize_report(self, directory_name): if not os.path.exists(directory_name): @@ -430,7 +436,7 @@ class CDash(Reporter): buildid = match.group(1) self.buildIds[self.current_package_name] = buildid - def print_cdash_link(self): + def finalize_report(self): if self.buildIds: print("View your build results here:") for package_name, buildid in iteritems(self.buildIds): @@ -440,3 +446,5 @@ class CDash(Reporter): build_url = build_url[0:build_url.find("submit.php")] build_url += "buildSummary.php?buildid={0}".format(buildid) print("{0}: {1}".format(package_name, build_url)) + if not self.success: + raise SpackError("Errors encountered, see above for more details") diff --git a/lib/spack/spack/reporters/junit.py b/lib/spack/spack/reporters/junit.py index 598b308934..9120850c4c 100644 --- a/lib/spack/spack/reporters/junit.py +++ b/lib/spack/spack/reporters/junit.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/resource.py b/lib/spack/spack/resource.py index 5cec54f6de..488c6a846d 100644 --- a/lib/spack/spack/resource.py +++ b/lib/spack/spack/resource.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/s3_handler.py b/lib/spack/spack/s3_handler.py index 24f0a46221..d906577bed 100644 --- a/lib/spack/spack/s3_handler.py +++ b/lib/spack/spack/s3_handler.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/schema/__init__.py b/lib/spack/spack/schema/__init__.py index 1ddca90cc6..804b7c989d 100644 --- a/lib/spack/spack/schema/__init__.py +++ b/lib/spack/spack/schema/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/schema/buildcache_spec.py b/lib/spack/spack/schema/buildcache_spec.py index 5ba07a27f1..f1c515852c 100644 --- a/lib/spack/spack/schema/buildcache_spec.py +++ b/lib/spack/spack/schema/buildcache_spec.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/schema/cdash.py b/lib/spack/spack/schema/cdash.py index 2df2ea3f32..41c0146b1d 100644 --- a/lib/spack/spack/schema/cdash.py +++ b/lib/spack/spack/schema/cdash.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/schema/compilers.py b/lib/spack/spack/schema/compilers.py index c994bef819..91ace05f4e 100644 --- a/lib/spack/spack/schema/compilers.py +++ b/lib/spack/spack/schema/compilers.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/schema/config.py b/lib/spack/spack/schema/config.py index 0f83eb86f4..f0669600e1 100644 --- a/lib/spack/spack/schema/config.py +++ b/lib/spack/spack/schema/config.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -78,6 +78,7 @@ properties = { 'install_missing_compilers': {'type': 'boolean'}, 'debug': {'type': 'boolean'}, 'checksum': {'type': 'boolean'}, + 'deprecated': {'type': 'boolean'}, 'locks': {'type': 'boolean'}, 'dirty': {'type': 'boolean'}, 'build_language': {'type': 'string'}, diff --git a/lib/spack/spack/schema/container.py b/lib/spack/spack/schema/container.py index 7e14efd75c..5223efa041 100644 --- a/lib/spack/spack/schema/container.py +++ b/lib/spack/spack/schema/container.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/schema/database_index.py b/lib/spack/spack/schema/database_index.py index eec3d4ce02..105f6ad6a3 100644 --- a/lib/spack/spack/schema/database_index.py +++ b/lib/spack/spack/schema/database_index.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/schema/env.py b/lib/spack/spack/schema/env.py index a7e3dc608f..82971505e4 100644 --- a/lib/spack/spack/schema/env.py +++ b/lib/spack/spack/schema/env.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/schema/environment.py b/lib/spack/spack/schema/environment.py index d251aeba96..6431916c56 100644 --- a/lib/spack/spack/schema/environment.py +++ b/lib/spack/spack/schema/environment.py @@ -1,4 +1,4 @@ -# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -38,11 +38,12 @@ def parse(config_obj): config_obj: a configuration dictionary conforming to the schema definition for environment modifications """ + import sys import spack.util.environment as ev - try: - from collections import Sequence # novm - except ImportError: + if sys.version_info >= (3, 5): from collections.abc import Sequence # novm + else: + from collections import Sequence # novm env = ev.EnvironmentModifications() for command, variable in config_obj.items(): diff --git a/lib/spack/spack/schema/gitlab_ci.py b/lib/spack/spack/schema/gitlab_ci.py index 79ffee2074..d6d8f564a3 100644 --- a/lib/spack/spack/schema/gitlab_ci.py +++ b/lib/spack/spack/schema/gitlab_ci.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -9,6 +9,8 @@ :lines: 13- """ +from llnl.util.lang import union_dicts + image_schema = { 'oneOf': [ { @@ -28,130 +30,126 @@ image_schema = { ], } -#: Properties for inclusion in other schemas -properties = { - 'gitlab-ci': { +runner_attributes_schema_items = { + 'image': image_schema, + 'tags': { + 'type': 'array', + 'items': {'type': 'string'} + }, + 'variables': { 'type': 'object', - 'additionalProperties': False, - 'required': ['mappings'], 'patternProperties': { - 'bootstrap': { - 'type': 'array', - 'items': { - 'anyOf': [ - { - 'type': 'string', - }, { - 'type': 'object', - 'additionalProperties': False, - 'required': ['name'], - 'properties': { - 'name': { - 'type': 'string', - }, - 'compiler-agnostic': { - 'type': 'boolean', - 'default': False, - }, - }, - }, - ], - }, + r'[\w\d\-_\.]+': { + 'type': 'string', }, - 'mappings': { - 'type': 'array', - 'items': { - 'type': 'object', - 'additionalProperties': False, - 'required': ['match'], - 'properties': { - 'match': { - 'type': 'array', - 'items': { + }, + }, + 'before_script': { + 'type': 'array', + 'items': {'type': 'string'} + }, + 'script': { + 'type': 'array', + 'items': {'type': 'string'} + }, + 'after_script': { + 'type': 'array', + 'items': {'type': 'string'} + }, +} + +runner_selector_schema = { + 'type': 'object', + 'additionalProperties': False, + 'required': ['tags'], + 'properties': runner_attributes_schema_items, +} + + +core_shared_properties = union_dicts( + runner_attributes_schema_items, + { + 'bootstrap': { + 'type': 'array', + 'items': { + 'anyOf': [ + { + 'type': 'string', + }, { + 'type': 'object', + 'additionalProperties': False, + 'required': ['name'], + 'properties': { + 'name': { 'type': 'string', }, - }, - 'runner-attributes': { - 'type': 'object', - 'additionalProperties': True, - 'required': ['tags'], - 'properties': { - 'image': image_schema, - 'tags': { - 'type': 'array', - 'items': {'type': 'string'} - }, - 'variables': { - 'type': 'object', - 'patternProperties': { - r'[\w\d\-_\.]+': { - 'type': 'string', - }, - }, - }, - 'before_script': { - 'type': 'array', - 'items': {'type': 'string'} - }, - 'script': { - 'type': 'array', - 'items': {'type': 'string'} - }, - 'after_script': { - 'type': 'array', - 'items': {'type': 'string'} - }, + 'compiler-agnostic': { + 'type': 'boolean', + 'default': False, }, }, }, - }, + ], }, - 'image': image_schema, - 'tags': { - 'type': 'array', - 'items': {'type': 'string'} - }, - 'variables': { - 'type': 'object', - 'patternProperties': { - r'[\w\d\-_\.]+': { - 'type': 'string', - }, - }, - }, - 'before_script': { - 'type': 'array', - 'items': {'type': 'string'} - }, - 'script': { - 'type': 'array', - 'items': {'type': 'string'} - }, - 'after_script': { - 'type': 'array', - 'items': {'type': 'string'} - }, - 'enable-artifacts-buildcache': { - 'type': 'boolean', - 'default': False, - }, - 'final-stage-rebuild-index': { + }, + 'mappings': { + 'type': 'array', + 'items': { 'type': 'object', 'additionalProperties': False, - 'required': ['tags'], + 'required': ['match'], 'properties': { - 'image': image_schema, - 'tags': { + 'match': { 'type': 'array', - 'default': [], - 'items': {'type': 'string'} + 'items': { + 'type': 'string', + }, }, + 'runner-attributes': runner_selector_schema, }, }, }, + 'service-job-attributes': runner_selector_schema, + 'rebuild-index': {'type': 'boolean'}, + 'broken-specs-url': {'type': 'string'}, }, +) + +gitlab_ci_properties = { + 'anyOf': [ + { + 'type': 'object', + 'additionalProperties': False, + 'required': ['mappings'], + 'properties': union_dicts( + core_shared_properties, + { + 'enable-artifacts-buildcache': { + 'type': 'boolean', + }, + }, + ), + }, + { + 'type': 'object', + 'additionalProperties': False, + 'required': ['mappings'], + 'properties': union_dicts( + core_shared_properties, + { + 'temporary-storage-url-prefix': { + 'type': 'string', + }, + }, + ), + }, + ] } +#: Properties for inclusion in other schemas +properties = { + 'gitlab-ci': gitlab_ci_properties, +} #: Full schema with metadata schema = { diff --git a/lib/spack/spack/schema/merged.py b/lib/spack/spack/schema/merged.py index e118acf286..a58f47cb16 100644 --- a/lib/spack/spack/schema/merged.py +++ b/lib/spack/spack/schema/merged.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/schema/mirrors.py b/lib/spack/spack/schema/mirrors.py index 5bb641df05..6dec5aac97 100644 --- a/lib/spack/spack/schema/mirrors.py +++ b/lib/spack/spack/schema/mirrors.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/schema/modules.py b/lib/spack/spack/schema/modules.py index f44ba7d97d..39db0bf9a7 100644 --- a/lib/spack/spack/schema/modules.py +++ b/lib/spack/spack/schema/modules.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/schema/packages.py b/lib/spack/spack/schema/packages.py index f2626c83be..e24f610c74 100644 --- a/lib/spack/spack/schema/packages.py +++ b/lib/spack/spack/schema/packages.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/schema/projections.py b/lib/spack/spack/schema/projections.py index bd0693f30e..cab512fe3b 100644 --- a/lib/spack/spack/schema/projections.py +++ b/lib/spack/spack/schema/projections.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/schema/repos.py b/lib/spack/spack/schema/repos.py index 6aa9bcf57e..44cf688a48 100644 --- a/lib/spack/spack/schema/repos.py +++ b/lib/spack/spack/schema/repos.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/schema/spec.py b/lib/spack/spack/schema/spec.py index e2c8efad4c..fecfcb2af4 100644 --- a/lib/spack/spack/schema/spec.py +++ b/lib/spack/spack/schema/spec.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/schema/upstreams.py b/lib/spack/spack/schema/upstreams.py index f5192aa096..7d721332eb 100644 --- a/lib/spack/spack/schema/upstreams.py +++ b/lib/spack/spack/schema/upstreams.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/solver/__init__.py b/lib/spack/spack/solver/__init__.py index 94f8ac4d9e..103eae6134 100644 --- a/lib/spack/spack/solver/__init__.py +++ b/lib/spack/spack/solver/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/solver/asp.py b/lib/spack/spack/solver/asp.py index dbfca6506d..99e556a764 100644 --- a/lib/spack/spack/solver/asp.py +++ b/lib/spack/spack/solver/asp.py @@ -22,7 +22,7 @@ try: # There may be a better way to detect this clingo_cffi = hasattr(clingo.Symbol, '_rep') except ImportError: - clingo = None + clingo = None # type: ignore clingo_cffi = False import llnl.util.lang @@ -208,6 +208,9 @@ class Result(object): self.answers = [] self.cores = [] + # names of optimization criteria + self.criteria = [] + def print_cores(self): for core in self.cores: tty.msg( @@ -355,6 +358,7 @@ class PyclingoDriver(object): return x.string or str(x) if result.satisfiable: + # build spec from the best model builder = SpecBuilder(specs) min_cost, best_model = min(models) tuples = [ @@ -362,8 +366,20 @@ class PyclingoDriver(object): for sym in best_model ] answers = builder.build_specs(tuples) + + # add best spec to the results result.answers.append((list(min_cost), 0, answers)) + # pull optimization criteria names out of the solution + criteria = [ + (int(args[0]), args[1]) for name, args in tuples + if name == "opt_criterion" + ] + result.criteria = [t[1] for t in sorted(criteria, reverse=True)] + + # record the number of models the solver considered + result.nmodels = len(models) + elif cores: symbols = dict( (a.literal, a.symbol) @@ -395,6 +411,7 @@ class SpackSolverSetup(object): self.gen = None # set by setup() self.possible_versions = {} self.versions_in_package_py = {} + self.deprecated_versions = {} self.versions_from_externals = {} self.possible_virtuals = None self.possible_compilers = [] @@ -465,6 +482,11 @@ class SpackSolverSetup(object): for i, v in enumerate(most_to_least_preferred): self.gen.fact(fn.version_declared(pkg.name, v, i)) + # Declare deprecated versions for this package, if any + deprecated = self.deprecated_versions[pkg.name] + for v in sorted(deprecated): + self.gen.fact(fn.deprecated_version(pkg.name, v)) + def spec_versions(self, spec): """Return list of clauses expressing spec's version constraints.""" spec = specify(spec) @@ -492,9 +514,7 @@ class SpackSolverSetup(object): def conflict_rules(self, pkg): for trigger, constraints in pkg.conflicts.items(): - trigger_id = self.condition( - spack.spec.Spec(trigger), name=pkg.name - ) + trigger_id = self.condition(spack.spec.Spec(trigger), name=pkg.name) self.gen.fact(fn.conflict_trigger(trigger_id)) for constraint, _ in constraints: @@ -950,16 +970,17 @@ class SpackSolverSetup(object): if value == '*': continue - # validate variant value - reserved_names = spack.directives.reserved_names - if not spec.virtual and vname not in reserved_names: - try: - variant_def = spec.package.variants[vname] - except KeyError: - msg = 'variant "{0}" not found in package "{1}"' - raise RuntimeError(msg.format(vname, spec.name)) - else: - variant_def.validate_or_raise(variant, spec.package) + # validate variant value only if spec not concrete + if not spec.concrete: + reserved_names = spack.directives.reserved_names + if not spec.virtual and vname not in reserved_names: + try: + variant_def = spec.package.variants[vname] + except KeyError: + msg = 'variant "{0}" not found in package "{1}"' + raise RuntimeError(msg.format(vname, spec.name)) + else: + variant_def.validate_or_raise(variant, spec.package) clauses.append(f.variant_value(spec.name, vname, value)) @@ -1005,12 +1026,16 @@ class SpackSolverSetup(object): self.possible_versions = collections.defaultdict(set) self.versions_in_package_py = collections.defaultdict(set) self.versions_from_externals = collections.defaultdict(set) + self.deprecated_versions = collections.defaultdict(set) for pkg_name in possible_pkgs: pkg = spack.repo.get(pkg_name) - for v in pkg.versions: + for v, version_info in pkg.versions.items(): self.versions_in_package_py[pkg_name].add(v) self.possible_versions[pkg_name].add(v) + deprecated = version_info.get('deprecated', False) + if deprecated: + self.deprecated_versions[pkg_name].add(v) for spec in specs: for dep in spec.traverse(): @@ -1378,7 +1403,10 @@ class SpackSolverSetup(object): ) for clause in self.spec_clauses(spec): self.gen.fact(clause) - + if clause.name == 'variant_set': + self.gen.fact(fn.variant_default_value_from_cli( + *clause.args + )) self.gen.h1("Variant Values defined in specs") self.define_variant_values() @@ -1535,6 +1563,10 @@ class SpecBuilder(object): check_same_flags(spec.compiler_flags, flags) spec.compiler_flags.update(flags) + def deprecated(self, pkg, version): + msg = 'using "{0}@{1}" which is a deprecated version' + tty.warn(msg.format(pkg, version)) + def build_specs(self, function_tuples): # Functions don't seem to be in particular order in output. Sort # them here so that directives that build objects (like node and @@ -1603,8 +1635,7 @@ def _develop_specs_from_env(spec, env): if not dev_info: return - path = dev_info['path'] - path = path if os.path.isabs(path) else os.path.join(env.path, path) + path = os.path.normpath(os.path.join(env.path, dev_info['path'])) if 'dev_path' in spec.variants: assert spec.variants['dev_path'].value == path diff --git a/lib/spack/spack/solver/concretize.lp b/lib/spack/spack/solver/concretize.lp index 145c00ab67..ec0d4fdf5a 100644 --- a/lib/spack/spack/solver/concretize.lp +++ b/lib/spack/spack/solver/concretize.lp @@ -1,3 +1,8 @@ +% Copyright 2013-2021 Lawrence Livermore National Security, LLC and other +% Spack Project Developers. See the top-level COPYRIGHT file for details. +% +% SPDX-License-Identifier: (Apache-2.0 OR MIT) + %============================================================================= % This logic program implements Spack's concretizer %============================================================================= @@ -14,6 +19,9 @@ version_declared(Package, Version) :- version_declared(Package, Version, _). 1 { version(Package, Version) : version_declared(Package, Version) } 1 :- node(Package). +% If we select a deprecated version, mark the package as deprecated +deprecated(Package, Version) :- version(Package, Version), deprecated_version(Package, Version). + possible_version_weight(Package, Weight) :- version(Package, Version), version_declared(Package, Version, Weight), not preferred_version_declared(Package, Version, _). @@ -32,6 +40,7 @@ version_satisfies(Package, Constraint) #defined preferred_version_declared/3. #defined version_satisfies/3. +#defined deprecated_version/2. %----------------------------------------------------------------------------- % Spec conditions and imposed constraints @@ -211,6 +220,7 @@ provider_weight(Package, 100) provider(Package, Virtual), not default_provider_preference(Virtual, Package, _). +#defined possible_provider/2. #defined provider_condition/3. #defined required_provider_condition/3. #defined required_provider_condition/4. @@ -397,14 +407,23 @@ variant_not_default(Package, Variant, Value, 0) external_with_variant_set(Package, Variant, Value), node(Package). -% The default value for a variant in a package is what is written -% in the package.py file, unless some preference is set in packages.yaml +% The default value for a variant in a package is what is prescribed: +% +% 1. On the command line +% 2. In packages.yaml (if there's no command line settings) +% 3. In the package.py file (if there are no settings in +% packages.yaml and the command line) +% variant_default_value(Package, Variant, Value) :- variant_default_value_from_package_py(Package, Variant, Value), - not variant_default_value_from_packages_yaml(Package, Variant, _). + not variant_default_value_from_packages_yaml(Package, Variant, _), + not variant_default_value_from_cli(Package, Variant, _). variant_default_value(Package, Variant, Value) - :- variant_default_value_from_packages_yaml(Package, Variant, Value). + :- variant_default_value_from_packages_yaml(Package, Variant, Value), + not variant_default_value_from_cli(Package, Variant, _). + +variant_default_value(Package, Variant, Value) :- variant_default_value_from_cli(Package, Variant, Value). % Treat 'none' in a special way - it cannot be combined with other % values even if the variant is multi-valued @@ -429,6 +448,7 @@ variant_single_value(Package, "dev_path") #defined variant_single_value/2. #defined variant_default_value/3. #defined variant_possible_value/3. +#defined variant_default_value_from_cli/3. #defined variant_default_value_from_packages_yaml/3. #defined variant_default_value_from_package_py/3. #defined variant_value_from_disjoint_sets/4. @@ -529,29 +549,19 @@ node_target_weight(Package, Weight) node_target(Package, Target), target_weight(Target, Package, Weight). -% compatibility rules for targets among nodes -node_target_match_pref(Dependency, Target) - :- depends_on(Package, Dependency), - node_target_match_pref(Package, Target), - not node_target_set(Dependency, _). - -node_target_match_pref(Dependency, Target) - :- depends_on(Package, Dependency), - node_target_set(Package, Target), - not node_target_match_pref(Package, Target), - not node_target_set(Dependency, _). - -node_target_match_pref(Dependency, Target) - :- depends_on(Package, Dependency), - root(Package), node_target(Package, Target), - not node_target_match_pref(Package, _). - -node_target_match(Package, 1) - :- node_target(Package, Target), node_target_match_pref(Package, Target). - derive_target_from_parent(Parent, Package) - :- depends_on(Parent, Package), not package_target_weight(_, Package, _). + :- depends_on(Parent, Package), + not package_target_weight(_, Package, _). + +% compatibility rules for targets among nodes +node_target_match(Parent, Dependency) + :- depends_on(Parent, Dependency), + node_target(Parent, Target), + node_target(Dependency, Target). +node_target_mismatch(Parent, Dependency) + :- depends_on(Parent, Dependency), + not node_target_match(Parent, Dependency). #defined node_target_set/2. #defined package_target_weight/3. @@ -599,22 +609,18 @@ node_compiler_version(Package, Compiler, Version) :- node_compiler_version_set(P % If a package and one of its dependencies don't have the % same compiler there's a mismatch. -compiler_mismatch(Package, Dependency) +compiler_match(Package, Dependency) :- depends_on(Package, Dependency), - node_compiler_version(Package, Compiler1, _), - node_compiler_version(Dependency, Compiler2, _), - Compiler1 != Compiler2. + node_compiler_version(Package, Compiler, Version), + node_compiler_version(Dependency, Compiler, Version). compiler_mismatch(Package, Dependency) :- depends_on(Package, Dependency), - node_compiler_version(Package, Compiler, Version1), - node_compiler_version(Dependency, Compiler, Version2), - Version1 != Version2. + not compiler_match(Package, Dependency). #defined node_compiler_set/2. #defined node_compiler_version_set/3. #defined compiler_supports_os/3. -#defined compiler_version_match/2. #defined allow_compiler/2. % compilers weighted by preference according to packages.yaml @@ -689,84 +695,101 @@ no_flags(Package, FlagType) %----------------------------------------------------------------------------- % How to optimize the spec (high to low priority) %----------------------------------------------------------------------------- -% weight root preferences higher -% -% TODO: how best to deal with this issue? It's not clear how best to -% weight all the constraints. Without this root preference, `spack solve -% hdf5` will pick mpich instead of openmpi, even if openmpi is the -% preferred provider, because openmpi has a version constraint on hwloc. -% It ends up choosing between settling for an old version of hwloc, or -% picking the second-best provider. This workaround weights root -% preferences higher so that hdf5's prefs are more important, but it's -% not clear this is a general solution. It would be nice to weight by -% distance to root, but that seems to slow down the solve a lot. -% -% One option is to make preferences hard constraints. Or maybe we need -% to look more closely at where a constraint came from and factor that -% into our weights. e.g., a non-default variant resulting from a version -% constraint counts like a version constraint. Needs more thought later. -% +% Each criterion below has: +% 1. an opt_criterion(ID, Name) fact that describes the criterion, and +% 2. a `#minimize{ 0@2 : #true }.` statement that ensures the criterion +% is displayed (clingo doesn't display sums over empty sets by default) -root(Package, 2) :- root(Package), node(Package). -root(Dependency, 1) :- not root(Dependency), node(Dependency). +% Minimize the number of deprecated versions being used +opt_criterion(16, "deprecated versions used"). +#minimize{ 0@16 : #true }. +#minimize{ 1@16,Package : deprecated(Package, _)}. % The highest priority is to minimize the: % 1. Version weight % 2. Number of variants with a non default value, if not set % for the root(Package) -#minimize { Weight@15 : root(Package),version_weight(Package, Weight)}. +opt_criterion(15, "version weight"). +#minimize{ 0@15 : #true }. +#minimize { Weight@15 : root(Package),version_weight(Package, Weight) }. + +opt_criterion(14, "number of non-default variants (roots)"). +#minimize{ 0@14 : #true }. #minimize { Weight@14,Package,Variant,Value : variant_not_default(Package, Variant, Value, Weight), root(Package) }. + % If the value is a multivalued variant there could be multiple % values set as default. Since a default value has a weight of 0 we % need to maximize their number below to ensure they're all set +opt_criterion(13, "multi-valued variants"). +#minimize{ 0@13 : #true }. #maximize { 1@13,Package,Variant,Value : variant_not_default(Package, Variant, Value, Weight), not variant_single_value(Package, Variant), root(Package) }. +opt_criterion(12, "preferred providers for roots"). +#minimize{ 0@12 : #true }. #minimize{ - Weight@13,Provider + Weight@12,Provider : provider_weight(Provider, Weight), root(Provider) }. % Try to use default variants or variants that have been set +opt_criterion(11, "number of non-default variants (non-roots)"). +#minimize{ 0@11 : #true }. #minimize { Weight@11,Package,Variant,Value : variant_not_default(Package, Variant, Value, Weight), not root(Package) }. + % Minimize the weights of the providers, i.e. use as much as % possible the most preferred providers +opt_criterion(9, "number of non-default providers (non-roots)"). +#minimize{ 0@9 : #true }. #minimize{ Weight@9,Provider : provider_weight(Provider, Weight), not root(Provider) }. + % If the value is a multivalued variant there could be multiple % values set as default. Since a default value has a weight of 0 we % need to maximize their number below to ensure they're all set +opt_criterion(8, "count of non-root multi-valued variants"). +#minimize{ 0@8 : #true }. #maximize { 1@8,Package,Variant,Value - : variant_not_default(Package, Variant, Value, Weight), + : variant_not_default(Package, Variant, Value, _), not variant_single_value(Package, Variant), not root(Package) }. % Try to minimize the number of compiler mismatches in the DAG. +opt_criterion(7, "compiler mismatches"). #minimize{ 0@7 : #true }. #minimize{ 1@7,Package,Dependency : compiler_mismatch(Package, Dependency) }. % Choose more recent versions for nodes +opt_criterion(6, "version badness"). +#minimize{ 0@6 : #true }. #minimize{ Weight@6,Package : version_weight(Package, Weight) }. % Try to use preferred compilers +opt_criterion(5, "non-preferred compilers"). +#minimize{ 0@5 : #true }. #minimize{ Weight@5,Package : compiler_weight(Package, Weight) }. -% Maximize the number of matches for targets in the DAG, try +% Minimize the number of mismatches for targets in the DAG, try % to select the preferred target. -#maximize{ Weight@4,Package : node_target_match(Package, Weight) }. +opt_criterion(4, "target mismatches"). +#minimize{ 0@4 : #true }. +#minimize{ 1@4,Package,Dependency : node_target_mismatch(Package, Dependency) }. + +opt_criterion(3, "non-preferred targets"). +#minimize{ 0@3 : #true }. #minimize{ Weight@3,Package : node_target_weight(Package, Weight) }. diff --git a/lib/spack/spack/solver/display.lp b/lib/spack/spack/solver/display.lp index fdc4e4088c..818ce2d90a 100644 --- a/lib/spack/spack/solver/display.lp +++ b/lib/spack/spack/solver/display.lp @@ -1,8 +1,16 @@ +% Copyright 2013-2021 Lawrence Livermore National Security, LLC and other +% Spack Project Developers. See the top-level COPYRIGHT file for details. +% +% SPDX-License-Identifier: (Apache-2.0 OR MIT) + %=============================================================================- % Display Results % % This section determines what parts of the model are printed at the end %============================================================================== + +% Spec-related functions. +% Used to build the result of the solve. #show node/1. #show depends_on/3. #show version/2. @@ -16,12 +24,10 @@ #show node_flag_compiler_default/1. #show node_flag_source/2. #show no_flags/2. - -#show variant_not_default/4. -#show provider_weight/2. -#show version_weight/2. -#show compiler_version_match/2. -#show compiler_weight/2. -#show node_target_match/2. -#show node_target_weight/2. #show external_spec_selected/2. + +% names of optimization criteria +#show opt_criterion/2. + +% deprecated packages +#show deprecated/2. diff --git a/lib/spack/spack/spec.py b/lib/spack/spack/spec.py index edb2c94300..609b20e238 100644 --- a/lib/spack/spack/spec.py +++ b/lib/spack/spack/spec.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -76,10 +76,8 @@ thing. Spack uses ~variant in directory names and in the canonical form of specs to avoid ambiguity. Both are provided because ~ can cause shell expansion when it is the first character in an id typed on the command line. """ -import base64 import sys import collections -import hashlib import itertools import operator import os @@ -108,6 +106,7 @@ import spack.solver import spack.store import spack.util.crypto import spack.util.executable +import spack.util.hash import spack.util.module_cmd as md import spack.util.prefix import spack.util.spack_json as sjson @@ -203,7 +202,7 @@ def colorize_spec(spec): return clr.colorize(re.sub(_separators, insert_color(), str(spec)) + '@.') -@lang.key_ordering +@lang.lazy_lexicographic_ordering class ArchSpec(object): def __init__(self, spec_or_platform_tuple=(None, None, None)): """ Architecture specification a package should be built with. @@ -252,8 +251,10 @@ class ArchSpec(object): return spec_like return ArchSpec(spec_like) - def _cmp_key(self): - return self.platform, self.os, self.target + def _cmp_iter(self): + yield self.platform + yield self.os + yield self.target def _dup(self, other): self.platform = other.platform @@ -534,7 +535,7 @@ class ArchSpec(object): return string in str(self) or string in self.target -@lang.key_ordering +@lang.lazy_lexicographic_ordering class CompilerSpec(object): """The CompilerSpec field represents the compiler or range of compiler versions that a package should be built with. CompilerSpecs have a @@ -623,8 +624,9 @@ class CompilerSpec(object): clone.versions = self.versions.copy() return clone - def _cmp_key(self): - return (self.name, self.versions) + def _cmp_iter(self): + yield self.name + yield self.versions def to_dict(self): d = syaml.syaml_dict([('name', self.name)]) @@ -648,7 +650,7 @@ class CompilerSpec(object): return str(self) -@lang.key_ordering +@lang.lazy_lexicographic_ordering class DependencySpec(object): """DependencySpecs connect two nodes in the DAG, and contain deptypes. @@ -686,10 +688,10 @@ class DependencySpec(object): self.deptypes + dp.canonical_deptype(type) ) - def _cmp_key(self): - return (self.parent.name if self.parent else None, - self.spec.name if self.spec else None, - self.deptypes) + def _cmp_iter(self): + yield self.parent.name if self.parent else None + yield self.spec.name if self.spec else None + yield self.deptypes def __str__(self): return "%s %s--> %s" % (self.parent.name if self.parent else None, @@ -747,8 +749,15 @@ class FlagMap(lang.HashableMap): clone[name] = value return clone - def _cmp_key(self): - return tuple((k, tuple(v)) for k, v in sorted(six.iteritems(self))) + def _cmp_iter(self): + for k, v in sorted(self.items()): + yield k + + def flags(): + for flag in v: + yield flag + + yield flags def __str__(self): sorted_keys = [k for k in sorted(self.keys()) if self[k] != []] @@ -1016,7 +1025,7 @@ class SpecBuildInterface(lang.ObjectWrapper): ) -@lang.key_ordering +@lang.lazy_lexicographic_ordering(set_hash=False) class Spec(object): #: Cache for spec's prefix, computed lazily in the corresponding property @@ -1060,7 +1069,7 @@ class Spec(object): self._hash = None self._build_hash = None - self._cmp_key_cache = None + self._dunder_hash = None self._package = None # Most of these are internal implementation details that can be @@ -1088,6 +1097,12 @@ class Spec(object): # external specs. None signal that it was not set yet. self.extra_attributes = None + # This attribute holds the original build copy of the spec if it is + # deployed differently than it was built. None signals that the spec + # is deployed "as built." + # Build spec should be the actual build spec unless marked dirty. + self._build_spec = None + if isinstance(spec_like, six.string_types): spec_list = SpecParser(self).parse(spec_like) if len(spec_list) > 1: @@ -1302,6 +1317,13 @@ class Spec(object): """ return self._concrete + @property + def spliced(self): + """Returns whether or not this Spec is being deployed as built i.e. + whether or not this Spec has ever been spliced. + """ + return any(s.build_spec is not s for s in self.traverse(root=True)) + def traverse(self, **kwargs): direction = kwargs.get('direction', 'children') depth = kwargs.get('depth', False) @@ -1369,7 +1391,16 @@ class Spec(object): cover = kwargs.get('cover', 'nodes') direction = kwargs.get('direction', 'children') order = kwargs.get('order', 'pre') - deptype = dp.canonical_deptype(deptype) + + # we don't want to run canonical_deptype every time through + # traverse, because it is somewhat expensive. This ensures we + # canonicalize only once. + canonical_deptype = kwargs.get("canonical_deptype", None) + if canonical_deptype is None: + deptype = dp.canonical_deptype(deptype) + kwargs["canonical_deptype"] = deptype + else: + deptype = canonical_deptype # Make sure kwargs have legal values; raise ValueError if not. def validate(name, val, allowed_values): @@ -1473,13 +1504,7 @@ class Spec(object): # this when we move to using package hashing on all specs. node_dict = self.to_node_dict(hash=hash) yaml_text = syaml.dump(node_dict, default_flow_style=True) - sha = hashlib.sha1(yaml_text.encode('utf-8')) - b32_hash = base64.b32encode(sha.digest()).lower() - - if sys.version_info[0] >= 3: - b32_hash = b32_hash.decode('utf-8') - - return b32_hash + return spack.util.hash.b32_hash(yaml_text) def _cached_hash(self, hash, length=None): """Helper function for storing a cached hash on the spec. @@ -1535,7 +1560,7 @@ class Spec(object): def dag_hash_bit_prefix(self, bits): """Get the first <bits> bits of the DAG hash as an integer type.""" - return base32_prefix_bits(self.dag_hash(), bits) + return spack.util.hash.base32_prefix_bits(self.dag_hash(), bits) def to_node_dict(self, hash=ht.dag_hash): """Create a dictionary representing the state of this Spec. @@ -1629,7 +1654,13 @@ class Spec(object): d['patches'] = variant._patches_in_order_of_appearance if hash.package_hash: - d['package_hash'] = self.package.content_hash() + package_hash = self.package.content_hash() + + # Full hashes are in bytes + if (not isinstance(package_hash, six.text_type) + and isinstance(package_hash, six.binary_type)): + package_hash = package_hash.decode('utf-8') + d['package_hash'] = package_hash deps = self.dependencies_dict(deptype=hash.deptype) if deps: @@ -1786,10 +1817,12 @@ class Spec(object): name = next(iter(node)) node = node[name] - spec = Spec(name, full_hash=node.get('full_hash', None)) + spec = Spec() + spec.name = name spec.namespace = node.get('namespace', None) spec._hash = node.get('hash', None) spec._build_hash = node.get('build_hash', None) + spec._full_hash = node.get('full_hash', None) if 'version' in node or 'versions' in node: spec.versions = vn.VersionList.from_dict(node) @@ -2263,6 +2296,8 @@ class Spec(object): # If replacement is external then trim the dependencies if replacement.external: if (spec._dependencies): + for dep in spec.dependencies(): + del dep._dependents[spec.name] changed = True spec._dependencies = DependencyMap() replacement._dependencies = DependencyMap() @@ -2536,17 +2571,27 @@ class Spec(object): else: self._old_concretize(tests) + def _mark_root_concrete(self, value=True): + """Mark just this spec (not dependencies) concrete.""" + if (not value) and self.concrete and self.package.installed: + return + self._normal = value + self._concrete = value + def _mark_concrete(self, value=True): """Mark this spec and its dependencies as concrete. Only for internal use -- client code should use "concretize" unless there is a need to force a spec to be concrete. """ + # if set to false, clear out all hashes (set to None or remove attr) + # may need to change references to respect None for s in self.traverse(): if (not value) and s.concrete and s.package.installed: continue - s._normal = value - s._concrete = value + elif not value: + s.clear_cached_hashes() + s._mark_root_concrete(value) def concretized(self, tests=False): """This is a non-destructive version of concretize(). @@ -3274,7 +3319,7 @@ class Spec(object): """Return list of any virtual deps in this spec.""" return [spec for spec in self.traverse() if spec.virtual] - @property + @property # type: ignore[misc] # decorated prop not supported in mypy @lang.memoized def patches(self): """Return patch objects for any patch sha256 sums on this Spec. @@ -3320,7 +3365,7 @@ class Spec(object): before possibly copying the dependencies of ``other`` onto ``self`` caches (bool or None): preserve cached fields such as - ``_normal``, ``_hash``, and ``_cmp_key_cache``. By + ``_normal``, ``_hash``, and ``_dunder_hash``. By default this is ``False`` if DAG structure would be changed by the copy, ``True`` if it's an exact copy. @@ -3357,6 +3402,7 @@ class Spec(object): self.compiler_flags = other.compiler_flags.copy() self.compiler_flags.spec = self self.variants = other.variants.copy() + self._build_spec = other._build_spec # FIXME: we manage _patches_in_order_of_appearance specially here # to keep it from leaking out of spec.py, but we should figure @@ -3393,13 +3439,13 @@ class Spec(object): if caches: self._hash = other._hash self._build_hash = other._build_hash - self._cmp_key_cache = other._cmp_key_cache + self._dunder_hash = other._dunder_hash self._normal = other._normal self._full_hash = other._full_hash else: self._hash = None self._build_hash = None - self._cmp_key_cache = None + self._dunder_hash = None self._normal = False self._full_hash = None @@ -3518,18 +3564,17 @@ class Spec(object): else: return any(s.satisfies(spec) for s in self.traverse(root=False)) - def sorted_deps(self): - """Return a list of all dependencies sorted by name.""" - deps = self.flat_dependencies() - return tuple(deps[name] for name in sorted(deps)) + def eq_dag(self, other, deptypes=True, vs=None, vo=None): + """True if the full dependency DAGs of specs are equal.""" + if vs is None: + vs = set() + if vo is None: + vo = set() - def _eq_dag(self, other, vs, vo, deptypes): - """Recursive helper for eq_dag and ne_dag. Does the actual DAG - traversal.""" vs.add(id(self)) vo.add(id(other)) - if self.ne_node(other): + if not self.eq_node(other): return False if len(self._dependencies) != len(other._dependencies): @@ -3557,58 +3602,38 @@ class Spec(object): continue # Recursive check for equality - if not s._eq_dag(o, vs, vo, deptypes): + if not s.eq_dag(o, deptypes, vs, vo): return False return True - def eq_dag(self, other, deptypes=True): - """True if the full dependency DAGs of specs are equal.""" - return self._eq_dag(other, set(), set(), deptypes) - - def ne_dag(self, other, deptypes=True): - """True if the full dependency DAGs of specs are not equal.""" - return not self.eq_dag(other, set(), set(), deptypes) - def _cmp_node(self): - """Comparison key for just *this node* and not its deps.""" - # Name or namespace None will lead to invalid comparisons for abstract - # specs. Replace them with the empty string, which is not a valid spec - # name nor namespace so it will not create spurious equalities. - return (self.name or '', - self.namespace or '', - tuple(self.versions), - self.variants, - self.architecture, - self.compiler, - self.compiler_flags) + """Yield comparable elements of just *this node* and not its deps.""" + yield self.name + yield self.namespace + yield self.versions + yield self.variants + yield self.compiler + yield self.compiler_flags + yield self.architecture def eq_node(self, other): """Equality with another spec, not including dependencies.""" - return self._cmp_node() == other._cmp_node() - - def ne_node(self, other): - """Inequality with another spec, not including dependencies.""" - return self._cmp_node() != other._cmp_node() - - def _cmp_key(self): - """This returns a key for the spec *including* DAG structure. - - The key is the concatenation of: - 1. A tuple describing this node in the DAG. - 2. The hash of each of this node's dependencies' cmp_keys. - """ - if self._cmp_key_cache: - return self._cmp_key_cache + return (other is not None) and lang.lazy_eq( + self._cmp_node, other._cmp_node + ) - dep_tuple = tuple( - (d.spec.name, hash(d.spec), tuple(sorted(d.deptypes))) - for name, d in sorted(self._dependencies.items())) + def _cmp_iter(self): + """Lazily yield components of self for comparison.""" + for item in self._cmp_node(): + yield item - key = (self._cmp_node(), dep_tuple) - if self._concrete: - self._cmp_key_cache = key - return key + def deps(): + for _, dep in sorted(self._dependencies.items()): + yield dep.spec.name + yield tuple(sorted(dep.deptypes)) + yield hash(dep.spec) + yield deps def colorized(self): return colorize_spec(self) @@ -3848,7 +3873,9 @@ class Spec(object): 'Format string terminated while reading attribute.' 'Missing terminating }.' ) - return out.getvalue() + + formatted_spec = out.getvalue() + return formatted_spec.strip() def old_format(self, format_string='$_$@$%@+$+$=', **kwargs): """ @@ -4104,12 +4131,12 @@ class Spec(object): kwargs.setdefault('color', None) return self.format(*args, **kwargs) - def dep_string(self): - return ''.join(" ^" + dep.format() for dep in self.sorted_deps()) - def __str__(self): - ret = self.format() + self.dep_string() - return ret.strip() + sorted_nodes = [self] + sorted( + self.traverse(root=False), key=lambda x: x.name + ) + spec_str = " ^".join(d.format() for d in sorted_nodes) + return spec_str.strip() def install_status(self): """Helper for tree to print DB install status.""" @@ -4217,6 +4244,105 @@ class Spec(object): # to give to the attribute the appropriate comparison semantic return self.architecture.target.microarchitecture + @property + def build_spec(self): + return self._build_spec or self + + @build_spec.setter + def build_spec(self, value): + self._build_spec = value + + def splice(self, other, transitive): + """Splices dependency "other" into this ("target") Spec, and return the + result as a concrete Spec. + If transitive, then other and its dependencies will be extrapolated to + a list of Specs and spliced in accordingly. + For example, let there exist a dependency graph as follows: + T + | \ + Z<-H + In this example, Spec T depends on H and Z, and H also depends on Z. + Suppose, however, that we wish to use a differently-built H, known as + H'. This function will splice in the new H' in one of two ways: + 1. transitively, where H' depends on the Z' it was built with, and the + new T* also directly depends on this new Z', or + 2. intransitively, where the new T* and H' both depend on the original + Z. + Since the Spec returned by this splicing function is no longer deployed + the same way it was built, any such changes are tracked by setting the + build_spec to point to the corresponding dependency from the original + Spec. + TODO: Extend this for non-concrete Specs. + """ + assert self.concrete + assert other.concrete + assert other.name in self + + # Multiple unique specs with the same name will collide, so the + # _dependents of these specs should not be trusted. + # Variants may also be ignored here for now... + + if transitive: + self_nodes = dict((s.name, s.copy(deps=False)) + for s in self.traverse(root=True) + if s.name not in other) + other_nodes = dict((s.name, s.copy(deps=False)) + for s in other.traverse(root=True)) + else: + # If we're not doing a transitive splice, then we only want the + # root of other. + self_nodes = dict((s.name, s.copy(deps=False)) + for s in self.traverse(root=True) + if s.name != other.name) + other_nodes = {other.name: other.copy(deps=False)} + + nodes = other_nodes.copy() + nodes.update(self_nodes) + + for name in nodes: + if name in self_nodes: + dependencies = self[name]._dependencies + for dep in dependencies: + nodes[name]._add_dependency(nodes[dep], + dependencies[dep].deptypes) + if any(dep not in self_nodes for dep in dependencies): + nodes[name].build_spec = self[name].build_spec + else: + dependencies = other[name]._dependencies + for dep in dependencies: + nodes[name]._add_dependency(nodes[dep], + dependencies[dep].deptypes) + if any(dep not in other_nodes for dep in dependencies): + nodes[name].build_spec = other[name].build_spec + + # Clear cached hashes + nodes[self.name].clear_cached_hashes() + return nodes[self.name] + + def clear_cached_hashes(self): + """ + Clears all cached hashes in a Spec, while preserving other properties. + """ + for attr in ht.SpecHashDescriptor.hash_types: + if hasattr(self, attr): + setattr(self, attr, None) + + def __hash__(self): + # If the spec is concrete, we leverage the DAG hash and just use + # a 64-bit prefix of it. The DAG hash has the advantage that it's + # computed once per concrete spec, and it's saved -- so if we + # read concrete specs we don't need to recompute the whole hash. + # This is good for large, unchanging specs. + if self.concrete: + if not self._dunder_hash: + self._dunder_hash = self.dag_hash_bit_prefix(64) + return self._dunder_hash + + # This is the normal hash for lazy_lexicographic_ordering. It's + # slow for large specs because it traverses the whole spec graph, + # so we hope it only runs on abstract specs, which are small. + return hash(lang.tuplify(self._cmp_iter)) + class LazySpecCache(collections.defaultdict): """Cache for Specs that uses a spec_like as key, and computes lazily @@ -4627,16 +4753,6 @@ def save_dependency_spec_yamls( fd.write(dep_spec.to_yaml(hash=ht.build_hash)) -def base32_prefix_bits(hash_string, bits): - """Return the first <bits> bits of a base32 string as an integer.""" - if bits > len(hash_string) * 5: - raise ValueError("Too many bits! Requested %d bit prefix of '%s'." - % (bits, hash_string)) - - hash_bytes = base64.b32decode(hash_string, casefold=True) - return spack.util.crypto.prefix_bits(hash_bytes, bits) - - class SpecParseError(spack.error.SpecError): """Wrapper for ParseError for when we're parsing specs.""" def __init__(self, parse_error): diff --git a/lib/spack/spack/spec_list.py b/lib/spack/spack/spec_list.py index 1aa0ab870e..4f7ae79ed9 100644 --- a/lib/spack/spack/spec_list.py +++ b/lib/spack/spack/spec_list.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/stage.py b/lib/spack/spack/stage.py index 719d408d84..c39b4d2f33 100644 --- a/lib/spack/spack/stage.py +++ b/lib/spack/spack/stage.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -16,6 +16,7 @@ import sys import tempfile from six import string_types from six import iteritems +from typing import Dict # novm import llnl.util.tty as tty from llnl.util.filesystem import mkdirp, can_access, install, install_tree @@ -221,7 +222,7 @@ class Stage(object): """ """Shared dict of all stage locks.""" - stage_locks = {} + stage_locks = {} # type: Dict[str, spack.util.lock.Lock] """Most staging is managed by Spack. DIYStage is one exception.""" managed_by_spack = True @@ -602,12 +603,12 @@ class Stage(object): """ Ensures the top-level (config:build_stage) directory exists. """ - # Emulate file permissions for tempfile.mkdtemp. + # User has full permissions and group has only read permissions if not os.path.exists(self.path): - mkdirp(self.path, mode=stat.S_IRWXU) + mkdirp(self.path, mode=stat.S_IRWXU | stat.S_IRGRP | stat.S_IXGRP) elif not os.path.isdir(self.path): os.remove(self.path) - mkdirp(self.path, mode=stat.S_IRWXU) + mkdirp(self.path, mode=stat.S_IRWXU | stat.S_IRGRP | stat.S_IXGRP) # Make sure we can actually do something with the stage we made. ensure_access(self.path) diff --git a/lib/spack/spack/store.py b/lib/spack/spack/store.py index 157ba00be6..5d05349b2d 100644 --- a/lib/spack/spack/store.py +++ b/lib/spack/spack/store.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -158,6 +158,8 @@ class Store(object): ): self.root = root self.unpadded_root = unpadded_root or root + self.projections = projections + self.hash_length = hash_length self.db = spack.database.Database( root, upstream_dbs=retrieve_upstream_dbs()) self.layout = spack.directory_layout.YamlDirectoryLayout( @@ -167,6 +169,27 @@ class Store(object): """Convenience function to reindex the store DB with its own layout.""" return self.db.reindex(self.layout) + def serialize(self): + """Return a pickle-able object that can be used to reconstruct + a store. + """ + return ( + self.root, self.unpadded_root, self.projections, self.hash_length + ) + + @staticmethod + def deserialize(token): + """Return a store reconstructed from a token created by + the serialize method. + + Args: + token: return value of the serialize method + + Returns: + Store object reconstructed from the token + """ + return Store(*token) + def _store(): """Get the singleton store instance.""" @@ -258,13 +281,12 @@ def use_store(store_or_path): """Use the store passed as argument within the context manager. Args: - store_or_path: either a Store object ot a path to where the - store resides + store_or_path: either a Store object ot a path to where the store resides Returns: Store object associated with the context manager's store """ - global store + global store, db, layout, root, unpadded_root # Normalize input arguments temporary_store = store_or_path @@ -272,8 +294,14 @@ def use_store(store_or_path): temporary_store = Store(store_or_path) # Swap the store with the one just constructed and return it + _ = store.db original_store, store = store, temporary_store + db, layout = store.db, store.layout + root, unpadded_root = store.root, store.unpadded_root + yield temporary_store # Restore the original store store = original_store + db, layout = original_store.db, original_store.layout + root, unpadded_root = original_store.root, original_store.unpadded_root diff --git a/lib/spack/spack/subprocess_context.py b/lib/spack/spack/subprocess_context.py index b8e157740f..3eee2125d2 100644 --- a/lib/spack/spack/subprocess_context.py +++ b/lib/spack/spack/subprocess_context.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -69,10 +69,12 @@ class PackageInstallContext(object): self.serialized_pkg = serialize(pkg) else: self.pkg = pkg + self.spack_working_dir = spack.main.spack_working_dir self.test_state = TestState() def restore(self): self.test_state.restore() + spack.main.spack_working_dir = self.spack_working_dir if _serialize: return pickle.load(self.serialized_pkg) else: @@ -91,18 +93,21 @@ class TestState(object): self.config = spack.config.config self.platform = spack.architecture.platform self.test_patches = store_patches() - - # TODO: transfer spack.store.store? note that you should not - # transfer spack.store.store and spack.store.db: 'db' is a - # shortcut that accesses the store (so transferring both can - # create an inconsistency). Some tests set 'db' directly, and - # others set 'store' + self.store_token = spack.store.store.serialize() def restore(self): if _serialize: spack.repo.path = spack.repo._path(self.repo_dirs) spack.config.config = self.config spack.architecture.platform = self.platform + + new_store = spack.store.Store.deserialize(self.store_token) + spack.store.store = new_store + spack.store.root = new_store.root + spack.store.unpadded_root = new_store.unpadded_root + spack.store.db = new_store.db + spack.store.layout = new_store.layout + self.test_patches.restore() diff --git a/lib/spack/spack/tengine.py b/lib/spack/spack/tengine.py index 15268e682d..fc2f19f57f 100644 --- a/lib/spack/spack/tengine.py +++ b/lib/spack/spack/tengine.py @@ -1,9 +1,10 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) import itertools import textwrap +from typing import List # novm import llnl.util.lang import six @@ -18,7 +19,7 @@ class ContextMeta(type): """ #: Keeps track of the context properties that have been added #: by the class that is being defined - _new_context_properties = [] + _new_context_properties = [] # type: List[str] def __new__(cls, name, bases, attr_dict): # Merge all the context properties that are coming from base classes diff --git a/lib/spack/spack/test/__init__.py b/lib/spack/spack/test/__init__.py index 9f87532b85..103eae6134 100644 --- a/lib/spack/spack/test/__init__.py +++ b/lib/spack/spack/test/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/abi.py b/lib/spack/spack/test/abi.py index dd41228941..ee2aa1f3e6 100644 --- a/lib/spack/spack/test/abi.py +++ b/lib/spack/spack/test/abi.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/architecture.py b/lib/spack/spack/test/architecture.py index 47e1e25fe8..e37818f10c 100644 --- a/lib/spack/spack/test/architecture.py +++ b/lib/spack/spack/test/architecture.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/bindist.py b/lib/spack/spack/test/bindist.py index 05cfda5019..1868d74ccc 100644 --- a/lib/spack/spack/test/bindist.py +++ b/lib/spack/spack/test/bindist.py @@ -1,96 +1,62 @@ -# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) - -""" -This test checks creating and install buildcaches -""" import os import sys +import platform + import py import pytest -import argparse -import platform -import spack.repo -import spack.store + import spack.binary_distribution as bindist -import spack.cmd.buildcache as buildcache -import spack.cmd.install as install -import spack.cmd.uninstall as uninstall -import spack.cmd.mirror as mirror +import spack.config import spack.hooks.sbang as sbang -from spack.main import SpackCommand +import spack.main import spack.mirror +import spack.repo +import spack.store import spack.util.gpg import spack.util.web as web_util + from spack.directory_layout import YamlDirectoryLayout from spack.spec import Spec - -def_install_path_scheme = '${ARCHITECTURE}/${COMPILERNAME}-${COMPILERVER}/${PACKAGE}-${VERSION}-${HASH}' # noqa: E501 -ndef_install_path_scheme = '${PACKAGE}/${VERSION}/${ARCHITECTURE}-${COMPILERNAME}-${COMPILERVER}-${HASH}' # noqa: E501 - -mirror_path_def = None -mirror_path_rel = None - -mirror_cmd = SpackCommand('mirror') -install_cmd = SpackCommand('install') -uninstall_cmd = SpackCommand('uninstall') -buildcache_cmd = SpackCommand('buildcache') +mirror_cmd = spack.main.SpackCommand('mirror') +install_cmd = spack.main.SpackCommand('install') +uninstall_cmd = spack.main.SpackCommand('uninstall') +buildcache_cmd = spack.main.SpackCommand('buildcache') @pytest.fixture(scope='function') def cache_directory(tmpdir): - old_cache_path = spack.caches.fetch_cache - tmpdir.ensure('fetch_cache', dir=True) - fsc = spack.fetch_strategy.FsCache(str(tmpdir.join('fetch_cache'))) - spack.config.caches = fsc - yield spack.config.caches - tmpdir.join('fetch_cache').remove() - spack.config.caches = old_cache_path - - -@pytest.fixture(scope='session') -def session_mirror_def(tmpdir_factory): - dir = tmpdir_factory.mktemp('mirror') - global mirror_path_rel - mirror_path_rel = dir - dir.ensure('build_cache', dir=True) - yield dir - dir.join('build_cache').remove() + fetch_cache_dir = tmpdir.ensure('fetch_cache', dir=True) + fsc = spack.fetch_strategy.FsCache(str(fetch_cache_dir)) + spack.config.caches, old_cache_path = fsc, spack.caches.fetch_cache + yield spack.config.caches -@pytest.fixture(scope='function') -def mirror_directory_def(session_mirror_def): - yield str(session_mirror_def) + fetch_cache_dir.remove() + spack.config.caches = old_cache_path -@pytest.fixture(scope='session') -def session_mirror_rel(tmpdir_factory): +@pytest.fixture(scope='module') +def mirror_dir(tmpdir_factory): dir = tmpdir_factory.mktemp('mirror') - global mirror_path_rel - mirror_path_rel = dir dir.ensure('build_cache', dir=True) - yield dir + yield str(dir) dir.join('build_cache').remove() @pytest.fixture(scope='function') -def mirror_directory_rel(session_mirror_rel): - yield(session_mirror_rel) - - -@pytest.fixture(scope='function') -def function_mirror(tmpdir): - mirror_dir = str(tmpdir.join('mirror')) - mirror_cmd('add', '--scope', 'site', 'test-mirror-func', - 'file://%s' % mirror_dir) +def test_mirror(mirror_dir): + mirror_url = 'file://%s' % mirror_dir + mirror_cmd('add', '--scope', 'site', 'test-mirror-func', mirror_url) yield mirror_dir mirror_cmd('rm', '--scope=site', 'test-mirror-func') -@pytest.fixture(scope='session') +@pytest.fixture(scope='module') def config_directory(tmpdir_factory): tmpdir = tmpdir_factory.mktemp('test_configs') # restore some sane defaults for packages and config @@ -116,8 +82,14 @@ def config_directory(tmpdir_factory): @pytest.fixture(scope='function') -def default_config(tmpdir_factory, config_directory, monkeypatch): - +def default_config( + tmpdir_factory, config_directory, monkeypatch, + install_mockery_mutable_config +): + # This fixture depends on install_mockery_mutable_config to ensure + # there is a clear order of initialization. The substitution of the + # config scopes here is done on top of the substitution that comes with + # install_mockery_mutable_config mutable_dir = tmpdir_factory.mktemp('mutable_config').join('tmp') config_directory.copy(mutable_dir) @@ -126,7 +98,7 @@ def default_config(tmpdir_factory, config_directory, monkeypatch): for name in ['site/%s' % platform.system().lower(), 'site', 'user']]) - monkeypatch.setattr(spack.config, 'config', cfg) + spack.config.config, old_config = cfg, spack.config.config # This is essential, otherwise the cache will create weird side effects # that will compromise subsequent tests if compilers.yaml is modified @@ -148,18 +120,25 @@ def default_config(tmpdir_factory, config_directory, monkeypatch): timeout = spack.config.get('config:connect_timeout') if not timeout: spack.config.set('config:connect_timeout', 10, scope='user') + yield spack.config.config + + spack.config.config = old_config mutable_dir.remove() @pytest.fixture(scope='function') def install_dir_default_layout(tmpdir): """Hooks a fake install directory with a default layout""" - real_store = spack.store.store - real_layout = spack.store.layout - spack.store.store = spack.store.Store(str(tmpdir.join('opt'))) - spack.store.layout = YamlDirectoryLayout(str(tmpdir.join('opt')), - path_scheme=def_install_path_scheme) # noqa: E501 + scheme = os.path.join( + '${architecture}', + '${compiler.name}-${compiler.version}', + '${name}-${version}-${hash}' + ) + real_store, real_layout = spack.store.store, spack.store.layout + opt_dir = tmpdir.join('opt') + spack.store.store = spack.store.Store(str(opt_dir)) + spack.store.layout = YamlDirectoryLayout(str(opt_dir), path_scheme=scheme) try: yield spack.store finally: @@ -170,11 +149,14 @@ def install_dir_default_layout(tmpdir): @pytest.fixture(scope='function') def install_dir_non_default_layout(tmpdir): """Hooks a fake install directory with a non-default layout""" - real_store = spack.store.store - real_layout = spack.store.layout - spack.store.store = spack.store.Store(str(tmpdir.join('opt'))) - spack.store.layout = YamlDirectoryLayout(str(tmpdir.join('opt')), - path_scheme=ndef_install_path_scheme) # noqa: E501 + scheme = os.path.join( + '${name}', '${version}', + '${architecture}-${compiler.name}-${compiler.version}-${hash}' + ) + real_store, real_layout = spack.store.store, spack.store.layout + opt_dir = tmpdir.join('opt') + spack.store.store = spack.store.Store(str(opt_dir)) + spack.store.layout = YamlDirectoryLayout(str(opt_dir), path_scheme=scheme) try: yield spack.store finally: @@ -190,308 +172,149 @@ else: @pytest.mark.requires_executables(*args) -@pytest.mark.disable_clean_stage_check @pytest.mark.maybeslow -@pytest.mark.usefixtures('default_config', 'cache_directory', - 'install_dir_default_layout') -def test_default_rpaths_create_install_default_layout(tmpdir, - mirror_directory_def, - install_mockery): +@pytest.mark.usefixtures( + 'default_config', 'cache_directory', 'install_dir_default_layout', + 'test_mirror' +) +def test_default_rpaths_create_install_default_layout(mirror_dir): """ Test the creation and installation of buildcaches with default rpaths into the default directory layout scheme. """ + gspec, cspec = Spec('garply').concretized(), Spec('corge').concretized() - gspec = Spec('garply') - gspec.concretize() - cspec = Spec('corge') - cspec.concretize() - - iparser = argparse.ArgumentParser() - install.setup_parser(iparser) - # Install some packages with dependent packages - iargs = iparser.parse_args(['--no-cache', cspec.name]) - install.install(iparser, iargs) - - global mirror_path_def - mirror_path_def = mirror_directory_def - mparser = argparse.ArgumentParser() - mirror.setup_parser(mparser) - margs = mparser.parse_args( - ['add', '--scope', 'site', 'test-mirror-def', 'file://%s' % mirror_path_def]) - mirror.mirror(mparser, margs) - margs = mparser.parse_args(['list']) - mirror.mirror(mparser, margs) - - # setup argument parser - parser = argparse.ArgumentParser() - buildcache.setup_parser(parser) - - # Set default buildcache args - create_args = ['create', '-a', '-u', '-d', str(mirror_path_def), - cspec.name] - install_args = ['install', '-a', '-u', cspec.name] + # Install 'corge' without using a cache + install_cmd('--no-cache', cspec.name) # Create a buildache - args = parser.parse_args(create_args) - buildcache.buildcache(parser, args) - # Test force overwrite create buildcache - create_args.insert(create_args.index('-a'), '-f') - args = parser.parse_args(create_args) - buildcache.buildcache(parser, args) - # create mirror index - args = parser.parse_args(['update-index', '-d', 'file://%s' % str(mirror_path_def)]) - buildcache.buildcache(parser, args) - # list the buildcaches in the mirror - args = parser.parse_args(['list', '-a', '-l', '-v']) - buildcache.buildcache(parser, args) + buildcache_cmd('create', '-au', '-d', mirror_dir, cspec.name) + # Test force overwrite create buildcache (-f option) + buildcache_cmd('create', '-auf', '-d', mirror_dir, cspec.name) + + # Create mirror index + mirror_url = 'file://{0}'.format(mirror_dir) + buildcache_cmd('update-index', '-d', mirror_url) + # List the buildcaches in the mirror + buildcache_cmd('list', '-alv') # Uninstall the package and deps - uparser = argparse.ArgumentParser() - uninstall.setup_parser(uparser) - uargs = uparser.parse_args(['-y', '--dependents', gspec.name]) - uninstall.uninstall(uparser, uargs) + uninstall_cmd('-y', '--dependents', gspec.name) - # test install - args = parser.parse_args(install_args) - buildcache.buildcache(parser, args) + # Test installing from build caches + buildcache_cmd('install', '-au', cspec.name) # This gives warning that spec is already installed - buildcache.buildcache(parser, args) - - # test overwrite install - install_args.insert(install_args.index('-a'), '-f') - args = parser.parse_args(install_args) - buildcache.buildcache(parser, args) + buildcache_cmd('install', '-au', cspec.name) - args = parser.parse_args(['keys', '-f']) - buildcache.buildcache(parser, args) + # Test overwrite install + buildcache_cmd('install', '-afu', cspec.name) - args = parser.parse_args(['list']) - buildcache.buildcache(parser, args) + buildcache_cmd('keys', '-f') + buildcache_cmd('list') - args = parser.parse_args(['list', '-a']) - buildcache.buildcache(parser, args) - - args = parser.parse_args(['list', '-l', '-v']) - buildcache.buildcache(parser, args) - bindist.clear_spec_cache() - spack.stage.purge() - margs = mparser.parse_args( - ['rm', '--scope', 'site', 'test-mirror-def']) - mirror.mirror(mparser, margs) + buildcache_cmd('list', '-a') + buildcache_cmd('list', '-l', '-v') @pytest.mark.requires_executables(*args) -@pytest.mark.disable_clean_stage_check @pytest.mark.maybeslow @pytest.mark.nomockstage -@pytest.mark.usefixtures('default_config', 'cache_directory', - 'install_dir_non_default_layout') -def test_default_rpaths_install_nondefault_layout(tmpdir, - install_mockery): +@pytest.mark.usefixtures( + 'default_config', 'cache_directory', 'install_dir_non_default_layout', + 'test_mirror' +) +def test_default_rpaths_install_nondefault_layout(mirror_dir): """ Test the creation and installation of buildcaches with default rpaths into the non-default directory layout scheme. """ - - gspec = Spec('garply') - gspec.concretize() - cspec = Spec('corge') - cspec.concretize() - - global mirror_path_def - mparser = argparse.ArgumentParser() - mirror.setup_parser(mparser) - margs = mparser.parse_args( - ['add', '--scope', 'site', 'test-mirror-def', 'file://%s' % mirror_path_def]) - mirror.mirror(mparser, margs) - - # setup argument parser - parser = argparse.ArgumentParser() - buildcache.setup_parser(parser) - - # Set default buildcache args - install_args = ['install', '-a', '-u', '%s' % cspec.name] + cspec = Spec('corge').concretized() # Install some packages with dependent packages # test install in non-default install path scheme - args = parser.parse_args(install_args) - buildcache.buildcache(parser, args) - # test force install in non-default install path scheme - install_args.insert(install_args.index('-a'), '-f') - args = parser.parse_args(install_args) - buildcache.buildcache(parser, args) + buildcache_cmd('install', '-au', cspec.name) - bindist.clear_spec_cache() - spack.stage.purge() - margs = mparser.parse_args( - ['rm', '--scope', 'site', 'test-mirror-def']) - mirror.mirror(mparser, margs) + # Test force install in non-default install path scheme + buildcache_cmd('install', '-auf', cspec.name) @pytest.mark.requires_executables(*args) -@pytest.mark.disable_clean_stage_check @pytest.mark.maybeslow @pytest.mark.nomockstage -@pytest.mark.usefixtures('default_config', 'cache_directory', - 'install_dir_default_layout') -def test_relative_rpaths_create_default_layout(tmpdir, - mirror_directory_rel, - install_mockery): +@pytest.mark.usefixtures( + 'default_config', 'cache_directory', 'install_dir_default_layout' +) +def test_relative_rpaths_create_default_layout(mirror_dir): """ Test the creation and installation of buildcaches with relative rpaths into the default directory layout scheme. """ - gspec = Spec('garply') - gspec.concretize() - cspec = Spec('corge') - cspec.concretize() + gspec, cspec = Spec('garply').concretized(), Spec('corge').concretized() - global mirror_path_rel - mirror_path_rel = mirror_directory_rel - # Install patchelf needed for relocate in linux test environment - iparser = argparse.ArgumentParser() - install.setup_parser(iparser) - # Install some packages with dependent packages - iargs = iparser.parse_args(['--no-cache', cspec.name]) - install.install(iparser, iargs) - - # setup argument parser - parser = argparse.ArgumentParser() - buildcache.setup_parser(parser) - - # set default buildcache args - create_args = ['create', '-a', '-u', '-r', '-d', - str(mirror_path_rel), - cspec.name] - - # create build cache with relatived rpaths - args = parser.parse_args(create_args) - buildcache.buildcache(parser, args) - # create mirror index - args = parser.parse_args(['update-index', '-d', 'file://%s' % str(mirror_path_rel)]) - buildcache.buildcache(parser, args) - # Uninstall the package and deps - uparser = argparse.ArgumentParser() - uninstall.setup_parser(uparser) - uargs = uparser.parse_args(['-y', '--dependents', gspec.name]) - uninstall.uninstall(uparser, uargs) + # Install 'corge' without using a cache + install_cmd('--no-cache', cspec.name) - bindist.clear_spec_cache() - spack.stage.purge() + # Create build cache with relative rpaths + buildcache_cmd( + 'create', '-aur', '-d', mirror_dir, cspec.name + ) + + # Create mirror index + mirror_url = 'file://%s' % mirror_dir + buildcache_cmd('update-index', '-d', mirror_url) + + # Uninstall the package and deps + uninstall_cmd('-y', '--dependents', gspec.name) @pytest.mark.requires_executables(*args) -@pytest.mark.disable_clean_stage_check @pytest.mark.maybeslow @pytest.mark.nomockstage -@pytest.mark.usefixtures('default_config', 'cache_directory', - 'install_dir_default_layout') -def test_relative_rpaths_install_default_layout(tmpdir, - install_mockery): +@pytest.mark.usefixtures( + 'default_config', 'cache_directory', 'install_dir_default_layout', + 'test_mirror' +) +def test_relative_rpaths_install_default_layout(mirror_dir): """ Test the creation and installation of buildcaches with relative rpaths into the default directory layout scheme. """ + gspec, cspec = Spec('garply').concretized(), Spec('corge').concretized() - gspec = Spec('garply') - gspec.concretize() - cspec = Spec('corge') - cspec.concretize() - - global mirror_path_rel - mparser = argparse.ArgumentParser() - mirror.setup_parser(mparser) - margs = mparser.parse_args( - ['add', '--scope', 'site', 'test-mirror-rel', 'file://%s' % mirror_path_rel]) - mirror.mirror(mparser, margs) - - iparser = argparse.ArgumentParser() - install.setup_parser(iparser) - - # setup argument parser - parser = argparse.ArgumentParser() - buildcache.setup_parser(parser) - - # set default buildcache args - install_args = ['install', '-a', '-u', '-f', - cspec.name] - - # install buildcache created with relativized rpaths - args = parser.parse_args(install_args) - buildcache.buildcache(parser, args) + # Install buildcache created with relativized rpaths + buildcache_cmd('install', '-auf', cspec.name) # This gives warning that spec is already installed - buildcache.buildcache(parser, args) + buildcache_cmd('install', '-auf', cspec.name) # Uninstall the package and deps - uparser = argparse.ArgumentParser() - uninstall.setup_parser(uparser) - uargs = uparser.parse_args(['-y', '--dependents', gspec.name]) - uninstall.uninstall(uparser, uargs) - - # install build cache - buildcache.buildcache(parser, args) + uninstall_cmd('-y', '--dependents', gspec.name) - # test overwrite install - install_args.insert(install_args.index('-a'), '-f') - args = parser.parse_args(install_args) - buildcache.buildcache(parser, args) + # Install build cache + buildcache_cmd('install', '-auf', cspec.name) - bindist.clear_spec_cache() - spack.stage.purge() - margs = mparser.parse_args( - ['rm', '--scope', 'site', 'test-mirror-rel']) - mirror.mirror(mparser, margs) + # Test overwrite install + buildcache_cmd('install', '-auf', cspec.name) @pytest.mark.requires_executables(*args) -@pytest.mark.disable_clean_stage_check @pytest.mark.maybeslow @pytest.mark.nomockstage -@pytest.mark.usefixtures('default_config', 'cache_directory', - 'install_dir_non_default_layout') -def test_relative_rpaths_install_nondefault(tmpdir, - install_mockery): +@pytest.mark.usefixtures( + 'default_config', 'cache_directory', 'install_dir_non_default_layout', + 'test_mirror' +) +def test_relative_rpaths_install_nondefault(mirror_dir): """ Test the installation of buildcaches with relativized rpaths into the non-default directory layout scheme. """ + cspec = Spec('corge').concretized() - gspec = Spec('garply') - gspec.concretize() - cspec = Spec('corge') - cspec.concretize() - - global mirror_path_rel - - mparser = argparse.ArgumentParser() - mirror.setup_parser(mparser) - margs = mparser.parse_args( - ['add', '--scope', 'site', 'test-mirror-rel', 'file://%s' % mirror_path_rel]) - mirror.mirror(mparser, margs) - - iparser = argparse.ArgumentParser() - install.setup_parser(iparser) - - # setup argument parser - parser = argparse.ArgumentParser() - buildcache.setup_parser(parser) - - # Set default buildcache args - install_args = ['install', '-a', '-u', '-f', '%s' % cspec.name] - - # test install in non-default install path scheme and relative path - args = parser.parse_args(install_args) - buildcache.buildcache(parser, args) - - bindist.clear_spec_cache() - spack.stage.purge() - margs = mparser.parse_args( - ['rm', '--scope', 'site', 'test-mirror-rel']) - mirror.mirror(mparser, margs) + # Test install in non-default install path scheme and relative path + buildcache_cmd('install', '-auf', cspec.name) @pytest.mark.skipif(not spack.util.gpg.has_gpg(), @@ -534,38 +357,20 @@ def test_push_and_fetch_keys(mock_gnupghome): @pytest.mark.requires_executables(*args) -@pytest.mark.disable_clean_stage_check @pytest.mark.maybeslow @pytest.mark.nomockstage -@pytest.mark.usefixtures('default_config', 'cache_directory', - 'install_dir_non_default_layout') -def test_built_spec_cache(tmpdir, - install_mockery): +@pytest.mark.usefixtures( + 'default_config', 'cache_directory', 'install_dir_non_default_layout', + 'test_mirror' +) +def test_built_spec_cache(mirror_dir): """ Because the buildcache list command fetches the buildcache index and uses it to populate the binary_distribution built spec cache, when this test calls get_mirrors_for_spec, it is testing the popluation of that cache from a buildcache index. """ - global mirror_path_rel - - mparser = argparse.ArgumentParser() - mirror.setup_parser(mparser) - margs = mparser.parse_args( - ['add', '--scope', 'site', 'test-mirror-rel', 'file://%s' % mirror_path_rel]) - mirror.mirror(mparser, margs) - - # setup argument parser - parser = argparse.ArgumentParser() - buildcache.setup_parser(parser) - - list_args = ['list', '-a', '-l'] - args = parser.parse_args(list_args) - buildcache.buildcache(parser, args) - - gspec = Spec('garply') - gspec.concretize() + buildcache_cmd('list', '-a', '-l') - cspec = Spec('corge') - cspec.concretize() + gspec, cspec = Spec('garply').concretized(), Spec('corge').concretized() full_hash_map = { 'garply': gspec.full_hash(), @@ -590,12 +395,6 @@ def test_built_spec_cache(tmpdir, assert(result['mirror_url'] not in cspec_mirrors) cspec_mirrors[result['mirror_url']] = True - bindist.clear_spec_cache() - - margs = mparser.parse_args( - ['rm', '--scope', 'site', 'test-mirror-rel']) - mirror.mirror(mparser, margs) - def fake_full_hash(spec): # Generate an arbitrary hash that is intended to be different than @@ -604,8 +403,11 @@ def fake_full_hash(spec): return 'tal4c7h4z0gqmixb1eqa92mjoybxn5l6' -def test_spec_needs_rebuild(install_mockery_mutable_config, mock_packages, - mock_fetch, monkeypatch, tmpdir): +@pytest.mark.usefixtures( + 'install_mockery_mutable_config', 'mock_packages', 'mock_fetch', + 'test_mirror' +) +def test_spec_needs_rebuild(monkeypatch, tmpdir): """Make sure needs_rebuild properly compares remote full_hash against locally computed one, avoiding unnecessary rebuilds""" @@ -613,8 +415,6 @@ def test_spec_needs_rebuild(install_mockery_mutable_config, mock_packages, mirror_dir = tmpdir.join('mirror_dir') mirror_url = 'file://{0}'.format(mirror_dir.strpath) - mirror_cmd('add', 'test-mirror', mirror_url) - s = Spec('libdwarf').concretized() # Install a package @@ -683,47 +483,49 @@ def test_generate_indices_exception(monkeypatch, capfd): assert expect in err -@pytest.mark.usefixtures('mock_fetch') -def test_update_sbang(tmpdir, install_mockery, function_mirror): - """ - Test the creation and installation of buildcaches with default rpaths +@pytest.mark.usefixtures('mock_fetch', 'install_mockery') +def test_update_sbang(tmpdir, test_mirror): + """Test the creation and installation of buildcaches with default rpaths into the non-default directory layout scheme, triggering an update of the sbang. """ - - # Save the original store and layout before we touch ANYTHING. - real_store = spack.store.store - real_layout = spack.store.layout - + scheme = os.path.join( + '${name}', '${version}', + '${architecture}-${compiler.name}-${compiler.version}-${hash}' + ) + spec_str = 'old-sbang' # Concretize a package with some old-fashioned sbang lines. - sspec = Spec('old-sbang') - sspec.concretize() + old_spec = Spec(spec_str).concretized() + old_spec_hash_str = '/{0}'.format(old_spec.dag_hash()) # Need a fake mirror with *function* scope. - mirror_dir = function_mirror + mirror_dir = test_mirror + mirror_url = 'file://{0}'.format(mirror_dir) - # Assumes all commands will concretize sspec the same way. - install_cmd('--no-cache', sspec.name) + # Assume all commands will concretize old_spec the same way. + install_cmd('--no-cache', old_spec.name) # Create a buildcache with the installed spec. - buildcache_cmd('create', '-u', '-a', '-d', mirror_dir, - '/%s' % sspec.dag_hash()) + buildcache_cmd('create', '-u', '-a', '-d', mirror_dir, old_spec_hash_str) # Need to force an update of the buildcache index - buildcache_cmd('update-index', '-d', 'file://%s' % mirror_dir) + buildcache_cmd('update-index', '-d', mirror_url) # Uninstall the original package. - uninstall_cmd('-y', '/%s' % sspec.dag_hash()) + uninstall_cmd('-y', old_spec_hash_str) - try: - # New install tree locations... - # Too fine-grained to do be done in a fixture - spack.store.store = spack.store.Store(str(tmpdir.join('newtree'))) - spack.store.layout = YamlDirectoryLayout(str(tmpdir.join('newtree')), - path_scheme=ndef_install_path_scheme) # noqa: E501 + # Switch the store to the new install tree locations + newtree_dir = tmpdir.join('newtree') + s = spack.store.Store(str(newtree_dir)) + s.layout = YamlDirectoryLayout(str(newtree_dir), path_scheme=scheme) + + with spack.store.use_store(s): + new_spec = Spec('old-sbang') + new_spec.concretize() + assert new_spec.dag_hash() == old_spec.dag_hash() # Install package from buildcache - buildcache_cmd('install', '-a', '-u', '-f', sspec.name) + buildcache_cmd('install', '-a', '-u', '-f', new_spec.name) # Continue blowing away caches bindist.clear_spec_cache() @@ -734,25 +536,19 @@ def test_update_sbang(tmpdir, install_mockery, function_mirror): #!/usr/bin/env python {1} - '''.format(sbang.sbang_shebang_line(), sspec.prefix.bin) +'''.format(sbang.sbang_shebang_line(), new_spec.prefix.bin) sbang_style_2_expected = '''{0} #!/usr/bin/env python {1} - '''.format(sbang.sbang_shebang_line(), sspec.prefix.bin) +'''.format(sbang.sbang_shebang_line(), new_spec.prefix.bin) - installed_script_style_1_path = \ - sspec.prefix.bin.join('sbang-style-1.sh') + installed_script_style_1_path = new_spec.prefix.bin.join('sbang-style-1.sh') assert sbang_style_1_expected == \ open(str(installed_script_style_1_path)).read() - installed_script_style_2_path = \ - sspec.prefix.bin.join('sbang-style-2.sh') + installed_script_style_2_path = new_spec.prefix.bin.join('sbang-style-2.sh') assert sbang_style_2_expected == \ open(str(installed_script_style_2_path)).read() - uninstall_cmd('-y', '/%s' % sspec.dag_hash()) - - finally: - spack.store.store = real_store - spack.store.layout = real_layout + uninstall_cmd('-y', '/%s' % new_spec.dag_hash()) diff --git a/lib/spack/spack/test/bootstrap.py b/lib/spack/spack/test/bootstrap.py index 56083b327f..97687ddb4d 100644 --- a/lib/spack/spack/test/bootstrap.py +++ b/lib/spack/spack/test/bootstrap.py @@ -22,5 +22,5 @@ def test_store_is_restored_correctly_after_bootstrap(mutable_config, tmpdir): # Test that within the context manager we use the bootstrap store # and that outside we restore the correct location with spack.bootstrap.ensure_bootstrap_configuration(): - assert str(spack.store.root) == spack.paths.user_bootstrap_store - assert str(spack.store.root) == user_path + assert spack.store.root == spack.paths.user_bootstrap_store + assert spack.store.root == user_path diff --git a/lib/spack/spack/test/build_distribution.py b/lib/spack/spack/test/build_distribution.py index 2d567aee14..873a57a0e9 100644 --- a/lib/spack/spack/test/build_distribution.py +++ b/lib/spack/spack/test/build_distribution.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/build_environment.py b/lib/spack/spack/test/build_environment.py index b374a7c8ce..7aca1dea59 100644 --- a/lib/spack/spack/test/build_environment.py +++ b/lib/spack/spack/test/build_environment.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -14,9 +14,9 @@ import spack.spec import spack.util.spack_yaml as syaml from spack.paths import build_env_path from spack.build_environment import dso_suffix, _static_to_shared_library +from spack.build_environment import determine_number_of_jobs from spack.util.executable import Executable from spack.util.environment import EnvironmentModifications - from llnl.util.filesystem import LibraryList, HeaderList @@ -379,3 +379,22 @@ def test_setting_dtags_based_on_config( dtags_to_add = modifications['SPACK_DTAGS_TO_ADD'][0] assert dtags_to_add.value == expected_flag + + +def test_build_jobs_sequential_is_sequential(): + assert determine_number_of_jobs( + parallel=False, command_line=8, config_default=8, max_cpus=8) == 1 + + +def test_build_jobs_command_line_overrides(): + assert determine_number_of_jobs( + parallel=True, command_line=10, config_default=1, max_cpus=1) == 10 + assert determine_number_of_jobs( + parallel=True, command_line=10, config_default=100, max_cpus=100) == 10 + + +def test_build_jobs_defaults(): + assert determine_number_of_jobs( + parallel=True, command_line=None, config_default=1, max_cpus=10) == 1 + assert determine_number_of_jobs( + parallel=True, command_line=None, config_default=100, max_cpus=10) == 10 diff --git a/lib/spack/spack/test/build_system_guess.py b/lib/spack/spack/test/build_system_guess.py index 6c2e3c6be5..ab9735fd48 100644 --- a/lib/spack/spack/test/build_system_guess.py +++ b/lib/spack/spack/test/build_system_guess.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/build_systems.py b/lib/spack/spack/test/build_systems.py index 1606ba8ebd..b02da380a8 100644 --- a/lib/spack/spack/test/build_systems.py +++ b/lib/spack/spack/test/build_systems.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/buildrequest.py b/lib/spack/spack/test/buildrequest.py index 5918838edc..22d7b5fff1 100644 --- a/lib/spack/spack/test/buildrequest.py +++ b/lib/spack/spack/test/buildrequest.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/buildtask.py b/lib/spack/spack/test/buildtask.py index f82ff6861a..44faf6244d 100644 --- a/lib/spack/spack/test/buildtask.py +++ b/lib/spack/spack/test/buildtask.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cache_fetch.py b/lib/spack/spack/test/cache_fetch.py index 3b4c3cb887..8771769307 100644 --- a/lib/spack/spack/test/cache_fetch.py +++ b/lib/spack/spack/test/cache_fetch.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cc.py b/lib/spack/spack/test/cc.py index 7b8d34fbde..20ec946835 100644 --- a/lib/spack/spack/test/cc.py +++ b/lib/spack/spack/test/cc.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -28,6 +28,7 @@ test_args = [ '-Wl,--end-group', '-Xlinker', '-rpath', '-Xlinker', '/third/rpath', '-Xlinker', '-rpath', '-Xlinker', '/fourth/rpath', + '-Wl,--rpath,/fifth/rpath', '-Wl,--rpath', '-Wl,/sixth/rpath', '-llib3', '-llib4', 'arg5', 'arg6'] @@ -45,11 +46,13 @@ test_library_paths = [ test_wl_rpaths = [ '-Wl,-rpath,/first/rpath', '-Wl,-rpath,/second/rpath', - '-Wl,-rpath,/third/rpath', '-Wl,-rpath,/fourth/rpath'] + '-Wl,-rpath,/third/rpath', '-Wl,-rpath,/fourth/rpath', + '-Wl,-rpath,/fifth/rpath', '-Wl,-rpath,/sixth/rpath'] test_rpaths = [ '-rpath', '/first/rpath', '-rpath', '/second/rpath', - '-rpath', '/third/rpath', '-rpath', '/fourth/rpath'] + '-rpath', '/third/rpath', '-rpath', '/fourth/rpath', + '-rpath', '/fifth/rpath', '-rpath', '/sixth/rpath'] test_args_without_paths = [ 'arg1', @@ -347,15 +350,15 @@ def test_ccld_deps_isystem(): with set_env(SPACK_INCLUDE_DIRS='xinc:yinc:zinc', SPACK_RPATH_DIRS='xlib:ylib:zlib', SPACK_LINK_DIRS='xlib:ylib:zlib'): - mytest_args = test_args + ['-isystemfooinc'] + mytest_args = test_args + ['-isystem', 'fooinc'] check_args( cc, mytest_args, [real_cc] + test_include_paths + - ['-isystemfooinc', - '-isystemxinc', - '-isystemyinc', - '-isystemzinc'] + + ['-isystem', 'fooinc', + '-isystem', 'xinc', + '-isystem', 'yinc', + '-isystem', 'zinc'] + test_library_paths + ['-Lxlib', '-Lylib', @@ -429,20 +432,20 @@ def test_ccld_with_system_dirs_isystem(): SPACK_RPATH_DIRS='xlib:ylib:zlib', SPACK_LINK_DIRS='xlib:ylib:zlib'): - sys_path_args = ['-isystem/usr/include', + sys_path_args = ['-isystem', '/usr/include', '-L/usr/local/lib', '-Wl,-rpath,/usr/lib64', - '-isystem/usr/local/include', + '-isystem', '/usr/local/include', '-L/lib64/'] check_args( cc, sys_path_args + test_args, [real_cc] + test_include_paths + - ['-isystemxinc', - '-isystemyinc', - '-isystemzinc'] + - ['-isystem/usr/include', - '-isystem/usr/local/include'] + + ['-isystem', 'xinc', + '-isystem', 'yinc', + '-isystem', 'zinc'] + + ['-isystem', '/usr/include', + '-isystem', '/usr/local/include'] + test_library_paths + ['-Lxlib', '-Lylib', @@ -619,3 +622,24 @@ def test_filter_enable_new_dtags(wrapper_flags): result = cc(*(test_args + ['-Wl,--enable-new-dtags']), output=str) result = result.strip().split('\n') assert '-Wl,--enable-new-dtags' not in result + + +@pytest.mark.regression('22643') +def test_linker_strips_loopopt(wrapper_flags): + with set_env(SPACK_TEST_COMMAND='dump-args'): + # ensure that -loopopt=0 is not present in ld mode + result = ld(*(test_args + ["-loopopt=0"]), output=str) + result = result.strip().split('\n') + assert '-loopopt=0' not in result + + # ensure that -loopopt=0 is not present in ccld mode + result = cc(*(test_args + ["-loopopt=0"]), output=str) + result = result.strip().split('\n') + assert '-loopopt=0' not in result + + # ensure that -loopopt=0 *is* present in cc mode + # The "-c" argument is needed for cc to be detected + # as compile only (cc) mode. + result = cc(*(test_args + ["-loopopt=0", "-c", "x.c"]), output=str) + result = result.strip().split('\n') + assert '-loopopt=0' in result diff --git a/lib/spack/spack/test/ci.py b/lib/spack/spack/test/ci.py index f1986b34d0..dae5066bf8 100644 --- a/lib/spack/spack/test/ci.py +++ b/lib/spack/spack/test/ci.py @@ -1,4 +1,4 @@ -# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -114,6 +114,7 @@ def test_get_concrete_specs(config, mock_packages): assert('archive-files' in spec_map) +@pytest.mark.maybeslow def test_register_cdash_build(): build_name = 'Some pkg' base_url = 'http://cdash.fake.org' diff --git a/lib/spack/spack/test/cmd/__init__.py b/lib/spack/spack/test/cmd/__init__.py index 9f87532b85..103eae6134 100644 --- a/lib/spack/spack/test/cmd/__init__.py +++ b/lib/spack/spack/test/cmd/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/activate.py b/lib/spack/spack/test/cmd/activate.py index a3546230f1..48806c5833 100644 --- a/lib/spack/spack/test/cmd/activate.py +++ b/lib/spack/spack/test/cmd/activate.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/analyze.py b/lib/spack/spack/test/cmd/analyze.py new file mode 100644 index 0000000000..6d164063ab --- /dev/null +++ b/lib/spack/spack/test/cmd/analyze.py @@ -0,0 +1,176 @@ +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +import os +import pytest + +import spack.config +import spack.package +import spack.cmd.install + +from spack.spec import Spec +import spack.util.spack_json as sjson +from spack.main import SpackCommand + +install = SpackCommand('install') +analyze = SpackCommand('analyze') + + +def test_test_package_not_installed(mock_fetch, install_mockery_mutable_config): + # We cannot run an analysis for a package not installed + out = analyze('run', 'libdwarf', fail_on_error=False) + assert "==> Error: Spec 'libdwarf' matches no installed packages.\n" in out + + +def test_analyzer_get_install_dir(mock_fetch, install_mockery_mutable_config): + """ + Test that we cannot get an analyzer directory without a spec package. + """ + spec = Spec('libdwarf').concretized() + assert 'libdwarf' in spack.analyzers.analyzer_base.get_analyzer_dir(spec) + + # Case 1: spec is missing attribute for package + with pytest.raises(SystemExit): + spack.analyzers.analyzer_base.get_analyzer_dir(None) + + class Packageless(object): + package = None + + # Case 2: spec has package attribute, but it's None + with pytest.raises(SystemExit): + spack.analyzers.analyzer_base.get_analyzer_dir(Packageless()) + + +def test_malformed_analyzer(mock_fetch, install_mockery_mutable_config): + """ + Test that an analyzer missing needed attributes is invalid. + """ + from spack.analyzers.analyzer_base import AnalyzerBase + + # Missing attribute description + class MyAnalyzer(AnalyzerBase): + name = "my_analyzer" + outfile = "my_analyzer_output.txt" + + spec = Spec('libdwarf').concretized() + with pytest.raises(SystemExit): + MyAnalyzer(spec) + + +def test_analyze_output(tmpdir, mock_fetch, install_mockery_mutable_config): + """ + Test that an analyzer errors if requested name does not exist. + """ + install('libdwarf') + install('python@3.8') + analyzer_dir = tmpdir.join('analyzers') + + # An analyzer that doesn't exist should not work + out = analyze('run', '-a', 'pusheen', 'libdwarf', fail_on_error=False) + assert '==> Error: Analyzer pusheen does not exist\n' in out + + # We will output to this analyzer directory + analyzer_dir = tmpdir.join('analyzers') + out = analyze('run', '-a', 'install_files', '-p', str(analyzer_dir), 'libdwarf') + + # Ensure that if we run again without over write, we don't run + out = analyze('run', '-a', 'install_files', '-p', str(analyzer_dir), 'libdwarf') + assert "skipping" in out + + # With overwrite it should run + out = analyze('run', '-a', 'install_files', '-p', str(analyzer_dir), + '--overwrite', 'libdwarf') + assert "==> Writing result to" in out + + +def _run_analyzer(name, package, tmpdir): + """ + A shared function to test that an analyzer runs. + + We return the output file for further inspection. + """ + analyzer = spack.analyzers.get_analyzer(name) + analyzer_dir = tmpdir.join('analyzers') + out = analyze('run', '-a', analyzer.name, '-p', str(analyzer_dir), package) + + assert "==> Writing result to" in out + assert "/%s/%s\n" % (analyzer.name, analyzer.outfile) in out + + # The output file should exist + output_file = out.strip('\n').split(' ')[-1].strip() + assert os.path.exists(output_file) + return output_file + + +def test_installfiles_analyzer(tmpdir, mock_fetch, install_mockery_mutable_config): + """ + test the install files analyzer + """ + install('libdwarf') + output_file = _run_analyzer("install_files", "libdwarf", tmpdir) + + # Ensure it's the correct content + with open(output_file, 'r') as fd: + content = sjson.load(fd.read()) + + basenames = set() + for key, attrs in content.items(): + basenames.add(os.path.basename(key)) + + # Check for a few expected files + for key in ['.spack', 'libdwarf', 'packages', 'repo.yaml', 'repos']: + assert key in basenames + + +def test_environment_analyzer(tmpdir, mock_fetch, install_mockery_mutable_config): + """ + test the environment variables analyzer. + """ + install('libdwarf') + output_file = _run_analyzer("environment_variables", "libdwarf", tmpdir) + with open(output_file, 'r') as fd: + content = sjson.load(fd.read()) + + # Check a few expected keys + for key in ['SPACK_CC', 'SPACK_COMPILER_SPEC', 'SPACK_ENV_PATH']: + assert key in content + + # The analyzer should return no result if the output file does not exist. + spec = Spec('libdwarf').concretized() + env_file = os.path.join(spec.package.prefix, '.spack', 'spack-build-env.txt') + assert os.path.exists(env_file) + os.remove(env_file) + analyzer = spack.analyzers.get_analyzer("environment_variables") + analyzer_dir = tmpdir.join('analyzers') + result = analyzer(spec, analyzer_dir).run() + assert "environment_variables" in result + assert not result['environment_variables'] + + +def test_list_analyzers(): + """ + test that listing analyzers shows all the possible analyzers. + """ + from spack.analyzers import analyzer_types + + # all cannot be an analyzer + assert "all" not in analyzer_types + + # All types should be present! + out = analyze('list-analyzers') + for analyzer_type in analyzer_types: + assert analyzer_type in out + + +def test_configargs_analyzer(tmpdir, mock_fetch, install_mockery_mutable_config): + """ + test the config args analyzer. + + Since we don't have any, this should return an empty result. + """ + install('libdwarf') + analyzer_dir = tmpdir.join('analyzers') + out = analyze('run', '-a', 'config_args', '-p', str(analyzer_dir), 'libdwarf') + assert out == '' diff --git a/lib/spack/spack/test/cmd/arch.py b/lib/spack/spack/test/cmd/arch.py index ec3c92a3d0..cf4969ce28 100644 --- a/lib/spack/spack/test/cmd/arch.py +++ b/lib/spack/spack/test/cmd/arch.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/blame.py b/lib/spack/spack/test/cmd/blame.py index 61bdf41084..5ca69bb7b4 100644 --- a/lib/spack/spack/test/cmd/blame.py +++ b/lib/spack/spack/test/cmd/blame.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/build_env.py b/lib/spack/spack/test/cmd/build_env.py index b7de759a2e..d73b5bb972 100644 --- a/lib/spack/spack/test/cmd/build_env.py +++ b/lib/spack/spack/test/cmd/build_env.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -6,9 +6,9 @@ from six.moves import cPickle import pytest -from spack.main import SpackCommand, SpackCommandError +from spack.main import SpackCommand -info = SpackCommand('build-env') +build_env = SpackCommand('build-env') @pytest.mark.parametrize('pkg', [ @@ -17,17 +17,24 @@ info = SpackCommand('build-env') ]) @pytest.mark.usefixtures('config') def test_it_just_runs(pkg): - info(*pkg) + build_env(*pkg) -@pytest.mark.parametrize('pkg,error_cls', [ - ('zlib libszip', SpackCommandError), - ('', IndexError) +@pytest.mark.usefixtures('config') +def test_error_when_multiple_specs_are_given(): + output = build_env('libelf libdwarf', fail_on_error=False) + assert 'only takes one spec' in output + + +@pytest.mark.parametrize('args', [ + ('--', '/bin/bash', '-c', 'echo test'), + ('--',), + (), ]) @pytest.mark.usefixtures('config') -def test_it_just_fails(pkg, error_cls): - with pytest.raises(error_cls): - info(pkg) +def test_build_env_requires_a_spec(args): + output = build_env(*args, fail_on_error=False) + assert 'requires a spec' in output _out_file = 'env.out' @@ -36,7 +43,7 @@ _out_file = 'env.out' @pytest.mark.usefixtures('config') def test_dump(tmpdir): with tmpdir.as_cwd(): - info('--dump', _out_file, 'zlib') + build_env('--dump', _out_file, 'zlib') with open(_out_file) as f: assert(any(line.startswith('PATH=') for line in f.readlines())) @@ -44,7 +51,7 @@ def test_dump(tmpdir): @pytest.mark.usefixtures('config') def test_pickle(tmpdir): with tmpdir.as_cwd(): - info('--pickle', _out_file, 'zlib') + build_env('--pickle', _out_file, 'zlib') environment = cPickle.load(open(_out_file, 'rb')) assert(type(environment) == dict) assert('PATH' in environment) diff --git a/lib/spack/spack/test/cmd/buildcache.py b/lib/spack/spack/test/cmd/buildcache.py index dcd9b822c0..5c056fb33b 100644 --- a/lib/spack/spack/test/cmd/buildcache.py +++ b/lib/spack/spack/test/cmd/buildcache.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -125,10 +125,6 @@ def test_buildcache_create_fails_on_noargs(tmpdir): buildcache('create', '-d', str(tmpdir), '--unsigned') -@pytest.mark.skipif( - os.environ.get('SPACK_TEST_SOLVER') == 'clingo', - reason='Test for Clingo are run in a container with root permissions' -) def test_buildcache_create_fail_on_perm_denied( install_mockery, mock_fetch, monkeypatch, tmpdir): """Ensure that buildcache create fails on permission denied error.""" diff --git a/lib/spack/spack/test/cmd/cd.py b/lib/spack/spack/test/cmd/cd.py index eda6994aec..c8766503b4 100644 --- a/lib/spack/spack/test/cmd/cd.py +++ b/lib/spack/spack/test/cmd/cd.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/ci.py b/lib/spack/spack/test/cmd/ci.py index 16af0e99e6..d1cac9e78c 100644 --- a/lib/spack/spack/test/cmd/ci.py +++ b/lib/spack/spack/test/cmd/ci.py @@ -1,4 +1,4 @@ -# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -7,10 +7,11 @@ import filecmp import json import os import pytest -from jsonschema import validate +from jsonschema import validate, ValidationError import spack import spack.ci as ci +import spack.compilers as compilers import spack.config import spack.environment as ev import spack.hash_types as ht @@ -19,7 +20,8 @@ import spack.paths as spack_paths import spack.repo as repo from spack.schema.buildcache_spec import schema as spec_yaml_schema from spack.schema.database_index import schema as db_idx_schema -from spack.spec import Spec +from spack.schema.gitlab_ci import schema as gitlab_ci_schema +from spack.spec import Spec, CompilerSpec from spack.util.mock_package import MockPackageMultiRepo import spack.util.executable as exe import spack.util.spack_yaml as syaml @@ -31,10 +33,14 @@ env_cmd = spack.main.SpackCommand('env') mirror_cmd = spack.main.SpackCommand('mirror') gpg_cmd = spack.main.SpackCommand('gpg') install_cmd = spack.main.SpackCommand('install') +uninstall_cmd = spack.main.SpackCommand('uninstall') buildcache_cmd = spack.main.SpackCommand('buildcache') git = exe.which('git', required=True) +pytestmark = pytest.mark.maybeslow + + @pytest.fixture() def env_deactivate(): yield @@ -77,13 +83,13 @@ and then 'd', 'b', and 'a' to be put in the next three stages, respectively. spec_a = Spec('a') spec_a.concretize() - spec_a_label = ci.spec_deps_key_label(spec_a)[1] - spec_b_label = ci.spec_deps_key_label(spec_a['b'])[1] - spec_c_label = ci.spec_deps_key_label(spec_a['c'])[1] - spec_d_label = ci.spec_deps_key_label(spec_a['d'])[1] - spec_e_label = ci.spec_deps_key_label(spec_a['e'])[1] - spec_f_label = ci.spec_deps_key_label(spec_a['f'])[1] - spec_g_label = ci.spec_deps_key_label(spec_a['g'])[1] + spec_a_label = ci.spec_deps_key(spec_a) + spec_b_label = ci.spec_deps_key(spec_a['b']) + spec_c_label = ci.spec_deps_key(spec_a['c']) + spec_d_label = ci.spec_deps_key(spec_a['d']) + spec_e_label = ci.spec_deps_key(spec_a['e']) + spec_f_label = ci.spec_deps_key(spec_a['f']) + spec_g_label = ci.spec_deps_key(spec_a['g']) spec_labels, dependencies, stages = ci.stage_spec_jobs([spec_a]) @@ -109,6 +115,7 @@ def test_ci_generate_with_env(tmpdir, mutable_mock_env_path, env_deactivate, install_mockery, mock_packages): """Make sure we can get a .gitlab-ci.yml from an environment file which has the gitlab-ci, cdash, and mirrors sections.""" + mirror_url = 'https://my.fake.mirror' filename = str(tmpdir.join('spack.yaml')) with open(filename, 'w') as f: f.write("""\ @@ -128,7 +135,7 @@ spack: - matrix: - [$old-gcc-pkgs] mirrors: - some-mirror: https://my.fake.mirror + some-mirror: {0} gitlab-ci: bootstrap: - name: bootstrap @@ -140,7 +147,7 @@ spack: tags: - donotcare image: donotcare - final-stage-rebuild-index: + service-job-attributes: image: donotcare tags: [donotcare] cdash: @@ -148,7 +155,7 @@ spack: url: https://my.fake.cdash project: Not used site: Nothing -""") +""".format(mirror_url)) with tmpdir.as_cwd(): env_cmd('create', 'test', './spack.yaml') outputfile = str(tmpdir.join('.gitlab-ci.yml')) @@ -170,16 +177,30 @@ spack: assert(yaml_contents['stages'][0] == 'stage-0') assert(yaml_contents['stages'][5] == 'stage-rebuild-index') + assert('rebuild-index' in yaml_contents) + rebuild_job = yaml_contents['rebuild-index'] + expected = 'spack buildcache update-index --keys -d {0}'.format( + mirror_url) + assert(rebuild_job['script'][0] == expected) + def _validate_needs_graph(yaml_contents, needs_graph, artifacts): for job_name, job_def in yaml_contents.items(): for needs_def_name, needs_list in needs_graph.items(): if job_name.startswith(needs_def_name): # check job needs against the expected needs definition + j_needs = job_def['needs'] + print('job {0} needs:'.format(needs_def_name)) + print([j['job'] for j in j_needs]) + print('expected:') + print([nl for nl in needs_list]) assert all([job_needs['job'][:job_needs['job'].index('/')] - in needs_list for job_needs in job_def['needs']]) + in needs_list for job_needs in j_needs]) + assert(all([nl in + [n['job'][:n['job'].index('/')] for n in j_needs] + for nl in needs_list])) assert all([job_needs['artifacts'] == artifacts - for job_needs in job_def['needs']]) + for job_needs in j_needs]) break @@ -369,7 +390,10 @@ spack: fake_token = 'notreallyatokenbutshouldnotmatter' os.environ['SPACK_CDASH_AUTH_TOKEN'] = fake_token copy_to_file = str(tmpdir.join('backup-ci.yml')) - output = ci_cmd('generate', '--copy-to', copy_to_file, output=str) + try: + output = ci_cmd('generate', '--copy-to', copy_to_file, output=str) + finally: + del os.environ['SPACK_CDASH_AUTH_TOKEN'] # That fake token should still have resulted in being unable to # register build group with cdash, but the workload should # still have been generated. @@ -533,7 +557,7 @@ spack: def test_ci_generate_for_pr_pipeline(tmpdir, mutable_mock_env_path, env_deactivate, install_mockery, - mock_packages): + mock_packages, monkeypatch): """Test that PR pipelines do not include a final stage job for rebuilding the mirror index, even if that job is specifically configured""" @@ -558,9 +582,10 @@ spack: runner-attributes: tags: - donotcare - final-stage-rebuild-index: + service-job-attributes: image: donotcare tags: [donotcare] + rebuild-index: False """) with tmpdir.as_cwd(): @@ -569,7 +594,14 @@ spack: with ev.read('test'): os.environ['SPACK_IS_PR_PIPELINE'] = 'True' - ci_cmd('generate', '--output-file', outputfile) + os.environ['SPACK_PR_BRANCH'] = 'fake-test-branch' + monkeypatch.setattr( + ci, 'SPACK_PR_MIRRORS_ROOT_URL', r"file:///fake/mirror") + try: + ci_cmd('generate', '--output-file', outputfile) + finally: + del os.environ['SPACK_IS_PR_PIPELINE'] + del os.environ['SPACK_PR_BRANCH'] with open(outputfile) as f: contents = f.read() @@ -579,10 +611,17 @@ spack: assert('rebuild-index' not in yaml_contents) + for ci_key in yaml_contents.keys(): + if ci_key.startswith('(specs) '): + job_object = yaml_contents[ci_key] + job_vars = job_object['variables'] + assert('SPACK_IS_PR_PIPELINE' in job_vars) + assert(job_vars['SPACK_IS_PR_PIPELINE'] == 'True') + def test_ci_generate_with_external_pkg(tmpdir, mutable_mock_env_path, env_deactivate, install_mockery, - mock_packages): + mock_packages, monkeypatch): """Make sure we do not generate jobs for external pkgs""" filename = str(tmpdir.join('spack.yaml')) with open(filename, 'w') as f: @@ -609,6 +648,8 @@ spack: outputfile = str(tmpdir.join('.gitlab-ci.yml')) with ev.read('test'): + monkeypatch.setattr( + ci, 'SPACK_PR_MIRRORS_ROOT_URL', r"file:///fake/mirror") ci_cmd('generate', '--output-file', outputfile) with open(outputfile) as f: @@ -712,6 +753,19 @@ spack: - $packages mirrors: test-mirror: {0} + gitlab-ci: + enable-artifacts-buildcache: True + mappings: + - match: + - patchelf + runner-attributes: + tags: + - donotcare + image: donotcare + service-job-attributes: + tags: + - nonbuildtag + image: basicimage """.format(mirror_url) print('spack.yaml:\n{0}\n'.format(spack_yaml_contents)) @@ -739,6 +793,48 @@ spack: buildcache_path = os.path.join(mirror_dir.strpath, 'build_cache') + # Now test the --prune-dag (default) option of spack ci generate + mirror_cmd('add', 'test-ci', mirror_url) + + outputfile_pruned = str(tmpdir.join('pruned_pipeline.yml')) + ci_cmd('generate', '--output-file', outputfile_pruned) + + with open(outputfile_pruned) as f: + contents = f.read() + yaml_contents = syaml.load(contents) + assert('no-specs-to-rebuild' in yaml_contents) + # Make sure there are no other spec jobs or rebuild-index + assert(len(yaml_contents.keys()) == 1) + the_elt = yaml_contents['no-specs-to-rebuild'] + assert('tags' in the_elt) + assert('nonbuildtag' in the_elt['tags']) + assert('image' in the_elt) + assert(the_elt['image'] == 'basicimage') + + outputfile_not_pruned = str(tmpdir.join('unpruned_pipeline.yml')) + ci_cmd('generate', '--no-prune-dag', '--output-file', + outputfile_not_pruned) + + # Test the --no-prune-dag option of spack ci generate + with open(outputfile_not_pruned) as f: + contents = f.read() + yaml_contents = syaml.load(contents) + + found_spec_job = False + + for ci_key in yaml_contents.keys(): + if '(specs) patchelf' in ci_key: + the_elt = yaml_contents[ci_key] + assert('variables' in the_elt) + job_vars = the_elt['variables'] + assert('SPACK_SPEC_NEEDS_REBUILD' in job_vars) + assert(job_vars['SPACK_SPEC_NEEDS_REBUILD'] == 'False') + found_spec_job = True + + assert(found_spec_job) + + mirror_cmd('rm', 'test-ci') + # Test generating buildcache index while we have bin mirror buildcache_cmd('update-index', '--mirror-url', mirror_url) index_path = os.path.join(buildcache_path, 'index.json') @@ -785,6 +881,26 @@ spack: assert(len(dl_dir_list) == 3) +def test_push_mirror_contents_exceptions(monkeypatch, capsys): + def faked(env, spec_yaml=None, packages=None, add_spec=True, + add_deps=True, output_location=os.getcwd(), + signing_key=None, force=False, make_relative=False, + unsigned=False, allow_root=False, rebuild_index=False): + raise Exception('Error: Access Denied') + + import spack.cmd.buildcache as buildcache + monkeypatch.setattr(buildcache, '_createtarball', faked) + + url = 'fakejunk' + ci.push_mirror_contents(None, None, None, url, None, None) + + captured = capsys.readouterr() + std_out = captured[0] + expect_msg = 'Permission problem writing to {0}'.format(url) + + assert(expect_msg in std_out) + + def test_ci_generate_override_runner_attrs(tmpdir, mutable_mock_env_path, env_deactivate, install_mockery, mock_packages, monkeypatch): @@ -839,7 +955,7 @@ spack: - custom main step after_script: - custom post step one - final-stage-rebuild-index: + service-job-attributes: image: donotcare tags: [donotcare] """) @@ -851,6 +967,8 @@ spack: with ev.read('test'): monkeypatch.setattr( spack.main, 'get_version', lambda: '0.15.3-416-12ad69eb1') + monkeypatch.setattr( + ci, 'SPACK_PR_MIRRORS_ROOT_URL', r"file:///fake/mirror") ci_cmd('generate', '--output-file', outputfile) with open(outputfile) as f: @@ -925,3 +1043,416 @@ spack: assert(the_elt['script'][0] == 'main step') assert(len(the_elt['after_script']) == 1) assert(the_elt['after_script'][0] == 'post step one') + + +def test_ci_generate_with_workarounds(tmpdir, mutable_mock_env_path, + env_deactivate, install_mockery, + mock_packages, monkeypatch): + """Make sure the post-processing cli workarounds do what they should""" + filename = str(tmpdir.join('spack.yaml')) + with open(filename, 'w') as f: + f.write("""\ +spack: + specs: + - callpath%gcc@3.0 + mirrors: + some-mirror: https://my.fake.mirror + gitlab-ci: + mappings: + - match: ['%gcc@3.0'] + runner-attributes: + tags: + - donotcare + image: donotcare + enable-artifacts-buildcache: true +""") + + with tmpdir.as_cwd(): + env_cmd('create', 'test', './spack.yaml') + outputfile = str(tmpdir.join('.gitlab-ci.yml')) + + with ev.read('test'): + monkeypatch.setattr( + ci, 'SPACK_PR_MIRRORS_ROOT_URL', r"file:///fake/mirror") + ci_cmd('generate', '--output-file', outputfile, '--dependencies') + + with open(outputfile) as f: + contents = f.read() + yaml_contents = syaml.load(contents) + + found_one = False + + for ci_key in yaml_contents.keys(): + if ci_key.startswith('(specs) '): + found_one = True + job_obj = yaml_contents[ci_key] + assert('needs' not in job_obj) + assert('dependencies' in job_obj) + + assert(found_one is True) + + +@pytest.mark.disable_clean_stage_check +def test_ci_rebuild_index(tmpdir, mutable_mock_env_path, env_deactivate, + install_mockery, mock_packages, mock_fetch, + mock_stage): + working_dir = tmpdir.join('working_dir') + + mirror_dir = working_dir.join('mirror') + mirror_url = 'file://{0}'.format(mirror_dir.strpath) + + spack_yaml_contents = """ +spack: + specs: + - callpath + mirrors: + test-mirror: {0} + gitlab-ci: + mappings: + - match: + - patchelf + runner-attributes: + tags: + - donotcare + image: donotcare +""".format(mirror_url) + + filename = str(tmpdir.join('spack.yaml')) + with open(filename, 'w') as f: + f.write(spack_yaml_contents) + + with tmpdir.as_cwd(): + env_cmd('create', 'test', './spack.yaml') + with ev.read('test'): + spec_map = ci.get_concrete_specs( + 'callpath', 'callpath', '', 'FIND_ANY') + concrete_spec = spec_map['callpath'] + spec_yaml = concrete_spec.to_yaml(hash=ht.build_hash) + yaml_path = str(tmpdir.join('spec.yaml')) + with open(yaml_path, 'w') as ypfd: + ypfd.write(spec_yaml) + + install_cmd('--keep-stage', '-f', yaml_path) + buildcache_cmd('create', '-u', '-a', '-f', '--mirror-url', + mirror_url, 'callpath') + ci_cmd('rebuild-index') + + buildcache_path = os.path.join(mirror_dir.strpath, 'build_cache') + index_path = os.path.join(buildcache_path, 'index.json') + with open(index_path) as idx_fd: + index_object = json.load(idx_fd) + validate(index_object, db_idx_schema) + + +def test_ci_generate_bootstrap_prune_dag( + install_mockery_mutable_config, mock_packages, mock_fetch, + mock_archive, mutable_config, monkeypatch, tmpdir, + mutable_mock_env_path, env_deactivate): + """Test compiler bootstrapping with DAG pruning. Specifically, make + sure that if we detect the bootstrapped compiler needs to be rebuilt, + we ensure the spec we want to build with that compiler is scheduled + for rebuild as well.""" + + # Create a temp mirror directory for buildcache usage + mirror_dir = tmpdir.join('mirror_dir') + mirror_url = 'file://{0}'.format(mirror_dir.strpath) + + # Install a compiler, because we want to put it in a buildcache + install_cmd('gcc@10.1.0%gcc@4.5.0') + + # Put installed compiler in the buildcache + buildcache_cmd('create', '-u', '-a', '-f', '-d', mirror_dir.strpath, + 'gcc@10.1.0%gcc@4.5.0') + + # Now uninstall the compiler + uninstall_cmd('-y', 'gcc@10.1.0%gcc@4.5.0') + + monkeypatch.setattr(spack.concretize.Concretizer, + 'check_for_compiler_existence', False) + spack.config.set('config:install_missing_compilers', True) + assert CompilerSpec('gcc@10.1.0') not in compilers.all_compiler_specs() + + # Configure the mirror where we put that buildcache w/ the compiler + mirror_cmd('add', 'test-mirror', mirror_url) + + install_cmd('--no-check-signature', 'a%gcc@10.1.0') + + # Put spec built with installed compiler in the buildcache + buildcache_cmd('create', '-u', '-a', '-f', '-d', mirror_dir.strpath, + 'a%gcc@10.1.0') + + # Now uninstall the spec + uninstall_cmd('-y', 'a%gcc@10.1.0') + + filename = str(tmpdir.join('spack.yaml')) + with open(filename, 'w') as f: + f.write("""\ +spack: + definitions: + - bootstrap: + - gcc@10.1.0%gcc@4.5.0 + specs: + - a%gcc@10.1.0 + mirrors: + atestm: {0} + gitlab-ci: + bootstrap: + - name: bootstrap + compiler-agnostic: true + mappings: + - match: + - arch=test-debian6-x86_64 + runner-attributes: + tags: + - donotcare + - match: + - arch=test-debian6-core2 + runner-attributes: + tags: + - meh +""".format(mirror_url)) + + # Without this monkeypatch, pipeline generation process would think that + # nothing in the environment needs rebuilding. With the monkeypatch, the + # process sees the compiler as needing a rebuild, which should then result + # in the specs built with that compiler needing a rebuild too. + def fake_get_mirrors_for_spec(spec=None, full_hash_match=False, + mirrors_to_check=None, index_only=False): + if spec.name == 'gcc': + return [] + else: + return [{ + 'spec': spec, + 'mirror_url': mirror_url, + }] + + with tmpdir.as_cwd(): + env_cmd('create', 'test', './spack.yaml') + outputfile = str(tmpdir.join('.gitlab-ci.yml')) + + with ev.read('test'): + monkeypatch.setattr( + ci, 'SPACK_PR_MIRRORS_ROOT_URL', r"file:///fake/mirror") + + ci_cmd('generate', '--output-file', outputfile) + + with open(outputfile) as of: + yaml_contents = of.read() + original_yaml_contents = syaml.load(yaml_contents) + + # without the monkeypatch, everything appears up to date and no + # rebuild jobs are generated. + assert(original_yaml_contents) + assert('no-specs-to-rebuild' in original_yaml_contents) + + monkeypatch.setattr(spack.binary_distribution, + 'get_mirrors_for_spec', + fake_get_mirrors_for_spec) + + ci_cmd('generate', '--output-file', outputfile) + + with open(outputfile) as of: + yaml_contents = of.read() + new_yaml_contents = syaml.load(yaml_contents) + + assert(new_yaml_contents) + + # This 'needs' graph reflects that even though specs 'a' and 'b' do + # not otherwise need to be rebuilt (thanks to DAG pruning), they + # both end up in the generated pipeline because the compiler they + # depend on is bootstrapped, and *does* need to be rebuilt. + needs_graph = { + '(bootstrap) gcc': [], + '(specs) b': [ + '(bootstrap) gcc', + ], + '(specs) a': [ + '(bootstrap) gcc', + '(specs) b', + ], + } + + _validate_needs_graph(new_yaml_contents, needs_graph, False) + + +def test_ci_subcommands_without_mirror(tmpdir, mutable_mock_env_path, + env_deactivate, mock_packages, + install_mockery): + """Make sure we catch if there is not a mirror and report an error""" + filename = str(tmpdir.join('spack.yaml')) + with open(filename, 'w') as f: + f.write("""\ +spack: + specs: + - archive-files + gitlab-ci: + mappings: + - match: + - archive-files + runner-attributes: + tags: + - donotcare + image: donotcare +""") + + with tmpdir.as_cwd(): + env_cmd('create', 'test', './spack.yaml') + outputfile = str(tmpdir.join('.gitlab-ci.yml')) + + with ev.read('test'): + # Check the 'generate' subcommand + output = ci_cmd('generate', '--output-file', outputfile, + output=str, fail_on_error=False) + ex = 'spack ci generate requires an env containing a mirror' + assert(ex in output) + + # Also check the 'rebuild-index' subcommand + output = ci_cmd('rebuild-index', output=str, fail_on_error=False) + ex = 'spack ci rebuild-index requires an env containing a mirror' + assert(ex in output) + + +def test_ensure_only_one_temporary_storage(): + """Make sure 'gitlab-ci' section of env does not allow specification of + both 'enable-artifacts-buildcache' and 'temporary-storage-url-prefix'.""" + gitlab_ci_template = """ + gitlab-ci: + {0} + mappings: + - match: + - notcheckedhere + runner-attributes: + tags: + - donotcare +""" + + enable_artifacts = 'enable-artifacts-buildcache: True' + temp_storage = 'temporary-storage-url-prefix: file:///temp/mirror' + specify_both = """{0} + {1} +""".format(enable_artifacts, temp_storage) + specify_neither = '' + + # User can specify "enable-artifacts-buildcache" (boolean) + yaml_obj = syaml.load(gitlab_ci_template.format(enable_artifacts)) + validate(yaml_obj, gitlab_ci_schema) + + # User can also specify "temporary-storage-url-prefix" (string) + yaml_obj = syaml.load(gitlab_ci_template.format(temp_storage)) + validate(yaml_obj, gitlab_ci_schema) + + # However, specifying both should fail to validate + yaml_obj = syaml.load(gitlab_ci_template.format(specify_both)) + with pytest.raises(ValidationError): + validate(yaml_obj, gitlab_ci_schema) + + # Specifying neither should be fine too, as neither of these properties + # should be required + yaml_obj = syaml.load(gitlab_ci_template.format(specify_neither)) + validate(yaml_obj, gitlab_ci_schema) + + +def test_ci_generate_temp_storage_url(tmpdir, mutable_mock_env_path, + env_deactivate, install_mockery, + mock_packages, monkeypatch): + """Verify correct behavior when using temporary-storage-url-prefix""" + filename = str(tmpdir.join('spack.yaml')) + with open(filename, 'w') as f: + f.write("""\ +spack: + specs: + - archive-files + mirrors: + some-mirror: https://my.fake.mirror + gitlab-ci: + temporary-storage-url-prefix: file:///work/temp/mirror + mappings: + - match: + - archive-files + runner-attributes: + tags: + - donotcare + image: donotcare +""") + + with tmpdir.as_cwd(): + env_cmd('create', 'test', './spack.yaml') + outputfile = str(tmpdir.join('.gitlab-ci.yml')) + + monkeypatch.setattr( + ci, 'SPACK_PR_MIRRORS_ROOT_URL', r"file:///fake/mirror") + + with ev.read('test'): + ci_cmd('generate', '--output-file', outputfile) + + with open(outputfile) as of: + pipeline_doc = syaml.load(of.read()) + + print(pipeline_doc) + + assert('cleanup' in pipeline_doc) + cleanup_job = pipeline_doc['cleanup'] + + assert('script' in cleanup_job) + cleanup_task = cleanup_job['script'][0] + + assert(cleanup_task.startswith('spack -d mirror destroy')) + + assert('stages' in pipeline_doc) + stages = pipeline_doc['stages'] + + # Cleanup job should be 2nd to last, just before rebuild-index + assert('stage' in cleanup_job) + assert(cleanup_job['stage'] == stages[-2]) + + +def test_ci_generate_read_broken_specs_url(tmpdir, mutable_mock_env_path, + env_deactivate, install_mockery, + mock_packages, monkeypatch): + """Verify that `broken-specs-url` works as intended""" + spec_a = Spec('a') + spec_a.concretize() + a_full_hash = spec_a.full_hash() + + spec_flattendeps = Spec('flatten-deps') + spec_flattendeps.concretize() + flattendeps_full_hash = spec_flattendeps.full_hash() + + # Mark 'a' as broken (but not 'flatten-deps') + broken_spec_a_path = str(tmpdir.join(a_full_hash)) + with open(broken_spec_a_path, 'w') as bsf: + bsf.write('') + + # Test that `spack ci generate` notices this broken spec and fails. + filename = str(tmpdir.join('spack.yaml')) + with open(filename, 'w') as f: + f.write("""\ +spack: + specs: + - flatten-deps + - a + mirrors: + some-mirror: https://my.fake.mirror + gitlab-ci: + broken-specs-url: "{0}" + mappings: + - match: + - archive-files + runner-attributes: + tags: + - donotcare + image: donotcare +""".format(tmpdir.strpath)) + + with tmpdir.as_cwd(): + env_cmd('create', 'test', './spack.yaml') + with ev.read('test'): + # Check output of the 'generate' subcommand + output = ci_cmd('generate', output=str, fail_on_error=False) + assert('known to be broken' in output) + + ex = '({0})'.format(a_full_hash) + assert(ex in output) + + ex = '({0})'.format(flattendeps_full_hash) + assert(ex not in output) diff --git a/lib/spack/spack/test/cmd/clean.py b/lib/spack/spack/test/cmd/clean.py index dcaf0c916c..daf561eea4 100644 --- a/lib/spack/spack/test/cmd/clean.py +++ b/lib/spack/spack/test/cmd/clean.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/commands.py b/lib/spack/spack/test/cmd/commands.py index 409dea7c51..2427511538 100644 --- a/lib/spack/spack/test/cmd/commands.py +++ b/lib/spack/spack/test/cmd/commands.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -118,48 +118,21 @@ def test_rst_with_header(tmpdir): def test_rst_update(tmpdir): update_file = tmpdir.join('output') - # not yet created when commands is run commands('--update', str(update_file)) assert update_file.exists() - with update_file.open() as f: - assert f.read() - - # created but older than commands - with update_file.open('w') as f: - f.write('empty\n') - update_file.setmtime(0) - commands('--update', str(update_file)) - assert update_file.exists() - with update_file.open() as f: - assert f.read() != 'empty\n' - - # newer than commands - with update_file.open('w') as f: - f.write('empty\n') - commands('--update', str(update_file)) - assert update_file.exists() - with update_file.open() as f: - assert f.read() == 'empty\n' def test_update_with_header(tmpdir): update_file = tmpdir.join('output') - # not yet created when commands is run commands('--update', str(update_file)) assert update_file.exists() - with update_file.open() as f: - assert f.read() fake_header = 'this is a header!\n\n' filename = tmpdir.join('header.txt') with filename.open('w') as f: f.write(fake_header) - # created, newer than commands, but older than header - commands('--update', str(update_file), '--header', str(filename)) - - # newer than commands and header commands('--update', str(update_file), '--header', str(filename)) @@ -229,7 +202,6 @@ def test_update_completion_arg(tmpdir, monkeypatch): old_file = old.read() with open(mock_args['bash']['update'], 'w') as mock: mock.write(old_file.replace("--update-completion", "")) - mock_bashfile.setmtime(0) # ensure mtime triggers update monkeypatch.setattr( spack.cmd.commands, 'update_completion_args', mock_args) diff --git a/lib/spack/spack/test/cmd/common/__init__.py b/lib/spack/spack/test/cmd/common/__init__.py index 9f87532b85..103eae6134 100644 --- a/lib/spack/spack/test/cmd/common/__init__.py +++ b/lib/spack/spack/test/cmd/common/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/common/arguments.py b/lib/spack/spack/test/cmd/common/arguments.py index 4f6cd1a527..8797626d60 100644 --- a/lib/spack/spack/test/cmd/common/arguments.py +++ b/lib/spack/spack/test/cmd/common/arguments.py @@ -1,14 +1,12 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) import argparse -import multiprocessing import pytest - import spack.cmd import spack.cmd.common.arguments as arguments import spack.config @@ -27,20 +25,15 @@ def job_parser(): yield p -@pytest.mark.parametrize("ncores", [1, 2, 4, 8, 16, 32]) -def test_setting_jobs_flag(job_parser, ncores, monkeypatch): - monkeypatch.setattr(multiprocessing, 'cpu_count', lambda: ncores) +def test_setting_jobs_flag(job_parser): namespace = job_parser.parse_args(['-j', '24']) - expected = min(24, ncores) - assert namespace.jobs == expected - assert spack.config.get('config:build_jobs') == expected + assert namespace.jobs == 24 + assert spack.config.get('config:build_jobs', scope='command_line') == 24 -@pytest.mark.parametrize("ncores", [1, 2, 4, 8, 16, 32]) -def test_omitted_job_flag(job_parser, ncores, monkeypatch): - monkeypatch.setattr(multiprocessing, 'cpu_count', lambda: ncores) +def test_omitted_job_flag(job_parser): namespace = job_parser.parse_args([]) - assert namespace.jobs == min(ncores, 16) + assert namespace.jobs is None assert spack.config.get('config:build_jobs') is None @@ -103,3 +96,21 @@ def test_multiple_env_match_raises_error(mock_packages, mutable_mock_env_path): spack.cmd.matching_spec_from_env(spack.cmd.parse_specs(['a'])[0]) assert 'matches multiple specs' in exc_info.value.message + + +@pytest.mark.usefixtures('config') +def test_root_and_dep_match_returns_root(mock_packages, mutable_mock_env_path): + e = ev.create('test') + e.add('b@0.9') + e.add('a foobar=bar') # Depends on b, should choose b@1.0 + e.concretize() + with e: + # This query matches the root b and b as a dependency of a. In that + # case the root instance should be preferred. + env_spec1 = spack.cmd.matching_spec_from_env( + spack.cmd.parse_specs(['b'])[0]) + assert env_spec1.satisfies('@0.9') + + env_spec2 = spack.cmd.matching_spec_from_env( + spack.cmd.parse_specs(['b@1.0'])[0]) + assert env_spec2 diff --git a/lib/spack/spack/test/cmd/compiler.py b/lib/spack/spack/test/cmd/compiler.py index c5c354221e..109695edd8 100644 --- a/lib/spack/spack/test/cmd/compiler.py +++ b/lib/spack/spack/test/cmd/compiler.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/concretize.py b/lib/spack/spack/test/cmd/concretize.py new file mode 100644 index 0000000000..ec8dbdf910 --- /dev/null +++ b/lib/spack/spack/test/cmd/concretize.py @@ -0,0 +1,55 @@ +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + + +import pytest +import spack.environment as ev +from spack.main import SpackCommand + + +# everything here uses the mock_env_path +pytestmark = pytest.mark.usefixtures( + 'mutable_mock_env_path', 'config', 'mutable_mock_repo') + +env = SpackCommand('env') +add = SpackCommand('add') +concretize = SpackCommand('concretize') + + +@pytest.mark.parametrize('concretization', ['separately', 'together']) +def test_concretize_all_test_dependencies(concretization): + """Check all test dependencies are concretized.""" + env('create', 'test') + + with ev.read('test') as e: + e.concretization = concretization + add('depb') + concretize('--test', 'all') + assert e.matching_spec('test-dependency') + + +@pytest.mark.parametrize('concretization', ['separately', 'together']) +def test_concretize_root_test_dependencies_not_recursive(concretization): + """Check that test dependencies are not concretized recursively.""" + env('create', 'test') + + with ev.read('test') as e: + e.concretization = concretization + add('depb') + concretize('--test', 'root') + assert e.matching_spec('test-dependency') is None + + +@pytest.mark.parametrize('concretization', ['separately', 'together']) +def test_concretize_root_test_dependencies_are_concretized(concretization): + """Check that root test dependencies are concretized.""" + env('create', 'test') + + with ev.read('test') as e: + e.concretization = concretization + add('a') + add('b') + concretize('--test', 'root') + assert e.matching_spec('test-dependency') diff --git a/lib/spack/spack/test/cmd/config.py b/lib/spack/spack/test/cmd/config.py index 7830f1c4e2..32996310a6 100644 --- a/lib/spack/spack/test/cmd/config.py +++ b/lib/spack/spack/test/cmd/config.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -11,6 +11,9 @@ import spack.config import spack.environment as ev import spack.main import spack.util.spack_yaml as syaml +import spack.spec +import spack.database +import spack.store config = spack.main.SpackCommand('config') env = spack.main.SpackCommand('env') @@ -84,6 +87,7 @@ repos: def test_config_edit(): """Ensure `spack config edit` edits the right paths.""" + dms = spack.config.default_modify_scope('compilers') dms_path = spack.config.config.scopes[dms].path user_path = spack.config.config.scopes['user'].path @@ -201,20 +205,27 @@ def test_config_add_override_leaf(mutable_empty_config): def test_config_add_update_dict(mutable_empty_config): - config('add', 'packages:all:compiler:[gcc]') - config('add', 'packages:all:version:1.0.0') + config('add', 'packages:all:version:[1.0.0]') output = config('get', 'packages') - expected = """packages: - all: - compiler: [gcc] - version: - - 1.0.0 -""" - + expected = 'packages:\n all:\n version: [1.0.0]\n' assert output == expected +def test_config_with_c_argument(mutable_empty_config): + + # I don't know how to add a spack argument to a Spack Command, so we test this way + config_file = 'config:install_root:root:/path/to/config.yaml' + parser = spack.main.make_argument_parser() + args = parser.parse_args(['-c', config_file]) + assert config_file in args.config_vars + + # Add the path to the config + config("add", args.config_vars[0], scope='command_line') + output = config("get", 'config') + assert "config:\n install_root:\n - root: /path/to/config.yaml" in output + + def test_config_add_ordered_dict(mutable_empty_config): config('add', 'mirrors:first:/path/to/first') config('add', 'mirrors:second:/path/to/second') @@ -645,3 +656,50 @@ def check_config_updated(data): assert isinstance(data['install_tree'], dict) assert data['install_tree']['root'] == '/fake/path' assert data['install_tree']['projections'] == {'all': '{name}-{version}'} + + +def test_config_prefer_upstream(tmpdir_factory, install_mockery, mock_fetch, + mutable_config, gen_mock_layout, monkeypatch): + """Check that when a dependency package is recorded as installed in + an upstream database that it is not reinstalled. + """ + + mock_db_root = str(tmpdir_factory.mktemp('mock_db_root')) + prepared_db = spack.database.Database(mock_db_root) + + upstream_layout = gen_mock_layout('/a/') + + for spec in [ + 'hdf5 +mpi', + 'hdf5 ~mpi', + 'boost+debug~icu+graph', + 'dependency-install', + 'patch']: + dep = spack.spec.Spec(spec) + dep.concretize() + prepared_db.add(dep, upstream_layout) + + downstream_db_root = str( + tmpdir_factory.mktemp('mock_downstream_db_root')) + db_for_test = spack.database.Database( + downstream_db_root, upstream_dbs=[prepared_db]) + monkeypatch.setattr(spack.store, 'db', db_for_test) + + output = config('prefer-upstream') + scope = spack.config.default_modify_scope('packages') + cfg_file = spack.config.config.get_config_filename(scope, 'packages') + packages = syaml.load(open(cfg_file))['packages'] + + # Make sure only the non-default variants are set. + assert packages['boost'] == { + 'compiler': ['gcc@4.5.0'], + 'variants': '+debug +graph', + 'version': ['1.63.0']} + assert packages['dependency-install'] == { + 'compiler': ['gcc@4.5.0'], 'version': ['2.0']} + # Ensure that neither variant gets listed for hdf5, since they conflict + assert packages['hdf5'] == { + 'compiler': ['gcc@4.5.0'], 'version': ['2.3']} + + # Make sure a message about the conflicting hdf5's was given. + assert '- hdf5' in output diff --git a/lib/spack/spack/test/cmd/create.py b/lib/spack/spack/test/cmd/create.py index 4262744317..39b076df08 100644 --- a/lib/spack/spack/test/cmd/create.py +++ b/lib/spack/spack/test/cmd/create.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/debug.py b/lib/spack/spack/test/cmd/debug.py index 2898ad670e..72190431de 100644 --- a/lib/spack/spack/test/cmd/debug.py +++ b/lib/spack/spack/test/cmd/debug.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -11,6 +11,7 @@ import os import os.path import spack.architecture as architecture +import spack.config from spack.main import SpackCommand, get_version from spack.util.executable import which @@ -53,3 +54,4 @@ def test_report(): assert get_version() in out assert platform.python_version() in out assert str(arch) in out + assert spack.config.get('config:concretizer') in out diff --git a/lib/spack/spack/test/cmd/dependencies.py b/lib/spack/spack/test/cmd/dependencies.py index 05d0556936..4c0de355ef 100644 --- a/lib/spack/spack/test/cmd/dependencies.py +++ b/lib/spack/spack/test/cmd/dependencies.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -51,8 +51,11 @@ def test_direct_installed_dependencies(mock_packages, database): with color_when(False): out = dependencies('--installed', 'mpileaks^mpich') - lines = [l for l in out.strip().split('\n') if not l.startswith('--')] - hashes = set([re.split(r'\s+', l)[0] for l in lines]) + lines = [ + line for line in out.strip().split('\n') + if not line.startswith('--') + ] + hashes = set([re.split(r'\s+', line)[0] for line in lines]) expected = set([spack.store.db.query_one(s).dag_hash(7) for s in ['mpich', 'callpath^mpich']]) @@ -65,8 +68,11 @@ def test_transitive_installed_dependencies(mock_packages, database): with color_when(False): out = dependencies('--installed', '--transitive', 'mpileaks^zmpi') - lines = [l for l in out.strip().split('\n') if not l.startswith('--')] - hashes = set([re.split(r'\s+', l)[0] for l in lines]) + lines = [ + line for line in out.strip().split('\n') + if not line.startswith('--') + ] + hashes = set([re.split(r'\s+', line)[0] for line in lines]) expected = set([spack.store.db.query_one(s).dag_hash(7) for s in ['zmpi', 'callpath^zmpi', 'fake', diff --git a/lib/spack/spack/test/cmd/dependents.py b/lib/spack/spack/test/cmd/dependents.py index bd87f92a65..376332cf88 100644 --- a/lib/spack/spack/test/cmd/dependents.py +++ b/lib/spack/spack/test/cmd/dependents.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/deprecate.py b/lib/spack/spack/test/cmd/deprecate.py index 7b38c177f5..370d267753 100644 --- a/lib/spack/spack/test/cmd/deprecate.py +++ b/lib/spack/spack/test/cmd/deprecate.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/dev_build.py b/lib/spack/spack/test/cmd/dev_build.py index 3067f5a0dd..11f4cbff82 100644 --- a/lib/spack/spack/test/cmd/dev_build.py +++ b/lib/spack/spack/test/cmd/dev_build.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -89,7 +89,7 @@ def test_dev_build_until_last_phase(tmpdir, mock_packages, install_mockery): assert os.path.exists(str(tmpdir)) -def test_dev_build_before_until(tmpdir, mock_packages, install_mockery): +def test_dev_build_before_until(tmpdir, mock_packages, install_mockery, capsys): spec = spack.spec.Spec('dev-build-test-install@0.0.0 dev_path=%s' % tmpdir) spec.concretize() @@ -103,13 +103,18 @@ def test_dev_build_before_until(tmpdir, mock_packages, install_mockery): bad_phase = 'phase_that_does_not_exist' not_allowed = 'is not a valid phase' - out = dev_build('-u', bad_phase, 'dev-build-test-install@0.0.0') + not_installed = 'was not installed' + out = dev_build('-u', bad_phase, 'dev-build-test-install@0.0.0', + fail_on_error=False) assert bad_phase in out assert not_allowed in out + assert not_installed in out - out = dev_build('-b', bad_phase, 'dev-build-test-install@0.0.0') + out = dev_build('-b', bad_phase, 'dev-build-test-install@0.0.0', + fail_on_error=False) assert bad_phase in out assert not_allowed in out + assert not_installed in out def print_spack_cc(*args): @@ -197,7 +202,7 @@ env: dev-build-test-install: spec: dev-build-test-install@0.0.0 path: %s -""" % build_dir) +""" % os.path.relpath(str(build_dir), start=str(envdir))) env('create', 'test', './spack.yaml') with ev.read('test'): @@ -323,7 +328,7 @@ env: dev-build-test-install: spec: dev-build-test-install@0.0.0 path: %s -""" % build_dir) +""" % os.path.relpath(str(build_dir), start=str(envdir))) env('create', 'test', './spack.yaml') with ev.read('test'): @@ -338,7 +343,7 @@ env: assert dep_spec.package.filename in os.listdir(dep_spec.prefix) assert os.path.exists(spec.prefix) - # Ensure variants set properly + # Ensure variants set properly; ensure build_dir is absolute and normalized for dep in (dep_spec, spec['dev-build-test-install']): assert dep.satisfies('dev_path=%s' % build_dir) assert spec.satisfies('^dev_path=*') diff --git a/lib/spack/spack/test/cmd/develop.py b/lib/spack/spack/test/cmd/develop.py index c539cd91c0..b6d8b11105 100644 --- a/lib/spack/spack/test/cmd/develop.py +++ b/lib/spack/spack/test/cmd/develop.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/env.py b/lib/spack/spack/test/cmd/env.py index f43ea4ff32..f6933dc349 100644 --- a/lib/spack/spack/test/cmd/env.py +++ b/lib/spack/spack/test/cmd/env.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -9,6 +9,7 @@ from six import StringIO import pytest import llnl.util.filesystem as fs +import llnl.util.link_tree import spack.hash_types as ht import spack.modules @@ -25,8 +26,10 @@ from spack.util.path import substitute_path_variables # everything here uses the mock_env_path -pytestmark = pytest.mark.usefixtures( - 'mutable_mock_env_path', 'config', 'mutable_mock_repo') +pytestmark = [ + pytest.mark.usefixtures('mutable_mock_env_path', 'config', 'mutable_mock_repo'), + pytest.mark.maybeslow +] env = SpackCommand('env') install = SpackCommand('install') @@ -198,6 +201,19 @@ def test_env_modifications_error_on_activate( assert "Warning: couldn't get environment settings" in err +def test_activate_adds_transitive_run_deps_to_path( + install_mockery, mock_fetch, monkeypatch): + env('create', 'test') + install = SpackCommand('install') + + e = ev.read('test') + with e: + install('depends-on-run-env') + + cmds = spack.environment.activate(e) + assert 'DEPENDENCY_ENV_VAR=1' in cmds + + def test_env_install_same_spec_twice(install_mockery, mock_fetch): env('create', 'test') @@ -1082,7 +1098,7 @@ def test_store_different_build_deps(): def test_env_updates_view_install( tmpdir, mock_stage, mock_fetch, install_mockery): - view_dir = tmpdir.mkdir('view') + view_dir = tmpdir.join('view') env('create', '--with-view=%s' % view_dir, 'test') with ev.read('test'): add('mpileaks') @@ -1093,12 +1109,13 @@ def test_env_updates_view_install( def test_env_view_fails( tmpdir, mock_packages, mock_stage, mock_fetch, install_mockery): - view_dir = tmpdir.mkdir('view') + view_dir = tmpdir.join('view') env('create', '--with-view=%s' % view_dir, 'test') with ev.read('test'): add('libelf') add('libelf cflags=-g') - with pytest.raises(RuntimeError, match='merge blocked by file'): + with pytest.raises(llnl.util.link_tree.MergeConflictError, + match='merge blocked by file'): install('--fake') @@ -1111,7 +1128,7 @@ def test_env_without_view_install( with pytest.raises(spack.environment.SpackEnvironmentError): test_env.default_view - view_dir = tmpdir.mkdir('view') + view_dir = tmpdir.join('view') with ev.read('test'): add('mpileaks') @@ -1146,7 +1163,7 @@ env: def test_env_updates_view_install_package( tmpdir, mock_stage, mock_fetch, install_mockery): - view_dir = tmpdir.mkdir('view') + view_dir = tmpdir.join('view') env('create', '--with-view=%s' % view_dir, 'test') with ev.read('test'): install('--fake', 'mpileaks') @@ -1156,7 +1173,7 @@ def test_env_updates_view_install_package( def test_env_updates_view_add_concretize( tmpdir, mock_stage, mock_fetch, install_mockery): - view_dir = tmpdir.mkdir('view') + view_dir = tmpdir.join('view') env('create', '--with-view=%s' % view_dir, 'test') install('--fake', 'mpileaks') with ev.read('test'): @@ -1168,7 +1185,7 @@ def test_env_updates_view_add_concretize( def test_env_updates_view_uninstall( tmpdir, mock_stage, mock_fetch, install_mockery): - view_dir = tmpdir.mkdir('view') + view_dir = tmpdir.join('view') env('create', '--with-view=%s' % view_dir, 'test') with ev.read('test'): install('--fake', 'mpileaks') @@ -1183,7 +1200,7 @@ def test_env_updates_view_uninstall( def test_env_updates_view_uninstall_referenced_elsewhere( tmpdir, mock_stage, mock_fetch, install_mockery): - view_dir = tmpdir.mkdir('view') + view_dir = tmpdir.join('view') env('create', '--with-view=%s' % view_dir, 'test') install('--fake', 'mpileaks') with ev.read('test'): @@ -1200,7 +1217,7 @@ def test_env_updates_view_uninstall_referenced_elsewhere( def test_env_updates_view_remove_concretize( tmpdir, mock_stage, mock_fetch, install_mockery): - view_dir = tmpdir.mkdir('view') + view_dir = tmpdir.join('view') env('create', '--with-view=%s' % view_dir, 'test') install('--fake', 'mpileaks') with ev.read('test'): @@ -1218,7 +1235,7 @@ def test_env_updates_view_remove_concretize( def test_env_updates_view_force_remove( tmpdir, mock_stage, mock_fetch, install_mockery): - view_dir = tmpdir.mkdir('view') + view_dir = tmpdir.join('view') env('create', '--with-view=%s' % view_dir, 'test') with ev.read('test'): install('--fake', 'mpileaks') @@ -2113,7 +2130,11 @@ def test_env_activate_default_view_root_unconditional(env_deactivate, viewdir = e.default_view.root out = env('activate', '--sh', 'test') - assert 'PATH=%s' % os.path.join(viewdir, 'bin') in out + viewdir_bin = os.path.join(viewdir, 'bin') + + assert "export PATH={0}".format(viewdir_bin) in out or \ + "export PATH='{0}".format(viewdir_bin) in out or \ + 'export PATH="{0}'.format(viewdir_bin) in out def test_concretize_user_specs_together(): @@ -2371,6 +2392,85 @@ spack: assert os.path.exists(str(spack_lock)) +def _setup_develop_packages(tmpdir): + """Sets up a structure ./init_env/spack.yaml, ./build_folder, ./dest_env + where spack.yaml has a relative develop path to build_folder""" + init_env = tmpdir.join('init_env') + build_folder = tmpdir.join('build_folder') + dest_env = tmpdir.join('dest_env') + + fs.mkdirp(str(init_env)) + fs.mkdirp(str(build_folder)) + fs.mkdirp(str(dest_env)) + + raw_yaml = """ +spack: + specs: ['mypkg1', 'mypkg2'] + develop: + mypkg1: + path: ../build_folder + spec: mypkg@main + mypkg2: + path: /some/other/path + spec: mypkg@main +""" + spack_yaml = init_env.join('spack.yaml') + spack_yaml.write(raw_yaml) + + return init_env, build_folder, dest_env, spack_yaml + + +def test_rewrite_rel_dev_path_new_dir(tmpdir): + """Relative develop paths should be rewritten for new environments in + a different directory from the original manifest file""" + _, build_folder, dest_env, spack_yaml = _setup_develop_packages(tmpdir) + + env('create', '-d', str(dest_env), str(spack_yaml)) + with ev.Environment(str(dest_env)) as e: + assert e.dev_specs['mypkg1']['path'] == str(build_folder) + assert e.dev_specs['mypkg2']['path'] == '/some/other/path' + + +def test_rewrite_rel_dev_path_named_env(tmpdir): + """Relative develop paths should by default be rewritten for new named + environment""" + _, build_folder, _, spack_yaml = _setup_develop_packages(tmpdir) + env('create', 'named_env', str(spack_yaml)) + with ev.read('named_env') as e: + assert e.dev_specs['mypkg1']['path'] == str(build_folder) + assert e.dev_specs['mypkg2']['path'] == '/some/other/path' + + +def test_rewrite_rel_dev_path_original_dir(tmpdir): + """Relative devevelop paths should not be rewritten when initializing an + environment with root path set to the same directory""" + init_env, _, _, spack_yaml = _setup_develop_packages(tmpdir) + with ev.Environment(str(init_env), str(spack_yaml)) as e: + assert e.dev_specs['mypkg1']['path'] == '../build_folder' + assert e.dev_specs['mypkg2']['path'] == '/some/other/path' + + +def test_rewrite_rel_dev_path_create_original_dir(tmpdir): + """Relative develop paths should not be rewritten when creating an + environment in the original directory""" + init_env, _, _, spack_yaml = _setup_develop_packages(tmpdir) + env('create', '-d', str(init_env), str(spack_yaml)) + with ev.Environment(str(init_env)) as e: + assert e.dev_specs['mypkg1']['path'] == '../build_folder' + assert e.dev_specs['mypkg2']['path'] == '/some/other/path' + + +def test_does_not_rewrite_rel_dev_path_when_keep_relative_is_set(tmpdir): + """Relative develop paths should not be rewritten when --keep-relative is + passed to create""" + _, _, _, spack_yaml = _setup_develop_packages(tmpdir) + env('create', '--keep-relative', 'named_env', str(spack_yaml)) + with ev.read('named_env') as e: + print(e.dev_specs) + assert e.dev_specs['mypkg1']['path'] == '../build_folder' + assert e.dev_specs['mypkg2']['path'] == '/some/other/path' + + @pytest.mark.regression('23440') def test_custom_version_concretize_together(tmpdir): # Custom versions should be permitted in specs when diff --git a/lib/spack/spack/test/cmd/extensions.py b/lib/spack/spack/test/cmd/extensions.py index ad993244ae..d2093fa887 100644 --- a/lib/spack/spack/test/cmd/extensions.py +++ b/lib/spack/spack/test/cmd/extensions.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/external.py b/lib/spack/spack/test/cmd/external.py index 53e1f9eaf2..3249a92894 100644 --- a/lib/spack/spack/test/cmd/external.py +++ b/lib/spack/spack/test/cmd/external.py @@ -1,7 +1,9 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) +import pytest + import os import os.path @@ -242,3 +244,26 @@ def test_new_entries_are_reported_correctly( # has been found output = external('find', 'gcc') assert 'No new external packages detected' in output + + +@pytest.mark.parametrize('command_args', [ + ('-t', 'build-tools'), + ('-t', 'build-tools', 'cmake'), +]) +def test_use_tags_for_detection( + command_args, mock_executable, mutable_config, monkeypatch +): + # Prepare an environment to detect a fake cmake + cmake_exe = mock_executable('cmake', output="echo cmake version 3.19.1") + prefix = os.path.dirname(cmake_exe) + monkeypatch.setenv('PATH', prefix) + + openssl_exe = mock_executable('openssl', output="OpenSSL 2.8.3") + prefix = os.path.dirname(openssl_exe) + monkeypatch.setenv('PATH', prefix) + + # Test that we detect specs + output = external('find', *command_args) + assert 'The following specs have been' in output + assert 'cmake' in output + assert 'openssl' not in output diff --git a/lib/spack/spack/test/cmd/fetch.py b/lib/spack/spack/test/cmd/fetch.py index d2f17eeb1b..8a2b51c57b 100644 --- a/lib/spack/spack/test/cmd/fetch.py +++ b/lib/spack/spack/test/cmd/fetch.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/find.py b/lib/spack/spack/test/cmd/find.py index 9d2203aaaf..762578a028 100644 --- a/lib/spack/spack/test/cmd/find.py +++ b/lib/spack/spack/test/cmd/find.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -89,7 +89,7 @@ def test_query_arguments(): @pytest.mark.usefixtures('database', 'mock_display') def test_tag1(parser, specs): - args = parser.parse_args(['--tags', 'tag1']) + args = parser.parse_args(['--tag', 'tag1']) spack.cmd.find.find(parser, args) assert len(specs) == 2 @@ -100,7 +100,7 @@ def test_tag1(parser, specs): @pytest.mark.db @pytest.mark.usefixtures('database', 'mock_display') def test_tag2(parser, specs): - args = parser.parse_args(['--tags', 'tag2']) + args = parser.parse_args(['--tag', 'tag2']) spack.cmd.find.find(parser, args) assert len(specs) == 1 @@ -110,7 +110,7 @@ def test_tag2(parser, specs): @pytest.mark.db @pytest.mark.usefixtures('database', 'mock_display') def test_tag2_tag3(parser, specs): - args = parser.parse_args(['--tags', 'tag2', '--tags', 'tag3']) + args = parser.parse_args(['--tag', 'tag2', '--tag', 'tag3']) spack.cmd.find.find(parser, args) assert len(specs) == 0 diff --git a/lib/spack/spack/test/cmd/flake8.py b/lib/spack/spack/test/cmd/flake8.py index d4bfc5618d..c1de26a278 100644 --- a/lib/spack/spack/test/cmd/flake8.py +++ b/lib/spack/spack/test/cmd/flake8.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -11,12 +11,12 @@ import sys from llnl.util.filesystem import FileFilter import spack.paths -from spack.cmd.flake8 import flake8, setup_parser, changed_files +from spack.cmd.style import style, setup_parser, changed_files from spack.repo import Repo from spack.util.executable import which -@pytest.fixture(scope='module') +@pytest.fixture(scope="module") def parser(): """Returns the parser for the ``flake8`` command""" parser = argparse.ArgumentParser() @@ -24,7 +24,7 @@ def parser(): return parser -@pytest.fixture(scope='module') +@pytest.fixture(scope="module") def flake8_package(): """Flake8 only checks files that have been modified. This fixture makes a small change to the ``flake8`` @@ -32,7 +32,7 @@ def flake8_package(): change on cleanup. """ repo = Repo(spack.paths.mock_packages_path) - filename = repo.filename_for_package_name('flake8') + filename = repo.filename_for_package_name("flake8") package = FileFilter(filename) # Make the change @@ -60,16 +60,18 @@ def test_changed_files(parser, flake8_package): # As of flake8 3.0.0, Python 2.6 and 3.3 are no longer supported # http://flake8.pycqa.org/en/latest/release-notes/3.0.0.html @pytest.mark.skipif( - sys.version_info[:2] <= (2, 6) or - (3, 0) <= sys.version_info[:2] <= (3, 3), - reason='flake8 no longer supports Python 2.6 or 3.3 and older') -@pytest.mark.skipif(not which('flake8'), reason='flake8 is not installed.') + sys.version_info[:2] <= (2, 6) or (3, 0) <= sys.version_info[:2] <= (3, 3), + reason="flake8 no longer supports Python 2.6 or 3.3 and older", +) +@pytest.mark.skipif(not which("flake8"), reason="flake8 is not installed.") def test_flake8(parser, flake8_package): # Only test the flake8_package that we modified # Otherwise, the unit tests would fail every time # the flake8 tests fail - args = parser.parse_args([flake8_package]) - flake8(parser, args) + args = parser.parse_args(["--no-mypy", flake8_package]) + style(parser, args) # Get even more coverage - args = parser.parse_args(['--output', '--root-relative', flake8_package]) - flake8(parser, args) + args = parser.parse_args( + ["--no-mypy", "--output", "--root-relative", flake8_package] + ) + style(parser, args) diff --git a/lib/spack/spack/test/cmd/gc.py b/lib/spack/spack/test/cmd/gc.py index 22c85a1d78..70df8af687 100644 --- a/lib/spack/spack/test/cmd/gc.py +++ b/lib/spack/spack/test/cmd/gc.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/gpg.py b/lib/spack/spack/test/cmd/gpg.py index 751790182b..099d53e039 100644 --- a/lib/spack/spack/test/cmd/gpg.py +++ b/lib/spack/spack/test/cmd/gpg.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/graph.py b/lib/spack/spack/test/cmd/graph.py index c402f35a9f..8d6da1049f 100644 --- a/lib/spack/spack/test/cmd/graph.py +++ b/lib/spack/spack/test/cmd/graph.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/help.py b/lib/spack/spack/test/cmd/help.py index 00edfa187a..8e0f6d02c9 100644 --- a/lib/spack/spack/test/cmd/help.py +++ b/lib/spack/spack/test/cmd/help.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/info.py b/lib/spack/spack/test/cmd/info.py index b57ef3257b..0ff6857617 100644 --- a/lib/spack/spack/test/cmd/info.py +++ b/lib/spack/spack/test/cmd/info.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/init_py_functions.py b/lib/spack/spack/test/cmd/init_py_functions.py index 2e63106283..56b0ef7171 100644 --- a/lib/spack/spack/test/cmd/init_py_functions.py +++ b/lib/spack/spack/test/cmd/init_py_functions.py @@ -1,4 +1,4 @@ -# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/install.py b/lib/spack/spack/test/cmd/install.py index 8e87f97c98..cbdea73931 100644 --- a/lib/spack/spack/test/cmd/install.py +++ b/lib/spack/spack/test/cmd/install.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -32,6 +32,7 @@ add = SpackCommand('add') mirror = SpackCommand('mirror') uninstall = SpackCommand('uninstall') buildcache = SpackCommand('buildcache') +find = SpackCommand('find') @pytest.fixture() @@ -119,7 +120,9 @@ def test_install_dirty_flag(arguments, expected): def test_package_output(tmpdir, capsys, install_mockery, mock_fetch): - """Ensure output printed from pkgs is captured by output redirection.""" + """ + Ensure output printed from pkgs is captured by output redirection. + """ # we can't use output capture here because it interferes with Spack's # logging. TODO: see whether we can get multiple log_outputs to work # when nested AND in pytest @@ -140,12 +143,15 @@ def test_package_output(tmpdir, capsys, install_mockery, mock_fetch): @pytest.mark.disable_clean_stage_check def test_install_output_on_build_error(mock_packages, mock_archive, mock_fetch, config, install_mockery, capfd): + """ + This test used to assume receiving full output, but since we've updated + spack to generate logs on the level of phases, it will only return the + last phase, install. + """ # capfd interferes with Spack's capturing with capfd.disabled(): - out = install('build-error', fail_on_error=False) - assert 'ProcessError' in out - assert 'configure: error: in /path/to/some/file:' in out - assert 'configure: error: cannot run C compiled programs.' in out + out = install('-v', 'build-error', fail_on_error=False) + assert 'Installing build-error' in out @pytest.mark.disable_clean_stage_check @@ -172,20 +178,17 @@ def test_install_with_source( @pytest.mark.disable_clean_stage_check def test_show_log_on_error(mock_packages, mock_archive, mock_fetch, config, install_mockery, capfd): - """Make sure --show-log-on-error works.""" + """ + Make sure --show-log-on-error works. + """ with capfd.disabled(): out = install('--show-log-on-error', 'build-error', fail_on_error=False) assert isinstance(install.error, spack.build_environment.ChildError) assert install.error.pkg.name == 'build-error' - assert 'Full build log:' in out - print(out) - - # Message shows up for ProcessError (1) and output (1) - errors = [line for line in out.split('\n') - if 'configure: error: cannot run C compiled programs' in line] - assert len(errors) == 2 + assert '==> Installing build-error' in out + assert 'See build log for details:' in out def test_install_overwrite( @@ -629,10 +632,11 @@ def test_cache_only_fails(tmpdir, mock_fetch, install_mockery, capfd): # libelf from cache fails to install, which automatically removes the # the libdwarf build task with capfd.disabled(): - out = install('--cache-only', 'libdwarf') + out = install('--cache-only', 'libdwarf', fail_on_error=False) assert 'Failed to install libelf' in out assert 'Skipping build of libdwarf' in out + assert 'was not installed' in out # Check that failure prefix locks are still cached failure_lock_prefixes = ','.join(spack.store.db._prefix_failures.keys()) @@ -709,8 +713,116 @@ def test_install_only_dependencies_of_all_in_env( assert os.path.exists(dep.prefix) +def test_install_no_add_in_env(tmpdir, mock_fetch, install_mockery, + mutable_mock_env_path): + # To test behavior of --no-add option, we create the following environment: + # + # mpileaks + # ^callpath + # ^dyninst + # ^libelf@0.8.13 # or latest, really + # ^libdwarf + # ^mpich + # libelf@0.8.10 + # a~bvv + # ^b + # a + # ^b + e = ev.create('test') + e.add('mpileaks') + e.add('libelf@0.8.10') # so env has both root and dep libelf specs + e.add('a') + e.add('a ~bvv') + e.concretize() + env_specs = e.all_specs() + + a_spec = None + b_spec = None + mpi_spec = None + + # First find and remember some target concrete specs in the environment + for e_spec in env_specs: + if e_spec.satisfies(Spec('a ~bvv')): + a_spec = e_spec + elif e_spec.name == 'b': + b_spec = e_spec + elif e_spec.satisfies(Spec('mpi')): + mpi_spec = e_spec + + assert(a_spec) + assert(a_spec.concrete) + + assert(b_spec) + assert(b_spec.concrete) + assert(b_spec not in e.roots()) + + assert(mpi_spec) + assert(mpi_spec.concrete) + + # Activate the environment + with e: + # Assert using --no-add with a spec not in the env fails + inst_out = install( + '--no-add', 'boost', fail_on_error=False, output=str) + + assert('no such spec exists in environment' in inst_out) + + # Ensure using --no-add with an ambiguous spec fails + with pytest.raises(ev.SpackEnvironmentError) as err: + inst_out = install( + '--no-add', 'a', output=str) + + assert('a matches multiple specs in the env' in str(err)) + + # With "--no-add", install an unambiguous dependency spec (that already + # exists as a dep in the environment) using --no-add and make sure it + # gets installed (w/ deps), but is not added to the environment. + install('--no-add', 'dyninst') + + find_output = find('-l', output=str) + assert('dyninst' in find_output) + assert('libdwarf' in find_output) + assert('libelf' in find_output) + assert('callpath' not in find_output) + + post_install_specs = e.all_specs() + assert all([s in env_specs for s in post_install_specs]) + + # Make sure we can install a concrete dependency spec from a spec.yaml + # file on disk, using the ``--no-add` option, and the spec is installed + # but not added as a root + mpi_spec_yaml_path = tmpdir.join('{0}.yaml'.format(mpi_spec.name)) + with open(mpi_spec_yaml_path.strpath, 'w') as fd: + fd.write(mpi_spec.to_yaml(hash=ht.full_hash)) + + install('--no-add', '-f', mpi_spec_yaml_path.strpath) + assert(mpi_spec not in e.roots()) + + find_output = find('-l', output=str) + assert(mpi_spec.name in find_output) + + # Without "--no-add", install an unambiguous depependency spec (that + # already exists as a dep in the environment) without --no-add and make + # sure it is added as a root of the environment as well as installed. + assert(b_spec not in e.roots()) + + install('b') + + assert(b_spec in e.roots()) + assert(b_spec not in e.uninstalled_specs()) + + # Without "--no-add", install a novel spec and make sure it is added + # as a root and installed. + install('bowtie') + + assert(any([s.name == 'bowtie' for s in e.roots()])) + assert(not any([s.name == 'bowtie' for s in e.uninstalled_specs()])) + + def test_install_help_does_not_show_cdash_options(capsys): - """Make sure `spack install --help` does not describe CDash arguments""" + """ + Make sure `spack install --help` does not describe CDash arguments + """ with pytest.raises(SystemExit): install('--help') captured = capsys.readouterr() @@ -738,6 +850,25 @@ def test_cdash_auth_token(tmpdir, install_mockery, capfd): assert 'Using CDash auth token from environment' in out +@pytest.mark.disable_clean_stage_check +def test_cdash_configure_warning(tmpdir, mock_fetch, install_mockery, capfd): + # capfd interferes with Spack's capturing of e.g., Build.xml output + with capfd.disabled(): + with tmpdir.as_cwd(): + # Test would fail if install raised an error. + install( + '--log-file=cdash_reports', + '--log-format=cdash', + 'configure-warning') + # Verify Configure.xml exists with expected contents. + report_dir = tmpdir.join('cdash_reports') + assert report_dir in tmpdir.listdir() + report_file = report_dir.join('Configure.xml') + assert report_file in report_dir.listdir() + content = report_file.open().read() + assert 'foo: No such file or directory' in content + + def test_compiler_bootstrap( install_mockery_mutable_config, mock_packages, mock_fetch, mock_archive, mutable_config, monkeypatch): @@ -753,7 +884,9 @@ def test_compiler_bootstrap( def test_compiler_bootstrap_from_binary_mirror( install_mockery_mutable_config, mock_packages, mock_fetch, mock_archive, mutable_config, monkeypatch, tmpdir): - """Make sure installing compiler from buildcache registers compiler""" + """ + Make sure installing compiler from buildcache registers compiler + """ # Create a temp mirror directory for buildcache usage mirror_dir = tmpdir.join('mirror_dir') @@ -884,3 +1017,23 @@ def test_cache_install_full_hash_match( uninstall('-y', s.name) mirror('rm', 'test-mirror') + + +def test_install_env_with_tests_all(tmpdir, mock_packages, mock_fetch, + install_mockery, mutable_mock_env_path): + env('create', 'test') + with ev.read('test'): + test_dep = Spec('test-dependency').concretized() + add('depb') + install('--test', 'all') + assert os.path.exists(test_dep.prefix) + + +def test_install_env_with_tests_root(tmpdir, mock_packages, mock_fetch, + install_mockery, mutable_mock_env_path): + env('create', 'test') + with ev.read('test'): + test_dep = Spec('test-dependency').concretized() + add('depb') + install('--test', 'root') + assert not os.path.exists(test_dep.prefix) diff --git a/lib/spack/spack/test/cmd/is_git_repo.py b/lib/spack/spack/test/cmd/is_git_repo.py index 724925e5e4..109f5030c7 100644 --- a/lib/spack/spack/test/cmd/is_git_repo.py +++ b/lib/spack/spack/test/cmd/is_git_repo.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/license.py b/lib/spack/spack/test/cmd/license.py index 614d093c5f..6063a8d9a8 100644 --- a/lib/spack/spack/test/cmd/license.py +++ b/lib/spack/spack/test/cmd/license.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -8,6 +8,7 @@ import re from llnl.util.filesystem import touch, mkdirp +import spack.cmd.license import spack.paths from spack.main import SpackCommand @@ -31,7 +32,7 @@ def test_verify(tmpdir): lgpl_header = source_dir.join('lgpl_header.py') with lgpl_header.open('w') as f: f.write("""\ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: LGPL-2.1-only @@ -48,13 +49,13 @@ def test_verify(tmpdir): correct_header = source_dir.join('correct_header.py') with correct_header.open('w') as f: f.write("""\ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) """) - out = license('verify', '--root', str(tmpdir), fail_on_error=False) + out = license('--root', str(tmpdir), 'verify', fail_on_error=False) assert str(no_header) in out assert str(lgpl_header) in out @@ -66,3 +67,28 @@ def test_verify(tmpdir): assert re.search(r'files with old license header:\s*1', out) assert license.returncode == 1 + + +def test_update_copyright_year(tmpdir): + source_dir = tmpdir.join('lib', 'spack', 'spack') + mkdirp(str(source_dir)) + + years = list(range(2018, 2021)) + + for year in years: + outdated = source_dir.join('header_%d.py' % year) + with outdated.open('w') as f: + f.write("""\ +# Copyright 2013-%d Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) +""" % year) + + license('--root', str(tmpdir), 'update-copyright-year') + + for year in years: + outdated = source_dir.join('header_%d.py' % year) + first_line = outdated.open().read().split("\n")[0] + assert str(year) not in first_line + assert spack.cmd.license.strict_date in first_line diff --git a/lib/spack/spack/test/cmd/list.py b/lib/spack/spack/test/cmd/list.py index 17f5a1b493..f99b7420a4 100644 --- a/lib/spack/spack/test/cmd/list.py +++ b/lib/spack/spack/test/cmd/list.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -33,15 +33,15 @@ def test_list_search_description(): def test_list_tags(): - output = list('--tags', 'proxy-app') + output = list('--tag', 'proxy-app') assert 'cloverleaf3d' in output assert 'hdf5' not in output - output = list('--tags', 'hpc') + output = list('--tag', 'hpc') assert 'nek5000' in output assert 'mfem' in output - output = list('--tags', 'HPC') + output = list('--tag', 'HPC') assert 'nek5000' in output assert 'mfem' in output diff --git a/lib/spack/spack/test/cmd/load.py b/lib/spack/spack/test/cmd/load.py index b1697bbfc4..620280e016 100644 --- a/lib/spack/spack/test/cmd/load.py +++ b/lib/spack/spack/test/cmd/load.py @@ -1,4 +1,4 @@ -# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/location.py b/lib/spack/spack/test/cmd/location.py index 2cefbd558f..98a13ce9da 100644 --- a/lib/spack/spack/test/cmd/location.py +++ b/lib/spack/spack/test/cmd/location.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -19,6 +19,7 @@ pytestmark = pytest.mark.usefixtures('config', 'database') # location prints out "locations of packages and spack directories" location = SpackCommand('location') +env = SpackCommand('env') @pytest.fixture @@ -86,6 +87,27 @@ def test_location_env(mock_test_env): assert location('--env', test_env_name).strip() == env_dir +def test_location_env_flag_interference(mutable_mock_env_path, tmpdir): + """ + Tests that specifying an active environment using `spack -e x location ...` + does not interfere with the location command flags. + """ + + # create two environments + env('create', 'first_env') + env('create', 'second_env') + + global_args = ['-e', 'first_env'] + + # `spack -e first_env location -e second_env` should print the env + # path of second_env + assert 'first_env' not in location('-e', 'second_env', global_args=global_args) + + # `spack -e first_env location --packages` should not print + # the environment path of first_env. + assert 'first_env' not in location('--packages', global_args=global_args) + + def test_location_env_missing(): """Tests spack location --env.""" missing_env_name = 'missing-env' diff --git a/lib/spack/spack/test/cmd/maintainers.py b/lib/spack/spack/test/cmd/maintainers.py index 4c566c7513..f6520b4a1b 100644 --- a/lib/spack/spack/test/cmd/maintainers.py +++ b/lib/spack/spack/test/cmd/maintainers.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/mark.py b/lib/spack/spack/test/cmd/mark.py index 5b488bbf3e..450bc1f30e 100644 --- a/lib/spack/spack/test/cmd/mark.py +++ b/lib/spack/spack/test/cmd/mark.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/mirror.py b/lib/spack/spack/test/cmd/mirror.py index 0957624cba..b72487e5ab 100644 --- a/lib/spack/spack/test/cmd/mirror.py +++ b/lib/spack/spack/test/cmd/mirror.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -14,6 +14,9 @@ mirror = SpackCommand('mirror') env = SpackCommand('env') add = SpackCommand('add') concretize = SpackCommand('concretize') +install = SpackCommand('install') +buildcache = SpackCommand('buildcache') +uninstall = SpackCommand('uninstall') @pytest.fixture @@ -183,3 +186,39 @@ def test_mirror_name_collision(tmp_scope): with pytest.raises(SpackCommandError): mirror('add', '--scope', tmp_scope, 'first', '1') + + +def test_mirror_destroy(install_mockery_mutable_config, + mock_packages, mock_fetch, mock_archive, + mutable_config, monkeypatch, tmpdir): + # Create a temp mirror directory for buildcache usage + mirror_dir = tmpdir.join('mirror_dir') + mirror_url = 'file://{0}'.format(mirror_dir.strpath) + mirror('add', 'atest', mirror_url) + + spec_name = 'libdwarf' + + # Put a binary package in a buildcache + install('--no-cache', spec_name) + buildcache('create', '-u', '-a', '-f', '-d', mirror_dir.strpath, spec_name) + + contents = os.listdir(mirror_dir.strpath) + assert('build_cache' in contents) + + # Destroy mirror by name + mirror('destroy', '-m', 'atest') + + assert(not os.path.exists(mirror_dir.strpath)) + + buildcache('create', '-u', '-a', '-f', '-d', mirror_dir.strpath, spec_name) + + contents = os.listdir(mirror_dir.strpath) + assert('build_cache' in contents) + + # Destroy mirror by url + mirror('destroy', '--mirror-url', mirror_url) + + assert(not os.path.exists(mirror_dir.strpath)) + + uninstall('-y', spec_name) + mirror('remove', 'atest') diff --git a/lib/spack/spack/test/cmd/module.py b/lib/spack/spack/test/cmd/module.py index 8be6c129c7..9acb21fdef 100644 --- a/lib/spack/spack/test/cmd/module.py +++ b/lib/spack/spack/test/cmd/module.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -23,7 +23,7 @@ def ensure_module_files_are_there( ): """Generate module files for module tests.""" module = spack.main.SpackCommand('module') - with spack.store.use_store(mock_store): + with spack.store.use_store(str(mock_store)): with spack.config.use_configuration(*mock_configuration_scopes): with spack.repo.use_repositories(mock_repo_path): module('tcl', 'refresh', '-y') diff --git a/lib/spack/spack/test/cmd/pkg.py b/lib/spack/spack/test/cmd/pkg.py index 5319e4e288..f9311b14bc 100644 --- a/lib/spack/spack/test/cmd/pkg.py +++ b/lib/spack/spack/test/cmd/pkg.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/print_shell_vars.py b/lib/spack/spack/test/cmd/print_shell_vars.py index 5f4f8164eb..3c56ee6e49 100644 --- a/lib/spack/spack/test/cmd/print_shell_vars.py +++ b/lib/spack/spack/test/cmd/print_shell_vars.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/providers.py b/lib/spack/spack/test/cmd/providers.py index dd0c8a9eb7..daab694032 100644 --- a/lib/spack/spack/test/cmd/providers.py +++ b/lib/spack/spack/test/cmd/providers.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/python.py b/lib/spack/spack/test/cmd/python.py index b1c9d3db00..d2ef35a85e 100644 --- a/lib/spack/spack/test/cmd/python.py +++ b/lib/spack/spack/test/cmd/python.py @@ -1,9 +1,10 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) import platform +import sys import pytest @@ -18,6 +19,11 @@ def test_python(): assert out.strip() == spack.spack_version +def test_python_interpreter_path(): + out = python('--path') + assert out.strip() == sys.executable + + def test_python_version(): out = python('-V') assert platform.python_version() in out diff --git a/lib/spack/spack/test/cmd/reindex.py b/lib/spack/spack/test/cmd/reindex.py index feb5545ae6..77c672251e 100644 --- a/lib/spack/spack/test/cmd/reindex.py +++ b/lib/spack/spack/test/cmd/reindex.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/repo.py b/lib/spack/spack/test/cmd/repo.py index 82fe872710..541ffeddf0 100644 --- a/lib/spack/spack/test/cmd/repo.py +++ b/lib/spack/spack/test/cmd/repo.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/resource.py b/lib/spack/spack/test/cmd/resource.py index 4fc5fd5c50..ba1bf48daf 100644 --- a/lib/spack/spack/test/cmd/resource.py +++ b/lib/spack/spack/test/cmd/resource.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/spec.py b/lib/spack/spack/test/cmd/spec.py index 3c50a66ce7..7d5cf12e75 100644 --- a/lib/spack/spack/test/cmd/spec.py +++ b/lib/spack/spack/test/cmd/spec.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/stage.py b/lib/spack/spack/test/cmd/stage.py new file mode 100644 index 0000000000..481694dc6e --- /dev/null +++ b/lib/spack/spack/test/cmd/stage.py @@ -0,0 +1,114 @@ +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +import pytest +from spack.main import SpackCommand +import spack.environment as ev +import spack.repo +from spack.version import Version + + +stage = SpackCommand('stage') +env = SpackCommand('env') + +pytestmark = pytest.mark.usefixtures('install_mockery', 'mock_packages') + + +def test_stage_spec(monkeypatch): + """Verify that staging specs works.""" + + expected = set(['trivial-install-test-package', 'mpileaks']) + + def fake_stage(pkg, mirror_only=False): + expected.remove(pkg.name) + + monkeypatch.setattr(spack.package.PackageBase, 'do_stage', fake_stage) + + stage('trivial-install-test-package', 'mpileaks') + + assert len(expected) == 0 + + +def test_stage_path(monkeypatch): + """Verify that --path only works with single specs.""" + + def fake_stage(pkg, mirror_only=False): + assert pkg.path == 'x' + + monkeypatch.setattr(spack.package.PackageBase, 'do_stage', fake_stage) + + stage('--path=x', 'trivial-install-test-package') + + +def test_stage_path_errors_multiple_specs(monkeypatch): + """Verify that --path only works with single specs.""" + + def fake_stage(pkg, mirror_only=False): + pass + + monkeypatch.setattr(spack.package.PackageBase, 'do_stage', fake_stage) + + with pytest.raises(spack.main.SpackCommandError): + stage('--path=x', 'trivial-install-test-package', 'mpileaks') + + +def test_stage_with_env_outside_env(mutable_mock_env_path, monkeypatch): + """Verify that stage concretizes specs not in environment instead of erroring.""" + + def fake_stage(pkg, mirror_only=False): + assert pkg.name == 'trivial-install-test-package' + assert pkg.path is None + + monkeypatch.setattr(spack.package.PackageBase, 'do_stage', fake_stage) + + e = ev.create('test') + e.add('mpileaks') + e.concretize() + + with e: + stage('trivial-install-test-package') + + +def test_stage_with_env_inside_env(mutable_mock_env_path, monkeypatch): + """Verify that stage filters specs in environment instead of reconcretizing.""" + + def fake_stage(pkg, mirror_only=False): + assert pkg.name == 'mpileaks' + assert pkg.version == Version('100.100') + + monkeypatch.setattr(spack.package.PackageBase, 'do_stage', fake_stage) + + e = ev.create('test') + e.add('mpileaks@100.100') + e.concretize() + + with e: + stage('mpileaks') + + +def test_stage_full_env(mutable_mock_env_path, monkeypatch): + """Verify that stage filters specs in environment.""" + + e = ev.create('test') + e.add('mpileaks@100.100') + e.concretize() + + # list all the package names that should be staged + expected = set() + for spec in e.specs_by_hash.values(): + for dep in spec.traverse(): + expected.add(dep.name) + + # pop the package name from the list instead of actually staging + def fake_stage(pkg, mirror_only=False): + expected.remove(pkg.name) + + monkeypatch.setattr(spack.package.PackageBase, 'do_stage', fake_stage) + + with e: + stage() + + # assert that all were staged + assert len(expected) == 0 diff --git a/lib/spack/spack/test/cmd/test.py b/lib/spack/spack/test/cmd/test.py index 4163853274..c0471fdfca 100644 --- a/lib/spack/spack/test/cmd/test.py +++ b/lib/spack/spack/test/cmd/test.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -11,6 +11,8 @@ import pytest import spack.config import spack.package import spack.cmd.install + +from spack.cmd.test import has_test_method from spack.main import SpackCommand install = SpackCommand('install') @@ -181,3 +183,32 @@ def test_test_help_cdash(mock_test_stage): """Make sure `spack test --help-cdash` describes CDash arguments""" out = spack_test('run', '--help-cdash') assert 'CDash URL' in out + + +def test_test_list_all(mock_packages): + """make sure `spack test list --all` returns all packages with tests""" + pkgs = spack_test("list", "--all").strip().split() + assert set(pkgs) == set([ + "printing-package", + "py-extension1", + "py-extension2", + "test-error", + "test-fail", + ]) + + +def test_test_list( + mock_packages, mock_archive, mock_fetch, install_mockery_mutable_config +): + pkg_with_tests = 'printing-package' + install(pkg_with_tests) + output = spack_test("list") + assert pkg_with_tests in output + + +def test_has_test_method_fails(capsys): + with pytest.raises(SystemExit): + has_test_method('printing-package') + + captured = capsys.readouterr()[1] + assert 'is not a class' in captured diff --git a/lib/spack/spack/test/cmd/undevelop.py b/lib/spack/spack/test/cmd/undevelop.py index 2e897e77f3..0c8f2df978 100644 --- a/lib/spack/spack/test/cmd/undevelop.py +++ b/lib/spack/spack/test/cmd/undevelop.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -39,9 +39,7 @@ env: assert not after.satisfies('dev_path=*') -def test_undevelop_nonexistent( - tmpdir, config, mock_packages, mutable_mock_env_path -): +def test_undevelop_nonexistent(tmpdir, config, mock_packages, mutable_mock_env_path): # setup environment envdir = tmpdir.mkdir('env') with envdir.as_cwd(): diff --git a/lib/spack/spack/test/cmd/uninstall.py b/lib/spack/spack/test/cmd/uninstall.py index a5adcdc74b..9a5d1587f0 100644 --- a/lib/spack/spack/test/cmd/uninstall.py +++ b/lib/spack/spack/test/cmd/uninstall.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/unit_test.py b/lib/spack/spack/test/cmd/unit_test.py index c5b8eb765e..1a273ff244 100644 --- a/lib/spack/spack/test/cmd/unit_test.py +++ b/lib/spack/spack/test/cmd/unit_test.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/url.py b/lib/spack/spack/test/cmd/url.py index 2350fc5fdb..c130840984 100644 --- a/lib/spack/spack/test/cmd/url.py +++ b/lib/spack/spack/test/cmd/url.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -71,7 +71,7 @@ def test_url_with_no_version_fails(): url('parse', 'http://www.netlib.org/voronoi/triangle.zip') -@pytest.mark.network +@pytest.mark.maybeslow @pytest.mark.skipif( sys.version_info < (2, 7), reason="Python 2.6 tests are run in a container, where " @@ -106,7 +106,7 @@ def test_url_list(): assert 0 < correct_version_urls < total_urls -@pytest.mark.network +@pytest.mark.maybeslow @pytest.mark.skipif( sys.version_info < (2, 7), reason="Python 2.6 tests are run in a container, where " diff --git a/lib/spack/spack/test/cmd/verify.py b/lib/spack/spack/test/cmd/verify.py index dda992fe49..de7544d9f1 100644 --- a/lib/spack/spack/test/cmd/verify.py +++ b/lib/spack/spack/test/cmd/verify.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd/versions.py b/lib/spack/spack/test/cmd/versions.py index 1e2bcbedb3..97f1745085 100644 --- a/lib/spack/spack/test/cmd/versions.py +++ b/lib/spack/spack/test/cmd/versions.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -6,45 +6,88 @@ import pytest from spack.main import SpackCommand +from spack.version import Version versions = SpackCommand('versions') +def test_safe_only_versions(): + """Only test the safe versions of a package. + (Using the deprecated command line argument) + """ + versions('--safe-only', 'zlib') + + def test_safe_versions(): """Only test the safe versions of a package.""" - versions('--safe-only', 'zlib') + versions('--safe', 'zlib') -@pytest.mark.network +@pytest.mark.maybeslow def test_remote_versions(): """Test a package for which remote versions should be available.""" versions('zlib') -@pytest.mark.network +@pytest.mark.maybeslow +def test_remote_versions_only(): + """Test a package for which remote versions should be available.""" + + versions('--remote', 'zlib') + + +@pytest.mark.usefixtures('mock_packages') +def test_new_versions_only(monkeypatch): + """Test a package for which new versions should be available.""" + from spack.pkg.builtin.mock.brillig import Brillig + + def mock_fetch_remote_versions(*args, **kwargs): + mock_remote_versions = { + # new version, we expect this to be in output: + Version('99.99.99'): {}, + # some packages use '3.2' equivalently to '3.2.0' + # thus '3.2.1' is considered to be a new version + # and expected in the output also + Version('3.2.1'): {}, # new version, we expect this to be in output + Version('3.2'): {}, + Version('1.0.0'): {}, + } + return mock_remote_versions + mock_versions = { + # already checksummed versions: + Version('3.2'): {}, + Version('1.0.0'): {}, + } + monkeypatch.setattr(Brillig, 'versions', mock_versions) + monkeypatch.setattr(Brillig, 'fetch_remote_versions', mock_fetch_remote_versions) + v = versions('--new', 'brillig') + assert(v.strip(' \n\t') == "99.99.99\n 3.2.1") + + +@pytest.mark.maybeslow def test_no_versions(): """Test a package for which no remote versions are available.""" versions('converge') -@pytest.mark.network +@pytest.mark.maybeslow def test_no_unchecksummed_versions(): """Test a package for which no unchecksummed versions are available.""" versions('bzip2') -@pytest.mark.network +@pytest.mark.maybeslow def test_versions_no_url(): """Test a package with versions but without a ``url`` attribute.""" versions('graphviz') -@pytest.mark.network +@pytest.mark.maybeslow def test_no_versions_no_url(): """Test a package without versions or a ``url`` attribute.""" diff --git a/lib/spack/spack/test/cmd/view.py b/lib/spack/spack/test/cmd/view.py index 4ff1592035..b5a503d278 100644 --- a/lib/spack/spack/test/cmd/view.py +++ b/lib/spack/spack/test/cmd/view.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/cmd_extensions.py b/lib/spack/spack/test/cmd_extensions.py index cb16609278..f611bd5c8f 100644 --- a/lib/spack/spack/test/cmd_extensions.py +++ b/lib/spack/spack/test/cmd_extensions.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/compilers/__init__.py b/lib/spack/spack/test/compilers/__init__.py index 9f87532b85..103eae6134 100644 --- a/lib/spack/spack/test/compilers/__init__.py +++ b/lib/spack/spack/test/compilers/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/compilers/basics.py b/lib/spack/spack/test/compilers/basics.py index 78d8484b3b..39bb65254d 100644 --- a/lib/spack/spack/test/compilers/basics.py +++ b/lib/spack/spack/test/compilers/basics.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -389,6 +389,7 @@ def test_cce_flags(): supported_flag_test("cxx_pic_flag", "-fPIC", "cce@9.1.0") supported_flag_test("f77_pic_flag", "-fPIC", "cce@9.1.0") supported_flag_test("fc_pic_flag", "-fPIC", "cce@9.1.0") + supported_flag_test("stdcxx_libs", (), "cce@1.0") supported_flag_test("debug_flags", ['-g', '-G0', '-G1', '-G2', '-Gfast'], 'cce@1.0') @@ -453,6 +454,8 @@ def test_aocc_flags(): '-Os', '-Oz', '-Og', '-O', '-O4'], 'aocc@2.2.0') + + supported_flag_test("stdcxx_libs", ("-lstdc++",), "aocc@2.2.0") supported_flag_test("openmp_flag", "-fopenmp", "aocc@2.2.0") supported_flag_test("cxx11_flag", "-std=c++11", "aocc@2.2.0") supported_flag_test("cxx14_flag", "-std=c++14", "aocc@2.2.0") @@ -464,6 +467,10 @@ def test_aocc_flags(): supported_flag_test("f77_pic_flag", "-fPIC", "aocc@2.2.0") supported_flag_test("fc_pic_flag", "-fPIC", "aocc@2.2.0") supported_flag_test("version_argument", "--version", "aocc@2.2.0") + flg = "-Wno-unused-command-line-argument -mllvm -eliminate-similar-expr=false" + supported_flag_test("cflags", flg, "aocc@3.0.0") + supported_flag_test("cxxflags", flg, "aocc@3.0.0") + supported_flag_test("fflags", flg, "aocc@3.0.0") def test_fj_flags(): @@ -471,14 +478,16 @@ def test_fj_flags(): supported_flag_test("cxx98_flag", "-std=c++98", "fj@4.0.0") supported_flag_test("cxx11_flag", "-std=c++11", "fj@4.0.0") supported_flag_test("cxx14_flag", "-std=c++14", "fj@4.0.0") + supported_flag_test("cxx17_flag", "-std=c++17", "fj@4.0.0") supported_flag_test("c99_flag", "-std=c99", "fj@4.0.0") supported_flag_test("c11_flag", "-std=c11", "fj@4.0.0") supported_flag_test("cc_pic_flag", "-KPIC", "fj@4.0.0") supported_flag_test("cxx_pic_flag", "-KPIC", "fj@4.0.0") supported_flag_test("f77_pic_flag", "-KPIC", "fj@4.0.0") supported_flag_test("fc_pic_flag", "-KPIC", "fj@4.0.0") - supported_flag_test("opt_flags", ['-O', '-O0', '-O1', '-O2', '-O3', '-O4'], + supported_flag_test("opt_flags", ['-O0', '-O1', '-O2', '-O3', '-Ofast'], 'fj@4.0.0') + supported_flag_test("debug_flags", "-g", "fj@4.0.0") def test_gcc_flags(): @@ -604,6 +613,7 @@ def test_pgi_flags(): supported_flag_test("cxx_pic_flag", "-fpic", "pgi@1.0") supported_flag_test("f77_pic_flag", "-fpic", "pgi@1.0") supported_flag_test("fc_pic_flag", "-fpic", "pgi@1.0") + supported_flag_test("stdcxx_libs", ("-pgc++libs",), "pgi@1.0") supported_flag_test("debug_flags", ['-g', '-gopt'], 'pgi@1.0') supported_flag_test("opt_flags", ['-O', '-O0', '-O1', '-O2', '-O3', '-O4'], 'pgi@1.0') diff --git a/lib/spack/spack/test/compilers/detection.py b/lib/spack/spack/test/compilers/detection.py index 47e078f242..1736f47ee5 100644 --- a/lib/spack/spack/test/compilers/detection.py +++ b/lib/spack/spack/test/compilers/detection.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -96,7 +96,11 @@ def test_apple_clang_version_detection( ('clang version 8.0.0-3 (tags/RELEASE_800/final)\n' 'Target: aarch64-unknown-linux-gnu\n' 'Thread model: posix\n' - 'InstalledDir: /usr/bin\n', '8.0.0') + 'InstalledDir: /usr/bin\n', '8.0.0'), + ('clang version 11.0.0\n' + 'Target: aarch64-unknown-linux-gnu\n' + 'Thread model: posix\n' + 'InstalledDir: /usr/bin\n', '11.0.0') ]) def test_clang_version_detection(version_str, expected_version): version = spack.compilers.clang.Clang.extract_version_from_output( @@ -317,6 +321,16 @@ def test_cray_frontend_compiler_detection( @pytest.mark.parametrize('version_str,expected_version', [ # This applies to C,C++ and FORTRAN compiler + ('AMD clang version 12.0.0 (CLANG: AOCC_3.0.0-Build#78 2020_12_10)' + '(based on LLVM Mirror.Version.12.0.0)\n' + 'Target: x86_64-unknown-linux-gnu\n' + 'Thread model: posix\n', '3.0.0' + ), + ('AMD clang version 11.0.0 (CLANG: AOCC_2.3.0-Build#85 2020_11_10)' + '(based on LLVM Mirror.Version.11.0.0)\n' + 'Target: x86_64-unknown-linux-gnu\n' + 'Thread model: posix\n', '2.3.0' + ), ('AMD clang version 10.0.0 (CLANG: AOCC_2.2.0-Build#93 2020_06_25)' '(based on LLVM Mirror.Version.10.0.0)\n' 'Target: x86_64-unknown-linux-gnu\n' diff --git a/lib/spack/spack/test/concretize.py b/lib/spack/spack/test/concretize.py index 6f87381e3f..f392387099 100644 --- a/lib/spack/spack/test/concretize.py +++ b/lib/spack/spack/test/concretize.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -660,8 +660,7 @@ class TestConcretize(object): abstract_specs = [Spec(x) for x in abstract_specs] concrete_specs = spack.concretize.concretize_specs_together( - *abstract_specs - ) + *abstract_specs) # Check there's only one configuration of each package in the DAG names = set( @@ -690,7 +689,7 @@ class TestConcretize(object): # Include targets to prevent regression on 20537 @pytest.mark.parametrize('spec, best_achievable', [ - ('mpileaks%gcc@4.4.7 target=x86_64:', 'core2'), + ('mpileaks%gcc@4.4.7 ^dyninst@10.2.1 target=x86_64:', 'core2'), ('mpileaks%gcc@4.8 target=x86_64:', 'haswell'), ('mpileaks%gcc@5.3.0 target=x86_64:', 'broadwell'), ('mpileaks%apple-clang@5.1.0 target=x86_64:', 'x86_64') @@ -925,6 +924,35 @@ class TestConcretize(object): assert s.external assert s.satisfies(expected) + @pytest.mark.regression('20976') + @pytest.mark.parametrize('compiler,spec_str,expected,xfailold', [ + ('gcc', 'external-common-python %clang', + '%clang ^external-common-openssl%gcc ^external-common-gdbm%clang', False), + ('clang', 'external-common-python', + '%clang ^external-common-openssl%clang ^external-common-gdbm%clang', True) + ]) + def test_compiler_in_nonbuildable_external_package( + self, compiler, spec_str, expected, xfailold + ): + """Check that the compiler of a non-buildable external package does not + spread to other dependencies, unless no other commpiler is specified.""" + packages_yaml = { + 'external-common-openssl': { + 'externals': [ + {'spec': 'external-common-openssl@1.1.1i%' + compiler, + 'prefix': '/usr'} + ], + 'buildable': False + } + } + spack.config.set('packages', packages_yaml) + + s = Spec(spec_str).concretized() + if xfailold and spack.config.get('config:concretizer') == 'original': + pytest.xfail('This only works on the ASP-based concretizer') + assert s.satisfies(expected) + assert 'external-common-perl' not in [d.name for d in s.dependencies()] + def test_external_packages_have_consistent_hash(self): if spack.config.get('config:concretizer') == 'original': pytest.skip('This tests needs the ASP-based concretizer') @@ -1182,3 +1210,37 @@ class TestConcretize(object): for node in s.traverse(): assert node.satisfies(expected_compiler) + + @pytest.mark.parametrize('spec_str,expected_dict', [ + # Check the defaults from the package (libs=shared) + ('multivalue-variant', { + 'libs=shared': True, + 'libs=static': False + }), + # Check that libs=static doesn't extend the default + ('multivalue-variant libs=static', { + 'libs=shared': False, + 'libs=static': True + }), + ]) + def test_multivalued_variants_from_cli(self, spec_str, expected_dict): + s = Spec(spec_str).concretized() + + for constraint, value in expected_dict.items(): + assert s.satisfies(constraint) == value + + @pytest.mark.regression('22351') + @pytest.mark.parametrize('spec_str,expected', [ + # Version 1.1.0 is deprecated and should not be selected, unless we + # explicitly asked for that + ('deprecated-versions', ['deprecated-versions@1.0.0']), + ('deprecated-versions@1.1.0', ['deprecated-versions@1.1.0']), + ]) + def test_deprecated_versions_not_selected(self, spec_str, expected): + if spack.config.get('config:concretizer') == 'original': + pytest.xfail('Known failure of the original concretizer') + + s = Spec(spec_str).concretized() + + for abstract_spec in expected: + assert abstract_spec in s diff --git a/lib/spack/spack/test/concretize_preferences.py b/lib/spack/spack/test/concretize_preferences.py index 9abac7221d..7abbaa6e92 100644 --- a/lib/spack/spack/test/concretize_preferences.py +++ b/lib/spack/spack/test/concretize_preferences.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/config.py b/lib/spack/spack/test/config.py index 9384eb5b47..d47851462a 100644 --- a/lib/spack/spack/test/config.py +++ b/lib/spack/spack/test/config.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -16,6 +16,7 @@ import pytest import spack.paths import spack.config import spack.main +import spack.environment import spack.schema.compilers import spack.schema.config import spack.schema.env @@ -276,6 +277,37 @@ def test_write_to_same_priority_file(mock_low_high_config, compiler_specs): repos_low = {'repos': ["/some/path"]} repos_high = {'repos': ["/some/other/path"]} +# Test setting config values via path in filename + + +def test_add_config_path(): + + # Try setting a new install tree root + path = "config:install_tree:root:/path/to/config.yaml" + spack.config.add(path, scope="command_line") + set_value = spack.config.get('config')['install_tree']['root'] + assert set_value == '/path/to/config.yaml' + + # Now a package:all setting + path = "packages:all:compiler:[gcc]" + spack.config.add(path, scope="command_line") + compilers = spack.config.get('packages')['all']['compiler'] + assert "gcc" in compilers + + +def test_add_config_filename(mock_low_high_config, tmpdir): + + config_yaml = tmpdir.join('config-filename.yaml') + config_yaml.ensure() + with config_yaml.open('w') as f: + syaml.dump_config(config_low, f) + + spack.config.add_from_file(str(config_yaml), scope="low") + assert "build_stage" in spack.config.get('config') + build_stages = spack.config.get('config')['build_stage'] + for stage in config_low['config']['build_stage']: + assert stage in build_stages + # repos def test_write_list_in_memory(mock_low_high_config): @@ -286,7 +318,12 @@ def test_write_list_in_memory(mock_low_high_config): assert config == repos_high['repos'] + repos_low['repos'] -def test_substitute_config_variables(mock_low_high_config): +class MockEnv(object): + def __init__(self, path): + self.path = path + + +def test_substitute_config_variables(mock_low_high_config, monkeypatch): prefix = spack.paths.prefix.lstrip('/') assert os.path.join( @@ -317,6 +354,33 @@ def test_substitute_config_variables(mock_low_high_config): '/foo/bar/baz', prefix, 'foo/bar/baz' ) != spack_path.canonicalize_path('/foo/bar/baz/${spack/foo/bar/baz/') + # $env replacement is a no-op when no environment is active + assert spack_path.canonicalize_path( + '/foo/bar/baz/$env' + ) == '/foo/bar/baz/$env' + + # Fake an active environment and $env is replaced properly + fake_env_path = '/quux/quuux' + monkeypatch.setattr(spack.environment, 'get_env', + lambda x, y: MockEnv(fake_env_path)) + assert spack_path.canonicalize_path( + '$env/foo/bar/baz' + ) == os.path.join(fake_env_path, 'foo/bar/baz') + + # relative paths without source information are relative to cwd + assert spack_path.canonicalize_path( + 'foo/bar/baz' + ) == os.path.abspath('foo/bar/baz') + + # relative paths with source information are relative to the file + spack.config.set( + 'config:module_roots', {'lmod': 'foo/bar/baz'}, scope='low') + spack.config.config.clear_caches() + path = spack.config.get('config:module_roots:lmod') + assert spack_path.canonicalize_path(path) == os.path.normpath( + os.path.join(mock_low_high_config.scopes['low'].path, + 'foo/bar/baz')) + packages_merge_low = { 'packages': { diff --git a/lib/spack/spack/test/config_values.py b/lib/spack/spack/test/config_values.py index a0bf7e52f8..367138f098 100644 --- a/lib/spack/spack/test/config_values.py +++ b/lib/spack/spack/test/config_values.py @@ -1,37 +1,31 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) +import pytest import spack.spec +import spack.store -def test_set_install_hash_length(mock_packages, mutable_config, monkeypatch, - tmpdir): - # spack.store.layout caches initial config values, so we monkeypatch - mutable_config.set('config:install_hash_length', 5) +@pytest.mark.parametrize('hash_length', [1, 2, 3, 4, 5, 9]) +@pytest.mark.use_fixtures('mock_packages') +def test_set_install_hash_length(hash_length, mutable_config, tmpdir): + mutable_config.set('config:install_hash_length', hash_length) mutable_config.set('config:install_tree', {'root': str(tmpdir)}) - monkeypatch.setattr(spack.store, 'store', spack.store._store()) - - spec = spack.spec.Spec('libelf').concretized() - prefix = spec.prefix - hash = prefix.rsplit('-')[-1] - - assert len(hash) == 5 - - mutable_config.set('config:install_hash_length', 9) - monkeypatch.setattr(spack.store, 'store', spack.store._store()) - - spec = spack.spec.Spec('libelf').concretized() - prefix = spec.prefix - hash = prefix.rsplit('-')[-1] - - assert len(hash) == 9 - - -def test_set_install_hash_length_upper_case(mock_packages, mutable_config, - monkeypatch, tmpdir): - # spack.store.layout caches initial config values, so we monkeypatch + # The call below is to reinitialize the directory layout associated + # with the store according to the configuration changes above (i.e. + # with the shortened hash) + store = spack.store._store() + with spack.store.use_store(store): + spec = spack.spec.Spec('libelf').concretized() + prefix = spec.prefix + hash_str = prefix.rsplit('-')[-1] + assert len(hash_str) == hash_length + + +@pytest.mark.use_fixtures('mock_packages') +def test_set_install_hash_length_upper_case(mutable_config, tmpdir): mutable_config.set('config:install_hash_length', 5) mutable_config.set( 'config:install_tree', @@ -42,10 +36,12 @@ def test_set_install_hash_length_upper_case(mock_packages, mutable_config, } } ) - monkeypatch.setattr(spack.store, 'store', spack.store._store()) - - spec = spack.spec.Spec('libelf').concretized() - prefix = spec.prefix - hash = prefix.rsplit('-')[-1] - - assert len(hash) == 5 + # The call below is to reinitialize the directory layout associated + # with the store according to the configuration changes above (i.e. + # with the shortened hash and projection) + store = spack.store._store() + with spack.store.use_store(store): + spec = spack.spec.Spec('libelf').concretized() + prefix = spec.prefix + hash_str = prefix.rsplit('-')[-1] + assert len(hash_str) == 5 diff --git a/lib/spack/spack/test/conftest.py b/lib/spack/spack/test/conftest.py index 297ae0d4f2..6cc3ad16db 100644 --- a/lib/spack/spack/test/conftest.py +++ b/lib/spack/spack/test/conftest.py @@ -563,12 +563,10 @@ def _populate(mock_db): pkg = spack.repo.get(s) pkg.do_install(fake=True, explicit=True) - # Transaction used to avoid repeated writes. - with mock_db.write_transaction(): - _install('mpileaks ^mpich') - _install('mpileaks ^mpich2') - _install('mpileaks ^zmpi') - _install('externaltest') + _install('mpileaks ^mpich') + _install('mpileaks ^mpich2') + _install('mpileaks ^zmpi') + _install('externaltest') @pytest.fixture(scope='session') @@ -605,38 +603,7 @@ def mock_store(tmpdir_factory, mock_repo_path, mock_configuration_scopes, # Make the DB filesystem read-only to ensure we can't modify entries store_path.join('.spack-db').chmod(mode=0o555, rec=1) - yield store - - store_path.join('.spack-db').chmod(mode=0o755, rec=1) - - -@pytest.fixture(scope='function') -def mutable_mock_store( - tmpdir_factory, mock_repo_path, mock_configuration_scopes, - _store_dir_and_cache -): - """Creates a read-only mock database with some packages installed note - that the ref count for dyninst here will be 3, as it's recycled - across each install. - - This does not actually activate the store for use by Spack -- see the - ``database`` fixture for that. - - """ - store_path, store_cache = _store_dir_and_cache - - # If the cache does not exist populate the store and create it - if not os.path.exists(str(store_cache.join('.spack-db'))): - with spack.config.use_configuration(*mock_configuration_scopes): - with spack.store.use_store(str(store_path)) as store: - with spack.repo.use_repositories(mock_repo_path): - _populate(store.db) - store_path.copy(store_cache, mode=True, stat=True) - - # Make the DB filesystem read-only to ensure we can't modify entries - store_path.join('.spack-db').chmod(mode=0o555, rec=1) - - yield store + yield store_path store_path.join('.spack-db').chmod(mode=0o755, rec=1) @@ -644,10 +611,9 @@ def mutable_mock_store( @pytest.fixture(scope='function') def database(mock_store, mock_packages, config, monkeypatch): """This activates the mock store, packages, AND config.""" - monkeypatch.setattr(spack.store, 'store', mock_store) - yield mock_store.db - # Force reading the database again between tests - mock_store.db.last_seen_verifier = '' + with spack.store.use_store(str(mock_store)) as store: + yield store.db + store.db.last_seen_verifier = '' @pytest.fixture(scope='function') @@ -718,20 +684,15 @@ def disable_compiler_execution(monkeypatch, request): @pytest.fixture(scope='function') -def install_mockery(tmpdir, config, mock_packages, monkeypatch): +def install_mockery(temporary_store, config, mock_packages): """Hooks a fake install directory, DB, and stage directory into Spack.""" - monkeypatch.setattr( - spack.store, 'store', spack.store.Store(str(tmpdir.join('opt')))) - # We use a fake package, so temporarily disable checksumming with spack.config.override('config:checksum', False): yield - tmpdir.join('opt').remove() - # Also wipe out any cached prefix failure locks (associated with # the session-scoped mock archive). - for pkg_id in list(spack.store.db._prefix_failures.keys()): + for pkg_id in list(temporary_store.db._prefix_failures.keys()): lock = spack.store.db._prefix_failures.pop(pkg_id, None) if lock: try: @@ -741,23 +702,28 @@ def install_mockery(tmpdir, config, mock_packages, monkeypatch): @pytest.fixture(scope='function') +def temporary_store(tmpdir): + """Hooks a temporary empty store for the test function.""" + temporary_store_path = tmpdir.join('opt') + with spack.store.use_store(str(temporary_store_path)) as s: + yield s + temporary_store_path.remove() + + +@pytest.fixture(scope='function') def install_mockery_mutable_config( - tmpdir, mutable_config, mock_packages, monkeypatch): + temporary_store, mutable_config, mock_packages +): """Hooks a fake install directory, DB, and stage directory into Spack. This is specifically for tests which want to use 'install_mockery' but also need to modify configuration (and hence would want to use 'mutable config'): 'install_mockery' does not support this. """ - monkeypatch.setattr( - spack.store, 'store', spack.store.Store(str(tmpdir.join('opt')))) - # We use a fake package, so temporarily disable checksumming with spack.config.override('config:checksum', False): yield - tmpdir.join('opt').remove() - @pytest.fixture() def mock_fetch(mock_archive, monkeypatch): @@ -1222,7 +1188,7 @@ repo: class MockBundle(object): has_code = False name = 'mock-bundle' - versions = {} + versions = {} # type: ignore @pytest.fixture diff --git a/lib/spack/spack/test/container/cli.py b/lib/spack/spack/test/container/cli.py index f9b3b43f83..26ae9f0c68 100644 --- a/lib/spack/spack/test/container/cli.py +++ b/lib/spack/spack/test/container/cli.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/container/conftest.py b/lib/spack/spack/test/container/conftest.py index 70a1ae2547..43e16cd180 100644 --- a/lib/spack/spack/test/container/conftest.py +++ b/lib/spack/spack/test/container/conftest.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/container/docker.py b/lib/spack/spack/test/container/docker.py index 6392f97db2..43a32b0720 100644 --- a/lib/spack/spack/test/container/docker.py +++ b/lib/spack/spack/test/container/docker.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/container/images.py b/lib/spack/spack/test/container/images.py index 6cec1cc592..d04bf60b08 100644 --- a/lib/spack/spack/test/container/images.py +++ b/lib/spack/spack/test/container/images.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/container/schema.py b/lib/spack/spack/test/container/schema.py index e86f9d3fe8..4bb0d574a9 100644 --- a/lib/spack/spack/test/container/schema.py +++ b/lib/spack/spack/test/container/schema.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/container/singularity.py b/lib/spack/spack/test/container/singularity.py index ab342cacec..12245e935e 100644 --- a/lib/spack/spack/test/container/singularity.py +++ b/lib/spack/spack/test/container/singularity.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/data/sourceme_first.sh b/lib/spack/spack/test/data/sourceme_first.sh index 8fa1774b17..6d473ea6d3 100644 --- a/lib/spack/spack/test/data/sourceme_first.sh +++ b/lib/spack/spack/test/data/sourceme_first.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/data/sourceme_lmod.sh b/lib/spack/spack/test/data/sourceme_lmod.sh index b71e338ec9..0ce370d09e 100644 --- a/lib/spack/spack/test/data/sourceme_lmod.sh +++ b/lib/spack/spack/test/data/sourceme_lmod.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/data/sourceme_parameters.sh b/lib/spack/spack/test/data/sourceme_parameters.sh index d11590d224..205a7006df 100644 --- a/lib/spack/spack/test/data/sourceme_parameters.sh +++ b/lib/spack/spack/test/data/sourceme_parameters.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/data/sourceme_second.sh b/lib/spack/spack/test/data/sourceme_second.sh index 3f16153627..79af1aea36 100644 --- a/lib/spack/spack/test/data/sourceme_second.sh +++ b/lib/spack/spack/test/data/sourceme_second.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/data/sourceme_unicode.sh b/lib/spack/spack/test/data/sourceme_unicode.sh index d819126bb6..b396dc43d7 100644 --- a/lib/spack/spack/test/data/sourceme_unicode.sh +++ b/lib/spack/spack/test/data/sourceme_unicode.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/data/sourceme_unset.sh b/lib/spack/spack/test/data/sourceme_unset.sh index 0366833838..2f1658529a 100644 --- a/lib/spack/spack/test/data/sourceme_unset.sh +++ b/lib/spack/spack/test/data/sourceme_unset.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/database.py b/lib/spack/spack/test/database.py index 8e0eac7cc0..cb9fbada2a 100644 --- a/lib/spack/spack/test/database.py +++ b/lib/spack/spack/test/database.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -38,16 +38,6 @@ pytestmark = pytest.mark.db @pytest.fixture() -def test_store(tmpdir): - real_store = spack.store.store - spack.store.store = spack.store.Store(str(tmpdir.join('test_store'))) - - yield - - spack.store.store = real_store - - -@pytest.fixture() def upstream_and_downstream_db(tmpdir_factory, gen_mock_layout): mock_db_root = str(tmpdir_factory.mktemp('mock_db_root')) upstream_write_db = spack.database.Database(mock_db_root) @@ -180,8 +170,8 @@ def test_add_to_upstream_after_downstream(upstream_and_downstream_db): spack.store.db = orig_db -@pytest.mark.usefixtures('config') -def test_cannot_write_upstream(tmpdir_factory, test_store, gen_mock_layout): +@pytest.mark.usefixtures('config', 'temporary_store') +def test_cannot_write_upstream(tmpdir_factory, gen_mock_layout): roots = [str(tmpdir_factory.mktemp(x)) for x in ['a', 'b']] layouts = [gen_mock_layout(x) for x in ['/ra/', '/rb/']] @@ -205,8 +195,8 @@ def test_cannot_write_upstream(tmpdir_factory, test_store, gen_mock_layout): upstream_dbs[0].add(spec, layouts[1]) -@pytest.mark.usefixtures('config') -def test_recursive_upstream_dbs(tmpdir_factory, test_store, gen_mock_layout): +@pytest.mark.usefixtures('config', 'temporary_store') +def test_recursive_upstream_dbs(tmpdir_factory, gen_mock_layout): roots = [str(tmpdir_factory.mktemp(x)) for x in ['a', 'b', 'c']] layouts = [gen_mock_layout(x) for x in ['/ra/', '/rb/', '/rc/']] @@ -648,10 +638,6 @@ def test_090_non_root_ref_counts(mutable_database): assert mpich_rec.ref_count == 0 -@pytest.mark.skipif( - os.environ.get('SPACK_TEST_SOLVER') == 'clingo', - reason='Test for Clingo are run in a container with root permissions' -) def test_100_no_write_with_exception_on_remove(database): def fail_while_writing(): with database.write_transaction(): @@ -669,10 +655,6 @@ def test_100_no_write_with_exception_on_remove(database): assert len(database.query('mpileaks ^zmpi', installed=any)) == 1 -@pytest.mark.skipif( - os.environ.get('SPACK_TEST_SOLVER') == 'clingo', - reason='Test for Clingo are run in a container with root permissions' -) def test_110_no_write_with_exception_on_install(database): def fail_while_writing(): with database.write_transaction(): diff --git a/lib/spack/spack/test/directives.py b/lib/spack/spack/test/directives.py index ae1b6b7e2a..55ec46d01d 100644 --- a/lib/spack/spack/test/directives.py +++ b/lib/spack/spack/test/directives.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/directory_layout.py b/lib/spack/spack/test/directory_layout.py index 3cb976a077..73446714ba 100644 --- a/lib/spack/spack/test/directory_layout.py +++ b/lib/spack/spack/test/directory_layout.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -7,6 +7,7 @@ This test verifies that the Spack directory layout works properly. """ import os +import os.path import pytest import spack.paths @@ -19,16 +20,6 @@ from spack.spec import Spec max_packages = 10 -@pytest.fixture() -def layout_and_dir(tmpdir): - """Returns a directory layout and the corresponding directory.""" - layout = YamlDirectoryLayout(str(tmpdir)) - old_layout = spack.store.layout - spack.store.layout = layout - yield layout, str(tmpdir) - spack.store.layout = old_layout - - def test_yaml_directory_layout_parameters(tmpdir, config): """This tests the various parameters that can be used to configure the install location """ @@ -84,14 +75,14 @@ def test_yaml_directory_layout_parameters(tmpdir, config): projections=projections_package7) -def test_read_and_write_spec(layout_and_dir, config, mock_packages): +def test_read_and_write_spec(temporary_store, config, mock_packages): """This goes through each package in spack and creates a directory for it. It then ensures that the spec for the directory's installed package can be read back in consistently, and finally that the directory can be removed by the directory layout. """ - layout, tmpdir = layout_and_dir + layout = temporary_store.layout packages = list(spack.repo.path.all_packages())[:max_packages] for pkg in packages: @@ -114,7 +105,7 @@ def test_read_and_write_spec(layout_and_dir, config, mock_packages): # Ensure directory has been created in right place. assert os.path.isdir(install_dir) - assert install_dir.startswith(str(tmpdir)) + assert install_dir.startswith(temporary_store.root) # Ensure spec file exists when directory is created assert os.path.isfile(spec_path) @@ -129,7 +120,7 @@ def test_read_and_write_spec(layout_and_dir, config, mock_packages): # TODO: fix this when we can concretize more loosely based on # TODO: what is installed. We currently omit these to # TODO: increase reuse of build dependencies. - stored_deptypes = ('link', 'run') + stored_deptypes = spack.hash_types.full_hash expected = spec.copy(deps=stored_deptypes) expected._mark_concrete() @@ -160,7 +151,7 @@ def test_read_and_write_spec(layout_and_dir, config, mock_packages): assert not os.path.exists(install_dir) -def test_handle_unknown_package(layout_and_dir, config, mock_packages): +def test_handle_unknown_package(temporary_store, config, mock_packages): """This test ensures that spack can at least do *some* operations with packages that are installed but that it does not know about. This is actually not such an uncommon @@ -171,7 +162,7 @@ def test_handle_unknown_package(layout_and_dir, config, mock_packages): information about installed packages' specs to uninstall or query them again if the package goes away. """ - layout, _ = layout_and_dir + layout = temporary_store.layout mock_db = spack.repo.RepoPath(spack.paths.mock_packages_path) not_in_mock = set.difference( @@ -209,9 +200,9 @@ def test_handle_unknown_package(layout_and_dir, config, mock_packages): assert spec.dag_hash() == spec_from_file.dag_hash() -def test_find(layout_and_dir, config, mock_packages): +def test_find(temporary_store, config, mock_packages): """Test that finding specs within an install layout works.""" - layout, _ = layout_and_dir + layout = temporary_store.layout packages = list(spack.repo.path.all_packages())[:max_packages] # Create install prefixes for all packages in the list diff --git a/lib/spack/spack/test/environment_modifications.py b/lib/spack/spack/test/environment_modifications.py index d1f5a4b791..014bf340bf 100644 --- a/lib/spack/spack/test/environment_modifications.py +++ b/lib/spack/spack/test/environment_modifications.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/fetch_strategy.py b/lib/spack/spack/test/fetch_strategy.py index 5470d88c13..d7f770eac3 100644 --- a/lib/spack/spack/test/fetch_strategy.py +++ b/lib/spack/spack/test/fetch_strategy.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/flag_handlers.py b/lib/spack/spack/test/flag_handlers.py index 6c7651ad19..8fac5a898c 100644 --- a/lib/spack/spack/test/flag_handlers.py +++ b/lib/spack/spack/test/flag_handlers.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/git_fetch.py b/lib/spack/spack/test/git_fetch.py index 8dc57da793..ad010e3ca5 100644 --- a/lib/spack/spack/test/git_fetch.py +++ b/lib/spack/spack/test/git_fetch.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/graph.py b/lib/spack/spack/test/graph.py index 7ad1a75a90..76c9524728 100644 --- a/lib/spack/spack/test/graph.py +++ b/lib/spack/spack/test/graph.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -122,3 +122,12 @@ o dyninst |/ o libelf ''' + + +def test_topo_sort_filtered(mock_packages): + """Test topo sort gives correct order when filtering link deps.""" + s = Spec('both-link-and-build-dep-a').normalized() + + topo = topological_sort(s, deptype=('link',)) + + assert topo == ['both-link-and-build-dep-a', 'both-link-and-build-dep-c'] diff --git a/lib/spack/spack/test/hg_fetch.py b/lib/spack/spack/test/hg_fetch.py index 2cdeace3e4..d0f7629cf8 100644 --- a/lib/spack/spack/test/hg_fetch.py +++ b/lib/spack/spack/test/hg_fetch.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/install.py b/lib/spack/spack/test/install.py index c2a8d0b100..77810ff0e3 100644 --- a/lib/spack/spack/test/install.py +++ b/lib/spack/spack/test/install.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -183,70 +183,70 @@ def test_flatten_deps( assert os.path.isdir(dependency_dir) -def test_installed_upstream_external( - tmpdir_factory, install_mockery, mock_fetch, gen_mock_layout, - monkeypatch): - """Check that when a dependency package is recorded as installed in - an upstream database that it is not reinstalled. +@pytest.fixture() +def install_upstream(tmpdir_factory, gen_mock_layout, install_mockery): + """Provides a function that installs a specified set of specs to an + upstream database. The function returns a store which points to the + upstream, as well as the upstream layout (for verifying that dependent + installs are using the upstream installs). """ mock_db_root = str(tmpdir_factory.mktemp('mock_db_root')) prepared_db = spack.database.Database(mock_db_root) - upstream_layout = gen_mock_layout('/a/') - dependency = spack.spec.Spec('externaltool') - dependency.concretize() - prepared_db.add(dependency, upstream_layout) - - downstream_db_root = str( - tmpdir_factory.mktemp('mock_downstream_db_root')) - db_for_test = spack.database.Database( - downstream_db_root, upstream_dbs=[prepared_db]) - monkeypatch.setattr(spack.store, 'db', db_for_test) - dependent = spack.spec.Spec('externaltest') - dependent.concretize() - - new_dependency = dependent['externaltool'] - assert new_dependency.external - assert new_dependency.prefix == '/path/to/external_tool' + def _install_upstream(*specs): + for spec_str in specs: + s = spack.spec.Spec(spec_str).concretized() + prepared_db.add(s, upstream_layout) - dependent.package.do_install() + downstream_root = str(tmpdir_factory.mktemp('mock_downstream_db_root')) + db_for_test = spack.database.Database( + downstream_root, upstream_dbs=[prepared_db] + ) + store = spack.store.Store(downstream_root) + store.db = db_for_test + return store, upstream_layout - assert not os.path.exists(new_dependency.prefix) - assert os.path.exists(dependent.prefix) + return _install_upstream -def test_installed_upstream(tmpdir_factory, install_mockery, mock_fetch, - gen_mock_layout, monkeypatch): +def test_installed_upstream_external(install_upstream, mock_fetch): """Check that when a dependency package is recorded as installed in - an upstream database that it is not reinstalled. + an upstream database that it is not reinstalled. """ - mock_db_root = str(tmpdir_factory.mktemp('mock_db_root')) - prepared_db = spack.database.Database(mock_db_root) + s, _ = install_upstream('externaltool') + with spack.store.use_store(s): + dependent = spack.spec.Spec('externaltest') + dependent.concretize() - upstream_layout = gen_mock_layout('/a/') + new_dependency = dependent['externaltool'] + assert new_dependency.external + assert new_dependency.prefix == '/path/to/external_tool' - dependency = spack.spec.Spec('dependency-install') - dependency.concretize() - prepared_db.add(dependency, upstream_layout) - - downstream_db_root = str( - tmpdir_factory.mktemp('mock_downstream_db_root')) - db_for_test = spack.database.Database( - downstream_db_root, upstream_dbs=[prepared_db]) - monkeypatch.setattr(spack.store, 'db', db_for_test) - dependent = spack.spec.Spec('dependent-install') - dependent.concretize() + dependent.package.do_install() + + assert not os.path.exists(new_dependency.prefix) + assert os.path.exists(dependent.prefix) + + +def test_installed_upstream(install_upstream, mock_fetch): + """Check that when a dependency package is recorded as installed in + an upstream database that it is not reinstalled. + """ + s, upstream_layout = install_upstream('dependency-install') + with spack.store.use_store(s): + dependency = spack.spec.Spec('dependency-install').concretized() + dependent = spack.spec.Spec('dependent-install').concretized() - new_dependency = dependent['dependency-install'] - assert new_dependency.package.installed_upstream - assert (new_dependency.prefix == - upstream_layout.path_for_spec(dependency)) + new_dependency = dependent['dependency-install'] + assert new_dependency.package.installed_upstream + assert (new_dependency.prefix == + upstream_layout.path_for_spec(dependency)) - dependent.package.do_install() + dependent.package.do_install() - assert not os.path.exists(new_dependency.prefix) - assert os.path.exists(dependent.prefix) + assert not os.path.exists(new_dependency.prefix) + assert os.path.exists(dependent.prefix) @pytest.mark.disable_clean_stage_check diff --git a/lib/spack/spack/test/installer.py b/lib/spack/spack/test/installer.py index 590c4a4a1a..861b0db8df 100644 --- a/lib/spack/spack/test/installer.py +++ b/lib/spack/spack/test/installer.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -231,7 +231,7 @@ def test_process_binary_cache_tarball_tar(install_mockery, monkeypatch, capfd): def test_try_install_from_binary_cache(install_mockery, mock_packages, monkeypatch, capsys): """Tests SystemExit path for_try_install_from_binary_cache.""" - def _mirrors_for_spec(spec, force, full_hash_match=False): + def _mirrors_for_spec(spec, full_hash_match=False): spec = spack.spec.Spec('mpi').concretized() return [{ 'mirror_url': 'notused', @@ -579,6 +579,32 @@ def test_clear_failures_errs(install_mockery, monkeypatch, capsys): monkeypatch.setattr(os, 'remove', orig_fn) +def test_combine_phase_logs(tmpdir): + """Write temporary files, and assert that combine phase logs works + to combine them into one file. We aren't currently using this function, + but it's available when the logs are refactored to be written separately. + """ + log_files = ['configure-out.txt', 'install-out.txt', 'build-out.txt'] + phase_log_files = [] + + # Create and write to dummy phase log files + for log_file in log_files: + phase_log_file = os.path.join(str(tmpdir), log_file) + with open(phase_log_file, 'w') as plf: + plf.write('Output from %s\n' % log_file) + phase_log_files.append(phase_log_file) + + # This is the output log we will combine them into + combined_log = os.path.join(str(tmpdir), "combined-out.txt") + spack.installer.combine_phase_logs(phase_log_files, combined_log) + with open(combined_log, 'r') as log_file: + out = log_file.read() + + # Ensure each phase log file is represented + for log_file in log_files: + assert "Output from %s\n" % log_file in out + + def test_check_deps_status_install_failure(install_mockery, monkeypatch): const_arg = installer_args(['a'], {}) installer = create_installer(const_arg) @@ -751,7 +777,11 @@ def test_requeue_task(install_mockery, capfd): installer = create_installer(const_arg) task = create_build_task(installer.build_requests[0].pkg) + # temporarily set tty debug messages on so we can test output + current_debug_level = tty.debug_level() + tty.set_debug(1) installer._requeue_task(task) + tty.set_debug(current_debug_level) ids = list(installer.build_tasks) assert len(ids) == 1 @@ -760,7 +790,7 @@ def test_requeue_task(install_mockery, capfd): assert qtask.sequence > task.sequence assert qtask.attempts == task.attempts + 1 - out = capfd.readouterr()[0] + out = capfd.readouterr()[1] assert 'Installing a' in out assert ' in progress by another process' in out @@ -879,7 +909,8 @@ def test_install_failed(install_mockery, monkeypatch, capsys): # Make sure the package is identified as failed monkeypatch.setattr(spack.database.Database, 'prefix_failed', _true) - installer.install() + with pytest.raises(inst.InstallError, match='request failed'): + installer.install() out = str(capsys.readouterr()) assert installer.build_requests[0].pkg_id in out @@ -894,7 +925,8 @@ def test_install_failed_not_fast(install_mockery, monkeypatch, capsys): # Make sure the package is identified as failed monkeypatch.setattr(spack.database.Database, 'prefix_failed', _true) - installer.install() + with pytest.raises(inst.InstallError, match='request failed'): + installer.install() out = str(capsys.readouterr()) assert 'failed to install' in out @@ -1046,7 +1078,9 @@ def test_install_lock_failures(install_mockery, monkeypatch, capfd): # Ensure don't continually requeue the task monkeypatch.setattr(inst.PackageInstaller, '_requeue_task', _requeued) - installer.install() + with pytest.raises(inst.InstallError, match='request failed'): + installer.install() + out = capfd.readouterr()[0] expected = ['write locked', 'read locked', 'requeued'] for exp, ln in zip(expected, out.split('\n')): @@ -1077,7 +1111,9 @@ def test_install_lock_installed_requeue(install_mockery, monkeypatch, capfd): # Ensure don't continually requeue the task monkeypatch.setattr(inst.PackageInstaller, '_requeue_task', _requeued) - installer.install() + with pytest.raises(inst.InstallError, match='request failed'): + installer.install() + assert b_pkg_id not in installer.installed out = capfd.readouterr()[0] @@ -1113,7 +1149,9 @@ def test_install_read_locked_requeue(install_mockery, monkeypatch, capfd): const_arg = installer_args(['b'], {}) installer = create_installer(const_arg) - installer.install() + with pytest.raises(inst.InstallError, match='request failed'): + installer.install() + assert 'b' not in installer.installed out = capfd.readouterr()[0] diff --git a/lib/spack/spack/test/link_paths.py b/lib/spack/spack/test/link_paths.py index 79a69942aa..4485bf3ae8 100644 --- a/lib/spack/spack/test/link_paths.py +++ b/lib/spack/spack/test/link_paths.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/llnl/util/__init__.py b/lib/spack/spack/test/llnl/util/__init__.py index 9f87532b85..103eae6134 100644 --- a/lib/spack/spack/test/llnl/util/__init__.py +++ b/lib/spack/spack/test/llnl/util/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/llnl/util/argparsewriter.py b/lib/spack/spack/test/llnl/util/argparsewriter.py index 127149bbaa..708110558e 100644 --- a/lib/spack/spack/test/llnl/util/argparsewriter.py +++ b/lib/spack/spack/test/llnl/util/argparsewriter.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/llnl/util/file_list.py b/lib/spack/spack/test/llnl/util/file_list.py index e4036fe492..4efec18067 100644 --- a/lib/spack/spack/test/llnl/util/file_list.py +++ b/lib/spack/spack/test/llnl/util/file_list.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/llnl/util/filesystem.py b/lib/spack/spack/test/llnl/util/filesystem.py index 501c07a1f6..5733be6b59 100644 --- a/lib/spack/spack/test/llnl/util/filesystem.py +++ b/lib/spack/spack/test/llnl/util/filesystem.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -588,3 +588,24 @@ def test_content_of_files_with_same_name(tmpdir): # and have not been mixed assert file1.read().strip() == 'file1' assert file2.read().strip() == 'file2' + + +def test_keep_modification_time(tmpdir): + file1 = tmpdir.ensure('file1') + file2 = tmpdir.ensure('file2') + + # Shift the modification time of the file 10 seconds back: + mtime1 = file1.mtime() - 10 + file1.setmtime(mtime1) + + with fs.keep_modification_time(file1.strpath, + file2.strpath, + 'non-existing-file'): + file1.write('file1') + file2.remove() + + # Assert that the modifications took place the modification time has not + # changed; + assert file1.read().strip() == 'file1' + assert not file2.exists() + assert int(mtime1) == int(file1.mtime()) diff --git a/lib/spack/spack/test/llnl/util/lang.py b/lib/spack/spack/test/llnl/util/lang.py index 83342dd6ed..99829f49d9 100644 --- a/lib/spack/spack/test/llnl/util/lang.py +++ b/lib/spack/spack/test/llnl/util/lang.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -6,6 +6,7 @@ import pytest import os.path +import sys from datetime import datetime, timedelta import llnl.util.lang @@ -27,7 +28,12 @@ value = 1 path = os.path.join('/usr', 'bin') """ m.write(content) - return str(m) + + yield str(m) + + # Don't leave garbage in the module system + if 'foo' in sys.modules: + del sys.modules['foo'] def test_pretty_date(): @@ -127,10 +133,22 @@ def test_match_predicate(): def test_load_modules_from_file(module_path): + # Check prerequisites + assert 'foo' not in sys.modules + + # Check that the module is loaded correctly from file foo = llnl.util.lang.load_module_from_file('foo', module_path) + assert 'foo' in sys.modules assert foo.value == 1 assert foo.path == os.path.join('/usr', 'bin') + # Check that the module is not reloaded a second time on subsequent calls + foo.value = 2 + foo = llnl.util.lang.load_module_from_file('foo', module_path) + assert 'foo' in sys.modules + assert foo.value == 2 + assert foo.path == os.path.join('/usr', 'bin') + def test_uniq(): assert [1, 2, 3] == llnl.util.lang.uniq([1, 2, 3]) diff --git a/lib/spack/spack/test/llnl/util/link_tree.py b/lib/spack/spack/test/llnl/util/link_tree.py index bebb3da805..af7012692f 100644 --- a/lib/spack/spack/test/llnl/util/link_tree.py +++ b/lib/spack/spack/test/llnl/util/link_tree.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/llnl/util/lock.py b/lib/spack/spack/test/llnl/util/lock.py index d4b61929c2..928bded6ca 100644 --- a/lib/spack/spack/test/llnl/util/lock.py +++ b/lib/spack/spack/test/llnl/util/lock.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -661,10 +661,6 @@ def test_upgrade_read_to_write(private_lock_path): assert lock._file is None -@pytest.mark.skipif( - os.environ.get('SPACK_TEST_SOLVER') == 'clingo', - reason='Test for Clingo are run in a container with root permissions' -) def test_upgrade_read_to_write_fails_with_readonly_file(private_lock_path): """Test that read-only file can be read-locked but not write-locked.""" # ensure lock file exists the first time @@ -1192,7 +1188,7 @@ def test_nested_reads(lock_path): class LockDebugOutput(object): def __init__(self, lock_path): self.lock_path = lock_path - self.host = socket.getfqdn() + self.host = socket.gethostname() def p1(self, barrier, q1, q2): # exchange pids diff --git a/lib/spack/spack/test/llnl/util/tty/__init__.py b/lib/spack/spack/test/llnl/util/tty/__init__.py index 9f87532b85..103eae6134 100644 --- a/lib/spack/spack/test/llnl/util/tty/__init__.py +++ b/lib/spack/spack/test/llnl/util/tty/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/llnl/util/tty/log.py b/lib/spack/spack/test/llnl/util/tty/log.py index 811b6c6642..080a6e1339 100644 --- a/lib/spack/spack/test/llnl/util/tty/log.py +++ b/lib/spack/spack/test/llnl/util/tty/log.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -11,10 +11,8 @@ import signal import sys import time -try: - import termios -except ImportError: - termios = None +from typing import Optional # novm +from types import ModuleType # novm import pytest @@ -25,6 +23,13 @@ from llnl.util.tty.pty import PseudoShell from spack.util.executable import which +termios = None # type: Optional[ModuleType] +try: + import termios as term_mod + termios = term_mod +except ImportError: + pass + @contextlib.contextmanager def nullcontext(): @@ -432,20 +437,14 @@ def test_foreground_background_output( with open(log_path) as log: log = log.read().strip().split("\n") - # Controller and minion process coordinate with locks such that the minion - # writes "off" when echo is off, and "on" when echo is on. The - # output should contain mostly "on" lines, but may contain an "off" - # or two. This is because the controller toggles echo by sending "v" on - # stdin to the minion, but this is not synchronized with our locks. - # It's good enough for a test, though. We allow at most 4 "off"'s in - # the output to account for the race. - # - # Originally we only allowed 2, but GitHub's macOS runners seem to be - # very slow, and frequently we get 3 "off"'s. Increased limit to 4 to - # account for this. Real errors should still be caught with this limit. + # Controller and minion process coordinate with locks such that the + # minion writes "off" when echo is off, and "on" when echo is on. The + # output should contain mostly "on" lines, but may contain "off" + # lines if the controller is slow. The important thing to observe + # here is that we started seeing 'on' in the end. assert ( ['forced output', 'on'] == uniq(output) or - output.count("off") <= 4 # if controller_fd is a bit slow + ['forced output', 'off', 'on'] == uniq(output) ) # log should be off for a while, then on, then off diff --git a/lib/spack/spack/test/llnl/util/tty/tty.py b/lib/spack/spack/test/llnl/util/tty/tty.py index b8366a9738..bc68543ec3 100644 --- a/lib/spack/spack/test/llnl/util/tty/tty.py +++ b/lib/spack/spack/test/llnl/util/tty/tty.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/main.py b/lib/spack/spack/test/main.py index c35a6e195b..068dce0bef 100644 --- a/lib/spack/spack/test/main.py +++ b/lib/spack/spack/test/main.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/make_executable.py b/lib/spack/spack/test/make_executable.py index f554fd038c..510487002a 100644 --- a/lib/spack/spack/test/make_executable.py +++ b/lib/spack/spack/test/make_executable.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/mirror.py b/lib/spack/spack/test/mirror.py index 8d3ec2d07b..2eef86e4f4 100644 --- a/lib/spack/spack/test/mirror.py +++ b/lib/spack/spack/test/mirror.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/module_parsing.py b/lib/spack/spack/test/module_parsing.py index 8dc06b058b..a1d0142b92 100644 --- a/lib/spack/spack/test/module_parsing.py +++ b/lib/spack/spack/test/module_parsing.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -123,3 +123,9 @@ def test_get_argument_from_module_line(): for bl in bad_lines: with pytest.raises(ValueError): get_path_args_from_module_line(bl) + + +def test_lmod_quote_parsing(): + lines = ['setenv("SOME_PARTICULAR_DIR","-L/opt/cray/pe/mpich/8.1.4/gtl/lib")'] + result = get_path_from_module_contents(lines, 'some-module') + assert '/opt/cray/pe/mpich/8.1.4/gtl' == result diff --git a/lib/spack/spack/test/modules/__init__.py b/lib/spack/spack/test/modules/__init__.py index 9f87532b85..103eae6134 100644 --- a/lib/spack/spack/test/modules/__init__.py +++ b/lib/spack/spack/test/modules/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/modules/common.py b/lib/spack/spack/test/modules/common.py index f2cb60e1db..0918cf2dfd 100644 --- a/lib/spack/spack/test/modules/common.py +++ b/lib/spack/spack/test/modules/common.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/modules/conftest.py b/lib/spack/spack/test/modules/conftest.py index 4ab129a5dd..dbfac6b0bc 100644 --- a/lib/spack/spack/test/modules/conftest.py +++ b/lib/spack/spack/test/modules/conftest.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/modules/lmod.py b/lib/spack/spack/test/modules/lmod.py index 474277f239..7239c487aa 100644 --- a/lib/spack/spack/test/modules/lmod.py +++ b/lib/spack/spack/test/modules/lmod.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/modules/tcl.py b/lib/spack/spack/test/modules/tcl.py index 70e3fc6c56..e5f2797e39 100644 --- a/lib/spack/spack/test/modules/tcl.py +++ b/lib/spack/spack/test/modules/tcl.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/multimethod.py b/lib/spack/spack/test/multimethod.py index 0e5bd5dad5..beb92bbb6d 100644 --- a/lib/spack/spack/test/multimethod.py +++ b/lib/spack/spack/test/multimethod.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/namespace_trie.py b/lib/spack/spack/test/namespace_trie.py index 2bf83c59f2..afe3af7386 100644 --- a/lib/spack/spack/test/namespace_trie.py +++ b/lib/spack/spack/test/namespace_trie.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/operating_system.py b/lib/spack/spack/test/operating_system.py index 97def3feda..c2fa52fa3e 100644 --- a/lib/spack/spack/test/operating_system.py +++ b/lib/spack/spack/test/operating_system.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/optional_deps.py b/lib/spack/spack/test/optional_deps.py index d9268e7e14..fe73a0ae8a 100644 --- a/lib/spack/spack/test/optional_deps.py +++ b/lib/spack/spack/test/optional_deps.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/package_class.py b/lib/spack/spack/test/package_class.py index 33e5eb1c0a..686c956186 100644 --- a/lib/spack/spack/test/package_class.py +++ b/lib/spack/spack/test/package_class.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/package_hash.py b/lib/spack/spack/test/package_hash.py index a8a160d96e..bcb1f6b804 100644 --- a/lib/spack/spack/test/package_hash.py +++ b/lib/spack/spack/test/package_hash.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/package_sanity.py b/lib/spack/spack/test/package_sanity.py index ded4d1c485..308706a5a7 100644 --- a/lib/spack/spack/test/package_sanity.py +++ b/lib/spack/spack/test/package_sanity.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -16,7 +16,7 @@ import spack.util.executable as executable import spack.variant # A few functions from this module are used to # do sanity checks only on packagess modified by a PR -import spack.cmd.flake8 as flake8 +import spack.cmd.style as style import spack.util.crypto as crypto import pickle @@ -207,7 +207,7 @@ def test_prs_update_old_api(): deprecated calls to any method. """ changed_package_files = [ - x for x in flake8.changed_files() if flake8.is_package(x) + x for x in style.changed_files() if style.is_package(x) ] failing = [] for file in changed_package_files: diff --git a/lib/spack/spack/test/packages.py b/lib/spack/spack/test/packages.py index ffaad396c1..c821d5b5c7 100644 --- a/lib/spack/spack/test/packages.py +++ b/lib/spack/spack/test/packages.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/packaging.py b/lib/spack/spack/test/packaging.py index 31ad3f1a7a..57446b6e2d 100644 --- a/lib/spack/spack/test/packaging.py +++ b/lib/spack/spack/test/packaging.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/patch.py b/lib/spack/spack/test/patch.py index 2dc781538a..57a5669c57 100644 --- a/lib/spack/spack/test/patch.py +++ b/lib/spack/spack/test/patch.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -124,9 +124,9 @@ def test_patch_order(mock_packages, config): spec = Spec('dep-diamond-patch-top') spec.concretize() - mid2_sha256 = 'mid21234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234' # noqa: E501 - mid1_sha256 = '0b62284961dab49887e31319843431ee5b037382ac02c4fe436955abef11f094' # noqa: E501 - top_sha256 = 'f7de2947c64cb6435e15fb2bef359d1ed5f6356b2aebb7b20535e3772904e6db' # noqa: E501 + mid2_sha256 = 'mid21234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234' + mid1_sha256 = '0b62284961dab49887e31319843431ee5b037382ac02c4fe436955abef11f094' + top_sha256 = 'f7de2947c64cb6435e15fb2bef359d1ed5f6356b2aebb7b20535e3772904e6db' dep = spec['patch'] patch_order = dep.variants['patches']._patches_in_order_of_appearance @@ -328,9 +328,25 @@ def test_write_and_read_sub_dags_with_patched_deps(mock_packages, config): spec.package.package_dir) -def test_file_patch_no_file(): +def test_patch_no_file(): + # Give it the attributes we need to construct the error message + FakePackage = collections.namedtuple( + 'FakePackage', ['name', 'namespace', 'fullname']) + fp = FakePackage('fake-package', 'test', 'fake-package') + with pytest.raises(ValueError, match='FilePatch:'): + spack.patch.FilePatch(fp, 'nonexistent_file', 0, '') + + patch = spack.patch.Patch(fp, 'nonexistent_file', 0, '') + patch.path = 'test' + with pytest.raises(spack.patch.NoSuchPatchError, match='No such patch:'): + patch.apply('') + + +@pytest.mark.parametrize('level', [-1, 0.0, '1']) +def test_invalid_level(level): # Give it the attributes we need to construct the error message FakePackage = collections.namedtuple('FakePackage', ['name', 'namespace']) fp = FakePackage('fake-package', 'test') - with pytest.raises(ValueError, match=r'FilePatch:.*'): - spack.patch.FilePatch(fp, 'nonexistent_file', 0, '') + with pytest.raises(ValueError, + match='Patch level needs to be a non-negative integer.'): + spack.patch.Patch(fp, 'nonexistent_file', level, '') diff --git a/lib/spack/spack/test/pattern.py b/lib/spack/spack/test/pattern.py index 582ece3d51..94f4e6d5f0 100644 --- a/lib/spack/spack/test/pattern.py +++ b/lib/spack/spack/test/pattern.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/permissions.py b/lib/spack/spack/test/permissions.py index 26974c8096..320ff2fdba 100644 --- a/lib/spack/spack/test/permissions.py +++ b/lib/spack/spack/test/permissions.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/provider_index.py b/lib/spack/spack/test/provider_index.py index 0855ac50ec..1f9f0a74e0 100644 --- a/lib/spack/spack/test/provider_index.py +++ b/lib/spack/spack/test/provider_index.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/relocate.py b/lib/spack/spack/test/relocate.py index afeb2162db..fb1ed4eb60 100644 --- a/lib/spack/spack/test/relocate.py +++ b/lib/spack/spack/test/relocate.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/repo.py b/lib/spack/spack/test/repo.py index d10349e6ec..df3015500d 100644 --- a/lib/spack/spack/test/repo.py +++ b/lib/spack/spack/test/repo.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -67,3 +67,15 @@ def test_repo_invisibles(mutable_mock_repo, extra_repo): with open(os.path.join(extra_repo.root, 'packages', '.invisible'), 'w'): pass extra_repo.all_package_names() + + +@pytest.mark.parametrize('attr_name,exists', [ + ('cmake', True), + ('__sphinx_mock__', False) +]) +@pytest.mark.regression('20661') +def test_namespace_hasattr(attr_name, exists, mutable_mock_repo): + # Check that we don't fail on 'hasattr' checks because + # of a custom __getattr__ implementation + nms = spack.repo.SpackNamespace('spack.pkg.builtin.mock') + assert hasattr(nms, attr_name) == exists diff --git a/lib/spack/spack/test/s3_fetch.py b/lib/spack/spack/test/s3_fetch.py index 70efad19ce..e339a1c173 100644 --- a/lib/spack/spack/test/s3_fetch.py +++ b/lib/spack/spack/test/s3_fetch.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/sbang.py b/lib/spack/spack/test/sbang.py index 9307d2bc4b..422ec3cdc0 100644 --- a/lib/spack/spack/test/sbang.py +++ b/lib/spack/spack/test/sbang.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -44,7 +44,7 @@ sbang_line = '#!/bin/sh %s/bin/sbang\n' % spack.store.store.unpadded_root last_line = "last!\n" -@pytest.fixture +@pytest.fixture # type: ignore[no-redef] def sbang_line(): yield '#!/bin/sh %s/bin/sbang\n' % spack.store.layout.root diff --git a/lib/spack/spack/test/schema.py b/lib/spack/spack/test/schema.py index 14c9014d61..1b9694a8e8 100644 --- a/lib/spack/spack/test/schema.py +++ b/lib/spack/spack/test/schema.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/spack_yaml.py b/lib/spack/spack/test/spack_yaml.py index fb4abbf1d8..8e4fe3c9f4 100644 --- a/lib/spack/spack/test/spack_yaml.py +++ b/lib/spack/spack/test/spack_yaml.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/spec_dag.py b/lib/spack/spack/test/spec_dag.py index 854495988b..ea16f655db 100644 --- a/lib/spack/spack/test/spec_dag.py +++ b/lib/spack/spack/test/spec_dag.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -13,6 +13,7 @@ import spack.package from spack.spec import Spec from spack.dependency import all_deptypes, Dependency, canonical_deptype from spack.util.mock_package import MockPackageMultiRepo +import spack.util.hash as hashutil def check_links(spec_to_check): @@ -705,17 +706,17 @@ class TestSpecDag(object): for c in test_hash]) for bits in (1, 2, 3, 4, 7, 8, 9, 16, 64, 117, 128, 160): - actual_int = spack.spec.base32_prefix_bits(test_hash, bits) + actual_int = hashutil.base32_prefix_bits(test_hash, bits) fmt = "#0%sb" % (bits + 2) actual = format(actual_int, fmt).replace('0b', '') assert expected[:bits] == actual with pytest.raises(ValueError): - spack.spec.base32_prefix_bits(test_hash, 161) + hashutil.base32_prefix_bits(test_hash, 161) with pytest.raises(ValueError): - spack.spec.base32_prefix_bits(test_hash, 256) + hashutil.base32_prefix_bits(test_hash, 256) def test_traversal_directions(self): """Make sure child and parent traversals of specs work.""" diff --git a/lib/spack/spack/test/spec_list.py b/lib/spack/spack/test/spec_list.py index 27567b4080..48796d0ad9 100644 --- a/lib/spack/spack/test/spec_list.py +++ b/lib/spack/spack/test/spec_list.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/spec_semantics.py b/lib/spack/spack/test/spec_semantics.py index b2161d4a0e..16a4cf77ce 100644 --- a/lib/spack/spack/test/spec_semantics.py +++ b/lib/spack/spack/test/spec_semantics.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -776,7 +776,7 @@ class TestSpecSematics(object): sigil_package_segments = [("{@VERSIONS}", '@' + str(spec.version)), ("{%compiler}", '%' + str(spec.compiler)), ("{arch=architecture}", - ' arch=' + str(spec.architecture))] + 'arch=' + str(spec.architecture))] compiler_segments = [("{compiler.name}", "name"), ("{compiler.version}", "versions")] @@ -798,7 +798,7 @@ class TestSpecSematics(object): for named_str, prop in package_segments: expected = getattr(spec, prop, "") actual = spec.format(named_str) - assert str(expected) == actual + assert str(expected).strip() == actual for named_str, expected in sigil_package_segments: actual = spec.format(named_str) @@ -985,6 +985,74 @@ class TestSpecSematics(object): assert 'avx512' not in spec.target assert spec.target < 'broadwell' + @pytest.mark.parametrize('transitive', [True, False]) + def test_splice(self, transitive): + # Tests the new splice function in Spec using a somewhat simple case + # with a variant with a conditional dependency. + # TODO: Test being able to splice in different provider for a virtual. + # Example: mvapich for mpich. + spec = Spec('splice-t') + dep = Spec('splice-h+foo') + spec.concretize() + dep.concretize() + # Sanity checking that these are not the same thing. + assert dep.dag_hash() != spec['splice-h'].dag_hash() + assert dep.build_hash() != spec['splice-h'].build_hash() + # Do the splice. + out = spec.splice(dep, transitive) + # Returned spec should still be concrete. + assert out.concrete + # Traverse the spec and assert that all dependencies are accounted for. + for node in spec.traverse(): + assert node.name in out + # If the splice worked, then the full hash of the spliced dep should + # now match the full hash of the build spec of the dependency from the + # returned spec. + out_h_build = out['splice-h'].build_spec + assert out_h_build.full_hash() == dep.full_hash() + # Transitivity should determine whether the transitive dependency was + # changed. + expected_z = dep['splice-z'] if transitive else spec['splice-z'] + assert out['splice-z'].full_hash() == expected_z.full_hash() + # Sanity check build spec of out should be the original spec. + assert (out['splice-t'].build_spec.full_hash() == + spec['splice-t'].full_hash()) + # Finally, the spec should know it's been spliced: + assert out.spliced + + @pytest.mark.parametrize('transitive', [True, False]) + def test_splice_input_unchanged(self, transitive): + spec = Spec('splice-t').concretized() + dep = Spec('splice-h+foo').concretized() + orig_spec_hash = spec.full_hash() + orig_dep_hash = dep.full_hash() + spec.splice(dep, transitive) + # Post-splice, dag hash should still be different; no changes should be + # made to these specs. + assert spec.full_hash() == orig_spec_hash + assert dep.full_hash() == orig_dep_hash + + @pytest.mark.parametrize('transitive', [True, False]) + def test_splice_subsequent(self, transitive): + spec = Spec('splice-t') + dep = Spec('splice-h+foo') + spec.concretize() + dep.concretize() + out = spec.splice(dep, transitive) + # Now we attempt a second splice. + dep = Spec('splice-z+bar') + dep.concretize() + # Transitivity shouldn't matter since Splice Z has no dependencies. + out2 = out.splice(dep, transitive) + assert out2.concrete + assert out2['splice-z'].build_hash() != spec['splice-z'].build_hash() + assert out2['splice-z'].build_hash() != out['splice-z'].build_hash() + assert out2['splice-z'].full_hash() != spec['splice-z'].full_hash() + assert out2['splice-z'].full_hash() != out['splice-z'].full_hash() + assert (out2['splice-t'].build_spec.full_hash() == + spec['splice-t'].full_hash()) + assert out2.spliced + @pytest.mark.parametrize('spec,constraint,expected_result', [ ('libelf target=haswell', 'target=broadwell', False), ('libelf target=haswell', 'target=haswell', True), diff --git a/lib/spack/spack/test/spec_syntax.py b/lib/spack/spack/test/spec_syntax.py index b59828206f..fd5108c809 100644 --- a/lib/spack/spack/test/spec_syntax.py +++ b/lib/spack/spack/test/spec_syntax.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -352,10 +352,10 @@ class TestSpecSyntax(object): def test_ambiguous_hash(self, mutable_database): x1 = Spec('a') x1.concretize() - x1._hash = 'xy' + x1._hash = 'xyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy' x2 = Spec('a') x2.concretize() - x2._hash = 'xx' + x2._hash = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' mutable_database.add(x1, spack.store.layout) mutable_database.add(x2, spack.store.layout) @@ -557,10 +557,6 @@ class TestSpecSyntax(object): with specfile.open('w') as f: f.write(s['libelf'].to_yaml(hash=ht.build_hash)) - print("") - print("") - print("PARSING HERE") - # Make sure we can use yaml path as dependency, e.g.: # "spack spec libdwarf ^ /path/to/libelf.yaml" specs = sp.parse('libdwarf ^ {0}'.format(specfile.strpath)) diff --git a/lib/spack/spack/test/spec_yaml.py b/lib/spack/spack/test/spec_yaml.py index b32fe09e03..74ebf0a541 100644 --- a/lib/spack/spack/test/spec_yaml.py +++ b/lib/spack/spack/test/spec_yaml.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -130,8 +130,13 @@ def test_to_record_dict(mock_packages, config): assert record[key] == value +@pytest.mark.parametrize("hash_type", [ + ht.dag_hash, + ht.build_hash, + ht.full_hash +]) def test_ordered_read_not_required_for_consistent_dag_hash( - config, mock_packages + hash_type, config, mock_packages ): """Make sure ordered serialization isn't required to preserve hashes. @@ -148,15 +153,15 @@ def test_ordered_read_not_required_for_consistent_dag_hash( # # Dict & corresponding YAML & JSON from the original spec. # - spec_dict = spec.to_dict() - spec_yaml = spec.to_yaml() - spec_json = spec.to_json() + spec_dict = spec.to_dict(hash=hash_type) + spec_yaml = spec.to_yaml(hash=hash_type) + spec_json = spec.to_json(hash=hash_type) # # Make a spec with reversed OrderedDicts for every # OrderedDict in the original. # - reversed_spec_dict = reverse_all_dicts(spec.to_dict()) + reversed_spec_dict = reverse_all_dicts(spec.to_dict(hash=hash_type)) # # Dump to YAML and JSON @@ -190,11 +195,13 @@ def test_ordered_read_not_required_for_consistent_dag_hash( reversed_json_string ) - # TODO: remove this when build deps are in provenance. - spec = spec.copy(deps=('link', 'run')) + # Strip spec if we stripped the yaml + spec = spec.copy(deps=hash_type.deptype) + # specs are equal to the original assert spec == round_trip_yaml_spec assert spec == round_trip_json_spec + assert spec == round_trip_reversed_yaml_spec assert spec == round_trip_reversed_json_spec assert round_trip_yaml_spec == round_trip_reversed_yaml_spec @@ -204,16 +211,18 @@ def test_ordered_read_not_required_for_consistent_dag_hash( assert spec.dag_hash() == round_trip_json_spec.dag_hash() assert spec.dag_hash() == round_trip_reversed_yaml_spec.dag_hash() assert spec.dag_hash() == round_trip_reversed_json_spec.dag_hash() - # full_hashes are equal - spec.concretize() - round_trip_yaml_spec.concretize() - round_trip_json_spec.concretize() - round_trip_reversed_yaml_spec.concretize() - round_trip_reversed_json_spec.concretize() - assert spec.full_hash() == round_trip_yaml_spec.full_hash() - assert spec.full_hash() == round_trip_json_spec.full_hash() - assert spec.full_hash() == round_trip_reversed_yaml_spec.full_hash() - assert spec.full_hash() == round_trip_reversed_json_spec.full_hash() + + # full_hashes are equal if we round-tripped by build_hash or full_hash + if hash_type in (ht.build_hash, ht.full_hash): + spec.concretize() + round_trip_yaml_spec.concretize() + round_trip_json_spec.concretize() + round_trip_reversed_yaml_spec.concretize() + round_trip_reversed_json_spec.concretize() + assert spec.full_hash() == round_trip_yaml_spec.full_hash() + assert spec.full_hash() == round_trip_json_spec.full_hash() + assert spec.full_hash() == round_trip_reversed_yaml_spec.full_hash() + assert spec.full_hash() == round_trip_reversed_json_spec.full_hash() @pytest.mark.parametrize("module", [ diff --git a/lib/spack/spack/test/stage.py b/lib/spack/spack/test/stage.py index 9dea091730..3e3ceeaaa0 100644 --- a/lib/spack/spack/test/stage.py +++ b/lib/spack/spack/test/stage.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/svn_fetch.py b/lib/spack/spack/test/svn_fetch.py index f435b07db2..87e9779ca9 100644 --- a/lib/spack/spack/test/svn_fetch.py +++ b/lib/spack/spack/test/svn_fetch.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/tengine.py b/lib/spack/spack/test/tengine.py index 01f2d65780..00216a7fdc 100644 --- a/lib/spack/spack/test/tengine.py +++ b/lib/spack/spack/test/tengine.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/test_activations.py b/lib/spack/spack/test/test_activations.py index e8ce6c55ed..a05032d5a2 100644 --- a/lib/spack/spack/test/test_activations.py +++ b/lib/spack/spack/test/test_activations.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/test_suite.py b/lib/spack/spack/test/test_suite.py index 1ec5106182..50c5989a3d 100644 --- a/lib/spack/spack/test/test_suite.py +++ b/lib/spack/spack/test/test_suite.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/url_fetch.py b/lib/spack/spack/test/url_fetch.py index 28b6c87385..9804e4eb72 100644 --- a/lib/spack/spack/test/url_fetch.py +++ b/lib/spack/spack/test/url_fetch.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/url_parse.py b/lib/spack/spack/test/url_parse.py index 846e24d242..332c988fca 100644 --- a/lib/spack/spack/test/url_parse.py +++ b/lib/spack/spack/test/url_parse.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/url_substitution.py b/lib/spack/spack/test/url_substitution.py index 08a3b99a4a..8fbfd48714 100644 --- a/lib/spack/spack/test/url_substitution.py +++ b/lib/spack/spack/test/url_substitution.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/util/__init__.py b/lib/spack/spack/test/util/__init__.py index 9f87532b85..103eae6134 100644 --- a/lib/spack/spack/test/util/__init__.py +++ b/lib/spack/spack/test/util/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/util/editor.py b/lib/spack/spack/test/util/editor.py index e1495a70cc..aa85323212 100644 --- a/lib/spack/spack/test/util/editor.py +++ b/lib/spack/spack/test/util/editor.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/util/environment.py b/lib/spack/spack/test/util/environment.py index a555c35c9c..b7b6a0edea 100644 --- a/lib/spack/spack/test/util/environment.py +++ b/lib/spack/spack/test/util/environment.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/util/executable.py b/lib/spack/spack/test/util/executable.py index ae2859ea4b..9bfa9a408a 100644 --- a/lib/spack/spack/test/util/executable.py +++ b/lib/spack/spack/test/util/executable.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/util/file_cache.py b/lib/spack/spack/test/util/file_cache.py index 0fec1a5dc9..c033ca49d9 100644 --- a/lib/spack/spack/test/util/file_cache.py +++ b/lib/spack/spack/test/util/file_cache.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/util/log_parser.py b/lib/spack/spack/test/util/log_parser.py index 0f829082b6..911226f381 100644 --- a/lib/spack/spack/test/util/log_parser.py +++ b/lib/spack/spack/test/util/log_parser.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/util/mock_package.py b/lib/spack/spack/test/util/mock_package.py index cca55bb534..2609d849db 100644 --- a/lib/spack/spack/test/util/mock_package.py +++ b/lib/spack/spack/test/util/mock_package.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/util/prefix.py b/lib/spack/spack/test/util/prefix.py index cee3ad028b..d7724f616e 100644 --- a/lib/spack/spack/test/util/prefix.py +++ b/lib/spack/spack/test/util/prefix.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/util/spack_lock_wrapper.py b/lib/spack/spack/test/util/spack_lock_wrapper.py index d7819c84cb..73a2e41eb0 100644 --- a/lib/spack/spack/test/util/spack_lock_wrapper.py +++ b/lib/spack/spack/test/util/spack_lock_wrapper.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/util/spack_yaml.py b/lib/spack/spack/test/util/spack_yaml.py index 23e02d7be6..34a923093f 100644 --- a/lib/spack/spack/test/util/spack_yaml.py +++ b/lib/spack/spack/test/util/spack_yaml.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -65,3 +65,31 @@ def test_config_blame_with_override(config): check_blame('verify_ssl', config_file, 13) check_blame('checksum', config_file, 14) check_blame('dirty', config_file, 15) + + +def test_config_blame_defaults(): + """check blame for an element from an override scope""" + files = {} + + def get_file_lines(filename): + if filename not in files: + with open(filename, "r") as f: + files[filename] = [""] + f.read().split("\n") + return files[filename] + + config_blame = config_cmd("blame", "config") + for line in config_blame.split("\n"): + # currently checking only simple lines with dict keys + match = re.match(r"^([^:]+):(\d+)\s+([^:]+):\s+(.*)", line) + + # check that matches are on the lines they say they are + if match: + filename, line, key, val = match.groups() + line = int(line) + + if val.lower() in ("true", "false"): + val = val.lower() + + lines = get_file_lines(filename) + assert key in lines[line] + assert val in lines[line] diff --git a/lib/spack/spack/test/util/util_gpg.py b/lib/spack/spack/test/util/util_gpg.py index 83fad508f9..987dcf7f0e 100644 --- a/lib/spack/spack/test/util/util_gpg.py +++ b/lib/spack/spack/test/util/util_gpg.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -65,6 +65,7 @@ fpr:::::::::ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ:""" @pytest.mark.skipif(not spack.util.gpg.GpgConstants.user_run_dir, reason='This test requires /var/run/user/$(id -u)') +@pytest.mark.requires_executables('gpg2') def test_really_long_gnupg_home_dir(tmpdir): N = 960 diff --git a/lib/spack/spack/test/util/util_string.py b/lib/spack/spack/test/util/util_string.py index bcaa7d6211..7c68895be2 100644 --- a/lib/spack/spack/test/util/util_string.py +++ b/lib/spack/spack/test/util/util_string.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/util/util_url.py b/lib/spack/spack/test/util/util_url.py index 5d0d9fa4eb..07a055c572 100644 --- a/lib/spack/spack/test/util/util_url.py +++ b/lib/spack/spack/test/util/util_url.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/variant.py b/lib/spack/spack/test/variant.py index 10e8ea7e7f..501d897321 100644 --- a/lib/spack/spack/test/variant.py +++ b/lib/spack/spack/test/variant.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/verification.py b/lib/spack/spack/test/verification.py index 009f407186..cbeba8f635 100644 --- a/lib/spack/spack/test/verification.py +++ b/lib/spack/spack/test/verification.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/versions.py b/lib/spack/spack/test/versions.py index 315788f7f3..a33f7e08f7 100644 --- a/lib/spack/spack/test/versions.py +++ b/lib/spack/spack/test/versions.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -373,6 +373,9 @@ def test_intersect_with_containment(): check_intersection('1.6:1.6.5', ':1.6.5', '1.6') check_intersection('1.6:1.6.5', '1.6', ':1.6.5') + check_intersection('11.2', '11', '11.2') + check_intersection('11.2', '11.2', '11') + def test_union_with_containment(): check_union(':1.6', '1.6.5', ':1.6') diff --git a/lib/spack/spack/test/views.py b/lib/spack/spack/test/views.py index e1dcec4c5c..18fd88b093 100644 --- a/lib/spack/spack/test/views.py +++ b/lib/spack/spack/test/views.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/test/web.py b/lib/spack/spack/test/web.py index 472ab0dab7..d5f76632a4 100644 --- a/lib/spack/spack/test/web.py +++ b/lib/spack/spack/test/web.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -6,10 +6,14 @@ import os import ordereddict_backport import pytest +import spack.config import spack.paths import spack.util.web +import spack.util.s3 from spack.version import ver +import llnl.util.tty as tty + def _create_url(relative_url): web_data_path = os.path.join(spack.paths.test_path, 'data', 'web') @@ -195,3 +199,82 @@ def test_list_url(tmpdir): 'file-0.txt', 'file-1.txt', 'file-2.txt'] + + +class MockPages(object): + def search(self, *args, **kwargs): + return [ + {'Key': 'keyone'}, + {'Key': 'keytwo'}, + {'Key': 'keythree'}, + ] + + +class MockPaginator(object): + def paginate(self, *args, **kwargs): + return MockPages() + + +class MockClientError(Exception): + def __init__(self): + self.response = {'Error': {'Code': 'NoSuchKey'}} + + +class MockS3Client(object): + def get_paginator(self, *args, **kwargs): + return MockPaginator() + + def delete_objects(self, *args, **kwargs): + return { + 'Errors': [ + {'Key': 'keyone', 'Message': 'Access Denied'} + ], + 'Deleted': [ + {'Key': 'keytwo'}, + {'Key': 'keythree'} + ], + } + + def delete_object(self, *args, **kwargs): + pass + + def get_object(self, Bucket=None, Key=None): + self.ClientError = MockClientError + if Bucket == 'my-bucket' and Key == 'subdirectory/my-file': + return True + raise self.ClientError + + +def test_remove_s3_url(monkeypatch, capfd): + fake_s3_url = 's3://my-bucket/subdirectory/mirror' + + def mock_create_s3_session(url): + return MockS3Client() + + monkeypatch.setattr( + spack.util.s3, 'create_s3_session', mock_create_s3_session) + + current_debug_level = tty.debug_level() + tty.set_debug(1) + + spack.util.web.remove_url(fake_s3_url, recursive=True) + err = capfd.readouterr()[1] + + tty.set_debug(current_debug_level) + + assert('Failed to delete keyone (Access Denied)' in err) + assert('Deleted keythree' in err) + assert('Deleted keytwo' in err) + + +def test_s3_url_exists(monkeypatch, capfd): + def mock_create_s3_session(url): + return MockS3Client() + monkeypatch.setattr( + spack.util.s3, 'create_s3_session', mock_create_s3_session) + + fake_s3_url_exists = 's3://my-bucket/subdirectory/my-file' + assert(spack.util.web.url_exists(fake_s3_url_exists)) + + fake_s3_url_does_not_exist = 's3://my-bucket/subdirectory/my-notfound-file' + assert(not spack.util.web.url_exists(fake_s3_url_does_not_exist)) diff --git a/lib/spack/spack/url.py b/lib/spack/spack/url.py index aa40524ae8..1fc0f2f522 100644 --- a/lib/spack/spack/url.py +++ b/lib/spack/spack/url.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -56,8 +56,12 @@ def find_list_urls(url): GitLab https://gitlab.\*/<repo>/<name>/tags BitBucket https://bitbucket.org/<repo>/<name>/downloads/?tab=tags CRAN https://\*.r-project.org/src/contrib/Archive/<name> + PyPI https://pypi.org/simple/<name>/ ========= ======================================================= + Note: this function is called by `spack versions`, `spack checksum`, + and `spack create`, but not by `spack fetch` or `spack install`. + Parameters: url (str): The download URL for the package @@ -91,6 +95,16 @@ def find_list_urls(url): # e.g. https://cloud.r-project.org/src/contrib/rgl_0.98.1.tar.gz (r'(.*\.r-project\.org/src/contrib)/([^_]+)', lambda m: m.group(1) + '/Archive/' + m.group(2)), + + # PyPI + # e.g. https://pypi.io/packages/source/n/numpy/numpy-1.19.4.zip + # e.g. https://www.pypi.io/packages/source/n/numpy/numpy-1.19.4.zip + # e.g. https://pypi.org/packages/source/n/numpy/numpy-1.19.4.zip + # e.g. https://pypi.python.org/packages/source/n/numpy/numpy-1.19.4.zip + # e.g. https://files.pythonhosted.org/packages/source/n/numpy/numpy-1.19.4.zip + # e.g. https://pypi.io/packages/py2.py3/o/opencensus-context/opencensus_context-0.1.1-py2.py3-none-any.whl + (r'(?:pypi|pythonhosted)[^/]+/packages/[^/]+/./([^/]+)', + lambda m: 'https://pypi.org/simple/' + m.group(1) + '/'), ] list_urls = set([os.path.dirname(url)]) diff --git a/lib/spack/spack/user_environment.py b/lib/spack/spack/user_environment.py index f08f21e0ec..7dd63dcdea 100644 --- a/lib/spack/spack/user_environment.py +++ b/lib/spack/spack/user_environment.py @@ -1,4 +1,4 @@ -# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -72,7 +72,7 @@ def environment_modifications_for_spec(spec, view=None): the view.""" spec = spec.copy() if view and not spec.external: - spec.prefix = prefix.Prefix(view.view().get_projection_for_spec(spec)) + spec.prefix = prefix.Prefix(view.get_projection_for_spec(spec)) # generic environment modifications determined by inspecting the spec # prefix diff --git a/lib/spack/spack/util/__init__.py b/lib/spack/spack/util/__init__.py index 9f87532b85..103eae6134 100644 --- a/lib/spack/spack/util/__init__.py +++ b/lib/spack/spack/util/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/util/classes.py b/lib/spack/spack/util/classes.py new file mode 100644 index 0000000000..b6465c7581 --- /dev/null +++ b/lib/spack/spack/util/classes.py @@ -0,0 +1,39 @@ +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +# Need this because of spack.util.string +from __future__ import absolute_import +from spack.util.naming import mod_to_class +from llnl.util.lang import memoized, list_modules +import llnl.util.tty as tty + +import inspect + +__all__ = [ + 'list_classes' +] + + +@memoized +def list_classes(parent_module, mod_path): + """Given a parent path (e.g., spack.platforms or spack.analyzers), + use list_modules to derive the module names, and then mod_to_class + to derive class names. Import the classes and return them in a list + """ + classes = [] + + for name in list_modules(mod_path): + mod_name = '%s.%s' % (parent_module, name) + class_name = mod_to_class(name) + mod = __import__(mod_name, fromlist=[class_name]) + if not hasattr(mod, class_name): + tty.die('No class %s defined in %s' % (class_name, mod_name)) + cls = getattr(mod, class_name) + if not inspect.isclass(cls): + tty.die('%s.%s is not a class' % (mod_name, class_name)) + + classes.append(cls) + + return classes diff --git a/lib/spack/spack/util/compression.py b/lib/spack/spack/util/compression.py index ebbe0519d0..a1a0e1d6f3 100644 --- a/lib/spack/spack/util/compression.py +++ b/lib/spack/spack/util/compression.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -22,6 +22,22 @@ def allowed_archive(path): return any(path.endswith(t) for t in ALLOWED_ARCHIVE_TYPES) +def _gunzip(archive_file): + """Like gunzip, but extracts in the current working directory + instead of in-place. + + Args: + archive_file (str): absolute path of the file to be decompressed + """ + import gzip + decompressed_file = os.path.basename(archive_file.strip('.gz')) + working_dir = os.getcwd() + destination_abspath = os.path.join(working_dir, decompressed_file) + with gzip.open(archive_file, "rb") as f_in: + with open(destination_abspath, "wb") as f_out: + f_out.write(f_in.read()) + + def decompressor_for(path, extension=None): """Get the appropriate decompressor for a path.""" if ((extension and re.match(r'\.?zip$', extension)) or @@ -30,8 +46,7 @@ def decompressor_for(path, extension=None): unzip.add_default_arg('-q') return unzip if extension and re.match(r'gz', extension): - gunzip = which('gunzip', required=True) - return gunzip + return _gunzip if extension and re.match(r'bz2', extension): bunzip2 = which('bunzip2', required=True) return bunzip2 diff --git a/lib/spack/spack/util/cpus.py b/lib/spack/spack/util/cpus.py new file mode 100644 index 0000000000..9fa10c87a8 --- /dev/null +++ b/lib/spack/spack/util/cpus.py @@ -0,0 +1,20 @@ +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +import os +import multiprocessing + + +def cpus_available(): + """ + Returns the number of CPUs available for the current process, or the number + of phyiscal CPUs when that information cannot be retrieved. The number + of available CPUs might differ from the number of physical CPUs when + using spack through Slurm or container runtimes. + """ + try: + return len(os.sched_getaffinity(0)) # novermin + except Exception: + return multiprocessing.cpu_count() diff --git a/lib/spack/spack/util/crypto.py b/lib/spack/spack/util/crypto.py index 7e264ff0c7..809b36ffea 100644 --- a/lib/spack/spack/util/crypto.py +++ b/lib/spack/spack/util/crypto.py @@ -1,14 +1,14 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) import sys import hashlib +from typing import Dict, Callable, Any # novm import llnl.util.tty as tty - #: Set of hash algorithms that Spack can use, mapped to digest size in bytes hashes = { 'md5': 16, @@ -30,7 +30,7 @@ _deprecated_hash_algorithms = ['md5'] #: cache of hash functions generated -_hash_functions = {} +_hash_functions = {} # type: Dict[str, Callable[[], Any]] class DeprecatedHash(object): diff --git a/lib/spack/spack/util/debug.py b/lib/spack/spack/util/debug.py index 1ff74f24cf..290ad832f8 100644 --- a/lib/spack/spack/util/debug.py +++ b/lib/spack/spack/util/debug.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/util/editor.py b/lib/spack/spack/util/editor.py index 00a095252e..9b0081b19a 100644 --- a/lib/spack/spack/util/editor.py +++ b/lib/spack/spack/util/editor.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/util/environment.py b/lib/spack/spack/util/environment.py index 6d299fa7ef..04158c119b 100644 --- a/lib/spack/spack/util/environment.py +++ b/lib/spack/spack/util/environment.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -9,7 +9,9 @@ import contextlib import inspect import json import os +import platform import re +import socket import sys import os.path @@ -139,6 +141,40 @@ def pickle_environment(path, environment=None): open(path, 'wb'), protocol=2) +def get_host_environment_metadata(): + """Get the host environment, reduce to a subset that we can store in + the install directory, and add the spack version. + """ + import spack.main + environ = get_host_environment() + return {"host_os": environ['os'], + "platform": environ['platform'], + "host_target": environ['target'], + "hostname": environ['hostname'], + "spack_version": spack.main.get_version(), + "kernel_version": platform.version()} + + +def get_host_environment(): + """Return a dictionary (lookup) with host information (not including the + os.environ). + """ + import spack.spec + import spack.architecture as architecture + arch = architecture.Arch( + architecture.platform(), 'default_os', 'default_target') + arch_spec = spack.spec.Spec('arch=%s' % arch) + return { + 'target': str(arch.target), + 'os': str(arch.os), + 'platform': str(arch.platform), + 'arch': arch_spec, + 'architecture': arch_spec, + 'arch_str': str(arch), + 'hostname': socket.gethostname() + } + + @contextlib.contextmanager def set_env(**kwargs): """Temporarily sets and restores environment variables. @@ -943,8 +979,7 @@ def environment_after_sourcing_files(*files, **kwargs): concatenate_on_success, dump_environment, ]) output = shell( - source_file_arguments, output=str, env=environment, - ignore_quotes=True + source_file_arguments, output=str, env=environment, ignore_quotes=True ) environment = json.loads(output) diff --git a/lib/spack/spack/util/executable.py b/lib/spack/spack/util/executable.py index 8baea6d238..c1e90ff25f 100644 --- a/lib/spack/spack/util/executable.py +++ b/lib/spack/spack/util/executable.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -92,8 +92,8 @@ class Executable(object): ignore_errors (int or list): A list of error codes to ignore. If these codes are returned, this process will not raise an exception even if ``fail_on_error`` is set to ``True`` - ignore_quotes (bool): If False, warn users that quotes are not - needed as Spack does not use a shell. Defaults to False. + ignore_quotes (bool): If False, warn users that quotes are not needed + as Spack does not use a shell. Defaults to False. input: Where to read stdin from output: Where to send stdout error: Where to send stderr @@ -168,14 +168,12 @@ class Executable(object): istream, close_istream = streamify(input, 'r') if not ignore_quotes: - quoted_args = [ - arg for arg in args if re.search(r'^"|^\'|"$|\'$', arg) - ] + quoted_args = [arg for arg in args if re.search(r'^"|^\'|"$|\'$', arg)] if quoted_args: tty.warn( "Quotes in command arguments can confuse scripts like" " configure.", - "These arguments may cause problems when executed:", + "The following arguments may cause problems when executed:", str("\n".join([" " + arg for arg in quoted_args])), "Quotes aren't needed because spack doesn't use a shell. " "Consider removing them.", diff --git a/lib/spack/spack/util/file_cache.py b/lib/spack/spack/util/file_cache.py index 9342f9772b..670b2ed698 100644 --- a/lib/spack/spack/util/file_cache.py +++ b/lib/spack/spack/util/file_cache.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/util/file_permissions.py b/lib/spack/spack/util/file_permissions.py index b133d2569e..7ed974816e 100644 --- a/lib/spack/spack/util/file_permissions.py +++ b/lib/spack/spack/util/file_permissions.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/util/gpg.py b/lib/spack/spack/util/gpg.py index eaa199e417..aa376bc041 100644 --- a/lib/spack/spack/util/gpg.py +++ b/lib/spack/spack/util/gpg.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -115,7 +115,7 @@ if not cached_property: return result -class GpgConstants(object): +class _GpgConstants(object): @cached_property def target_version(self): return spack.version.Version('2') @@ -244,7 +244,7 @@ class GpgConstants(object): pass -GpgConstants = GpgConstants() +GpgConstants = _GpgConstants() def ensure_gpg(reevaluate=False): @@ -382,7 +382,7 @@ def gpg(*args, **kwargs): return get_global_gpg_instance()(*args, **kwargs) -gpg.name = 'gpg' +gpg.name = 'gpg' # type: ignore[attr-defined] @functools.wraps(Gpg.create) diff --git a/lib/spack/spack/util/hash.py b/lib/spack/spack/util/hash.py new file mode 100644 index 0000000000..da81284ea6 --- /dev/null +++ b/lib/spack/spack/util/hash.py @@ -0,0 +1,31 @@ +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +import base64 +import hashlib +import sys + +import spack.util.crypto + + +def b32_hash(content): + """Return the b32 encoded sha1 hash of the input string as a string.""" + sha = hashlib.sha1(content.encode('utf-8')) + b32_hash = base64.b32encode(sha.digest()).lower() + + if sys.version_info[0] >= 3: + b32_hash = b32_hash.decode('utf-8') + + return b32_hash + + +def base32_prefix_bits(hash_string, bits): + """Return the first <bits> bits of a base32 string as an integer.""" + if bits > len(hash_string) * 5: + raise ValueError("Too many bits! Requested %d bit prefix of '%s'." + % (bits, hash_string)) + + hash_bytes = base64.b32decode(hash_string, casefold=True) + return spack.util.crypto.prefix_bits(hash_bytes, bits) diff --git a/lib/spack/spack/util/imp/__init__.py b/lib/spack/spack/util/imp/__init__.py index 27e2b00f9f..975b9a49a7 100644 --- a/lib/spack/spack/util/imp/__init__.py +++ b/lib/spack/spack/util/imp/__init__.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/util/imp/imp_importer.py b/lib/spack/spack/util/imp/imp_importer.py index 810e1d01a3..a9b5cd641a 100644 --- a/lib/spack/spack/util/imp/imp_importer.py +++ b/lib/spack/spack/util/imp/imp_importer.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/util/imp/importlib_importer.py b/lib/spack/spack/util/imp/importlib_importer.py index c230f4c50a..4c8a2673e5 100644 --- a/lib/spack/spack/util/imp/importlib_importer.py +++ b/lib/spack/spack/util/imp/importlib_importer.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/util/lock.py b/lib/spack/spack/util/lock.py index b748aa056a..2f465f590a 100644 --- a/lib/spack/spack/util/lock.py +++ b/lib/spack/spack/util/lock.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -15,7 +15,7 @@ import spack.error import spack.paths -class Lock(llnl.util.lock.Lock): +class Lock(llnl.util.lock.Lock): # type: ignore[no-redef] """Lock that can be disabled. This overrides the ``_lock()`` and ``_unlock()`` methods from diff --git a/lib/spack/spack/util/log_parse.py b/lib/spack/spack/util/log_parse.py index 4b01c18c91..7973bfd6b0 100644 --- a/lib/spack/spack/util/log_parse.py +++ b/lib/spack/spack/util/log_parse.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -43,7 +43,7 @@ def parse_log_events(stream, context=6, jobs=None, profile=False): #: lazily constructed CTest log parser -parse_log_events.ctest_parser = None +parse_log_events.ctest_parser = None # type: ignore[attr-defined] def _wrap(text, width): @@ -107,7 +107,7 @@ def make_log_context(log_events, width=None): for i in range(start, event.end): # wrap to width lines = _wrap(event[i], wrap_width) - lines[1:] = [indent + l for l in lines[1:]] + lines[1:] = [indent + ln for ln in lines[1:]] wrapped_line = line_fmt % (i, '\n'.join(lines)) if i in error_lines: diff --git a/lib/spack/spack/util/mock_package.py b/lib/spack/spack/util/mock_package.py index 734a93c469..847177ae05 100644 --- a/lib/spack/spack/util/mock_package.py +++ b/lib/spack/spack/util/mock_package.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -81,8 +81,8 @@ class MockPackageMultiRepo(object): def __init__(self): self.spec_to_pkg = {} - self.namespace = '' - self.full_namespace = 'spack.pkg.mock' + self.namespace = 'mock' # repo namespace + self.full_namespace = 'spack.pkg.mock' # python import namespace def get(self, spec): if not isinstance(spec, spack.spec.Spec): @@ -92,6 +92,10 @@ class MockPackageMultiRepo(object): return self.spec_to_pkg[spec.name] def get_pkg_class(self, name): + namespace, _, name = name.rpartition(".") + if namespace and namespace != self.namespace: + raise spack.repo.InvalidNamespaceError( + "bad namespace: %s" % self.namespace) return self.spec_to_pkg[name] def exists(self, name): diff --git a/lib/spack/spack/util/module_cmd.py b/lib/spack/spack/util/module_cmd.py index bc994fd4b4..fa6996fcbd 100644 --- a/lib/spack/spack/util/module_cmd.py +++ b/lib/spack/spack/util/module_cmd.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -155,7 +155,7 @@ def path_from_modules(modules): if candidate_path and not os.path.exists(candidate_path): msg = ("Extracted path from module does not exist " - "[module={0}, path={0}]") + "[module={0}, path={1}]") tty.warn(msg.format(module_name, candidate_path)) # If anything is found, then it's the best choice. This means @@ -195,9 +195,13 @@ def get_path_from_module_contents(text, module_name): def match_flag_and_strip(line, flag, strip=[]): flag_idx = line.find(flag) if flag_idx >= 0: - end = line.find(' ', flag_idx) - if end >= 0: - path = line[flag_idx + len(flag):end] + # Search for the first occurence of any separator marking the end of + # the path. + separators = (' ', '"', "'") + occurrences = [line.find(s, flag_idx) for s in separators] + indices = [idx for idx in occurrences if idx >= 0] + if indices: + path = line[flag_idx + len(flag):min(indices)] else: path = line[flag_idx + len(flag):] path = strip_path(path, strip) diff --git a/lib/spack/spack/util/naming.py b/lib/spack/spack/util/naming.py index 1cc6d6109f..db5d1d4907 100644 --- a/lib/spack/spack/util/naming.py +++ b/lib/spack/spack/util/naming.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/util/package_hash.py b/lib/spack/spack/util/package_hash.py index adea1a498b..462f43d223 100644 --- a/lib/spack/spack/util/package_hash.py +++ b/lib/spack/spack/util/package_hash.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/util/path.py b/lib/spack/spack/util/path.py index 182f93c585..0735133b35 100644 --- a/lib/spack/spack/util/path.py +++ b/lib/spack/spack/util/path.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -17,7 +17,7 @@ import llnl.util.tty as tty from llnl.util.lang import memoized import spack.paths - +import spack.util.spack_yaml as syaml __all__ = [ 'substitute_config_variables', @@ -72,12 +72,22 @@ def substitute_config_variables(path): - $spack The Spack instance's prefix - $user The current user's username - $tempdir Default temporary directory returned by tempfile.gettempdir() + - $env The active Spack environment. These are substituted case-insensitively into the path, and users can - use either ``$var`` or ``${var}`` syntax for the variables. - + use either ``$var`` or ``${var}`` syntax for the variables. $env is only + replaced if there is an active environment, and should only be used in + environment yaml files. """ - # Look up replacements for re.sub in the replacements dict. + import spack.environment as ev # break circular + env = ev.get_env({}, '') + if env: + replacements.update({'env': env.path}) + else: + # If a previous invocation added env, remove it + replacements.pop('env', None) + + # Look up replacements def repl(match): m = match.group(0).strip('${}') return replacements.get(m.lower(), match.group(0)) @@ -132,7 +142,19 @@ def add_padding(path, length): def canonicalize_path(path): """Same as substitute_path_variables, but also take absolute path.""" - path = substitute_path_variables(path) - path = os.path.abspath(path) + # Get file in which path was written in case we need to make it absolute + # relative to that path. + filename = None + if isinstance(path, syaml.syaml_str): + filename = os.path.dirname(path._start_mark.name) + assert path._start_mark.name == path._end_mark.name - return path + path = substitute_path_variables(path) + if not os.path.isabs(path): + if filename: + path = os.path.join(filename, path) + else: + path = os.path.abspath(path) + tty.debug("Using current working directory as base for abspath") + + return os.path.normpath(path) diff --git a/lib/spack/spack/util/pattern.py b/lib/spack/spack/util/pattern.py index c0569f5f55..fc4b324b17 100644 --- a/lib/spack/spack/util/pattern.py +++ b/lib/spack/spack/util/pattern.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/util/prefix.py b/lib/spack/spack/util/prefix.py index 10ec50d30c..b4542c7eb0 100644 --- a/lib/spack/spack/util/prefix.py +++ b/lib/spack/spack/util/prefix.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/util/s3.py b/lib/spack/spack/util/s3.py index c9ed22f72c..31a18093e1 100644 --- a/lib/spack/spack/util/s3.py +++ b/lib/spack/spack/util/s3.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -22,6 +22,7 @@ def create_s3_session(url): # want to require boto as a dependency unless the user actually wants to # access S3 mirrors. from boto3 import Session + from botocore.exceptions import ClientError session = Session() @@ -41,4 +42,6 @@ def create_s3_session(url): s3_client_args["config"] = Config(signature_version=UNSIGNED) - return session.client('s3', **s3_client_args) + client = session.client('s3', **s3_client_args) + client.ClientError = ClientError + return client diff --git a/lib/spack/spack/util/spack_json.py b/lib/spack/spack/util/spack_json.py index 01f773058b..62fd9ca824 100644 --- a/lib/spack/spack/util/spack_json.py +++ b/lib/spack/spack/util/spack_json.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -38,10 +38,14 @@ def dump(data, stream=None): def _strify(data, ignore_dicts=False): + """Converts python 2 unicodes to str in JSON data.""" + # this is a no-op in python 3 + if sys.version_info[0] >= 3: + return data + # if this is a unicode string in python 2, return its string representation - if sys.version_info[0] < 3: - if isinstance(data, string_types): - return data.encode('utf-8') + if isinstance(data, string_types): + return data.encode('utf-8') # if this is a list of values, return list of byteified values if isinstance(data, list): diff --git a/lib/spack/spack/util/spack_yaml.py b/lib/spack/spack/util/spack_yaml.py index 4e61e02a4e..24eea88b77 100644 --- a/lib/spack/spack/util/spack_yaml.py +++ b/lib/spack/spack/util/spack_yaml.py @@ -13,7 +13,9 @@ """ import ctypes +import re import sys +from typing import List # novm from ordereddict_backport import OrderedDict from six import string_types, StringIO @@ -184,6 +186,12 @@ class OrderedLineDumper(RoundTripDumper): """Make the dumper NEVER print YAML aliases.""" return True + def represent_data(self, data): + result = super(OrderedLineDumper, self).represent_data(data) + if data is None: + result.value = syaml_str("null") + return result + def represent_str(self, data): if hasattr(data, 'override') and data.override: data = data + ':' @@ -226,7 +234,7 @@ def file_line(mark): #: This is nasty but YAML doesn't give us many ways to pass arguments -- #: yaml.dump() takes a class (not an instance) and instantiates the dumper #: itself, so we can't just pass an instance -_annotations = [] +_annotations = [] # type: List[str] class LineAnnotationDumper(OrderedLineDumper): @@ -261,19 +269,18 @@ class LineAnnotationDumper(OrderedLineDumper): def represent_data(self, data): """Force syaml_str to be passed through with marks.""" result = super(LineAnnotationDumper, self).represent_data(data) - if isinstance(result.value, string_types): + if data is None: + result.value = syaml_str("null") + elif isinstance(result.value, string_types): result.value = syaml_str(data) if markable(result.value): mark(result.value, data) return result - def write_stream_start(self): - super(LineAnnotationDumper, self).write_stream_start() - _annotations.append(colorize('@K{---}')) - def write_line_break(self): super(LineAnnotationDumper, self).write_line_break() - if not self.saved: + if self.saved is None: + _annotations.append(colorize('@K{---}')) return # append annotations at the end of each line @@ -321,7 +328,10 @@ def dump_annotated(data, stream=None, *args, **kwargs): sio = StringIO() yaml.dump(data, sio, *args, **kwargs) - lines = sio.getvalue().rstrip().split('\n') + + # write_line_break() is not called by YAML for empty lines, so we + # skip empty lines here with \n+. + lines = re.split(r"\n+", sio.getvalue().rstrip()) getvalue = None if stream is None: diff --git a/lib/spack/spack/util/string.py b/lib/spack/spack/util/string.py index 7284cef2d3..6137c32d5a 100644 --- a/lib/spack/spack/util/string.py +++ b/lib/spack/spack/util/string.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/util/url.py b/lib/spack/spack/util/url.py index ab5503229f..f8abf4fa96 100644 --- a/lib/spack/spack/util/url.py +++ b/lib/spack/spack/util/url.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/util/web.py b/lib/spack/spack/util/web.py index 1b415f5de2..f59f3bb064 100644 --- a/lib/spack/spack/util/web.py +++ b/lib/spack/spack/util/web.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -20,17 +20,6 @@ import six from six.moves.urllib.error import URLError from six.moves.urllib.request import urlopen, Request -try: - # Python 2 had these in the HTMLParser package. - from HTMLParser import HTMLParser, HTMLParseError # novm -except ImportError: - # In Python 3, things moved to html.parser - from html.parser import HTMLParser - - # Also, HTMLParseError is deprecated and never raised. - class HTMLParseError(Exception): - pass - from llnl.util.filesystem import mkdirp import llnl.util.tty as tty @@ -45,6 +34,16 @@ import llnl.util.lang from spack.util.compression import ALLOWED_ARCHIVE_TYPES +if sys.version_info < (3, 0): + # Python 2 had these in the HTMLParser package. + from HTMLParser import HTMLParser, HTMLParseError # novm +else: + # In Python 3, things moved to html.parser + from html.parser import HTMLParser + + # Also, HTMLParseError is deprecated and never raised. + class HTMLParseError(Exception): + pass # Timeout in seconds for web requests _timeout = 10 @@ -211,11 +210,10 @@ def url_exists(url): if url.scheme == 's3': s3 = s3_util.create_s3_session(url) - from botocore.exceptions import ClientError try: - s3.get_object(Bucket=url.netloc, Key=url.path) + s3.get_object(Bucket=url.netloc, Key=url.path.lstrip('/')) return True - except ClientError as err: + except s3.ClientError as err: if err.response['Error']['Code'] == 'NoSuchKey': return False raise err @@ -229,17 +227,57 @@ def url_exists(url): return False -def remove_url(url): +def _debug_print_delete_results(result): + if 'Deleted' in result: + for d in result['Deleted']: + tty.debug('Deleted {0}'.format(d['Key'])) + if 'Errors' in result: + for e in result['Errors']: + tty.debug('Failed to delete {0} ({1})'.format( + e['Key'], e['Message'])) + + +def remove_url(url, recursive=False): url = url_util.parse(url) local_path = url_util.local_file_path(url) if local_path: - os.remove(local_path) + if recursive: + shutil.rmtree(local_path) + else: + os.remove(local_path) return if url.scheme == 's3': s3 = s3_util.create_s3_session(url) - s3.delete_object(Bucket=url.netloc, Key=url.path) + bucket = url.netloc + if recursive: + # Because list_objects_v2 can only return up to 1000 items + # at a time, we have to paginate to make sure we get it all + prefix = url.path.strip('/') + paginator = s3.get_paginator('list_objects_v2') + pages = paginator.paginate(Bucket=bucket, Prefix=prefix) + + delete_request = {'Objects': []} + for item in pages.search('Contents'): + if not item: + continue + + delete_request['Objects'].append({'Key': item['Key']}) + + # Make sure we do not try to hit S3 with a list of more + # than 1000 items + if len(delete_request['Objects']) >= 1000: + r = s3.delete_objects(Bucket=bucket, Delete=delete_request) + _debug_print_delete_results(r) + delete_request = {'Objects': []} + + # Delete any items that remain + if len(delete_request['Objects']): + r = s3.delete_objects(Bucket=bucket, Delete=delete_request) + _debug_print_delete_results(r) + else: + s3.delete_object(Bucket=bucket, Key=url.path) return # Don't even try for other URL schemes. @@ -549,7 +587,10 @@ def find_versions_of_archive( # .sha256 # .sig # However, SourceForge downloads still need to end in '/download'. - url_regex += r'(\/download)?$' + url_regex += r'(\/download)?' + # PyPI adds #sha256=... to the end of the URL + url_regex += '(#sha256=.*)?' + url_regex += '$' regexes.append(url_regex) diff --git a/lib/spack/spack/variant.py b/lib/spack/spack/variant.py index 8fa52f9738..0929a5ea44 100644 --- a/lib/spack/spack/variant.py +++ b/lib/spack/spack/variant.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -12,6 +12,12 @@ import inspect import itertools import re from six import StringIO +import sys + +if sys.version_info >= (3, 5): + from collections.abc import Sequence # novm +else: + from collections import Sequence import llnl.util.tty.color import llnl.util.lang as lang @@ -20,11 +26,6 @@ from spack.util.string import comma_or import spack.directives import spack.error as error -try: - from collections.abc import Sequence # novm -except ImportError: - from collections import Sequence - special_variant_values = [None, 'none', '*'] @@ -200,7 +201,7 @@ def implicit_variant_conversion(method): return convert -@lang.key_ordering +@lang.lazy_lexicographic_ordering class AbstractVariant(object): """A variant that has not yet decided who it wants to be. It behaves like a multi valued variant which **could** do things. @@ -281,8 +282,14 @@ class AbstractVariant(object): # to a set self._value = tuple(sorted(set(value))) - def _cmp_key(self): - return self.name, self.value + def _cmp_iter(self): + yield self.name + + value = self._value + if not isinstance(value, tuple): + value = (value,) + value = tuple(str(x) for x in value) + yield value def copy(self): """Returns an instance of a variant equivalent to self diff --git a/lib/spack/spack/verify.py b/lib/spack/spack/verify.py index 528fa7903b..b8d76e03e2 100644 --- a/lib/spack/spack/verify.py +++ b/lib/spack/spack/verify.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) diff --git a/lib/spack/spack/version.py b/lib/spack/spack/version.py index 44e03edb5c..82c0afd53e 100644 --- a/lib/spack/spack/version.py +++ b/lib/spack/spack/version.py @@ -1,4 +1,4 @@ -# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other +# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Spack Project Developers. See the top-level COPYRIGHT file for details. # # SPDX-License-Identifier: (Apache-2.0 OR MIT) @@ -37,7 +37,10 @@ from spack.util.spack_yaml import syaml_dict __all__ = ['Version', 'VersionRange', 'VersionList', 'ver'] # Valid version characters -VALID_VERSION = r'[A-Za-z0-9_.-]' +VALID_VERSION = re.compile(r'[A-Za-z0-9_.-]') + +# regex for version segments +SEGMENT_REGEX = re.compile(r'[a-zA-Z]+|[0-9]+') # Infinity-like versions. The order in the list implies the comparison rules infinity_versions = ['develop', 'main', 'master', 'head', 'trunk'] @@ -97,9 +100,10 @@ class Version(object): """Class to represent versions""" def __init__(self, string): - string = str(string) + if not isinstance(string, str): + string = str(string) - if not re.match(VALID_VERSION, string): + if not VALID_VERSION.match(string): raise ValueError("Bad characters in version string: %s" % string) # preserve the original string, but trimmed. @@ -107,12 +111,11 @@ class Version(object): self.string = string # Split version into alphabetical and numeric segments - segment_regex = r'[a-zA-Z]+|[0-9]+' - segments = re.findall(segment_regex, string) + segments = SEGMENT_REGEX.findall(string) self.version = tuple(int_if_int(seg) for seg in segments) # Store the separators from the original version string as well. - self.separators = tuple(re.split(segment_regex, string)[1:]) + self.separators = tuple(SEGMENT_REGEX.split(string)[1:]) @property def dotted(self): @@ -369,8 +372,10 @@ class Version(object): @coerced def intersection(self, other): - if self == other: + if self in other: # also covers `self == other` return self + elif other in self: + return other else: return VersionList() |