diff options
Diffstat (limited to 'lib')
| -rw-r--r-- | lib/spack/docs/basic_usage.rst | 242 | ||||
| -rw-r--r-- | lib/spack/docs/build_systems/pythonpackage.rst | 42 | ||||
| -rw-r--r-- | lib/spack/spack/bootstrap/_common.py | 10 | ||||
| -rw-r--r-- | lib/spack/spack/bootstrap/environment.py | 62 | ||||
| -rw-r--r-- | lib/spack/spack/build_systems/cmake.py | 13 | ||||
| -rw-r--r-- | lib/spack/spack/build_systems/python.py | 40 | ||||
| -rw-r--r-- | lib/spack/spack/directives.py | 6 | ||||
| -rw-r--r-- | lib/spack/spack/hooks/windows_runtime_linkage.py | 8 | ||||
| -rw-r--r-- | lib/spack/spack/installer.py | 4 | ||||
| -rw-r--r-- | lib/spack/spack/spec.py | 95 | ||||
| -rw-r--r-- | lib/spack/spack/test/cmd/extensions.py | 14 | ||||
| -rw-r--r-- | lib/spack/spack/util/executable.py | 14 |
12 files changed, 257 insertions, 293 deletions
diff --git a/lib/spack/docs/basic_usage.rst b/lib/spack/docs/basic_usage.rst index f26c6e5683..e49ca3073e 100644 --- a/lib/spack/docs/basic_usage.rst +++ b/lib/spack/docs/basic_usage.rst @@ -865,7 +865,7 @@ There are several different ways to use Spack packages once you have installed them. As you've seen, spack packages are installed into long paths with hashes, and you need a way to get them into your path. The easiest way is to use :ref:`spack load <cmd-spack-load>`, which is -described in the next section. +described in this section. Some more advanced ways to use Spack packages include: @@ -959,7 +959,86 @@ use ``spack find --loaded``. You can also use ``spack load --list`` to get the same output, but it does not have the full set of query options that ``spack find`` offers. -We'll learn more about Spack's spec syntax in the next section. +We'll learn more about Spack's spec syntax in :ref:`a later section <sec-specs>`. + + +.. _extensions: + +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Python packages and virtual environments +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Spack can install a large number of Python packages. Their names are +typically prefixed with ``py-``. Installing and using them is no +different from any other package: + +.. code-block:: console + + $ spack install py-numpy + $ spack load py-numpy + $ python3 + >>> import numpy + +The ``spack load`` command sets the ``PATH`` variable so that the right Python +executable is used, and makes sure that ``numpy`` and its dependencies can be +located in the ``PYTHONPATH``. + +Spack is different from other Python package managers in that it installs +every package into its *own* prefix. This is in contrast to ``pip``, which +installs all packages into the same prefix, be it in a virtual environment +or not. + +For many users, **virtual environments** are more convenient than repeated +``spack load`` commands, particularly when working with multiple Python +packages. Fortunately Spack supports environments itself, which together +with a view are no different from Python virtual environments. + +The recommended way of working with Python extensions such as ``py-numpy`` +is through :ref:`Environments <environments>`. The following example creates +a Spack environment with ``numpy`` in the current working directory. It also +puts a filesystem view in ``./view``, which is a more traditional combined +prefix for all packages in the environment. + +.. code-block:: console + + $ spack env create --with-view view --dir . + $ spack -e . add py-numpy + $ spack -e . concretize + $ spack -e . install + +Now you can activate the environment and start using the packages: + +.. code-block:: console + + $ spack env activate . + $ python3 + >>> import numpy + +The environment view is also a virtual environment, which is useful if you are +sharing the environment with others who are unfamiliar with Spack. They can +either use the Python executable directly: + +.. code-block:: console + + $ ./view/bin/python3 + >>> import numpy + +or use the activation script: + +.. code-block:: console + + $ source ./view/bin/activate + $ python3 + >>> import numpy + +In general, there should not be much difference between ``spack env activate`` +and using the virtual environment. The main advantage of ``spack env activate`` +is that it knows about more packages than just Python packages, and it may set +additional runtime variables that are not covered by the virtual environment +activation script. + +See :ref:`environments` for a more in-depth description of Spack +environments and customizations to views. .. _sec-specs: @@ -1705,165 +1784,6 @@ check only local packages (as opposed to those used transparently from ``upstream`` spack instances) and the ``-j,--json`` option to output machine-readable json data for any errors. - -.. _extensions: - ---------------------------- -Extensions & Python support ---------------------------- - -Spack's installation model assumes that each package will live in its -own install prefix. However, certain packages are typically installed -*within* the directory hierarchy of other packages. For example, -`Python <https://www.python.org>`_ packages are typically installed in the -``$prefix/lib/python-2.7/site-packages`` directory. - -In Spack, installation prefixes are immutable, so this type of installation -is not directly supported. However, it is possible to create views that -allow you to merge install prefixes of multiple packages into a single new prefix. -Views are a convenient way to get a more traditional filesystem structure. -Using *extensions*, you can ensure that Python packages always share the -same prefix in the view as Python itself. Suppose you have -Python installed like so: - -.. code-block:: console - - $ spack find python - ==> 1 installed packages. - -- linux-debian7-x86_64 / gcc@4.4.7 -------------------------------- - python@2.7.8 - -.. _cmd-spack-extensions: - -^^^^^^^^^^^^^^^^^^^^ -``spack extensions`` -^^^^^^^^^^^^^^^^^^^^ - -You can find extensions for your Python installation like this: - -.. code-block:: console - - $ spack extensions python - ==> python@2.7.8%gcc@4.4.7 arch=linux-debian7-x86_64-703c7a96 - ==> 36 extensions: - geos py-ipython py-pexpect py-pyside py-sip - py-basemap py-libxml2 py-pil py-pytz py-six - py-biopython py-mako py-pmw py-rpy2 py-sympy - py-cython py-matplotlib py-pychecker py-scientificpython py-virtualenv - py-dateutil py-mpi4py py-pygments py-scikit-learn - py-epydoc py-mx py-pylint py-scipy - py-gnuplot py-nose py-pyparsing py-setuptools - py-h5py py-numpy py-pyqt py-shiboken - - ==> 12 installed: - -- linux-debian7-x86_64 / gcc@4.4.7 -------------------------------- - py-dateutil@2.4.0 py-nose@1.3.4 py-pyside@1.2.2 - py-dateutil@2.4.0 py-numpy@1.9.1 py-pytz@2014.10 - py-ipython@2.3.1 py-pygments@2.0.1 py-setuptools@11.3.1 - py-matplotlib@1.4.2 py-pyparsing@2.0.3 py-six@1.9.0 - -The extensions are a subset of what's returned by ``spack list``, and -they are packages like any other. They are installed into their own -prefixes, and you can see this with ``spack find --paths``: - -.. code-block:: console - - $ spack find --paths py-numpy - ==> 1 installed packages. - -- linux-debian7-x86_64 / gcc@4.4.7 -------------------------------- - py-numpy@1.9.1 ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/py-numpy@1.9.1-66733244 - -However, even though this package is installed, you cannot use it -directly when you run ``python``: - -.. code-block:: console - - $ spack load python - $ python - Python 2.7.8 (default, Feb 17 2015, 01:35:25) - [GCC 4.4.7 20120313 (Red Hat 4.4.7-11)] on linux2 - Type "help", "copyright", "credits" or "license" for more information. - >>> import numpy - Traceback (most recent call last): - File "<stdin>", line 1, in <module> - ImportError: No module named numpy - >>> - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Using Extensions in Environments -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The recommended way of working with extensions such as ``py-numpy`` -above is through :ref:`Environments <environments>`. For example, -the following creates an environment in the current working directory -with a filesystem view in the ``./view`` directory: - -.. code-block:: console - - $ spack env create --with-view view --dir . - $ spack -e . add py-numpy - $ spack -e . concretize - $ spack -e . install - -We recommend environments for two reasons. Firstly, environments -can be activated (requires :ref:`shell-support`): - -.. code-block:: console - - $ spack env activate . - -which sets all the right environment variables such as ``PATH`` and -``PYTHONPATH``. This ensures that - -.. code-block:: console - - $ python - >>> import numpy - -works. Secondly, even without shell support, the view ensures -that Python can locate its extensions: - -.. code-block:: console - - $ ./view/bin/python - >>> import numpy - -See :ref:`environments` for a more in-depth description of Spack -environments and customizations to views. - -^^^^^^^^^^^^^^^^^^^^ -Using ``spack load`` -^^^^^^^^^^^^^^^^^^^^ - -A more traditional way of using Spack and extensions is ``spack load`` -(requires :ref:`shell-support`). This will add the extension to ``PYTHONPATH`` -in your current shell, and Python itself will be available in the ``PATH``: - -.. code-block:: console - - $ spack load py-numpy - $ python - >>> import numpy - -The loaded packages can be checked using ``spack find --loaded`` - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Loading Extensions via Modules -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Apart from ``spack env activate`` and ``spack load``, you can load numpy -through your environment modules (using ``environment-modules`` or -``lmod``). This will also add the extension to the ``PYTHONPATH`` in -your current shell. - -.. code-block:: console - - $ module load <name of numpy module> - -If you do not know the name of the specific numpy module you wish to -load, you can use the ``spack module tcl|lmod loads`` command to get -the name of the module from the Spack spec. - ----------------------- Filesystem requirements ----------------------- diff --git a/lib/spack/docs/build_systems/pythonpackage.rst b/lib/spack/docs/build_systems/pythonpackage.rst index 372d4ad47c..9512b08885 100644 --- a/lib/spack/docs/build_systems/pythonpackage.rst +++ b/lib/spack/docs/build_systems/pythonpackage.rst @@ -718,23 +718,45 @@ command-line tool, or C/C++/Fortran program with optional Python modules? The former should be prepended with ``py-``, while the latter should not. -"""""""""""""""""""""" -extends vs. depends_on -"""""""""""""""""""""" +"""""""""""""""""""""""""""""" +``extends`` vs. ``depends_on`` +"""""""""""""""""""""""""""""" -This is very similar to the naming dilemma above, with a slight twist. As mentioned in the :ref:`Packaging Guide <packaging_extensions>`, ``extends`` and ``depends_on`` are very similar, but ``extends`` ensures that the extension and extendee share the same prefix in views. This allows the user to import a Python module without having to add that module to ``PYTHONPATH``. -When deciding between ``extends`` and ``depends_on``, the best rule of -thumb is to check the installation prefix. If Python libraries are -installed to ``<prefix>/lib/pythonX.Y/site-packages``, then you -should use ``extends``. If Python libraries are installed elsewhere -or the only files that get installed reside in ``<prefix>/bin``, then -don't use ``extends``. +Additionally, ``extends("python")`` adds a dependency on the package +``python-venv``. This improves isolation from the system, whether +it's during the build or at runtime: user and system site packages +cannot accidentally be used by any package that ``extends("python")``. + +As a rule of thumb: if a package does not install any Python modules +of its own, and merely puts a Python script in the ``bin`` directory, +then there is no need for ``extends``. If the package installs modules +in the ``site-packages`` directory, it requires ``extends``. + +""""""""""""""""""""""""""""""""""""" +Executing ``python`` during the build +""""""""""""""""""""""""""""""""""""" + +Whenever you need to execute a Python command or pass the path of the +Python interpreter to the build system, it is best to use the global +variable ``python`` directly. For example: + +.. code-block:: python + + @run_before("install") + def recythonize(self): + python("setup.py", "clean") # use the `python` global + +As mentioned in the previous section, ``extends("python")`` adds an +automatic dependency on ``python-venv``, which is a virtual environment +that guarantees build isolation. The ``python`` global always refers to +the correct Python interpreter, whether the package uses ``extends("python")`` +or ``depends_on("python")``. ^^^^^^^^^^^^^^^^^^^^^ Alternatives to Spack diff --git a/lib/spack/spack/bootstrap/_common.py b/lib/spack/spack/bootstrap/_common.py index 2ce53d3165..5c3ca93e94 100644 --- a/lib/spack/spack/bootstrap/_common.py +++ b/lib/spack/spack/bootstrap/_common.py @@ -54,10 +54,14 @@ def _try_import_from_store( installed_specs = spack.store.STORE.db.query(query_spec, installed=True) for candidate_spec in installed_specs: - pkg = candidate_spec["python"].package + # previously bootstrapped specs may not have a python-venv dependency. + if candidate_spec.dependencies("python-venv"): + python, *_ = candidate_spec.dependencies("python-venv") + else: + python, *_ = candidate_spec.dependencies("python") module_paths = [ - os.path.join(candidate_spec.prefix, pkg.purelib), - os.path.join(candidate_spec.prefix, pkg.platlib), + os.path.join(candidate_spec.prefix, python.package.purelib), + os.path.join(candidate_spec.prefix, python.package.platlib), ] path_before = list(sys.path) diff --git a/lib/spack/spack/bootstrap/environment.py b/lib/spack/spack/bootstrap/environment.py index f1af8990e8..13942ba86f 100644 --- a/lib/spack/spack/bootstrap/environment.py +++ b/lib/spack/spack/bootstrap/environment.py @@ -3,13 +3,11 @@ # # SPDX-License-Identifier: (Apache-2.0 OR MIT) """Bootstrap non-core Spack dependencies from an environment.""" -import glob import hashlib import os import pathlib import sys -import warnings -from typing import List +from typing import Iterable, List import archspec.cpu @@ -28,6 +26,16 @@ from .core import _add_externals_if_missing class BootstrapEnvironment(spack.environment.Environment): """Environment to install dependencies of Spack for a given interpreter and architecture""" + def __init__(self) -> None: + if not self.spack_yaml().exists(): + self._write_spack_yaml_file() + super().__init__(self.environment_root()) + + # Remove python package roots created before python-venv was introduced + for s in self.concrete_roots(): + if "python" in s.package.extendees and not s.dependencies("python-venv"): + self.deconcretize(s) + @classmethod def spack_dev_requirements(cls) -> List[str]: """Spack development requirements""" @@ -59,31 +67,19 @@ class BootstrapEnvironment(spack.environment.Environment): return cls.environment_root().joinpath("view") @classmethod - def pythonpaths(cls) -> List[str]: - """Paths to be added to sys.path or PYTHONPATH""" - python_dir_part = f"python{'.'.join(str(x) for x in sys.version_info[:2])}" - glob_expr = str(cls.view_root().joinpath("**", python_dir_part, "**")) - result = glob.glob(glob_expr) - if not result: - msg = f"Cannot find any Python path in {cls.view_root()}" - warnings.warn(msg) - return result - - @classmethod - def bin_dirs(cls) -> List[pathlib.Path]: + def bin_dir(cls) -> pathlib.Path: """Paths to be added to PATH""" - return [cls.view_root().joinpath("bin")] + return cls.view_root().joinpath("bin") + + def python_dirs(self) -> Iterable[pathlib.Path]: + python = next(s for s in self.all_specs_generator() if s.name == "python-venv").package + return {self.view_root().joinpath(p) for p in (python.platlib, python.purelib)} @classmethod def spack_yaml(cls) -> pathlib.Path: """Environment spack.yaml file""" return cls.environment_root().joinpath("spack.yaml") - def __init__(self) -> None: - if not self.spack_yaml().exists(): - self._write_spack_yaml_file() - super().__init__(self.environment_root()) - def update_installations(self) -> None: """Update the installations of this environment.""" log_enabled = tty.is_debug() or tty.is_verbose() @@ -100,21 +96,13 @@ class BootstrapEnvironment(spack.environment.Environment): self.install_all() self.write(regenerate=True) - def update_syspath_and_environ(self) -> None: - """Update ``sys.path`` and the PATH, PYTHONPATH environment variables to point to - the environment view. - """ - # Do minimal modifications to sys.path and environment variables. In particular, pay - # attention to have the smallest PYTHONPATH / sys.path possible, since that may impact - # the performance of the current interpreter - sys.path.extend(self.pythonpaths()) - os.environ["PATH"] = os.pathsep.join( - [str(x) for x in self.bin_dirs()] + os.environ.get("PATH", "").split(os.pathsep) - ) - os.environ["PYTHONPATH"] = os.pathsep.join( - os.environ.get("PYTHONPATH", "").split(os.pathsep) - + [str(x) for x in self.pythonpaths()] - ) + def load(self) -> None: + """Update PATH and sys.path.""" + # Make executables available (shouldn't need PYTHONPATH) + os.environ["PATH"] = f"{self.bin_dir()}{os.pathsep}{os.environ.get('PATH', '')}" + + # Spack itself imports pytest + sys.path.extend(str(p) for p in self.python_dirs()) def _write_spack_yaml_file(self) -> None: tty.msg( @@ -164,4 +152,4 @@ def ensure_environment_dependencies() -> None: _add_externals_if_missing() with BootstrapEnvironment() as env: env.update_installations() - env.update_syspath_and_environ() + env.load() diff --git a/lib/spack/spack/build_systems/cmake.py b/lib/spack/spack/build_systems/cmake.py index b6e66e136c..a64904715e 100644 --- a/lib/spack/spack/build_systems/cmake.py +++ b/lib/spack/spack/build_systems/cmake.py @@ -39,16 +39,11 @@ def _maybe_set_python_hints(pkg: spack.package_base.PackageBase, args: List[str] """Set the PYTHON_EXECUTABLE, Python_EXECUTABLE, and Python3_EXECUTABLE CMake variables if the package has Python as build or link dep and ``find_python_hints`` is set to True. See ``find_python_hints`` for context.""" - if not getattr(pkg, "find_python_hints", False): + if not getattr(pkg, "find_python_hints", False) or not pkg.spec.dependencies( + "python", dt.BUILD | dt.LINK + ): return - pythons = pkg.spec.dependencies("python", dt.BUILD | dt.LINK) - if len(pythons) != 1: - return - try: - python_executable = pythons[0].package.command.path - except RuntimeError: - return - + python_executable = pkg.spec["python"].command.path args.extend( [ CMakeBuilder.define("PYTHON_EXECUTABLE", python_executable), diff --git a/lib/spack/spack/build_systems/python.py b/lib/spack/spack/build_systems/python.py index 1f650be98a..c94e2db700 100644 --- a/lib/spack/spack/build_systems/python.py +++ b/lib/spack/spack/build_systems/python.py @@ -120,6 +120,12 @@ class PythonExtension(spack.package_base.PackageBase): """ return [] + @property + def python_spec(self): + """Get python-venv if it exists or python otherwise.""" + python, *_ = self.spec.dependencies("python-venv") or self.spec.dependencies("python") + return python + def view_file_conflicts(self, view, merge_map): """Report all file conflicts, excepting special cases for python. Specifically, this does not report errors for duplicate @@ -138,16 +144,17 @@ class PythonExtension(spack.package_base.PackageBase): return conflicts def add_files_to_view(self, view, merge_map, skip_if_exists=True): - # Patch up shebangs to the python linked in the view only if python is built by Spack. - if not self.extendee_spec or self.extendee_spec.external: + # Patch up shebangs if the package extends Python and we put a Python interpreter in the + # view. + python = self.python_spec + if not self.extendee_spec or python.external: return super().add_files_to_view(view, merge_map, skip_if_exists) # We only patch shebangs in the bin directory. copied_files: Dict[Tuple[int, int], str] = {} # File identifier -> source delayed_links: List[Tuple[str, str]] = [] # List of symlinks from merge map - bin_dir = self.spec.prefix.bin - python_prefix = self.extendee_spec.prefix + for src, dst in merge_map.items(): if skip_if_exists and os.path.lexists(dst): continue @@ -168,7 +175,7 @@ class PythonExtension(spack.package_base.PackageBase): copied_files[(s.st_dev, s.st_ino)] = dst shutil.copy2(src, dst) fs.filter_file( - python_prefix, os.path.abspath(view.get_projection_for_spec(self.spec)), dst + python.prefix, os.path.abspath(view.get_projection_for_spec(self.spec)), dst ) else: view.link(src, dst) @@ -199,14 +206,13 @@ class PythonExtension(spack.package_base.PackageBase): ignore_namespace = True bin_dir = self.spec.prefix.bin - global_view = self.extendee_spec.prefix == view.get_projection_for_spec(self.spec) to_remove = [] for src, dst in merge_map.items(): if ignore_namespace and namespace_init(dst): continue - if global_view or not fs.path_contains_subdirectory(src, bin_dir): + if not fs.path_contains_subdirectory(src, bin_dir): to_remove.append(dst) else: os.remove(dst) @@ -371,8 +377,9 @@ class PythonPackage(PythonExtension): # Headers should only be in include or platlib, but no harm in checking purelib too include = self.prefix.join(self.spec["python"].package.include).join(name) - platlib = self.prefix.join(self.spec["python"].package.platlib).join(name) - purelib = self.prefix.join(self.spec["python"].package.purelib).join(name) + python = self.python_spec + platlib = self.prefix.join(python.package.platlib).join(name) + purelib = self.prefix.join(python.package.purelib).join(name) headers_list = map(fs.find_all_headers, [include, platlib, purelib]) headers = functools.reduce(operator.add, headers_list) @@ -391,8 +398,9 @@ class PythonPackage(PythonExtension): name = self.spec.name[3:] # Libraries should only be in platlib, but no harm in checking purelib too - platlib = self.prefix.join(self.spec["python"].package.platlib).join(name) - purelib = self.prefix.join(self.spec["python"].package.purelib).join(name) + python = self.python_spec + platlib = self.prefix.join(python.package.platlib).join(name) + purelib = self.prefix.join(python.package.purelib).join(name) find_all_libraries = functools.partial(fs.find_all_libraries, recursive=True) libs_list = map(find_all_libraries, [platlib, purelib]) @@ -504,6 +512,8 @@ class PythonPipBuilder(BaseBuilder): def install(self, pkg: PythonPackage, spec: Spec, prefix: Prefix) -> None: """Install everything from build directory.""" + pip = spec["python"].command + pip.add_default_arg("-m", "pip") args = PythonPipBuilder.std_args(pkg) + [f"--prefix={prefix}"] @@ -519,14 +529,6 @@ class PythonPipBuilder(BaseBuilder): else: args.append(".") - pip = spec["python"].command - # Hide user packages, since we don't have build isolation. This is - # necessary because pip / setuptools may run hooks from arbitrary - # packages during the build. There is no equivalent variable to hide - # system packages, so this is not reliable for external Python. - pip.add_default_env("PYTHONNOUSERSITE", "1") - pip.add_default_arg("-m") - pip.add_default_arg("pip") with fs.working_dir(self.build_directory): pip(*args) diff --git a/lib/spack/spack/directives.py b/lib/spack/spack/directives.py index 4991040142..b69f83a75d 100644 --- a/lib/spack/spack/directives.py +++ b/lib/spack/spack/directives.py @@ -662,6 +662,7 @@ def _execute_redistribute( @directive(("extendees", "dependencies")) def extends(spec, when=None, type=("build", "run"), patches=None): """Same as depends_on, but also adds this package to the extendee list. + In case of Python, also adds a dependency on python-venv. keyword arguments can be passed to extends() so that extension packages can pass parameters to the extendee's extension @@ -677,6 +678,11 @@ def extends(spec, when=None, type=("build", "run"), patches=None): _depends_on(pkg, spec, when=when, type=type, patches=patches) spec_obj = spack.spec.Spec(spec) + # When extending python, also add a dependency on python-venv. This is done so that + # Spack environment views are Python virtual environments. + if spec_obj.name == "python" and not pkg.name == "python-venv": + _depends_on(pkg, "python-venv", when=when, type=("build", "run")) + # TODO: the values of the extendees dictionary are not used. Remove in next refactor. pkg.extendees[spec_obj.name] = (spec_obj, None) diff --git a/lib/spack/spack/hooks/windows_runtime_linkage.py b/lib/spack/spack/hooks/windows_runtime_linkage.py new file mode 100644 index 0000000000..5bb3744910 --- /dev/null +++ b/lib/spack/spack/hooks/windows_runtime_linkage.py @@ -0,0 +1,8 @@ +# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + + +def post_install(spec, explicit=None): + spec.package.windows_establish_runtime_linkage() diff --git a/lib/spack/spack/installer.py b/lib/spack/spack/installer.py index 1f33a7c6b0..289a48568d 100644 --- a/lib/spack/spack/installer.py +++ b/lib/spack/spack/installer.py @@ -1698,10 +1698,6 @@ class PackageInstaller: spack.package_base.PackageBase._verbose = spack.build_environment.start_build_process( pkg, build_process, install_args ) - # Currently this is how RPATH-like behavior is achieved on Windows, after install - # establish runtime linkage via Windows Runtime link object - # Note: this is a no-op on non Windows platforms - pkg.windows_establish_runtime_linkage() # Note: PARENT of the build process adds the new package to # the database, so that we don't need to re-read from file. spack.store.STORE.db.add(pkg.spec, spack.store.STORE.layout, explicit=explicit) diff --git a/lib/spack/spack/spec.py b/lib/spack/spack/spec.py index 1f30f7e923..27c762bd40 100644 --- a/lib/spack/spack/spec.py +++ b/lib/spack/spack/spec.py @@ -1030,16 +1030,13 @@ class _EdgeMap(collections.abc.Mapping): self.edges.clear() -def _command_default_handler(descriptor, spec, cls): +def _command_default_handler(spec: "Spec"): """Default handler when looking for the 'command' attribute. Tries to search for ``spec.name`` in the ``spec.home.bin`` directory. Parameters: - descriptor (ForwardQueryToPackage): descriptor that triggered the call - spec (Spec): spec that is being queried - cls (type(spec)): type of spec, to match the signature of the - descriptor ``__get__`` method + spec: spec that is being queried Returns: Executable: An executable of the command @@ -1052,22 +1049,17 @@ def _command_default_handler(descriptor, spec, cls): if fs.is_exe(path): return spack.util.executable.Executable(path) - else: - msg = "Unable to locate {0} command in {1}" - raise RuntimeError(msg.format(spec.name, home.bin)) + raise RuntimeError(f"Unable to locate {spec.name} command in {home.bin}") -def _headers_default_handler(descriptor, spec, cls): +def _headers_default_handler(spec: "Spec"): """Default handler when looking for the 'headers' attribute. Tries to search for ``*.h`` files recursively starting from ``spec.package.home.include``. Parameters: - descriptor (ForwardQueryToPackage): descriptor that triggered the call - spec (Spec): spec that is being queried - cls (type(spec)): type of spec, to match the signature of the - descriptor ``__get__`` method + spec: spec that is being queried Returns: HeaderList: The headers in ``prefix.include`` @@ -1080,12 +1072,10 @@ def _headers_default_handler(descriptor, spec, cls): if headers: return headers - else: - msg = "Unable to locate {0} headers in {1}" - raise spack.error.NoHeadersError(msg.format(spec.name, home)) + raise spack.error.NoHeadersError(f"Unable to locate {spec.name} headers in {home}") -def _libs_default_handler(descriptor, spec, cls): +def _libs_default_handler(spec: "Spec"): """Default handler when looking for the 'libs' attribute. Tries to search for ``lib{spec.name}`` recursively starting from @@ -1093,10 +1083,7 @@ def _libs_default_handler(descriptor, spec, cls): ``{spec.name}`` instead. Parameters: - descriptor (ForwardQueryToPackage): descriptor that triggered the call - spec (Spec): spec that is being queried - cls (type(spec)): type of spec, to match the signature of the - descriptor ``__get__`` method + spec: spec that is being queried Returns: LibraryList: The libraries found @@ -1135,27 +1122,33 @@ def _libs_default_handler(descriptor, spec, cls): if libs: return libs - msg = "Unable to recursively locate {0} libraries in {1}" - raise spack.error.NoLibrariesError(msg.format(spec.name, home)) + raise spack.error.NoLibrariesError( + f"Unable to recursively locate {spec.name} libraries in {home}" + ) class ForwardQueryToPackage: """Descriptor used to forward queries from Spec to Package""" - def __init__(self, attribute_name, default_handler=None): + def __init__( + self, + attribute_name: str, + default_handler: Optional[Callable[["Spec"], Any]] = None, + _indirect: bool = False, + ) -> None: """Create a new descriptor. Parameters: - attribute_name (str): name of the attribute to be - searched for in the Package instance - default_handler (callable, optional): default function to be - called if the attribute was not found in the Package - instance + attribute_name: name of the attribute to be searched for in the Package instance + default_handler: default function to be called if the attribute was not found in the + Package instance + _indirect: temporarily added to redirect a query to another package. """ self.attribute_name = attribute_name self.default = default_handler + self.indirect = _indirect - def __get__(self, instance, cls): + def __get__(self, instance: "SpecBuildInterface", cls): """Retrieves the property from Package using a well defined chain of responsibility. @@ -1177,13 +1170,18 @@ class ForwardQueryToPackage: indicating a query failure, e.g. that library files were not found in a 'libs' query. """ - pkg = instance.package + # TODO: this indirection exist solely for `spec["python"].command` to actually return + # spec["python-venv"].command. It should be removed when `python` is a virtual. + if self.indirect and instance.indirect_spec: + pkg = instance.indirect_spec.package + else: + pkg = instance.wrapped_obj.package try: query = instance.last_query except AttributeError: # There has been no query yet: this means # a spec is trying to access its own attributes - _ = instance[instance.name] # NOQA: ignore=F841 + _ = instance.wrapped_obj[instance.wrapped_obj.name] # NOQA: ignore=F841 query = instance.last_query callbacks_chain = [] @@ -1195,7 +1193,8 @@ class ForwardQueryToPackage: callbacks_chain.append(lambda: getattr(pkg, self.attribute_name)) # Final resort : default callback if self.default is not None: - callbacks_chain.append(lambda: self.default(self, instance, cls)) + _default = self.default # make mypy happy + callbacks_chain.append(lambda: _default(instance.wrapped_obj)) # Trigger the callbacks in order, the first one producing a # value wins @@ -1254,25 +1253,33 @@ QueryState = collections.namedtuple("QueryState", ["name", "extra_parameters", " class SpecBuildInterface(lang.ObjectWrapper): # home is available in the base Package so no default is needed home = ForwardQueryToPackage("home", default_handler=None) - - command = ForwardQueryToPackage("command", default_handler=_command_default_handler) - headers = ForwardQueryToPackage("headers", default_handler=_headers_default_handler) - libs = ForwardQueryToPackage("libs", default_handler=_libs_default_handler) + command = ForwardQueryToPackage( + "command", default_handler=_command_default_handler, _indirect=True + ) - def __init__(self, spec, name, query_parameters): + def __init__(self, spec: "Spec", name: str, query_parameters: List[str], _parent: "Spec"): super().__init__(spec) # Adding new attributes goes after super() call since the ObjectWrapper # resets __dict__ to behave like the passed object original_spec = getattr(spec, "wrapped_obj", spec) self.wrapped_obj = original_spec - self.token = original_spec, name, query_parameters + self.token = original_spec, name, query_parameters, _parent is_virtual = spack.repo.PATH.is_virtual(name) self.last_query = QueryState( name=name, extra_parameters=query_parameters, isvirtual=is_virtual ) + # TODO: this ad-hoc logic makes `spec["python"].command` return + # `spec["python-venv"].command` and should be removed when `python` is a virtual. + self.indirect_spec = None + if spec.name == "python": + python_venvs = _parent.dependencies("python-venv") + if not python_venvs: + return + self.indirect_spec = python_venvs[0] + def __reduce__(self): return SpecBuildInterface, self.token @@ -4137,7 +4144,7 @@ class Spec: raise spack.error.SpecError("Spec version is not concrete: " + str(self)) return self.versions[0] - def __getitem__(self, name): + def __getitem__(self, name: str): """Get a dependency from the spec by its name. This call implicitly sets a query state in the package being retrieved. The behavior of packages may be influenced by additional query parameters that are @@ -4146,7 +4153,7 @@ class Spec: Note that if a virtual package is queried a copy of the Spec is returned while for non-virtual a reference is returned. """ - query_parameters = name.split(":") + query_parameters: List[str] = name.split(":") if len(query_parameters) > 2: raise KeyError("key has more than one ':' symbol. At most one is admitted.") @@ -4169,7 +4176,7 @@ class Spec: ) try: - value = next( + child: Spec = next( itertools.chain( # Regular specs (x for x in order() if x.name == name), @@ -4186,9 +4193,9 @@ class Spec: raise KeyError(f"No spec with name {name} in {self}") if self._concrete: - return SpecBuildInterface(value, name, query_parameters) + return SpecBuildInterface(child, name, query_parameters, _parent=self) - return value + return child def __contains__(self, spec): """True if this spec or some dependency satisfies the spec. diff --git a/lib/spack/spack/test/cmd/extensions.py b/lib/spack/spack/test/cmd/extensions.py index 1f6ed95b56..5869e46642 100644 --- a/lib/spack/spack/test/cmd/extensions.py +++ b/lib/spack/spack/test/cmd/extensions.py @@ -33,21 +33,23 @@ def test_extensions(mock_packages, python_database, config, capsys): packages = extensions("-s", "packages", "python") installed = extensions("-s", "installed", "python") assert "==> python@2.7.11" in output - assert "==> 2 extensions" in output + assert "==> 3 extensions" in output assert "py-extension1" in output assert "py-extension2" in output + assert "python-venv" in output - assert "==> 2 extensions" in packages + assert "==> 3 extensions" in packages assert "py-extension1" in packages assert "py-extension2" in packages + assert "python-venv" in packages assert "installed" not in packages - assert ("%s installed" % (ni if ni else "None")) in output - assert ("%s installed" % (ni if ni else "None")) in installed + assert f"{ni if ni else 'None'} installed" in output + assert f"{ni if ni else 'None'} installed" in installed - check_output(2) + check_output(3) ext2.package.do_uninstall(force=True) - check_output(1) + check_output(2) def test_extensions_no_arguments(mock_packages): diff --git a/lib/spack/spack/util/executable.py b/lib/spack/spack/util/executable.py index f160051674..afb8bcaa39 100644 --- a/lib/spack/spack/util/executable.py +++ b/lib/spack/spack/util/executable.py @@ -39,6 +39,20 @@ class Executable: """Add default argument(s) to the command.""" self.exe.extend(args) + def with_default_args(self, *args): + """Same as add_default_arg, but returns a copy of the executable.""" + new = self.copy() + new.add_default_arg(*args) + return new + + def copy(self): + """Return a copy of this Executable.""" + new = Executable(self.exe[0]) + new.exe[:] = self.exe + new.default_env.update(self.default_env) + new.default_envmod.extend(self.default_envmod) + return new + def add_default_env(self, key, value): """Set an environment variable when the command is run. |