summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
Diffstat
-rw-r--r--lib/spack/docs/environments.rst126
-rw-r--r--lib/spack/spack/cmd/env.py76
-rw-r--r--lib/spack/spack/cmd/find.py22
-rw-r--r--lib/spack/spack/environment/__init__.py126
-rw-r--r--lib/spack/spack/environment/environment.py366
-rw-r--r--lib/spack/spack/schema/env.py1
-rw-r--r--lib/spack/spack/test/cmd/env.py348
-rw-r--r--lib/spack/spack/test/cmd/find.py81
-rwxr-xr-xshare/spack/spack-completion.bash6
-rwxr-xr-xshare/spack/spack-completion.fish12
10 files changed, 1116 insertions, 48 deletions
diff --git a/lib/spack/docs/environments.rst b/lib/spack/docs/environments.rst
index 78a903a77e..193258e047 100644
--- a/lib/spack/docs/environments.rst
+++ b/lib/spack/docs/environments.rst
@@ -460,6 +460,125 @@ Sourcing that file in Bash will make the environment available to the
user; and can be included in ``.bashrc`` files, etc. The ``loads``
file may also be copied out of the environment, renamed, etc.
+
+.. _environment_include_concrete:
+
+------------------------------
+Included Concrete Environments
+------------------------------
+
+Spack environments can create an environment based off of information in already
+established environments. You can think of it as a combination of existing
+environments. It will gather information from the existing environment's
+``spack.lock`` and use that during the creation of this included concrete
+environment. When an included concrete environment is created it will generate
+a ``spack.lock`` file for the newly created environment.
+
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Creating included environments
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+To create a combined concrete environment, you must have at least one existing
+concrete environment. You will use the command ``spack env create`` with the
+argument ``--include-concrete`` followed by the name or path of the environment
+you'd like to include. Here is an example of how to create a combined environment
+from the command line.
+
+.. code-block:: console
+
+ $ spack env create myenv
+ $ spack -e myenv add python
+ $ spack -e myenv concretize
+ $ spack env create --include-concrete myenv included_env
+
+
+You can also include an environment directly in the ``spack.yaml`` file. It
+involves adding the ``include_concrete`` heading in the yaml followed by the
+absolute path to the independent environments.
+
+.. code-block:: yaml
+
+ spack:
+ specs: []
+ concretizer:
+ unify: true
+ include_concrete:
+ - /absolute/path/to/environment1
+ - /absolute/path/to/environment2
+
+
+Once the ``spack.yaml`` has been updated you must concretize the environment to
+get the concrete specs from the included environments.
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Updating an included environment
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+If changes were made to the base environment and you want that reflected in the
+included environment you will need to reconcretize both the base environment and the
+included environment for the change to be implemented. For example:
+
+.. code-block:: console
+
+ $ spack env create myenv
+ $ spack -e myenv add python
+ $ spack -e myenv concretize
+ $ spack env create --include-concrete myenv included_env
+
+
+ $ spack -e myenv find
+ ==> In environment myenv
+ ==> Root specs
+ python
+
+ ==> 0 installed packages
+
+
+ $ spack -e included_env find
+ ==> In environment included_env
+ ==> No root specs
+ ==> Included specs
+ python
+
+ ==> 0 installed packages
+
+Here we see that ``included_env`` has access to the python package through
+the ``myenv`` environment. But if we were to add another spec to ``myenv``,
+``included_env`` will not be able to access the new information.
+
+.. code-block:: console
+
+ $ spack -e myenv add perl
+ $ spack -e myenv concretize
+ $ spack -e myenv find
+ ==> In environment myenv
+ ==> Root specs
+ perl python
+
+ ==> 0 installed packages
+
+
+ $ spack -e included_env find
+ ==> In environment included_env
+ ==> No root specs
+ ==> Included specs
+ python
+
+ ==> 0 installed packages
+
+It isn't until you run the ``spack concretize`` command that the combined
+environment will get the updated information from the reconcretized base environmennt.
+
+.. code-block:: console
+
+ $ spack -e included_env concretize
+ $ spack -e included_env find
+ ==> In environment included_env
+ ==> No root specs
+ ==> Included specs
+ perl python
+
+ ==> 0 installed packages
+
.. _environment-configuration:
------------------------
@@ -811,6 +930,7 @@ For example, the following environment has three root packages:
This allows for a much-needed reduction in redundancy between packages
and constraints.
+
----------------
Filesystem Views
----------------
@@ -1044,7 +1164,7 @@ other targets to depend on the environment installation.
A typical workflow is as follows:
-.. code:: console
+.. code-block:: console
spack env create -d .
spack -e . add perl
@@ -1137,7 +1257,7 @@ its dependencies. This can be useful when certain flags should only apply to
dependencies. Below we show a use case where a spec is installed with verbose
output (``spack install --verbose``) while its dependencies are installed silently:
-.. code:: console
+.. code-block:: console
$ spack env depfile -o Makefile
@@ -1159,7 +1279,7 @@ This can be accomplished through the generated ``[<prefix>/]SPACK_PACKAGE_IDS``
variable. Assuming we have an active and concrete environment, we generate the
associated ``Makefile`` with a prefix ``example``:
-.. code:: console
+.. code-block:: console
$ spack env depfile -o env.mk --make-prefix example
diff --git a/lib/spack/spack/cmd/env.py b/lib/spack/spack/cmd/env.py
index 13f67e74e6..2ccb88fd1a 100644
--- a/lib/spack/spack/cmd/env.py
+++ b/lib/spack/spack/cmd/env.py
@@ -10,7 +10,7 @@ import shutil
import sys
import tempfile
from pathlib import Path
-from typing import Optional
+from typing import List, Optional
import llnl.string as string
import llnl.util.filesystem as fs
@@ -87,6 +87,9 @@ def env_create_setup_parser(subparser):
default=None,
help="either a lockfile (must end with '.json' or '.lock') or a manifest file",
)
+ subparser.add_argument(
+ "--include-concrete", action="append", help="name of old environment to copy specs from"
+ )
def env_create(args):
@@ -104,12 +107,17 @@ def env_create(args):
# the environment should not include a view.
with_view = None
+ include_concrete = None
+ if hasattr(args, "include_concrete"):
+ include_concrete = args.include_concrete
+
env = _env_create(
args.env_name,
init_file=args.envfile,
dir=args.dir or os.path.sep in args.env_name or args.env_name in (".", ".."),
with_view=with_view,
keep_relative=args.keep_relative,
+ include_concrete=include_concrete,
)
# Generate views, only really useful for environments created from spack.lock files.
@@ -123,31 +131,43 @@ def _env_create(
dir: bool = False,
with_view: Optional[str] = None,
keep_relative: bool = False,
+ include_concrete: Optional[List[str]] = None,
):
"""Create a new environment, with an optional yaml description.
Arguments:
- name_or_path: name of the environment to create, or path to it
- init_file: optional initialization file -- can be a JSON lockfile (*.lock, *.json) or YAML
- manifest file
- dir: if True, create an environment in a directory instead of a managed environment
- keep_relative: if True, develop paths are copied verbatim into the new environment file,
- otherwise they may be made absolute if the new environment is in a different location
+ name_or_path (str): name of the environment to create, or path to it
+ init_file (str or file): optional initialization file -- can be
+ a JSON lockfile (*.lock, *.json) or YAML manifest file
+ dir (bool): if True, create an environment in a directory instead
+ of a named environment
+ keep_relative (bool): if True, develop paths are copied verbatim into
+ the new environment file, otherwise they may be made absolute if the
+ new environment is in a different location
+ include_concrete (list): list of the included concrete environments
"""
if not dir:
env = ev.create(
- name_or_path, init_file=init_file, with_view=with_view, keep_relative=keep_relative
+ name_or_path,
+ init_file=init_file,
+ with_view=with_view,
+ keep_relative=keep_relative,
+ include_concrete=include_concrete,
)
tty.msg(
colorize(
- f"Created environment @c{{{cescape(env.name)}}} in: @c{{{cescape(env.path)}}}"
+ f"Created environment @c{{{cescape(name_or_path)}}} in: @c{{{cescape(env.path)}}}"
)
)
else:
env = ev.create_in_dir(
- name_or_path, init_file=init_file, with_view=with_view, keep_relative=keep_relative
+ name_or_path,
+ init_file=init_file,
+ with_view=with_view,
+ keep_relative=keep_relative,
+ include_concrete=include_concrete,
)
- tty.msg(colorize(f"Created anonymous environment in: @c{{{cescape(env.path)}}}"))
+ tty.msg(colorize(f"Created independent environment in: @c{{{cescape(env.path)}}}"))
tty.msg(f"Activate with: {colorize(f'@c{{spack env activate {cescape(name_or_path)}}}')}")
return env
@@ -434,6 +454,12 @@ def env_remove_setup_parser(subparser):
"""remove an existing environment"""
subparser.add_argument("rm_env", metavar="env", nargs="+", help="environment(s) to remove")
arguments.add_common_arguments(subparser, ["yes_to_all"])
+ subparser.add_argument(
+ "-f",
+ "--force",
+ action="store_true",
+ help="remove the environment even if it is included in another environment",
+ )
def env_remove(args):
@@ -443,13 +469,35 @@ def env_remove(args):
and manifests embedded in repositories should be removed manually.
"""
read_envs = []
+ valid_envs = []
bad_envs = []
- for env_name in args.rm_env:
+ invalid_envs = []
+
+ for env_name in ev.all_environment_names():
try:
env = ev.read(env_name)
- read_envs.append(env)
+ valid_envs.append(env_name)
+
+ if env_name in args.rm_env:
+ read_envs.append(env)
except (spack.config.ConfigFormatError, ev.SpackEnvironmentConfigError):
- bad_envs.append(env_name)
+ invalid_envs.append(env_name)
+
+ if env_name in args.rm_env:
+ bad_envs.append(env_name)
+
+ # Check if env is linked to another before trying to remove
+ for name in valid_envs:
+ # don't check if environment is included to itself
+ if name == env_name:
+ continue
+ environ = ev.Environment(ev.root(name))
+ if ev.root(env_name) in environ.included_concrete_envs:
+ msg = f'Environment "{env_name}" is being used by environment "{name}"'
+ if args.force:
+ tty.warn(msg)
+ else:
+ tty.die(msg)
if not args.yes_to_all:
environments = string.plural(len(args.rm_env), "environment", show_n=False)
diff --git a/lib/spack/spack/cmd/find.py b/lib/spack/spack/cmd/find.py
index d1917a73b5..c4e2c77552 100644
--- a/lib/spack/spack/cmd/find.py
+++ b/lib/spack/spack/cmd/find.py
@@ -3,6 +3,7 @@
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+import copy
import sys
import llnl.util.lang
@@ -271,6 +272,27 @@ def display_env(env, args, decorator, results):
print()
+ if env.included_concrete_envs:
+ tty.msg("Included specs")
+
+ # Root specs cannot be displayed with prefixes, since those are not
+ # set for abstract specs. Same for hashes
+ root_args = copy.copy(args)
+ root_args.paths = False
+
+ # Roots are displayed with variants, etc. so that we can see
+ # specifically what the user asked for.
+ cmd.display_specs(
+ env.included_user_specs,
+ root_args,
+ decorator=lambda s, f: color.colorize("@*{%s}" % f),
+ namespace=True,
+ show_flags=True,
+ show_full_compiler=True,
+ variants=True,
+ )
+ print()
+
if args.show_concretized:
tty.msg("Concretized roots")
cmd.display_specs(env.specs_by_hash.values(), args, decorator=decorator)
diff --git a/lib/spack/spack/environment/__init__.py b/lib/spack/spack/environment/__init__.py
index e6521aed87..fb083594e0 100644
--- a/lib/spack/spack/environment/__init__.py
+++ b/lib/spack/spack/environment/__init__.py
@@ -34,6 +34,9 @@ contents have. Lockfiles are JSON-formatted and their top-level sections are:
* ``spec``: a string representation of the abstract spec that was concretized
4. ``concrete_specs``: a dictionary containing the specs in the environment.
+ 5. ``include_concrete`` (dictionary): an optional dictionary that includes the roots
+ and concrete specs from the included environments, keyed by the path to that
+ environment
Compatibility
-------------
@@ -50,26 +53,37 @@ upgrade Spack to use them.
- ``v2``
- ``v3``
- ``v4``
+ - ``v5``
* - ``v0.12:0.14``
- ✅
-
-
-
+ -
* - ``v0.15:0.16``
- ✅
- ✅
-
-
+ -
* - ``v0.17``
- ✅
- ✅
- ✅
-
+ -
* - ``v0.18:``
- ✅
- ✅
- ✅
- ✅
+ -
+ * - ``v0.22:``
+ - ✅
+ - ✅
+ - ✅
+ - ✅
+ - ✅
Version 1
---------
@@ -334,6 +348,118 @@ the commit or version.
}
}
}
+
+
+Version 5
+---------
+
+Version 5 doesn't change the top-level lockfile format, but an optional dictionary is
+added. The dictionary has the ``root`` and ``concrete_specs`` of the included
+environments, which are keyed by the path to that environment. Since this is optional
+if the environment does not have any included environments ``include_concrete`` will
+not be a part of the lockfile.
+
+.. code-block:: json
+
+ {
+ "_meta": {
+ "file-type": "spack-lockfile",
+ "lockfile-version": 5,
+ "specfile-version": 3
+ },
+ "roots": [
+ {
+ "hash": "<dag_hash 1>",
+ "spec": "<abstract spec 1>"
+ },
+ {
+ "hash": "<dag_hash 2>",
+ "spec": "<abstract spec 2>"
+ }
+ ],
+ "concrete_specs": {
+ "<dag_hash 1>": {
+ "... <spec dict attributes> ...": { },
+ "dependencies": [
+ {
+ "name": "depname_1",
+ "hash": "<dag_hash for depname_1>",
+ "type": ["build", "link"]
+ },
+ {
+ "name": "depname_2",
+ "hash": "<dag_hash for depname_2>",
+ "type": ["build", "link"]
+ }
+ ],
+ "hash": "<dag_hash 1>",
+ },
+ "<daghash 2>": {
+ "... <spec dict attributes> ...": { },
+ "dependencies": [
+ {
+ "name": "depname_3",
+ "hash": "<dag_hash for depname_3>",
+ "type": ["build", "link"]
+ },
+ {
+ "name": "depname_4",
+ "hash": "<dag_hash for depname_4>",
+ "type": ["build", "link"]
+ }
+ ],
+ "hash": "<dag_hash 2>"
+ }
+ }
+ "include_concrete": {
+ "<path to environment>": {
+ "roots": [
+ {
+ "hash": "<dag_hash 1>",
+ "spec": "<abstract spec 1>"
+ },
+ {
+ "hash": "<dag_hash 2>",
+ "spec": "<abstract spec 2>"
+ }
+ ],
+ "concrete_specs": {
+ "<dag_hash 1>": {
+ "... <spec dict attributes> ...": { },
+ "dependencies": [
+ {
+ "name": "depname_1",
+ "hash": "<dag_hash for depname_1>",
+ "type": ["build", "link"]
+ },
+ {
+ "name": "depname_2",
+ "hash": "<dag_hash for depname_2>",
+ "type": ["build", "link"]
+ }
+ ],
+ "hash": "<dag_hash 1>",
+ },
+ "<daghash 2>": {
+ "... <spec dict attributes> ...": { },
+ "dependencies": [
+ {
+ "name": "depname_3",
+ "hash": "<dag_hash for depname_3>",
+ "type": ["build", "link"]
+ },
+ {
+ "name": "depname_4",
+ "hash": "<dag_hash for depname_4>",
+ "type": ["build", "link"]
+ }
+ ],
+ "hash": "<dag_hash 2>"
+ }
+ }
+ }
+ }
+ }
"""
from .environment import (
diff --git a/lib/spack/spack/environment/environment.py b/lib/spack/spack/environment/environment.py
index f14919adb6..562413c234 100644
--- a/lib/spack/spack/environment/environment.py
+++ b/lib/spack/spack/environment/environment.py
@@ -16,7 +16,7 @@ import time
import urllib.parse
import urllib.request
import warnings
-from typing import Dict, Iterable, List, Optional, Set, Tuple, Union
+from typing import Any, Dict, Iterable, List, Optional, Set, Tuple, Union
import llnl.util.filesystem as fs
import llnl.util.tty as tty
@@ -159,6 +159,8 @@ user_speclist_name = "specs"
default_view_name = "default"
# Default behavior to link all packages into views (vs. only root packages)
default_view_link = "all"
+# The name for any included concrete specs
+included_concrete_name = "include_concrete"
def installed_specs():
@@ -293,6 +295,7 @@ def create(
init_file: Optional[Union[str, pathlib.Path]] = None,
with_view: Optional[Union[str, pathlib.Path, bool]] = None,
keep_relative: bool = False,
+ include_concrete: Optional[List[str]] = None,
) -> "Environment":
"""Create a managed environment in Spack and returns it.
@@ -309,10 +312,15 @@ def create(
string, it specifies the path to the view
keep_relative: if True, develop paths are copied verbatim into the new environment file,
otherwise they are made absolute
+ include_concrete: list of concrete environment names/paths to be included
"""
environment_dir = environment_dir_from_name(name, exists_ok=False)
return create_in_dir(
- environment_dir, init_file=init_file, with_view=with_view, keep_relative=keep_relative
+ environment_dir,
+ init_file=init_file,
+ with_view=with_view,
+ keep_relative=keep_relative,
+ include_concrete=include_concrete,
)
@@ -321,6 +329,7 @@ def create_in_dir(
init_file: Optional[Union[str, pathlib.Path]] = None,
with_view: Optional[Union[str, pathlib.Path, bool]] = None,
keep_relative: bool = False,
+ include_concrete: Optional[List[str]] = None,
) -> "Environment":
"""Create an environment in the directory passed as input and returns it.
@@ -334,6 +343,7 @@ def create_in_dir(
string, it specifies the path to the view
keep_relative: if True, develop paths are copied verbatim into the new environment file,
otherwise they are made absolute
+ include_concrete: concrete environment names/paths to be included
"""
initialize_environment_dir(root, envfile=init_file)
@@ -346,6 +356,12 @@ def create_in_dir(
if with_view is not None:
manifest.set_default_view(with_view)
+ if include_concrete is not None:
+ set_included_envs_to_env_paths(include_concrete)
+ validate_included_envs_exists(include_concrete)
+ validate_included_envs_concrete(include_concrete)
+ manifest.set_include_concrete(include_concrete)
+
manifest.flush()
except (spack.config.ConfigFormatError, SpackEnvironmentConfigError) as e:
@@ -367,6 +383,14 @@ def create_in_dir(
return env
+ # Must be done after environment is initialized
+ if include_concrete:
+ manifest.set_include_concrete(include_concrete)
+
+ if not keep_relative and init_file is not None and str(init_file).endswith(manifest_name):
+ init_file = pathlib.Path(init_file)
+ manifest.absolutify_dev_paths(init_file.parent)
+
def _rewrite_relative_dev_paths_on_relocation(env, init_file_dir):
"""When initializing the environment from a manifest file and we plan
@@ -419,6 +443,67 @@ def ensure_env_root_path_exists():
fs.mkdirp(env_root_path())
+def set_included_envs_to_env_paths(include_concrete: List[str]) -> None:
+ """If the included environment(s) is the environment name
+ it is replaced by the path to the environment
+
+ Args:
+ include_concrete: list of env name or path to env"""
+
+ for i, env_name in enumerate(include_concrete):
+ if is_env_dir(env_name):
+ include_concrete[i] = env_name
+ elif exists(env_name):
+ include_concrete[i] = root(env_name)
+
+
+def validate_included_envs_exists(include_concrete: List[str]) -> None:
+ """Checks that all of the included environments exist
+
+ Args:
+ include_concrete: list of already existing concrete environments to include
+
+ Raises:
+ SpackEnvironmentError: if any of the included environments do not exist
+ """
+
+ missing_envs = set()
+
+ for i, env_name in enumerate(include_concrete):
+ if not is_env_dir(env_name):
+ missing_envs.add(env_name)
+
+ if missing_envs:
+ msg = "The following environment(s) are missing: {0}".format(", ".join(missing_envs))
+ raise SpackEnvironmentError(msg)
+
+
+def validate_included_envs_concrete(include_concrete: List[str]) -> None:
+ """Checks that all of the included environments are concrete
+
+ Args:
+ include_concrete: list of already existing concrete environments to include
+
+ Raises:
+ SpackEnvironmentError: if any of the included environments are not concrete
+ """
+
+ non_concrete_envs = set()
+
+ for env_path in include_concrete:
+ if not os.path.exists(Environment(env_path).lock_path):
+ non_concrete_envs.add(Environment(env_path).name)
+
+ if non_concrete_envs:
+ msg = "The following environment(s) are not concrete: {0}\n" "Please run:".format(
+ ", ".join(non_concrete_envs)
+ )
+ for env in non_concrete_envs:
+ msg += f"\n\t`spack -e {env} concretize`"
+
+ raise SpackEnvironmentError(msg)
+
+
def all_environment_names():
"""List the names of environments that currently exist."""
# just return empty if the env path does not exist. A read-only
@@ -821,6 +906,18 @@ class Environment:
self.specs_by_hash: Dict[str, Spec] = {}
#: Repository for this environment (memoized)
self._repo = None
+
+ #: Environment paths for concrete (lockfile) included environments
+ self.included_concrete_envs: List[str] = []
+ #: First-level included concretized spec data from/to the lockfile.
+ self.included_concrete_spec_data: Dict[str, Dict[str, List[str]]] = {}
+ #: User specs from included environments from the last concretization
+ self.included_concretized_user_specs: Dict[str, List[Spec]] = {}
+ #: Roots from included environments with the last concretization, in order
+ self.included_concretized_order: Dict[str, List[str]] = {}
+ #: Concretized specs by hash from the included environments
+ self.included_specs_by_hash: Dict[str, Dict[str, Spec]] = {}
+
#: Previously active environment
self._previous_active = None
self._dev_specs = None
@@ -858,7 +955,7 @@ class Environment:
if os.path.exists(self.lock_path):
with open(self.lock_path) as f:
- read_lock_version = self._read_lockfile(f)
+ read_lock_version = self._read_lockfile(f)["_meta"]["lockfile-version"]
if read_lock_version == 1:
tty.debug(f"Storing backup of {self.lock_path} at {self._lock_backup_v1_path}")
@@ -926,6 +1023,20 @@ class Environment:
if self.views == dict():
self.views[default_view_name] = ViewDescriptor(self.path, self.view_path_default)
+ def _process_concrete_includes(self):
+ """Extract and load into memory included concrete spec data."""
+ self.included_concrete_envs = self.manifest[TOP_LEVEL_KEY].get(included_concrete_name, [])
+
+ if self.included_concrete_envs:
+ if os.path.exists(self.lock_path):
+ with open(self.lock_path) as f:
+ data = self._read_lockfile(f)
+
+ if included_concrete_name in data:
+ self.included_concrete_spec_data = data[included_concrete_name]
+ else:
+ self.include_concrete_envs()
+
def _construct_state_from_manifest(self):
"""Set up user specs and views from the manifest file."""
self.spec_lists = collections.OrderedDict()
@@ -942,6 +1053,31 @@ class Environment:
self.spec_lists[user_speclist_name] = user_specs
self._process_view(spack.config.get("view", True))
+ self._process_concrete_includes()
+
+ def all_concretized_user_specs(self) -> List[Spec]:
+ """Returns all of the concretized user specs of the environment and
+ its included environment(s)."""
+ concretized_user_specs = self.concretized_user_specs[:]
+ for included_specs in self.included_concretized_user_specs.values():
+ for included in included_specs:
+ # Don't duplicate included spec(s)
+ if included not in concretized_user_specs:
+ concretized_user_specs.append(included)
+
+ return concretized_user_specs
+
+ def all_concretized_orders(self) -> List[str]:
+ """Returns all of the concretized order of the environment and
+ its included environment(s)."""
+ concretized_order = self.concretized_order[:]
+ for included_concretized_order in self.included_concretized_order.values():
+ for included in included_concretized_order:
+ # Don't duplicate included spec(s)
+ if included not in concretized_order:
+ concretized_order.append(included)
+
+ return concretized_order
@property
def user_specs(self):
@@ -966,6 +1102,26 @@ class Environment:
dev_specs[name] = local_entry
return dev_specs
+ @property
+ def included_user_specs(self) -> SpecList:
+ """Included concrete user (or root) specs from last concretization."""
+ spec_list = SpecList()
+
+ if not self.included_concrete_envs:
+ return spec_list
+
+ def add_root_specs(included_concrete_specs):
+ # add specs from the include *and* any nested includes it may have
+ for env, info in included_concrete_specs.items():
+ for root_list in info["roots"]:
+ spec_list.add(root_list["spec"])
+
+ if "include_concrete" in info:
+ add_root_specs(info["include_concrete"])
+
+ add_root_specs(self.included_concrete_spec_data)
+ return spec_list
+
def clear(self, re_read=False):
"""Clear the contents of the environment
@@ -977,9 +1133,15 @@ class Environment:
self.spec_lists[user_speclist_name] = SpecList()
self._dev_specs = {}
- self.concretized_user_specs = [] # user specs from last concretize
self.concretized_order = [] # roots of last concretize, in order
+ self.concretized_user_specs = [] # user specs from last concretize
self.specs_by_hash = {} # concretized specs by hash
+
+ self.included_concrete_spec_data = {} # concretized specs from lockfile of included envs
+ self.included_concretized_order = {} # root specs of the included envs, keyed by env path
+ self.included_concretized_user_specs = {} # user specs from last concretize's included env
+ self.included_specs_by_hash = {} # concretized specs by hash from the included envs
+
self.invalidate_repository_cache()
self._previous_active = None # previously active environment
if not re_read:
@@ -1033,6 +1195,55 @@ class Environment:
"""Name of the config scope of this environment's manifest file."""
return self.manifest.scope_name
+ def include_concrete_envs(self):
+ """Copy and save the included envs' specs internally"""
+
+ lockfile_meta = None
+ root_hash_seen = set()
+ concrete_hash_seen = set()
+ self.included_concrete_spec_data = {}
+
+ for env_path in self.included_concrete_envs:
+ # Check that environment exists
+ if not is_env_dir(env_path):
+ raise SpackEnvironmentError(f"Unable to find env at {env_path}")
+
+ env = Environment(env_path)
+
+ with open(env.lock_path) as f:
+ lockfile_as_dict = env._read_lockfile(f)
+
+ # Lockfile_meta must match each env and use at least format version 5
+ if lockfile_meta is None:
+ lockfile_meta = lockfile_as_dict["_meta"]
+ elif lockfile_meta != lockfile_as_dict["_meta"]:
+ raise SpackEnvironmentError("All lockfile _meta values must match")
+ elif lockfile_meta["lockfile-version"] < 5:
+ raise SpackEnvironmentError("The lockfile format must be at version 5 or higher")
+
+ # Copy unique root specs from env
+ self.included_concrete_spec_data[env_path] = {"roots": []}
+ for root_dict in lockfile_as_dict["roots"]:
+ if root_dict["hash"] not in root_hash_seen:
+ self.included_concrete_spec_data[env_path]["roots"].append(root_dict)
+ root_hash_seen.add(root_dict["hash"])
+
+ # Copy unique concrete specs from env
+ for concrete_spec in lockfile_as_dict["concrete_specs"]:
+ if concrete_spec not in concrete_hash_seen:
+ self.included_concrete_spec_data[env_path].update(
+ {"concrete_specs": lockfile_as_dict["concrete_specs"]}
+ )
+ concrete_hash_seen.add(concrete_spec)
+
+ if "include_concrete" in lockfile_as_dict.keys():
+ self.included_concrete_spec_data[env_path]["include_concrete"] = lockfile_as_dict[
+ "include_concrete"
+ ]
+
+ self._read_lockfile_dict(self._to_lockfile_dict())
+ self.write()
+
def destroy(self):
"""Remove this environment from Spack entirely."""
shutil.rmtree(self.path)
@@ -1232,6 +1443,10 @@ class Environment:
for spec in set(self.concretized_user_specs) - set(self.user_specs):
self.deconcretize(spec, concrete=False)
+ # If a combined env, check updated spec is in the linked envs
+ if self.included_concrete_envs:
+ self.include_concrete_envs()
+
# Pick the right concretization strategy
if self.unify == "when_possible":
return self._concretize_together_where_possible(tests=tests)
@@ -1704,8 +1919,14 @@ class Environment:
of per spec."""
installed, uninstalled = [], []
with spack.store.STORE.db.read_transaction():
- for concretized_hash in self.concretized_order:
- spec = self.specs_by_hash[concretized_hash]
+ for concretized_hash in self.all_concretized_orders():
+ if concretized_hash in self.specs_by_hash:
+ spec = self.specs_by_hash[concretized_hash]
+ else:
+ for env_path in self.included_specs_by_hash.keys():
+ if concretized_hash in self.included_specs_by_hash[env_path]:
+ spec = self.included_specs_by_hash[env_path][concretized_hash]
+ break
if not spec.installed or (
spec.satisfies("dev_path=*") or spec.satisfies("^dev_path=*")
):
@@ -1785,8 +2006,14 @@ class Environment:
def concretized_specs(self):
"""Tuples of (user spec, concrete spec) for all concrete specs."""
- for s, h in zip(self.concretized_user_specs, self.concretized_order):
- yield (s, self.specs_by_hash[h])
+ for s, h in zip(self.all_concretized_user_specs(), self.all_concretized_orders()):
+ if h in self.specs_by_hash:
+ yield (s, self.specs_by_hash[h])
+ else:
+ for env_path in self.included_specs_by_hash.keys():
+ if h in self.included_specs_by_hash[env_path]:
+ yield (s, self.included_specs_by_hash[env_path][h])
+ break
def concrete_roots(self):
"""Same as concretized_specs, except it returns the list of concrete
@@ -1915,8 +2142,7 @@ class Environment:
If these specs appear under different user_specs, only one copy
is added to the list returned.
"""
- specs = [self.specs_by_hash[h] for h in self.concretized_order]
-
+ specs = [self.specs_by_hash[h] for h in self.all_concretized_orders()]
if recurse_dependencies:
specs.extend(
traverse.traverse_nodes(
@@ -1961,31 +2187,76 @@ class Environment:
"concrete_specs": concrete_specs,
}
+ if self.included_concrete_envs:
+ data[included_concrete_name] = self.included_concrete_spec_data
+
return data
def _read_lockfile(self, file_or_json):
"""Read a lockfile from a file or from a raw string."""
lockfile_dict = sjson.load(file_or_json)
self._read_lockfile_dict(lockfile_dict)
- return lockfile_dict["_meta"]["lockfile-version"]
+ return lockfile_dict
+
+ def set_included_concretized_user_specs(
+ self,
+ env_name: str,
+ env_info: Dict[str, Dict[str, Any]],
+ included_json_specs_by_hash: Dict[str, Dict[str, Any]],
+ ) -> Dict[str, Dict[str, Any]]:
+ """Sets all of the concretized user specs from included environments
+ to include those from nested included environments.
+
+ Args:
+ env_name: the name (technically the path) of the included environment
+ env_info: included concrete environment data
+ included_json_specs_by_hash: concrete spec data keyed by hash
+
+ Returns: updated specs_by_hash
+ """
+ self.included_concretized_order[env_name] = []
+ self.included_concretized_user_specs[env_name] = []
+
+ def add_specs(name, info, specs_by_hash):
+ # Add specs from the environment as well as any of its nested
+ # environments.
+ for root_info in info["roots"]:
+ self.included_concretized_order[name].append(root_info["hash"])
+ self.included_concretized_user_specs[name].append(Spec(root_info["spec"]))
+ if "concrete_specs" in info:
+ specs_by_hash.update(info["concrete_specs"])
+
+ if included_concrete_name in info:
+ for included_name, included_info in info[included_concrete_name].items():
+ if included_name not in self.included_concretized_order:
+ self.included_concretized_order[included_name] = []
+ self.included_concretized_user_specs[included_name] = []
+ add_specs(included_name, included_info, specs_by_hash)
+
+ add_specs(env_name, env_info, included_json_specs_by_hash)
+ return included_json_specs_by_hash
def _read_lockfile_dict(self, d):
"""Read a lockfile dictionary into this environment."""
self.specs_by_hash = {}
+ self.included_specs_by_hash = {}
+ self.included_concretized_user_specs = {}
+ self.included_concretized_order = {}
roots = d["roots"]
self.concretized_user_specs = [Spec(r["spec"]) for r in roots]
self.concretized_order = [r["hash"] for r in roots]
json_specs_by_hash = d["concrete_specs"]
+ included_json_specs_by_hash = {}
- # Track specs by their lockfile key. Currently spack uses the finest
- # grained hash as the lockfile key, while older formats used the build
- # hash or a previous incarnation of the DAG hash (one that did not
- # include build deps or package hash).
- specs_by_hash = {}
+ if included_concrete_name in d:
+ for env_name, env_info in d[included_concrete_name].items():
+ included_json_specs_by_hash.update(
+ self.set_included_concretized_user_specs(
+ env_name, env_info, included_json_specs_by_hash
+ )
+ )
- # Track specs by their DAG hash, allows handling DAG hash collisions
- first_seen = {}
current_lockfile_format = d["_meta"]["lockfile-version"]
try:
reader = READER_CLS[current_lockfile_format]
@@ -1998,6 +2269,39 @@ class Environment:
msg += " You need to use a newer Spack version."
raise SpackEnvironmentError(msg)
+ first_seen, self.concretized_order = self.filter_specs(
+ reader, json_specs_by_hash, self.concretized_order
+ )
+
+ for spec_dag_hash in self.concretized_order:
+ self.specs_by_hash[spec_dag_hash] = first_seen[spec_dag_hash]
+
+ if any(self.included_concretized_order.values()):
+ first_seen = {}
+
+ for env_name, concretized_order in self.included_concretized_order.items():
+ filtered_spec, self.included_concretized_order[env_name] = self.filter_specs(
+ reader, included_json_specs_by_hash, concretized_order
+ )
+ first_seen.update(filtered_spec)
+
+ for env_path, spec_hashes in self.included_concretized_order.items():
+ self.included_specs_by_hash[env_path] = {}
+ for spec_dag_hash in spec_hashes:
+ self.included_specs_by_hash[env_path].update(
+ {spec_dag_hash: first_seen[spec_dag_hash]}
+ )
+
+ def filter_specs(self, reader, json_specs_by_hash, order_concretized):
+ # Track specs by their lockfile key. Currently spack uses the finest
+ # grained hash as the lockfile key, while older formats used the build
+ # hash or a previous incarnation of the DAG hash (one that did not
+ # include build deps or package hash).
+ specs_by_hash = {}
+
+ # Track specs by their DAG hash, allows handling DAG hash collisions
+ first_seen = {}
+
# First pass: Put each spec in the map ignoring dependencies
for lockfile_key, node_dict in json_specs_by_hash.items():
spec = reader.from_node_dict(node_dict)
@@ -2020,7 +2324,8 @@ class Environment:
# keep. This is only required as long as we support older lockfile
# formats where the mapping from DAG hash to lockfile key is possibly
# one-to-many.
- for lockfile_key in self.concretized_order:
+
+ for lockfile_key in order_concretized:
for s in specs_by_hash[lockfile_key].traverse():
if s.dag_hash() not in first_seen:
first_seen[s.dag_hash()] = s
@@ -2028,12 +2333,10 @@ class Environment:
# Now make sure concretized_order and our internal specs dict
# contains the keys used by modern spack (i.e. the dag_hash
# that includes build deps and package hash).
- self.concretized_order = [
- specs_by_hash[h_key].dag_hash() for h_key in self.concretized_order
- ]
- for spec_dag_hash in self.concretized_order:
- self.specs_by_hash[spec_dag_hash] = first_seen[spec_dag_hash]
+ order_concretized = [specs_by_hash[h_key].dag_hash() for h_key in order_concretized]
+
+ return first_seen, order_concretized
def write(self, regenerate: bool = True) -> None:
"""Writes an in-memory environment to its location on disk.
@@ -2046,7 +2349,7 @@ class Environment:
regenerate: regenerate views and run post-write hooks as well as writing if True.
"""
self.manifest_uptodate_or_warn()
- if self.specs_by_hash:
+ if self.specs_by_hash or self.included_concrete_envs:
self.ensure_env_directory_exists(dot_env=True)
self.update_environment_repository()
self.manifest.flush()
@@ -2545,6 +2848,19 @@ class EnvironmentManifestFile(collections.abc.Mapping):
raise SpackEnvironmentError(msg) from e
self.changed = True
+ def set_include_concrete(self, include_concrete: List[str]) -> None:
+ """Sets the included concrete environments in the manifest to the value(s) passed as input.
+
+ Args:
+ include_concrete: list of already existing concrete environments to include
+ """
+ self.pristine_configuration[included_concrete_name] = []
+
+ for env_path in include_concrete:
+ self.pristine_configuration[included_concrete_name].append(env_path)
+
+ self.changed = True
+
def add_definition(self, user_spec: str, list_name: str) -> None:
"""Appends a user spec to the first active definition matching the name passed as argument.
diff --git a/lib/spack/spack/schema/env.py b/lib/spack/spack/schema/env.py
index d2df795a3d..8b37f3e236 100644
--- a/lib/spack/spack/schema/env.py
+++ b/lib/spack/spack/schema/env.py
@@ -35,6 +35,7 @@ properties: Dict[str, Any] = {
{
"include": {"type": "array", "default": [], "items": {"type": "string"}},
"specs": spec_list_schema,
+ "include_concrete": {"type": "array", "default": [], "items": {"type": "string"}},
},
),
}
diff --git a/lib/spack/spack/test/cmd/env.py b/lib/spack/spack/test/cmd/env.py
index a05222db6b..e1136e3bbe 100644
--- a/lib/spack/spack/test/cmd/env.py
+++ b/lib/spack/spack/test/cmd/env.py
@@ -60,6 +60,27 @@ module = SpackCommand("module")
sep = os.sep
+def setup_combined_multiple_env():
+ env("create", "test1")
+ test1 = ev.read("test1")
+ with test1:
+ add("zlib")
+ test1.concretize()
+ test1.write()
+
+ env("create", "test2")
+ test2 = ev.read("test2")
+ with test2:
+ add("libelf")
+ test2.concretize()
+ test2.write()
+
+ env("create", "--include-concrete", "test1", "--include-concrete", "test2", "combined_env")
+ combined = ev.read("combined_env")
+
+ return test1, test2, combined
+
+
@pytest.fixture()
def environment_from_manifest(tmp_path):
"""Returns a new environment named 'test' from the content of a manifest file."""
@@ -369,6 +390,29 @@ def test_env_install_single_spec(install_mockery, mock_fetch):
assert e.specs_by_hash[e.concretized_order[0]].name == "cmake-client"
+@pytest.mark.parametrize("unify", [True, False, "when_possible"])
+def test_env_install_include_concrete_env(unify, install_mockery, mock_fetch):
+ test1, test2, combined = setup_combined_multiple_env()
+
+ combined.concretize()
+ combined.write()
+
+ combined.unify = unify
+
+ with combined:
+ install()
+
+ test1_roots = test1.concretized_order
+ test2_roots = test2.concretized_order
+ combined_included_roots = combined.included_concretized_order
+
+ for spec in combined.all_specs():
+ assert spec.installed
+
+ assert test1_roots == combined_included_roots[test1.path]
+ assert test2_roots == combined_included_roots[test2.path]
+
+
def test_env_roots_marked_explicit(install_mockery, mock_fetch):
install = SpackCommand("install")
install("dependent-install")
@@ -557,6 +601,41 @@ def test_remove_command():
assert "mpileaks@" not in find("--show-concretized")
+def test_bad_remove_included_env():
+ env("create", "test")
+ test = ev.read("test")
+
+ with test:
+ add("mpileaks")
+
+ test.concretize()
+ test.write()
+
+ env("create", "--include-concrete", "test", "combined_env")
+
+ with pytest.raises(SpackCommandError):
+ env("remove", "test")
+
+
+def test_force_remove_included_env():
+ env("create", "test")
+ test = ev.read("test")
+
+ with test:
+ add("mpileaks")
+
+ test.concretize()
+ test.write()
+
+ env("create", "--include-concrete", "test", "combined_env")
+
+ rm_output = env("remove", "-f", "-y", "test")
+ list_output = env("list")
+
+ assert '"test" is being used by environment "combined_env"' in rm_output
+ assert "test" not in list_output
+
+
def test_environment_status(capsys, tmpdir):
with tmpdir.as_cwd():
with capsys.disabled():
@@ -1636,6 +1715,275 @@ def test_env_without_view_install(tmpdir, mock_stage, mock_fetch, install_mocker
check_mpileaks_and_deps_in_view(view_dir)
+@pytest.mark.parametrize("env_name", [True, False])
+def test_env_include_concrete_env_yaml(env_name):
+ env("create", "test")
+ test = ev.read("test")
+
+ with test:
+ add("mpileaks")
+ test.concretize()
+ test.write()
+
+ environ = "test" if env_name else test.path
+
+ env("create", "--include-concrete", environ, "combined_env")
+
+ combined = ev.read("combined_env")
+ combined_yaml = combined.manifest["spack"]
+
+ assert "include_concrete" in combined_yaml
+ assert test.path in combined_yaml["include_concrete"]
+
+
+def test_env_bad_include_concrete_env():
+ with pytest.raises(ev.SpackEnvironmentError):
+ env("create", "--include-concrete", "nonexistant_env", "combined_env")
+
+
+def test_env_not_concrete_include_concrete_env():
+ env("create", "test")
+ test = ev.read("test")
+
+ with test:
+ add("mpileaks")
+
+ with pytest.raises(ev.SpackEnvironmentError):
+ env("create", "--include-concrete", "test", "combined_env")
+
+
+def test_env_multiple_include_concrete_envs():
+ test1, test2, combined = setup_combined_multiple_env()
+
+ combined_yaml = combined.manifest["spack"]
+
+ assert test1.path in combined_yaml["include_concrete"][0]
+ assert test2.path in combined_yaml["include_concrete"][1]
+
+ # No local specs in the combined env
+ assert not combined_yaml["specs"]
+
+
+def test_env_include_concrete_envs_lockfile():
+ test1, test2, combined = setup_combined_multiple_env()
+
+ combined_yaml = combined.manifest["spack"]
+
+ assert "include_concrete" in combined_yaml
+ assert test1.path in combined_yaml["include_concrete"]
+
+ with open(combined.lock_path) as f:
+ lockfile_as_dict = combined._read_lockfile(f)
+
+ assert set(
+ entry["hash"] for entry in lockfile_as_dict["include_concrete"][test1.path]["roots"]
+ ) == set(test1.specs_by_hash)
+ assert set(
+ entry["hash"] for entry in lockfile_as_dict["include_concrete"][test2.path]["roots"]
+ ) == set(test2.specs_by_hash)
+
+
+def test_env_include_concrete_add_env():
+ test1, test2, combined = setup_combined_multiple_env()
+
+ # crete new env & crecretize
+ env("create", "new")
+ new_env = ev.read("new")
+ with new_env:
+ add("mpileaks")
+
+ new_env.concretize()
+ new_env.write()
+
+ # add new env to combined
+ combined.included_concrete_envs.append(new_env.path)
+
+ # assert thing haven't changed yet
+ with open(combined.lock_path) as f:
+ lockfile_as_dict = combined._read_lockfile(f)
+
+ assert new_env.path not in lockfile_as_dict["include_concrete"].keys()
+
+ # concretize combined env with new env
+ combined.concretize()
+ combined.write()
+
+ # assert changes
+ with open(combined.lock_path) as f:
+ lockfile_as_dict = combined._read_lockfile(f)
+
+ assert new_env.path in lockfile_as_dict["include_concrete"].keys()
+
+
+def test_env_include_concrete_remove_env():
+ test1, test2, combined = setup_combined_multiple_env()
+
+ # remove test2 from combined
+ combined.included_concrete_envs = [test1.path]
+
+ # assert test2 is still in combined's lockfile
+ with open(combined.lock_path) as f:
+ lockfile_as_dict = combined._read_lockfile(f)
+
+ assert test2.path in lockfile_as_dict["include_concrete"].keys()
+
+ # reconcretize combined
+ combined.concretize()
+ combined.write()
+
+ # assert test2 is not in combined's lockfile
+ with open(combined.lock_path) as f:
+ lockfile_as_dict = combined._read_lockfile(f)
+
+ assert test2.path not in lockfile_as_dict["include_concrete"].keys()
+
+
+@pytest.mark.parametrize("unify", [True, False, "when_possible"])
+def test_env_include_concrete_env_reconcretized(unify):
+ """Double check to make sure that concrete_specs for the local specs is empty
+ after recocnretizing.
+ """
+ _, _, combined = setup_combined_multiple_env()
+
+ combined.unify = unify
+
+ with open(combined.lock_path) as f:
+ lockfile_as_dict = combined._read_lockfile(f)
+
+ assert not lockfile_as_dict["roots"]
+ assert not lockfile_as_dict["concrete_specs"]
+
+ combined.concretize()
+ combined.write()
+
+ with open(combined.lock_path) as f:
+ lockfile_as_dict = combined._read_lockfile(f)
+
+ assert not lockfile_as_dict["roots"]
+ assert not lockfile_as_dict["concrete_specs"]
+
+
+def test_concretize_include_concrete_env():
+ test1, _, combined = setup_combined_multiple_env()
+
+ with test1:
+ add("mpileaks")
+ test1.concretize()
+ test1.write()
+
+ assert Spec("mpileaks") in test1.concretized_user_specs
+ assert Spec("mpileaks") not in combined.included_concretized_user_specs[test1.path]
+
+ combined.concretize()
+ combined.write()
+
+ assert Spec("mpileaks") in combined.included_concretized_user_specs[test1.path]
+
+
+def test_concretize_nested_include_concrete_envs():
+ env("create", "test1")
+ test1 = ev.read("test1")
+ with test1:
+ add("zlib")
+ test1.concretize()
+ test1.write()
+
+ env("create", "--include-concrete", "test1", "test2")
+ test2 = ev.read("test2")
+ with test2:
+ add("libelf")
+ test2.concretize()
+ test2.write()
+
+ env("create", "--include-concrete", "test2", "test3")
+ test3 = ev.read("test3")
+
+ with open(test3.lock_path) as f:
+ lockfile_as_dict = test3._read_lockfile(f)
+
+ assert test2.path in lockfile_as_dict["include_concrete"]
+ assert test1.path in lockfile_as_dict["include_concrete"][test2.path]["include_concrete"]
+
+ assert Spec("zlib") in test3.included_concretized_user_specs[test1.path]
+
+
+def test_concretize_nested_included_concrete():
+ """Confirm that nested included environments use specs concretized at
+ environment creation time and change with reconcretization."""
+ env("create", "test1")
+ test1 = ev.read("test1")
+ with test1:
+ add("zlib")
+ test1.concretize()
+ test1.write()
+
+ # test2 should include test1 with zlib
+ env("create", "--include-concrete", "test1", "test2")
+ test2 = ev.read("test2")
+ with test2:
+ add("libelf")
+ test2.concretize()
+ test2.write()
+
+ assert Spec("zlib") in test2.included_concretized_user_specs[test1.path]
+
+ # Modify/re-concretize test1 to replace zlib with mpileaks
+ with test1:
+ remove("zlib")
+ add("mpileaks")
+ test1.concretize()
+ test1.write()
+
+ # test3 should include the latest concretization of test1
+ env("create", "--include-concrete", "test1", "test3")
+ test3 = ev.read("test3")
+ with test3:
+ add("callpath")
+ test3.concretize()
+ test3.write()
+
+ included_specs = test3.included_concretized_user_specs[test1.path]
+ assert len(included_specs) == 1
+ assert Spec("mpileaks") in included_specs
+
+ # The last concretization of test4's included environments should have test2
+ # with the original concretized test1 spec and test3 with the re-concretized
+ # test1 spec.
+ env("create", "--include-concrete", "test2", "--include-concrete", "test3", "test4")
+ test4 = ev.read("test4")
+
+ def included_included_spec(path1, path2):
+ included_path1 = test4.included_concrete_spec_data[path1]
+ included_path2 = included_path1["include_concrete"][path2]
+ return included_path2["roots"][0]["spec"]
+
+ included_test2_test1 = included_included_spec(test2.path, test1.path)
+ assert "zlib" in included_test2_test1
+
+ included_test3_test1 = included_included_spec(test3.path, test1.path)
+ assert "mpileaks" in included_test3_test1
+
+ # test4's concretized specs should reflect the original concretization.
+ concrete_specs = [s for s, _ in test4.concretized_specs()]
+ expected = [Spec(s) for s in ["libelf", "zlib", "mpileaks", "callpath"]]
+ assert all(s in concrete_specs for s in expected)
+
+ # Re-concretize test2 to reflect the new concretization of included test1
+ # to remove zlib and write it out so it can be picked up by test4.
+ # Re-concretize test4 to reflect the re-concretization of included test2
+ # and ensure that its included specs are up-to-date
+ test2.concretize()
+ test2.write()
+ test4.concretize()
+
+ concrete_specs = [s for s, _ in test4.concretized_specs()]
+ assert Spec("zlib") not in concrete_specs
+
+ # Expecting mpileaks to appear only once
+ expected = [Spec(s) for s in ["libelf", "mpileaks", "callpath"]]
+ assert len(concrete_specs) == 3 and all(s in concrete_specs for s in expected)
+
+
def test_env_config_view_default(
environment_from_manifest, mock_stage, mock_fetch, install_mockery
):
diff --git a/lib/spack/spack/test/cmd/find.py b/lib/spack/spack/test/cmd/find.py
index e42a938e38..37a0a7ff14 100644
--- a/lib/spack/spack/test/cmd/find.py
+++ b/lib/spack/spack/test/cmd/find.py
@@ -349,6 +349,87 @@ def test_find_prefix_in_env(
# Would throw error on regression
+def test_find_specs_include_concrete_env(mutable_mock_env_path, config, mutable_mock_repo, tmpdir):
+ path = tmpdir.join("spack.yaml")
+
+ with tmpdir.as_cwd():
+ with open(str(path), "w") as f:
+ f.write(
+ """\
+spack:
+ specs:
+ - mpileaks
+"""
+ )
+ env("create", "test1", "spack.yaml")
+
+ test1 = ev.read("test1")
+ test1.concretize()
+ test1.write()
+
+ with tmpdir.as_cwd():
+ with open(str(path), "w") as f:
+ f.write(
+ """\
+spack:
+ specs:
+ - libelf
+"""
+ )
+ env("create", "test2", "spack.yaml")
+
+ test2 = ev.read("test2")
+ test2.concretize()
+ test2.write()
+
+ env("create", "--include-concrete", "test1", "--include-concrete", "test2", "combined_env")
+
+ with ev.read("combined_env"):
+ output = find()
+
+ assert "No root specs" in output
+ assert "Included specs" in output
+ assert "mpileaks" in output
+ assert "libelf" in output
+
+
+def test_find_specs_nested_include_concrete_env(
+ mutable_mock_env_path, config, mutable_mock_repo, tmpdir
+):
+ path = tmpdir.join("spack.yaml")
+
+ with tmpdir.as_cwd():
+ with open(str(path), "w") as f:
+ f.write(
+ """\
+spack:
+ specs:
+ - mpileaks
+"""
+ )
+ env("create", "test1", "spack.yaml")
+
+ test1 = ev.read("test1")
+ test1.concretize()
+ test1.write()
+
+ env("create", "--include-concrete", "test1", "test2")
+ test2 = ev.read("test2")
+ test2.add("libelf")
+ test2.concretize()
+ test2.write()
+
+ env("create", "--include-concrete", "test2", "test3")
+
+ with ev.read("test3"):
+ output = find()
+
+ assert "No root specs" in output
+ assert "Included specs" in output
+ assert "mpileaks" in output
+ assert "libelf" in output
+
+
def test_find_loaded(database, working_env):
output = find("--loaded", "--group")
assert output == ""
diff --git a/share/spack/spack-completion.bash b/share/spack/spack-completion.bash
index 387b364189..1f73849fc8 100755
--- a/share/spack/spack-completion.bash
+++ b/share/spack/spack-completion.bash
@@ -1052,7 +1052,7 @@ _spack_env_deactivate() {
_spack_env_create() {
if $list_options
then
- SPACK_COMPREPLY="-h --help -d --dir --keep-relative --without-view --with-view"
+ SPACK_COMPREPLY="-h --help -d --dir --keep-relative --without-view --with-view --include-concrete"
else
_environments
fi
@@ -1061,7 +1061,7 @@ _spack_env_create() {
_spack_env_remove() {
if $list_options
then
- SPACK_COMPREPLY="-h --help -y --yes-to-all"
+ SPACK_COMPREPLY="-h --help -y --yes-to-all -f --force"
else
_environments
fi
@@ -1070,7 +1070,7 @@ _spack_env_remove() {
_spack_env_rm() {
if $list_options
then
- SPACK_COMPREPLY="-h --help -y --yes-to-all"
+ SPACK_COMPREPLY="-h --help -y --yes-to-all -f --force"
else
_environments
fi
diff --git a/share/spack/spack-completion.fish b/share/spack/spack-completion.fish
index 3818b12f1b..63abb4864e 100755
--- a/share/spack/spack-completion.fish
+++ b/share/spack/spack-completion.fish
@@ -1538,7 +1538,7 @@ complete -c spack -n '__fish_spack_using_command env deactivate' -l pwsh -f -a s
complete -c spack -n '__fish_spack_using_command env deactivate' -l pwsh -d 'print pwsh commands to activate the environment'
# spack env create
-set -g __fish_spack_optspecs_spack_env_create h/help d/dir keep-relative without-view with-view=
+set -g __fish_spack_optspecs_spack_env_create h/help d/dir keep-relative without-view with-view= include-concrete=
complete -c spack -n '__fish_spack_using_command_pos 0 env create' -f -a '(__fish_spack_environments)'
complete -c spack -n '__fish_spack_using_command env create' -s h -l help -f -a help
complete -c spack -n '__fish_spack_using_command env create' -s h -l help -d 'show this help message and exit'
@@ -1550,22 +1550,28 @@ complete -c spack -n '__fish_spack_using_command env create' -l without-view -f
complete -c spack -n '__fish_spack_using_command env create' -l without-view -d 'do not maintain a view for this environment'
complete -c spack -n '__fish_spack_using_command env create' -l with-view -r -f -a with_view
complete -c spack -n '__fish_spack_using_command env create' -l with-view -r -d 'specify that this environment should maintain a view at the specified path (by default the view is maintained in the environment directory)'
+complete -c spack -n '__fish_spack_using_command env create' -l include-concrete -r -f -a include_concrete
+complete -c spack -n '__fish_spack_using_command env create' -l include-concrete -r -d 'name of old environment to copy specs from'
# spack env remove
-set -g __fish_spack_optspecs_spack_env_remove h/help y/yes-to-all
+set -g __fish_spack_optspecs_spack_env_remove h/help y/yes-to-all f/force
complete -c spack -n '__fish_spack_using_command_pos_remainder 0 env remove' -f -a '(__fish_spack_environments)'
complete -c spack -n '__fish_spack_using_command env remove' -s h -l help -f -a help
complete -c spack -n '__fish_spack_using_command env remove' -s h -l help -d 'show this help message and exit'
complete -c spack -n '__fish_spack_using_command env remove' -s y -l yes-to-all -f -a yes_to_all
complete -c spack -n '__fish_spack_using_command env remove' -s y -l yes-to-all -d 'assume "yes" is the answer to every confirmation request'
+complete -c spack -n '__fish_spack_using_command env remove' -s f -l force -f -a force
+complete -c spack -n '__fish_spack_using_command env remove' -s f -l force -d 'remove the environment even if it is included in another environment'
# spack env rm
-set -g __fish_spack_optspecs_spack_env_rm h/help y/yes-to-all
+set -g __fish_spack_optspecs_spack_env_rm h/help y/yes-to-all f/force
complete -c spack -n '__fish_spack_using_command_pos_remainder 0 env rm' -f -a '(__fish_spack_environments)'
complete -c spack -n '__fish_spack_using_command env rm' -s h -l help -f -a help
complete -c spack -n '__fish_spack_using_command env rm' -s h -l help -d 'show this help message and exit'
complete -c spack -n '__fish_spack_using_command env rm' -s y -l yes-to-all -f -a yes_to_all
complete -c spack -n '__fish_spack_using_command env rm' -s y -l yes-to-all -d 'assume "yes" is the answer to every confirmation request'
+complete -c spack -n '__fish_spack_using_command env rm' -s f -l force -f -a force
+complete -c spack -n '__fish_spack_using_command env rm' -s f -l force -d 'remove the environment even if it is included in another environment'
# spack env rename
set -g __fish_spack_optspecs_spack_env_rename h/help d/dir f/force
generated by cgit v1.2.3-70-g09d2 (git 2.46.0) at 2025-03-31 05:10:38 +0000