diff options
author | Harmen Stoppels <harmenstoppels@gmail.com> | 2022-11-07 15:38:24 +0100 |
---|---|---|
committer | GitHub <noreply@github.com> | 2022-11-07 15:38:24 +0100 |
commit | 2ab974f5301746f20ff19cb991951158a9c498fd (patch) | |
tree | f233b1d4bb6e8a00de24d772c7f1e89826aa8396 | |
parent | e045dabb3afbc6b2af78fe7deeb92d0fe995c814 (diff) | |
download | spack-2ab974f5301746f20ff19cb991951158a9c498fd.tar.gz spack-2ab974f5301746f20ff19cb991951158a9c498fd.tar.bz2 spack-2ab974f5301746f20ff19cb991951158a9c498fd.tar.xz spack-2ab974f5301746f20ff19cb991951158a9c498fd.zip |
concretizer:unify:true as a default (#31787)
`spack env create` enables a view by default (in a weird hidden
directory, but well...). This is asking for trouble with the other
default of `concretizer:unify:false`, since having different flavors of
the same spec in an environment, leads to collision errors when
generating the view.
A change of defaults would improve user experience:
However, `unify:true` makes most sense, since any time the issue is
brought up in Slack, the user changes the concretization config, since
it wasn't the intention to have different flavors of the same spec, and
install times are decreased.
Further we improve the docs and drop the duplicate root spec limitation
-rw-r--r-- | etc/spack/defaults/concretizer.yaml | 2 | ||||
-rw-r--r-- | lib/spack/docs/environments.rst | 55 | ||||
-rw-r--r-- | lib/spack/spack/environment/environment.py | 33 | ||||
-rw-r--r-- | lib/spack/spack/test/cmd/env.py | 5 |
4 files changed, 50 insertions, 45 deletions
diff --git a/etc/spack/defaults/concretizer.yaml b/etc/spack/defaults/concretizer.yaml index 7311354c28..f455aa723d 100644 --- a/etc/spack/defaults/concretizer.yaml +++ b/etc/spack/defaults/concretizer.yaml @@ -33,4 +33,4 @@ concretizer: # environments can always be activated. When "false" perform concretization separately # on each root spec, allowing different versions and variants of the same package in # an environment. - unify: false
\ No newline at end of file + unify: true
\ No newline at end of file diff --git a/lib/spack/docs/environments.rst b/lib/spack/docs/environments.rst index b417e26f3e..9f75f789ae 100644 --- a/lib/spack/docs/environments.rst +++ b/lib/spack/docs/environments.rst @@ -519,27 +519,33 @@ available from the yaml file. ^^^^^^^^^^^^^^^^^^^ Spec concretization ^^^^^^^^^^^^^^^^^^^ -An environment can be concretized in three different modes and the behavior active under any environment -is determined by the ``concretizer:unify`` property. By default specs are concretized *separately*, one after the other: +An environment can be concretized in three different modes and the behavior active under +any environment is determined by the ``concretizer:unify`` configuration option. + +The *default* mode is to unify all specs: .. code-block:: yaml spack: specs: - - hdf5~mpi - hdf5+mpi - zlib@1.2.8 concretizer: - unify: false + unify: true + +This means that any package in the environment corresponds to a single concrete spec. In +the above example, when ``hdf5`` depends down the line of ``zlib``, it is required to +take ``zlib@1.2.8`` instead of a newer version. This mode of concretization is +particularly useful when environment views are used: if every package occurs in +only one flavor, it is usually possible to merge all install directories into a view. -This mode of operation permits to deploy a full software stack where multiple configurations of the same package -need to be installed alongside each other using the best possible selection of transitive dependencies. The downside -is that redundancy of installations is disregarded completely, and thus environments might be more bloated than -strictly needed. In the example above, for instance, if a version of ``zlib`` newer than ``1.2.8`` is known to Spack, -then it will be used for both ``hdf5`` installations. +A downside of unified concretization is that it can be overly strict. For example, a +concretization error would happen when both ``hdf5+mpi`` and ``hdf5~mpi`` are specified +in an environment. -If redundancy of the environment is a concern, Spack provides a way to install it *together where possible*, -i.e. trying to maximize reuse of dependencies across different specs: +The second mode is to *unify when possible*: this makes concretization of root specs +more independendent. Instead of requiring reuse of dependencies across different root +specs, it is only maximized: .. code-block:: yaml @@ -551,26 +557,27 @@ i.e. trying to maximize reuse of dependencies across different specs: concretizer: unify: when_possible -Also in this case Spack allows having multiple configurations of the same package, but privileges the reuse of -specs over other factors. Going back to our example, this means that both ``hdf5`` installations will use -``zlib@1.2.8`` as a dependency even if newer versions of that library are available. -Central installations done at HPC centers by system administrators or user support groups are a common case -that fits either of these two modes. +This means that both ``hdf5`` installations will use ``zlib@1.2.8`` as a dependency even +if newer versions of that library are available. -Environments can also be configured to concretize all the root specs *together*, in a self-consistent way, to -ensure that each package in the environment comes with a single configuration: +The third mode of operation is to concretize root specs entirely independently by +disabling unified concretization: .. code-block:: yaml spack: specs: + - hdf5~mpi - hdf5+mpi - zlib@1.2.8 concretizer: - unify: true + unify: false + +In this example ``hdf5`` is concretized separately, and does not consider ``zlib@1.2.8`` +as a constraint or preference. Instead, it will take the latest possible version. -This mode of operation is usually what is required by software developers that want to deploy their development -environment and have a single view of it in the filesystem. +The last two concretization options are typically useful for system administrators and +user support groups providing a large software stack for their HPC center. .. note:: @@ -581,10 +588,10 @@ environment and have a single view of it in the filesystem. .. admonition:: Re-concretization of user specs - When concretizing specs *together* or *together where possible* the entire set of specs will be + When using *unified* concretization (when possible), the entire set of specs will be re-concretized after any addition of new user specs, to ensure that - the environment remains consistent / minimal. When instead the specs are concretized - separately only the new specs will be re-concretized after any addition. + the environment remains consistent / minimal. When instead unified concretization is + disabled, only the new specs will be concretized after any addition. ^^^^^^^^^^^^^ Spec Matrices diff --git a/lib/spack/spack/environment/environment.py b/lib/spack/spack/environment/environment.py index 44d6785456..0265d68a35 100644 --- a/lib/spack/spack/environment/environment.py +++ b/lib/spack/spack/environment/environment.py @@ -1322,30 +1322,25 @@ class Environment(object): if user_specs_did_not_change: return [] - # Check that user specs don't have duplicate packages - counter = collections.defaultdict(int) - for user_spec in self.user_specs: - counter[user_spec.name] += 1 - - duplicates = [] - for name, count in counter.items(): - if count > 1: - duplicates.append(name) - - if duplicates: - msg = ( - "environment that are configured to concretize specs" - " together cannot contain more than one spec for each" - " package [{0}]".format(", ".join(duplicates)) - ) - raise SpackEnvironmentError(msg) - # Proceed with concretization self.concretized_user_specs = [] self.concretized_order = [] self.specs_by_hash = {} - concrete_specs = spack.concretize.concretize_specs_together(*self.user_specs, tests=tests) + try: + concrete_specs = spack.concretize.concretize_specs_together( + *self.user_specs, tests=tests + ) + except spack.error.UnsatisfiableSpecError as e: + # "Enhance" the error message for multiple root specs, suggest a less strict + # form of concretization. + if len(self.user_specs) > 1: + e.message += ( + ". Consider setting `concretizer:unify` to `when_possible` " + "or `false` to relax the concretizer strictness." + ) + raise + concretized_specs = [x for x in zip(self.user_specs, concrete_specs)] for abstract, concrete in concretized_specs: self._add_concrete_spec(abstract, concrete) diff --git a/lib/spack/spack/test/cmd/env.py b/lib/spack/spack/test/cmd/env.py index 6257431ee9..801ff04669 100644 --- a/lib/spack/spack/test/cmd/env.py +++ b/lib/spack/spack/test/cmd/env.py @@ -18,6 +18,7 @@ import llnl.util.link_tree import spack.cmd.env import spack.environment as ev import spack.environment.shell +import spack.error import spack.modules import spack.paths import spack.repo @@ -2403,7 +2404,9 @@ def test_duplicate_packages_raise_when_concretizing_together(): e.add("mpileaks~opt") e.add("mpich") - with pytest.raises(ev.SpackEnvironmentError, match=r"cannot contain more"): + with pytest.raises( + spack.error.UnsatisfiableSpecError, match=r"relax the concretizer strictness" + ): e.concretize() |