summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorMassimiliano Culpo <massimiliano.culpo@gmail.com>2022-04-26 02:19:51 +0200
committerGitHub <noreply@github.com>2022-04-25 17:19:51 -0700
commitb9d6a5103dd2f96c1a3e3e729435bee4cd30eac1 (patch)
tree0e45fbe3a327a7ff468254fc202b3ad4bff6ff46
parent834f8e04ca362195f9a971d1f90b43ce52f78cc4 (diff)
downloadspack-b9d6a5103dd2f96c1a3e3e729435bee4cd30eac1.tar.gz
spack-b9d6a5103dd2f96c1a3e3e729435bee4cd30eac1.tar.bz2
spack-b9d6a5103dd2f96c1a3e3e729435bee4cd30eac1.tar.xz
spack-b9d6a5103dd2f96c1a3e3e729435bee4cd30eac1.zip
ASP-based solver: allow configuring target selection (#29835)
* ASP-based solver: allow configuring target selection This commit adds a new "concretizer:targets" configuration section, and two options under it. - "concretizer:targets:granularity" allows switching from considering only generic targets to consider all possible microarchitectures. - "concretizer:targets:host_compatible" instead controls whether we can concretize for microarchitectures that are incompatible with the current host. * Add documentation * Add unit-tests
-rw-r--r--etc/spack/defaults/concretizer.yaml13
-rw-r--r--lib/spack/docs/build_settings.rst69
-rw-r--r--lib/spack/spack/schema/concretizer.py10
-rw-r--r--lib/spack/spack/solver/asp.py66
-rw-r--r--lib/spack/spack/test/concretize.py42
-rw-r--r--lib/spack/spack/test/data/config/concretizer.yaml5
6 files changed, 162 insertions, 43 deletions
diff --git a/etc/spack/defaults/concretizer.yaml b/etc/spack/defaults/concretizer.yaml
index 52200062c8..2aadccd6cd 100644
--- a/etc/spack/defaults/concretizer.yaml
+++ b/etc/spack/defaults/concretizer.yaml
@@ -15,3 +15,16 @@ concretizer:
# as possible, rather than building. If `false`, we'll always give you a fresh
# concretization.
reuse: false
+ # Options that tune which targets are considered for concretization. The
+ # concretization process is very sensitive to the number targets, and the time
+ # needed to reach a solution increases noticeably with the number of targets
+ # considered.
+ targets:
+ # Determine whether we want to target specific or generic microarchitectures.
+ # An example of the first kind might be for instance "skylake" or "bulldozer",
+ # while generic microarchitectures are for instance "aarch64" or "x86_64_v4".
+ granularity: microarchitectures
+ # If "false" allow targets that are incompatible with the current host (for
+ # instance concretize with target "icelake" while running on "haswell").
+ # If "true" only allow targets that are compatible with the host.
+ host_compatible: false
diff --git a/lib/spack/docs/build_settings.rst b/lib/spack/docs/build_settings.rst
index 38eb078cbe..3b49375b30 100644
--- a/lib/spack/docs/build_settings.rst
+++ b/lib/spack/docs/build_settings.rst
@@ -219,26 +219,29 @@ Concretizer options
but you can also use ``concretizer.yaml`` to customize aspects of the
algorithm it uses to select the dependencies you install:
-.. _code-block: yaml
+.. literalinclude:: _spack_root/etc/spack/defaults/concretizer.yaml
+ :language: yaml
- concretizer:
- # Whether to consider installed packages or packages from buildcaches when
- # concretizing specs. If `true`, we'll try to use as many installs/binaries
- # as possible, rather than building. If `false`, we'll always give you a fresh
- # concretization.
- reuse: false
-
-^^^^^^^^^^^^^^^^
-``reuse``
-^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Reuse already installed packages
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-This controls whether Spack will prefer to use installed packages (``true``), or
+The ``reuse`` attribute controls whether Spack will prefer to use installed packages (``true``), or
whether it will do a "fresh" installation and prefer the latest settings from
-``package.py`` files and ``packages.yaml`` (``false``). .
+``package.py`` files and ``packages.yaml`` (``false``).
+You can use:
+
+.. code-block:: console
+
+ % spack install --reuse <spec>
+
+to enable reuse for a single installation, and you can use:
-You can use ``spack install --reuse`` to enable reuse for a single installation,
-and you can use ``spack install --fresh`` to do a fresh install if ``reuse`` is
-enabled by default.
+.. code-block:: console
+
+ spack install --fresh <spec>
+
+to do a fresh install if ``reuse`` is enabled by default.
.. note::
@@ -246,6 +249,40 @@ enabled by default.
in the next Spack release. You will still be able to use ``spack install --fresh``
to get the old behavior.
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Selection of the target microarchitectures
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The options under the ``targets`` attribute control which targets are considered during a solve.
+Currently the options in this section are only configurable from the ``concretization.yaml`` file
+and there are no corresponding command line arguments to enable them for a single solve.
+
+The ``granularity`` option can take two possible values: ``microarchitectures`` and ``generic``.
+If set to:
+
+.. code-block:: yaml
+
+ concretizer:
+ targets:
+ granularity: microarchitectures
+
+Spack will consider all the microarchitectures known to ``archspec`` to label nodes for
+compatibility. If instead the option is set to:
+
+.. code-block:: yaml
+
+ concretizer:
+ targets:
+ granularity: generic
+
+Spack will consider only generic microarchitectures. For instance, when running on an
+Haswell node, Spack will consider ``haswell`` as the best target in the former case and
+``x86_64_v3`` as the best target in the latter case.
+
+The ``host_compatible`` option is a Boolean option that determines whether or not the
+microarchitectures considered during the solve are constrained to be compatible with the
+host Spack is currently running on. For instance, if this option is set to ``true``, a
+user cannot concretize for ``target=icelake`` while running on an Haswell node.
.. _package-preferences:
diff --git a/lib/spack/spack/schema/concretizer.py b/lib/spack/spack/schema/concretizer.py
index e3bdde4adf..9138ab7685 100644
--- a/lib/spack/spack/schema/concretizer.py
+++ b/lib/spack/spack/schema/concretizer.py
@@ -15,6 +15,16 @@ properties = {
'additionalProperties': False,
'properties': {
'reuse': {'type': 'boolean'},
+ 'targets': {
+ 'type': 'object',
+ 'properties': {
+ 'host_compatible': {'type': 'boolean'},
+ 'granularity': {
+ 'type': 'string',
+ 'enum': ['generic', 'microarchitectures']
+ }
+ }
+ },
}
}
}
diff --git a/lib/spack/spack/solver/asp.py b/lib/spack/spack/solver/asp.py
index 78b465dddc..7b082382db 100644
--- a/lib/spack/spack/solver/asp.py
+++ b/lib/spack/spack/solver/asp.py
@@ -1413,23 +1413,48 @@ class SpackSolverSetup(object):
self.gen.h2('Target compatibility')
- compatible_targets = [uarch] + uarch.ancestors
- additional_targets_in_family = sorted([
- t for t in archspec.cpu.TARGETS.values()
- if (t.family.name == uarch.family.name and
- t not in compatible_targets)
- ], key=lambda x: len(x.ancestors), reverse=True)
- compatible_targets += additional_targets_in_family
+ # Construct the list of targets which are compatible with the host
+ candidate_targets = [uarch] + uarch.ancestors
+
+ # Get configuration options
+ granularity = spack.config.get('concretizer:targets:granularity')
+ host_compatible = spack.config.get('concretizer:targets:host_compatible')
+
+ # Add targets which are not compatible with the current host
+ if not host_compatible:
+ additional_targets_in_family = sorted([
+ t for t in archspec.cpu.TARGETS.values()
+ if (t.family.name == uarch.family.name and
+ t not in candidate_targets)
+ ], key=lambda x: len(x.ancestors), reverse=True)
+ candidate_targets += additional_targets_in_family
+
+ # Check if we want only generic architecture
+ if granularity == 'generic':
+ candidate_targets = [t for t in candidate_targets if t.vendor == 'generic']
+
compilers = self.possible_compilers
- # this loop can be used to limit the number of targets
- # considered. Right now we consider them all, but it seems that
- # many targets can make things slow.
- # TODO: investigate this.
+ # Add targets explicitly requested from specs
+ for spec in specs:
+ if not spec.architecture or not spec.architecture.target:
+ continue
+
+ target = archspec.cpu.TARGETS.get(spec.target.name)
+ if not target:
+ self.target_ranges(spec, None)
+ continue
+
+ if target not in candidate_targets and not host_compatible:
+ candidate_targets.append(target)
+ for ancestor in target.ancestors:
+ if ancestor not in candidate_targets:
+ candidate_targets.append(ancestor)
+
best_targets = set([uarch.family.name])
for compiler in sorted(compilers):
supported = self._supported_targets(
- compiler.name, compiler.version, compatible_targets
+ compiler.name, compiler.version, candidate_targets
)
# If we can't find supported targets it may be due to custom
@@ -1442,7 +1467,7 @@ class SpackSolverSetup(object):
supported = self._supported_targets(
compiler.name,
compiler_obj.real_version,
- compatible_targets
+ candidate_targets
)
if not supported:
@@ -1458,21 +1483,8 @@ class SpackSolverSetup(object):
compiler.name, compiler.version, uarch.family.name
))
- # add any targets explicitly mentioned in specs
- for spec in specs:
- if not spec.architecture or not spec.architecture.target:
- continue
-
- target = archspec.cpu.TARGETS.get(spec.target.name)
- if not target:
- self.target_ranges(spec, None)
- continue
-
- if target not in compatible_targets:
- compatible_targets.append(target)
-
i = 0
- for target in compatible_targets:
+ for target in candidate_targets:
self.gen.fact(fn.target(target.name))
self.gen.fact(fn.target_family(target.name, target.family.name))
for parent in sorted(target.parents):
diff --git a/lib/spack/spack/test/concretize.py b/lib/spack/spack/test/concretize.py
index 010d269507..32b94257bc 100644
--- a/lib/spack/spack/test/concretize.py
+++ b/lib/spack/spack/test/concretize.py
@@ -1526,6 +1526,48 @@ class TestConcretize(object):
s = Spec('conditional-values-in-variant@1.60.0').concretized()
assert 'cxxstd' in s.variants
+ def test_target_granularity(self):
+ if spack.config.get('config:concretizer') == 'original':
+ pytest.skip(
+ 'Original concretizer cannot account for target granularity'
+ )
+
+ # The test architecture uses core2 as the default target. Check that when
+ # we configure Spack for "generic" granularity we concretize for x86_64
+ s = Spec('python')
+ assert s.concretized().satisfies('target=core2')
+ with spack.config.override('concretizer:targets', {'granularity': 'generic'}):
+ assert s.concretized().satisfies('target=x86_64')
+
+ def test_host_compatible_concretization(self):
+ if spack.config.get('config:concretizer') == 'original':
+ pytest.skip(
+ 'Original concretizer cannot account for host compatibility'
+ )
+
+ # Check that after setting "host_compatible" to false we cannot concretize.
+ # Here we use "k10" to set a target non-compatible with the current host
+ # to avoid a lot of boilerplate when mocking the test platform. The issue
+ # is that the defaults for the test platform are very old, so there's no
+ # compiler supporting e.g. icelake etc.
+ s = Spec('python target=k10')
+ assert s.concretized()
+ with spack.config.override('concretizer:targets', {'host_compatible': True}):
+ with pytest.raises(spack.error.SpackError):
+ s.concretized()
+
+ def test_add_microarchitectures_on_explicit_request(self):
+ if spack.config.get('config:concretizer') == 'original':
+ pytest.skip(
+ 'Original concretizer cannot account for host compatibility'
+ )
+
+ # Check that if we consider only "generic" targets, we can still solve for
+ # specific microarchitectures on explicit requests
+ with spack.config.override('concretizer:targets', {'granularity': 'generic'}):
+ s = Spec('python target=k10').concretized()
+ assert s.satisfies('target=k10')
+
@pytest.mark.regression('29201')
def test_delete_version_and_reuse(
self, mutable_database, repo_with_changing_recipe
diff --git a/lib/spack/spack/test/data/config/concretizer.yaml b/lib/spack/spack/test/data/config/concretizer.yaml
new file mode 100644
index 0000000000..b5b1c712db
--- /dev/null
+++ b/lib/spack/spack/test/data/config/concretizer.yaml
@@ -0,0 +1,5 @@
+concretizer:
+ # reuse is missing on purpose, see "test_concretizer_arguments"
+ targets:
+ granularity: microarchitectures
+ host_compatible: false