diff options
author | Massimiliano Culpo <massimiliano.culpo@gmail.com> | 2020-07-31 13:07:48 +0200 |
---|---|---|
committer | Peter Scheibel <scheibel1@llnl.gov> | 2020-08-10 11:59:05 -0700 |
commit | c0d490ffbe7268e72b3214764d7b03aee9f65502 (patch) | |
tree | 4b186cfc39499cf346e7c61c23259b842aed15e0 /lib/spack/docs/packaging_guide.rst | |
parent | 193e8333fa23a2e9b44d44a80e153d9a27033860 (diff) | |
download | spack-c0d490ffbe7268e72b3214764d7b03aee9f65502.tar.gz spack-c0d490ffbe7268e72b3214764d7b03aee9f65502.tar.bz2 spack-c0d490ffbe7268e72b3214764d7b03aee9f65502.tar.xz spack-c0d490ffbe7268e72b3214764d7b03aee9f65502.zip |
Simplify the detection protocol for packages
Packages can implement “detect_version” to support detection
of external instances of a package. This is generally easier
than implementing “determine_spec_details”. The API for
determine_version is similar: for example you can return
“None” to indicate that an executable is not an instance
of a package.
Users may implement a “determine_variants” method for a package.
When doing external detection, executables are grouped by version
and each group results in a single invocation of “determine_variants”
for the associated spec. The method returns a string specifying
the variants for the package. The method may additionally return
a dictionary representing extra attributes for the package.
These will be stored in the spec yaml and can be retrieved
from self.spec.extra_attributes
The Spack GCC package has been updated with an implementation
of “determine_variants” which adds the following extra
attributes to the package: c, cxx, fortran
Diffstat (limited to 'lib/spack/docs/packaging_guide.rst')
-rw-r--r-- | lib/spack/docs/packaging_guide.rst | 239 |
1 files changed, 221 insertions, 18 deletions
diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst index d3a888b1fc..52f690c4d6 100644 --- a/lib/spack/docs/packaging_guide.rst +++ b/lib/spack/docs/packaging_guide.rst @@ -4054,21 +4054,223 @@ File functions Making a package discoverable with ``spack external find`` ---------------------------------------------------------- -To make a package discoverable with -:ref:`spack external find <cmd-spack-external-find>` you must -define one or more executables associated with the package and must -implement a method to generate a Spec when given an executable. +The simplest way to make a package discoverable with +:ref:`spack external find <cmd-spack-external-find>` is to: -The executables are specified as a package level ``executables`` -attribute which is a list of strings (see example below); each string -is treated as a regular expression (e.g. 'gcc' would match 'gcc', 'gcc-8.3', -'my-weird-gcc', etc.). +1. Define the executables associated with the package +2. Implement a method to determine the versions of these executables -The method ``determine_spec_details`` has the following signature: +^^^^^^^^^^^^^^^^^ +Minimal detection +^^^^^^^^^^^^^^^^^ + +The first step is fairly simple, as it requires only to +specify a package level ``executables`` attribute: + +.. code-block:: python + + class Foo(Package): + # Each string provided here is treated as a regular expression, and + # would match for example 'foo', 'foobar', and 'bazfoo'. + executables = ['foo'] + +This attribute must be a list of strings. Each string is a regular +expression (e.g. 'gcc' would match 'gcc', 'gcc-8.3', 'my-weird-gcc', etc.) to +determine a set of system executables that might be part or this package. Note +that to match only executables named 'gcc' the regular expression ``'^gcc$'`` +must be used. + +Finally to determine the version of each executable the ``determine_version`` +method must be implemented: + +.. code-block:: python + + @classmethod + def determine_version(cls, exe): + """Return either the version of the executable passed as argument + or ``None`` if the version cannot be determined. + + Args: + exe (str): absolute path to the executable being examined + """ + +This method receives as input the path to a single executable and must return +as output its version as a string; if the user cannot determine the version +or determines that the executable is not an instance of the package, they can +return None and the exe will be discarded as a candidate. +Implementing the two steps above is mandatory, and gives the package the +basic ability to detect if a spec is present on the system at a given version. + +.. note:: + Any executable for which the ``determine_version`` method returns ``None`` + will be discarded and won't appear in later stages of the workflow described below. + +^^^^^^^^^^^^^^^^^^^^^^^^ +Additional functionality +^^^^^^^^^^^^^^^^^^^^^^^^ + +Besides the two mandatory steps described above, there are also optional +methods that can be implemented to either increase the amount of details +being detected or improve the robustness of the detection logic in a package. + +"""""""""""""""""""""""""""""" +Variants and custom attributes +"""""""""""""""""""""""""""""" + +The ``determine_variants`` method can be optionally implemented in a package +to detect additional details of the spec: + +.. code-block:: python + + @classmethod + def determine_variants(cls, exes, version_str): + """Return either a variant string, a tuple of a variant string + and a dictionary of extra attributes that will be recorded in + packages.yaml or a list of those items. + + Args: + exes (list of str): list of executables (absolute paths) that + live in the same prefix and share the same version + version_str (str): version associated with the list of + executables, as detected by ``determine_version`` + """ + +This method takes as input a list of executables that live in the same prefix and +share the same version string, and returns either: + +1. A variant string +2. A tuple of a variant string and a dictionary of extra attributes +3. A list of items matching either 1 or 2 (if multiple specs are detected + from the set of executables) + +If extra attributes are returned, they will be recorded in ``packages.yaml`` +and be available for later reuse. As an example, the ``gcc`` package will record +by default the different compilers found and an entry in ``packages.yaml`` +would look like: + +.. code-block:: yaml + + packages: + gcc: + externals: + - spec: 'gcc@9.0.1 languages=c,c++,fortran' + prefix: /usr + extra_attributes: + compilers: + c: /usr/bin/x86_64-linux-gnu-gcc-9 + c++: /usr/bin/x86_64-linux-gnu-g++-9 + fortran: /usr/bin/x86_64-linux-gnu-gfortran-9 + +This allows us, for instance, to keep track of executables that would be named +differently if built by Spack (e.g. ``x86_64-linux-gnu-gcc-9`` +instead of just ``gcc``). + +.. TODO: we need to gather some more experience on overriding 'prefix' + and other special keywords in extra attributes, but as soon as we are + confident that this is the way to go we should document the process. + See https://github.com/spack/spack/pull/16526#issuecomment-653783204 + +""""""""""""""""""""""""""" +Filter matching executables +""""""""""""""""""""""""""" + +Sometimes defining the appropriate regex for the ``executables`` +attribute might prove to be difficult, especially if one has to +deal with corner cases or exclude "red herrings". To help keeping +the regular expressions as simple as possible, each package can +optionally implement a ``filter_executables`` method: + +.. code-block:: python + + @classmethod + def filter_detected_exes(cls, prefix, exes_in_prefix): + """Return a filtered list of the executables in prefix""" + +which takes as input a prefix and a list of matching executables and +returns a filtered list of said executables. + +Using this method has the advantage of allowing custom logic for +filtering, and does not restrict the user to regular expressions +only. Consider the case of detecting the GNU C++ compiler. If we +try to search for executables that match ``g++``, that would have +the unwanted side effect of selecting also ``clang++`` - which is +a C++ compiler provided by another package - if present on the system. +Trying to select executables that contain ``g++`` but not ``clang`` +would be quite complicated to do using regex only. Employing the +``filter_detected_exes`` method it becomes: + +.. code-block:: python + + class Gcc(Package): + executables = ['g++'] + + def filter_detected_exes(cls, prefix, exes_in_prefix): + return [x for x in exes_in_prefix if 'clang' not in x] + +Another possibility that this method opens is to apply certain +filtering logic when specific conditions are met (e.g. take some +decisions on an OS and not on another). + +^^^^^^^^^^^^^^^^^^ +Validate detection +^^^^^^^^^^^^^^^^^^ + +To increase detection robustness, packagers may also implement a method +to validate the detected Spec objects: + +.. code-block:: python + + @classmethod + def validate_detected_spec(cls, spec, extra_attributes): + """Validate a detected spec. Raise an exception if validation fails.""" + +This method receives a detected spec along with its extra attributes and can be +used to check that certain conditions are met by the spec. Packagers can either +use assertions or raise an ``InvalidSpecDetected`` exception when the check fails. +In case the conditions are not honored the spec will be discarded and any message +associated with the assertion or the exception will be logged as the reason for +discarding it. + +As an example, a package that wants to check that the ``compilers`` attribute is +in the extra attributes can implement this method like this: + +.. code-block:: python + + @classmethod + def validate_detected_spec(cls, spec, extra_attributes): + """Check that 'compilers' is in the extra attributes.""" + msg = ('the extra attribute "compilers" must be set for ' + 'the detected spec "{0}"'.format(spec)) + assert 'compilers' in extra_attributes, msg + +or like this: + +.. code-block:: python + + @classmethod + def validate_detected_spec(cls, spec, extra_attributes): + """Check that 'compilers' is in the extra attributes.""" + if 'compilers' not in extra_attributes: + msg = ('the extra attribute "compilers" must be set for ' + 'the detected spec "{0}"'.format(spec)) + raise InvalidSpecDetected(msg) + +.. _determine_spec_details: + +^^^^^^^^^^^^^^^^^^^^^^^^^ +Custom detection workflow +^^^^^^^^^^^^^^^^^^^^^^^^^ + +In the rare case when the mechanisms described so far don't fit the +detection of a package, the implementation of all the methods above +can be disregarded and instead a custom ``determine_spec_details`` +method can be implemented directly in the package class (note that +the definition of the ``executables`` attribute is still required): .. code-block:: python - def determine_spec_details(prefix, exes_in_prefix): + @classmethod + def determine_spec_details(cls, prefix, exes_in_prefix): # exes_in_prefix = a set of paths, each path is an executable # prefix = a prefix that is common to each path in exes_in_prefix @@ -4076,14 +4278,13 @@ The method ``determine_spec_details`` has the following signature: # the package. Return one or more Specs for each instance of the # package which is thought to be installed in the provided prefix -``determine_spec_details`` takes as parameters a set of discovered -executables (which match those specified by the user) as well as a -common prefix shared by all of those executables. The function must -return one or more Specs associated with the executables (it can also -return ``None`` to indicate that no provided executables are associated -with the package). +This method takes as input a set of discovered executables (which match +those specified by the user) as well as a common prefix shared by all +of those executables. The function must return one or more :py:class:`spack.spec.Spec` associated +with the executables (it can also return ``None`` to indicate that no +provided executables are associated with the package). -Say for example we have a package called ``foo-package`` which +As an example, consider a made-up package called ``foo-package`` which builds an executable called ``foo``. ``FooPackage`` would appear as follows: @@ -4110,7 +4311,9 @@ follows: exe = spack.util.executable.Executable(exe_path) output = exe('--version') version_str = ... # parse output for version string - return Spec('foo-package@{0}'.format(version_str)) + return Spec.from_detection( + 'foo-package@{0}'.format(version_str) + ) .. _package-lifecycle: |