cmake: use CMAKE_INSTALL_RPATH_USE_LINK_PATH (#29703)

* cmake: use CMAKE_INSTALL_RPATH_USE_LINK_PATH

Spack has a heuristic to add rpaths for packages it knows are required,
but it's really a heuristic, and it does not work when the dependencies
put their libraries in a different folder than `<prefix>/lib{64,}`.

CMake patches binaries after install with the "install rpaths", which by
default are provided by Spack and its heuristic through
`CMAKE_INSTALL_RPATH`.

CMake however knows better what libraries are effectively being linked
to, and has an option to include those in the install rpath too, through
`CMAKE_INSTALL_RPATH_USE_LINK_PATH`.

These two CMake options are complementary, repeated rpaths seem to be
filtered, and the "use link path" paths are appended to Spack's
heuristic "install rpath".

So, it seems like a good idea to enable "use link path" by default, so
that:
- `dlopen` by library name uses Spack's heuristic search paths
- linked libraries in non-standard locations within a prefix get an
rpath thanks to CMake.

* docs

2 files changed, 80 insertions, 31 deletions

diff --git a/lib/spack/docs/build_systems/cmakepackage.rst b/lib/spack/docs/build_systems/cmakepackage.rst
index c565375b14..9544a7df73 100644
--- a/lib/spack/docs/build_systems/cmakepackage.rst
+++ b/lib/spack/docs/build_systems/cmakepackage.rst
@@ -159,6 +159,85 @@ and CMake simply ignores the empty command line argument. For example the follow
 will generate ``'cmake' '-DEXAMPLE=ON' ...`` when `@2.0: +example` is met, but will
 result in ``'cmake' '' ...`` when the spec version is below ``2.0``.
 
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+CMake arguments provided by Spack
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following default arguments are controlled by Spack:
+
+
+``CMAKE_INSTALL_PREFIX``
+------------------------
+
+Is set to the the package's install directory.
+
+
+``CMAKE_PREFIX_PATH``
+---------------------
+
+CMake finds dependencies through calls to ``find_package()``, ``find_program()``,
+``find_library()``, ``find_file()``, and ``find_path()``, which use a list of search
+paths from ``CMAKE_PREFIX_PATH``. Spack sets this variable to a list of prefixes of the
+spec's transitive dependencies.
+
+For troubleshooting cases where CMake fails to find a dependency, add the
+``--debug-find`` flag to ``cmake_args``.
+
+``CMAKE_BUILD_TYPE``
+--------------------
+
+Every CMake-based package accepts a ``-DCMAKE_BUILD_TYPE`` flag to
+dictate which level of optimization to use. In order to ensure
+uniformity across packages, the ``CMakePackage`` base class adds
+a variant to control this:
+
+.. code-block:: python
+
+   variant('build_type', default='RelWithDebInfo',
+           description='CMake build type',
+           values=('Debug', 'Release', 'RelWithDebInfo', 'MinSizeRel'))
+
+However, not every CMake package accepts all four of these options.
+Grep the ``CMakeLists.txt`` file to see if the default values are
+missing or replaced. For example, the
+`dealii <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/dealii/package.py>`_
+package overrides the default variant with:
+
+.. code-block:: python
+
+   variant('build_type', default='DebugRelease',
+           description='The build type to build',
+           values=('Debug', 'Release', 'DebugRelease'))
+
+For more information on ``CMAKE_BUILD_TYPE``, see:
+https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html
+
+
+``CMAKE_INSTALL_RPATH`` and ``CMAKE_INSTALL_RPATH_USE_LINK_PATH=ON``
+--------------------------------------------------------------------
+
+CMake uses different RPATHs during the build and after installation, so that executables
+can locate the libraries they're linked to during the build, and installed executables
+do not have RPATHs to build directories. In Spack, we have to make sure that RPATHs are
+set properly after installation.
+
+Spack sets ``CMAKE_INSTALL_RPATH`` to a list of ``<prefix>/lib`` or ``<prefix>/lib64``
+directories of the spec's link-type dependencies. Apart from that, it sets
+``-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON``, which should add RPATHs for directories of
+linked libraries not in the directories covered by ``CMAKE_INSTALL_RPATH``.
+
+Usually it's enough to set only ``-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON``, but the
+reason to provide both options is that packages may dynamically open shared libraries,
+which CMake cannot detect. In those cases, the RPATHs from ``CMAKE_INSTALL_RPATH`` are
+used as search paths.
+
+.. note::
+
+   Some packages provide stub libraries, which contain an interface for linking without
+   an implementation. When using such libraries, it's best to override the option
+   ``-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=OFF`` in ``cmake_args``, so that stub libraries
+   are not used at runtime.
+
 
 ^^^^^^^^^^
 Generators
@@ -196,36 +275,6 @@ generators, but it should be simple to add support for alternative
 generators. For more information on CMake generators, see:
 https://cmake.org/cmake/help/latest/manual/cmake-generators.7.html
 
-^^^^^^^^^^^^^^^^
-CMAKE_BUILD_TYPE
-^^^^^^^^^^^^^^^^
-
-Every CMake-based package accepts a ``-DCMAKE_BUILD_TYPE`` flag to
-dictate which level of optimization to use. In order to ensure
-uniformity across packages, the ``CMakePackage`` base class adds
-a variant to control this:
-
-.. code-block:: python
-
-   variant('build_type', default='RelWithDebInfo',
-           description='CMake build type',
-           values=('Debug', 'Release', 'RelWithDebInfo', 'MinSizeRel'))
-
-However, not every CMake package accepts all four of these options.
-Grep the ``CMakeLists.txt`` file to see if the default values are
-missing or replaced. For example, the
-`dealii <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/dealii/package.py>`_
-package overrides the default variant with:
-
-.. code-block:: python
-
-   variant('build_type', default='DebugRelease',
-           description='The build type to build',
-           values=('Debug', 'Release', 'DebugRelease'))
-
-For more information on ``CMAKE_BUILD_TYPE``, see:
-https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html
-
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 CMakeLists.txt in a sub-directory
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
diff --git a/lib/spack/spack/build_systems/cmake.py b/lib/spack/spack/build_systems/cmake.py
index 6e6b5e20b0..f934947604 100644
--- a/lib/spack/spack/build_systems/cmake.py
+++ b/lib/spack/spack/build_systems/cmake.py
@@ -193,7 +193,7 @@ class CMakePackage(PackageBase):
 
         # Set up CMake rpath
         args.extend([
-            define('CMAKE_INSTALL_RPATH_USE_LINK_PATH', False),
+            define('CMAKE_INSTALL_RPATH_USE_LINK_PATH', True),
             define('CMAKE_INSTALL_RPATH',
                    spack.build_environment.get_rpaths(pkg)),
             define('CMAKE_PREFIX_PATH',