summaryrefslogtreecommitdiff
path: root/lib
diff options
context:
space:
mode:
Diffstat (limited to 'lib')
-rw-r--r--lib/spack/docs/packaging_guide.rst730
1 files changed, 447 insertions, 283 deletions
diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst
index 9b08a7d498..75546d943e 100644
--- a/lib/spack/docs/packaging_guide.rst
+++ b/lib/spack/docs/packaging_guide.rst
@@ -405,70 +405,22 @@ For tarball downloads, Spack can currently support checksums using the
MD5, SHA-1, SHA-224, SHA-256, SHA-384, and SHA-512 algorithms. It
determines the algorithm to use based on the hash length.
------------------------
-Package Version Numbers
------------------------
-
-Most Spack versions are numeric, a tuple of integers; for example,
-``apex@0.1``, ``ferret@6.96`` or ``py-netcdf@1.2.3.1``. Spack knows
-how to compare and sort numeric versions.
-
-Some Spack versions involve slight extensions of numeric syntax; for
-example, ``py-sphinx-rtd-theme@0.1.10a0``. In this case, numbers are
-always considered to be "newer" than letters. This is for consistency
-with `RPM <https://bugzilla.redhat.com/show_bug.cgi?id=50977>`_.
-
-Spack versions may also be arbitrary non-numeric strings; any string
-here will suffice; for example, ``@develop``, ``@master``, ``@local``.
-The following rules determine the sort order of numeric
-vs. non-numeric versions:
-
-#. The non-numeric versions ``@develop`` is considered greatest (newest).
-
-#. Numeric versions are all less than ``@develop`` version, and are
- sorted numerically.
-
-#. All other non-numeric versions are less than numeric versions, and
- are sorted alphabetically.
-
-The logic behind this sort order is two-fold:
-
-#. Non-numeric versions are usually used for special cases while
- developing or debugging a piece of software. Keeping most of them
- less than numeric versions ensures that Spack choose numeric
- versions by default whenever possible.
-
-#. The most-recent development version of a package will usually be
- newer than any released numeric versions. This allows the
- ``develop`` version to satisfy dependencies like ``depends_on(abc,
- when="@x.y.z:")``
-
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Concretization Version Selection
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-When concretizing, many versions might match a user-supplied spec.
-For example, the spec ``python`` matches all available versions of the
-package ``python``. Similarly, ``python@3:`` matches all versions of
-Python3. Given a set of versions that match a spec, Spack
-concretization uses the following priorities to decide which one to
-use:
+---------------------
+Versions and fetching
+---------------------
-#. If the user provided a list of versions in ``packages.yaml``, the
- first matching version in that list will be used.
+The most straightforward way to add new versions to your package is to
+add a line like this in the package class:
-#. If one or more versions is specified as ``preferred=True``, in
- either ``packages.yaml`` or ``package.py``, the largest matching
- version will be used. ("Latest" is defined by the sort order
- above).
+.. code-block:: python
+ :linenos:
-#. If no preferences in particular are specified in the package or in
- ``packages.yaml``, then the largest matching non-develop version
- will be used. By avoiding ``@develop``, this prevents users from
- accidentally installing a ``@develop`` version.
+ class Foo(Package):
+ url = 'http://example.com/foo-1.0.tar.gz'
+ version('8.2.1', '4136d7b4c04df68b686570afa26988ac')
+ ...
-#. If all else fails and ``@develop`` is the only matching version, it
- will be used.
+Versions should be listed with the newest version first.
^^^^^^^^^^^^^
Date Versions
@@ -484,24 +436,6 @@ Alternately, you might use a hybrid release-version / date scheme.
For example, ``@1.3.2016.08.31`` would mean the version from the
``1.3`` branch, as of August 31, 2016.
-
--------------------
-Adding new versions
--------------------
-
-The most straightforward way to add new versions to your package is to
-add a line like this in the package class:
-
-.. code-block:: python
- :linenos:
-
- class Foo(Package):
- url = 'http://example.com/foo-1.0.tar.gz'
- version('8.2.1', '4136d7b4c04df68b686570afa26988ac')
- ...
-
-Versions should be listed with the newest version first.
-
^^^^^^^^^^^^
Version URLs
^^^^^^^^^^^^
@@ -566,6 +500,37 @@ way to guess the URL systematically.
When you supply a custom URL for a version, Spack uses that URL
*verbatim* and does not perform extrapolation.
+^^^^^^^^^^^^^^^^^^^^^
+PyPI and version URLs
+^^^^^^^^^^^^^^^^^^^^^
+
+In addition to their developer websites, many python packages are hosted at the
+`Python Package Index (PyPi) <https://pypi.python.org/pypi>`_. Although links to
+these individual files are typically `generated using a hash
+<https://bitbucket.org/pypa/pypi/issues/438>`_ it is often possible to find a
+reliable link of the format
+
+.. code-block:: sh
+
+ https://pypi.python.org/packages/source/<first letter of package>/<package>/<package>-<version>.<extension>
+
+Packages hosted on GitHub and the like are often developer versions that do not
+contain all of the files (e.g. configuration scripts) necessary to support
+compilation. For this reason it is ideal to link to a repository such as PyPi
+if possible.
+
+More recently, sources are being indexed at `pypi.io <https://pypi.io>`_ as
+well. Links obtained from this site follow a similar pattern, namely
+
+.. code-block:: sh
+
+ https://pypi.io/packages/source/<first letter of package>/<package>/<package>-<version>.<extension>
+
+These links currently redirect back to `pypi.python.org
+<https://pypi.python.org>`_, but this `may change in the future
+<https://bitbucket.org/pypa/pypi/issues/438#comment-27243225>`_.
+
+
^^^^^^^^^^^^^^^^^^^^^^^^
Skipping the expand step
^^^^^^^^^^^^^^^^^^^^^^^^
@@ -596,6 +561,79 @@ it executable, then runs it with some arguments.
installer = Executable(self.stage.archive_file)
installer('--prefix=%s' % prefix, 'arg1', 'arg2', 'etc.')
+^^^^^^^^^^^^^^^^
+Download caching
+^^^^^^^^^^^^^^^^
+
+Spack maintains a cache (described :ref:`here <caching>`) which saves files
+retrieved during package installations to avoid re-downloading in the case that
+a package is installed with a different specification (but the same version) or
+reinstalled on account of a change in the hashing scheme.
+
+^^^^^^^^^^^^^^^^^^
+Version comparison
+^^^^^^^^^^^^^^^^^^
+
+Most Spack versions are numeric, a tuple of integers; for example,
+``apex@0.1``, ``ferret@6.96`` or ``py-netcdf@1.2.3.1``. Spack knows
+how to compare and sort numeric versions.
+
+Some Spack versions involve slight extensions of numeric syntax; for
+example, ``py-sphinx-rtd-theme@0.1.10a0``. In this case, numbers are
+always considered to be "newer" than letters. This is for consistency
+with `RPM <https://bugzilla.redhat.com/show_bug.cgi?id=50977>`_.
+
+Spack versions may also be arbitrary non-numeric strings; any string
+here will suffice; for example, ``@develop``, ``@master``, ``@local``.
+The following rules determine the sort order of numeric
+vs. non-numeric versions:
+
+#. The non-numeric versions ``@develop`` is considered greatest (newest).
+
+#. Numeric versions are all less than ``@develop`` version, and are
+ sorted numerically.
+
+#. All other non-numeric versions are less than numeric versions, and
+ are sorted alphabetically.
+
+The logic behind this sort order is two-fold:
+
+#. Non-numeric versions are usually used for special cases while
+ developing or debugging a piece of software. Keeping most of them
+ less than numeric versions ensures that Spack choose numeric
+ versions by default whenever possible.
+
+#. The most-recent development version of a package will usually be
+ newer than any released numeric versions. This allows the
+ ``develop`` version to satisfy dependencies like ``depends_on(abc,
+ when="@x.y.z:")``
+
+^^^^^^^^^^^^^^^^^
+Version selection
+^^^^^^^^^^^^^^^^^
+
+When concretizing, many versions might match a user-supplied spec.
+For example, the spec ``python`` matches all available versions of the
+package ``python``. Similarly, ``python@3:`` matches all versions of
+Python3. Given a set of versions that match a spec, Spack
+concretization uses the following priorities to decide which one to
+use:
+
+#. If the user provided a list of versions in ``packages.yaml``, the
+ first matching version in that list will be used.
+
+#. If one or more versions is specified as ``preferred=True``, in
+ either ``packages.yaml`` or ``package.py``, the largest matching
+ version will be used. ("Latest" is defined by the sort order
+ above).
+
+#. If no preferences in particular are specified in the package or in
+ ``packages.yaml``, then the largest matching non-develop version
+ will be used. By avoiding ``@develop``, this prevents users from
+ accidentally installing a ``@develop`` version.
+
+#. If all else fails and ``@develop`` is the only matching version, it
+ will be used.
^^^^^^^^^^^^^
``spack md5``
@@ -693,9 +731,9 @@ versions. See the documentation on `attribute_list_url`_ and
.. _vcs-fetch:
-------------------------------
-Fetching from VCS repositories
-------------------------------
+-------------------------------
+Fetching from code repositories
+-------------------------------
For some packages, source code is provided in a Version Control System
(VCS) repository rather than in a tarball. Spack can fetch packages
@@ -793,9 +831,9 @@ Submodules
.. _github-fetch:
-""""""
+^^^^^^
GitHub
-""""""
+^^^^^^
If a project is hosted on GitHub, *any* valid Git branch, tag or hash
may be downloaded as a tarball. This is accomplished simply by
@@ -875,38 +913,8 @@ Fetching a revision
Subversion branches are handled as part of the directory structure, so
you can check out a branch or tag by changing the ``url``.
------------------------------------------
-Standard repositories for python packages
------------------------------------------
-
-In addition to their developer websites, many python packages are hosted at the
-`Python Package Index (PyPi) <https://pypi.python.org/pypi>`_. Although links to
-these individual files are typically `generated using a hash
-<https://bitbucket.org/pypa/pypi/issues/438>`_ it is often possible to find a
-reliable link of the format
-
-.. code-block:: sh
-
- https://pypi.python.org/packages/source/<first letter of package>/<package>/<package>-<version>.<extension>
-
-Packages hosted on GitHub and the like are often developer versions that do not
-contain all of the files (e.g. configuration scripts) necessary to support
-compilation. For this reason it is ideal to link to a repository such as PyPi
-if possible.
-
-More recently, sources are being indexed at `pypi.io <https://pypi.io>`_ as
-well. Links obtained from this site follow a similar pattern, namely
-
-.. code-block:: sh
-
- https://pypi.io/packages/source/<first letter of package>/<package>/<package>-<version>.<extension>
-
-These links currently redirect back to `pypi.python.org
-<https://pypi.python.org>`_, but this `may change in the future
-<https://bitbucket.org/pypa/pypi/issues/438#comment-27243225>`_.
-
-------------------------------------------------
-Expanding additional resources in the source tree
+Resources (expanding extra tarballs)
-------------------------------------------------
Some packages (most notably compilers) provide optional features if additional
@@ -926,15 +934,6 @@ Based on the keywords present among the arguments the appropriate ``FetchStrateg
will be used for the resource. The keyword ``destination`` is relative to the source
root of the package and should point to where the resource is to be expanded.
-------------------------------------------------------
-Automatic caching of files fetched during installation
-------------------------------------------------------
-
-Spack maintains a cache (described :ref:`here <caching>`) which saves files
-retrieved during package installations to avoid re-downloading in the case that
-a package is installed with a different specification (but the same version) or
-reinstalled on account of a change in the hashing scheme.
-
.. _license:
-----------------
@@ -1099,20 +1098,28 @@ structure like this:
package.py
ad_lustre_rwcontig_open_source.patch
-If you supply a URL instead of a filename, the patch will be fetched
-from the URL and then applied to your source code.
+If you supply a URL instead of a filename, you need to supply a checksum,
+like this:
-.. warning::
+.. code-block:: python
+
+ patch('http://www.nwchem-sw.org/images/Tddft_mxvec20.patch.gz',
+ md5='f91c6a04df56e228fe946291d2f38c9a')
- It is generally better to use a filename rather than a URL for your
- patch. Patches fetched from URLs are not currently checksummed,
- and adding checksums for them is tedious for the package builder.
- File patches go into the spack repository, which gives you git's
- integrity guarantees. URL patches may be removed in a future spack
- version.
+This directive provides an ``md5`` checksum. You can use other hashing
+algorihtms like ``sha256`` as well. The patch will be fetched from the
+URL, checked, and applied to your source code. You can use the ``spack
+md5`` command to generate a checksum for a patch file.
``patch`` can take two options keyword arguments. They are:
+""""""""""""""""""""""""""""""""""""""
+``md5``, ``sha256``, ``sha512``, etc.
+""""""""""""""""""""""""""""""""""""""
+
+Use one of these when you supply a patch to be downloaded from a remote
+site. The downloaded file will be validated using the given checksum.
+
""""""""
``when``
""""""""
@@ -1960,7 +1967,7 @@ See the :ref:`concretization-preferences` section for more details.
.. _install-method:
------------------
-Inconsistent Specs
+Conflicting Specs
------------------
Suppose a user needs to install package C, which depends on packages A
@@ -2151,7 +2158,7 @@ built with. These parameters give you access to this type of information.
.. _install-environment:
-----------------------
-The install environment
+The build environment
-----------------------
In general, you should not have to do much differently in your install
@@ -2170,6 +2177,17 @@ The remainder of the section covers the way Spack's build environment
works.
^^^^^^^^^^^^^^^^^^^^^
+Forking ``install()``
+^^^^^^^^^^^^^^^^^^^^^
+
+To give packagers free reign over their install environment, Spack forks
+a new process each time it invokes a package's ``install()`` method.
+This allows packages to have a sandboxed build environment, without
+impacting the environments ofother jobs that the main Spack process runs.
+Packages are free to change the environment or to modify Spack internals,
+because each ``install()`` call has its own dedicated process.
+
+^^^^^^^^^^^^^^^^^^^^^
Environment variables
^^^^^^^^^^^^^^^^^^^^^
@@ -2190,6 +2208,10 @@ The Compiler environment variables that Spack sets are:
``FC`` Fortran 90 and above compiler
============ ===============================
+Spack sets these variables so that they point to *compiler
+wrappers*. These are covered in :ref:`their own section
+<compiler-wrappers>` below.
+
All of these are standard variables respected by most build systems.
If your project uses ``Autotools`` or ``CMake``, then it should pick
them up automatically when you run ``configure`` or ``cmake`` in the
@@ -2237,158 +2259,102 @@ if you want to run commands in that environment to test them out, you
can use the :ref:`cmd-spack-env` command, documented
below.
-.. _compiler-wrappers:
-
^^^^^^^^^^^^^^^^^^^^^
-Compiler interceptors
+Failing the build
^^^^^^^^^^^^^^^^^^^^^
-As mentioned, ``CC``, ``CXX``, ``F77``, and ``FC`` are set to point to
-Spack's compiler wrappers. These are simply called ``cc``, ``c++``,
-``f77``, and ``f90``, and they live in ``$SPACK_ROOT/lib/spack/env``.
-
-``$SPACK_ROOT/lib/spack/env`` is added first in the ``PATH``
-environment variable when ``install()`` runs so that system compilers
-are not picked up instead.
+Sometimes you don't want a package to successfully install unless some
+condition is true. You can explicitly cause the build to fail from
+``install()`` by raising an ``InstallError``, for example:
-All of these compiler wrappers point to a single compiler wrapper
-script that figures out which *real* compiler it should be building
-with. This comes either from spec `concretization
-<abstract-and-concrete>`_ or from a user explicitly asking for a
-particular compiler using, e.g., ``%intel`` on the command line.
+.. code-block:: python
-In addition to invoking the right compiler, the compiler wrappers add
-flags to the compile line so that dependencies can be easily found.
-These flags are added for each dependency, if they exist:
+ if spec.architecture.startswith('darwin'):
+ raise InstallError('This package does not build on Mac OS X!')
-Compile-time library search paths
-* ``-L$dep_prefix/lib``
-* ``-L$dep_prefix/lib64``
+.. _shell-wrappers:
-Runtime library search paths (RPATHs)
-* ``$rpath_flag$dep_prefix/lib``
-* ``$rpath_flag$dep_prefix/lib64``
+^^^^^^^^^^^^^^^^^^^^^^^
+Shell command functions
+^^^^^^^^^^^^^^^^^^^^^^^
-Include search paths
-* ``-I$dep_prefix/include``
+Recall the install method from ``libelf``:
-An example of this would be the ``libdwarf`` build, which has one
-dependency: ``libelf``. Every call to ``cc`` in the ``libdwarf``
-build will have ``-I$LIBELF_PREFIX/include``,
-``-L$LIBELF_PREFIX/lib``, and ``$rpath_flag$LIBELF_PREFIX/lib``
-inserted on the command line. This is done transparently to the
-project's build system, which will just think it's using a system
-where ``libelf`` is readily available. Because of this, you **do
-not** have to insert extra ``-I``, ``-L``, etc. on the command line.
+.. literalinclude:: ../../../var/spack/repos/builtin/packages/libelf/package.py
+ :pyobject: Libelf.install
+ :linenos:
-Another useful consequence of this is that you often do *not* have to
-add extra parameters on the ``configure`` line to get autotools to
-find dependencies. The ``libdwarf`` install method just calls
-configure like this:
+Normally in Python, you'd have to write something like this in order
+to execute shell commands:
.. code-block:: python
- configure("--prefix=" + prefix)
+ import subprocess
+ subprocess.check_call('configure', '--prefix={0}'.format(prefix))
-Because of the ``-L`` and ``-I`` arguments, configure will
-successfully find ``libdwarf.h`` and ``libdwarf.so``, without the
-packager having to provide ``--with-libdwarf=/path/to/libdwarf`` on
-the command line.
+We've tried to make this a bit easier by providing callable wrapper
+objects for some shell commands. By default, ``configure``,
+``cmake``, and ``make`` wrappers are are provided, so you can call
+them more naturally in your package files.
-.. note::
+If you need other commands, you can use ``which`` to get them:
- For most compilers, ``$rpath_flag`` is ``-Wl,-rpath,``. However, NAG
- passes its flags to GCC instead of passing them directly to the linker.
- Therefore, its ``$rpath_flag`` is doubly wrapped: ``-Wl,-Wl,,-rpath,``.
- ``$rpath_flag`` can be overriden on a compiler specific basis in
- ``lib/spack/spack/compilers/$compiler.py``.
+.. code-block:: python
-The compiler wrappers also pass the compiler flags specified by the user from
-the command line (``cflags``, ``cxxflags``, ``fflags``, ``cppflags``, ``ldflags``,
-and/or ``ldlibs``). They do not override the canonical autotools flags with the
-same names (but in ALL-CAPS) that may be passed into the build by particularly
-challenging package scripts.
+ sed = which('sed')
+ sed('s/foo/bar/', filename)
+
+The ``which`` function will search the ``PATH`` for the application.
+
+Callable wrappers also allow spack to provide some special features.
+For example, in Spack, ``make`` is parallel by default, and Spack
+figures out the number of cores on your machine and passes an
+appropriate value for ``-j<numjobs>`` when it calls ``make`` (see the
+``parallel`` `package attribute <attribute_parallel>`). In
+a package file, you can supply a keyword argument, ``parallel=False``,
+to the ``make`` wrapper to disable parallel make. In the ``libelf``
+package, this allows us to avoid race conditions in the library's
+build system.
^^^^^^^^^^^^^^
Compiler flags
^^^^^^^^^^^^^^
-In rare circumstances such as compiling and running small unit tests, a package
-developer may need to know what are the appropriate compiler flags to enable
-features like ``OpenMP``, ``c++11``, ``c++14`` and alike. To that end the
-compiler classes in ``spack`` implement the following **properties**:
-``openmp_flag``, ``cxx11_flag``, ``cxx14_flag``, which can be accessed in a
-package by ``self.compiler.cxx11_flag`` and alike. Note that the implementation
-is such that if a given compiler version does not support this feature, an
-error will be produced. Therefore package developers can also use these properties
-to assert that a compiler supports the requested feature. This is handy when a
-package supports additional variants like
+In rare circumstances such as compiling and running small unit tests, a
+package developer may need to know what are the appropriate compiler
+flags to enable features like ``OpenMP``, ``c++11``, ``c++14`` and
+alike. To that end the compiler classes in ``spack`` implement the
+following **properties**: ``openmp_flag``, ``cxx11_flag``,
+``cxx14_flag``, which can be accessed in a package by
+``self.compiler.cxx11_flag`` and alike. Note that the implementation is
+such that if a given compiler version does not support this feature, an
+error will be produced. Therefore package developers can also use these
+properties to assert that a compiler supports the requested feature. This
+is handy when a package supports additional variants like
.. code-block:: python
variant('openmp', default=True, description="Enable OpenMP support.")
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Message Parsing Interface (MPI)
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-It is common for high performance computing software/packages to use ``MPI``.
-As a result of conretization, a given package can be built using different
-implementations of MPI such as ``Openmpi``, ``MPICH`` or ``IntelMPI``.
-In some scenarios, to configure a package, one has to provide it with appropriate MPI
-compiler wrappers such as ``mpicc``, ``mpic++``.
-However different implementations of ``MPI`` may have different names for those
-wrappers. In order to make package's ``install()`` method indifferent to the
-choice ``MPI`` implementation, each package which implements ``MPI`` sets up
-``self.spec.mpicc``, ``self.spec.mpicxx``, ``self.spec.mpifc`` and ``self.spec.mpif77``
-to point to ``C``, ``C++``, ``Fortran 90`` and ``Fortran 77`` ``MPI`` wrappers.
-Package developers are advised to use these variables, for example ``self.spec['mpi'].mpicc``
-instead of hard-coding ``join_path(self.spec['mpi'].prefix.bin, 'mpicc')`` for
-the reasons outlined above.
-
^^^^^^^^^^^^^^^^^^^^^^^^^
Blas and Lapack libraries
^^^^^^^^^^^^^^^^^^^^^^^^^
-Different packages provide implementation of ``Blas`` and ``Lapack`` routines.
-The names of the resulting static and/or shared libraries differ from package
-to package. In order to make the ``install()`` method independent of the
-choice of ``Blas`` implementation, each package which provides it
-sets up ``self.spec.blas_libs`` to point to the correct ``Blas`` libraries.
-The same applies to packages which provide ``Lapack``. Package developers are advised to
-use these variables, for example ``spec['blas'].blas_libs.joined()`` instead of
-hard-coding ``join_path(spec['blas'].prefix.lib, 'libopenblas.so')``.
-
-^^^^^^^^^^^^^^^^^^^^^
-Forking ``install()``
-^^^^^^^^^^^^^^^^^^^^^
-
-To give packagers free reign over their install environment, Spack
-forks a new process each time it invokes a package's ``install()``
-method. This allows packages to have their own completely sandboxed
-build environment, without impacting other jobs that the main Spack
-process runs. Packages are free to change the environment or to
-modify Spack internals, because each ``install()`` call has its own
-dedicated process.
+Different packages provide implementation of ``Blas`` and ``Lapack``
+routines. The names of the resulting static and/or shared libraries
+differ from package to package. In order to make the ``install()`` method
+independent of the choice of ``Blas`` implementation, each package which
+provides it sets up ``self.spec.blas_libs`` to point to the correct
+``Blas`` libraries. The same applies to packages which provide
+``Lapack``. Package developers are advised to use these variables, for
+example ``spec['blas'].blas_libs.joined()`` instead of hard-coding
+``join_path(spec['blas'].prefix.lib, 'libopenblas.so')``.
.. _prefix-objects:
------------------
-Failing the build
------------------
-
-Sometimes you don't want a package to successfully install unless some
-condition is true. You can explicitly cause the build to fail from
-``install()`` by raising an ``InstallError``, for example:
-
-.. code-block:: python
-
- if spec.architecture.startswith('darwin'):
- raise InstallError('This package does not build on Mac OS X!')
-
---------------
+^^^^^^^^^^^^^^^^^^^^^
Prefix objects
---------------
+^^^^^^^^^^^^^^^^^^^^^
Spack passes the ``prefix`` parameter to the install method so that
you can pass it to ``configure``, ``cmake``, or some other installer,
@@ -2657,50 +2623,248 @@ method (the one without the ``@when`` decorator) will be called.
versions. There's not much we can do to get around this because of
the way decorators work.
+.. _compiler-wrappers:
-.. _shell-wrappers:
+---------------------
+Compiler wrappers
+---------------------
------------------------
-Shell command functions
------------------------
+As mentioned, ``CC``, ``CXX``, ``F77``, and ``FC`` are set to point to
+Spack's compiler wrappers. These are simply called ``cc``, ``c++``,
+``f77``, and ``f90``, and they live in ``$SPACK_ROOT/lib/spack/env``.
-Recall the install method from ``libelf``:
+``$SPACK_ROOT/lib/spack/env`` is added first in the ``PATH``
+environment variable when ``install()`` runs so that system compilers
+are not picked up instead.
-.. literalinclude:: ../../../var/spack/repos/builtin/packages/libelf/package.py
- :pyobject: Libelf.install
- :linenos:
+All of these compiler wrappers point to a single compiler wrapper
+script that figures out which *real* compiler it should be building
+with. This comes either from spec `concretization
+<abstract-and-concrete>`_ or from a user explicitly asking for a
+particular compiler using, e.g., ``%intel`` on the command line.
-Normally in Python, you'd have to write something like this in order
-to execute shell commands:
+In addition to invoking the right compiler, the compiler wrappers add
+flags to the compile line so that dependencies can be easily found.
+These flags are added for each dependency, if they exist:
+
+Compile-time library search paths
+* ``-L$dep_prefix/lib``
+* ``-L$dep_prefix/lib64``
+
+Runtime library search paths (RPATHs)
+* ``$rpath_flag$dep_prefix/lib``
+* ``$rpath_flag$dep_prefix/lib64``
+
+Include search paths
+* ``-I$dep_prefix/include``
+
+An example of this would be the ``libdwarf`` build, which has one
+dependency: ``libelf``. Every call to ``cc`` in the ``libdwarf``
+build will have ``-I$LIBELF_PREFIX/include``,
+``-L$LIBELF_PREFIX/lib``, and ``$rpath_flag$LIBELF_PREFIX/lib``
+inserted on the command line. This is done transparently to the
+project's build system, which will just think it's using a system
+where ``libelf`` is readily available. Because of this, you **do
+not** have to insert extra ``-I``, ``-L``, etc. on the command line.
+
+Another useful consequence of this is that you often do *not* have to
+add extra parameters on the ``configure`` line to get autotools to
+find dependencies. The ``libdwarf`` install method just calls
+configure like this:
.. code-block:: python
- import subprocess
- subprocess.check_call('configure', '--prefix={0}'.format(prefix))
+ configure("--prefix=" + prefix)
-We've tried to make this a bit easier by providing callable wrapper
-objects for some shell commands. By default, ``configure``,
-``cmake``, and ``make`` wrappers are are provided, so you can call
-them more naturally in your package files.
+Because of the ``-L`` and ``-I`` arguments, configure will
+successfully find ``libdwarf.h`` and ``libdwarf.so``, without the
+packager having to provide ``--with-libdwarf=/path/to/libdwarf`` on
+the command line.
-If you need other commands, you can use ``which`` to get them:
+.. note::
+
+ For most compilers, ``$rpath_flag`` is ``-Wl,-rpath,``. However, NAG
+ passes its flags to GCC instead of passing them directly to the linker.
+ Therefore, its ``$rpath_flag`` is doubly wrapped: ``-Wl,-Wl,,-rpath,``.
+ ``$rpath_flag`` can be overriden on a compiler specific basis in
+ ``lib/spack/spack/compilers/$compiler.py``.
+
+The compiler wrappers also pass the compiler flags specified by the user from
+the command line (``cflags``, ``cxxflags``, ``fflags``, ``cppflags``, ``ldflags``,
+and/or ``ldlibs``). They do not override the canonical autotools flags with the
+same names (but in ALL-CAPS) that may be passed into the build by particularly
+challenging package scripts.
+
+---------------------
+MPI support in Spack
+---------------------
+
+It is common for high performance computing software/packages to use the
+Message Passing Interface ( ``MPI``). As a result of conretization, a
+given package can be built using different implementations of MPI such as
+``Openmpi``, ``MPICH`` or ``IntelMPI``. That is, when your package
+declares that it ``depends_on('mpi')``, it can be built with any of these
+``mpi`` implementations. In some scenarios, to configure a package, one
+has to provide it with appropriate MPI compiler wrappers such as
+``mpicc``, ``mpic++``. However different implementations of ``MPI`` may
+have different names for those wrappers.
+
+Spack provides an idiomatic way to use MPI compilers in your package. To
+use MPI wrappers to compile your whole build, do this in your
+``install()`` method:
.. code-block:: python
- sed = which('sed')
- sed('s/foo/bar/', filename)
+ env['CC'] = spec['mpi'].mpicc
+ env['CXX'] = spec['mpi'].mpicxx
+ env['F77'] = spec['mpi'].mpif77
+ env['FC'] = spec['mpi'].mpifc
-The ``which`` function will search the ``PATH`` for the application.
+That's all. A longer explanation of why this works is below.
-Callable wrappers also allow spack to provide some special features.
-For example, in Spack, ``make`` is parallel by default, and Spack
-figures out the number of cores on your machine and passes an
-appropriate value for ``-j<numjobs>`` when it calls ``make`` (see the
-``parallel`` `package attribute <attribute_parallel>`). In
-a package file, you can supply a keyword argument, ``parallel=False``,
-to the ``make`` wrapper to disable parallel make. In the ``libelf``
-package, this allows us to avoid race conditions in the library's
-build system.
+We don't try to force any particular build method on packagers. The
+decision to use MPI wrappers depends on the way the package is written,
+on common practice, and on "what works". Loosely, There are three types
+of MPI builds:
+
+ 1. Some build systems work well without the wrappers and can treat MPI
+ as an external library, where the person doing the build has to
+ supply includes/libs/etc. This is fairly uncommon.
+
+ 2. Others really want the wrappers and assume you're using an MPI
+ "compiler" – i.e., they have no mechanism to add MPI
+ includes/libraries/etc.
+
+ 3. CMake's ``FindMPI`` needs the compiler wrappers, but it uses them to
+ extract ``–I`` / ``-L`` / ``-D`` arguments, then treats MPI like a
+ regular library.
+
+Note that some CMake builds fall into case 2 because they either don't
+know about or don't like CMake's ``FindMPI`` support – they just assume
+an MPI compiler. Also, some autotools builds fall into case 3 (e.g. `here
+is an autotools version of CMake's FindMPI
+<https://github.com/tgamblin/libra/blob/master/m4/lx_find_mpi.m4>`_).
+
+Given all of this, we leave the use of the wrappers up to the packager.
+Spack will support all three ways of building MPI packages.
+
+^^^^^^^^^^^^^^^^^^^^^
+Packaging Conventions
+^^^^^^^^^^^^^^^^^^^^^
+
+As mentioned above, in the ``install()`` method, ``CC``, ``CXX``,
+``F77``, and ``FC`` point to Spack's wrappers around the chosen compiler.
+Spack's wrappers are not the MPI compiler wrappers, though they do
+automatically add ``–I``, ``–L``, and ``–Wl,-rpath`` args for
+dependencies in a similar way. The MPI wrappers are a bit different in
+that they also add ``-l`` arguments for the MPI libraries, and some add
+special ``-D`` arguments to trigger build options in MPI programs.
+
+For case 1 above, you generally don't need to do more than patch your
+Makefile or add configure args as you normally would.
+
+For case 3, you don't need to do much of anything, as Spack puts the MPI
+compiler wrappers in the PATH, and the build will find them and
+interrogate them.
+
+For case 2, things are a bit more complicated, as you'll need to tell the
+build to use the MPI compiler wrappers instead of Spack's compiler
+wrappers. All it takes some lines like this:
+
+.. code-block:: python
+
+ env['CC'] = spec['mpi'].mpicc
+ env['CXX'] = spec['mpi'].mpicxx
+ env['F77'] = spec['mpi'].mpif77
+ env['FC'] = spec['mpi'].mpifc
+
+Or, if you pass CC, CXX, etc. directly to your build with, e.g.,
+`--with-cc=<path>`, you'll want to substitute `spec['mpi'].mpicc` in
+there instead, e.g.:
+
+.. code-block:: python
+
+ configure('—prefix=%s' % prefix,
+ '—with-cc=%s' % spec['mpi'].mpicc)
+
+Now, you may think that doing this will lose the includes, library paths,
+and RPATHs that Spack's compiler wrapper get you, but we've actually set
+things up so that the MPI compiler wrappers use Spack's compiler wrappers
+when run from within Spack. So using the MPI wrappers should really be as
+simple as the code above.
+
+^^^^^^^^^^^^^^^^^^^^^
+``spec['mpi']``
+^^^^^^^^^^^^^^^^^^^^^
+
+Ok, so how does all this work?
+
+If your package has a virtual dependency like ``mpi``, then referring to
+``spec['mpi']`` within ``install()`` will get you the concrete ``mpi``
+implementation in your dependency DAG. That is a spec object just like
+the one passed to install, only the MPI implementations all set some
+additional properties on it to help you out. E.g., in mvapich2, you'll
+find this:
+
+.. code-block:: python
+
+ def setup_dependent_package(self, module, dep_spec):
+ self.spec.mpicc = join_path(self.prefix.bin, 'mpicc')
+ # … etc …
+
+That code allows the mvapich2 package to associate an ``mpicc`` property
+with the ``mvapich2`` node in the DAG, so that dependents can access it.
+``openmpi`` and ``mpich`` do similar things. So, no matter what MPI
+you're using, spec['mpi'].mpicc gets you the location of the MPI
+compilers. This allows us to have a fairly simple polymorphic interface
+for information about virtual dependencies like MPI.
+
+^^^^^^^^^^^^^^^^^^^^^
+Wrapping wrappers
+^^^^^^^^^^^^^^^^^^^^^
+
+Spack likes to use its own compiler wrappers to make it easy to add
+``RPATHs`` to builds, and to try hard to ensure that your builds use the
+right dependencies. This doesn't play nicely by default with MPI, so we
+have to do a couple tricks.
+
+ 1. If we build MPI with Spack's wrappers, mpicc and friends will be
+ installed with hard-coded paths to Spack's wrappers, and using them
+ from outside of Spack will fail because they only work within Spack.
+ To fix this, we patch mpicc and friends to use the regular
+ compilers. Look at the filter_compilers method in mpich, openmpi,
+ or mvapich2 for details.
+
+ 2. We still want to use the Spack compiler wrappers when Spack is
+ calling mpicc. Luckily, wrappers in all mainstream MPI
+ implementations provide environment variables that allow us to
+ dynamically set the compiler to be used by mpicc, mpicxx, etc.
+ Denis pasted some code from this below – Spack's build environment
+ sets ``MPICC``, ``MPICXX``, etc. for mpich derivatives and
+ ``OMPI_CC``, ``OMPI_CXX``, etc. for OpenMPI. This makes the MPI
+ compiler wrappers use the Spack compiler wrappers so that your
+ dependencies still get proper RPATHs even if you use the MPI
+ wrappers.
+
+^^^^^^^^^^^^^^^^^^^^^
+MPI on Cray machines
+^^^^^^^^^^^^^^^^^^^^^
+
+The Cray programming environment notably uses ITS OWN compiler wrappers,
+which function like MPI wrappers. On Cray systems, the ``CC``, ``cc``,
+and ``ftn`` wrappers ARE the MPI compiler wrappers, and it's assumed that
+you'll use them for all of your builds. So on Cray we don't bother with
+``mpicc``, ``mpicxx``, etc, Spack MPI implementations set
+``spec['mpi'].mpicc`` to point to Spack's wrappers, which wrap the Cray
+wrappers, which wrap the regular compilers and include MPI flags. That
+may seem complicated, but for packagers, that means the same code for
+using MPI wrappers will work, even on even on a Cray:
+
+.. code-block:: python
+
+ env['CC'] = spec['mpi'].mpicc
+
+This is because on Cray, ``spec['mpi'].mpicc`` is just ``spack_cc``.
.. _sanity-checks:
@@ -2963,9 +3127,9 @@ File functions
.. _package-lifecycle:
------------------------
-Coding Style Guidelines
------------------------
+-----------------------------
+Style guidelines for packages
+-----------------------------
The following guidelines are provided, in the interests of making
Spack packages work in a consistent manner: