summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
-rw-r--r--lib/spack/docs/basic_usage.rst268
-rw-r--r--lib/spack/docs/packaging_guide.rst199
-rw-r--r--lib/spack/llnl/util/filesystem.py2
3 files changed, 428 insertions, 41 deletions
diff --git a/lib/spack/docs/basic_usage.rst b/lib/spack/docs/basic_usage.rst
index 71f3da8610..0a0c2c678c 100644
--- a/lib/spack/docs/basic_usage.rst
+++ b/lib/spack/docs/basic_usage.rst
@@ -10,25 +10,6 @@ Only a small subset of commands are needed for typical usage.
This section covers a small set of subcommands that should cover most
general use cases for Spack.
-Getting Help
------------------------
-
-``spack help``
-~~~~~~~~~~~~~~~~~~~~~~
-
-The ``help`` subcommand will print out out a list of all of
-``spack``'s options and subcommands:
-
-.. command-output:: spack help
-
-Adding an argument, e.g. ``spack help <subcommand>``, will print out
-usage information for a particular subcommand:
-
-.. command-output:: spack help install
-
-Alternately, you can use ``spack -h`` in place of ``spack help``, or
-``spack <subcommand> -h`` to get help on a particular subcommand.
-
Listing available packages
------------------------------
@@ -327,19 +308,19 @@ completely remove the directory in which the package was installed.
spack uninstall mpich
If there are still installed packages that depend on the package to be
-uninstalled, spack will issue a warning. In general, it is safer to
-remove dependent packages *before* removing their dependencies. Not
-doing so risks breaking packages on your system. To remove a package
-without regard for its dependencies, run ``spack uninstall -f
-<package>`` to override the warning.
+uninstalled, spack will refuse to uninstall. If you know what you're
+doing, you can override this with ``spack uninstall -f <package>``.
+However, running this risks breaking other installed packages. In
+general, it is safer to remove dependent packages *before* removing
+their dependencies.
A line like ``spack uninstall mpich`` may be ambiguous, if multiple
``mpich`` configurations are installed. For example, if both
``mpich@3.0.2`` and ``mpich@3.1`` are installed, it could refer to
either one, and Spack cannot determine which one to uninstall. Spack
-will ask you to provide a version number to remove any ambiguity. For
-example, ``spack uninstall mpich@3.1`` is unambiguous in the
-above scenario.
+will ask you to provide a version number to remove the ambiguity. For
+example, ``spack uninstall mpich@3.1`` is unambiguous in the above
+scenario.
.. _sec-specs:
@@ -657,3 +638,236 @@ add a version specifier to the spec:
Notice that the package versions that provide insufficient MPI
versions are now filtered out.
+
+.. _shell-support:
+
+Environment Modules
+-------------------------------
+
+.. note::
+
+ Environment module support is currently experimental and should not
+ be considered a stable feature of Spack. In particular, the
+ interface and/or generated module names may change in future
+ versions.
+
+Spack provides some limited integration with environment module
+systems to make it easier to use the packages it provides.
+
+You can enable shell support by sourcing some files in the
+``/share/spack`` directory.
+
+For ``bash`` or ``ksh``, run:
+
+.. code-block:: sh
+
+ . $SPACK_ROOT/share/spack/setup-env.sh
+
+For ``csh`` and ``tcsh`` run:
+
+.. code-block:: csh
+
+ setenv SPACK_ROOT /path/to/spack
+ source $SPACK_ROOT/share/spack/setup-env.csh
+
+You can put the above code in your ``.bashrc`` or ``.cshrc``, and
+Spack's shell support will be available on the command line.
+
+
+-------------------------------
+
+
+When you install a package with Spack, it automatically generates an
+environment module that lets you add the package to your environment.
+
+Currently, Spack supports the generation of `TCL Modules
+<http://wiki.tcl.tk/12999>`_ and `Dotkit
+<https://computing.llnl.gov/?set=jobs&page=dotkit>`_. Generated
+module files for each of these systems can be found in these
+directories:
+
+ * ``$SPACK_ROOT/share/spack/modules``
+ * ``$SPACK_ROOT/share/spack/dotkit``
+
+The directories are automatically added to your ``MODULEPATH`` and
+``DK_NODE`` environment variables when you enable Spack's `shell
+support <shell-support_>`_.
+
+Using Modules & Dotkits
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you have shell support enabled you should be able to run either
+``module avail`` or ``use -l spack`` to see what modules/dotkits have
+been installed. Here is sample output of those programs, showing lots
+of installed packages.
+
+ .. code-block:: sh
+
+ $ module avail
+
+ ------- /g/g21/gamblin2/src/spack/share/spack/modules/chaos_5_x86_64_ib --------
+ adept-utils@1.0%gcc@4.4.7-5adef8da libelf@0.8.13%gcc@4.4.7
+ automaded@1.0%gcc@4.4.7-d9691bb0 libelf@0.8.13%intel@15.0.0
+ boost@1.55.0%gcc@4.4.7 mpc@1.0.2%gcc@4.4.7-559607f5
+ callpath@1.0.1%gcc@4.4.7-5dce4318 mpfr@3.1.2%gcc@4.4.7
+ dyninst@8.1.2%gcc@4.4.7-b040c20e mpich@3.0.4%gcc@4.4.7
+ gcc@4.9.1%gcc@4.4.7-93ab98c5 mpich@3.0.4%gcc@4.9.0
+ gmp@6.0.0a%gcc@4.4.7 mrnet@4.1.0%gcc@4.4.7-72b7881d
+ graphlib@2.0.0%gcc@4.4.7 netgauge@2.4.6%gcc@4.9.0-27912b7b
+ launchmon@1.0.1%gcc@4.4.7 stat@2.1.0%gcc@4.4.7-51101207
+ libNBC@1.1.1%gcc@4.9.0-27912b7b sundials@2.5.0%gcc@4.9.0-27912b7b
+ libdwarf@20130729%gcc@4.4.7-b52fac98
+
+ .. code-block:: sh
+
+ $ use -l spack
+
+ spack ----------
+ adept-utils@1.0%gcc@4.4.7-5adef8da - adept-utils @1.0
+ automaded@1.0%gcc@4.4.7-d9691bb0 - automaded @1.0
+ boost@1.55.0%gcc@4.4.7 - boost @1.55.0
+ callpath@1.0.1%gcc@4.4.7-5dce4318 - callpath @1.0.1
+ dyninst@8.1.2%gcc@4.4.7-b040c20e - dyninst @8.1.2
+ gmp@6.0.0a%gcc@4.4.7 - gmp @6.0.0a
+ libNBC@1.1.1%gcc@4.9.0-27912b7b - libNBC @1.1.1
+ libdwarf@20130729%gcc@4.4.7-b52fac98 - libdwarf @20130729
+ libelf@0.8.13%gcc@4.4.7 - libelf @0.8.13
+ libelf@0.8.13%intel@15.0.0 - libelf @0.8.13
+ mpc@1.0.2%gcc@4.4.7-559607f5 - mpc @1.0.2
+ mpfr@3.1.2%gcc@4.4.7 - mpfr @3.1.2
+ mpich@3.0.4%gcc@4.4.7 - mpich @3.0.4
+ mpich@3.0.4%gcc@4.9.0 - mpich @3.0.4
+ netgauge@2.4.6%gcc@4.9.0-27912b7b - netgauge @2.4.6
+ sundials@2.5.0%gcc@4.9.0-27912b7b - sundials @2.5.0
+
+The names here should look familiar, they're the same ones from
+``spack find``. You *can* use the names here directly. For example,
+you could type either of these commands to load the callpath module
+(assuming dotkit and modules are installed):
+
+.. code-block:: sh
+
+ use callpath@1.0.1%gcc@4.4.7-5dce4318
+
+.. code-block:: sh
+
+ module load callpath@1.0.1%gcc@4.4.7-5dce4318
+
+Neither of these is particularly pretty, easy to remember, or
+easy to type. Luckily, Spack has its own interface for using modules
+and dotkits. You can use the same spec syntax you're used to:
+
+ ========================= ==========================
+ Modules Dotkit
+ ========================= ==========================
+ ``spack load <spec>`` ``spack use <spec>``
+ ``spack unload <spec>`` ``spack unuse <spec>``
+ ========================= ==========================
+
+And you can use the same shortened names you use everywhere else in
+Spack. For example, this will add the ``mpich`` package built with
+``gcc`` to your path:
+
+.. code-block:: sh
+
+ $ spack install mpich %gcc@4.4.7
+
+ # ... wait for install ...
+
+ $ spack use mpich %gcc@4.4.7
+ Prepending: mpich@3.0.4%gcc@4.4.7 (ok)
+ $ which mpicc
+ ~/src/spack/opt/chaos_5_x86_64_ib/gcc@4.4.7/mpich@3.0.4/bin/mpicc
+
+Or, similarly with modules, you could type:
+
+.. code-block:: sh
+
+ $ spack load mpich %gcc@4.4.7
+
+These commands will add appropriate directories to your ``PATH``,
+``MANPATH``, and ``LD_LIBRARY_PATH``. When you no longer want to use
+a package, you can type unload or unuse similarly:
+
+.. code-block:: sh
+
+ $ spack unload mpich %gcc@4.4.7 # modules
+ $ spack unuse mpich %gcc@4.4.7 # dotkit
+
+.. note::
+
+ These ``use``, ``unuse``, ``load``, and ``unload`` subcommands are
+ only available if you have enabled Spack's shell support *and* you
+ have dotkit or modules installed on your machine.
+
+Ambiguous module names
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+If a spec used with load/unload or use/unuse is ambiguous (i.e. more
+than one installed package matches it), then Spack will warn you:
+
+.. code-block:: sh
+
+ $ spack load libelf
+ ==> Error: Multiple matches for spec libelf. Choose one:
+ libelf@0.8.13%gcc@4.4.7=chaos_5_x86_64_ib
+ libelf@0.8.13%intel@15.0.0=chaos_5_x86_64_ib
+
+You can either type the ``spack load`` command again with a fully
+qualified argument, or you can add just enough extra constraints to
+identify one package. For example, above, the key differentiator is
+that one ``libelf`` is built with the Intel compiler, while the other
+used ``gcc``. You could therefore just type:
+
+.. code-block:: sh
+
+ $ spack load libelf %intel
+
+To identify just the one built with the Intel compiler.
+
+
+Regenerating Module files
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Module and dotkit files are generated when packages are installed, and
+are placed in the following directories under the Spack root:
+
+ * ``$SPACK_ROOT/share/spack/modules``
+ * ``$SPACK_ROOT/share/spack/dotkit``
+
+Sometimes you may need to regenerate the modules files. For example,
+if newer, fancier module support is added to Spack at some later date,
+you may want to regenerate all the modules to take advantage of these
+new features.
+
+``spack module refresh``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Running ``spack module refresh`` will remove the
+``share/spack/modules`` and ``share/spack/dotkit`` directories, then
+regenerate all module and dotkit files from scratch:
+
+.. code-block:: sh
+
+ $ spack module refresh
+ ==> Regenerating tcl module files.
+ ==> Regenerating dotkit module files.
+
+Getting Help
+-----------------------
+
+``spack help``
+~~~~~~~~~~~~~~~~~~~~~~
+
+If you don't find what you need here, the ``help`` subcommand will
+print out out a list of *all* of ``spack``'s options and subcommands:
+
+.. command-output:: spack help
+
+Adding an argument, e.g. ``spack help <subcommand>``, will print out
+usage information for a particular subcommand:
+
+.. command-output:: spack help install
+
+Alternately, you can use ``spack -h`` in place of ``spack help``, or
+``spack <subcommand> -h`` to get help on a particular subcommand.
diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst
index 1f95e56d2a..ec2ca4d099 100644
--- a/lib/spack/docs/packaging_guide.rst
+++ b/lib/spack/docs/packaging_guide.rst
@@ -768,9 +768,9 @@ information about the package, and to determine where to download its
source code.
Spack uses the tarball URL to extrapolate where to find other tarballs
-of the same package (e.g. in `spack-checksum`_, but this does not
-always work. This section covers ways you can tell Spack to find
-tarballs elsewhere.
+of the same package (e.g. in `spack checksum <spack-checksum_>`_, but
+this does not always work. This section covers ways you can tell
+Spack to find tarballs elsewhere.
.. _attribute_list_url:
@@ -778,8 +778,9 @@ tarballs elsewhere.
~~~~~~~~~~~~~~~~~~~~~
When spack tries to find available versions of packages (e.g. with
-`spack-checksum`_), it spiders the parent directory of the tarball in
-the ``url`` attribute. For example, for libelf, the url is:
+`spack checksum <spack-checksum_>`_), it spiders the parent directory
+of the tarball in the ``url`` attribute. For example, for libelf, the
+url is:
.. code-block:: python
@@ -1335,8 +1336,8 @@ If your build system does *not* automatically pick these variables up
from the environment, then you can simply pass them on the command
line or use a patch as part of your build process to get the correct
compilers into the project's build system. There are also some file
-editing commands you can use -- these are described later in
-`filtering-files`_.
+editing commands you can use -- these are described later in the
+`section on file manipulation <file-manipulation_>`_.
In addition to the compiler variables, these variables are set before
entering ``install()`` so that packages can locate dependencies
@@ -1466,9 +1467,28 @@ yourself, e.g.:
Most of the standard UNIX directory names are attributes on the
-``prefix`` object. See :py:class:`spack.prefix.Prefix` for a full
-list.
-
+``prefix`` object. Here is a full list:
+
+ ========================= ================================================
+ Prefix Attribute Location
+ ========================= ================================================
+ ``prefix.bin`` ``$prefix/bin``
+ ``prefix.sbin`` ``$prefix/sbin``
+ ``prefix.etc`` ``$prefix/etc``
+ ``prefix.include`` ``$prefix/include``
+ ``prefix.lib`` ``$prefix/lib``
+ ``prefix.lib64`` ``$prefix/lib64``
+ ``prefix.libexec`` ``$prefix/libexec``
+ ``prefix.share`` ``$prefix/share``
+ ``prefix.doc`` ``$prefix/doc``
+ ``prefix.info`` ``$prefix/info``
+
+ ``prefix.man`` ``$prefix/man``
+ ``prefix.man[1-8]`` ``$prefix/man/man[1-8]``
+
+ ``prefix.share_man`` ``$prefix/share/man``
+ ``prefix.share_man[1-8]`` ``$prefix/share/man[1-8]``
+ ========================= ================================================
.. _spec-objects:
@@ -1678,11 +1698,10 @@ method (the one without the ``@when`` decorator) will be called.
the way decorators work.
-
.. _shell-wrappers:
-Shell command wrappers
--------------------------
+Shell command functions
+----------------------------
Recall the install method from ``libelf``:
@@ -1730,9 +1749,161 @@ to the ``make`` wrapper to disable parallel make. In the ``libelf``
package, this allows us to avoid race conditions in the library's
build system.
+
+.. _file-manipulation:
+
+File manipulation functions
+------------------------------
+
+Many builds are not perfect. If a build lacks an install target, or if
+it does not use systems like CMake or autotools, which have standard
+ways of setting compilers and options, you may need to edit files or
+install some files yourself to get them working with Spack.
+
+You can do this with standard Python code, and Python has rich
+libraries with functions for file manipulation and filtering. Spack
+also provides a number of convenience functions of its own to make
+your life even easier. These functions are described in this section.
+
+All of the functions in this section can be included by simply
+running:
+
+.. code-block:: python
+
+ from spack import *
+
+This is already part of the boilerplate for packages created with
+``spack create`` or ``spack edit``.
+
+
+Filtering functions
+~~~~~~~~~~~~~~~~~~~~~~
+
+:py:func:`filter_file(regex, repl, *filenames, **kwargs) <spack.filter_file>`
+ Works like ``sed`` but with Python regular expression syntax. Takes
+ a regular expression, a replacement, and a set of files. ``repl``
+ can be a raw string or a callable function. If it is a raw string,
+ it can contain ``\1``, ``\2``, etc. to refer to capture groups in
+ the regular expression. If it is a callable, it is passed the
+ Python ``MatchObject`` and should return a suitable replacement
+ string for the particular match.
+
+ Examples:
+
+ #. Replacing ``#!/usr/bin/perl`` with ``#!/usr/bin/env perl`` in ``bib2xhtml``:
+
+ .. code-block:: python
+
+ filter_file(r'#!/usr/bin/perl',
+ '#!/usr/bin/env perl', join_path(prefix.bin, 'bib2xhtml'))
+
+ #. Switching the compilers used by ``mpich``'s MPI wrapper scripts from
+ ``cc``, etc. to the compilers used by the Spack build:
+
+ .. code-block:: python
+
+ filter_file('CC="cc"', 'CC="%s"' % self.compiler.cc,
+ join_path(prefix.bin, 'mpicc'))
+
+ filter_file('CXX="c++"', 'CXX="%s"' % self.compiler.cxx,
+ join_path(prefix.bin, 'mpicxx'))
+
+:py:func:`change_sed_delimiter(old_delim, new_delim, *filenames) <spack.change_sed_delim>`
+ Some packages, like TAU, have a build system that can't install
+ into directories with, e.g. '@' in the name, because they use
+ hard-coded ``sed`` commands in their build.
+
+ ``change_sed_delimiter`` finds all ``sed`` search/replace commands
+ and change the delimiter. e.g., if the file contains commands
+ that look like ``s///``, you can use this to change them to
+ ``s@@@``.
+
+ Example of changing ``s///`` to ``s@@@`` in TAU:
+
+ .. code-block:: python
+
+ change_sed_delimiter('@', ';', 'configure')
+ change_sed_delimiter('@', ';', 'utils/FixMakefile')
+ change_sed_delimiter('@', ';', 'utils/FixMakefile.sed.default')
+
+
+File functions
+~~~~~~~~~~~~~~~~~~~~~~
+
+:py:func:`ancestor(dir, n=1) <spack.ancestor>`
+ Get the n\ :sup:`th` ancestor of the directory ``dir``.
+
+:py:func:`can_access(path) <spack.can_access>`
+ True if we can read and write to the file at ``path``. Same as
+ native python ``os.access(file_name, os.R_OK|os.W_OK)``.
+
+:py:func:`install(src, dest) <spack.install>`
+ Install a file to a particular location. For example, install a
+ header into the ``include`` directory under the install ``prefix``:
+
+ .. code-block:: python
+
+ install('my-header.h', join_path(prefix.include))
+
+:py:func:`join_path(prefix, *args) <spack.join_path>` Like
+ ``os.path.join``, this joins paths using the OS path separator.
+ However, this version allows an arbitrary number of arguments, so
+ you can string together many path components.
+
+:py:func:`mkdirp(*paths) <spack.mkdirp>`
+ Create each of the directories in ``paths``, creating any parent
+ directories if they do not exist.
+
+:py:func:`working_dir(dirname, kwargs) <spack.working_dir>`
+ This is a Python `Context Manager
+ <https://docs.python.org/2/library/contextlib.html>`_ that makes it
+ easier to work with subdirectories in builds. You use this with the
+ Python ``with`` statement to change into a working directory, and
+ when the with block is done, you change back to the original
+ directory. Think of it as a safe ``pushd`` / ``popd`` combination,
+ where ``popd`` is guaranteed to be called at the end, even if
+ exceptions are thrown.
+
+ Example usage:
+
+ #. The ``libdwarf`` build first runs ``configure`` and ``make`` in a
+ subdirectory called ``libdwarf``. It then implements the
+ installation code itself. This is natural with ``working_dir``:
+
+ .. code-block:: python
+
+ with working_dir('libdwarf'):
+ configure("--prefix=" + prefix, "--enable-shared")
+ make()
+ install('libdwarf.a', prefix.lib)
+
+ #. Many CMake builds require that you build "out of source", that
+ is, in a subdirectory. You can handle creating and ``cd``'ing to
+ the subdirectory like the LLVM package does:
+
+ .. code-block:: python
+
+ with working_dir('spack-build', create=True):
+ cmake('..',
+ '-DLLVM_REQUIRES_RTTI=1',
+ '-DPYTHON_EXECUTABLE=/usr/bin/python',
+ '-DPYTHON_INCLUDE_DIR=/usr/include/python2.6',
+ '-DPYTHON_LIBRARY=/usr/lib64/libpython2.6.so',
+ *std_cmake_args)
+ make()
+ make("install")
+
+ The ``create=True`` keyword argument causes the command to create
+ the directory if it does not exist.
+
+
+:py:func:`touch(path) <spack.touch>`
+ Create an empty file at ``path``.
+
+
.. _pacakge-lifecycle:
-Useful Packaging Commands
+Package Workflow Commands
---------------------------------
When you are building packages, you will likely not get things
diff --git a/lib/spack/llnl/util/filesystem.py b/lib/spack/llnl/util/filesystem.py
index a70111b915..dc722297ec 100644
--- a/lib/spack/llnl/util/filesystem.py
+++ b/lib/spack/llnl/util/filesystem.py
@@ -144,6 +144,7 @@ def expand_user(path):
def mkdirp(*paths):
+ """Creates a directory, as well as parent directories if needed."""
for path in paths:
if not os.path.exists(path):
os.makedirs(path)
@@ -163,6 +164,7 @@ def working_dir(dirname, **kwargs):
def touch(path):
+ """Creates an empty file at the specified path."""
with closing(open(path, 'a')) as file:
os.utime(path, None)