Merge branch 'features/new-docs' into develop

3 files changed, 428 insertions, 41 deletions

diff --git a/lib/spack/docs/basic_usage.rst b/lib/spack/docs/basic_usage.rst
index 71f3da8610..0a0c2c678c 100644
--- a/lib/spack/docs/basic_usage.rst
+++ b/lib/spack/docs/basic_usage.rst
@@ -10,25 +10,6 @@ Only a small subset of commands are needed for typical usage.
 This section covers a small set of subcommands that should cover most
 general use cases for Spack.
 
-Getting Help
------------------------
-
-``spack help``
-~~~~~~~~~~~~~~~~~~~~~~
-
-The ``help`` subcommand will print out out a list of all of
-``spack``'s options and subcommands:
-
-.. command-output:: spack help
-
-Adding an argument, e.g. ``spack help <subcommand>``, will print out
-usage information for a particular subcommand:
-
-.. command-output:: spack help install
-
-Alternately, you can use ``spack -h`` in place of ``spack help``, or
-``spack <subcommand> -h`` to get help on a particular subcommand.
-
 
 Listing available packages
 ------------------------------
@@ -327,19 +308,19 @@ completely remove the directory in which the package was installed.
    spack uninstall mpich
 
 If there are still installed packages that depend on the package to be
-uninstalled, spack will issue a warning.  In general, it is safer to
-remove dependent packages *before* removing their dependencies.  Not
-doing so risks breaking packages on your system.  To remove a package
-without regard for its dependencies, run ``spack uninstall -f
-<package>`` to override the warning.
+uninstalled, spack will refuse to uninstall.  If you know what you're
+doing, you can override this with ``spack uninstall -f <package>``.
+However, running this risks breaking other installed packages. In
+general, it is safer to remove dependent packages *before* removing
+their dependencies.
 
 A line like ``spack uninstall mpich`` may be ambiguous, if multiple
 ``mpich`` configurations are installed.  For example, if both
 ``mpich@3.0.2`` and ``mpich@3.1`` are installed, it could refer to
 either one, and Spack cannot determine which one to uninstall.  Spack
-will ask you to provide a version number to remove any ambiguity.  For
-example, ``spack uninstall mpich@3.1`` is unambiguous in the
-above scenario.
+will ask you to provide a version number to remove the ambiguity.  For
+example, ``spack uninstall mpich@3.1`` is unambiguous in the above
+scenario.
 
 
 .. _sec-specs:
@@ -657,3 +638,236 @@ add a version specifier to the spec:
 
 Notice that the package versions that provide insufficient MPI
 versions are now filtered out.
+
+.. _shell-support:
+
+Environment Modules
+-------------------------------
+
+.. note::
+
+   Environment module support is currently experimental and should not
+   be considered a stable feature of Spack.  In particular, the
+   interface and/or generated module names may change in future
+   versions.
+
+Spack provides some limited integration with environment module
+systems to make it easier to use the packages it provides.
+
+You can enable shell support by sourcing some files in the
+``/share/spack`` directory.
+
+For ``bash`` or ``ksh``, run:
+
+.. code-block:: sh
+
+   . $SPACK_ROOT/share/spack/setup-env.sh
+
+For ``csh`` and ``tcsh`` run:
+
+.. code-block:: csh
+
+   setenv SPACK_ROOT /path/to/spack
+   source $SPACK_ROOT/share/spack/setup-env.csh
+
+You can put the above code in your ``.bashrc`` or ``.cshrc``, and
+Spack's shell support will be available on the command line.
+
+
+-------------------------------
+
+
+When you install a package with Spack, it automatically generates an
+environment module that lets you add the package to your environment.
+
+Currently, Spack supports the generation of `TCL Modules
+<http://wiki.tcl.tk/12999>`_ and `Dotkit
+<https://computing.llnl.gov/?set=jobs&page=dotkit>`_.  Generated
+module files for each of these systems can be found in these
+directories:
+
+  * ``$SPACK_ROOT/share/spack/modules``
+  * ``$SPACK_ROOT/share/spack/dotkit``
+
+The directories are automatically added to your ``MODULEPATH`` and
+``DK_NODE`` environment variables when you enable Spack's `shell
+support <shell-support_>`_.
+
+Using Modules & Dotkits
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you have shell support enabled you should be able to run either
+``module avail`` or ``use -l spack`` to see what modules/dotkits have
+been installed.  Here is sample output of those programs, showing lots
+of installed packages.
+
+  .. code-block:: sh
+
+     $ module avail
+
+     ------- /g/g21/gamblin2/src/spack/share/spack/modules/chaos_5_x86_64_ib --------
+     adept-utils@1.0%gcc@4.4.7-5adef8da   libelf@0.8.13%gcc@4.4.7
+     automaded@1.0%gcc@4.4.7-d9691bb0     libelf@0.8.13%intel@15.0.0
+     boost@1.55.0%gcc@4.4.7               mpc@1.0.2%gcc@4.4.7-559607f5
+     callpath@1.0.1%gcc@4.4.7-5dce4318    mpfr@3.1.2%gcc@4.4.7
+     dyninst@8.1.2%gcc@4.4.7-b040c20e     mpich@3.0.4%gcc@4.4.7
+     gcc@4.9.1%gcc@4.4.7-93ab98c5         mpich@3.0.4%gcc@4.9.0
+     gmp@6.0.0a%gcc@4.4.7                 mrnet@4.1.0%gcc@4.4.7-72b7881d
+     graphlib@2.0.0%gcc@4.4.7             netgauge@2.4.6%gcc@4.9.0-27912b7b
+     launchmon@1.0.1%gcc@4.4.7            stat@2.1.0%gcc@4.4.7-51101207
+     libNBC@1.1.1%gcc@4.9.0-27912b7b      sundials@2.5.0%gcc@4.9.0-27912b7b
+     libdwarf@20130729%gcc@4.4.7-b52fac98
+
+  .. code-block:: sh
+
+     $ use -l spack
+
+     spack ----------
+       adept-utils@1.0%gcc@4.4.7-5adef8da - adept-utils @1.0
+       automaded@1.0%gcc@4.4.7-d9691bb0 - automaded @1.0
+       boost@1.55.0%gcc@4.4.7 - boost @1.55.0
+       callpath@1.0.1%gcc@4.4.7-5dce4318 - callpath @1.0.1
+       dyninst@8.1.2%gcc@4.4.7-b040c20e - dyninst @8.1.2
+       gmp@6.0.0a%gcc@4.4.7 - gmp @6.0.0a
+       libNBC@1.1.1%gcc@4.9.0-27912b7b - libNBC @1.1.1
+       libdwarf@20130729%gcc@4.4.7-b52fac98 - libdwarf @20130729
+       libelf@0.8.13%gcc@4.4.7 - libelf @0.8.13
+       libelf@0.8.13%intel@15.0.0 - libelf @0.8.13
+       mpc@1.0.2%gcc@4.4.7-559607f5 - mpc @1.0.2
+       mpfr@3.1.2%gcc@4.4.7 - mpfr @3.1.2
+       mpich@3.0.4%gcc@4.4.7 - mpich @3.0.4
+       mpich@3.0.4%gcc@4.9.0 - mpich @3.0.4
+       netgauge@2.4.6%gcc@4.9.0-27912b7b - netgauge @2.4.6
+       sundials@2.5.0%gcc@4.9.0-27912b7b - sundials @2.5.0
+
+The names here should look familiar, they're the same ones from
+``spack find``.  You *can* use the names here directly.  For example,
+you could type either of these commands to load the callpath module
+(assuming dotkit and modules are installed):
+
+.. code-block:: sh
+
+   use callpath@1.0.1%gcc@4.4.7-5dce4318
+
+.. code-block:: sh
+
+   module load callpath@1.0.1%gcc@4.4.7-5dce4318
+
+Neither of these is particularly pretty, easy to remember, or
+easy to type.  Luckily, Spack has its own interface for using modules
+and dotkits.  You can use the same spec syntax you're used to:
+
+  =========================  ==========================
+  Modules                    Dotkit
+  =========================  ==========================
+  ``spack load <spec>``      ``spack use <spec>``
+  ``spack unload <spec>``    ``spack unuse <spec>``
+  =========================  ==========================
+
+And you can use the same shortened names you use everywhere else in
+Spack.  For example, this will add the ``mpich`` package built with
+``gcc`` to your path:
+
+.. code-block:: sh
+
+   $ spack install mpich %gcc@4.4.7
+
+   # ... wait for install ...
+
+   $ spack use mpich %gcc@4.4.7
+   Prepending: mpich@3.0.4%gcc@4.4.7 (ok)
+   $ which mpicc
+   ~/src/spack/opt/chaos_5_x86_64_ib/gcc@4.4.7/mpich@3.0.4/bin/mpicc
+
+Or, similarly with modules, you could type:
+
+.. code-block:: sh
+
+   $ spack load mpich %gcc@4.4.7
+
+These commands will add appropriate directories to your ``PATH``,
+``MANPATH``, and ``LD_LIBRARY_PATH``.  When you no longer want to use
+a package, you can type unload or unuse similarly:
+
+.. code-block:: sh
+
+   $ spack unload mpich %gcc@4.4.7    # modules
+   $ spack unuse mpich %gcc@4.4.7     # dotkit
+
+.. note::
+
+   These ``use``, ``unuse``, ``load``, and ``unload`` subcommands are
+   only available if you have enabled Spack's shell support *and* you
+   have dotkit or modules installed on your machine.
+
+Ambiguous module names
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+If a spec used with load/unload or use/unuse is ambiguous (i.e. more
+than one installed package matches it), then Spack will warn you:
+
+.. code-block:: sh
+
+   $ spack load libelf
+   ==> Error: Multiple matches for spec libelf.  Choose one:
+   libelf@0.8.13%gcc@4.4.7=chaos_5_x86_64_ib
+   libelf@0.8.13%intel@15.0.0=chaos_5_x86_64_ib
+
+You can either type the ``spack load`` command again with a fully
+qualified argument, or you can add just enough extra constraints to
+identify one package.  For example, above, the key differentiator is
+that one ``libelf`` is built with the Intel compiler, while the other
+used ``gcc``.  You could therefore just type:
+
+.. code-block:: sh
+
+   $ spack load libelf %intel
+
+To identify just the one built with the Intel compiler.
+
+
+Regenerating Module files
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Module and dotkit files are generated when packages are installed, and
+are placed in the following directories under the Spack root:
+
+  * ``$SPACK_ROOT/share/spack/modules``
+  * ``$SPACK_ROOT/share/spack/dotkit``
+
+Sometimes you may need to regenerate the modules files.  For example,
+if newer, fancier module support is added to Spack at some later date,
+you may want to regenerate all the modules to take advantage of these
+new features.
+
+``spack module refresh``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Running ``spack module refresh`` will remove the
+``share/spack/modules`` and ``share/spack/dotkit`` directories, then
+regenerate all module and dotkit files from scratch:
+
+.. code-block:: sh
+
+   $ spack module refresh
+   ==> Regenerating tcl module files.
+   ==> Regenerating dotkit module files.
+
+Getting Help
+-----------------------
+
+``spack help``
+~~~~~~~~~~~~~~~~~~~~~~
+
+If you don't find what you need here, the ``help`` subcommand will
+print out out a list of *all* of ``spack``'s options and subcommands:
+
+.. command-output:: spack help
+
+Adding an argument, e.g. ``spack help <subcommand>``, will print out
+usage information for a particular subcommand:
+
+.. command-output:: spack help install
+
+Alternately, you can use ``spack -h`` in place of ``spack help``, or
+``spack <subcommand> -h`` to get help on a particular subcommand.
diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst
index 1f95e56d2a..ec2ca4d099 100644
--- a/lib/spack/docs/packaging_guide.rst
+++ b/lib/spack/docs/packaging_guide.rst
@@ -768,9 +768,9 @@ information about the package, and to determine where to download its
 source code.
 
 Spack uses the tarball URL to extrapolate where to find other tarballs
-of the same package (e.g. in `spack-checksum`_, but this does not
-always work.  This section covers ways you can tell Spack to find
-tarballs elsewhere.
+of the same package (e.g. in `spack checksum <spack-checksum_>`_, but
+this does not always work.  This section covers ways you can tell
+Spack to find tarballs elsewhere.
 
 .. _attribute_list_url:
 
@@ -778,8 +778,9 @@ tarballs elsewhere.
 ~~~~~~~~~~~~~~~~~~~~~
 
 When spack tries to find available versions of packages (e.g. with
-`spack-checksum`_), it spiders the parent directory of the tarball in
-the ``url`` attribute.  For example, for libelf, the url is:
+`spack checksum <spack-checksum_>`_), it spiders the parent directory
+of the tarball in the ``url`` attribute.  For example, for libelf, the
+url is:
 
 .. code-block:: python
 
@@ -1335,8 +1336,8 @@ If your build system does *not* automatically pick these variables up
 from the environment, then you can simply pass them on the command
 line or use a patch as part of your build process to get the correct
 compilers into the project's build system.  There are also some file
-editing commands you can use -- these are described later in
-`filtering-files`_.
+editing commands you can use -- these are described later in the
+`section on file manipulation <file-manipulation_>`_.
 
 In addition to the compiler variables, these variables are set before
 entering ``install()`` so that packages can locate dependencies
@@ -1466,9 +1467,28 @@ yourself, e.g.:
 
 
 Most of the standard UNIX directory names are attributes on the
-``prefix`` object.  See :py:class:`spack.prefix.Prefix` for a full
-list.
-
+``prefix`` object.  Here is a full list:
+
+  =========================  ================================================
+  Prefix Attribute           Location
+  =========================  ================================================
+  ``prefix.bin``             ``$prefix/bin``
+  ``prefix.sbin``            ``$prefix/sbin``
+  ``prefix.etc``             ``$prefix/etc``
+  ``prefix.include``         ``$prefix/include``
+  ``prefix.lib``             ``$prefix/lib``
+  ``prefix.lib64``           ``$prefix/lib64``
+  ``prefix.libexec``         ``$prefix/libexec``
+  ``prefix.share``           ``$prefix/share``
+  ``prefix.doc``             ``$prefix/doc``
+  ``prefix.info``            ``$prefix/info``
+
+  ``prefix.man``             ``$prefix/man``
+  ``prefix.man[1-8]``        ``$prefix/man/man[1-8]``
+
+  ``prefix.share_man``       ``$prefix/share/man``
+  ``prefix.share_man[1-8]``  ``$prefix/share/man[1-8]``
+  =========================  ================================================
 
 .. _spec-objects:
 
@@ -1678,11 +1698,10 @@ method (the one without the ``@when`` decorator) will be called.
    the way decorators work.
 
 
-
 .. _shell-wrappers:
 
-Shell command wrappers
--------------------------
+Shell command functions
+----------------------------
 
 Recall the install method from ``libelf``:
 
@@ -1730,9 +1749,161 @@ to the ``make`` wrapper to disable parallel make.  In the ``libelf``
 package, this allows us to avoid race conditions in the library's
 build system.
 
+
+.. _file-manipulation:
+
+File manipulation functions
+------------------------------
+
+Many builds are not perfect. If a build lacks an install target, or if
+it does not use systems like CMake or autotools, which have standard
+ways of setting compilers and options, you may need to edit files or
+install some files yourself to get them working with Spack.
+
+You can do this with standard Python code, and Python has rich
+libraries with functions for file manipulation and filtering.  Spack
+also provides a number of convenience functions of its own to make
+your life even easier.  These functions are described in this section.
+
+All of the functions in this section can be included by simply
+running:
+
+.. code-block:: python
+
+   from spack import *
+
+This is already part of the boilerplate for packages created with
+``spack create`` or ``spack edit``.
+
+
+Filtering functions
+~~~~~~~~~~~~~~~~~~~~~~
+
+:py:func:`filter_file(regex, repl, *filenames, **kwargs) <spack.filter_file>`
+  Works like ``sed`` but with Python regular expression syntax.  Takes
+  a regular expression, a replacement, and a set of files.  ``repl``
+  can be a raw string or a callable function.  If it is a raw string,
+  it can contain ``\1``, ``\2``, etc. to refer to capture groups in
+  the regular expression.  If it is a callable, it is passed the
+  Python ``MatchObject`` and should return a suitable replacement
+  string for the particular match.
+
+  Examples:
+
+  #. Replacing ``#!/usr/bin/perl`` with ``#!/usr/bin/env perl`` in ``bib2xhtml``:
+
+     .. code-block:: python
+
+        filter_file(r'#!/usr/bin/perl',
+                    '#!/usr/bin/env perl', join_path(prefix.bin, 'bib2xhtml'))
+
+  #. Switching the compilers used by ``mpich``'s MPI wrapper scripts from
+     ``cc``, etc. to the compilers used by the Spack build:
+
+     .. code-block:: python
+
+        filter_file('CC="cc"', 'CC="%s"' % self.compiler.cc,
+                    join_path(prefix.bin, 'mpicc'))
+
+        filter_file('CXX="c++"', 'CXX="%s"' % self.compiler.cxx,
+                    join_path(prefix.bin, 'mpicxx'))
+
+:py:func:`change_sed_delimiter(old_delim, new_delim, *filenames) <spack.change_sed_delim>`
+    Some packages, like TAU, have a build system that can't install
+    into directories with, e.g. '@' in the name, because they use
+    hard-coded ``sed`` commands in their build.
+
+    ``change_sed_delimiter`` finds all ``sed`` search/replace commands
+    and change the delimiter.  e.g., if the file contains commands
+    that look like ``s///``, you can use this to change them to
+    ``s@@@``.
+
+    Example of changing ``s///`` to ``s@@@`` in TAU:
+
+    .. code-block:: python
+
+       change_sed_delimiter('@', ';', 'configure')
+       change_sed_delimiter('@', ';', 'utils/FixMakefile')
+       change_sed_delimiter('@', ';', 'utils/FixMakefile.sed.default')
+
+
+File functions
+~~~~~~~~~~~~~~~~~~~~~~
+
+:py:func:`ancestor(dir, n=1) <spack.ancestor>`
+  Get the n\ :sup:`th` ancestor of the directory ``dir``.
+
+:py:func:`can_access(path) <spack.can_access>`
+  True if we can read and write to the file at ``path``.  Same as
+  native python ``os.access(file_name, os.R_OK|os.W_OK)``.
+
+:py:func:`install(src, dest) <spack.install>`
+  Install a file to a particular location.  For example, install a
+  header into the ``include`` directory under the install ``prefix``:
+
+  .. code-block:: python
+
+     install('my-header.h', join_path(prefix.include))
+
+:py:func:`join_path(prefix, *args) <spack.join_path>` Like
+  ``os.path.join``, this joins paths using the OS path separator.
+  However, this version allows an arbitrary number of arguments, so
+  you can string together many path components.
+
+:py:func:`mkdirp(*paths) <spack.mkdirp>`
+  Create each of the directories in ``paths``, creating any parent
+  directories if they do not exist.
+
+:py:func:`working_dir(dirname, kwargs) <spack.working_dir>`
+  This is a Python `Context Manager
+  <https://docs.python.org/2/library/contextlib.html>`_ that makes it
+  easier to work with subdirectories in builds.  You use this with the
+  Python ``with`` statement to change into a working directory, and
+  when the with block is done, you change back to the original
+  directory.  Think of it as a safe ``pushd`` / ``popd`` combination,
+  where ``popd`` is guaranteed to be called at the end, even if
+  exceptions are thrown.
+
+  Example usage:
+
+  #. The ``libdwarf`` build first runs ``configure`` and ``make`` in a
+     subdirectory called ``libdwarf``.  It then implements the
+     installation code itself.  This is natural with ``working_dir``:
+
+     .. code-block:: python
+
+        with working_dir('libdwarf'):
+            configure("--prefix=" + prefix, "--enable-shared")
+            make()
+            install('libdwarf.a',  prefix.lib)
+
+  #. Many CMake builds require that you build "out of source", that
+     is, in a subdirectory.  You can handle creating and ``cd``'ing to
+     the subdirectory like the LLVM package does:
+
+     .. code-block:: python
+
+        with working_dir('spack-build', create=True):
+            cmake('..',
+                  '-DLLVM_REQUIRES_RTTI=1',
+                  '-DPYTHON_EXECUTABLE=/usr/bin/python',
+                  '-DPYTHON_INCLUDE_DIR=/usr/include/python2.6',
+                  '-DPYTHON_LIBRARY=/usr/lib64/libpython2.6.so',
+                  *std_cmake_args)
+            make()
+            make("install")
+
+     The ``create=True`` keyword argument causes the command to create
+     the directory if it does not exist.
+
+
+:py:func:`touch(path) <spack.touch>`
+  Create an empty file at ``path``.
+
+
 .. _pacakge-lifecycle:
 
-Useful Packaging Commands
+Package Workflow Commands
 ---------------------------------
 
 When you are building packages, you will likely not get things
diff --git a/lib/spack/llnl/util/filesystem.py b/lib/spack/llnl/util/filesystem.py
index a70111b915..dc722297ec 100644
--- a/lib/spack/llnl/util/filesystem.py
+++ b/lib/spack/llnl/util/filesystem.py
@@ -144,6 +144,7 @@ def expand_user(path):
 
 
 def mkdirp(*paths):
+    """Creates a directory, as well as parent directories if needed."""
     for path in paths:
         if not os.path.exists(path):
             os.makedirs(path)
@@ -163,6 +164,7 @@ def working_dir(dirname, **kwargs):
 
 
 def touch(path):
+    """Creates an empty file at the specified path."""
     with closing(open(path, 'a')) as file:
         os.utime(path, None)