diff options
| -rw-r--r-- | lib/spack/docs/basic_usage.rst | 268 | ||||
| -rw-r--r-- | lib/spack/docs/packaging_guide.rst | 199 | ||||
| -rw-r--r-- | lib/spack/llnl/util/filesystem.py | 2 |
3 files changed, 428 insertions, 41 deletions
diff --git a/lib/spack/docs/basic_usage.rst b/lib/spack/docs/basic_usage.rst index 71f3da8610..0a0c2c678c 100644 --- a/lib/spack/docs/basic_usage.rst +++ b/lib/spack/docs/basic_usage.rst @@ -10,25 +10,6 @@ Only a small subset of commands are needed for typical usage. This section covers a small set of subcommands that should cover most general use cases for Spack. -Getting Help ------------------------ - -``spack help`` -~~~~~~~~~~~~~~~~~~~~~~ - -The ``help`` subcommand will print out out a list of all of -``spack``'s options and subcommands: - -.. command-output:: spack help - -Adding an argument, e.g. ``spack help <subcommand>``, will print out -usage information for a particular subcommand: - -.. command-output:: spack help install - -Alternately, you can use ``spack -h`` in place of ``spack help``, or -``spack <subcommand> -h`` to get help on a particular subcommand. - Listing available packages ------------------------------ @@ -327,19 +308,19 @@ completely remove the directory in which the package was installed. spack uninstall mpich If there are still installed packages that depend on the package to be -uninstalled, spack will issue a warning. In general, it is safer to -remove dependent packages *before* removing their dependencies. Not -doing so risks breaking packages on your system. To remove a package -without regard for its dependencies, run ``spack uninstall -f -<package>`` to override the warning. +uninstalled, spack will refuse to uninstall. If you know what you're +doing, you can override this with ``spack uninstall -f <package>``. +However, running this risks breaking other installed packages. In +general, it is safer to remove dependent packages *before* removing +their dependencies. A line like ``spack uninstall mpich`` may be ambiguous, if multiple ``mpich`` configurations are installed. For example, if both ``mpich@3.0.2`` and ``mpich@3.1`` are installed, it could refer to either one, and Spack cannot determine which one to uninstall. Spack -will ask you to provide a version number to remove any ambiguity. For -example, ``spack uninstall mpich@3.1`` is unambiguous in the -above scenario. +will ask you to provide a version number to remove the ambiguity. For +example, ``spack uninstall mpich@3.1`` is unambiguous in the above +scenario. .. _sec-specs: @@ -657,3 +638,236 @@ add a version specifier to the spec: Notice that the package versions that provide insufficient MPI versions are now filtered out. + +.. _shell-support: + +Environment Modules +------------------------------- + +.. note:: + + Environment module support is currently experimental and should not + be considered a stable feature of Spack. In particular, the + interface and/or generated module names may change in future + versions. + +Spack provides some limited integration with environment module +systems to make it easier to use the packages it provides. + +You can enable shell support by sourcing some files in the +``/share/spack`` directory. + +For ``bash`` or ``ksh``, run: + +.. code-block:: sh + + . $SPACK_ROOT/share/spack/setup-env.sh + +For ``csh`` and ``tcsh`` run: + +.. code-block:: csh + + setenv SPACK_ROOT /path/to/spack + source $SPACK_ROOT/share/spack/setup-env.csh + +You can put the above code in your ``.bashrc`` or ``.cshrc``, and +Spack's shell support will be available on the command line. + + +------------------------------- + + +When you install a package with Spack, it automatically generates an +environment module that lets you add the package to your environment. + +Currently, Spack supports the generation of `TCL Modules +<http://wiki.tcl.tk/12999>`_ and `Dotkit +<https://computing.llnl.gov/?set=jobs&page=dotkit>`_. Generated +module files for each of these systems can be found in these +directories: + + * ``$SPACK_ROOT/share/spack/modules`` + * ``$SPACK_ROOT/share/spack/dotkit`` + +The directories are automatically added to your ``MODULEPATH`` and +``DK_NODE`` environment variables when you enable Spack's `shell +support <shell-support_>`_. + +Using Modules & Dotkits +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you have shell support enabled you should be able to run either +``module avail`` or ``use -l spack`` to see what modules/dotkits have +been installed. Here is sample output of those programs, showing lots +of installed packages. + + .. code-block:: sh + + $ module avail + + ------- /g/g21/gamblin2/src/spack/share/spack/modules/chaos_5_x86_64_ib -------- + adept-utils@1.0%gcc@4.4.7-5adef8da libelf@0.8.13%gcc@4.4.7 + automaded@1.0%gcc@4.4.7-d9691bb0 libelf@0.8.13%intel@15.0.0 + boost@1.55.0%gcc@4.4.7 mpc@1.0.2%gcc@4.4.7-559607f5 + callpath@1.0.1%gcc@4.4.7-5dce4318 mpfr@3.1.2%gcc@4.4.7 + dyninst@8.1.2%gcc@4.4.7-b040c20e mpich@3.0.4%gcc@4.4.7 + gcc@4.9.1%gcc@4.4.7-93ab98c5 mpich@3.0.4%gcc@4.9.0 + gmp@6.0.0a%gcc@4.4.7 mrnet@4.1.0%gcc@4.4.7-72b7881d + graphlib@2.0.0%gcc@4.4.7 netgauge@2.4.6%gcc@4.9.0-27912b7b + launchmon@1.0.1%gcc@4.4.7 stat@2.1.0%gcc@4.4.7-51101207 + libNBC@1.1.1%gcc@4.9.0-27912b7b sundials@2.5.0%gcc@4.9.0-27912b7b + libdwarf@20130729%gcc@4.4.7-b52fac98 + + .. code-block:: sh + + $ use -l spack + + spack ---------- + adept-utils@1.0%gcc@4.4.7-5adef8da - adept-utils @1.0 + automaded@1.0%gcc@4.4.7-d9691bb0 - automaded @1.0 + boost@1.55.0%gcc@4.4.7 - boost @1.55.0 + callpath@1.0.1%gcc@4.4.7-5dce4318 - callpath @1.0.1 + dyninst@8.1.2%gcc@4.4.7-b040c20e - dyninst @8.1.2 + gmp@6.0.0a%gcc@4.4.7 - gmp @6.0.0a + libNBC@1.1.1%gcc@4.9.0-27912b7b - libNBC @1.1.1 + libdwarf@20130729%gcc@4.4.7-b52fac98 - libdwarf @20130729 + libelf@0.8.13%gcc@4.4.7 - libelf @0.8.13 + libelf@0.8.13%intel@15.0.0 - libelf @0.8.13 + mpc@1.0.2%gcc@4.4.7-559607f5 - mpc @1.0.2 + mpfr@3.1.2%gcc@4.4.7 - mpfr @3.1.2 + mpich@3.0.4%gcc@4.4.7 - mpich @3.0.4 + mpich@3.0.4%gcc@4.9.0 - mpich @3.0.4 + netgauge@2.4.6%gcc@4.9.0-27912b7b - netgauge @2.4.6 + sundials@2.5.0%gcc@4.9.0-27912b7b - sundials @2.5.0 + +The names here should look familiar, they're the same ones from +``spack find``. You *can* use the names here directly. For example, +you could type either of these commands to load the callpath module +(assuming dotkit and modules are installed): + +.. code-block:: sh + + use callpath@1.0.1%gcc@4.4.7-5dce4318 + +.. code-block:: sh + + module load callpath@1.0.1%gcc@4.4.7-5dce4318 + +Neither of these is particularly pretty, easy to remember, or +easy to type. Luckily, Spack has its own interface for using modules +and dotkits. You can use the same spec syntax you're used to: + + ========================= ========================== + Modules Dotkit + ========================= ========================== + ``spack load <spec>`` ``spack use <spec>`` + ``spack unload <spec>`` ``spack unuse <spec>`` + ========================= ========================== + +And you can use the same shortened names you use everywhere else in +Spack. For example, this will add the ``mpich`` package built with +``gcc`` to your path: + +.. code-block:: sh + + $ spack install mpich %gcc@4.4.7 + + # ... wait for install ... + + $ spack use mpich %gcc@4.4.7 + Prepending: mpich@3.0.4%gcc@4.4.7 (ok) + $ which mpicc + ~/src/spack/opt/chaos_5_x86_64_ib/gcc@4.4.7/mpich@3.0.4/bin/mpicc + +Or, similarly with modules, you could type: + +.. code-block:: sh + + $ spack load mpich %gcc@4.4.7 + +These commands will add appropriate directories to your ``PATH``, +``MANPATH``, and ``LD_LIBRARY_PATH``. When you no longer want to use +a package, you can type unload or unuse similarly: + +.. code-block:: sh + + $ spack unload mpich %gcc@4.4.7 # modules + $ spack unuse mpich %gcc@4.4.7 # dotkit + +.. note:: + + These ``use``, ``unuse``, ``load``, and ``unload`` subcommands are + only available if you have enabled Spack's shell support *and* you + have dotkit or modules installed on your machine. + +Ambiguous module names +~~~~~~~~~~~~~~~~~~~~~~~~ + +If a spec used with load/unload or use/unuse is ambiguous (i.e. more +than one installed package matches it), then Spack will warn you: + +.. code-block:: sh + + $ spack load libelf + ==> Error: Multiple matches for spec libelf. Choose one: + libelf@0.8.13%gcc@4.4.7=chaos_5_x86_64_ib + libelf@0.8.13%intel@15.0.0=chaos_5_x86_64_ib + +You can either type the ``spack load`` command again with a fully +qualified argument, or you can add just enough extra constraints to +identify one package. For example, above, the key differentiator is +that one ``libelf`` is built with the Intel compiler, while the other +used ``gcc``. You could therefore just type: + +.. code-block:: sh + + $ spack load libelf %intel + +To identify just the one built with the Intel compiler. + + +Regenerating Module files +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Module and dotkit files are generated when packages are installed, and +are placed in the following directories under the Spack root: + + * ``$SPACK_ROOT/share/spack/modules`` + * ``$SPACK_ROOT/share/spack/dotkit`` + +Sometimes you may need to regenerate the modules files. For example, +if newer, fancier module support is added to Spack at some later date, +you may want to regenerate all the modules to take advantage of these +new features. + +``spack module refresh`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Running ``spack module refresh`` will remove the +``share/spack/modules`` and ``share/spack/dotkit`` directories, then +regenerate all module and dotkit files from scratch: + +.. code-block:: sh + + $ spack module refresh + ==> Regenerating tcl module files. + ==> Regenerating dotkit module files. + +Getting Help +----------------------- + +``spack help`` +~~~~~~~~~~~~~~~~~~~~~~ + +If you don't find what you need here, the ``help`` subcommand will +print out out a list of *all* of ``spack``'s options and subcommands: + +.. command-output:: spack help + +Adding an argument, e.g. ``spack help <subcommand>``, will print out +usage information for a particular subcommand: + +.. command-output:: spack help install + +Alternately, you can use ``spack -h`` in place of ``spack help``, or +``spack <subcommand> -h`` to get help on a particular subcommand. diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst index 1f95e56d2a..ec2ca4d099 100644 --- a/lib/spack/docs/packaging_guide.rst +++ b/lib/spack/docs/packaging_guide.rst @@ -768,9 +768,9 @@ information about the package, and to determine where to download its source code. Spack uses the tarball URL to extrapolate where to find other tarballs -of the same package (e.g. in `spack-checksum`_, but this does not -always work. This section covers ways you can tell Spack to find -tarballs elsewhere. +of the same package (e.g. in `spack checksum <spack-checksum_>`_, but +this does not always work. This section covers ways you can tell +Spack to find tarballs elsewhere. .. _attribute_list_url: @@ -778,8 +778,9 @@ tarballs elsewhere. ~~~~~~~~~~~~~~~~~~~~~ When spack tries to find available versions of packages (e.g. with -`spack-checksum`_), it spiders the parent directory of the tarball in -the ``url`` attribute. For example, for libelf, the url is: +`spack checksum <spack-checksum_>`_), it spiders the parent directory +of the tarball in the ``url`` attribute. For example, for libelf, the +url is: .. code-block:: python @@ -1335,8 +1336,8 @@ If your build system does *not* automatically pick these variables up from the environment, then you can simply pass them on the command line or use a patch as part of your build process to get the correct compilers into the project's build system. There are also some file -editing commands you can use -- these are described later in -`filtering-files`_. +editing commands you can use -- these are described later in the +`section on file manipulation <file-manipulation_>`_. In addition to the compiler variables, these variables are set before entering ``install()`` so that packages can locate dependencies @@ -1466,9 +1467,28 @@ yourself, e.g.: Most of the standard UNIX directory names are attributes on the -``prefix`` object. See :py:class:`spack.prefix.Prefix` for a full -list. - +``prefix`` object. Here is a full list: + + ========================= ================================================ + Prefix Attribute Location + ========================= ================================================ + ``prefix.bin`` ``$prefix/bin`` + ``prefix.sbin`` ``$prefix/sbin`` + ``prefix.etc`` ``$prefix/etc`` + ``prefix.include`` ``$prefix/include`` + ``prefix.lib`` ``$prefix/lib`` + ``prefix.lib64`` ``$prefix/lib64`` + ``prefix.libexec`` ``$prefix/libexec`` + ``prefix.share`` ``$prefix/share`` + ``prefix.doc`` ``$prefix/doc`` + ``prefix.info`` ``$prefix/info`` + + ``prefix.man`` ``$prefix/man`` + ``prefix.man[1-8]`` ``$prefix/man/man[1-8]`` + + ``prefix.share_man`` ``$prefix/share/man`` + ``prefix.share_man[1-8]`` ``$prefix/share/man[1-8]`` + ========================= ================================================ .. _spec-objects: @@ -1678,11 +1698,10 @@ method (the one without the ``@when`` decorator) will be called. the way decorators work. - .. _shell-wrappers: -Shell command wrappers -------------------------- +Shell command functions +---------------------------- Recall the install method from ``libelf``: @@ -1730,9 +1749,161 @@ to the ``make`` wrapper to disable parallel make. In the ``libelf`` package, this allows us to avoid race conditions in the library's build system. + +.. _file-manipulation: + +File manipulation functions +------------------------------ + +Many builds are not perfect. If a build lacks an install target, or if +it does not use systems like CMake or autotools, which have standard +ways of setting compilers and options, you may need to edit files or +install some files yourself to get them working with Spack. + +You can do this with standard Python code, and Python has rich +libraries with functions for file manipulation and filtering. Spack +also provides a number of convenience functions of its own to make +your life even easier. These functions are described in this section. + +All of the functions in this section can be included by simply +running: + +.. code-block:: python + + from spack import * + +This is already part of the boilerplate for packages created with +``spack create`` or ``spack edit``. + + +Filtering functions +~~~~~~~~~~~~~~~~~~~~~~ + +:py:func:`filter_file(regex, repl, *filenames, **kwargs) <spack.filter_file>` + Works like ``sed`` but with Python regular expression syntax. Takes + a regular expression, a replacement, and a set of files. ``repl`` + can be a raw string or a callable function. If it is a raw string, + it can contain ``\1``, ``\2``, etc. to refer to capture groups in + the regular expression. If it is a callable, it is passed the + Python ``MatchObject`` and should return a suitable replacement + string for the particular match. + + Examples: + + #. Replacing ``#!/usr/bin/perl`` with ``#!/usr/bin/env perl`` in ``bib2xhtml``: + + .. code-block:: python + + filter_file(r'#!/usr/bin/perl', + '#!/usr/bin/env perl', join_path(prefix.bin, 'bib2xhtml')) + + #. Switching the compilers used by ``mpich``'s MPI wrapper scripts from + ``cc``, etc. to the compilers used by the Spack build: + + .. code-block:: python + + filter_file('CC="cc"', 'CC="%s"' % self.compiler.cc, + join_path(prefix.bin, 'mpicc')) + + filter_file('CXX="c++"', 'CXX="%s"' % self.compiler.cxx, + join_path(prefix.bin, 'mpicxx')) + +:py:func:`change_sed_delimiter(old_delim, new_delim, *filenames) <spack.change_sed_delim>` + Some packages, like TAU, have a build system that can't install + into directories with, e.g. '@' in the name, because they use + hard-coded ``sed`` commands in their build. + + ``change_sed_delimiter`` finds all ``sed`` search/replace commands + and change the delimiter. e.g., if the file contains commands + that look like ``s///``, you can use this to change them to + ``s@@@``. + + Example of changing ``s///`` to ``s@@@`` in TAU: + + .. code-block:: python + + change_sed_delimiter('@', ';', 'configure') + change_sed_delimiter('@', ';', 'utils/FixMakefile') + change_sed_delimiter('@', ';', 'utils/FixMakefile.sed.default') + + +File functions +~~~~~~~~~~~~~~~~~~~~~~ + +:py:func:`ancestor(dir, n=1) <spack.ancestor>` + Get the n\ :sup:`th` ancestor of the directory ``dir``. + +:py:func:`can_access(path) <spack.can_access>` + True if we can read and write to the file at ``path``. Same as + native python ``os.access(file_name, os.R_OK|os.W_OK)``. + +:py:func:`install(src, dest) <spack.install>` + Install a file to a particular location. For example, install a + header into the ``include`` directory under the install ``prefix``: + + .. code-block:: python + + install('my-header.h', join_path(prefix.include)) + +:py:func:`join_path(prefix, *args) <spack.join_path>` Like + ``os.path.join``, this joins paths using the OS path separator. + However, this version allows an arbitrary number of arguments, so + you can string together many path components. + +:py:func:`mkdirp(*paths) <spack.mkdirp>` + Create each of the directories in ``paths``, creating any parent + directories if they do not exist. + +:py:func:`working_dir(dirname, kwargs) <spack.working_dir>` + This is a Python `Context Manager + <https://docs.python.org/2/library/contextlib.html>`_ that makes it + easier to work with subdirectories in builds. You use this with the + Python ``with`` statement to change into a working directory, and + when the with block is done, you change back to the original + directory. Think of it as a safe ``pushd`` / ``popd`` combination, + where ``popd`` is guaranteed to be called at the end, even if + exceptions are thrown. + + Example usage: + + #. The ``libdwarf`` build first runs ``configure`` and ``make`` in a + subdirectory called ``libdwarf``. It then implements the + installation code itself. This is natural with ``working_dir``: + + .. code-block:: python + + with working_dir('libdwarf'): + configure("--prefix=" + prefix, "--enable-shared") + make() + install('libdwarf.a', prefix.lib) + + #. Many CMake builds require that you build "out of source", that + is, in a subdirectory. You can handle creating and ``cd``'ing to + the subdirectory like the LLVM package does: + + .. code-block:: python + + with working_dir('spack-build', create=True): + cmake('..', + '-DLLVM_REQUIRES_RTTI=1', + '-DPYTHON_EXECUTABLE=/usr/bin/python', + '-DPYTHON_INCLUDE_DIR=/usr/include/python2.6', + '-DPYTHON_LIBRARY=/usr/lib64/libpython2.6.so', + *std_cmake_args) + make() + make("install") + + The ``create=True`` keyword argument causes the command to create + the directory if it does not exist. + + +:py:func:`touch(path) <spack.touch>` + Create an empty file at ``path``. + + .. _pacakge-lifecycle: -Useful Packaging Commands +Package Workflow Commands --------------------------------- When you are building packages, you will likely not get things diff --git a/lib/spack/llnl/util/filesystem.py b/lib/spack/llnl/util/filesystem.py index a70111b915..dc722297ec 100644 --- a/lib/spack/llnl/util/filesystem.py +++ b/lib/spack/llnl/util/filesystem.py @@ -144,6 +144,7 @@ def expand_user(path): def mkdirp(*paths): + """Creates a directory, as well as parent directories if needed.""" for path in paths: if not os.path.exists(path): os.makedirs(path) @@ -163,6 +164,7 @@ def working_dir(dirname, **kwargs): def touch(path): + """Creates an empty file at the specified path.""" with closing(open(path, 'a')) as file: os.utime(path, None) |