diff options
23 files changed, 945 insertions, 130 deletions
@@ -35,7 +35,7 @@ parser.add_argument('-m', '--mock', action='store_true', dest='mock', # each command module implements a parser() function, to which we pass its # subparser for setup. -subparsers = parser.add_subparsers(metavar='COMMAND', dest="command") +subparsers = parser.add_subparsers(metavar='SUBCOMMAND', dest="command") import spack.cmd for cmd in spack.cmd.commands: diff --git a/lib/spack/docs/.gitignore b/lib/spack/docs/.gitignore new file mode 100644 index 0000000000..4d5300fbb9 --- /dev/null +++ b/lib/spack/docs/.gitignore @@ -0,0 +1,2 @@ +spack*.rst +_build diff --git a/lib/spack/docs/Makefile b/lib/spack/docs/Makefile index c413ed2694..99d1cbaff6 100644 --- a/lib/spack/docs/Makefile +++ b/lib/spack/docs/Makefile @@ -7,6 +7,9 @@ SPHINXBUILD = sphinx-build PAPER = BUILDDIR = _build +export PYTHONPATH = ../../spack +APIDOC_FILES = spack*.rst + # Internal variables. PAPEROPT_a4 = -D latex_paper_size=a4 PAPEROPT_letter = -D latex_paper_size=letter @@ -14,10 +17,13 @@ ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . # the i18n builder cannot share the environment and doctrees with the others I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . -.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext +.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext apidoc all: html +apidoc: + sphinx-apidoc -T -o . $(PYTHONPATH)/spack + help: @echo "Please use \`make <target>' where <target> is one of" @echo " html to make standalone HTML files" @@ -41,9 +47,9 @@ help: @echo " doctest to run all doctests embedded in the documentation (if enabled)" clean: - -rm -rf $(BUILDDIR)/* + -rm -rf $(BUILDDIR)/* $(APIDOC_FILES) -html: +html: apidoc $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." diff --git a/lib/spack/docs/_themes/sphinx_rtd_theme/footer.html b/lib/spack/docs/_themes/sphinx_rtd_theme/footer.html index e42d753ff6..6007d5eb90 100644 --- a/lib/spack/docs/_themes/sphinx_rtd_theme/footer.html +++ b/lib/spack/docs/_themes/sphinx_rtd_theme/footer.html @@ -24,7 +24,7 @@ {%- if last_updated %} {% trans last_updated=last_updated|e %}Last updated on {{ last_updated }}.{% endtrans %} {%- endif %} - </p> - + {% trans %}<a href="https://www.github.com/snide/sphinx_rtd_theme">Sphinx theme</a> provided by <a href="http://readthedocs.org">Read the Docs</a>{% endtrans %} + </p> </footer> diff --git a/lib/spack/docs/basic_usage.rst b/lib/spack/docs/basic_usage.rst new file mode 100644 index 0000000000..425fe0c288 --- /dev/null +++ b/lib/spack/docs/basic_usage.rst @@ -0,0 +1,233 @@ +Basic usage +===================== + +Nearly everything you do wtih spack will involve the ``spack`` +command. Like many well-known tools (``git``, ``cvs``, ``svn``, +``yum``, ``port``, ``apt-get``, etc.), ``spack`` is generally called +with a *subcommand* indicating the action you want to perform. + +Getting Help +----------------------- + + +``spack help`` +~~~~~~~~~~~~~~~~~~~~~~ + +The first subcommand you should know is ``spack help``. Run with no +arguments, it will give a list of all spack options and subcommands: + +.. command-output:: spack help + +If you want help on the usage of a particular subcommand, you can pass +it as an argument to ``spack help``: + +.. command-output:: spack help install + +Alternately, you can use ``spack -h`` in place of ``spack help``, or +``spack <subcommand> -h`` to get help on a particular subcommand. + + +Viewing available packages +------------------------------ + +The first thing you will likely want to do with spack is find out what +software is available to install. There are two main commands for +this: ``spack list`` and ``spack info``. + + +``spack list`` +~~~~~~~~~~~~~~~~ + +The ``spack list`` command does what you might expect. it prints out a +list of all the available packages you can install. Use it like +this: + +.. command-output:: spack list + +The packages are listed by name in alphabetical order. If you just +want to see *installed* packages, you should use ``spack list -i`` + + +``spack info`` +~~~~~~~~~~~~~~~~ + +To get information on a particular package from the full list, you can +run ``spack info <package name>``. e.g., for ``mpich``: + +.. command-output:: spack info mpich + +This gives basic information about the package, such as where it can +be downloaded, what other packages it depends on, virtual package +information, and a text description, if one is available. We'll give +more details on dependencies and virtual dependencies later in this +guide. + + +Installing and uninstalling +------------------------------ + +``spack install`` +~~~~~~~~~~~~~~~~~~~~~ + +You can install any package from ``spack list``, using ``spack +install``. In the simplest case, if you just want the latest version +and you don't care about any configuration, you can just run ``spack +install <package>``: + +.. code-block:: sh + + spack install mpileaks + +This will fetch the tarball for ``mpileaks``, expand it, verify that +it was donwloaded without errors, build the package, and install it in +its own directory in ``$SPACK_HOME/opt``. If the requested packages +depends on other packages in order to build, then they will also be +fetched and installed. + +Spack also allows you to ask for *specific* configurations of a +package. For example, if you want to install something with a +specific version, you can add ``@`` after the package name, followed +by the version you want: + +.. code-block:: sh + + spack install mpich@3.0.4 + +You can install as many versions of the same pacakge as you want, and +they will not interfere with each other. Spack installs each package +into its own unique prefix. If you or another user links a library +against soething you install using Spack, it will continue to work +until you explicitly uninstall it. + +The version isn't all that you can customize on a spack command line. +Spack can install many configurations, with different versions, +compilers, compiler versions, compile-time options (variants), and +even architectures (e.g., on a machine that requires cross-compiling). +Spack is also unique in that it lets you customize the *dependencies* +you build a package with. That is, you could have two configurations +of the same version of a package: one built with boost 1.39.0, and the +other version built with version 1.43.0. + +Spack calls the descriptor used to refer to a particular package +configuration a **spec**. In the command lines above, both +``mpileaks`` and ``mpileaks@3.0.4`` are specs. Specs and their syntax +are covered in more detail in :ref:`sec-specs`. + + + + +``spack uninstall`` +~~~~~~~~~~~~~~~~~~~~~ + +To uninstall a package, just type ``spack uninstall <package>``. This +will completely remove the directory in which the package was installed. + +.. code-block:: sh + + spack uninstall mpich + +If there are other installed packages depend on the package you're +uninstalling, spack will issue a warning to this effect. In general, +you should remove the other packages *before* removing the package +they depend on, or you risk breaking packages on your system. If you +still want to remove the package without regard for its dependencies, +you can run ``spack uninstall -f <package>`` to override Spack's +warning. + +If you have more than one version of the same package installed, spack +may not be able to figure out which on eyou want uninstalled. For +example, if you have both ``mpich@3.0.2`` and ``mpich@3.1`` installed, +and you type ``spack uninstall mpich``, then Spack will not know which +one you're referring to, and it will ask you to be more specific by +providing a version to differentiate, For example, ``spack uninstall +mpich@3.1`` is unambiguous. + + +.. _sec-specs: + +Specs +------------------------- + +Dependencies +------------------------- + +Virtual dependencies +------------------------- + +Versions, compilers, and architectures +---------------------------------------- + +``spack versions`` +~~~~~~~~~~~~~~~~~~~~~~~~ + +``spack compilers`` +~~~~~~~~~~~~~~~~~~~~~~~~ + +Architectures +~~~~~~~~~~~~~~~~~~~~~~~~ + +Spack's specs allow insatllations for multiple architectures to coexist +within the same prefix. It is also intended to support multiple +architecutres for cross-compilation. + + +Package lifecycle +------------------------------ + +The ``spack install`` command performs a number of tasks before it +finally installs each package. It downloads an archive, expands it in +a temporary directory, and only then performs the installation. Spack +has several commands that allow finer-grained control over each of +these stages of the build process. + + +``spack fetch`` +~~~~~~~~~~~~~~~~~ + +This is the first step of ``spack install``. It takes a spec and +determines the correct download URL to use for the requested package +version. It then downloads the archive, checks it against an MD5 +checksum, and stores it in a staging directory if the check was +successful. The staging directory will be located under +``$SPACK_HOME/var/spack``. + +If run after the archive has already been downloaded, ``spack fetch`` +is idempotent and will not download the archive again. + +``spack stage`` +~~~~~~~~~~~~~~~~~ + +This is the second step in installation after ``spack fetch``. It +expands the downloaded archive in its temporary directory, where it +will be built by ``spack install``. If the archive has already been +expanded, then this command does nothing. + +``spack clean`` +~~~~~~~~~~~~~~~~~ + +This command has several variations, each undoing one of the +installation tasks. They are: + +``spack clean`` + Runs ``make clean`` in the expanded archive directory. This is useful + if an attempted build failed, and something needs to be changed to get + a package to build. If a particular package does not have a ``make clean`` + target, this will do nothing. + +``spack clean -w`` or ``spack clean --work`` + This deletes the entire build directory and re-expands it from the downloaded + archive. This is useful if a package does not support a proper ``make clean`` + target. + +``spack clean -d`` or ``spack clean --dist`` + This deletes the build directory *and* the downloaded archive. If + ``fetch``, ``stage``, or ``install`` are run again after this, the + process will start from scratch, and the archive archive will be + downloaded again. Useful if somehow a bad archive is downloaded + accidentally and needs to be cleaned out of the staging area. + + + + +``spack purge`` +~~~~~~~~~~~~~~~~~ diff --git a/lib/spack/docs/conf.py b/lib/spack/docs/conf.py index df472fd4c4..68b70f359f 100644 --- a/lib/spack/docs/conf.py +++ b/lib/spack/docs/conf.py @@ -16,7 +16,10 @@ import sys, os # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. -#sys.path.insert(0, os.path.abspath('.')) +sys.path.insert(0, os.path.abspath('exts')) +os.environ['PATH'] += os.pathsep + '../../../bin' + +todo_include_todos = True # -- General configuration ----------------------------------------------------- @@ -25,7 +28,10 @@ import sys, os # Add any Sphinx extension module names here, as strings. They can be extensions # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. -extensions = ['sphinx.ext.autodoc'] +extensions = ['sphinx.ext.autodoc', + 'sphinx.ext.graphviz', + 'sphinx.ext.todo', + 'sphinxcontrib.programoutput'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] diff --git a/lib/spack/docs/developer_guide.rst b/lib/spack/docs/developer_guide.rst new file mode 100644 index 0000000000..79bb71b182 --- /dev/null +++ b/lib/spack/docs/developer_guide.rst @@ -0,0 +1,31 @@ +Developer Guide +===================== + + +Spec objects +------------------------- + +Package objects +------------------------- + +Stage objects +------------------------- + +Writing commands +------------------------- + +Unit tests +------------------------- + +Unit testing +------------------------- + + +Developer commands +------------------------- + +``spack doc`` +~~~~~~~~~~~~~~~~~ + +``spack test`` +~~~~~~~~~~~~~~~~~ diff --git a/lib/spack/docs/exts/sphinxcontrib/LICENSE b/lib/spack/docs/exts/sphinxcontrib/LICENSE new file mode 100644 index 0000000000..43b87a5992 --- /dev/null +++ b/lib/spack/docs/exts/sphinxcontrib/LICENSE @@ -0,0 +1,25 @@ +Copyright (c) 2010, 2011, 2012 Sebastian Wiesner <lunaryorn@googlemail.com> +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + +* Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/lib/spack/docs/exts/sphinxcontrib/__init__.py b/lib/spack/docs/exts/sphinxcontrib/__init__.py new file mode 100644 index 0000000000..591cf0e16e --- /dev/null +++ b/lib/spack/docs/exts/sphinxcontrib/__init__.py @@ -0,0 +1,9 @@ +# -*- coding: utf-8 -*- +""" + sphinxcontrib + ~~~~~~~~~~~~~ + + Contains 3rd party Sphinx extensions. +""" + +__import__('pkg_resources').declare_namespace(__name__) diff --git a/lib/spack/docs/exts/sphinxcontrib/programoutput.py b/lib/spack/docs/exts/sphinxcontrib/programoutput.py new file mode 100644 index 0000000000..3f6a4f1595 --- /dev/null +++ b/lib/spack/docs/exts/sphinxcontrib/programoutput.py @@ -0,0 +1,263 @@ +# -*- coding: utf-8 -*- +# Copyright (c) 2010, 2011, 2012, Sebastian Wiesner <lunaryorn@gmail.com> +# All rights reserved. + +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: + +# 1. Redistributions of source code must retain the above copyright notice, +# this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. + +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE +# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +# POSSIBILITY OF SUCH DAMAGE. + +""" + sphinxcontrib.programoutput + =========================== + + This extension provides a directive to include the output of commands as + literal block while building the docs. + + .. moduleauthor:: Sebastian Wiesner <lunaryorn@gmail.com> +""" + +from __future__ import (print_function, division, unicode_literals, + absolute_import) + +import sys +import os +import shlex +from subprocess import Popen, PIPE, STDOUT +from collections import defaultdict, namedtuple + +from docutils import nodes +from docutils.parsers import rst +from docutils.parsers.rst.directives import flag, unchanged, nonnegative_int + + +__version__ = '0.9' + + +class program_output(nodes.Element): + pass + + +def _slice(value): + parts = [int(v.strip()) for v in value.split(',')] + if len(parts) > 2: + raise ValueError('too many slice parts') + return tuple((parts + [None] * 2)[:2]) + + +class ProgramOutputDirective(rst.Directive): + has_content = False + final_argument_whitespace = True + required_arguments = 1 + + option_spec = dict(shell=flag, prompt=flag, nostderr=flag, + ellipsis=_slice, extraargs=unchanged, + returncode=nonnegative_int, cwd=unchanged) + + def run(self): + env = self.state.document.settings.env + + node = program_output() + node.line = self.lineno + node['command'] = self.arguments[0] + + if self.name == 'command-output': + node['show_prompt'] = True + else: + node['show_prompt'] = 'prompt' in self.options + + node['hide_standard_error'] = 'nostderr' in self.options + node['extraargs'] = self.options.get('extraargs', '') + _, cwd = env.relfn2path(self.options.get('cwd', '/')) + node['working_directory'] = cwd + node['use_shell'] = 'shell' in self.options + node['returncode'] = self.options.get('returncode', 0) + if 'ellipsis' in self.options: + node['strip_lines'] = self.options['ellipsis'] + return [node] + + +_Command = namedtuple( + 'Command', 'command shell hide_standard_error working_directory') + + +class Command(_Command): + """ + A command to be executed. + """ + + def __new__(cls, command, shell=False, hide_standard_error=False, + working_directory='/'): + if isinstance(command, list): + command = tuple(command) + # `chdir()` resolves symlinks, so we need to resolve them too for + # caching to make sure that different symlinks to the same directory + # don't result in different cache keys. Also normalize paths to make + # sure that identical paths are also equal as strings. + working_directory = os.path.normpath(os.path.realpath( + working_directory)) + return _Command.__new__(cls, command, shell, hide_standard_error, + working_directory) + + @classmethod + def from_program_output_node(cls, node): + """ + Create a command from a :class:`program_output` node. + """ + extraargs = node.get('extraargs', '') + command = (node['command'] + ' ' + extraargs).strip() + return cls(command, node['use_shell'], + node['hide_standard_error'], node['working_directory']) + + def execute(self): + """ + Execute this command. + + Return the :class:`~subprocess.Popen` object representing the running + command. + """ + if self.shell: + if sys.version_info[0] < 3 and isinstance(self.command, unicode): + command = self.command.encode(sys.getfilesystemencoding()) + else: + command = self.command + else: + if sys.version_info[0] < 3 and isinstance(self.command, unicode): + command = shlex.split(self.command.encode( + sys.getfilesystemencoding())) + elif isinstance(self.command, str): + command = shlex.split(self.command) + else: + command = self.command + return Popen(command, shell=self.shell, stdout=PIPE, + stderr=PIPE if self.hide_standard_error else STDOUT, + cwd=self.working_directory) + + def get_output(self): + """ + Get the output of this command. + + Return a tuple ``(returncode, output)``. ``returncode`` is the + integral return code of the process, ``output`` is the output as + unicode string, with final trailing spaces and new lines stripped. + """ + process = self.execute() + output = process.communicate()[0].decode( + sys.getfilesystemencoding(), 'replace').rstrip() + return process.returncode, output + + def __str__(self): + if isinstance(self.command, tuple): + return repr(list(self.command)) + return repr(self.command) + + +class ProgramOutputCache(defaultdict): + """ + Execute command and cache their output. + + This class is a mapping. Its keys are :class:`Command` objects represeting + command invocations. Its values are tuples of the form ``(returncode, + output)``, where ``returncode`` is the integral return code of the command, + and ``output`` is the output as unicode string. + + The first time, a key is retrieved from this object, the command is + invoked, and its result is cached. Subsequent access to the same key + returns the cached value. + """ + + def __missing__(self, command): + """ + Called, if a command was not found in the cache. + + ``command`` is an instance of :class:`Command`. + """ + result = command.get_output() + self[command] = result + return result + + +def run_programs(app, doctree): + """ + Execute all programs represented by ``program_output`` nodes in + ``doctree``. Each ``program_output`` node in ``doctree`` is then + replaced with a node, that represents the output of this program. + + The program output is retrieved from the cache in + ``app.env.programoutput_cache``. + """ + if app.config.programoutput_use_ansi: + # enable ANSI support, if requested by config + from sphinxcontrib.ansi import ansi_literal_block + node_class = ansi_literal_block + else: + node_class = nodes.literal_block + + cache = app.env.programoutput_cache + + for node in doctree.traverse(program_output): + command = Command.from_program_output_node(node) + try: + returncode, output = cache[command] + except EnvironmentError as error: + error_message = 'Command {0} failed: {1}'.format(command, error) + error_node = doctree.reporter.error(error_message, base_node=node) + node.replace_self(error_node) + else: + if returncode != node['returncode']: + app.warn('Unexpected return code {0} from command {1}'.format( + returncode, command)) + + # replace lines with ..., if ellipsis is specified + if 'strip_lines' in node: + lines = output.splitlines() + start, stop = node['strip_lines'] + lines[start:stop] = ['...'] + output = '\n'.join(lines) + + if node['show_prompt']: + tmpl = app.config.programoutput_prompt_template + output = tmpl.format(command=node['command'], output=output, + returncode=returncode) + + new_node = node_class(output, output) + new_node['language'] = 'text' + node.replace_self(new_node) + + +def init_cache(app): + """ + Initialize the cache for program output at + ``app.env.programoutput_cache``, if not already present (e.g. being + loaded from a pickled environment). + + The cache is of type :class:`ProgramOutputCache`. + """ + if not hasattr(app.env, 'programoutput_cache'): + app.env.programoutput_cache = ProgramOutputCache() + + +def setup(app): + app.add_config_value('programoutput_use_ansi', False, 'env') + app.add_config_value('programoutput_prompt_template', + '$ {command}\n{output}', 'env') + app.add_directive('program-output', ProgramOutputDirective) + app.add_directive('command-output', ProgramOutputDirective) + app.connect(str('builder-inited'), init_cache) + app.connect(str('doctree-read'), run_programs) diff --git a/lib/spack/docs/features.rst b/lib/spack/docs/features.rst new file mode 100644 index 0000000000..7039563698 --- /dev/null +++ b/lib/spack/docs/features.rst @@ -0,0 +1,96 @@ +Feature Overview +================== + +This is an overview of features that make Spack different from other +`package managers <http://en.wikipedia.org/wiki/Package_management_system>`_ +and `port systems <http://en.wikipedia.org/wiki/Ports_collection>`_. + +Simple package installation +---------------------------- + +Installing packages is easy with Spack when you just want the default +version. This installs the latest version of mpileaks and all of its +dependencies: + +.. code-block:: sh + + $ spack install mpileaks + +Custom versions & configurations +------------------------------------------- + +If there's some aspect of your package that you want to customize, you +can do that too. + +.. code-block:: sh + + # Install a particular version by appending @ + $ spack install mpileaks@1.1.2 + + # Or your favorite compiler (and its version), with % + $ spack install mpileaks@1.1.2 %gcc@4.7.3 + + # Add some special compile-time options with + + $ spack install mpileaks@1.1.2 %gcc@4.7.3 +debug + + # Cross-compile for a different architecture with = + $ spack install mpileaks@1.1.2 =bgqos_0 + +Customize dependencies +------------------------------------- + +You can customize package dependencies with ``^``. Suppose that +``mpileaks`` depends indirectly on ``libelf`` and ``libdwarf``. Using +``^``, you can add custom configurations for the dependencies, too. + +.. code-block:: sh + + # Install mpileaks and link it with specific versions of libelf and libdwarf + $ spack install mpileaks@1.1.2 %gcc@4.7.3 +debug ^libelf@0.8.12 ^libdwarf@20130729+debug + + +Non-destructive installs +------------------------------------- + +Spack installs every unique package configuration in its own prefix, +so you can install as many different versions and configurations as +you want. New installs will not break existing ones. + + +Packages can peacefully coexist +------------------------------------- + +Spack uses ``RPATH`` everywhere, so users do not need to customize +``LD_LIBRARY_PATH``. If you use a library or run a program, it will +run the way you built it. + + +Creating packages is easy +------------------------------------- + +To create your own packages, give spack the tarball URL. Spack +creates all the boilerplate for you. + +.. code-block:: sh + + $ spack create http://scalability.llnl.gov/mpileaks/downloads/mpileaks-1.0.tar.gz + +Creates ``mpileaks.py``: + +.. code-block:: python + + from spack import * + + class Mpileaks(Package): + homepage = "http://www.example.com/" + url = "http://scalability.llnl.gov/mpileaks/downloads/mpileaks-1.0.tar.gz" + md5 = "4136d7b4c04df68b686570afa26988ac" + + def install(self, prefix): + configure("--prefix=%s" % prefix) + make() + make("install") + +Packages are pure python, so you have complete freedom when writing +build code. Spack also provides a number of feature that make it +easier to write packages. diff --git a/lib/spack/docs/getting_started.rst b/lib/spack/docs/getting_started.rst new file mode 100644 index 0000000000..ea7b949fe6 --- /dev/null +++ b/lib/spack/docs/getting_started.rst @@ -0,0 +1,44 @@ +Getting Started +==================== + +Download +-------------------- + +Getting spack is easy. Clone it using `git <http://git-scm.com->`_ +with the following command: + +.. code-block:: sh + + $ git clone ssh://git@cz-stash.llnl.gov:7999/scale/spack.git + +This will create a directory called ``spack``. We'll assume that the +full path to this directory is in some environment called +``SPACK_HOME``. Add ``$SPACK_HOME/bin`` to your path and you're ready +to go: + +.. code-block:: sh + + $ export PATH=spack/bin:$SPACK_HOME + $ spack install mpich + +In general, most of your interactions with Spack will be through the +``spack`` command. + + +Install +-------------------- + +You don't need to install Spack; it's ready to run as soon as you +clone it from git. + +You may want to run it out of a prefix other than the git repository +you cloned. The ``spack bootstrap`` command provides this +functionality. To install spack in a new directory, simply type: + +.. code-block:: sh + + $ spack bootstrap /my/favorite/prefix + +This will install a new spack script in /my/favorite/prefix/bin, which +you can use just like you would the regular spack script. Each copy +of spack installs packages into its own ``$PREFIX/opt`` directory. diff --git a/lib/spack/docs/index.rst b/lib/spack/docs/index.rst index 089c1fe5ee..092d696983 100644 --- a/lib/spack/docs/index.rst +++ b/lib/spack/docs/index.rst @@ -3,15 +3,39 @@ You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. -Welcome to Spack's documentation! +Spack Documentation ================================= -Contents: +Spack builds and installs software the way you want it. Other tools +let you install the latest version once. Spack lets you install the +versions you want, built with the compilers, libraries, and options +you want. Spack is non-destructive; installing a new version does not +break your old installs. See the :doc:`features` for more highlights. + +Get spack and install your first package: + +.. code-block:: sh + + $ git clone ssh://git@cz-stash.llnl.gov:7999/scale/spack.git + $ cd spack/bin + $ ./spack install mpich + +If you're new to spack and want to start using it, see :doc:`getting_started`, +or refer to the full manual below. + +Table of Contents +--------------------- .. toctree:: :maxdepth: 2 - + features + getting_started + basic_usage + packaging_guide + site_configuration + developer_guide + API Docs <spack> Indices and tables ================== @@ -19,4 +43,3 @@ Indices and tables * :ref:`genindex` * :ref:`modindex` * :ref:`search` - diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst new file mode 100644 index 0000000000..371c85df4c --- /dev/null +++ b/lib/spack/docs/packaging_guide.rst @@ -0,0 +1,35 @@ +Packaging Guide +===================== + + +Package files +------------------------- + + +Dependencies +------------------------- + + +Virtual dependencies +------------------------- + + +Virtual dependencies +------------------------- + + + +Packaging commands +------------------------- + +``spack edit`` +~~~~~~~~~~~~~~~~~~~~ + +``spack create`` +~~~~~~~~~~~~~~~~~~~~ + +``spack checksum`` +~~~~~~~~~~~~~~~~~~~~ + +``spack graph`` +~~~~~~~~~~~~~~~~~~~~ diff --git a/lib/spack/docs/site_configuration.rst b/lib/spack/docs/site_configuration.rst new file mode 100644 index 0000000000..bc790db257 --- /dev/null +++ b/lib/spack/docs/site_configuration.rst @@ -0,0 +1,9 @@ +Site-specific configuration +=================================== + +Temporary space +---------------------------- + + +Concretization policies +---------------------------- diff --git a/lib/spack/spack/cmd/install-spack.py b/lib/spack/spack/cmd/bootstrap.py index a8663e97ac..66dda5dbc9 100644 --- a/lib/spack/spack/cmd/install-spack.py +++ b/lib/spack/spack/cmd/bootstrap.py @@ -17,7 +17,7 @@ def get_origin_url(): return origin_url.strip() -def install_spack(parser, args): +def bootstrap(parser, args): origin_url = get_origin_url() prefix = args.prefix @@ -28,7 +28,7 @@ def install_spack(parser, args): files_in_the_way = os.listdir(prefix) if files_in_the_way: - tty.die("There are already files there! Delete these files before installing spack.", + tty.die("There are already files there! Delete these files before boostrapping spack.", *files_in_the_way) tty.msg("Installing:", @@ -41,5 +41,5 @@ def install_spack(parser, args): check_call(['git', 'fetch', 'origin', 'master:refs/remotes/origin/master', '-n', '-q']) check_call(['git', 'reset', '--hard', 'origin/master', '-q']) - tty.msg("Successfully installed spack in %s" % prefix, + tty.msg("Successfully created a new spack in %s" % prefix, "Run %s/bin/spack to use this installation." % prefix) diff --git a/lib/spack/spack/cmd/clean.py b/lib/spack/spack/cmd/clean.py index 52eaf2893f..ff5ae4ba2c 100644 --- a/lib/spack/spack/cmd/clean.py +++ b/lib/spack/spack/cmd/clean.py @@ -9,9 +9,9 @@ description = "Remove staged files for packages" def setup_parser(subparser): subparser.add_argument('-c', "--clean", action="store_true", dest='clean', - help="run make clean in the stage directory (default)") + help="run make clean in the build directory (default)") subparser.add_argument('-w', "--work", action="store_true", dest='work', - help="delete and re-expand the entire stage directory") + help="delete the build directory and re-expand it from its archive.") subparser.add_argument('-d', "--dist", action="store_true", dest='dist', help="delete the downloaded archive.") subparser.add_argument('packages', nargs=argparse.REMAINDER, diff --git a/lib/spack/spack/cmd/sys-type.py b/lib/spack/spack/cmd/sys_type.py index 7f57fb16ff..7f57fb16ff 100644 --- a/lib/spack/spack/cmd/sys-type.py +++ b/lib/spack/spack/cmd/sys_type.py diff --git a/lib/spack/spack/package.py b/lib/spack/spack/package.py index a28bfae411..69392efdd5 100644 --- a/lib/spack/spack/package.py +++ b/lib/spack/spack/package.py @@ -35,8 +35,8 @@ from spack.util.web import get_pages class Package(object): """This is the superclass for all spack packages. - The Package class - ================== + ***The Package class*** + Package is where the bulk of the work of installing packages is done. A package defines how to fetch, verfiy (via, e.g., md5), build, and @@ -53,35 +53,39 @@ class Package(object): in this directory. Spack automatically scans the python files there and figures out which one to import when you invoke it. - An example package - ==================== + **An example package** + Let's look at the cmake package to start with. This package lives in $prefix/lib/spack/spack/packages/cmake.py: - from spack import * - class Cmake(Package): - homepage = 'https://www.cmake.org' - url = 'http://www.cmake.org/files/v2.8/cmake-2.8.10.2.tar.gz' - md5 = '097278785da7182ec0aea8769d06860c' + .. code-block:: python + + from spack import * + class Cmake(Package): + homepage = 'https://www.cmake.org' + url = 'http://www.cmake.org/files/v2.8/cmake-2.8.10.2.tar.gz' + md5 = '097278785da7182ec0aea8769d06860c' - def install(self, prefix): - configure('--prefix=%s' % prefix, - '--parallel=%s' % make_jobs) - make() - make('install') + def install(self, prefix): + configure('--prefix=%s' % prefix, + '--parallel=%s' % make_jobs) + make() + make('install') + + **Naming conventions** - Naming conventions - --------------------- There are two names you should care about: - 1. The module name, 'cmake'. - - User will refers to this name, e.g. 'spack install cmake'. - - Corresponds to the name of the file, 'cmake.py', and it can - include _, -, and numbers (it can even start with a number). + 1. The module name, ``cmake``. + + * User will refers to this name, e.g. 'spack install cmake'. + * Corresponds to the name of the file, 'cmake.py', and it can + include ``_``, ``-``, and numbers (it can even start with a + number). - 2. The class name, "Cmake". This is formed by converting -'s or _'s - in the module name to camel case. If the name starts with a number, - we prefix the class name with 'Num_'. Examples: + 2. The class name, "Cmake". This is formed by converting `-` or + ``_`` in the module name to camel case. If the name starts with + a number, we prefix the class name with ``Num_``. Examples: Module Name Class Name foo_bar FooBar @@ -91,23 +95,28 @@ class Package(object): The class name is what spack looks for when it loads a package module. - Required Attributes - --------------------- + **Required Attributes** + Aside from proper naming, here is the bare minimum set of things you need when you make a package: - homepage informational URL, so that users know what they're - installing. - url URL of the source archive that spack will fetch. + homepage + informational URL, so that users know what they're + installing. - md5 md5 hash of the source archive, so that we can - verify that it was downloaded securely and correctly. + url + URL of the source archive that spack will fetch. - install() This function tells spack how to build and install the - software it downloaded. + md5 + md5 hash of the source archive, so that we can + verify that it was downloaded securely and correctly. + + install() + This function tells spack how to build and install the + software it downloaded. + + **Optional Attributes** - Optional Attributes - --------------------- You can also optionally add these attributes, if needed: list_url Webpage to scrape for available version strings. Default is the @@ -121,8 +130,8 @@ class Package(object): your package needs special version formatting in its URL. boost is an example of a package that needs this. - Creating Packages - =================== + ***Creating Packages*** + As a package creator, you can probably ignore most of the preceding information, because you can use the 'spack create' command to do it all automatically. @@ -130,8 +139,8 @@ class Package(object): You as the package creator generally only have to worry about writing your install function and specifying dependencies. - spack create - ---------------- + **spack create** + Most software comes in nicely packaged tarballs, like this one: http://www.cmake.org/files/v2.8/cmake-2.8.10.2.tar.gz @@ -147,15 +156,17 @@ class Package(object): Once this skeleton code is generated, spack pops up the new package in your $EDITOR so that you can modify the parts that need changes. - Dependencies - --------------- + **Dependencies** + If your package requires another in order to build, you can specify that like this: - class Stackwalker(Package): - ... - depends_on("libdwarf") - ... + .. code-block:: python + + class Stackwalker(Package): + ... + depends_on("libdwarf") + ... This tells spack that before it builds stackwalker, it needs to build the libdwarf package as well. Note that this is the module name, not @@ -170,8 +181,8 @@ class Package(object): it will find the dependencies automatically. - The Install Function - ---------------------- + **The Install Function** + The install function is designed so that someone not too terribly familiar with Python could write a package installer. For example, we put a number of commands in install scope that you can use almost like shell commands. @@ -195,47 +206,58 @@ class Package(object): them are created and set on the module. - Parallel Builds - ------------------- + **Parallel Builds** + By default, Spack will run make in parallel when you run make() in your install function. Spack figures out how many cores are available on your system and runs make with -j<cores>. If you do not want this behavior, you can explicitly mark a package not to use parallel make: - class SomePackage(Package): - ... - parallel = False - ... + .. code-block:: python + + class SomePackage(Package): + ... + parallel = False + ... This changes thd default behavior so that make is sequential. If you still want to build some parts in parallel, you can do this in your install function: - make(parallel=True) + .. code-block:: python + + make(parallel=True) Likewise, if you do not supply parallel = True in your Package, you can keep the default parallel behavior and run make like this when you want a sequential build: - make(parallel=False) + .. code-block:: python + + make(parallel=False) + + **Package Lifecycle** - Package Lifecycle - ================== This section is really only for developers of new spack commands. A package's lifecycle over a run of Spack looks something like this: - p = Package() # Done for you by spack + .. code-block:: python + + p = Package() # Done for you by spack - p.do_fetch() # called by spack commands in spack/cmd. - p.do_stage() # see spack.stage.Stage docs. - p.do_install() # calls package's install() function - p.do_uninstall() + p.do_fetch() # called by spack commands in spack/cmd. + p.do_stage() # see spack.stage.Stage docs. + p.do_install() # calls package's install() function + p.do_uninstall() There are also some other commands that clean the build area: - p.do_clean() # runs make clean - p.do_clean_work() # removes the build directory and + + .. code-block:: python + + p.do_clean() # runs make clean + p.do_clean_work() # removes the build directory and # re-expands the archive. - p.do_clean_dist() # removes the stage directory entirely + p.do_clean_dist() # removes the stage directory entirely The convention used here is that a do_* function is intended to be called internally by Spack commands (in spack.cmd). These aren't for package @@ -244,6 +266,7 @@ class Package(object): Package creators override functions like install() (all of them do this), clean() (some of them do this), and others to provide custom behavior. + """ # diff --git a/lib/spack/spack/relations.py b/lib/spack/spack/relations.py index 4be4bb6cad..b6913d69b8 100644 --- a/lib/spack/spack/relations.py +++ b/lib/spack/spack/relations.py @@ -68,9 +68,11 @@ def _caller_locals(): del stack -def _ensure_caller_is_spack_package(): - """Make sure that the caller is a spack package. If it's not, - raise ScopeError. if it is, return its name.""" +def _get_calling_package_name(): + """Make sure that the caller is a class definition, and return + the module's name. This is useful for getting the name of + spack packages from inside a relation function. + """ stack = inspect.stack() try: # get calling function name (the relation) @@ -85,9 +87,6 @@ def _ensure_caller_is_spack_package(): raise ScopeError(relation) module_name = caller_locals['__module__'] - if not module_name.startswith(packages_module()): - raise ScopeError(relation) - base_name = module_name.split('.')[-1] return base_name @@ -121,7 +120,7 @@ def _parse_local_spec(spec_like, pkg_name): """Adds a dependencies local variable in the locals of the calling class, based on args. """ def depends_on(*specs): - pkg = _ensure_caller_is_spack_package() + pkg = _get_calling_package_name() dependencies = _caller_locals().setdefault('dependencies', {}) for string in specs: @@ -136,7 +135,7 @@ def provides(*specs, **kwargs): 'mpi', other packages can declare that they depend on "mpi", and spack can use the providing package to satisfy the dependency. """ - pkg = _ensure_caller_is_spack_package() + pkg = _get_calling_package_name() spec_string = kwargs.get('when', pkg) provider_spec = _parse_local_spec(spec_string, pkg) @@ -170,7 +169,7 @@ class ScopeError(RelationError): def __init__(self, relation): super(ScopeError, self).__init__( relation, - "Cannot inovke '%s' from outside of a Spack package!" % relation) + "Must invoke '%s' from inside a class definition!" % relation) class CircularReferenceError(RelationError): diff --git a/lib/spack/spack/spec.py b/lib/spack/spack/spec.py index d7c642a81e..17ef523f57 100644 --- a/lib/spack/spack/spec.py +++ b/lib/spack/spack/spec.py @@ -6,8 +6,10 @@ package configuration a "spec". The syntax looks like this: - spack install mpileaks ^openmpi @1.2:1.4 +debug %intel @12.1 - 0 1 2 3 4 5 +.. code-block:: sh + + $ spack install mpileaks ^openmpi @1.2:1.4 +debug %intel @12.1 =bgqos_0 + 0 1 2 3 4 5 6 The first part of this is the command, 'spack install'. The rest of the line is a spec for a particular installation of the mpileaks package. @@ -37,7 +39,10 @@ line is a spec for a particular installation of the mpileaks package. if it comes immediately after the compiler name. Otherwise it will be associated with the current package spec. -Here is the EBNF grammar for a spec: +6. The architecture to build with. This is needed on machines where + cross-compilation is required + +Here is the EBNF grammar for a spec:: spec-list = { spec [ dep-list ] } dep_list = { ^ spec } @@ -80,8 +85,7 @@ from spack.util.string import * """This map determines the coloring of specs when using color output. We make the fields different colors to enhance readability. - See spack.color for descriptions of the color codes. -""" + See spack.color for descriptions of the color codes. """ color_formats = {'%' : '@g', # compiler '@' : '@c', # version '=' : '@m', # architecture @@ -129,8 +133,7 @@ def colorize_spec(spec): class Compiler(object): """The Compiler field represents the compiler or range of compiler versions that a package should be built with. Compilers have a - name and a version list. - """ + name and a version list. """ def __init__(self, name, version=None): if name not in spack.compilers.supported_compilers(): raise UnknownCompilerError(name) @@ -348,10 +351,11 @@ class Spec(object): @property def virtual(self): """Right now, a spec is virtual if no package exists with its name. + TODO: revisit this -- might need to use a separate namespace and - be more explicit about this. - Possible idea: just use conventin and make virtual deps all - caps, e.g., MPI vs mpi. + be more explicit about this. + Possible idea: just use conventin and make virtual deps all + caps, e.g., MPI vs mpi. """ return not packages.exists(self.name) @@ -463,13 +467,15 @@ class Spec(object): and normalize again to include the provider's (potentially virtual) dependencies. Repeat until there are no virtual deps. - TODO: If a provider depends on something that conflicts with - other dependencies in the spec being expanded, this can - produce a conflicting spec. For example, if mpich depends - on hwloc@:1.3 but something in the spec needs hwloc1.4:, - then we should choose an MPI other than mpich. Cases like - this are infrequent, but should implement this before it is - a problem. + .. todo:: + + If a provider depends on something that conflicts with + other dependencies in the spec being expanded, this can + produce a conflicting spec. For example, if mpich depends + on hwloc@:1.3 but something in the spec needs hwloc1.4:, + then we should choose an MPI other than mpich. Cases like + this are infrequent, but should implement this before it is + a problem. """ while True: virtuals =[v for v in self.preorder_traversal() if v.virtual] @@ -625,16 +631,19 @@ class Spec(object): """When specs are parsed, any dependencies specified are hanging off the root, and ONLY the ones that were explicitly provided are there. Normalization turns a partial flat spec into a DAG, where: - 1) ALL dependencies of the root package are in the DAG. - 2) Each node's dependencies dict only contains its direct deps. - 3) There is only ONE unique spec for each package in the DAG. - - This includes virtual packages. If there a non-virtual - package that provides a virtual package that is in the spec, - then we replace the virtual package with the non-virtual one. - 4) The spec DAG matches package DAG. + + 1. ALL dependencies of the root package are in the DAG. + 2. Each node's dependencies dict only contains its direct deps. + 3. There is only ONE unique spec for each package in the DAG. + + * This includes virtual packages. If there a non-virtual + package that provides a virtual package that is in the spec, + then we replace the virtual package with the non-virtual one. + + 4. The spec DAG matches package DAG. TODO: normalize should probably implement some form of cycle detection, - to ensure that the spec is actually a DAG. + to ensure that the spec is actually a DAG. """ # Ensure first that all packages in the DAG exist. self.validate_package_names() diff --git a/lib/spack/spack/stage.py b/lib/spack/spack/stage.py index d38413d155..9c5248e9ca 100644 --- a/lib/spack/spack/stage.py +++ b/lib/spack/spack/stage.py @@ -18,12 +18,16 @@ class Stage(object): expanded, and built before being installed. It also handles downloading the archive. A stage's lifecycle looks like this: - setup() Create the stage directory. - fetch() Fetch a source archive into the stage. - expand_archive() Expand the source archive. - <install> Build and install the archive. This is handled - by the Package class. - destroy() Remove the stage once the package has been installed. + setup() + Create the stage directory. + fetch() + Fetch a source archive into the stage. + expand_archive() + Expand the source archive. + <install> + Build and install the archive. This is handled by the Package class. + destroy() + Remove the stage once the package has been installed. If spack.use_tmp_stage is True, spack will attempt to create stages in a tmp directory. Otherwise, stages are created directly in diff --git a/lib/spack/spack/version.py b/lib/spack/spack/version.py index 904992f832..5564d80a4e 100644 --- a/lib/spack/spack/version.py +++ b/lib/spack/spack/version.py @@ -1,15 +1,15 @@ """ This file implements Version and version-ish objects. These are: - Version - A single version of a package. - VersionRange - A range of versions of a package. - VersionList - A list of Versions and VersionRanges. +Version + A single version of a package. +VersionRange + A range of versions of a package. +VersionList + A list of Versions and VersionRanges. All of these types support the following operations, which can -be called on any of the types: +be called on any of the types:: __eq__, __ne__, __lt__, __gt__, __ge__, __le__, __hash__ __contains__ @@ -18,8 +18,6 @@ be called on any of the types: union intersection concrete - True if the Version, VersionRange or VersionList represents - a single version. """ import os import sys |