From 91d6f0b4815d57768422a07c1ecc7a743222caf6 Mon Sep 17 00:00:00 2001 From: Todd Gamblin Date: Sun, 12 Jan 2014 18:22:19 +0100 Subject: Developer documentation --- lib/spack/docs/basic_usage.rst | 2 +- lib/spack/docs/developer_guide.rst | 274 +++++++++++++++++++++++++++++++++- lib/spack/docs/packaging_guide.rst | 6 +- lib/spack/docs/site_configuration.rst | 3 + lib/spack/spack/compilation.py | 10 ++ lib/spack/spack/version.py | 2 +- 6 files changed, 292 insertions(+), 5 deletions(-) diff --git a/lib/spack/docs/basic_usage.rst b/lib/spack/docs/basic_usage.rst index fc2312134e..4c527d91d2 100644 --- a/lib/spack/docs/basic_usage.rst +++ b/lib/spack/docs/basic_usage.rst @@ -1,4 +1,4 @@ -.. _basic_usage: +.. _basic-usage: Basic usage ===================== diff --git a/lib/spack/docs/developer_guide.rst b/lib/spack/docs/developer_guide.rst index 47b98a211d..1f0977b4de 100644 --- a/lib/spack/docs/developer_guide.rst +++ b/lib/spack/docs/developer_guide.rst @@ -3,8 +3,258 @@ Developer Guide ===================== -This guide is intended for people who want to work on Spack's inner -workings. Right now it's pretty sparse. +This guide is intended for people who want to work on Spack itself. +If you just want to develop pacakges, see the :ref:`packaging-guide`. + +It is assumed that you've read the :ref:`basic-usage` and +:ref:`packaging-guide` sections, and that you're familiar with the +concepts discussed there. If you're not, we recommend reading those +first. + +Overview +----------------------- + +Spack is designed with three separate roles in mind: + + #. **Users**, who need to install software *without* knowing all the + details about how it is built. + #. **Packagers** who know how a particular software package is + built and encode this information in package files. + #. **Developers** who work on Spack, add new features, and try to + make the jobs of packagers and users easier. + +Users could be end users installing software in their home directory, +or administrators installing software to a shared directory on a +shared machine. Packagers could be administrators who want to +automate software builds, or application developers who want to make +their software more accessible to users. + +As you might expect, there are many types of users with different +levels of sophistication, and Spack is designed to accommodate both +simple and complex use cases for packages. A user who only knows that +he needs a certain package should be able to type something simple, +like ``spack install ``, and get the package that he +wants. If a user wants to ask for a specific version, use particular +compilers, or build several versions with different configurations, +then that should be possible with a minimal amount of additional +specification. + +This gets us to the two key concepts in Spack's software design: + + #. **Specs**: expressions for describing builds of software, and + #. **Packages**: Python modules that build software according to a + spec. + +A package is a template for building particular software, and a spec +as a descriptor for one or more instances of that template. Users +express the configuration they want using a spec, and a package turns +the spec into a complete build. + +The obvious difficulty with this design is that users underspecify +what they want. To build a software package, the package object needs +a *complete* specification. In Spack, if a spec describes only one +instance of a package, then we say it is **concrete**. If a spec +could describes many instances, (i.e. it is underspecified in one way +or another), then we say it is **abstract**. + +Spack's job is to take an *abstract* spec from the user, find a +*concrete* spec that satisfies the constraints, and hand the task of +building the software off to the package object. The rest of this +document describes all the pieces that come together to make that +happen. + + +Directory Structure +------------------------- + +So that you can familiarize yourself with the project, we'll start +with a high level view of Spack's directory structure:: + + spack/ <- installation root + bin/ + spack <- main spack executable + var/ + spack/ <- build & stage directories + opt/ + spack/ <- packages are installed here + lib/ + spack/ + docs/ <- source for this documentation + env/ <- compiler wrappers for build environment + + spack/ <- spack module; contains Python code + cmd/ <- each file in here is a spack subcommand + compilers/ <- compiler description files + packages/ <- each file in here is a spack package + test/ <- unit test modules + util/ <- common code + +Spack is designed so that it could live within a `standard UNIX +directory hierarchy `_, so ``lib``, +``var``, and ``opt`` all contain a ``spack`` subdirectory in case +Spack is installed alongside other software. Most of the insteresting +parts of Spack live in ``lib/spack``. Files under ``var`` are created +as needed, so there is no ``var`` directory when you initially clone +Spack from the repository. + +Spack has *one* directory layout and there is no install process. +version and the source code. Most Python programs don't look like +this (they use distutils, ``setup.py``, etc.) but we wanted to make +Spack *very* easy to use. The simple layout spares users from the +need to install Spack into a Python environment. Many users don't +have write access to a Python installation, and installing an entire +new instance of Python to bootstrap Spack would be very complicated. +Users should not have to install install a big, complicated package to +use the thing that's supposed to spare them from the details of big, +complicated packages. The end result is that Spack works out of the +box: clone it and add ``bin`` to your PATH and you're ready to go. + + +Code Structure +------------------------- + +This section gives an overview of the various Python modules in Spack, +grouped by functionality. + +Package-related modules +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:mod:`spack.package` + Contains the :class:`Package ` class, which + is the superclass for all packages in Spack. Methods on ``Package`` + implement all phases of the :ref:`package lifecycle + ` and manage the build process. + +:mod:`spack.packages` + Contains all of the packages in Spack and methods for managing them. + Functions like :func:`packages.get ` and + :func:`class_name_for_package_name + ` handle mapping packge module + names to class names and dynamically instantiating packages by name + from module files. + +:mod:`spack.relations` + *Relations* are relationships between packages, like + :func:`depends_on ` and :func:`provides + `. See :ref:`dependencies` and + :ref:`virtual-dependencies`. + +:mod:`spack.multimethod` + Implementation of the :func:`@when ` + decorator, which allows :ref:`multimethods ` in + packages. + + +Spec-related modules +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:mod:`spack.spec` + Contains :class:`Spec ` and :class:`SpecParser + `. Also implements most of the logic for + normalization and concretization of specs. + +:mod:`spack.parse` + Contains some base classes for implementing simple recursive descent + parsers: :class:`Parser ` and :class:`Lexer + `. Used by :class:`SpecParser + `. + +:mod:`spack.concretize` + Contains :class:`DefaultConcretizer + ` implementation, which allows + site administrators to change Spack's :ref:`concretization-policies`. + +:mod:`spack.version` + Implements a simple :class:`Version ` class + with simple comparison semantics. Also implements + :class:`VersionRange ` and + :class:`VersionList `. All three are + comparable with each other and offer union and intersection + operations. Spack uses these classes to compare versions and to + manage version constraints on specs. Comparison semantics are + similar to the ``LooseVersion`` class in ``distutils`` and to the + way RPM compares version strings. + +:mod:`spack.compilers` + Submodules contains descriptors for all valid compilers in Spack. + This is used by the build system to set up the build environment. + + .. warning:: + + Not yet implemented. Currently has two compiler descriptions, + but compilers aren't fully integrated with the build process + yet. + +:mod:`spack.architecture` + :func:`architecture.sys_type ` is used + to determine the host architecture while building. + + .. warning:: + + Not yet implemented. Should eventually have architecture + descriptions for cross-compiling. + + +Build environment +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:mod:`spack.stage` + Handles creating temporary directories for builds. + +:mod:`spack.compilation` + This contains utility functions used by the compiler wrapper script, + ``cc``. + +:mod:`spack.directory_layout` + Classes that control the way an installation directory is laid out. + Create more implementations of this to change the hierarchy and + naming scheme in ``$spack_prefix/opt`` + +Spack Subcommands +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:mod:`spack.cmd` + Each module in this package implements a Spack subcommand. See + :ref:`writing commands ` for details. + +Unit tests +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:mod:`spack.test` + Implements Spack's test suite. Add a module and put its name in + the test suite in ``__init__.py`` to add more unit tests. + +:mod:`spack.test.mock_packages` + This is a fake package hierarchy used to mock up packages for + Spack's test suite. + +Other Modules +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:mod:`spack.globals` + Includes global settings for Spack. the default policy classes for + things like :ref:`temporary space ` and + :ref:`concretization `. + +:mod:`spack.tty` + Basic output functions for all of the messages Spack writes to the + terminal. + +:mod:`spack.color` + Implements a color formatting syntax used by ``spack.tty``. + +:mod:`spack.url` + URL parsing, for deducing names and versions of packages from + tarball URLs. + +:mod:`spack.util` + In this package are a number of utility modules for the rest of + Spack. + +:mod:`spack.error` + :class:`SpackError `, the base class for + Spack's exception hierarchy. + Spec objects ------------------------- @@ -12,9 +262,29 @@ Spec objects Package objects ------------------------- + +Most spack commands +look something like this: + + #. Parse an abstract spec (or specs) from the command line, + #. *Normalize* the spec based on information in package files, + #. *Concretize* the spec according to some customizable policies, + #. Instantiate a package based on the spec, and + #. Call methods (e.g., ``install()``) on the package object. + + + +The information in Package files is used at all stages in this +process. + + +Conceptually, packages are overloaded. They contain + Stage objects ------------------------- +.. _writing-commands: + Writing commands ------------------------- diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst index fa3f5f43e2..39b41f8756 100644 --- a/lib/spack/docs/packaging_guide.rst +++ b/lib/spack/docs/packaging_guide.rst @@ -6,7 +6,7 @@ Packaging Guide This guide is intended for developers or administrators who want to *package* their software so that Spack can install it. We assume that you have at least some familiarty with Python, and that you've read -the :ref:`basic usage guide `, especially the part +the :ref:`basic usage guide `, especially the part about :ref:`specs `. There are two key parts of Spack: @@ -459,6 +459,8 @@ this: :end-before: versions +.. _dependencies: + Dependencies ------------------------------ @@ -513,6 +515,7 @@ on the command line to find specs or to install specs with particular constraints, and package authors can use it to describe relationships between packages. +.. _virtual-dependencies: Virtual dependencies ----------------------------- @@ -993,6 +996,7 @@ do all the same things you'd do with the package's own spec: mpicc = new_path(my_mpi.prefix.bin, 'mpicc') +.. _multimethods: Multimethods and ``@when`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/lib/spack/docs/site_configuration.rst b/lib/spack/docs/site_configuration.rst index ae52df2254..2658e0a310 100644 --- a/lib/spack/docs/site_configuration.rst +++ b/lib/spack/docs/site_configuration.rst @@ -3,6 +3,8 @@ Site-specific configuration =================================== +.. _temp-space: + Temporary space ---------------------------- @@ -47,6 +49,7 @@ in the first directory it finds to which it has write access. Add more elements to the list to indicate where your own site's temporary directory is. +.. _concretization-policies: Concretization policies ---------------------------- diff --git a/lib/spack/spack/compilation.py b/lib/spack/spack/compilation.py index c4561a4122..3a469376a8 100644 --- a/lib/spack/spack/compilation.py +++ b/lib/spack/spack/compilation.py @@ -22,6 +22,16 @@ # along with this program; if not, write to the Free Software Foundation, # Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA ############################################################################## +"""\ +The ``compilation`` module contains utility functions used by the compiler +wrapper script. + +.. todo:: + + Think about moving this into the script to increase compilation + speed. + +""" import os import sys diff --git a/lib/spack/spack/version.py b/lib/spack/spack/version.py index a99a5a7b4d..c433fa8962 100644 --- a/lib/spack/spack/version.py +++ b/lib/spack/spack/version.py @@ -23,7 +23,7 @@ # Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA ############################################################################## """ -This file implements Version and version-ish objects. These are: +This module implements Version and version-ish objects. These are: Version A single version of a package. -- cgit v1.2.3-60-g2f50