From d0870865817054288fbebb7dda626b6e8aeb0114 Mon Sep 17 00:00:00 2001 From: Todd Gamblin Date: Sat, 21 May 2016 14:33:05 -0700 Subject: update documentation. --- lib/spack/docs/configuration.rst | 233 ++++++++++++++++++++++++++++++++++ lib/spack/docs/index.rst | 2 +- lib/spack/docs/packaging_guide.rst | 60 +-------- lib/spack/docs/site_configuration.rst | 173 ------------------------- 4 files changed, 238 insertions(+), 230 deletions(-) create mode 100644 lib/spack/docs/configuration.rst delete mode 100644 lib/spack/docs/site_configuration.rst (limited to 'lib') diff --git a/lib/spack/docs/configuration.rst b/lib/spack/docs/configuration.rst new file mode 100644 index 0000000000..c0b79a7f44 --- /dev/null +++ b/lib/spack/docs/configuration.rst @@ -0,0 +1,233 @@ +.. _configuration: + +Configuration +=================================== + +.. _temp-space: + +Temporary space +---------------------------- + +.. warning:: Temporary space configuration will eventually be moved to + configuration files, but currently these settings are in + ``lib/spack/spack/__init__.py`` + +By default, Spack will try to do all of its building in temporary +space. There are two main reasons for this. First, Spack is designed +to run out of a user's home directory, and on may systems the home +directory is network mounted and potentially not a very fast +filesystem. We create build stages in a temporary directory to avoid +this. Second, many systems impose quotas on home directories, and +``/tmp`` or similar directories often have more available space. This +helps conserve space for installations in users' home directories. + +You can customize temporary directories by editing +``lib/spack/spack/__init__.py``. Specifically, find this part of the file: + +.. code-block:: python + + # Whether to build in tmp space or directly in the stage_path. + # If this is true, then spack will make stage directories in + # a tmp filesystem, and it will symlink them into stage_path. + use_tmp_stage = True + + # Locations to use for staging and building, in order of preference + # Use a %u to add a username to the stage paths here, in case this + # is a shared filesystem. Spack will use the first of these paths + # that it can create. + tmp_dirs = ['/nfs/tmp2/%u/spack-stage', + '/var/tmp/%u/spack-stage', + '/tmp/%u/spack-stage'] + +The ``use_tmp_stage`` variable controls whether Spack builds +**directly** inside the ``var/spack/`` directory. Normally, Spack +will try to find a temporary directory for a build, then it *symlinks* +that temporary directory into ``var/spack/`` so that you can keep +track of what temporary directories Spack is using. + +The ``tmp_dirs`` variable is a list of paths Spack should search when +trying to find a temporary directory. They can optionally contain a +``%u``, which will substitute the current user's name into the path. +The list is searched in order, and Spack will create a temporary stage +in the first directory it finds to which it has write access. Add +more elements to the list to indicate where your own site's temporary +directory is. + + +External Packages +---------------------------- +Spack can be configured to use externally-installed +packages rather than building its own packages. This may be desirable +if machines ship with system packages, such as a customized MPI +that should be used instead of Spack building its own MPI. + +External packages are configured through the ``packages.yaml`` file found +in a Spack installation's ``etc/spack/`` or a user's ``~/.spack/`` +directory. Here's an example of an external configuration: + +.. code-block:: yaml + + packages: + openmpi: + paths: + openmpi@1.4.3%gcc@4.4.7=chaos_5_x86_64_ib: /opt/openmpi-1.4.3 + openmpi@1.4.3%gcc@4.4.7=chaos_5_x86_64_ib+debug: /opt/openmpi-1.4.3-debug + openmpi@1.6.5%intel@10.1=chaos_5_x86_64_ib: /opt/openmpi-1.6.5-intel + +This example lists three installations of OpenMPI, one built with gcc, +one built with gcc and debug information, and another built with Intel. +If Spack is asked to build a package that uses one of these MPIs as a +dependency, it will use the the pre-installed OpenMPI in +the given directory. + +Each ``packages.yaml`` begins with a ``packages:`` token, followed +by a list of package names. To specify externals, add a ``paths`` +token under the package name, which lists externals in a +``spec : /path`` format. Each spec should be as +well-defined as reasonably possible. If a +package lacks a spec component, such as missing a compiler or +package version, then Spack will guess the missing component based +on its most-favored packages, and it may guess incorrectly. + +Each package version and compilers listed in an external should +have entries in Spack's packages and compiler configuration, even +though the package and compiler may not every be built. + +The packages configuration can tell Spack to use an external location +for certain package versions, but it does not restrict Spack to using +external packages. In the above example, if an OpenMPI 1.8.4 became +available Spack may choose to start building and linking with that version +rather than continue using the pre-installed OpenMPI versions. + +To prevent this, the ``packages.yaml`` configuration also allows packages +to be flagged as non-buildable. The previous example could be modified to +be: + +.. code-block:: yaml + + packages: + openmpi: + paths: + openmpi@1.4.3%gcc@4.4.7=chaos_5_x86_64_ib: /opt/openmpi-1.4.3 + openmpi@1.4.3%gcc@4.4.7=chaos_5_x86_64_ib+debug: /opt/openmpi-1.4.3-debug + openmpi@1.6.5%intel@10.1=chaos_5_x86_64_ib: /opt/openmpi-1.6.5-intel + buildable: False + +The addition of the ``buildable`` flag tells Spack that it should never build +its own version of OpenMPI, and it will instead always rely on a pre-built +OpenMPI. Similar to ``paths``, ``buildable`` is specified as a property under +a package name. + +The ``buildable`` does not need to be paired with external packages. +It could also be used alone to forbid packages that may be +buggy or otherwise undesirable. + + +Concretization Preferences +-------------------------------- + +Spack can be configured to prefer certain compilers, package +versions, depends_on, and variants during concretization. +The preferred configuration can be controlled via the +``~/.spack/packages.yaml`` file for user configuations, or the +``etc/spack/packages.yaml`` site configuration. + + +Here's an example packages.yaml file that sets preferred packages: + +.. code-block:: sh + + packages: + dyninst: + compiler: [gcc@4.9] + variants: +debug + gperftools: + version: [2.2, 2.4, 2.3] + all: + compiler: [gcc@4.4.7, gcc@4.6:, intel, clang, pgi] + providers: + mpi: [mvapich, mpich, openmpi] + + +At a high level, this example is specifying how packages should be +concretized. The dyninst package should prefer using gcc 4.9 and +be built with debug options. The gperftools package should prefer version +2.2 over 2.4. Every package on the system should prefer mvapich for +its MPI and gcc 4.4.7 (except for Dyninst, which overrides this by preferring gcc 4.9). +These options are used to fill in implicit defaults. Any of them can be overwritten +on the command line if explicitly requested. + +Each packages.yaml file begins with the string ``packages:`` and +package names are specified on the next level. The special string ``all`` +applies settings to each package. Underneath each package name is +one or more components: ``compiler``, ``variants``, ``version``, +or ``providers``. Each component has an ordered list of spec +``constraints``, with earlier entries in the list being preferred over +later entries. + +Sometimes a package installation may have constraints that forbid +the first concretization rule, in which case Spack will use the first +legal concretization rule. Going back to the example, if a user +requests gperftools 2.3 or later, then Spack will install version 2.4 +as the 2.4 version of gperftools is preferred over 2.3. + +An explicit concretization rule in the preferred section will always +take preference over unlisted concretizations. In the above example, +xlc isn't listed in the compiler list. Every listed compiler from +gcc to pgi will thus be preferred over the xlc compiler. + +The syntax for the ``provider`` section differs slightly from other +concretization rules. A provider lists a value that packages may +``depend_on`` (e.g, mpi) and a list of rules for fulfilling that +dependency. + + + +Profiling +------------------ + +Spack has some limited built-in support for profiling, and can report +statistics using standard Python timing tools. To use this feature, +supply ``-p`` to Spack on the command line, before any subcommands. + +.. _spack-p: + +``spack -p`` +~~~~~~~~~~~~~~~~~ + +``spack -p`` output looks like this: + +.. code-block:: sh + + $ spack -p graph dyninst + o dyninst + |\ + | |\ + | o | libdwarf + |/ / + o | libelf + / + o boost + + 307670 function calls (305943 primitive calls) in 0.127 seconds + + Ordered by: internal time + + ncalls tottime percall cumtime percall filename:lineno(function) + 853 0.021 0.000 0.066 0.000 inspect.py:472(getmodule) + 51197 0.011 0.000 0.018 0.000 inspect.py:51(ismodule) + 73961 0.010 0.000 0.010 0.000 {isinstance} + 1762 0.006 0.000 0.053 0.000 inspect.py:440(getsourcefile) + 32075 0.006 0.000 0.006 0.000 {hasattr} + 1760 0.004 0.000 0.004 0.000 {posix.stat} + 2240 0.004 0.000 0.004 0.000 {posix.lstat} + 2602 0.004 0.000 0.011 0.000 inspect.py:398(getfile) + 771 0.004 0.000 0.077 0.000 inspect.py:518(findsource) + 2656 0.004 0.000 0.004 0.000 {method 'match' of '_sre.SRE_Pattern' objects} + 30772 0.003 0.000 0.003 0.000 {method 'get' of 'dict' objects} + ... + +The bottom of the output shows the top most time consuming functions, +slowest on top. The profiling support is from Python's built-in tool, +`cProfile +`_. diff --git a/lib/spack/docs/index.rst b/lib/spack/docs/index.rst index d6ce52b747..98ed9ff0fe 100644 --- a/lib/spack/docs/index.rst +++ b/lib/spack/docs/index.rst @@ -47,7 +47,7 @@ Table of Contents basic_usage packaging_guide mirrors - site_configuration + configuration developer_guide command_index package_list diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst index 650e0ee3b2..63c411ffb5 100644 --- a/lib/spack/docs/packaging_guide.rst +++ b/lib/spack/docs/packaging_guide.rst @@ -1648,8 +1648,8 @@ point will Spack call the ``install()`` method for your package. Concretization in Spack is based on certain selection policies that tell Spack how to select, e.g., a version, when one is not specified explicitly. Concretization policies are discussed in more detail in -:ref:`site-configuration`. Sites using Spack can customize them to -match the preferences of their own users. +:ref:`configuration`. Sites using Spack can customize them to match +the preferences of their own users. .. _spack-spec: @@ -1682,60 +1682,8 @@ be concretized on their system. For example, one user may prefer packages built with OpenMPI and the Intel compiler. Another user may prefer packages be built with MVAPICH and GCC. -Spack can be configured to prefer certain compilers, package -versions, depends_on, and variants during concretization. -The preferred configuration can be controlled via the -``~/.spack/packages.yaml`` file for user configuations, or the -``etc/spack/packages.yaml`` site configuration. - - -Here's an example packages.yaml file that sets preferred packages: - -.. code-block:: sh - - packages: - dyninst: - compiler: [gcc@4.9] - variants: +debug - gperftools: - version: [2.2, 2.4, 2.3] - all: - compiler: [gcc@4.4.7, gcc@4.6:, intel, clang, pgi] - providers: - mpi: [mvapich, mpich, openmpi] - - -At a high level, this example is specifying how packages should be -concretized. The dyninst package should prefer using gcc 4.9 and -be built with debug options. The gperftools package should prefer version -2.2 over 2.4. Every package on the system should prefer mvapich for -its MPI and gcc 4.4.7 (except for Dyninst, which overrides this by preferring gcc 4.9). -These options are used to fill in implicit defaults. Any of them can be overwritten -on the command line if explicitly requested. - -Each packages.yaml file begins with the string ``packages:`` and -package names are specified on the next level. The special string ``all`` -applies settings to each package. Underneath each package name is -one or more components: ``compiler``, ``variants``, ``version``, -or ``providers``. Each component has an ordered list of spec -``constraints``, with earlier entries in the list being preferred over -later entries. - -Sometimes a package installation may have constraints that forbid -the first concretization rule, in which case Spack will use the first -legal concretization rule. Going back to the example, if a user -requests gperftools 2.3 or later, then Spack will install version 2.4 -as the 2.4 version of gperftools is preferred over 2.3. - -An explicit concretization rule in the preferred section will always -take preference over unlisted concretizations. In the above example, -xlc isn't listed in the compiler list. Every listed compiler from -gcc to pgi will thus be preferred over the xlc compiler. - -The syntax for the ``provider`` section differs slightly from other -concretization rules. A provider lists a value that packages may -``depend_on`` (e.g, mpi) and a list of rules for fulfilling that -dependency. +See the `documentation in the config section `_ +for more details. .. _install-method: diff --git a/lib/spack/docs/site_configuration.rst b/lib/spack/docs/site_configuration.rst deleted file mode 100644 index 3abfa21a9d..0000000000 --- a/lib/spack/docs/site_configuration.rst +++ /dev/null @@ -1,173 +0,0 @@ -.. _site-configuration: - -Site configuration -=================================== - -.. _temp-space: - -Temporary space ----------------------------- - -.. warning:: Temporary space configuration will eventually be moved to - configuration files, but currently these settings are in - ``lib/spack/spack/__init__.py`` - -By default, Spack will try to do all of its building in temporary -space. There are two main reasons for this. First, Spack is designed -to run out of a user's home directory, and on may systems the home -directory is network mounted and potentially not a very fast -filesystem. We create build stages in a temporary directory to avoid -this. Second, many systems impose quotas on home directories, and -``/tmp`` or similar directories often have more available space. This -helps conserve space for installations in users' home directories. - -You can customize temporary directories by editing -``lib/spack/spack/__init__.py``. Specifically, find this part of the file: - -.. code-block:: python - - # Whether to build in tmp space or directly in the stage_path. - # If this is true, then spack will make stage directories in - # a tmp filesystem, and it will symlink them into stage_path. - use_tmp_stage = True - - # Locations to use for staging and building, in order of preference - # Use a %u to add a username to the stage paths here, in case this - # is a shared filesystem. Spack will use the first of these paths - # that it can create. - tmp_dirs = ['/nfs/tmp2/%u/spack-stage', - '/var/tmp/%u/spack-stage', - '/tmp/%u/spack-stage'] - -The ``use_tmp_stage`` variable controls whether Spack builds -**directly** inside the ``var/spack/`` directory. Normally, Spack -will try to find a temporary directory for a build, then it *symlinks* -that temporary directory into ``var/spack/`` so that you can keep -track of what temporary directories Spack is using. - -The ``tmp_dirs`` variable is a list of paths Spack should search when -trying to find a temporary directory. They can optionally contain a -``%u``, which will substitute the current user's name into the path. -The list is searched in order, and Spack will create a temporary stage -in the first directory it finds to which it has write access. Add -more elements to the list to indicate where your own site's temporary -directory is. - - -External Packages -~~~~~~~~~~~~~~~~~~~~~ -Spack can be configured to use externally-installed -packages rather than building its own packages. This may be desirable -if machines ship with system packages, such as a customized MPI -that should be used instead of Spack building its own MPI. - -External packages are configured through the ``packages.yaml`` file found -in a Spack installation's ``etc/spack/`` or a user's ``~/.spack/`` -directory. Here's an example of an external configuration: - -.. code-block:: yaml - - packages: - openmpi: - paths: - openmpi@1.4.3%gcc@4.4.7=chaos_5_x86_64_ib: /opt/openmpi-1.4.3 - openmpi@1.4.3%gcc@4.4.7=chaos_5_x86_64_ib+debug: /opt/openmpi-1.4.3-debug - openmpi@1.6.5%intel@10.1=chaos_5_x86_64_ib: /opt/openmpi-1.6.5-intel - -This example lists three installations of OpenMPI, one built with gcc, -one built with gcc and debug information, and another built with Intel. -If Spack is asked to build a package that uses one of these MPIs as a -dependency, it will use the the pre-installed OpenMPI in -the given directory. - -Each ``packages.yaml`` begins with a ``packages:`` token, followed -by a list of package names. To specify externals, add a ``paths`` -token under the package name, which lists externals in a -``spec : /path`` format. Each spec should be as -well-defined as reasonably possible. If a -package lacks a spec component, such as missing a compiler or -package version, then Spack will guess the missing component based -on its most-favored packages, and it may guess incorrectly. - -Each package version and compilers listed in an external should -have entries in Spack's packages and compiler configuration, even -though the package and compiler may not every be built. - -The packages configuration can tell Spack to use an external location -for certain package versions, but it does not restrict Spack to using -external packages. In the above example, if an OpenMPI 1.8.4 became -available Spack may choose to start building and linking with that version -rather than continue using the pre-installed OpenMPI versions. - -To prevent this, the ``packages.yaml`` configuration also allows packages -to be flagged as non-buildable. The previous example could be modified to -be: - -.. code-block:: yaml - - packages: - openmpi: - paths: - openmpi@1.4.3%gcc@4.4.7=chaos_5_x86_64_ib: /opt/openmpi-1.4.3 - openmpi@1.4.3%gcc@4.4.7=chaos_5_x86_64_ib+debug: /opt/openmpi-1.4.3-debug - openmpi@1.6.5%intel@10.1=chaos_5_x86_64_ib: /opt/openmpi-1.6.5-intel - buildable: False - -The addition of the ``buildable`` flag tells Spack that it should never build -its own version of OpenMPI, and it will instead always rely on a pre-built -OpenMPI. Similar to ``paths``, ``buildable`` is specified as a property under -a package name. - -The ``buildable`` does not need to be paired with external packages. -It could also be used alone to forbid packages that may be -buggy or otherwise undesirable. - - -Profiling -~~~~~~~~~~~~~~~~~~~~~ - -Spack has some limited built-in support for profiling, and can report -statistics using standard Python timing tools. To use this feature, -supply ``-p`` to Spack on the command line, before any subcommands. - -.. _spack-p: - -``spack -p`` -^^^^^^^^^^^^^^^^^^ - -``spack -p`` output looks like this: - -.. code-block:: sh - - $ spack -p graph dyninst - o dyninst - |\ - | |\ - | o | libdwarf - |/ / - o | libelf - / - o boost - - 307670 function calls (305943 primitive calls) in 0.127 seconds - - Ordered by: internal time - - ncalls tottime percall cumtime percall filename:lineno(function) - 853 0.021 0.000 0.066 0.000 inspect.py:472(getmodule) - 51197 0.011 0.000 0.018 0.000 inspect.py:51(ismodule) - 73961 0.010 0.000 0.010 0.000 {isinstance} - 1762 0.006 0.000 0.053 0.000 inspect.py:440(getsourcefile) - 32075 0.006 0.000 0.006 0.000 {hasattr} - 1760 0.004 0.000 0.004 0.000 {posix.stat} - 2240 0.004 0.000 0.004 0.000 {posix.lstat} - 2602 0.004 0.000 0.011 0.000 inspect.py:398(getfile) - 771 0.004 0.000 0.077 0.000 inspect.py:518(findsource) - 2656 0.004 0.000 0.004 0.000 {method 'match' of '_sre.SRE_Pattern' objects} - 30772 0.003 0.000 0.003 0.000 {method 'get' of 'dict' objects} - ... - -The bottom of the output shows the top most time consuming functions, -slowest on top. The profiling support is from Python's built-in tool, -`cProfile -`_. -- cgit v1.2.3-70-g09d2