From 3b1e6d4bbce4ba98f1da7ffb0e69e77477ee2e18 Mon Sep 17 00:00:00 2001 From: Todd Gamblin Date: Tue, 24 Jun 2014 17:01:29 -0700 Subject: Updated docs. --- lib/spack/docs/basic_usage.rst | 104 ++++++++++++++++- lib/spack/docs/packaging_guide.rst | 8 +- lib/spack/docs/site_configuration.rst | 207 ++++++++++++++++++++++------------ 3 files changed, 232 insertions(+), 87 deletions(-) diff --git a/lib/spack/docs/basic_usage.rst b/lib/spack/docs/basic_usage.rst index 65637f5c74..f7f74a1623 100644 --- a/lib/spack/docs/basic_usage.rst +++ b/lib/spack/docs/basic_usage.rst @@ -78,17 +78,109 @@ Packages like ``mpich`` use traditional version numbers like like ``20130729``. Versions can contain numbers, letters, dashes, underscores, and periods. +Compiler Configuration +----------------------------------- + +Spack has the ability to build packages with multiple compilers and +compiler versions. Spack searches for compilers on your machine +automatically the first time it is run. It does this by inspecting +your path. + ``spack compilers`` -~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~ -You can see supported compilers by running ``spack compilers``. The -output will depend on the platform you run it on. +You can see which compilers spack has found by running ``spack +compilers`` or ``spack compiler list``:: + + $ spack compilers + ==> Available compilers + -- gcc --------------------------------------------------------- + gcc@4.9.0 gcc@4.8.0 gcc@4.7.0 gcc@4.6.2 gcc@4.4.7 + gcc@4.8.2 gcc@4.7.1 gcc@4.6.3 gcc@4.6.1 gcc@4.1.2 + -- intel ------------------------------------------------------- + intel@15.0.0 intel@14.0.0 intel@13.0.0 intel@12.1.0 intel@10.0 + intel@14.0.3 intel@13.1.1 intel@12.1.5 intel@12.0.4 intel@9.1 + intel@14.0.2 intel@13.1.0 intel@12.1.3 intel@11.1 + intel@14.0.1 intel@13.0.1 intel@12.1.2 intel@10.1 + -- clang ------------------------------------------------------- + clang@3.4 clang@3.3 clang@3.2 clang@3.1 + -- pgi --------------------------------------------------------- + pgi@14.3-0 pgi@13.2-0 pgi@12.1-0 pgi@10.9-0 pgi@8.0-1 + pgi@13.10-0 pgi@13.1-1 pgi@11.10-0 pgi@10.2-0 pgi@7.1-3 + pgi@13.6-0 pgi@12.8-0 pgi@11.1-0 pgi@9.0-4 pgi@7.0-6 + +Any of these compilers can be used to build Spack packages. More on +how this is done is in :ref:`sec-specs`. + +``spack compiler add`` +~~~~~~~~~~~~~~~~~~~~~~~ -.. command-output:: spack compilers +If you do not see a compiler in this list, but you want to use it with +Spack, you can simply run ``spack compiler add`` with the path to +where the compiler is installed. For example:: + $ spack compiler add /usr/local/tools/ic-13.0.079 + ==> Added 1 new compiler to /Users/gamblin2/.spackconfig + intel@13.0.079 -Seeing installed packages ------------------------------------ +Or you can run ``spack compiler add`` with no arguments to force +autodetection. This is useful if you do not know where compilers +live, but new compilers have been added to your ``PATH``. For +example, using dotkit, you might do this:: + + $ use gcc-4.9.0 + $ spack compiler add + ==> Added 1 new compiler to /Users/gamblin2/.spackconfig + gcc@4.9.0 + + +``spack compiler info`` +~~~~~~~~~~~~~~~~~~~~~~~ + +If you want to see specifics on a particular compiler, you can run +``spack compiler info`` on it:: + + $ spack compiler info intel@12.1.3 + intel@12.1.3: + cc = /usr/local/bin/icc-12.1.293 + cxx = /usr/local/bin/icpc-12.1.293 + f77 = /usr/local/bin/ifort-12.1.293 + fc = /usr/local/bin/ifort-12.1.293 + +This shows which C, C++, and Fortran compilers were detected by Spack. + + +Manual configuration +~~~~~~~~~~~~~~~~~~~~~~~ + +If autodetection fails, you can manually conigure a compiler by +editing your ``~/.spackconfig`` file. You can do this by running +``spack config edit``, which will open the file in your ``$EDITOR``. + +Each compiler configuration in the file looks like this:: + + ... + [compiler "intel@15.0.0"] + cc = /usr/local/bin/icc-15.0.024-beta + cxx = /usr/local/bin/icpc-15.0.024-beta + f77 = /usr/local/bin/ifort-15.0.024-beta + fc = /usr/local/bin/ifort-15.0.024-beta + ... + +For compilers, like ``clang``, that do not support Fortran, you can simply +put ``None`` for ``f77`` and ``fc``:: + + [compiler "clang@3.3svn"] + cc = /usr/bin/clang + cxx = /usr/bin/clang++ + f77 = None + fc = None + +Once you save the file, the configured compilers will show up in the +list displayed when you run ``spack compilers``. + + +Seeing installed packages ----------------------------------- ``spack find`` ~~~~~~~~~~~~~~~~~~~~~~ diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst index da944b33c8..023896b6c0 100644 --- a/lib/spack/docs/packaging_guide.rst +++ b/lib/spack/docs/packaging_guide.rst @@ -753,8 +753,6 @@ This is due to Spack's compiler interceptors. Environment variables ~~~~~~~~~~~~~~~~~~~~~~~~~~ -.. warning:: Environment variable setting is not fully implemented. - Spack sets a number of standard environment variables so that build systems use its compiler wrappers for their builds. The standard enviroment variables are: @@ -764,10 +762,8 @@ enviroment variables are: ======================= ============================= ``CC`` C compiler ``CXX`` C++ compiler - ``CPP`` C Preprocessor ``F77`` Fortran 77 compiler - ``F90`` Fortran 90 compiler - ``F95`` Fortran 95 compiler + ``FC`` Fortran 90 and above compiler ``CMAKE_PREFIX_PATH`` Path to dependency prefixes for CMake ======================= ============================= @@ -787,8 +783,6 @@ correct compilers into the project's build system. Forked process ~~~~~~~~~~~~~~~~~~~~~ -.. warning:: This is not implemented yet. - To give packages free reign over how they install things, how they modify the environemnt, and how they use Spack's internal APIs, we fork a new process each time we invoke ``install()``. This allows diff --git a/lib/spack/docs/site_configuration.rst b/lib/spack/docs/site_configuration.rst index 2a42cc3d1a..c7465d759b 100644 --- a/lib/spack/docs/site_configuration.rst +++ b/lib/spack/docs/site_configuration.rst @@ -8,6 +8,10 @@ Site-specific configuration Temporary space ---------------------------- +.. deprecated:: + + This will be moved to configuration files. + By default, Spack will try to do all of its building in temporary space. There are two main reasons for this. First, Spack is designed to run out of a user's home directory, and on may systems the home @@ -18,7 +22,7 @@ this. Second, many systems impose quotas on home directories, and helps conserve space for installations in users' home directories. You can customize temporary directories by editing -``lib/spack/spack/globals.py``. Specifically, find this part of the file: +``lib/spack/spack/__init__.py``. Specifically, find this part of the file: .. code-block:: python @@ -32,7 +36,7 @@ You can customize temporary directories by editing # is a shared filesystem. Spack will use the first of these paths # that it can create. tmp_dirs = ['/nfs/tmp2/%u/spack-stage', - '/var/tmp/%u/spcak-stage', + '/var/tmp/%u/spack-stage', '/tmp/%u/spack-stage'] The ``use_tmp_stage`` variable controls whether Spack builds @@ -91,100 +95,155 @@ package's fetch URL. ``spack mirror`` ~~~~~~~~~~~~~~~~~~~~~~~ -You can create a mirror using the ``spack mirror`` command, assuming -you're on a machine where you can access the internet. ``spack -mirror`` will iterate through all of Spack's packages and download the -safe ones into a directory structure like the one above. Here is what -it looks like: +Mirrors are managed with the ``spack mirror`` command. The help for +``spack mirror`` looks like this:: -.. code-block:: bash + $ spack mirror -h + usage: spack mirror [-h] SUBCOMMAND ... - $ spack mirror mirror-dir - ==> No safe (checksummed) versions for package callpath. Skipping. - ==> Trying to fetch from http://www.cmake.org/files/v2.8/cmake-2.8.10.2.tar.gz - ################################################################ 90.2% - ==> Added mirror-dir/cmake/cmake-2.8.10.2.tar.gz to mirror - ==> Trying to fetch from http://www.dyninst.org/sites/default/files/downloads/dyninst/8.1.2/DyninstAPI-8.1.2.tgz - ########################################################### 82.0% - ==> Added mirror-dir/dyninst/DyninstAPI-8.1.2.tgz to mirror - ==> Trying to fetch from http://www.dyninst.org/sites/default/files/downloads/dyninst/8.1.2/DyninstAPI-8.1.1.tgz - ######################################################################## 100.0% - ==> Added mirror-dir/dyninst/DyninstAPI-8.1.1.tgz to mirror + positional arguments: + SUBCOMMAND + create Create a directory to be used as a spack mirror, and fill + it with package archives. + add Add a mirror to Spack. + remove Remove a mirror by name. + list Print out available mirrors to the console. - ... + optional arguments: + -h, --help show this help message and exit + +The ``create`` command actually builds a mirror by fetching all of its +packages from the internet and checksumming them. + +The other three commands are for managing mirror configuration. They +control the URL(s) from which Spack downloads its packages. + + +``spack mirror create`` +~~~~~~~~~~~~~~~~~~~~~~~ + +You can create a mirror using the ``spack mirror create`` command, assuming +you're on a machine where you can access the internet. + +The command will iterate through all of Spack's packages and download +the safe ones into a directory structure like the one above. Here is +what it looks like: + +.. code-block:: bash + + $ spack mirror create libelf libdwarf + ==> Created new mirror in spack-mirror-2014-06-24 + ==> Trying to fetch from http://www.mr511.de/software/libelf-0.8.13.tar.gz + ########################################################## 81.6% + ==> Checksum passed for libelf@0.8.13 + ==> Added spack-mirror-2014-06-24/libelf/libelf-0.8.13.tar.gz to mirror ==> Trying to fetch from http://www.mr511.de/software/libelf-0.8.12.tar.gz - ############################################################## 86.5% - ==> Added mirror-dir/libelf/libelf-0.8.12.tar.gz to mirror - ==> Trying to fetch from http://download.savannah.gnu.org/releases/libunwind/libunwind-1.1.tar.gz - ################################################################ 89.3% - ==> Added mirror-dir/libunwind/libunwind-1.1.tar.gz to mirror - ==> Trying to fetch from http://www.mpich.org/static/downloads/3.0.4/mpich-3.0.4.tar.gz - ##################################################################### 96.4% - ==> Added mirror-dir/mpich/mpich-3.0.4.tar.gz to mirror - ==> No safe (checksummed) versions for package mpileaks. Skipping. - ==> Trying to fetch from http://mvapich.cse.ohio-state.edu/download/mvapich2/mv2/mvapich2-1.9.tgz - ####################################################################### 99.2% - ==> Added mirror-dir/mvapich2/mvapich2-1.9.tgz to mirror - ==> Created Spack mirror in mirror-dir - -Once this is done, you can tar up the ``mirror-dir`` directory and + ###################################################################### 98.6% + ==> Checksum passed for libelf@0.8.12 + ==> Added spack-mirror-2014-06-24/libelf/libelf-0.8.12.tar.gz to mirror + ==> Trying to fetch from http://www.prevanders.net/libdwarf-20130207.tar.gz + ###################################################################### 97.3% + ==> Checksum passed for libdwarf@20130207 + ==> Added spack-mirror-2014-06-24/libdwarf/libdwarf-20130207.tar.gz to mirror + ==> Trying to fetch from http://www.prevanders.net/libdwarf-20130126.tar.gz + ######################################################## 78.9% + ==> Checksum passed for libdwarf@20130126 + ==> Added spack-mirror-2014-06-24/libdwarf/libdwarf-20130126.tar.gz to mirror + ==> Trying to fetch from http://www.prevanders.net/libdwarf-20130729.tar.gz + ############################################################# 84.7% + ==> Checksum passed for libdwarf@20130729 + ==> Added spack-mirror-2014-06-24/libdwarf/libdwarf-20130729.tar.gz to mirror + +Once this is done, you can tar up the ``spack-mirror-2014-06-24`` directory and copy it over to the machine you want it hosted on. -Normally, ``spack mirror`` downloads all the archives it has checksums -for. If you want to only create a mirror for a subset of packages, -you can do that by supplying a list of package names on the command -line after ``spack mirror``. +Custom package sets +^^^^^^^^^^^^^^^^^^^^^^^^ +Normally, ``spack mirror create`` downloads all the archives it has +checksums for. If you want to only create a mirror for a subset of +packages, you can do that by supplying a list of package specs on the +command line after ``spack mirror create``. For example, this +command:: -Setting up a mirror -~~~~~~~~~~~~~~~~~~~~~~~~~~~ + $ spack mirror create libelf@0.8.12: boost@1.44: -Once you have a mirrror, you need to let spack know about it. Find -this section in ``globals.py``: +Will create a mirror for libelf versions greater than or equal to +0.8.12 and boost versions greater than or equal to 1.44. -.. code-block:: python +Mirror files +^^^^^^^^^^^^^^^^^^^^^^^^ - # - # Places to download tarballs from. Examples: - # - # For a local directory: - # mirrors = ['file:///Users/gamblin2/spack-mirror'] - # - # For a website: - # mirrors = ['http://spackports.org/spack-mirror/'] - # - # For no mirrors: - # mirrors = [] - # - mirrors = [] +If you have a *very* large number of packages you want to mirror, you +can supply a file with specs in it, one per line:: -Change the list of mirrors to include the location where you copied -your directory created by ``spack mirror``. If it's on a local -filesystem, you want to use a ``file://`` URL. If it's on a private -web server, you will need to use a ``http://`` or ``https://`` URL. + $ cat specs.txt + libdwarf + libelf@0.8.12: + boost@1.44: + boost@1.39.0 + ... + $ spack mirror create -f specs.txt + ... -Mirror precedence -~~~~~~~~~~~~~~~~~~~~~~~~~ +This is useful if there is a specific suite of software managed by +your site. + + +``spack mirror add`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~ -If you have specified mirrors in ``globals.py``, then Spack will try -to find an archive in each mirror in the list, in order, before it -downloads from the URL in a package file. +Once you have a mirrror, you need to let spack know about it. This is +relatively simple. First, figure out the URL for the mirror. If it's +a file, you can use a file URL like this one:: -You can test whether a mirror is working properly by first setting it -in ``globals.py``, then running ``spack fetch`` to test fetching the -archive. Example: + file:///Users/gamblin2/spack-mirror-2014-06-24 + +That points to the directory on the local filesystem. If it were on a +web server, you could use a URL like this one: + + https://example.com/some/web-hosted/directory/spack-mirror-2014-06-24 + +Spack will use the URL as the root for all of the packages it fetches. +You can tell your Spack installation to use that mirror like this: .. code-block:: bash - $ spack fetch dyninst - ==> Trying to fetch from file:///Users/gamblin2/mirror-dir/dyninst/DyninstAPI-8.1.2.tgz + $ spack mirror add local_filesystem file:///Users/gamblin2/spack-mirror-2014-06-24 + +Each mirror has a name so that you can refer to it again later. + +``spack mirror list`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you want to see all the mirrors Spack knows about you can run ``spack mirror list``:: + + $ spack mirror list + local_filesystem file:///Users/gamblin2/spack-mirror-2014-06-24 + +``spack mirror remove`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +And, if you want to remove a mirror, just remove it by name:: + + $ spack mirror remove local_filesystem + $ spack mirror list + ==> No mirrors configured. + +Mirror precedence +~~~~~~~~~~~~~~~~~~~~~~~~~ - ==> Checksum passed for dyninst +Adding a mirror really just adds a section in ``~/.spackconfig``:: -If the mirror setup worked, you should see the mirror URL in the fetch -output, like the ``file://`` URL above. + [mirror "local_filesystem"] + url = file:///Users/gamblin2/spack-mirror-2014-06-24 + [mirror "remote_server"] + url = https://example.com/some/web-hosted/directory/spack-mirror-2014-06-24 +If you want to change the order in which mirrors are searched for +packages, you can edit this file and reorder the sections. Spack will +search the topmost mirror first and the bottom-most mirror last. .. _concretization-policies: -- cgit v1.2.3-60-g2f50