From bdd1069a9ee8ed5ddc5fee5006782543c04002b3 Mon Sep 17 00:00:00 2001 From: Todd Gamblin Date: Sun, 13 Nov 2016 13:29:28 -0700 Subject: Add SC16 tutorial to the Spack docs. --- lib/spack/docs/basic_usage_tutorial.rst | 1253 -------------------- lib/spack/docs/index.rst | 6 +- lib/spack/docs/module_file_tutorial.rst | 982 --------------- lib/spack/docs/package_tutorial.rst | 463 -------- lib/spack/docs/packaging_tutorial/0.package.py | 64 - lib/spack/docs/packaging_tutorial/1.package.py | 49 - lib/spack/docs/packaging_tutorial/2.package.py | 21 - lib/spack/docs/packaging_tutorial/3.package.py | 17 - lib/spack/docs/packaging_tutorial/4.package.py | 26 - lib/spack/docs/tutorial/.gitattributes | 1 + lib/spack/docs/tutorial/Spack-SC16-Tutorial.pdf | Bin 0 -> 7067546 bytes lib/spack/docs/tutorial/examples/0.package.py | 64 + lib/spack/docs/tutorial/examples/1.package.py | 49 + lib/spack/docs/tutorial/examples/2.package.py | 21 + lib/spack/docs/tutorial/examples/3.package.py | 17 + lib/spack/docs/tutorial/examples/4.package.py | 26 + .../docs/tutorial/sc16-tutorial-slide-preview.png | Bin 0 -> 71641 bytes lib/spack/docs/tutorial_sc16.rst | 48 + lib/spack/docs/tutorial_sc16_modules.rst | 982 +++++++++++++++ lib/spack/docs/tutorial_sc16_packaging.rst | 463 ++++++++ lib/spack/docs/tutorial_sc16_spack_basics.rst | 1253 ++++++++++++++++++++ 21 files changed, 2926 insertions(+), 2879 deletions(-) delete mode 100644 lib/spack/docs/basic_usage_tutorial.rst delete mode 100644 lib/spack/docs/module_file_tutorial.rst delete mode 100644 lib/spack/docs/package_tutorial.rst delete mode 100644 lib/spack/docs/packaging_tutorial/0.package.py delete mode 100644 lib/spack/docs/packaging_tutorial/1.package.py delete mode 100644 lib/spack/docs/packaging_tutorial/2.package.py delete mode 100644 lib/spack/docs/packaging_tutorial/3.package.py delete mode 100644 lib/spack/docs/packaging_tutorial/4.package.py create mode 100644 lib/spack/docs/tutorial/.gitattributes create mode 100644 lib/spack/docs/tutorial/Spack-SC16-Tutorial.pdf create mode 100644 lib/spack/docs/tutorial/examples/0.package.py create mode 100644 lib/spack/docs/tutorial/examples/1.package.py create mode 100644 lib/spack/docs/tutorial/examples/2.package.py create mode 100644 lib/spack/docs/tutorial/examples/3.package.py create mode 100644 lib/spack/docs/tutorial/examples/4.package.py create mode 100644 lib/spack/docs/tutorial/sc16-tutorial-slide-preview.png create mode 100644 lib/spack/docs/tutorial_sc16.rst create mode 100644 lib/spack/docs/tutorial_sc16_modules.rst create mode 100644 lib/spack/docs/tutorial_sc16_packaging.rst create mode 100644 lib/spack/docs/tutorial_sc16_spack_basics.rst diff --git a/lib/spack/docs/basic_usage_tutorial.rst b/lib/spack/docs/basic_usage_tutorial.rst deleted file mode 100644 index cf8c654c0f..0000000000 --- a/lib/spack/docs/basic_usage_tutorial.rst +++ /dev/null @@ -1,1253 +0,0 @@ -.. _basics_tutorial: - -========================================= -Basic Installation Tutorial -========================================= - -This tutorial will guide you through the process of installing software -using Spack. We will first cover the `spack install` command, focusing on -the power of the spec syntax and the flexibility it gives to users. We -will also cover the `spack find` command for viewing installed packages -and the `spack uninstall` command. Finally, we will touch on how Spack -manages compilers, especially as it relates to using Spack-built -compilers within Spack. We will include full output from all of the -commands demonstrated, although we will frequently call attention to only -small portions of that output (or merely to the fact that it -succeeded). The provided output is all from a cluster running Red Hat -Enterprise Linux. - -.. _basics-tutorial-install: - ----------------- -Installing Spack ----------------- - -Spack works out of the box. Simply clone spack and get going. - -.. code-block:: console - - becker33:~$ git clone https://github.com/LLNL/spack.git - Initialized empty Git repository in /g/g0/becker33/testclone/spack/.git/ - remote: Counting objects: 47125, done. - remote: Compressing objects: 100% (68/68), done. - remote: Total 47125 (delta 16), reused 2 (delta 2), pack-reused 47047 - Receiving objects: 100% (47125/47125), 12.02 MiB | 2.11 MiB/s, done. - Resolving deltas: 100% (23044/23044), done. - becker33:~$ cd spack - -Then add Spack to your path. - -.. code-block:: console - - becker33:spack$ export PATH=~/spack/bin/spack:$PATH - -You're good to go! - ------------------ -What is in Spack? ------------------ - -The ``spack list`` command shows available packages. - -.. code-block:: console - - becker33:spack$ spack list - ==> 1016 packages. - abinit hwloc piranha r-rjava - ack hydra pixman r-rjson - activeharmony hypre pkg-config r-rjsonio - ... - -The ``spack list`` command can also take a query string. Spack -automatically adds wildcards to both ends of the string. For example, -we can view all available python packages. - -.. code-block:: console - - becker33:spack$ spack list py - ==> 129 packages. - py-3to2 py-epydoc py-nestle py-pycparser py-six - py-alabaster py-flake8 py-netcdf py-pydatalog py-sncosmo - py-argcomplete py-funcsigs py-networkx py-pyelftools py-snowballstemmer - ... - -------------------- -Installing Packages -------------------- - -Installing a package with Spack is very simple. To install a piece of -software, simply type ``spack install `` - -.. code-block:: console - - $ spack install libelf - ==> Installing libelf - ==> Trying to fetch from file:///g/g0/becker33/SC16-spack/spack/var/spack/cache/libelf/libelf-0.8.13.tar.gz - curl: (37) Couldn't open file /g/g0/becker33/SC16-spack/spack/var/spack/cache/libelf/libelf-0.8.13.tar.gz - ==> Fetching from file:///g/g0/becker33/SC16-spack/spack/var/spack/cache/libelf/libelf-0.8.13.tar.gz failed. - ==> Trying to fetch from http://www.mr511.de/software/libelf-0.8.13.tar.gz - ################################################################################################################################################################################# 100.0% - ==> Staging archive: /g/g0/becker33/SC16-spack/spack/var/spack/stage/libelf-0.8.13-csrt4qxfkhjgn5xg3zjpkir7xdnszl2a/libelf-0.8.13.tar.gz - ==> Created stage in /g/g0/becker33/SC16-spack/spack/var/spack/stage/libelf-0.8.13-csrt4qxfkhjgn5xg3zjpkir7xdnszl2a - ==> No patches needed for libelf - ==> Building libelf [Package] - ==> Executing phase : 'install' - ==> Successfully installed libelf - Fetch: 1.21s. Build: 8.42s. Total: 9.62s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libelf-0.8.13-csrt4qxfkhjgn5xg3zjpkir7xdnszl2a - - -Spack's spec syntax is the interface by which we can request specific -configurations of the package. The ``%`` sigil is used to specify -compilers. - -.. code-block:: console - - $ spack install libelf %intel - ==> Installing libelf - ==> Trying to fetch from file:///g/g0/becker33/SC16-spack/spack/var/spack/cache/libelf/libelf-0.8.13.tar.gz - ################################################################################################################################################################################# 100.0% - ==> Staging archive: /g/g0/becker33/SC16-spack/spack/var/spack/stage/libelf-0.8.13-7wgp32xksatkvw2tbssmehw2t5tnxndj/libelf-0.8.13.tar.gz - ==> Created stage in /g/g0/becker33/SC16-spack/spack/var/spack/stage/libelf-0.8.13-7wgp32xksatkvw2tbssmehw2t5tnxndj - ==> No patches needed for libelf - ==> Building libelf [Package] - ==> Executing phase : 'install' - ==> Successfully installed libelf - Fetch: 0.09s. Build: 50.64s. Total: 50.72s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/intel-16.0.3/libelf-0.8.13-7wgp32xksatkvw2tbssmehw2t5tnxndj - -Note that this installation is located separately from the previous -one. We will discuss this in more detail later, but this is part of what -allows Spack to support arbitrarily versioned software. - -You can check for particular versions before requesting them. We will -use the ``spack versions`` command to see the available versions, and then -install a different version of ``libelf``. - -.. code-block:: console - - $ spack versions libelf - ==> Safe versions (already checksummed): - 0.8.13 - 0.8.12 - ==> Remote versions (not yet checksummed): - 0.8.11 - 0.8.10 - 0.8.9 - 0.8.8 - 0.8.7 - 0.8.6 - 0.8.5 - 0.8.4 - 0.8.3 - 0.8.2 - 0.8.0 - 0.7.0 - 0.6.4 - 0.5.2 - - -The ``@`` sigil is used to specify versions, both of packages and of -compilers. - -.. code-block:: console - - $ spack install libelf @0.8.12 - ==> Installing libelf - ==> Trying to fetch from file:///g/g0/becker33/SC16-spack/spack/var/spack/cache/libelf/libelf-0.8.12.tar.gz - curl: (37) Couldn't open file /g/g0/becker33/SC16-spack/spack/var/spack/cache/libelf/libelf-0.8.12.tar.gz - ==> Fetching from file:///g/g0/becker33/SC16-spack/spack/var/spack/cache/libelf/libelf-0.8.12.tar.gz failed. - ==> Trying to fetch from http://www.mr511.de/software/libelf-0.8.12.tar.gz - ################################################################################################################################################################################# 100.0% - ==> Staging archive: /g/g0/becker33/SC16-spack/spack/var/spack/stage/libelf-0.8.12-ipggckv6i7h44iryzfa4dwdela32a7fy/libelf-0.8.12.tar.gz - ==> Created stage in /g/g0/becker33/SC16-spack/spack/var/spack/stage/libelf-0.8.12-ipggckv6i7h44iryzfa4dwdela32a7fy - ==> No patches needed for libelf - ==> Building libelf [Package] - ==> Executing phase : 'install' - ==> Successfully installed libelf - Fetch: 1.12s. Build: 7.88s. Total: 9.00s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libelf-0.8.12-ipggckv6i7h44iryzfa4dwdela32a7fy - - - - $ spack install libelf %intel@15.0.4 - ==> Installing libelf - ==> Trying to fetch from file:///g/g0/becker33/SC16-spack/spack/var/spack/cache/libelf/libelf-0.8.13.tar.gz - ################################################################################################################################################################################# 100.0% - ==> Staging archive: /g/g0/becker33/SC16-spack/spack/var/spack/stage/libelf-0.8.13-w33hrejdyqu2j2gggdswitls2zv6kdsi/libelf-0.8.13.tar.gz - ==> Created stage in /g/g0/becker33/SC16-spack/spack/var/spack/stage/libelf-0.8.13-w33hrejdyqu2j2gggdswitls2zv6kdsi - ==> No patches needed for libelf - ==> Building libelf [Package] - ==> Executing phase : 'install' - ==> Successfully installed libelf - Fetch: 0.09s. Build: 55.51s. Total: 55.60s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/intel-15.0.4/libelf-0.8.13-w33hrejdyqu2j2gggdswitls2zv6kdsi - - -The spec syntax also includes compiler flags. Spack accepts ``cppflags``, -``cflags``, ``cxxflags``, ``fflags``, ``ldflags``, and ``ldlibs`` -parameters. The values of these fields must be escape-quoted with ``\"`` -on the command line. These values are injected into the compile line -automatically by the Spack compiler wrappers. - -.. code-block:: console - - $ spack install libelf @0.8.12 cppflags=\"-O3\" - ==> Installing libelf - ==> Trying to fetch from file:///g/g0/becker33/SC16-spack/spack/var/spack/cache/libelf/libelf-0.8.12.tar.gz - ################################################################################################################################################################################# 100.0% - ==> Staging archive: /g/g0/becker33/SC16-spack/spack/var/spack/stage/libelf-0.8.12-vrv2ttbd34xlfoxy4jwt6qsjrcbalmmw/libelf-0.8.12.tar.gz - ==> Created stage in /g/g0/becker33/SC16-spack/spack/var/spack/stage/libelf-0.8.12-vrv2ttbd34xlfoxy4jwt6qsjrcbalmmw - ==> No patches needed for libelf - ==> Building libelf [Package] - ==> Executing phase : 'install' - ==> Successfully installed libelf - Fetch: 0.04s. Build: 7.95s. Total: 7.99s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libelf-0.8.12-vrv2ttbd34xlfoxy4jwt6qsjrcbalmmw - - -The ``spack find`` command is used to query installed packages. Note that -some packages appear identical with the default output. The ``-l`` flag -shows the hash of each package, and the ``-f`` flag shows any non-empty -compiler flags of those packages. - -.. code-block:: console - - $ spack find - ==> 5 installed packages. - -- linux-redhat6-x86_64 / gcc@4.4.7 ----------------------------- - libelf@0.8.12 - libelf@0.8.12 - libelf@0.8.13 - - -- linux-redhat6-x86_64 / intel@15.0.4 -------------------------- - libelf@0.8.13 - - -- linux-redhat6-x86_64 / intel@16.0.3 -------------------------- - libelf@0.8.13 - - - - $ spack find -lf - ==> 5 installed packages. - -- linux-redhat6-x86_64 / gcc@4.4.7 ----------------------------- - ipggckv libelf@0.8.12%gcc - - vrv2ttb libelf@0.8.12%gcc cppflags="-O3" - - csrt4qx libelf@0.8.13%gcc - - - -- linux-redhat6-x86_64 / intel@15.0.4 -------------------------- - w33hrej libelf@0.8.13%intel - - - -- linux-redhat6-x86_64 / intel@16.0.3 -------------------------- - 7wgp32x libelf@0.8.13%intel - - -Spack generates a hash for each spec. This hash is a function of the full -provenance of the package, so any change to the spec affects the -hash. Spack uses this value to compare specs and to generate unique -installation directories for every combinatorial version. As we move into -more complicated packages with software dependencies, we can see that -Spack reuses existing packages to satisfy a dependency only when the -existing package's hash matches the desired spec. - -.. code-block:: console - - $ spack install libdwarf - ==> Installing libdwarf - ==> libelf is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libelf-0.8.13-csrt4qxfkhjgn5xg3zjpkir7xdnszl2a - ==> Can not find version 20160507 in url_list - ==> Trying to fetch from file:///g/g0/becker33/SC16-spack/spack/var/spack/cache/libdwarf/libdwarf-20160507.tar.gz - curl: (37) Couldn't open file /g/g0/becker33/SC16-spack/spack/var/spack/cache/libdwarf/libdwarf-20160507.tar.gz - ==> Fetching from file:///g/g0/becker33/SC16-spack/spack/var/spack/cache/libdwarf/libdwarf-20160507.tar.gz failed. - ==> Trying to fetch from http://www.prevanders.net/libdwarf-20160507.tar.gz - ################################################################################################################################################################################# 100.0% - ==> Staging archive: /g/g0/becker33/SC16-spack/spack/var/spack/stage/libdwarf-20160507-yfx6p3g3rkmqvcqbmtb34o6pln7pqvcz/libdwarf-20160507.tar.gz - ==> Created stage in /g/g0/becker33/SC16-spack/spack/var/spack/stage/libdwarf-20160507-yfx6p3g3rkmqvcqbmtb34o6pln7pqvcz - ==> No patches needed for libdwarf - ==> Building libdwarf [Package] - ==> Executing phase : 'install' - ==> Successfully installed libdwarf - Fetch: 1.56s. Build: 33.59s. Total: 35.15s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libdwarf-20160507-yfx6p3g3rkmqvcqbmtb34o6pln7pqvcz - - -Dependencies can be explicitly requested using the ``^`` sigil. Note that -the spec syntax is recursive. Anything we could specify about the -top-level package, we can also specify about a dependency using ``^``. - -.. code-block:: console - - $ spack install libdwarf ^libelf @0.8.12 %intel - ==> Installing libdwarf - ==> Installing libelf - ==> Trying to fetch from file:///g/g0/becker33/SC16-spack/spack/var/spack/cache/libelf/libelf-0.8.12.tar.gz - ################################################################################################################################################################################# 100.0% - ==> Staging archive: /g/g0/becker33/SC16-spack/spack/var/spack/stage/libelf-0.8.12-4blbe3qxqct3ymrfoxxnxysmybvbxay7/libelf-0.8.12.tar.gz - ==> Created stage in /g/g0/becker33/SC16-spack/spack/var/spack/stage/libelf-0.8.12-4blbe3qxqct3ymrfoxxnxysmybvbxay7 - ==> No patches needed for libelf - ==> Building libelf [Package] - ==> Executing phase : 'install' - ==> Successfully installed libelf - Fetch: 0.04s. Build: 52.16s. Total: 52.19s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/intel-16.0.3/libelf-0.8.12-4blbe3qxqct3ymrfoxxnxysmybvbxay7 - ==> Can not find version 20160507 in url_list - ==> Trying to fetch from file:///g/g0/becker33/SC16-spack/spack/var/spack/cache/libdwarf/libdwarf-20160507.tar.gz - ################################################################################################################################################################################# 100.0% - ==> Staging archive: /g/g0/becker33/SC16-spack/spack/var/spack/stage/libdwarf-20160507-csruprgucaujkfkrcywhwou7nbeis5fo/libdwarf-20160507.tar.gz - ==> Created stage in /g/g0/becker33/SC16-spack/spack/var/spack/stage/libdwarf-20160507-csruprgucaujkfkrcywhwou7nbeis5fo - ==> No patches needed for libdwarf - ==> Building libdwarf [Package] - ==> Executing phase : 'install' - ==> Successfully installed libdwarf - Fetch: 0.40s. Build: 2m 17.29s. Total: 2m 17.69s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/intel-16.0.3/libdwarf-20160507-csruprgucaujkfkrcywhwou7nbeis5fo - - -Packages can also be referred to from the command line by their package -hash. Using the ``spack find -lf`` command earlier we saw that the hash -of our optimized installation of libelf (``cppflags=\"-O3\"``) began with -``vrv2ttb``. We can now explicitly build with that package without typing -the entire spec, by using the ``/`` sigil to refer to it by hash. As with -other tools like git, you do not need to specify an *entire* hash on the -command line. You can specify just enough digits to identify a hash -uniquely. If a hash prefix is ambiguous (i.e., two or more installed -packages share the prefix) then spack will report an error. - -.. code-block:: console - - $ spack install libdwarf ^/vrv2ttb - ==> Installing libdwarf - ==> libelf is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libelf-0.8.12-vrv2ttbd34xlfoxy4jwt6qsjrcbalmmw - ==> Can not find version 20160507 in url_list - ==> Trying to fetch from file:///g/g0/becker33/SC16-spack/spack/var/spack/cache/libdwarf/libdwarf-20160507.tar.gz - #################################################################################################################################################################################################################################################### 100.0% - ==> Staging archive: /g/g0/becker33/SC16-spack/spack/var/spack/stage/libdwarf-20160507-dtg3tgnp7htccoly26gduqlrgvnwcp5t/libdwarf-20160507.tar.gz - ==> Created stage in /g/g0/becker33/SC16-spack/spack/var/spack/stage/libdwarf-20160507-dtg3tgnp7htccoly26gduqlrgvnwcp5t - ==> No patches needed for libdwarf - ==> Building libdwarf [Package] - ==> Executing phase : 'install' - ==> Successfully installed libdwarf - Fetch: 0.96s. Build: 24.03s. Total: 24.99s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libdwarf-20160507-dtg3tgnp7htccoly26gduqlrgvnwcp5t - - -The ``spack find`` command can also take a ``-d`` flag, which can show -dependency information. Note that each package has a top-level entry, -even if it also appears as a dependency. - -.. code-block:: console - - $ spack find -ldf - ==> 9 installed packages. - -- linux-redhat6-x86_64 / gcc@4.4.7 ----------------------------- - dtg3tgn libdwarf@20160507%gcc - vrv2ttb ^libelf@0.8.12%gcc cppflags="-O3" - - yfx6p3g libdwarf@20160507%gcc - csrt4qx ^libelf@0.8.13%gcc - - ipggckv libelf@0.8.12%gcc - - vrv2ttb libelf@0.8.12%gcc cppflags="-O3" - - csrt4qx libelf@0.8.13%gcc - - - -- linux-redhat6-x86_64 / intel@15.0.4 -------------------------- - w33hrej libelf@0.8.13%intel - - - -- linux-redhat6-x86_64 / intel@16.0.3 -------------------------- - csruprg libdwarf@20160507%intel - 4blbe3q ^libelf@0.8.12%intel - - 4blbe3q libelf@0.8.12%intel - - 7wgp32x libelf@0.8.13%intel - - -As we get to more complex packages, full installs will take too long to -build in the time allotted for this tutorial. Our collaborators at CERN -have been working on binary caching for Spack, which would allow for very -fast installs of previously built packages. We are still working out the -security ramifications of the feature, but it is coming soon. - -For now, we will switch to doing "fake" installs. When supplied with the -``--fake`` flag (primarily used for debugging), Spack computes build -metadata the same way it normally would, but it does not download the -source or run the install script for a pacakge. We can use this to -quickly demonstrate some of the more advanced Spack features in our -limited tutorial time. - -``HDF5`` is an example of a more complicated package, with an MPI -dependency. If we install it "out of the box," it will build with -``openmpi``. - -.. code-block:: console - - $ spack install --fake hdf5 - ==> Installing hdf5 - ==> Installing zlib - ==> Building zlib [Package] - ==> Successfully installed zlib - Fetch: . Build: 0.11s. Total: 0.11s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/zlib-1.2.8-ayc4jq7vxuzge5n444gutvskeytfdruh - ==> Installing openmpi - ==> Installing hwloc - ==> Installing libpciaccess - ==> Installing util-macros - ==> Building util-macros [Package] - ==> Successfully installed util-macros - Fetch: . Build: 0.11s. Total: 0.11s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/util-macros-1.19.0-pc6zhs4cnkmg2cv4et4fizsp6scuvacg - ==> Installing libtool - ==> Installing m4 - ==> Installing libsigsegv - ==> Building libsigsegv [Package] - ==> Successfully installed libsigsegv - Fetch: . Build: 0.11s. Total: 0.11s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libsigsegv-2.10-q4cok3yber7lhf3jswg6mysg7oi53unh - ==> Building m4 [Package] - ==> Successfully installed m4 - Fetch: . Build: 0.23s. Total: 0.23s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/m4-1.4.17-qijdzvhjyybrtwbqm73vykhmkaqro3je - ==> Building libtool [Package] - ==> Successfully installed libtool - Fetch: . Build: 0.11s. Total: 0.11s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libtool-2.4.6-rdx5nkfjwlvcanz5il3ys2pe34j4vxx5 - ==> Installing pkg-config - ==> Building pkg-config [Package] - ==> Successfully installed pkg-config - Fetch: . Build: 0.11s. Total: 0.11s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/pkg-config-0.29.1-wpjnlzahdw6ahkrgmqyeugkj2zhv4tui - ==> Building libpciaccess [Package] - ==> Successfully installed libpciaccess - Fetch: . Build: 0.10s. Total: 0.10s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libpciaccess-0.13.4-m2f6fpm22rpprq2ihkmfx6llf363264m - ==> Building hwloc [Package] - ==> Successfully installed hwloc - Fetch: . Build: 0.23s. Total: 0.23s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/hwloc-1.11.4-xpb6hbl2hsze25cgdgfnoppn6rchhzaz - ==> Building openmpi [Package] - ==> Successfully installed openmpi - Fetch: . Build: 0.35s. Total: 0.35s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openmpi-2.0.1-j4cgoq4furxvr73pq72r2qgywgksw3qn - ==> Building hdf5 [AutotoolsPackage] - ==> Successfully installed hdf5 - Fetch: . Build: 0.61s. Total: 0.61s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/hdf5-1.10.0-patch1-ezvtnox35albuaxqryuondweyjgeo6es - - -Spack packages can also have variants. Boolean variants can be specified -using the ``+`` and ``~`` or ``-`` sigils. There are two sigils for -``False`` to avoid conflicts with shell parsing in different -situations. Variants (boolean or otherwise) can also be specified using -the same syntax as compiler flags. Here we can install HDF5 without MPI -support. - -.. code-block:: console - - $ spack install --fake hdf5~mpi - ==> Installing hdf5 - ==> zlib is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/zlib-1.2.8-ayc4jq7vxuzge5n444gutvskeytfdruh - ==> Building hdf5 [AutotoolsPackage] - ==> Successfully installed hdf5 - Fetch: . Build: 0.22s. Total: 0.22s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/hdf5-1.10.0-patch1-twppaioxqn6lti4grgopnmhwcq3h2rpw - - -We might also want to install HDF5 with a different MPI -implementation. While MPI is not a package itself, packages can depend on -abstract interfaces like MPI. Spack handles these through "virtual -dependencies." A package, such as HDF5, can depend on the MPI -interface. Other packages (``openmpi``, ``mpich``, ``mvapich``, etc.) -provide the MPI interface. Any of these providers can be requested for -an MPI dependency. For example, we can build HDF5 with MPI support -provided by mpich by specifying a dependency on ``mpich``. Spack also -supports versioning of virtual dependencies. A package can depend on the -MPI interface at version 3, and provider packages specify what version of -the interface *they* provide. The partial spec ``^mpi@3`` can be safisfied -by any of several providers. - -.. code-block:: console - - $ spack install --fake hdf5+mpi ^mpich - ==> Installing hdf5 - ==> mpich is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/mpich-3.2-5jlp2ndnsb67txggraglu47vjmayx5za - ==> zlib is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/zlib-1.2.8-ayc4jq7vxuzge5n444gutvskeytfdruh - ==> Building hdf5 [AutotoolsPackage] - ==> Successfully installed hdf5 - Fetch: . Build: 0.38s. Total: 0.38s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/hdf5-1.10.0-patch1-j36yfw25i6gdd3q4vwlupgkpwic4ua6m - - -We'll do a quick check in on what we have installed so far. - -.. code-block:: console - - $ spack find -ldf - ==> 22 installed packages. - -- linux-redhat6-x86_64 / gcc@4.4.7 ----------------------------- - twppaio hdf5@1.10.0-patch1%gcc - ayc4jq7 ^zlib@1.2.8%gcc - - j36yfw2 hdf5@1.10.0-patch1%gcc - 5jlp2nd ^mpich@3.2%gcc - ayc4jq7 ^zlib@1.2.8%gcc - - ezvtnox hdf5@1.10.0-patch1%gcc - j4cgoq4 ^openmpi@2.0.1%gcc - xpb6hbl ^hwloc@1.11.4%gcc - m2f6fpm ^libpciaccess@0.13.4%gcc - ayc4jq7 ^zlib@1.2.8%gcc - - xpb6hbl hwloc@1.11.4%gcc - m2f6fpm ^libpciaccess@0.13.4%gcc - - dtg3tgn libdwarf@20160507%gcc - vrv2ttb ^libelf@0.8.12%gcc cppflags="-O3" - - yfx6p3g libdwarf@20160507%gcc - csrt4qx ^libelf@0.8.13%gcc - - ipggckv libelf@0.8.12%gcc - - vrv2ttb libelf@0.8.12%gcc cppflags="-O3" - - csrt4qx libelf@0.8.13%gcc - - m2f6fpm libpciaccess@0.13.4%gcc - - q4cok3y libsigsegv@2.10%gcc - - rdx5nkf libtool@2.4.6%gcc - - qijdzvh m4@1.4.17%gcc - q4cok3y ^libsigsegv@2.10%gcc - - 5jlp2nd mpich@3.2%gcc - - j4cgoq4 openmpi@2.0.1%gcc - xpb6hbl ^hwloc@1.11.4%gcc - m2f6fpm ^libpciaccess@0.13.4%gcc - - wpjnlza pkg-config@0.29.1%gcc - - pc6zhs4 util-macros@1.19.0%gcc - - ayc4jq7 zlib@1.2.8%gcc - - - -- linux-redhat6-x86_64 / intel@15.0.4 -------------------------- - w33hrej libelf@0.8.13%intel - - - -- linux-redhat6-x86_64 / intel@16.0.3 -------------------------- - csruprg libdwarf@20160507%intel - 4blbe3q ^libelf@0.8.12%intel - - 4blbe3q libelf@0.8.12%intel - - 7wgp32x libelf@0.8.13%intel - - -Spack models the dependencies of packages as a directed acyclic graph -(DAG). The ``spack find -d`` command shows the tree representation of -that graph. We can also use the ``spack graph`` command to view the entire -DAG as a graph. - -.. code-block:: console - - $ spack graph hdf5+mpi ^mpich - o hdf5 - |\ - o | zlib - / - o mpich - -You may also have noticed that there are some packages shown in the -``spack find -d`` output that we didn't install explicitly. These are -dependencies that were installed implicitly. A few packages installed -implicitly are not shown as dependencies in the ``spack find -d`` -output. These are build dependencies. For example, ``libpciaccess`` is a -dependency of openmpi and requires m4 to build. Spack will build `m4`` as -part of the installation of ``openmpi``, but it does not become a part of -the DAG because it is not linked in at run time. Spack handles build -dependencies differently because of their different (less strict) -consistency requirements. It is entirely possible to have two packages -using different versions of a dependency to build, which obviously cannot -be done with linked dependencies. - -``HDF5`` is more complicated than our basic example of libelf and -libdwarf, but it's still within the realm of software that an experienced -HPC user could reasonably expect to install given a bit of time. Now -let's look at a more complicated package. - -.. code-block:: console - - $ spack install --fake trilinos - ==> Installing trilinos - ==> Installing superlu-dist - ==> openmpi is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openmpi-2.0.1-j4cgoq4furxvr73pq72r2qgywgksw3qn - ==> Installing parmetis - ==> openmpi is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openmpi-2.0.1-j4cgoq4furxvr73pq72r2qgywgksw3qn - ==> Installing cmake - ==> Installing bzip2 - ==> Building bzip2 [Package] - ==> Successfully installed bzip2 - Fetch: . Build: 0.11s. Total: 0.11s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/bzip2-1.0.6-gll2xsahysy7ji5gkmfxwkofdt3mwjhs - ==> expat is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/expat-2.2.0-mg5kwd3kluxdgorj32vzbp7aksg3vqej - ==> Installing ncurses - ==> Building ncurses [Package] - ==> Successfully installed ncurses - Fetch: . Build: 0.11s. Total: 0.11s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/ncurses-6.0-fttg4astvrtq2buey4wq66tnyu7bgj2c - ==> zlib is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/zlib-1.2.8-ayc4jq7vxuzge5n444gutvskeytfdruh - ==> Installing openssl - ==> zlib is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/zlib-1.2.8-ayc4jq7vxuzge5n444gutvskeytfdruh - ==> Building openssl [Package] - ==> Successfully installed openssl - Fetch: . Build: 0.23s. Total: 0.23s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openssl-1.0.2j-kt5xyk2dkho6tzadnqlbnbujmljprylg - ==> Installing libarchive - ==> Installing lzma - ==> Building lzma [Package] - ==> Successfully installed lzma - Fetch: . Build: 0.11s. Total: 0.11s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/lzma-4.32.7-hah2cdo3zbulz6yg5do6dvnfn6en5v5c - ==> Installing nettle - ==> m4 is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/m4-1.4.17-qijdzvhjyybrtwbqm73vykhmkaqro3je - ==> Installing gmp - ==> m4 is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/m4-1.4.17-qijdzvhjyybrtwbqm73vykhmkaqro3je - ==> Building gmp [AutotoolsPackage] - ==> Successfully installed gmp - Fetch: . Build: 0.11s. Total: 0.11s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/gmp-6.1.1-uwn4gfdtq3sywy5uf4f7znrh66oybikf - ==> Building nettle [Package] - ==> Successfully installed nettle - Fetch: . Build: 0.18s. Total: 0.18s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/nettle-3.2-w4ieitifcmrldo4ra7as63apagzf56ja - ==> bzip2 is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/bzip2-1.0.6-gll2xsahysy7ji5gkmfxwkofdt3mwjhs - ==> expat is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/expat-2.2.0-mg5kwd3kluxdgorj32vzbp7aksg3vqej - ==> Installing libxml2 - ==> Installing xz - ==> Building xz [Package] - ==> Successfully installed xz - Fetch: . Build: 0.11s. Total: 0.11s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/xz-5.2.2-bxh6cpyqqozazm5okvjqk23sww3gccnf - ==> zlib is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/zlib-1.2.8-ayc4jq7vxuzge5n444gutvskeytfdruh - ==> Building libxml2 [Package] - ==> Successfully installed libxml2 - Fetch: . Build: 0.35s. Total: 0.35s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libxml2-2.9.4-un323rppyu5qipkegyf7flmymvtmunrx - ==> zlib is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/zlib-1.2.8-ayc4jq7vxuzge5n444gutvskeytfdruh - ==> Installing lz4 - ==> Building lz4 [Package] - ==> Successfully installed lz4 - Fetch: . Build: 0.12s. Total: 0.12s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/lz4-131-ivy2fcaw7ywujx74weebdi5bsm7q4vkc - ==> openssl is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openssl-1.0.2j-kt5xyk2dkho6tzadnqlbnbujmljprylg - ==> xz is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/xz-5.2.2-bxh6cpyqqozazm5okvjqk23sww3gccnf - ==> Installing lzo - ==> Building lzo [AutotoolsPackage] - ==> Successfully installed lzo - Fetch: . Build: 0.11s. Total: 0.11s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/lzo-2.09-dlgnm74ozo6baactkft5oah2jre2ri2i - ==> Building libarchive [Package] - ==> Successfully installed libarchive - Fetch: . Build: 1.35s. Total: 1.35s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libarchive-3.2.1-biq3kebw7vel7njusill7vv7mjldkqjv - ==> xz is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/xz-5.2.2-bxh6cpyqqozazm5okvjqk23sww3gccnf - ==> Installing curl - ==> zlib is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/zlib-1.2.8-ayc4jq7vxuzge5n444gutvskeytfdruh - ==> openssl is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openssl-1.0.2j-kt5xyk2dkho6tzadnqlbnbujmljprylg - ==> Building curl [Package] - ==> Successfully installed curl - Fetch: . Build: 0.36s. Total: 0.36s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/curl-7.50.3-oze4gqutj4x2isbkcn5ob2bhhxbskod4 - ==> Building cmake [Package] - ==> Successfully installed cmake - Fetch: . Build: 1.64s. Total: 1.64s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/cmake-3.6.1-n2nkknrku6dvuneo3rjumim7axt7n36e - ==> Installing metis - ==> cmake is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/cmake-3.6.1-n2nkknrku6dvuneo3rjumim7axt7n36e - ==> Building metis [Package] - ==> Successfully installed metis - Fetch: . Build: 0.11s. Total: 0.11s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/metis-5.1.0-ithifyl4xvqbn76js23wsb4tjnztrbdv - ==> Building parmetis [Package] - ==> Successfully installed parmetis - Fetch: . Build: 0.62s. Total: 0.62s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/parmetis-4.0.3-rtg6hml5t6acdcnxomn3l5zfiful4d2t - ==> Installing openblas - ==> Building openblas [Package] - ==> Successfully installed openblas - Fetch: . Build: 0.11s. Total: 0.11s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openblas-0.2.19-bwofa7fhff6od5zn56vy3j4eeyupsqgt - ==> metis is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/metis-5.1.0-ithifyl4xvqbn76js23wsb4tjnztrbdv - ==> Building superlu-dist [Package] - ==> Successfully installed superlu-dist - Fetch: . Build: 0.85s. Total: 0.85s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/superlu-dist-5.1.1-25r6jlvkpjnkiuwt2rtbzhk3l3htuxs7 - ==> cmake is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/cmake-3.6.1-n2nkknrku6dvuneo3rjumim7axt7n36e - ==> Installing glm - ==> cmake is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/cmake-3.6.1-n2nkknrku6dvuneo3rjumim7axt7n36e - ==> Building glm [Package] - ==> Successfully installed glm - Fetch: . Build: 0.12s. Total: 0.12s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/glm-0.9.7.1-7a6oho4aerz7vftxd5ur7lywscht2iry - ==> Installing hypre - ==> openmpi is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openmpi-2.0.1-j4cgoq4furxvr73pq72r2qgywgksw3qn - ==> openblas is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openblas-0.2.19-bwofa7fhff6od5zn56vy3j4eeyupsqgt - ==> Building hypre [Package] - ==> Successfully installed hypre - Fetch: . Build: 0.61s. Total: 0.61s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/hypre-2.11.1-lf7hcejiiww5peesh57quda72z67veit - ==> metis is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/metis-5.1.0-ithifyl4xvqbn76js23wsb4tjnztrbdv - ==> Installing netlib-scalapack - ==> openmpi is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openmpi-2.0.1-j4cgoq4furxvr73pq72r2qgywgksw3qn - ==> cmake is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/cmake-3.6.1-n2nkknrku6dvuneo3rjumim7axt7n36e - ==> openblas is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openblas-0.2.19-bwofa7fhff6od5zn56vy3j4eeyupsqgt - ==> Building netlib-scalapack [Package] - ==> Successfully installed netlib-scalapack - Fetch: . Build: 0.61s. Total: 0.61s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/netlib-scalapack-2.0.2-dvcanz2qq4dfcexznbhbmzbxfj43uz4q - ==> Installing suite-sparse - ==> Installing tbb - ==> Building tbb [Package] - ==> Successfully installed tbb - Fetch: . Build: 0.12s. Total: 0.12s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/tbb-4.4.4-zawzkkhrmdonbjpj3a5bb6gkgnqlrjeu - ==> openblas is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openblas-0.2.19-bwofa7fhff6od5zn56vy3j4eeyupsqgt - ==> metis is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/metis-5.1.0-ithifyl4xvqbn76js23wsb4tjnztrbdv - ==> Building suite-sparse [Package] - ==> Successfully installed suite-sparse - Fetch: . Build: 0.49s. Total: 0.49s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/suite-sparse-4.5.3-lvur6hriy2j7xfjwh5punp3exwpynzm6 - ==> openmpi is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openmpi-2.0.1-j4cgoq4furxvr73pq72r2qgywgksw3qn - ==> Installing netcdf - ==> m4 is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/m4-1.4.17-qijdzvhjyybrtwbqm73vykhmkaqro3je - ==> curl is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/curl-7.50.3-oze4gqutj4x2isbkcn5ob2bhhxbskod4 - ==> zlib is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/zlib-1.2.8-ayc4jq7vxuzge5n444gutvskeytfdruh - ==> hdf5 is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/hdf5-1.10.0-patch1-ezvtnox35albuaxqryuondweyjgeo6es - ==> Building netcdf [Package] - ==> Successfully installed netcdf - Fetch: . Build: 0.90s. Total: 0.90s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/netcdf-4.4.1-tcl4zbrmdfrit2cqlaxig6xieu5h552j - ==> Installing mumps - ==> netlib-scalapack is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/netlib-scalapack-2.0.2-dvcanz2qq4dfcexznbhbmzbxfj43uz4q - ==> openmpi is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openmpi-2.0.1-j4cgoq4furxvr73pq72r2qgywgksw3qn - ==> openblas is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openblas-0.2.19-bwofa7fhff6od5zn56vy3j4eeyupsqgt - ==> Building mumps [Package] - ==> Successfully installed mumps - Fetch: . Build: 0.74s. Total: 0.74s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/mumps-5.0.2-kr5r4nnx5tfcacxnk3ii5dsxbe6pu5fy - ==> Installing matio - ==> Building matio [Package] - ==> Successfully installed matio - Fetch: . Build: 0.11s. Total: 0.11s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/matio-1.5.2-4zrozucookychlvc4q53omp2zyfk2bed - ==> Installing boost - ==> bzip2 is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/bzip2-1.0.6-gll2xsahysy7ji5gkmfxwkofdt3mwjhs - ==> zlib is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/zlib-1.2.8-ayc4jq7vxuzge5n444gutvskeytfdruh - ==> Building boost [Package] - ==> Successfully installed boost - Fetch: . Build: 0.35s. Total: 0.35s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/boost-1.62.0-je7eqvzt74kezwhh55y5lwt5dy6pnali - ==> parmetis is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/parmetis-4.0.3-rtg6hml5t6acdcnxomn3l5zfiful4d2t - ==> openblas is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openblas-0.2.19-bwofa7fhff6od5zn56vy3j4eeyupsqgt - ==> hdf5 is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/hdf5-1.10.0-patch1-ezvtnox35albuaxqryuondweyjgeo6es - ==> Building trilinos [Package] - ==> Successfully installed trilinos - Fetch: . Build: 2.63s. Total: 2.63s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/trilinos-12.8.1-uvd6dfd7x4uyvck4awo3r3frudihn4ar - - -Now we're starting to see the power of Spack. Trilinos has 11 top level -dependecies, many of which have dependencies of their own. Installing -more complex packages can take days or weeks even for an experienced -user. Although we've done a fake installation for the tutorial, a real -installation of trilinos using Spack takes about 3 hours (depending on -the system), but only 20 seconds of programmer time. - -Spack manages constistency of the entire DAG. Every MPI dependency will -be satisfied by the same configuration of MPI, etc. If we install -``trilinos`` again specifying a dependency on our previous HDF5 built -with ``mpich``: - -.. code-block:: console - - $ spack install --fake trilinos ^hdf5+mpi ^mpich - ==> Installing trilinos - ==> Installing superlu-dist - ==> mpich is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/mpich-3.2-5jlp2ndnsb67txggraglu47vjmayx5za - ==> metis is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/metis-5.1.0-ithifyl4xvqbn76js23wsb4tjnztrbdv - ==> Installing parmetis - ==> mpich is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/mpich-3.2-5jlp2ndnsb67txggraglu47vjmayx5za - ==> metis is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/metis-5.1.0-ithifyl4xvqbn76js23wsb4tjnztrbdv - ==> cmake is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/cmake-3.6.1-n2nkknrku6dvuneo3rjumim7axt7n36e - ==> Building parmetis [Package] - ==> Successfully installed parmetis - Fetch: . Build: 0.38s. Total: 0.38s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/parmetis-4.0.3-43kbtni6p5y446c6qdkybq4htj7ot4zn - ==> openblas is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openblas-0.2.19-bwofa7fhff6od5zn56vy3j4eeyupsqgt - ==> Building superlu-dist [Package] - ==> Successfully installed superlu-dist - Fetch: . Build: 0.61s. Total: 0.61s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/superlu-dist-5.1.1-46uuupehmonx5jicc6xnegnud2n5jqyl - ==> cmake is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/cmake-3.6.1-n2nkknrku6dvuneo3rjumim7axt7n36e - ==> glm is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/glm-0.9.7.1-7a6oho4aerz7vftxd5ur7lywscht2iry - ==> Installing hypre - ==> mpich is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/mpich-3.2-5jlp2ndnsb67txggraglu47vjmayx5za - ==> openblas is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openblas-0.2.19-bwofa7fhff6od5zn56vy3j4eeyupsqgt - ==> Building hypre [Package] - ==> Successfully installed hypre - Fetch: . Build: 0.37s. Total: 0.37s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/hypre-2.11.1-6ajnyymoivs5apajd7thjisae36jv4lz - ==> metis is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/metis-5.1.0-ithifyl4xvqbn76js23wsb4tjnztrbdv - ==> Installing netlib-scalapack - ==> mpich is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/mpich-3.2-5jlp2ndnsb67txggraglu47vjmayx5za - ==> cmake is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/cmake-3.6.1-n2nkknrku6dvuneo3rjumim7axt7n36e - ==> openblas is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openblas-0.2.19-bwofa7fhff6od5zn56vy3j4eeyupsqgt - ==> Building netlib-scalapack [Package] - ==> Successfully installed netlib-scalapack - Fetch: . Build: 0.37s. Total: 0.37s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/netlib-scalapack-2.0.2-dayeep27omm26wksd3iqvbu3gezc2eoh - ==> suite-sparse is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/suite-sparse-4.5.3-lvur6hriy2j7xfjwh5punp3exwpynzm6 - ==> Installing netcdf - ==> m4 is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/m4-1.4.17-qijdzvhjyybrtwbqm73vykhmkaqro3je - ==> curl is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/curl-7.50.3-oze4gqutj4x2isbkcn5ob2bhhxbskod4 - ==> zlib is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/zlib-1.2.8-ayc4jq7vxuzge5n444gutvskeytfdruh - ==> hdf5 is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/hdf5-1.10.0-patch1-j36yfw25i6gdd3q4vwlupgkpwic4ua6m - ==> Building netcdf [Package] - ==> Successfully installed netcdf - Fetch: . Build: 0.67s. Total: 0.67s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/netcdf-4.4.1-gfemi4jk4qltvp33xhtpkam7dozbqvhq - ==> Installing mumps - ==> mpich is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/mpich-3.2-5jlp2ndnsb67txggraglu47vjmayx5za - ==> netlib-scalapack is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/netlib-scalapack-2.0.2-dayeep27omm26wksd3iqvbu3gezc2eoh - ==> openblas is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openblas-0.2.19-bwofa7fhff6od5zn56vy3j4eeyupsqgt - ==> Building mumps [Package] - ==> Successfully installed mumps - Fetch: . Build: 0.49s. Total: 0.49s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/mumps-5.0.2-w7t5pl3jhhwitfiyer63zj6zv7idkt3m - ==> mpich is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/mpich-3.2-5jlp2ndnsb67txggraglu47vjmayx5za - ==> matio is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/matio-1.5.2-4zrozucookychlvc4q53omp2zyfk2bed - ==> boost is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/boost-1.62.0-je7eqvzt74kezwhh55y5lwt5dy6pnali - ==> parmetis is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/parmetis-4.0.3-43kbtni6p5y446c6qdkybq4htj7ot4zn - ==> openblas is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openblas-0.2.19-bwofa7fhff6od5zn56vy3j4eeyupsqgt - ==> hdf5 is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/hdf5-1.10.0-patch1-j36yfw25i6gdd3q4vwlupgkpwic4ua6m - ==> Building trilinos [Package] - ==> Successfully installed trilinos - Fetch: . Build: 2.42s. Total: 2.42s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/trilinos-12.8.1-ffwrpxnq7lhiw2abxn2u7ffr4jjsdwep - -We see that every package in the trilinos DAG that depends on MPI now -uses ``mpich``. - -.. code-block:: console - - $ spack find -d trilinos - ==> 2 installed packages. - -- linux-redhat6-x86_64 / gcc@4.4.7 ----------------------------- - trilinos@12.8.1 - ^boost@1.62.0 - ^bzip2@1.0.6 - ^zlib@1.2.8 - ^glm@0.9.7.1 - ^hdf5@1.10.0-patch1 - ^mpich@3.2 - ^hypre@2.11.1 - ^openblas@0.2.19 - ^matio@1.5.2 - ^metis@5.1.0 - ^mumps@5.0.2 - ^netlib-scalapack@2.0.2 - ^netcdf@4.4.1 - ^curl@7.50.3 - ^openssl@1.0.2j - ^parmetis@4.0.3 - ^suite-sparse@4.5.3 - ^tbb@4.4.4 - ^superlu-dist@5.1.1 - - trilinos@12.8.1 - ^boost@1.62.0 - ^bzip2@1.0.6 - ^zlib@1.2.8 - ^glm@0.9.7.1 - ^hdf5@1.10.0-patch1 - ^openmpi@2.0.1 - ^hwloc@1.11.4 - ^libpciaccess@0.13.4 - ^hypre@2.11.1 - ^openblas@0.2.19 - ^matio@1.5.2 - ^metis@5.1.0 - ^mumps@5.0.2 - ^netlib-scalapack@2.0.2 - ^netcdf@4.4.1 - ^curl@7.50.3 - ^openssl@1.0.2j - ^parmetis@4.0.3 - ^suite-sparse@4.5.3 - ^tbb@4.4.4 - ^superlu-dist@5.1.1 - - -As we discussed before, the ``spack find -d`` command shows the -dependency information as a tree. While that is often sufficient, many -complicated packages, including trilinos, have dependencies that -cannot be fully represented as a tree. Again, the ``spack graph`` -command shows the full DAG of the dependency information. - -.. code-block:: console - - $ spack graph trilinos - o trilinos - |\ - | |\ - | | |\ - | | | |\ - | | | | |\ - | | | | | |\ - | | | | | | |\ - | o | | | | | | netcdf - | |\ \ \ \ \ \ \ - | | |\ \ \ \ \ \ \ - | | | o | | | | | | curl - | | |/| | | | | | | - | |/| | | | | | | | - | | | o | | | | | | openssl - | | |/ / / / / / / - | |/| | | | | | | - | | o | | | | | | hdf5 - | |/| | | | | | | - | | |/ / / / / / - | o | | | | | | zlib - | / / / / / / - o | | | | | | swig - o | | | | | | pcre - / / / / / / - o | | | | | mpi - / / / / / - o | | | | matio - / / / / - o | | | lapack - / / / - o | | glm - / / - o | boost - / - o blas - - -You can control how the output is displayed with a number of options. - -The ASCII output from ``spack graph`` can be difficult to parse for -complicated packages. The output can be changed to the ``graphviz`` -``.dot`` format using the `--dot` flag. - -.. code-block:: console - - $ spack graph --dot trilinos | dot -Tpdf trilinos_graph.pdf - -.. _basics-tutorial-uninstall: - ---------------------- -Uninstalling Packages ---------------------- - -Earlier we installed many configurations each of libelf and -libdwarf. Now we will go through and uninstall some of those packages -that we didn't really need. - -.. code-block:: console - - $ spack find -d libdwarf - ==> 3 installed packages. - -- linux-redhat6-x86_64 / gcc@4.4.7 ----------------------------- - libdwarf@20160507 - ^libelf@0.8.12 - - libdwarf@20160507 - ^libelf@0.8.13 - - - -- linux-redhat6-x86_64 / intel@16.0.3 -------------------------- - libdwarf@20160507 - ^libelf@0.8.12 - - $ spack find libelf - ==> 6 installed packages. - -- linux-redhat6-x86_64 / gcc@4.4.7 ----------------------------- - libelf@0.8.12 libelf@0.8.12 libelf@0.8.13 - - -- linux-redhat6-x86_64 / intel@15.0.4 -------------------------- - libelf@0.8.13 - - -- linux-redhat6-x86_64 / intel@16.0.3 -------------------------- - libelf@0.8.12 libelf@0.8.13 - - -We can uninstall packages by spec using the same syntax as install. - -.. code-block:: console - - $ spack uninstall libelf%intel@15.0.4 - ==> The following packages will be uninstalled : - - -- linux-redhat6-x86_64 / intel@15.0.4 -------------------------- - w33hrej libelf@0.8.13%intel - - - ==> Do you want to proceed ? [y/n] - y - ==> Successfully uninstalled libelf@0.8.13%intel@15.0.4 arch=linux-redhat6-x86_64-w33hrej - - - - $ spack find -lf libelf - ==> 5 installed packages. - -- linux-redhat6-x86_64 / gcc@4.4.7 ----------------------------- - ipggckv libelf@0.8.12%gcc - - vrv2ttb libelf@0.8.12%gcc cppflags="-O3" - - csrt4qx libelf@0.8.13%gcc - - - -- linux-redhat6-x86_64 / intel@16.0.3 -------------------------- - 4blbe3q libelf@0.8.12%intel - - 7wgp32x libelf@0.8.13%intel - - -We can uninstall packages by referring only to their hash. - - -We can use either ``-f`` (force) or ``-d`` (remove dependents as well) to -remove packages that are required by another installed package. - - - $ spack uninstall /4blb - ==> Error: Will not uninstall libelf@0.8.12%intel@16.0.3-4blbe3q - - The following packages depend on it: - -- linux-redhat6-x86_64 / intel@16.0.3 -------------------------- - csruprg libdwarf@20160507%intel - - - ==> Error: You can use spack uninstall --dependents to uninstall these dependencies as well - $ spack uninstall -d /4blb - ==> The following packages will be uninstalled : - - -- linux-redhat6-x86_64 / intel@16.0.3 -------------------------- - csruprg libdwarf@20160507%intel - - 4blbe3q libelf@0.8.12%intel - - - ==> Do you want to proceed ? [y/n] - y - ==> Successfully uninstalled libdwarf@20160507%intel@16.0.3 arch=linux-redhat6-x86_64-csruprg - ==> Successfully uninstalled libelf@0.8.12%intel@16.0.3 arch=linux-redhat6-x86_64-4blbe3q - - -Spack will not uninstall packages that are not sufficiently -specified. The ``-a`` (all) flag can be used to uninstall multiple -packages at once. - - - $ spack uninstall trilinos - ==> Error: trilinos matches multiple packages: - - -- linux-redhat6-x86_64 / gcc@4.4.7 ----------------------------- - ffwrpxn trilinos@12.8.1%gcc+boost~debug+hdf5+hypre+metis+mumps~python+shared+suite-sparse+superlu-dist - - uvd6dfd trilinos@12.8.1%gcc+boost~debug+hdf5+hypre+metis+mumps~python+shared+suite-sparse+superlu-dist - - - ==> Error: You can either: - a) Use a more specific spec, or - b) use spack uninstall -a to uninstall ALL matching specs. - - - - $ spack uninstall /ffwr - ==> The following packages will be uninstalled : - - -- linux-redhat6-x86_64 / gcc@4.4.7 ----------------------------- - ffwrpxn trilinos@12.8.1%gcc+boost~debug+hdf5+hypre+metis+mumps~python+shared+suite-sparse+superlu-dist - - - ==> Do you want to proceed ? [y/n] - y - ==> Successfully uninstalled trilinos@12.8.1%gcc@4.4.7+boost~debug+hdf5+hypre+metis+mumps~python+shared+suite-sparse+superlu-dist arch=linux-redhat6-x86_64-ffwrpxn - ------------------------------ -Advanced ``spack find`` Usage ------------------------------ - -We will go over some additional uses for the `spack find` command not -already covered in the :ref:`basics-tutorial-install` and -:ref:`basics-tutorial-uninstall` sections. - -The ``spack find`` command can accept what we call "anonymous specs." -These are expressions in spec syntax that do not contain a package -name. For example, `spack find %intel` will return every package built -with the intel compiler, and ``spack find cppflags=\\"-O3\\"`` will -return every package which was built with ``cppflags=\\"-O3\\"``. - -.. code-block:: console - - $ spack find %intel - ==> 1 installed packages. - -- linux-redhat6-x86_64 / intel@16.0.3 -------------------------- - libelf@0.8.13 - - - - $ spack find cppflags=\"-O3\" - ==> 1 installed packages. - -- linux-redhat6-x86_64 / gcc@4.4.7 ----------------------------- - libelf@0.8.12 - - -The ``find`` command can also show which packages were installed -explicitly (rather than pulled in as a dependency) using the ``-e`` -flag. The ``-E`` flag shows implicit installs only. The ``find`` command can -also show the path to which a spack package was installed using the ``-p`` -command. - -.. code-block:: console - - $ spack find -pe - ==> 10 installed packages. - -- linux-redhat6-x86_64 / gcc@4.4.7 ----------------------------- - hdf5@1.10.0-patch1 /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/hdf5-1.10.0-patch1-twppaioxqn6lti4grgopnmhwcq3h2rpw - hdf5@1.10.0-patch1 /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/hdf5-1.10.0-patch1-j36yfw25i6gdd3q4vwlupgkpwic4ua6m - hdf5@1.10.0-patch1 /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/hdf5-1.10.0-patch1-ezvtnox35albuaxqryuondweyjgeo6es - libdwarf@20160507 /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libdwarf-20160507-dtg3tgnp7htccoly26gduqlrgvnwcp5t - libdwarf@20160507 /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libdwarf-20160507-yfx6p3g3rkmqvcqbmtb34o6pln7pqvcz - libelf@0.8.12 /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libelf-0.8.12-ipggckv6i7h44iryzfa4dwdela32a7fy - libelf@0.8.12 /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libelf-0.8.12-vrv2ttbd34xlfoxy4jwt6qsjrcbalmmw - libelf@0.8.13 /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libelf-0.8.13-csrt4qxfkhjgn5xg3zjpkir7xdnszl2a - trilinos@12.8.1 /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/trilinos-12.8.1-uvd6dfd7x4uyvck4awo3r3frudihn4ar - - -- linux-redhat6-x86_64 / intel@16.0.3 -------------------------- - libelf@0.8.13 /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/intel-16.0.3/libelf-0.8.13-7wgp32xksatkvw2tbssmehw2t5tnxndj - - ---------------------- -Customizing Compilers ---------------------- - - -Spack manages a list of available compilers on the system, detected -automatically from from the user's ``PATH`` variable. The ``spack -compilers`` command is an alias for the command ``spack compiler list``. - -.. code-block:: console - - $ spack compilers - ==> Available compilers - -- gcc ---------------------------------------------------------- - gcc@4.4.7 - - -- intel -------------------------------------------------------- - intel@16.0.3 intel@15.0.1 intel@14.0.0 intel@12.1.3 intel@10.0 - intel@16.0.2 intel@15.0.0 intel@13.1.1 intel@12.1.2 intel@9.1 - intel@16.0.1 intel@14.0.4 intel@13.1.0 intel@12.1.0 - intel@16.0.0 intel@14.0.3 intel@13.0.1 intel@12.0.4 - intel@15.0.4 intel@14.0.2 intel@13.0.0 intel@11.1 - intel@15.0.3 intel@14.0.1 intel@12.1.5 intel@10.1 - - -- pgi ---------------------------------------------------------- - pgi@16.5-0 pgi@15.7-0 pgi@14.7-0 pgi@13.2-0 pgi@11.10-0 pgi@9.0-4 - pgi@16.3-0 pgi@15.5-0 pgi@14.3-0 pgi@13.1-1 pgi@11.1-0 pgi@8.0-1 - pgi@16.1-0 pgi@15.1-0 pgi@13.10-0 pgi@12.8-0 pgi@10.9-0 pgi@7.1-3 - pgi@15.10-0 pgi@14.10-0 pgi@13.6-0 pgi@12.1-0 pgi@10.2-0 pgi@7.0-6 - -The compilers are maintained in a YAML file that can be hand-edited -for special cases. Spack also has tools to add compilers, and -compilers built with Spack can be added to the configuration. - -.. code-block:: console - - $ spack install --fake gcc@6.1.0 - ==> Installing gcc - ==> gmp is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/gmp-6.1.1-uwn4gfdtq3sywy5uf4f7znrh66oybikf - ==> Installing isl - ==> gmp is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/gmp-6.1.1-uwn4gfdtq3sywy5uf4f7znrh66oybikf - ==> Building isl [Package] - ==> Successfully installed isl - Fetch: . Build: 0.19s. Total: 0.19s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/isl-0.14-hs2w7mjjjaakkmbbv5yvfqf7yyzhorl6 - ==> Installing mpc - ==> gmp is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/gmp-6.1.1-uwn4gfdtq3sywy5uf4f7znrh66oybikf - ==> Installing mpfr - ==> gmp is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/gmp-6.1.1-uwn4gfdtq3sywy5uf4f7znrh66oybikf - ==> Building mpfr [Package] - ==> Successfully installed mpfr - Fetch: . Build: 0.17s. Total: 0.17s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/mpfr-3.1.4-7kt5ij437khredfq4bvnyu22t3fmtfvt - ==> Building mpc [Package] - ==> Successfully installed mpc - Fetch: . Build: 0.28s. Total: 0.28s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/mpc-1.0.3-g5taq6lt3zuy5l2jtggi5lctxnl4la5u - ==> Installing binutils - ==> m4 is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/m4-1.4.17-qijdzvhjyybrtwbqm73vykhmkaqro3je - ==> Installing bison - ==> m4 is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/m4-1.4.17-qijdzvhjyybrtwbqm73vykhmkaqro3je - ==> Building bison [Package] - ==> Successfully installed bison - Fetch: . Build: 0.12s. Total: 0.12s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/bison-3.0.4-hkhfysfvq5l6rsns67g2htmkpxauvnwa - ==> Installing flex - ==> m4 is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/m4-1.4.17-qijdzvhjyybrtwbqm73vykhmkaqro3je - ==> bison is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/bison-3.0.4-hkhfysfvq5l6rsns67g2htmkpxauvnwa - ==> Building flex [Package] - ==> Successfully installed flex - Fetch: . Build: 0.11s. Total: 0.11s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/flex-2.6.0-qd6d73rdfrozdrsdpimvl4tj7d5ps7qg - ==> Building binutils [Package] - ==> Successfully installed binutils - Fetch: . Build: 0.11s. Total: 0.11s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/binutils-2.27-iq2hry3gvaxszmwwbnll7njgdgaek56o - ==> mpfr is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/mpfr-3.1.4-7kt5ij437khredfq4bvnyu22t3fmtfvt - ==> Building gcc [Package] - ==> Successfully installed gcc - Fetch: . Build: 0.66s. Total: 0.66s. - [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/gcc-6.1.0-j5576zbsot2ydljlthjzhsirsesnogvh - - - - $ spack find -p gcc - ==> 1 installed packages. - -- linux-redhat6-x86_64 / gcc@4.4.7 ----------------------------- - gcc@6.1.0 /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/gcc-6.1.0-j5576zbsot2ydljlthjzhsirsesnogvh - - -If we had done a "real" install of gcc, we could add it to our -configuration now using the `spack compiler add` command, but we would -also be waiting for it to install. If we run the command now, it will -return no new compilers. - -.. code-block:: console - - $ spack compiler add /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/gcc-6.1.0-j5576zbsot2ydljlthjzhsirsesnogvh/bin - ==> Found no new compilers - -If we had done a real install, the output would have been as follows: - -.. code-block:: console - - $ spack compiler add /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/gcc-6.1.0-j5576zbsot2ydljlthjzhsirsesnogvh/bin - ==> Added 1 new compiler to /g/g0/becker33/.spack/linux/compilers.yaml - gcc@6.1.0 diff --git a/lib/spack/docs/index.rst b/lib/spack/docs/index.rst index 65fb995d1e..ed2ff96ed0 100644 --- a/lib/spack/docs/index.rst +++ b/lib/spack/docs/index.rst @@ -61,11 +61,9 @@ or refer to the full manual below. .. toctree:: :maxdepth: 2 - :caption: Step-by-step + :caption: Tutorial - basic_usage_tutorial - module_file_tutorial - package_tutorial + tutorial_sc16 .. toctree:: :maxdepth: 2 diff --git a/lib/spack/docs/module_file_tutorial.rst b/lib/spack/docs/module_file_tutorial.rst deleted file mode 100644 index 83df756566..0000000000 --- a/lib/spack/docs/module_file_tutorial.rst +++ /dev/null @@ -1,982 +0,0 @@ -.. _module_tutorial: - -============================= -Module Configuration Tutorial -============================= - -This tutorial will guide you through the customization of both -content and naming of module files generated by Spack. - -Starting from the default Spack settings you will add an increasing -number of directives to the ``modules.yaml`` configuration file to -satisfy a number of constraints that mimic those that you may encounter -in a typical production environment at HPC sites. - -Even though the focus will be for the most part on customizing -TCL non-hierarchical module files, everything -you'll see applies also to other kinds of module files generated by Spack. - -The generation of Lua hierarchical -module files will be addressed at the end of the tutorial, -and you'll see that with minor modifications -to an existing ``modules.yaml`` written for TCL -non-hierarchical modules you'll get almost -for free the possibility to try a hierarchical layout. - -Let's start! - -.. _module_file_tutorial_prerequisites: - -------------- -Prerequisites -------------- - -Before proceeding further ensure: - -- you have LMod or Environment Modules available -- have :ref:`shell support ` activated in Spack - -If you need to install Lmod or Environment module you can refer -to the documentation :ref:`here `. - - -^^^^^^^^^^^^^^^^^^ -Add a new compiler -^^^^^^^^^^^^^^^^^^ - -Spack automatically scans the environment to search for available -compilers on first use. On a Ubuntu 14.04 a fresh clone will show -something like this: - -.. code-block:: console - - $ uname -a - Linux nuvolari 4.4.0-45-generic #66~14.04.1-Ubuntu SMP Wed Oct 19 15:05:38 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux - - $ spack compilers - ==> Available compilers - -- gcc ---------------------------------------------------------- - gcc@4.8 - -For the purpose of building a limited set of packages with some features -that will help showcasing the capabilities of -module customization the first thing we need is to build a new compiler: - -.. code-block:: console - - $ spack install gcc@6.2.0 - # ... - # Wait a long time - # ... - -Then we can use shell support for modules to add it to the list of known compilers: - -.. code-block:: console - - # The name of the generated module may vary - $ module load gcc-6.2.0-gcc-4.8-twd5nqg - - $ spack compiler add - ==> Added 1 new compiler to /home/mculpo/.spack/linux/compilers.yaml - gcc@6.2.0 - - $ spack compilers - ==> Available compilers - -- gcc ---------------------------------------------------------- - gcc@6.2.0 gcc@4.8 - -Note that the final 7 digits hash at the end of the generated module may vary depending -on architecture or package version. - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Build software that will be used in the tutorial -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Next you should install a few modules that will be used in the tutorial: - -.. code-block:: console - - $ spack install netlib-scalapack ^openmpi ^openblas - # ... - -The packages you need to install are: - -- ``netlib-scalapack ^openmpi ^openblas`` -- ``netlib-scalapack ^mpich ^openblas`` -- ``netlib-scalapack ^openmpi ^netlib-lapack`` -- ``netlib-scalapack ^mpich ^netlib-lapack`` -- ``py-scipy ^openblas`` - -In the end your environment should look something like: - -.. code-block:: console - - $ module avail - - ------------------------------------------------------------------------ /home/mculpo/wdir/spack/share/spack/modules/linux-Ubuntu14-x86_64 ------------------------------------------------------------------------ - binutils-2.27-gcc-4.8-dz3xevw libpciaccess-0.13.4-gcc-6.2.0-eo2siet lzo-2.09-gcc-6.2.0-jcngz72 netlib-scalapack-2.0.2-gcc-6.2.0-wnimqhw python-2.7.12-gcc-6.2.0-qu7rc5p - bzip2-1.0.6-gcc-6.2.0-csoc2mq libsigsegv-2.10-gcc-4.8-avb6azw m4-1.4.17-gcc-4.8-iggewke netlib-scalapack-2.0.2-gcc-6.2.0-wojunhq sqlite-3.8.5-gcc-6.2.0-td3zfe7 - cmake-3.5.2-gcc-6.2.0-6poypqg libsigsegv-2.10-gcc-6.2.0-g3qpmbi m4-1.4.17-gcc-6.2.0-lhgqa6s nettle-3.2-gcc-6.2.0-djdthlh tcl-8.6.5-gcc-4.8-atddxu7 - curl-7.50.3-gcc-6.2.0-2ffacqm libtool-2.4.6-gcc-6.2.0-kiepac6 mpc-1.0.3-gcc-4.8-lylv7lk openblas-0.2.19-gcc-6.2.0-js33umc util-macros-1.19.0-gcc-6.2.0-uoukuqk - expat-2.2.0-gcc-6.2.0-bxqnjar libxml2-2.9.4-gcc-6.2.0-3k4ykbe mpfr-3.1.4-gcc-4.8-bldfx3w openmpi-2.0.1-gcc-6.2.0-s3qbtby xz-5.2.2-gcc-6.2.0-t5lk6in - gcc-6.2.0-gcc-4.8-twd5nqg lmod-6.4.5-gcc-4.8-7v7bh7b mpich-3.2-gcc-6.2.0-5n5xoep openssl-1.0.2j-gcc-6.2.0-hibnfda zlib-1.2.8-gcc-4.8-bds4ies - gmp-6.1.1-gcc-4.8-uq52e2n lua-5.3.2-gcc-4.8-xozf2hx ncurses-6.0-gcc-4.8-u62fit4 pkg-config-0.29.1-gcc-6.2.0-rslsgcs zlib-1.2.8-gcc-6.2.0-asydrba - gmp-6.1.1-gcc-6.2.0-3cfh3hi lua-luafilesystem-1_6_3-gcc-4.8-sbzejlz ncurses-6.0-gcc-6.2.0-7tb426s py-nose-1.3.7-gcc-6.2.0-4gl5c42 - hwloc-1.11.4-gcc-6.2.0-3ostwel lua-luaposix-33.4.0-gcc-4.8-xf7y2p5 netlib-lapack-3.6.1-gcc-6.2.0-mirer2l py-numpy-1.11.1-gcc-6.2.0-i3rpk4e - isl-0.14-gcc-4.8-cq73t5m lz4-131-gcc-6.2.0-cagoem4 netlib-scalapack-2.0.2-gcc-6.2.0-6bqlxqy py-scipy-0.18.1-gcc-6.2.0-e6uljfi - libarchive-3.2.1-gcc-6.2.0-2b54aos lzma-4.32.7-gcc-6.2.0-sfmeynw netlib-scalapack-2.0.2-gcc-6.2.0-hpqb3dp py-setuptools-25.2.0-gcc-6.2.0-hkqauaa - ------------------------------------------------- -Filter unwanted modifications to the environment ------------------------------------------------- - -The non-hierarchical TCL module files that have been generated so far -follow the default rules for module generation, which are given -:ref:`here ` in the reference part of the manual. Taking a -look at the ``gcc`` module you'll see something like: - -.. code-block:: console - - $ module show gcc-6.2.0-gcc-4.8-twd5nqg - --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - /home/mculpo/wdir/spack/share/spack/modules/linux-Ubuntu14-x86_64/gcc-6.2.0-gcc-4.8-twd5nqg: - --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - whatis("gcc @6.2.0 ") - prepend_path("PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/bin") - prepend_path("CMAKE_PREFIX_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/") - prepend_path("MANPATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/share/man") - prepend_path("PKG_CONFIG_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/lib64/pkgconfig") - prepend_path("LIBRARY_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/lib64") - prepend_path("LD_LIBRARY_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/lib64") - prepend_path("CPATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/include") - help([[The GNU Compiler Collection includes front ends for C, C++, Objective-C, - Fortran, and Java. - ]]) - -As expected, a few environment variables representing paths will be modified -by the modules according to the default prefix inspection rules. - -Consider now the case that your site has decided that e.g. ``CPATH`` and -``LIBRARY_PATH`` modifications should not be present in module files. What you can -do to abide by the rules is to create a configuration file ``~/.spack/modules.yaml`` -with the following content: - -.. code-block:: yaml - - modules: - tcl: - all: - filter: - environment_blacklist: ['CPATH', 'LIBRARY_PATH'] - -Next you should regenerate all the module files: - -.. code-block:: console - - $ spack module refresh --module-type tcl - ==> You are about to regenerate tcl module files for: - - -- linux-Ubuntu14-x86_64 / gcc@4.8 ------------------------------ - dz3xevw binutils@2.27 uq52e2n gmp@6.1.1 avb6azw libsigsegv@2.10 xozf2hx lua@5.3.2 xf7y2p5 lua-luaposix@33.4.0 lylv7lk mpc@1.0.3 u62fit4 ncurses@6.0 bds4ies zlib@1.2.8 - twd5nqg gcc@6.2.0 cq73t5m isl@0.14 7v7bh7b lmod@6.4.5 sbzejlz lua-luafilesystem@1_6_3 iggewke m4@1.4.17 bldfx3w mpfr@3.1.4 atddxu7 tcl@8.6.5 - - ... - - ==> Do you want to proceed ? [y/n] - y - ==> Regenerating tcl module files - -If you take a look now at the module for ``gcc`` you'll see that the unwanted -paths have disappeared: - -.. code-block:: console - - $ module show gcc-6.2.0-gcc-4.8-twd5nqg - --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - /home/mculpo/wdir/spack/share/spack/modules/linux-Ubuntu14-x86_64/gcc-6.2.0-gcc-4.8-twd5nqg: - --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - whatis("gcc @6.2.0 ") - prepend_path("PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/bin") - prepend_path("CMAKE_PREFIX_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/") - prepend_path("MANPATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/share/man") - prepend_path("PKG_CONFIG_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/lib64/pkgconfig") - prepend_path("LD_LIBRARY_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/lib64") - help([[The GNU Compiler Collection includes front ends for C, C++, Objective-C, - Fortran, and Java. - ]]) - ----------------------------------------------- -Prevent some module files from being generated ----------------------------------------------- - -Another common request at many sites is to avoid exposing software that -is only needed as an intermediate step when building a newer stack. -Let's try to prevent the generation of -module files for anything that is compiled with ``gcc@4.8`` (the OS provided compiler). - -To do this you should add a ``blacklist`` keyword to the configuration file: - -.. code-block:: yaml - :emphasize-lines: 3,4 - - modules: - tcl: - blacklist: - - '%gcc@4.8' - all: - filter: - environment_blacklist: ['CPATH', 'LIBRARY_PATH'] - -and regenerate the module files: - -.. code-block:: console - - $ spack module refresh --module-type tcl --delete-tree - ==> You are about to regenerate tcl module files for: - - -- linux-Ubuntu14-x86_64 / gcc@4.8 ------------------------------ - dz3xevw binutils@2.27 uq52e2n gmp@6.1.1 avb6azw libsigsegv@2.10 xozf2hx lua@5.3.2 xf7y2p5 lua-luaposix@33.4.0 lylv7lk mpc@1.0.3 u62fit4 ncurses@6.0 bds4ies zlib@1.2.8 - twd5nqg gcc@6.2.0 cq73t5m isl@0.14 7v7bh7b lmod@6.4.5 sbzejlz lua-luafilesystem@1_6_3 iggewke m4@1.4.17 bldfx3w mpfr@3.1.4 atddxu7 tcl@8.6.5 - - -- linux-Ubuntu14-x86_64 / gcc@6.2.0 ---------------------------- - csoc2mq bzip2@1.0.6 2b54aos libarchive@3.2.1 sfmeynw lzma@4.32.7 wnimqhw netlib-scalapack@2.0.2 s3qbtby openmpi@2.0.1 hkqauaa py-setuptools@25.2.0 - 6poypqg cmake@3.5.2 eo2siet libpciaccess@0.13.4 jcngz72 lzo@2.09 6bqlxqy netlib-scalapack@2.0.2 hibnfda openssl@1.0.2j qu7rc5p python@2.7.12 - 2ffacqm curl@7.50.3 g3qpmbi libsigsegv@2.10 lhgqa6s m4@1.4.17 wojunhq netlib-scalapack@2.0.2 rslsgcs pkg-config@0.29.1 td3zfe7 sqlite@3.8.5 - bxqnjar expat@2.2.0 kiepac6 libtool@2.4.6 5n5xoep mpich@3.2 hpqb3dp netlib-scalapack@2.0.2 4gl5c42 py-nose@1.3.7 uoukuqk util-macros@1.19.0 - 3cfh3hi gmp@6.1.1 3k4ykbe libxml2@2.9.4 7tb426s ncurses@6.0 djdthlh nettle@3.2 i3rpk4e py-numpy@1.11.1 t5lk6in xz@5.2.2 - 3ostwel hwloc@1.11.4 cagoem4 lz4@131 mirer2l netlib-lapack@3.6.1 js33umc openblas@0.2.19 e6uljfi py-scipy@0.18.1 asydrba zlib@1.2.8 - - ==> Do you want to proceed ? [y/n] - y - - $ module avail - - ------------------------------------------------------------------------ /home/mculpo/wdir/spack/share/spack/modules/linux-Ubuntu14-x86_64 ------------------------------------------------------------------------ - bzip2-1.0.6-gcc-6.2.0-csoc2mq libsigsegv-2.10-gcc-6.2.0-g3qpmbi ncurses-6.0-gcc-6.2.0-7tb426s openmpi-2.0.1-gcc-6.2.0-s3qbtby sqlite-3.8.5-gcc-6.2.0-td3zfe7 - cmake-3.5.2-gcc-6.2.0-6poypqg libtool-2.4.6-gcc-6.2.0-kiepac6 netlib-lapack-3.6.1-gcc-6.2.0-mirer2l openssl-1.0.2j-gcc-6.2.0-hibnfda util-macros-1.19.0-gcc-6.2.0-uoukuqk - curl-7.50.3-gcc-6.2.0-2ffacqm libxml2-2.9.4-gcc-6.2.0-3k4ykbe netlib-scalapack-2.0.2-gcc-6.2.0-6bqlxqy pkg-config-0.29.1-gcc-6.2.0-rslsgcs xz-5.2.2-gcc-6.2.0-t5lk6in - expat-2.2.0-gcc-6.2.0-bxqnjar lz4-131-gcc-6.2.0-cagoem4 netlib-scalapack-2.0.2-gcc-6.2.0-hpqb3dp py-nose-1.3.7-gcc-6.2.0-4gl5c42 zlib-1.2.8-gcc-6.2.0-asydrba - gmp-6.1.1-gcc-6.2.0-3cfh3hi lzma-4.32.7-gcc-6.2.0-sfmeynw netlib-scalapack-2.0.2-gcc-6.2.0-wnimqhw py-numpy-1.11.1-gcc-6.2.0-i3rpk4e - hwloc-1.11.4-gcc-6.2.0-3ostwel lzo-2.09-gcc-6.2.0-jcngz72 netlib-scalapack-2.0.2-gcc-6.2.0-wojunhq py-scipy-0.18.1-gcc-6.2.0-e6uljfi - libarchive-3.2.1-gcc-6.2.0-2b54aos m4-1.4.17-gcc-6.2.0-lhgqa6s nettle-3.2-gcc-6.2.0-djdthlh py-setuptools-25.2.0-gcc-6.2.0-hkqauaa - libpciaccess-0.13.4-gcc-6.2.0-eo2siet mpich-3.2-gcc-6.2.0-5n5xoep openblas-0.2.19-gcc-6.2.0-js33umc python-2.7.12-gcc-6.2.0-qu7rc5p - -This time it is convenient to pass the option ``--delete-tree`` to the command that -regenerates the module files to instruct it to delete the existing tree and regenerate -a new one instead of overwriting the files in the existing directory. - -If you pay careful attention you'll see though that we went too far in blacklisting modules: -the module for ``gcc@6.2.0`` disappeared as it was bootstrapped with ``gcc@4.8``. To specify -exceptions to the blacklist rules you can use ``whitelist``: - -.. code-block:: yaml - :emphasize-lines: 3,4 - - modules: - tcl: - whitelist: - - gcc - blacklist: - - '%gcc@4.8' - all: - filter: - environment_blacklist: ['CPATH', 'LIBRARY_PATH'] - -``whitelist`` rules always have precedence over ``blacklist`` rules. If you regenerate the modules again: - -.. code-block:: console - - $ spack module refresh --module-type tcl -y - -you'll see that now the module for ``gcc@6.2.0`` has reappeared: - -.. code-block:: console - - $ module avail gcc-6.2.0-gcc-4.8-twd5nqg - - ------------------------------------------------------------------------ /home/mculpo/wdir/spack/share/spack/modules/linux-Ubuntu14-x86_64 ------------------------------------------------------------------------ - gcc-6.2.0-gcc-4.8-twd5nqg - -------------------------- -Change module file naming -------------------------- - -The next step in making module files more user-friendly is to -improve their naming scheme. -To reduce the length of the hash or remove it altogether you can -use the ``hash_length`` keyword in the configuration file: - -.. TODO: give reasons to remove hashes if they are not evident enough? - -.. code-block:: yaml - :emphasize-lines: 3 - - modules: - tcl: - hash_length: 0 - whitelist: - - gcc - blacklist: - - '%gcc@4.8' - all: - filter: - environment_blacklist: ['CPATH', 'LIBRARY_PATH'] - -If you try to regenerate the module files now you will get an error: - -.. code-block:: console - - $ spack module refresh --module-type tcl --delete-tree -y - ==> Error: Name clashes detected in module files: - - file : /home/mculpo/wdir/spack/share/spack/modules/linux-Ubuntu14-x86_64/netlib-scalapack-2.0.2-gcc-6.2.0 - spec : netlib-scalapack@2.0.2%gcc@6.2.0~fpic+shared arch=linux-Ubuntu14-x86_64 - spec : netlib-scalapack@2.0.2%gcc@6.2.0~fpic+shared arch=linux-Ubuntu14-x86_64 - spec : netlib-scalapack@2.0.2%gcc@6.2.0~fpic+shared arch=linux-Ubuntu14-x86_64 - spec : netlib-scalapack@2.0.2%gcc@6.2.0~fpic+shared arch=linux-Ubuntu14-x86_64 - - ==> Error: Operation aborted - -.. note:: - We try to check for errors upfront! - In Spack we check for errors upfront whenever possible, so don't worry about your module files: - as a name clash was detected nothing has been changed on disk. - -The problem here is that without -the hashes the four different flavors of ``netlib-scalapack`` map to the same module file -name. We have the possibility to add suffixes to differentiate them: - -.. code-block:: yaml - :emphasize-lines: 9-11,14-17 - - modules: - tcl: - hash_length: 0 - whitelist: - - gcc - blacklist: - - '%gcc@4.8' - all: - suffixes: - '^openblas': openblas - '^netlib-lapack': netlib - filter: - environment_blacklist: ['CPATH', 'LIBRARY_PATH'] - netlib-scalapack: - suffixes: - '^openmpi': openmpi - '^mpich': mpich - -As you can see it is possible to specify rules that applies only to a -restricted set of packages using :ref:`anonymous specs `. -Regenerating module files now we obtain: - -.. code-block:: console - - $ spack module refresh --module-type tcl --delete-tree -y - ==> Regenerating tcl module files - $ module avail - - ------------------------------------------------------------------------ /home/mculpo/wdir/spack/share/spack/modules/linux-Ubuntu14-x86_64 ------------------------------------------------------------------------ - bzip2-1.0.6-gcc-6.2.0 libpciaccess-0.13.4-gcc-6.2.0 mpich-3.2-gcc-6.2.0 openblas-0.2.19-gcc-6.2.0 python-2.7.12-gcc-6.2.0 - cmake-3.5.2-gcc-6.2.0 libsigsegv-2.10-gcc-6.2.0 ncurses-6.0-gcc-6.2.0 openmpi-2.0.1-gcc-6.2.0 sqlite-3.8.5-gcc-6.2.0 - curl-7.50.3-gcc-6.2.0 libtool-2.4.6-gcc-6.2.0 netlib-lapack-3.6.1-gcc-6.2.0 openssl-1.0.2j-gcc-6.2.0 util-macros-1.19.0-gcc-6.2.0 - expat-2.2.0-gcc-6.2.0 libxml2-2.9.4-gcc-6.2.0 netlib-scalapack-2.0.2-gcc-6.2.0-netlib-mpich pkg-config-0.29.1-gcc-6.2.0 xz-5.2.2-gcc-6.2.0 - gcc-6.2.0-gcc-4.8 lz4-131-gcc-6.2.0 netlib-scalapack-2.0.2-gcc-6.2.0-netlib-openmpi py-nose-1.3.7-gcc-6.2.0 zlib-1.2.8-gcc-6.2.0 - gmp-6.1.1-gcc-6.2.0 lzma-4.32.7-gcc-6.2.0 netlib-scalapack-2.0.2-gcc-6.2.0-openblas-mpich py-numpy-1.11.1-gcc-6.2.0-openblas - hwloc-1.11.4-gcc-6.2.0 lzo-2.09-gcc-6.2.0 netlib-scalapack-2.0.2-gcc-6.2.0-openblas-openmpi py-scipy-0.18.1-gcc-6.2.0-openblas - libarchive-3.2.1-gcc-6.2.0 m4-1.4.17-gcc-6.2.0 nettle-3.2-gcc-6.2.0 py-setuptools-25.2.0-gcc-6.2.0 - -Finally we can set a ``naming_scheme`` to prevent users from loading -modules that refer to different flavors of the same library/application: - -.. code-block:: yaml - :emphasize-lines: 4,10,11 - - modules: - tcl: - hash_length: 0 - naming_scheme: '${PACKAGE}/${VERSION}-${COMPILERNAME}-${COMPILERVER}' - whitelist: - - gcc - blacklist: - - '%gcc@4.8' - all: - conflict: - - '${PACKAGE}' - suffixes: - '^openblas': openblas - '^netlib-lapack': netlib - filter: - environment_blacklist: ['CPATH', 'LIBRARY_PATH'] - netlib-scalapack: - suffixes: - '^openmpi': openmpi - '^mpich': mpich - -The final result should look like: - -.. code-block:: console - - $ module avail - - ------------------------------------------------------------------------ /home/mculpo/wdir/spack/share/spack/modules/linux-Ubuntu14-x86_64 ------------------------------------------------------------------------ - bzip2/1.0.6-gcc-6.2.0 libpciaccess/0.13.4-gcc-6.2.0 mpich/3.2-gcc-6.2.0 openblas/0.2.19-gcc-6.2.0 python/2.7.12-gcc-6.2.0 - cmake/3.5.2-gcc-6.2.0 libsigsegv/2.10-gcc-6.2.0 ncurses/6.0-gcc-6.2.0 openmpi/2.0.1-gcc-6.2.0 sqlite/3.8.5-gcc-6.2.0 - curl/7.50.3-gcc-6.2.0 libtool/2.4.6-gcc-6.2.0 netlib-lapack/3.6.1-gcc-6.2.0 openssl/1.0.2j-gcc-6.2.0 util-macros/1.19.0-gcc-6.2.0 - expat/2.2.0-gcc-6.2.0 libxml2/2.9.4-gcc-6.2.0 netlib-scalapack/2.0.2-gcc-6.2.0-netlib-mpich pkg-config/0.29.1-gcc-6.2.0 xz/5.2.2-gcc-6.2.0 - gcc/6.2.0-gcc-4.8 lz4/131-gcc-6.2.0 netlib-scalapack/2.0.2-gcc-6.2.0-netlib-openmpi py-nose/1.3.7-gcc-6.2.0 zlib/1.2.8-gcc-6.2.0 - gmp/6.1.1-gcc-6.2.0 lzma/4.32.7-gcc-6.2.0 netlib-scalapack/2.0.2-gcc-6.2.0-openblas-mpich py-numpy/1.11.1-gcc-6.2.0-openblas - hwloc/1.11.4-gcc-6.2.0 lzo/2.09-gcc-6.2.0 netlib-scalapack/2.0.2-gcc-6.2.0-openblas-openmpi (D) py-scipy/0.18.1-gcc-6.2.0-openblas - libarchive/3.2.1-gcc-6.2.0 m4/1.4.17-gcc-6.2.0 nettle/3.2-gcc-6.2.0 py-setuptools/25.2.0-gcc-6.2.0 - -.. note:: - TCL specific directive - The directives ``naming_scheme`` and ``conflict`` are TCL specific and do not apply - to the ``dotkit`` or ``lmod`` sections in the configuration file. - ------------------------------------- -Add custom environment modifications ------------------------------------- - -At many sites it is customary to set an environment variable in a -package's module file that points to the folder in which the package -is installed. You can achieve this with Spack by adding an -``environment`` directive to the configuration file: - -.. code-block:: yaml - :emphasize-lines: 17-19 - - modules: - tcl: - hash_length: 0 - naming_scheme: '${PACKAGE}/${VERSION}-${COMPILERNAME}-${COMPILERVER}' - whitelist: - - gcc - blacklist: - - '%gcc@4.8' - all: - conflict: - - '${PACKAGE}' - suffixes: - '^openblas': openblas - '^netlib-lapack': netlib - filter: - environment_blacklist: ['CPATH', 'LIBRARY_PATH'] - environment: - set: - '${PACKAGE}_ROOT': '${PREFIX}' - netlib-scalapack: - suffixes: - '^openmpi': openmpi - '^mpich': mpich - -There are many variable tokens available to use in the ``environment`` -and ``naming_scheme`` directives, such as ``${PACKAGE}``, -``${VERSION}``, etc. (see the :meth:`~spack.spec.Spec.format` API -documentation for the complete list). - -Regenerating the module files should result in something like: - -.. code-block:: console - :emphasize-lines: 14 - - $ spack module refresh -y --module-type tcl - ==> Regenerating tcl module files - - $ module show gcc - --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - /home/mculpo/wdir/spack/share/spack/modules/linux-Ubuntu14-x86_64/gcc/6.2.0-gcc-4.8: - --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - whatis("gcc @6.2.0 ") - prepend_path("PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/bin") - prepend_path("CMAKE_PREFIX_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/") - prepend_path("MANPATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/share/man") - prepend_path("PKG_CONFIG_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/lib64/pkgconfig") - prepend_path("LD_LIBRARY_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/lib64") - setenv("GCC_ROOT","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u") - conflict("gcc") - help([[The GNU Compiler Collection includes front ends for C, C++, Objective-C, - Fortran, and Java. - ]]) - -As you see the ``gcc`` module has the environment variable ``GCC_ROOT`` set. - -Sometimes it's also useful to apply environment modifications selectively and target -only certain packages. You can, for instance set the common variables ``CC``, ``CXX``, -etc. in the ``gcc`` module file and apply other custom modifications to the -``openmpi`` modules as follows: - -.. code-block:: yaml - :emphasize-lines: 20-32 - - modules: - tcl: - hash_length: 0 - naming_scheme: '${PACKAGE}/${VERSION}-${COMPILERNAME}-${COMPILERVER}' - whitelist: - - gcc - blacklist: - - '%gcc@4.8' - all: - conflict: - - '${PACKAGE}' - suffixes: - '^openblas': openblas - '^netlib-lapack': netlib - filter: - environment_blacklist: ['CPATH', 'LIBRARY_PATH'] - environment: - set: - '${PACKAGE}_ROOT': '${PREFIX}' - gcc: - environment: - set: - CC: gcc - CXX: g++ - FC: gfortran - F90: gfortran - F77: gfortran - openmpi: - environment: - set: - SLURM_MPI_TYPE: pmi2 - OMPI_MCA_btl_openib_warn_default_gid_prefix: '0' - netlib-scalapack: - suffixes: - '^openmpi': openmpi - '^mpich': mpich - -This time we will be more selective and regenerate only the ``gcc`` and -``openmpi`` module files: - -.. code-block:: console - - $ spack module refresh -y --module-type tcl gcc - ==> Regenerating tcl module files - - $ spack module refresh -y --module-type tcl openmpi - ==> Regenerating tcl module files - - $ module show gcc - --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - /home/mculpo/wdir/spack/share/spack/modules/linux-Ubuntu14-x86_64/gcc/6.2.0-gcc-4.8: - --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - whatis("gcc @6.2.0 ") - prepend_path("PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/bin") - prepend_path("CMAKE_PREFIX_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/") - prepend_path("MANPATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/share/man") - prepend_path("PKG_CONFIG_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/lib64/pkgconfig") - prepend_path("LD_LIBRARY_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/lib64") - setenv("GCC_ROOT","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u") - setenv("CC","gcc") - setenv("CXX","g++") - setenv("F90","gfortran") - setenv("FC","gfortran") - setenv("F77","gfortran") - conflict("gcc") - help([[The GNU Compiler Collection includes front ends for C, C++, Objective-C, - Fortran, and Java. - ]]) - - $ module show openmpi - --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - /home/mculpo/wdir/spack/share/spack/modules/linux-Ubuntu14-x86_64/openmpi/2.0.1-gcc-6.2.0: - --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - whatis("openmpi @2.0.1 ") - prepend_path("PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-6.2.0/openmpi-2.0.1-s3qbtbyh3y5y4gkchmhcuak7th44l53w/bin") - prepend_path("CMAKE_PREFIX_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-6.2.0/openmpi-2.0.1-s3qbtbyh3y5y4gkchmhcuak7th44l53w/") - prepend_path("LD_LIBRARY_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-6.2.0/openmpi-2.0.1-s3qbtbyh3y5y4gkchmhcuak7th44l53w/lib") - prepend_path("PKG_CONFIG_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-6.2.0/openmpi-2.0.1-s3qbtbyh3y5y4gkchmhcuak7th44l53w/lib/pkgconfig") - prepend_path("MANPATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-6.2.0/openmpi-2.0.1-s3qbtbyh3y5y4gkchmhcuak7th44l53w/share/man") - setenv("SLURM_MPI_TYPE","pmi2") - setenv("OMPI_MCA_BTL_OPENIB_WARN_DEFAULT_GID_PREFIX","0") - setenv("OPENMPI_ROOT","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-6.2.0/openmpi-2.0.1-s3qbtbyh3y5y4gkchmhcuak7th44l53w") - conflict("openmpi") - help([[The Open MPI Project is an open source Message Passing Interface - implementation that is developed and maintained by a consortium of - academic, research, and industry partners. Open MPI is therefore able to - combine the expertise, technologies, and resources from all across the - High Performance Computing community in order to build the best MPI - library available. Open MPI offers advantages for system and software - vendors, application developers and computer science researchers. - ]]) - - ---------------------- -Autoload dependencies ---------------------- - -Spack can also generate module files that contain code to load the -dependencies automatically. You can, for instance generate python -modules that load their dependencies by adding the ``autoload`` -directive and assigning it the value ``direct``: - -.. code-block:: yaml - :emphasize-lines: 37,38 - - modules: - tcl: - hash_length: 0 - naming_scheme: '${PACKAGE}/${VERSION}-${COMPILERNAME}-${COMPILERVER}' - whitelist: - - gcc - blacklist: - - '%gcc@4.8' - all: - conflict: - - '${PACKAGE}' - suffixes: - '^openblas': openblas - '^netlib-lapack': netlib - filter: - environment_blacklist: ['CPATH', 'LIBRARY_PATH'] - environment: - set: - '${PACKAGE}_ROOT': '${PREFIX}' - gcc: - environment: - set: - CC: gcc - CXX: g++ - FC: gfortran - F90: gfortran - F77: gfortran - openmpi: - environment: - set: - SLURM_MPI_TYPE: pmi2 - OMPI_MCA_btl_openib_warn_default_gid_prefix: '0' - netlib-scalapack: - suffixes: - '^openmpi': openmpi - '^mpich': mpich - ^python: - autoload: 'direct' - -and regenerating the module files for every package that depends on ``python``: - -.. code-block:: console - - $ spack module refresh -y --module-type tcl ^python - ==> Regenerating tcl module files - -Now the ``py-scipy`` module will be: - -.. code-block:: tcl - - #%Module1.0 - ## Module file created by spack (https://github.com/LLNL/spack) on 2016-11-02 20:53:21.283547 - ## - ## py-scipy@0.18.1%gcc@6.2.0 arch=linux-Ubuntu14-x86_64-e6uljfi - ## - module-whatis "py-scipy @0.18.1" - - proc ModulesHelp { } { - puts stderr "SciPy (pronounced "Sigh Pie") is a Scientific Library for Python. It" - puts stderr "provides many user-friendly and efficient numerical routines such as" - puts stderr "routines for numerical integration and optimization." - } - - if ![ is-loaded python/2.7.12-gcc-6.2.0 ] { - puts stderr "Autoloading python/2.7.12-gcc-6.2.0" - module load python/2.7.12-gcc-6.2.0 - } - - if ![ is-loaded openblas/0.2.19-gcc-6.2.0 ] { - puts stderr "Autoloading openblas/0.2.19-gcc-6.2.0" - module load openblas/0.2.19-gcc-6.2.0 - } - - if ![ is-loaded py-numpy/1.11.1-gcc-6.2.0-openblas ] { - puts stderr "Autoloading py-numpy/1.11.1-gcc-6.2.0-openblas" - module load py-numpy/1.11.1-gcc-6.2.0-openblas - } - - prepend-path CMAKE_PREFIX_PATH "/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-6.2.0/py-scipy-0.18.1-e6uljfiffgym4xvj6wveevqxfqnfb3gh/" - prepend-path LD_LIBRARY_PATH "/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-6.2.0/py-scipy-0.18.1-e6uljfiffgym4xvj6wveevqxfqnfb3gh/lib" - prepend-path PYTHONPATH "/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-6.2.0/py-scipy-0.18.1-e6uljfiffgym4xvj6wveevqxfqnfb3gh/lib/python2.7/site-packages" - setenv PY_SCIPY_ROOT "/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-6.2.0/py-scipy-0.18.1-e6uljfiffgym4xvj6wveevqxfqnfb3gh" - conflict py-scipy - -and will contain code to autoload all the dependencies: - -.. code-block:: console - - $ module load py-scipy - Autoloading python/2.7.12-gcc-6.2.0 - Autoloading openblas/0.2.19-gcc-6.2.0 - Autoloading py-numpy/1.11.1-gcc-6.2.0-openblas - ------------------------------ -Lua hierarchical module files ------------------------------ - -In the final part of this tutorial you will modify ``modules.yaml`` to generate -Lua hierarchical module files. You will see that most of the directives used before -are also valid in the ``lmod`` context. - -^^^^^^^^^^^^^^^^^ -Core/Compiler/MPI -^^^^^^^^^^^^^^^^^ - -.. warning:: - Only LMod supports Lua hierarchical module files - For this part of the tutorial you need to be using LMod to - manage your environment. - -The most common hierarchy is the so called ``Core/Compiler/MPI``. To have an idea -how a hierarchy is organized you may refer to the -`Lmod guide `_. -Since ``lmod`` is not enabled by default, you need to add it to the list of -enabled module file generators. The other things you need to do are: - -- change the ``tcl`` tag to ``lmod`` -- remove ``tcl`` specific directives (``naming_scheme`` and ``conflict``) -- set which compilers are considered ``core`` -- remove the ``mpi`` related suffixes (as they will be substituted by hierarchies) - -After modifications the configuration file will be: - -.. code-block:: yaml - :emphasize-lines: 2-6 - - modules: - enable:: - - lmod - lmod: - core_compilers: - - 'gcc@4.8' - hash_length: 0 - whitelist: - - gcc - blacklist: - - '%gcc@4.8' - all: - suffixes: - '^openblas': openblas - '^netlib-lapack': netlib - filter: - environment_blacklist: ['CPATH', 'LIBRARY_PATH'] - environment: - set: - '${PACKAGE}_ROOT': '${PREFIX}' - gcc: - environment: - set: - CC: gcc - CXX: g++ - FC: gfortran - F90: gfortran - F77: gfortran - openmpi: - environment: - set: - SLURM_MPI_TYPE: pmi2 - OMPI_MCA_btl_openib_warn_default_gid_prefix: '0' - - -.. note:: - The double colon - The double colon after ``enable`` is intentional and it serves the - purpose of overriding the default list of enabled generators so - that only ``lmod`` will be active (see :ref:`the reference - manual ` for a more detailed explanation of - config scopes). - -The directive ``core_compilers`` accepts a list of compilers : everything built -using these compilers will create a module in the ``Core`` part of the hierarchy. It is -common practice to put the OS provided compilers in the list and only build common utilities -and other compilers in ``Core``. - -If you regenerate the module files - -.. code-block:: console - - $ spack module refresh --module-type lmod --delete-tree -y - -and update ``MODULEPATH`` to point to the ``Core`` folder, and -list the available modules, you'll see: - -.. code-block:: console - - $ module unuse /home/mculpo/wdir/spack/share/spack/modules/linux-Ubuntu14-x86_64 - $ module use /home/mculpo/wdir/spack/share/spack/lmod/linux-Ubuntu14-x86_64/Core - $ module avail - - ----------------------------------------------------------------------- /home/mculpo/wdir/spack/share/spack/lmod/linux-Ubuntu14-x86_64/Core ----------------------------------------------------------------------- - gcc/6.2.0 - -The only module visible now is ``gcc``. Loading that you will make -visible the ``Compiler`` part of the software stack that was built with ``gcc/6.2.0``: - -.. code-block:: console - - $ module load gcc - $ module avail - - -------------------------------------------------------------------- /home/mculpo/wdir/spack/share/spack/lmod/linux-Ubuntu14-x86_64/gcc/6.2.0 --------------------------------------------------------------------- - binutils/2.27 curl/7.50.3 hwloc/1.11.4 libtool/2.4.6 lzo/2.09 netlib-lapack/3.6.1 openssl/1.0.2j py-scipy/0.18.1-openblas util-macros/1.19.0 - bison/3.0.4 expat/2.2.0 libarchive/3.2.1 libxml2/2.9.4 m4/1.4.17 nettle/3.2 pkg-config/0.29.1 py-setuptools/25.2.0 xz/5.2.2 - bzip2/1.0.6 flex/2.6.0 libpciaccess/0.13.4 lz4/131 mpich/3.2 openblas/0.2.19 py-nose/1.3.7 python/2.7.12 zlib/1.2.8 - cmake/3.6.1 gmp/6.1.1 libsigsegv/2.10 lzma/4.32.7 ncurses/6.0 openmpi/2.0.1 py-numpy/1.11.1-openblas sqlite/3.8.5 - - ----------------------------------------------------------------------- /home/mculpo/wdir/spack/share/spack/lmod/linux-Ubuntu14-x86_64/Core ----------------------------------------------------------------------- - gcc/6.2.0 (L) - -The same holds true for the ``MPI`` part of the stack, that you can enable by loading -either ``mpich`` or ``openmpi``. The nice features of LMod will become evident -once you'll try switching among different stacks: - -.. code-block:: console - - $ module load mpich - $ module avail - - ----------------------------------------------------------- /home/mculpo/wdir/spack/share/spack/lmod/linux-Ubuntu14-x86_64/mpich/3.2-5n5xoep/gcc/6.2.0 ------------------------------------------------------------ - netlib-scalapack/2.0.2-netlib netlib-scalapack/2.0.2-openblas (D) - - -------------------------------------------------------------------- /home/mculpo/wdir/spack/share/spack/lmod/linux-Ubuntu14-x86_64/gcc/6.2.0 --------------------------------------------------------------------- - binutils/2.27 curl/7.50.3 hwloc/1.11.4 libtool/2.4.6 lzo/2.09 netlib-lapack/3.6.1 openssl/1.0.2j py-scipy/0.18.1-openblas util-macros/1.19.0 - bison/3.0.4 expat/2.2.0 libarchive/3.2.1 libxml2/2.9.4 m4/1.4.17 nettle/3.2 pkg-config/0.29.1 py-setuptools/25.2.0 xz/5.2.2 - bzip2/1.0.6 flex/2.6.0 libpciaccess/0.13.4 lz4/131 mpich/3.2 (L) openblas/0.2.19 py-nose/1.3.7 python/2.7.12 zlib/1.2.8 - cmake/3.6.1 gmp/6.1.1 libsigsegv/2.10 lzma/4.32.7 ncurses/6.0 openmpi/2.0.1 py-numpy/1.11.1-openblas sqlite/3.8.5 - - ----------------------------------------------------------------------- /home/mculpo/wdir/spack/share/spack/lmod/linux-Ubuntu14-x86_64/Core ----------------------------------------------------------------------- - gcc/6.2.0 (L) - - $ module load openblas netlib-scalapack/2.0.2-openblas - $ module list - - Currently Loaded Modules: - 1) gcc/6.2.0 2) mpich/3.2 3) openblas/0.2.19 4) netlib-scalapack/2.0.2-openblas - - $ module load openmpi - - Lmod is automatically replacing "mpich/3.2" with "openmpi/2.0.1" - - - Due to MODULEPATH changes the following have been reloaded: - 1) netlib-scalapack/2.0.2-openblas - -This layout is already a great improvement over the usual non-hierarchical layout, -but it still has an asymmetry: ``LAPACK`` providers are semantically the same as ``MPI`` -providers, but they are still not part of the hierarchy. We'll see a possible solution -next. - -.. Activate lmod and turn the previous modifications into lmod: - Add core compilers - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Extend the hierarchy to other virtual providers -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. warning:: - This is an experimental feature - Having a hierarchy deeper than ``Core``/``Compiler``/``MPI`` is an experimental - feature, still not fully supported by ``module spider``, - see `here `_. Furthermore its use - with hierarchies more complex than ``Core``/``Compiler``/``MPI``/``LAPACK`` - has not been thoroughly tested in production environments. - -Spack permits you to generate Lua hierarchical module files where users -can add an arbitrary list of virtual providers to the triplet -``Core``/``Compiler``/``MPI``. A configuration file like: - -.. code-block:: yaml - :emphasize-lines: 7,8 - - modules: - enable:: - - lmod - lmod: - core_compilers: - - 'gcc@4.8' - hierarchical_scheme: - - lapack - hash_length: 0 - whitelist: - - gcc - blacklist: - - '%gcc@4.8' - - readline - all: - filter: - environment_blacklist: ['CPATH', 'LIBRARY_PATH'] - environment: - set: - '${PACKAGE}_ROOT': '${PREFIX}' - gcc: - environment: - set: - CC: gcc - CXX: g++ - FC: gfortran - F90: gfortran - F77: gfortran - openmpi: - environment: - set: - SLURM_MPI_TYPE: pmi2 - OMPI_MCA_btl_openib_warn_default_gid_prefix: '0' - -will add ``lapack`` providers to the mix. After the usual regeneration of module files: - -.. code-block:: console - - $ module purge - $ spack module refresh --module-type lmod --delete-tree -y - ==> Regenerating lmod module files - -you will have something like: - -.. code-block:: console - - $ module load gcc - $ module load openblas - $ module load openmpi - $ module avail - - --------------------------------------------- /home/mculpo/wdir/spack/share/spack/lmod/linux-Ubuntu14-x86_64/openblas/0.2.19-js33umc/openmpi/2.0.1-s3qbtby/gcc/6.2.0 ---------------------------------------------- - netlib-scalapack/2.0.2 - - -------------------------------------------------------- /home/mculpo/wdir/spack/share/spack/lmod/linux-Ubuntu14-x86_64/openblas/0.2.19-js33umc/gcc/6.2.0 --------------------------------------------------------- - py-numpy/1.11.1 py-scipy/0.18.1 - - -------------------------------------------------------------------- /home/mculpo/wdir/spack/share/spack/lmod/linux-Ubuntu14-x86_64/gcc/6.2.0 --------------------------------------------------------------------- - binutils/2.27 curl/7.50.3 hwloc/1.11.4 libtool/2.4.6 lzo/2.09 netlib-lapack/3.6.1 openssl/1.0.2j python/2.7.12 zlib/1.2.8 - bison/3.0.4 expat/2.2.0 libarchive/3.2.1 libxml2/2.9.4 m4/1.4.17 nettle/3.2 pkg-config/0.29.1 sqlite/3.8.5 - bzip2/1.0.6 flex/2.6.0 libpciaccess/0.13.4 lz4/131 mpich/3.2 openblas/0.2.19 (L) py-nose/1.3.7 util-macros/1.19.0 - cmake/3.6.1 gmp/6.1.1 libsigsegv/2.10 lzma/4.32.7 ncurses/6.0 openmpi/2.0.1 (L) py-setuptools/25.2.0 xz/5.2.2 - - ----------------------------------------------------------------------- /home/mculpo/wdir/spack/share/spack/lmod/linux-Ubuntu14-x86_64/Core ----------------------------------------------------------------------- - gcc/6.2.0 (L) - -Now both the ``MPI`` and the ``LAPACK`` providers are handled by LMod as hierarchies: - -.. code-block:: console - - $ module load py-numpy netlib-scalapack - $ module load mpich - - Lmod is automatically replacing "openmpi/2.0.1" with "mpich/3.2" - - - Due to MODULEPATH changes the following have been reloaded: - 1) netlib-scalapack/2.0.2 - - $ module load netlib-lapack - - Lmod is automatically replacing "openblas/0.2.19" with "netlib-lapack/3.6.1" - - - Inactive Modules: - 1) py-numpy - - Due to MODULEPATH changes the following have been reloaded: - 1) netlib-scalapack/2.0.2 - -making the use of tags to differentiate them unnecessary. -Note that because we only compiled ``py-numpy`` with ``openblas`` the module -is made inactive when we switch the ``LAPACK`` provider. The user -environment will now be consistent by design! diff --git a/lib/spack/docs/package_tutorial.rst b/lib/spack/docs/package_tutorial.rst deleted file mode 100644 index e955780345..0000000000 --- a/lib/spack/docs/package_tutorial.rst +++ /dev/null @@ -1,463 +0,0 @@ -.. _package_tutorial: - -========================= -Package Creation Tutorial -========================= - -This tutorial will walk you through the steps behind building a simple -package installation script. We'll focus building an mpileaks package, -which is a MPI debugging tool. By creating a package file we're -essentially giving Spack a recipe for how to build a particular piece of -software. We're describing some of the software's dependencies, where to -find the package, what commands and options are used to build the package -from source, and more. Once we've specified a package's recipe, we can -ask Spack to build that package in many different ways. - -This tutorial assumes you have a basic familiarity with some of the Spack -commands, and that you have a working version of Spack installed. If -not, we suggest looking at Spack's *Getting Started* guide. This -tutorial also assumes you have at least a beginner's-level familiarity -with Python. - -Also note that this document is a tutorial. It can help you get started -with packaging, but is not intended to be complete. See Spack's -:ref:`packaging-guide` for more complete documentation on this topic. - ---------------- -Getting Started ---------------- - -A few things before we get started: - -- We'll refer to the Spack installation location via the environment - variable ``SPACK_ROOT``. You should point ``SPACK_ROOT`` at wherever - you have Spack installed. -- Add ``$SPACK_ROOT/bin`` to your ``PATH`` before you start. -- Make sure your ``EDITOR`` environment variable is set to some text - editor you like. -- We'll be writting Python code as part of this tutorial. You can find - successive versions of the Python code in - ``$SPACK_ROOT/lib/spack/docs/packaging_tutorial``. - -------------------------- -Creating the Package File -------------------------- - -Spack comes with a handy command to create a new package: ``spack create`` - -This command is given the location of a package's source code, downloads -the code, and sets up some basic packaging infrastructure for you. The -mpileaks source code can be found on github, and here's what happens when -we run ``spack create`` on it: - -.. code-block:: console - - % spack create -f https://github.com/hpc/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz - ==> This looks like a URL for mpileaks version 1.0 - ==> Creating template for package mpileaks - ==> Downloading... - ==> Fetching https://github.com/hpc/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz - ###################################################################################### 100.0% - -And Spack should spawn a text editor with this file: - -.. literalinclude:: packaging_tutorial/0.package.py - :start-after: # flake8: noqa - :language: python - -Spack has created this file in -``$SPACK_ROOT/var/spack/repos/builtin/packages/mpileaks/package.py``. Take a -moment to look over the file. There's a few placeholders that Spack has -created, which we'll fill in as part of this tutorial: - -- We'll document some information about this package in the comments. -- We'll fill in the dependency list for this package. -- We'll fill in some of the configuration arguments needed to build this - package. - -For the moment, exit your editor and let's see what happens when we try -to build this package: - -.. code-block:: console - - % spack install mpileaks - ==> Installing mpileaks - ==> Using cached archive: /usr/workspace/wsa/legendre/spack/var/spack/cache/mpileaks/mpileaks-1.0.tar.gz - ==> Staging archive: /usr/workspace/wsa/legendre/spack/var/spack/stage/mpileaks-1.0-hufwhwpq5benv3sslie6ryflk5s6nm35/mpileaks-1.0.tar.gz - ==> Created stage in /usr/workspace/wsa/legendre/spack/var/spack/stage/mpileaks-1.0-hufwhwpq5benv3sslie6ryflk5s6nm35 - ==> Ran patch() for mpileaks - ==> Building mpileaks [AutotoolsPackage] - ==> Executing phase : 'autoreconf' - ==> Executing phase : 'configure' - ==> Error: ProcessError: Command exited with status 1: - './configure' '--prefix=/usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/mpileaks-1.0-hufwhwpq5benv3sslie6ryflk5s6nm35' - /usr/workspace/wsa/legendre/spack/lib/spack/spack/build_systems/autotools.py:150, in configure: - 145 def configure(self, spec, prefix): - 146 """Runs configure with the arguments specified in `configure_args` - 147 and an appropriately set prefix - 148 """ - 149 options = ['--prefix={0}'.format(prefix)] + self.configure_args() - >> 150 inspect.getmodule(self).configure(*options) - - See build log for details: - /tmp/legendre/spack-stage/spack-stage-8HVzqu/mpileaks-1.0/spack-build.out - -This obviously didn't work; we need to fill in the package-specific -information. Specifically, Spack didn't try to build any of mpileaks' -dependencies, nor did it use the proper configure arguments. Let's start -fixing things - ---------------------- -Package Documentation ---------------------- - -We can bring the ``package.py`` file back into our ``EDITOR`` with the -``spack edit`` command: - -.. code-block:: console - - % spack edit mpileaks - -Let's remove some of the ``TODO`` comments, and add links to the mpileaks -homepage and document what mpileaks does. I'm also going to cut out the -Copyright clause at this point to keep this tutorial document shorter, -but you shouldn't do that normally. The results of these changes can be -found in ``$SPACK_ROOT/lib/spack/docs/packaging_tutorial/1.package.py`` -and are below. Make these changes to your ``package.py``: - -.. literalinclude:: packaging_tutorial/1.package.py - :start-after: # flake8: noqa - :language: python - -We've filled in the comment that describes what this package does and -added a link to the web site. That won't help us build yet, but it will -allow Spack to provide some documentation on this package to other users: - -.. code-block:: console - - % spack info mpileaks - AutotoolsPackage: mpileaks - Homepage: https://github.com/hpc/mpileaks - - Safe versions: - 1.0 https://github.com/hpc/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz - - Variants: - None - - Installation Phases: - autoreconf configure build install - - Build Dependencies: - None - - Link Dependencies: - None - - Run Dependencies: - None - - Virtual Packages: - None - - Description: - Tool to detect and report MPI objects like MPI_Requests and - MPI_Datatypes - -As we fill in more information about this package the spack info command -will become more informative. Now let's start making this package build. - ------------- -Dependencies ------------- - -The mpileaks packages depends on three other package: ``MPI``, -``adept-utils``, and ``callpath``. Let's add those via the -``depends_on`` command in our ``package.py`` (this version is in -``$SPACK_ROOT/lib/spack/docs/packaging_tutorial/2.package.py``): - -.. literalinclude:: packaging_tutorial/2.package.py - :start-after: # flake8: noqa - :language: python - -Now when we go to build mpileaks, Spack will fetch and build these -dependencies before building mpileaks. Note that the mpi dependency is a -different kind of beast than the adept-utils and callpath dependencies; -there is no mpi package available in Spack. Instead mpi is a virtual -dependency. Spack may satisfy that dependency by installing packages -such as ``openmpi`` or ``mvapich``. See the :ref:`packaging-guide` for more -information on virtual dependencies. - -Now when we try to install this package a lot more happens: - -.. code-block:: console - - % spack install mpileaks - ==> Installing mpileaks - ==> openmpi is already installed in /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/openmpi-2.0.1-5ee5j34c2y4kb5c3joygrgahidqnwhnz - ==> callpath is already installed in /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/callpath-1.0.2-zm4pf3gasgxeibyu2y262suktvaazube - ==> adept-utils is already installed in /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/adept-utils-1.0.1-7p7ezxwtajdglj6cmojy2vybjct4j4jz - ==> Using cached archive: /usr/workspace/wsa/legendre/spack/var/spack/cache/mpileaks/mpileaks-1.0.tar.gz - ==> Already staged mpileaks-1.0-eum4hmnlt6ovalwjnciaygfb3beja4gk in /usr/workspace/wsa/legendre/spack/var/spack/stage/mpileaks-1.0-eum4hmnlt6ovalwjnciaygfb3beja4gk - ==> Already patched mpileaks - ==> Building mpileaks [AutotoolsPackage] - ==> Executing phase : 'autoreconf' - ==> Executing phase : 'configure' - ==> Error: ProcessError: Command exited with status 1: - './configure' '--prefix=/usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/mpileaks-1.0-eum4hmnlt6ovalwjnciaygfb3beja4gk' - /usr/workspace/wsa/legendre/spack/lib/spack/spack/build_systems/autotools.py:150, in configure: - 145 def configure(self, spec, prefix): - 146 """Runs configure with the arguments specified in `configure_args` - 147 and an appropriately set prefix - 148 """ - 149 options = ['--prefix={0}'.format(prefix)] + self.configure_args() - >> 150 inspect.getmodule(self).configure(*options) - - See build log for details: - /tmp/legendre/spack-stage/spack-stage-7V5yyk/mpileaks-1.0/spack-build.out - -Note that this command may take a while to run and produce more output if -you don't have an MPI already installed or configured in Spack. - -Now Spack has identified and made sure all of our dependencies have been -built. It found the ``openmpi`` package that will satisfy our ``mpi`` -dependency, and the callpath and ``adept-utils`` package to satisfy our -concrete dependencies. - ------------------------- -Debugging Package Builds ------------------------- - -Our ``mpileaks`` package is still not building. It may be obvious to -many of you that we're still missing the configure options. But let's -pretend we're not all intelligent developers and use this opportunity -spend some time debugging. We a few options that can tell us about -what's going wrong: - -As per the error message, Spack has given us a ``spack-build.out`` debug log: - -.. code-block:: console - - ==> './configure' '--prefix=/usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/mpileaks-1.0-eum4hmnlt6ovalwjnciaygfb3beja4gk' - checking metadata... no - checking installation directory variables... yes - checking for a BSD-compatible install... /usr/bin/install -c - checking whether build environment is sane... yes - checking for a thread-safe mkdir -p... /usr/bin/mkdir -p - checking for gawk... gawk - checking whether make sets $(MAKE)... yes - checking for gcc... /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc - checking for C compiler default output file name... a.out - checking whether the C compiler works... yes - checking whether we are cross compiling... no - checking for suffix of executables... - checking for suffix of object files... o - checking whether we are using the GNU C compiler... yes - checking whether /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc accepts -g... yes - checking for /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc option to accept ISO C89... none needed - checking for style of include used by make... GNU - checking dependency style of /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc... gcc3 - checking whether /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc and cc understand -c and -o together... yes - checking whether we are using the GNU C++ compiler... yes - checking whether /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/g++ accepts -g... yes - checking dependency style of /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/g++... gcc3 - checking for /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/openmpi-2.0.1-5ee5j34c2y4kb5c3joygrgahidqnwhnz/bin/mpicc... /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/openmpi-2.0.1-5ee5j34c2y4kb5c3joygrgahidqnwhnz/bin/mpicc - Checking whether /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/openmpi-2.0.1-5ee5j34c2y4kb5c3joygrgahidqnwhnz/bin/mpicc responds to '-showme:compile'... yes - configure: error: unable to locate ``adept-utils`` installation - -This gives us the output from the build, and it's fairly obvious that -mpileaks isn't finding it's ``adept-utils`` package. Spack has -automatically added the include and library directories of -``adept-utils`` to the compiler's search path, but some packages like -mpileaks can sometimes be picky and still want things spelled out on -their command line. But let's continue to pretend we're not brilliant -developers, and explore some other debugging paths: - -We can also enter the build area and try to manually run the build: - -.. code-block:: console - - % spack env mpileaks tcsh - % spack cd mpileaks - -The ``spack env`` command spawned a new shell that contains the same -environment that Spack used to build the mpileaks package (you can -substitute tcsh for your favorite shell). The ``spack cd`` command -changed our working dirctory to the last attempted build for mpileaks. -From here we can manually re-run the build: - -.. code-block:: console - - % ./configure - checking metadata... no - checking installation directory variables... yes - checking for a BSD-compatible install... /usr/bin/install -c - checking whether build environment is sane... yes - checking for a thread-safe mkdir -p... /usr/bin/mkdir -p - checking for gawk... gawk - checking whether make sets $(MAKE)... yes - checking for gcc... /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc - checking for C compiler default output file name... a.out - checking whether the C compiler works... yes - checking whether we are cross compiling... no - checking for suffix of executables... - checking for suffix of object files... o - checking whether we are using the GNU C compiler... yes - checking whether /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc accepts -g... yes - checking for /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc option to accept ISO C89... none needed - checking for style of include used by make... GNU - checking dependency style of /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc... gcc3 - checking whether /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc and cc understand -c and -o together... yes - checking whether we are using the GNU C++ compiler... yes - checking whether /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/g++ accepts -g... yes - checking dependency style of /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/g++... gcc3 - checking for /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/openmpi-2.0.1-5ee5j34c2y4kb5c3joygrgahidqnwhnz/bin/mpicc... /usr/workspace/wsa /legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/openmpi-2.0.1-5ee5j34c2y4kb5c3joygrgahidqnwhnz/bin/mpicc - Checking whether /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/openmpi-2.0.1-5ee5j34c2y4kb5c3joygrgahidqnwhnz/bin/mpicc responds to '-showme:compile'... yes - configure: error: unable to locate adept-utils installation - -We're seeing the same error, but now we're in a shell where we can run -the command ourselves and debug as needed. We could, for example, run -``./configure --help`` to see what options we can use to specify -dependencies. - -We can use the ``exit`` command to leave the shell spawned by ``spack -env``. - ------------------------------- -Specifying Configure Arguments ------------------------------- - -Let's add the configure arguments to the mpileaks' ``package.py``. This -version can be found in -``$SPACK_ROOT/lib/spack/docs/packaging_tutorial/3.package.py``: - -.. literalinclude:: packaging_tutorial/3.package.py - :start-after: # flake8: noqa - :language: python - -This is all we need for working mpileaks! If we install now we'll see: - -.. code-block:: console - - % spack install mpileaks - spack install mpileaks - ==> Installing mpileaks - ==> openmpi is already installed in /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/openmpi-2.0.1-5ee5j34c2y4kb5c3joygrgahidqnwhnz - ==> callpath is already installed in /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/callpath-1.0.2-zm4pf3gasgxeibyu2y262suktvaazube - ==> adept-utils is already installed in /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/adept-utils-1.0.1-7p7ezxwtajdglj6cmojy2vybjct4j4jz - ==> Using cached archive: /usr/workspace/wsa/legendre/spack/var/spack/cache/mpileaks/mpileaks-1.0.tar.gz - ==> Already staged mpileaks-1.0-eum4hmnlt6ovalwjnciaygfb3beja4gk in /usr/workspace/wsa/legendre/spack/var/spack/stage/mpileaks-1.0-eum4hmnlt6ovalwjnciaygfb3beja4gk - ==> Already patched mpileaks - ==> Building mpileaks [AutotoolsPackage] - ==> Executing phase : 'autoreconf' - ==> Executing phase : 'configure' - ==> Executing phase : 'build' - ==> Executing phase : 'install' - ==> Successfully installed mpileaks - Fetch: 0.00s. Build: 14.08s. Total: 14.08s. - [+] /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/mpileaks-1.0-eum4hmnlt6ovalwjnciaygfb3beja4gk - -We took a few shortcuts for this package that are worth highlighting. -Spack automatically detected that mpileaks was an autotools-based package -when we ran spack create. If this had been a CMake-based package we -would have been filling in a ``cmake_args`` function instead of -``configure_args``. If spack hadn't been able to detect the build -system, we'd be filling in a generic install method that would manually -be calling build commands, such as is found in the ``zlib`` package: - -.. code-block:: python - - def install(self, spec, prefix): - configure('--prefix={0}'.format(prefix)) - make() - make('install') - --------- -Variants --------- - -We have a successful mpileaks build, but let's take some time to improve -it. ``mpileaks`` has a build-time option to truncate parts of the stack -that it walks. Let's add a variant to allow users to set this when they -build in Spack. - -To do this, we'll add a variant to our package, as per the following (see -``$SPACK_ROOT/lib/spack/docs/packaging_tutorial/4.package.py``): - -.. literalinclude:: packaging_tutorial/4.package.py - :start-after: # flake8: noqa - :language: python - -We've added the variant ``stackstart``, and given it a default value of -``0``. If we install now we can see the stackstart variant added to the -configure line (output truncated for length): - -.. code-block:: console - - % spack install --verbose mpileaks stackstart=4 - ==> Installing mpileaks - ==> openmpi is already installed in /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/openmpi-2.0.1-5ee5j34c2y4kb5c3joygrgahidqnwhnz - ==> callpath is already installed in /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/callpath-1.0.2-zm4pf3gasgxeibyu2y262suktvaazube - ==> adept-utils is already installed in /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/adept-utils-1.0.1-7p7ezxwtajdglj6cmojy2vybjct4j4jz - ==> Using cached archive: /usr/workspace/wsa/legendre/spack/var/spack/cache/mpileaks/mpileaks-1.0.tar.gz - ==> Staging archive: /usr/workspace/wsa/legendre/spack/var/spack/stage/mpileaks-1.0-otqo2opkhan5ksujt6tpmdftydrieig7/mpileaks-1.0.tar.gz - ==> Created stage in /usr/workspace/wsa/legendre/spack/var/spack/stage/mpileaks-1.0-otqo2opkhan5ksujt6tpmdftydrieig7 - ==> Ran patch() for mpileaks - ==> Building mpileaks [AutotoolsPackage] - ==> Executing phase : 'autoreconf' - ==> Executing phase : 'configure' - ==> './configure' '--prefix=/usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/mpileaks-1.0-otqo2opkhan5ksujt6tpmdftydrieig7' '--with-adept-utils=/usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/adept-utils-1.0.1-7p7ezxwtajdglj6cmojy2vybjct4j4jz' '--with-callpath=/usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/callpath-1.0.2-zm4pf3gasgxeibyu2y262suktvaazube' '--with-stack-start-c=4' '--with-stack-start-fortran=4' - ---------------- -The Spec Object ---------------- - -This tutorial has glossed over a few important features, which weren't -too relevant for mpleaks but may be useful for other packages. There -were several places we references the ``self.spec`` object. This is a -powerful class for querying information about what we're building. For -example, you could use the spec to query information about how a -package's dependencies were built, or what compiler was being used, or -what version of a package is being installed. Full documentation can be -found in the :ref:`packaging-guide`, but here's some quick snippets with -common queries: - -- Am I building ``mpileaks`` version ``1.1`` or greater? - -.. code-block:: python - - if self.spec.satisfies('@1.1:'): - # Do things needed for 1.1+ - -- Is ``openmpi`` the MPI I'm building with? - -.. code-block:: python - - if self.spec['mpi'].name == 'openmpi': - # Do openmpi things - -- Am I building with ``gcc`` version less than ``5.0.0``: - -.. code-block:: python - - if self.spec.satisfies('%gcc@:5.0.0'): - # Add arguments specific to gcc's earlier than 5.0.0 - -- Am I built with the ``debug`` variant: - -.. code-block:: python - - if self.spec.satisfies('+debug'): - # Add -g option to configure flags - -- Is my ``dyninst`` dependency greater than version ``8.0``? - -.. code-block:: python - - if self.spec['dyninst'].satisfies('@8.0:'): - # Use newest dyninst options - -More examples can be found in the thousands of packages already added to -Spack in ``$SPACK_ROOT/var/spack/repos/builtin/packages``. - -Good Luck! diff --git a/lib/spack/docs/packaging_tutorial/0.package.py b/lib/spack/docs/packaging_tutorial/0.package.py deleted file mode 100644 index ad6fd4d077..0000000000 --- a/lib/spack/docs/packaging_tutorial/0.package.py +++ /dev/null @@ -1,64 +0,0 @@ -# flake8: noqa -############################################################################## -# Copyright (c) 2013-2016, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory. -# -# This file is part of Spack. -# Created by Todd Gamblin, tgamblin@llnl.gov, All rights reserved. -# LLNL-CODE-647188 -# -# For details, see https://github.com/llnl/spack -# Please also see the LICENSE file for our notice and the LGPL. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU Lesser General Public License (as -# published by the Free Software Foundation) version 2.1, February 1999. -# -# This program is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the IMPLIED WARRANTY OF -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the terms and -# conditions of the GNU Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this program; if not, write to the Free Software -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA -############################################################################## -# -# This is a template package file for Spack. We've put "FIXME" -# next to all the things you'll want to change. Once you've handled -# them, you can save this file and test your package like this: -# -# spack install mpileaks -# -# You can edit this file again by typing: -# -# spack edit mpileaks -# -# See the Spack documentation for more information on packaging. -# If you submit this package back to Spack as a pull request, -# please first remove this boilerplate and all FIXME comments. -# -from spack import * - - -class Mpileaks(AutotoolsPackage): - """FIXME: Put a proper description of your package here.""" - - # FIXME: Add a proper url for your package's homepage here. - homepage = "http://www.example.com" - url = "https://github.com/hpc/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz" - - version('1.0', '8838c574b39202a57d7c2d68692718aa') - - # FIXME: Add dependencies if required. - # depends_on('m4', type='build') - # depends_on('autoconf', type='build') - # depends_on('automake', type='build') - # depends_on('libtool', type='build') - # depends_on('foo') - - def configure_args(self): - # FIXME: Add arguments other than --prefix - # FIXME: If not needed delete the function - args = [] - return args diff --git a/lib/spack/docs/packaging_tutorial/1.package.py b/lib/spack/docs/packaging_tutorial/1.package.py deleted file mode 100644 index afacf3bbe8..0000000000 --- a/lib/spack/docs/packaging_tutorial/1.package.py +++ /dev/null @@ -1,49 +0,0 @@ -############################################################################## -# Copyright (c) 2013-2016, Lawrence Livermore National Security, LLC. -# Produced at the Lawrence Livermore National Laboratory. -# -# This file is part of Spack. -# Created by Todd Gamblin, tgamblin@llnl.gov, All rights reserved. -# LLNL-CODE-647188 -# -# For details, see https://github.com/llnl/spack -# Please also see the LICENSE file for our notice and the LGPL. -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU Lesser General Public License (as -# published by the Free Software Foundation) version 2.1, February 1999. -# -# This program is distributed in the hope that it will be useful, but -# WITHOUT ANY WARRANTY; without even the IMPLIED WARRANTY OF -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the terms and -# conditions of the GNU Lesser General Public License for more details. -# -# You should have received a copy of the GNU Lesser General Public -# License along with this program; if not, write to the Free Software -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA -############################################################################## -# flake8: noqa - -from spack import * - - -class Mpileaks(AutotoolsPackage): - """Tool to detect and report MPI objects like MPI_Requests and MPI_Datatypes""" - - homepage = "https://github.com/hpc/mpileaks" - url = "https://github.com/hpc/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz" - - version('1.0', '8838c574b39202a57d7c2d68692718aa') - - # FIXME: Add dependencies if required. - # depends_on('m4', type='build') - # depends_on('autoconf', type='build') - # depends_on('automake', type='build') - # depends_on('libtool', type='build') - # depends_on('foo') - - def configure_args(self): - # FIXME: Add arguments other than --prefix - # FIXME: If not needed delete the function - args = [] - return args diff --git a/lib/spack/docs/packaging_tutorial/2.package.py b/lib/spack/docs/packaging_tutorial/2.package.py deleted file mode 100644 index f7000cd1b6..0000000000 --- a/lib/spack/docs/packaging_tutorial/2.package.py +++ /dev/null @@ -1,21 +0,0 @@ -# flake8: noqa - -from spack import * - -class Mpileaks(AutotoolsPackage): - """Tool to detect and report MPI objects like MPI_Requests and MPI_Datatypes""" - - homepage = "https://github.com/hpc/mpileaks" - url = "https://github.com/hpc/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz" - - version('1.0', '8838c574b39202a57d7c2d68692718aa') - - depends_on('mpi') - depends_on('adept-utils') - depends_on('callpath') - - def configure_args(self): - # FIXME: Add arguments other than --prefix - # FIXME: If not needed delete the function - args = [] - return args diff --git a/lib/spack/docs/packaging_tutorial/3.package.py b/lib/spack/docs/packaging_tutorial/3.package.py deleted file mode 100644 index 6e7e5e2f3c..0000000000 --- a/lib/spack/docs/packaging_tutorial/3.package.py +++ /dev/null @@ -1,17 +0,0 @@ -# flake8: noqa - -class Mpileaks(AutotoolsPackage): - """Tool to detect and report MPI objects like MPI_Requests and MPI_Datatypes""" - homepage = "https://github.com/hpc/mpileaks" - url = "https://github.com/hpc/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz" - - version('1.0', '8838c574b39202a57d7c2d68692718aa') - - depends_on('mpi') - depends_on('adept-utils') - depends_on('callpath') - - def configure_args(self): - args = ['--with-adept-utils=%s' % self.spec['adept-utils'].prefix, - '--with-callpath=%s' % self.spec['callpath'].prefix] - return args diff --git a/lib/spack/docs/packaging_tutorial/4.package.py b/lib/spack/docs/packaging_tutorial/4.package.py deleted file mode 100644 index 3294b86e25..0000000000 --- a/lib/spack/docs/packaging_tutorial/4.package.py +++ /dev/null @@ -1,26 +0,0 @@ -# flake8: noqa - -from spack import * - -class Mpileaks(AutotoolsPackage): - """Tool to detect and report MPI objects like MPI_Requests and MPI_Datatypes""" - - homepage = "https://github.com/hpc/mpileaks" - url = "https://github.com/hpc/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz" - - version('1.0', '8838c574b39202a57d7c2d68692718aa') - - variant('stackstart', default=0, description='Specify the number of stack frames to truncate.') - - depends_on('mpi') - depends_on('adept-utils') - depends_on('callpath') - - def configure_args(self): - args = ['--with-adept-utils=%s' % self.spec['adept-utils'].prefix, - '--with-callpath=%s' % self.spec['callpath'].prefix] - stackstart = int(self.spec.variants['stackstart'].value) - if stackstart: - args.extend(['--with-stack-start-c=%s' % stackstart, - '--with-stack-start-fortran=%s' % stackstart]) - return args diff --git a/lib/spack/docs/tutorial/.gitattributes b/lib/spack/docs/tutorial/.gitattributes new file mode 100644 index 0000000000..71905cea7d --- /dev/null +++ b/lib/spack/docs/tutorial/.gitattributes @@ -0,0 +1 @@ +Spack-SC16-Tutorial.pdf filter=lfs diff=lfs merge=lfs -text diff --git a/lib/spack/docs/tutorial/Spack-SC16-Tutorial.pdf b/lib/spack/docs/tutorial/Spack-SC16-Tutorial.pdf new file mode 100644 index 0000000000..12dd4fe06f Binary files /dev/null and b/lib/spack/docs/tutorial/Spack-SC16-Tutorial.pdf differ diff --git a/lib/spack/docs/tutorial/examples/0.package.py b/lib/spack/docs/tutorial/examples/0.package.py new file mode 100644 index 0000000000..ad6fd4d077 --- /dev/null +++ b/lib/spack/docs/tutorial/examples/0.package.py @@ -0,0 +1,64 @@ +# flake8: noqa +############################################################################## +# Copyright (c) 2013-2016, Lawrence Livermore National Security, LLC. +# Produced at the Lawrence Livermore National Laboratory. +# +# This file is part of Spack. +# Created by Todd Gamblin, tgamblin@llnl.gov, All rights reserved. +# LLNL-CODE-647188 +# +# For details, see https://github.com/llnl/spack +# Please also see the LICENSE file for our notice and the LGPL. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU Lesser General Public License (as +# published by the Free Software Foundation) version 2.1, February 1999. +# +# This program is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the IMPLIED WARRANTY OF +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the terms and +# conditions of the GNU Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +############################################################################## +# +# This is a template package file for Spack. We've put "FIXME" +# next to all the things you'll want to change. Once you've handled +# them, you can save this file and test your package like this: +# +# spack install mpileaks +# +# You can edit this file again by typing: +# +# spack edit mpileaks +# +# See the Spack documentation for more information on packaging. +# If you submit this package back to Spack as a pull request, +# please first remove this boilerplate and all FIXME comments. +# +from spack import * + + +class Mpileaks(AutotoolsPackage): + """FIXME: Put a proper description of your package here.""" + + # FIXME: Add a proper url for your package's homepage here. + homepage = "http://www.example.com" + url = "https://github.com/hpc/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz" + + version('1.0', '8838c574b39202a57d7c2d68692718aa') + + # FIXME: Add dependencies if required. + # depends_on('m4', type='build') + # depends_on('autoconf', type='build') + # depends_on('automake', type='build') + # depends_on('libtool', type='build') + # depends_on('foo') + + def configure_args(self): + # FIXME: Add arguments other than --prefix + # FIXME: If not needed delete the function + args = [] + return args diff --git a/lib/spack/docs/tutorial/examples/1.package.py b/lib/spack/docs/tutorial/examples/1.package.py new file mode 100644 index 0000000000..afacf3bbe8 --- /dev/null +++ b/lib/spack/docs/tutorial/examples/1.package.py @@ -0,0 +1,49 @@ +############################################################################## +# Copyright (c) 2013-2016, Lawrence Livermore National Security, LLC. +# Produced at the Lawrence Livermore National Laboratory. +# +# This file is part of Spack. +# Created by Todd Gamblin, tgamblin@llnl.gov, All rights reserved. +# LLNL-CODE-647188 +# +# For details, see https://github.com/llnl/spack +# Please also see the LICENSE file for our notice and the LGPL. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU Lesser General Public License (as +# published by the Free Software Foundation) version 2.1, February 1999. +# +# This program is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the IMPLIED WARRANTY OF +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the terms and +# conditions of the GNU Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +############################################################################## +# flake8: noqa + +from spack import * + + +class Mpileaks(AutotoolsPackage): + """Tool to detect and report MPI objects like MPI_Requests and MPI_Datatypes""" + + homepage = "https://github.com/hpc/mpileaks" + url = "https://github.com/hpc/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz" + + version('1.0', '8838c574b39202a57d7c2d68692718aa') + + # FIXME: Add dependencies if required. + # depends_on('m4', type='build') + # depends_on('autoconf', type='build') + # depends_on('automake', type='build') + # depends_on('libtool', type='build') + # depends_on('foo') + + def configure_args(self): + # FIXME: Add arguments other than --prefix + # FIXME: If not needed delete the function + args = [] + return args diff --git a/lib/spack/docs/tutorial/examples/2.package.py b/lib/spack/docs/tutorial/examples/2.package.py new file mode 100644 index 0000000000..f7000cd1b6 --- /dev/null +++ b/lib/spack/docs/tutorial/examples/2.package.py @@ -0,0 +1,21 @@ +# flake8: noqa + +from spack import * + +class Mpileaks(AutotoolsPackage): + """Tool to detect and report MPI objects like MPI_Requests and MPI_Datatypes""" + + homepage = "https://github.com/hpc/mpileaks" + url = "https://github.com/hpc/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz" + + version('1.0', '8838c574b39202a57d7c2d68692718aa') + + depends_on('mpi') + depends_on('adept-utils') + depends_on('callpath') + + def configure_args(self): + # FIXME: Add arguments other than --prefix + # FIXME: If not needed delete the function + args = [] + return args diff --git a/lib/spack/docs/tutorial/examples/3.package.py b/lib/spack/docs/tutorial/examples/3.package.py new file mode 100644 index 0000000000..6e7e5e2f3c --- /dev/null +++ b/lib/spack/docs/tutorial/examples/3.package.py @@ -0,0 +1,17 @@ +# flake8: noqa + +class Mpileaks(AutotoolsPackage): + """Tool to detect and report MPI objects like MPI_Requests and MPI_Datatypes""" + homepage = "https://github.com/hpc/mpileaks" + url = "https://github.com/hpc/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz" + + version('1.0', '8838c574b39202a57d7c2d68692718aa') + + depends_on('mpi') + depends_on('adept-utils') + depends_on('callpath') + + def configure_args(self): + args = ['--with-adept-utils=%s' % self.spec['adept-utils'].prefix, + '--with-callpath=%s' % self.spec['callpath'].prefix] + return args diff --git a/lib/spack/docs/tutorial/examples/4.package.py b/lib/spack/docs/tutorial/examples/4.package.py new file mode 100644 index 0000000000..3294b86e25 --- /dev/null +++ b/lib/spack/docs/tutorial/examples/4.package.py @@ -0,0 +1,26 @@ +# flake8: noqa + +from spack import * + +class Mpileaks(AutotoolsPackage): + """Tool to detect and report MPI objects like MPI_Requests and MPI_Datatypes""" + + homepage = "https://github.com/hpc/mpileaks" + url = "https://github.com/hpc/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz" + + version('1.0', '8838c574b39202a57d7c2d68692718aa') + + variant('stackstart', default=0, description='Specify the number of stack frames to truncate.') + + depends_on('mpi') + depends_on('adept-utils') + depends_on('callpath') + + def configure_args(self): + args = ['--with-adept-utils=%s' % self.spec['adept-utils'].prefix, + '--with-callpath=%s' % self.spec['callpath'].prefix] + stackstart = int(self.spec.variants['stackstart'].value) + if stackstart: + args.extend(['--with-stack-start-c=%s' % stackstart, + '--with-stack-start-fortran=%s' % stackstart]) + return args diff --git a/lib/spack/docs/tutorial/sc16-tutorial-slide-preview.png b/lib/spack/docs/tutorial/sc16-tutorial-slide-preview.png new file mode 100644 index 0000000000..e7f9b2e323 Binary files /dev/null and b/lib/spack/docs/tutorial/sc16-tutorial-slide-preview.png differ diff --git a/lib/spack/docs/tutorial_sc16.rst b/lib/spack/docs/tutorial_sc16.rst new file mode 100644 index 0000000000..e8cf3aeebe --- /dev/null +++ b/lib/spack/docs/tutorial_sc16.rst @@ -0,0 +1,48 @@ +.. _spack-101: + +============================= +Spack 101 +============================= + +This is a 3-hour introduction to Spack with lectures and live demos. It +was presented as a tutorial at `Supercomputing 2016 +`_. You can use these materials to teach +a course on Spack at your own site, or you can just skip ahead and read +the live demo scripts to see how Spack is used in practice. + +.. _sc16-slides: + +.. rubric:: Slides + +.. figure:: tutorial/sc16-tutorial-slide-preview.png + :target: _downloads/Spack-SC16-Tutorial.pdf + :height: 72px + :align: left + :alt: Slide Preview + +:download:`Download Slides `. + +**Full citation:** Todd Gamblin, Massimiliano Culpo, Gregory Becker, Matt +Legendre, Greg Lee, Elizabeth Fischer, and Benedikt Hegner. +`Managing HPC Software Complexity with Spack +`_. +Tutorial presented at Supercomputing 2016. November 13, 2016, Salt Lake +City, UT, USA. + +.. _sc16-live-demos: + +.. rubric:: Live Demos + +These scripts will take you step-by-step through basic Spack tasks. They +correspond to sections in the slides above. + + 1. :ref:`basics-tutorial` + 2. :ref:`packaging-tutorial` + 3. :ref:`modules-tutorial` + +Full contents: + +.. toctree:: + tutorial_sc16_spack_basics + tutorial_sc16_packaging + tutorial_sc16_modules diff --git a/lib/spack/docs/tutorial_sc16_modules.rst b/lib/spack/docs/tutorial_sc16_modules.rst new file mode 100644 index 0000000000..1e786674bb --- /dev/null +++ b/lib/spack/docs/tutorial_sc16_modules.rst @@ -0,0 +1,982 @@ +.. _modules-tutorial: + +============================= +Module Configuration Tutorial +============================= + +This tutorial will guide you through the customization of both +content and naming of module files generated by Spack. + +Starting from the default Spack settings you will add an increasing +number of directives to the ``modules.yaml`` configuration file to +satisfy a number of constraints that mimic those that you may encounter +in a typical production environment at HPC sites. + +Even though the focus will be for the most part on customizing +TCL non-hierarchical module files, everything +you'll see applies also to other kinds of module files generated by Spack. + +The generation of Lua hierarchical +module files will be addressed at the end of the tutorial, +and you'll see that with minor modifications +to an existing ``modules.yaml`` written for TCL +non-hierarchical modules you'll get almost +for free the possibility to try a hierarchical layout. + +Let's start! + +.. _module_file_tutorial_prerequisites: + +------------- +Prerequisites +------------- + +Before proceeding further ensure: + +- you have LMod or Environment Modules available +- have :ref:`shell support ` activated in Spack + +If you need to install Lmod or Environment module you can refer +to the documentation :ref:`here `. + + +^^^^^^^^^^^^^^^^^^ +Add a new compiler +^^^^^^^^^^^^^^^^^^ + +Spack automatically scans the environment to search for available +compilers on first use. On a Ubuntu 14.04 a fresh clone will show +something like this: + +.. code-block:: console + + $ uname -a + Linux nuvolari 4.4.0-45-generic #66~14.04.1-Ubuntu SMP Wed Oct 19 15:05:38 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux + + $ spack compilers + ==> Available compilers + -- gcc ---------------------------------------------------------- + gcc@4.8 + +For the purpose of building a limited set of packages with some features +that will help showcasing the capabilities of +module customization the first thing we need is to build a new compiler: + +.. code-block:: console + + $ spack install gcc@6.2.0 + # ... + # Wait a long time + # ... + +Then we can use shell support for modules to add it to the list of known compilers: + +.. code-block:: console + + # The name of the generated module may vary + $ module load gcc-6.2.0-gcc-4.8-twd5nqg + + $ spack compiler add + ==> Added 1 new compiler to /home/mculpo/.spack/linux/compilers.yaml + gcc@6.2.0 + + $ spack compilers + ==> Available compilers + -- gcc ---------------------------------------------------------- + gcc@6.2.0 gcc@4.8 + +Note that the final 7 digits hash at the end of the generated module may vary depending +on architecture or package version. + +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Build software that will be used in the tutorial +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Next you should install a few modules that will be used in the tutorial: + +.. code-block:: console + + $ spack install netlib-scalapack ^openmpi ^openblas + # ... + +The packages you need to install are: + +- ``netlib-scalapack ^openmpi ^openblas`` +- ``netlib-scalapack ^mpich ^openblas`` +- ``netlib-scalapack ^openmpi ^netlib-lapack`` +- ``netlib-scalapack ^mpich ^netlib-lapack`` +- ``py-scipy ^openblas`` + +In the end your environment should look something like: + +.. code-block:: console + + $ module avail + + ------------------------------------------------------------------------ /home/mculpo/wdir/spack/share/spack/modules/linux-Ubuntu14-x86_64 ------------------------------------------------------------------------ + binutils-2.27-gcc-4.8-dz3xevw libpciaccess-0.13.4-gcc-6.2.0-eo2siet lzo-2.09-gcc-6.2.0-jcngz72 netlib-scalapack-2.0.2-gcc-6.2.0-wnimqhw python-2.7.12-gcc-6.2.0-qu7rc5p + bzip2-1.0.6-gcc-6.2.0-csoc2mq libsigsegv-2.10-gcc-4.8-avb6azw m4-1.4.17-gcc-4.8-iggewke netlib-scalapack-2.0.2-gcc-6.2.0-wojunhq sqlite-3.8.5-gcc-6.2.0-td3zfe7 + cmake-3.5.2-gcc-6.2.0-6poypqg libsigsegv-2.10-gcc-6.2.0-g3qpmbi m4-1.4.17-gcc-6.2.0-lhgqa6s nettle-3.2-gcc-6.2.0-djdthlh tcl-8.6.5-gcc-4.8-atddxu7 + curl-7.50.3-gcc-6.2.0-2ffacqm libtool-2.4.6-gcc-6.2.0-kiepac6 mpc-1.0.3-gcc-4.8-lylv7lk openblas-0.2.19-gcc-6.2.0-js33umc util-macros-1.19.0-gcc-6.2.0-uoukuqk + expat-2.2.0-gcc-6.2.0-bxqnjar libxml2-2.9.4-gcc-6.2.0-3k4ykbe mpfr-3.1.4-gcc-4.8-bldfx3w openmpi-2.0.1-gcc-6.2.0-s3qbtby xz-5.2.2-gcc-6.2.0-t5lk6in + gcc-6.2.0-gcc-4.8-twd5nqg lmod-6.4.5-gcc-4.8-7v7bh7b mpich-3.2-gcc-6.2.0-5n5xoep openssl-1.0.2j-gcc-6.2.0-hibnfda zlib-1.2.8-gcc-4.8-bds4ies + gmp-6.1.1-gcc-4.8-uq52e2n lua-5.3.2-gcc-4.8-xozf2hx ncurses-6.0-gcc-4.8-u62fit4 pkg-config-0.29.1-gcc-6.2.0-rslsgcs zlib-1.2.8-gcc-6.2.0-asydrba + gmp-6.1.1-gcc-6.2.0-3cfh3hi lua-luafilesystem-1_6_3-gcc-4.8-sbzejlz ncurses-6.0-gcc-6.2.0-7tb426s py-nose-1.3.7-gcc-6.2.0-4gl5c42 + hwloc-1.11.4-gcc-6.2.0-3ostwel lua-luaposix-33.4.0-gcc-4.8-xf7y2p5 netlib-lapack-3.6.1-gcc-6.2.0-mirer2l py-numpy-1.11.1-gcc-6.2.0-i3rpk4e + isl-0.14-gcc-4.8-cq73t5m lz4-131-gcc-6.2.0-cagoem4 netlib-scalapack-2.0.2-gcc-6.2.0-6bqlxqy py-scipy-0.18.1-gcc-6.2.0-e6uljfi + libarchive-3.2.1-gcc-6.2.0-2b54aos lzma-4.32.7-gcc-6.2.0-sfmeynw netlib-scalapack-2.0.2-gcc-6.2.0-hpqb3dp py-setuptools-25.2.0-gcc-6.2.0-hkqauaa + +------------------------------------------------ +Filter unwanted modifications to the environment +------------------------------------------------ + +The non-hierarchical TCL module files that have been generated so far +follow the default rules for module generation, which are given +:ref:`here ` in the reference part of the manual. Taking a +look at the ``gcc`` module you'll see something like: + +.. code-block:: console + + $ module show gcc-6.2.0-gcc-4.8-twd5nqg + --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + /home/mculpo/wdir/spack/share/spack/modules/linux-Ubuntu14-x86_64/gcc-6.2.0-gcc-4.8-twd5nqg: + --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + whatis("gcc @6.2.0 ") + prepend_path("PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/bin") + prepend_path("CMAKE_PREFIX_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/") + prepend_path("MANPATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/share/man") + prepend_path("PKG_CONFIG_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/lib64/pkgconfig") + prepend_path("LIBRARY_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/lib64") + prepend_path("LD_LIBRARY_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/lib64") + prepend_path("CPATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/include") + help([[The GNU Compiler Collection includes front ends for C, C++, Objective-C, + Fortran, and Java. + ]]) + +As expected, a few environment variables representing paths will be modified +by the modules according to the default prefix inspection rules. + +Consider now the case that your site has decided that e.g. ``CPATH`` and +``LIBRARY_PATH`` modifications should not be present in module files. What you can +do to abide by the rules is to create a configuration file ``~/.spack/modules.yaml`` +with the following content: + +.. code-block:: yaml + + modules: + tcl: + all: + filter: + environment_blacklist: ['CPATH', 'LIBRARY_PATH'] + +Next you should regenerate all the module files: + +.. code-block:: console + + $ spack module refresh --module-type tcl + ==> You are about to regenerate tcl module files for: + + -- linux-Ubuntu14-x86_64 / gcc@4.8 ------------------------------ + dz3xevw binutils@2.27 uq52e2n gmp@6.1.1 avb6azw libsigsegv@2.10 xozf2hx lua@5.3.2 xf7y2p5 lua-luaposix@33.4.0 lylv7lk mpc@1.0.3 u62fit4 ncurses@6.0 bds4ies zlib@1.2.8 + twd5nqg gcc@6.2.0 cq73t5m isl@0.14 7v7bh7b lmod@6.4.5 sbzejlz lua-luafilesystem@1_6_3 iggewke m4@1.4.17 bldfx3w mpfr@3.1.4 atddxu7 tcl@8.6.5 + + ... + + ==> Do you want to proceed ? [y/n] + y + ==> Regenerating tcl module files + +If you take a look now at the module for ``gcc`` you'll see that the unwanted +paths have disappeared: + +.. code-block:: console + + $ module show gcc-6.2.0-gcc-4.8-twd5nqg + --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + /home/mculpo/wdir/spack/share/spack/modules/linux-Ubuntu14-x86_64/gcc-6.2.0-gcc-4.8-twd5nqg: + --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + whatis("gcc @6.2.0 ") + prepend_path("PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/bin") + prepend_path("CMAKE_PREFIX_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/") + prepend_path("MANPATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/share/man") + prepend_path("PKG_CONFIG_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/lib64/pkgconfig") + prepend_path("LD_LIBRARY_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/lib64") + help([[The GNU Compiler Collection includes front ends for C, C++, Objective-C, + Fortran, and Java. + ]]) + +---------------------------------------------- +Prevent some module files from being generated +---------------------------------------------- + +Another common request at many sites is to avoid exposing software that +is only needed as an intermediate step when building a newer stack. +Let's try to prevent the generation of +module files for anything that is compiled with ``gcc@4.8`` (the OS provided compiler). + +To do this you should add a ``blacklist`` keyword to the configuration file: + +.. code-block:: yaml + :emphasize-lines: 3,4 + + modules: + tcl: + blacklist: + - '%gcc@4.8' + all: + filter: + environment_blacklist: ['CPATH', 'LIBRARY_PATH'] + +and regenerate the module files: + +.. code-block:: console + + $ spack module refresh --module-type tcl --delete-tree + ==> You are about to regenerate tcl module files for: + + -- linux-Ubuntu14-x86_64 / gcc@4.8 ------------------------------ + dz3xevw binutils@2.27 uq52e2n gmp@6.1.1 avb6azw libsigsegv@2.10 xozf2hx lua@5.3.2 xf7y2p5 lua-luaposix@33.4.0 lylv7lk mpc@1.0.3 u62fit4 ncurses@6.0 bds4ies zlib@1.2.8 + twd5nqg gcc@6.2.0 cq73t5m isl@0.14 7v7bh7b lmod@6.4.5 sbzejlz lua-luafilesystem@1_6_3 iggewke m4@1.4.17 bldfx3w mpfr@3.1.4 atddxu7 tcl@8.6.5 + + -- linux-Ubuntu14-x86_64 / gcc@6.2.0 ---------------------------- + csoc2mq bzip2@1.0.6 2b54aos libarchive@3.2.1 sfmeynw lzma@4.32.7 wnimqhw netlib-scalapack@2.0.2 s3qbtby openmpi@2.0.1 hkqauaa py-setuptools@25.2.0 + 6poypqg cmake@3.5.2 eo2siet libpciaccess@0.13.4 jcngz72 lzo@2.09 6bqlxqy netlib-scalapack@2.0.2 hibnfda openssl@1.0.2j qu7rc5p python@2.7.12 + 2ffacqm curl@7.50.3 g3qpmbi libsigsegv@2.10 lhgqa6s m4@1.4.17 wojunhq netlib-scalapack@2.0.2 rslsgcs pkg-config@0.29.1 td3zfe7 sqlite@3.8.5 + bxqnjar expat@2.2.0 kiepac6 libtool@2.4.6 5n5xoep mpich@3.2 hpqb3dp netlib-scalapack@2.0.2 4gl5c42 py-nose@1.3.7 uoukuqk util-macros@1.19.0 + 3cfh3hi gmp@6.1.1 3k4ykbe libxml2@2.9.4 7tb426s ncurses@6.0 djdthlh nettle@3.2 i3rpk4e py-numpy@1.11.1 t5lk6in xz@5.2.2 + 3ostwel hwloc@1.11.4 cagoem4 lz4@131 mirer2l netlib-lapack@3.6.1 js33umc openblas@0.2.19 e6uljfi py-scipy@0.18.1 asydrba zlib@1.2.8 + + ==> Do you want to proceed ? [y/n] + y + + $ module avail + + ------------------------------------------------------------------------ /home/mculpo/wdir/spack/share/spack/modules/linux-Ubuntu14-x86_64 ------------------------------------------------------------------------ + bzip2-1.0.6-gcc-6.2.0-csoc2mq libsigsegv-2.10-gcc-6.2.0-g3qpmbi ncurses-6.0-gcc-6.2.0-7tb426s openmpi-2.0.1-gcc-6.2.0-s3qbtby sqlite-3.8.5-gcc-6.2.0-td3zfe7 + cmake-3.5.2-gcc-6.2.0-6poypqg libtool-2.4.6-gcc-6.2.0-kiepac6 netlib-lapack-3.6.1-gcc-6.2.0-mirer2l openssl-1.0.2j-gcc-6.2.0-hibnfda util-macros-1.19.0-gcc-6.2.0-uoukuqk + curl-7.50.3-gcc-6.2.0-2ffacqm libxml2-2.9.4-gcc-6.2.0-3k4ykbe netlib-scalapack-2.0.2-gcc-6.2.0-6bqlxqy pkg-config-0.29.1-gcc-6.2.0-rslsgcs xz-5.2.2-gcc-6.2.0-t5lk6in + expat-2.2.0-gcc-6.2.0-bxqnjar lz4-131-gcc-6.2.0-cagoem4 netlib-scalapack-2.0.2-gcc-6.2.0-hpqb3dp py-nose-1.3.7-gcc-6.2.0-4gl5c42 zlib-1.2.8-gcc-6.2.0-asydrba + gmp-6.1.1-gcc-6.2.0-3cfh3hi lzma-4.32.7-gcc-6.2.0-sfmeynw netlib-scalapack-2.0.2-gcc-6.2.0-wnimqhw py-numpy-1.11.1-gcc-6.2.0-i3rpk4e + hwloc-1.11.4-gcc-6.2.0-3ostwel lzo-2.09-gcc-6.2.0-jcngz72 netlib-scalapack-2.0.2-gcc-6.2.0-wojunhq py-scipy-0.18.1-gcc-6.2.0-e6uljfi + libarchive-3.2.1-gcc-6.2.0-2b54aos m4-1.4.17-gcc-6.2.0-lhgqa6s nettle-3.2-gcc-6.2.0-djdthlh py-setuptools-25.2.0-gcc-6.2.0-hkqauaa + libpciaccess-0.13.4-gcc-6.2.0-eo2siet mpich-3.2-gcc-6.2.0-5n5xoep openblas-0.2.19-gcc-6.2.0-js33umc python-2.7.12-gcc-6.2.0-qu7rc5p + +This time it is convenient to pass the option ``--delete-tree`` to the command that +regenerates the module files to instruct it to delete the existing tree and regenerate +a new one instead of overwriting the files in the existing directory. + +If you pay careful attention you'll see though that we went too far in blacklisting modules: +the module for ``gcc@6.2.0`` disappeared as it was bootstrapped with ``gcc@4.8``. To specify +exceptions to the blacklist rules you can use ``whitelist``: + +.. code-block:: yaml + :emphasize-lines: 3,4 + + modules: + tcl: + whitelist: + - gcc + blacklist: + - '%gcc@4.8' + all: + filter: + environment_blacklist: ['CPATH', 'LIBRARY_PATH'] + +``whitelist`` rules always have precedence over ``blacklist`` rules. If you regenerate the modules again: + +.. code-block:: console + + $ spack module refresh --module-type tcl -y + +you'll see that now the module for ``gcc@6.2.0`` has reappeared: + +.. code-block:: console + + $ module avail gcc-6.2.0-gcc-4.8-twd5nqg + + ------------------------------------------------------------------------ /home/mculpo/wdir/spack/share/spack/modules/linux-Ubuntu14-x86_64 ------------------------------------------------------------------------ + gcc-6.2.0-gcc-4.8-twd5nqg + +------------------------- +Change module file naming +------------------------- + +The next step in making module files more user-friendly is to +improve their naming scheme. +To reduce the length of the hash or remove it altogether you can +use the ``hash_length`` keyword in the configuration file: + +.. TODO: give reasons to remove hashes if they are not evident enough? + +.. code-block:: yaml + :emphasize-lines: 3 + + modules: + tcl: + hash_length: 0 + whitelist: + - gcc + blacklist: + - '%gcc@4.8' + all: + filter: + environment_blacklist: ['CPATH', 'LIBRARY_PATH'] + +If you try to regenerate the module files now you will get an error: + +.. code-block:: console + + $ spack module refresh --module-type tcl --delete-tree -y + ==> Error: Name clashes detected in module files: + + file : /home/mculpo/wdir/spack/share/spack/modules/linux-Ubuntu14-x86_64/netlib-scalapack-2.0.2-gcc-6.2.0 + spec : netlib-scalapack@2.0.2%gcc@6.2.0~fpic+shared arch=linux-Ubuntu14-x86_64 + spec : netlib-scalapack@2.0.2%gcc@6.2.0~fpic+shared arch=linux-Ubuntu14-x86_64 + spec : netlib-scalapack@2.0.2%gcc@6.2.0~fpic+shared arch=linux-Ubuntu14-x86_64 + spec : netlib-scalapack@2.0.2%gcc@6.2.0~fpic+shared arch=linux-Ubuntu14-x86_64 + + ==> Error: Operation aborted + +.. note:: + We try to check for errors upfront! + In Spack we check for errors upfront whenever possible, so don't worry about your module files: + as a name clash was detected nothing has been changed on disk. + +The problem here is that without +the hashes the four different flavors of ``netlib-scalapack`` map to the same module file +name. We have the possibility to add suffixes to differentiate them: + +.. code-block:: yaml + :emphasize-lines: 9-11,14-17 + + modules: + tcl: + hash_length: 0 + whitelist: + - gcc + blacklist: + - '%gcc@4.8' + all: + suffixes: + '^openblas': openblas + '^netlib-lapack': netlib + filter: + environment_blacklist: ['CPATH', 'LIBRARY_PATH'] + netlib-scalapack: + suffixes: + '^openmpi': openmpi + '^mpich': mpich + +As you can see it is possible to specify rules that applies only to a +restricted set of packages using :ref:`anonymous specs `. +Regenerating module files now we obtain: + +.. code-block:: console + + $ spack module refresh --module-type tcl --delete-tree -y + ==> Regenerating tcl module files + $ module avail + + ------------------------------------------------------------------------ /home/mculpo/wdir/spack/share/spack/modules/linux-Ubuntu14-x86_64 ------------------------------------------------------------------------ + bzip2-1.0.6-gcc-6.2.0 libpciaccess-0.13.4-gcc-6.2.0 mpich-3.2-gcc-6.2.0 openblas-0.2.19-gcc-6.2.0 python-2.7.12-gcc-6.2.0 + cmake-3.5.2-gcc-6.2.0 libsigsegv-2.10-gcc-6.2.0 ncurses-6.0-gcc-6.2.0 openmpi-2.0.1-gcc-6.2.0 sqlite-3.8.5-gcc-6.2.0 + curl-7.50.3-gcc-6.2.0 libtool-2.4.6-gcc-6.2.0 netlib-lapack-3.6.1-gcc-6.2.0 openssl-1.0.2j-gcc-6.2.0 util-macros-1.19.0-gcc-6.2.0 + expat-2.2.0-gcc-6.2.0 libxml2-2.9.4-gcc-6.2.0 netlib-scalapack-2.0.2-gcc-6.2.0-netlib-mpich pkg-config-0.29.1-gcc-6.2.0 xz-5.2.2-gcc-6.2.0 + gcc-6.2.0-gcc-4.8 lz4-131-gcc-6.2.0 netlib-scalapack-2.0.2-gcc-6.2.0-netlib-openmpi py-nose-1.3.7-gcc-6.2.0 zlib-1.2.8-gcc-6.2.0 + gmp-6.1.1-gcc-6.2.0 lzma-4.32.7-gcc-6.2.0 netlib-scalapack-2.0.2-gcc-6.2.0-openblas-mpich py-numpy-1.11.1-gcc-6.2.0-openblas + hwloc-1.11.4-gcc-6.2.0 lzo-2.09-gcc-6.2.0 netlib-scalapack-2.0.2-gcc-6.2.0-openblas-openmpi py-scipy-0.18.1-gcc-6.2.0-openblas + libarchive-3.2.1-gcc-6.2.0 m4-1.4.17-gcc-6.2.0 nettle-3.2-gcc-6.2.0 py-setuptools-25.2.0-gcc-6.2.0 + +Finally we can set a ``naming_scheme`` to prevent users from loading +modules that refer to different flavors of the same library/application: + +.. code-block:: yaml + :emphasize-lines: 4,10,11 + + modules: + tcl: + hash_length: 0 + naming_scheme: '${PACKAGE}/${VERSION}-${COMPILERNAME}-${COMPILERVER}' + whitelist: + - gcc + blacklist: + - '%gcc@4.8' + all: + conflict: + - '${PACKAGE}' + suffixes: + '^openblas': openblas + '^netlib-lapack': netlib + filter: + environment_blacklist: ['CPATH', 'LIBRARY_PATH'] + netlib-scalapack: + suffixes: + '^openmpi': openmpi + '^mpich': mpich + +The final result should look like: + +.. code-block:: console + + $ module avail + + ------------------------------------------------------------------------ /home/mculpo/wdir/spack/share/spack/modules/linux-Ubuntu14-x86_64 ------------------------------------------------------------------------ + bzip2/1.0.6-gcc-6.2.0 libpciaccess/0.13.4-gcc-6.2.0 mpich/3.2-gcc-6.2.0 openblas/0.2.19-gcc-6.2.0 python/2.7.12-gcc-6.2.0 + cmake/3.5.2-gcc-6.2.0 libsigsegv/2.10-gcc-6.2.0 ncurses/6.0-gcc-6.2.0 openmpi/2.0.1-gcc-6.2.0 sqlite/3.8.5-gcc-6.2.0 + curl/7.50.3-gcc-6.2.0 libtool/2.4.6-gcc-6.2.0 netlib-lapack/3.6.1-gcc-6.2.0 openssl/1.0.2j-gcc-6.2.0 util-macros/1.19.0-gcc-6.2.0 + expat/2.2.0-gcc-6.2.0 libxml2/2.9.4-gcc-6.2.0 netlib-scalapack/2.0.2-gcc-6.2.0-netlib-mpich pkg-config/0.29.1-gcc-6.2.0 xz/5.2.2-gcc-6.2.0 + gcc/6.2.0-gcc-4.8 lz4/131-gcc-6.2.0 netlib-scalapack/2.0.2-gcc-6.2.0-netlib-openmpi py-nose/1.3.7-gcc-6.2.0 zlib/1.2.8-gcc-6.2.0 + gmp/6.1.1-gcc-6.2.0 lzma/4.32.7-gcc-6.2.0 netlib-scalapack/2.0.2-gcc-6.2.0-openblas-mpich py-numpy/1.11.1-gcc-6.2.0-openblas + hwloc/1.11.4-gcc-6.2.0 lzo/2.09-gcc-6.2.0 netlib-scalapack/2.0.2-gcc-6.2.0-openblas-openmpi (D) py-scipy/0.18.1-gcc-6.2.0-openblas + libarchive/3.2.1-gcc-6.2.0 m4/1.4.17-gcc-6.2.0 nettle/3.2-gcc-6.2.0 py-setuptools/25.2.0-gcc-6.2.0 + +.. note:: + TCL specific directive + The directives ``naming_scheme`` and ``conflict`` are TCL specific and do not apply + to the ``dotkit`` or ``lmod`` sections in the configuration file. + +------------------------------------ +Add custom environment modifications +------------------------------------ + +At many sites it is customary to set an environment variable in a +package's module file that points to the folder in which the package +is installed. You can achieve this with Spack by adding an +``environment`` directive to the configuration file: + +.. code-block:: yaml + :emphasize-lines: 17-19 + + modules: + tcl: + hash_length: 0 + naming_scheme: '${PACKAGE}/${VERSION}-${COMPILERNAME}-${COMPILERVER}' + whitelist: + - gcc + blacklist: + - '%gcc@4.8' + all: + conflict: + - '${PACKAGE}' + suffixes: + '^openblas': openblas + '^netlib-lapack': netlib + filter: + environment_blacklist: ['CPATH', 'LIBRARY_PATH'] + environment: + set: + '${PACKAGE}_ROOT': '${PREFIX}' + netlib-scalapack: + suffixes: + '^openmpi': openmpi + '^mpich': mpich + +There are many variable tokens available to use in the ``environment`` +and ``naming_scheme`` directives, such as ``${PACKAGE}``, +``${VERSION}``, etc. (see the :meth:`~spack.spec.Spec.format` API +documentation for the complete list). + +Regenerating the module files should result in something like: + +.. code-block:: console + :emphasize-lines: 14 + + $ spack module refresh -y --module-type tcl + ==> Regenerating tcl module files + + $ module show gcc + --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + /home/mculpo/wdir/spack/share/spack/modules/linux-Ubuntu14-x86_64/gcc/6.2.0-gcc-4.8: + --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + whatis("gcc @6.2.0 ") + prepend_path("PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/bin") + prepend_path("CMAKE_PREFIX_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/") + prepend_path("MANPATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/share/man") + prepend_path("PKG_CONFIG_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/lib64/pkgconfig") + prepend_path("LD_LIBRARY_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/lib64") + setenv("GCC_ROOT","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u") + conflict("gcc") + help([[The GNU Compiler Collection includes front ends for C, C++, Objective-C, + Fortran, and Java. + ]]) + +As you see the ``gcc`` module has the environment variable ``GCC_ROOT`` set. + +Sometimes it's also useful to apply environment modifications selectively and target +only certain packages. You can, for instance set the common variables ``CC``, ``CXX``, +etc. in the ``gcc`` module file and apply other custom modifications to the +``openmpi`` modules as follows: + +.. code-block:: yaml + :emphasize-lines: 20-32 + + modules: + tcl: + hash_length: 0 + naming_scheme: '${PACKAGE}/${VERSION}-${COMPILERNAME}-${COMPILERVER}' + whitelist: + - gcc + blacklist: + - '%gcc@4.8' + all: + conflict: + - '${PACKAGE}' + suffixes: + '^openblas': openblas + '^netlib-lapack': netlib + filter: + environment_blacklist: ['CPATH', 'LIBRARY_PATH'] + environment: + set: + '${PACKAGE}_ROOT': '${PREFIX}' + gcc: + environment: + set: + CC: gcc + CXX: g++ + FC: gfortran + F90: gfortran + F77: gfortran + openmpi: + environment: + set: + SLURM_MPI_TYPE: pmi2 + OMPI_MCA_btl_openib_warn_default_gid_prefix: '0' + netlib-scalapack: + suffixes: + '^openmpi': openmpi + '^mpich': mpich + +This time we will be more selective and regenerate only the ``gcc`` and +``openmpi`` module files: + +.. code-block:: console + + $ spack module refresh -y --module-type tcl gcc + ==> Regenerating tcl module files + + $ spack module refresh -y --module-type tcl openmpi + ==> Regenerating tcl module files + + $ module show gcc + --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + /home/mculpo/wdir/spack/share/spack/modules/linux-Ubuntu14-x86_64/gcc/6.2.0-gcc-4.8: + --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + whatis("gcc @6.2.0 ") + prepend_path("PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/bin") + prepend_path("CMAKE_PREFIX_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/") + prepend_path("MANPATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/share/man") + prepend_path("PKG_CONFIG_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/lib64/pkgconfig") + prepend_path("LD_LIBRARY_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u/lib64") + setenv("GCC_ROOT","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-4.8/gcc-6.2.0-twd5nqg33hrrssqclcfi5k42eccwxz5u") + setenv("CC","gcc") + setenv("CXX","g++") + setenv("F90","gfortran") + setenv("FC","gfortran") + setenv("F77","gfortran") + conflict("gcc") + help([[The GNU Compiler Collection includes front ends for C, C++, Objective-C, + Fortran, and Java. + ]]) + + $ module show openmpi + --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + /home/mculpo/wdir/spack/share/spack/modules/linux-Ubuntu14-x86_64/openmpi/2.0.1-gcc-6.2.0: + --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + whatis("openmpi @2.0.1 ") + prepend_path("PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-6.2.0/openmpi-2.0.1-s3qbtbyh3y5y4gkchmhcuak7th44l53w/bin") + prepend_path("CMAKE_PREFIX_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-6.2.0/openmpi-2.0.1-s3qbtbyh3y5y4gkchmhcuak7th44l53w/") + prepend_path("LD_LIBRARY_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-6.2.0/openmpi-2.0.1-s3qbtbyh3y5y4gkchmhcuak7th44l53w/lib") + prepend_path("PKG_CONFIG_PATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-6.2.0/openmpi-2.0.1-s3qbtbyh3y5y4gkchmhcuak7th44l53w/lib/pkgconfig") + prepend_path("MANPATH","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-6.2.0/openmpi-2.0.1-s3qbtbyh3y5y4gkchmhcuak7th44l53w/share/man") + setenv("SLURM_MPI_TYPE","pmi2") + setenv("OMPI_MCA_BTL_OPENIB_WARN_DEFAULT_GID_PREFIX","0") + setenv("OPENMPI_ROOT","/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-6.2.0/openmpi-2.0.1-s3qbtbyh3y5y4gkchmhcuak7th44l53w") + conflict("openmpi") + help([[The Open MPI Project is an open source Message Passing Interface + implementation that is developed and maintained by a consortium of + academic, research, and industry partners. Open MPI is therefore able to + combine the expertise, technologies, and resources from all across the + High Performance Computing community in order to build the best MPI + library available. Open MPI offers advantages for system and software + vendors, application developers and computer science researchers. + ]]) + + +--------------------- +Autoload dependencies +--------------------- + +Spack can also generate module files that contain code to load the +dependencies automatically. You can, for instance generate python +modules that load their dependencies by adding the ``autoload`` +directive and assigning it the value ``direct``: + +.. code-block:: yaml + :emphasize-lines: 37,38 + + modules: + tcl: + hash_length: 0 + naming_scheme: '${PACKAGE}/${VERSION}-${COMPILERNAME}-${COMPILERVER}' + whitelist: + - gcc + blacklist: + - '%gcc@4.8' + all: + conflict: + - '${PACKAGE}' + suffixes: + '^openblas': openblas + '^netlib-lapack': netlib + filter: + environment_blacklist: ['CPATH', 'LIBRARY_PATH'] + environment: + set: + '${PACKAGE}_ROOT': '${PREFIX}' + gcc: + environment: + set: + CC: gcc + CXX: g++ + FC: gfortran + F90: gfortran + F77: gfortran + openmpi: + environment: + set: + SLURM_MPI_TYPE: pmi2 + OMPI_MCA_btl_openib_warn_default_gid_prefix: '0' + netlib-scalapack: + suffixes: + '^openmpi': openmpi + '^mpich': mpich + ^python: + autoload: 'direct' + +and regenerating the module files for every package that depends on ``python``: + +.. code-block:: console + + $ spack module refresh -y --module-type tcl ^python + ==> Regenerating tcl module files + +Now the ``py-scipy`` module will be: + +.. code-block:: tcl + + #%Module1.0 + ## Module file created by spack (https://github.com/LLNL/spack) on 2016-11-02 20:53:21.283547 + ## + ## py-scipy@0.18.1%gcc@6.2.0 arch=linux-Ubuntu14-x86_64-e6uljfi + ## + module-whatis "py-scipy @0.18.1" + + proc ModulesHelp { } { + puts stderr "SciPy (pronounced "Sigh Pie") is a Scientific Library for Python. It" + puts stderr "provides many user-friendly and efficient numerical routines such as" + puts stderr "routines for numerical integration and optimization." + } + + if ![ is-loaded python/2.7.12-gcc-6.2.0 ] { + puts stderr "Autoloading python/2.7.12-gcc-6.2.0" + module load python/2.7.12-gcc-6.2.0 + } + + if ![ is-loaded openblas/0.2.19-gcc-6.2.0 ] { + puts stderr "Autoloading openblas/0.2.19-gcc-6.2.0" + module load openblas/0.2.19-gcc-6.2.0 + } + + if ![ is-loaded py-numpy/1.11.1-gcc-6.2.0-openblas ] { + puts stderr "Autoloading py-numpy/1.11.1-gcc-6.2.0-openblas" + module load py-numpy/1.11.1-gcc-6.2.0-openblas + } + + prepend-path CMAKE_PREFIX_PATH "/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-6.2.0/py-scipy-0.18.1-e6uljfiffgym4xvj6wveevqxfqnfb3gh/" + prepend-path LD_LIBRARY_PATH "/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-6.2.0/py-scipy-0.18.1-e6uljfiffgym4xvj6wveevqxfqnfb3gh/lib" + prepend-path PYTHONPATH "/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-6.2.0/py-scipy-0.18.1-e6uljfiffgym4xvj6wveevqxfqnfb3gh/lib/python2.7/site-packages" + setenv PY_SCIPY_ROOT "/home/mculpo/wdir/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-6.2.0/py-scipy-0.18.1-e6uljfiffgym4xvj6wveevqxfqnfb3gh" + conflict py-scipy + +and will contain code to autoload all the dependencies: + +.. code-block:: console + + $ module load py-scipy + Autoloading python/2.7.12-gcc-6.2.0 + Autoloading openblas/0.2.19-gcc-6.2.0 + Autoloading py-numpy/1.11.1-gcc-6.2.0-openblas + +----------------------------- +Lua hierarchical module files +----------------------------- + +In the final part of this tutorial you will modify ``modules.yaml`` to generate +Lua hierarchical module files. You will see that most of the directives used before +are also valid in the ``lmod`` context. + +^^^^^^^^^^^^^^^^^ +Core/Compiler/MPI +^^^^^^^^^^^^^^^^^ + +.. warning:: + Only LMod supports Lua hierarchical module files + For this part of the tutorial you need to be using LMod to + manage your environment. + +The most common hierarchy is the so called ``Core/Compiler/MPI``. To have an idea +how a hierarchy is organized you may refer to the +`Lmod guide `_. +Since ``lmod`` is not enabled by default, you need to add it to the list of +enabled module file generators. The other things you need to do are: + +- change the ``tcl`` tag to ``lmod`` +- remove ``tcl`` specific directives (``naming_scheme`` and ``conflict``) +- set which compilers are considered ``core`` +- remove the ``mpi`` related suffixes (as they will be substituted by hierarchies) + +After modifications the configuration file will be: + +.. code-block:: yaml + :emphasize-lines: 2-6 + + modules: + enable:: + - lmod + lmod: + core_compilers: + - 'gcc@4.8' + hash_length: 0 + whitelist: + - gcc + blacklist: + - '%gcc@4.8' + all: + suffixes: + '^openblas': openblas + '^netlib-lapack': netlib + filter: + environment_blacklist: ['CPATH', 'LIBRARY_PATH'] + environment: + set: + '${PACKAGE}_ROOT': '${PREFIX}' + gcc: + environment: + set: + CC: gcc + CXX: g++ + FC: gfortran + F90: gfortran + F77: gfortran + openmpi: + environment: + set: + SLURM_MPI_TYPE: pmi2 + OMPI_MCA_btl_openib_warn_default_gid_prefix: '0' + + +.. note:: + The double colon + The double colon after ``enable`` is intentional and it serves the + purpose of overriding the default list of enabled generators so + that only ``lmod`` will be active (see :ref:`the reference + manual ` for a more detailed explanation of + config scopes). + +The directive ``core_compilers`` accepts a list of compilers : everything built +using these compilers will create a module in the ``Core`` part of the hierarchy. It is +common practice to put the OS provided compilers in the list and only build common utilities +and other compilers in ``Core``. + +If you regenerate the module files + +.. code-block:: console + + $ spack module refresh --module-type lmod --delete-tree -y + +and update ``MODULEPATH`` to point to the ``Core`` folder, and +list the available modules, you'll see: + +.. code-block:: console + + $ module unuse /home/mculpo/wdir/spack/share/spack/modules/linux-Ubuntu14-x86_64 + $ module use /home/mculpo/wdir/spack/share/spack/lmod/linux-Ubuntu14-x86_64/Core + $ module avail + + ----------------------------------------------------------------------- /home/mculpo/wdir/spack/share/spack/lmod/linux-Ubuntu14-x86_64/Core ----------------------------------------------------------------------- + gcc/6.2.0 + +The only module visible now is ``gcc``. Loading that you will make +visible the ``Compiler`` part of the software stack that was built with ``gcc/6.2.0``: + +.. code-block:: console + + $ module load gcc + $ module avail + + -------------------------------------------------------------------- /home/mculpo/wdir/spack/share/spack/lmod/linux-Ubuntu14-x86_64/gcc/6.2.0 --------------------------------------------------------------------- + binutils/2.27 curl/7.50.3 hwloc/1.11.4 libtool/2.4.6 lzo/2.09 netlib-lapack/3.6.1 openssl/1.0.2j py-scipy/0.18.1-openblas util-macros/1.19.0 + bison/3.0.4 expat/2.2.0 libarchive/3.2.1 libxml2/2.9.4 m4/1.4.17 nettle/3.2 pkg-config/0.29.1 py-setuptools/25.2.0 xz/5.2.2 + bzip2/1.0.6 flex/2.6.0 libpciaccess/0.13.4 lz4/131 mpich/3.2 openblas/0.2.19 py-nose/1.3.7 python/2.7.12 zlib/1.2.8 + cmake/3.6.1 gmp/6.1.1 libsigsegv/2.10 lzma/4.32.7 ncurses/6.0 openmpi/2.0.1 py-numpy/1.11.1-openblas sqlite/3.8.5 + + ----------------------------------------------------------------------- /home/mculpo/wdir/spack/share/spack/lmod/linux-Ubuntu14-x86_64/Core ----------------------------------------------------------------------- + gcc/6.2.0 (L) + +The same holds true for the ``MPI`` part of the stack, that you can enable by loading +either ``mpich`` or ``openmpi``. The nice features of LMod will become evident +once you'll try switching among different stacks: + +.. code-block:: console + + $ module load mpich + $ module avail + + ----------------------------------------------------------- /home/mculpo/wdir/spack/share/spack/lmod/linux-Ubuntu14-x86_64/mpich/3.2-5n5xoep/gcc/6.2.0 ------------------------------------------------------------ + netlib-scalapack/2.0.2-netlib netlib-scalapack/2.0.2-openblas (D) + + -------------------------------------------------------------------- /home/mculpo/wdir/spack/share/spack/lmod/linux-Ubuntu14-x86_64/gcc/6.2.0 --------------------------------------------------------------------- + binutils/2.27 curl/7.50.3 hwloc/1.11.4 libtool/2.4.6 lzo/2.09 netlib-lapack/3.6.1 openssl/1.0.2j py-scipy/0.18.1-openblas util-macros/1.19.0 + bison/3.0.4 expat/2.2.0 libarchive/3.2.1 libxml2/2.9.4 m4/1.4.17 nettle/3.2 pkg-config/0.29.1 py-setuptools/25.2.0 xz/5.2.2 + bzip2/1.0.6 flex/2.6.0 libpciaccess/0.13.4 lz4/131 mpich/3.2 (L) openblas/0.2.19 py-nose/1.3.7 python/2.7.12 zlib/1.2.8 + cmake/3.6.1 gmp/6.1.1 libsigsegv/2.10 lzma/4.32.7 ncurses/6.0 openmpi/2.0.1 py-numpy/1.11.1-openblas sqlite/3.8.5 + + ----------------------------------------------------------------------- /home/mculpo/wdir/spack/share/spack/lmod/linux-Ubuntu14-x86_64/Core ----------------------------------------------------------------------- + gcc/6.2.0 (L) + + $ module load openblas netlib-scalapack/2.0.2-openblas + $ module list + + Currently Loaded Modules: + 1) gcc/6.2.0 2) mpich/3.2 3) openblas/0.2.19 4) netlib-scalapack/2.0.2-openblas + + $ module load openmpi + + Lmod is automatically replacing "mpich/3.2" with "openmpi/2.0.1" + + + Due to MODULEPATH changes the following have been reloaded: + 1) netlib-scalapack/2.0.2-openblas + +This layout is already a great improvement over the usual non-hierarchical layout, +but it still has an asymmetry: ``LAPACK`` providers are semantically the same as ``MPI`` +providers, but they are still not part of the hierarchy. We'll see a possible solution +next. + +.. Activate lmod and turn the previous modifications into lmod: + Add core compilers + +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Extend the hierarchy to other virtual providers +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. warning:: + This is an experimental feature + Having a hierarchy deeper than ``Core``/``Compiler``/``MPI`` is an experimental + feature, still not fully supported by ``module spider``, + see `here `_. Furthermore its use + with hierarchies more complex than ``Core``/``Compiler``/``MPI``/``LAPACK`` + has not been thoroughly tested in production environments. + +Spack permits you to generate Lua hierarchical module files where users +can add an arbitrary list of virtual providers to the triplet +``Core``/``Compiler``/``MPI``. A configuration file like: + +.. code-block:: yaml + :emphasize-lines: 7,8 + + modules: + enable:: + - lmod + lmod: + core_compilers: + - 'gcc@4.8' + hierarchical_scheme: + - lapack + hash_length: 0 + whitelist: + - gcc + blacklist: + - '%gcc@4.8' + - readline + all: + filter: + environment_blacklist: ['CPATH', 'LIBRARY_PATH'] + environment: + set: + '${PACKAGE}_ROOT': '${PREFIX}' + gcc: + environment: + set: + CC: gcc + CXX: g++ + FC: gfortran + F90: gfortran + F77: gfortran + openmpi: + environment: + set: + SLURM_MPI_TYPE: pmi2 + OMPI_MCA_btl_openib_warn_default_gid_prefix: '0' + +will add ``lapack`` providers to the mix. After the usual regeneration of module files: + +.. code-block:: console + + $ module purge + $ spack module refresh --module-type lmod --delete-tree -y + ==> Regenerating lmod module files + +you will have something like: + +.. code-block:: console + + $ module load gcc + $ module load openblas + $ module load openmpi + $ module avail + + --------------------------------------------- /home/mculpo/wdir/spack/share/spack/lmod/linux-Ubuntu14-x86_64/openblas/0.2.19-js33umc/openmpi/2.0.1-s3qbtby/gcc/6.2.0 ---------------------------------------------- + netlib-scalapack/2.0.2 + + -------------------------------------------------------- /home/mculpo/wdir/spack/share/spack/lmod/linux-Ubuntu14-x86_64/openblas/0.2.19-js33umc/gcc/6.2.0 --------------------------------------------------------- + py-numpy/1.11.1 py-scipy/0.18.1 + + -------------------------------------------------------------------- /home/mculpo/wdir/spack/share/spack/lmod/linux-Ubuntu14-x86_64/gcc/6.2.0 --------------------------------------------------------------------- + binutils/2.27 curl/7.50.3 hwloc/1.11.4 libtool/2.4.6 lzo/2.09 netlib-lapack/3.6.1 openssl/1.0.2j python/2.7.12 zlib/1.2.8 + bison/3.0.4 expat/2.2.0 libarchive/3.2.1 libxml2/2.9.4 m4/1.4.17 nettle/3.2 pkg-config/0.29.1 sqlite/3.8.5 + bzip2/1.0.6 flex/2.6.0 libpciaccess/0.13.4 lz4/131 mpich/3.2 openblas/0.2.19 (L) py-nose/1.3.7 util-macros/1.19.0 + cmake/3.6.1 gmp/6.1.1 libsigsegv/2.10 lzma/4.32.7 ncurses/6.0 openmpi/2.0.1 (L) py-setuptools/25.2.0 xz/5.2.2 + + ----------------------------------------------------------------------- /home/mculpo/wdir/spack/share/spack/lmod/linux-Ubuntu14-x86_64/Core ----------------------------------------------------------------------- + gcc/6.2.0 (L) + +Now both the ``MPI`` and the ``LAPACK`` providers are handled by LMod as hierarchies: + +.. code-block:: console + + $ module load py-numpy netlib-scalapack + $ module load mpich + + Lmod is automatically replacing "openmpi/2.0.1" with "mpich/3.2" + + + Due to MODULEPATH changes the following have been reloaded: + 1) netlib-scalapack/2.0.2 + + $ module load netlib-lapack + + Lmod is automatically replacing "openblas/0.2.19" with "netlib-lapack/3.6.1" + + + Inactive Modules: + 1) py-numpy + + Due to MODULEPATH changes the following have been reloaded: + 1) netlib-scalapack/2.0.2 + +making the use of tags to differentiate them unnecessary. +Note that because we only compiled ``py-numpy`` with ``openblas`` the module +is made inactive when we switch the ``LAPACK`` provider. The user +environment will now be consistent by design! diff --git a/lib/spack/docs/tutorial_sc16_packaging.rst b/lib/spack/docs/tutorial_sc16_packaging.rst new file mode 100644 index 0000000000..41c938327e --- /dev/null +++ b/lib/spack/docs/tutorial_sc16_packaging.rst @@ -0,0 +1,463 @@ +.. _packaging-tutorial: + +========================= +Package Creation Tutorial +========================= + +This tutorial will walk you through the steps behind building a simple +package installation script. We'll focus building an mpileaks package, +which is a MPI debugging tool. By creating a package file we're +essentially giving Spack a recipe for how to build a particular piece of +software. We're describing some of the software's dependencies, where to +find the package, what commands and options are used to build the package +from source, and more. Once we've specified a package's recipe, we can +ask Spack to build that package in many different ways. + +This tutorial assumes you have a basic familiarity with some of the Spack +commands, and that you have a working version of Spack installed. If +not, we suggest looking at Spack's *Getting Started* guide. This +tutorial also assumes you have at least a beginner's-level familiarity +with Python. + +Also note that this document is a tutorial. It can help you get started +with packaging, but is not intended to be complete. See Spack's +:ref:`packaging-guide` for more complete documentation on this topic. + +--------------- +Getting Started +--------------- + +A few things before we get started: + +- We'll refer to the Spack installation location via the environment + variable ``SPACK_ROOT``. You should point ``SPACK_ROOT`` at wherever + you have Spack installed. +- Add ``$SPACK_ROOT/bin`` to your ``PATH`` before you start. +- Make sure your ``EDITOR`` environment variable is set to some text + editor you like. +- We'll be writting Python code as part of this tutorial. You can find + successive versions of the Python code in + ``$SPACK_ROOT/lib/spack/docs/tutorial/examples``. + +------------------------- +Creating the Package File +------------------------- + +Spack comes with a handy command to create a new package: ``spack create`` + +This command is given the location of a package's source code, downloads +the code, and sets up some basic packaging infrastructure for you. The +mpileaks source code can be found on github, and here's what happens when +we run ``spack create`` on it: + +.. code-block:: console + + % spack create -f https://github.com/hpc/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz + ==> This looks like a URL for mpileaks version 1.0 + ==> Creating template for package mpileaks + ==> Downloading... + ==> Fetching https://github.com/hpc/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz + ###################################################################################### 100.0% + +And Spack should spawn a text editor with this file: + +.. literalinclude:: tutorial/examples/0.package.py + :start-after: # flake8: noqa + :language: python + +Spack has created this file in +``$SPACK_ROOT/var/spack/repos/builtin/packages/mpileaks/package.py``. Take a +moment to look over the file. There's a few placeholders that Spack has +created, which we'll fill in as part of this tutorial: + +- We'll document some information about this package in the comments. +- We'll fill in the dependency list for this package. +- We'll fill in some of the configuration arguments needed to build this + package. + +For the moment, exit your editor and let's see what happens when we try +to build this package: + +.. code-block:: console + + % spack install mpileaks + ==> Installing mpileaks + ==> Using cached archive: /usr/workspace/wsa/legendre/spack/var/spack/cache/mpileaks/mpileaks-1.0.tar.gz + ==> Staging archive: /usr/workspace/wsa/legendre/spack/var/spack/stage/mpileaks-1.0-hufwhwpq5benv3sslie6ryflk5s6nm35/mpileaks-1.0.tar.gz + ==> Created stage in /usr/workspace/wsa/legendre/spack/var/spack/stage/mpileaks-1.0-hufwhwpq5benv3sslie6ryflk5s6nm35 + ==> Ran patch() for mpileaks + ==> Building mpileaks [AutotoolsPackage] + ==> Executing phase : 'autoreconf' + ==> Executing phase : 'configure' + ==> Error: ProcessError: Command exited with status 1: + './configure' '--prefix=/usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/mpileaks-1.0-hufwhwpq5benv3sslie6ryflk5s6nm35' + /usr/workspace/wsa/legendre/spack/lib/spack/spack/build_systems/autotools.py:150, in configure: + 145 def configure(self, spec, prefix): + 146 """Runs configure with the arguments specified in `configure_args` + 147 and an appropriately set prefix + 148 """ + 149 options = ['--prefix={0}'.format(prefix)] + self.configure_args() + >> 150 inspect.getmodule(self).configure(*options) + + See build log for details: + /tmp/legendre/spack-stage/spack-stage-8HVzqu/mpileaks-1.0/spack-build.out + +This obviously didn't work; we need to fill in the package-specific +information. Specifically, Spack didn't try to build any of mpileaks' +dependencies, nor did it use the proper configure arguments. Let's start +fixing things + +--------------------- +Package Documentation +--------------------- + +We can bring the ``package.py`` file back into our ``EDITOR`` with the +``spack edit`` command: + +.. code-block:: console + + % spack edit mpileaks + +Let's remove some of the ``TODO`` comments, and add links to the mpileaks +homepage and document what mpileaks does. I'm also going to cut out the +Copyright clause at this point to keep this tutorial document shorter, +but you shouldn't do that normally. The results of these changes can be +found in ``$SPACK_ROOT/lib/spack/docs/tutorial/examples/1.package.py`` +and are below. Make these changes to your ``package.py``: + +.. literalinclude:: tutorial/examples/1.package.py + :start-after: # flake8: noqa + :language: python + +We've filled in the comment that describes what this package does and +added a link to the web site. That won't help us build yet, but it will +allow Spack to provide some documentation on this package to other users: + +.. code-block:: console + + % spack info mpileaks + AutotoolsPackage: mpileaks + Homepage: https://github.com/hpc/mpileaks + + Safe versions: + 1.0 https://github.com/hpc/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz + + Variants: + None + + Installation Phases: + autoreconf configure build install + + Build Dependencies: + None + + Link Dependencies: + None + + Run Dependencies: + None + + Virtual Packages: + None + + Description: + Tool to detect and report MPI objects like MPI_Requests and + MPI_Datatypes + +As we fill in more information about this package the spack info command +will become more informative. Now let's start making this package build. + +------------ +Dependencies +------------ + +The mpileaks packages depends on three other package: ``MPI``, +``adept-utils``, and ``callpath``. Let's add those via the +``depends_on`` command in our ``package.py`` (this version is in +``$SPACK_ROOT/lib/spack/docs/tutorial/examples/2.package.py``): + +.. literalinclude:: tutorial/examples/2.package.py + :start-after: # flake8: noqa + :language: python + +Now when we go to build mpileaks, Spack will fetch and build these +dependencies before building mpileaks. Note that the mpi dependency is a +different kind of beast than the adept-utils and callpath dependencies; +there is no mpi package available in Spack. Instead mpi is a virtual +dependency. Spack may satisfy that dependency by installing packages +such as ``openmpi`` or ``mvapich``. See the :ref:`packaging-guide` for more +information on virtual dependencies. + +Now when we try to install this package a lot more happens: + +.. code-block:: console + + % spack install mpileaks + ==> Installing mpileaks + ==> openmpi is already installed in /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/openmpi-2.0.1-5ee5j34c2y4kb5c3joygrgahidqnwhnz + ==> callpath is already installed in /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/callpath-1.0.2-zm4pf3gasgxeibyu2y262suktvaazube + ==> adept-utils is already installed in /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/adept-utils-1.0.1-7p7ezxwtajdglj6cmojy2vybjct4j4jz + ==> Using cached archive: /usr/workspace/wsa/legendre/spack/var/spack/cache/mpileaks/mpileaks-1.0.tar.gz + ==> Already staged mpileaks-1.0-eum4hmnlt6ovalwjnciaygfb3beja4gk in /usr/workspace/wsa/legendre/spack/var/spack/stage/mpileaks-1.0-eum4hmnlt6ovalwjnciaygfb3beja4gk + ==> Already patched mpileaks + ==> Building mpileaks [AutotoolsPackage] + ==> Executing phase : 'autoreconf' + ==> Executing phase : 'configure' + ==> Error: ProcessError: Command exited with status 1: + './configure' '--prefix=/usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/mpileaks-1.0-eum4hmnlt6ovalwjnciaygfb3beja4gk' + /usr/workspace/wsa/legendre/spack/lib/spack/spack/build_systems/autotools.py:150, in configure: + 145 def configure(self, spec, prefix): + 146 """Runs configure with the arguments specified in `configure_args` + 147 and an appropriately set prefix + 148 """ + 149 options = ['--prefix={0}'.format(prefix)] + self.configure_args() + >> 150 inspect.getmodule(self).configure(*options) + + See build log for details: + /tmp/legendre/spack-stage/spack-stage-7V5yyk/mpileaks-1.0/spack-build.out + +Note that this command may take a while to run and produce more output if +you don't have an MPI already installed or configured in Spack. + +Now Spack has identified and made sure all of our dependencies have been +built. It found the ``openmpi`` package that will satisfy our ``mpi`` +dependency, and the callpath and ``adept-utils`` package to satisfy our +concrete dependencies. + +------------------------ +Debugging Package Builds +------------------------ + +Our ``mpileaks`` package is still not building. It may be obvious to +many of you that we're still missing the configure options. But let's +pretend we're not all intelligent developers and use this opportunity +spend some time debugging. We a few options that can tell us about +what's going wrong: + +As per the error message, Spack has given us a ``spack-build.out`` debug log: + +.. code-block:: console + + ==> './configure' '--prefix=/usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/mpileaks-1.0-eum4hmnlt6ovalwjnciaygfb3beja4gk' + checking metadata... no + checking installation directory variables... yes + checking for a BSD-compatible install... /usr/bin/install -c + checking whether build environment is sane... yes + checking for a thread-safe mkdir -p... /usr/bin/mkdir -p + checking for gawk... gawk + checking whether make sets $(MAKE)... yes + checking for gcc... /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc + checking for C compiler default output file name... a.out + checking whether the C compiler works... yes + checking whether we are cross compiling... no + checking for suffix of executables... + checking for suffix of object files... o + checking whether we are using the GNU C compiler... yes + checking whether /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc accepts -g... yes + checking for /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc option to accept ISO C89... none needed + checking for style of include used by make... GNU + checking dependency style of /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc... gcc3 + checking whether /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc and cc understand -c and -o together... yes + checking whether we are using the GNU C++ compiler... yes + checking whether /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/g++ accepts -g... yes + checking dependency style of /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/g++... gcc3 + checking for /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/openmpi-2.0.1-5ee5j34c2y4kb5c3joygrgahidqnwhnz/bin/mpicc... /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/openmpi-2.0.1-5ee5j34c2y4kb5c3joygrgahidqnwhnz/bin/mpicc + Checking whether /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/openmpi-2.0.1-5ee5j34c2y4kb5c3joygrgahidqnwhnz/bin/mpicc responds to '-showme:compile'... yes + configure: error: unable to locate ``adept-utils`` installation + +This gives us the output from the build, and it's fairly obvious that +mpileaks isn't finding it's ``adept-utils`` package. Spack has +automatically added the include and library directories of +``adept-utils`` to the compiler's search path, but some packages like +mpileaks can sometimes be picky and still want things spelled out on +their command line. But let's continue to pretend we're not brilliant +developers, and explore some other debugging paths: + +We can also enter the build area and try to manually run the build: + +.. code-block:: console + + % spack env mpileaks tcsh + % spack cd mpileaks + +The ``spack env`` command spawned a new shell that contains the same +environment that Spack used to build the mpileaks package (you can +substitute tcsh for your favorite shell). The ``spack cd`` command +changed our working dirctory to the last attempted build for mpileaks. +From here we can manually re-run the build: + +.. code-block:: console + + % ./configure + checking metadata... no + checking installation directory variables... yes + checking for a BSD-compatible install... /usr/bin/install -c + checking whether build environment is sane... yes + checking for a thread-safe mkdir -p... /usr/bin/mkdir -p + checking for gawk... gawk + checking whether make sets $(MAKE)... yes + checking for gcc... /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc + checking for C compiler default output file name... a.out + checking whether the C compiler works... yes + checking whether we are cross compiling... no + checking for suffix of executables... + checking for suffix of object files... o + checking whether we are using the GNU C compiler... yes + checking whether /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc accepts -g... yes + checking for /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc option to accept ISO C89... none needed + checking for style of include used by make... GNU + checking dependency style of /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc... gcc3 + checking whether /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc and cc understand -c and -o together... yes + checking whether we are using the GNU C++ compiler... yes + checking whether /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/g++ accepts -g... yes + checking dependency style of /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/g++... gcc3 + checking for /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/openmpi-2.0.1-5ee5j34c2y4kb5c3joygrgahidqnwhnz/bin/mpicc... /usr/workspace/wsa /legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/openmpi-2.0.1-5ee5j34c2y4kb5c3joygrgahidqnwhnz/bin/mpicc + Checking whether /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/openmpi-2.0.1-5ee5j34c2y4kb5c3joygrgahidqnwhnz/bin/mpicc responds to '-showme:compile'... yes + configure: error: unable to locate adept-utils installation + +We're seeing the same error, but now we're in a shell where we can run +the command ourselves and debug as needed. We could, for example, run +``./configure --help`` to see what options we can use to specify +dependencies. + +We can use the ``exit`` command to leave the shell spawned by ``spack +env``. + +------------------------------ +Specifying Configure Arguments +------------------------------ + +Let's add the configure arguments to the mpileaks' ``package.py``. This +version can be found in +``$SPACK_ROOT/lib/spack/docs/tutorial/examples/3.package.py``: + +.. literalinclude:: tutorial/examples/3.package.py + :start-after: # flake8: noqa + :language: python + +This is all we need for working mpileaks! If we install now we'll see: + +.. code-block:: console + + % spack install mpileaks + spack install mpileaks + ==> Installing mpileaks + ==> openmpi is already installed in /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/openmpi-2.0.1-5ee5j34c2y4kb5c3joygrgahidqnwhnz + ==> callpath is already installed in /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/callpath-1.0.2-zm4pf3gasgxeibyu2y262suktvaazube + ==> adept-utils is already installed in /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/adept-utils-1.0.1-7p7ezxwtajdglj6cmojy2vybjct4j4jz + ==> Using cached archive: /usr/workspace/wsa/legendre/spack/var/spack/cache/mpileaks/mpileaks-1.0.tar.gz + ==> Already staged mpileaks-1.0-eum4hmnlt6ovalwjnciaygfb3beja4gk in /usr/workspace/wsa/legendre/spack/var/spack/stage/mpileaks-1.0-eum4hmnlt6ovalwjnciaygfb3beja4gk + ==> Already patched mpileaks + ==> Building mpileaks [AutotoolsPackage] + ==> Executing phase : 'autoreconf' + ==> Executing phase : 'configure' + ==> Executing phase : 'build' + ==> Executing phase : 'install' + ==> Successfully installed mpileaks + Fetch: 0.00s. Build: 14.08s. Total: 14.08s. + [+] /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/mpileaks-1.0-eum4hmnlt6ovalwjnciaygfb3beja4gk + +We took a few shortcuts for this package that are worth highlighting. +Spack automatically detected that mpileaks was an autotools-based package +when we ran spack create. If this had been a CMake-based package we +would have been filling in a ``cmake_args`` function instead of +``configure_args``. If spack hadn't been able to detect the build +system, we'd be filling in a generic install method that would manually +be calling build commands, such as is found in the ``zlib`` package: + +.. code-block:: python + + def install(self, spec, prefix): + configure('--prefix={0}'.format(prefix)) + make() + make('install') + +-------- +Variants +-------- + +We have a successful mpileaks build, but let's take some time to improve +it. ``mpileaks`` has a build-time option to truncate parts of the stack +that it walks. Let's add a variant to allow users to set this when they +build in Spack. + +To do this, we'll add a variant to our package, as per the following (see +``$SPACK_ROOT/lib/spack/docs/tutorial/examples/4.package.py``): + +.. literalinclude:: tutorial/examples/4.package.py + :start-after: # flake8: noqa + :language: python + +We've added the variant ``stackstart``, and given it a default value of +``0``. If we install now we can see the stackstart variant added to the +configure line (output truncated for length): + +.. code-block:: console + + % spack install --verbose mpileaks stackstart=4 + ==> Installing mpileaks + ==> openmpi is already installed in /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/openmpi-2.0.1-5ee5j34c2y4kb5c3joygrgahidqnwhnz + ==> callpath is already installed in /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/callpath-1.0.2-zm4pf3gasgxeibyu2y262suktvaazube + ==> adept-utils is already installed in /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/adept-utils-1.0.1-7p7ezxwtajdglj6cmojy2vybjct4j4jz + ==> Using cached archive: /usr/workspace/wsa/legendre/spack/var/spack/cache/mpileaks/mpileaks-1.0.tar.gz + ==> Staging archive: /usr/workspace/wsa/legendre/spack/var/spack/stage/mpileaks-1.0-otqo2opkhan5ksujt6tpmdftydrieig7/mpileaks-1.0.tar.gz + ==> Created stage in /usr/workspace/wsa/legendre/spack/var/spack/stage/mpileaks-1.0-otqo2opkhan5ksujt6tpmdftydrieig7 + ==> Ran patch() for mpileaks + ==> Building mpileaks [AutotoolsPackage] + ==> Executing phase : 'autoreconf' + ==> Executing phase : 'configure' + ==> './configure' '--prefix=/usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/mpileaks-1.0-otqo2opkhan5ksujt6tpmdftydrieig7' '--with-adept-utils=/usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/adept-utils-1.0.1-7p7ezxwtajdglj6cmojy2vybjct4j4jz' '--with-callpath=/usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/callpath-1.0.2-zm4pf3gasgxeibyu2y262suktvaazube' '--with-stack-start-c=4' '--with-stack-start-fortran=4' + +--------------- +The Spec Object +--------------- + +This tutorial has glossed over a few important features, which weren't +too relevant for mpleaks but may be useful for other packages. There +were several places we references the ``self.spec`` object. This is a +powerful class for querying information about what we're building. For +example, you could use the spec to query information about how a +package's dependencies were built, or what compiler was being used, or +what version of a package is being installed. Full documentation can be +found in the :ref:`packaging-guide`, but here's some quick snippets with +common queries: + +- Am I building ``mpileaks`` version ``1.1`` or greater? + +.. code-block:: python + + if self.spec.satisfies('@1.1:'): + # Do things needed for 1.1+ + +- Is ``openmpi`` the MPI I'm building with? + +.. code-block:: python + + if self.spec['mpi'].name == 'openmpi': + # Do openmpi things + +- Am I building with ``gcc`` version less than ``5.0.0``: + +.. code-block:: python + + if self.spec.satisfies('%gcc@:5.0.0'): + # Add arguments specific to gcc's earlier than 5.0.0 + +- Am I built with the ``debug`` variant: + +.. code-block:: python + + if self.spec.satisfies('+debug'): + # Add -g option to configure flags + +- Is my ``dyninst`` dependency greater than version ``8.0``? + +.. code-block:: python + + if self.spec['dyninst'].satisfies('@8.0:'): + # Use newest dyninst options + +More examples can be found in the thousands of packages already added to +Spack in ``$SPACK_ROOT/var/spack/repos/builtin/packages``. + +Good Luck! diff --git a/lib/spack/docs/tutorial_sc16_spack_basics.rst b/lib/spack/docs/tutorial_sc16_spack_basics.rst new file mode 100644 index 0000000000..d83d6e4064 --- /dev/null +++ b/lib/spack/docs/tutorial_sc16_spack_basics.rst @@ -0,0 +1,1253 @@ +.. _basics-tutorial: + +========================================= +Basic Installation Tutorial +========================================= + +This tutorial will guide you through the process of installing software +using Spack. We will first cover the `spack install` command, focusing on +the power of the spec syntax and the flexibility it gives to users. We +will also cover the `spack find` command for viewing installed packages +and the `spack uninstall` command. Finally, we will touch on how Spack +manages compilers, especially as it relates to using Spack-built +compilers within Spack. We will include full output from all of the +commands demonstrated, although we will frequently call attention to only +small portions of that output (or merely to the fact that it +succeeded). The provided output is all from a cluster running Red Hat +Enterprise Linux. + +.. _basics-tutorial-install: + +---------------- +Installing Spack +---------------- + +Spack works out of the box. Simply clone spack and get going. + +.. code-block:: console + + becker33:~$ git clone https://github.com/LLNL/spack.git + Initialized empty Git repository in /g/g0/becker33/testclone/spack/.git/ + remote: Counting objects: 47125, done. + remote: Compressing objects: 100% (68/68), done. + remote: Total 47125 (delta 16), reused 2 (delta 2), pack-reused 47047 + Receiving objects: 100% (47125/47125), 12.02 MiB | 2.11 MiB/s, done. + Resolving deltas: 100% (23044/23044), done. + becker33:~$ cd spack + +Then add Spack to your path. + +.. code-block:: console + + becker33:spack$ export PATH=~/spack/bin/spack:$PATH + +You're good to go! + +----------------- +What is in Spack? +----------------- + +The ``spack list`` command shows available packages. + +.. code-block:: console + + becker33:spack$ spack list + ==> 1016 packages. + abinit hwloc piranha r-rjava + ack hydra pixman r-rjson + activeharmony hypre pkg-config r-rjsonio + ... + +The ``spack list`` command can also take a query string. Spack +automatically adds wildcards to both ends of the string. For example, +we can view all available python packages. + +.. code-block:: console + + becker33:spack$ spack list py + ==> 129 packages. + py-3to2 py-epydoc py-nestle py-pycparser py-six + py-alabaster py-flake8 py-netcdf py-pydatalog py-sncosmo + py-argcomplete py-funcsigs py-networkx py-pyelftools py-snowballstemmer + ... + +------------------- +Installing Packages +------------------- + +Installing a package with Spack is very simple. To install a piece of +software, simply type ``spack install `` + +.. code-block:: console + + $ spack install libelf + ==> Installing libelf + ==> Trying to fetch from file:///g/g0/becker33/SC16-spack/spack/var/spack/cache/libelf/libelf-0.8.13.tar.gz + curl: (37) Couldn't open file /g/g0/becker33/SC16-spack/spack/var/spack/cache/libelf/libelf-0.8.13.tar.gz + ==> Fetching from file:///g/g0/becker33/SC16-spack/spack/var/spack/cache/libelf/libelf-0.8.13.tar.gz failed. + ==> Trying to fetch from http://www.mr511.de/software/libelf-0.8.13.tar.gz + ################################################################################################################################################################################# 100.0% + ==> Staging archive: /g/g0/becker33/SC16-spack/spack/var/spack/stage/libelf-0.8.13-csrt4qxfkhjgn5xg3zjpkir7xdnszl2a/libelf-0.8.13.tar.gz + ==> Created stage in /g/g0/becker33/SC16-spack/spack/var/spack/stage/libelf-0.8.13-csrt4qxfkhjgn5xg3zjpkir7xdnszl2a + ==> No patches needed for libelf + ==> Building libelf [Package] + ==> Executing phase : 'install' + ==> Successfully installed libelf + Fetch: 1.21s. Build: 8.42s. Total: 9.62s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libelf-0.8.13-csrt4qxfkhjgn5xg3zjpkir7xdnszl2a + + +Spack's spec syntax is the interface by which we can request specific +configurations of the package. The ``%`` sigil is used to specify +compilers. + +.. code-block:: console + + $ spack install libelf %intel + ==> Installing libelf + ==> Trying to fetch from file:///g/g0/becker33/SC16-spack/spack/var/spack/cache/libelf/libelf-0.8.13.tar.gz + ################################################################################################################################################################################# 100.0% + ==> Staging archive: /g/g0/becker33/SC16-spack/spack/var/spack/stage/libelf-0.8.13-7wgp32xksatkvw2tbssmehw2t5tnxndj/libelf-0.8.13.tar.gz + ==> Created stage in /g/g0/becker33/SC16-spack/spack/var/spack/stage/libelf-0.8.13-7wgp32xksatkvw2tbssmehw2t5tnxndj + ==> No patches needed for libelf + ==> Building libelf [Package] + ==> Executing phase : 'install' + ==> Successfully installed libelf + Fetch: 0.09s. Build: 50.64s. Total: 50.72s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/intel-16.0.3/libelf-0.8.13-7wgp32xksatkvw2tbssmehw2t5tnxndj + +Note that this installation is located separately from the previous +one. We will discuss this in more detail later, but this is part of what +allows Spack to support arbitrarily versioned software. + +You can check for particular versions before requesting them. We will +use the ``spack versions`` command to see the available versions, and then +install a different version of ``libelf``. + +.. code-block:: console + + $ spack versions libelf + ==> Safe versions (already checksummed): + 0.8.13 + 0.8.12 + ==> Remote versions (not yet checksummed): + 0.8.11 + 0.8.10 + 0.8.9 + 0.8.8 + 0.8.7 + 0.8.6 + 0.8.5 + 0.8.4 + 0.8.3 + 0.8.2 + 0.8.0 + 0.7.0 + 0.6.4 + 0.5.2 + + +The ``@`` sigil is used to specify versions, both of packages and of +compilers. + +.. code-block:: console + + $ spack install libelf @0.8.12 + ==> Installing libelf + ==> Trying to fetch from file:///g/g0/becker33/SC16-spack/spack/var/spack/cache/libelf/libelf-0.8.12.tar.gz + curl: (37) Couldn't open file /g/g0/becker33/SC16-spack/spack/var/spack/cache/libelf/libelf-0.8.12.tar.gz + ==> Fetching from file:///g/g0/becker33/SC16-spack/spack/var/spack/cache/libelf/libelf-0.8.12.tar.gz failed. + ==> Trying to fetch from http://www.mr511.de/software/libelf-0.8.12.tar.gz + ################################################################################################################################################################################# 100.0% + ==> Staging archive: /g/g0/becker33/SC16-spack/spack/var/spack/stage/libelf-0.8.12-ipggckv6i7h44iryzfa4dwdela32a7fy/libelf-0.8.12.tar.gz + ==> Created stage in /g/g0/becker33/SC16-spack/spack/var/spack/stage/libelf-0.8.12-ipggckv6i7h44iryzfa4dwdela32a7fy + ==> No patches needed for libelf + ==> Building libelf [Package] + ==> Executing phase : 'install' + ==> Successfully installed libelf + Fetch: 1.12s. Build: 7.88s. Total: 9.00s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libelf-0.8.12-ipggckv6i7h44iryzfa4dwdela32a7fy + + + + $ spack install libelf %intel@15.0.4 + ==> Installing libelf + ==> Trying to fetch from file:///g/g0/becker33/SC16-spack/spack/var/spack/cache/libelf/libelf-0.8.13.tar.gz + ################################################################################################################################################################################# 100.0% + ==> Staging archive: /g/g0/becker33/SC16-spack/spack/var/spack/stage/libelf-0.8.13-w33hrejdyqu2j2gggdswitls2zv6kdsi/libelf-0.8.13.tar.gz + ==> Created stage in /g/g0/becker33/SC16-spack/spack/var/spack/stage/libelf-0.8.13-w33hrejdyqu2j2gggdswitls2zv6kdsi + ==> No patches needed for libelf + ==> Building libelf [Package] + ==> Executing phase : 'install' + ==> Successfully installed libelf + Fetch: 0.09s. Build: 55.51s. Total: 55.60s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/intel-15.0.4/libelf-0.8.13-w33hrejdyqu2j2gggdswitls2zv6kdsi + + +The spec syntax also includes compiler flags. Spack accepts ``cppflags``, +``cflags``, ``cxxflags``, ``fflags``, ``ldflags``, and ``ldlibs`` +parameters. The values of these fields must be escape-quoted with ``\"`` +on the command line. These values are injected into the compile line +automatically by the Spack compiler wrappers. + +.. code-block:: console + + $ spack install libelf @0.8.12 cppflags=\"-O3\" + ==> Installing libelf + ==> Trying to fetch from file:///g/g0/becker33/SC16-spack/spack/var/spack/cache/libelf/libelf-0.8.12.tar.gz + ################################################################################################################################################################################# 100.0% + ==> Staging archive: /g/g0/becker33/SC16-spack/spack/var/spack/stage/libelf-0.8.12-vrv2ttbd34xlfoxy4jwt6qsjrcbalmmw/libelf-0.8.12.tar.gz + ==> Created stage in /g/g0/becker33/SC16-spack/spack/var/spack/stage/libelf-0.8.12-vrv2ttbd34xlfoxy4jwt6qsjrcbalmmw + ==> No patches needed for libelf + ==> Building libelf [Package] + ==> Executing phase : 'install' + ==> Successfully installed libelf + Fetch: 0.04s. Build: 7.95s. Total: 7.99s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libelf-0.8.12-vrv2ttbd34xlfoxy4jwt6qsjrcbalmmw + + +The ``spack find`` command is used to query installed packages. Note that +some packages appear identical with the default output. The ``-l`` flag +shows the hash of each package, and the ``-f`` flag shows any non-empty +compiler flags of those packages. + +.. code-block:: console + + $ spack find + ==> 5 installed packages. + -- linux-redhat6-x86_64 / gcc@4.4.7 ----------------------------- + libelf@0.8.12 + libelf@0.8.12 + libelf@0.8.13 + + -- linux-redhat6-x86_64 / intel@15.0.4 -------------------------- + libelf@0.8.13 + + -- linux-redhat6-x86_64 / intel@16.0.3 -------------------------- + libelf@0.8.13 + + + + $ spack find -lf + ==> 5 installed packages. + -- linux-redhat6-x86_64 / gcc@4.4.7 ----------------------------- + ipggckv libelf@0.8.12%gcc + + vrv2ttb libelf@0.8.12%gcc cppflags="-O3" + + csrt4qx libelf@0.8.13%gcc + + + -- linux-redhat6-x86_64 / intel@15.0.4 -------------------------- + w33hrej libelf@0.8.13%intel + + + -- linux-redhat6-x86_64 / intel@16.0.3 -------------------------- + 7wgp32x libelf@0.8.13%intel + + +Spack generates a hash for each spec. This hash is a function of the full +provenance of the package, so any change to the spec affects the +hash. Spack uses this value to compare specs and to generate unique +installation directories for every combinatorial version. As we move into +more complicated packages with software dependencies, we can see that +Spack reuses existing packages to satisfy a dependency only when the +existing package's hash matches the desired spec. + +.. code-block:: console + + $ spack install libdwarf + ==> Installing libdwarf + ==> libelf is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libelf-0.8.13-csrt4qxfkhjgn5xg3zjpkir7xdnszl2a + ==> Can not find version 20160507 in url_list + ==> Trying to fetch from file:///g/g0/becker33/SC16-spack/spack/var/spack/cache/libdwarf/libdwarf-20160507.tar.gz + curl: (37) Couldn't open file /g/g0/becker33/SC16-spack/spack/var/spack/cache/libdwarf/libdwarf-20160507.tar.gz + ==> Fetching from file:///g/g0/becker33/SC16-spack/spack/var/spack/cache/libdwarf/libdwarf-20160507.tar.gz failed. + ==> Trying to fetch from http://www.prevanders.net/libdwarf-20160507.tar.gz + ################################################################################################################################################################################# 100.0% + ==> Staging archive: /g/g0/becker33/SC16-spack/spack/var/spack/stage/libdwarf-20160507-yfx6p3g3rkmqvcqbmtb34o6pln7pqvcz/libdwarf-20160507.tar.gz + ==> Created stage in /g/g0/becker33/SC16-spack/spack/var/spack/stage/libdwarf-20160507-yfx6p3g3rkmqvcqbmtb34o6pln7pqvcz + ==> No patches needed for libdwarf + ==> Building libdwarf [Package] + ==> Executing phase : 'install' + ==> Successfully installed libdwarf + Fetch: 1.56s. Build: 33.59s. Total: 35.15s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libdwarf-20160507-yfx6p3g3rkmqvcqbmtb34o6pln7pqvcz + + +Dependencies can be explicitly requested using the ``^`` sigil. Note that +the spec syntax is recursive. Anything we could specify about the +top-level package, we can also specify about a dependency using ``^``. + +.. code-block:: console + + $ spack install libdwarf ^libelf @0.8.12 %intel + ==> Installing libdwarf + ==> Installing libelf + ==> Trying to fetch from file:///g/g0/becker33/SC16-spack/spack/var/spack/cache/libelf/libelf-0.8.12.tar.gz + ################################################################################################################################################################################# 100.0% + ==> Staging archive: /g/g0/becker33/SC16-spack/spack/var/spack/stage/libelf-0.8.12-4blbe3qxqct3ymrfoxxnxysmybvbxay7/libelf-0.8.12.tar.gz + ==> Created stage in /g/g0/becker33/SC16-spack/spack/var/spack/stage/libelf-0.8.12-4blbe3qxqct3ymrfoxxnxysmybvbxay7 + ==> No patches needed for libelf + ==> Building libelf [Package] + ==> Executing phase : 'install' + ==> Successfully installed libelf + Fetch: 0.04s. Build: 52.16s. Total: 52.19s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/intel-16.0.3/libelf-0.8.12-4blbe3qxqct3ymrfoxxnxysmybvbxay7 + ==> Can not find version 20160507 in url_list + ==> Trying to fetch from file:///g/g0/becker33/SC16-spack/spack/var/spack/cache/libdwarf/libdwarf-20160507.tar.gz + ################################################################################################################################################################################# 100.0% + ==> Staging archive: /g/g0/becker33/SC16-spack/spack/var/spack/stage/libdwarf-20160507-csruprgucaujkfkrcywhwou7nbeis5fo/libdwarf-20160507.tar.gz + ==> Created stage in /g/g0/becker33/SC16-spack/spack/var/spack/stage/libdwarf-20160507-csruprgucaujkfkrcywhwou7nbeis5fo + ==> No patches needed for libdwarf + ==> Building libdwarf [Package] + ==> Executing phase : 'install' + ==> Successfully installed libdwarf + Fetch: 0.40s. Build: 2m 17.29s. Total: 2m 17.69s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/intel-16.0.3/libdwarf-20160507-csruprgucaujkfkrcywhwou7nbeis5fo + + +Packages can also be referred to from the command line by their package +hash. Using the ``spack find -lf`` command earlier we saw that the hash +of our optimized installation of libelf (``cppflags=\"-O3\"``) began with +``vrv2ttb``. We can now explicitly build with that package without typing +the entire spec, by using the ``/`` sigil to refer to it by hash. As with +other tools like git, you do not need to specify an *entire* hash on the +command line. You can specify just enough digits to identify a hash +uniquely. If a hash prefix is ambiguous (i.e., two or more installed +packages share the prefix) then spack will report an error. + +.. code-block:: console + + $ spack install libdwarf ^/vrv2ttb + ==> Installing libdwarf + ==> libelf is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libelf-0.8.12-vrv2ttbd34xlfoxy4jwt6qsjrcbalmmw + ==> Can not find version 20160507 in url_list + ==> Trying to fetch from file:///g/g0/becker33/SC16-spack/spack/var/spack/cache/libdwarf/libdwarf-20160507.tar.gz + #################################################################################################################################################################################################################################################### 100.0% + ==> Staging archive: /g/g0/becker33/SC16-spack/spack/var/spack/stage/libdwarf-20160507-dtg3tgnp7htccoly26gduqlrgvnwcp5t/libdwarf-20160507.tar.gz + ==> Created stage in /g/g0/becker33/SC16-spack/spack/var/spack/stage/libdwarf-20160507-dtg3tgnp7htccoly26gduqlrgvnwcp5t + ==> No patches needed for libdwarf + ==> Building libdwarf [Package] + ==> Executing phase : 'install' + ==> Successfully installed libdwarf + Fetch: 0.96s. Build: 24.03s. Total: 24.99s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libdwarf-20160507-dtg3tgnp7htccoly26gduqlrgvnwcp5t + + +The ``spack find`` command can also take a ``-d`` flag, which can show +dependency information. Note that each package has a top-level entry, +even if it also appears as a dependency. + +.. code-block:: console + + $ spack find -ldf + ==> 9 installed packages. + -- linux-redhat6-x86_64 / gcc@4.4.7 ----------------------------- + dtg3tgn libdwarf@20160507%gcc + vrv2ttb ^libelf@0.8.12%gcc cppflags="-O3" + + yfx6p3g libdwarf@20160507%gcc + csrt4qx ^libelf@0.8.13%gcc + + ipggckv libelf@0.8.12%gcc + + vrv2ttb libelf@0.8.12%gcc cppflags="-O3" + + csrt4qx libelf@0.8.13%gcc + + + -- linux-redhat6-x86_64 / intel@15.0.4 -------------------------- + w33hrej libelf@0.8.13%intel + + + -- linux-redhat6-x86_64 / intel@16.0.3 -------------------------- + csruprg libdwarf@20160507%intel + 4blbe3q ^libelf@0.8.12%intel + + 4blbe3q libelf@0.8.12%intel + + 7wgp32x libelf@0.8.13%intel + + +As we get to more complex packages, full installs will take too long to +build in the time allotted for this tutorial. Our collaborators at CERN +have been working on binary caching for Spack, which would allow for very +fast installs of previously built packages. We are still working out the +security ramifications of the feature, but it is coming soon. + +For now, we will switch to doing "fake" installs. When supplied with the +``--fake`` flag (primarily used for debugging), Spack computes build +metadata the same way it normally would, but it does not download the +source or run the install script for a pacakge. We can use this to +quickly demonstrate some of the more advanced Spack features in our +limited tutorial time. + +``HDF5`` is an example of a more complicated package, with an MPI +dependency. If we install it "out of the box," it will build with +``openmpi``. + +.. code-block:: console + + $ spack install --fake hdf5 + ==> Installing hdf5 + ==> Installing zlib + ==> Building zlib [Package] + ==> Successfully installed zlib + Fetch: . Build: 0.11s. Total: 0.11s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/zlib-1.2.8-ayc4jq7vxuzge5n444gutvskeytfdruh + ==> Installing openmpi + ==> Installing hwloc + ==> Installing libpciaccess + ==> Installing util-macros + ==> Building util-macros [Package] + ==> Successfully installed util-macros + Fetch: . Build: 0.11s. Total: 0.11s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/util-macros-1.19.0-pc6zhs4cnkmg2cv4et4fizsp6scuvacg + ==> Installing libtool + ==> Installing m4 + ==> Installing libsigsegv + ==> Building libsigsegv [Package] + ==> Successfully installed libsigsegv + Fetch: . Build: 0.11s. Total: 0.11s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libsigsegv-2.10-q4cok3yber7lhf3jswg6mysg7oi53unh + ==> Building m4 [Package] + ==> Successfully installed m4 + Fetch: . Build: 0.23s. Total: 0.23s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/m4-1.4.17-qijdzvhjyybrtwbqm73vykhmkaqro3je + ==> Building libtool [Package] + ==> Successfully installed libtool + Fetch: . Build: 0.11s. Total: 0.11s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libtool-2.4.6-rdx5nkfjwlvcanz5il3ys2pe34j4vxx5 + ==> Installing pkg-config + ==> Building pkg-config [Package] + ==> Successfully installed pkg-config + Fetch: . Build: 0.11s. Total: 0.11s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/pkg-config-0.29.1-wpjnlzahdw6ahkrgmqyeugkj2zhv4tui + ==> Building libpciaccess [Package] + ==> Successfully installed libpciaccess + Fetch: . Build: 0.10s. Total: 0.10s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libpciaccess-0.13.4-m2f6fpm22rpprq2ihkmfx6llf363264m + ==> Building hwloc [Package] + ==> Successfully installed hwloc + Fetch: . Build: 0.23s. Total: 0.23s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/hwloc-1.11.4-xpb6hbl2hsze25cgdgfnoppn6rchhzaz + ==> Building openmpi [Package] + ==> Successfully installed openmpi + Fetch: . Build: 0.35s. Total: 0.35s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openmpi-2.0.1-j4cgoq4furxvr73pq72r2qgywgksw3qn + ==> Building hdf5 [AutotoolsPackage] + ==> Successfully installed hdf5 + Fetch: . Build: 0.61s. Total: 0.61s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/hdf5-1.10.0-patch1-ezvtnox35albuaxqryuondweyjgeo6es + + +Spack packages can also have variants. Boolean variants can be specified +using the ``+`` and ``~`` or ``-`` sigils. There are two sigils for +``False`` to avoid conflicts with shell parsing in different +situations. Variants (boolean or otherwise) can also be specified using +the same syntax as compiler flags. Here we can install HDF5 without MPI +support. + +.. code-block:: console + + $ spack install --fake hdf5~mpi + ==> Installing hdf5 + ==> zlib is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/zlib-1.2.8-ayc4jq7vxuzge5n444gutvskeytfdruh + ==> Building hdf5 [AutotoolsPackage] + ==> Successfully installed hdf5 + Fetch: . Build: 0.22s. Total: 0.22s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/hdf5-1.10.0-patch1-twppaioxqn6lti4grgopnmhwcq3h2rpw + + +We might also want to install HDF5 with a different MPI +implementation. While MPI is not a package itself, packages can depend on +abstract interfaces like MPI. Spack handles these through "virtual +dependencies." A package, such as HDF5, can depend on the MPI +interface. Other packages (``openmpi``, ``mpich``, ``mvapich``, etc.) +provide the MPI interface. Any of these providers can be requested for +an MPI dependency. For example, we can build HDF5 with MPI support +provided by mpich by specifying a dependency on ``mpich``. Spack also +supports versioning of virtual dependencies. A package can depend on the +MPI interface at version 3, and provider packages specify what version of +the interface *they* provide. The partial spec ``^mpi@3`` can be safisfied +by any of several providers. + +.. code-block:: console + + $ spack install --fake hdf5+mpi ^mpich + ==> Installing hdf5 + ==> mpich is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/mpich-3.2-5jlp2ndnsb67txggraglu47vjmayx5za + ==> zlib is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/zlib-1.2.8-ayc4jq7vxuzge5n444gutvskeytfdruh + ==> Building hdf5 [AutotoolsPackage] + ==> Successfully installed hdf5 + Fetch: . Build: 0.38s. Total: 0.38s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/hdf5-1.10.0-patch1-j36yfw25i6gdd3q4vwlupgkpwic4ua6m + + +We'll do a quick check in on what we have installed so far. + +.. code-block:: console + + $ spack find -ldf + ==> 22 installed packages. + -- linux-redhat6-x86_64 / gcc@4.4.7 ----------------------------- + twppaio hdf5@1.10.0-patch1%gcc + ayc4jq7 ^zlib@1.2.8%gcc + + j36yfw2 hdf5@1.10.0-patch1%gcc + 5jlp2nd ^mpich@3.2%gcc + ayc4jq7 ^zlib@1.2.8%gcc + + ezvtnox hdf5@1.10.0-patch1%gcc + j4cgoq4 ^openmpi@2.0.1%gcc + xpb6hbl ^hwloc@1.11.4%gcc + m2f6fpm ^libpciaccess@0.13.4%gcc + ayc4jq7 ^zlib@1.2.8%gcc + + xpb6hbl hwloc@1.11.4%gcc + m2f6fpm ^libpciaccess@0.13.4%gcc + + dtg3tgn libdwarf@20160507%gcc + vrv2ttb ^libelf@0.8.12%gcc cppflags="-O3" + + yfx6p3g libdwarf@20160507%gcc + csrt4qx ^libelf@0.8.13%gcc + + ipggckv libelf@0.8.12%gcc + + vrv2ttb libelf@0.8.12%gcc cppflags="-O3" + + csrt4qx libelf@0.8.13%gcc + + m2f6fpm libpciaccess@0.13.4%gcc + + q4cok3y libsigsegv@2.10%gcc + + rdx5nkf libtool@2.4.6%gcc + + qijdzvh m4@1.4.17%gcc + q4cok3y ^libsigsegv@2.10%gcc + + 5jlp2nd mpich@3.2%gcc + + j4cgoq4 openmpi@2.0.1%gcc + xpb6hbl ^hwloc@1.11.4%gcc + m2f6fpm ^libpciaccess@0.13.4%gcc + + wpjnlza pkg-config@0.29.1%gcc + + pc6zhs4 util-macros@1.19.0%gcc + + ayc4jq7 zlib@1.2.8%gcc + + + -- linux-redhat6-x86_64 / intel@15.0.4 -------------------------- + w33hrej libelf@0.8.13%intel + + + -- linux-redhat6-x86_64 / intel@16.0.3 -------------------------- + csruprg libdwarf@20160507%intel + 4blbe3q ^libelf@0.8.12%intel + + 4blbe3q libelf@0.8.12%intel + + 7wgp32x libelf@0.8.13%intel + + +Spack models the dependencies of packages as a directed acyclic graph +(DAG). The ``spack find -d`` command shows the tree representation of +that graph. We can also use the ``spack graph`` command to view the entire +DAG as a graph. + +.. code-block:: console + + $ spack graph hdf5+mpi ^mpich + o hdf5 + |\ + o | zlib + / + o mpich + +You may also have noticed that there are some packages shown in the +``spack find -d`` output that we didn't install explicitly. These are +dependencies that were installed implicitly. A few packages installed +implicitly are not shown as dependencies in the ``spack find -d`` +output. These are build dependencies. For example, ``libpciaccess`` is a +dependency of openmpi and requires m4 to build. Spack will build `m4`` as +part of the installation of ``openmpi``, but it does not become a part of +the DAG because it is not linked in at run time. Spack handles build +dependencies differently because of their different (less strict) +consistency requirements. It is entirely possible to have two packages +using different versions of a dependency to build, which obviously cannot +be done with linked dependencies. + +``HDF5`` is more complicated than our basic example of libelf and +libdwarf, but it's still within the realm of software that an experienced +HPC user could reasonably expect to install given a bit of time. Now +let's look at a more complicated package. + +.. code-block:: console + + $ spack install --fake trilinos + ==> Installing trilinos + ==> Installing superlu-dist + ==> openmpi is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openmpi-2.0.1-j4cgoq4furxvr73pq72r2qgywgksw3qn + ==> Installing parmetis + ==> openmpi is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openmpi-2.0.1-j4cgoq4furxvr73pq72r2qgywgksw3qn + ==> Installing cmake + ==> Installing bzip2 + ==> Building bzip2 [Package] + ==> Successfully installed bzip2 + Fetch: . Build: 0.11s. Total: 0.11s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/bzip2-1.0.6-gll2xsahysy7ji5gkmfxwkofdt3mwjhs + ==> expat is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/expat-2.2.0-mg5kwd3kluxdgorj32vzbp7aksg3vqej + ==> Installing ncurses + ==> Building ncurses [Package] + ==> Successfully installed ncurses + Fetch: . Build: 0.11s. Total: 0.11s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/ncurses-6.0-fttg4astvrtq2buey4wq66tnyu7bgj2c + ==> zlib is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/zlib-1.2.8-ayc4jq7vxuzge5n444gutvskeytfdruh + ==> Installing openssl + ==> zlib is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/zlib-1.2.8-ayc4jq7vxuzge5n444gutvskeytfdruh + ==> Building openssl [Package] + ==> Successfully installed openssl + Fetch: . Build: 0.23s. Total: 0.23s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openssl-1.0.2j-kt5xyk2dkho6tzadnqlbnbujmljprylg + ==> Installing libarchive + ==> Installing lzma + ==> Building lzma [Package] + ==> Successfully installed lzma + Fetch: . Build: 0.11s. Total: 0.11s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/lzma-4.32.7-hah2cdo3zbulz6yg5do6dvnfn6en5v5c + ==> Installing nettle + ==> m4 is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/m4-1.4.17-qijdzvhjyybrtwbqm73vykhmkaqro3je + ==> Installing gmp + ==> m4 is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/m4-1.4.17-qijdzvhjyybrtwbqm73vykhmkaqro3je + ==> Building gmp [AutotoolsPackage] + ==> Successfully installed gmp + Fetch: . Build: 0.11s. Total: 0.11s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/gmp-6.1.1-uwn4gfdtq3sywy5uf4f7znrh66oybikf + ==> Building nettle [Package] + ==> Successfully installed nettle + Fetch: . Build: 0.18s. Total: 0.18s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/nettle-3.2-w4ieitifcmrldo4ra7as63apagzf56ja + ==> bzip2 is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/bzip2-1.0.6-gll2xsahysy7ji5gkmfxwkofdt3mwjhs + ==> expat is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/expat-2.2.0-mg5kwd3kluxdgorj32vzbp7aksg3vqej + ==> Installing libxml2 + ==> Installing xz + ==> Building xz [Package] + ==> Successfully installed xz + Fetch: . Build: 0.11s. Total: 0.11s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/xz-5.2.2-bxh6cpyqqozazm5okvjqk23sww3gccnf + ==> zlib is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/zlib-1.2.8-ayc4jq7vxuzge5n444gutvskeytfdruh + ==> Building libxml2 [Package] + ==> Successfully installed libxml2 + Fetch: . Build: 0.35s. Total: 0.35s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libxml2-2.9.4-un323rppyu5qipkegyf7flmymvtmunrx + ==> zlib is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/zlib-1.2.8-ayc4jq7vxuzge5n444gutvskeytfdruh + ==> Installing lz4 + ==> Building lz4 [Package] + ==> Successfully installed lz4 + Fetch: . Build: 0.12s. Total: 0.12s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/lz4-131-ivy2fcaw7ywujx74weebdi5bsm7q4vkc + ==> openssl is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openssl-1.0.2j-kt5xyk2dkho6tzadnqlbnbujmljprylg + ==> xz is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/xz-5.2.2-bxh6cpyqqozazm5okvjqk23sww3gccnf + ==> Installing lzo + ==> Building lzo [AutotoolsPackage] + ==> Successfully installed lzo + Fetch: . Build: 0.11s. Total: 0.11s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/lzo-2.09-dlgnm74ozo6baactkft5oah2jre2ri2i + ==> Building libarchive [Package] + ==> Successfully installed libarchive + Fetch: . Build: 1.35s. Total: 1.35s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libarchive-3.2.1-biq3kebw7vel7njusill7vv7mjldkqjv + ==> xz is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/xz-5.2.2-bxh6cpyqqozazm5okvjqk23sww3gccnf + ==> Installing curl + ==> zlib is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/zlib-1.2.8-ayc4jq7vxuzge5n444gutvskeytfdruh + ==> openssl is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openssl-1.0.2j-kt5xyk2dkho6tzadnqlbnbujmljprylg + ==> Building curl [Package] + ==> Successfully installed curl + Fetch: . Build: 0.36s. Total: 0.36s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/curl-7.50.3-oze4gqutj4x2isbkcn5ob2bhhxbskod4 + ==> Building cmake [Package] + ==> Successfully installed cmake + Fetch: . Build: 1.64s. Total: 1.64s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/cmake-3.6.1-n2nkknrku6dvuneo3rjumim7axt7n36e + ==> Installing metis + ==> cmake is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/cmake-3.6.1-n2nkknrku6dvuneo3rjumim7axt7n36e + ==> Building metis [Package] + ==> Successfully installed metis + Fetch: . Build: 0.11s. Total: 0.11s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/metis-5.1.0-ithifyl4xvqbn76js23wsb4tjnztrbdv + ==> Building parmetis [Package] + ==> Successfully installed parmetis + Fetch: . Build: 0.62s. Total: 0.62s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/parmetis-4.0.3-rtg6hml5t6acdcnxomn3l5zfiful4d2t + ==> Installing openblas + ==> Building openblas [Package] + ==> Successfully installed openblas + Fetch: . Build: 0.11s. Total: 0.11s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openblas-0.2.19-bwofa7fhff6od5zn56vy3j4eeyupsqgt + ==> metis is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/metis-5.1.0-ithifyl4xvqbn76js23wsb4tjnztrbdv + ==> Building superlu-dist [Package] + ==> Successfully installed superlu-dist + Fetch: . Build: 0.85s. Total: 0.85s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/superlu-dist-5.1.1-25r6jlvkpjnkiuwt2rtbzhk3l3htuxs7 + ==> cmake is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/cmake-3.6.1-n2nkknrku6dvuneo3rjumim7axt7n36e + ==> Installing glm + ==> cmake is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/cmake-3.6.1-n2nkknrku6dvuneo3rjumim7axt7n36e + ==> Building glm [Package] + ==> Successfully installed glm + Fetch: . Build: 0.12s. Total: 0.12s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/glm-0.9.7.1-7a6oho4aerz7vftxd5ur7lywscht2iry + ==> Installing hypre + ==> openmpi is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openmpi-2.0.1-j4cgoq4furxvr73pq72r2qgywgksw3qn + ==> openblas is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openblas-0.2.19-bwofa7fhff6od5zn56vy3j4eeyupsqgt + ==> Building hypre [Package] + ==> Successfully installed hypre + Fetch: . Build: 0.61s. Total: 0.61s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/hypre-2.11.1-lf7hcejiiww5peesh57quda72z67veit + ==> metis is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/metis-5.1.0-ithifyl4xvqbn76js23wsb4tjnztrbdv + ==> Installing netlib-scalapack + ==> openmpi is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openmpi-2.0.1-j4cgoq4furxvr73pq72r2qgywgksw3qn + ==> cmake is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/cmake-3.6.1-n2nkknrku6dvuneo3rjumim7axt7n36e + ==> openblas is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openblas-0.2.19-bwofa7fhff6od5zn56vy3j4eeyupsqgt + ==> Building netlib-scalapack [Package] + ==> Successfully installed netlib-scalapack + Fetch: . Build: 0.61s. Total: 0.61s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/netlib-scalapack-2.0.2-dvcanz2qq4dfcexznbhbmzbxfj43uz4q + ==> Installing suite-sparse + ==> Installing tbb + ==> Building tbb [Package] + ==> Successfully installed tbb + Fetch: . Build: 0.12s. Total: 0.12s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/tbb-4.4.4-zawzkkhrmdonbjpj3a5bb6gkgnqlrjeu + ==> openblas is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openblas-0.2.19-bwofa7fhff6od5zn56vy3j4eeyupsqgt + ==> metis is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/metis-5.1.0-ithifyl4xvqbn76js23wsb4tjnztrbdv + ==> Building suite-sparse [Package] + ==> Successfully installed suite-sparse + Fetch: . Build: 0.49s. Total: 0.49s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/suite-sparse-4.5.3-lvur6hriy2j7xfjwh5punp3exwpynzm6 + ==> openmpi is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openmpi-2.0.1-j4cgoq4furxvr73pq72r2qgywgksw3qn + ==> Installing netcdf + ==> m4 is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/m4-1.4.17-qijdzvhjyybrtwbqm73vykhmkaqro3je + ==> curl is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/curl-7.50.3-oze4gqutj4x2isbkcn5ob2bhhxbskod4 + ==> zlib is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/zlib-1.2.8-ayc4jq7vxuzge5n444gutvskeytfdruh + ==> hdf5 is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/hdf5-1.10.0-patch1-ezvtnox35albuaxqryuondweyjgeo6es + ==> Building netcdf [Package] + ==> Successfully installed netcdf + Fetch: . Build: 0.90s. Total: 0.90s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/netcdf-4.4.1-tcl4zbrmdfrit2cqlaxig6xieu5h552j + ==> Installing mumps + ==> netlib-scalapack is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/netlib-scalapack-2.0.2-dvcanz2qq4dfcexznbhbmzbxfj43uz4q + ==> openmpi is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openmpi-2.0.1-j4cgoq4furxvr73pq72r2qgywgksw3qn + ==> openblas is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openblas-0.2.19-bwofa7fhff6od5zn56vy3j4eeyupsqgt + ==> Building mumps [Package] + ==> Successfully installed mumps + Fetch: . Build: 0.74s. Total: 0.74s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/mumps-5.0.2-kr5r4nnx5tfcacxnk3ii5dsxbe6pu5fy + ==> Installing matio + ==> Building matio [Package] + ==> Successfully installed matio + Fetch: . Build: 0.11s. Total: 0.11s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/matio-1.5.2-4zrozucookychlvc4q53omp2zyfk2bed + ==> Installing boost + ==> bzip2 is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/bzip2-1.0.6-gll2xsahysy7ji5gkmfxwkofdt3mwjhs + ==> zlib is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/zlib-1.2.8-ayc4jq7vxuzge5n444gutvskeytfdruh + ==> Building boost [Package] + ==> Successfully installed boost + Fetch: . Build: 0.35s. Total: 0.35s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/boost-1.62.0-je7eqvzt74kezwhh55y5lwt5dy6pnali + ==> parmetis is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/parmetis-4.0.3-rtg6hml5t6acdcnxomn3l5zfiful4d2t + ==> openblas is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openblas-0.2.19-bwofa7fhff6od5zn56vy3j4eeyupsqgt + ==> hdf5 is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/hdf5-1.10.0-patch1-ezvtnox35albuaxqryuondweyjgeo6es + ==> Building trilinos [Package] + ==> Successfully installed trilinos + Fetch: . Build: 2.63s. Total: 2.63s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/trilinos-12.8.1-uvd6dfd7x4uyvck4awo3r3frudihn4ar + + +Now we're starting to see the power of Spack. Trilinos has 11 top level +dependecies, many of which have dependencies of their own. Installing +more complex packages can take days or weeks even for an experienced +user. Although we've done a fake installation for the tutorial, a real +installation of trilinos using Spack takes about 3 hours (depending on +the system), but only 20 seconds of programmer time. + +Spack manages constistency of the entire DAG. Every MPI dependency will +be satisfied by the same configuration of MPI, etc. If we install +``trilinos`` again specifying a dependency on our previous HDF5 built +with ``mpich``: + +.. code-block:: console + + $ spack install --fake trilinos ^hdf5+mpi ^mpich + ==> Installing trilinos + ==> Installing superlu-dist + ==> mpich is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/mpich-3.2-5jlp2ndnsb67txggraglu47vjmayx5za + ==> metis is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/metis-5.1.0-ithifyl4xvqbn76js23wsb4tjnztrbdv + ==> Installing parmetis + ==> mpich is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/mpich-3.2-5jlp2ndnsb67txggraglu47vjmayx5za + ==> metis is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/metis-5.1.0-ithifyl4xvqbn76js23wsb4tjnztrbdv + ==> cmake is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/cmake-3.6.1-n2nkknrku6dvuneo3rjumim7axt7n36e + ==> Building parmetis [Package] + ==> Successfully installed parmetis + Fetch: . Build: 0.38s. Total: 0.38s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/parmetis-4.0.3-43kbtni6p5y446c6qdkybq4htj7ot4zn + ==> openblas is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openblas-0.2.19-bwofa7fhff6od5zn56vy3j4eeyupsqgt + ==> Building superlu-dist [Package] + ==> Successfully installed superlu-dist + Fetch: . Build: 0.61s. Total: 0.61s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/superlu-dist-5.1.1-46uuupehmonx5jicc6xnegnud2n5jqyl + ==> cmake is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/cmake-3.6.1-n2nkknrku6dvuneo3rjumim7axt7n36e + ==> glm is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/glm-0.9.7.1-7a6oho4aerz7vftxd5ur7lywscht2iry + ==> Installing hypre + ==> mpich is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/mpich-3.2-5jlp2ndnsb67txggraglu47vjmayx5za + ==> openblas is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openblas-0.2.19-bwofa7fhff6od5zn56vy3j4eeyupsqgt + ==> Building hypre [Package] + ==> Successfully installed hypre + Fetch: . Build: 0.37s. Total: 0.37s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/hypre-2.11.1-6ajnyymoivs5apajd7thjisae36jv4lz + ==> metis is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/metis-5.1.0-ithifyl4xvqbn76js23wsb4tjnztrbdv + ==> Installing netlib-scalapack + ==> mpich is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/mpich-3.2-5jlp2ndnsb67txggraglu47vjmayx5za + ==> cmake is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/cmake-3.6.1-n2nkknrku6dvuneo3rjumim7axt7n36e + ==> openblas is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openblas-0.2.19-bwofa7fhff6od5zn56vy3j4eeyupsqgt + ==> Building netlib-scalapack [Package] + ==> Successfully installed netlib-scalapack + Fetch: . Build: 0.37s. Total: 0.37s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/netlib-scalapack-2.0.2-dayeep27omm26wksd3iqvbu3gezc2eoh + ==> suite-sparse is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/suite-sparse-4.5.3-lvur6hriy2j7xfjwh5punp3exwpynzm6 + ==> Installing netcdf + ==> m4 is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/m4-1.4.17-qijdzvhjyybrtwbqm73vykhmkaqro3je + ==> curl is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/curl-7.50.3-oze4gqutj4x2isbkcn5ob2bhhxbskod4 + ==> zlib is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/zlib-1.2.8-ayc4jq7vxuzge5n444gutvskeytfdruh + ==> hdf5 is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/hdf5-1.10.0-patch1-j36yfw25i6gdd3q4vwlupgkpwic4ua6m + ==> Building netcdf [Package] + ==> Successfully installed netcdf + Fetch: . Build: 0.67s. Total: 0.67s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/netcdf-4.4.1-gfemi4jk4qltvp33xhtpkam7dozbqvhq + ==> Installing mumps + ==> mpich is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/mpich-3.2-5jlp2ndnsb67txggraglu47vjmayx5za + ==> netlib-scalapack is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/netlib-scalapack-2.0.2-dayeep27omm26wksd3iqvbu3gezc2eoh + ==> openblas is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openblas-0.2.19-bwofa7fhff6od5zn56vy3j4eeyupsqgt + ==> Building mumps [Package] + ==> Successfully installed mumps + Fetch: . Build: 0.49s. Total: 0.49s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/mumps-5.0.2-w7t5pl3jhhwitfiyer63zj6zv7idkt3m + ==> mpich is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/mpich-3.2-5jlp2ndnsb67txggraglu47vjmayx5za + ==> matio is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/matio-1.5.2-4zrozucookychlvc4q53omp2zyfk2bed + ==> boost is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/boost-1.62.0-je7eqvzt74kezwhh55y5lwt5dy6pnali + ==> parmetis is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/parmetis-4.0.3-43kbtni6p5y446c6qdkybq4htj7ot4zn + ==> openblas is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/openblas-0.2.19-bwofa7fhff6od5zn56vy3j4eeyupsqgt + ==> hdf5 is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/hdf5-1.10.0-patch1-j36yfw25i6gdd3q4vwlupgkpwic4ua6m + ==> Building trilinos [Package] + ==> Successfully installed trilinos + Fetch: . Build: 2.42s. Total: 2.42s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/trilinos-12.8.1-ffwrpxnq7lhiw2abxn2u7ffr4jjsdwep + +We see that every package in the trilinos DAG that depends on MPI now +uses ``mpich``. + +.. code-block:: console + + $ spack find -d trilinos + ==> 2 installed packages. + -- linux-redhat6-x86_64 / gcc@4.4.7 ----------------------------- + trilinos@12.8.1 + ^boost@1.62.0 + ^bzip2@1.0.6 + ^zlib@1.2.8 + ^glm@0.9.7.1 + ^hdf5@1.10.0-patch1 + ^mpich@3.2 + ^hypre@2.11.1 + ^openblas@0.2.19 + ^matio@1.5.2 + ^metis@5.1.0 + ^mumps@5.0.2 + ^netlib-scalapack@2.0.2 + ^netcdf@4.4.1 + ^curl@7.50.3 + ^openssl@1.0.2j + ^parmetis@4.0.3 + ^suite-sparse@4.5.3 + ^tbb@4.4.4 + ^superlu-dist@5.1.1 + + trilinos@12.8.1 + ^boost@1.62.0 + ^bzip2@1.0.6 + ^zlib@1.2.8 + ^glm@0.9.7.1 + ^hdf5@1.10.0-patch1 + ^openmpi@2.0.1 + ^hwloc@1.11.4 + ^libpciaccess@0.13.4 + ^hypre@2.11.1 + ^openblas@0.2.19 + ^matio@1.5.2 + ^metis@5.1.0 + ^mumps@5.0.2 + ^netlib-scalapack@2.0.2 + ^netcdf@4.4.1 + ^curl@7.50.3 + ^openssl@1.0.2j + ^parmetis@4.0.3 + ^suite-sparse@4.5.3 + ^tbb@4.4.4 + ^superlu-dist@5.1.1 + + +As we discussed before, the ``spack find -d`` command shows the +dependency information as a tree. While that is often sufficient, many +complicated packages, including trilinos, have dependencies that +cannot be fully represented as a tree. Again, the ``spack graph`` +command shows the full DAG of the dependency information. + +.. code-block:: console + + $ spack graph trilinos + o trilinos + |\ + | |\ + | | |\ + | | | |\ + | | | | |\ + | | | | | |\ + | | | | | | |\ + | o | | | | | | netcdf + | |\ \ \ \ \ \ \ + | | |\ \ \ \ \ \ \ + | | | o | | | | | | curl + | | |/| | | | | | | + | |/| | | | | | | | + | | | o | | | | | | openssl + | | |/ / / / / / / + | |/| | | | | | | + | | o | | | | | | hdf5 + | |/| | | | | | | + | | |/ / / / / / + | o | | | | | | zlib + | / / / / / / + o | | | | | | swig + o | | | | | | pcre + / / / / / / + o | | | | | mpi + / / / / / + o | | | | matio + / / / / + o | | | lapack + / / / + o | | glm + / / + o | boost + / + o blas + + +You can control how the output is displayed with a number of options. + +The ASCII output from ``spack graph`` can be difficult to parse for +complicated packages. The output can be changed to the ``graphviz`` +``.dot`` format using the `--dot` flag. + +.. code-block:: console + + $ spack graph --dot trilinos | dot -Tpdf trilinos_graph.pdf + +.. _basics-tutorial-uninstall: + +--------------------- +Uninstalling Packages +--------------------- + +Earlier we installed many configurations each of libelf and +libdwarf. Now we will go through and uninstall some of those packages +that we didn't really need. + +.. code-block:: console + + $ spack find -d libdwarf + ==> 3 installed packages. + -- linux-redhat6-x86_64 / gcc@4.4.7 ----------------------------- + libdwarf@20160507 + ^libelf@0.8.12 + + libdwarf@20160507 + ^libelf@0.8.13 + + + -- linux-redhat6-x86_64 / intel@16.0.3 -------------------------- + libdwarf@20160507 + ^libelf@0.8.12 + + $ spack find libelf + ==> 6 installed packages. + -- linux-redhat6-x86_64 / gcc@4.4.7 ----------------------------- + libelf@0.8.12 libelf@0.8.12 libelf@0.8.13 + + -- linux-redhat6-x86_64 / intel@15.0.4 -------------------------- + libelf@0.8.13 + + -- linux-redhat6-x86_64 / intel@16.0.3 -------------------------- + libelf@0.8.12 libelf@0.8.13 + + +We can uninstall packages by spec using the same syntax as install. + +.. code-block:: console + + $ spack uninstall libelf%intel@15.0.4 + ==> The following packages will be uninstalled : + + -- linux-redhat6-x86_64 / intel@15.0.4 -------------------------- + w33hrej libelf@0.8.13%intel + + + ==> Do you want to proceed ? [y/n] + y + ==> Successfully uninstalled libelf@0.8.13%intel@15.0.4 arch=linux-redhat6-x86_64-w33hrej + + + + $ spack find -lf libelf + ==> 5 installed packages. + -- linux-redhat6-x86_64 / gcc@4.4.7 ----------------------------- + ipggckv libelf@0.8.12%gcc + + vrv2ttb libelf@0.8.12%gcc cppflags="-O3" + + csrt4qx libelf@0.8.13%gcc + + + -- linux-redhat6-x86_64 / intel@16.0.3 -------------------------- + 4blbe3q libelf@0.8.12%intel + + 7wgp32x libelf@0.8.13%intel + + +We can uninstall packages by referring only to their hash. + + +We can use either ``-f`` (force) or ``-d`` (remove dependents as well) to +remove packages that are required by another installed package. + + + $ spack uninstall /4blb + ==> Error: Will not uninstall libelf@0.8.12%intel@16.0.3-4blbe3q + + The following packages depend on it: + -- linux-redhat6-x86_64 / intel@16.0.3 -------------------------- + csruprg libdwarf@20160507%intel + + + ==> Error: You can use spack uninstall --dependents to uninstall these dependencies as well + $ spack uninstall -d /4blb + ==> The following packages will be uninstalled : + + -- linux-redhat6-x86_64 / intel@16.0.3 -------------------------- + csruprg libdwarf@20160507%intel + + 4blbe3q libelf@0.8.12%intel + + + ==> Do you want to proceed ? [y/n] + y + ==> Successfully uninstalled libdwarf@20160507%intel@16.0.3 arch=linux-redhat6-x86_64-csruprg + ==> Successfully uninstalled libelf@0.8.12%intel@16.0.3 arch=linux-redhat6-x86_64-4blbe3q + + +Spack will not uninstall packages that are not sufficiently +specified. The ``-a`` (all) flag can be used to uninstall multiple +packages at once. + + + $ spack uninstall trilinos + ==> Error: trilinos matches multiple packages: + + -- linux-redhat6-x86_64 / gcc@4.4.7 ----------------------------- + ffwrpxn trilinos@12.8.1%gcc+boost~debug+hdf5+hypre+metis+mumps~python+shared+suite-sparse+superlu-dist + + uvd6dfd trilinos@12.8.1%gcc+boost~debug+hdf5+hypre+metis+mumps~python+shared+suite-sparse+superlu-dist + + + ==> Error: You can either: + a) Use a more specific spec, or + b) use spack uninstall -a to uninstall ALL matching specs. + + + + $ spack uninstall /ffwr + ==> The following packages will be uninstalled : + + -- linux-redhat6-x86_64 / gcc@4.4.7 ----------------------------- + ffwrpxn trilinos@12.8.1%gcc+boost~debug+hdf5+hypre+metis+mumps~python+shared+suite-sparse+superlu-dist + + + ==> Do you want to proceed ? [y/n] + y + ==> Successfully uninstalled trilinos@12.8.1%gcc@4.4.7+boost~debug+hdf5+hypre+metis+mumps~python+shared+suite-sparse+superlu-dist arch=linux-redhat6-x86_64-ffwrpxn + +----------------------------- +Advanced ``spack find`` Usage +----------------------------- + +We will go over some additional uses for the `spack find` command not +already covered in the :ref:`basics-tutorial-install` and +:ref:`basics-tutorial-uninstall` sections. + +The ``spack find`` command can accept what we call "anonymous specs." +These are expressions in spec syntax that do not contain a package +name. For example, `spack find %intel` will return every package built +with the intel compiler, and ``spack find cppflags=\\"-O3\\"`` will +return every package which was built with ``cppflags=\\"-O3\\"``. + +.. code-block:: console + + $ spack find %intel + ==> 1 installed packages. + -- linux-redhat6-x86_64 / intel@16.0.3 -------------------------- + libelf@0.8.13 + + + + $ spack find cppflags=\"-O3\" + ==> 1 installed packages. + -- linux-redhat6-x86_64 / gcc@4.4.7 ----------------------------- + libelf@0.8.12 + + +The ``find`` command can also show which packages were installed +explicitly (rather than pulled in as a dependency) using the ``-e`` +flag. The ``-E`` flag shows implicit installs only. The ``find`` command can +also show the path to which a spack package was installed using the ``-p`` +command. + +.. code-block:: console + + $ spack find -pe + ==> 10 installed packages. + -- linux-redhat6-x86_64 / gcc@4.4.7 ----------------------------- + hdf5@1.10.0-patch1 /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/hdf5-1.10.0-patch1-twppaioxqn6lti4grgopnmhwcq3h2rpw + hdf5@1.10.0-patch1 /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/hdf5-1.10.0-patch1-j36yfw25i6gdd3q4vwlupgkpwic4ua6m + hdf5@1.10.0-patch1 /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/hdf5-1.10.0-patch1-ezvtnox35albuaxqryuondweyjgeo6es + libdwarf@20160507 /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libdwarf-20160507-dtg3tgnp7htccoly26gduqlrgvnwcp5t + libdwarf@20160507 /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libdwarf-20160507-yfx6p3g3rkmqvcqbmtb34o6pln7pqvcz + libelf@0.8.12 /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libelf-0.8.12-ipggckv6i7h44iryzfa4dwdela32a7fy + libelf@0.8.12 /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libelf-0.8.12-vrv2ttbd34xlfoxy4jwt6qsjrcbalmmw + libelf@0.8.13 /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/libelf-0.8.13-csrt4qxfkhjgn5xg3zjpkir7xdnszl2a + trilinos@12.8.1 /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/trilinos-12.8.1-uvd6dfd7x4uyvck4awo3r3frudihn4ar + + -- linux-redhat6-x86_64 / intel@16.0.3 -------------------------- + libelf@0.8.13 /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/intel-16.0.3/libelf-0.8.13-7wgp32xksatkvw2tbssmehw2t5tnxndj + + +--------------------- +Customizing Compilers +--------------------- + + +Spack manages a list of available compilers on the system, detected +automatically from from the user's ``PATH`` variable. The ``spack +compilers`` command is an alias for the command ``spack compiler list``. + +.. code-block:: console + + $ spack compilers + ==> Available compilers + -- gcc ---------------------------------------------------------- + gcc@4.4.7 + + -- intel -------------------------------------------------------- + intel@16.0.3 intel@15.0.1 intel@14.0.0 intel@12.1.3 intel@10.0 + intel@16.0.2 intel@15.0.0 intel@13.1.1 intel@12.1.2 intel@9.1 + intel@16.0.1 intel@14.0.4 intel@13.1.0 intel@12.1.0 + intel@16.0.0 intel@14.0.3 intel@13.0.1 intel@12.0.4 + intel@15.0.4 intel@14.0.2 intel@13.0.0 intel@11.1 + intel@15.0.3 intel@14.0.1 intel@12.1.5 intel@10.1 + + -- pgi ---------------------------------------------------------- + pgi@16.5-0 pgi@15.7-0 pgi@14.7-0 pgi@13.2-0 pgi@11.10-0 pgi@9.0-4 + pgi@16.3-0 pgi@15.5-0 pgi@14.3-0 pgi@13.1-1 pgi@11.1-0 pgi@8.0-1 + pgi@16.1-0 pgi@15.1-0 pgi@13.10-0 pgi@12.8-0 pgi@10.9-0 pgi@7.1-3 + pgi@15.10-0 pgi@14.10-0 pgi@13.6-0 pgi@12.1-0 pgi@10.2-0 pgi@7.0-6 + +The compilers are maintained in a YAML file that can be hand-edited +for special cases. Spack also has tools to add compilers, and +compilers built with Spack can be added to the configuration. + +.. code-block:: console + + $ spack install --fake gcc@6.1.0 + ==> Installing gcc + ==> gmp is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/gmp-6.1.1-uwn4gfdtq3sywy5uf4f7znrh66oybikf + ==> Installing isl + ==> gmp is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/gmp-6.1.1-uwn4gfdtq3sywy5uf4f7znrh66oybikf + ==> Building isl [Package] + ==> Successfully installed isl + Fetch: . Build: 0.19s. Total: 0.19s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/isl-0.14-hs2w7mjjjaakkmbbv5yvfqf7yyzhorl6 + ==> Installing mpc + ==> gmp is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/gmp-6.1.1-uwn4gfdtq3sywy5uf4f7znrh66oybikf + ==> Installing mpfr + ==> gmp is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/gmp-6.1.1-uwn4gfdtq3sywy5uf4f7znrh66oybikf + ==> Building mpfr [Package] + ==> Successfully installed mpfr + Fetch: . Build: 0.17s. Total: 0.17s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/mpfr-3.1.4-7kt5ij437khredfq4bvnyu22t3fmtfvt + ==> Building mpc [Package] + ==> Successfully installed mpc + Fetch: . Build: 0.28s. Total: 0.28s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/mpc-1.0.3-g5taq6lt3zuy5l2jtggi5lctxnl4la5u + ==> Installing binutils + ==> m4 is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/m4-1.4.17-qijdzvhjyybrtwbqm73vykhmkaqro3je + ==> Installing bison + ==> m4 is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/m4-1.4.17-qijdzvhjyybrtwbqm73vykhmkaqro3je + ==> Building bison [Package] + ==> Successfully installed bison + Fetch: . Build: 0.12s. Total: 0.12s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/bison-3.0.4-hkhfysfvq5l6rsns67g2htmkpxauvnwa + ==> Installing flex + ==> m4 is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/m4-1.4.17-qijdzvhjyybrtwbqm73vykhmkaqro3je + ==> bison is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/bison-3.0.4-hkhfysfvq5l6rsns67g2htmkpxauvnwa + ==> Building flex [Package] + ==> Successfully installed flex + Fetch: . Build: 0.11s. Total: 0.11s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/flex-2.6.0-qd6d73rdfrozdrsdpimvl4tj7d5ps7qg + ==> Building binutils [Package] + ==> Successfully installed binutils + Fetch: . Build: 0.11s. Total: 0.11s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/binutils-2.27-iq2hry3gvaxszmwwbnll7njgdgaek56o + ==> mpfr is already installed in /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/mpfr-3.1.4-7kt5ij437khredfq4bvnyu22t3fmtfvt + ==> Building gcc [Package] + ==> Successfully installed gcc + Fetch: . Build: 0.66s. Total: 0.66s. + [+] /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/gcc-6.1.0-j5576zbsot2ydljlthjzhsirsesnogvh + + + + $ spack find -p gcc + ==> 1 installed packages. + -- linux-redhat6-x86_64 / gcc@4.4.7 ----------------------------- + gcc@6.1.0 /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/gcc-6.1.0-j5576zbsot2ydljlthjzhsirsesnogvh + + +If we had done a "real" install of gcc, we could add it to our +configuration now using the `spack compiler add` command, but we would +also be waiting for it to install. If we run the command now, it will +return no new compilers. + +.. code-block:: console + + $ spack compiler add /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/gcc-6.1.0-j5576zbsot2ydljlthjzhsirsesnogvh/bin + ==> Found no new compilers + +If we had done a real install, the output would have been as follows: + +.. code-block:: console + + $ spack compiler add /g/g0/becker33/SC16-spack/spack/opt/spack/linux-redhat6-x86_64/gcc-4.4.7/gcc-6.1.0-j5576zbsot2ydljlthjzhsirsesnogvh/bin + ==> Added 1 new compiler to /g/g0/becker33/.spack/linux/compilers.yaml + gcc@6.1.0 -- cgit v1.2.3-60-g2f50