diff options
| author | Todd Gamblin <tgamblin@llnl.gov> | 2019-10-25 21:49:27 -0700 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2019-10-25 21:49:27 -0700 |
| commit | 847703c13d0fd69be0b06d355308c47f4a060e66 (patch) | |
| tree | 5061dc9f0873408ceef7ff25d57b394ac522189f /lib | |
| parent | 4af448724f4f933b5306c4cdd2f7522d64dc8719 (diff) | |
| download | spack-847703c13d0fd69be0b06d355308c47f4a060e66.tar.gz spack-847703c13d0fd69be0b06d355308c47f4a060e66.tar.bz2 spack-847703c13d0fd69be0b06d355308c47f4a060e66.tar.xz spack-847703c13d0fd69be0b06d355308c47f4a060e66.zip | |
tutorial: move tutorial to standalone site (#13450)
* docs: add a spack environment for building the docs
* docs: remove tutorial and link to spack-tutorial.readthedocs.io
The tutorial now has its own standalone website, versioned by instances
of the tutorial. Link to that instead of versioning it directly with Spack.
Diffstat (limited to 'lib')
31 files changed, 28 insertions, 7565 deletions
diff --git a/lib/spack/docs/.gitignore b/lib/spack/docs/.gitignore index 0b6258af50..1624dfeaea 100644 --- a/lib/spack/docs/.gitignore +++ b/lib/spack/docs/.gitignore @@ -3,3 +3,5 @@ command_index.rst spack*.rst llnl*.rst _build +.spack-env +spack.lock diff --git a/lib/spack/docs/build_systems/intelpackage.rst b/lib/spack/docs/build_systems/intelpackage.rst index 30c567373e..f7d975899e 100644 --- a/lib/spack/docs/build_systems/intelpackage.rst +++ b/lib/spack/docs/build_systems/intelpackage.rst @@ -120,7 +120,7 @@ version numbers seen with most other Spack packages. For example, we have: ... Preferred version: professional.2018.3 http:... - + Safe versions: professional.2018.3 http:... ... @@ -728,7 +728,7 @@ For packages that contain a compiler, follow `the previous section .. code-block:: console - $ spack install intel-mpi@2018.3.199 + $ spack install intel-mpi@2018.3.199 $ spack install intel-mpi@2018.3.199 %intel@18 4. To prepare the new packages for use with client packages, @@ -802,7 +802,7 @@ by one of the following means: Configure the order of compilers in the appropriate ``packages.yaml`` file, under either an ``all:`` or client-package-specific entry, in a ``compiler:`` list. Consult the Spack documentation for - :ref:`Configuring Package Preferences <configs-tutorial-package-prefs>` + `Configuring Package Preferences <https://spack-tutorial.readthedocs.io/en/latest/tutorial_configuration.html#configuring-package-preferences>`_ and :ref:`Concretization Preferences <concretization-preferences>`. @@ -851,7 +851,7 @@ client packages, edit the ``packages.yaml`` file. Customize, either in the the virtual packages and whose values are the Spack specs that satisfy the virtual package, in order of decreasing preference. To learn more about the ``providers:`` settings, see the Spack tutorial for -:ref:`Configuring Package Preferences <configs-tutorial-package-prefs>` +`Configuring Package Preferences <https://spack-tutorial.readthedocs.io/en/latest/tutorial_configuration.html#configuring-package-preferences>`_ and the section :ref:`Concretization Preferences <concretization-preferences>`. @@ -972,7 +972,7 @@ a *virtual* ``mkl`` package is declared in Spack. .. code-block:: python self.spec['blas'].headers.include_flags - + and to generate linker options (``-L<dir> -llibname ...``), use the same as above, .. code-block:: python diff --git a/lib/spack/docs/conf.py b/lib/spack/docs/conf.py index cacbbb3a41..e5305a9317 100644 --- a/lib/spack/docs/conf.py +++ b/lib/spack/docs/conf.py @@ -159,7 +159,7 @@ gettext_uuid = False # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. -exclude_patterns = ['_build', '_spack_root'] +exclude_patterns = ['_build', '_spack_root', '.spack-env'] # The reST default role (used for this markup: `text`) to use for all documents. #default_role = None diff --git a/lib/spack/docs/index.rst b/lib/spack/docs/index.rst index 5c93285cbb..48af67982a 100644 --- a/lib/spack/docs/index.rst +++ b/lib/spack/docs/index.rst @@ -55,7 +55,7 @@ or refer to the full manual below. getting_started basic_usage workflows - tutorial + Tutorial: Spack 101 <https://spack-tutorial.readthedocs.io> known_issues .. toctree:: diff --git a/lib/spack/docs/spack.yaml b/lib/spack/docs/spack.yaml new file mode 100644 index 0000000000..bade961fce --- /dev/null +++ b/lib/spack/docs/spack.yaml @@ -0,0 +1,19 @@ +# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other +# Spack Project Developers. See the top-level COPYRIGHT file for details. +# +# SPDX-License-Identifier: (Apache-2.0 OR MIT) + +# +# These are requirements for building the documentation. You can run +# these commands in this directory to install Sphinx and its plugins, +# then build the docs: +# +# spack install +# spack env activate . +# make +# +spack: + specs: + - py-sphinx + - py-sphinxcontrib-programoutput + - py-sphinx-rtd-theme diff --git a/lib/spack/docs/tutorial.rst b/lib/spack/docs/tutorial.rst deleted file mode 100644 index 4ece87ade0..0000000000 --- a/lib/spack/docs/tutorial.rst +++ /dev/null @@ -1,75 +0,0 @@ -.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other - Spack Project Developers. See the top-level COPYRIGHT file for details. - - SPDX-License-Identifier: (Apache-2.0 OR MIT) - -.. _spack-101: - -============================= -Tutorial: Spack 101 -============================= - -This is a full-day introduction to Spack with lectures and live demos. -It was last presented at the `Practice and Experience in Advanced -Research Computing Conference (PEARC19) -<https://www.pearc19.pearc.org/>`_ on July 31, 2019. - -You can use these materials to teach a course on Spack at your own site, -or you can just skip ahead and read the live demo scripts to see how -Spack is used in practice. - -.. _sc16-slides: - -.. rubric:: Slides - -.. figure:: tutorial/sc16-tutorial-slide-preview.png - :target: https://spack.io/slides/spack-pearc19-tutorial-slides.pdf - :height: 72px - :align: left - :alt: Slide Preview - -`Download Slides <https://spack.io/slides/spack-pearc19-tutorial-slides.pdf>`_. - -**Full citation:** Levi Baber, Gregory Becker, Adam J. Stewart, and Todd -Gamblin. Managing HPC Software Complexity with Spack. Tutorial presented -at the Practice and Experience in Advanced Research Computing Conference -(PEARC19). July 31, 2019. Chicago, IL, USA. - -.. _sc16-live-demos: - -.. rubric:: Live Demos - -We provide scripts that take you step-by-step through basic Spack tasks. -They correspond to sections in the slides above. You can use one of the -following methods to run through the scripts: - - 1. We provide the `spack/tutorial - <https://hub.docker.com/r/spack/tutorial>`_ container image on - Docker Hub that you can use to do the tutorial on your local - machine. You can invoke ``docker run -it spack/tutorial`` to start - using the container. - - 2. When we host the tutorial, we also provision VM instances in `AWS - <https://aws.amazon.com/>`_, so that users who are unfamiliar with - Docker can simply log into a VPM to do the demo exercises. - -You should now be ready to run through our demo scripts: - - 1. :ref:`basics-tutorial` - 2. :ref:`configs-tutorial` - 3. :ref:`packaging-tutorial` - 4. :ref:`environments-tutorial` - 5. :ref:`modules-tutorial` - 6. :ref:`build-systems-tutorial` - 7. :ref:`advanced-packaging-tutorial` - -Full contents: - -.. toctree:: - tutorial_basics - tutorial_configuration - tutorial_packaging - tutorial_environments - tutorial_modules - tutorial_buildsystems - tutorial_advanced_packaging diff --git a/lib/spack/docs/tutorial/examples/0.package.py b/lib/spack/docs/tutorial/examples/0.package.py deleted file mode 100644 index 990ba7d3d8..0000000000 --- a/lib/spack/docs/tutorial/examples/0.package.py +++ /dev/null @@ -1,39 +0,0 @@ -# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other -# Spack Project Developers. See the top-level COPYRIGHT file for details. -# -# SPDX-License-Identifier: (Apache-2.0 OR MIT) - -# -# This is a template package file for Spack. We've put "FIXME" -# next to all the things you'll want to change. Once you've handled -# them, you can save this file and test your package like this: -# -# spack install mpileaks -# -# You can edit this file again by typing: -# -# spack edit mpileaks -# -# See the Spack documentation for more information on packaging. -# If you submit this package back to Spack as a pull request, -# please first remove this boilerplate and all FIXME comments. -# -from spack import * - - -class Mpileaks(Package): - """FIXME: Put a proper description of your package here.""" - - # FIXME: Add a proper url for your package's homepage here. - homepage = "http://www.example.com" - url = "https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz" - - version('1.0', sha256='2e34cc4505556d1c1f085758e26f2f8eea0972db9382f051b2dcfb1d7d9e1825') - - # FIXME: Add dependencies if required. - # depends_on('foo') - - def install(self, spec, prefix): - # FIXME: Unknown build system - make() - make('install') diff --git a/lib/spack/docs/tutorial/examples/1.package.py b/lib/spack/docs/tutorial/examples/1.package.py deleted file mode 100644 index f2d515fa4a..0000000000 --- a/lib/spack/docs/tutorial/examples/1.package.py +++ /dev/null @@ -1,24 +0,0 @@ -# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other -# Spack Project Developers. See the top-level COPYRIGHT file for details. -# -# SPDX-License-Identifier: (Apache-2.0 OR MIT) - -from spack import * - - -class Mpileaks(Package): - """Tool to detect and report MPI objects like MPI_Requests and - MPI_Datatypes.""" - - homepage = "https://github.com/LLNL/mpileaks" - url = "https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz" - - version('1.0', sha256='2e34cc4505556d1c1f085758e26f2f8eea0972db9382f051b2dcfb1d7d9e1825') - - # FIXME: Add dependencies if required. - # depends_on('foo') - - def install(self, spec, prefix): - # FIXME: Unknown build system - make() - make('install') diff --git a/lib/spack/docs/tutorial/examples/2.package.py b/lib/spack/docs/tutorial/examples/2.package.py deleted file mode 100644 index d0f1e6d062..0000000000 --- a/lib/spack/docs/tutorial/examples/2.package.py +++ /dev/null @@ -1,25 +0,0 @@ -# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other -# Spack Project Developers. See the top-level COPYRIGHT file for details. -# -# SPDX-License-Identifier: (Apache-2.0 OR MIT) - -from spack import * - - -class Mpileaks(Package): - """Tool to detect and report MPI objects like MPI_Requests and - MPI_Datatypes.""" - - homepage = "https://github.com/LLNL/mpileaks" - url = "https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz" - - version('1.0', sha256='2e34cc4505556d1c1f085758e26f2f8eea0972db9382f051b2dcfb1d7d9e1825') - - depends_on('mpi') - depends_on('adept-utils') - depends_on('callpath') - - def install(self, spec, prefix): - # FIXME: Unknown build system - make() - make('install') diff --git a/lib/spack/docs/tutorial/examples/3.package.py b/lib/spack/docs/tutorial/examples/3.package.py deleted file mode 100644 index 05fd7aa00a..0000000000 --- a/lib/spack/docs/tutorial/examples/3.package.py +++ /dev/null @@ -1,25 +0,0 @@ -# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other -# Spack Project Developers. See the top-level COPYRIGHT file for details. -# -# SPDX-License-Identifier: (Apache-2.0 OR MIT) - -from spack import * - - -class Mpileaks(Package): - """Tool to detect and report MPI objects like MPI_Requests and - MPI_Datatypes.""" - - homepage = "https://github.com/LLNL/mpileaks" - url = "https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz" - - version('1.0', sha256='2e34cc4505556d1c1f085758e26f2f8eea0972db9382f051b2dcfb1d7d9e1825') - - depends_on('mpi') - depends_on('adept-utils') - depends_on('callpath') - - def install(self, spec, prefix): - configure() - make() - make('install') diff --git a/lib/spack/docs/tutorial/examples/4.package.py b/lib/spack/docs/tutorial/examples/4.package.py deleted file mode 100644 index 626e7d3fec..0000000000 --- a/lib/spack/docs/tutorial/examples/4.package.py +++ /dev/null @@ -1,27 +0,0 @@ -# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other -# Spack Project Developers. See the top-level COPYRIGHT file for details. -# -# SPDX-License-Identifier: (Apache-2.0 OR MIT) - -from spack import * - - -class Mpileaks(Package): - """Tool to detect and report MPI objects like MPI_Requests and - MPI_Datatypes.""" - - homepage = "https://github.com/LLNL/mpileaks" - url = "https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz" - - version('1.0', sha256='2e34cc4505556d1c1f085758e26f2f8eea0972db9382f051b2dcfb1d7d9e1825') - - depends_on('mpi') - depends_on('adept-utils') - depends_on('callpath') - - def install(self, spec, prefix): - configure('--prefix={0}'.format(prefix), - '--with-adept-utils={0}'.format(spec['adept-utils'].prefix), - '--with-callpath={0}'.format(spec['callpath'].prefix)) - make() - make('install') diff --git a/lib/spack/docs/tutorial/examples/5.package.py b/lib/spack/docs/tutorial/examples/5.package.py deleted file mode 100644 index 9a1dec2079..0000000000 --- a/lib/spack/docs/tutorial/examples/5.package.py +++ /dev/null @@ -1,42 +0,0 @@ -# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other -# Spack Project Developers. See the top-level COPYRIGHT file for details. -# -# SPDX-License-Identifier: (Apache-2.0 OR MIT) - -from spack import * - - -class Mpileaks(Package): - """Tool to detect and report MPI objects like MPI_Requests and - MPI_Datatypes.""" - - homepage = "https://github.com/LLNL/mpileaks" - url = "https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz" - - version('1.0', sha256='2e34cc4505556d1c1f085758e26f2f8eea0972db9382f051b2dcfb1d7d9e1825') - - variant('stackstart', values=int, default=0, - description='Specify the number of stack frames to truncate') - - depends_on('mpi') - depends_on('adept-utils') - depends_on('callpath') - - def install(self, spec, prefix): - stackstart = int(spec.variants['stackstart'].value) - - args = [ - '--prefix={0}'.format(prefix), - '--with-adept-utils={0}'.format(spec['adept-utils'].prefix), - '--with-callpath={0}'.format(spec['callpath'].prefix), - ] - - if stackstart: - args.extend([ - '--with-stack-start-c={0}'.format(stackstart), - '--with-stack-start-fortran={0}'.format(stackstart) - ]) - - configure(*args) - make() - make('install') diff --git a/lib/spack/docs/tutorial/examples/Autotools/0.package.py b/lib/spack/docs/tutorial/examples/Autotools/0.package.py deleted file mode 100644 index 213965e5c6..0000000000 --- a/lib/spack/docs/tutorial/examples/Autotools/0.package.py +++ /dev/null @@ -1,27 +0,0 @@ -# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other -# Spack Project Developers. See the top-level COPYRIGHT file for details. -# -# SPDX-License-Identifier: (Apache-2.0 OR MIT) - -from spack import * - - -class Mpileaks(AutotoolsPackage): - """Tool to detect and report leaked MPI objects like MPI_Requests and - MPI_Datatypes.""" - - homepage = "https://github.com/hpc/mpileaks" - url = "https://github.com/hpc/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz" - - version('1.0', '8838c574b39202a57d7c2d68692718aa') - - depends_on("mpi") - depends_on("adept-utils") - depends_on("callpath") - - def install(self, spec, prefix): - configure("--prefix=" + prefix, - "--with-adept-utils=" + spec['adept-utils'].prefix, - "--with-callpath=" + spec['callpath'].prefix) - make() - make("install") diff --git a/lib/spack/docs/tutorial/examples/Autotools/1.package.py b/lib/spack/docs/tutorial/examples/Autotools/1.package.py deleted file mode 100644 index 639f130596..0000000000 --- a/lib/spack/docs/tutorial/examples/Autotools/1.package.py +++ /dev/null @@ -1,32 +0,0 @@ -# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other -# Spack Project Developers. See the top-level COPYRIGHT file for details. -# -# SPDX-License-Identifier: (Apache-2.0 OR MIT) - -from spack import * - - -class Mpileaks(AutotoolsPackage): - """Tool to detect and report leaked MPI objects like MPI_Requests and - MPI_Datatypes.""" - - homepage = "https://github.com/hpc/mpileaks" - url = "https://github.com/hpc/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz" - - version('1.0', '8838c574b39202a57d7c2d68692718aa') - - variant("stackstart", values=int, default=0, - description="Specify the number of stack frames to truncate") - - depends_on("mpi") - depends_on("adept-utils") - depends_on("callpath") - - def configure_args(self): - stackstart = int(self.spec.variants['stackstart'].value) - args = ["--with-adept-utils=" + spec['adept-utils'].prefix, - "--with-callpath=" + spec['callpath'].prefix] - if stackstart: - args.extend(['--with-stack-start-c=%s' % stackstart, - '--with-stack-start-fortran=%s' % stackstart]) - return args diff --git a/lib/spack/docs/tutorial/examples/Cmake/0.package.py b/lib/spack/docs/tutorial/examples/Cmake/0.package.py deleted file mode 100644 index b8c96a1929..0000000000 --- a/lib/spack/docs/tutorial/examples/Cmake/0.package.py +++ /dev/null @@ -1,41 +0,0 @@ -# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other -# Spack Project Developers. See the top-level COPYRIGHT file for details. -# -# SPDX-License-Identifier: (Apache-2.0 OR MIT) - -# -# This is a template package file for Spack. We've put "FIXME" -# next to all the things you'll want to change. Once you've handled -# them, you can save this file and test your package like this: -# -# spack install callpath -# -# You can edit this file again by typing: -# -# spack edit callpath -# -# See the Spack documentation for more information on packaging. -# If you submit this package back to Spack as a pull request, -# please first remove this boilerplate and all FIXME comments. -# -from spack import * - - -class Callpath(CMakePackage): - """FIXME: Put a proper description of your package here.""" - - # FIXME: Add a proper url for your package's homepage here. - homepage = "http://www.example.com" - url = "https://github.com/llnl/callpath/archive/v1.0.1.tar.gz" - - version('1.0.3', 'c89089b3f1c1ba47b09b8508a574294a') - - # FIXME: Add dependencies if required. - # depends_on('foo') - - def cmake_args(self): - # FIXME: Add arguments other than - # FIXME: CMAKE_INSTALL_PREFIX and CMAKE_BUILD_TYPE - # FIXME: If not needed delete this function - args = [] - return args diff --git a/lib/spack/docs/tutorial/examples/Cmake/1.package.py b/lib/spack/docs/tutorial/examples/Cmake/1.package.py deleted file mode 100644 index 8a0d9d0e82..0000000000 --- a/lib/spack/docs/tutorial/examples/Cmake/1.package.py +++ /dev/null @@ -1,23 +0,0 @@ -# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other -# Spack Project Developers. See the top-level COPYRIGHT file for details. -# -# SPDX-License-Identifier: (Apache-2.0 OR MIT) - -from spack import * - - -class Callpath(CMakePackage): - """Library for representing callpaths consistently in - distributed-memory performance tools.""" - - homepage = "https://github.com/llnl/callpath" - url = "https://github.com/llnl/callpath/archive/v1.0.3.tar.gz" - - version('1.0.3', 'c89089b3f1c1ba47b09b8508a574294a') - - depends_on("elf", type="link") - depends_on("libdwarf") - depends_on("dyninst") - depends_on("adept-utils") - depends_on("mpi") - depends_on("cmake@2.8:", type="build") diff --git a/lib/spack/docs/tutorial/examples/Cmake/2.package.py b/lib/spack/docs/tutorial/examples/Cmake/2.package.py deleted file mode 100644 index 2b4faca90d..0000000000 --- a/lib/spack/docs/tutorial/examples/Cmake/2.package.py +++ /dev/null @@ -1,33 +0,0 @@ -# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other -# Spack Project Developers. See the top-level COPYRIGHT file for details. -# -# SPDX-License-Identifier: (Apache-2.0 OR MIT) - -from spack import * - - -class Callpath(CMakePackage): - """Library for representing callpaths consistently in - distributed-memory performance tools.""" - - homepage = "https://github.com/llnl/callpath" - url = "https://github.com/llnl/callpath/archive/v1.0.3.tar.gz" - - version('1.0.3', 'c89089b3f1c1ba47b09b8508a574294a') - - depends_on("elf", type="link") - depends_on("libdwarf") - depends_on("dyninst") - depends_on("adept-utils") - depends_on("mpi") - depends_on("cmake@2.8:", type="build") - - def cmake_args(self): - args = ["-DCALLPATH_WALKER=dyninst"] - - if self.spec.satisfies("^dyninst@9.3.0:"): - std.flag = self.compiler.cxx_flag - args.append("-DCMAKE_CXX_FLAGS='{0}' -fpermissive'".format( - std_flag)) - - return args diff --git a/lib/spack/docs/tutorial/examples/Makefile/0.package.py b/lib/spack/docs/tutorial/examples/Makefile/0.package.py deleted file mode 100644 index 8ad252f5b3..0000000000 --- a/lib/spack/docs/tutorial/examples/Makefile/0.package.py +++ /dev/null @@ -1,26 +0,0 @@ -# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other -# Spack Project Developers. See the top-level COPYRIGHT file for details. -# -# SPDX-License-Identifier: (Apache-2.0 OR MIT) - -from spack import * - - -class Bowtie(MakefilePackage): - """FIXME: Put a proper description of your package here.""" - - # FIXME: Add a proper url for your package's homepage here. - homepage = "http://www.example.com" - url = "https://downloads.sourceforge.net/project/bowtie-bio/bowtie/1.2.1.1/bowtie-1.2.1.1-src.zip" - - version('1.2.1.1', 'ec06265730c5f587cd58bcfef6697ddf') - - # FIXME: Add dependencies if required. - # depends_on('foo') - - def edit(self, spec, prefix): - # FIXME: Edit the Makefile if necessary - # FIXME: If not needed delete this function - # makefile = FileFilter('Makefile') - # makefile.filter('CC = .*', 'CC = cc') - return diff --git a/lib/spack/docs/tutorial/examples/Makefile/1.package.py b/lib/spack/docs/tutorial/examples/Makefile/1.package.py deleted file mode 100644 index 16988dc1b0..0000000000 --- a/lib/spack/docs/tutorial/examples/Makefile/1.package.py +++ /dev/null @@ -1,27 +0,0 @@ -# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other -# Spack Project Developers. See the top-level COPYRIGHT file for details. -# -# SPDX-License-Identifier: (Apache-2.0 OR MIT) - -from spack import * - - -class Bowtie(MakefilePackage): - """Bowtie is an ultrafast, memory efficient short read aligner - for short DNA sequences (reads) from next-gen sequencers.""" - - homepage = "https://sourceforge.net/projects/bowtie-bio/" - url = "https://downloads.sourceforge.net/project/bowtie-bio/bowtie/1.2.1.1/bowtie-1.2.1.1-src.zip" - - version('1.2.1.1', 'ec06265730c5f587cd58bcfef6697ddf') - - variant("tbb", default=False, description="Use Intel thread building block") - - depends_on("tbb", when="+tbb") - - def edit(self, spec, prefix): - # FIXME: Edit the Makefile if necessary - # FIXME: If not needed delete this function - # makefile = FileFilter('Makefile') - # makefile.filter('CC = .*', 'CC = cc') - return diff --git a/lib/spack/docs/tutorial/examples/Makefile/2.package.py b/lib/spack/docs/tutorial/examples/Makefile/2.package.py deleted file mode 100644 index 0973b2028e..0000000000 --- a/lib/spack/docs/tutorial/examples/Makefile/2.package.py +++ /dev/null @@ -1,25 +0,0 @@ -# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other -# Spack Project Developers. See the top-level COPYRIGHT file for details. -# -# SPDX-License-Identifier: (Apache-2.0 OR MIT) - -from spack import * - - -class Bowtie(MakefilePackage): - """Bowtie is an ultrafast, memory efficient short read aligner - for short DNA sequences (reads) from next-gen sequencers.""" - - homepage = "https://sourceforge.net/projects/bowtie-bio/" - url = "https://downloads.sourceforge.net/project/bowtie-bio/bowtie/1.2.1.1/bowtie-1.2.1.1-src.zip" - - version('1.2.1.1', 'ec06265730c5f587cd58bcfef6697ddf') - - variant("tbb", default=False, description="Use Intel thread building block") - - depends_on("tbb", when="+tbb") - - def edit(self, spec, prefix): - makefile = FileFilter("Makefile") - makefile.filter('CC= .*', 'CC = ' + env['CC']) - makefile.filter('CXX = .*', 'CXX = ' + env['CXX']) diff --git a/lib/spack/docs/tutorial/examples/Makefile/3.package.py b/lib/spack/docs/tutorial/examples/Makefile/3.package.py deleted file mode 100644 index 6a1942e627..0000000000 --- a/lib/spack/docs/tutorial/examples/Makefile/3.package.py +++ /dev/null @@ -1,36 +0,0 @@ -# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other -# Spack Project Developers. See the top-level COPYRIGHT file for details. -# -# SPDX-License-Identifier: (Apache-2.0 OR MIT) - -from spack import * - - -class Bowtie(MakefilePackage): - """Bowtie is an ultrafast, memory efficient short read aligner - for short DNA sequences (reads) from next-gen sequencers.""" - - homepage = "https://sourceforge.net/projects/bowtie-bio/" - url = "https://downloads.sourceforge.net/project/bowtie-bio/bowtie/1.2.1.1/bowtie-1.2.1.1-src.zip" - - version('1.2.1.1', 'ec06265730c5f587cd58bcfef6697ddf') - - variant("tbb", default=False, description="Use Intel thread building block") - - depends_on("tbb", when="+tbb") - - def edit(self, spec, prefix): - makefile = FileFilter("Makefile") - makefile.filter('CC= .*', 'CC = ' + env['CC']) - makefile.filter('CXX = .*', 'CXX = ' + env['CXX']) - - @property - def build_targets(self): - if "+tbb" in spec: - return [] - else: - return ["NO_TBB=1"] - - @property - def install_targets(self): - return ['prefix={0}'.format(self.prefix), 'install'] diff --git a/lib/spack/docs/tutorial/examples/PyPackage/0.package.py b/lib/spack/docs/tutorial/examples/PyPackage/0.package.py deleted file mode 100644 index 9ad94189d9..0000000000 --- a/lib/spack/docs/tutorial/examples/PyPackage/0.package.py +++ /dev/null @@ -1,41 +0,0 @@ -# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other -# Spack Project Developers. See the top-level COPYRIGHT file for details. -# -# SPDX-License-Identifier: (Apache-2.0 OR MIT) - -# -# This is a template package file for Spack. We've put "FIXME" -# next to all the things you'll want to change. Once you've handled -# them, you can save this file and test your package like this: -# -# spack install py-pandas -# -# You can edit this file again by typing: -# -# spack edit py-pandas -# -# See the Spack documentation for more information on packaging. -# If you submit this package back to Spack as a pull request, -# please first remove this boilerplate and all FIXME comments. -# -from spack import * - - -class PyPandas(PythonPackage): - """FIXME: Put a proper description of your package here.""" - - # FIXME: Add a proper url for your package's homepage here. - homepage = "http://www.example.com" - url = "https://pypi.io/packages/source/p/pandas/pandas-0.19.0.tar.gz" - - version('0.19.0', 'bc9bb7188e510b5d44fbdd249698a2c3') - - # FIXME: Add dependencies if required. - # depends_on('py-setuptools', type='build') - # depends_on('py-foo', type=('build', 'run')) - - def build_args(self, spec, prefix): - # FIXME: Add arguments other than --prefix - # FIXME: If not needed delete this function - args = [] - return args diff --git a/lib/spack/docs/tutorial/examples/PyPackage/1.package.py b/lib/spack/docs/tutorial/examples/PyPackage/1.package.py deleted file mode 100644 index 8ac335d4a6..0000000000 --- a/lib/spack/docs/tutorial/examples/PyPackage/1.package.py +++ /dev/null @@ -1,32 +0,0 @@ -# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other -# Spack Project Developers. See the top-level COPYRIGHT file for details. -# -# SPDX-License-Identifier: (Apache-2.0 OR MIT) - -from spack import * - - -class PyPandas(PythonPackage): - """pandas is a Python package providing fast, flexible, and expressive - data structures designed to make working with relational or - labeled data both easy and intuitive. It aims to be the - fundamental high-level building block for doing practical, real - world data analysis in Python. Additionally, it has the broader - goal of becoming the most powerful and flexible open source data - analysis / manipulation tool available in any language. - """ - homepage = "http://pandas.pydata.org/" - url = "https://pypi.io/packages/source/p/pandas/pandas-0.19.0.tar.gz" - - version('0.19.0', 'bc9bb7188e510b5d44fbdd249698a2c3') - version('0.18.0', 'f143762cd7a59815e348adf4308d2cf6') - version('0.16.1', 'fac4f25748f9610a3e00e765474bdea8') - version('0.16.0', 'bfe311f05dc0c351f8955fbd1e296e73') - - depends_on('py-dateutil', type=('build', 'run')) - depends_on('py-numpy', type=('build', 'run')) - depends_on('py-setuptools', type='build') - depends_on('py-cython', type='build') - depends_on('py-pytz', type=('build', 'run')) - depends_on('py-numexpr', type=('build', 'run')) - depends_on('py-bottleneck', type=('build', 'run')) diff --git a/lib/spack/docs/tutorial/sc16-tutorial-slide-preview.png b/lib/spack/docs/tutorial/sc16-tutorial-slide-preview.png Binary files differdeleted file mode 100644 index e7f9b2e323..0000000000 --- a/lib/spack/docs/tutorial/sc16-tutorial-slide-preview.png +++ /dev/null diff --git a/lib/spack/docs/tutorial_advanced_packaging.rst b/lib/spack/docs/tutorial_advanced_packaging.rst deleted file mode 100644 index 876a6e8f70..0000000000 --- a/lib/spack/docs/tutorial_advanced_packaging.rst +++ /dev/null @@ -1,515 +0,0 @@ -.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other - Spack Project Developers. See the top-level COPYRIGHT file for details. - - SPDX-License-Identifier: (Apache-2.0 OR MIT) - -.. _advanced-packaging-tutorial: - -============================ -Advanced Topics in Packaging -============================ - -Spack tries to automatically configure packages with information from -dependencies such that all you need to do is to list the dependencies -(i.e., with the ``depends_on`` directive) and the build system (for example -by deriving from :code:`CmakePackage`). - -However, there are many special cases. Often you need to retrieve details -about dependencies to set package-specific configuration options, or to -define package-specific environment variables used by the package's build -system. This tutorial covers how to retrieve build information from -dependencies, and how you can automatically provide important information to -dependents in your package. - ----------------------- -Setup for the tutorial ----------------------- - -.. note:: - - We do not recommend doing this section of the tutorial in a - production Spack instance. - -The tutorial uses custom package definitions with missing sections that -will be filled in during the tutorial. These package definitions are stored -in a separate package repository, which can be enabled with: - -.. code-block:: console - - $ spack repo add --scope=site var/spack/repos/tutorial - -This section of the tutorial may also require a newer version of -gcc. If you have not already installed gcc@7.2.0 and added it to your -configuration, you can do so with: - -.. code-block:: console - - $ spack install gcc@7.2.0 %gcc@5.4.0 - $ spack compiler add --scope=site `spack location -i gcc@7.2.0 %gcc@5.4.0` - -If you are using the tutorial docker image, all dependency packages -will have been installed. Otherwise, to install these packages you can use -the following commands: - -.. code-block:: console - - $ spack install openblas - $ spack install netlib-lapack - $ spack install mpich - -Now, you are ready to set your preferred ``EDITOR`` and continue with -the rest of the tutorial. - -.. note:: - - Several of these packages depend on an MPI implementation. You can use - OpenMPI if you install it from scratch, but this is slow (>10 min.). - A binary cache of MPICH may be provided, in which case you can force - the package to use it and install quickly. All tutorial examples with - packages that depend on MPICH include the spec syntax for building with it - -.. _adv_pkg_tutorial_start: - ---------------------------------------- -Modifying a package's build environment ---------------------------------------- - -Spack sets up several environment variables like ``PATH`` by default to aid in -building a package, but many packages make use of environment variables which -convey specific information about their dependencies (e.g., ``MPICC``). -This section covers how to update your Spack packages so that package-specific -environment variables are defined at build-time. - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Set environment variables in dependent packages at build-time -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Dependencies can set environment variables that are required when their -dependents build. For example, when a package depends on a python extension -like py-numpy, Spack's ``python`` package will add it to ``PYTHONPATH`` -so it is available at build time; this is required because the default setup -that spack does is not sufficient for python to import modules. - -Any package can override the -:py:func:`setup_dependent_build_environment <spack.package.PackageBase.setup_dependent_build_environment>` -method to setup the build environment for a dependent. -This method takes as an argument a :py:class:`EnvironmentModifications <spack.util.environment.EnvironmentModifications>` -object which includes convenience methods to update the environment. For -example, an MPI implementation can set ``MPICC`` for packages that depend on it: - -.. code-block:: python - - def setup_dependent_build_environment(self, env, dependent_spec): - env.set('MPICC', join_path(self.prefix.bin, 'mpicc')) - -In this case packages that depend on ``mpi`` will have ``MPICC`` defined in -their environment when they build. This section is focused on setting up the -build-time environment but it's worth noting that a similar method called -:py:func:`setup_dependent_run_environment <spack.package.PackageBase.setup_dependent_run_environment>` -can be used to code modifications that will be included in Spack's automatically-generated -module files. - -We can practice by editing the ``mpich`` package to set the ``MPICC`` -environment variable in the build-time environment of dependent packages. - -.. code-block:: console - - root@advanced-packaging-tutorial:/# spack edit mpich - -Once you're finished, the method should look like this: - -.. code-block:: python - - def setup_dependent_build_environment(self, env, dependent_spec): - env.set('MPICC', join_path(self.prefix.bin, 'mpicc')) - env.set('MPICXX', join_path(self.prefix.bin, 'mpic++')) - env.set('MPIF77', join_path(self.prefix.bin, 'mpif77')) - env.set('MPIF90', join_path(self.prefix.bin, 'mpif90')) - - env.set('MPICH_CC', spack_cc) - env.set('MPICH_CXX', spack_cxx) - env.set('MPICH_F77', spack_f77) - env.set('MPICH_F90', spack_fc) - env.set('MPICH_FC', spack_fc) - -At this point we can, for instance, install ``netlib-scalapack`` with -``mpich``: - -.. code-block:: console - - root@advanced-packaging-tutorial:/# spack install netlib-scalapack ^mpich - ... - ==> Created stage in /usr/local/var/spack/stage/netlib-scalapack-2.0.2-km7tsbgoyyywonyejkjoojskhc5knz3z - ==> No patches needed for netlib-scalapack - ==> Building netlib-scalapack [CMakePackage] - ==> Executing phase: 'cmake' - ==> Executing phase: 'build' - ==> Executing phase: 'install' - ==> Successfully installed netlib-scalapack - Fetch: 0.01s. Build: 3m 59.86s. Total: 3m 59.87s. - [+] /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/netlib-scalapack-2.0.2-km7tsbgoyyywonyejkjoojskhc5knz3z - - -and double check the environment logs to verify that every variable was -set to the correct value. - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Set environment variables in your own package -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Packages can override the -:py:func:`setup_build_environment <spack.package.PackageBase.setup_build_environment>` -or the -:py:func:`setup_run_environment <spack.package.PackageBase.setup_run_environment>` -methods to modify their own build-time or run-time environment respectively. -An example of a package that overrides both methods is ``qt``: - -.. code-block:: python - - def setup_build_environment(self, env): - env.set('MAKEFLAGS', '-j{0}'.format(make_jobs)) - - def setup_run_environment(self, env): - env.set('QTDIR', self.prefix) - -When ``qt`` builds, ``MAKEFLAGS`` will be defined in the environment. Likewise, when a -module file is created for ``qt`` it will contain commands to define ``QTDIR`` appropriately. - -To contrast with ``qt``'s -:py:func:`setup_dependent_build_environment <spack.package.PackageBase.setup_dependent_build_environment>` -function: - -.. code-block:: python - - def setup_dependent_build_environment(self, env, dependent_spec): - env.set('QTDIR', self.prefix) - -Let's see how it works by completing the ``elpa`` package: - -.. code-block:: console - - root@advanced-packaging-tutorial:/# spack edit elpa - -In the end your method should look like: - -.. code-block:: python - - def setup_build_environment(self, env): - spec = self.spec - - env.set('CC', spec['mpi'].mpicc) - env.set('FC', spec['mpi'].mpifc) - env.set('CXX', spec['mpi'].mpicxx) - env.set('SCALAPACK_LDFLAGS', spec['scalapack'].libs.joined()) - - env.append_flags('LDFLAGS', spec['lapack'].libs.search_flags) - env.append_flags('LIBS', spec['lapack'].libs.link_flags) - -At this point it's possible to proceed with the installation of ``elpa ^mpich`` - ------------------------------- -Retrieving library information ------------------------------- - -Although Spack attempts to help packages locate their dependency libraries -automatically (e.g. by setting ``PKG_CONFIG_PATH`` and ``CMAKE_PREFIX_PATH``), -a package may have unique configuration options that are required to locate -libraries. When a package needs information about dependency libraries, the -general approach in Spack is to query the dependencies for the locations of -their libraries and set configuration options accordingly. By default most -Spack packages know how to automatically locate their libraries. This section -covers how to retrieve library information from dependencies and how to locate -libraries when the default logic doesn't work. - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Accessing dependency libraries -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -If you need to access the libraries of a dependency, you can do so -via the ``libs`` property of the spec, for example in the ``arpack-ng`` -package: - -.. code-block:: python - - def install(self, spec, prefix): - lapack_libs = spec['lapack'].libs.joined(';') - blas_libs = spec['blas'].libs.joined(';') - - cmake(*[ - '-DLAPACK_LIBRARIES={0}'.format(lapack_libs), - '-DBLAS_LIBRARIES={0}'.format(blas_libs) - ], '.') - -Note that ``arpack-ng`` is querying virtual dependencies, which Spack -automatically resolves to the installed implementation (e.g. ``openblas`` -for ``blas``). - -We've started work on a package for ``armadillo``. You should open it, -read through the comment that starts with ``# TUTORIAL:`` and complete -the ``cmake_args`` section: - -.. code-block:: console - - root@advanced-packaging-tutorial:/# spack edit armadillo - -If you followed the instructions in the package, when you are finished your -``cmake_args`` method should look like: - -.. code-block:: python - - def cmake_args(self): - spec = self.spec - - return [ - # ARPACK support - '-DARPACK_LIBRARY={0}'.format(spec['arpack-ng'].libs.joined(";")), - # BLAS support - '-DBLAS_LIBRARY={0}'.format(spec['blas'].libs.joined(";")), - # LAPACK support - '-DLAPACK_LIBRARY={0}'.format(spec['lapack'].libs.joined(";")), - # SuperLU support - '-DSuperLU_INCLUDE_DIR={0}'.format(spec['superlu'].prefix.include), - '-DSuperLU_LIBRARY={0}'.format(spec['superlu'].libs.joined(";")), - # HDF5 support - '-DDETECT_HDF5={0}'.format('ON' if '+hdf5' in spec else 'OFF') - ] - -As you can see, getting the list of libraries that your dependencies provide -is as easy as accessing the their ``libs`` attribute. Furthermore, the interface -remains the same whether you are querying regular or virtual dependencies. - -At this point you can complete the installation of ``armadillo`` using ``openblas`` -as a LAPACK provider (``armadillo ^openblas ^mpich``): - -.. code-block:: console - - root@advanced-packaging-tutorial:/# spack install armadillo ^openblas ^mpich - ==> pkg-config is already installed in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/pkg-config-0.29.2-ae2hwm7q57byfbxtymts55xppqwk7ecj - ... - ==> superlu is already installed in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/superlu-5.2.1-q2mbtw2wo4kpzis2e2n227ip2fquxrno - ==> Installing armadillo - ==> Using cached archive: /usr/local/var/spack/cache/armadillo/armadillo-8.100.1.tar.xz - ==> Staging archive: /usr/local/var/spack/stage/armadillo-8.100.1-n2eojtazxbku6g4l5izucwwgnpwz77r4/armadillo-8.100.1.tar.xz - ==> Created stage in /usr/local/var/spack/stage/armadillo-8.100.1-n2eojtazxbku6g4l5izucwwgnpwz77r4 - ==> Applied patch undef_linux.patch - ==> Building armadillo [CMakePackage] - ==> Executing phase: 'cmake' - ==> Executing phase: 'build' - ==> Executing phase: 'install' - ==> Successfully installed armadillo - Fetch: 0.01s. Build: 3.96s. Total: 3.98s. - [+] /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/armadillo-8.100.1-n2eojtazxbku6g4l5izucwwgnpwz77r4 - -Hopefully the installation went fine and the code we added expanded to the right list -of semicolon separated libraries (you are encouraged to open ``armadillo``'s -build logs to double check). - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Providing libraries to dependents -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Spack provides a default implementation for ``libs`` which often works -out of the box. A user can write a package definition without having to -implement a ``libs`` property and dependents can retrieve its libraries -as shown in the above section. However, the default implementation assumes that -libraries follow the naming scheme ``lib<package name>.so`` (or e.g. -``lib<package name>.a`` for static libraries). Packages which don't -follow this naming scheme must implement this function themselves, e.g. -``opencv``: - -.. code-block:: python - - @property - def libs(self): - shared = "+shared" in self.spec - return find_libraries( - "libopencv_*", root=self.prefix, shared=shared, recurse=True - ) - -This issue is common for packages which implement an interface (i.e. -virtual package providers in Spack). If we try to build another version of -``armadillo`` tied to ``netlib-lapack`` (``armadillo ^netlib-lapack ^mpich``) -we'll notice that this time the installation won't complete: - -.. code-block:: console - - root@advanced-packaging-tutorial:/# spack install armadillo ^netlib-lapack ^mpich - ==> pkg-config is already installed in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/pkg-config-0.29.2-ae2hwm7q57byfbxtymts55xppqwk7ecj - ... - ==> openmpi is already installed in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.0.0-yo5qkfvumpmgmvlbalqcadu46j5bd52f - ==> Installing arpack-ng - ==> Using cached archive: /usr/local/var/spack/cache/arpack-ng/arpack-ng-3.5.0.tar.gz - ==> Already staged arpack-ng-3.5.0-bloz7cqirpdxj33pg7uj32zs5likz2un in /usr/local/var/spack/stage/arpack-ng-3.5.0-bloz7cqirpdxj33pg7uj32zs5likz2un - ==> No patches needed for arpack-ng - ==> Building arpack-ng [Package] - ==> Executing phase: 'install' - ==> Error: RuntimeError: Unable to recursively locate netlib-lapack libraries in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/netlib-lapack-3.6.1-jjfe23wgt7nkjnp2adeklhseg3ftpx6z - RuntimeError: RuntimeError: Unable to recursively locate netlib-lapack libraries in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/netlib-lapack-3.6.1-jjfe23wgt7nkjnp2adeklhseg3ftpx6z - - /usr/local/var/spack/repos/builtin/packages/arpack-ng/package.py:105, in install: - 5 options.append('-DCMAKE_INSTALL_NAME_DIR:PATH=%s/lib' % prefix) - 6 - 7 # Make sure we use Spack's blas/lapack: - >> 8 lapack_libs = spec['lapack'].libs.joined(';') - 9 blas_libs = spec['blas'].libs.joined(';') - 10 - 11 options.extend([ - - See build log for details: - /usr/local/var/spack/stage/arpack-ng-3.5.0-bloz7cqirpdxj33pg7uj32zs5likz2un/arpack-ng-3.5.0/spack-build-out.txt - -Unlike ``openblas`` which provides a library named ``libopenblas.so``, -``netlib-lapack`` provides ``liblapack.so``, so it needs to implement -customized library search logic. Let's edit it: - -.. code-block:: console - - root@advanced-packaging-tutorial:/# spack edit netlib-lapack - -and follow the instructions in the ``# TUTORIAL:`` comment as before. -What we need to implement is: - -.. code-block:: python - - @property - def lapack_libs(self): - shared = True if '+shared' in self.spec else False - return find_libraries( - 'liblapack', root=self.prefix, shared=shared, recursive=True - ) - -i.e., a property that returns the correct list of libraries for the LAPACK interface. - -We use the name ``lapack_libs`` rather than ``libs`` because -``netlib-lapack`` can also provide ``blas``, and when it does it is provided -as a separate library file. Using this name ensures that when -dependents ask for ``lapack`` libraries, ``netlib-lapack`` will retrieve only -the libraries associated with the ``lapack`` interface. Now we can finally -install ``armadillo ^netlib-lapack ^mpich``: - -.. code-block:: console - - root@advanced-packaging-tutorial:/# spack install armadillo ^netlib-lapack ^mpich - ... - - ==> Building armadillo [CMakePackage] - ==> Executing phase: 'cmake' - ==> Executing phase: 'build' - ==> Executing phase: 'install' - ==> Successfully installed armadillo - Fetch: 0.01s. Build: 3.75s. Total: 3.76s. - [+] /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/armadillo-8.100.1-sxmpu5an4dshnhickh6ykchyfda7jpyn - -Since each implementation of a virtual package is responsible for locating the -libraries associated with the interfaces it provides, dependents do not need -to include special-case logic for different implementations and for example -need only ask for :code:`spec['blas'].libs`. - ----------------------- -Other Packaging Topics ----------------------- - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Attach attributes to other packages -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Build tools usually also provide a set of executables that can be used -when another package is being installed. Spack gives you the opportunity -to monkey-patch dependent modules and attach attributes to them. This -helps make the packager experience as similar as possible to what would -have been the manual installation of the same package. - -An example here is the ``automake`` package, which overrides -:py:func:`setup_dependent_package <spack.package.PackageBase.setup_dependent_package>`: - -.. code-block:: python - - def setup_dependent_package(self, module, dependent_spec): - # Automake is very likely to be a build dependency, - # so we add the tools it provides to the dependent module - executables = ['aclocal', 'automake'] - for name in executables: - setattr(module, name, self._make_executable(name)) - -so that every other package that depends on it can use directly ``aclocal`` -and ``automake`` with the usual function call syntax of :py:class:`Executable <spack.util.executable.Executable>`: - -.. code-block:: python - - aclocal('--force') - -^^^^^^^^^^^^^^^^^^^^^^^ -Extra query parameters -^^^^^^^^^^^^^^^^^^^^^^^ - -An advanced feature of the Spec's build-interface protocol is the support -for extra parameters after the subscript key. In fact, any of the keys used in the query -can be followed by a comma-separated list of extra parameters which can be -inspected by the package receiving the request to fine-tune a response. - -Let's look at an example and try to install ``netcdf ^mpich``: - -.. code-block:: console - - root@advanced-packaging-tutorial:/# spack install netcdf ^mpich - ==> libsigsegv is already installed in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/libsigsegv-2.11-fypapcprssrj3nstp6njprskeyynsgaz - ==> m4 is already installed in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/m4-1.4.18-r5envx3kqctwwflhd4qax4ahqtt6x43a - ... - ==> Error: AttributeError: 'list' object has no attribute 'search_flags' - AttributeError: AttributeError: 'list' object has no attribute 'search_flags' - - /usr/local/var/spack/repos/builtin/packages/netcdf/package.py:207, in configure_args: - 50 # used instead. - 51 hdf5_hl = self.spec['hdf5:hl'] - 52 CPPFLAGS.append(hdf5_hl.headers.cpp_flags) - >> 53 LDFLAGS.append(hdf5_hl.libs.search_flags) - 54 - 55 if '+parallel-netcdf' in self.spec: - 56 config_args.append('--enable-pnetcdf') - - See build log for details: - /usr/local/var/spack/stage/netcdf-4.4.1.1-gk2xxhbqijnrdwicawawcll4t3c7dvoj/netcdf-4.4.1.1/spack-build-out.txt - -We can see from the error that ``netcdf`` needs to know how to link the *high-level interface* -of ``hdf5``, and thus passes the extra parameter ``hl`` after the request to retrieve it. -Clearly the implementation in the ``hdf5`` package is not complete, and we need to fix it: - -.. code-block:: console - - root@advanced-packaging-tutorial:/# spack edit hdf5 - -If you followed the instructions correctly, the code added to the -``lib`` property should be similar to: - -.. code-block:: python - :emphasize-lines: 1 - - query_parameters = self.spec.last_query.extra_parameters - key = tuple(sorted(query_parameters)) - libraries = query2libraries[key] - shared = '+shared' in self.spec - return find_libraries( - libraries, root=self.prefix, shared=shared, recurse=True - ) - -where we highlighted the line retrieving the extra parameters. Now we can successfully -complete the installation of ``netcdf ^mpich``: - -.. code-block:: console - - root@advanced-packaging-tutorial:/# spack install netcdf ^mpich - ==> libsigsegv is already installed in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/libsigsegv-2.11-fypapcprssrj3nstp6njprskeyynsgaz - ==> m4 is already installed in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/m4-1.4.18-r5envx3kqctwwflhd4qax4ahqtt6x43a - ... - ==> Installing netcdf - ==> Using cached archive: /usr/local/var/spack/cache/netcdf/netcdf-4.4.1.1.tar.gz - ==> Already staged netcdf-4.4.1.1-gk2xxhbqijnrdwicawawcll4t3c7dvoj in /usr/local/var/spack/stage/netcdf-4.4.1.1-gk2xxhbqijnrdwicawawcll4t3c7dvoj - ==> Already patched netcdf - ==> Building netcdf [AutotoolsPackage] - ==> Executing phase: 'autoreconf' - ==> Executing phase: 'configure' - ==> Executing phase: 'build' - ==> Executing phase: 'install' - ==> Successfully installed netcdf - Fetch: 0.01s. Build: 24.61s. Total: 24.62s. - [+] /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/netcdf-4.4.1.1-gk2xxhbqijnrdwicawawcll4t3c7dvoj diff --git a/lib/spack/docs/tutorial_basics.rst b/lib/spack/docs/tutorial_basics.rst deleted file mode 100644 index 9a6a2074b3..0000000000 --- a/lib/spack/docs/tutorial_basics.rst +++ /dev/null @@ -1,1736 +0,0 @@ -.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other - Spack Project Developers. See the top-level COPYRIGHT file for details. - - SPDX-License-Identifier: (Apache-2.0 OR MIT) - -.. _basics-tutorial: - -========================================= -Basic Installation Tutorial -========================================= - -This tutorial will guide you through the process of installing -software using Spack. We will first cover the `spack install` command, -focusing on the power of the spec syntax and the flexibility it gives -to users. We will also cover the `spack find` command for viewing -installed packages and the `spack uninstall` command. Finally, we will -touch on how Spack manages compilers, especially as it relates to -using Spack-built compilers within Spack. We will include full output -from all of the commands demonstrated, although we will frequently -call attention to only small portions of that output (or merely to the -fact that it succeeded). The provided output is all from an AWS -instance running Ubuntu 16.04 - -.. _basics-tutorial-install: - ----------------- -Installing Spack ----------------- - -Spack works out of the box. Simply clone spack and get going. We will -clone Spack and immediately checkout the most recent release, v0.12. - -.. code-block:: console - - $ git clone https://github.com/spack/spack - git clone https://github.com/spack/spack - Cloning into 'spack'... - remote: Enumerating objects: 68, done. - remote: Counting objects: 100% (68/68), done. - remote: Compressing objects: 100% (56/56), done. - remote: Total 135389 (delta 40), reused 16 (delta 9), pack-reused 135321 - Receiving objects: 100% (135389/135389), 47.31 MiB | 1.01 MiB/s, done. - Resolving deltas: 100% (64414/64414), done. - Checking connectivity... done. - $ cd spack - $ git checkout releases/v0.12 - Branch releases/v0.12 set up to track remote branch releases/v0.12 from origin. - Switched to a new branch 'releases/v0.12' - -Next add Spack to your path. Spack has some nice command line -integration tools, so instead of simply appending to your ``PATH`` -variable, source the spack setup script. Then add Spack to your path. - -.. code-block:: console - - $ . share/spack/setup-env.sh - -You're good to go! - ------------------ -What is in Spack? ------------------ - -The ``spack list`` command shows available packages. - -.. code-block:: console - - $ spack list - ==> 2907 packages. - abinit libgpuarray py-espresso r-mlrmbo - abyss libgridxc py-espressopp r-mmwrweek - accfft libgtextutils py-et-xmlfile r-mnormt - ... - -The ``spack list`` command can also take a query string. Spack -automatically adds wildcards to both ends of the string. For example, -we can view all available python packages. - -.. code-block:: console - - $ spack list py- - ==> 479 packages. - lumpy-sv py-funcsigs py-numpydoc py-utililib - perl-file-copy-recursive py-functools32 py-olefile py-pywavelets - py-3to2 py-future py-ont-fast5-api py-pyyaml - ... - -------------------- -Installing Packages -------------------- - -Installing a package with Spack is very simple. To install a piece of -software, simply type ``spack install <package_name>``. - -.. code-block:: console - - $ spack install zlib - ==> Installing zlib - ==> Searching for binary cache of zlib - ==> Warning: No Spack mirrors are currently configured - ==> No binary for zlib found: installing from source - ==> Fetching http://zlib.net/fossils/zlib-1.2.11.tar.gz - ######################################################################## 100.0% - ==> Staging archive: /home/spack1/spack/var/spack/stage/zlib-1.2.11-5nus6knzumx4ik2yl44jxtgtsl7d54xb/zlib-1.2.11.tar.gz - ==> Created stage in /home/spack1/spack/var/spack/stage/zlib-1.2.11-5nus6knzumx4ik2yl44jxtgtsl7d54xb - ==> No patches needed for zlib - ==> Building zlib [Package] - ==> Executing phase: 'install' - ==> Successfully installed zlib - Fetch: 3.27s. Build: 2.18s. Total: 5.44s. - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/zlib-1.2.11-5nus6knzumx4ik2yl44jxtgtsl7d54xb - -Spack can install software either from source or from a binary -cache. Packages in the binary cache are signed with GPG for -security. For the tutorial we have prepared a binary cache so you -don't have to wait on slow compilation from source. To be able to -install from the binary cache, we will need to configure Spack with -the location of the binary cache and trust the GPG key that the binary -cache was signed with. - -.. code-block:: console - - $ spack mirror add tutorial /mirror - $ spack gpg trust /mirror/public.key - gpg: keybox '/home/spack1/spack/opt/spack/gpg/pubring.kbx' created - gpg: /home/spack1/spack/opt/spack/gpg/trustdb.gpg: trustdb created - gpg: key 3B7C69B2: public key "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" imported - gpg: Total number processed: 1 - gpg: imported: 1 - -You'll learn more about configuring Spack later in the tutorial, but -for now you will be able to install the rest of the packages in the -tutorial from a binary cache using the same ``spack install`` -command. By default this will install the binary cached version if it -exists and fall back on installing from source. - -Spack's spec syntax is the interface by which we can request specific -configurations of the package. The ``%`` sigil is used to specify -compilers. - -.. code-block:: console - - $ spack install zlib %clang - ==> Installing zlib - ==> Searching for binary cache of zlib - ==> Finding buildcaches in /mirror/build_cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64-gcc-7.2.0-texinfo-6.5-cuqnfgfhhmudqp5f7upmld6ax7pratzw.spec.yaml - ######################################################################## 100.0% - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64-gcc-4.7-zlib-1.2.11-bq2wtdxakpjytk2tjr7qu23i4py2fi2r.spec.yaml - ######################################################################## 100.0% - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64-gcc-5.4.0-dyninst-9.3.2-bu6s2jzievsjkwtcnrtimc5b625j5omf.spec.yaml - ######################################################################## 100.0% - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64-gcc-7.2.0-openmpi-3.1.3-do5xfer2whhk7gc26atgs3ozr3ljbvs4.spec.yaml - ... - ==> Installing zlib from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/clang-3.8.0-2ubuntu4/zlib-1.2.11/linux-ubuntu16.04-x86_64-clang-3.8.0-2ubuntu4-zlib-1.2.11-4pt75q7qq6lygf3hgnona4lyc2uwedul.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:08:01 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed zlib from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/clang-3.8.0-2ubuntu4/zlib-1.2.11-4pt75q7qq6lygf3hgnona4lyc2uwedul - -Note that this installation is located separately from the previous -one. We will discuss this in more detail later, but this is part of what -allows Spack to support arbitrarily versioned software. - -You can check for particular versions before requesting them. We will -use the ``spack versions`` command to see the available versions, and then -install a different version of ``zlib``. - -.. code-block:: console - - $ spack versions zlib - ==> Safe versions (already checksummed): - 1.2.11 1.2.8 1.2.3 - ==> Remote versions (not yet checksummed): - 1.2.10 1.2.7 1.2.5.1 1.2.4.2 1.2.3.7 - ... - -The ``@`` sigil is used to specify versions, both of packages and of -compilers. - -.. code-block:: console - - $ spack install zlib@1.2.8 - ==> Installing zlib - ==> Searching for binary cache of zlib - ==> Finding buildcaches in /mirror/build_cache - ==> Installing zlib from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/zlib-1.2.8/linux-ubuntu16.04-x86_64-gcc-5.4.0-zlib-1.2.8-bkyl5bhuep6fmhuxzkmhqy25qefjcvzc.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:18:30 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed zlib from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/zlib-1.2.8-bkyl5bhuep6fmhuxzkmhqy25qefjcvzc - - $ spack install zlib %gcc@4.7 - ==> Installing zlib - ==> Searching for binary cache of zlib - ==> Finding buildcaches in /mirror/build_cache - ==> Installing zlib from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-4.7/zlib-1.2.11/linux-ubuntu16.04-x86_64-gcc-4.7-zlib-1.2.11-bq2wtdxakpjytk2tjr7qu23i4py2fi2r.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 04:55:30 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed zlib from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-4.7/zlib-1.2.11-bq2wtdxakpjytk2tjr7qu23i4py2fi2r - -The spec syntax also includes compiler flags. Spack accepts -``cppflags``, ``cflags``, ``cxxflags``, ``fflags``, ``ldflags``, and -``ldlibs`` parameters. The values of these fields must be quoted on -the command line if they include spaces. These values are injected -into the compile line automatically by the Spack compiler wrappers. - -.. code-block:: console - - $ spack install zlib @1.2.8 cppflags=-O3 - ==> Installing zlib - ==> Searching for binary cache of zlib - ==> Finding buildcaches in /mirror/build_cache - ==> Installing zlib from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/zlib-1.2.8/linux-ubuntu16.04-x86_64-gcc-5.4.0-zlib-1.2.8-64mns5mvdacqvlashkf7v6lqrxixhmxu.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:31:54 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed zlib from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/zlib-1.2.8-64mns5mvdacqvlashkf7v6lqrxixhmxu - -The ``spack find`` command is used to query installed packages. Note that -some packages appear identical with the default output. The ``-l`` flag -shows the hash of each package, and the ``-f`` flag shows any non-empty -compiler flags of those packages. - -.. code-block:: console - - $ spack find - ==> 5 installed packages. - -- linux-ubuntu16.04-x86_64 / clang@3.8.0-2ubuntu4 -------------- - zlib@1.2.11 - - -- linux-ubuntu16.04-x86_64 / gcc@4.7 --------------------------- - zlib@1.2.11 - - -- linux-ubuntu16.04-x86_64 / gcc@5.4.0 ------------------------- - zlib@1.2.8 zlib@1.2.8 zlib@1.2.11 - - - $ spack find -lf - ==> 5 installed packages. - -- linux-ubuntu16.04-x86_64 / clang@3.8.0-2ubuntu4 -------------- - 4pt75q7 zlib@1.2.11%clang - - - -- linux-ubuntu16.04-x86_64 / gcc@4.7 --------------------------- - bq2wtdx zlib@1.2.11%gcc - - - -- linux-ubuntu16.04-x86_64 / gcc@5.4.0 ------------------------- - bkyl5bh zlib@1.2.8%gcc - - 64mns5m zlib@1.2.8%gcc cppflags="-O3" - - 5nus6kn zlib@1.2.11%gcc - -Spack generates a hash for each spec. This hash is a function of the full -provenance of the package, so any change to the spec affects the -hash. Spack uses this value to compare specs and to generate unique -installation directories for every combinatorial version. As we move into -more complicated packages with software dependencies, we can see that -Spack reuses existing packages to satisfy a dependency only when the -existing package's hash matches the desired spec. - -.. code-block:: console - - $ spack install tcl - ==> zlib is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/zlib-1.2.11-5nus6knzumx4ik2yl44jxtgtsl7d54xb - ==> Installing tcl - ==> Searching for binary cache of tcl - ==> Finding buildcaches in /mirror/build_cache - ==> Installing tcl from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/tcl-8.6.8/linux-ubuntu16.04-x86_64-gcc-5.4.0-tcl-8.6.8-qhwyccywhx2i6s7ob2gvjrjtj3rnfuqt.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:07:15 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed tcl from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/tcl-8.6.8-qhwyccywhx2i6s7ob2gvjrjtj3rnfuqt - -Dependencies can be explicitly requested using the ``^`` sigil. Note that -the spec syntax is recursive. Anything we could specify about the -top-level package, we can also specify about a dependency using ``^``. - -.. code-block:: console - - $ spack install tcl ^zlib @1.2.8 %clang - ==> Installing zlib - ==> Searching for binary cache of zlib - ==> Finding buildcaches in /mirror/build_cache - ==> Installing zlib from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/clang-3.8.0-2ubuntu4/zlib-1.2.8/linux-ubuntu16.04-x86_64-clang-3.8.0-2ubuntu4-zlib-1.2.8-i426yu3o6lyau5fv5ljwsajfkqxj5rl5.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:09:01 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed zlib from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/clang-3.8.0-2ubuntu4/zlib-1.2.8-i426yu3o6lyau5fv5ljwsajfkqxj5rl5 - ==> Installing tcl - ==> Searching for binary cache of tcl - ==> Installing tcl from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/clang-3.8.0-2ubuntu4/tcl-8.6.8/linux-ubuntu16.04-x86_64-clang-3.8.0-2ubuntu4-tcl-8.6.8-6wc66etr7y6hgibp2derrdkf763exwvc.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:10:21 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed tcl from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/clang-3.8.0-2ubuntu4/tcl-8.6.8-6wc66etr7y6hgibp2derrdkf763exwvc - -Packages can also be referred to from the command line by their package -hash. Using the ``spack find -lf`` command earlier we saw that the hash -of our optimized installation of zlib (``cppflags="-O3"``) began with -``64mns5m``. We can now explicitly build with that package without typing -the entire spec, by using the ``/`` sigil to refer to it by hash. As with -other tools like git, you do not need to specify an *entire* hash on the -command line. You can specify just enough digits to identify a hash -uniquely. If a hash prefix is ambiguous (i.e., two or more installed -packages share the prefix) then spack will report an error. - -.. code-block:: console - - $ spack install tcl ^/64mn - ==> zlib is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/zlib-1.2.8-64mns5mvdacqvlashkf7v6lqrxixhmxu - ==> Installing tcl - ==> Searching for binary cache of tcl - ==> Finding buildcaches in /mirror/build_cache - ==> Installing tcl from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/tcl-8.6.8/linux-ubuntu16.04-x86_64-gcc-5.4.0-tcl-8.6.8-am4pbatrtga3etyusg2akmsvrswwxno2.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:11:53 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed tcl from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/tcl-8.6.8-am4pbatrtga3etyusg2akmsvrswwxno2 - -The ``spack find`` command can also take a ``-d`` flag, which can show -dependency information. Note that each package has a top-level entry, -even if it also appears as a dependency. - -.. code-block:: console - - $ spack find -ldf - ==> 9 installed packages - -- linux-ubuntu16.04-x86_64 / clang@3.8.0-2ubuntu4 -------------- - 6wc66et tcl@8.6.8%clang - i426yu3 ^zlib@1.2.8%clang - - i426yu3 zlib@1.2.8%clang - - 4pt75q7 zlib@1.2.11%clang - - - -- linux-ubuntu16.04-x86_64 / gcc@4.7 --------------------------- - bq2wtdx zlib@1.2.11%gcc - - - -- linux-ubuntu16.04-x86_64 / gcc@5.4.0 ------------------------- - am4pbat tcl@8.6.8%gcc - 64mns5m ^zlib@1.2.8%gcc cppflags="-O3" - - qhwyccy tcl@8.6.8%gcc - 5nus6kn ^zlib@1.2.11%gcc - - bkyl5bh zlib@1.2.8%gcc - - 64mns5m zlib@1.2.8%gcc cppflags="-O3" - - 5nus6kn zlib@1.2.11%gcc - - -Let's move on to slightly more complicated packages. ``HDF5`` is a -good example of a more complicated package, with an MPI dependency. If -we install it "out of the box," it will build with ``openmpi``. - -.. code-block:: console - - $ spack install hdf5 - ==> Installing libsigsegv - ==> Searching for binary cache of libsigsegv - ==> Finding buildcaches in /mirror/build_cache - ==> Installing libsigsegv from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/libsigsegv-2.11/linux-ubuntu16.04-x86_64-gcc-5.4.0-libsigsegv-2.11-fypapcprssrj3nstp6njprskeyynsgaz.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:08:01 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed libsigsegv from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/libsigsegv-2.11-fypapcprssrj3nstp6njprskeyynsgaz - ==> Installing m4 - ==> Searching for binary cache of m4 - ==> Installing m4 from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/m4-1.4.18/linux-ubuntu16.04-x86_64-gcc-5.4.0-m4-1.4.18-suf5jtcfehivwfesrc5hjy72r4nukyel.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:24:11 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed m4 from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/m4-1.4.18-suf5jtcfehivwfesrc5hjy72r4nukyel - ==> Installing libtool - ==> Searching for binary cache of libtool - ==> Installing libtool from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/libtool-2.4.6/linux-ubuntu16.04-x86_64-gcc-5.4.0-libtool-2.4.6-o2pfwjf44353ajgr42xqtvzyvqsazkgu.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:12:47 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed libtool from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/libtool-2.4.6-o2pfwjf44353ajgr42xqtvzyvqsazkgu - ==> Installing pkgconf - ==> Searching for binary cache of pkgconf - ==> Installing pkgconf from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/pkgconf-1.4.2/linux-ubuntu16.04-x86_64-gcc-5.4.0-pkgconf-1.4.2-fovrh7alpft646n6mhis5mml6k6e5f4v.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:00:47 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed pkgconf from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/pkgconf-1.4.2-fovrh7alpft646n6mhis5mml6k6e5f4v - ==> Installing util-macros - ==> Searching for binary cache of util-macros - ==> Installing util-macros from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/util-macros-1.19.1/linux-ubuntu16.04-x86_64-gcc-5.4.0-util-macros-1.19.1-milz7fmttmptcic2qdk5cnel7ll5sybr.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:31:54 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed util-macros from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/util-macros-1.19.1-milz7fmttmptcic2qdk5cnel7ll5sybr - ==> Installing libpciaccess - ==> Searching for binary cache of libpciaccess - ==> Installing libpciaccess from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/libpciaccess-0.13.5/linux-ubuntu16.04-x86_64-gcc-5.4.0-libpciaccess-0.13.5-5urc6tcjae26fbbd2wyfohoszhgxtbmc.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:09:34 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed libpciaccess from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/libpciaccess-0.13.5-5urc6tcjae26fbbd2wyfohoszhgxtbmc - ==> Installing xz - ==> Searching for binary cache of xz - ==> Installing xz from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/xz-5.2.4/linux-ubuntu16.04-x86_64-gcc-5.4.0-xz-5.2.4-teneqii2xv5u6zl5r6qi3pwurc6pmypz.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:05:03 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed xz from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/xz-5.2.4-teneqii2xv5u6zl5r6qi3pwurc6pmypz - ==> zlib is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/zlib-1.2.11-5nus6knzumx4ik2yl44jxtgtsl7d54xb - ==> Installing libxml2 - ==> Searching for binary cache of libxml2 - ==> Installing libxml2 from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/libxml2-2.9.8/linux-ubuntu16.04-x86_64-gcc-5.4.0-libxml2-2.9.8-wpexsphdmfayxqxd4up5vgwuqgu5woo7.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 04:56:04 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed libxml2 from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/libxml2-2.9.8-wpexsphdmfayxqxd4up5vgwuqgu5woo7 - ==> Installing ncurses - ==> Searching for binary cache of ncurses - ==> Installing ncurses from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/ncurses-6.1/linux-ubuntu16.04-x86_64-gcc-5.4.0-ncurses-6.1-3o765ourmesfrji6yeclb4wb5w54aqbh.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:04:49 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed ncurses from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/ncurses-6.1-3o765ourmesfrji6yeclb4wb5w54aqbh - ==> Installing readline - ==> Searching for binary cache of readline - ==> Installing readline from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/readline-7.0/linux-ubuntu16.04-x86_64-gcc-5.4.0-readline-7.0-nxhwrg7xwc6nbsm2v4ezwe63l6nfidbi.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:04:56 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed readline from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/readline-7.0-nxhwrg7xwc6nbsm2v4ezwe63l6nfidbi - ==> Installing gdbm - ==> Searching for binary cache of gdbm - ==> Installing gdbm from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/gdbm-1.14.1/linux-ubuntu16.04-x86_64-gcc-5.4.0-gdbm-1.14.1-q4fpyuo7ouhkeq6d3oabtrppctpvxmes.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:18:34 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed gdbm from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gdbm-1.14.1-q4fpyuo7ouhkeq6d3oabtrppctpvxmes - ==> Installing perl - ==> Searching for binary cache of perl - ==> Installing perl from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/perl-5.26.2/linux-ubuntu16.04-x86_64-gcc-5.4.0-perl-5.26.2-ic2kyoadgp3dxfejcbllyplj2wf524fo.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:12:45 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed perl from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/perl-5.26.2-ic2kyoadgp3dxfejcbllyplj2wf524fo - ==> Installing autoconf - ==> Searching for binary cache of autoconf - ==> Installing autoconf from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/autoconf-2.69/linux-ubuntu16.04-x86_64-gcc-5.4.0-autoconf-2.69-3sx2gxeibc4oasqd4o5h6lnwpcpsgd2q.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:24:03 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed autoconf from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/autoconf-2.69-3sx2gxeibc4oasqd4o5h6lnwpcpsgd2q - ==> Installing automake - ==> Searching for binary cache of automake - ==> Installing automake from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/automake-1.16.1/linux-ubuntu16.04-x86_64-gcc-5.4.0-automake-1.16.1-rymw7imfehycqxzj4nuy2oiw3abegooy.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:12:03 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed automake from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/automake-1.16.1-rymw7imfehycqxzj4nuy2oiw3abegooy - ==> Installing numactl - ==> Searching for binary cache of numactl - ==> Installing numactl from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/numactl-2.0.11/linux-ubuntu16.04-x86_64-gcc-5.4.0-numactl-2.0.11-ft463odrombnxlc3qew4omckhlq7tqgc.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:30:34 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed numactl from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/numactl-2.0.11-ft463odrombnxlc3qew4omckhlq7tqgc - ==> Installing hwloc - ==> Searching for binary cache of hwloc - ==> Installing hwloc from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/hwloc-1.11.9/linux-ubuntu16.04-x86_64-gcc-5.4.0-hwloc-1.11.9-43tkw5mt6huhv37vqnybqgxtkodbsava.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:08:00 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed hwloc from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/hwloc-1.11.9-43tkw5mt6huhv37vqnybqgxtkodbsava - ==> Installing openmpi - ==> Searching for binary cache of openmpi - ==> Installing openmpi from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.1.3/linux-ubuntu16.04-x86_64-gcc-5.4.0-openmpi-3.1.3-3njc4q5pqdpptq6jvqjrezkffwokv2sx.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:01:54 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed openmpi from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.1.3-3njc4q5pqdpptq6jvqjrezkffwokv2sx - ==> Installing hdf5 - ==> Searching for binary cache of hdf5 - ==> Installing hdf5 from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/hdf5-1.10.4/linux-ubuntu16.04-x86_64-gcc-5.4.0-hdf5-1.10.4-ozyvmhzdew66byarohm4p36ep7wtcuiw.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:23:04 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed hdf5 from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/hdf5-1.10.4-ozyvmhzdew66byarohm4p36ep7wtcuiw - -Spack packages can also have build options, called variants. Boolean -variants can be specified using the ``+`` and ``~`` or ``-`` -sigils. There are two sigils for ``False`` to avoid conflicts with -shell parsing in different situations. Variants (boolean or otherwise) -can also be specified using the same syntax as compiler flags. Here -we can install HDF5 without MPI support. - -.. code-block:: console - - $ spack install hdf5~mpi - ==> zlib is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/zlib-1.2.11-5nus6knzumx4ik2yl44jxtgtsl7d54xb - ==> Installing hdf5 - ==> Searching for binary cache of hdf5 - ==> Finding buildcaches in /mirror/build_cache - ==> Installing hdf5 from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/hdf5-1.10.4/linux-ubuntu16.04-x86_64-gcc-5.4.0-hdf5-1.10.4-5vcv5r67vpjzenq4apyebshclelnzuja.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:23:24 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed hdf5 from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/hdf5-1.10.4-5vcv5r67vpjzenq4apyebshclelnzuja - -We might also want to install HDF5 with a different MPI -implementation. While MPI is not a package itself, packages can depend on -abstract interfaces like MPI. Spack handles these through "virtual -dependencies." A package, such as HDF5, can depend on the MPI -interface. Other packages (``openmpi``, ``mpich``, ``mvapich``, etc.) -provide the MPI interface. Any of these providers can be requested for -an MPI dependency. For example, we can build HDF5 with MPI support -provided by mpich by specifying a dependency on ``mpich``. Spack also -supports versioning of virtual dependencies. A package can depend on the -MPI interface at version 3, and provider packages specify what version of -the interface *they* provide. The partial spec ``^mpi@3`` can be safisfied -by any of several providers. - -.. code-block:: console - - $ spack install hdf5+hl+mpi ^mpich - ==> libsigsegv is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/libsigsegv-2.11-fypapcprssrj3nstp6njprskeyynsgaz - ==> m4 is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/m4-1.4.18-suf5jtcfehivwfesrc5hjy72r4nukyel - ==> pkgconf is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/pkgconf-1.4.2-fovrh7alpft646n6mhis5mml6k6e5f4v - ==> ncurses is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/ncurses-6.1-3o765ourmesfrji6yeclb4wb5w54aqbh - ==> readline is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/readline-7.0-nxhwrg7xwc6nbsm2v4ezwe63l6nfidbi - ==> gdbm is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gdbm-1.14.1-q4fpyuo7ouhkeq6d3oabtrppctpvxmes - ==> perl is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/perl-5.26.2-ic2kyoadgp3dxfejcbllyplj2wf524fo - ==> autoconf is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/autoconf-2.69-3sx2gxeibc4oasqd4o5h6lnwpcpsgd2q - ==> automake is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/automake-1.16.1-rymw7imfehycqxzj4nuy2oiw3abegooy - ==> libtool is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/libtool-2.4.6-o2pfwjf44353ajgr42xqtvzyvqsazkgu - ==> Installing texinfo - ==> Searching for binary cache of texinfo - ==> Finding buildcaches in /mirror/build_cache - ==> Installing texinfo from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/texinfo-6.5/linux-ubuntu16.04-x86_64-gcc-5.4.0-texinfo-6.5-zs7a2pcwhq6ho2cj2x26uxfktwkpyucn.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:18:29 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed texinfo from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/texinfo-6.5-zs7a2pcwhq6ho2cj2x26uxfktwkpyucn - ==> Installing findutils - ==> Searching for binary cache of findutils - ==> Installing findutils from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/findutils-4.6.0/linux-ubuntu16.04-x86_64-gcc-5.4.0-findutils-4.6.0-d4iajxsopzrlcjtasahxqeyjkjv5jx4v.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:07:17 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed findutils from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/findutils-4.6.0-d4iajxsopzrlcjtasahxqeyjkjv5jx4v - ==> Installing mpich - ==> Searching for binary cache of mpich - ==> Installing mpich from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/mpich-3.2.1/linux-ubuntu16.04-x86_64-gcc-5.4.0-mpich-3.2.1-p3f7p2r5ntrynqibosglxvhwyztiwqs5.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:23:57 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed mpich from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/mpich-3.2.1-p3f7p2r5ntrynqibosglxvhwyztiwqs5 - ==> zlib is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/zlib-1.2.11-5nus6knzumx4ik2yl44jxtgtsl7d54xb - ==> Installing hdf5 - ==> Searching for binary cache of hdf5 - ==> Installing hdf5 from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/hdf5-1.10.4/linux-ubuntu16.04-x86_64-gcc-5.4.0-hdf5-1.10.4-xxd7syhgej6onpyfyewxqcqe7ltkt7ob.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:07:32 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed hdf5 from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/hdf5-1.10.4-xxd7syhgej6onpyfyewxqcqe7ltkt7ob - -We'll do a quick check in on what we have installed so far. - -.. code-block:: console - - $ spack find -ldf - ==> 32 installed packages - -- linux-ubuntu16.04-x86_64 / clang@3.8.0-2ubuntu4 -------------- - 6wc66et tcl@8.6.8%clang - i426yu3 ^zlib@1.2.8%clang - - i426yu3 zlib@1.2.8%clang - - 4pt75q7 zlib@1.2.11%clang - - - -- linux-ubuntu16.04-x86_64 / gcc@4.7 --------------------------- - bq2wtdx zlib@1.2.11%gcc - - - -- linux-ubuntu16.04-x86_64 / gcc@5.4.0 ------------------------- - 3sx2gxe autoconf@2.69%gcc - suf5jtc ^m4@1.4.18%gcc - fypapcp ^libsigsegv@2.11%gcc - ic2kyoa ^perl@5.26.2%gcc - q4fpyuo ^gdbm@1.14.1%gcc - nxhwrg7 ^readline@7.0%gcc - 3o765ou ^ncurses@6.1%gcc - - rymw7im automake@1.16.1%gcc - ic2kyoa ^perl@5.26.2%gcc - q4fpyuo ^gdbm@1.14.1%gcc - nxhwrg7 ^readline@7.0%gcc - 3o765ou ^ncurses@6.1%gcc - - d4iajxs findutils@4.6.0%gcc - - q4fpyuo gdbm@1.14.1%gcc - nxhwrg7 ^readline@7.0%gcc - 3o765ou ^ncurses@6.1%gcc - - 5vcv5r6 hdf5@1.10.4%gcc - 5nus6kn ^zlib@1.2.11%gcc - - ozyvmhz hdf5@1.10.4%gcc - 3njc4q5 ^openmpi@3.1.3%gcc - 43tkw5m ^hwloc@1.11.9%gcc - 5urc6tc ^libpciaccess@0.13.5%gcc - wpexsph ^libxml2@2.9.8%gcc - teneqii ^xz@5.2.4%gcc - 5nus6kn ^zlib@1.2.11%gcc - ft463od ^numactl@2.0.11%gcc - - xxd7syh hdf5@1.10.4%gcc - p3f7p2r ^mpich@3.2.1%gcc - 5nus6kn ^zlib@1.2.11%gcc - - 43tkw5m hwloc@1.11.9%gcc - 5urc6tc ^libpciaccess@0.13.5%gcc - wpexsph ^libxml2@2.9.8%gcc - teneqii ^xz@5.2.4%gcc - 5nus6kn ^zlib@1.2.11%gcc - ft463od ^numactl@2.0.11%gcc - - 5urc6tc libpciaccess@0.13.5%gcc - - fypapcp libsigsegv@2.11%gcc - - o2pfwjf libtool@2.4.6%gcc - - wpexsph libxml2@2.9.8%gcc - teneqii ^xz@5.2.4%gcc - 5nus6kn ^zlib@1.2.11%gcc - - suf5jtc m4@1.4.18%gcc - fypapcp ^libsigsegv@2.11%gcc - - p3f7p2r mpich@3.2.1%gcc - - 3o765ou ncurses@6.1%gcc - - ft463od numactl@2.0.11%gcc - - 3njc4q5 openmpi@3.1.3%gcc - 43tkw5m ^hwloc@1.11.9%gcc - 5urc6tc ^libpciaccess@0.13.5%gcc - wpexsph ^libxml2@2.9.8%gcc - teneqii ^xz@5.2.4%gcc - 5nus6kn ^zlib@1.2.11%gcc - ft463od ^numactl@2.0.11%gcc - - ic2kyoa perl@5.26.2%gcc - q4fpyuo ^gdbm@1.14.1%gcc - nxhwrg7 ^readline@7.0%gcc - 3o765ou ^ncurses@6.1%gcc - - fovrh7a pkgconf@1.4.2%gcc - - nxhwrg7 readline@7.0%gcc - 3o765ou ^ncurses@6.1%gcc - - am4pbat tcl@8.6.8%gcc - 64mns5m ^zlib@1.2.8%gcc cppflags="-O3" - - qhwyccy tcl@8.6.8%gcc - 5nus6kn ^zlib@1.2.11%gcc - - zs7a2pc texinfo@6.5%gcc - ic2kyoa ^perl@5.26.2%gcc - q4fpyuo ^gdbm@1.14.1%gcc - nxhwrg7 ^readline@7.0%gcc - 3o765ou ^ncurses@6.1%gcc - - milz7fm util-macros@1.19.1%gcc - - teneqii xz@5.2.4%gcc - - bkyl5bh zlib@1.2.8%gcc - - 64mns5m zlib@1.2.8%gcc cppflags="-O3" - - 5nus6kn zlib@1.2.11%gcc - - -Spack models the dependencies of packages as a directed acyclic graph -(DAG). The ``spack find -d`` command shows the tree representation of -that graph. We can also use the ``spack graph`` command to view the entire -DAG as a graph. - -.. code-block:: console - - $ spack graph hdf5+hl+mpi ^mpich - o hdf5 - |\ - o | zlib - / - o mpich - o findutils - |\ - | |\ - | | |\ - | | | |\ - o | | | | texinfo - | | | o | automake - | |_|/| | - |/| | | | - | | | |/ - | | | o autoconf - | |_|/| - |/| |/ - | |/| - o | | perl - o | | gdbm - o | | readline - o | | ncurses - o | | pkgconf - / / - | o libtool - |/ - o m4 - o libsigsegv - -You may also have noticed that there are some packages shown in the -``spack find -d`` output that we didn't install explicitly. These are -dependencies that were installed implicitly. A few packages installed -implicitly are not shown as dependencies in the ``spack find -d`` -output. These are build dependencies. For example, ``libpciaccess`` is a -dependency of openmpi and requires ``m4`` to build. Spack will build ``m4`` as -part of the installation of ``openmpi``, but it does not become a part of -the DAG because it is not linked in at run time. Spack handles build -dependencies differently because of their different (less strict) -consistency requirements. It is entirely possible to have two packages -using different versions of a dependency to build, which obviously cannot -be done with linked dependencies. - -``HDF5`` is more complicated than our basic example of zlib and -openssl, but it's still within the realm of software that an experienced -HPC user could reasonably expect to install given a bit of time. Now -let's look at an even more complicated package. - -.. code-block:: console - - $ spack install trilinos - ==> Installing diffutils - ==> Searching for binary cache of diffutils - ==> Finding buildcaches in /mirror/build_cache - ==> Installing diffutils from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/diffutils-3.6/linux-ubuntu16.04-x86_64-gcc-5.4.0-diffutils-3.6-2rhuivgjrna2nrxhntyde6md2khcvs34.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:30:17 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed diffutils from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/diffutils-3.6-2rhuivgjrna2nrxhntyde6md2khcvs34 - ==> Installing bzip2 - ==> Searching for binary cache of bzip2 - ==> Installing bzip2 from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/bzip2-1.0.6/linux-ubuntu16.04-x86_64-gcc-5.4.0-bzip2-1.0.6-ufczdvsqt6edesm36xiucyry7myhj7e7.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:34:37 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed bzip2 from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/bzip2-1.0.6-ufczdvsqt6edesm36xiucyry7myhj7e7 - ==> zlib is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/zlib-1.2.11-5nus6knzumx4ik2yl44jxtgtsl7d54xb - ==> Installing boost - ==> Searching for binary cache of boost - ==> Installing boost from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/boost-1.68.0/linux-ubuntu16.04-x86_64-gcc-5.4.0-boost-1.68.0-zbgfxapchxa4awxdwpleubfuznblxzvt.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 04:58:55 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed boost from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/boost-1.68.0-zbgfxapchxa4awxdwpleubfuznblxzvt - ==> pkgconf is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/pkgconf-1.4.2-fovrh7alpft646n6mhis5mml6k6e5f4v - ==> ncurses is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/ncurses-6.1-3o765ourmesfrji6yeclb4wb5w54aqbh - ==> readline is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/readline-7.0-nxhwrg7xwc6nbsm2v4ezwe63l6nfidbi - ==> gdbm is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gdbm-1.14.1-q4fpyuo7ouhkeq6d3oabtrppctpvxmes - ==> perl is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/perl-5.26.2-ic2kyoadgp3dxfejcbllyplj2wf524fo - ==> Installing openssl - ==> Searching for binary cache of openssl - ==> Installing openssl from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/openssl-1.0.2o/linux-ubuntu16.04-x86_64-gcc-5.4.0-openssl-1.0.2o-b4y3w3bsyvjla6eesv4vt6aplpfrpsha.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:24:10 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed openssl from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openssl-1.0.2o-b4y3w3bsyvjla6eesv4vt6aplpfrpsha - ==> Installing cmake - ==> Searching for binary cache of cmake - ==> Installing cmake from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/cmake-3.12.3/linux-ubuntu16.04-x86_64-gcc-5.4.0-cmake-3.12.3-otafqzhh4xnlq2mpakch7dr3tjfsrjnx.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:33:15 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed cmake from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/cmake-3.12.3-otafqzhh4xnlq2mpakch7dr3tjfsrjnx - ==> Installing glm - ==> Searching for binary cache of glm - ==> Installing glm from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/glm-0.9.7.1/linux-ubuntu16.04-x86_64-gcc-5.4.0-glm-0.9.7.1-jnw622jwcbsymzj2fsx22omjl7tmvaws.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:30:33 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed glm from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/glm-0.9.7.1-jnw622jwcbsymzj2fsx22omjl7tmvaws - ==> libsigsegv is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/libsigsegv-2.11-fypapcprssrj3nstp6njprskeyynsgaz - ==> m4 is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/m4-1.4.18-suf5jtcfehivwfesrc5hjy72r4nukyel - ==> libtool is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/libtool-2.4.6-o2pfwjf44353ajgr42xqtvzyvqsazkgu - ==> util-macros is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/util-macros-1.19.1-milz7fmttmptcic2qdk5cnel7ll5sybr - ==> libpciaccess is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/libpciaccess-0.13.5-5urc6tcjae26fbbd2wyfohoszhgxtbmc - ==> xz is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/xz-5.2.4-teneqii2xv5u6zl5r6qi3pwurc6pmypz - ==> libxml2 is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/libxml2-2.9.8-wpexsphdmfayxqxd4up5vgwuqgu5woo7 - ==> autoconf is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/autoconf-2.69-3sx2gxeibc4oasqd4o5h6lnwpcpsgd2q - ==> automake is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/automake-1.16.1-rymw7imfehycqxzj4nuy2oiw3abegooy - ==> numactl is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/numactl-2.0.11-ft463odrombnxlc3qew4omckhlq7tqgc - ==> hwloc is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/hwloc-1.11.9-43tkw5mt6huhv37vqnybqgxtkodbsava - ==> openmpi is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.1.3-3njc4q5pqdpptq6jvqjrezkffwokv2sx - ==> Installing hdf5 - ==> Searching for binary cache of hdf5 - ==> Installing hdf5 from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/hdf5-1.10.4/linux-ubuntu16.04-x86_64-gcc-5.4.0-hdf5-1.10.4-oqwnui7wtovuf2id4vjwcxfmxlzjus6y.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:09:10 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed hdf5 from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/hdf5-1.10.4-oqwnui7wtovuf2id4vjwcxfmxlzjus6y - ==> Installing openblas - ==> Searching for binary cache of openblas - ==> Installing openblas from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/openblas-0.3.3/linux-ubuntu16.04-x86_64-gcc-5.4.0-openblas-0.3.3-cyeg2yiitpuqglhvbox5gtbgsim2v5vn.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:32:04 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed openblas from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openblas-0.3.3-cyeg2yiitpuqglhvbox5gtbgsim2v5vn - ==> Installing hypre - ==> Searching for binary cache of hypre - ==> Installing hypre from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/hypre-2.15.1/linux-ubuntu16.04-x86_64-gcc-5.4.0-hypre-2.15.1-fshksdpecwiq7r6vawfswpboedhbisju.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:07:34 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed hypre from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/hypre-2.15.1-fshksdpecwiq7r6vawfswpboedhbisju - ==> Installing matio - ==> Searching for binary cache of matio - ==> Installing matio from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/matio-1.5.9/linux-ubuntu16.04-x86_64-gcc-5.4.0-matio-1.5.9-lmzdgssvobdljw52mtahelu2ju7osh6h.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:05:13 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed matio from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/matio-1.5.9-lmzdgssvobdljw52mtahelu2ju7osh6h - ==> Installing metis - ==> Searching for binary cache of metis - ==> Installing metis from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/metis-5.1.0/linux-ubuntu16.04-x86_64-gcc-5.4.0-metis-5.1.0-3wnvp4ji3wwu4v4vymszrhx6naehs6jc.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:31:42 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed metis from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/metis-5.1.0-3wnvp4ji3wwu4v4vymszrhx6naehs6jc - ==> Installing netlib-scalapack - ==> Searching for binary cache of netlib-scalapack - ==> Installing netlib-scalapack from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/netlib-scalapack-2.0.2/linux-ubuntu16.04-x86_64-gcc-5.4.0-netlib-scalapack-2.0.2-wotpfwfctgfkzzn2uescucxvvbg3tm6b.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:07:22 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed netlib-scalapack from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/netlib-scalapack-2.0.2-wotpfwfctgfkzzn2uescucxvvbg3tm6b - ==> Installing mumps - ==> Searching for binary cache of mumps - ==> Installing mumps from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/mumps-5.1.1/linux-ubuntu16.04-x86_64-gcc-5.4.0-mumps-5.1.1-acsg2dzroox2swssgc5cwgkvdy6jcm5q.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:18:32 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed mumps from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/mumps-5.1.1-acsg2dzroox2swssgc5cwgkvdy6jcm5q - ==> Installing netcdf - ==> Searching for binary cache of netcdf - ==> Installing netcdf from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/netcdf-4.6.1/linux-ubuntu16.04-x86_64-gcc-5.4.0-netcdf-4.6.1-mhm4izpogf4mrjidyskb6ewtzxdi7t6g.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:11:57 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed netcdf from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/netcdf-4.6.1-mhm4izpogf4mrjidyskb6ewtzxdi7t6g - ==> Installing parmetis - ==> Searching for binary cache of parmetis - ==> Installing parmetis from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/parmetis-4.0.3/linux-ubuntu16.04-x86_64-gcc-5.4.0-parmetis-4.0.3-uv6h3sqx6quqg22hxesi2mw2un3kw6b7.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:12:04 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed parmetis from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/parmetis-4.0.3-uv6h3sqx6quqg22hxesi2mw2un3kw6b7 - ==> Installing suite-sparse - ==> Searching for binary cache of suite-sparse - ==> Installing suite-sparse from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/suite-sparse-5.3.0/linux-ubuntu16.04-x86_64-gcc-5.4.0-suite-sparse-5.3.0-zaau4kifha2enpdcn3mjlrqym7hm7yon.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:22:54 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed suite-sparse from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/suite-sparse-5.3.0-zaau4kifha2enpdcn3mjlrqym7hm7yon - ==> Installing trilinos - ==> Searching for binary cache of trilinos - ==> Installing trilinos from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/trilinos-12.12.1/linux-ubuntu16.04-x86_64-gcc-5.4.0-trilinos-12.12.1-rlsruavxqvwk2tgxzxboclbo6ykjf54r.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:18:10 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed trilinos from binary cache - -Now we're starting to see the power of Spack. Trilinos in its default -configuration has 23 top level dependecies, many of which have -dependencies of their own. Installing more complex packages can take -days or weeks even for an experienced user. Although we've done a -binary installation for the tutorial, a source installation of -trilinos using Spack takes about 3 hours (depending on the system), -but only 20 seconds of programmer time. - -Spack manages constistency of the entire DAG. Every MPI dependency will -be satisfied by the same configuration of MPI, etc. If we install -``trilinos`` again specifying a dependency on our previous HDF5 built -with ``mpich``: - -.. code-block:: console - - $ spack install trilinos +hdf5 ^hdf5+hl+mpi ^mpich - ==> diffutils is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/diffutils-3.6-2rhuivgjrna2nrxhntyde6md2khcvs34 - ==> bzip2 is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/bzip2-1.0.6-ufczdvsqt6edesm36xiucyry7myhj7e7 - ==> zlib is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/zlib-1.2.11-5nus6knzumx4ik2yl44jxtgtsl7d54xb - ==> boost is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/boost-1.68.0-zbgfxapchxa4awxdwpleubfuznblxzvt - ==> pkgconf is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/pkgconf-1.4.2-fovrh7alpft646n6mhis5mml6k6e5f4v - ==> ncurses is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/ncurses-6.1-3o765ourmesfrji6yeclb4wb5w54aqbh - ==> readline is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/readline-7.0-nxhwrg7xwc6nbsm2v4ezwe63l6nfidbi - ==> gdbm is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gdbm-1.14.1-q4fpyuo7ouhkeq6d3oabtrppctpvxmes - ==> perl is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/perl-5.26.2-ic2kyoadgp3dxfejcbllyplj2wf524fo - ==> openssl is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openssl-1.0.2o-b4y3w3bsyvjla6eesv4vt6aplpfrpsha - ==> cmake is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/cmake-3.12.3-otafqzhh4xnlq2mpakch7dr3tjfsrjnx - ==> glm is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/glm-0.9.7.1-jnw622jwcbsymzj2fsx22omjl7tmvaws - ==> libsigsegv is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/libsigsegv-2.11-fypapcprssrj3nstp6njprskeyynsgaz - ==> m4 is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/m4-1.4.18-suf5jtcfehivwfesrc5hjy72r4nukyel - ==> autoconf is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/autoconf-2.69-3sx2gxeibc4oasqd4o5h6lnwpcpsgd2q - ==> automake is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/automake-1.16.1-rymw7imfehycqxzj4nuy2oiw3abegooy - ==> libtool is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/libtool-2.4.6-o2pfwjf44353ajgr42xqtvzyvqsazkgu - ==> texinfo is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/texinfo-6.5-zs7a2pcwhq6ho2cj2x26uxfktwkpyucn - ==> findutils is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/findutils-4.6.0-d4iajxsopzrlcjtasahxqeyjkjv5jx4v - ==> mpich is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/mpich-3.2.1-p3f7p2r5ntrynqibosglxvhwyztiwqs5 - ==> hdf5 is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/hdf5-1.10.4-xxd7syhgej6onpyfyewxqcqe7ltkt7ob - ==> openblas is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openblas-0.3.3-cyeg2yiitpuqglhvbox5gtbgsim2v5vn - ==> Installing hypre - ==> Searching for binary cache of hypre - ==> Finding buildcaches in /mirror/build_cache - ==> Installing hypre from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/hypre-2.15.1/linux-ubuntu16.04-x86_64-gcc-5.4.0-hypre-2.15.1-obewuozolon7tkdg4cfxc6ae2tzkronb.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:34:36 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed hypre from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/hypre-2.15.1-obewuozolon7tkdg4cfxc6ae2tzkronb - ==> Installing matio - ==> Searching for binary cache of matio - ==> Installing matio from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/matio-1.5.9/linux-ubuntu16.04-x86_64-gcc-5.4.0-matio-1.5.9-gvyqldhifflmvcrtui3b6s64jcczsxxh.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:25:11 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed matio from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/matio-1.5.9-gvyqldhifflmvcrtui3b6s64jcczsxxh - ==> metis is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/metis-5.1.0-3wnvp4ji3wwu4v4vymszrhx6naehs6jc - ==> Installing netlib-scalapack - ==> Searching for binary cache of netlib-scalapack - ==> Installing netlib-scalapack from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/netlib-scalapack-2.0.2/linux-ubuntu16.04-x86_64-gcc-5.4.0-netlib-scalapack-2.0.2-p7iln2pcosw2ipyqoyr7ie6lpva2oj7r.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:32:20 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed netlib-scalapack from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/netlib-scalapack-2.0.2-p7iln2pcosw2ipyqoyr7ie6lpva2oj7r - ==> Installing mumps - ==> Searching for binary cache of mumps - ==> Installing mumps from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/mumps-5.1.1/linux-ubuntu16.04-x86_64-gcc-5.4.0-mumps-5.1.1-cumcj5a75cagsznpjrgretxdg6okxaur.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:33:18 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed mumps from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/mumps-5.1.1-cumcj5a75cagsznpjrgretxdg6okxaur - ==> Installing netcdf - ==> Searching for binary cache of netcdf - ==> Installing netcdf from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/netcdf-4.6.1/linux-ubuntu16.04-x86_64-gcc-5.4.0-netcdf-4.6.1-wmmx5sgwfds34v7bkkhiduar5yecrnnd.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:24:01 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed netcdf from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/netcdf-4.6.1-wmmx5sgwfds34v7bkkhiduar5yecrnnd - ==> Installing parmetis - ==> Searching for binary cache of parmetis - ==> Installing parmetis from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/parmetis-4.0.3/linux-ubuntu16.04-x86_64-gcc-5.4.0-parmetis-4.0.3-jehtatan4y2lcobj6waoqv66jj4libtz.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:07:41 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed parmetis from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/parmetis-4.0.3-jehtatan4y2lcobj6waoqv66jj4libtz - ==> suite-sparse is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/suite-sparse-5.3.0-zaau4kifha2enpdcn3mjlrqym7hm7yon - ==> Installing trilinos - ==> Searching for binary cache of trilinos - ==> Installing trilinos from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/trilinos-12.12.1/linux-ubuntu16.04-x86_64-gcc-5.4.0-trilinos-12.12.1-kqc52moweigxqxzwzfqajc6ocxwdwn4w.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:30:15 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed trilinos from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/trilinos-12.12.1-kqc52moweigxqxzwzfqajc6ocxwdwn4w - - -We see that every package in the trilinos DAG that depends on MPI now -uses ``mpich``. - -.. code-block:: console - - $ spack find -d trilinos - ==> 2 installed packages - -- linux-ubuntu16.04-x86_64 / gcc@5.4.0 ------------------------- - trilinos@12.12.1 - ^boost@1.68.0 - ^bzip2@1.0.6 - ^zlib@1.2.11 - ^glm@0.9.7.1 - ^hdf5@1.10.4 - ^openmpi@3.1.3 - ^hwloc@1.11.9 - ^libpciaccess@0.13.5 - ^libxml2@2.9.8 - ^xz@5.2.4 - ^numactl@2.0.11 - ^hypre@2.15.1 - ^openblas@0.3.3 - ^matio@1.5.9 - ^metis@5.1.0 - ^mumps@5.1.1 - ^netlib-scalapack@2.0.2 - ^netcdf@4.6.1 - ^parmetis@4.0.3 - ^suite-sparse@5.3.0 - - trilinos@12.12.1 - ^boost@1.68.0 - ^bzip2@1.0.6 - ^zlib@1.2.11 - ^glm@0.9.7.1 - ^hdf5@1.10.4 - ^mpich@3.2.1 - ^hypre@2.15.1 - ^openblas@0.3.3 - ^matio@1.5.9 - ^metis@5.1.0 - ^mumps@5.1.1 - ^netlib-scalapack@2.0.2 - ^netcdf@4.6.1 - ^parmetis@4.0.3 - ^suite-sparse@5.3.0 - - -As we discussed before, the ``spack find -d`` command shows the -dependency information as a tree. While that is often sufficient, many -complicated packages, including trilinos, have dependencies that -cannot be fully represented as a tree. Again, the ``spack graph`` -command shows the full DAG of the dependency information. - -.. code-block:: console - - $ spack graph trilinos - o trilinos - |\ - | |\ - | | |\ - | | | |\ - | | | | |\ - | | | | | |\ - | | | | | | |\ - | | | | | | | |\ - | | | | | | | | |\ - | | | | | | | | | |\ - | | | | | | | | | | |\ - | | | | | | | | | | | |\ - | | | | | | | | | | | | |\ - o | | | | | | | | | | | | | suite-sparse - |\ \ \ \ \ \ \ \ \ \ \ \ \ \ - | |_|_|/ / / / / / / / / / / - |/| | | | | | | | | | | | | - | |\ \ \ \ \ \ \ \ \ \ \ \ \ - | | |_|_|_|_|_|/ / / / / / / - | |/| | | | | | | | | | | | - | | | |_|_|_|_|_|_|_|/ / / - | | |/| | | | | | | | | | - | | | o | | | | | | | | | parmetis - | | |/| | | | | | | | | | - | |/|/| | | | | | | | | | - | | | |/ / / / / / / / / - | | | | | | o | | | | | mumps - | |_|_|_|_|/| | | | | | - |/| | | |_|/| | | | | | - | | | |/| |/ / / / / / - | | | | |/| | | | | | - | | | | o | | | | | | netlib-scalapack - | |_|_|/| | | | | | | - |/| | |/| | | | | | | - | | |/|/ / / / / / / - | o | | | | | | | | metis - | |/ / / / / / / / - | | | | | | | o | glm - | | |_|_|_|_|/ / - | |/| | | | | | - | o | | | | | | cmake - | |\ \ \ \ \ \ \ - | o | | | | | | | openssl - | |\ \ \ \ \ \ \ \ - | | | | | o | | | | netcdf - | | |_|_|/| | | | | - | |/| | |/| | | | | - | | | | | |\ \ \ \ \ - | | | | | | | |_|/ / - | | | | | | |/| | | - | | | | | | | o | | matio - | | |_|_|_|_|/| | | - | |/| | | | |/ / / - | | | | | | | o | hypre - | |_|_|_|_|_|/| | - |/| | | | |_|/ / - | | | | |/| | | - | | | | | | o | hdf5 - | | |_|_|_|/| | - | |/| | | |/ / - | | | | |/| | - | | | | o | | openmpi - | | |_|/| | | - | |/| | | | | - | | | | |\ \ \ - | | | | | o | | hwloc - | | | | |/| | | - | | | | | |\ \ \ - | | | | | | |\ \ \ - | | | | | | o | | | libxml2 - | | |_|_|_|/| | | | - | |/| | | |/| | | | - | | | | | | | | | o boost - | | |_|_|_|_|_|_|/| - | |/| | | | | | | | - | o | | | | | | | | zlib - | / / / / / / / / - | | | | | o | | | xz - | | | | | / / / - | | | | | o | | libpciaccess - | | | | |/| | | - | | | | | |\ \ \ - | | | | | o | | | util-macros - | | | | | / / / - | | | o | | | | numactl - | | | |\ \ \ \ \ - | | | | |_|_|/ / - | | | |/| | | | - | | | | |\ \ \ \ - | | | | | |_|/ / - | | | | |/| | | - | | | | | |\ \ \ - | | | | | o | | | automake - | | |_|_|/| | | | - | |/| | | | | | | - | | | | | |/ / / - | | | | | o | | autoconf - | | |_|_|/| | | - | |/| | |/ / / - | | | |/| | | - | o | | | | | perl - | o | | | | | gdbm - | o | | | | | readline - | |/ / / / / - | o | | | | ncurses - | | |_|/ / - | |/| | | - | o | | | pkgconf - | / / / - o | | | openblas - / / / - | o | libtool - |/ / - o | m4 - o | libsigsegv - / - o bzip2 - o diffutils - -You can control how the output is displayed with a number of options. - -The ASCII output from ``spack graph`` can be difficult to parse for -complicated packages. The output can be changed to the ``graphviz`` -``.dot`` format using the ``--dot`` flag. - -.. code-block:: console - - $ spack graph --dot trilinos | dot -Tpdf trilinos_graph.pdf - -.. _basics-tutorial-uninstall: - ---------------------- -Uninstalling Packages ---------------------- - -Earlier we installed many configurations each of zlib and tcl. Now we -will go through and uninstall some of those packages that we didn't -really need. - -.. code-block:: console - - $ spack find -d tcl - ==> 3 installed packages - -- linux-ubuntu16.04-x86_64 / clang@3.8.0-2ubuntu4 -------------- - tcl@8.6.8 - ^zlib@1.2.8 - - - -- linux-ubuntu16.04-x86_64 / gcc@5.4.0 ------------------------- - tcl@8.6.8 - ^zlib@1.2.8 - - tcl@8.6.8 - ^zlib@1.2.11 - - - $ spack find zlib - ==> 6 installed packages. - -- linux-ubuntu16.04-x86_64 / clang@3.8.0-2ubuntu4 -------------- - zlib@1.2.8 zlib@1.2.11 - - -- linux-ubuntu16.04-x86_64 / gcc@4.7 --------------------------- - zlib@1.2.11 - - -- linux-ubuntu16.04-x86_64 / gcc@5.4.0 ------------------------- - zlib@1.2.8 zlib@1.2.8 zlib@1.2.11 - -We can uninstall packages by spec using the same syntax as install. - -.. code-block:: console - - $ spack uninstall zlib %gcc@4.7 - ==> The following packages will be uninstalled: - - -- linux-ubuntu16.04-x86_64 / gcc@4.7 --------------------------- - bq2wtdx zlib@1.2.11%gcc+optimize+pic+shared - - ==> Do you want to proceed? [y/N] y - ==> Successfully uninstalled zlib@1.2.11%gcc@4.7+optimize+pic+shared arch=linux-ubuntu16.04-x86_64 /bq2wtdx - - $ spack find -lf zlib - ==> 5 installed packages. - -- linux-ubuntu16.04-x86_64 / clang@3.8.0-2ubuntu4 -------------- - i426yu3 zlib@1.2.8%clang - 4pt75q7 zlib@1.2.11%clang - - - -- linux-ubuntu16.04-x86_64 / gcc@5.4.0 ------------------------- - bkyl5bh zlib@1.2.8%gcc - 64mns5m zlib@1.2.8%gcc cppflags="-O3" - 5nus6kn zlib@1.2.11%gcc - -We can also uninstall packages by referring only to their hash. - -We can use either ``-f`` (force) or ``-R`` (remove dependents as well) to -remove packages that are required by another installed package. - -.. code-block:: console - - $ spack uninstall zlib/i426 - ==> Error: Will not uninstall zlib@1.2.8%clang@3.8.0-2ubuntu4/i426yu3 - - The following packages depend on it: - -- linux-ubuntu16.04-x86_64 / clang@3.8.0-2ubuntu4 -------------- - 6wc66et tcl@8.6.8%clang - - ==> Error: Use \`spack uninstall --dependents\` to uninstall these dependencies as well. - - $ spack uninstall -R zlib/i426 - ==> The following packages will be uninstalled: - - -- linux-ubuntu16.04-x86_64 / clang@3.8.0-2ubuntu4 -------------- - 6wc66et tcl@8.6.8%clang - i426yu3 zlib@1.2.8%clang+optimize+pic+shared - ==> Do you want to proceed? [y/N] y - ==> Successfully uninstalled tcl@8.6.8%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64 /6wc66et - ==> Successfully uninstalled zlib@1.2.8%clang@3.8.0-2ubuntu4+optimize+pic+shared arch=linux-ubuntu16.04-x86_64 /i426yu3 - -Spack will not uninstall packages that are not sufficiently -specified. The ``-a`` (all) flag can be used to uninstall multiple -packages at once. - -.. code-block:: console - - $ spack uninstall trilinos - ==> Error: trilinos matches multiple packages: - - -- linux-ubuntu16.04-x86_64 / gcc@5.4.0 ------------------------- - rlsruav trilinos@12.12.1%gcc~alloptpkgs+amesos+amesos2+anasazi+aztec+belos+boost build_type=RelWithDebInfo ~cgns~complex~dtk+epetra+epetraext+exodus+explicit_template_instantiation~float+fortran~fortrilinos+gtest+hdf5+hypre+ifpack+ifpack2~intrepid~intrepid2~isorropia+kokkos+metis~minitensor+ml+muelu+mumps~nox~openmp~phalanx~piro~pnetcdf~python~rol~rythmos+sacado~shards+shared~stk+suite-sparse~superlu~superlu-dist~teko~tempus+teuchos+tpetra~x11~xsdkflags~zlib+zoltan+zoltan2 - kqc52mo trilinos@12.12.1%gcc~alloptpkgs+amesos+amesos2+anasazi+aztec+belos+boost build_type=RelWithDebInfo ~cgns~complex~dtk+epetra+epetraext+exodus+explicit_template_instantiation~float+fortran~fortrilinos+gtest+hdf5+hypre+ifpack+ifpack2~intrepid~intrepid2~isorropia+kokkos+metis~minitensor+ml+muelu+mumps~nox~openmp~phalanx~piro~pnetcdf~python~rol~rythmos+sacado~shards+shared~stk+suite-sparse~superlu~superlu-dist~teko~tempus+teuchos+tpetra~x11~xsdkflags~zlib+zoltan+zoltan2 - - ==> Error: You can either: - a) use a more specific spec, or - b) use `spack uninstall --all` to uninstall ALL matching specs. - - - $ spack uninstall /rlsr - ==> The following packages will be uninstalled: - - -- linux-ubuntu16.04-x86_64 / gcc@5.4.0 ------------------------- - rlsruav trilinos@12.12.1%gcc~alloptpkgs+amesos+amesos2+anasazi+aztec+belos+boost build_type=RelWithDebInfo ~cgns~complex~dtk+epetra+epetraext+exodus+explicit_template_instantiation~float+fortran~fortrilinos+gtest+hdf5+hypre+ifpack+ifpack2~intrepid~intrepid2~isorropia+kokkos+metis~minitensor+ml+muelu+mumps~nox~openmp~phalanx~piro~pnetcdf~python~rol~rythmos+sacado~shards+shared~stk+suite-sparse~superlu~superlu-dist~teko~tempus+teuchos+tpetra~x11~xsdkflags~zlib+zoltan+zoltan2 - ==> Do you want to proceed? [y/N] y - ==> Successfully uninstalled trilinos@12.12.1%gcc@5.4.0~alloptpkgs+amesos+amesos2+anasazi+aztec+belos+boost build_type=RelWithDebInfo ~cgns~complex~dtk+epetra+epetraext+exodus+explicit_template_instantiation~float+fortran~fortrilinos+gtest+hdf5+hypre+ifpack+ifpack2~intrepid~intrepid2~isorropia+kokkos+metis~minitensor+ml+muelu+mumps~nox~openmp~phalanx~piro~pnetcdf~python~rol~rythmos+sacado~shards+shared~stk+suite-sparse~superlu~superlu-dist~teko~tempus+teuchos+tpetra~x11~xsdkflags~zlib+zoltan+zoltan2 arch=linux-ubuntu16.04-x86_64 /rlsruav - ------------------------------ -Advanced ``spack find`` Usage ------------------------------ - -We will go over some additional uses for the ``spack find`` command not -already covered in the :ref:`basics-tutorial-install` and -:ref:`basics-tutorial-uninstall` sections. - -The ``spack find`` command can accept what we call "anonymous specs." -These are expressions in spec syntax that do not contain a package -name. For example, ``spack find ^mpich`` will return every installed -package that depends on mpich, and ``spack find cppflags="-O3"`` will -return every package which was built with ``cppflags="-O3"``. - -.. code-block:: console - - $ spack find ^mpich - ==> 8 installed packages - -- linux-ubuntu16.04-x86_64 / gcc@5.4.0 ------------------------- - hdf5@1.10.4 matio@1.5.9 netcdf@4.6.1 parmetis@4.0.3 - hypre@2.15.1 mumps@5.1.1 netlib-scalapack@2.0.2 trilinos@12.12.1 - - $ spack find cppflags=-O3 - ==> 1 installed packages. - -- linux-ubuntu16.04-x86_64 / gcc@5.4.0 ------------------------- - zlib@1.2.8 - -The ``find`` command can also show which packages were installed -explicitly (rather than pulled in as a dependency) using the ``-x`` -flag. The ``-X`` flag shows implicit installs only. The ``find`` command can -also show the path to which a spack package was installed using the ``-p`` -command. - -.. code-block:: console - - $ spack find -px - ==> 10 installed packages - -- linux-ubuntu16.04-x86_64 / clang@3.8.0-2ubuntu4 -------------- - zlib@1.2.11 /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/clang-3.8.0-2ubuntu4/zlib-1.2.11-4pt75q7qq6lygf3hgnona4lyc2uwedul - - -- linux-ubuntu16.04-x86_64 / gcc@5.4.0 ------------------------- - hdf5@1.10.4 /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/hdf5-1.10.4-5vcv5r67vpjzenq4apyebshclelnzuja - hdf5@1.10.4 /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/hdf5-1.10.4-ozyvmhzdew66byarohm4p36ep7wtcuiw - hdf5@1.10.4 /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/hdf5-1.10.4-xxd7syhgej6onpyfyewxqcqe7ltkt7ob - tcl@8.6.8 /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/tcl-8.6.8-am4pbatrtga3etyusg2akmsvrswwxno2 - tcl@8.6.8 /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/tcl-8.6.8-qhwyccywhx2i6s7ob2gvjrjtj3rnfuqt - trilinos@12.12.1 /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/trilinos-12.12.1-kqc52moweigxqxzwzfqajc6ocxwdwn4w - zlib@1.2.8 /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/zlib-1.2.8-bkyl5bhuep6fmhuxzkmhqy25qefjcvzc - zlib@1.2.8 /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/zlib-1.2.8-64mns5mvdacqvlashkf7v6lqrxixhmxu - zlib@1.2.11 /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/zlib-1.2.11-5nus6knzumx4ik2yl44jxtgtsl7d54xb - ---------------------- -Customizing Compilers ---------------------- - - -Spack manages a list of available compilers on the system, detected -automatically from from the user's ``PATH`` variable. The ``spack -compilers`` command is an alias for the command ``spack compiler list``. - -.. code-block:: console - - $ spack compilers - ==> Available compilers - -- clang ubuntu16.04-x86_64 ------------------------------------- - clang@3.8.0-2ubuntu4 clang@3.7.1-2ubuntu2 - - -- gcc ubuntu16.04-x86_64 --------------------------------------- - gcc@5.4.0 gcc@4.7 - -The compilers are maintained in a YAML file. Later in the tutorial you -will learn how to configure compilers by hand for special cases. Spack -also has tools to add compilers, and compilers built with Spack can be -added to the configuration. - -.. code-block:: console - - $ spack install gcc @7.2.0 - ==> libsigsegv is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/libsigsegv-2.11-fypapcprssrj3nstp6njprskeyynsgaz - ==> m4 is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/m4-1.4.18-suf5jtcfehivwfesrc5hjy72r4nukyel - ==> pkgconf is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/pkgconf-1.4.2-fovrh7alpft646n6mhis5mml6k6e5f4v - ==> ncurses is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/ncurses-6.1-3o765ourmesfrji6yeclb4wb5w54aqbh - ==> readline is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/readline-7.0-nxhwrg7xwc6nbsm2v4ezwe63l6nfidbi - ==> gdbm is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gdbm-1.14.1-q4fpyuo7ouhkeq6d3oabtrppctpvxmes - ==> perl is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/perl-5.26.2-ic2kyoadgp3dxfejcbllyplj2wf524fo - ==> autoconf is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/autoconf-2.69-3sx2gxeibc4oasqd4o5h6lnwpcpsgd2q - ==> automake is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/automake-1.16.1-rymw7imfehycqxzj4nuy2oiw3abegooy - ==> libtool is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/libtool-2.4.6-o2pfwjf44353ajgr42xqtvzyvqsazkgu - ==> Installing gmp - ==> Searching for binary cache of gmp - ==> Finding buildcaches in /mirror/build_cache - ==> Installing gmp from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/gmp-6.1.2/linux-ubuntu16.04-x86_64-gcc-5.4.0-gmp-6.1.2-qc4qcfz4monpllc3nqupdo7vwinf73sw.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:18:16 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed gmp from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gmp-6.1.2-qc4qcfz4monpllc3nqupdo7vwinf73sw - ==> Installing isl - ==> Searching for binary cache of isl - ==> Installing isl from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/isl-0.18/linux-ubuntu16.04-x86_64-gcc-5.4.0-isl-0.18-vttqoutnsmjpm3ogb52rninksc7hq5ax.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:05:19 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed isl from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/isl-0.18-vttqoutnsmjpm3ogb52rninksc7hq5ax - ==> Installing mpfr - ==> Searching for binary cache of mpfr - ==> Installing mpfr from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/mpfr-3.1.6/linux-ubuntu16.04-x86_64-gcc-5.4.0-mpfr-3.1.6-jnt2nnp5pmvikbw7opueajlbwbhmjxyv.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:32:07 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed mpfr from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/mpfr-3.1.6-jnt2nnp5pmvikbw7opueajlbwbhmjxyv - ==> Installing mpc - ==> Searching for binary cache of mpc - ==> Installing mpc from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/mpc-1.1.0/linux-ubuntu16.04-x86_64-gcc-5.4.0-mpc-1.1.0-iuf3gc3zpgr4n4mditnxhff6x3joxi27.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:30:35 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed mpc from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/mpc-1.1.0-iuf3gc3zpgr4n4mditnxhff6x3joxi27 - ==> zlib is already installed in /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/zlib-1.2.11-5nus6knzumx4ik2yl44jxtgtsl7d54xb - Installing gcc - ==> Searching for binary cache of gcc - ==> Finding buildcaches in /mirror/build_cache - ==> Installing gcc from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0/linux-ubuntu16.04-x86_64-gcc-5.4.0-gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs.spack - ######################################################################## 100.0% - gpg: Signature made Sat Nov 10 05:22:47 2018 UTC using RSA key ID 3B7C69B2 - gpg: Good signature from "sc-tutorial (GPG created for Spack) <becker33@llnl.gov>" [unknown] - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 95C7 1787 7AC0 0FFD AA8F D6E9 9CFA 4A45 3B7C 69B2 - ==> Successfully installed gcc from binary cache - [+] /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs - - $ spack find -p gcc - spack find -p gcc - ==> 1 installed package - -- linux-ubuntu16.04-x86_64 / gcc@5.4.0 ------------------------- - gcc@7.2.0 /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs - -We can add gcc to Spack as an available compiler using the ``spack -compiler add`` command. This will allow future packages to build with -gcc@7.2.0. - -.. code-block:: console - - $ spack compiler add `spack location -i gcc@7.2.0` - ==> Added 1 new compiler to /home/ubuntu/.spack/linux/compilers.yaml - gcc@7.2.0 - ==> Compilers are defined in the following files: - /home/ubuntu/.spack/linux/compilers.yaml - -We can also remove compilers from our configuration using ``spack compiler remove <compiler_spec>`` - -.. code-block:: console - - $ spack compiler remove gcc@7.2.0 - ==> Removed compiler gcc@7.2.0 diff --git a/lib/spack/docs/tutorial_buildsystems.rst b/lib/spack/docs/tutorial_buildsystems.rst deleted file mode 100644 index 9f3785e7ee..0000000000 --- a/lib/spack/docs/tutorial_buildsystems.rst +++ /dev/null @@ -1,807 +0,0 @@ -.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other - Spack Project Developers. See the top-level COPYRIGHT file for details. - - SPDX-License-Identifier: (Apache-2.0 OR MIT) - -.. _build-systems-tutorial: - -============================== -Spack Package Build Systems -============================== - -You may begin to notice after writing a couple of package template files a -pattern emerge for some packages. For example, you may find yourself writing -an :code:`install()` method that invokes: :code:`configure`, :code:`cmake`, -:code:`make`, :code:`make install`. You may also find yourself writing -:code:`"prefix=" + prefix` as an argument to :code:`configure` or :code:`cmake`. -Rather than having you repeat these lines for all packages, Spack has -classes that can take care of these patterns. In addition, -these package files allow for finer grained control of these build systems. -In this section, we will describe each build system and give examples on -how these can be manipulated to install a package. - ------------------------ -Package Class Hierarchy ------------------------ - -.. graphviz:: - - digraph G { - - node [ - shape = "record" - ] - edge [ - arrowhead = "empty" - ] - - PackageBase -> Package [dir=back] - PackageBase -> MakefilePackage [dir=back] - PackageBase -> AutotoolsPackage [dir=back] - PackageBase -> CMakePackage [dir=back] - PackageBase -> PythonPackage [dir=back] - } - -The above diagram gives a high level view of the class hierarchy and how each -package relates. Each subclass inherits from the :code:`PackageBaseClass` -super class. The bulk of the work is done in this super class which includes -fetching, extracting to a staging directory and installing. Each subclass -then adds additional build-system-specific functionality. In the following -sections, we will go over examples of how to utilize each subclass and to see -how powerful these abstractions are when packaging. - ------------------ -Package ------------------ - -We've already seen examples of a :code:`Package` class in our walkthrough for writing -package files, so we won't be spending much time with them here. Briefly, -the Package class allows for abitrary control over the build process, whereas -subclasses rely on certain patterns (e.g. :code:`configure` :code:`make` -:code:`make install`) to be useful. :code:`Package` classes are particularly useful -for packages that have a non-conventional way of being built since the packager -can utilize some of Spack's helper functions to customize the building and -installing of a package. - -------------------- -Autotools -------------------- - -As we have seen earlier, packages using :code:`Autotools` use :code:`configure`, -:code:`make` and :code:`make install` commands to execute the build and -install process. In our :code:`Package` class, your typical build incantation will -consist of the following: - -.. code-block:: python - - def install(self, spec, prefix): - configure("--prefix=" + prefix) - make() - make("install") - -You'll see that this looks similar to what we wrote in our packaging tutorial. - -The :code:`Autotools` subclass aims to simplify writing package files and provides -convenience methods to manipulate each of the different phases for a :code:`Autotools` -build system. - -:code:`Autotools` packages consist of four phases: - -1. :code:`autoreconf()` -2. :code:`configure()` -3. :code:`build()` -4. :code:`install()` - - -Each of these phases have sensible defaults. Let's take a quick look at some -the internals of the :code:`Autotools` class: - -.. code-block:: console - - $ spack edit --build-system autotools - - -This will open the :code:`AutotoolsPackage` file in your text editor. - -.. note:: - The examples showing code for these classes is abridged to avoid having - long examples. We only show what is relevant to the packager. - - -.. literalinclude:: _spack_root/lib/spack/spack/build_systems/autotools.py - :language: python - :emphasize-lines: 33,36,54 - :lines: 30-76,240-248 - :linenos: - - -Important to note are the highlighted lines. These properties allow the -packager to set what build targets and install targets they want for their -package. If, for example, we wanted to add as our build target :code:`foo` -then we can append to our :code:`build_targets` property: - -.. code-block:: python - - build_targets = ["foo"] - -Which is similiar to invoking make in our Package - -.. code-block:: python - - make("foo") - -This is useful if we have packages that ignore environment variables and need -a command-line argument. - -Another thing to take note of is in the :code:`configure()` method. -Here we see that the :code:`prefix` argument is already included since it is a -common pattern amongst packages using :code:`Autotools`. We then only have to -override :code:`configure_args()`, which will then return it's output to -to :code:`configure()`. Then, :code:`configure()` will append the common -arguments - -Packagers also have the option to run :code:`autoreconf` in case a package -needs to update the build system and generate a new :code:`configure`. Though, -for the most part this will be unnecessary. - -Let's look at the :code:`mpileaks` package.py file that we worked on earlier: - -.. code-block:: console - - $ spack edit mpileaks - -Notice that mpileaks is a :code:`Package` class but uses the :code:`Autotools` -build system. Although this package is acceptable let's make this into an -:code:`AutotoolsPackage` class and simplify it further. - -.. literalinclude:: tutorial/examples/Autotools/0.package.py - :language: python - :emphasize-lines: 9 - :linenos: - -We first inherit from the :code:`AutotoolsPackage` class. - - -Although we could keep the :code:`install()` method, most of it can be handled -by the :code:`AutotoolsPackage` base class. In fact, the only thing that needs -to be overridden is :code:`configure_args()`. - -.. literalinclude:: tutorial/examples/Autotools/1.package.py - :language: python - :emphasize-lines: 25,26,27,28,29,30,31,32 - :linenos: - -Since Spack takes care of setting the prefix for us we can exclude that as -an argument to :code:`configure`. Our packages look simpler, and the packager -does not need to worry about whether they have properly included :code:`configure` -and :code:`make`. - -This version of the :code:`mpileaks` package installs the same as the previous, -but the :code:`AutotoolsPackage` class lets us do it with a cleaner looking -package file. - ------------------ -Makefile ------------------ - -Packages that utilize :code:`Make` or a :code:`Makefile` usually require you -to edit a :code:`Makefile` to set up platform and compiler specific variables. -These packages are handled by the :code:`Makefile` subclass which provides -convenience methods to help write these types of packages. - -A :code:`MakefilePackage` class has three phases that can be overridden. These include: - - 1. :code:`edit()` - 2. :code:`build()` - 3. :code:`install()` - -Packagers then have the ability to control how a :code:`Makefile` is edited, and -what targets to include for the build phase or install phase. - -Let's also take a look inside the :code:`MakefilePackage` class: - -.. code-block:: console - - $ spack edit --build-system makefile - -Take note of the following: - - -.. literalinclude:: _spack_root/lib/spack/spack/build_systems/makefile.py - :language: python - :lines: 14,43-61,70-88 - :emphasize-lines: 21,27,34 - :linenos: - -Similar to :code:`Autotools`, :code:`MakefilePackage` class has properties -that can be set by the packager. We can also override the different -methods highlighted. - - -Let's try to recreate the Bowtie_ package: - -.. _Bowtie: http://bowtie-bio.sourceforge.net/index.shtml - - -.. code-block:: console - - $ spack create -f https://downloads.sourceforge.net/project/bowtie-bio/bowtie/1.2.1.1/bowtie-1.2.1.1-src.zip - ==> This looks like a URL for bowtie - ==> Found 1 version of bowtie: - - 1.2.1.1 https://downloads.sourceforge.net/project/bowtie-bio/bowtie/1.2.1.1/bowtie-1.2.1.1-src.zip - - ==> How many would you like to checksum? (default is 1, q to abort) 1 - ==> Downloading... - ==> Fetching https://downloads.sourceforge.net/project/bowtie-bio/bowtie/1.2.1.1/bowtie-1.2.1.1-src.zip - ######################################################################## 100.0% - ==> Checksummed 1 version of bowtie - ==> This package looks like it uses the makefile build system - ==> Created template for bowtie package - ==> Created package file: /Users/mamelara/spack/var/spack/repos/builtin/packages/bowtie/package.py - -Once the fetching is completed, Spack will open up your text editor in the -usual fashion and create a template of a :code:`MakefilePackage` package.py. - -.. literalinclude:: tutorial/examples/Makefile/0.package.py - :language: python - :linenos: - -Spack was successfully able to detect that :code:`Bowtie` uses :code:`Make`. -Let's add in the rest of our details for our package: - -.. literalinclude:: tutorial/examples/Makefile/1.package.py - :language: python - :emphasize-lines: 10,11,13,14,18,20 - :linenos: - -As we mentioned earlier, most packages using a :code:`Makefile` have hard-coded -variables that must be edited. These variables are fine if you happen to not -care about setup or types of compilers used but Spack is designed to work with -any compiler. The :code:`MakefilePackage` subclass makes it easy to edit -these :code:`Makefiles` by having an :code:`edit()` method that -can be overridden. - -Let's take a look at the default :code:`Makefile` that :code:`Bowtie` provides. -If we look inside, we see that :code:`CC` and :code:`CXX` point to our GNU -compiler: - -.. code-block:: console - - $ spack stage bowtie - -.. note:: - As usual make sure you have shell support activated with spack: - :code:`source /path/to/spack_root/spack/share/spack/setup-env.sh` - -.. code-block:: console - - $ spack cd -s bowtie - $ cd bowtie-1.2 - $ vim Makefile - - -.. code-block:: make - - CPP = g++ -w - CXX = $(CPP) - CC = gcc - LIBS = $(LDFLAGS) -lz - HEADERS = $(wildcard *.h) - -To fix this, we need to use the :code:`edit()` method to write our custom -:code:`Makefile`. - -.. literalinclude:: tutorial/examples/Makefile/2.package.py - :language: python - :emphasize-lines: 23,24,25 - :linenos: - -Here we use a :code:`FileFilter` object to edit our :code:`Makefile`. It takes -in a regular expression and then replaces :code:`CC` and :code:`CXX` to whatever -Spack sets :code:`CC` and :code:`CXX` environment variables to. This allows us to -build :code:`Bowtie` with whatever compiler we specify through Spack's -:code:`spec` syntax. - -Let's change the build and install phases of our package: - -.. literalinclude:: tutorial/examples/Makefile/3.package.py - :language: python - :emphasize-lines: 28,29,30,31,32,35,36 - :linenos: - -Here demonstrate another strategy that we can use to manipulate our package -We can provide command-line arguments to :code:`make()`. Since :code:`Bowtie` -can use :code:`tbb` we can either add :code:`NO_TBB=1` as a argument to prevent -:code:`tbb` support or we can just invoke :code:`make` with no arguments. - -:code:`Bowtie` requires our :code:`install_target` to provide a path to -the install directory. We can do this by providing :code:`prefix=` as a command -line argument to :code:`make()`. - -Let's look at a couple of other examples and go through them: - -.. code-block:: console - - $ spack edit esmf - -Some packages allow environment variables to be set and will honor them. -Packages that use :code:`?=` for assignment in their :code:`Makefile` -can be set using environment variables. In our :code:`esmf` example we -set two environment variables in our :code:`edit()` method: - -.. code-block:: python - - def edit(self, spec, prefix): - for var in os.environ: - if var.startswith('ESMF_'): - os.environ.pop(var) - - # More code ... - - if self.compiler.name == 'gcc': - os.environ['ESMF_COMPILER'] = 'gfortran' - elif self.compiler.name == 'intel': - os.environ['ESMF_COMPILER'] = 'intel' - elif self.compiler.name == 'clang': - os.environ['ESMF_COMPILER'] = 'gfortranclang' - elif self.compiler.name == 'nag': - os.environ['ESMF_COMPILER'] = 'nag' - elif self.compiler.name == 'pgi': - os.environ['ESMF_COMPILER'] = 'pgi' - else: - msg = "The compiler you are building with, " - msg += "'{0}', is not supported by ESMF." - raise InstallError(msg.format(self.compiler.name)) - -As you may have noticed, we didn't really write anything to the :code:`Makefile` -but rather we set environment variables that will override variables set in -the :code:`Makefile`. - -Some packages include a configuration file that sets certain compiler variables, -platform specific variables, and the location of dependencies or libraries. -If the file is simple and only requires a couple of changes, we can overwrite -those entries with our :code:`FileFilter` object. If the configuration involves -complex changes, we can write a new configuration file from scratch. - -Let's look at an example of this in the :code:`elk` package: - -.. code-block:: console - - $ spack edit elk - -.. code-block:: python - - def edit(self, spec, prefix): - # Dictionary of configuration options - config = { - 'MAKE': 'make', - 'AR': 'ar' - } - - # Compiler-specific flags - flags = '' - if self.compiler.name == 'intel': - flags = '-O3 -ip -unroll -no-prec-div' - elif self.compiler.name == 'gcc': - flags = '-O3 -ffast-math -funroll-loops' - elif self.compiler.name == 'pgi': - flags = '-O3 -lpthread' - elif self.compiler.name == 'g95': - flags = '-O3 -fno-second-underscore' - elif self.compiler.name == 'nag': - flags = '-O4 -kind=byte -dusty -dcfuns' - elif self.compiler.name == 'xl': - flags = '-O3' - config['F90_OPTS'] = flags - config['F77_OPTS'] = flags - - # BLAS/LAPACK support - # Note: BLAS/LAPACK must be compiled with OpenMP support - # if the +openmp variant is chosen - blas = 'blas.a' - lapack = 'lapack.a' - if '+blas' in spec: - blas = spec['blas'].libs.joined() - if '+lapack' in spec: - lapack = spec['lapack'].libs.joined() - # lapack must come before blas - config['LIB_LPK'] = ' '.join([lapack, blas]) - - # FFT support - if '+fft' in spec: - config['LIB_FFT'] = join_path(spec['fftw'].prefix.lib, - 'libfftw3.so') - config['SRC_FFT'] = 'zfftifc_fftw.f90' - else: - config['LIB_FFT'] = 'fftlib.a' - config['SRC_FFT'] = 'zfftifc.f90' - - # MPI support - if '+mpi' in spec: - config['F90'] = spec['mpi'].mpifc - config['F77'] = spec['mpi'].mpif77 - else: - config['F90'] = spack_fc - config['F77'] = spack_f77 - config['SRC_MPI'] = 'mpi_stub.f90' - - # OpenMP support - if '+openmp' in spec: - config['F90_OPTS'] += ' ' + self.compiler.openmp_flag - config['F77_OPTS'] += ' ' + self.compiler.openmp_flag - else: - config['SRC_OMP'] = 'omp_stub.f90' - - # Libxc support - if '+libxc' in spec: - config['LIB_libxc'] = ' '.join([ - join_path(spec['libxc'].prefix.lib, 'libxcf90.so'), - join_path(spec['libxc'].prefix.lib, 'libxc.so') - ]) - config['SRC_libxc'] = ' '.join([ - 'libxc_funcs.f90', - 'libxc.f90', - 'libxcifc.f90' - ]) - else: - config['SRC_libxc'] = 'libxcifc_stub.f90' - - # Write configuration options to include file - with open('make.inc', 'w') as inc: - for key in config: - inc.write('{0} = {1}\n'.format(key, config[key])) - -:code:`config` is just a dictionary that we can add key-value pairs to. By the -end of the :code:`edit()` method we write the contents of our dictionary to -:code:`make.inc`. - ---------------- -CMake ---------------- - -CMake_ is another common build system that has been gaining popularity. It works -in a similar manner to :code:`Autotools` but with differences in variable names, -the number of configuration options available, and the handling of shared libraries. -Typical build incantations look like this: - -.. _CMake: https://cmake.org - -.. code-block:: python - - def install(self, spec, prefix): - cmake("-DCMAKE_INSTALL_PREFIX:PATH=/path/to/install_dir ..") - make() - make("install") - -As you can see from the example above, it's very similar to invoking -:code:`configure` and :code:`make` in an :code:`Autotools` build system. However, -the variable names and options differ. Most options in CMake are prefixed -with a :code:`'-D'` flag to indicate a configuration setting. - -In the :code:`CMakePackage` class we can override the following phases: - -1. :code:`cmake()` -2. :code:`build()` -3. :code:`install()` - -The :code:`CMakePackage` class also provides sensible defaults so we only need to -override :code:`cmake_args()`. - -Let's look at these defaults in the :code:`CMakePackage` class in the :code:`_std_args()` method: - -.. code-block:: console - - $ spack edit --build-system cmake - -.. literalinclude:: _spack_root/lib/spack/spack/build_systems/cmake.py - :language: python - :lines: 102-147 - :emphasize-lines: 10,18,24,36,37,38,44 - :linenos: - -Some :code:`CMake` packages use different generators. Spack is able to support -Unix-Makefile_ generators as well as Ninja_ generators. - -.. _Unix-Makefile: https://cmake.org/cmake/help/v3.4/generator/Unix%20Makefiles.html -.. _Ninja: https://cmake.org/cmake/help/v3.4/generator/Ninja.html - -If no generator is specified Spack will default to :code:`Unix Makefiles`. - -Next we setup the build type. In :code:`CMake` you can specify the build type -that you want. Options include: - -1. :code:`empty` -2. :code:`Debug` -3. :code:`Release` -4. :code:`RelWithDebInfo` -5. :code:`MinSizeRel` - -With these options you can specify whether you want your executable to have -the debug version only, release version or the release with debug information. -Release executables tend to be more optimized than Debug. In Spack, we set -the default as RelWithDebInfo unless otherwise specified through a variant. - -Spack then automatically sets up the :code:`-DCMAKE_INSTALL_PREFIX` path, -appends the build type (:code:`RelWithDebInfo` default), and then specifies a verbose -:code:`Makefile`. - -Next we add the :code:`rpaths` to :code:`-DCMAKE_INSTALL_RPATH:STRING`. - - -Finally we add to :code:`-DCMAKE_PREFIX_PATH:STRING` the locations of all our -dependencies so that :code:`CMake` can find them. - -In the end our :code:`cmake` line will look like this (example is :code:`xrootd`): - -.. code-block:: console - - $ cmake $HOME/spack/var/spack/stage/xrootd-4.6.0-4ydm74kbrp4xmcgda5upn33co5pwddyk/xrootd-4.6.0 -G Unix Makefiles -DCMAKE_INSTALL_PREFIX:PATH=$HOME/spack/opt/spack/darwin-sierra-x86_64/clang-9.0.0-apple/xrootd-4.6.0-4ydm74kbrp4xmcgda5upn33co5pwddyk -DCMAKE_BUILD_TYPE:STRING=RelWithDebInfo -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON -DCMAKE_FIND_FRAMEWORK:STRING=LAST -DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=FALSE -DCMAKE_INSTALL_RPATH:STRING=$HOME/spack/opt/spack/darwin-sierra-x86_64/clang-9.0.0-apple/xrootd-4.6.0-4ydm74kbrp4xmcgda5upn33co5pwddyk/lib:$HOME/spack/opt/spack/darwin-sierra-x86_64/clang-9.0.0-apple/xrootd-4.6.0-4ydm74kbrp4xmcgda5upn33co5pwddyk/lib64 -DCMAKE_PREFIX_PATH:STRING=$HOME/spack/opt/spack/darwin-sierra-x86_64/clang-9.0.0-apple/cmake-3.9.4-hally3vnbzydiwl3skxcxcbzsscaasx5 - -We can see now how :code:`CMake` takes care of a lot of the boilerplate code -that would have to be otherwise typed in. - -Let's try to recreate callpath_: - -.. _callpath: https://github.com/LLNL/callpath.git - -.. code-block:: console - - $ spack create -f https://github.com/llnl/callpath/archive/v1.0.3.tar.gz - ==> This looks like a URL for callpath - ==> Found 4 versions of callpath: - - 1.0.3 https://github.com/LLNL/callpath/archive/v1.0.3.tar.gz - 1.0.2 https://github.com/LLNL/callpath/archive/v1.0.2.tar.gz - 1.0.1 https://github.com/LLNL/callpath/archive/v1.0.1.tar.gz - 1.0 https://github.com/LLNL/callpath/archive/v1.0.tar.gz - - ==> How many would you like to checksum? (default is 1, q to abort) 1 - ==> Downloading... - ==> Fetching https://github.com/LLNL/callpath/archive/v1.0.3.tar.gz - ######################################################################## 100.0% - ==> Checksummed 1 version of callpath - ==> This package looks like it uses the cmake build system - ==> Created template for callpath package - ==> Created package file: /Users/mamelara/spack/var/spack/repos/builtin/packages/callpath/package.py - - -which then produces the following template: - -.. literalinclude:: tutorial/examples/Cmake/0.package.py - :language: python - :linenos: - -Again we fill in the details: - -.. literalinclude:: tutorial/examples/Cmake/1.package.py - :language: python - :linenos: - :emphasize-lines: 9,13,14,18,19,20,21,22,23 - -As mentioned earlier, Spack will use sensible defaults to prevent repeated code -and to make writing :code:`CMake` package files simpler. - -In callpath, we want to add options to :code:`CALLPATH_WALKER` as well as add -compiler flags. We add the following options like so: - -.. literalinclude:: tutorial/examples/Cmake/2.package.py - :language: python - :linenos: - :emphasize-lines: 26,30,31 - -Now we can control our build options using :code:`cmake_args()`. If defaults are -sufficient enough for the package, we can leave this method out. - -:code:`CMakePackage` classes allow for control of other features in the -build system. For example, you can specify the path to the "out of source" -build directory and also point to the root of the :code:`CMakeLists.txt` file if it -is placed in a non-standard location. - -A good example of a package that has its :code:`CMakeLists.txt` file located at a -different location is found in :code:`spades`. - -.. code-block:: console - - $ spack edit spades - -.. code-block:: python - - root_cmakelists_dir = "src" - -Here :code:`root_cmakelists_dir` will tell Spack where to find the location -of :code:`CMakeLists.txt`. In this example, it is located a directory level below in -the :code:`src` directory. - -Some :code:`CMake` packages also require the :code:`install` phase to be -overridden. For example, let's take a look at :code:`sniffles`. - -.. code-block:: console - - $ spack edit sniffles - -In the :code:`install()` method, we have to manually install our targets -so we override the :code:`install()` method to do it for us: - -.. code-block:: python - - # the build process doesn't actually install anything, do it by hand - def install(self, spec, prefix): - mkdir(prefix.bin) - src = "bin/sniffles-core-{0}".format(spec.version.dotted) - binaries = ['sniffles', 'sniffles-debug'] - for b in binaries: - install(join_path(src, b), join_path(prefix.bin, b)) - - --------------- -PythonPackage --------------- - -Python extensions and modules are built differently from source than most -applications. Python uses a :code:`setup.py` script to install Python modules. -The script consists of a call to :code:`setup()` which provides the information -required to build a module to Distutils. If you're familiar with pip or -easy_install, setup.py does the same thing. - -These modules are usually installed using the following line: - -.. code-block:: console - - $ python setup.py install - -There are also a list of commands and phases that you can call. To see the full -list you can run: - -.. code-block:: console - - $ python setup.py --help-commands - Standard commands: - build build everything needed to install - build_py "build" pure Python modules (copy to build directory) - build_ext build C/C++ extensions (compile/link to build directory) - build_clib build C/C++ libraries used by Python extensions - build_scripts "build" scripts (copy and fixup #! line) - clean (no description available) - install install everything from build directory - install_lib install all Python modules (extensions and pure Python) - install_headers install C/C++ header files - install_scripts install scripts (Python or otherwise) - install_data install data files - sdist create a source distribution (tarball, zip file, etc.) - register register the distribution with the Python package index - bdist create a built (binary) distribution - bdist_dumb create a "dumb" built distribution - bdist_rpm create an RPM distribution - bdist_wininst create an executable installer for MS Windows - upload upload binary package to PyPI - check perform some checks on the package - - -We can write package files for Python packages using the :code:`Package` class, -but the class brings with it a lot of methods that are useless for Python packages. -Instead, Spack has a :code:`PythonPackage` subclass that allows packagers -of Python modules to be able to invoke :code:`setup.py` and use :code:`Distutils`, -which is much more familiar to a typical python user. - -To see the defaults that Spack has for each a methods, we will take a look -at the :code:`PythonPackage` class: - -.. code-block:: console - - $ spack edit --build-system python - -We see the following: - - -.. literalinclude:: _spack_root/lib/spack/spack/build_systems/python.py - :language: python - :lines: 19,146-357 - :linenos: - -Each of these methods have sensible defaults or they can be overridden. - -We will write a package file for Pandas_: - -.. _pandas: https://pandas.pydata.org - -.. code-block:: console - - $ spack create -f https://pypi.io/packages/source/p/pandas/pandas-0.19.0.tar.gz - ==> This looks like a URL for pandas - ==> Warning: Spack was unable to fetch url list due to a certificate verification problem. You can try running spack -k, which will not check SSL certificates. Use this at your own risk. - ==> Found 1 version of pandas: - - 0.19.0 https://pypi.io/packages/source/p/pandas/pandas-0.19.0.tar.gz - - ==> How many would you like to checksum? (default is 1, q to abort) 1 - ==> Downloading... - ==> Fetching https://pypi.io/packages/source/p/pandas/pandas-0.19.0.tar.gz - ######################################################################## 100.0% - ==> Checksummed 1 version of pandas - ==> This package looks like it uses the python build system - ==> Changing package name from pandas to py-pandas - ==> Created template for py-pandas package - ==> Created package file: /Users/mamelara/spack/var/spack/repos/builtin/packages/py-pandas/package.py - -And we are left with the following template: - -.. literalinclude:: tutorial/examples/PyPackage/0.package.py - :language: python - :linenos: - -As you can see this is not any different than any package template that we have -written. We have the choice of providing build options or using the sensible -defaults - -Luckily for us, there is no need to provide build args. - -Next we need to find the dependencies of a package. Dependencies are usually -listed in :code:`setup.py`. You can find the dependencies by searching for -:code:`install_requires` keyword in that file. Here it is for :code:`Pandas`: - -.. code-block:: python - - # ... code - if sys.version_info[0] >= 3: - - setuptools_kwargs = { - 'zip_safe': False, - 'install_requires': ['python-dateutil >= 2', - 'pytz >= 2011k', - 'numpy >= %s' % min_numpy_ver], - 'setup_requires': ['numpy >= %s' % min_numpy_ver], - } - if not _have_setuptools: - sys.exit("need setuptools/distribute for Py3k" - "\n$ pip install distribute") - - # ... more code - -You can find a more comprehensive list at the Pandas documentation_. - -.. _documentation: https://pandas.pydata.org/pandas-docs/stable/install.html - - -By reading the documentation and :code:`setup.py` we found that :code:`Pandas` -depends on :code:`python-dateutil`, :code:`pytz`, and :code:`numpy`, :code:`numexpr`, -and finally :code:`bottleneck`. - -Here is the completed :code:`Pandas` script: - -.. literalinclude:: tutorial/examples/PyPackage/1.package.py - :language: python - :linenos: - -It is quite important to declare all the dependencies of a Python package. -Spack can "activate" Python packages to prevent the user from having to -load each dependency module explictly. If a dependency is missed, Spack will -be unable to properly activate the package and it will cause an issue. To -learn more about extensions go to :ref:`cmd-spack-extensions`. - -From this example, you can see that building Python modules is made easy -through the :code:`PythonPackage` class. - -------------------- -Other Build Systems -------------------- - -Although we won't get in depth with any of the other build systems that Spack -supports, it is worth mentioning that Spack does provide subclasses -for the following build systems: - -1. :code:`IntelPackage` -2. :code:`SconsPackage` -3. :code:`WafPackage` -4. :code:`RPackage` -5. :code:`PerlPackage` -6. :code:`QMakePackage` - - -Each of these classes have their own abstractions to help assist in writing -package files. For whatever doesn't fit nicely into the other build-systems, -you can use the :code:`Package` class. - -Hopefully by now you can see how we aim to make packaging simple and -robust through these classes. If you want to learn more about these build -systems, check out :ref:`installation_procedure` in the Packaging Guide. diff --git a/lib/spack/docs/tutorial_configuration.rst b/lib/spack/docs/tutorial_configuration.rst deleted file mode 100644 index 879bb94967..0000000000 --- a/lib/spack/docs/tutorial_configuration.rst +++ /dev/null @@ -1,951 +0,0 @@ -.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other - Spack Project Developers. See the top-level COPYRIGHT file for details. - - SPDX-License-Identifier: (Apache-2.0 OR MIT) - -.. _configs-tutorial: - -====================== -Configuration Tutorial -====================== - -This tutorial will guide you through various configuration options -that allow you to customize Spack's behavior with respect to -software installation. We will first cover the configuration file -hierarchy. Then, we will cover configuration options for compilers, -focusing on how they can be used to extend Spack's compiler auto-detection. -Next, we will cover the packages configuration file, focusing on -how it can be used to override default build options as well as -specify external package installations to use. Finally, we will -briefly touch on the config configuration file, which manages more -high-level Spack configuration options. - -For all of these features, we will demonstrate how we build up a full -configuration file. For some, we will then demonstrate how the -configuration affects the install command, and for others we will use -the ``spack spec`` command to demonstrate how the configuration -changes have affected Spack's concretization algorithm. The provided -output is all from a server running Ubuntu version 16.04. - -.. _configs-tutorial-scopes: - --------------------- -Configuration Scopes --------------------- - -Depending on your use case, you may want to provide configuration -settings common to everyone on your team, or you may want to set -default behaviors specific to a single user account. Spack provides -six configuration *scopes* to handle this customization. These scopes, -in order of decreasing priority, are: - -============ =================================================== -Scope Directory -============ =================================================== -Command-line N/A -Custom Custom directory, specified with ``--config-scope`` -User ``~/.spack/`` -Site ``$SPACK_ROOT/etc/spack/`` -System ``/etc/spack/`` -Defaults ``$SPACK_ROOT/etc/spack/defaults/`` -============ =================================================== - -Spack's default configuration settings reside in -``$SPACK_ROOT/etc/spack/defaults``. These are useful for reference, -but should never be directly edited. To override these settings, -create new configuration files in any of the higher-priority -configuration scopes. - -A particular cluster may have multiple Spack installations associated -with different projects. To provide settings common to all Spack -installations, put your configuration files in ``/etc/spack``. -To provide settings specific to a particular Spack installation, -you can use the ``$SPACK_ROOT/etc/spack`` directory. - -For settings specific to a particular user, you will want to add -configuration files to the ``~/.spack`` directory. When Spack first -checked for compilers on your system, you may have noticed that it -placed your compiler configuration in this directory. - -Configuration settings can also be placed in a custom location, -which is then specified on the command line via ``--config-scope``. -An example use case is managing two sets of configurations, one for -development and another for production preferences. - -Settings specified on the command line have precedence over all -other configuration scopes. - -You can also use ``spack config blame <config>`` for displaying -the effective configuration. Spack will show from which scopes -the configuration has been assembled. - -^^^^^^^^^^^^^^^^^^^^^^^^ -Platform-specific Scopes -^^^^^^^^^^^^^^^^^^^^^^^^ - -Some facilities manage multiple platforms from a single shared -file system. In order to handle this, each of the configuration -scopes listed above has two *sub-scopes*: platform-specific and -platform-independent. For example, compiler settings can be stored -in ``compilers.yaml`` configuration files in the following locations: - -#. ``~/.spack/<platform>/compilers.yaml`` -#. ``~/.spack/compilers.yaml`` -#. ``$SPACK_ROOT/etc/spack/<platform>/compilers.yaml`` -#. ``$SPACK_ROOT/etc/spack/compilers.yaml`` -#. ``/etc/spack/<platform>/compilers.yaml`` -#. ``/etc/spack/compilers.yaml`` -#. ``$SPACK_ROOT/etc/defaults/<platform>/compilers.yaml`` -#. ``$SPACK_ROOT/etc/defaults/compilers.yaml`` - -These files are listed in decreasing order of precedence, so files in -``~/.spack/<platform>`` will override settings in ``~/.spack``. - ------------ -YAML Format ------------ - -Spack configurations are YAML dictionaries. Every configuration file -begins with a top-level dictionary that tells Spack which -configuration set it modifies. When Spack checks its configuration, -the configuration scopes are updated as dictionaries in increasing -order of precedence, allowing higher precedence files to override -lower. YAML dictionaries use a colon ":" to specify key-value -pairs. Spack extends YAML syntax slightly to allow a double-colon -"::" to specify a key-value pair. When a double-colon is used to -specify a key-value pair, instead of adding that section, Spack -replaces what was in that section with the new value. For example, -consider a user's compilers configuration file as follows: - -.. code-block:: yaml - - compilers:: - - compiler: - environment: {} - extra_rpaths: [] - flags: {} - modules: [] - operating_system: ubuntu16.04 - paths: - cc: /usr/bin/gcc - cxx: /usr/bin/g++ - f77: /usr/bin/gfortran - fc: /usr/bin/gfortran - spec: gcc@5.4.0 - target: x86_64 - - -This ensures that no other compilers are used, as the user configuration -scope is the last scope searched and the ``compilers::`` line replaces -all previous configuration files information. If the same -configuration file had a single colon instead of the double colon, it -would add the GCC version 5.4.0 compiler to whatever other compilers -were listed in other configuration files. - -.. _configs-tutorial-compilers: - ----------------------- -Compiler Configuration ----------------------- - -For most tasks, we can use Spack with the compilers auto-detected the -first time Spack runs on a system. As discussed in the basic -installation tutorial, we can also tell Spack where compilers are -located using the ``spack compiler add`` command. However, in some -circumstances we want even more fine-grained control over the -compilers available. This section will teach you how to exercise that -control using the compilers configuration file. - -We will start by opening the compilers configuration file: - -.. code-block:: console - - $ spack config edit compilers - - -.. code-block:: yaml - - compilers: - - compiler: - environment: {} - extra_rpaths: [] - flags: {} - modules: [] - operating_system: ubuntu16.04 - paths: - cc: /usr/bin/clang-3.7 - cxx: /usr/bin/clang++-3.7 - f77: null - fc: null - spec: clang@3.7.1-2ubuntu2 - target: x86_64 - - compiler: - environment: {} - extra_rpaths: [] - flags: {} - modules: [] - operating_system: ubuntu16.04 - paths: - cc: /usr/bin/clang - cxx: /usr/bin/clang++ - f77: null - fc: null - spec: clang@3.8.0-2ubuntu4 - target: x86_64 - - compiler: - environment: {} - extra_rpaths: [] - flags: {} - modules: [] - operating_system: ubuntu16.04 - paths: - cc: /usr/bin/gcc-4.7 - cxx: /usr/bin/g++-4.7 - f77: /usr/bin/gfortran-4.7 - fc: /usr/bin/gfortran-4.7 - spec: gcc@4.7 - target: x86_64 - - compiler: - environment: {} - extra_rpaths: [] - flags: {} - modules: [] - operating_system: ubuntu16.04 - paths: - cc: /usr/bin/gcc - cxx: /usr/bin/g++ - f77: /usr/bin/gfortran - fc: /usr/bin/gfortran - spec: gcc@5.4.0 - target: x86_64 - - -This specifies two versions of the GCC compiler and two versions of the -Clang compiler with no Flang compiler. Now suppose we have a code that -we want to compile with the Clang compiler for C/C++ code, but with -gfortran for Fortran components. We can do this by adding another entry -to the ``compilers.yaml`` file. - -.. code-block:: yaml - - - compiler: - environment: {} - extra_rpaths: [] - flags: {} - modules: [] - operating_system: ubuntu16.04 - paths: - cc: /usr/bin/clang - cxx: /usr/bin/clang++ - f77: /usr/bin/gfortran - fc: /usr/bin/gfortran - spec: clang@3.8.0-gfortran - target: x86_64 - - -Let's talk about the sections of this compiler entry that we've changed. -The biggest change we've made is to the ``paths`` section. This lists -the paths to the compilers to use for each language/specification. -In this case, we point to the Clang compiler for C/C++ and the gfortran -compiler for both specifications of Fortran. We've also changed the -``spec`` entry for this compiler. The ``spec`` entry is effectively the -name of the compiler for Spack. It consists of a name and a version -number, separated by the ``@`` sigil. The name must be one of the supported -compiler names in Spack (gcc, intel, pgi, xl, xl_r, clang, nag, cce, arm). -The version number can be an arbitrary string of alphanumeric characters, -as well as ``-``, ``.``, and ``_``. The ``target`` and ``operating_system`` -sections we leave unchanged. These sections specify when Spack can use -different compilers, and are primarily useful for configuration files that -will be used across multiple systems. - -We can verify that our new compiler works by invoking it now: - -.. code-block:: console - - $ spack install --no-cache zlib %clang@3.8.0-gfortran - ... - - -This new compiler also works on Fortran codes: - -.. code-block:: console - - $ spack install --no-cache cfitsio~bzip2 %clang@3.8.0-gfortran - ... - - -^^^^^^^^^^^^^^ -Compiler Flags -^^^^^^^^^^^^^^ - -Some compilers may require specific compiler flags to work properly in -a particular computing environment. Spack provides configuration -options for setting compiler flags every time a specific compiler is -invoked. These flags become part of the package spec and therefore of -the build provenance. As on the command line, the flags are set -through the implicit build variables ``cflags``, ``cxxflags``, ``cppflags``, -``fflags``, ``ldflags``, and ``ldlibs``. - -Let's open our compilers configuration file again and add a compiler flag: - -.. code-block:: yaml - - - compiler: - environment: {} - extra_rpaths: [] - flags: - cppflags: -g - modules: [] - operating_system: ubuntu16.04 - paths: - cc: /usr/bin/clang - cxx: /usr/bin/clang++ - f77: /usr/bin/gfortran - fc: /usr/bin/gfortran - spec: clang@3.8.0-gfortran - target: x86_64 - - -We can test this out using the ``spack spec`` command to show how the -spec is concretized: - -.. code-block:: console - - $ spack spec cfitsio %clang@3.8.0-gfortran - Input spec - -------------------------------- - cfitsio%clang@3.8.0-gfortran - - Normalized - -------------------------------- - cfitsio%clang@3.8.0-gfortran - - Concretized - -------------------------------- - cfitsio@3.410%clang@3.8.0-gfortran cppflags="-g" +bzip2+shared arch=linux-ubuntu16.04-x86_64 - ^bzip2@1.0.6%clang@3.8.0-gfortran cppflags="-g" +shared arch=linux-ubuntu16.04-x86_64 - - -We can see that ``cppflags="-g"`` has been added to every node in the DAG. - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Advanced Compiler Configuration -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -There are three fields of the compiler configuration entry that we -have not yet talked about. - -The ``modules`` field of the compiler is used primarily on Cray systems, -but can be useful on any system that has compilers that are only -useful when a particular module is loaded. Any modules in the -``modules`` field of the compiler configuration will be loaded as part -of the build environment for packages using that compiler. - -The ``extra_rpaths`` field of the compiler configuration is used for -compilers that do not rpath all of their dependencies by -default. Since compilers are often installed externally to Spack, -Spack is unable to manage compiler dependencies and enforce -rpath usage. This can lead to packages not finding link dependencies -imposed by the compiler properly. For compilers that impose link -dependencies on the resulting executables that are not rpath'ed into -the executable automatically, the ``extra_rpaths`` field of the compiler -configuration tells Spack which dependencies to rpath into every -executable created by that compiler. The executables will then be able -to find the link dependencies imposed by the compiler. As an example, -this field can be set by: - -.. code-block:: yaml - - - compiler: - ... - extra_rpaths: - - /apps/intel/ComposerXE2017/compilers_and_libraries_2017.5.239/linux/compiler/lib/intel64_lin - ... - - -The ``environment`` field of the compiler configuration is used for -compilers that require environment variables to be set during build -time. For example, if your Intel compiler suite requires the -``INTEL_LICENSE_FILE`` environment variable to point to the proper -license server, you can set this in ``compilers.yaml`` as follows: - -.. code-block:: yaml - - - compiler: - environment: - set: - INTEL_LICENSE_FILE: 1713@license4 - ... - - -In addition to ``set``, ``environment`` also supports ``unset``, -``prepend-path``, and ``append-path``. - -.. _configs-tutorial-package-prefs: - -------------------------------- -Configuring Package Preferences -------------------------------- - -Package preferences in Spack are managed through the ``packages.yaml`` -configuration file. First, we will look at the default -``packages.yaml`` file. - -.. code-block:: console - - $ spack config --scope defaults edit packages - - -.. literalinclude:: _spack_root/etc/spack/defaults/packages.yaml - :language: yaml - - -This sets the default preferences for compilers and for providers of -virtual packages. To illustrate how this works, suppose we want to -change the preferences to prefer the Clang compiler and to prefer -MPICH over OpenMPI. Currently, we prefer GCC and OpenMPI. - -.. code-block:: console - - $ spack spec hdf5 - Input spec - -------------------------------- - hdf5 - - Concretized - -------------------------------- - hdf5@1.10.4%gcc@5.4.0~cxx~debug~fortran~hl+mpi+pic+shared~szip~threadsafe arch=linux-ubuntu16.04-x86_64 - ^openmpi@3.1.3%gcc@5.4.0~cuda+cxx_exceptions fabrics= ~java~legacylaunchers~memchecker~pmi schedulers= ~sqlite3~thread_multiple+vt arch=linux-ubuntu16.04-x86_64 - ^hwloc@1.11.9%gcc@5.4.0~cairo~cuda+libxml2+pci+shared arch=linux-ubuntu16.04-x86_64 - ^libpciaccess@0.13.5%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ^libtool@2.4.6%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ^m4@1.4.18%gcc@5.4.0 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu16.04-x86_64 - ^libsigsegv@2.11%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ^pkgconf@1.4.2%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ^util-macros@1.19.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ^libxml2@2.9.8%gcc@5.4.0~python arch=linux-ubuntu16.04-x86_64 - ^xz@5.2.4%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ^zlib@1.2.11%gcc@5.4.0+optimize+pic+shared arch=linux-ubuntu16.04-x86_64 - ^numactl@2.0.11%gcc@5.4.0 patches=592f30f7f5f757dfc239ad0ffd39a9a048487ad803c26b419e0f96b8cda08c1a arch=linux-ubuntu16.04-x86_64 - ^autoconf@2.69%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ^perl@5.26.2%gcc@5.4.0+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=linux-ubuntu16.04-x86_64 - ^gdbm@1.14.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ^readline@7.0%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ^ncurses@6.1%gcc@5.4.0~symlinks~termlib arch=linux-ubuntu16.04-x86_64 - ^automake@1.16.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - - -Now we will open the packages configuration file and update our -preferences. - -.. code-block:: console - - $ spack config edit packages - - -.. code-block:: yaml - - packages: - all: - compiler: [clang, gcc, intel, pgi, xl, nag] - providers: - mpi: [mpich, openmpi] - - -Because of the configuration scoping we discussed earlier, this -overrides the default settings just for these two items. - -.. code-block:: console - - $ spack spec hdf5 - Input spec - -------------------------------- - hdf5 - - Concretized - -------------------------------- - hdf5@1.10.4%clang@3.8.0-2ubuntu4~cxx~debug~fortran~hl+mpi+pic+shared~szip~threadsafe arch=linux-ubuntu16.04-x86_64 - ^mpich@3.2.1%clang@3.8.0-2ubuntu4 device=ch3 +hydra netmod=tcp +pmi+romio~verbs arch=linux-ubuntu16.04-x86_64 - ^findutils@4.6.0%clang@3.8.0-2ubuntu4 patches=84b916c0bf8c51b7e7b28417692f0ad3e7030d1f3c248ba77c42ede5c1c5d11e,bd9e4e5cc280f9753ae14956c4e4aa17fe7a210f55dd6c84aa60b12d106d47a2 arch=linux-ubuntu16.04-x86_64 - ^autoconf@2.69%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64 - ^m4@1.4.18%clang@3.8.0-2ubuntu4 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu16.04-x86_64 - ^libsigsegv@2.11%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64 - ^perl@5.26.2%clang@3.8.0-2ubuntu4+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=linux-ubuntu16.04-x86_64 - ^gdbm@1.14.1%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64 - ^readline@7.0%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64 - ^ncurses@6.1%clang@3.8.0-2ubuntu4~symlinks~termlib arch=linux-ubuntu16.04-x86_64 - ^pkgconf@1.4.2%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64 - ^automake@1.16.1%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64 - ^libtool@2.4.6%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64 - ^texinfo@6.5%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64 - ^zlib@1.2.11%clang@3.8.0-2ubuntu4+optimize+pic+shared arch=linux-ubuntu16.04-x86_64 - - -^^^^^^^^^^^^^^^^^^^ -Variant Preferences -^^^^^^^^^^^^^^^^^^^ - -The packages configuration file can also set variant preferences for -package variants. For example, let's change our preferences to build all -packages without shared libraries. We will accomplish this by turning -off the ``shared`` variant on all packages that have one. - -.. code-block:: yaml - - packages: - all: - compiler: [clang, gcc, intel, pgi, xl, nag] - providers: - mpi: [mpich, openmpi] - variants: ~shared - - -We can check the effect of this command with ``spack spec hdf5`` again. - -.. code-block:: console - - $ spack spec hdf5 - Input spec - -------------------------------- - hdf5 - - Concretized - -------------------------------- - hdf5@1.10.4%clang@3.8.0-2ubuntu4~cxx~debug~fortran~hl+mpi+pic~shared~szip~threadsafe arch=linux-ubuntu16.04-x86_64 - ^mpich@3.2.1%clang@3.8.0-2ubuntu4 device=ch3 +hydra netmod=tcp +pmi+romio~verbs arch=linux-ubuntu16.04-x86_64 - ^findutils@4.6.0%clang@3.8.0-2ubuntu4 patches=84b916c0bf8c51b7e7b28417692f0ad3e7030d1f3c248ba77c42ede5c1c5d11e,bd9e4e5cc280f9753ae14956c4e4aa17fe7a210f55dd6c84aa60b12d106d47a2 arch=linux-ubuntu16.04-x86_64 - ^autoconf@2.69%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64 - ^m4@1.4.18%clang@3.8.0-2ubuntu4 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu16.04-x86_64 - ^libsigsegv@2.11%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64 - ^perl@5.26.2%clang@3.8.0-2ubuntu4+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac ~shared+threads arch=linux-ubuntu16.04-x86_64 - ^gdbm@1.14.1%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64 - ^readline@7.0%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64 - ^ncurses@6.1%clang@3.8.0-2ubuntu4~symlinks~termlib arch=linux-ubuntu16.04-x86_64 - ^pkgconf@1.4.2%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64 - ^automake@1.16.1%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64 - ^libtool@2.4.6%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64 - ^texinfo@6.5%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64 - ^zlib@1.2.11%clang@3.8.0-2ubuntu4+optimize+pic~shared arch=linux-ubuntu16.04-x86_64 - - -So far we have only made global changes to the package preferences. As -we've seen throughout this tutorial, HDF5 builds with MPI enabled by -default in Spack. If we were working on a project that would routinely -need serial HDF5, that might get annoying quickly, having to type -``hdf5~mpi`` all the time. Instead, we'll update our preferences for -HDF5. - -.. code-block:: yaml - - packages: - all: - compiler: [clang, gcc, intel, pgi, xl, nag] - providers: - mpi: [mpich, openmpi] - variants: ~shared - hdf5: - variants: ~mpi - - -Now hdf5 will concretize without an MPI dependency by default. - -.. code-block:: console - - $ spack spec hdf5 - Input spec - -------------------------------- - hdf5 - - Concretized - -------------------------------- - hdf5@1.10.4%clang@3.8.0-2ubuntu4~cxx~debug~fortran~hl~mpi+pic+shared~szip~threadsafe arch=linux-ubuntu16.04-x86_64 - ^zlib@1.2.11%clang@3.8.0-2ubuntu4+optimize+pic~shared arch=linux-ubuntu16.04-x86_64 - - -In general, every attribute that we can set for all packages we can -set separately for an individual package. - -^^^^^^^^^^^^^^^^^ -External Packages -^^^^^^^^^^^^^^^^^ - -The packages configuration file also controls when Spack will build -against an externally installed package. On these systems we have a -pre-installed zlib. - -.. code-block:: yaml - - packages: - all: - compiler: [clang, gcc, intel, pgi, xl, nag] - providers: - mpi: [mpich, openmpi] - variants: ~shared - hdf5: - variants: ~mpi - zlib: - paths: - zlib@1.2.8%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64: /usr - - -Here, we've told Spack that zlib 1.2.8 is installed on our system. -We've also told it the installation prefix where zlib can be found. -We don't know exactly which variants it was built with, but that's -okay. - -.. code-block:: console - - $ spack spec hdf5 - Input spec - -------------------------------- - hdf5 - - Concretized - -------------------------------- - hdf5@1.10.4%gcc@5.4.0~cxx~debug~fortran~hl~mpi+pic+shared~szip~threadsafe arch=linux-ubuntu16.04-x86_64 - ^zlib@1.2.8%gcc@5.4.0+optimize+pic~shared arch=linux-ubuntu16.04-x86_64 - - -You'll notice that Spack is now using the external zlib installation, -but the compiler used to build zlib is now overriding our compiler -preference of clang. If we explicitly specify Clang: - -.. code-block:: console - - $ spack spec hdf5 %clang - Input spec - -------------------------------- - hdf5%clang - - Concretized - -------------------------------- - hdf5@1.10.4%clang@3.8.0-2ubuntu4~cxx~debug~fortran~hl~mpi+pic+shared~szip~threadsafe arch=linux-ubuntu16.04-x86_64 - ^zlib@1.2.11%clang@3.8.0-2ubuntu4+optimize+pic~shared arch=linux-ubuntu16.04-x86_64 - - -Spack concretizes to both HDF5 and zlib being built with Clang. -This has a side-effect of rebuilding zlib. If we want to force -Spack to use the system zlib, we have two choices. We can either -specify it on the command line, or we can tell Spack that it's -not allowed to build its own zlib. We'll go with the latter. - -.. code-block:: yaml - - packages: - all: - compiler: [clang, gcc, intel, pgi, xl, nag] - providers: - mpi: [mpich, openmpi] - variants: ~shared - hdf5: - variants: ~mpi - zlib: - paths: - zlib@1.2.8%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64: /usr - buildable: False - - -Now Spack will be forced to choose the external zlib. - -.. code-block:: console - - $ spack spec hdf5 %clang - Input spec - -------------------------------- - hdf5%clang - - Concretized - -------------------------------- - hdf5@1.10.4%clang@3.8.0-2ubuntu4~cxx~debug~fortran~hl~mpi+pic+shared~szip~threadsafe arch=linux-ubuntu16.04-x86_64 - ^zlib@1.2.8%gcc@5.4.0+optimize+pic~shared arch=linux-ubuntu16.04-x86_64 - - -This gets slightly more complicated with virtual dependencies. Suppose -we don't want to build our own MPI, but we now want a parallel version -of HDF5? Well, fortunately we have MPICH installed on these systems. - -.. code-block:: yaml - - packages: - all: - compiler: [clang, gcc, intel, pgi, xl, nag] - providers: - mpi: [mpich, openmpi] - variants: ~shared - hdf5: - variants: ~mpi - zlib: - paths: - zlib@1.2.8%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64: /usr - buildable: False - mpich: - paths: - mpich@3.2%gcc@5.4.0 device=ch3 +hydra netmod=tcp +pmi+romio~verbs arch=linux-ubuntu16.04-x86_64: /usr - buildable: False - - -If we concretize ``hdf5+mpi`` with this configuration file, we will just -build with an alternate MPI implementation. - -.. code-block:: console - - $ spack spec hdf5+mpi %clang - Input spec - -------------------------------- - hdf5%clang+mpi - - Concretized - -------------------------------- - hdf5@1.10.4%clang@3.8.0-2ubuntu4~cxx~debug~fortran~hl+mpi+pic+shared~szip~threadsafe arch=linux-ubuntu16.04-x86_64 - ^openmpi@3.1.3%clang@3.8.0-2ubuntu4~cuda+cxx_exceptions fabrics= ~java~legacylaunchers~memchecker~pmi schedulers= ~sqlite3~thread_multiple+vt arch=linux-ubuntu16.04-x86_64 - ^hwloc@1.11.9%clang@3.8.0-2ubuntu4~cairo~cuda+libxml2+pci~shared arch=linux-ubuntu16.04-x86_64 - ^libpciaccess@0.13.5%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64 - ^libtool@2.4.6%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64 - ^m4@1.4.18%clang@3.8.0-2ubuntu4 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu16.04-x86_64 - ^libsigsegv@2.11%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64 - ^pkgconf@1.4.2%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64 - ^util-macros@1.19.1%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64 - ^libxml2@2.9.8%clang@3.8.0-2ubuntu4~python arch=linux-ubuntu16.04-x86_64 - ^xz@5.2.4%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64 - ^zlib@1.2.8%gcc@5.4.0+optimize+pic~shared arch=linux-ubuntu16.04-x86_64 - ^numactl@2.0.11%clang@3.8.0-2ubuntu4 patches=592f30f7f5f757dfc239ad0ffd39a9a048487ad803c26b419e0f96b8cda08c1a arch=linux-ubuntu16.04-x86_64 - ^autoconf@2.69%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64 - ^perl@5.26.2%clang@3.8.0-2ubuntu4+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac ~shared+threads arch=linux-ubuntu16.04-x86_64 - ^gdbm@1.14.1%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64 - ^readline@7.0%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64 - ^ncurses@6.1%clang@3.8.0-2ubuntu4~symlinks~termlib arch=linux-ubuntu16.04-x86_64 - ^automake@1.16.1%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64 - - -We have only expressed a preference for MPICH over other MPI -implementations, and Spack will happily build with one we haven't -forbid it from building. We could resolve this by requesting -``hdf5+mpi%clang^mpich`` explicitly, or we can configure Spack not to -use any other MPI implementation. Since we're focused on -configurations here and the former can get tedious, we'll need to -modify our ``packages.yaml`` file again. - -While we're at it, we can configure HDF5 to build with MPI by default -again. - -.. code-block:: yaml - - packages: - all: - compiler: [clang, gcc, intel, pgi, xl, nag] - providers: - mpi: [mpich, openmpi] - variants: ~shared - zlib: - paths: - zlib@1.2.8%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64: /usr - buildable: False - mpich: - paths: - mpich@3.2%gcc@5.4.0 device=ch3 +hydra netmod=tcp +pmi+romio~verbs arch=linux-ubuntu16.04-x86_64: /usr - buildable: False - openmpi: - buildable: False - mvapich2: - buildable: False - intel-mpi: - buildable: False - intel-parallel-studio: - buildable: False - spectrum-mpi: - buildable: False - mpilander: - buildable: False - charm: - buildable: False - charmpp: - buildable: False - - -Now that we have configured Spack not to build any of the possible -providers for MPI, we can try again. - -.. code-block:: console - - $ spack spec hdf5 %clang - Input spec - -------------------------------- - hdf5%clang - - Concretized - -------------------------------- - hdf5@1.10.4%clang@3.8.0-2ubuntu4~cxx~debug~fortran~hl+mpi+pic~shared~szip~threadsafe arch=linux-ubuntu16.04-x86_64 - ^mpich@3.2%gcc@5.4.0 device=ch3 +hydra netmod=tcp +pmi+romio~verbs arch=linux-ubuntu16.04-x86_64 - ^zlib@1.2.8%gcc@5.4.0+optimize+pic~shared arch=linux-ubuntu16.04-x86_64 - - -By configuring most of our package preferences in ``packages.yaml``, -we can cut down on the amount of work we need to do when specifying -a spec on the command line. In addition to compiler and variant -preferences, we can specify version preferences as well. Except for -selecting providers via `^`, anything that you can specify on the -command line can be specified in ``packages.yaml`` with the exact -same spec syntax. - -^^^^^^^^^^^^^^^^^^^^^^^^ -Installation Permissions -^^^^^^^^^^^^^^^^^^^^^^^^ - -The ``packages.yaml`` file also controls the default permissions -to use when installing a package. You'll notice that by default, -the installation prefix will be world readable but only user writable. - -Let's say we need to install ``converge``, a licensed software package. -Since a specific research group, ``fluid_dynamics``, pays for this -license, we want to ensure that only members of this group can access -the software. We can do this like so: - -.. code-block:: yaml - - packages: - converge: - permissions: - read: group - group: fluid_dynamics - - -Now, only members of the ``fluid_dynamics`` group can use any -``converge`` installations. - -.. warning:: - - Make sure to delete or move the ``packages.yaml`` you have been - editing up to this point. Otherwise, it will change the hashes - of your packages, leading to differences in the output of later - tutorial sections. - - ------------------ -High-level Config ------------------ - -In addition to compiler and package settings, Spack allows customization -of several high-level settings. These settings are stored in the generic -``config.yaml`` configuration file. You can see the default settings by -running: - -.. code-block:: console - - $ spack config --scope defaults edit config - - -.. literalinclude:: _spack_root/etc/spack/defaults/config.yaml - :language: yaml - - -As you can see, many of the directories Spack uses can be customized. -For example, you can tell Spack to install packages to a prefix -outside of the ``$SPACK_ROOT`` hierarchy. Module files can be -written to a central location if you are using multiple Spack -instances. If you have a fast scratch file system, you can run builds -from this file system with the following ``config.yaml``: - -.. code-block:: yaml - - config: - build_stage: - - /scratch/$user/spack-stage - - -.. note:: - - It is important to distinguish the build stage directory from other - directories in your scratch space to ensure ``spack clean`` does not - inadvertently remove unrelated files. Spack prepends ``spack-stage-`` - to temporary staging directory names to reduce this risk. Using a - combination of ``spack`` and or ``stage`` in each specified path, as - shown in the default settings and documented examples, will add - another layer of protection. See :ref:`config-yaml` for details. - - -On systems with compilers that absolutely *require* environment variables -like ``LD_LIBRARY_PATH``, it is possible to prevent Spack from cleaning -the build environment with the ``dirty`` setting: - -.. code-block:: yaml - - config: - dirty: true - - -However, this is strongly discouraged, as it can pull unwanted libraries -into the build. - -One last setting that may be of interest to many users is the ability -to customize the parallelism of Spack builds. By default, Spack -installs all packages in parallel with the number of jobs equal to the -number of cores on the node (up to a maximum of 16). For example, on a -node with 16 cores, this will look like: - -.. code-block:: console - - $ spack install --no-cache --verbose --overwrite zlib - ==> Installing zlib - ==> Using cached archive: /home/user/spack/var/spack/cache/zlib/zlib-1.2.11.tar.gz - ==> Staging archive: /home/user/spack/var/spack/stage/zlib-1.2.11-5nus6knzumx4ik2yl44jxtgtsl7d54xb/zlib-1.2.11.tar.gz - ==> Created stage in /home/user/spack/var/spack/stage/zlib-1.2.11-5nus6knzumx4ik2yl44jxtgtsl7d54xb - ==> No patches needed for zlib - ==> Building zlib [Package] - ==> Executing phase: 'install' - ==> './configure' '--prefix=/home/user/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/zlib-1.2.11-5nus6knzumx4ik2yl44jxtgtsl7d54xb' - ... - ==> 'make' '-j16' - ... - ==> 'make' '-j16' 'install' - ... - ==> Successfully installed zlib - Fetch: 0.00s. Build: 1.03s. Total: 1.03s. - [+] /home/user/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/zlib-1.2.11-5nus6knzumx4ik2yl44jxtgtsl7d54xb - - -As you can see, we are building with all 16 cores on the node. If you are -on a shared login node, this can slow down the system for other users. If -you have a strict ulimit or restriction on the number of available licenses, -you may not be able to build at all with this many cores. On nodes with 64+ -cores, you may not see a significant speedup of the build anyway. To limit -the number of cores our build uses, set ``build_jobs`` like so: - -.. code-block:: yaml - - config: - build_jobs: 2 - - -If we uninstall and reinstall zlib, we see that it now uses only 2 cores: - -.. code-block:: console - - $ spack install --no-cache --verbose --overwrite zlib - ==> Installing zlib - ==> Using cached archive: /home/user/spack/var/spack/cache/zlib/zlib-1.2.11.tar.gz - ==> Staging archive: /home/user/spack/var/spack/stage/zlib-1.2.11-5nus6knzumx4ik2yl44jxtgtsl7d54xb/zlib-1.2.11.tar.gz - ==> Created stage in /home/user/spack/var/spack/stage/zlib-1.2.11-5nus6knzumx4ik2yl44jxtgtsl7d54xb - ==> No patches needed for zlib - ==> Building zlib [Package] - ==> Executing phase: 'install' - ==> './configure' '--prefix=/home/user/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/zlib-1.2.11-5nus6knzumx4ik2yl44jxtgtsl7d54xb' - ... - ==> 'make' '-j2' - ... - ==> 'make' '-j2' 'install' - ... - ==> Successfully installed zlib - Fetch: 0.00s. Build: 1.03s. Total: 1.03s. - [+] /home/user/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/zlib-1.2.11-5nus6knzumx4ik2yl44jxtgtsl7d54xb - - -Obviously, if you want to build everything in serial for whatever reason, -you would set ``build_jobs`` to 1. - --------- -Examples --------- - -For examples of how other sites configure Spack, see -https://github.com/spack/spack-configs. If you use Spack at your site -and want to share your config files, feel free to submit a pull request! diff --git a/lib/spack/docs/tutorial_environments.rst b/lib/spack/docs/tutorial_environments.rst deleted file mode 100644 index 4b86dbd5ac..0000000000 --- a/lib/spack/docs/tutorial_environments.rst +++ /dev/null @@ -1,815 +0,0 @@ -.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other - Spack Project Developers. See the top-level COPYRIGHT file for details. - - SPDX-License-Identifier: (Apache-2.0 OR MIT) - -.. _environments-tutorial: - -===================== -Environments Tutorial -===================== - -We've shown you how to install and remove packages with Spack. You can -use :ref:`cmd-spack-install` to install packages, -:ref:`cmd-spack-uninstall` to remove them, and :ref:`cmd-spack-find` to -look at and query what is installed. We've also shown you how to -customize Spack's installation with configuration files like -:ref:`packages.yaml <build-settings>`. - -If you build a lot of software, or if you work on multiple projects, -managing everything in one place can be overwhelming. The default ``spack -find`` output may contain many packages, but you may want to *just* focus -on packages for a particular project. Moreover, you may want to include -special configuration with your package groups, e.g., to build all the -packages in the same group the same way. - -Spack **environments** provide a way to handle these problems. - -------------------- -Environment basics -------------------- - -Let's look at the output of ``spack find`` at this point in the tutorial. - -.. code-block:: console - - $ spack find - ==> 70 installed packages - -- linux-ubuntu16.04-x86_64 / clang@3.8.0-2ubuntu4 -------------- - tcl@8.6.8 zlib@1.2.8 zlib@1.2.11 - - -- linux-ubuntu16.04-x86_64 / gcc@4.7 --------------------------- - zlib@1.2.11 - - -- linux-ubuntu16.04-x86_64 / gcc@5.4.0 ------------------------- - adept-utils@1.0.1 hdf5@1.10.4 mpc@1.1.0 perl@5.26.2 - autoconf@2.69 hdf5@1.10.4 mpfr@3.1.6 pkgconf@1.4.2 - automake@1.16.1 hdf5@1.10.4 mpich@3.2.1 readline@7.0 - boost@1.68.0 hwloc@1.11.9 mpileaks@1.0 suite-sparse@5.3.0 - bzip2@1.0.6 hypre@2.15.1 mumps@5.1.1 tar@1.30 - callpath@1.0.4 hypre@2.15.1 mumps@5.1.1 tcl@8.6.8 - cmake@3.12.3 isl@0.18 ncurses@6.1 tcl@8.6.8 - diffutils@3.6 libdwarf@20180129 netcdf@4.6.1 texinfo@6.5 - dyninst@9.3.2 libiberty@2.31.1 netcdf@4.6.1 trilinos@12.12.1 - elfutils@0.173 libpciaccess@0.13.5 netlib-scalapack@2.0.2 trilinos@12.12.1 - findutils@4.6.0 libsigsegv@2.11 netlib-scalapack@2.0.2 util-macros@1.19.1 - gcc@7.2.0 libtool@2.4.6 numactl@2.0.11 xz@5.2.4 - gdbm@1.14.1 libxml2@2.9.8 openblas@0.3.3 zlib@1.2.8 - gettext@0.19.8.1 m4@1.4.18 openmpi@3.1.3 zlib@1.2.8 - glm@0.9.7.1 matio@1.5.9 openssl@1.0.2o zlib@1.2.11 - gmp@6.1.2 matio@1.5.9 parmetis@4.0.3 - hdf5@1.10.4 metis@5.1.0 parmetis@4.0.3 - - -This is a complete, but cluttered view. There are packages built with -both ``openmpi`` and ``mpich``, as well as multiple variants of other -packages, like ``zlib``. The query mechanism we learned about in ``spack -find`` can help, but it would be nice if we could start from a clean -slate without losing what we've already done. - - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Creating and activating environments -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The ``spack env`` command can help. Let's create a new environment: - -.. code-block:: console - - $ spack env create myproject - ==> Created environment 'myproject' in ~/spack/var/spack/environments/myproject - -An environment is a virtualized ``spack`` instance that you can use for a -specific purpose. You can see the environments we've created so far like this: - -.. code-block:: console - - $ spack env list - ==> 1 environments - myproject - -And you can **activate** an environment with ``spack env activate``: - -.. code-block:: console - - $ spack env activate myproject - -Once you enter an environment, ``spack find`` shows only what is in the -current environment. That's nothing, so far: - -.. code-block:: console - - $ spack find - ==> In environment myproject - ==> No root specs - - ==> 0 installed packages - -The ``spack find`` output is still *slightly* different. It tells you -that you're in the ``myproject`` environment, so that you don't panic -when you see that there is nothing installed. It also says that there -are *no root specs*. We'll get back to what that means later. - -If you *only* want to check what environment you are in, you can use -``spack env status``: - -.. code-block:: console - - $ spack env status - ==> In environment myproject - -And, if you want to leave this environment and go back to normal Spack, -you can use ``spack env deactivate``. We like to use the -``despacktivate`` alias (which Spack sets up automatically) for short: - -.. code-block:: console - - $ despacktivate # short alias for `spack env deactivate` - $ spack env status - ==> No active environment - $ spack find - netcdf@4.6.1 readline@7.0 zlib@1.2.11 - diffutils@3.6 hdf5@1.10.4 m4@1.4.18 netcdf@4.6.1 suite-sparse@5.3.0 - dyninst@10.0.0 hwloc@1.11.9 matio@1.5.9 netlib-scalapack@2.0.2 tar@1.30 - elfutils@0.173 hypre@2.15.1 matio@1.5.9 netlib-scalapack@2.0.2 tcl@8.6.8 - findutils@4.6.0 hypre@2.15.1 metis@5.1.0 numactl@2.0.11 tcl@8.6.8 - gcc@7.2.0 intel-tbb@2019 mpc@1.1.0 openblas@0.3.3 texinfo@6.5~ - - -^^^^^^^^^^^^^^^^^^^ -Installing packages -^^^^^^^^^^^^^^^^^^^ - -Ok, now that we understand how creation and activation work, let's go -back to ``myproject`` and *install* a few packages: - -.. code-block:: console - - $ spack env activate myproject - $ spack install tcl - ==> tcl is already installed in ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/tcl-8.6.8-qhwyccywhx2i6s7ob2gvjrjtj3rnfuqt - $ spack install trilinos - ==> trilinos is already installed in ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/trilinos-12.12.1-rlsruavxqvwk2tgxzxboclbo6ykjf54r - $ spack find - ==> In environment myproject - ==> Root specs - tcl trilinos - - ==> 22 installed packages - -- linux-ubuntu16.04-x86_64 / gcc@5.4.0 ------------------------- - boost@1.68.0 hwloc@1.11.9 matio@1.5.9 netlib-scalapack@2.0.2 parmetis@4.0.3 xz@5.2.4 - bzip2@1.0.6 hypre@2.15.1 metis@5.1.0 numactl@2.0.11 suite-sparse@5.3.0 zlib@1.2.11 - glm@0.9.7.1 libpciaccess@0.13.5 mumps@5.1.1 openblas@0.3.3 tcl@8.6.8 - hdf5@1.10.4 libxml2@2.9.8 netcdf@4.6.1 openmpi@3.1.3 trilinos@12.12.1 - -We've installed ``tcl`` and ``trilinos`` in our environment, along with -all of their dependencies. We call ``tcl`` and ``trilinos`` the -**roots** because we asked for them explicitly. The other 20 packages -listed under "installed packages" are present because they were needed as -dependencies. So, these are the roots of the packages' dependency graph. - -The "<package> is already installed" messages above are generated because -we already installed these packages in previous steps of the tutorial, -and we don't have to rebuild them to put them in an environment. - -Now let's create *another* project. We'll call this one ``myproject2``: - -.. code-block:: console - - $ spack env create myproject2 - ==> Created environment 'myproject2' in ~/spack/var/spack/environments/myproject2 - $ spack env activate myproject2 - $ spack install hdf5 - ==> hdf5 is already installed in ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/hdf5-1.10.4-ozyvmhzdew66byarohm4p36ep7wtcuiw - $ spack install trilinos - ==> trilinos is already installed in ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/trilinos-12.12.1-rlsruavxqvwk2tgxzxboclbo6ykjf54r - $ spack find - ==> In environment myproject2 - ==> Root specs - hdf5 trilinos - - ==> 22 installed packages - -- linux-ubuntu16.04-x86_64 / gcc@5.4.0 ------------------------- - boost@1.68.0 hdf5@1.10.4 libxml2@2.9.8 netcdf@4.6.1 openmpi@3.1.3 xz@5.2.4 - bzip2@1.0.6 hwloc@1.11.9 matio@1.5.9 netlib-scalapack@2.0.2 parmetis@4.0.3 zlib@1.2.11 - glm@0.9.7.1 hypre@2.15.1 metis@5.1.0 numactl@2.0.11 suite-sparse@5.3.0 - hdf5@1.10.4 libpciaccess@0.13.5 mumps@5.1.1 openblas@0.3.3 trilinos@12.12.1 - -Now we have two environments: one with ``tcl`` and ``trilinos``, and -another with ``hdf5`` and ``trilinos``. - -We can uninstall trilinos from ``myproject2`` as you would expect: - -.. code-block:: console - - $ spack uninstall trilinos - ==> The following packages will be uninstalled: - - -- linux-ubuntu16.04-x86_64 / gcc@5.4.0 ------------------------- - rlsruav trilinos@12.12.1%gcc~alloptpkgs+amesos+amesos2+anasazi+aztec+belos+boost build_type=RelWithDebInfo ~cgns~complex~dtk+epetra+epetraext+exodus+explicit_template_instantiation~float+fortran~fortrilinos+gtest+hdf5+hypre+ifpack+ifpack2~intrepid~intrepid2~isorropia+kokkos+metis~minitensor+ml+muelu+mumps~nox~openmp~phalanx~piro~pnetcdf~python~rol~rythmos+sacado~shards+shared~stk+suite-sparse~superlu~superlu-dist~teko~tempus+teuchos+tpetra~x11~xsdkflags~zlib+zoltan+zoltan2 - ==> Do you want to proceed? [y/N] y - $ spack find - ==> In environment myproject2 - ==> Root specs - hdf5 - - ==> 8 installed packages - -- linux-ubuntu16.04-x86_64 / gcc@5.4.0 ------------------------- - hdf5@1.10.4 libpciaccess@0.13.5 numactl@2.0.11 xz@5.2.4 - hwloc@1.11.9 libxml2@2.9.8 openmpi@3.1.3 zlib@1.2.11 - -Now there is only one root spec, ``hdf5``, which requires fewer -additional dependencies. - -However, we still needed ``trilinos`` for the ``myproject`` environment! -What happened to it? Let's switch back and see. - -.. code-block:: console - - $ despacktivate - $ spack env activate myproject - $ spack find - ==> In environment myproject - ==> Root specs - tcl trilinos - - ==> 22 installed packages - -- linux-ubuntu16.04-x86_64 / gcc@5.4.0 ------------------------- - boost@1.68.0 hwloc@1.11.9 matio@1.5.9 netlib-scalapack@2.0.2 parmetis@4.0.3 xz@5.2.4 - bzip2@1.0.6 hypre@2.15.1 metis@5.1.0 numactl@2.0.11 suite-sparse@5.3.0 zlib@1.2.11 - glm@0.9.7.1 libpciaccess@0.13.5 mumps@5.1.1 openblas@0.3.3 tcl@8.6.8 - hdf5@1.10.4 libxml2@2.9.8 netcdf@4.6.1 openmpi@3.1.3 trilinos@12.12.1 - - -Spack is smart enough to realize that ``trilinos`` is still present in -the other environment. Trilinos won't *actually* be uninstalled unless -it is no longer needed by any environments or packages. If it is still -needed, it is only removed from the environment. - -------------------------------- -Dealing with many specs at once -------------------------------- - -In the above examples, we just used ``install`` and ``uninstall``. There -are other ways to deal with groups of packages, as well. - -^^^^^^^^^^^^^ -Adding specs -^^^^^^^^^^^^^ - -Let's go back to our first ``myproject`` environment and *add* a few specs instead of installing them: - -.. code-block:: console - - $ spack add hdf5 - ==> Adding hdf5 to environment myproject - $ spack add gmp - ==> Adding mumps to environment myproject - $ spack find - ==> In environment myproject - ==> Root specs - gmp hdf5 tcl trilinos - - ==> 22 installed packages - -- linux-ubuntu16.04-x86_64 / gcc@5.4.0 ------------------------- - boost@1.68.0 hwloc@1.11.9 matio@1.5.9 netlib-scalapack@2.0.2 parmetis@4.0.3 xz@5.2.4 - bzip2@1.0.6 hypre@2.15.1 metis@5.1.0 numactl@2.0.11 suite-sparse@5.3.0 zlib@1.2.11 - glm@0.9.7.1 libpciaccess@0.13.5 mumps@5.1.1 openblas@0.3.3 tcl@8.6.8 - hdf5@1.10.4 libxml2@2.9.8 netcdf@4.6.1 openmpi@3.1.3 trilinos@12.12.1 - -Let's take a close look at what happened. The two packages we added, -``hdf5`` and ``gmp``, are present, but they're not installed in the -environment yet. ``spack add`` just adds *roots* to the environment, but -it does not automatically install them. - -We can install *all* the as-yet uninstalled packages in an environment by -simply running ``spack install`` with no arguments: - -.. code-block:: console - - $ spack install - ==> Concretizing hdf5 - [+] ozyvmhz hdf5@1.10.4%gcc@5.4.0~cxx~debug~fortran~hl+mpi+pic+shared~szip~threadsafe arch=linux-ubuntu16.04-x86_64 - [+] 3njc4q5 ^openmpi@3.1.3%gcc@5.4.0~cuda+cxx_exceptions fabrics= ~java~legacylaunchers~memchecker~pmi schedulers= ~sqlite3~thread_multiple+vt arch=linux-ubuntu16.04-x86_64 - [+] 43tkw5m ^hwloc@1.11.9%gcc@5.4.0~cairo~cuda+libxml2+pci+shared arch=linux-ubuntu16.04-x86_64 - [+] 5urc6tc ^libpciaccess@0.13.5%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] o2pfwjf ^libtool@2.4.6%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] suf5jtc ^m4@1.4.18%gcc@5.4.0 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu16.04-x86_64 - [+] fypapcp ^libsigsegv@2.11%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] fovrh7a ^pkgconf@1.4.2%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] milz7fm ^util-macros@1.19.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] wpexsph ^libxml2@2.9.8%gcc@5.4.0~python arch=linux-ubuntu16.04-x86_64 - [+] teneqii ^xz@5.2.4%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] 5nus6kn ^zlib@1.2.11%gcc@5.4.0+optimize+pic+shared arch=linux-ubuntu16.04-x86_64 - [+] ft463od ^numactl@2.0.11%gcc@5.4.0 patches=592f30f7f5f757dfc239ad0ffd39a9a048487ad803c26b419e0f96b8cda08c1a arch=linux-ubuntu16.04-x86_64 - [+] 3sx2gxe ^autoconf@2.69%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] ic2kyoa ^perl@5.26.2%gcc@5.4.0+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=linux-ubuntu16.04-x86_64 - [+] q4fpyuo ^gdbm@1.14.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] nxhwrg7 ^readline@7.0%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] 3o765ou ^ncurses@6.1%gcc@5.4.0~symlinks~termlib arch=linux-ubuntu16.04-x86_64 - [+] rymw7im ^automake@1.16.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ==> Concretizing gmp - [+] qc4qcfz gmp@6.1.2%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] 3sx2gxe ^autoconf@2.69%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] suf5jtc ^m4@1.4.18%gcc@5.4.0 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu16.04-x86_64 - [+] fypapcp ^libsigsegv@2.11%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] ic2kyoa ^perl@5.26.2%gcc@5.4.0+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=linux-ubuntu16.04-x86_64 - [+] q4fpyuo ^gdbm@1.14.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] nxhwrg7 ^readline@7.0%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] 3o765ou ^ncurses@6.1%gcc@5.4.0~symlinks~termlib arch=linux-ubuntu16.04-x86_64 - [+] fovrh7a ^pkgconf@1.4.2%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] rymw7im ^automake@1.16.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] o2pfwjf ^libtool@2.4.6%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ==> Installing environment myproject - ==> tcl is already installed in ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/tcl-8.6.8-qhwyccywhx2i6s7ob2gvjrjtj3rnfuqt - ==> trilinos is already installed in ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/trilinos-12.12.1-rlsruavxqvwk2tgxzxboclbo6ykjf54r - ==> hdf5 is already installed in ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/hdf5-1.10.4-ozyvmhzdew66byarohm4p36ep7wtcuiw - ==> gmp is already installed in ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gmp-6.1.2-qc4qcfz4monpllc3nqupdo7vwinf73sw - -Spack will concretize the new roots, and install everything you added to -the environment. Now we can see the installed roots in the output of -``spack find``: - -.. code-block:: console - - $ spack find - ==> In environment myproject - ==> Root specs - gmp hdf5 tcl trilinos - - ==> 24 installed packages - -- linux-ubuntu16.04-x86_64 / gcc@5.4.0 ------------------------- - boost@1.68.0 hdf5@1.10.4 libpciaccess@0.13.5 mumps@5.1.1 openblas@0.3.3 tcl@8.6.8 - bzip2@1.0.6 hdf5@1.10.4 libxml2@2.9.8 netcdf@4.6.1 openmpi@3.1.3 trilinos@12.12.1 - glm@0.9.7.1 hwloc@1.11.9 matio@1.5.9 netlib-scalapack@2.0.2 parmetis@4.0.3 xz@5.2.4 - gmp@6.1.2 hypre@2.15.1 metis@5.1.0 numactl@2.0.11 suite-sparse@5.3.0 zlib@1.2.11 - -We can build whole environments this way, by adding specs and installing -all at once, or we can install them with the usual ``install`` and -``uninstall`` portions. The advantage to doing them all at once is that -we don't have to write a script outside of Spack to automate this, and we -can kick off a large build of many packages easily. - -^^^^^^^^^^^^^^^^^^^^^ -Configuration -^^^^^^^^^^^^^^^^^^^^^ - -So far, ``myproject`` does not have any special configuration associated -with it. The specs concretize using Spack's defaults: - -.. code-block:: console - - $ spack spec hypre - Input spec - -------------------------------- - hypre - - Concretized - -------------------------------- - hypre@2.15.1%gcc@5.4.0~debug~int64+internal-superlu+mpi+shared arch=linux-ubuntu16.04-x86_64 - ^openblas@0.3.3%gcc@5.4.0 cpu_target= ~ilp64 patches=47cfa7a952ac7b2e4632c73ae199d69fb54490627b66a62c681e21019c4ddc9d,714aea33692304a50bd0ccde42590c176c82ded4a8ac7f06e573dc8071929c33 +pic+shared threads=none ~virtual_machine arch=linux-ubuntu16.04-x86_64 - ^openmpi@3.1.3%gcc@5.4.0~cuda+cxx_exceptions fabrics= ~java~legacylaunchers~memchecker~pmi schedulers= ~sqlite3~thread_multiple+vt arch=linux-ubuntu16.04-x86_64 - ^hwloc@1.11.9%gcc@5.4.0~cairo~cuda+libxml2+pci+shared arch=linux-ubuntu16.04-x86_64 - ^libpciaccess@0.13.5%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ^libtool@2.4.6%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ^m4@1.4.18%gcc@5.4.0 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu16.04-x86_64 - ^libsigsegv@2.11%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ^pkgconf@1.4.2%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ^util-macros@1.19.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ^libxml2@2.9.8%gcc@5.4.0~python arch=linux-ubuntu16.04-x86_64 - ^xz@5.2.4%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ^zlib@1.2.11%gcc@5.4.0+optimize+pic+shared arch=linux-ubuntu16.04-x86_64 - ^numactl@2.0.11%gcc@5.4.0 patches=592f30f7f5f757dfc239ad0ffd39a9a048487ad803c26b419e0f96b8cda08c1a arch=linux-ubuntu16.04-x86_64 - ^autoconf@2.69%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ^perl@5.26.2%gcc@5.4.0+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=linux-ubuntu16.04-x86_64 - ^gdbm@1.14.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ^readline@7.0%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ^ncurses@6.1%gcc@5.4.0~symlinks~termlib arch=linux-ubuntu16.04-x86_64 - ^automake@1.16.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - -You may want to add extra configuration to your environment. You can see -how your environment is configured using ``spack config get``: - -.. code-block:: console - - $ spack config get - # This is a Spack Environment file. - # - # It describes a set of packages to be installed, along with - # configuration settings. - spack: - # add package specs to the `specs` list - specs: [tcl, trilinos, hdf5, gmp] - -It turns out that this is a special configuration format where Spack -stores the state for the environment. Currently, the file is just a -``spack:`` header and a list of ``specs``. These are the roots. - -You can edit this file to add your own custom configuration. Spack -provides a shortcut to do that: - -.. code-block:: console - - spack config edit - -You should now see the same file, and edit it to look like this: - -.. code-block:: yaml - - # This is a Spack Environment file. - # - # It describes a set of packages to be installed, along with - # configuration settings. - spack: - packages: - all: - providers: - mpi: [mpich] - - # add package specs to the `specs` list - specs: [tcl, trilinos, hdf5, gmp] - -Now if we run ``spack spec`` again in the environment, specs will concretize with ``mpich`` as the MPI implementation: - -.. code-block:: console - - $ spack spec hypre - Input spec - -------------------------------- - hypre - - Concretized - -------------------------------- - hypre@2.15.1%gcc@5.4.0~debug~int64+internal-superlu+mpi+shared arch=linux-ubuntu16.04-x86_64 - ^mpich@3.2.1%gcc@5.4.0 device=ch3 +hydra netmod=tcp +pmi+romio~verbs arch=linux-ubuntu16.04-x86_64 - ^findutils@4.6.0%gcc@5.4.0 patches=84b916c0bf8c51b7e7b28417692f0ad3e7030d1f3c248ba77c42ede5c1c5d11e,bd9e4e5cc280f9753ae14956c4e4aa17fe7a210f55dd6c84aa60b12d106d47a2 arch=linux-ubuntu16.04-x86_64 - ^autoconf@2.69%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ^m4@1.4.18%gcc@5.4.0 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu16.04-x86_64 - ^libsigsegv@2.11%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ^perl@5.26.2%gcc@5.4.0+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=linux-ubuntu16.04-x86_64 - ^gdbm@1.14.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ^readline@7.0%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ^ncurses@6.1%gcc@5.4.0~symlinks~termlib arch=linux-ubuntu16.04-x86_64 - ^pkgconf@1.4.2%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ^automake@1.16.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ^libtool@2.4.6%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ^texinfo@6.5%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ^openblas@0.3.3%gcc@5.4.0 cpu_target= ~ilp64 patches=47cfa7a952ac7b2e4632c73ae199d69fb54490627b66a62c681e21019c4ddc9d,714aea33692304a50bd0ccde42590c176c82ded4a8ac7f06e573dc8071929c33 +pic+shared threads=none ~virtual_machine arch=linux-ubuntu16.04-x86_64 - -In addition to the ``specs`` section, an environment's configuration can -contain any of the configuration options from Spack's various config -sections. You can add custom repositories, a custom install location, -custom compilers, or custom external packages, in addition to the ``package`` -preferences we show here. - -But now we have a problem. We already installed part of this environment -with openmpi, but now we want to install it with ``mpich``. - -You can run ``spack concretize`` inside of an environment to concretize -all of its specs. We can run it here: - -.. code-block:: console - - $ spack concretize -f - ==> Concretizing tcl - [+] qhwyccy tcl@8.6.8%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] 5nus6kn ^zlib@1.2.11%gcc@5.4.0+optimize+pic+shared arch=linux-ubuntu16.04-x86_64 - ==> Concretizing trilinos - [+] kqc52mo trilinos@12.12.1%gcc@5.4.0~alloptpkgs+amesos+amesos2+anasazi+aztec+belos+boost build_type=RelWithDebInfo ~cgns~complex~dtk+epetra+epetraext+exodus+explicit_template_instantiation~float+fortran~fortrilinos+gtest+hdf5+hypre+ifpack+ifpack2~intrepid~intrepid2~isorropia+kokkos+metis~minitensor+ml+muelu+mumps~nox~openmp~phalanx~piro~pnetcdf~python~rol~rythmos+sacado~shards+shared~stk+suite-sparse~superlu~superlu-dist~teko~tempus+teuchos+tpetra~x11~xsdkflags~zlib+zoltan+zoltan2 arch=linux-ubuntu16.04-x86_64 - [+] zbgfxap ^boost@1.68.0%gcc@5.4.0+atomic+chrono~clanglibcpp cxxstd=default +date_time~debug+exception+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy patches=2ab6c72d03dec6a4ae20220a9dfd5c8c572c5294252155b85c6874d97c323199 +program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave arch=linux-ubuntu16.04-x86_64 - [+] ufczdvs ^bzip2@1.0.6%gcc@5.4.0+shared arch=linux-ubuntu16.04-x86_64 - [+] 2rhuivg ^diffutils@3.6%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] 5nus6kn ^zlib@1.2.11%gcc@5.4.0+optimize+pic+shared arch=linux-ubuntu16.04-x86_64 - [+] otafqzh ^cmake@3.12.3%gcc@5.4.0~doc+ncurses+openssl+ownlibs patches=dd3a40d4d92f6b2158b87d6fb354c277947c776424aa03f6dc8096cf3135f5d0 ~qt arch=linux-ubuntu16.04-x86_64 - [+] 3o765ou ^ncurses@6.1%gcc@5.4.0~symlinks~termlib arch=linux-ubuntu16.04-x86_64 - [+] fovrh7a ^pkgconf@1.4.2%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] b4y3w3b ^openssl@1.0.2o%gcc@5.4.0+systemcerts arch=linux-ubuntu16.04-x86_64 - [+] ic2kyoa ^perl@5.26.2%gcc@5.4.0+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=linux-ubuntu16.04-x86_64 - [+] q4fpyuo ^gdbm@1.14.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] nxhwrg7 ^readline@7.0%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] jnw622j ^glm@0.9.7.1%gcc@5.4.0 build_type=RelWithDebInfo arch=linux-ubuntu16.04-x86_64 - [+] xxd7syh ^hdf5@1.10.4%gcc@5.4.0~cxx~debug~fortran+hl+mpi+pic+shared~szip~threadsafe arch=linux-ubuntu16.04-x86_64 - [+] p3f7p2r ^mpich@3.2.1%gcc@5.4.0 device=ch3 +hydra netmod=tcp +pmi+romio~verbs arch=linux-ubuntu16.04-x86_64 - [+] d4iajxs ^findutils@4.6.0%gcc@5.4.0 patches=84b916c0bf8c51b7e7b28417692f0ad3e7030d1f3c248ba77c42ede5c1c5d11e,bd9e4e5cc280f9753ae14956c4e4aa17fe7a210f55dd6c84aa60b12d106d47a2 arch=linux-ubuntu16.04-x86_64 - [+] 3sx2gxe ^autoconf@2.69%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] suf5jtc ^m4@1.4.18%gcc@5.4.0 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu16.04-x86_64 - [+] fypapcp ^libsigsegv@2.11%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] rymw7im ^automake@1.16.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] o2pfwjf ^libtool@2.4.6%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] zs7a2pc ^texinfo@6.5%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] obewuoz ^hypre@2.15.1%gcc@5.4.0~debug~int64~internal-superlu+mpi+shared arch=linux-ubuntu16.04-x86_64 - [+] cyeg2yi ^openblas@0.3.3%gcc@5.4.0 cpu_target= ~ilp64 patches=47cfa7a952ac7b2e4632c73ae199d69fb54490627b66a62c681e21019c4ddc9d,714aea33692304a50bd0ccde42590c176c82ded4a8ac7f06e573dc8071929c33 +pic+shared threads=none ~virtual_machine arch=linux-ubuntu16.04-x86_64 - [+] gvyqldh ^matio@1.5.9%gcc@5.4.0+hdf5+shared+zlib arch=linux-ubuntu16.04-x86_64 - [+] 3wnvp4j ^metis@5.1.0%gcc@5.4.0 build_type=Release ~gdb~int64 patches=4991da938c1d3a1d3dea78e49bbebecba00273f98df2a656e38b83d55b281da1 ~real64+shared arch=linux-ubuntu16.04-x86_64 - [+] cumcj5a ^mumps@5.1.1%gcc@5.4.0+complex+double+float~int64~metis+mpi~parmetis~ptscotch~scotch+shared arch=linux-ubuntu16.04-x86_64 - [+] p7iln2p ^netlib-scalapack@2.0.2%gcc@5.4.0 build_type=RelWithDebInfo ~pic+shared arch=linux-ubuntu16.04-x86_64 - [+] wmmx5sg ^netcdf@4.6.1%gcc@5.4.0~dap~hdf4 maxdims=1024 maxvars=8192 +mpi~parallel-netcdf+shared arch=linux-ubuntu16.04-x86_64 - [+] jehtata ^parmetis@4.0.3%gcc@5.4.0 build_type=RelWithDebInfo ~gdb patches=4f892531eb0a807eb1b82e683a416d3e35154a455274cf9b162fb02054d11a5b,50ed2081bc939269689789942067c58b3e522c269269a430d5d34c00edbc5870,704b84f7c7444d4372cb59cca6e1209df4ef3b033bc4ee3cf50f369bce972a9d +shared arch=linux-ubuntu16.04-x86_64 - [+] zaau4ki ^suite-sparse@5.3.0%gcc@5.4.0~cuda~openmp+pic~tbb arch=linux-ubuntu16.04-x86_64 - ==> Concretizing hdf5 - - zjgyn3w hdf5@1.10.4%gcc@5.4.0~cxx~debug~fortran~hl+mpi+pic+shared~szip~threadsafe arch=linux-ubuntu16.04-x86_64 - [+] p3f7p2r ^mpich@3.2.1%gcc@5.4.0 device=ch3 +hydra netmod=tcp +pmi+romio~verbs arch=linux-ubuntu16.04-x86_64 - [+] d4iajxs ^findutils@4.6.0%gcc@5.4.0 patches=84b916c0bf8c51b7e7b28417692f0ad3e7030d1f3c248ba77c42ede5c1c5d11e,bd9e4e5cc280f9753ae14956c4e4aa17fe7a210f55dd6c84aa60b12d106d47a2 arch=linux-ubuntu16.04-x86_64 - [+] 3sx2gxe ^autoconf@2.69%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] suf5jtc ^m4@1.4.18%gcc@5.4.0 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu16.04-x86_64 - [+] fypapcp ^libsigsegv@2.11%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] ic2kyoa ^perl@5.26.2%gcc@5.4.0+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=linux-ubuntu16.04-x86_64 - [+] q4fpyuo ^gdbm@1.14.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] nxhwrg7 ^readline@7.0%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] 3o765ou ^ncurses@6.1%gcc@5.4.0~symlinks~termlib arch=linux-ubuntu16.04-x86_64 - [+] fovrh7a ^pkgconf@1.4.2%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] rymw7im ^automake@1.16.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] o2pfwjf ^libtool@2.4.6%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] zs7a2pc ^texinfo@6.5%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] 5nus6kn ^zlib@1.2.11%gcc@5.4.0+optimize+pic+shared arch=linux-ubuntu16.04-x86_64 - ==> Concretizing gmp - [+] qc4qcfz gmp@6.1.2%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] 3sx2gxe ^autoconf@2.69%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] suf5jtc ^m4@1.4.18%gcc@5.4.0 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu16.04-x86_64 - [+] fypapcp ^libsigsegv@2.11%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] ic2kyoa ^perl@5.26.2%gcc@5.4.0+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=linux-ubuntu16.04-x86_64 - [+] q4fpyuo ^gdbm@1.14.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] nxhwrg7 ^readline@7.0%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] 3o765ou ^ncurses@6.1%gcc@5.4.0~symlinks~termlib arch=linux-ubuntu16.04-x86_64 - [+] fovrh7a ^pkgconf@1.4.2%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] rymw7im ^automake@1.16.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] o2pfwjf ^libtool@2.4.6%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - -Now, all the specs in the environment are concrete and ready to be -installed with ``mpich`` as the MPI implementation. - -Normally, we could just run ``spack config edit``, edit the environment -configuration, ``spack add`` some specs, and ``spack install``. - -But, when we already have installed packages in the environment, we have -to force everything in the environment to be re-concretized using ``spack -concretize -f``. *Then* we can re-run ``spack install``. - - ------------------------------------ -``spack.yaml`` and ``spack.lock`` ------------------------------------ - -So far we've shown you how to interact with environments from the command -line, but they also have a file-based interface that can be used by -developers and admins to manage workflows for projects. - -In this section we'll dive a little deeper to see how environments are -implemented, and how you could use this in your day-to-day development. - -^^^^^^^^^^^^^^ -``spack.yaml`` -^^^^^^^^^^^^^^ - -Earlier, we changed an environment's configuration using ``spack config -edit``. We were actually editing a special file called ``spack.yaml``. -Let's take a look. - -We can get directly to the current environment's location using ``spack cd``: - -.. code-block:: console - - $ spack cd -e myproject - $ pwd - ~/spack/var/spack/environments/myproject - $ ls - spack.lock spack.yaml - -We notice two things here. First, the environment is just a directory -inside of ``var/spack/environments`` within the Spack installation. -Second, it contains two important files: ``spack.yaml`` and -``spack.lock``. - -``spack.yaml`` is the configuration file for environments that we've -already seen, but it does not *have* to live inside Spack. If you create -an environment using ``spack env create``, it is *managed* by -Spack in the ``var/spack/environments`` directory, and you can refer to -it by name. - -You can actually put a ``spack.yaml`` file *anywhere*, and you can use it -to bundle an environment, or a list of dependencies to install, with your -project. Let's make a simple project: - -.. code-block:: console - - $ cd - $ mkdir code - $ cd code - $ spack env create -d . - ==> Created environment in ~/code - -Here, we made a new directory called *code*, and we used the ``-d`` -option to create an environment in it. - -What really happened? - -.. code-block:: console - - $ ls - spack.yaml - $ cat spack.yaml - # This is a Spack Environment file. - # - # It describes a set of packages to be installed, along with - # configuration settings. - spack: - # add package specs to the `specs` list - specs: [] - -Spack just created a ``spack.yaml`` file in the code directory, with an -empty list of root specs. Now we have a Spack environment, *in a -directory*, that we can use to manage dependencies. Suppose your project -depends on ``boost``, ``trilinos``, and ``openmpi``. You can add these -to your spec list: - -.. code-block:: yaml - - # This is a Spack Environment file. - # - # It describes a set of packages to be installed, along with - # configuration settings. - spack: - # add package specs to the `specs` list - specs: - - boost - - trilinos - - openmpi - -And now *anyone* who uses the *code* repository can use this format to -install the project's dependencies. They need only clone the repository, -``cd`` into it, and type ``spack install``: - -.. code-block:: console - - $ spack install - ==> Concretizing boost - [+] zbgfxap boost@1.68.0%gcc@5.4.0+atomic+chrono~clanglibcpp cxxstd=default +date_time~debug+exception+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy patches=2ab6c72d03dec6a4ae20220a9dfd5c8c572c5294252155b85c6874d97c323199 +program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave arch=linux-ubuntu16.04-x86_64 - [+] ufczdvs ^bzip2@1.0.6%gcc@5.4.0+shared arch=linux-ubuntu16.04-x86_64 - [+] 2rhuivg ^diffutils@3.6%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] 5nus6kn ^zlib@1.2.11%gcc@5.4.0+optimize+pic+shared arch=linux-ubuntu16.04-x86_64 - ==> Concretizing trilinos - [+] rlsruav trilinos@12.12.1%gcc@5.4.0~alloptpkgs+amesos+amesos2+anasazi+aztec+belos+boost build_type=RelWithDebInfo ~cgns~complex~dtk+epetra+epetraext+exodus+explicit_template_instantiation~float+fortran~fortrilinos+gtest+hdf5+hypre+ifpack+ifpack2~intrepid~intrepid2~isorropia+kokkos+metis~minitensor+ml+muelu+mumps~nox~openmp~phalanx~piro~pnetcdf~python~rol~rythmos+sacado~shards+shared~stk+suite-sparse~superlu~superlu-dist~teko~tempus+teuchos+tpetra~x11~xsdkflags~zlib+zoltan+zoltan2 arch=linux-ubuntu16.04-x86_64 - [+] zbgfxap ^boost@1.68.0%gcc@5.4.0+atomic+chrono~clanglibcpp cxxstd=default +date_time~debug+exception+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy patches=2ab6c72d03dec6a4ae20220a9dfd5c8c572c5294252155b85c6874d97c323199 +program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave arch=linux-ubuntu16.04-x86_64 - [+] ufczdvs ^bzip2@1.0.6%gcc@5.4.0+shared arch=linux-ubuntu16.04-x86_64 - [+] 2rhuivg ^diffutils@3.6%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] 5nus6kn ^zlib@1.2.11%gcc@5.4.0+optimize+pic+shared arch=linux-ubuntu16.04-x86_64 - [+] otafqzh ^cmake@3.12.3%gcc@5.4.0~doc+ncurses+openssl+ownlibs patches=dd3a40d4d92f6b2158b87d6fb354c277947c776424aa03f6dc8096cf3135f5d0 ~qt arch=linux-ubuntu16.04-x86_64 - [+] 3o765ou ^ncurses@6.1%gcc@5.4.0~symlinks~termlib arch=linux-ubuntu16.04-x86_64 - [+] fovrh7a ^pkgconf@1.4.2%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] b4y3w3b ^openssl@1.0.2o%gcc@5.4.0+systemcerts arch=linux-ubuntu16.04-x86_64 - [+] ic2kyoa ^perl@5.26.2%gcc@5.4.0+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=linux-ubuntu16.04-x86_64 - [+] q4fpyuo ^gdbm@1.14.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] nxhwrg7 ^readline@7.0%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] jnw622j ^glm@0.9.7.1%gcc@5.4.0 build_type=RelWithDebInfo arch=linux-ubuntu16.04-x86_64 - [+] oqwnui7 ^hdf5@1.10.4%gcc@5.4.0~cxx~debug~fortran+hl+mpi+pic+shared~szip~threadsafe arch=linux-ubuntu16.04-x86_64 - [+] 3njc4q5 ^openmpi@3.1.3%gcc@5.4.0~cuda+cxx_exceptions fabrics= ~java~legacylaunchers~memchecker~pmi schedulers= ~sqlite3~thread_multiple+vt arch=linux-ubuntu16.04-x86_64 - [+] 43tkw5m ^hwloc@1.11.9%gcc@5.4.0~cairo~cuda+libxml2+pci+shared arch=linux-ubuntu16.04-x86_64 - [+] 5urc6tc ^libpciaccess@0.13.5%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] o2pfwjf ^libtool@2.4.6%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] suf5jtc ^m4@1.4.18%gcc@5.4.0 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu16.04-x86_64 - [+] fypapcp ^libsigsegv@2.11%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] milz7fm ^util-macros@1.19.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] wpexsph ^libxml2@2.9.8%gcc@5.4.0~python arch=linux-ubuntu16.04-x86_64 - [+] teneqii ^xz@5.2.4%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] ft463od ^numactl@2.0.11%gcc@5.4.0 patches=592f30f7f5f757dfc239ad0ffd39a9a048487ad803c26b419e0f96b8cda08c1a arch=linux-ubuntu16.04-x86_64 - [+] 3sx2gxe ^autoconf@2.69%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] rymw7im ^automake@1.16.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] fshksdp ^hypre@2.15.1%gcc@5.4.0~debug~int64~internal-superlu+mpi+shared arch=linux-ubuntu16.04-x86_64 - [+] cyeg2yi ^openblas@0.3.3%gcc@5.4.0 cpu_target= ~ilp64 patches=47cfa7a952ac7b2e4632c73ae199d69fb54490627b66a62c681e21019c4ddc9d,714aea33692304a50bd0ccde42590c176c82ded4a8ac7f06e573dc8071929c33 +pic+shared threads=none ~virtual_machine arch=linux-ubuntu16.04-x86_64 - [+] lmzdgss ^matio@1.5.9%gcc@5.4.0+hdf5+shared+zlib arch=linux-ubuntu16.04-x86_64 - [+] 3wnvp4j ^metis@5.1.0%gcc@5.4.0 build_type=Release ~gdb~int64 patches=4991da938c1d3a1d3dea78e49bbebecba00273f98df2a656e38b83d55b281da1 ~real64+shared arch=linux-ubuntu16.04-x86_64 - [+] acsg2dz ^mumps@5.1.1%gcc@5.4.0+complex+double+float~int64~metis+mpi~parmetis~ptscotch~scotch+shared arch=linux-ubuntu16.04-x86_64 - [+] wotpfwf ^netlib-scalapack@2.0.2%gcc@5.4.0 build_type=RelWithDebInfo ~pic+shared arch=linux-ubuntu16.04-x86_64 - [+] mhm4izp ^netcdf@4.6.1%gcc@5.4.0~dap~hdf4 maxdims=1024 maxvars=8192 +mpi~parallel-netcdf+shared arch=linux-ubuntu16.04-x86_64 - [+] uv6h3sq ^parmetis@4.0.3%gcc@5.4.0 build_type=RelWithDebInfo ~gdb patches=4f892531eb0a807eb1b82e683a416d3e35154a455274cf9b162fb02054d11a5b,50ed2081bc939269689789942067c58b3e522c269269a430d5d34c00edbc5870,704b84f7c7444d4372cb59cca6e1209df4ef3b033bc4ee3cf50f369bce972a9d +shared arch=linux-ubuntu16.04-x86_64 - [+] zaau4ki ^suite-sparse@5.3.0%gcc@5.4.0~cuda~openmp+pic~tbb arch=linux-ubuntu16.04-x86_64 - ==> Concretizing openmpi - [+] 3njc4q5 openmpi@3.1.3%gcc@5.4.0~cuda+cxx_exceptions fabrics= ~java~legacylaunchers~memchecker~pmi schedulers= ~sqlite3~thread_multiple+vt arch=linux-ubuntu16.04-x86_64 - [+] 43tkw5m ^hwloc@1.11.9%gcc@5.4.0~cairo~cuda+libxml2+pci+shared arch=linux-ubuntu16.04-x86_64 - [+] 5urc6tc ^libpciaccess@0.13.5%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] o2pfwjf ^libtool@2.4.6%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] suf5jtc ^m4@1.4.18%gcc@5.4.0 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu16.04-x86_64 - [+] fypapcp ^libsigsegv@2.11%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] fovrh7a ^pkgconf@1.4.2%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] milz7fm ^util-macros@1.19.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] wpexsph ^libxml2@2.9.8%gcc@5.4.0~python arch=linux-ubuntu16.04-x86_64 - [+] teneqii ^xz@5.2.4%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] 5nus6kn ^zlib@1.2.11%gcc@5.4.0+optimize+pic+shared arch=linux-ubuntu16.04-x86_64 - [+] ft463od ^numactl@2.0.11%gcc@5.4.0 patches=592f30f7f5f757dfc239ad0ffd39a9a048487ad803c26b419e0f96b8cda08c1a arch=linux-ubuntu16.04-x86_64 - [+] 3sx2gxe ^autoconf@2.69%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] ic2kyoa ^perl@5.26.2%gcc@5.4.0+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=linux-ubuntu16.04-x86_64 - [+] q4fpyuo ^gdbm@1.14.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] nxhwrg7 ^readline@7.0%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - [+] 3o765ou ^ncurses@6.1%gcc@5.4.0~symlinks~termlib arch=linux-ubuntu16.04-x86_64 - [+] rymw7im ^automake@1.16.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64 - ==> Installing environment ~/code - ==> boost is already installed in ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/boost-1.68.0-zbgfxapchxa4awxdwpleubfuznblxzvt - ==> trilinos is already installed in ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/trilinos-12.12.1-rlsruavxqvwk2tgxzxboclbo6ykjf54r - ==> openmpi is already installed in ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.1.3-3njc4q5pqdpptq6jvqjrezkffwokv2sx - - -Spack concretizes the specs in the ``spack.yaml`` file and installs them. - -What happened here? If you ``cd`` into a directory that has a -``spack.yaml`` file in it, Spack considers this directory's environment -to be activated. The directory does not have to live within Spack; it -can be anywhere. - -So, from ``~/code``, we can actually manipulate ``spack.yaml`` using -``spack add`` and ``spack remove`` (just like managed environments): - -.. code-block:: console - - $ spack add hdf5@5.5.1 - ==> Adding hdf5 to environment ~/code - $ cat spack.yaml - # This is a Spack Environment file. - # - # It describes a set of packages to be installed, along with - # configuration settings. - spack: - # add package specs to the `specs` list - specs: - - boost - - trilinos - - openmpi - - hdf5@5.5.1 - - $ spack remove hdf5 - ==> Removing hdf5 from environment ~/code - $ cat spack.yaml - # This is a Spack Environment file. - # - # It describes a set of packages to be installed, along with - # configuration settings. - spack: - # add package specs to the `specs` list - specs: - - boost - - trilinos - - openmpi - - -^^^^^^^^^^^^^^ -``spack.lock`` -^^^^^^^^^^^^^^ - -Okay, we've covered managed environments, environments in directories, and -the last thing we'll cover is ``spack.lock``. You may remember that when -we ran ``spack install``, Spack concretized all the specs in the -``spack.yaml`` file and installed them. - -Whenever we concretize Specs in an environment, all concrete specs in the -environment are written out to a ``spack.lock`` file *alongside* -``spack.yaml``. The ``spack.lock`` file is not really human-readable -like the ``spack.yaml`` file. It is a ``json`` format that contains all -the information that we need to *reproduce* the build of an -environment: - -.. code-block:: console - - $ head spack.lock - { - "concrete_specs": { - "teneqii2xv5u6zl5r6qi3pwurc6pmypz": { - "xz": { - "version": "5.2.4", - "arch": { - "platform": "linux", - "platform_os": "ubuntu16.04", - "target": "x86_64" - }, - ... - -``spack.yaml`` and ``spack.lock`` correspond to two fundamental concepts -in Spack, but for environments: - - * ``spack.yaml`` is the set of *abstract* specs and configuration that - you want to install. - * ``spack.lock`` is the set of all fully *concretized* specs generated - from concretizing ``spack.yaml`` - -Using either of these, you can recreate an environment that someone else -built. ``spack env create`` takes an extra optional argument, which can -be either a ``spack.yaml`` or a ``spack.lock`` file: - -.. code-block:: console - - $ spack env create my-project spack.yaml - - $ spack env create my-project spack.lock - -Both of these create a new environment called ``my-project``, but which -one you choose to use depends on your needs: - -#. copying the yaml file allows someone else to build your *requirements*, - potentially a different way. - -#. copying the lock file allows someone else to rebuild your - *installation* exactly as you built it. - -The first use case can *re-concretize* the same specs on new platforms in -order to build, but it will preserve the abstract requirements. The -second use case (currently) requires you to be on the same machine, but -it retains all decisions made during concretization and is faithful to a -prior install. diff --git a/lib/spack/docs/tutorial_modules.rst b/lib/spack/docs/tutorial_modules.rst deleted file mode 100644 index 686da9f26e..0000000000 --- a/lib/spack/docs/tutorial_modules.rst +++ /dev/null @@ -1,1575 +0,0 @@ -.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other - Spack Project Developers. See the top-level COPYRIGHT file for details. - - SPDX-License-Identifier: (Apache-2.0 OR MIT) - -.. _modules-tutorial: - -============ -Module Files -============ - -In this tutorial, we'll introduce a few concepts that are fundamental -to the generation of module files with Spack, and we'll guide you through -the customization of both module files content and their layout on disk. In the end you -should have a clear understanding of: - - * What are module files and how they work - * How Spack generates them - * Which commands are available to ease their maintenance - * How it is possible to customize them in all aspects - -.. _module_file_tutorial_overview: - -------------------- -Modules at a glance -------------------- - -Let's start by summarizing what module files are and how you can use -them to modify your environment. The idea is to give enough information so that -people without any previous exposure to them will be able to follow the tutorial -later on. We'll also give a high-level view of how module files are generated -in Spack. If you are already familiar with these topics you can quickly skim -through this section or move directly to :ref:`module_file_tutorial_prerequisites`. - -.. _module_file_tutorial_what_are_modules: - -^^^^^^^^^^^^^^^^^^^^^^ -What are module files? -^^^^^^^^^^^^^^^^^^^^^^ - -Module files are an easy way to modify your environment in a controlled manner -during a shell session. In general, they contain the information needed to run an -application or use a library, and they work in conjunction with a tool that -interprets them. -Typical module files instruct this tool to modify the environment variables when a -module file is loaded: - - .. code-block:: console - - $ module show zlib - ------------------------------------------------------------------- - /home/mculpo/PycharmProjects/spack/share/spack/modules/linux-ubuntu14.04-x86_64/zlib/1.2.11-gcc-7.2.0-linux-ubuntu14.04-x86_64-co2px3k: - - module-whatis A free, general-purpose, legally unencumbered lossless data-compression library. - prepend-path MANPATH /home/mculpo/PycharmProjects/spack/opt/spack/linux-ubuntu14.04-x86_64/gcc-7.2.0/zlib-1.2.11-co2px3k53m76lm6tofylh2mur2hnicux/share/man - prepend-path LIBRARY_PATH /home/mculpo/PycharmProjects/spack/opt/spack/linux-ubuntu14.04-x86_64/gcc-7.2.0/zlib-1.2.11-co2px3k53m76lm6tofylh2mur2hnicux/lib - prepend-path LD_LIBRARY_PATH /home/mculpo/PycharmProjects/spack/opt/spack/linux-ubuntu14.04-x86_64/gcc-7.2.0/zlib-1.2.11-co2px3k53m76lm6tofylh2mur2hnicux/lib - prepend-path CPATH /home/mculpo/PycharmProjects/spack/opt/spack/linux-ubuntu14.04-x86_64/gcc-7.2.0/zlib-1.2.11-co2px3k53m76lm6tofylh2mur2hnicux/include - prepend-path PKG_CONFIG_PATH /home/mculpo/PycharmProjects/spack/opt/spack/linux-ubuntu14.04-x86_64/gcc-7.2.0/zlib-1.2.11-co2px3k53m76lm6tofylh2mur2hnicux/lib/pkgconfig - prepend-path CMAKE_PREFIX_PATH /home/mculpo/PycharmProjects/spack/opt/spack/linux-ubuntu14.04-x86_64/gcc-7.2.0/zlib-1.2.11-co2px3k53m76lm6tofylh2mur2hnicux/ - ------------------------------------------------------------------- - - $ echo $LD_LIBRARY_PATH - - $ module load zlib - $ echo $LD_LIBRARY_PATH - /home/mculpo/PycharmProjects/spack/opt/spack/linux-ubuntu14.04-x86_64/gcc-7.2.0/zlib-1.2.11-co2px3k53m76lm6tofylh2mur2hnicux/lib - -and to undo the modifications when the same module file is unloaded: - - .. code-block:: console - - $ module unload zlib - $ echo $LD_LIBRARY_PATH - - $ - -Different formats exist for module files, and different tools -provide various levels of support for them. Spack can natively generate: - -1. Non-hierarchical module files written in TCL -2. Hierarchical module files written in Lua - -and can build `environment-modules <http://modules.sourceforge.net/>`_ -and `lmod <http://lmod.readthedocs.io/en/latest>`_ as support tools. -Which of the formats or tools best suits one's needs depends on each particular -use-case. For the sake of illustration, we'll be working on -both formats using ``lmod``. - -.. seealso:: - Environment modules - This is the original tool that provided modules support. Its first - version was coded in C in the early '90s and was later substituted by a version - completely coded in TCL - the one Spack is distributing. More details on - its features are given in the `homepage of the project <http://modules.sourceforge.net/>`_ - or in its `github page <https://github.com/cea-hpc/modules>`_. The tool is able to - interpret the non-hierarchical TCL modulefiles written by Spack. - - Lmod - Lmod is a module system written in Lua, designed to easily handle hierarchies of - module files. It's a drop-in replacement of Environment Modules and works with - both of the module file formats generated by Spack. - Despite being fully compatible with Environment Modules there are many features that - are unique to Lmod. These features are either - `targeted towards safety <http://lmod.readthedocs.io/en/latest/010_user.html#safety-features>`_ - or meant to - `extend the module system functionality <http://lmod.readthedocs.io/en/latest/010_user.html#module-hierarchy>`_. - - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -How do we generate module files? -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Before we dive into the hands-on sections it's worth spending a couple of words to explain how -module files are generated by Spack. The following diagram provides a high-level view -of the process: - - -.. image:: module_file_generation.* - -The red dashed line above represents Spack's boundaries, the blue one Spack's dependencies [#f1]_. -Module files are generated by combining: - - * the configuration details in ``config.yaml`` and ``modules.yaml`` - * the information contained in Spack packages (and processed by the module subpackage) - * a set of template files - -with `Jinja2 <http://jinja.pocoo.org/docs/2.9/>`_, an external template engine -that stamps out each particular module file. As Spack serves very diverse needs -this process has many points of customization, and we'll explore most of -them in the next sections. - -.. [#f1] Spack vendors its dependencies! This means that Spack comes with a copy of - each one of its dependencies, including ``Jinja2``, and is already configured to use them. - -.. _module_file_tutorial_prerequisites: - ----------------------- -Setup for the tutorial ----------------------- - -In order to showcase the capabilities of Spack's module file generation, we need -a representative set of software to work with. This set must include different -flavors of the same packages installed alongside each other and some -:ref:`external packages <sec-external-packages>`. - -The purpose of this setup is not to make our life harder but to demonstrate -how Spack can help with similar situations, as they will happen on real HPC clusters. -For instance, it's often preferable for Spack to use vendor-provided MPI -implementations than to build one itself. - -To keep the set of software we're dealing with manageable, we're going -to uninstall everything from earlier in the tutorial. - -.. code-block: console - - $ spack uninstall -ay - -^^^^^^^^^^^^^^^^^^^ -Build a module tool -^^^^^^^^^^^^^^^^^^^ - -The first thing that we need is the module tool. In this case we -choose ``lmod`` as it can work with both hierarchical and non-hierarchical -module file layouts. - -.. code-block:: console - - $ bin/spack install lmod - -Once the module tool is installed we need to have it available in the -current shell. As the installation directories are definitely not easy -to remember, we'll employ the command ``spack location`` to retrieve the -``lmod`` prefix directly from Spack: - -.. code-block:: console - - $ . $(spack location -i lmod)/lmod/lmod/init/bash - -Now we can re-source the setup file and Spack modules will be put in -our module path. - -.. code-block:: console - - $ . share/spack/setup-env.sh - -.. FIXME: this needs bootstrap support for ``lmod`` - -.. FIXME: check the docs here, update them if necessary - If you need to install Lmod or Environment module you can refer - to the documentation :ref:`here <InstallEnvironmentModules>`. - - -^^^^^^^^^^^^^^^^^^ -Add a new compiler -^^^^^^^^^^^^^^^^^^ - -The second step is to build a recent compiler. On first use, Spack -scans the environment and automatically locates the compiler(s) -already available on the system. For this tutorial, however, we want -to use ``gcc@7.2.0``. - - -.. code-block:: console - - $ spack install gcc@7.2.0 - ... - Wait a long time - ... - -Once ``gcc`` is installed we can use shell support to load it and make -it readily available: - -.. code-block:: console - - $ spack load gcc@7.2.0 - -It may not be apparent, but the last command employed the module files -generated automatically by Spack. What happens under the hood when you use -the ``spack load`` command is: - -1. the spec passed as argument is translated into a module file name -2. the current module tool is used to load that module file - -You can use this command to double check: - -.. code-block:: console - - $ module list - Currently Loaded Modules: - 1) gcc-7.2.0-gcc-5.4.0-b7smjjc - -Note that the 7-digit hash at the end of the generated module may vary depending -on architecture or package version. Now that we have ``gcc@7.2.0`` in ``PATH`` we -can finally add it to the list of compilers known to Spack: - -.. code-block:: console - - $ spack compiler add - ==> Added 1 new compiler to /home/spack1/.spack/linux/compilers.yaml - gcc@7.2.0 - ==> Compilers are defined in the following files: - /home/spack1/.spack/linux/compilers.yaml - - $ spack compiler list - ==> Available compilers - -- clang ubuntu16.04-x86_64 ------------------------------------- - clang@3.8.0-2ubuntu4 clang@3.7.1-2ubuntu2 - - -- gcc ubuntu16.04-x86_64 --------------------------------------- - gcc@7.2.0 gcc@5.4.0 gcc@4.7 - - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Build the software that will be used in the tutorial -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Finally, we should use Spack to install the packages used in the examples: - -.. code-block:: console - - $ spack install netlib-scalapack ^openmpi ^openblas - $ spack install netlib-scalapack ^mpich ^openblas - $ spack install netlib-scalapack ^openmpi ^netlib-lapack - $ spack install netlib-scalapack ^mpich ^netlib-lapack - $ spack install py-scipy ^openblas - - -.. _module_file_tutorial_non_hierarchical: - ------------------------------ -Non-hierarchical module files ------------------------------ - -If you arrived to this point you should have an environment that looks similar to: - -.. code-block:: console - - $ module avail - - ----------------------------------------------- /home/spack1/spack/share/spack/modules/linux-ubuntu16.04-x86_64 ----------------------------------------------- - autoconf-2.69-gcc-5.4.0-3sx2gxe libsigsegv-2.11-gcc-7.2.0-g67xpfd openssl-1.0.2o-gcc-5.4.0-b4y3w3b - autoconf-2.69-gcc-7.2.0-yb2makb libtool-2.4.6-gcc-5.4.0-o2pfwjf openssl-1.0.2o-gcc-7.2.0-cvldq3v - automake-1.16.1-gcc-5.4.0-rymw7im libtool-2.4.6-gcc-7.2.0-kt2udm6 pcre-8.42-gcc-5.4.0-gt5lgzi - automake-1.16.1-gcc-7.2.0-qoowd5q libxml2-2.9.8-gcc-5.4.0-wpexsph perl-5.26.2-gcc-5.4.0-ic2kyoa - bzip2-1.0.6-gcc-5.4.0-ufczdvs libxml2-2.9.8-gcc-7.2.0-47gf5kk perl-5.26.2-gcc-7.2.0-fdwz5yu - bzip2-1.0.6-gcc-7.2.0-mwamumj lmod-7.8-gcc-5.4.0-kmhks3p pkgconf-1.4.2-gcc-5.4.0-fovrh7a - cmake-3.12.3-gcc-7.2.0-obqgn2v lua-5.3.4-gcc-5.4.0-cpfeo2w pkgconf-1.4.2-gcc-7.2.0-yoxwmgb - curl-7.60.0-gcc-5.4.0-vzqreb2 lua-luafilesystem-1_6_3-gcc-5.4.0-alakjim py-numpy-1.15.2-gcc-7.2.0-wbwtcxf - diffutils-3.6-gcc-5.4.0-2rhuivg lua-luaposix-33.4.0-gcc-5.4.0-7wqhwoc py-scipy-1.1.0-gcc-7.2.0-d5n3cph - diffutils-3.6-gcc-7.2.0-eauxwi7 m4-1.4.18-gcc-5.4.0-suf5jtc py-setuptools-40.4.3-gcc-7.2.0-5dbwfwn - expat-2.2.5-gcc-5.4.0-emyv67q m4-1.4.18-gcc-7.2.0-wdzvagl python-2.7.15-gcc-7.2.0-ucmr2mn - findutils-4.6.0-gcc-7.2.0-ca4b7zq mpc-1.1.0-gcc-5.4.0-iuf3gc3 readline-7.0-gcc-5.4.0-nxhwrg7 - gcc-7.2.0-gcc-5.4.0-b7smjjc (L) mpfr-3.1.6-gcc-5.4.0-jnt2nnp readline-7.0-gcc-7.2.0-ccruj2i - gdbm-1.14.1-gcc-5.4.0-q4fpyuo mpich-3.2.1-gcc-7.2.0-vt5xcat sqlite-3.23.1-gcc-7.2.0-5ltus3a - gdbm-1.14.1-gcc-7.2.0-zk5lhob ncurses-6.1-gcc-5.4.0-3o765ou tar-1.30-gcc-5.4.0-dk7lrpo - gettext-0.19.8.1-gcc-5.4.0-tawgous ncurses-6.1-gcc-7.2.0-xcgzqdv tcl-8.6.8-gcc-5.4.0-qhwyccy - git-2.19.1-gcc-5.4.0-p3gjnfa netlib-lapack-3.8.0-gcc-7.2.0-fj7nayd texinfo-6.5-gcc-7.2.0-cuqnfgf - gmp-6.1.2-gcc-5.4.0-qc4qcfz netlib-scalapack-2.0.2-gcc-7.2.0-67nmj7g unzip-6.0-gcc-5.4.0-ba23fbg - hwloc-1.11.9-gcc-7.2.0-gbyc65s netlib-scalapack-2.0.2-gcc-7.2.0-6jgjbyg util-macros-1.19.1-gcc-7.2.0-t62kozq - isl-0.18-gcc-5.4.0-vttqout netlib-scalapack-2.0.2-gcc-7.2.0-prgo67d xz-5.2.4-gcc-5.4.0-teneqii - libbsd-0.8.6-gcc-5.4.0-f4qkkwm netlib-scalapack-2.0.2-gcc-7.2.0-zxpt252 xz-5.2.4-gcc-7.2.0-rql5kog - libiconv-1.15-gcc-5.4.0-u2x3umv numactl-2.0.11-gcc-7.2.0-rifwktk zlib-1.2.11-gcc-5.4.0-5nus6kn - libpciaccess-0.13.5-gcc-7.2.0-riipwi2 openblas-0.3.3-gcc-7.2.0-xxoxfh4 zlib-1.2.11-gcc-7.2.0-ezuwp4p - libsigsegv-2.11-gcc-5.4.0-fypapcp openmpi-3.1.3-gcc-7.2.0-do5xfer - - Where: - L: Module is loaded - - Use "module spider" to find all possible modules. - Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys". - -The non-hierarchical module files that have been generated so far -follow :ref:`the default rules for module generation <modules-yaml>`. -Taking a look at the ``gcc`` module you'll see, for example: - -.. code-block:: console - - $ module show gcc-7.2.0-gcc-5.4.0-b7smjjc - ------------------------------------------------------------------------------------------------------------------------------------------------------------------- - /home/spack1/spack/share/spack/modules/linux-ubuntu16.04-x86_64/gcc-7.2.0-gcc-5.4.0-b7smjjc: - ------------------------------------------------------------------------------------------------------------------------------------------------------------------- - whatis("The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Ada, and Go, as well as libraries for these languages. ") - prepend_path("PATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/bin") - prepend_path("MANPATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/share/man") - prepend_path("LD_LIBRARY_PATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/lib") - prepend_path("LIBRARY_PATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/lib") - prepend_path("LD_LIBRARY_PATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/lib64") - prepend_path("LIBRARY_PATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/lib64") - prepend_path("CPATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/include") - prepend_path("CMAKE_PREFIX_PATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/") - setenv("CC","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/bin/gcc") - setenv("CXX","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/bin/g++") - setenv("FC","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/bin/gfortran") - setenv("F77","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/bin/gfortran") - setenv("F90","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/bin/gfortran") - help([[The GNU Compiler Collection includes front ends for C, C++, Objective-C, - Fortran, Ada, and Go, as well as libraries for these languages. - ]]) - -As expected, a few environment variables representing paths will be modified -by the module file according to the default prefix inspection rules. - - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Filter unwanted modifications to the environment -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Now consider the case that your site has decided that ``CPATH`` and -``LIBRARY_PATH`` modifications should not be present in module files. What you can -do to abide by the rules is to create a configuration file ``~/.spack/modules.yaml`` -with the following content: - -.. code-block:: yaml - - modules: - tcl: - all: - filter: - environment_blacklist: ['CPATH', 'LIBRARY_PATH'] - -Next you should regenerate all the module files: - -.. code-block:: console - - $ spack module tcl refresh - ==> You are about to regenerate tcl module files for: - - -- linux-ubuntu16.04-x86_64 / gcc@5.4.0 ------------------------- - 3sx2gxe autoconf@2.69 b7smjjc gcc@7.2.0 f4qkkwm libbsd@0.8.6 cpfeo2w lua@5.3.4 3o765ou ncurses@6.1 dk7lrpo tar@1.30 - rymw7im automake@1.16.1 q4fpyuo gdbm@1.14.1 u2x3umv libiconv@1.15 alakjim lua-luafilesystem@1_6_3 b4y3w3b openssl@1.0.2o qhwyccy tcl@8.6.8 - ufczdvs bzip2@1.0.6 tawgous gettext@0.19.8.1 fypapcp libsigsegv@2.11 7wqhwoc lua-luaposix@33.4.0 gt5lgzi pcre@8.42 ba23fbg unzip@6.0 - vzqreb2 curl@7.60.0 p3gjnfa git@2.19.1 o2pfwjf libtool@2.4.6 suf5jtc m4@1.4.18 ic2kyoa perl@5.26.2 teneqii xz@5.2.4 - 2rhuivg diffutils@3.6 qc4qcfz gmp@6.1.2 wpexsph libxml2@2.9.8 iuf3gc3 mpc@1.1.0 fovrh7a pkgconf@1.4.2 5nus6kn zlib@1.2.11 - emyv67q expat@2.2.5 vttqout isl@0.18 kmhks3p lmod@7.8 jnt2nnp mpfr@3.1.6 nxhwrg7 readline@7.0 - - -- linux-ubuntu16.04-x86_64 / gcc@7.2.0 ------------------------- - yb2makb autoconf@2.69 riipwi2 libpciaccess@0.13.5 6jgjbyg netlib-scalapack@2.0.2 fdwz5yu perl@5.26.2 cuqnfgf texinfo@6.5 - qoowd5q automake@1.16.1 g67xpfd libsigsegv@2.11 zxpt252 netlib-scalapack@2.0.2 yoxwmgb pkgconf@1.4.2 t62kozq util-macros@1.19.1 - mwamumj bzip2@1.0.6 kt2udm6 libtool@2.4.6 67nmj7g netlib-scalapack@2.0.2 wbwtcxf py-numpy@1.15.2 rql5kog xz@5.2.4 - obqgn2v cmake@3.12.3 47gf5kk libxml2@2.9.8 prgo67d netlib-scalapack@2.0.2 d5n3cph py-scipy@1.1.0 ezuwp4p zlib@1.2.11 - eauxwi7 diffutils@3.6 wdzvagl m4@1.4.18 rifwktk numactl@2.0.11 5dbwfwn py-setuptools@40.4.3 - ca4b7zq findutils@4.6.0 vt5xcat mpich@3.2.1 xxoxfh4 openblas@0.3.3 ucmr2mn python@2.7.15 - zk5lhob gdbm@1.14.1 xcgzqdv ncurses@6.1 do5xfer openmpi@3.1.3 ccruj2i readline@7.0 - gbyc65s hwloc@1.11.9 fj7nayd netlib-lapack@3.8.0 cvldq3v openssl@1.0.2o 5ltus3a sqlite@3.23.1 - - ==> Do you want to proceed? [y/n] y - ==> Regenerating tcl module files - -If you take a look now at the module for ``gcc`` you'll see that the unwanted -paths have disappeared: - -.. code-block:: console - - $ module show gcc-7.2.0-gcc-5.4.0-b7smjjc - ------------------------------------------------------------------------------------------------------------------------------------------------------------------- - /home/spack1/spack/share/spack/modules/linux-ubuntu16.04-x86_64/gcc-7.2.0-gcc-5.4.0-b7smjjc: - ------------------------------------------------------------------------------------------------------------------------------------------------------------------- - whatis("The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Ada, and Go, as well as libraries for these languages. ") - prepend_path("PATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/bin") - prepend_path("MANPATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/share/man") - prepend_path("LD_LIBRARY_PATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/lib") - prepend_path("LD_LIBRARY_PATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/lib64") - prepend_path("CMAKE_PREFIX_PATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/") - setenv("CC","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/bin/gcc") - setenv("CXX","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/bin/g++") - setenv("FC","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/bin/gfortran") - setenv("F77","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/bin/gfortran") - setenv("F90","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/bin/gfortran") - help([[The GNU Compiler Collection includes front ends for C, C++, Objective-C, - Fortran, Ada, and Go, as well as libraries for these languages. - ]]) - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Prevent some module files from being generated -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Another common request at many sites is to avoid exposing software that -is only needed as an intermediate step when building a newer stack. -Let's try to prevent the generation of -module files for anything that is compiled with ``gcc@5.4.0`` (the OS provided compiler). - -To do this you should add a ``blacklist`` keyword to ``~/.spack/modules.yaml``: - -.. code-block:: yaml - :emphasize-lines: 3,4 - - modules: - tcl: - blacklist: - - '%gcc@5.4.0' - all: - filter: - environment_blacklist: ['CPATH', 'LIBRARY_PATH'] - -and regenerate the module files: - -This time it is convenient to pass the option ``--delete-tree`` to the command that -regenerates the module files to instruct it to delete the existing tree and regenerate -a new one instead of overwriting the files in the existing directory. - -.. code-block:: console - - $ spack module tcl refresh --delete-tree - ==> You are about to regenerate tcl module files for: - - -- linux-ubuntu16.04-x86_64 / gcc@5.4.0 ------------------------- - 3sx2gxe autoconf@2.69 b7smjjc gcc@7.2.0 f4qkkwm libbsd@0.8.6 cpfeo2w lua@5.3.4 3o765ou ncurses@6.1 dk7lrpo tar@1.30 - rymw7im automake@1.16.1 q4fpyuo gdbm@1.14.1 u2x3umv libiconv@1.15 alakjim lua-luafilesystem@1_6_3 b4y3w3b openssl@1.0.2o qhwyccy tcl@8.6.8 - ufczdvs bzip2@1.0.6 tawgous gettext@0.19.8.1 fypapcp libsigsegv@2.11 7wqhwoc lua-luaposix@33.4.0 gt5lgzi pcre@8.42 ba23fbg unzip@6.0 - vzqreb2 curl@7.60.0 p3gjnfa git@2.19.1 o2pfwjf libtool@2.4.6 suf5jtc m4@1.4.18 ic2kyoa perl@5.26.2 teneqii xz@5.2.4 - 2rhuivg diffutils@3.6 qc4qcfz gmp@6.1.2 wpexsph libxml2@2.9.8 iuf3gc3 mpc@1.1.0 fovrh7a pkgconf@1.4.2 5nus6kn zlib@1.2.11 - emyv67q expat@2.2.5 vttqout isl@0.18 kmhks3p lmod@7.8 jnt2nnp mpfr@3.1.6 nxhwrg7 readline@7.0 - - -- linux-ubuntu16.04-x86_64 / gcc@7.2.0 ------------------------- - yb2makb autoconf@2.69 riipwi2 libpciaccess@0.13.5 6jgjbyg netlib-scalapack@2.0.2 fdwz5yu perl@5.26.2 cuqnfgf texinfo@6.5 - qoowd5q automake@1.16.1 g67xpfd libsigsegv@2.11 zxpt252 netlib-scalapack@2.0.2 yoxwmgb pkgconf@1.4.2 t62kozq util-macros@1.19.1 - mwamumj bzip2@1.0.6 kt2udm6 libtool@2.4.6 67nmj7g netlib-scalapack@2.0.2 wbwtcxf py-numpy@1.15.2 rql5kog xz@5.2.4 - obqgn2v cmake@3.12.3 47gf5kk libxml2@2.9.8 prgo67d netlib-scalapack@2.0.2 d5n3cph py-scipy@1.1.0 ezuwp4p zlib@1.2.11 - eauxwi7 diffutils@3.6 wdzvagl m4@1.4.18 rifwktk numactl@2.0.11 5dbwfwn py-setuptools@40.4.3 - ca4b7zq findutils@4.6.0 vt5xcat mpich@3.2.1 xxoxfh4 openblas@0.3.3 ucmr2mn python@2.7.15 - zk5lhob gdbm@1.14.1 xcgzqdv ncurses@6.1 do5xfer openmpi@3.1.3 ccruj2i readline@7.0 - gbyc65s hwloc@1.11.9 fj7nayd netlib-lapack@3.8.0 cvldq3v openssl@1.0.2o 5ltus3a sqlite@3.23.1 - - ==> Do you want to proceed? [y/n] y - ==> Regenerating tcl module files - - $ module avail - - ----------------------------------------------- /home/spack1/spack/share/spack/modules/linux-ubuntu16.04-x86_64 ----------------------------------------------- - autoconf-2.69-gcc-7.2.0-yb2makb m4-1.4.18-gcc-7.2.0-wdzvagl perl-5.26.2-gcc-7.2.0-fdwz5yu - automake-1.16.1-gcc-7.2.0-qoowd5q mpich-3.2.1-gcc-7.2.0-vt5xcat pkgconf-1.4.2-gcc-7.2.0-yoxwmgb - bzip2-1.0.6-gcc-7.2.0-mwamumj ncurses-6.1-gcc-7.2.0-xcgzqdv py-numpy-1.15.2-gcc-7.2.0-wbwtcxf - cmake-3.12.3-gcc-7.2.0-obqgn2v netlib-lapack-3.8.0-gcc-7.2.0-fj7nayd py-scipy-1.1.0-gcc-7.2.0-d5n3cph - diffutils-3.6-gcc-7.2.0-eauxwi7 netlib-scalapack-2.0.2-gcc-7.2.0-67nmj7g py-setuptools-40.4.3-gcc-7.2.0-5dbwfwn - findutils-4.6.0-gcc-7.2.0-ca4b7zq netlib-scalapack-2.0.2-gcc-7.2.0-6jgjbyg python-2.7.15-gcc-7.2.0-ucmr2mn - gdbm-1.14.1-gcc-7.2.0-zk5lhob netlib-scalapack-2.0.2-gcc-7.2.0-prgo67d readline-7.0-gcc-7.2.0-ccruj2i - hwloc-1.11.9-gcc-7.2.0-gbyc65s netlib-scalapack-2.0.2-gcc-7.2.0-zxpt252 sqlite-3.23.1-gcc-7.2.0-5ltus3a - libpciaccess-0.13.5-gcc-7.2.0-riipwi2 numactl-2.0.11-gcc-7.2.0-rifwktk texinfo-6.5-gcc-7.2.0-cuqnfgf - libsigsegv-2.11-gcc-7.2.0-g67xpfd openblas-0.3.3-gcc-7.2.0-xxoxfh4 util-macros-1.19.1-gcc-7.2.0-t62kozq - libtool-2.4.6-gcc-7.2.0-kt2udm6 openmpi-3.1.3-gcc-7.2.0-do5xfer xz-5.2.4-gcc-7.2.0-rql5kog - libxml2-2.9.8-gcc-7.2.0-47gf5kk openssl-1.0.2o-gcc-7.2.0-cvldq3v zlib-1.2.11-gcc-7.2.0-ezuwp4p - - Use "module spider" to find all possible modules. - Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys". - -If you look closely you'll see though that we went too far in blacklisting modules: -the module for ``gcc@7.2.0`` disappeared as it was bootstrapped with ``gcc@5.4.0``. To specify -exceptions to the blacklist rules you can use ``whitelist``: - -.. code-block:: yaml - :emphasize-lines: 3,4 - - modules: - tcl: - whitelist: - - gcc - blacklist: - - '%gcc@5.4.0' - all: - filter: - environment_blacklist: ['CPATH', 'LIBRARY_PATH'] - -``whitelist`` rules always have precedence over ``blacklist`` rules. If you regenerate the modules again: - -.. code-block:: console - - $ spack module tcl refresh -y - ==> Regenerating tcl module files - - -you'll see that now the module for ``gcc@7.2.0`` has reappeared: - -.. code-block:: console - - $ module avail gcc-7.2.0-gcc-5.4.0-b7smjjc - - -------------------------------------------- /home/spack1/spack/share/spack/modules/linux-ubuntu16.04-x86_64 --------------------------------------------- - gcc-7.2.0-gcc-5.4.0-b7smjjc - - Use "module spider" to find all possible modules. - Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys". - -An additional possibility that you can leverage to unclutter the environment -is that of preventing the generation of module files for implicitly installed -packages. In this case all one needs to do is to add the following line: - -.. code-block:: yaml - :emphasize-lines: 3 - - modules: - tcl: - blacklist_implicits: true - whitelist: - - gcc - blacklist: - - '%gcc@5.4.0' - all: - filter: - environment_blacklist: ['CPATH', 'LIBRARY_PATH'] - -to ``modules.yaml`` and regenerate the module file tree as above. - -^^^^^^^^^^^^^^^^^^^^^^^^^ -Change module file naming -^^^^^^^^^^^^^^^^^^^^^^^^^ - -The next step in making module files more user-friendly is to -improve their naming scheme. -To reduce the length of the hash or remove it altogether you can -use the ``hash_length`` keyword in the configuration file: - -.. code-block:: yaml - :emphasize-lines: 3 - - modules: - tcl: - hash_length: 0 - whitelist: - - gcc - blacklist: - - '%gcc@5.4.0' - all: - filter: - environment_blacklist: ['CPATH', 'LIBRARY_PATH'] - -If you try to regenerate the module files now you will get an error: - -.. code-block:: console - - $ spack module tcl refresh --delete-tree -y - ==> Error: Name clashes detected in module files: - - file: /home/spack1/spack/share/spack/modules/linux-ubuntu16.04-x86_64/netlib-scalapack-2.0.2-gcc-7.2.0 - spec: netlib-scalapack@2.0.2%gcc@7.2.0 build_type=RelWithDebInfo ~pic+shared arch=linux-ubuntu16.04-x86_64 - spec: netlib-scalapack@2.0.2%gcc@7.2.0 build_type=RelWithDebInfo ~pic+shared arch=linux-ubuntu16.04-x86_64 - spec: netlib-scalapack@2.0.2%gcc@7.2.0 build_type=RelWithDebInfo ~pic+shared arch=linux-ubuntu16.04-x86_64 - spec: netlib-scalapack@2.0.2%gcc@7.2.0 build_type=RelWithDebInfo ~pic+shared arch=linux-ubuntu16.04-x86_64 - - ==> Error: Operation aborted - -.. note:: - We try to check for errors upfront! - In Spack we check for errors upfront whenever possible, so don't worry about your module files: - as a name clash was detected nothing has been changed on disk. - -The problem here is that without -the hashes the four different flavors of ``netlib-scalapack`` map to the same module file -name. We can add suffixes to differentiate them: - -.. code-block:: yaml - :emphasize-lines: 9-11,14-17 - - modules: - tcl: - hash_length: 0 - whitelist: - - gcc - blacklist: - - '%gcc@5.4.0' - all: - suffixes: - '^openblas': openblas - '^netlib-lapack': netlib - filter: - environment_blacklist: ['CPATH', 'LIBRARY_PATH'] - netlib-scalapack: - suffixes: - '^openmpi': openmpi - '^mpich': mpich - -As you can see it is possible to specify rules that apply only to a -restricted set of packages using :ref:`anonymous specs <anonymous_specs>`. -Regenerating module files now we obtain: - -.. code-block:: console - - $ spack module tcl refresh --delete-tree -y - ==> Regenerating tcl module files - $ module avail - - ----------------------------------------------- /home/spack1/spack/share/spack/modules/linux-ubuntu16.04-x86_64 ----------------------------------------------- - autoconf-2.69-gcc-7.2.0 m4-1.4.18-gcc-7.2.0 pkgconf-1.4.2-gcc-7.2.0 - automake-1.16.1-gcc-7.2.0 mpich-3.2.1-gcc-7.2.0 py-numpy-1.15.2-gcc-7.2.0-openblas - bzip2-1.0.6-gcc-7.2.0 ncurses-6.1-gcc-7.2.0 py-scipy-1.1.0-gcc-7.2.0-openblas - cmake-3.12.3-gcc-7.2.0 netlib-lapack-3.8.0-gcc-7.2.0 py-setuptools-40.4.3-gcc-7.2.0 - diffutils-3.6-gcc-7.2.0 netlib-scalapack-2.0.2-gcc-7.2.0-netlib-mpich python-2.7.15-gcc-7.2.0 - findutils-4.6.0-gcc-7.2.0 netlib-scalapack-2.0.2-gcc-7.2.0-netlib-openmpi readline-7.0-gcc-7.2.0 - gcc-7.2.0-gcc-5.4.0 netlib-scalapack-2.0.2-gcc-7.2.0-openblas-mpich sqlite-3.23.1-gcc-7.2.0 - gdbm-1.14.1-gcc-7.2.0 netlib-scalapack-2.0.2-gcc-7.2.0-openblas-openmpi texinfo-6.5-gcc-7.2.0 - hwloc-1.11.9-gcc-7.2.0 numactl-2.0.11-gcc-7.2.0 util-macros-1.19.1-gcc-7.2.0 - libpciaccess-0.13.5-gcc-7.2.0 openblas-0.3.3-gcc-7.2.0 xz-5.2.4-gcc-7.2.0 - libsigsegv-2.11-gcc-7.2.0 openmpi-3.1.3-gcc-7.2.0 zlib-1.2.11-gcc-7.2.0 - libtool-2.4.6-gcc-7.2.0 openssl-1.0.2o-gcc-7.2.0 - libxml2-2.9.8-gcc-7.2.0 perl-5.26.2-gcc-7.2.0 - - Use "module spider" to find all possible modules. - Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys". - -Finally we can set a ``naming_scheme`` to prevent users from loading -modules that refer to different flavors of the same library/application: - -.. code-block:: yaml - :emphasize-lines: 4,10,11 - - modules: - tcl: - hash_length: 0 - naming_scheme: '{name}/{version}-{compiler.name}-{compiler.version}' - whitelist: - - gcc - blacklist: - - '%gcc@5.4.0' - all: - conflict: - - '{name}' - suffixes: - '^openblas': openblas - '^netlib-lapack': netlib - filter: - environment_blacklist: ['CPATH', 'LIBRARY_PATH'] - netlib-scalapack: - suffixes: - '^openmpi': openmpi - '^mpich': mpich - -The final result should look like: - -.. code-block:: console - - $ spack module tcl refresh --delete-tree -y - ==> Regenerating tcl module files - $ module avail - - ----------------------------------------------- /home/spack1/spack/share/spack/modules/linux-ubuntu16.04-x86_64 ----------------------------------------------- - autoconf/2.69-gcc-7.2.0 m4/1.4.18-gcc-7.2.0 pkgconf/1.4.2-gcc-7.2.0 - automake/1.16.1-gcc-7.2.0 mpich/3.2.1-gcc-7.2.0 py-numpy/1.15.2-gcc-7.2.0-openblas - bzip2/1.0.6-gcc-7.2.0 ncurses/6.1-gcc-7.2.0 py-scipy/1.1.0-gcc-7.2.0-openblas - cmake/3.12.3-gcc-7.2.0 netlib-lapack/3.8.0-gcc-7.2.0 py-setuptools/40.4.3-gcc-7.2.0 - diffutils/3.6-gcc-7.2.0 netlib-scalapack/2.0.2-gcc-7.2.0-netlib-mpich python/2.7.15-gcc-7.2.0 - findutils/4.6.0-gcc-7.2.0 netlib-scalapack/2.0.2-gcc-7.2.0-netlib-openmpi readline/7.0-gcc-7.2.0 - gcc/7.2.0-gcc-5.4.0 netlib-scalapack/2.0.2-gcc-7.2.0-openblas-mpich sqlite/3.23.1-gcc-7.2.0 - gdbm/1.14.1-gcc-7.2.0 netlib-scalapack/2.0.2-gcc-7.2.0-openblas-openmpi (D) texinfo/6.5-gcc-7.2.0 - hwloc/1.11.9-gcc-7.2.0 numactl/2.0.11-gcc-7.2.0 util-macros/1.19.1-gcc-7.2.0 - libpciaccess/0.13.5-gcc-7.2.0 openblas/0.3.3-gcc-7.2.0 xz/5.2.4-gcc-7.2.0 - libsigsegv/2.11-gcc-7.2.0 openmpi/3.1.3-gcc-7.2.0 zlib/1.2.11-gcc-7.2.0 - libtool/2.4.6-gcc-7.2.0 openssl/1.0.2o-gcc-7.2.0 - libxml2/2.9.8-gcc-7.2.0 perl/5.26.2-gcc-7.2.0 - - Where: - D: Default Module - - Use "module spider" to find all possible modules. - Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys". - -.. note:: - TCL specific directive - The directives ``naming_scheme`` and ``conflict`` are TCL specific and - can't be used in the ``lmod`` section of the configuration file. - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Add custom environment modifications -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -At many sites it is customary to set an environment variable in a -package's module file that points to the folder in which the package -is installed. You can achieve this with Spack by adding an -``environment`` directive to the configuration file: - -.. code-block:: yaml - :emphasize-lines: 17-19 - - modules: - tcl: - hash_length: 0 - naming_scheme: '{name}/{version}-{compiler.name}-{compiler.version}' - whitelist: - - gcc - blacklist: - - '%gcc@5.4.0' - all: - conflict: - - '{name}' - suffixes: - '^openblas': openblas - '^netlib-lapack': netlib - filter: - environment_blacklist: ['CPATH', 'LIBRARY_PATH'] - environment: - set: - '{name}_ROOT': '{prefix}' - netlib-scalapack: - suffixes: - '^openmpi': openmpi - '^mpich': mpich - -Under the hood Spack uses the :meth:`~spack.spec.Spec.format` API to substitute -tokens in either environment variable names or values. There are two caveats though: - -- The set of allowed tokens in variable names is restricted to - ``name``, ``version``, ``compiler``, ``compiler.name``, - ``compiler.version``, ``architecture`` -- Any token expanded in a variable name is made uppercase, but other than that - case sensitivity is preserved - -Regenerating the module files results in something like: - -.. code-block:: console - :emphasize-lines: 15 - - $ spack module tcl refresh -y - ==> Regenerating tcl module files - - $ module show gcc - ------------------------------------------------------------------------------------------------------------------------------------------------------------------- - /home/spack1/spack/share/spack/modules/linux-ubuntu16.04-x86_64/gcc/7.2.0-gcc-5.4.0: - ------------------------------------------------------------------------------------------------------------------------------------------------------------------- - whatis("The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Ada, and Go, as well as libraries for these languages. ") - conflict("gcc") - prepend_path("PATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/bin") - prepend_path("MANPATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/share/man") - prepend_path("LD_LIBRARY_PATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/lib") - prepend_path("LD_LIBRARY_PATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/lib64") - prepend_path("CMAKE_PREFIX_PATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/") - setenv("CC","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/bin/gcc") - setenv("CXX","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/bin/g++") - setenv("FC","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/bin/gfortran") - setenv("F77","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/bin/gfortran") - setenv("F90","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/bin/gfortran") - setenv("GCC_ROOT","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs") - help([[The GNU Compiler Collection includes front ends for C, C++, Objective-C, - Fortran, Ada, and Go, as well as libraries for these languages. - ]]) - -As you can see, the ``gcc`` module has the environment variable ``GCC_ROOT`` set. - -Sometimes it's also useful to apply environment modifications selectively and target -only certain packages. You can, for instance set the common variables ``CC``, ``CXX``, -etc. in the ``gcc`` module file and apply other custom modifications to the -``openmpi`` modules as follows: - -.. code-block:: yaml - :emphasize-lines: 20-32 - - modules: - tcl: - hash_length: 0 - naming_scheme: '{name}/{version}-{compiler.name}-{compiler.version}' - whitelist: - - gcc - blacklist: - - '%gcc@5.4.0' - all: - conflict: - - '{name}' - suffixes: - '^openblas': openblas - '^netlib-lapack': netlib - filter: - environment_blacklist: ['CPATH', 'LIBRARY_PATH'] - environment: - set: - '{name}_ROOT': '{prefix}' - gcc: - environment: - set: - CC: gcc - CXX: g++ - FC: gfortran - F90: gfortran - F77: gfortran - openmpi: - environment: - set: - SLURM_MPI_TYPE: pmi2 - OMPI_MCA_btl_openib_warn_default_gid_prefix: '0' - netlib-scalapack: - suffixes: - '^openmpi': openmpi - '^mpich': mpich - -This time we will be more selective and regenerate only the ``gcc`` and -``openmpi`` module files: - -.. code-block:: console - - $ spack module tcl refresh -y gcc - ==> Regenerating tcl module files - - $ spack module tcl refresh -y openmpi - ==> Regenerating tcl module files - - $ module show gcc - ------------------------------------------------------------------------------------------------------------------------------------------------------------------- - /home/spack1/spack/share/spack/modules/linux-ubuntu16.04-x86_64/gcc/7.2.0-gcc-5.4.0: - ------------------------------------------------------------------------------------------------------------------------------------------------------------------- - whatis("The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Ada, and Go, as well as libraries for these languages. ") - conflict("gcc") - prepend_path("PATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/bin") - prepend_path("MANPATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/share/man") - prepend_path("LD_LIBRARY_PATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/lib") - prepend_path("LD_LIBRARY_PATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/lib64") - prepend_path("CMAKE_PREFIX_PATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/") - setenv("CC","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/bin/gcc") - setenv("CXX","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/bin/g++") - setenv("FC","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/bin/gfortran") - setenv("F77","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/bin/gfortran") - setenv("F90","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs/bin/gfortran") - setenv("GCC_ROOT","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gcc-7.2.0-b7smjjcsmwe5u5fcsvjmonlhlzzctnfs") - setenv("CC","gcc") - setenv("CXX","g++'") - setenv("FC","gfortran") - setenv("F77","gfortran") - setenv("F90","gfortran") - help([[The GNU Compiler Collection includes front ends for C, C++, Objective-C, - Fortran, Ada, and Go, as well as libraries for these languages. - ]]) - - $ module show openmpi - ------------------------------------------------------------------------------------------------------------------------------------------------------------------- - /home/spack1/spack/share/spack/modules/linux-ubuntu16.04-x86_64/openmpi/3.1.3-gcc-7.2.0: - ------------------------------------------------------------------------------------------------------------------------------------------------------------------- - whatis("An open source Message Passing Interface implementation. ") - conflict("openmpi") - prepend_path("PATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-7.2.0/openmpi-3.1.3-do5xfer2whhk7gc26atgs3ozr3ljbvs4/bin") - prepend_path("MANPATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-7.2.0/openmpi-3.1.3-do5xfer2whhk7gc26atgs3ozr3ljbvs4/share/man") - prepend_path("LD_LIBRARY_PATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-7.2.0/openmpi-3.1.3-do5xfer2whhk7gc26atgs3ozr3ljbvs4/lib") - prepend_path("PKG_CONFIG_PATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-7.2.0/openmpi-3.1.3-do5xfer2whhk7gc26atgs3ozr3ljbvs4/lib/pkgconfig") - prepend_path("CMAKE_PREFIX_PATH","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-7.2.0/openmpi-3.1.3-do5xfer2whhk7gc26atgs3ozr3ljbvs4/") - setenv("OPENMPI_ROOT","/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-7.2.0/openmpi-3.1.3-do5xfer2whhk7gc26atgs3ozr3ljbvs4") - setenv("SLURM_MPI_TYPE","pmi2") - setenv("OMPI_MCA_btl_openib_warn_default_gid_prefix","0") - help([[An open source Message Passing Interface implementation. The Open MPI - Project is an open source Message Passing Interface implementation that - is developed and maintained by a consortium of academic, research, and - industry partners. Open MPI is therefore able to combine the expertise, - technologies, and resources from all across the High Performance - Computing community in order to build the best MPI library available. - Open MPI offers advantages for system and software vendors, application - developers and computer science researchers. - ]]) - - -^^^^^^^^^^^^^^^^^^^^^ -Autoload dependencies -^^^^^^^^^^^^^^^^^^^^^ - -Spack can also generate module files that contain code to load the -dependencies automatically. You can, for instance generate python -modules that load their dependencies by adding the ``autoload`` -directive and assigning it the value ``direct``: - -.. code-block:: yaml - :emphasize-lines: 3,38,39 - - modules: - tcl: - verbose: True - hash_length: 0 - naming_scheme: '{name}/{version}-{compiler.name}-{compiler.version}' - whitelist: - - gcc - blacklist: - - '%gcc@5.4.0' - all: - conflict: - - '{name}' - suffixes: - '^openblas': openblas - '^netlib-lapack': netlib - filter: - environment_blacklist: ['CPATH', 'LIBRARY_PATH'] - environment: - set: - '{name}_ROOT': '{prefix}' - gcc: - environment: - set: - CC: gcc - CXX: g++ - FC: gfortran - F90: gfortran - F77: gfortran - openmpi: - environment: - set: - SLURM_MPI_TYPE: pmi2 - OMPI_MCA_btl_openib_warn_default_gid_prefix: '0' - netlib-scalapack: - suffixes: - '^openmpi': openmpi - '^mpich': mpich - ^python: - autoload: 'direct' - -and regenerating the module files for every package that depends on ``python``: - -.. code-block:: console - - root@module-file-tutorial:/# spack module tcl refresh -y ^python - ==> Regenerating tcl module files - -Now the ``py-scipy`` module will be: - -.. code-block:: tcl - - #%Module1.0 - ## Module file created by spack (https://github.com/spack/spack) on 2018-11-11 22:10:48.834221 - ## - ## py-scipy@1.1.0%gcc@7.2.0 arch=linux-ubuntu16.04-x86_64 /d5n3cph - ## - - - module-whatis "SciPy (pronounced 'Sigh Pie') is a Scientific Library for Python. It provides many user-friendly and efficient numerical routines such as routines for numerical integration and optimization." - - proc ModulesHelp { } { - puts stderr "SciPy (pronounced "Sigh Pie") is a Scientific Library for Python. It" - puts stderr "provides many user-friendly and efficient numerical routines such as" - puts stderr "routines for numerical integration and optimization." - } - - if { [ module-info mode load ] && ![ is-loaded python/2.7.15-gcc-7.2.0 ] } { - puts stderr "Autoloading python/2.7.15-gcc-7.2.0" - module load python/2.7.15-gcc-7.2.0 - } - if { [ module-info mode load ] && ![ is-loaded openblas/0.3.3-gcc-7.2.0 ] } { - puts stderr "Autoloading openblas/0.3.3-gcc-7.2.0" - module load openblas/0.3.3-gcc-7.2.0 - } - if { [ module-info mode load ] && ![ is-loaded py-numpy/1.15.2-gcc-7.2.0-openblas ] } { - puts stderr "Autoloading py-numpy/1.15.2-gcc-7.2.0-openblas" - module load py-numpy/1.15.2-gcc-7.2.0-openblas - } - conflict py-scipy - - prepend-path LD_LIBRARY_PATH "/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-7.2.0/py-scipy-1.1.0-d5n3cphk2lx2v74ypwb6h7tna7vvgdyn/lib" - prepend-path CMAKE_PREFIX_PATH "/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-7.2.0/py-scipy-1.1.0-d5n3cphk2lx2v74ypwb6h7tna7vvgdyn/" - prepend-path PYTHONPATH "/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-7.2.0/py-scipy-1.1.0-d5n3cphk2lx2v74ypwb6h7tna7vvgdyn/lib/python2.7/site-packages" - setenv PY_SCIPY_ROOT "/home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-7.2.0/py-scipy-1.1.0-d5n3cphk2lx2v74ypwb6h7tna7vvgdyn" - -and will contain code to autoload all the dependencies: - -.. code-block:: console - - $ module load py-scipy - Autoloading python/2.7.15-gcc-7.2.0 - Autoloading openblas/0.3.3-gcc-7.2.0 - Autoloading py-numpy/1.15.2-gcc-7.2.0-openblas - -In case messages are unwanted during the autoload procedure, it will be -sufficient to omit the line setting ``verbose: True`` in the configuration file above. - -------------------------- -Hierarchical module files -------------------------- - -So far we worked with non-hierarchical module files, i.e. with module files -that are all generated in the same root directory and don't attempt to -dynamically modify the ``MODULEPATH``. This results in a flat module structure where -all the software is visible at the same time: - -.. code-block:: console - - $ module avail - - ----------------------------------------------- /home/spack1/spack/share/spack/modules/linux-ubuntu16.04-x86_64 ----------------------------------------------- - autoconf/2.69-gcc-7.2.0 m4/1.4.18-gcc-7.2.0 pkgconf/1.4.2-gcc-7.2.0 - automake/1.16.1-gcc-7.2.0 mpich/3.2.1-gcc-7.2.0 py-numpy/1.15.2-gcc-7.2.0-openblas (L) - bzip2/1.0.6-gcc-7.2.0 ncurses/6.1-gcc-7.2.0 py-scipy/1.1.0-gcc-7.2.0-openblas (L) - cmake/3.12.3-gcc-7.2.0 netlib-lapack/3.8.0-gcc-7.2.0 py-setuptools/40.4.3-gcc-7.2.0 - diffutils/3.6-gcc-7.2.0 netlib-scalapack/2.0.2-gcc-7.2.0-netlib-mpich python/2.7.15-gcc-7.2.0 (L) - findutils/4.6.0-gcc-7.2.0 netlib-scalapack/2.0.2-gcc-7.2.0-netlib-openmpi readline/7.0-gcc-7.2.0 - gcc/7.2.0-gcc-5.4.0 netlib-scalapack/2.0.2-gcc-7.2.0-openblas-mpich sqlite/3.23.1-gcc-7.2.0 - gdbm/1.14.1-gcc-7.2.0 netlib-scalapack/2.0.2-gcc-7.2.0-openblas-openmpi (D) texinfo/6.5-gcc-7.2.0 - hwloc/1.11.9-gcc-7.2.0 numactl/2.0.11-gcc-7.2.0 util-macros/1.19.1-gcc-7.2.0 - libpciaccess/0.13.5-gcc-7.2.0 openblas/0.3.3-gcc-7.2.0 (L) xz/5.2.4-gcc-7.2.0 - libsigsegv/2.11-gcc-7.2.0 openmpi/3.1.3-gcc-7.2.0 zlib/1.2.11-gcc-7.2.0 - libtool/2.4.6-gcc-7.2.0 openssl/1.0.2o-gcc-7.2.0 - libxml2/2.9.8-gcc-7.2.0 perl/5.26.2-gcc-7.2.0 - - Where: - L: Module is loaded - D: Default Module - - Use "module spider" to find all possible modules. - Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys". - -This layout is quite simple to deploy, but you can see from the above snippet -that nothing prevents users from loading incompatible sets of modules: - -.. code-block:: console - - $ module purge - $ module load netlib-lapack/3.8.0-gcc-7.2.0 openblas/0.3.3-gcc-7.2.0 - $ module list - - Currently Loaded Modules: - 1) netlib-lapack/3.8.0-gcc-7.2.0 2) openblas/0.3.3-gcc-7.2.0 - -Even if ``conflicts`` directives are carefully placed in module files, they: - - - won't enforce a consistent environment, but will just report an error - - need constant updates, for instance as soon as a new compiler or MPI library is installed - -`Hierarchical module files <http://lmod.readthedocs.io/en/latest/080_hierarchy.html>`_ try to -overcome these shortcomings by showing at start-up only a restricted view of what is -available on the system: more specifically only the software that has been installed with -OS provided compilers. Among this software there will be other - usually more recent - compilers -that, once loaded, will prepend new directories to ``MODULEPATH`` unlocking all the software -that was compiled with them. This "unlocking" idea can then be extended arbitrarily to -virtual dependencies, as we'll see in the following section. - -^^^^^^^^^^^^^^^^^ -Core/Compiler/MPI -^^^^^^^^^^^^^^^^^ - -The most widely used hierarchy is the so called ``Core/Compiler/MPI`` where, on top -of the compilers, different MPI libraries also unlock software linked to them. -There are just a few steps needed to adapt the ``modules.yaml`` file we used previously: - - #. enable the ``lmod`` file generator - #. change the ``tcl`` tag to ``lmod`` - #. remove ``tcl`` specific directives (``naming_scheme`` and ``conflict``) - #. declare which compilers are considered ``core_compilers`` - #. remove the ``mpi`` related suffixes (as they will be substituted by hierarchies) - -After these modifications your configuration file should look like: - -.. code-block:: yaml - :emphasize-lines: 2-8 - - modules: - enable:: - - lmod - lmod: - core_compilers: - - 'gcc@5.4.0' - hierarchy: - - mpi - hash_length: 0 - whitelist: - - gcc - blacklist: - - '%gcc@5.4.0' - all: - suffixes: - '^openblas': openblas - '^netlib-lapack': netlib - filter: - environment_blacklist: ['CPATH', 'LIBRARY_PATH'] - environment: - set: - '{name}_ROOT': '{prefix}' - gcc: - environment: - set: - CC: gcc - CXX: g++ - FC: gfortran - F90: gfortran - F77: gfortran - openmpi: - environment: - set: - SLURM_MPI_TYPE: pmi2 - OMPI_MCA_btl_openib_warn_default_gid_prefix: '0' - - -.. note:: - Double colon in configuration files - The double colon after ``enable`` is intentional and it serves the - purpose of overriding the default list of enabled generators so - that only ``lmod`` will be active (see :ref:`config-overrides` for more - details). - -The directive ``core_compilers`` accepts a list of compilers. Everything built -using these compilers will create a module in the ``Core`` part of the hierarchy, -which is the entry point for hierarchical module files. It is -common practice to put the OS provided compilers in the list and only build common utilities -and other compilers with them. - -If we now regenerate the module files: - -.. code-block:: console - - $ spack module lmod refresh --delete-tree -y - ==> Regenerating lmod module files - -and update ``MODULEPATH`` to point to the ``Core``: - -.. code-block:: console - - $ module purge - $ module unuse $HOME/spack/share/spack/modules/linux-ubuntu16.04-x86_64 - $ module use $HOME/spack/share/spack/lmod/linux-ubuntu16.04-x86_64/Core - -asking for the available modules will return: - -.. code-block:: console - - $ module avail - - ----------------------------------------------------------- share/spack/lmod/linux-ubuntu16.04-x86_64/Core ------------------------------------------------------------ - gcc/7.2.0 - - Use "module spider" to find all possible modules. - Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys". - -Unsurprisingly, the only visible module is ``gcc``. Loading that we'll unlock -the ``Compiler`` part of the hierarchy: - -.. code-block:: console - - $ module load gcc - $ module avail - - ------------------------------------------- /home/spack1/spack/share/spack/lmod/linux-ubuntu16.04-x86_64/gcc/7.2.0 -------------------------------------------- - autoconf/2.69 findutils/4.6.0 libtool/2.4.6 netlib-lapack/3.8.0 perl/5.26.2 python/2.7.15 xz/5.2.4 - automake/1.16.1 gdbm/1.14.1 libxml2/2.9.8 numactl/2.0.11 pkgconf/1.4.2 readline/7.0 zlib/1.2.11 - bzip2/1.0.6 hwloc/1.11.9 m4/1.4.18 openblas/0.3.3 py-numpy/1.15.2-openblas sqlite/3.23.1 - cmake/3.12.3 libpciaccess/0.13.5 mpich/3.2.1 openmpi/3.1.3 py-scipy/1.1.0-openblas texinfo/6.5 - diffutils/3.6 libsigsegv/2.11 ncurses/6.1 openssl/1.0.2o py-setuptools/40.4.3 util-macros/1.19.1 - - ----------------------------------------------------------- share/spack/lmod/linux-ubuntu16.04-x86_64/Core ------------------------------------------------------------ - gcc/7.2.0 (L) - - Where: - L: Module is loaded - - Use "module spider" to find all possible modules. - Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys". - -The same holds true also for the ``MPI`` part, that you can enable by loading -either ``mpich`` or ``openmpi``. Let's start by loading ``mpich``: - -.. code-block:: console - - $ module load mpich - $ module avail - - --------------------------------- /home/spack1/spack/share/spack/lmod/linux-ubuntu16.04-x86_64/mpich/3.2.1-vt5xcat/gcc/7.2.0 ---------------------------------- - netlib-scalapack/2.0.2-netlib netlib-scalapack/2.0.2-openblas (D) - - ------------------------------------------- /home/spack1/spack/share/spack/lmod/linux-ubuntu16.04-x86_64/gcc/7.2.0 -------------------------------------------- - autoconf/2.69 findutils/4.6.0 libtool/2.4.6 netlib-lapack/3.8.0 perl/5.26.2 python/2.7.15 xz/5.2.4 - automake/1.16.1 gdbm/1.14.1 libxml2/2.9.8 numactl/2.0.11 pkgconf/1.4.2 readline/7.0 zlib/1.2.11 - bzip2/1.0.6 hwloc/1.11.9 m4/1.4.18 openblas/0.3.3 py-numpy/1.15.2-openblas sqlite/3.23.1 - cmake/3.12.3 libpciaccess/0.13.5 mpich/3.2.1 (L) openmpi/3.1.3 py-scipy/1.1.0-openblas texinfo/6.5 - diffutils/3.6 libsigsegv/2.11 ncurses/6.1 openssl/1.0.2o py-setuptools/40.4.3 util-macros/1.19.1 - - ----------------------------------------------------------- share/spack/lmod/linux-ubuntu16.04-x86_64/Core ------------------------------------------------------------ - gcc/7.2.0 (L) - - Where: - L: Module is loaded - D: Default Module - - Use "module spider" to find all possible modules. - Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys". - - - root@module-file-tutorial:/# module load openblas netlib-scalapack/2.0.2-openblas - root@module-file-tutorial:/# module list - - Currently Loaded Modules: - 1) gcc/7.2.0 2) mpich/3.2.1 3) openblas/0.3.3 4) netlib-scalapack/2.0.2-openblas - -At this point we can showcase the improved consistency that a hierarchical layout provides -over a non-hierarchical one: - -.. code-block:: console - - $ module load openmpi - - Lmod is automatically replacing "mpich/3.2.1" with "openmpi/3.1.3". - - - Due to MODULEPATH changes, the following have been reloaded: - 1) netlib-scalapack/2.0.2-openblas - -``Lmod`` took care of swapping the MPI provider for us, and it also substituted the -``netlib-scalapack`` module to conform to the change in the MPI. -In this way we can't accidentally pull-in two different MPI providers at the -same time or load a module file for a package linked to ``openmpi`` when ``mpich`` is also loaded. -Consistency for compilers and MPI is ensured by the tool. - - -^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Add LAPACK to the hierarchy -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The hierarchy just shown is already a great improvement over non-hierarchical layouts, -but it still has an asymmetry: ``LAPACK`` providers cover the same semantic role -as ``MPI`` providers, but yet they are not part of the hierarchy. - -To be more practical, this means that although we have gained an improved consistency in -our environment when it comes to ``MPI``, we still have the same problems as we had before -for ``LAPACK`` implementations: - -.. code-block:: console - - root@module-file-tutorial:/# module list - - Currently Loaded Modules: - 1) gcc/7.2.0 2) openblas/0.3.3 3) openmpi/3.1.3 4) netlib-scalapack/2.0.2-openblas - - root@module-file-tutorial:/# module load netlib-scalapack/2.0.2-netlib - - The following have been reloaded with a version change: - 1) netlib-scalapack/2.0.2-openblas => netlib-scalapack/2.0.2-netlib - - root@module-file-tutorial:/# module list - - Currently Loaded Modules: - 1) gcc/7.2.0 2) openblas/0.3.3 3) openmpi/3.1.3 4) netlib-scalapack/2.0.2-netlib - -Hierarchies that are deeper than ``Core``/``Compiler``/``MPI`` are -probably still considered "unusual" or "impractical" at many sites, mainly because -module files are written manually and keeping track of the combinations -among multiple providers quickly becomes quite involved. - -For instance, having both ``MPI`` and ``LAPACK`` in the hierarchy -means we must classify software into one of four categories: - - #. Software that doesn't depend on ``MPI`` or ``LAPACK`` - #. Software that depends only on ``MPI`` - #. Software that depends only on ``LAPACK`` - #. Software that depends on both - -to decide when to show it to the user. The situation becomes more involved as the number of virtual -dependencies in the hierarchy increases. - -We can take advantage of the DAG that Spack maintains for the installed software and solve -this combinatorial problem in a clean and automated way. -In some sense Spack's ability to manage this combinatorial complexity makes deeper -hierarchies feasible. - -Coming back to our example, let's add ``lapack`` to the hierarchy and remove any remaining suffix: - -.. code-block:: yaml - :emphasize-lines: 9 - - modules: - enable:: - - lmod - lmod: - core_compilers: - - 'gcc@5.4.0' - hierarchy: - - mpi - - lapack - hash_length: 0 - whitelist: - - gcc - blacklist: - - '%gcc@5.4.0' - all: - filter: - environment_blacklist: ['CPATH', 'LIBRARY_PATH'] - environment: - set: - '{name}_ROOT': '{prefix}' - gcc: - environment: - set: - CC: gcc - CXX: g++ - FC: gfortran - F90: gfortran - F77: gfortran - openmpi: - environment: - set: - SLURM_MPI_TYPE: pmi2 - OMPI_MCA_btl_openib_warn_default_gid_prefix: '0' - -After module files have been regenerated as usual: - -.. code-block:: console - - root@module-file-tutorial:/# module purge - - root@module-file-tutorial:/# spack module lmod refresh --delete-tree -y - ==> Regenerating lmod module files - -we can see that now we have additional components in the hierarchy: - -.. code-block:: console - - $ module load gcc - $ module load openblas - $ module avail - - -------------------------------- /home/spack1/spack/share/spack/lmod/linux-ubuntu16.04-x86_64/openblas/0.3.3-xxoxfh4/gcc/7.2.0 -------------------------------- - py-numpy/1.15.2 py-scipy/1.1.0 - - ------------------------------------------- /home/spack1/spack/share/spack/lmod/linux-ubuntu16.04-x86_64/gcc/7.2.0 -------------------------------------------- - autoconf/2.69 findutils/4.6.0 libtool/2.4.6 netlib-lapack/3.8.0 perl/5.26.2 sqlite/3.23.1 - automake/1.16.1 gdbm/1.14.1 libxml2/2.9.8 numactl/2.0.11 pkgconf/1.4.2 texinfo/6.5 - bzip2/1.0.6 hwloc/1.11.9 m4/1.4.18 openblas/0.3.3 (L) py-setuptools/40.4.3 util-macros/1.19.1 - cmake/3.12.3 libpciaccess/0.13.5 mpich/3.2.1 openmpi/3.1.3 python/2.7.15 xz/5.2.4 - diffutils/3.6 libsigsegv/2.11 ncurses/6.1 openssl/1.0.2o readline/7.0 zlib/1.2.11 - - ----------------------------------------------------------- share/spack/lmod/linux-ubuntu16.04-x86_64/Core ------------------------------------------------------------ - gcc/7.2.0 (L) - - Where: - L: Module is loaded - - Use "module spider" to find all possible modules. - Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys". - - - $ module load openmpi - $ module avail - - --------------------- /home/spack1/spack/share/spack/lmod/linux-ubuntu16.04-x86_64/openmpi/3.1.3-do5xfer/openblas/0.3.3-xxoxfh4/gcc/7.2.0 --------------------- - netlib-scalapack/2.0.2 - - -------------------------------- /home/spack1/spack/share/spack/lmod/linux-ubuntu16.04-x86_64/openblas/0.3.3-xxoxfh4/gcc/7.2.0 -------------------------------- - py-numpy/1.15.2 py-scipy/1.1.0 - - ------------------------------------------- /home/spack1/spack/share/spack/lmod/linux-ubuntu16.04-x86_64/gcc/7.2.0 -------------------------------------------- - autoconf/2.69 findutils/4.6.0 libtool/2.4.6 netlib-lapack/3.8.0 perl/5.26.2 sqlite/3.23.1 - automake/1.16.1 gdbm/1.14.1 libxml2/2.9.8 numactl/2.0.11 pkgconf/1.4.2 texinfo/6.5 - bzip2/1.0.6 hwloc/1.11.9 m4/1.4.18 openblas/0.3.3 (L) py-setuptools/40.4.3 util-macros/1.19.1 - cmake/3.12.3 libpciaccess/0.13.5 mpich/3.2.1 openmpi/3.1.3 (L) python/2.7.15 xz/5.2.4 - diffutils/3.6 libsigsegv/2.11 ncurses/6.1 openssl/1.0.2o readline/7.0 zlib/1.2.11 - - ---------------------------------------------- /home/spack1/spack/share/spack/lmod/linux-ubuntu16.04-x86_64/Core ---------------------------------------------- - gcc/7.2.0 (L) - - Where: - L: Module is loaded - - Use "module spider" to find all possible modules. - Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys". - -Both ``MPI`` and ``LAPACK`` providers will now benefit from the same safety features: - -.. code-block:: console - - $ module load py-numpy netlib-scalapack - $ module load mpich - - Lmod is automatically replacing "openmpi/3.1.3" with "mpich/3.2.1". - - - Due to MODULEPATH changes, the following have been reloaded: - 1) netlib-scalapack/2.0.2 - - $ module load netlib-lapack - - Lmod is automatically replacing "openblas/0.3.3" with "netlib-lapack/3.8.0". - - - Inactive Modules: - 1) py-numpy - - Due to MODULEPATH changes, the following have been reloaded: - 1) netlib-scalapack/2.0.2 - -Because we only compiled ``py-numpy`` with ``openblas`` the module -is made inactive when we switch the ``LAPACK`` provider. The user -environment is now consistent by design! - ----------------------- -Working with templates ----------------------- - -As briefly mentioned in the introduction, Spack uses `Jinja2 <http://jinja.pocoo.org/docs/2.9/>`_ -to generate each individual module file. -This means that you have all of its flexibility and power when it comes to -customizing what gets generated! - -^^^^^^^^^^^^^^^^^^^^^ -Module file templates -^^^^^^^^^^^^^^^^^^^^^ - -The templates that Spack uses to generate module files are stored in the -``share/spack/templates/module`` directory within the Spack prefix, and -they all share the same common structure. Usually, they start with a -header that identifies the type of module being generated. In the case of -hierarchical module files it's: - -.. literalinclude:: _spack_root/share/spack/templates/modules/modulefile.lua - :language: jinja - :lines: 1-6 - -The statements within double curly brackets ``{{ ... }}`` denote -`expressions <http://jinja.pocoo.org/docs/2.9/templates/#expressions>`_ -that will be evaluated and substituted at module generation time. -The rest of the file is then divided into -`blocks <http://jinja.pocoo.org/docs/2.9/templates/#template-inheritance>`_ -that can be overridden or extended by users, if need be. -`Control structures <http://jinja.pocoo.org/docs/2.9/templates/#list-of-control-structures>`_ -, delimited by ``{% ... %}``, -are also permitted in the template language: - -.. literalinclude:: _spack_root/share/spack/templates/modules/modulefile.lua - :language: jinja - :lines: 73-88 - -The locations where Spack looks for templates are specified -in ``config.yaml``: - -.. literalinclude:: _spack_root/etc/spack/defaults/config.yaml - :language: yaml - :lines: 21-24 - -and can be extended by users to employ custom templates, as we'll see next. - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Extend the default templates -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Let's assume one of our software is protected by group membership: -allowed users belong to the same linux group, and access is granted at group level. -Wouldn't it be nice if people that are not -yet entitled to use it could receive a helpful message at module load time -that tells them who to contact in your organization to be inserted in the group? - -To automate the generation of module files with such site-specific behavior -we'll start by extending the list of locations where Spack looks for module -files. Let's create the file ``~/.spack/config.yaml`` with the content: - -.. code-block:: yaml - - config: - template_dirs: - - $HOME/.spack/templates - -This tells Spack to also search another location when looking for template files. -Next, we need to create our custom template extension in the folder listed above: - -.. code-block:: jinja - - {% extends "modules/modulefile.lua" %} - {% block footer %} - -- Access is granted only to specific groups - if not isDir("{{ spec.prefix }}") then - LmodError ( - "You don't have the necessary rights to run \"{{ spec.name }}\".\n\n", - "\tPlease write an e-mail to 1234@foo.com if you need further information on how to get access to it.\n" - ) - end - {% endblock %} - -Let's name this file ``group-restricted.lua``. The line: - -.. code-block:: jinja - - {% extends "modules/modulefile.lua" %} - -tells Jinja2 that we are reusing the standard template for hierarchical module files. -The section: - -.. code-block:: jinja - - {% block footer %} - -- Access is granted only to specific groups - if not isDir("{{ spec.prefix }}") then - LmodError ( - "You don't have the necessary rights to run \"{{ spec.name }}\".\n\n", - "\tPlease write an e-mail to 1234@foo.com if you need further information on how to get access to it.\n" - ) - end - {% endblock %} - -overrides the ``footer`` block. -Finally, we need to add a couple of lines in ``modules.yaml`` to tell Spack which specs -need to use the new custom template. For the sake of illustration let's assume -it's ``netlib-scalapack``: - -.. code-block:: yaml - :emphasize-lines: 35-36 - - modules: - enable:: - - lmod - lmod: - core_compilers: - - 'gcc@5.4.0' - hierarchy: - - mpi - - lapack - hash_length: 0 - whitelist: - - gcc - blacklist: - - '%gcc@5.4.0' - - readline - all: - filter: - environment_blacklist: ['CPATH', 'LIBRARY_PATH'] - environment: - set: - '{name}_ROOT': '{prefix}' - gcc: - environment: - set: - CC: gcc - CXX: g++ - FC: gfortran - F90: gfortran - F77: gfortran - openmpi: - environment: - set: - SLURM_MPI_TYPE: pmi2 - OMPI_MCA_btl_openib_warn_default_gid_prefix: '0' - netlib-scalapack: - template: 'group-restricted.lua' - -If we regenerate the module files one last time: - -.. code-block:: console - - root@module-file-tutorial:/# spack module lmod refresh -y netlib-scalapack - ==> Regenerating lmod module files - -we'll find the following at the end of each ``netlib-scalapack`` module file: - -.. code-block:: lua - - -- Access is granted only to specific groups - if not isDir("/usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-7.2.0/netlib-scalapack-2.0.2-d3lertflood3twaor44eam2kcr4l72ag") then - LmodError ( - "You don't have the necessary rights to run \"netlib-scalapack\".\n\n", - "\tPlease write an e-mail to 1234@foo.com if you need further information on how to get access to it.\n" - ) - end - -and every user that doesn't have access to the software will now be redirected to -the right e-mail address where to ask for it! diff --git a/lib/spack/docs/tutorial_packaging.rst b/lib/spack/docs/tutorial_packaging.rst deleted file mode 100644 index ca693ba14f..0000000000 --- a/lib/spack/docs/tutorial_packaging.rst +++ /dev/null @@ -1,559 +0,0 @@ -.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other - Spack Project Developers. See the top-level COPYRIGHT file for details. - - SPDX-License-Identifier: (Apache-2.0 OR MIT) - -.. _packaging-tutorial: - -========================= -Package Creation Tutorial -========================= - -This tutorial will walk you through the steps behind building a simple -package installation script. We'll focus on writing a package for -mpileaks, an MPI debugging tool. By creating a package file we're -essentially giving Spack a recipe for how to build a particular piece of -software. We're describing some of the software's dependencies, where to -find the package, what commands and options are used to build the package -from source, and more. Once we've specified a package's recipe, we can -ask Spack to build that package in many different ways. - -This tutorial assumes you have a basic familiarity with some of the Spack -commands, and that you have a working version of Spack installed. If -not, we suggest looking at Spack's :ref:`getting_started` guide. This -tutorial also assumes you have at least a beginner's-level familiarity -with Python. - -Also note that this document is a tutorial. It can help you get started -with packaging, but is not intended to be complete. See Spack's -:ref:`packaging-guide` for more complete documentation on this topic. - ---------------- -Getting Started ---------------- - -A few things before we get started: - -- We'll refer to the Spack installation location via the environment - variable ``SPACK_ROOT``. You should point ``SPACK_ROOT`` at wherever - you have Spack installed. -- Add ``$SPACK_ROOT/bin`` to your ``PATH`` before you start. -- Make sure your ``EDITOR`` environment variable is set to your - preferred text editor. -- We'll be writing Python code as part of this tutorial. You can find - successive versions of the Python code in - ``$SPACK_ROOT/lib/spack/docs/tutorial/examples``. - -------------------------- -Creating the Package File -------------------------- - -We will use a separate package repository for the tutorial. Package -repositories allow you to separate sets of packages that take -precedence over one another. We will use the tutorial repo that ships -with Spack to avoid breaking the builtin Spack packages. - -.. code-block:: console - - $ spack repo add $SPACK_ROOT/var/spack/repos/tutorial/ - ==> Added repo with namespace 'tutorial'. - -Spack comes with a handy command to create a new package: ``spack create``. - -This command is given the location of a package's source code, downloads -the code, and sets up some basic packaging infrastructure for you. The -mpileaks source code can be found on GitHub, and here's what happens when -we run ``spack create`` on it: - -.. code-block:: console - - $ spack create -t generic https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz - ==> This looks like a URL for mpileaks - ==> Found 1 version of mpileaks: - - 1.0 https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz - - ==> How many would you like to checksum? (default is 1, q to abort) 1 - ==> Downloading... - ==> Fetching https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz - ############################################################################# 100.0% - ==> Checksummed 1 version of mpileaks - ==> Using specified package template: 'generic' - ==> Created template for mpileaks package - ==> Created package file: ~/spack/var/spack/repos/tutorial/packages/mpileaks/package.py - -Spack should spawn a text editor with this file: - -.. literalinclude:: tutorial/examples/0.package.py - :language: python - -Spack has created this file in -``$SPACK_ROOT/var/spack/repos/tutorial/packages/mpileaks/package.py``. Take a -moment to look over the file. There's a few placeholders that Spack has -created, which we'll fill in as part of this tutorial: - -- We'll document some information about this package in the comments. -- We'll fill in the dependency list for this package. -- We'll fill in some of the configuration arguments needed to build this - package. - -For the moment, exit your editor and let's see what happens when we try -to build this package: - -.. code-block:: console - - $ spack install mpileaks - ==> Installing mpileaks - ==> Searching for binary cache of mpileaks - ==> Warning: No Spack mirrors are currently configured - ==> No binary for mpileaks found: installing from source - ==> Fetching https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz - ############################################################################# 100.0% - ==> Staging archive: ~/spack/var/spack/stage/mpileaks-1.0-sv75n3u5ev6mljwcezisz3slooozbbxu/mpileaks-1.0.tar.gz - ==> Created stage in ~/spack/var/spack/stage/mpileaks-1.0-sv75n3u5ev6mljwcezisz3slooozbbxu - ==> No patches needed for mpileaks - ==> Building mpileaks [Package] - ==> Executing phase: 'install' - ==> Error: ProcessError: Command exited with status 2: - 'make' '-j16' - - 1 error found in build log: - 1 ==> Executing phase: 'install' - 2 ==> 'make' '-j16' - >> 3 make: *** No targets specified and no makefile found. Stop. - - See build log for details: - ~/spack/var/spack/stage/mpileaks-1.0-sv75n3u5ev6mljwcezisz3slooozbbxu/spack-build-out.txt - -This obviously didn't work; we need to fill in the package-specific -information. Specifically, Spack didn't try to build any of mpileaks' -dependencies, nor did it use the proper configure arguments. Let's start -fixing things. - ---------------------- -Package Documentation ---------------------- - -We can bring the ``package.py`` file back into our ``EDITOR`` with the -``spack edit`` command: - -.. code-block:: console - - $ spack edit mpileaks - -Let's remove some of the ``FIXME`` comments, add links to the mpileaks -homepage, and document what mpileaks does. I'm also going to cut out the -Copyright clause at this point to keep this tutorial document shorter, -but you shouldn't do that normally. The results of these changes can be -found in ``$SPACK_ROOT/lib/spack/docs/tutorial/examples/1.package.py`` -and are displayed below. Make these changes to your ``package.py``: - -.. literalinclude:: tutorial/examples/1.package.py - :lines: 6- - :language: python - -We've filled in the comment that describes what this package does and -added a link to its website. That won't help us build yet, but it will -allow Spack to provide some documentation on this package to other users: - -.. code-block:: console - - $ spack info mpileaks - Package: mpileaks - - Description: - Tool to detect and report MPI objects like MPI_Requests and - MPI_Datatypes. - - Homepage: https://github.com/LLNL/mpileaks - - Tags: - None - - Preferred version: - 1.0 https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz - - Safe versions: - 1.0 https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz - - Variants: - None - - Installation Phases: - install - - Build Dependencies: - None - - Link Dependencies: - None - - Run Dependencies: - None - - Virtual Packages: - None - -As we fill in more information about this package the ``spack info`` command -will become more informative. Now let's start making this package build. - ------------- -Dependencies ------------- - -The mpileaks package depends on three other packages: ``mpi``, -``adept-utils``, and ``callpath``. Let's add those via the -``depends_on`` command in our ``package.py`` (this version is in -``$SPACK_ROOT/lib/spack/docs/tutorial/examples/2.package.py``): - -.. literalinclude:: tutorial/examples/2.package.py - :lines: 6- - :language: python - -Now when we go to build mpileaks, Spack will fetch and build these -dependencies before building mpileaks. Note that the mpi dependency is a -different kind of beast than the adept-utils and callpath dependencies; -there is no mpi package available in Spack. Instead mpi is a *virtual -dependency*. Spack may satisfy that dependency by installing packages -such as ``openmpi`` or ``mvapich2``. See the :ref:`packaging-guide` for more -information on virtual dependencies. - -Now when we try to install this package, a lot more happens: - -.. code-block:: console - - $ spack install mpileaks - ... - ==> Successfully installed libdwarf from binary cache - [+] ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/libdwarf-20180129-p4jeflorwlnkoq2vpuyocwrbcht2ayak - ==> Installing callpath - ==> Searching for binary cache of callpath - ==> Installing callpath from binary cache - ==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/callpath-1.0.4/linux-ubuntu16.04-x86_64-gcc-5.4.0-callpath-1.0.4-empvyxdkc4j4pwg7gznwhbiumruey66x.spack - ######################################################################## 100.0% - ==> Successfully installed callpath from binary cache - [+] ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/callpath-1.0.4-empvyxdkc4j4pwg7gznwhbiumruey66x - ==> Installing mpileaks - ==> Searching for binary cache of mpileaks - ==> No binary for mpileaks found: installing from source - ==> Using cached archive: ~/spack/var/spack/cache/mpileaks/mpileaks-1.0.tar.gz - ==> Staging archive: ~/spack/var/spack/stage/mpileaks-1.0-csoikctsalli4cdkkdk377gprkc472rb/mpileaks-1.0.tar.gz - ==> Created stage in ~/spack/var/spack/stage/mpileaks-1.0-csoikctsalli4cdkkdk377gprkc472rb - ==> No patches needed for mpileaks - ==> Building mpileaks [Package] - ==> Executing phase: 'install' - ==> Error: ProcessError: Command exited with status 2: - 'make' '-j16' - - 1 error found in build log: - 1 ==> Executing phase: 'install' - 2 ==> 'make' '-j16' - >> 3 make: *** No targets specified and no makefile found. Stop. - - See build log for details: - ~/spack/var/spack/stage/mpileaks-1.0-csoikctsalli4cdkkdk377gprkc472rb/mpileaks-1.0/spack-build-out.txt - -Note that this command may take a while to run and produce more output if -you don't have an MPI already installed or configured in Spack. - -Now Spack has identified and made sure all of our dependencies have been -built. It found the ``openmpi`` package that will satisfy our ``mpi`` -dependency, and the ``callpath`` and ``adept-utils`` package to satisfy our -concrete dependencies. - ------------------------- -Debugging Package Builds ------------------------- - -Our ``mpileaks`` package is still not building. It may be obvious to -many of you that we never ran the configure script. Let's add a -call to ``configure()`` to the top of the install routine. The resulting -``package.py`` is in ``$SPACK_ROOT/lib/spack/docs/tutorial/examples/3.package.py``: - -.. literalinclude:: tutorial/examples/3.package.py - :lines: 6- - :language: python - -If we re-run we still get errors: - -.. code-block:: console - - $ spack install mpileaks - ... - ==> Installing mpileaks - ==> Searching for binary cache of mpileaks - ==> Finding buildcaches in /mirror/build_cache - ==> No binary for mpileaks found: installing from source - ==> Using cached archive: ~/spack/var/spack/cache/mpileaks/mpileaks-1.0.tar.gz - ==> Staging archive: ~/spack/var/spack/stage/mpileaks-1.0-csoikctsalli4cdkkdk377gprkc472rb/mpileaks-1.0.tar.gz - ==> Created stage in ~/spack/var/spack/stage/mpileaks-1.0-csoikctsalli4cdkkdk377gprkc472rb - ==> No patches needed for mpileaks - ==> Building mpileaks [Package] - ==> Executing phase: 'install' - ==> Error: ProcessError: Command exited with status 1: - './configure' - - 1 error found in build log: - 25 checking for ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.1.3-3 - njc4q5pqdpptq6jvqjrezkffwokv2sx/bin/mpicc... ~/spack/opt/spack/linux-ubuntu16.04- - x86_64/gcc-5.4.0/openmpi-3.1.3-3njc4q5pqdpptq6jvqjrezkffwokv2sx/bin/mpicc - 26 Checking whether ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.1 - .3-3njc4q5pqdpptq6jvqjrezkffwokv2sx/bin/mpicc responds to '-showme:compile'... no - 27 Checking whether ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.1 - .3-3njc4q5pqdpptq6jvqjrezkffwokv2sx/bin/mpicc responds to '-showme'... no - 28 Checking whether ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.1 - .3-3njc4q5pqdpptq6jvqjrezkffwokv2sx/bin/mpicc responds to '-compile-info'... no - 29 Checking whether ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.1 - .3-3njc4q5pqdpptq6jvqjrezkffwokv2sx/bin/mpicc responds to '-show'... no - 30 ./configure: line 4809: Echo: command not found - >> 31 configure: error: unable to locate adept-utils installation - - See build log for details: - ~/spack/var/spack/stage/mpileaks-1.0-csoikctsalli4cdkkdk377gprkc472rb/mpileaks-1.0/spack-build-out.txt - -Again, the problem may be obvious. But let's pretend we're not -all experienced Autotools developers and use this opportunity to spend some -time debugging. We have a few options that can tell us about -what's going wrong: - -As per the error message, Spack has given us a ``spack-build-out.txt`` debug -log: - -.. code-block:: console - - ==> Executing phase: 'install' - ==> './configure' - checking metadata... no - checking installation directory variables... yes - checking for a BSD-compatible install... /usr/bin/install -c - checking whether build environment is sane... yes - checking for a thread-safe mkdir -p... /bin/mkdir -p - checking for gawk... gawk - checking whether make sets $(MAKE)... yes - checking for gcc... /home/spack1/spack/lib/spack/env/gcc/gcc - checking for C compiler default output file name... a.out - checking whether the C compiler works... yes - checking whether we are cross compiling... no - checking for suffix of executables... - checking for suffix of object files... o - checking whether we are using the GNU C compiler... yes - checking whether /home/spack1/spack/lib/spack/env/gcc/gcc accepts -g... yes - checking for /home/spack1/spack/lib/spack/env/gcc/gcc option to accept ISO C89... none needed - checking for style of include used by make... GNU - checking dependency style of /home/spack1/spack/lib/spack/env/gcc/gcc... gcc3 - checking whether /home/spack1/spack/lib/spack/env/gcc/gcc and cc understand -c and -o together... yes - checking whether we are using the GNU C++ compiler... yes - checking whether /home/spack1/spack/lib/spack/env/gcc/g++ accepts -g... yes - checking dependency style of /home/spack1/spack/lib/spack/env/gcc/g++... gcc3 - checking for /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.0.0-yo5qkfvumpmgmvlbalqcadu46j5bd52f/bin/mpicc... /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.0.0-yo5qkfvumpmgmvlbalqcadu46j5bd52f/bin/mpicc - Checking whether /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.0.0-yo5qkfvumpmgmvlbalqcadu46j5bd52f/bin/mpicc responds to '-showme:compile'... yes - configure: error: unable to locate adept-utils installation - -This gives us the output from the build, and mpileaks isn't -finding its ``adept-utils`` package. Spack has -automatically added the include and library directories of -``adept-utils`` to the compiler's search path, but some packages like -mpileaks can sometimes be picky and still want things spelled out on -their command line. But let's continue to pretend we're not experienced -developers, and explore some other debugging paths: - -We can also enter the build area and try to manually run the build: - -.. code-block:: console - - $ spack build-env mpileaks bash - $ spack cd mpileaks - -The ``spack build-env`` command spawned a new shell that contains the same -environment that Spack used to build the mpileaks package (you can -substitute bash for your favorite shell). The ``spack cd`` command -changed our working dirctory to the last attempted build for mpileaks. -From here we can manually re-run the build: - -.. code-block:: console - - $ ./configure - checking metadata... no - checking installation directory variables... yes - checking for a BSD-compatible install... /usr/bin/install -c - checking whether build environment is sane... yes - checking for a thread-safe mkdir -p... /bin/mkdir -p - checking for gawk... gawk - checking whether make sets $(MAKE)... yes - checking for gcc... /home/spack1/spack/lib/spack/env/gcc/gcc - checking for C compiler default output file name... a.out - checking whether the C compiler works... yes - checking whether we are cross compiling... no - checking for suffix of executables... - checking for suffix of object files... o - checking whether we are using the GNU C compiler... yes - checking whether /home/spack1/spack/lib/spack/env/gcc/gcc accepts -g... yes - checking for /home/spack1/spack/lib/spack/env/gcc/gcc option to accept ISO C89... none needed - checking for style of include used by make... GNU - checking dependency style of /home/spack1/spack/lib/spack/env/gcc/gcc... gcc3 - checking whether /home/spack1/spack/lib/spack/env/gcc/gcc and cc understand -c and -o together... yes - checking whether we are using the GNU C++ compiler... yes - checking whether /home/spack1/spack/lib/spack/env/gcc/g++ accepts -g... yes - checking dependency style of /home/spack1/spack/lib/spack/env/gcc/g++... gcc3 - checking for /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.0.0-yo5qkfvumpmgmvlbalqcadu46j5bd52f/bin/mpicc... /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.0.0-yo5qkfvumpmgmvlbalqcadu46j5bd52f/bin/mpicc - Checking whether /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.0.0-yo5qkfvumpmgmvlbalqcadu46j5bd52f/bin/mpicc responds to '-showme:compile'... yes - configure: error: unable to locate adept-utils installation - -We're seeing the same error, but now we're in a shell where we can run -the command ourselves and debug as needed. We could, for example, run -``./configure --help`` to see what options we can use to specify -dependencies. - -We can use the ``exit`` command to leave the shell spawned by ``spack -build-env``. - ------------------------------- -Specifying Configure Arguments ------------------------------- - -Let's add the configure arguments to the mpileaks' ``package.py``. This -version can be found in -``$SPACK_ROOT/lib/spack/docs/tutorial/examples/4.package.py``: - -.. literalinclude:: tutorial/examples/4.package.py - :lines: 6- - :language: python - -This is all we need for a working mpileaks package! If we install now we'll see: - -.. code-block:: console - - $ spack install mpileaks - ... - ==> Installing mpileaks - ==> Searching for binary cache of mpileaks - ==> Finding buildcaches in /mirror/build_cache - ==> No binary for mpileaks found: installing from source - ==> Using cached archive: ~/spack/var/spack/cache/mpileaks/mpileaks-1.0.tar.gz - ==> Staging archive: ~/spack/var/spack/stage/mpileaks-1.0-csoikctsalli4cdkkdk377gprkc472rb/mpileaks-1.0.tar.gz - ==> Created stage in ~/spack/var/spack/stage/mpileaks-1.0-csoikctsalli4cdkkdk377gprkc472rb - ==> No patches needed for mpileaks - ==> Building mpileaks [Package] - ==> Executing phase: 'install' - ==> Successfully installed mpileaks - Fetch: 0.00s. Build: 9.41s. Total: 9.41s. - [+] ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/mpileaks-1.0-csoikctsalli4cdkkdk377gprkc472rb - -There are some special circumstances in this package that are worth highlighting. -Normally, Spack would have automatically detected that mpileaks was an -Autotools-based package when we ran ``spack create`` and made it an ``AutoToolsPackage`` -class (except we added the ``-t generic`` option to skip this). Instead of -a full install routine we would have just written: - -.. code-block:: python - - def configure_args(self): - return [ - '--with-adept-utils={0}'.format(self.spec['adept-utils'].prefix), - '--with-callpath={0}'.format(self.spec['callpath'].prefix) - ] - -Similarly, if this had been a CMake-based package we -would have been filling in a ``cmake_args`` function instead of -``configure_args``. There are similar default package types for -many build environments that will be discussed later in the tutorial. - --------- -Variants --------- - -We have a successful mpileaks build, but let's take some time to improve -it. ``mpileaks`` has a build-time option to truncate parts of the stack -that it walks. Let's add a variant to allow users to set this when they -build mpileaks with Spack. - -To do this, we'll add a variant to our package, as per the following (see -``$SPACK_ROOT/lib/spack/docs/tutorial/examples/5.package.py``): - -.. literalinclude:: tutorial/examples/5.package.py - :lines: 6- - :language: python - -We've added the variant ``stackstart``, and given it a default value of -``0``. If we install now we can see the stackstart variant added to the -configure line (output truncated for length): - -.. code-block:: console - - $ spack install --verbose mpileaks stackstart=4 - ... - ==> Installing mpileaks - ==> Searching for binary cache of mpileaks - ==> Finding buildcaches in /mirror/build_cache - ==> No binary for mpileaks found: installing from source - ==> Using cached archive: ~/spack/var/spack/cache/mpileaks/mpileaks-1.0.tar.gz - ==> Staging archive: ~/spack/var/spack/stage/mpileaks-1.0-meufjojkxve3l7rci2mbud3faidgplto/mpileaks-1.0.tar.gz - ==> Created stage in ~/spack/var/spack/stage/mpileaks-1.0-meufjojkxve3l7rci2mbud3faidgplto - ==> No patches needed for mpileaks - ==> Building mpileaks [Package] - ==> Executing phase: 'install' - ==> './configure' '--prefix=~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/mpileaks-1.0-meufjojkxve3l7rci2mbud3faidgplto' '--with-adept-utils=~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/adept-utils-1.0.1-7tippnvo5g76wpijk7x5kwfpr3iqiaen' '--with-callpath=~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/callpath-1.0.4-empvyxdkc4j4pwg7gznwhbiumruey66x' '--with-stack-start-c=4' '--with-stack-start-fortran=4' - ---------------- -The Spec Object ---------------- - -This tutorial has glossed over a few important features, which weren't -too relevant for mpileaks but may be useful for other packages. There -were several places we reference the ``self.spec`` object. This is a -powerful class for querying information about what we're building. For -example, you could use the spec to query information about how a -package's dependencies were built, or what compiler was being used, or -what version of a package is being installed. Full documentation can be -found in the :ref:`packaging-guide`, but here's some quick snippets with -common queries: - -- Am I building ``mpileaks`` version ``1.1`` or greater? - -.. code-block:: python - - if self.spec.satisfies('@1.1:'): - # Do things needed for 1.1+ - -- Is ``openmpi`` the MPI I'm building with? - -.. code-block:: python - - if self.spec['mpi'].name == 'openmpi': - # Do openmpi things - -- Am I building with ``gcc`` version less than ``5.0.0``: - -.. code-block:: python - - if self.spec.satisfies('%gcc@:5.0.0'): - # Add arguments specific to gcc's earlier than 5.0.0 - -- Am I building with the ``debug`` variant: - -.. code-block:: python - - if self.spec.satisfies('+debug'): - # Add -g option to configure flags - -- Is my ``dyninst`` dependency greater than version ``8.0``? - -.. code-block:: python - - if self.spec['dyninst'].satisfies('@8.0:'): - # Use newest dyninst options - -More examples can be found in the thousands of packages already added to -Spack in ``$SPACK_ROOT/var/spack/repos/builtin/packages``. - -Good Luck! - -To ensure that future sections of the tutorial run properly, please -uninstall mpileaks and remove the tutorial repo from your -configuration. - -.. code-block:: console - - $ spack uninstall -ay mpileaks - $ spack repo remove tutorial - $ rm -rf $SPACK_ROOT/var/spack/repos/tutorial/packages/mpileaks |
