summaryrefslogtreecommitdiff
path: root/lib
diff options
context:
space:
mode:
authorMassimiliano Culpo <massimiliano.culpo@gmail.com>2022-03-18 10:41:27 +0100
committerGitHub <noreply@github.com>2022-03-18 10:41:27 +0100
commit2fa495154ee8afe33a863c57ba0f9584d2324fd6 (patch)
tree8639719c1f9450ee666d794cec4d709945426da8 /lib
parent0ce8b9d3983acaf2469b3b10af8bd1b72798026b (diff)
downloadspack-2fa495154ee8afe33a863c57ba0f9584d2324fd6.tar.gz
spack-2fa495154ee8afe33a863c57ba0f9584d2324fd6.tar.bz2
spack-2fa495154ee8afe33a863c57ba0f9584d2324fd6.tar.xz
spack-2fa495154ee8afe33a863c57ba0f9584d2324fd6.zip
Split the workflow section and remove outdated advices (#29344)
This PR removes a few outdated sections from the "Basics" part of the documentation. It also makes a few topic under the environment section more prominent by removing an unneeded spack.yaml subsection and promoting everything under it.
Diffstat (limited to 'lib')
-rw-r--r--lib/spack/docs/basic_usage.rst4
-rw-r--r--lib/spack/docs/environments.rst123
-rw-r--r--lib/spack/docs/index.rst2
-rw-r--r--lib/spack/docs/module_file_support.rst5
-rw-r--r--lib/spack/docs/packaging_guide.rst2
-rw-r--r--lib/spack/docs/replace_conda_homebrew.rst206
-rw-r--r--lib/spack/docs/workflows.rst1193
7 files changed, 287 insertions, 1248 deletions
diff --git a/lib/spack/docs/basic_usage.rst b/lib/spack/docs/basic_usage.rst
index e7375c43da..96fee4a125 100644
--- a/lib/spack/docs/basic_usage.rst
+++ b/lib/spack/docs/basic_usage.rst
@@ -1723,8 +1723,8 @@ Activating Extensions in a View
Another way to use extensions is to create a view, which merges the
python installation along with the extensions into a single prefix.
-See :ref:`filesystem-views` for a more in-depth description of views and
-:ref:`cmd-spack-view` for usage of the ``spack view`` command.
+See :ref:`configuring_environment_views` for a more in-depth description
+of views.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Activating Extensions Globally
diff --git a/lib/spack/docs/environments.rst b/lib/spack/docs/environments.rst
index 4e4a75b88c..b18648006e 100644
--- a/lib/spack/docs/environments.rst
+++ b/lib/spack/docs/environments.rst
@@ -384,18 +384,11 @@ Sourcing that file in Bash will make the environment available to the
user; and can be included in ``.bashrc`` files, etc. The ``loads``
file may also be copied out of the environment, renamed, etc.
-----------
-spack.yaml
-----------
-
-Spack environments can be customized at finer granularity by editing
-the ``spack.yaml`` manifest file directly.
-
.. _environment-configuration:
-^^^^^^^^^^^^^^^^^^^^^^^^
+------------------------
Configuring Environments
-^^^^^^^^^^^^^^^^^^^^^^^^
+------------------------
A variety of Spack behaviors are changed through Spack configuration
files, covered in more detail in the :ref:`configuration`
@@ -417,9 +410,9 @@ environment can be specified by ``env:NAME`` (to affect environment
``foo``, set ``--scope env:foo``). These commands will automatically
manipulate configuration inline in the ``spack.yaml`` file.
-"""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^
Inline configurations
-"""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^
Inline Environment-scope configuration is done using the same yaml
format as standard Spack configuration scopes, covered in the
@@ -440,9 +433,9 @@ a ``packages.yaml`` file) could contain:
This configuration sets the default compiler for all packages to
``intel``.
-"""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^
Included configurations
-"""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^
Spack environments allow an ``include`` heading in their yaml
schema. This heading pulls in external configuration files and applies
@@ -462,9 +455,9 @@ to make small changes to an individual Environment. Included configs
listed earlier will have higher precedence, as the included configs are
applied in reverse order.
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+-------------------------------
Manually Editing the Specs List
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+-------------------------------
The list of abstract/root specs in the Environment is maintained in
the ``spack.yaml`` manifest under the heading ``specs``.
@@ -482,9 +475,9 @@ Appending to this list in the yaml is identical to using the ``spack
add`` command from the command line. However, there is more power
available from the yaml file.
-"""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^
Spec concretization
-"""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^
Specs can be concretized separately or together, as already
explained in :ref:`environments_concretization`. The behavior active
@@ -510,9 +503,9 @@ which can currently take either one of the two allowed values ``together`` or ``
the environment remains consistent. When instead the specs are concretized
separately only the new specs will be re-concretized after any addition.
-"""""""""""""
+^^^^^^^^^^^^^
Spec Matrices
-"""""""""""""
+^^^^^^^^^^^^^
Entries in the ``specs`` list can be individual abstract specs or a
spec matrix.
@@ -572,9 +565,9 @@ This allows one to create toolchains out of combinations of
constraints and apply them somewhat indiscriminately to packages,
without regard for the applicability of the constraint.
-""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^
Spec List References
-""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^
The last type of possible entry in the specs list is a reference.
@@ -674,9 +667,9 @@ The valid variables for a ``when`` clause are:
#. ``hostname``. The hostname of the system (if ``hostname`` is an
executable in the user's PATH).
-""""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^^
SpecLists as Constraints
-""""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^^
Dependencies and compilers in Spack can be both packages in an
environment and constraints on other packages. References to SpecLists
@@ -708,33 +701,32 @@ For example, the following environment has three root packages:
This allows for a much-needed reduction in redundancy between packages
and constraints.
-^^^^^^^^^^^^^^^^^^^^^^^^^
-Environment-managed Views
-^^^^^^^^^^^^^^^^^^^^^^^^^
+----------------
+Filesystem Views
+----------------
-Spack Environments can define filesystem views of their software,
-which are maintained as packages and can be installed and uninstalled from
-the Environment. Filesystem views provide an access point for packages
-from the filesystem for users who want to access those packages
-directly. For more information on filesystem views, see the section
-:ref:`filesystem-views`.
-
-Spack Environment managed views are updated every time the environment
-is written out to the lock file ``spack.lock``, so the concrete
-environment and the view are always compatible.
+Spack Environments can define filesystem views, which provide a direct access point
+for software similar to the directory hierarchy that might exist under ``/usr/local``.
+Filesystem views are updated every time the environment is written out to the lock
+file ``spack.lock``, so the concrete environment and the view are always compatible.
+The files of the view's installed packages are brought into the view by symbolic or
+hard links, referencing the original Spack installation, or by copy.
.. _configuring_environment_views:
-"""""""""""""""""""""""""""""
-Configuring environment views
-"""""""""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Configuration in ``spack.yaml``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The Spack Environment manifest file has a top-level keyword
-``view``. Each entry under that heading is a view descriptor, headed
-by a name. The view descriptor contains the root of the view, and
+``view``. Each entry under that heading is a **view descriptor**, headed
+by a name. Any number of views may be defined under the ``view`` heading.
+The view descriptor contains the root of the view, and
optionally the projections for the view, ``select`` and
``exclude`` lists for the view and link information via ``link`` and
-``link_type``. For example, in the following manifest
+``link_type``.
+
+For example, in the following manifest
file snippet we define a view named ``mpis``, rooted at
``/path/to/view`` in which all projections use the package name,
version, and compiler name to determine the path for a given
@@ -759,8 +751,7 @@ directories.
link: all
link_type: symlink
-For more information on using view projections, see the section on
-:ref:`adding_projections_to_views`. The default for the ``select`` and
+The default for the ``select`` and
``exclude`` values is to select everything and exclude nothing. The
default projection is the default view projection (``{}``). The ``link``
attribute allows the following values:
@@ -780,8 +771,6 @@ of ``hardlink`` or ``copy``.
when the environment is not activated, and linked libraries will be located
*outside* of the view thanks to rpaths.
-Any number of views may be defined under the ``view`` heading in a
-Spack Environment.
There are two shorthands for environments with a single view. If the
environment at ``/path/to/env`` has a single view, with a root at
@@ -847,9 +836,47 @@ regenerate`` will regenerate the views for the environment. This will
apply any updates in the environment configuration that have not yet
been applied.
-""""""""""""""""""""""""""""
+.. _view_projections:
+
+""""""""""""""""
+View Projections
+""""""""""""""""
+The default projection into a view is to link every package into the
+root of the view. The projections attribute is a mapping of partial specs to
+spec format strings, defined by the :meth:`~spack.spec.Spec.format`
+function, as shown in the example below:
+
+.. code-block:: yaml
+
+ projections:
+ zlib: {name}-{version}
+ ^mpi: {name}-{version}/{^mpi.name}-{^mpi.version}-{compiler.name}-{compiler.version}
+ all: {name}-{version}/{compiler.name}-{compiler.version}
+
+The entries in the projections configuration file must all be either
+specs or the keyword ``all``. For each spec, the projection used will
+be the first non-``all`` entry that the spec satisfies, or ``all`` if
+there is an entry for ``all`` and no other entry is satisfied by the
+spec. Where the keyword ``all`` appears in the file does not
+matter.
+
+Given the example above, the spec ``zlib@1.2.8``
+will be linked into ``/my/view/zlib-1.2.8/``, the spec
+``hdf5@1.8.10+mpi %gcc@4.9.3 ^mvapich2@2.2`` will be linked into
+``/my/view/hdf5-1.8.10/mvapich2-2.2-gcc-4.9.3``, and the spec
+``hdf5@1.8.10~mpi %gcc@4.9.3`` will be linked into
+``/my/view/hdf5-1.8.10/gcc-4.9.3``.
+
+If the keyword ``all`` does not appear in the projections
+configuration file, any spec that does not satisfy any entry in the
+file will be linked into the root of the view as in a single-prefix
+view. Any entries that appear below the keyword ``all`` in the
+projections configuration file will not be used, as all specs will use
+the projection under ``all`` before reaching those entries.
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Activating environment views
-""""""""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The ``spack env activate`` command will put the default view for the
environment into the user's path, in addition to activating the
diff --git a/lib/spack/docs/index.rst b/lib/spack/docs/index.rst
index 1d292eb02d..36f19ab643 100644
--- a/lib/spack/docs/index.rst
+++ b/lib/spack/docs/index.rst
@@ -54,8 +54,8 @@ or refer to the full manual below.
features
getting_started
basic_usage
- workflows
Tutorial: Spack 101 <https://spack-tutorial.readthedocs.io>
+ replace_conda_homebrew
known_issues
.. toctree::
diff --git a/lib/spack/docs/module_file_support.rst b/lib/spack/docs/module_file_support.rst
index 72e28b653e..5ce69dfc91 100644
--- a/lib/spack/docs/module_file_support.rst
+++ b/lib/spack/docs/module_file_support.rst
@@ -378,7 +378,7 @@ most likely via the ``+blas`` variant specification.
The most heavyweight solution to module naming is to change the entire
naming convention for module files. This uses the projections format
-covered in :ref:`adding_projections_to_views`.
+covered in :ref:`view_projections`.
.. code-block:: yaml
@@ -540,8 +540,7 @@ configuration:
#. The configuration is for an :ref:`environment <environments>` and
will never be applied outside the environment,
-#. The environment in question is configured to use a :ref:`view
- <filesystem-views>`,
+#. The environment in question is configured to use a view,
#. The :ref:`environment view is configured
<configuring_environment_views>` with a projection that ensures
every package is linked to a unique directory,
diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst
index bdf557a346..eb62fa8bf8 100644
--- a/lib/spack/docs/packaging_guide.rst
+++ b/lib/spack/docs/packaging_guide.rst
@@ -2543,7 +2543,7 @@ from being linked in at activation time.
Views
-----
-As covered in :ref:`filesystem-views`, the ``spack view`` command can be
+The ``spack view`` command can be
used to symlink a number of packages into a merged prefix. The methods of
``PackageViewMixin`` can be overridden to customize how packages are added
to views. Generally this can be used to create copies of specific files rather
diff --git a/lib/spack/docs/replace_conda_homebrew.rst b/lib/spack/docs/replace_conda_homebrew.rst
new file mode 100644
index 0000000000..f2f81a4799
--- /dev/null
+++ b/lib/spack/docs/replace_conda_homebrew.rst
@@ -0,0 +1,206 @@
+.. Copyright 2013-2022 Lawrence Livermore National Security, LLC and other
+ Spack Project Developers. See the top-level COPYRIGHT file for details.
+
+ SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+=====================================
+Using Spack to Replace Homebrew/Conda
+=====================================
+
+Spack is an incredibly powerful package manager, designed for supercomputers
+where users have diverse installation needs. But Spack can also be used to
+handle simple single-user installations on your laptop. Most macOS users are
+already familiar with package managers like Homebrew and Conda, where all
+installed packages are symlinked to a single central location like ``/usr/local``.
+In this section, we will show you how to emulate the behavior of Homebrew/Conda
+using :ref:`environments`!
+
+-----
+Setup
+-----
+
+First, let's create a new environment. We'll assume that Spack is already set up
+correctly, and that you've already sourced the setup script for your shell.
+To create a new environment, simply run:
+
+.. code-block:: console
+
+ $ spack env create myenv
+
+Here, *myenv* can be anything you want to name your environment. Next, we can add
+a list of packages we would like to install into our environment. Let's say we
+want a newer version of Bash than the one that comes with macOS, and we want a
+few Python libraries. We can run:
+
+.. code-block:: console
+
+ $ spack -e myenv add bash@5 python py-numpy py-scipy py-matplotlib
+
+Each package can be listed on a separate line, or combined into a single line like we did above.
+Notice that we're explicitly asking for Bash 5 here. You can use any spec
+you would normally use on the command line with other Spack commands.
+
+Next, we want to manually configure a couple of things:
+
+.. code-block:: console
+
+ $ spack -e myenv config edit
+
+.. code-block:: yaml
+
+ # This is a Spack Environment file.
+ #
+ # It describes a set of packages to be installed, along with
+ # configuration settings.
+ spack:
+ # add package specs to the `specs` list
+ specs: [bash@5, python, py-numpy, py-scipy, py-matplotlib]
+ view: true
+
+You can see the packages we added earlier in the ``specs:`` section. If you
+ever want to add more packages, you can either use ``spack add`` or manually
+edit this file.
+
+We also need to change the ``concretization:`` option. By default, Spack
+concretizes each spec *separately*, allowing multiple versions of the same
+package to coexist. Since we want a single consistent environment, we want to
+concretize all of the specs *together*.
+
+Here is what your ``spack.yaml`` looks like with this new setting:
+
+.. code-block:: yaml
+
+ # This is a Spack Environment file.
+ #
+ # It describes a set of packages to be installed, along with
+ # configuration settings.
+ spack:
+ # add package specs to the `specs` list
+ specs: [bash@5, python, py-numpy, py-scipy, py-matplotlib]
+ view: true
+ concretization: together
+
+^^^^^^^^^^^^^^^^
+Symlink location
+^^^^^^^^^^^^^^^^
+
+Spack symlinks all installations to ``/Users/me/spack/var/spack/environments/myenv/.spack-env/view``,
+which is the default when ``view: true``.
+You can actually change this to any directory you want. For example, Homebrew
+uses ``/usr/local``, while Conda uses ``/Users/me/anaconda``. In order to access
+files in these locations, you need to update ``PATH`` and other environment variables
+to point to them. Activating the Spack environment does this automatically, but
+you can also manually set them in your ``.bashrc``.
+
+.. warning::
+
+ There are several reasons why you shouldn't use ``/usr/local``:
+
+ 1. If you are on macOS 10.11+ (El Capitan and newer), Apple makes it hard
+ for you. You may notice permissions issues on ``/usr/local`` due to their
+ `System Integrity Protection <https://support.apple.com/en-us/HT204899>`_.
+ By default, users don't have permissions to install anything in ``/usr/local``,
+ and you can't even change this using ``sudo chown`` or ``sudo chmod``.
+ 2. Other package managers like Homebrew will try to install things to the
+ same directory. If you plan on using Homebrew in conjunction with Spack,
+ don't symlink things to ``/usr/local``.
+ 3. If you are on a shared workstation, or don't have sudo privileges, you
+ can't do this.
+
+ If you still want to do this anyway, there are several ways around SIP.
+ You could disable SIP by booting into recovery mode and running
+ ``csrutil disable``, but this is not recommended, as it can open up your OS
+ to security vulnerabilities. Another technique is to run ``spack concretize``
+ and ``spack install`` using ``sudo``. This is also not recommended.
+
+ The safest way I've found is to create your installation directories using
+ sudo, then change ownership back to the user like so:
+
+ .. code-block:: bash
+
+ for directory in .spack bin contrib include lib man share
+ do
+ sudo mkdir -p /usr/local/$directory
+ sudo chown $(id -un):$(id -gn) /usr/local/$directory
+ done
+
+ Depending on the packages you install in your environment, the exact list of
+ directories you need to create may vary. You may also find some packages
+ like Java libraries that install a single file to the installation prefix
+ instead of in a subdirectory. In this case, the action is the same, just replace
+ ``mkdir -p`` with ``touch`` in the for-loop above.
+
+ But again, it's safer just to use the default symlink location.
+
+
+------------
+Installation
+------------
+
+To actually concretize the environment, run:
+
+.. code-block:: console
+
+ $ spack -e myenv concretize
+
+This will tell you which if any packages are already installed, and alert you
+to any conflicting specs.
+
+To actually install these packages and symlink them to your ``view:``
+directory, simply run:
+
+.. code-block:: console
+
+ $ spack -e myenv install
+ $ spack env activate myenv
+
+Now, when you type ``which python3``, it should find the one you just installed.
+
+In order to change the default shell to our newer Bash installation, we first
+need to add it to this list of acceptable shells. Run:
+
+.. code-block:: console
+
+ $ sudo vim /etc/shells
+
+and add the absolute path to your bash executable. Then run:
+
+.. code-block:: console
+
+ $ chsh -s /path/to/bash
+
+Now, when you log out and log back in, ``echo $SHELL`` should point to the
+newer version of Bash.
+
+---------------------------
+Updating Installed Packages
+---------------------------
+
+Let's say you upgraded to a new version of macOS, or a new version of Python
+was released, and you want to rebuild your entire software stack. To do this,
+simply run the following commands:
+
+.. code-block:: console
+
+ $ spack env activate myenv
+ $ spack concretize --force
+ $ spack install
+
+The ``--force`` flag tells Spack to overwrite its previous concretization
+decisions, allowing you to choose a new version of Python. If any of the new
+packages like Bash are already installed, ``spack install`` won't re-install
+them, it will keep the symlinks in place.
+
+--------------
+Uninstallation
+--------------
+
+If you decide that Spack isn't right for you, uninstallation is simple.
+Just run:
+
+.. code-block:: console
+
+ $ spack env activate myenv
+ $ spack uninstall --all
+
+This will uninstall all packages in your environment and remove the symlinks.
diff --git a/lib/spack/docs/workflows.rst b/lib/spack/docs/workflows.rst
deleted file mode 100644
index 5535c4e624..0000000000
--- a/lib/spack/docs/workflows.rst
+++ /dev/null
@@ -1,1193 +0,0 @@
-.. Copyright 2013-2022 Lawrence Livermore National Security, LLC and other
- Spack Project Developers. See the top-level COPYRIGHT file for details.
-
- SPDX-License-Identifier: (Apache-2.0 OR MIT)
-
-=========
-Workflows
-=========
-
-The process of using Spack involves building packages, running
-binaries from those packages, and developing software that depends on
-those packages. For example, one might use Spack to build the
-``netcdf`` package, use ``spack load`` to run the ``ncdump`` binary, and
-finally, write a small C program to read/write a particular NetCDF file.
-
-Spack supports a variety of workflows to suit a variety of situations
-and user preferences, there is no single way to do all these things.
-This chapter demonstrates different workflows that have been
-developed, pointing out the pros and cons of them.
-
------------
-Definitions
------------
-
-First some basic definitions.
-
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Package, Concrete Spec, Installed Package
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-In Spack, a package is an abstract recipe to build one piece of software.
-Spack packages may be used to build, in principle, any version of that
-software with any set of variants. Examples of packages include
-``curl`` and ``zlib``.
-
-A package may be *instantiated* to produce a concrete spec; one
-possible realization of a particular package, out of combinatorially
-many other realizations. For example, here is a concrete spec
-instantiated from ``curl``:
-
-.. command-output:: spack spec curl
-
-Spack's core concretization algorithm generates concrete specs by
-instantiating packages from its repo, based on a set of "hints",
-including user input and the ``packages.yaml`` file. This algorithm
-may be accessed at any time with the ``spack spec`` command.
-
-Every time Spack installs a package, that installation corresponds to
-a concrete spec. Only a vanishingly small fraction of possible
-concrete specs will be installed at any one Spack site.
-
-^^^^^^^^^^^^^^^
-Consistent Sets
-^^^^^^^^^^^^^^^
-
-A set of Spack specs is said to be *consistent* if each package is
-only instantiated one way within it --- that is, if two specs in the
-set have the same package, then they must also have the same version,
-variant, compiler, etc. For example, the following set is consistent:
-
-.. code-block:: console
-
- curl@7.50.1%gcc@5.3.0 arch=linux-SuSE11-x86_64
- ^openssl@1.0.2k%gcc@5.3.0 arch=linux-SuSE11-x86_64
- ^zlib@1.2.8%gcc@5.3.0 arch=linux-SuSE11-x86_64
- zlib@1.2.8%gcc@5.3.0 arch=linux-SuSE11-x86_64
-
-The following set is not consistent:
-
-.. code-block:: console
-
- curl@7.50.1%gcc@5.3.0 arch=linux-SuSE11-x86_64
- ^openssl@1.0.2k%gcc@5.3.0 arch=linux-SuSE11-x86_64
- ^zlib@1.2.8%gcc@5.3.0 arch=linux-SuSE11-x86_64
- zlib@1.2.7%gcc@5.3.0 arch=linux-SuSE11-x86_64
-
-The compatibility of a set of installed packages determines what may
-be done with it. It is always possible to ``spack load`` any set of
-installed packages, whether or not they are consistent, and run their
-binaries from the command line. However, a set of installed packages
-can only be linked together in one binary if it is consistent.
-
-If the user produces a series of ``spack spec`` or ``spack load``
-commands, in general there is no guarantee of consistency between
-them. Spack's concretization procedure guarantees that the results of
-any *single* ``spack spec`` call will be consistent. Therefore, the
-best way to ensure a consistent set of specs is to create a Spack
-package with dependencies, and then instantiate that package. We will
-use this technique below.
-
------------------
-Building Packages
------------------
-
-Suppose you are tasked with installing a set of software packages on a
-system in order to support one application -- both a core application
-program, plus software to prepare input and analyze output. The
-required software might be summed up as a series of ``spack install``
-commands placed in a script. If needed, this script can always be run
-again in the future. For example:
-
-.. code-block:: sh
-
- #!/bin/sh
- spack install modele-utils
- spack install emacs
- spack install ncview
- spack install nco
- spack install modele-control
- spack install py-numpy
-
-In most cases, this script will not correctly install software
-according to your specific needs: choices need to be made for
-variants, versions and virtual dependency choices may be needed. It
-*is* possible to specify these choices by extending specs on the
-command line; however, the same choices must be specified repeatedly.
-For example, if you wish to use ``openmpi`` to satisfy the ``mpi``
-dependency, then ``^openmpi`` will have to appear on *every* ``spack
-install`` line that uses MPI. It can get repetitive fast.
-
-Customizing Spack installation options is easier to do in the
-``~/.spack/packages.yaml`` file. In this file, you can specify
-preferred versions and variants to use for packages. For example:
-
-.. code-block:: yaml
-
- packages:
- python:
- version: [3.5.1]
- modele-utils:
- version: [cmake]
-
- everytrace:
- version: [develop]
- eigen:
- variants: ~suitesparse
- netcdf:
- variants: +mpi
-
- all:
- compiler: [gcc@5.3.0]
- providers:
- mpi: [openmpi]
- blas: [openblas]
- lapack: [openblas]
-
-
-This approach will work as long as you are building packages for just
-one application.
-
-^^^^^^^^^^^^^^^^^^^^^
-Multiple Applications
-^^^^^^^^^^^^^^^^^^^^^
-
-Suppose instead you're building multiple inconsistent applications.
-For example, users want package A to be built with ``openmpi`` and
-package B with ``mpich`` --- but still share many other lower-level
-dependencies. In this case, a single ``packages.yaml`` file will not
-work. Plans are to implement *per-project* ``packages.yaml`` files.
-In the meantime, one could write shell scripts to switch
-``packages.yaml`` between multiple versions as needed, using symlinks.
-
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Combinatorial Sets of Installs
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Suppose that you are now tasked with systematically building many
-incompatible versions of packages. For example, you need to build
-``petsc`` 9 times for 3 different MPI implementations on 3 different
-compilers, in order to support user needs. In this case, you will
-need to either create 9 different ``packages.yaml`` files; or more
-likely, create 9 different ``spack install`` command lines with the
-correct options in the spec. Here is a real-life example of this kind
-of usage:
-
-.. code-block:: sh
-
- #!/bin/bash
-
- compilers=(
- %gcc
- %intel
- %pgi
- )
-
- mpis=(
- openmpi+psm~verbs
- openmpi~psm+verbs
- mvapich2+psm~mrail
- mvapich2~psm+mrail
- mpich+verbs
- )
-
- for compiler in "${compilers[@]}"
- do
- # Serial installs
- spack install szip $compiler
- spack install hdf $compiler
- spack install hdf5 $compiler
- spack install netcdf $compiler
- spack install netcdf-fortran $compiler
- spack install ncview $compiler
-
- # Parallel installs
- for mpi in "${mpis[@]}"
- do
- spack install $mpi $compiler
- spack install hdf5~cxx+mpi $compiler ^$mpi
- spack install parallel-netcdf $compiler ^$mpi
- done
- done
-
-------------------------------
-Running Binaries from Packages
-------------------------------
-
-Once Spack packages have been built, the next step is to use them. As
-with building packages, there are many ways to use them, depending on
-the use case.
-
-^^^^^^^^^^^^
-Find and Run
-^^^^^^^^^^^^
-
-The simplest way to run a Spack binary is to find it and run it!
-In many cases, nothing more is needed because Spack builds binaries
-with RPATHs. Spack installation directories may be found with ``spack
-location --install-dir`` commands. For example:
-
-.. code-block:: console
-
- $ spack location --install-dir cmake
- ~/spack/opt/spack/linux-SuSE11-x86_64/gcc-5.3.0/cmake-3.6.0-7cxrynb6esss6jognj23ak55fgxkwtx7
-
-This gives the root of the Spack package; relevant binaries may be
-found within it. For example:
-
-.. code-block:: console
-
- $ CMAKE=`spack location --install-dir cmake`/bin/cmake
-
-
-Standard UNIX tools can find binaries as well. For example:
-
-.. code-block:: console
-
- $ find ~/spack/opt -name cmake | grep bin
- ~/spack/opt/spack/linux-SuSE11-x86_64/gcc-5.3.0/cmake-3.6.0-7cxrynb6esss6jognj23ak55fgxkwtx7/bin/cmake
-
-These methods are suitable, for example, for setting up build
-processes or GUIs that need to know the location of particular tools.
-However, other more powerful methods are generally preferred for user
-environments.
-
-
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Using ``spack load`` to Manage the User Environment
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Suppose that Spack has been used to install a set of command-line
-programs, which users now wish to use. One can in principle put a
-number of ``spack load`` commands into ``.bashrc``, for example, to
-load a set of Spack packages:
-
-.. code-block:: sh
-
- spack load modele-utils
- spack load emacs
- spack load ncview
- spack load nco
- spack load modele-control
-
-Although simple load scripts like this are useful in many cases, they
-have some drawbacks:
-
-1. The set of packages loaded by them will in general not be
- consistent. They are a decent way to load commands to be called
- from command shells. See below for better ways to assemble a
- consistent set of packages for building application programs.
-
-2. The ``spack spec`` and ``spack install`` commands use a
- sophisticated concretization algorithm that chooses the "best"
- among several options, taking into account ``packages.yaml`` file.
- The ``spack load`` and ``spack module tcl loads`` commands, on the
- other hand, are not very smart: if the user-supplied spec matches
- more than one installed package, then ``spack module tcl loads`` will
- fail. This default behavior may change in the future. For now,
- the workaround is to either be more specific on any failing ``spack load``
- commands or to use ``spack load --first`` to allow spack to load the
- first matching spec.
-
-
-""""""""""""""""""""""
-Generated Load Scripts
-""""""""""""""""""""""
-
-Another problem with using `spack load` is, it can be slow; a typical
-user environment could take several seconds to load, and would not be
-appropriate to put into ``.bashrc`` directly. This is because it
-requires the full start-up overhead of python/Spack for each command.
-In some circumstances it is preferable to use a series of ``spack
-module tcl loads`` (or ``spack module lmod loads``) commands to
-pre-compute which modules to load. This will generate the modulenames
-to load the packages using environment modules, rather than Spack's
-built-in support for environment modifications. These can be put in a
-script that is run whenever installed Spack packages change. For
-example:
-
-.. code-block:: sh
-
- #!/bin/sh
- #
- # Generate module load commands in ~/env/spackenv
-
- cat <<EOF | /bin/sh >$HOME/env/spackenv
- FIND='spack module tcl loads --prefix linux-SuSE11-x86_64/'
-
- \$FIND modele-utils
- \$FIND emacs
- \$FIND ncview
- \$FIND nco
- \$FIND modele-control
- EOF
-
-The output of this file is written in ``~/env/spackenv``:
-
-.. code-block:: sh
-
- # binutils@2.25%gcc@5.3.0+gold~krellpatch~libiberty arch=linux-SuSE11-x86_64
- module load linux-SuSE11-x86_64/binutils-2.25-gcc-5.3.0-6w5d2t4
- # python@2.7.12%gcc@5.3.0~tk~ucs4 arch=linux-SuSE11-x86_64
- module load linux-SuSE11-x86_64/python-2.7.12-gcc-5.3.0-2azoju2
- # ncview@2.1.7%gcc@5.3.0 arch=linux-SuSE11-x86_64
- module load linux-SuSE11-x86_64/ncview-2.1.7-gcc-5.3.0-uw3knq2
- # nco@4.5.5%gcc@5.3.0 arch=linux-SuSE11-x86_64
- module load linux-SuSE11-x86_64/nco-4.5.5-gcc-5.3.0-7aqmimu
- # modele-control@develop%gcc@5.3.0 arch=linux-SuSE11-x86_64
- module load linux-SuSE11-x86_64/modele-control-develop-gcc-5.3.0-7rddsij
- # zlib@1.2.8%gcc@5.3.0 arch=linux-SuSE11-x86_64
- module load linux-SuSE11-x86_64/zlib-1.2.8-gcc-5.3.0-fe5onbi
- # curl@7.50.1%gcc@5.3.0 arch=linux-SuSE11-x86_64
- module load linux-SuSE11-x86_64/curl-7.50.1-gcc-5.3.0-4vlev55
- # hdf5@1.10.0-patch1%gcc@5.3.0+cxx~debug+fortran+mpi+shared~szip~threadsafe arch=linux-SuSE11-x86_64
- module load linux-SuSE11-x86_64/hdf5-1.10.0-patch1-gcc-5.3.0-pwnsr4w
- # netcdf@4.4.1%gcc@5.3.0~hdf4+mpi arch=linux-SuSE11-x86_64
- module load linux-SuSE11-x86_64/netcdf-4.4.1-gcc-5.3.0-rl5canv
- # netcdf-fortran@4.4.4%gcc@5.3.0 arch=linux-SuSE11-x86_64
- module load linux-SuSE11-x86_64/netcdf-fortran-4.4.4-gcc-5.3.0-stdk2xq
- # modele-utils@cmake%gcc@5.3.0+aux+diags+ic arch=linux-SuSE11-x86_64
- module load linux-SuSE11-x86_64/modele-utils-cmake-gcc-5.3.0-idyjul5
- # everytrace@develop%gcc@5.3.0+fortran+mpi arch=linux-SuSE11-x86_64
- module load linux-SuSE11-x86_64/everytrace-develop-gcc-5.3.0-p5wmb25
-
-Users may now put ``source ~/env/spackenv`` into ``.bashrc``.
-
-.. note ::
-
- Some module systems put a prefix on the names of modules created
- by Spack. For example, that prefix is ``linux-SuSE11-x86_64/`` in
- the above case. If a prefix is not needed, you may omit the
- ``--prefix`` flag from ``spack module tcl loads``.
-
-
-"""""""""""""""""""""""
-Transitive Dependencies
-"""""""""""""""""""""""
-
-In the script above, each ``spack module tcl loads`` command generates a
-*single* ``module load`` line. Transitive dependencies do not usually
-need to be loaded, only modules the user needs in ``$PATH``. This is
-because Spack builds binaries with RPATH. Spack's RPATH policy has
-some nice features:
-
-#. Modules for multiple inconsistent applications may be loaded
- simultaneously. In the above example (Multiple Applications),
- package A and package B can coexist together in the user's $PATH,
- even though they use different MPIs.
-
-#. RPATH eliminates a whole class of strange errors that can happen
- in non-RPATH binaries when the wrong ``LD_LIBRARY_PATH`` is
- loaded.
-
-#. Recursive module systems such as LMod are not necessary.
-
-#. Modules are not needed at all to execute binaries. If a path to a
- binary is known, it may be executed. For example, the path for a
- Spack-built compiler can be given to an IDE without requiring the
- IDE to load that compiler's module.
-
-Unfortunately, Spack's RPATH support does not work in every case. For example:
-
-#. Software comes in many forms --- not just compiled ELF binaries,
- but also as interpreted code in Python, R, JVM bytecode, etc.
- Those systems almost universally use an environment variable
- analogous to ``LD_LIBRARY_PATH`` to dynamically load libraries.
-
-#. Although Spack generally builds binaries with RPATH, it does not
- currently do so for compiled Python extensions (for example,
- ``py-numpy``). Any libraries that these extensions depend on
- (``blas`` in this case, for example) must be specified in the
- ``LD_LIBRARY_PATH``.`
-
-#. In some cases, Spack-generated binaries end up without a
- functional RPATH for no discernible reason.
-
-In cases where RPATH support doesn't make things "just work," it can
-be necessary to load a module's dependencies as well as the module
-itself. This is done by adding the ``--dependencies`` flag to the
-``spack module tcl loads`` command. For example, the following line,
-added to the script above, would be used to load SciPy, along with
-Numpy, core Python, BLAS/LAPACK and anything else needed:
-
-.. code-block:: sh
-
- spack module tcl loads --dependencies py-scipy
-
-^^^^^^^^^^^^^^
-Dummy Packages
-^^^^^^^^^^^^^^
-
-As an alternative to a series of ``module load`` commands, one might
-consider dummy packages as a way to create a *consistent* set of
-packages that may be loaded as one unit. The idea here is pretty
-simple:
-
-#. Create a package (say, ``mydummy``) with no URL and no
- ``install()`` method, just dependencies.
-
-#. Run ``spack install mydummy`` to install.
-
-An advantage of this method is the set of packages produced will be
-consistent. This means that you can reliably build software against
-it. A disadvantage is the set of packages will be consistent; this
-means you cannot load up two applications this way if they are not
-consistent with each other.
-
-.. _filesystem-views:
-
-^^^^^^^^^^^^^^^^
-Filesystem Views
-^^^^^^^^^^^^^^^^
-
-Filesystem views offer an alternative to environment modules, another
-way to assemble packages in a useful way and load them into a user's
-environment.
-
-A single-prefix filesystem view is a single directory tree that is the
-union of the directory hierarchies of a number of installed packages;
-it is similar to the directory hierarchy that might exist under
-``/usr/local``. The files of the view's installed packages are
-brought into the view by symbolic or hard links, referencing the
-original Spack installation.
-
-A combinatorial filesystem view can contain more software than a
-single-prefix view. Combinatorial filesystem views are created by
-defining a projection for each spec or set of specs. The syntax for
-this will be discussed in the section for the ``spack view`` command
-under `adding_projections_to_views`_.
-
-The projection for a spec or set of specs specifies the naming scheme
-for the directory structure under the root of the view into which the
-package will be linked. For example, the spec ``zlib@1.2.8%gcc@4.4.7``
-could be projected to ``MYVIEW/zlib-1.2.8-gcc``.
-
-When software is built and installed, absolute paths are frequently
-"baked into" the software, making it non-relocatable. This happens
-not just in RPATHs, but also in shebangs, configuration files, and
-assorted other locations.
-
-Therefore, programs run out of a Spack view will typically still look
-in the original Spack-installed location for shared libraries and
-other resources. This behavior is not easily changed; in general,
-there is no way to know where absolute paths might be written into an
-installed package, and how to relocate it. Therefore, the original
-Spack tree must be kept in place for a filesystem view to work, even
-if the view is built with hardlinks.
-
-.. FIXME: reference the relocation work of Hegner and Gartung (PR #1013)
-
-.. _cmd-spack-view:
-
-""""""""""""""
-``spack view``
-""""""""""""""
-
-A filesystem view is created, and packages are linked in, by the ``spack
-view`` command's ``symlink`` and ``hardlink`` sub-commands. The
-``spack view remove`` command can be used to unlink some or all of the
-filesystem view.
-
-The following example creates a filesystem view based
-on an installed ``cmake`` package and then removes from the view the
-files in the ``cmake`` package while retaining its dependencies.
-
-.. code-block:: console
-
- $ spack view --verbose symlink myview cmake@3.5.2
- ==> Linking package: "ncurses"
- ==> Linking package: "zlib"
- ==> Linking package: "openssl"
- ==> Linking package: "cmake"
-
- $ ls myview/
- bin doc etc include lib share
-
- $ ls myview/bin/
- captoinfo clear cpack ctest infotocap openssl tabs toe tset
- ccmake cmake c_rehash infocmp ncurses6-config reset tic tput
-
- $ spack view --verbose --dependencies false rm myview cmake@3.5.2
- ==> Removing package: "cmake"
-
- $ ls myview/bin/
- captoinfo c_rehash infotocap openssl tabs toe tset
- clear infocmp ncurses6-config reset tic tput
-
-.. note::
-
- If the set of packages being included in a view is inconsistent,
- then it is possible that two packages will provide the same file. Any
- conflicts of this type are handled on a first-come-first-served basis,
- and a warning is printed.
-
-.. note::
-
- When packages are removed from a view, empty directories are
- purged.
-
-.. _adding_projections_to_views:
-
-""""""""""""""""""""""""""""
-Controlling View Projections
-""""""""""""""""""""""""""""
-
-The default projection into a view is to link every package into the
-root of the view. This can be changed by adding a ``projections.yaml``
-configuration file to the view. The projection configuration file for
-a view located at ``/my/view`` is stored in
-``/my/view/.spack/projections.yaml``.
-
-When creating a view, the projection configuration file can also be
-specified from the command line using the ``--projection-file`` option
-to the ``spack view`` command.
-
-The projections configuration file is a mapping of partial specs to
-spec format strings, defined by the :meth:`~spack.spec.Spec.format`
-function, as shown in the example below.
-
-.. code-block:: yaml
-
- projections:
- zlib: {name}-{version}
- ^mpi: {name}-{version}/{^mpi.name}-{^mpi.version}-{compiler.name}-{compiler.version}
- all: {name}-{version}/{compiler.name}-{compiler.version}
-
-The entries in the projections configuration file must all be either
-specs or the keyword ``all``. For each spec, the projection used will
-be the first non-``all`` entry that the spec satisfies, or ``all`` if
-there is an entry for ``all`` and no other entry is satisfied by the
-spec. Where the keyword ``all`` appears in the file does not
-matter. Given the example above, any spec satisfying ``zlib@1.2.8``
-will be linked into ``/my/view/zlib-1.2.8/``, any spec satisfying
-``hdf5@1.8.10+mpi %gcc@4.9.3 ^mvapich2@2.2`` will be linked into
-``/my/view/hdf5-1.8.10/mvapich2-2.2-gcc-4.9.3``, and any spec
-satisfying ``hdf5@1.8.10~mpi %gcc@4.9.3`` will be linked into
-``/my/view/hdf5-1.8.10/gcc-4.9.3``.
-
-If the keyword ``all`` does not appear in the projections
-configuration file, any spec that does not satisfy any entry in the
-file will be linked into the root of the view as in a single-prefix
-view. Any entries that appear below the keyword ``all`` in the
-projections configuration file will not be used, as all specs will use
-the projection under ``all`` before reaching those entries.
-
-""""""""""""""""""
-Fine-Grain Control
-""""""""""""""""""
-
-The ``--exclude`` and ``--dependencies`` option flags allow for
-fine-grained control over which packages and dependencies do or not
-get included in a view. For example, suppose you are developing the
-``appsy`` package. You wish to build against a view of all ``appsy``
-dependencies, but not ``appsy`` itself:
-
-.. code-block:: console
-
- $ spack view --dependencies yes --exclude appsy symlink /path/to/MYVIEW/ appsy
-
-Alternately, you wish to create a view whose purpose is to provide
-binary executables to end users. You only need to include
-applications they might want, and not those applications'
-dependencies. In this case, you might use:
-
-.. code-block:: console
-
- $ spack view --dependencies no symlink /path/to/MYVIEW/ cmake
-
-
-"""""""""""""""""""""""
-Hybrid Filesystem Views
-"""""""""""""""""""""""
-
-Although filesystem views are usually created by Spack, users are free
-to add to them by other means. For example, imagine a filesystem
-view, created by Spack, that looks something like:
-
-.. code-block:: console
-
- /path/to/MYVIEW/bin/programA -> /path/to/spack/.../bin/programA
- /path/to/MYVIEW/lib/libA.so -> /path/to/spack/.../lib/libA.so
-
-Now, the user may add to this view by non-Spack means; for example, by
-running a classic install script. For example:
-
-.. code-block:: console
-
- $ tar -xf B.tar.gz
- $ cd B/
- $ ./configure --prefix=/path/to/MYVIEW \
- --with-A=/path/to/MYVIEW
- $ make && make install
-
-The result is a hybrid view:
-
-.. code-block:: console
-
- /path/to/MYVIEW/bin/programA -> /path/to/spack/.../bin/programA
- /path/to/MYVIEW/bin/programB
- /path/to/MYVIEW/lib/libA.so -> /path/to/spack/.../lib/libA.so
- /path/to/MYVIEW/lib/libB.so
-
-In this case, real files coexist, interleaved with the "view"
-symlinks. At any time one can delete ``/path/to/MYVIEW`` or use
-``spack view`` to manage it surgically. None of this will affect the
-real Spack install area.
-
-^^^^^^^^^^^^^^^^^^
-Global Activations
-^^^^^^^^^^^^^^^^^^
-
-:ref:`cmd-spack-activate` may be used as an alternative to loading
-Python (and similar systems) packages directly or creating a view.
-If extensions are globally activated, then ``spack load python`` will
-also load all the extensions activated for the given ``python``.
-This reduces the need for users to load a large number of packages.
-
-However, Spack global activations have two potential drawbacks:
-
-#. Activated packages that involve compiled C extensions may still
- need their dependencies to be loaded manually. For example,
- ``spack load openblas`` might be required to make ``py-numpy``
- work.
-
-#. Global activations "break" a core feature of Spack, which is that
- multiple versions of a package can co-exist side-by-side. For example,
- suppose you wish to run a Python package in two different
- environments but the same basic Python --- one with
- ``py-numpy@1.7`` and one with ``py-numpy@1.8``. Spack extensions
- will not support this potential debugging use case.
-
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Discussion: Running Binaries
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Modules, extension packages and filesystem views are all ways to
-assemble sets of Spack packages into a useful environment. They are
-all semantically similar, in that conflicting installed packages
-cannot simultaneously be loaded, activated or included in a view.
-
-With all of these approaches, there is no guarantee that the
-environment created will be consistent. It is possible, for example,
-to simultaneously load application A that uses OpenMPI and application
-B that uses MPICH. Both applications will run just fine in this
-inconsistent environment because they rely on RPATHs, not the
-environment, to find their dependencies.
-
-In general, environments set up using modules vs. views will work
-similarly. Both can be used to set up ephemeral or long-lived
-testing/development environments. Operational differences between the
-two approaches can make one or the other preferable in certain
-environments:
-
-* Filesystem views do not require environment module infrastructure.
- Although Spack can install ``environment-modules``, users might be
- hostile to its use. Filesystem views offer a good solution for
- sysadmins serving users who just "want all the stuff I need in one
- place" and don't want to hear about Spack.
-
-* Although modern build systems will find dependencies wherever they
- might be, some applications with hand-built make files expect their
- dependencies to be in one place. One common problem is makefiles
- that assume that ``netcdf`` and ``netcdf-fortran`` are installed in
- the same tree. Or, one might use an IDE that requires tedious
- configuration of dependency paths; and it's easier to automate that
- administration in a view-building script than in the IDE itself.
- For all these cases, a view will be preferable to other ways to
- assemble an environment.
-
-* On systems with I-node quotas, modules might be preferable to views
- and extension packages.
-
-* Views and activated extensions maintain state that is semantically
- equivalent to the information in a ``spack module tcl loads`` script.
- Administrators might find things easier to maintain without the
- added "heavyweight" state of a view.
-
--------------------------------------
-Using Spack to Replace Homebrew/Conda
--------------------------------------
-
-Spack is an incredibly powerful package manager, designed for supercomputers
-where users have diverse installation needs. But Spack can also be used to
-handle simple single-user installations on your laptop. Most macOS users are
-already familiar with package managers like Homebrew and Conda, where all
-installed packages are symlinked to a single central location like ``/usr/local``.
-In this section, we will show you how to emulate the behavior of Homebrew/Conda
-using :ref:`environments`!
-
-^^^^^
-Setup
-^^^^^
-
-First, let's create a new environment. We'll assume that Spack is already set up
-correctly, and that you've already sourced the setup script for your shell.
-To create a new environment, simply run:
-
-.. code-block:: console
-
- $ spack env create myenv
- ==> Updating view at /Users/me/spack/var/spack/environments/myenv/.spack-env/view
- ==> Created environment 'myenv' in /Users/me/spack/var/spack/environments/myenv
- $ spack env activate myenv
-
-Here, *myenv* can be anything you want to name your environment. Next, we can add
-a list of packages we would like to install into our environment. Let's say we
-want a newer version of Bash than the one that comes with macOS, and we want a
-few Python libraries. We can run:
-
-.. code-block:: console
-
- $ spack add bash
- ==> Adding bash to environment myenv
- ==> Updating view at /Users/me/spack/var/spack/environments/myenv/.spack-env/view
- $ spack add python@3:
- ==> Adding python@3: to environment myenv
- ==> Updating view at /Users/me/spack/var/spack/environments/myenv/.spack-env/view
- $ spack add py-numpy py-scipy py-matplotlib
- ==> Adding py-numpy to environment myenv
- ==> Adding py-scipy to environment myenv
- ==> Adding py-matplotlib to environment myenv
- ==> Updating view at /Users/me/spack/var/spack/environments/myenv/.spack-env/view
-
-Each package can be listed on a separate line, or combined into a single line.
-Notice that we're explicitly asking for Python 3 here. You can use any spec
-you would normally use on the command line with other Spack commands.
-
-Next, we want to manually configure a couple of things. In the ``myenv``
-directory, we can find the ``spack.yaml`` that actually defines our environment.
-
-.. code-block:: console
-
- $ vim ~/spack/var/spack/environments/myenv/spack.yaml
-
-.. code-block:: yaml
-
- # This is a Spack Environment file.
- #
- # It describes a set of packages to be installed, along with
- # configuration settings.
- spack:
- # add package specs to the `specs` list
- specs: [bash, 'python@3:', py-numpy, py-scipy, py-matplotlib]
- view:
- default:
- root: /Users/me/spack/var/spack/environments/myenv/.spack-env/view
- projections: {}
- config: {}
- mirrors: {}
- modules:
- enable: []
- packages: {}
- repos: []
- upstreams: {}
- definitions: []
- concretization: separately
-
-You can see the packages we added earlier in the ``specs:`` section. If you
-ever want to add more packages, you can either use ``spack add`` or manually
-edit this file.
-
-We also need to change the ``concretization:`` option. By default, Spack
-concretizes each spec *separately*, allowing multiple versions of the same
-package to coexist. Since we want a single consistent environment, we want to
-concretize all of the specs *together*.
-
-Here is what your ``spack.yaml`` looks like with these new settings, and with
-some of the sections we don't plan on using removed:
-
-.. code-block:: diff
-
- spack:
- - specs: [bash, 'python@3:', py-numpy, py-scipy, py-matplotlib]
- + specs:
- + - bash
- + - 'python@3:'
- + - py-numpy
- + - py-scipy
- + - py-matplotlib
- - view:
- - default:
- - root: /Users/me/spack/var/spack/environments/myenv/.spack-env/view
- - projections: {}
- + view: /Users/me/spack/var/spack/environments/myenv/.spack-env/view
- - config: {}
- - mirrors: {}
- - modules:
- - enable: []
- - packages: {}
- - repos: []
- - upstreams: {}
- - definitions: []
- + concretization: together
- - concretization: separately
-
-""""""""""""""""
-Symlink location
-""""""""""""""""
-
-In the ``spack.yaml`` file above, you'll notice that by default, Spack symlinks
-all installations to ``/Users/me/spack/var/spack/environments/myenv/.spack-env/view``.
-You can actually change this to any directory you want. For example, Homebrew
-uses ``/usr/local``, while Conda uses ``/Users/me/anaconda``. In order to access
-files in these locations, you need to update ``PATH`` and other environment variables
-to point to them. Activating the Spack environment does this automatically, but
-you can also manually set them in your ``.bashrc``.
-
-.. warning::
-
- There are several reasons why you shouldn't use ``/usr/local``:
-
- 1. If you are on macOS 10.11+ (El Capitan and newer), Apple makes it hard
- for you. You may notice permissions issues on ``/usr/local`` due to their
- `System Integrity Protection <https://support.apple.com/en-us/HT204899>`_.
- By default, users don't have permissions to install anything in ``/usr/local``,
- and you can't even change this using ``sudo chown`` or ``sudo chmod``.
- 2. Other package managers like Homebrew will try to install things to the
- same directory. If you plan on using Homebrew in conjunction with Spack,
- don't symlink things to ``/usr/local``.
- 3. If you are on a shared workstation, or don't have sudo privileges, you
- can't do this.
-
- If you still want to do this anyway, there are several ways around SIP.
- You could disable SIP by booting into recovery mode and running
- ``csrutil disable``, but this is not recommended, as it can open up your OS
- to security vulnerabilities. Another technique is to run ``spack concretize``
- and ``spack install`` using ``sudo``. This is also not recommended.
-
- The safest way I've found is to create your installation directories using
- sudo, then change ownership back to the user like so:
-
- .. code-block:: bash
-
- for directory in .spack bin contrib include lib man share
- do
- sudo mkdir -p /usr/local/$directory
- sudo chown $(id -un):$(id -gn) /usr/local/$directory
- done
-
- Depending on the packages you install in your environment, the exact list of
- directories you need to create may vary. You may also find some packages
- like Java libraries that install a single file to the installation prefix
- instead of in a subdirectory. In this case, the action is the same, just replace
- ``mkdir -p`` with ``touch`` in the for-loop above.
-
- But again, it's safer just to use the default symlink location.
-
-
-^^^^^^^^^^^^
-Installation
-^^^^^^^^^^^^
-
-To actually concretize the environment, run:
-
-.. code-block:: console
-
- $ spack concretize
-
-This will tell you which if any packages are already installed, and alert you
-to any conflicting specs.
-
-To actually install these packages and symlink them to your ``view:``
-directory, simply run:
-
-.. code-block:: console
-
- $ spack install
-
-Now, when you type ``which python3``, it should find the one you just installed.
-
-In order to change the default shell to our newer Bash installation, we first
-need to add it to this list of acceptable shells. Run:
-
-.. code-block:: console
-
- $ sudo vim /etc/shells
-
-and add the absolute path to your bash executable. Then run:
-
-.. code-block:: console
-
- $ chsh -s /path/to/bash
-
-Now, when you log out and log back in, ``echo $SHELL`` should point to the
-newer version of Bash.
-
-^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Updating Installed Packages
-^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Let's say you upgraded to a new version of macOS, or a new version of Python
-was released, and you want to rebuild your entire software stack. To do this,
-simply run the following commands:
-
-.. code-block:: console
-
- $ spack env activate myenv
- $ spack concretize --force
- $ spack install
-
-The ``--force`` flag tells Spack to overwrite its previous concretization
-decisions, allowing you to choose a new version of Python. If any of the new
-packages like Bash are already installed, ``spack install`` won't re-install
-them, it will keep the symlinks in place.
-
-^^^^^^^^^^^^^^
-Uninstallation
-^^^^^^^^^^^^^^
-
-If you decide that Spack isn't right for you, uninstallation is simple.
-Just run:
-
-.. code-block:: console
-
- $ spack env activate myenv
- $ spack uninstall --all
-
-This will uninstall all packages in your environment and remove the symlinks.
-
-------------------------
-Using Spack on Travis-CI
-------------------------
-
-Spack can be deployed as a provider for userland software in
-`Travis-CI <https://http://travis-ci.org>`_.
-
-A starting-point for a ``.travis.yml`` file can look as follows.
-It uses `caching <https://docs.travis-ci.com/user/caching/>`_ for
-already built environments, so make sure to clean the Travis cache if
-you run into problems.
-
-The main points that are implemented below:
-
-#. Travis is detected as having up to 34 cores available, but only 2
- are actually allocated for the user. We limit the parallelism of
- the spack builds in the config.
- (The Travis yaml parser is a bit buggy on the echo command.)
-
-#. Without control for the user, Travis jobs will run on various
- ``x86_64`` microarchitectures. If you plan to cache build results,
- e.g. to accelerate dependency builds, consider building for the
- generic ``x86_64`` target only.
- Limiting the microarchitecture will also find more packages when
- working with the
- `E4S Spack build cache <https://oaciss.uoregon.edu/e4s/e4s_buildcache_inventory.html>`_.
-
-#. Builds over 10 minutes need to be prefixed with ``travis_wait``.
- Alternatively, generate output once with ``spack install -v``.
-
-#. Travis builds are non-interactive. This prevents using bash
- aliases and functions for modules. We fix that by sourcing
- ``/etc/profile`` first (or running everything in a subshell with
- ``bash -l -c '...'``).
-
-.. code-block:: yaml
-
- language: cpp
- sudo: false
- dist: trusty
-
- cache:
- apt: true
- directories:
- - $HOME/.cache
-
- addons:
- apt:
- sources:
- - ubuntu-toolchain-r-test
- packages:
- - g++-4.9
- - environment-modules
-
- env:
- global:
- - SPACK_ROOT: $HOME/.cache/spack
- - PATH: $PATH:$HOME/.cache/spack/bin
-
- before_install:
- - export CXX=g++-4.9
- - export CC=gcc-4.9
- - export FC=gfortran-4.9
- - export CXXFLAGS="-std=c++11"
-
- install:
- - |
- if ! which spack >/dev/null; then
- mkdir -p $SPACK_ROOT &&
- git clone --depth 50 https://github.com/spack/spack.git $SPACK_ROOT &&
- printf "config:\n build_jobs: 2\n" > $SPACK_ROOT/etc/spack/config.yaml &&
- printf "packages:\n all:\n target: ['x86_64']\n" \
- > $SPACK_ROOT/etc/spack/packages.yaml;
- fi
- - travis_wait spack install cmake@3.7.2~openssl~ncurses
- - travis_wait spack install boost@1.62.0~graph~iostream~locale~log~wave
- - spack clean -a
- - source /etc/profile &&
- source $SPACK_ROOT/share/spack/setup-env.sh
- - spack load cmake
- - spack load boost
-
- script:
- - mkdir -p $HOME/build
- - cd $HOME/build
- - cmake $TRAVIS_BUILD_DIR
- - make -j 2
- - make test
-
-------------------
-Upstream Bug Fixes
-------------------
-
-It is not uncommon to discover a bug in an upstream project while
-trying to build with Spack. Typically, the bug is in a package that
-serves a dependency to something else. This section describes
-procedure to work around and ultimately resolve these bugs, while not
-delaying the Spack user's main goal.
-
-^^^^^^^^^^^^^^^^^
-Buggy New Version
-^^^^^^^^^^^^^^^^^
-
-Sometimes, the old version of a package works fine, but a new version
-is buggy. For example, it was once found that `Adios did not build
-with hdf5@1.10 <https://github.com/spack/spack/issues/1683>`_. If the
-old version of ``hdf5`` will work with ``adios``, the suggested
-procedure is:
-
-#. Revert ``adios`` to the old version of ``hdf5``. Put in its
- ``adios/package.py``:
-
- .. code-block:: python
-
- # Adios does not build with HDF5 1.10
- # See: https://github.com/spack/spack/issues/1683
- depends_on('hdf5@:1.9')
-
-#. Determine whether the problem is with ``hdf5`` or ``adios``, and
- report the problem to the appropriate upstream project. In this
- case, the problem was with ``adios``.
-
-#. Once a new version of ``adios`` comes out with the bugfix, modify
- ``adios/package.py`` to reflect it:
-
- .. code-block:: python
-
- # Adios up to v1.10.0 does not build with HDF5 1.10
- # See: https://github.com/spack/spack/issues/1683
- depends_on('hdf5@:1.9', when='@:1.10.0')
- depends_on('hdf5', when='@1.10.1:')
-
-^^^^^^^^^^^^^^^^
-No Version Works
-^^^^^^^^^^^^^^^^
-
-Sometimes, *no* existing versions of a dependency work for a build.
-This typically happens when developing a new project: only then does
-the developer notice that existing versions of a dependency are all
-buggy, or the non-buggy versions are all missing a critical feature.
-
-In the long run, the upstream project will hopefully fix the bug and
-release a new version. But that could take a while, even if a bugfix
-has already been pushed to the project's repository. In the meantime,
-the Spack user needs things to work.
-
-The solution is to create an unofficial Spack release of the project,
-as soon as the bug is fixed in *some* repository. A study of the `Git
-history <https://github.com/citibeth/spack/commits/efischer/develop/var/spack/repos/builtin/packages/py-proj/package.py>`_
-of ``py-proj/package.py`` is instructive here:
-
-#. On `April 1 <https://github.com/citibeth/spack/commit/44a1d6a96706affe6ef0a11c3a780b91d21d105a>`_, an initial bugfix was identified for the PyProj project
- and a pull request submitted to PyProj. Because the upstream
- authors had not yet fixed the bug, the ``py-proj`` Spack package
- downloads from a forked repository, set up by the package's author.
- A non-numeric version number is used to make it easy to upgrade the
- package without recomputing checksums; however, this is an
- untrusted download method and should not be distributed. The
- package author has now become, temporarily, a maintainer of the
- upstream project:
-
- .. code-block:: python
-
- # We need the benefits of this PR
- # https://github.com/jswhit/pyproj/pull/54
- version('citibeth-latlong2',
- git='https://github.com/citibeth/pyproj.git',
- branch='latlong2')
-
-
-#. By May 14, the upstream project had accepted a pull request with
- the required bugfix. At this point, the forked repository was
- deleted. However, the upstream project still had not released a
- new version with a bugfix. Therefore, a Spack-only release was
- created by specifying the desired hash in the main project
- repository. The version number ``@1.9.5.1.1`` was chosen for this
- "release" because it's a descendent of the officially released
- version ``@1.9.5.1``. This is a trusted download method, and can
- be released to the Spack community:
-
- .. code-block:: python
-
- # This is not a tagged release of pyproj.
- # The changes in this "version" fix some bugs, especially with Python3 use.
- version('1.9.5.1.1', 'd035e4bc704d136db79b43ab371b27d2',
- url='https://www.github.com/jswhit/pyproj/tarball/0be612cc9f972e38b50a90c946a9b353e2ab140f')
-
- .. note::
-
- It would have been simpler to use Spack's Git download method,
- which is also a trusted download in this case:
-
- .. code-block:: python
-
- # This is not a tagged release of pyproj.
- # The changes in this "version" fix some bugs, especially with Python3 use.
- version('1.9.5.1.1',
- git='https://github.com/jswhit/pyproj.git',
- commit='0be612cc9f972e38b50a90c946a9b353e2ab140f')
-
- .. note::
-
- In this case, the upstream project fixed the bug in its
- repository in a relatively timely manner. If that had not been
- the case, the numbered version in this step could have been
- released from the forked repository.
-
-
-#. The author of the Spack package has now become an unofficial
- release engineer for the upstream project. Depending on the
- situation, it may be advisable to put ``preferred=True`` on the
- latest *officially released* version.
-
-#. As of August 31, the upstream project still had not made a new
- release with the bugfix. In the meantime, Spack-built ``py-proj``
- provides the bugfix needed by packages depending on it. As long as
- this works, there is no particular need for the upstream project to
- make a new official release.
-
-#. If the upstream project releases a new official version with the
- bugfix, then the unofficial ``version()`` line should be removed
- from the Spack package.
-
-^^^^^^^
-Patches
-^^^^^^^
-
-Spack's source patching mechanism provides another way to fix bugs in
-upstream projects. This has advantages and disadvantages compared to the procedures above.
-
-Advantages:
-
- 1. It can fix bugs in existing released versions, and (probably)
- future releases as well.
-
- 2. It is lightweight, does not require a new fork to be set up.
-
-Disadvantages:
-
- 1. It is harder to develop and debug a patch, vs. a branch in a
- repository. The user loses the automation provided by version
- control systems.
-
- 2. Although patches of a few lines work OK, large patch files can be
- hard to create and maintain.