From 2fa495154ee8afe33a863c57ba0f9584d2324fd6 Mon Sep 17 00:00:00 2001 From: Massimiliano Culpo Date: Fri, 18 Mar 2022 10:41:27 +0100 Subject: Split the workflow section and remove outdated advices (#29344) This PR removes a few outdated sections from the "Basics" part of the documentation. It also makes a few topic under the environment section more prominent by removing an unneeded spack.yaml subsection and promoting everything under it. --- lib/spack/docs/basic_usage.rst | 4 +- lib/spack/docs/environments.rst | 123 +-- lib/spack/docs/index.rst | 2 +- lib/spack/docs/module_file_support.rst | 5 +- lib/spack/docs/packaging_guide.rst | 2 +- lib/spack/docs/replace_conda_homebrew.rst | 206 +++++ lib/spack/docs/workflows.rst | 1193 ----------------------------- 7 files changed, 287 insertions(+), 1248 deletions(-) create mode 100644 lib/spack/docs/replace_conda_homebrew.rst delete mode 100644 lib/spack/docs/workflows.rst (limited to 'lib') diff --git a/lib/spack/docs/basic_usage.rst b/lib/spack/docs/basic_usage.rst index e7375c43da..96fee4a125 100644 --- a/lib/spack/docs/basic_usage.rst +++ b/lib/spack/docs/basic_usage.rst @@ -1723,8 +1723,8 @@ Activating Extensions in a View Another way to use extensions is to create a view, which merges the python installation along with the extensions into a single prefix. -See :ref:`filesystem-views` for a more in-depth description of views and -:ref:`cmd-spack-view` for usage of the ``spack view`` command. +See :ref:`configuring_environment_views` for a more in-depth description +of views. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Activating Extensions Globally diff --git a/lib/spack/docs/environments.rst b/lib/spack/docs/environments.rst index 4e4a75b88c..b18648006e 100644 --- a/lib/spack/docs/environments.rst +++ b/lib/spack/docs/environments.rst @@ -384,18 +384,11 @@ Sourcing that file in Bash will make the environment available to the user; and can be included in ``.bashrc`` files, etc. The ``loads`` file may also be copied out of the environment, renamed, etc. ----------- -spack.yaml ----------- - -Spack environments can be customized at finer granularity by editing -the ``spack.yaml`` manifest file directly. - .. _environment-configuration: -^^^^^^^^^^^^^^^^^^^^^^^^ +------------------------ Configuring Environments -^^^^^^^^^^^^^^^^^^^^^^^^ +------------------------ A variety of Spack behaviors are changed through Spack configuration files, covered in more detail in the :ref:`configuration` @@ -417,9 +410,9 @@ environment can be specified by ``env:NAME`` (to affect environment ``foo``, set ``--scope env:foo``). These commands will automatically manipulate configuration inline in the ``spack.yaml`` file. -""""""""""""""""""""" +^^^^^^^^^^^^^^^^^^^^^ Inline configurations -""""""""""""""""""""" +^^^^^^^^^^^^^^^^^^^^^ Inline Environment-scope configuration is done using the same yaml format as standard Spack configuration scopes, covered in the @@ -440,9 +433,9 @@ a ``packages.yaml`` file) could contain: This configuration sets the default compiler for all packages to ``intel``. -""""""""""""""""""""""" +^^^^^^^^^^^^^^^^^^^^^^^ Included configurations -""""""""""""""""""""""" +^^^^^^^^^^^^^^^^^^^^^^^ Spack environments allow an ``include`` heading in their yaml schema. This heading pulls in external configuration files and applies @@ -462,9 +455,9 @@ to make small changes to an individual Environment. Included configs listed earlier will have higher precedence, as the included configs are applied in reverse order. -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +------------------------------- Manually Editing the Specs List -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +------------------------------- The list of abstract/root specs in the Environment is maintained in the ``spack.yaml`` manifest under the heading ``specs``. @@ -482,9 +475,9 @@ Appending to this list in the yaml is identical to using the ``spack add`` command from the command line. However, there is more power available from the yaml file. -""""""""""""""""""" +^^^^^^^^^^^^^^^^^^^ Spec concretization -""""""""""""""""""" +^^^^^^^^^^^^^^^^^^^ Specs can be concretized separately or together, as already explained in :ref:`environments_concretization`. The behavior active @@ -510,9 +503,9 @@ which can currently take either one of the two allowed values ``together`` or `` the environment remains consistent. When instead the specs are concretized separately only the new specs will be re-concretized after any addition. -""""""""""""" +^^^^^^^^^^^^^ Spec Matrices -""""""""""""" +^^^^^^^^^^^^^ Entries in the ``specs`` list can be individual abstract specs or a spec matrix. @@ -572,9 +565,9 @@ This allows one to create toolchains out of combinations of constraints and apply them somewhat indiscriminately to packages, without regard for the applicability of the constraint. -"""""""""""""""""""" +^^^^^^^^^^^^^^^^^^^^ Spec List References -"""""""""""""""""""" +^^^^^^^^^^^^^^^^^^^^ The last type of possible entry in the specs list is a reference. @@ -674,9 +667,9 @@ The valid variables for a ``when`` clause are: #. ``hostname``. The hostname of the system (if ``hostname`` is an executable in the user's PATH). -"""""""""""""""""""""""" +^^^^^^^^^^^^^^^^^^^^^^^^ SpecLists as Constraints -"""""""""""""""""""""""" +^^^^^^^^^^^^^^^^^^^^^^^^ Dependencies and compilers in Spack can be both packages in an environment and constraints on other packages. References to SpecLists @@ -708,33 +701,32 @@ For example, the following environment has three root packages: This allows for a much-needed reduction in redundancy between packages and constraints. -^^^^^^^^^^^^^^^^^^^^^^^^^ -Environment-managed Views -^^^^^^^^^^^^^^^^^^^^^^^^^ +---------------- +Filesystem Views +---------------- -Spack Environments can define filesystem views of their software, -which are maintained as packages and can be installed and uninstalled from -the Environment. Filesystem views provide an access point for packages -from the filesystem for users who want to access those packages -directly. For more information on filesystem views, see the section -:ref:`filesystem-views`. - -Spack Environment managed views are updated every time the environment -is written out to the lock file ``spack.lock``, so the concrete -environment and the view are always compatible. +Spack Environments can define filesystem views, which provide a direct access point +for software similar to the directory hierarchy that might exist under ``/usr/local``. +Filesystem views are updated every time the environment is written out to the lock +file ``spack.lock``, so the concrete environment and the view are always compatible. +The files of the view's installed packages are brought into the view by symbolic or +hard links, referencing the original Spack installation, or by copy. .. _configuring_environment_views: -""""""""""""""""""""""""""""" -Configuring environment views -""""""""""""""""""""""""""""" +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Configuration in ``spack.yaml`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The Spack Environment manifest file has a top-level keyword -``view``. Each entry under that heading is a view descriptor, headed -by a name. The view descriptor contains the root of the view, and +``view``. Each entry under that heading is a **view descriptor**, headed +by a name. Any number of views may be defined under the ``view`` heading. +The view descriptor contains the root of the view, and optionally the projections for the view, ``select`` and ``exclude`` lists for the view and link information via ``link`` and -``link_type``. For example, in the following manifest +``link_type``. + +For example, in the following manifest file snippet we define a view named ``mpis``, rooted at ``/path/to/view`` in which all projections use the package name, version, and compiler name to determine the path for a given @@ -759,8 +751,7 @@ directories. link: all link_type: symlink -For more information on using view projections, see the section on -:ref:`adding_projections_to_views`. The default for the ``select`` and +The default for the ``select`` and ``exclude`` values is to select everything and exclude nothing. The default projection is the default view projection (``{}``). The ``link`` attribute allows the following values: @@ -780,8 +771,6 @@ of ``hardlink`` or ``copy``. when the environment is not activated, and linked libraries will be located *outside* of the view thanks to rpaths. -Any number of views may be defined under the ``view`` heading in a -Spack Environment. There are two shorthands for environments with a single view. If the environment at ``/path/to/env`` has a single view, with a root at @@ -847,9 +836,47 @@ regenerate`` will regenerate the views for the environment. This will apply any updates in the environment configuration that have not yet been applied. -"""""""""""""""""""""""""""" +.. _view_projections: + +"""""""""""""""" +View Projections +"""""""""""""""" +The default projection into a view is to link every package into the +root of the view. The projections attribute is a mapping of partial specs to +spec format strings, defined by the :meth:`~spack.spec.Spec.format` +function, as shown in the example below: + +.. code-block:: yaml + + projections: + zlib: {name}-{version} + ^mpi: {name}-{version}/{^mpi.name}-{^mpi.version}-{compiler.name}-{compiler.version} + all: {name}-{version}/{compiler.name}-{compiler.version} + +The entries in the projections configuration file must all be either +specs or the keyword ``all``. For each spec, the projection used will +be the first non-``all`` entry that the spec satisfies, or ``all`` if +there is an entry for ``all`` and no other entry is satisfied by the +spec. Where the keyword ``all`` appears in the file does not +matter. + +Given the example above, the spec ``zlib@1.2.8`` +will be linked into ``/my/view/zlib-1.2.8/``, the spec +``hdf5@1.8.10+mpi %gcc@4.9.3 ^mvapich2@2.2`` will be linked into +``/my/view/hdf5-1.8.10/mvapich2-2.2-gcc-4.9.3``, and the spec +``hdf5@1.8.10~mpi %gcc@4.9.3`` will be linked into +``/my/view/hdf5-1.8.10/gcc-4.9.3``. + +If the keyword ``all`` does not appear in the projections +configuration file, any spec that does not satisfy any entry in the +file will be linked into the root of the view as in a single-prefix +view. Any entries that appear below the keyword ``all`` in the +projections configuration file will not be used, as all specs will use +the projection under ``all`` before reaching those entries. + +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Activating environment views -"""""""""""""""""""""""""""" +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The ``spack env activate`` command will put the default view for the environment into the user's path, in addition to activating the diff --git a/lib/spack/docs/index.rst b/lib/spack/docs/index.rst index 1d292eb02d..36f19ab643 100644 --- a/lib/spack/docs/index.rst +++ b/lib/spack/docs/index.rst @@ -54,8 +54,8 @@ or refer to the full manual below. features getting_started basic_usage - workflows Tutorial: Spack 101 + replace_conda_homebrew known_issues .. toctree:: diff --git a/lib/spack/docs/module_file_support.rst b/lib/spack/docs/module_file_support.rst index 72e28b653e..5ce69dfc91 100644 --- a/lib/spack/docs/module_file_support.rst +++ b/lib/spack/docs/module_file_support.rst @@ -378,7 +378,7 @@ most likely via the ``+blas`` variant specification. The most heavyweight solution to module naming is to change the entire naming convention for module files. This uses the projections format -covered in :ref:`adding_projections_to_views`. +covered in :ref:`view_projections`. .. code-block:: yaml @@ -540,8 +540,7 @@ configuration: #. The configuration is for an :ref:`environment ` and will never be applied outside the environment, -#. The environment in question is configured to use a :ref:`view - `, +#. The environment in question is configured to use a view, #. The :ref:`environment view is configured ` with a projection that ensures every package is linked to a unique directory, diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst index bdf557a346..eb62fa8bf8 100644 --- a/lib/spack/docs/packaging_guide.rst +++ b/lib/spack/docs/packaging_guide.rst @@ -2543,7 +2543,7 @@ from being linked in at activation time. Views ----- -As covered in :ref:`filesystem-views`, the ``spack view`` command can be +The ``spack view`` command can be used to symlink a number of packages into a merged prefix. The methods of ``PackageViewMixin`` can be overridden to customize how packages are added to views. Generally this can be used to create copies of specific files rather diff --git a/lib/spack/docs/replace_conda_homebrew.rst b/lib/spack/docs/replace_conda_homebrew.rst new file mode 100644 index 0000000000..f2f81a4799 --- /dev/null +++ b/lib/spack/docs/replace_conda_homebrew.rst @@ -0,0 +1,206 @@ +.. Copyright 2013-2022 Lawrence Livermore National Security, LLC and other + Spack Project Developers. See the top-level COPYRIGHT file for details. + + SPDX-License-Identifier: (Apache-2.0 OR MIT) + +===================================== +Using Spack to Replace Homebrew/Conda +===================================== + +Spack is an incredibly powerful package manager, designed for supercomputers +where users have diverse installation needs. But Spack can also be used to +handle simple single-user installations on your laptop. Most macOS users are +already familiar with package managers like Homebrew and Conda, where all +installed packages are symlinked to a single central location like ``/usr/local``. +In this section, we will show you how to emulate the behavior of Homebrew/Conda +using :ref:`environments`! + +----- +Setup +----- + +First, let's create a new environment. We'll assume that Spack is already set up +correctly, and that you've already sourced the setup script for your shell. +To create a new environment, simply run: + +.. code-block:: console + + $ spack env create myenv + +Here, *myenv* can be anything you want to name your environment. Next, we can add +a list of packages we would like to install into our environment. Let's say we +want a newer version of Bash than the one that comes with macOS, and we want a +few Python libraries. We can run: + +.. code-block:: console + + $ spack -e myenv add bash@5 python py-numpy py-scipy py-matplotlib + +Each package can be listed on a separate line, or combined into a single line like we did above. +Notice that we're explicitly asking for Bash 5 here. You can use any spec +you would normally use on the command line with other Spack commands. + +Next, we want to manually configure a couple of things: + +.. code-block:: console + + $ spack -e myenv config edit + +.. code-block:: yaml + + # This is a Spack Environment file. + # + # It describes a set of packages to be installed, along with + # configuration settings. + spack: + # add package specs to the `specs` list + specs: [bash@5, python, py-numpy, py-scipy, py-matplotlib] + view: true + +You can see the packages we added earlier in the ``specs:`` section. If you +ever want to add more packages, you can either use ``spack add`` or manually +edit this file. + +We also need to change the ``concretization:`` option. By default, Spack +concretizes each spec *separately*, allowing multiple versions of the same +package to coexist. Since we want a single consistent environment, we want to +concretize all of the specs *together*. + +Here is what your ``spack.yaml`` looks like with this new setting: + +.. code-block:: yaml + + # This is a Spack Environment file. + # + # It describes a set of packages to be installed, along with + # configuration settings. + spack: + # add package specs to the `specs` list + specs: [bash@5, python, py-numpy, py-scipy, py-matplotlib] + view: true + concretization: together + +^^^^^^^^^^^^^^^^ +Symlink location +^^^^^^^^^^^^^^^^ + +Spack symlinks all installations to ``/Users/me/spack/var/spack/environments/myenv/.spack-env/view``, +which is the default when ``view: true``. +You can actually change this to any directory you want. For example, Homebrew +uses ``/usr/local``, while Conda uses ``/Users/me/anaconda``. In order to access +files in these locations, you need to update ``PATH`` and other environment variables +to point to them. Activating the Spack environment does this automatically, but +you can also manually set them in your ``.bashrc``. + +.. warning:: + + There are several reasons why you shouldn't use ``/usr/local``: + + 1. If you are on macOS 10.11+ (El Capitan and newer), Apple makes it hard + for you. You may notice permissions issues on ``/usr/local`` due to their + `System Integrity Protection `_. + By default, users don't have permissions to install anything in ``/usr/local``, + and you can't even change this using ``sudo chown`` or ``sudo chmod``. + 2. Other package managers like Homebrew will try to install things to the + same directory. If you plan on using Homebrew in conjunction with Spack, + don't symlink things to ``/usr/local``. + 3. If you are on a shared workstation, or don't have sudo privileges, you + can't do this. + + If you still want to do this anyway, there are several ways around SIP. + You could disable SIP by booting into recovery mode and running + ``csrutil disable``, but this is not recommended, as it can open up your OS + to security vulnerabilities. Another technique is to run ``spack concretize`` + and ``spack install`` using ``sudo``. This is also not recommended. + + The safest way I've found is to create your installation directories using + sudo, then change ownership back to the user like so: + + .. code-block:: bash + + for directory in .spack bin contrib include lib man share + do + sudo mkdir -p /usr/local/$directory + sudo chown $(id -un):$(id -gn) /usr/local/$directory + done + + Depending on the packages you install in your environment, the exact list of + directories you need to create may vary. You may also find some packages + like Java libraries that install a single file to the installation prefix + instead of in a subdirectory. In this case, the action is the same, just replace + ``mkdir -p`` with ``touch`` in the for-loop above. + + But again, it's safer just to use the default symlink location. + + +------------ +Installation +------------ + +To actually concretize the environment, run: + +.. code-block:: console + + $ spack -e myenv concretize + +This will tell you which if any packages are already installed, and alert you +to any conflicting specs. + +To actually install these packages and symlink them to your ``view:`` +directory, simply run: + +.. code-block:: console + + $ spack -e myenv install + $ spack env activate myenv + +Now, when you type ``which python3``, it should find the one you just installed. + +In order to change the default shell to our newer Bash installation, we first +need to add it to this list of acceptable shells. Run: + +.. code-block:: console + + $ sudo vim /etc/shells + +and add the absolute path to your bash executable. Then run: + +.. code-block:: console + + $ chsh -s /path/to/bash + +Now, when you log out and log back in, ``echo $SHELL`` should point to the +newer version of Bash. + +--------------------------- +Updating Installed Packages +--------------------------- + +Let's say you upgraded to a new version of macOS, or a new version of Python +was released, and you want to rebuild your entire software stack. To do this, +simply run the following commands: + +.. code-block:: console + + $ spack env activate myenv + $ spack concretize --force + $ spack install + +The ``--force`` flag tells Spack to overwrite its previous concretization +decisions, allowing you to choose a new version of Python. If any of the new +packages like Bash are already installed, ``spack install`` won't re-install +them, it will keep the symlinks in place. + +-------------- +Uninstallation +-------------- + +If you decide that Spack isn't right for you, uninstallation is simple. +Just run: + +.. code-block:: console + + $ spack env activate myenv + $ spack uninstall --all + +This will uninstall all packages in your environment and remove the symlinks. diff --git a/lib/spack/docs/workflows.rst b/lib/spack/docs/workflows.rst deleted file mode 100644 index 5535c4e624..0000000000 --- a/lib/spack/docs/workflows.rst +++ /dev/null @@ -1,1193 +0,0 @@ -.. Copyright 2013-2022 Lawrence Livermore National Security, LLC and other - Spack Project Developers. See the top-level COPYRIGHT file for details. - - SPDX-License-Identifier: (Apache-2.0 OR MIT) - -========= -Workflows -========= - -The process of using Spack involves building packages, running -binaries from those packages, and developing software that depends on -those packages. For example, one might use Spack to build the -``netcdf`` package, use ``spack load`` to run the ``ncdump`` binary, and -finally, write a small C program to read/write a particular NetCDF file. - -Spack supports a variety of workflows to suit a variety of situations -and user preferences, there is no single way to do all these things. -This chapter demonstrates different workflows that have been -developed, pointing out the pros and cons of them. - ------------ -Definitions ------------ - -First some basic definitions. - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Package, Concrete Spec, Installed Package -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -In Spack, a package is an abstract recipe to build one piece of software. -Spack packages may be used to build, in principle, any version of that -software with any set of variants. Examples of packages include -``curl`` and ``zlib``. - -A package may be *instantiated* to produce a concrete spec; one -possible realization of a particular package, out of combinatorially -many other realizations. For example, here is a concrete spec -instantiated from ``curl``: - -.. command-output:: spack spec curl - -Spack's core concretization algorithm generates concrete specs by -instantiating packages from its repo, based on a set of "hints", -including user input and the ``packages.yaml`` file. This algorithm -may be accessed at any time with the ``spack spec`` command. - -Every time Spack installs a package, that installation corresponds to -a concrete spec. Only a vanishingly small fraction of possible -concrete specs will be installed at any one Spack site. - -^^^^^^^^^^^^^^^ -Consistent Sets -^^^^^^^^^^^^^^^ - -A set of Spack specs is said to be *consistent* if each package is -only instantiated one way within it --- that is, if two specs in the -set have the same package, then they must also have the same version, -variant, compiler, etc. For example, the following set is consistent: - -.. code-block:: console - - curl@7.50.1%gcc@5.3.0 arch=linux-SuSE11-x86_64 - ^openssl@1.0.2k%gcc@5.3.0 arch=linux-SuSE11-x86_64 - ^zlib@1.2.8%gcc@5.3.0 arch=linux-SuSE11-x86_64 - zlib@1.2.8%gcc@5.3.0 arch=linux-SuSE11-x86_64 - -The following set is not consistent: - -.. code-block:: console - - curl@7.50.1%gcc@5.3.0 arch=linux-SuSE11-x86_64 - ^openssl@1.0.2k%gcc@5.3.0 arch=linux-SuSE11-x86_64 - ^zlib@1.2.8%gcc@5.3.0 arch=linux-SuSE11-x86_64 - zlib@1.2.7%gcc@5.3.0 arch=linux-SuSE11-x86_64 - -The compatibility of a set of installed packages determines what may -be done with it. It is always possible to ``spack load`` any set of -installed packages, whether or not they are consistent, and run their -binaries from the command line. However, a set of installed packages -can only be linked together in one binary if it is consistent. - -If the user produces a series of ``spack spec`` or ``spack load`` -commands, in general there is no guarantee of consistency between -them. Spack's concretization procedure guarantees that the results of -any *single* ``spack spec`` call will be consistent. Therefore, the -best way to ensure a consistent set of specs is to create a Spack -package with dependencies, and then instantiate that package. We will -use this technique below. - ------------------ -Building Packages ------------------ - -Suppose you are tasked with installing a set of software packages on a -system in order to support one application -- both a core application -program, plus software to prepare input and analyze output. The -required software might be summed up as a series of ``spack install`` -commands placed in a script. If needed, this script can always be run -again in the future. For example: - -.. code-block:: sh - - #!/bin/sh - spack install modele-utils - spack install emacs - spack install ncview - spack install nco - spack install modele-control - spack install py-numpy - -In most cases, this script will not correctly install software -according to your specific needs: choices need to be made for -variants, versions and virtual dependency choices may be needed. It -*is* possible to specify these choices by extending specs on the -command line; however, the same choices must be specified repeatedly. -For example, if you wish to use ``openmpi`` to satisfy the ``mpi`` -dependency, then ``^openmpi`` will have to appear on *every* ``spack -install`` line that uses MPI. It can get repetitive fast. - -Customizing Spack installation options is easier to do in the -``~/.spack/packages.yaml`` file. In this file, you can specify -preferred versions and variants to use for packages. For example: - -.. code-block:: yaml - - packages: - python: - version: [3.5.1] - modele-utils: - version: [cmake] - - everytrace: - version: [develop] - eigen: - variants: ~suitesparse - netcdf: - variants: +mpi - - all: - compiler: [gcc@5.3.0] - providers: - mpi: [openmpi] - blas: [openblas] - lapack: [openblas] - - -This approach will work as long as you are building packages for just -one application. - -^^^^^^^^^^^^^^^^^^^^^ -Multiple Applications -^^^^^^^^^^^^^^^^^^^^^ - -Suppose instead you're building multiple inconsistent applications. -For example, users want package A to be built with ``openmpi`` and -package B with ``mpich`` --- but still share many other lower-level -dependencies. In this case, a single ``packages.yaml`` file will not -work. Plans are to implement *per-project* ``packages.yaml`` files. -In the meantime, one could write shell scripts to switch -``packages.yaml`` between multiple versions as needed, using symlinks. - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Combinatorial Sets of Installs -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Suppose that you are now tasked with systematically building many -incompatible versions of packages. For example, you need to build -``petsc`` 9 times for 3 different MPI implementations on 3 different -compilers, in order to support user needs. In this case, you will -need to either create 9 different ``packages.yaml`` files; or more -likely, create 9 different ``spack install`` command lines with the -correct options in the spec. Here is a real-life example of this kind -of usage: - -.. code-block:: sh - - #!/bin/bash - - compilers=( - %gcc - %intel - %pgi - ) - - mpis=( - openmpi+psm~verbs - openmpi~psm+verbs - mvapich2+psm~mrail - mvapich2~psm+mrail - mpich+verbs - ) - - for compiler in "${compilers[@]}" - do - # Serial installs - spack install szip $compiler - spack install hdf $compiler - spack install hdf5 $compiler - spack install netcdf $compiler - spack install netcdf-fortran $compiler - spack install ncview $compiler - - # Parallel installs - for mpi in "${mpis[@]}" - do - spack install $mpi $compiler - spack install hdf5~cxx+mpi $compiler ^$mpi - spack install parallel-netcdf $compiler ^$mpi - done - done - ------------------------------- -Running Binaries from Packages ------------------------------- - -Once Spack packages have been built, the next step is to use them. As -with building packages, there are many ways to use them, depending on -the use case. - -^^^^^^^^^^^^ -Find and Run -^^^^^^^^^^^^ - -The simplest way to run a Spack binary is to find it and run it! -In many cases, nothing more is needed because Spack builds binaries -with RPATHs. Spack installation directories may be found with ``spack -location --install-dir`` commands. For example: - -.. code-block:: console - - $ spack location --install-dir cmake - ~/spack/opt/spack/linux-SuSE11-x86_64/gcc-5.3.0/cmake-3.6.0-7cxrynb6esss6jognj23ak55fgxkwtx7 - -This gives the root of the Spack package; relevant binaries may be -found within it. For example: - -.. code-block:: console - - $ CMAKE=`spack location --install-dir cmake`/bin/cmake - - -Standard UNIX tools can find binaries as well. For example: - -.. code-block:: console - - $ find ~/spack/opt -name cmake | grep bin - ~/spack/opt/spack/linux-SuSE11-x86_64/gcc-5.3.0/cmake-3.6.0-7cxrynb6esss6jognj23ak55fgxkwtx7/bin/cmake - -These methods are suitable, for example, for setting up build -processes or GUIs that need to know the location of particular tools. -However, other more powerful methods are generally preferred for user -environments. - - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Using ``spack load`` to Manage the User Environment -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Suppose that Spack has been used to install a set of command-line -programs, which users now wish to use. One can in principle put a -number of ``spack load`` commands into ``.bashrc``, for example, to -load a set of Spack packages: - -.. code-block:: sh - - spack load modele-utils - spack load emacs - spack load ncview - spack load nco - spack load modele-control - -Although simple load scripts like this are useful in many cases, they -have some drawbacks: - -1. The set of packages loaded by them will in general not be - consistent. They are a decent way to load commands to be called - from command shells. See below for better ways to assemble a - consistent set of packages for building application programs. - -2. The ``spack spec`` and ``spack install`` commands use a - sophisticated concretization algorithm that chooses the "best" - among several options, taking into account ``packages.yaml`` file. - The ``spack load`` and ``spack module tcl loads`` commands, on the - other hand, are not very smart: if the user-supplied spec matches - more than one installed package, then ``spack module tcl loads`` will - fail. This default behavior may change in the future. For now, - the workaround is to either be more specific on any failing ``spack load`` - commands or to use ``spack load --first`` to allow spack to load the - first matching spec. - - -"""""""""""""""""""""" -Generated Load Scripts -"""""""""""""""""""""" - -Another problem with using `spack load` is, it can be slow; a typical -user environment could take several seconds to load, and would not be -appropriate to put into ``.bashrc`` directly. This is because it -requires the full start-up overhead of python/Spack for each command. -In some circumstances it is preferable to use a series of ``spack -module tcl loads`` (or ``spack module lmod loads``) commands to -pre-compute which modules to load. This will generate the modulenames -to load the packages using environment modules, rather than Spack's -built-in support for environment modifications. These can be put in a -script that is run whenever installed Spack packages change. For -example: - -.. code-block:: sh - - #!/bin/sh - # - # Generate module load commands in ~/env/spackenv - - cat <$HOME/env/spackenv - FIND='spack module tcl loads --prefix linux-SuSE11-x86_64/' - - \$FIND modele-utils - \$FIND emacs - \$FIND ncview - \$FIND nco - \$FIND modele-control - EOF - -The output of this file is written in ``~/env/spackenv``: - -.. code-block:: sh - - # binutils@2.25%gcc@5.3.0+gold~krellpatch~libiberty arch=linux-SuSE11-x86_64 - module load linux-SuSE11-x86_64/binutils-2.25-gcc-5.3.0-6w5d2t4 - # python@2.7.12%gcc@5.3.0~tk~ucs4 arch=linux-SuSE11-x86_64 - module load linux-SuSE11-x86_64/python-2.7.12-gcc-5.3.0-2azoju2 - # ncview@2.1.7%gcc@5.3.0 arch=linux-SuSE11-x86_64 - module load linux-SuSE11-x86_64/ncview-2.1.7-gcc-5.3.0-uw3knq2 - # nco@4.5.5%gcc@5.3.0 arch=linux-SuSE11-x86_64 - module load linux-SuSE11-x86_64/nco-4.5.5-gcc-5.3.0-7aqmimu - # modele-control@develop%gcc@5.3.0 arch=linux-SuSE11-x86_64 - module load linux-SuSE11-x86_64/modele-control-develop-gcc-5.3.0-7rddsij - # zlib@1.2.8%gcc@5.3.0 arch=linux-SuSE11-x86_64 - module load linux-SuSE11-x86_64/zlib-1.2.8-gcc-5.3.0-fe5onbi - # curl@7.50.1%gcc@5.3.0 arch=linux-SuSE11-x86_64 - module load linux-SuSE11-x86_64/curl-7.50.1-gcc-5.3.0-4vlev55 - # hdf5@1.10.0-patch1%gcc@5.3.0+cxx~debug+fortran+mpi+shared~szip~threadsafe arch=linux-SuSE11-x86_64 - module load linux-SuSE11-x86_64/hdf5-1.10.0-patch1-gcc-5.3.0-pwnsr4w - # netcdf@4.4.1%gcc@5.3.0~hdf4+mpi arch=linux-SuSE11-x86_64 - module load linux-SuSE11-x86_64/netcdf-4.4.1-gcc-5.3.0-rl5canv - # netcdf-fortran@4.4.4%gcc@5.3.0 arch=linux-SuSE11-x86_64 - module load linux-SuSE11-x86_64/netcdf-fortran-4.4.4-gcc-5.3.0-stdk2xq - # modele-utils@cmake%gcc@5.3.0+aux+diags+ic arch=linux-SuSE11-x86_64 - module load linux-SuSE11-x86_64/modele-utils-cmake-gcc-5.3.0-idyjul5 - # everytrace@develop%gcc@5.3.0+fortran+mpi arch=linux-SuSE11-x86_64 - module load linux-SuSE11-x86_64/everytrace-develop-gcc-5.3.0-p5wmb25 - -Users may now put ``source ~/env/spackenv`` into ``.bashrc``. - -.. note :: - - Some module systems put a prefix on the names of modules created - by Spack. For example, that prefix is ``linux-SuSE11-x86_64/`` in - the above case. If a prefix is not needed, you may omit the - ``--prefix`` flag from ``spack module tcl loads``. - - -""""""""""""""""""""""" -Transitive Dependencies -""""""""""""""""""""""" - -In the script above, each ``spack module tcl loads`` command generates a -*single* ``module load`` line. Transitive dependencies do not usually -need to be loaded, only modules the user needs in ``$PATH``. This is -because Spack builds binaries with RPATH. Spack's RPATH policy has -some nice features: - -#. Modules for multiple inconsistent applications may be loaded - simultaneously. In the above example (Multiple Applications), - package A and package B can coexist together in the user's $PATH, - even though they use different MPIs. - -#. RPATH eliminates a whole class of strange errors that can happen - in non-RPATH binaries when the wrong ``LD_LIBRARY_PATH`` is - loaded. - -#. Recursive module systems such as LMod are not necessary. - -#. Modules are not needed at all to execute binaries. If a path to a - binary is known, it may be executed. For example, the path for a - Spack-built compiler can be given to an IDE without requiring the - IDE to load that compiler's module. - -Unfortunately, Spack's RPATH support does not work in every case. For example: - -#. Software comes in many forms --- not just compiled ELF binaries, - but also as interpreted code in Python, R, JVM bytecode, etc. - Those systems almost universally use an environment variable - analogous to ``LD_LIBRARY_PATH`` to dynamically load libraries. - -#. Although Spack generally builds binaries with RPATH, it does not - currently do so for compiled Python extensions (for example, - ``py-numpy``). Any libraries that these extensions depend on - (``blas`` in this case, for example) must be specified in the - ``LD_LIBRARY_PATH``.` - -#. In some cases, Spack-generated binaries end up without a - functional RPATH for no discernible reason. - -In cases where RPATH support doesn't make things "just work," it can -be necessary to load a module's dependencies as well as the module -itself. This is done by adding the ``--dependencies`` flag to the -``spack module tcl loads`` command. For example, the following line, -added to the script above, would be used to load SciPy, along with -Numpy, core Python, BLAS/LAPACK and anything else needed: - -.. code-block:: sh - - spack module tcl loads --dependencies py-scipy - -^^^^^^^^^^^^^^ -Dummy Packages -^^^^^^^^^^^^^^ - -As an alternative to a series of ``module load`` commands, one might -consider dummy packages as a way to create a *consistent* set of -packages that may be loaded as one unit. The idea here is pretty -simple: - -#. Create a package (say, ``mydummy``) with no URL and no - ``install()`` method, just dependencies. - -#. Run ``spack install mydummy`` to install. - -An advantage of this method is the set of packages produced will be -consistent. This means that you can reliably build software against -it. A disadvantage is the set of packages will be consistent; this -means you cannot load up two applications this way if they are not -consistent with each other. - -.. _filesystem-views: - -^^^^^^^^^^^^^^^^ -Filesystem Views -^^^^^^^^^^^^^^^^ - -Filesystem views offer an alternative to environment modules, another -way to assemble packages in a useful way and load them into a user's -environment. - -A single-prefix filesystem view is a single directory tree that is the -union of the directory hierarchies of a number of installed packages; -it is similar to the directory hierarchy that might exist under -``/usr/local``. The files of the view's installed packages are -brought into the view by symbolic or hard links, referencing the -original Spack installation. - -A combinatorial filesystem view can contain more software than a -single-prefix view. Combinatorial filesystem views are created by -defining a projection for each spec or set of specs. The syntax for -this will be discussed in the section for the ``spack view`` command -under `adding_projections_to_views`_. - -The projection for a spec or set of specs specifies the naming scheme -for the directory structure under the root of the view into which the -package will be linked. For example, the spec ``zlib@1.2.8%gcc@4.4.7`` -could be projected to ``MYVIEW/zlib-1.2.8-gcc``. - -When software is built and installed, absolute paths are frequently -"baked into" the software, making it non-relocatable. This happens -not just in RPATHs, but also in shebangs, configuration files, and -assorted other locations. - -Therefore, programs run out of a Spack view will typically still look -in the original Spack-installed location for shared libraries and -other resources. This behavior is not easily changed; in general, -there is no way to know where absolute paths might be written into an -installed package, and how to relocate it. Therefore, the original -Spack tree must be kept in place for a filesystem view to work, even -if the view is built with hardlinks. - -.. FIXME: reference the relocation work of Hegner and Gartung (PR #1013) - -.. _cmd-spack-view: - -"""""""""""""" -``spack view`` -"""""""""""""" - -A filesystem view is created, and packages are linked in, by the ``spack -view`` command's ``symlink`` and ``hardlink`` sub-commands. The -``spack view remove`` command can be used to unlink some or all of the -filesystem view. - -The following example creates a filesystem view based -on an installed ``cmake`` package and then removes from the view the -files in the ``cmake`` package while retaining its dependencies. - -.. code-block:: console - - $ spack view --verbose symlink myview cmake@3.5.2 - ==> Linking package: "ncurses" - ==> Linking package: "zlib" - ==> Linking package: "openssl" - ==> Linking package: "cmake" - - $ ls myview/ - bin doc etc include lib share - - $ ls myview/bin/ - captoinfo clear cpack ctest infotocap openssl tabs toe tset - ccmake cmake c_rehash infocmp ncurses6-config reset tic tput - - $ spack view --verbose --dependencies false rm myview cmake@3.5.2 - ==> Removing package: "cmake" - - $ ls myview/bin/ - captoinfo c_rehash infotocap openssl tabs toe tset - clear infocmp ncurses6-config reset tic tput - -.. note:: - - If the set of packages being included in a view is inconsistent, - then it is possible that two packages will provide the same file. Any - conflicts of this type are handled on a first-come-first-served basis, - and a warning is printed. - -.. note:: - - When packages are removed from a view, empty directories are - purged. - -.. _adding_projections_to_views: - -"""""""""""""""""""""""""""" -Controlling View Projections -"""""""""""""""""""""""""""" - -The default projection into a view is to link every package into the -root of the view. This can be changed by adding a ``projections.yaml`` -configuration file to the view. The projection configuration file for -a view located at ``/my/view`` is stored in -``/my/view/.spack/projections.yaml``. - -When creating a view, the projection configuration file can also be -specified from the command line using the ``--projection-file`` option -to the ``spack view`` command. - -The projections configuration file is a mapping of partial specs to -spec format strings, defined by the :meth:`~spack.spec.Spec.format` -function, as shown in the example below. - -.. code-block:: yaml - - projections: - zlib: {name}-{version} - ^mpi: {name}-{version}/{^mpi.name}-{^mpi.version}-{compiler.name}-{compiler.version} - all: {name}-{version}/{compiler.name}-{compiler.version} - -The entries in the projections configuration file must all be either -specs or the keyword ``all``. For each spec, the projection used will -be the first non-``all`` entry that the spec satisfies, or ``all`` if -there is an entry for ``all`` and no other entry is satisfied by the -spec. Where the keyword ``all`` appears in the file does not -matter. Given the example above, any spec satisfying ``zlib@1.2.8`` -will be linked into ``/my/view/zlib-1.2.8/``, any spec satisfying -``hdf5@1.8.10+mpi %gcc@4.9.3 ^mvapich2@2.2`` will be linked into -``/my/view/hdf5-1.8.10/mvapich2-2.2-gcc-4.9.3``, and any spec -satisfying ``hdf5@1.8.10~mpi %gcc@4.9.3`` will be linked into -``/my/view/hdf5-1.8.10/gcc-4.9.3``. - -If the keyword ``all`` does not appear in the projections -configuration file, any spec that does not satisfy any entry in the -file will be linked into the root of the view as in a single-prefix -view. Any entries that appear below the keyword ``all`` in the -projections configuration file will not be used, as all specs will use -the projection under ``all`` before reaching those entries. - -"""""""""""""""""" -Fine-Grain Control -"""""""""""""""""" - -The ``--exclude`` and ``--dependencies`` option flags allow for -fine-grained control over which packages and dependencies do or not -get included in a view. For example, suppose you are developing the -``appsy`` package. You wish to build against a view of all ``appsy`` -dependencies, but not ``appsy`` itself: - -.. code-block:: console - - $ spack view --dependencies yes --exclude appsy symlink /path/to/MYVIEW/ appsy - -Alternately, you wish to create a view whose purpose is to provide -binary executables to end users. You only need to include -applications they might want, and not those applications' -dependencies. In this case, you might use: - -.. code-block:: console - - $ spack view --dependencies no symlink /path/to/MYVIEW/ cmake - - -""""""""""""""""""""""" -Hybrid Filesystem Views -""""""""""""""""""""""" - -Although filesystem views are usually created by Spack, users are free -to add to them by other means. For example, imagine a filesystem -view, created by Spack, that looks something like: - -.. code-block:: console - - /path/to/MYVIEW/bin/programA -> /path/to/spack/.../bin/programA - /path/to/MYVIEW/lib/libA.so -> /path/to/spack/.../lib/libA.so - -Now, the user may add to this view by non-Spack means; for example, by -running a classic install script. For example: - -.. code-block:: console - - $ tar -xf B.tar.gz - $ cd B/ - $ ./configure --prefix=/path/to/MYVIEW \ - --with-A=/path/to/MYVIEW - $ make && make install - -The result is a hybrid view: - -.. code-block:: console - - /path/to/MYVIEW/bin/programA -> /path/to/spack/.../bin/programA - /path/to/MYVIEW/bin/programB - /path/to/MYVIEW/lib/libA.so -> /path/to/spack/.../lib/libA.so - /path/to/MYVIEW/lib/libB.so - -In this case, real files coexist, interleaved with the "view" -symlinks. At any time one can delete ``/path/to/MYVIEW`` or use -``spack view`` to manage it surgically. None of this will affect the -real Spack install area. - -^^^^^^^^^^^^^^^^^^ -Global Activations -^^^^^^^^^^^^^^^^^^ - -:ref:`cmd-spack-activate` may be used as an alternative to loading -Python (and similar systems) packages directly or creating a view. -If extensions are globally activated, then ``spack load python`` will -also load all the extensions activated for the given ``python``. -This reduces the need for users to load a large number of packages. - -However, Spack global activations have two potential drawbacks: - -#. Activated packages that involve compiled C extensions may still - need their dependencies to be loaded manually. For example, - ``spack load openblas`` might be required to make ``py-numpy`` - work. - -#. Global activations "break" a core feature of Spack, which is that - multiple versions of a package can co-exist side-by-side. For example, - suppose you wish to run a Python package in two different - environments but the same basic Python --- one with - ``py-numpy@1.7`` and one with ``py-numpy@1.8``. Spack extensions - will not support this potential debugging use case. - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Discussion: Running Binaries -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Modules, extension packages and filesystem views are all ways to -assemble sets of Spack packages into a useful environment. They are -all semantically similar, in that conflicting installed packages -cannot simultaneously be loaded, activated or included in a view. - -With all of these approaches, there is no guarantee that the -environment created will be consistent. It is possible, for example, -to simultaneously load application A that uses OpenMPI and application -B that uses MPICH. Both applications will run just fine in this -inconsistent environment because they rely on RPATHs, not the -environment, to find their dependencies. - -In general, environments set up using modules vs. views will work -similarly. Both can be used to set up ephemeral or long-lived -testing/development environments. Operational differences between the -two approaches can make one or the other preferable in certain -environments: - -* Filesystem views do not require environment module infrastructure. - Although Spack can install ``environment-modules``, users might be - hostile to its use. Filesystem views offer a good solution for - sysadmins serving users who just "want all the stuff I need in one - place" and don't want to hear about Spack. - -* Although modern build systems will find dependencies wherever they - might be, some applications with hand-built make files expect their - dependencies to be in one place. One common problem is makefiles - that assume that ``netcdf`` and ``netcdf-fortran`` are installed in - the same tree. Or, one might use an IDE that requires tedious - configuration of dependency paths; and it's easier to automate that - administration in a view-building script than in the IDE itself. - For all these cases, a view will be preferable to other ways to - assemble an environment. - -* On systems with I-node quotas, modules might be preferable to views - and extension packages. - -* Views and activated extensions maintain state that is semantically - equivalent to the information in a ``spack module tcl loads`` script. - Administrators might find things easier to maintain without the - added "heavyweight" state of a view. - -------------------------------------- -Using Spack to Replace Homebrew/Conda -------------------------------------- - -Spack is an incredibly powerful package manager, designed for supercomputers -where users have diverse installation needs. But Spack can also be used to -handle simple single-user installations on your laptop. Most macOS users are -already familiar with package managers like Homebrew and Conda, where all -installed packages are symlinked to a single central location like ``/usr/local``. -In this section, we will show you how to emulate the behavior of Homebrew/Conda -using :ref:`environments`! - -^^^^^ -Setup -^^^^^ - -First, let's create a new environment. We'll assume that Spack is already set up -correctly, and that you've already sourced the setup script for your shell. -To create a new environment, simply run: - -.. code-block:: console - - $ spack env create myenv - ==> Updating view at /Users/me/spack/var/spack/environments/myenv/.spack-env/view - ==> Created environment 'myenv' in /Users/me/spack/var/spack/environments/myenv - $ spack env activate myenv - -Here, *myenv* can be anything you want to name your environment. Next, we can add -a list of packages we would like to install into our environment. Let's say we -want a newer version of Bash than the one that comes with macOS, and we want a -few Python libraries. We can run: - -.. code-block:: console - - $ spack add bash - ==> Adding bash to environment myenv - ==> Updating view at /Users/me/spack/var/spack/environments/myenv/.spack-env/view - $ spack add python@3: - ==> Adding python@3: to environment myenv - ==> Updating view at /Users/me/spack/var/spack/environments/myenv/.spack-env/view - $ spack add py-numpy py-scipy py-matplotlib - ==> Adding py-numpy to environment myenv - ==> Adding py-scipy to environment myenv - ==> Adding py-matplotlib to environment myenv - ==> Updating view at /Users/me/spack/var/spack/environments/myenv/.spack-env/view - -Each package can be listed on a separate line, or combined into a single line. -Notice that we're explicitly asking for Python 3 here. You can use any spec -you would normally use on the command line with other Spack commands. - -Next, we want to manually configure a couple of things. In the ``myenv`` -directory, we can find the ``spack.yaml`` that actually defines our environment. - -.. code-block:: console - - $ vim ~/spack/var/spack/environments/myenv/spack.yaml - -.. code-block:: yaml - - # This is a Spack Environment file. - # - # It describes a set of packages to be installed, along with - # configuration settings. - spack: - # add package specs to the `specs` list - specs: [bash, 'python@3:', py-numpy, py-scipy, py-matplotlib] - view: - default: - root: /Users/me/spack/var/spack/environments/myenv/.spack-env/view - projections: {} - config: {} - mirrors: {} - modules: - enable: [] - packages: {} - repos: [] - upstreams: {} - definitions: [] - concretization: separately - -You can see the packages we added earlier in the ``specs:`` section. If you -ever want to add more packages, you can either use ``spack add`` or manually -edit this file. - -We also need to change the ``concretization:`` option. By default, Spack -concretizes each spec *separately*, allowing multiple versions of the same -package to coexist. Since we want a single consistent environment, we want to -concretize all of the specs *together*. - -Here is what your ``spack.yaml`` looks like with these new settings, and with -some of the sections we don't plan on using removed: - -.. code-block:: diff - - spack: - - specs: [bash, 'python@3:', py-numpy, py-scipy, py-matplotlib] - + specs: - + - bash - + - 'python@3:' - + - py-numpy - + - py-scipy - + - py-matplotlib - - view: - - default: - - root: /Users/me/spack/var/spack/environments/myenv/.spack-env/view - - projections: {} - + view: /Users/me/spack/var/spack/environments/myenv/.spack-env/view - - config: {} - - mirrors: {} - - modules: - - enable: [] - - packages: {} - - repos: [] - - upstreams: {} - - definitions: [] - + concretization: together - - concretization: separately - -"""""""""""""""" -Symlink location -"""""""""""""""" - -In the ``spack.yaml`` file above, you'll notice that by default, Spack symlinks -all installations to ``/Users/me/spack/var/spack/environments/myenv/.spack-env/view``. -You can actually change this to any directory you want. For example, Homebrew -uses ``/usr/local``, while Conda uses ``/Users/me/anaconda``. In order to access -files in these locations, you need to update ``PATH`` and other environment variables -to point to them. Activating the Spack environment does this automatically, but -you can also manually set them in your ``.bashrc``. - -.. warning:: - - There are several reasons why you shouldn't use ``/usr/local``: - - 1. If you are on macOS 10.11+ (El Capitan and newer), Apple makes it hard - for you. You may notice permissions issues on ``/usr/local`` due to their - `System Integrity Protection `_. - By default, users don't have permissions to install anything in ``/usr/local``, - and you can't even change this using ``sudo chown`` or ``sudo chmod``. - 2. Other package managers like Homebrew will try to install things to the - same directory. If you plan on using Homebrew in conjunction with Spack, - don't symlink things to ``/usr/local``. - 3. If you are on a shared workstation, or don't have sudo privileges, you - can't do this. - - If you still want to do this anyway, there are several ways around SIP. - You could disable SIP by booting into recovery mode and running - ``csrutil disable``, but this is not recommended, as it can open up your OS - to security vulnerabilities. Another technique is to run ``spack concretize`` - and ``spack install`` using ``sudo``. This is also not recommended. - - The safest way I've found is to create your installation directories using - sudo, then change ownership back to the user like so: - - .. code-block:: bash - - for directory in .spack bin contrib include lib man share - do - sudo mkdir -p /usr/local/$directory - sudo chown $(id -un):$(id -gn) /usr/local/$directory - done - - Depending on the packages you install in your environment, the exact list of - directories you need to create may vary. You may also find some packages - like Java libraries that install a single file to the installation prefix - instead of in a subdirectory. In this case, the action is the same, just replace - ``mkdir -p`` with ``touch`` in the for-loop above. - - But again, it's safer just to use the default symlink location. - - -^^^^^^^^^^^^ -Installation -^^^^^^^^^^^^ - -To actually concretize the environment, run: - -.. code-block:: console - - $ spack concretize - -This will tell you which if any packages are already installed, and alert you -to any conflicting specs. - -To actually install these packages and symlink them to your ``view:`` -directory, simply run: - -.. code-block:: console - - $ spack install - -Now, when you type ``which python3``, it should find the one you just installed. - -In order to change the default shell to our newer Bash installation, we first -need to add it to this list of acceptable shells. Run: - -.. code-block:: console - - $ sudo vim /etc/shells - -and add the absolute path to your bash executable. Then run: - -.. code-block:: console - - $ chsh -s /path/to/bash - -Now, when you log out and log back in, ``echo $SHELL`` should point to the -newer version of Bash. - -^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Updating Installed Packages -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Let's say you upgraded to a new version of macOS, or a new version of Python -was released, and you want to rebuild your entire software stack. To do this, -simply run the following commands: - -.. code-block:: console - - $ spack env activate myenv - $ spack concretize --force - $ spack install - -The ``--force`` flag tells Spack to overwrite its previous concretization -decisions, allowing you to choose a new version of Python. If any of the new -packages like Bash are already installed, ``spack install`` won't re-install -them, it will keep the symlinks in place. - -^^^^^^^^^^^^^^ -Uninstallation -^^^^^^^^^^^^^^ - -If you decide that Spack isn't right for you, uninstallation is simple. -Just run: - -.. code-block:: console - - $ spack env activate myenv - $ spack uninstall --all - -This will uninstall all packages in your environment and remove the symlinks. - ------------------------- -Using Spack on Travis-CI ------------------------- - -Spack can be deployed as a provider for userland software in -`Travis-CI `_. - -A starting-point for a ``.travis.yml`` file can look as follows. -It uses `caching `_ for -already built environments, so make sure to clean the Travis cache if -you run into problems. - -The main points that are implemented below: - -#. Travis is detected as having up to 34 cores available, but only 2 - are actually allocated for the user. We limit the parallelism of - the spack builds in the config. - (The Travis yaml parser is a bit buggy on the echo command.) - -#. Without control for the user, Travis jobs will run on various - ``x86_64`` microarchitectures. If you plan to cache build results, - e.g. to accelerate dependency builds, consider building for the - generic ``x86_64`` target only. - Limiting the microarchitecture will also find more packages when - working with the - `E4S Spack build cache `_. - -#. Builds over 10 minutes need to be prefixed with ``travis_wait``. - Alternatively, generate output once with ``spack install -v``. - -#. Travis builds are non-interactive. This prevents using bash - aliases and functions for modules. We fix that by sourcing - ``/etc/profile`` first (or running everything in a subshell with - ``bash -l -c '...'``). - -.. code-block:: yaml - - language: cpp - sudo: false - dist: trusty - - cache: - apt: true - directories: - - $HOME/.cache - - addons: - apt: - sources: - - ubuntu-toolchain-r-test - packages: - - g++-4.9 - - environment-modules - - env: - global: - - SPACK_ROOT: $HOME/.cache/spack - - PATH: $PATH:$HOME/.cache/spack/bin - - before_install: - - export CXX=g++-4.9 - - export CC=gcc-4.9 - - export FC=gfortran-4.9 - - export CXXFLAGS="-std=c++11" - - install: - - | - if ! which spack >/dev/null; then - mkdir -p $SPACK_ROOT && - git clone --depth 50 https://github.com/spack/spack.git $SPACK_ROOT && - printf "config:\n build_jobs: 2\n" > $SPACK_ROOT/etc/spack/config.yaml && - printf "packages:\n all:\n target: ['x86_64']\n" \ - > $SPACK_ROOT/etc/spack/packages.yaml; - fi - - travis_wait spack install cmake@3.7.2~openssl~ncurses - - travis_wait spack install boost@1.62.0~graph~iostream~locale~log~wave - - spack clean -a - - source /etc/profile && - source $SPACK_ROOT/share/spack/setup-env.sh - - spack load cmake - - spack load boost - - script: - - mkdir -p $HOME/build - - cd $HOME/build - - cmake $TRAVIS_BUILD_DIR - - make -j 2 - - make test - ------------------- -Upstream Bug Fixes ------------------- - -It is not uncommon to discover a bug in an upstream project while -trying to build with Spack. Typically, the bug is in a package that -serves a dependency to something else. This section describes -procedure to work around and ultimately resolve these bugs, while not -delaying the Spack user's main goal. - -^^^^^^^^^^^^^^^^^ -Buggy New Version -^^^^^^^^^^^^^^^^^ - -Sometimes, the old version of a package works fine, but a new version -is buggy. For example, it was once found that `Adios did not build -with hdf5@1.10 `_. If the -old version of ``hdf5`` will work with ``adios``, the suggested -procedure is: - -#. Revert ``adios`` to the old version of ``hdf5``. Put in its - ``adios/package.py``: - - .. code-block:: python - - # Adios does not build with HDF5 1.10 - # See: https://github.com/spack/spack/issues/1683 - depends_on('hdf5@:1.9') - -#. Determine whether the problem is with ``hdf5`` or ``adios``, and - report the problem to the appropriate upstream project. In this - case, the problem was with ``adios``. - -#. Once a new version of ``adios`` comes out with the bugfix, modify - ``adios/package.py`` to reflect it: - - .. code-block:: python - - # Adios up to v1.10.0 does not build with HDF5 1.10 - # See: https://github.com/spack/spack/issues/1683 - depends_on('hdf5@:1.9', when='@:1.10.0') - depends_on('hdf5', when='@1.10.1:') - -^^^^^^^^^^^^^^^^ -No Version Works -^^^^^^^^^^^^^^^^ - -Sometimes, *no* existing versions of a dependency work for a build. -This typically happens when developing a new project: only then does -the developer notice that existing versions of a dependency are all -buggy, or the non-buggy versions are all missing a critical feature. - -In the long run, the upstream project will hopefully fix the bug and -release a new version. But that could take a while, even if a bugfix -has already been pushed to the project's repository. In the meantime, -the Spack user needs things to work. - -The solution is to create an unofficial Spack release of the project, -as soon as the bug is fixed in *some* repository. A study of the `Git -history `_ -of ``py-proj/package.py`` is instructive here: - -#. On `April 1 `_, an initial bugfix was identified for the PyProj project - and a pull request submitted to PyProj. Because the upstream - authors had not yet fixed the bug, the ``py-proj`` Spack package - downloads from a forked repository, set up by the package's author. - A non-numeric version number is used to make it easy to upgrade the - package without recomputing checksums; however, this is an - untrusted download method and should not be distributed. The - package author has now become, temporarily, a maintainer of the - upstream project: - - .. code-block:: python - - # We need the benefits of this PR - # https://github.com/jswhit/pyproj/pull/54 - version('citibeth-latlong2', - git='https://github.com/citibeth/pyproj.git', - branch='latlong2') - - -#. By May 14, the upstream project had accepted a pull request with - the required bugfix. At this point, the forked repository was - deleted. However, the upstream project still had not released a - new version with a bugfix. Therefore, a Spack-only release was - created by specifying the desired hash in the main project - repository. The version number ``@1.9.5.1.1`` was chosen for this - "release" because it's a descendent of the officially released - version ``@1.9.5.1``. This is a trusted download method, and can - be released to the Spack community: - - .. code-block:: python - - # This is not a tagged release of pyproj. - # The changes in this "version" fix some bugs, especially with Python3 use. - version('1.9.5.1.1', 'd035e4bc704d136db79b43ab371b27d2', - url='https://www.github.com/jswhit/pyproj/tarball/0be612cc9f972e38b50a90c946a9b353e2ab140f') - - .. note:: - - It would have been simpler to use Spack's Git download method, - which is also a trusted download in this case: - - .. code-block:: python - - # This is not a tagged release of pyproj. - # The changes in this "version" fix some bugs, especially with Python3 use. - version('1.9.5.1.1', - git='https://github.com/jswhit/pyproj.git', - commit='0be612cc9f972e38b50a90c946a9b353e2ab140f') - - .. note:: - - In this case, the upstream project fixed the bug in its - repository in a relatively timely manner. If that had not been - the case, the numbered version in this step could have been - released from the forked repository. - - -#. The author of the Spack package has now become an unofficial - release engineer for the upstream project. Depending on the - situation, it may be advisable to put ``preferred=True`` on the - latest *officially released* version. - -#. As of August 31, the upstream project still had not made a new - release with the bugfix. In the meantime, Spack-built ``py-proj`` - provides the bugfix needed by packages depending on it. As long as - this works, there is no particular need for the upstream project to - make a new official release. - -#. If the upstream project releases a new official version with the - bugfix, then the unofficial ``version()`` line should be removed - from the Spack package. - -^^^^^^^ -Patches -^^^^^^^ - -Spack's source patching mechanism provides another way to fix bugs in -upstream projects. This has advantages and disadvantages compared to the procedures above. - -Advantages: - - 1. It can fix bugs in existing released versions, and (probably) - future releases as well. - - 2. It is lightweight, does not require a new fork to be set up. - -Disadvantages: - - 1. It is harder to develop and debug a patch, vs. a branch in a - repository. The user loses the automation provided by version - control systems. - - 2. Although patches of a few lines work OK, large patch files can be - hard to create and maintain. -- cgit v1.2.3-60-g2f50