diff options
author | Elizabeth Fischer <elizabeth.fischer@columbia.edu> | 2016-08-29 23:06:38 -0400 |
---|---|---|
committer | Elizabeth Fischer <elizabeth.fischer@columbia.edu> | 2016-08-29 23:06:38 -0400 |
commit | 5450f9b91736b4e4b16eaaae920f3a9f1bcf28c7 (patch) | |
tree | 0de74340f61f6142fb703ed04e765f9e728758ef | |
parent | 72a3d35d0c415fc33cb07399e51a0c469a53f0a5 (diff) | |
download | spack-5450f9b91736b4e4b16eaaae920f3a9f1bcf28c7.tar.gz spack-5450f9b91736b4e4b16eaaae920f3a9f1bcf28c7.tar.bz2 spack-5450f9b91736b4e4b16eaaae920f3a9f1bcf28c7.tar.xz spack-5450f9b91736b4e4b16eaaae920f3a9f1bcf28c7.zip |
1. Rewrite of "Getting Started": everything you need to set up Spack, even on old/ornery systems. This is not a reference manual section; items covered here are covered more systematically elsewhere in the manual. Some sections were moved here from elsewhere.
2. Beginning to write three methods of application developer support. Two methods were moved from elsewhere.
-rw-r--r-- | lib/spack/docs/application_developer_suppport.rst | 229 | ||||
-rw-r--r-- | lib/spack/docs/basic_usage.rst | 483 | ||||
-rw-r--r-- | lib/spack/docs/getting_started.rst | 683 | ||||
-rw-r--r-- | lib/spack/docs/index.rst | 1 | ||||
-rw-r--r-- | lib/spack/docs/packaging_guide.rst | 105 |
5 files changed, 942 insertions, 559 deletions
diff --git a/lib/spack/docs/application_developer_suppport.rst b/lib/spack/docs/application_developer_suppport.rst new file mode 100644 index 0000000000..113bdf2ecb --- /dev/null +++ b/lib/spack/docs/application_developer_suppport.rst @@ -0,0 +1,229 @@ +Application Developer Support +=============================== + +Three ways to use your Spack packages. + +Setting up a User Environment +-------------------------------- + +Show how to build a script with a bunch of ``module loads`` commands. + + +Filesystem Views +------------------------------- + +.. Maybe this is not the right location for this documentation. + +The Spack installation area allows for many package installation trees +to coexist and gives the user choices as to what versions and variants +of packages to use. To use them, the user must rely on a way to +aggregate a subset of those packages. The section on Environment +Modules gives one good way to do that which relies on setting various +environment variables. An alternative way to aggregate is through +**filesystem views**. + +A filesystem view is a single directory tree which is the union of the +directory hierarchies of the individual package installation trees +that have been included. The files of the view's installed packages +are brought into the view by symbolic or hard links back to their +location in the original Spack installation area. As the view is +formed, any clashes due to a file having the exact same path in its +package installation tree are handled in a first-come-first-served +basis and a warning is printed. Packages and their dependencies can +be both added and removed. During removal, empty directories will be +purged. These operations can be limited to pertain to just the +packages listed by the user or to exclude specific dependencies and +they allow for software installed outside of Spack to coexist inside +the filesystem view tree. + +By its nature, a filesystem view represents a particular choice of one +set of packages among all the versions and variants that are available +in the Spack installation area. It is thus equivalent to the +directory hiearchy that might exist under ``/usr/local``. While this +limits a view to including only one version/variant of any package, it +provides the benefits of having a simpler and traditional layout which +may be used without any particular knowledge that its packages were +built by Spack. + +Views can be used for a variety of purposes including: + +- A central installation in a traditional layout, eg ``/usr/local`` maintained over time by the sysadmin. +- A self-contained installation area which may for the basis of a top-level atomic versioning scheme, eg ``/opt/pro`` vs ``/opt/dev``. +- Providing an atomic and monolithic binary distribution, eg for delivery as a single tarball. +- Producing ephemeral testing or developing environments. + +Using Filesystem Views +~~~~~~~~~~~~~~~~~~~~~~ + +A filesystem view is created and packages are linked in by the ``spack +view`` command's ``symlink`` and ``hardlink`` sub-commands. The +``spack view remove`` command can be used to unlink some or all of the +filesystem view. + +The following example creates a filesystem view based +on an installed ``cmake`` package and then removes from the view the +files in the ``cmake`` package while retaining its dependencies. + +.. code-block:: sh + + + $ spack view -v symlink myview cmake@3.5.2 + ==> Linking package: "ncurses" + ==> Linking package: "zlib" + ==> Linking package: "openssl" + ==> Linking package: "cmake" + + $ ls myview/ + bin doc etc include lib share + + $ ls myview/bin/ + captoinfo clear cpack ctest infotocap openssl tabs toe tset + ccmake cmake c_rehash infocmp ncurses6-config reset tic tput + + $ spack view -v -d false rm myview cmake@3.5.2 + ==> Removing package: "cmake" + + $ ls myview/bin/ + captoinfo c_rehash infotocap openssl tabs toe tset + clear infocmp ncurses6-config reset tic tput + + +Limitations of Filesystem Views +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This section describes some limitations that should be considered in +using filesystems views. + +Filesystem views are merely organizational. The binary executable +programs, shared libraries and other build products found in a view +are mere links into the "real" Spack installation area. If a view is +built with symbolic links it requires the Spack-installed package to +be kept in place. Building a view with hardlinks removes this +requirement but any internal paths (eg, rpath or ``#!`` interpreter +specifications) will still require the Spack-installed package files +to be in place. + +.. FIXME: reference the relocation work of Hegner and Gartung. + +As described above, when a view is built only a single instance of a +file may exist in the unified filesystem tree. If more than one +package provides a file at the same path (relative to its own root) +then it is the first package added to the view that "wins". A warning +is printed and it is up to the user to determine if the conflict +matters. + +It is up to the user to assure a consistent view is produced. In +particular if the user excludes packages, limits the following of +dependencies or removes packages the view may become inconsistent. In +particular, if two packages require the same sub-tree of dependencies, +removing one package (recursively) will remove its dependencies and +leave the other package broken. + + + +Build System Configuration Support +---------------------------------- + +Imagine a developer creating a CMake-based (or Autotools) project in a local +directory, which depends on libraries A-Z. Once Spack has installed +those dependencies, one would like to run ``cmake`` with appropriate +command line and environment so CMake can find them. The ``spack +setup`` command does this conveniently, producing a CMake +configuration that is essentially the same as how Spack *would have* +configured the project. This can be demonstrated with a usage +example: + +.. code-block:: bash + + cd myproject + spack setup myproject@local + mkdir build; cd build + ../spconfig.py .. + make + make install + +Notes: + * Spack must have ``myproject/package.py`` in its repository for + this to work. + * ``spack setup`` produces the executable script ``spconfig.py`` in + the local directory, and also creates the module file for the + package. ``spconfig.py`` is normally run from the user's + out-of-source build directory. + * The version number given to ``spack setup`` is arbitrary, just + like ``spack diy``. ``myproject/package.py`` does not need to + have any valid downloadable versions listed (typical when a + project is new). + * spconfig.py produces a CMake configuration that *does not* use the + Spack wrappers. Any resulting binaries *will not* use RPATH, + unless the user has enabled it. This is recommended for + development purposes, not production. + * ``spconfig.py`` is human readable, and can serve as a developer + reference of what dependencies are being used. + * ``make install`` installs the package into the Spack repository, + where it may be used by other Spack packages. + * CMake-generated makefiles re-run CMake in some circumstances. Use + of ``spconfig.py`` breaks this behavior, requiring the developer + to manually re-run ``spconfig.py`` when a ``CMakeLists.txt`` file + has changed. + +CMakePackage +~~~~~~~~~~~~ + +In order ot enable ``spack setup`` functionality, the author of +``myproject/package.py`` must subclass from ``CMakePackage`` instead +of the standard ``Package`` superclass. Because CMake is +standardized, the packager does not need to tell Spack how to run +``cmake; make; make install``. Instead the packager only needs to +create (optional) methods ``configure_args()`` and ``configure_env()``, which +provide the arguments (as a list) and extra environment variables (as +a dict) to provide to the ``cmake`` command. Usually, these will +translate variant flags into CMake definitions. For example: + +.. code-block:: python + + def configure_args(self): + spec = self.spec + return [ + '-DUSE_EVERYTRACE=%s' % ('YES' if '+everytrace' in spec else 'NO'), + '-DBUILD_PYTHON=%s' % ('YES' if '+python' in spec else 'NO'), + '-DBUILD_GRIDGEN=%s' % ('YES' if '+gridgen' in spec else 'NO'), + '-DBUILD_COUPLER=%s' % ('YES' if '+coupler' in spec else 'NO'), + '-DUSE_PISM=%s' % ('YES' if '+pism' in spec else 'NO')] + +If needed, a packager may also override methods defined in +``StagedPackage`` (see below). + + +StagedPackage +~~~~~~~~~~~~~ + +``CMakePackage`` is implemented by subclassing the ``StagedPackage`` +superclass, which breaks down the standard ``Package.install()`` +method into several sub-stages: ``setup``, ``configure``, ``build`` +and ``install``. Details: + +* Instead of implementing the standard ``install()`` method, package + authors implement the methods for the sub-stages + ``install_setup()``, ``install_configure()``, + ``install_build()``, and ``install_install()``. + +* The ``spack install`` command runs the sub-stages ``configure``, + ``build`` and ``install`` in order. (The ``setup`` stage is + not run by default; see below). +* The ``spack setup`` command runs the sub-stages ``setup`` + and a dummy install (to create the module file). +* The sub-stage install methods take no arguments (other than + ``self``). The arguments ``spec`` and ``prefix`` to the standard + ``install()`` method may be accessed via ``self.spec`` and + ``self.prefix``. + +GNU Autotools +~~~~~~~~~~~~~ + +The ``setup`` functionality is currently only available for +CMake-based packages. Extending this functionality to GNU +Autotools-based packages would be easy (and should be done by a +developer who actively uses Autotools). Packages that use +non-standard build systems can gain ``setup`` functionality by +subclassing ``StagedPackage`` directly. + diff --git a/lib/spack/docs/basic_usage.rst b/lib/spack/docs/basic_usage.rst index 42117378a1..a21a238ffa 100644 --- a/lib/spack/docs/basic_usage.rst +++ b/lib/spack/docs/basic_usage.rst @@ -368,154 +368,6 @@ will find every installed package with a 'debug' compile-time option enabled. The full spec syntax is discussed in detail in :ref:`sec-specs`. -.. _compiler-config: - -Compiler configuration ------------------------------------ - -Spack has the ability to build packages with multiple compilers and -compiler versions. Spack searches for compilers on your machine -automatically the first time it is run. It does this by inspecting -your path. - -.. _spack-compilers: - -``spack compilers`` -~~~~~~~~~~~~~~~~~~~~~~~ - -You can see which compilers spack has found by running ``spack -compilers`` or ``spack compiler list``:: - - $ spack compilers - ==> Available compilers - -- gcc --------------------------------------------------------- - gcc@4.9.0 gcc@4.8.0 gcc@4.7.0 gcc@4.6.2 gcc@4.4.7 - gcc@4.8.2 gcc@4.7.1 gcc@4.6.3 gcc@4.6.1 gcc@4.1.2 - -- intel ------------------------------------------------------- - intel@15.0.0 intel@14.0.0 intel@13.0.0 intel@12.1.0 intel@10.0 - intel@14.0.3 intel@13.1.1 intel@12.1.5 intel@12.0.4 intel@9.1 - intel@14.0.2 intel@13.1.0 intel@12.1.3 intel@11.1 - intel@14.0.1 intel@13.0.1 intel@12.1.2 intel@10.1 - -- clang ------------------------------------------------------- - clang@3.4 clang@3.3 clang@3.2 clang@3.1 - -- pgi --------------------------------------------------------- - pgi@14.3-0 pgi@13.2-0 pgi@12.1-0 pgi@10.9-0 pgi@8.0-1 - pgi@13.10-0 pgi@13.1-1 pgi@11.10-0 pgi@10.2-0 pgi@7.1-3 - pgi@13.6-0 pgi@12.8-0 pgi@11.1-0 pgi@9.0-4 pgi@7.0-6 - -Any of these compilers can be used to build Spack packages. More on -how this is done is in :ref:`sec-specs`. - -.. _spack-compiler-add: - -``spack compiler add`` -~~~~~~~~~~~~~~~~~~~~~~~ - -An alias for ``spack compiler find``. - -.. _spack-compiler-find: - -``spack compiler find`` -~~~~~~~~~~~~~~~~~~~~~~~ - -If you do not see a compiler in this list, but you want to use it with -Spack, you can simply run ``spack compiler find`` with the path to -where the compiler is installed. For example:: - - $ spack compiler find /usr/local/tools/ic-13.0.079 - ==> Added 1 new compiler to /Users/gamblin2/.spack/compilers.yaml - intel@13.0.079 - -Or you can run ``spack compiler find`` with no arguments to force -auto-detection. This is useful if you do not know where compilers are -installed, but you know that new compilers have been added to your -``PATH``. For example, using dotkit, you might do this:: - - $ module load gcc-4.9.0 - $ spack compiler find - ==> Added 1 new compiler to /Users/gamblin2/.spack/compilers.yaml - gcc@4.9.0 - -This loads the environment module for gcc-4.9.0 to add it to -``PATH``, and then it adds the compiler to Spack. - -.. _spack-compiler-info: - -``spack compiler info`` -~~~~~~~~~~~~~~~~~~~~~~~ - -If you want to see specifics on a particular compiler, you can run -``spack compiler info`` on it:: - - $ spack compiler info intel@15 - intel@15.0.0: - cc = /usr/local/bin/icc-15.0.090 - cxx = /usr/local/bin/icpc-15.0.090 - f77 = /usr/local/bin/ifort-15.0.090 - fc = /usr/local/bin/ifort-15.0.090 - -This shows which C, C++, and Fortran compilers were detected by Spack. -Notice also that we didn\'t have to be too specific about the -version. We just said ``intel@15``, and information about the only -matching Intel compiler was displayed. - - -Manual compiler configuration -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -If auto-detection fails, you can manually configure a compiler by -editing your ``~/.spack/compilers.yaml`` file. You can do this by running -``spack config edit compilers``, which will open the file in your ``$EDITOR``. - -Each compiler configuration in the file looks like this:: - - ... - compilers: - - compiler: - modules = [] - operating_system: OS - paths: - cc: /usr/local/bin/icc-15.0.024-beta - cxx: /usr/local/bin/icpc-15.0.024-beta - f77: /usr/local/bin/ifort-15.0.024-beta - fc: /usr/local/bin/ifort-15.0.024-beta - - spec: intel@15.0.0: - -For compilers, like ``clang``, that do not support Fortran, put -``None`` for ``f77`` and ``fc``:: - - clang@3.3svn: - cc: /usr/bin/clang - cxx: /usr/bin/clang++ - f77: None - fc: None - -Once you save the file, the configured compilers will show up in the -list displayed by ``spack compilers``. - -You can also add compiler flags to manually configured compilers. The -valid flags are ``cflags``, ``cxxflags``, ``fflags``, ``cppflags``, -``ldflags``, and ``ldlibs``. For example:: - - ... - compilers: - - compiler: - ... - intel@15.0.0: - cc: /usr/local/bin/icc-15.0.024-beta - cxx: /usr/local/bin/icpc-15.0.024-beta - f77: /usr/local/bin/ifort-15.0.024-beta - fc: /usr/local/bin/ifort-15.0.024-beta - cppflags: -O3 -fPIC - ... - -These flags will be treated by spack as if they were enterred from -the command line each time this compiler is used. The compiler wrappers -then inject those flags into the compiler command. Compiler flags -enterred from the command line will be discussed in more detail in the -following section. - .. _sec-specs: Specs & dependencies @@ -905,47 +757,18 @@ Integration with module systems interface and/or generated module names may change in future versions. -Spack provides some integration with -`Environment Modules <http://modules.sourceforge.net/>`_ -and `Dotkit <https://computing.llnl.gov/?set=jobs&page=dotkit>`_ to make -it easier to use the packages it installed. - - +Spack provides some integration with `Environment Modules +<http://modules.sourceforge.net/>`_ to make it easier to use the +packages it installs. If your system does not already have +Environment Modules, see InstallEnvironmentModules_. -Installing Environment Modules -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -In order to use Spack's generated environment modules, you must have -installed the *Environment Modules* package. On many Linux -distributions, this can be installed from the vendor's repository: - -.. code-block:: sh - - yum install environment-modules # (Fedora/RHEL/CentOS) - apt-get install environment-modules # (Ubuntu/Debian) - -If your Linux distribution does not have -Environment Modules, you can get it with Spack: - -.. code-block:: sh - - spack install environment-modules - - -In this case to activate it automatically you need to add the following two -lines to your ``.bashrc`` profile (or similar): - -.. code-block:: sh - - MODULES_HOME=`spack location -i environment-modules` - source ${MODULES_HOME}/Modules/init/bash - -If you use a Unix shell other than ``bash``, modify the commands above -accordingly and source the appropriate file in -``${MODULES_HOME}/Modules/init/``. +.. note:: + Spack also supports `Dotkit + <https://computing.llnl.gov/?set=jobs&page=dotkit>`_, which is used + by some systems. If you system does not already have a module + system installed, you should use Environment Modules or LMod. -.. TODO : Add a similar section on how to install dotkit ? Spack and module systems ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -1170,13 +993,13 @@ Module Commands for Shell Scripts Although Spack is flexible, the ``module`` command is much faster. This could become an issue when emitting a series of ``spack load`` -commands inside a shell script. By adding the ``--shell`` flag, -``spack module find`` may also be used to generate code that can be -cut-and-pasted into a shell script. For example: +commands inside a shell script. The ``spack module loads`` may also +be used to generate code that can be cut-and-pasted into a shell +script. For example: .. code-block:: sh - $ spack module find tcl --dependencies --shell py-numpy git + $ spack module loads --dependencies py-numpy git # bzip2@1.0.6%gcc@4.9.3=linux-x86_64 module load bzip2-1.0.6-gcc-4.9.3-ktnrhkrmbbtlvnagfatrarzjojmkvzsx # ncurses@6.0%gcc@4.9.3=linux-x86_64 @@ -1213,7 +1036,7 @@ This script may be directly executed in bash via .. code-block :: sh - source <( spack module find tcl --dependencies --shell py-numpy git ) + source <( spack module loads --dependencies py-numpy git ) Regenerating Module files @@ -1276,7 +1099,7 @@ cut-and-pasted into a shell script. For example: .. code-block:: sh - $ spack module find tcl --dependencies --shell py-numpy git + $ spack module loads --dependencies py-numpy git # bzip2@1.0.6%gcc@4.9.3=linux-x86_64 module load bzip2-1.0.6-gcc-4.9.3-ktnrhkrmbbtlvnagfatrarzjojmkvzsx # ncurses@6.0%gcc@4.9.3=linux-x86_64 @@ -1313,14 +1136,40 @@ This script may be directly executed in bash via .. code-block :: sh - source <( spack module find tcl --dependencies --shell py-numpy git ) + source <( spack module loads --dependencies py-numpy git ) + + +Module Prefixes +```````````````` + +On some systems, modules are automatically prefixed with a certain +string; ``spack module loads`` needs to know about that prefix when it +issues ``module load`` commands. Add the ``--prefix`` option to your +``spack module loads`` commands if this is necessary. +For example, consider the following on one system: + +..code-block:: sh + + $ module avail + linux-SuSE11-x86_64/antlr-2.7.7-gcc-5.3.0-bdpl46y + + $ spack module loads antlr # WRONG! + # antlr@2.7.7%gcc@5.3.0~csharp+cxx~java~python arch=linux-SuSE11-x86_64 + module load antlr-2.7.7-gcc-5.3.0-bdpl46y + + $ spack module loads --prefix linux-SuSE11-x86_64/ antlr + # antlr@2.7.7%gcc@5.3.0~csharp+cxx~java~python arch=linux-SuSE11-x86_64 + module load linux-SuSE11-x86_64/antlr-2.7.7-gcc-5.3.0-bdpl46y Regenerating Module files ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Module and dotkit files are generated when packages are installed, and -are placed in the following directories under the Spack root: +are placed in the directory ``share/spack/modules`` under the Spack +root. The command ``spack refresh`` will regenerate them all without +re-building the packages; for example, if module format or options +have changed. Configuration files ^^^^^^^^^^^^^^^^^^^ @@ -1502,116 +1351,6 @@ regenerate all module and dotkit files from scratch: .. _extensions: -Filesystem Views -------------------------------- - -.. Maybe this is not the right location for this documentation. - -The Spack installation area allows for many package installation trees -to coexist and gives the user choices as to what versions and variants -of packages to use. To use them, the user must rely on a way to -aggregate a subset of those packages. The section on Environment -Modules gives one good way to do that which relies on setting various -environment variables. An alternative way to aggregate is through -**filesystem views**. - -A filesystem view is a single directory tree which is the union of the -directory hierarchies of the individual package installation trees -that have been included. The files of the view's installed packages -are brought into the view by symbolic or hard links back to their -location in the original Spack installation area. As the view is -formed, any clashes due to a file having the exact same path in its -package installation tree are handled in a first-come-first-served -basis and a warning is printed. Packages and their dependencies can -be both added and removed. During removal, empty directories will be -purged. These operations can be limited to pertain to just the -packages listed by the user or to exclude specific dependencies and -they allow for software installed outside of Spack to coexist inside -the filesystem view tree. - -By its nature, a filesystem view represents a particular choice of one -set of packages among all the versions and variants that are available -in the Spack installation area. It is thus equivalent to the -directory hiearchy that might exist under ``/usr/local``. While this -limits a view to including only one version/variant of any package, it -provides the benefits of having a simpler and traditional layout which -may be used without any particular knowledge that its packages were -built by Spack. - -Views can be used for a variety of purposes including: - -- A central installation in a traditional layout, eg ``/usr/local`` maintained over time by the sysadmin. -- A self-contained installation area which may for the basis of a top-level atomic versioning scheme, eg ``/opt/pro`` vs ``/opt/dev``. -- Providing an atomic and monolithic binary distribution, eg for delivery as a single tarball. -- Producing ephemeral testing or developing environments. - -Using Filesystem Views -~~~~~~~~~~~~~~~~~~~~~~ - -A filesystem view is created and packages are linked in by the ``spack -view`` command's ``symlink`` and ``hardlink`` sub-commands. The -``spack view remove`` command can be used to unlink some or all of the -filesystem view. - -The following example creates a filesystem view based -on an installed ``cmake`` package and then removes from the view the -files in the ``cmake`` package while retaining its dependencies. - -.. code-block:: sh - - - $ spack view -v symlink myview cmake@3.5.2 - ==> Linking package: "ncurses" - ==> Linking package: "zlib" - ==> Linking package: "openssl" - ==> Linking package: "cmake" - - $ ls myview/ - bin doc etc include lib share - - $ ls myview/bin/ - captoinfo clear cpack ctest infotocap openssl tabs toe tset - ccmake cmake c_rehash infocmp ncurses6-config reset tic tput - - $ spack view -v -d false rm myview cmake@3.5.2 - ==> Removing package: "cmake" - - $ ls myview/bin/ - captoinfo c_rehash infotocap openssl tabs toe tset - clear infocmp ncurses6-config reset tic tput - - -Limitations of Filesystem Views -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -This section describes some limitations that should be considered in -using filesystems views. - -Filesystem views are merely organizational. The binary executable -programs, shared libraries and other build products found in a view -are mere links into the "real" Spack installation area. If a view is -built with symbolic links it requires the Spack-installed package to -be kept in place. Building a view with hardlinks removes this -requirement but any internal paths (eg, rpath or ``#!`` interpreter -specifications) will still require the Spack-installed package files -to be in place. - -.. FIXME: reference the relocation work of Hegner and Gartung. - -As described above, when a view is built only a single instance of a -file may exist in the unified filesystem tree. If more than one -package provides a file at the same path (relative to its own root) -then it is the first package added to the view that "wins". A warning -is printed and it is up to the user to determine if the conflict -matters. - -It is up to the user to assure a consistent view is produced. In -particular if the user excludes packages, limits the following of -dependencies or removes packages the view may become inconsistent. In -particular, if two packages require the same sub-tree of dependencies, -removing one package (recursively) will remove its dependencies and -leave the other package broken. - @@ -1876,142 +1615,6 @@ A nicer error message is TBD in future versions of Spack. .. _spack-cray: -Spack on Cray ------------------------------ - -Spack differs slightly when used on a Cray system. The architecture spec -can differentiate between the front-end and back-end processor and operating system. -For example, on Edison at NERSC, the back-end target processor -is \"Ivy Bridge\", so you can specify to use the back-end this way: - -.. code-block:: sh - - spack install zlib target=ivybridge - -You can also use the operating system to build against the back-end: - -.. code-block:: sh - - spack install zlib os=CNL10 - -Notice that the name includes both the operating system name and the major -version number concatenated together. - -Alternatively, if you want to build something for the front-end, -you can specify the front-end target processor. The processor for a login node -on Edison is \"Sandy bridge\" so we specify on the command line like so: - -.. code-block:: sh - - spack install zlib target=sandybridge - -And the front-end operating system is: - -.. code-block:: sh - - spack install zlib os=SuSE11 - - - -Cray compiler detection -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Spack can detect compilers using two methods. For the front-end, we treat -everything the same. The difference lies in back-end compiler detection. -Back-end compiler detection is made via the Tcl module avail command. -Once it detects the compiler it writes the appropriate PrgEnv and compiler -module name to compilers.yaml and sets the paths to each compiler with Cray\'s -compiler wrapper names (i.e. cc, CC, ftn). During build time, Spack will load -the correct PrgEnv and compiler module and will call appropriate wrapper. - -The compilers.yaml config file will also differ. There is a -modules section that is filled with the compiler\'s Programming Environment -and module name. On other systems, this field is empty []:: - - ... - - compiler: - modules: - - PrgEnv-intel - - intel/15.0.109 - ... - -As mentioned earlier, the compiler paths will look different on a Cray system. -Since most compilers are invoked using cc, CC and ftn, the paths for each -compiler are replaced with their respective Cray compiler wrapper names:: - - ... - paths: - cc: cc - cxx: CC - f77: ftn - fc: ftn - ... - -As opposed to an explicit path to the compiler executable. This allows Spack -to call the Cray compiler wrappers during build time. - -For more on compiler configuration, check out :ref:`compiler-config`. - -Spack sets the default Cray link type to dynamic, to better match other -other platforms. Individual packages can enable static linking (which is the -default outside of Spack on cray systems) using the -static flag. - -Setting defaults and using Cray modules -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -If you want to use default compilers for each PrgEnv and also be able -to load cray external modules, you will need to set up a packages.yaml. - -Here\'s an example of an external configuration for cray modules: - -.. code-block:: yaml - - packages: - mpi: - modules: - mpich@7.3.1%gcc@5.2.0 arch=cray_xc-haswell-CNL10: cray-mpich - mpich@7.3.1%intel@16.0.0.109 arch=cray_xc-haswell-CNL10: cray-mpich - -This tells Spack that for whatever package that depends on mpi, load the -cray-mpich module into the environment. You can then be able to use whatever -environment variables, libraries, etc, that are brought into the environment -via module load. - -You can set the default compiler that Spack can use for each compiler type. -If you want to use the Cray defaults, then set them under *all:* in packages.yaml. -In the compiler field, set the compiler specs in your order of preference. -Whenever you build with that compiler type, Spack will concretize to that version. - -Here is an example of a full packages.yaml used at NERSC - -.. code-block:: yaml - - packages: - mpi: - modules: - mpich@7.3.1%gcc@5.2.0 arch=cray_xc-CNL10-ivybridge: cray-mpich - mpich@7.3.1%intel@16.0.0.109 arch=cray_xc-SuSE11-ivybridge: cray-mpich - buildable: False - netcdf: - modules: - netcdf@4.3.3.1%gcc@5.2.0 arch=cray_xc-CNL10-ivybridge: cray-netcdf - netcdf@4.3.3.1%intel@16.0.0.109 arch=cray_xc-CNL10-ivybridge: cray-netcdf - buildable: False - hdf5: - modules: - hdf5@1.8.14%gcc@5.2.0 arch=cray_xc-CNL10-ivybridge: cray-hdf5 - hdf5@1.8.14%intel@16.0.0.109 arch=cray_xc-CNL10-ivybridge: cray-hdf5 - buildable: False - all: - compiler: [gcc@5.2.0, intel@16.0.0.109] - -Here we tell spack that whenever we want to build with gcc use version 5.2.0 or -if we want to build with intel compilers, use version 16.0.0.109. We add a spec -for each compiler type for each cray modules. This ensures that for each -compiler on our system we can use that external module. - - -For more on external packages check out the section :ref:`sec-external_packages`. Getting Help ----------------------- diff --git a/lib/spack/docs/getting_started.rst b/lib/spack/docs/getting_started.rst index 1fcfe7aa46..441cba3a65 100644 --- a/lib/spack/docs/getting_started.rst +++ b/lib/spack/docs/getting_started.rst @@ -1,7 +1,21 @@ Getting Started ==================== -Download +Prerequisites +--------------- + +Spack has the following minimum requirements, which must be installed +before Spack is run: + +1. Operating System: GNU/Linux or Macintosh +2. Python 2.6 or 2.7 +3. A C/C++ compiler + +These requirements can be easily installed on most modern Linux +systems; on Macintosh, XCode is required. + + +Installation -------------------- Getting spack is easy. You can clone it from the `github repository @@ -11,10 +25,16 @@ Getting spack is easy. You can clone it from the `github repository $ git clone https://github.com/llnl/spack.git -This will create a directory called ``spack``. We'll assume that the -full path to this directory is in the ``SPACK_ROOT`` environment -variable. Add ``$SPACK_ROOT/bin`` to your path and you're ready to -go: +This will create a directory called ``spack``. If you are using Spack +for a specific purpose, you might have received different instructions +on how to download Spack; if so, please follow those instructions. + +Add Spack to Shell +~~~~~~~~~~~~~~~~~~~ + +We'll assume that the full path to your downloaded Spack directory is +in the ``SPACK_ROOT`` environment variable. Add ``$SPACK_ROOT/bin`` +to your path and you're ready to go: .. code-block:: sh @@ -50,11 +70,29 @@ included. This procedure will avoid many strange build errors that no one knows how to fix. -Installation --------------------- +Although Spack will work as soon as you clone it, it won't necessarily +be able to install any packages. That is because Spack relies + +Check Installation +~~~~~~~~~~~~~~~~~~~~~ -You don't need to install Spack; it's ready to run as soon as you -clone it from git. +With Spack installed, you should be able to run some basic Spack commands. For example: + +.. code-block:: sh + + $ spack spec netcdf + ... + netcdf@4.4.1%gcc@5.3.0~hdf4+mpi arch=linux-SuSE11-x86_64 + ^curl@7.50.1%gcc@5.3.0 arch=linux-SuSE11-x86_64 + ^openssl@system%gcc@5.3.0 arch=linux-SuSE11-x86_64 + ^zlib@1.2.8%gcc@5.3.0 arch=linux-SuSE11-x86_64 + ^hdf5@1.10.0-patch1%gcc@5.3.0+cxx~debug+fortran+mpi+shared~szip~threadsafe arch=linux-SuSE11-x86_64 + ^openmpi@1.10.1%gcc@5.3.0~mxm~pmi~psm~psm2~slurm~sqlite3~thread_multiple~tm+verbs+vt arch=linux-SuSE11-x86_64 + ^m4@1.4.17%gcc@5.3.0+sigsegv arch=linux-SuSE11-x86_64 + ^libsigsegv@2.10%gcc@5.3.0 arch=linux-SuSE11-x86_64 + +Optional: Alternate Prefix +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You may want to run it out of a prefix other than the git repository you cloned. The ``spack bootstrap`` command provides this @@ -69,10 +107,496 @@ which you can use just like you would the regular spack script. Each copy of spack installs packages into its own ``$PREFIX/opt`` directory. -Bootstrapping --------------- -Although Spack itself does not needinstallation, it is + + +.. _compiler-config: + +Compiler configuration +----------------------------------- + +Spack has the ability to build packages with multiple compilers and +compiler versions. Spack searches for compilers on your machine +automatically the first time it is run. It does this by inspecting +your path. + +.. _spack-compilers: + +``spack compilers`` +~~~~~~~~~~~~~~~~~~~~~~~ + +You can see which compilers spack has found by running ``spack +compilers`` or ``spack compiler list``:: + + $ spack compilers + ==> Available compilers + -- gcc --------------------------------------------------------- + gcc@4.9.0 gcc@4.8.0 gcc@4.7.0 gcc@4.6.2 gcc@4.4.7 + gcc@4.8.2 gcc@4.7.1 gcc@4.6.3 gcc@4.6.1 gcc@4.1.2 + -- intel ------------------------------------------------------- + intel@15.0.0 intel@14.0.0 intel@13.0.0 intel@12.1.0 intel@10.0 + intel@14.0.3 intel@13.1.1 intel@12.1.5 intel@12.0.4 intel@9.1 + intel@14.0.2 intel@13.1.0 intel@12.1.3 intel@11.1 + intel@14.0.1 intel@13.0.1 intel@12.1.2 intel@10.1 + -- clang ------------------------------------------------------- + clang@3.4 clang@3.3 clang@3.2 clang@3.1 + -- pgi --------------------------------------------------------- + pgi@14.3-0 pgi@13.2-0 pgi@12.1-0 pgi@10.9-0 pgi@8.0-1 + pgi@13.10-0 pgi@13.1-1 pgi@11.10-0 pgi@10.2-0 pgi@7.1-3 + pgi@13.6-0 pgi@12.8-0 pgi@11.1-0 pgi@9.0-4 pgi@7.0-6 + +Any of these compilers can be used to build Spack packages. More on +how this is done is in :ref:`sec-specs`. + +.. _spack-compiler-add: + +``spack compiler add`` +~~~~~~~~~~~~~~~~~~~~~~~ + +An alias for ``spack compiler find``. + +.. _spack-compiler-find: + +``spack compiler find`` +~~~~~~~~~~~~~~~~~~~~~~~ + +If you do not see a compiler in this list, but you want to use it with +Spack, you can simply run ``spack compiler find`` with the path to +where the compiler is installed. For example:: + + $ spack compiler find /usr/local/tools/ic-13.0.079 + ==> Added 1 new compiler to /Users/gamblin2/.spack/compilers.yaml + intel@13.0.079 + +Or you can run ``spack compiler find`` with no arguments to force +auto-detection. This is useful if you do not know where compilers are +installed, but you know that new compilers have been added to your +``PATH``. For example, using dotkit, you might do this:: + + $ module load gcc-4.9.0 + $ spack compiler find + ==> Added 1 new compiler to /Users/gamblin2/.spack/compilers.yaml + gcc@4.9.0 + +This loads the environment module for gcc-4.9.0 to add it to +``PATH``, and then it adds the compiler to Spack. + +.. _spack-compiler-info: + +``spack compiler info`` +~~~~~~~~~~~~~~~~~~~~~~~ + +If you want to see specifics on a particular compiler, you can run +``spack compiler info`` on it:: + + $ spack compiler info intel@15 + intel@15.0.0: + cc = /usr/local/bin/icc-15.0.090 + cxx = /usr/local/bin/icpc-15.0.090 + f77 = /usr/local/bin/ifort-15.0.090 + fc = /usr/local/bin/ifort-15.0.090 + +This shows which C, C++, and Fortran compilers were detected by Spack. +Notice also that we didn\'t have to be too specific about the +version. We just said ``intel@15``, and information about the only +matching Intel compiler was displayed. + + +Manual Compiler Configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If auto-detection fails, you can manually configure a compiler by +editing your ``~/.spack/compilers.yaml`` file. You can do this by running +``spack config edit compilers``, which will open the file in your ``$EDITOR``. + +Each compiler configuration in the file looks like this:: + + ... + compilers: + - compiler: + modules = [] + operating_system: OS + paths: + cc: /usr/local/bin/icc-15.0.024-beta + cxx: /usr/local/bin/icpc-15.0.024-beta + f77: /usr/local/bin/ifort-15.0.024-beta + fc: /usr/local/bin/ifort-15.0.024-beta + + spec: intel@15.0.0: + +For compilers, like ``clang``, that do not support Fortran, put +``None`` for ``f77`` and ``fc``:: + + clang@3.3svn: + cc: /usr/bin/clang + cxx: /usr/bin/clang++ + f77: None + fc: None + +Once you save the file, the configured compilers will show up in the +list displayed by ``spack compilers``. + +You can also add compiler flags to manually configured compilers. The +valid flags are ``cflags``, ``cxxflags``, ``fflags``, ``cppflags``, +``ldflags``, and ``ldlibs``. For example:: + + ... + compilers: + - compiler: + ... + intel@15.0.0: + cc: /usr/local/bin/icc-15.0.024-beta + cxx: /usr/local/bin/icpc-15.0.024-beta + f77: /usr/local/bin/ifort-15.0.024-beta + fc: /usr/local/bin/ifort-15.0.024-beta + cppflags: -O3 -fPIC + ... + +These flags will be treated by spack as if they were enterred from +the command line each time this compiler is used. The compiler wrappers +then inject those flags into the compiler command. Compiler flags +enterred from the command line will be discussed in more detail in the +following section. + + +Build Your Own Compiler +~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you are particular about which compiler/version you use, you might +wish to have Spack build it for you. For example: + +.. code-block:: + + spack install gcc@4.9.3 + +Once that has finished, you will need to add it to your +``compilers.yaml`` file. If this is your preferred compiler, in +general future Spack builds will use it. + +.. note:: + + If you are building your own compiler, it can be useful to have a + Spack instance just for that. For example, create a new Spack in + ``~/spack-tools`` and then run ``~/spack-tools/bin/spack install + gcc@4.9.3``. Once the compiler is built, don't build anything + more in that Spack instance; instead, create a new "real" Spack + instance, configure Spack to use the compiler you've just built, + and then build your application software in the new Spack + instance. + + This tip is useful because sometimes you will find yourself + rebuilding may pacakges due to Spack updates. Sometimes, you + might even delete your entire Spack installation and start fresh. + If your compiler was built in a separate Spack installation, you + will never have to rebuild it --- as long as you wish to continue + using that version of the compiler. + + +Compilers Requiring Modules +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Many installed compilers will work regardless of the environment they +are called with. However, some installed compilers require +``$LD_LIBRARY_PATH`` or other environment variables to be set in order +to run; Intel compilers are known for this. In such a case, you +should tell Spack which module(s) to load in order to run the chosen +compiler. Spack will load this module into the environment ONLY when +the compiler is run, and NOT in general for a package's ``install()`` +method. See, for example, this ``compilers.yaml`` file: + +.. code-block:: yaml + + compilers: + - compiler: + modules: [other/comp/gcc-5.3-sp3] + operating_system: SuSE11 + paths: + cc: /usr/local/other/SLES11.3/gcc/5.3.0/bin/gcc + cxx: /usr/local/other/SLES11.3/gcc/5.3.0/bin/g++ + f77: /usr/local/other/SLES11.3/gcc/5.3.0/bin/gfortran + fc: /usr/local/other/SLES11.3/gcc/5.3.0/bin/gfortran + spec: gcc@5.3.0 + +Some compilers require a module to be loaded not just to run, but also +to execute any code built with the compiler, breaking packages that +execute any bits of code they just compiled. Such compilers should be +taken behind the barn and put out of their misery. If that is not +possible or practical, the user (and anyone running code built by that +compiler) will need to load the compiler's module into their +environment. And ``spack install --dirty`` will need to be used. + +Compiler Verification +~~~~~~~~~~~~~~~~~~~~~~ + +You can verify that your compilers are configured properly by installing a +simple package. For example: + +.. code-block:: sh + + spack install zlib%gcc@5.3.0 + + +System Packages +----------------- + +Once compilers are configured, one needs to determine which +pre-installed system packages, if any, to use in builds. This is +configured in the file `~/.spack/packages.yaml`. For example, to use +an OpenMPI installed in /opt/local, one would use: + +.. code-block:: yaml + + packages: + openmpi: + paths: + openmpi@1.10.1: /opt/local + buildable: False + +In general, Spack is easier to use and more reliable if it builds all +its own dependencies. However, there are two packages for which one +commonly needs to use system versions: + +MPI +~~~ + +On supercomputers, sysadmins have already built MPI versions that take +into account the specifics of that computer's hardware. Unless you +know how they were built and can choose the correct Spack variants, +you are unlikely to get a working MPI from Spack. Instead, use an +appropraite pre-installed MPI. + +If you choose a pre-installed MPI, you should consider using the +pre-installed compiler used to build that MPI; see above on +``compilers.yaml``. + +OpenSSL +~~~~~~~~ + +The ``openssl`` package underlies much of modern security in a modern +OS; an attacker can easily "pwn" any computer on which can modify SSL. +Therefore, any `openssl` used on a system should be created in a +"trusted environment" --- for example, that of the OS vendor. + +OpenSSL is also updated by the OS vendor from time to time, in +response to security problems discovered in the wider community. It +is in everyone's best interest to use any newly updated versions as +soon as they come out. Modern Linux installations have standard +procedures for security updates without user involvement. + +Spack running at user-level is not a trusted environment, nor do Spack +users generally keep up-to-date on the latest security holes in SSL. +For these reasons, any Spack-installed OpenSSL should be considered +untrusted. + +As long as the system-provided SSL works, it is better to use it. One can check if it works by trying to download an ``https://``. For example: + +.. code-block:: sh + + curl -O https://github.com/ImageMagick/ImageMagick/archive/7.0.2-7.tar.gz + +As long as it works, the recommended way to tell Spack to use the +system-supplied OpenSSL is to add the following to ``packages.yaml``. +Note that the ``@system`` "version" means "I don't care what version +it is, just use what is there." This is appropraite for OpenSSL, +which has a stable API. + + +.. code-block:: yaml + + packages: + # Recommended for security reasons + # Do not install OpenSSL as non-root user. + openssl: + paths: + openssl@system: /usr + version: [system] + buildable: False + + + +Utilities Configuration +------------------------- + +Although Spack does not need installation *per se*, it does rely on +other packages to be available on its host system. If those packages +are out of date or missing, then Spack will not work. Sometimes, an +appeal to the system's package manager can fix such problems. If not, +the solution is have Spack install the required packages, and then +have Spack use them. + +For example, if `curl` doesn't work, one could use the following steps +to provide Spack a working `curl`: + +.. code-block:: sh + + $ spack install curl + $ spack load curl + +or alternately: + +.. code-block:: sh + + $ spack module loads curl >>~/.bashrc + +or if environment modules don't work: + +.. code-block:: sh + + $ export PATH=`spack location -i curl`/bin:$PATH + + +External commands are used by Spack in two places: within core Spack, +and in the package recipes. The bootstrapping procedure for these two +cases is somewhat different, and is treated separately below. + +Core Spack Utilities +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Core Spack uses the following packages, aminly to download and unpack +source code, and to load generated environment modules: ``curl``, +``env``, ``git``, ``go``, ``hg``, ``svn``, ``tar``, ``unzip``, +``patch``, ``environment-modules``. + +As long as the user's environment is set up to successfully run these +programs from outside of Spack, they should work inside of Spack as +well. They can generally be activated as in the `curl` example above; +or some systems might already have an appropriate hand-built +environment module that may be loaded. Either way works. + +A few notes on specific programs in this list: + +cURL, git, Mercurial, etc. +``````````````````````````` + +Spack depends on cURL to download tarballs, the format that most +Spack-installed packages come in. Your system's cURL should always be +able to download unencrypted ``http://``. However, the cURL on some +systems has problems with SSL-enabled ``https://`` URLs, due to +outdated / insecure versions of OpenSSL on those systems. This will +prevent Spack from installing any software requiring ``https://`` +until a new cURL has been installed, using the technique above. + +.. note:: + + ``curl`` depends on ``openssl`` and ``zlib``, both of which are + downloadable from non-SSL sources. Unfortunately, this + Spack-built cURL should be considered untrustworthy for + ``https://`` sources becuase it relies on an OpenSSL built in user + space. Luckily, Spack verifies checksums of the software it + installs, and does not rely on a secure SSL implementation. + + If your version of ``curl`` is not trustworthy, then you should + not use it outside of Spack. Instead of putting it in your + ``.bashrc``, you might wish to create a short shell script that + loads the appropariate module(s) and then launches Spack. + +Some packages use source code control systems as their download +method: ``git``, ``hg``, ``svn`` and occasionally ``go``. If you had +to install a new ``curl``, then chances are the system-supplied +version of these other programs will also not work, because they also +rely on OpenSSL. Once ``curl`` has been installed, the others should +also be installable. + + +.. _InstallEnvironmentModules: + +Environment Modules +```````````````````` + +In order to use Spack's generated environment modules, you must have +installed the *Environment Modules* package. On many Linux +distributions, this can be installed from the vendor's repository. +For example: ```yum install environment-modules`` +(Fedora/RHEL/CentOS). If your Linux distribution does not have +Environment Modules, you can get it with Spack: + +1. Consider using system tcl (as long as your system has Tcl version 8.0 or later): + 1. Identify its location using ``which tclsh`` + 2. Identify its version using ``echo 'puts $tcl_version;exit 0' | tclsh`` + 3. Add to ``~/.spack/packages.yaml`` and modify as appropriate: + + .. code-block:: yaml + + packages: + tcl: + paths: + tcl@8.5: /usr + version: [8.5] + buildable: False + +2. Install with:: + .. code-block:: sh + + spack install environment-modules + +3. Activate with the following script (or apply the updates to your + ``.bashrc`` file manually):: + + .. code-block:: sh + + TMP=`tempfile` + echo >$TMP + MODULE_HOME=`spack location -i environment-modules` + MODULE_VERSION=`ls -1 $MODULE_HOME/Modules | head -1` + ${MODULE_HOME}/Modules/${MODULE_VERSION}/bin/add.modules <$TMP + cp .bashrc $TMP + echo "MODULE_VERSION=${MODULE_VERSION}" > .bashrc + cat $TMP >>.bashrc + +This adds to your ``.bashrc`` (or similar) files, enabling Environment +Modules when you log in. Re-load your .bashrc (or log out and in +again), and then test that the ``module`` command is found with: + +.. code-block:: sh + + module avail + + +Package Utilities +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Spack can also encounter bootstrapping problems inside a package's +``install()`` method. In this case, Spack will normally be running +inside a *sanitized build environment*. This includes all of the +package's dependencies, but none of the environment Spack inherited +from the user: if you load a module or modify ``$PATH`` before +launching Spack, it will have no effect. + +In this case, you will likley need to use the ``--dirty`` flag when +running ``spack install``, causing Spack to **not** santize the build +environment. You are now responsible for making sure that environment +does not do strange things to Spack or its installs. + +Another way to get Spack to use its own version of something is to add that something to a package that needs it. For example: + +.. code-block:: python + + depends_on('binutils', type='build') + +This is considered best practice for some common build dependencies, +such as ``autotools`` (if the ``autoreconf`` command is needed) and +``cmake`` --- ``cmake`` especially, because different packages require +a different version of CMake. + +However, adding ``depends_on('binutils')`` to every package is not +considered a best practice because every package written in +C/C++/Fortran would need it. Loading a recent ``binutils`` into your +environment is preferable here. + +binutils +~~~~~~~~~ + +# https://groups.google.com/forum/#!topic/spack/i_7l_kEEveI + +Sometimes, strange error messages can happen while building a package. +For exmaple, ``ld`` might crash. Or one receives a message like: + +.. code-block:: + + ld: final link failed: Nonrepresentable section on output + +These problems are often caused by an outdated ``binutils`` on your +system: bootstrap as described above. Install Environment Modules ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -114,8 +638,139 @@ again), and then test that the ``module`` command is found with: module avail +Spack on Cray +----------------------------- + +Spack differs slightly when used on a Cray system. The architecture spec +can differentiate between the front-end and back-end processor and operating system. +For example, on Edison at NERSC, the back-end target processor +is \"Ivy Bridge\", so you can specify to use the back-end this way: +.. code-block:: sh + spack install zlib target=ivybridge -git -binutils +You can also use the operating system to build against the back-end: + +.. code-block:: sh + + spack install zlib os=CNL10 + +Notice that the name includes both the operating system name and the major +version number concatenated together. + +Alternatively, if you want to build something for the front-end, +you can specify the front-end target processor. The processor for a login node +on Edison is \"Sandy bridge\" so we specify on the command line like so: + +.. code-block:: sh + + spack install zlib target=sandybridge + +And the front-end operating system is: + +.. code-block:: sh + + spack install zlib os=SuSE11 + + + +Cray compiler detection +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Spack can detect compilers using two methods. For the front-end, we treat +everything the same. The difference lies in back-end compiler detection. +Back-end compiler detection is made via the Tcl module avail command. +Once it detects the compiler it writes the appropriate PrgEnv and compiler +module name to compilers.yaml and sets the paths to each compiler with Cray\'s +compiler wrapper names (i.e. cc, CC, ftn). During build time, Spack will load +the correct PrgEnv and compiler module and will call appropriate wrapper. + +The compilers.yaml config file will also differ. There is a +modules section that is filled with the compiler\'s Programming Environment +and module name. On other systems, this field is empty []:: + + ... + - compiler: + modules: + - PrgEnv-intel + - intel/15.0.109 + ... + +As mentioned earlier, the compiler paths will look different on a Cray system. +Since most compilers are invoked using cc, CC and ftn, the paths for each +compiler are replaced with their respective Cray compiler wrapper names:: + + ... + paths: + cc: cc + cxx: CC + f77: ftn + fc: ftn + ... + +As opposed to an explicit path to the compiler executable. This allows Spack +to call the Cray compiler wrappers during build time. + +For more on compiler configuration, check out :ref:`compiler-config`. + +Spack sets the default Cray link type to dynamic, to better match other +other platforms. Individual packages can enable static linking (which is the +default outside of Spack on cray systems) using the -static flag. + +Setting defaults and using Cray modules +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you want to use default compilers for each PrgEnv and also be able +to load cray external modules, you will need to set up a packages.yaml. + +Here\'s an example of an external configuration for cray modules: + +.. code-block:: yaml + + packages: + mpi: + modules: + mpich@7.3.1%gcc@5.2.0 arch=cray_xc-haswell-CNL10: cray-mpich + mpich@7.3.1%intel@16.0.0.109 arch=cray_xc-haswell-CNL10: cray-mpich + +This tells Spack that for whatever package that depends on mpi, load the +cray-mpich module into the environment. You can then be able to use whatever +environment variables, libraries, etc, that are brought into the environment +via module load. + +You can set the default compiler that Spack can use for each compiler type. +If you want to use the Cray defaults, then set them under *all:* in packages.yaml. +In the compiler field, set the compiler specs in your order of preference. +Whenever you build with that compiler type, Spack will concretize to that version. + +Here is an example of a full packages.yaml used at NERSC + +.. code-block:: yaml + + packages: + mpi: + modules: + mpich@7.3.1%gcc@5.2.0 arch=cray_xc-CNL10-ivybridge: cray-mpich + mpich@7.3.1%intel@16.0.0.109 arch=cray_xc-SuSE11-ivybridge: cray-mpich + buildable: False + netcdf: + modules: + netcdf@4.3.3.1%gcc@5.2.0 arch=cray_xc-CNL10-ivybridge: cray-netcdf + netcdf@4.3.3.1%intel@16.0.0.109 arch=cray_xc-CNL10-ivybridge: cray-netcdf + buildable: False + hdf5: + modules: + hdf5@1.8.14%gcc@5.2.0 arch=cray_xc-CNL10-ivybridge: cray-hdf5 + hdf5@1.8.14%intel@16.0.0.109 arch=cray_xc-CNL10-ivybridge: cray-hdf5 + buildable: False + all: + compiler: [gcc@5.2.0, intel@16.0.0.109] + +Here we tell spack that whenever we want to build with gcc use version 5.2.0 or +if we want to build with intel compilers, use version 16.0.0.109. We add a spec +for each compiler type for each cray modules. This ensures that for each +compiler on our system we can use that external module. + + +For more on external packages check out the section :ref:`sec-external_packages`. diff --git a/lib/spack/docs/index.rst b/lib/spack/docs/index.rst index a5bbd4e23b..10661e1730 100644 --- a/lib/spack/docs/index.rst +++ b/lib/spack/docs/index.rst @@ -46,6 +46,7 @@ Table of Contents getting_started basic_usage packaging_guide + application_developer_support mirrors configuration developer_guide diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst index 879beb2476..5ec6ee4cb9 100644 --- a/lib/spack/docs/packaging_guide.rst +++ b/lib/spack/docs/packaging_guide.rst @@ -2999,108 +2999,3 @@ might write: CXXFLAGS += -I$DWARF_PREFIX/include CXXFLAGS += -L$DWARF_PREFIX/lib -Build System Configuration Support ----------------------------------- - -Imagine a developer creating a CMake-based (or Autotools) project in a local -directory, which depends on libraries A-Z. Once Spack has installed -those dependencies, one would like to run ``cmake`` with appropriate -command line and environment so CMake can find them. The ``spack -setup`` command does this conveniently, producing a CMake -configuration that is essentially the same as how Spack *would have* -configured the project. This can be demonstrated with a usage -example: - -.. code-block:: bash - - cd myproject - spack setup myproject@local - mkdir build; cd build - ../spconfig.py .. - make - make install - -Notes: - * Spack must have ``myproject/package.py`` in its repository for - this to work. - * ``spack setup`` produces the executable script ``spconfig.py`` in - the local directory, and also creates the module file for the - package. ``spconfig.py`` is normally run from the user's - out-of-source build directory. - * The version number given to ``spack setup`` is arbitrary, just - like ``spack diy``. ``myproject/package.py`` does not need to - have any valid downloadable versions listed (typical when a - project is new). - * spconfig.py produces a CMake configuration that *does not* use the - Spack wrappers. Any resulting binaries *will not* use RPATH, - unless the user has enabled it. This is recommended for - development purposes, not production. - * ``spconfig.py`` is human readable, and can serve as a developer - reference of what dependencies are being used. - * ``make install`` installs the package into the Spack repository, - where it may be used by other Spack packages. - * CMake-generated makefiles re-run CMake in some circumstances. Use - of ``spconfig.py`` breaks this behavior, requiring the developer - to manually re-run ``spconfig.py`` when a ``CMakeLists.txt`` file - has changed. - -CMakePackage -~~~~~~~~~~~~ - -In order ot enable ``spack setup`` functionality, the author of -``myproject/package.py`` must subclass from ``CMakePackage`` instead -of the standard ``Package`` superclass. Because CMake is -standardized, the packager does not need to tell Spack how to run -``cmake; make; make install``. Instead the packager only needs to -create (optional) methods ``configure_args()`` and ``configure_env()``, which -provide the arguments (as a list) and extra environment variables (as -a dict) to provide to the ``cmake`` command. Usually, these will -translate variant flags into CMake definitions. For example: - -.. code-block:: python - - def configure_args(self): - spec = self.spec - return [ - '-DUSE_EVERYTRACE=%s' % ('YES' if '+everytrace' in spec else 'NO'), - '-DBUILD_PYTHON=%s' % ('YES' if '+python' in spec else 'NO'), - '-DBUILD_GRIDGEN=%s' % ('YES' if '+gridgen' in spec else 'NO'), - '-DBUILD_COUPLER=%s' % ('YES' if '+coupler' in spec else 'NO'), - '-DUSE_PISM=%s' % ('YES' if '+pism' in spec else 'NO')] - -If needed, a packager may also override methods defined in -``StagedPackage`` (see below). - - -StagedPackage -~~~~~~~~~~~~~ - -``CMakePackage`` is implemented by subclassing the ``StagedPackage`` -superclass, which breaks down the standard ``Package.install()`` -method into several sub-stages: ``setup``, ``configure``, ``build`` -and ``install``. Details: - -* Instead of implementing the standard ``install()`` method, package - authors implement the methods for the sub-stages - ``install_setup()``, ``install_configure()``, - ``install_build()``, and ``install_install()``. - -* The ``spack install`` command runs the sub-stages ``configure``, - ``build`` and ``install`` in order. (The ``setup`` stage is - not run by default; see below). -* The ``spack setup`` command runs the sub-stages ``setup`` - and a dummy install (to create the module file). -* The sub-stage install methods take no arguments (other than - ``self``). The arguments ``spec`` and ``prefix`` to the standard - ``install()`` method may be accessed via ``self.spec`` and - ``self.prefix``. - -GNU Autotools -~~~~~~~~~~~~~ - -The ``setup`` functionality is currently only available for -CMake-based packages. Extending this functionality to GNU -Autotools-based packages would be easy (and should be done by a -developer who actively uses Autotools). Packages that use -non-standard build systems can gain ``setup`` functionality by -subclassing ``StagedPackage`` directly. |