diff options
Diffstat (limited to 'lib/spack/docs/workflows.rst')
| -rw-r--r-- | lib/spack/docs/workflows.rst | 394 |
1 files changed, 0 insertions, 394 deletions
diff --git a/lib/spack/docs/workflows.rst b/lib/spack/docs/workflows.rst index 7deb5eff54..4a56563283 100644 --- a/lib/spack/docs/workflows.rst +++ b/lib/spack/docs/workflows.rst @@ -703,400 +703,6 @@ environments: Administrators might find things easier to maintain without the added "heavyweight" state of a view. ------------------------------- -Developing Software with Spack ------------------------------- - -For any project, one needs to assemble an -environment of that application's dependencies. You might consider -loading a series of modules or creating a filesystem view. This -approach, while obvious, has some serious drawbacks: - -1. There is no guarantee that an environment created this way will be - consistent. Your application could end up with dependency A - expecting one version of MPI, and dependency B expecting another. - The linker will not be happy... - -2. Suppose you need to debug a package deep within your software DAG. - If you build that package with a manual environment, then it - becomes difficult to have Spack auto-build things that depend on - it. That could be a serious problem, depending on how deep the - package in question is in your dependency DAG. - -3. At its core, Spack is a sophisticated concretization algorithm that - matches up packages with appropriate dependencies and creates a - *consistent* environment for the package it's building. Writing a - list of ``spack load`` commands for your dependencies is at least - as hard as writing the same list of ``depends_on()`` declarations - in a Spack package. But it makes no use of Spack concretization - and is more error-prone. - -4. Spack provides an automated, systematic way not just to find a - packages's dependencies --- but also to build other packages on - top. Any Spack package can become a dependency for another Spack - package, offering a powerful vision of software re-use. If you - build your package A outside of Spack, then your ability to use it - as a building block for other packages in an automated way is - diminished: other packages depending on package A will not - be able to use Spack to fulfill that dependency. - -5. If you are reading this manual, you probably love Spack. You're - probably going to write a Spack package for your software so - prospective users can install it with the least amount of pain. - Why should you go to additional work to find dependencies in your - development environment? Shouldn't Spack be able to help you build - your software based on the package you've already written? - -In this section, we show how Spack can be used in the software -development process to greatest effect, and how development packages -can be seamlessly integrated into the Spack ecosystem. We will show -how this process works by example, assuming the software you are -creating is called ``mylib``. - -^^^^^^^^^^^^^^^^^^^^^ -Write the CMake Build -^^^^^^^^^^^^^^^^^^^^^ - -For now, the techniques in this section only work for CMake-based -projects, although they could be easily extended to other build -systems in the future. We will therefore assume you are using CMake -to build your project. - -The ``CMakeLists.txt`` file should be written as normal. A few caveats: - -1. Your project should produce binaries with RPATHs. This will ensure - that they work the same whether built manually or automatically by - Spack. For example: - -.. code-block:: cmake - - # enable @rpath in the install name for any shared library being built - # note: it is planned that a future version of CMake will enable this by default - set(CMAKE_MACOSX_RPATH 1) - - # Always use full RPATH - # http://www.cmake.org/Wiki/CMake_RPATH_handling - # http://www.kitware.com/blog/home/post/510 - - # use, i.e. don't skip the full RPATH for the build tree - SET(CMAKE_SKIP_BUILD_RPATH FALSE) - - # when building, don't use the install RPATH already - # (but later on when installing) - SET(CMAKE_BUILD_WITH_INSTALL_RPATH FALSE) - - # add the automatically determined parts of the RPATH - # which point to directories outside the build tree to the install RPATH - SET(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE) - - # the RPATH to be used when installing, but only if it's not a system directory - LIST(FIND CMAKE_PLATFORM_IMPLICIT_LINK_DIRECTORIES "${CMAKE_INSTALL_PREFIX}/lib" isSystemDir) - IF("${isSystemDir}" STREQUAL "-1") - SET(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_PREFIX}/lib") - ENDIF("${isSystemDir}" STREQUAL "-1") - - -2. Spack provides a CMake variable called - ``SPACK_TRANSITIVE_INCLUDE_PATH``, which contains the ``include/`` - directory for all of your project's transitive dependencies. It - can be useful if your project ``#include``s files from package B, - which ``#include`` files from package C, but your project only - lists project B as a dependency. This works in traditional - single-tree build environments, in which B and C's include files - live in the same place. In order to make it work with Spack as - well, you must add the following to ``CMakeLists.txt``. It will - have no effect when building without Spack: - - .. code-block:: cmake - - # Include all the transitive dependencies determined by Spack. - # If we're not running with Spack, this does nothing... - include_directories($ENV{SPACK_TRANSITIVE_INCLUDE_PATH}) - - .. note:: - - Note that this feature is controversial and could break with - future versions of GNU ld. The best practice is to make sure - anything you ``#include`` is listed as a dependency in your - CMakeLists.txt (and Spack package). - -.. _write-the-spack-package: - -^^^^^^^^^^^^^^^^^^^^^^^ -Write the Spack Package -^^^^^^^^^^^^^^^^^^^^^^^ - -The Spack package also needs to be written, in tandem with setting up -the build (for example, CMake). The most important part of this task -is declaring dependencies. Here is an example of the Spack package -for the ``mylib`` package (ellipses for brevity): - -.. code-block:: python - - class Mylib(CMakePackage): - """Misc. reusable utilities used by Myapp.""" - - homepage = "https://github.com/citibeth/mylib" - url = "https://github.com/citibeth/mylib/tarball/123" - - version('0.1.2', '3a6acd70085e25f81b63a7e96c504ef9') - version('develop', git='https://github.com/citibeth/mylib.git', - branch='develop') - - variant('everytrace', default=False, - description='Report errors through Everytrace') - ... - - extends('python') - - depends_on('eigen') - depends_on('everytrace', when='+everytrace') - depends_on('proj', when='+proj') - ... - depends_on('cmake', type='build') - depends_on('doxygen', type='build') - - def cmake_args(self): - spec = self.spec - return [ - '-DUSE_EVERYTRACE=%s' % ('YES' if '+everytrace' in spec else 'NO'), - '-DUSE_PROJ4=%s' % ('YES' if '+proj' in spec else 'NO'), - ... - '-DUSE_UDUNITS2=%s' % ('YES' if '+udunits2' in spec else 'NO'), - '-DUSE_GTEST=%s' % ('YES' if '+googletest' in spec else 'NO')] - -This is a standard Spack package that can be used to install -``mylib`` in a production environment. The list of dependencies in -the Spack package will generally be a repeat of the list of CMake -dependencies. This package also has some features that allow it to be -used for development: - -1. It subclasses ``CMakePackage`` instead of ``Package``. This - eliminates the need to write an ``install()`` method, which is - defined in the superclass. Instead, one just needs to write the - ``configure_args()`` method. That method should return the - arguments needed for the ``cmake`` command (beyond the standard - CMake arguments, which Spack will include already). These - arguments are typically used to turn features on/off in the build. - -2. It specifies a non-checksummed version ``develop``. Running - ``spack install mylib@develop`` the ``@develop`` version will - install the latest version off the develop branch. This method of - download is useful for the developer of a project while it is in - active development; however, it should only be used by developers - who control and trust the repository in question! - -3. The ``url``, ``url_for_version()`` and ``homepage`` attributes are - not used in development. Don't worry if you don't have any, or if - they are behind a firewall. - -^^^^^^^^^^^^^^^^ -Build with Spack -^^^^^^^^^^^^^^^^ - -Now that you have a Spack package, you can use Spack to find its -dependencies automatically. For example: - -.. code-block:: console - - $ cd mylib - $ spack setup mylib@local - -The result will be a file ``spconfig.py`` in the top-level -``mylib/`` directory. It is a short script that calls CMake with the -dependencies and options determined by Spack --- similar to what -happens in ``spack install``, but now written out in script form. -From a developer's point of view, you can think of ``spconfig.py`` as -a stand-in for the ``cmake`` command. - -.. note:: - - You can invent any "version" you like for the ``spack setup`` - command. - -.. note:: - - Although ``spack setup`` does not build your package, it does - create and install a module file, and mark in the database that - your package has been installed. This can lead to errors, of - course, if you don't subsequently install your package. - Also... you will need to ``spack uninstall`` before you run - ``spack setup`` again. - - -You can now build your project as usual with CMake: - -.. code-block:: console - - $ mkdir build; cd build - $ ../spconfig.py .. # Instead of cmake .. - $ make - $ make install - -Once your ``make install`` command is complete, your package will be -installed, just as if you'd run ``spack install``. Except you can now -edit, re-build and re-install as often as needed, without checking -into Git or downloading tarballs. - -.. note:: - - The build you get this way will be *almost* the same as the build - from ``spack install``. The only difference is, you will not be - using Spack's compiler wrappers. This difference has not caused - problems in our experience, as long as your project sets - RPATHs as shown above. You DO use RPATHs, right? - -^^^^^^^^^^^^^^^^^^^^ -Build Other Software -^^^^^^^^^^^^^^^^^^^^ - -Now that you've built ``mylib`` with Spack, you might want to build -another package that depends on it --- for example, ``myapp``. This -is accomplished easily enough: - -.. code-block:: console - - $ spack install myapp ^mylib@local - -Note that auto-built software has now been installed *on top of* -manually-built software, without breaking Spack's "web." This -property is useful if you need to debug a package deep in the -dependency hierarchy of your application. It is a *big* advantage of -using ``spack setup`` to build your package's environment. - -If you feel your software is stable, you might wish to install it with -``spack install`` and skip the source directory. You can just use, -for example: - -.. code-block:: console - - $ spack install mylib@develop - -.. _release-your-software: - -^^^^^^^^^^^^^^^^^^^^^ -Release Your Software -^^^^^^^^^^^^^^^^^^^^^ - -You are now ready to release your software as a tarball with a -numbered version, and a Spack package that can build it. If you're -hosted on GitHub, this process will be a bit easier. - -#. Put tag(s) on the version(s) in your GitHub repo you want to be - release versions. For example, a tag ``v0.1.0`` for version 0.1.0. - -#. Set the ``url`` in your ``package.py`` to download a tarball for - the appropriate version. GitHub will give you a tarball for any - commit in the repo, if you tickle it the right way. For example: - - .. code-block:: python - - url = 'https://github.com/citibeth/mylib/tarball/v0.1.2' - -#. Use Spack to determine your version's hash, and cut'n'paste it into - your ``package.py``: - - .. code-block:: console - - $ spack checksum mylib 0.1.2 - ==> Found 1 versions of mylib - 0.1.2 https://github.com/citibeth/mylib/tarball/v0.1.2 - - How many would you like to checksum? (default is 5, q to abort) - ==> Downloading... - ==> Trying to fetch from https://github.com/citibeth/mylib/tarball/v0.1.2 - ######################################################################## 100.0% - ==> Checksummed new versions of mylib: - version('0.1.2', '3a6acd70085e25f81b63a7e96c504ef9') - -#. You should now be able to install released version 0.1.2 of your package with: - - .. code-block:: console - - $ spack install mylib@0.1.2 - -#. There is no need to remove the `develop` version from your package. - Spack concretization will always prefer numbered version to - non-numeric versions. Users will only get it if they ask for it. - -^^^^^^^^^^^^^^^^^^^^^^^^ -Distribute Your Software -^^^^^^^^^^^^^^^^^^^^^^^^ - -Once you've released your software, other people will want to build -it; and you will need to tell them how. In the past, that has meant a -few paragraphs of prose explaining which dependencies to install. But -now you use Spack, and those instructions are written in executable -Python code. But your software has many dependencies, and you know -Spack is the best way to install it: - -#. First, you will want to fork Spack's ``develop`` branch. Your aim - is to provide a stable version of Spack that you KNOW will install - your software. If you make changes to Spack in the process, you - will want to submit pull requests to Spack core. - -#. Add your software's ``package.py`` to that fork. You should submit - a pull request for this as well, unless you don't want the public - to know about your software. - -#. Prepare instructions that read approximately as follows: - - #. Download Spack from your forked repo. - - #. Install Spack; see :ref:`getting_started`. - - #. Set up an appropriate ``packages.yaml`` file. You should tell - your users to include in this file whatever versions/variants - are needed to make your software work correctly (assuming those - are not already in your ``packages.yaml``). - - #. Run ``spack install mylib``. - - #. Run this script to generate the ``module load`` commands or - filesystem view needed to use this software. - -#. Be aware that your users might encounter unexpected bootstrapping - issues on their machines, especially if they are running on older - systems. The :ref:`getting_started` section should cover this, but - there could always be issues. - -^^^^^^^^^^^^^^^^^^^ -Other Build Systems -^^^^^^^^^^^^^^^^^^^ - -``spack setup`` currently only supports CMake-based builds, in -packages that subclass ``CMakePackage``. The intent is that this -mechanism should support a wider range of build systems; for example, -GNU Autotools. Someone well-versed in Autotools is needed to develop -this patch and test it out. - -Python Distutils is another popular build system that should get -``spack setup`` support. For non-compiled languages like Python, -``spack diy`` may be used. Even better is to put the source directory -directly in the user's ``PYTHONPATH``. Then, edits in source files -are immediately available to run without any install process at all! - -^^^^^^^^^^ -Conclusion -^^^^^^^^^^ - -The ``spack setup`` development workflow provides better automation, -flexibility and safety than workflows relying on environment modules -or filesystem views. However, it has some drawbacks: - -#. It currently works only with projects that use the CMake build - system. Support for other build systems is not hard to build, but - will require a small amount of effort for each build system to be - supported. It might not work well with some IDEs. - -#. It only works with packages that sub-class ``StagedPackage``. - Currently, most Spack packages do not. Converting them is not - hard; but must be done on a package-by-package basis. - -#. It requires that users are comfortable with Spack, as they - integrate Spack explicitly in their workflow. Not all users are - willing to do this. - ------------------------------------- Using Spack to Replace Homebrew/Conda ------------------------------------- |