diff options
| -rw-r--r-- | lib/spack/docs/config_yaml.rst | 4 | ||||
| -rw-r--r-- | lib/spack/docs/known_issues.rst | 13 | ||||
| -rw-r--r-- | lib/spack/docs/packaging_guide.rst | 116 | ||||
| -rw-r--r-- | lib/spack/docs/workflows.rst | 394 | ||||
| -rw-r--r-- | lib/spack/spack/cmd/setup.py | 3 | ||||
| -rw-r--r-- | lib/spack/spack/installer.py | 1 |
6 files changed, 5 insertions, 526 deletions
diff --git a/lib/spack/docs/config_yaml.rst b/lib/spack/docs/config_yaml.rst index 865845b274..bcb41527a9 100644 --- a/lib/spack/docs/config_yaml.rst +++ b/lib/spack/docs/config_yaml.rst @@ -99,7 +99,7 @@ username is not already in the path, Spack will append the value of ``$user`` to the selected ``build_stage`` path. .. warning:: We highly recommend specifying ``build_stage`` paths that - distinguish between staging and other activities to ensure + distinguish between staging and other activities to ensure ``spack clean`` does not inadvertently remove unrelated files. Spack prepends ``spack-stage-`` to temporary staging directory names to reduce this risk. Using a combination of ``spack`` and or ``stage`` in @@ -223,7 +223,7 @@ To build all software in serial, set ``build_jobs`` to 1. -------------------- When set to ``true`` Spack will use ccache to cache compiles. This is -useful specifically in two cases: (1) when using ``spack setup``, and (2) +useful specifically in two cases: (1) when using ``spack dev-build``, and (2) when building the same package with many different variants. The default is ``false``. diff --git a/lib/spack/docs/known_issues.rst b/lib/spack/docs/known_issues.rst index 7d47137392..531b3bcc5a 100644 --- a/lib/spack/docs/known_issues.rst +++ b/lib/spack/docs/known_issues.rst @@ -48,16 +48,3 @@ A workaround is to explicitly activate the variants of dependencies as well: See https://github.com/spack/spack/issues/267 and https://github.com/spack/spack/issues/2546 for further details. - - ----------------------------- -``spack setup`` doesn't work ----------------------------- - -**Status:** Work in progress - -Spack provides a ``setup`` command that is useful for the development of -software outside of Spack. Unfortunately, this command no longer works. -See https://github.com/spack/spack/issues/2597 and -https://github.com/spack/spack/issues/2662 for details. This is expected -to be fixed by https://github.com/spack/spack/pull/2664. diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst index 752a9c8aa7..cb5835c5c2 100644 --- a/lib/spack/docs/packaging_guide.rst +++ b/lib/spack/docs/packaging_guide.rst @@ -4677,119 +4677,3 @@ might write: DWARF_PREFIX = $(spack location --install-dir libdwarf) CXXFLAGS += -I$DWARF_PREFIX/include CXXFLAGS += -L$DWARF_PREFIX/lib - -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Build System Configuration Support -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Imagine a developer creating a CMake or Autotools-based project in a -local directory, which depends on libraries A-Z. Once Spack has -installed those dependencies, one would like to run ``cmake`` with -appropriate command line and environment so CMake can find them. The -``spack setup`` command does this conveniently, producing a CMake -configuration that is essentially the same as how Spack *would have* -configured the project. This can be demonstrated with a usage -example: - -.. code-block:: console - - $ cd myproject - $ spack setup myproject@local - $ mkdir build; cd build - $ ../spconfig.py .. - $ make - $ make install - -Notes: - -* Spack must have ``myproject/package.py`` in its repository for - this to work. -* ``spack setup`` produces the executable script ``spconfig.py`` in - the local directory, and also creates the module file for the - package. ``spconfig.py`` is normally run from the user's - out-of-source build directory. -* The version number given to ``spack setup`` is arbitrary, just - like ``spack diy``. ``myproject/package.py`` does not need to - have any valid downloadable versions listed (typical when a - project is new). -* spconfig.py produces a CMake configuration that *does not* use the - Spack wrappers. Any resulting binaries *will not* use RPATH, - unless the user has enabled it. This is recommended for - development purposes, not production. -* ``spconfig.py`` is human readable, and can serve as a developer - reference of what dependencies are being used. -* ``make install`` installs the package into the Spack repository, - where it may be used by other Spack packages. -* CMake-generated makefiles re-run CMake in some circumstances. Use - of ``spconfig.py`` breaks this behavior, requiring the developer - to manually re-run ``spconfig.py`` when a ``CMakeLists.txt`` file - has changed. - -^^^^^^^^^^^^ -CMakePackage -^^^^^^^^^^^^ - -In order to enable ``spack setup`` functionality, the author of -``myproject/package.py`` must subclass from ``CMakePackage`` instead -of the standard ``Package`` superclass. Because CMake is -standardized, the packager does not need to tell Spack how to run -``cmake; make; make install``. Instead the packager only needs to -create (optional) methods ``configure_args()`` and ``configure_env()``, which -provide the arguments (as a list) and extra environment variables (as -a dict) to provide to the ``cmake`` command. Usually, these will -translate variant flags into CMake definitions. For example: - -.. code-block:: python - - def cmake_args(self): - spec = self.spec - return [ - '-DUSE_EVERYTRACE=%s' % ('YES' if '+everytrace' in spec else 'NO'), - '-DBUILD_PYTHON=%s' % ('YES' if '+python' in spec else 'NO'), - '-DBUILD_GRIDGEN=%s' % ('YES' if '+gridgen' in spec else 'NO'), - '-DBUILD_COUPLER=%s' % ('YES' if '+coupler' in spec else 'NO'), - '-DUSE_PISM=%s' % ('YES' if '+pism' in spec else 'NO') - ] - -If needed, a packager may also override methods defined in -``StagedPackage`` (see below). - -^^^^^^^^^^^^^ -StagedPackage -^^^^^^^^^^^^^ - -``CMakePackage`` is implemented by subclassing the ``StagedPackage`` -superclass, which breaks down the standard ``Package.install()`` -method into several sub-stages: ``setup``, ``configure``, ``build`` -and ``install``. Details: - -* Instead of implementing the standard ``install()`` method, package - authors implement the methods for the sub-stages - ``install_setup()``, ``install_configure()``, - ``install_build()``, and ``install_install()``. - -* The ``spack install`` command runs the sub-stages ``configure``, - ``build`` and ``install`` in order. (The ``setup`` stage is - not run by default; see below). -* The ``spack setup`` command runs the sub-stages ``setup`` - and a dummy install (to create the module file). -* The sub-stage install methods take no arguments (other than - ``self``). The arguments ``spec`` and ``prefix`` to the standard - ``install()`` method may be accessed via ``self.spec`` and - ``self.prefix``. - -^^^^^^^^^^^^^ -GNU Autotools -^^^^^^^^^^^^^ - -The ``setup`` functionality is currently only available for -CMake-based packages. Extending this functionality to GNU -Autotools-based packages would be easy (and should be done by a -developer who actively uses Autotools). Packages that use -non-standard build systems can gain ``setup`` functionality by -subclassing ``StagedPackage`` directly. - -.. Emacs local variables - Local Variables: - fill-column: 79 - End: diff --git a/lib/spack/docs/workflows.rst b/lib/spack/docs/workflows.rst index 7deb5eff54..4a56563283 100644 --- a/lib/spack/docs/workflows.rst +++ b/lib/spack/docs/workflows.rst @@ -703,400 +703,6 @@ environments: Administrators might find things easier to maintain without the added "heavyweight" state of a view. ------------------------------- -Developing Software with Spack ------------------------------- - -For any project, one needs to assemble an -environment of that application's dependencies. You might consider -loading a series of modules or creating a filesystem view. This -approach, while obvious, has some serious drawbacks: - -1. There is no guarantee that an environment created this way will be - consistent. Your application could end up with dependency A - expecting one version of MPI, and dependency B expecting another. - The linker will not be happy... - -2. Suppose you need to debug a package deep within your software DAG. - If you build that package with a manual environment, then it - becomes difficult to have Spack auto-build things that depend on - it. That could be a serious problem, depending on how deep the - package in question is in your dependency DAG. - -3. At its core, Spack is a sophisticated concretization algorithm that - matches up packages with appropriate dependencies and creates a - *consistent* environment for the package it's building. Writing a - list of ``spack load`` commands for your dependencies is at least - as hard as writing the same list of ``depends_on()`` declarations - in a Spack package. But it makes no use of Spack concretization - and is more error-prone. - -4. Spack provides an automated, systematic way not just to find a - packages's dependencies --- but also to build other packages on - top. Any Spack package can become a dependency for another Spack - package, offering a powerful vision of software re-use. If you - build your package A outside of Spack, then your ability to use it - as a building block for other packages in an automated way is - diminished: other packages depending on package A will not - be able to use Spack to fulfill that dependency. - -5. If you are reading this manual, you probably love Spack. You're - probably going to write a Spack package for your software so - prospective users can install it with the least amount of pain. - Why should you go to additional work to find dependencies in your - development environment? Shouldn't Spack be able to help you build - your software based on the package you've already written? - -In this section, we show how Spack can be used in the software -development process to greatest effect, and how development packages -can be seamlessly integrated into the Spack ecosystem. We will show -how this process works by example, assuming the software you are -creating is called ``mylib``. - -^^^^^^^^^^^^^^^^^^^^^ -Write the CMake Build -^^^^^^^^^^^^^^^^^^^^^ - -For now, the techniques in this section only work for CMake-based -projects, although they could be easily extended to other build -systems in the future. We will therefore assume you are using CMake -to build your project. - -The ``CMakeLists.txt`` file should be written as normal. A few caveats: - -1. Your project should produce binaries with RPATHs. This will ensure - that they work the same whether built manually or automatically by - Spack. For example: - -.. code-block:: cmake - - # enable @rpath in the install name for any shared library being built - # note: it is planned that a future version of CMake will enable this by default - set(CMAKE_MACOSX_RPATH 1) - - # Always use full RPATH - # http://www.cmake.org/Wiki/CMake_RPATH_handling - # http://www.kitware.com/blog/home/post/510 - - # use, i.e. don't skip the full RPATH for the build tree - SET(CMAKE_SKIP_BUILD_RPATH FALSE) - - # when building, don't use the install RPATH already - # (but later on when installing) - SET(CMAKE_BUILD_WITH_INSTALL_RPATH FALSE) - - # add the automatically determined parts of the RPATH - # which point to directories outside the build tree to the install RPATH - SET(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE) - - # the RPATH to be used when installing, but only if it's not a system directory - LIST(FIND CMAKE_PLATFORM_IMPLICIT_LINK_DIRECTORIES "${CMAKE_INSTALL_PREFIX}/lib" isSystemDir) - IF("${isSystemDir}" STREQUAL "-1") - SET(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_PREFIX}/lib") - ENDIF("${isSystemDir}" STREQUAL "-1") - - -2. Spack provides a CMake variable called - ``SPACK_TRANSITIVE_INCLUDE_PATH``, which contains the ``include/`` - directory for all of your project's transitive dependencies. It - can be useful if your project ``#include``s files from package B, - which ``#include`` files from package C, but your project only - lists project B as a dependency. This works in traditional - single-tree build environments, in which B and C's include files - live in the same place. In order to make it work with Spack as - well, you must add the following to ``CMakeLists.txt``. It will - have no effect when building without Spack: - - .. code-block:: cmake - - # Include all the transitive dependencies determined by Spack. - # If we're not running with Spack, this does nothing... - include_directories($ENV{SPACK_TRANSITIVE_INCLUDE_PATH}) - - .. note:: - - Note that this feature is controversial and could break with - future versions of GNU ld. The best practice is to make sure - anything you ``#include`` is listed as a dependency in your - CMakeLists.txt (and Spack package). - -.. _write-the-spack-package: - -^^^^^^^^^^^^^^^^^^^^^^^ -Write the Spack Package -^^^^^^^^^^^^^^^^^^^^^^^ - -The Spack package also needs to be written, in tandem with setting up -the build (for example, CMake). The most important part of this task -is declaring dependencies. Here is an example of the Spack package -for the ``mylib`` package (ellipses for brevity): - -.. code-block:: python - - class Mylib(CMakePackage): - """Misc. reusable utilities used by Myapp.""" - - homepage = "https://github.com/citibeth/mylib" - url = "https://github.com/citibeth/mylib/tarball/123" - - version('0.1.2', '3a6acd70085e25f81b63a7e96c504ef9') - version('develop', git='https://github.com/citibeth/mylib.git', - branch='develop') - - variant('everytrace', default=False, - description='Report errors through Everytrace') - ... - - extends('python') - - depends_on('eigen') - depends_on('everytrace', when='+everytrace') - depends_on('proj', when='+proj') - ... - depends_on('cmake', type='build') - depends_on('doxygen', type='build') - - def cmake_args(self): - spec = self.spec - return [ - '-DUSE_EVERYTRACE=%s' % ('YES' if '+everytrace' in spec else 'NO'), - '-DUSE_PROJ4=%s' % ('YES' if '+proj' in spec else 'NO'), - ... - '-DUSE_UDUNITS2=%s' % ('YES' if '+udunits2' in spec else 'NO'), - '-DUSE_GTEST=%s' % ('YES' if '+googletest' in spec else 'NO')] - -This is a standard Spack package that can be used to install -``mylib`` in a production environment. The list of dependencies in -the Spack package will generally be a repeat of the list of CMake -dependencies. This package also has some features that allow it to be -used for development: - -1. It subclasses ``CMakePackage`` instead of ``Package``. This - eliminates the need to write an ``install()`` method, which is - defined in the superclass. Instead, one just needs to write the - ``configure_args()`` method. That method should return the - arguments needed for the ``cmake`` command (beyond the standard - CMake arguments, which Spack will include already). These - arguments are typically used to turn features on/off in the build. - -2. It specifies a non-checksummed version ``develop``. Running - ``spack install mylib@develop`` the ``@develop`` version will - install the latest version off the develop branch. This method of - download is useful for the developer of a project while it is in - active development; however, it should only be used by developers - who control and trust the repository in question! - -3. The ``url``, ``url_for_version()`` and ``homepage`` attributes are - not used in development. Don't worry if you don't have any, or if - they are behind a firewall. - -^^^^^^^^^^^^^^^^ -Build with Spack -^^^^^^^^^^^^^^^^ - -Now that you have a Spack package, you can use Spack to find its -dependencies automatically. For example: - -.. code-block:: console - - $ cd mylib - $ spack setup mylib@local - -The result will be a file ``spconfig.py`` in the top-level -``mylib/`` directory. It is a short script that calls CMake with the -dependencies and options determined by Spack --- similar to what -happens in ``spack install``, but now written out in script form. -From a developer's point of view, you can think of ``spconfig.py`` as -a stand-in for the ``cmake`` command. - -.. note:: - - You can invent any "version" you like for the ``spack setup`` - command. - -.. note:: - - Although ``spack setup`` does not build your package, it does - create and install a module file, and mark in the database that - your package has been installed. This can lead to errors, of - course, if you don't subsequently install your package. - Also... you will need to ``spack uninstall`` before you run - ``spack setup`` again. - - -You can now build your project as usual with CMake: - -.. code-block:: console - - $ mkdir build; cd build - $ ../spconfig.py .. # Instead of cmake .. - $ make - $ make install - -Once your ``make install`` command is complete, your package will be -installed, just as if you'd run ``spack install``. Except you can now -edit, re-build and re-install as often as needed, without checking -into Git or downloading tarballs. - -.. note:: - - The build you get this way will be *almost* the same as the build - from ``spack install``. The only difference is, you will not be - using Spack's compiler wrappers. This difference has not caused - problems in our experience, as long as your project sets - RPATHs as shown above. You DO use RPATHs, right? - -^^^^^^^^^^^^^^^^^^^^ -Build Other Software -^^^^^^^^^^^^^^^^^^^^ - -Now that you've built ``mylib`` with Spack, you might want to build -another package that depends on it --- for example, ``myapp``. This -is accomplished easily enough: - -.. code-block:: console - - $ spack install myapp ^mylib@local - -Note that auto-built software has now been installed *on top of* -manually-built software, without breaking Spack's "web." This -property is useful if you need to debug a package deep in the -dependency hierarchy of your application. It is a *big* advantage of -using ``spack setup`` to build your package's environment. - -If you feel your software is stable, you might wish to install it with -``spack install`` and skip the source directory. You can just use, -for example: - -.. code-block:: console - - $ spack install mylib@develop - -.. _release-your-software: - -^^^^^^^^^^^^^^^^^^^^^ -Release Your Software -^^^^^^^^^^^^^^^^^^^^^ - -You are now ready to release your software as a tarball with a -numbered version, and a Spack package that can build it. If you're -hosted on GitHub, this process will be a bit easier. - -#. Put tag(s) on the version(s) in your GitHub repo you want to be - release versions. For example, a tag ``v0.1.0`` for version 0.1.0. - -#. Set the ``url`` in your ``package.py`` to download a tarball for - the appropriate version. GitHub will give you a tarball for any - commit in the repo, if you tickle it the right way. For example: - - .. code-block:: python - - url = 'https://github.com/citibeth/mylib/tarball/v0.1.2' - -#. Use Spack to determine your version's hash, and cut'n'paste it into - your ``package.py``: - - .. code-block:: console - - $ spack checksum mylib 0.1.2 - ==> Found 1 versions of mylib - 0.1.2 https://github.com/citibeth/mylib/tarball/v0.1.2 - - How many would you like to checksum? (default is 5, q to abort) - ==> Downloading... - ==> Trying to fetch from https://github.com/citibeth/mylib/tarball/v0.1.2 - ######################################################################## 100.0% - ==> Checksummed new versions of mylib: - version('0.1.2', '3a6acd70085e25f81b63a7e96c504ef9') - -#. You should now be able to install released version 0.1.2 of your package with: - - .. code-block:: console - - $ spack install mylib@0.1.2 - -#. There is no need to remove the `develop` version from your package. - Spack concretization will always prefer numbered version to - non-numeric versions. Users will only get it if they ask for it. - -^^^^^^^^^^^^^^^^^^^^^^^^ -Distribute Your Software -^^^^^^^^^^^^^^^^^^^^^^^^ - -Once you've released your software, other people will want to build -it; and you will need to tell them how. In the past, that has meant a -few paragraphs of prose explaining which dependencies to install. But -now you use Spack, and those instructions are written in executable -Python code. But your software has many dependencies, and you know -Spack is the best way to install it: - -#. First, you will want to fork Spack's ``develop`` branch. Your aim - is to provide a stable version of Spack that you KNOW will install - your software. If you make changes to Spack in the process, you - will want to submit pull requests to Spack core. - -#. Add your software's ``package.py`` to that fork. You should submit - a pull request for this as well, unless you don't want the public - to know about your software. - -#. Prepare instructions that read approximately as follows: - - #. Download Spack from your forked repo. - - #. Install Spack; see :ref:`getting_started`. - - #. Set up an appropriate ``packages.yaml`` file. You should tell - your users to include in this file whatever versions/variants - are needed to make your software work correctly (assuming those - are not already in your ``packages.yaml``). - - #. Run ``spack install mylib``. - - #. Run this script to generate the ``module load`` commands or - filesystem view needed to use this software. - -#. Be aware that your users might encounter unexpected bootstrapping - issues on their machines, especially if they are running on older - systems. The :ref:`getting_started` section should cover this, but - there could always be issues. - -^^^^^^^^^^^^^^^^^^^ -Other Build Systems -^^^^^^^^^^^^^^^^^^^ - -``spack setup`` currently only supports CMake-based builds, in -packages that subclass ``CMakePackage``. The intent is that this -mechanism should support a wider range of build systems; for example, -GNU Autotools. Someone well-versed in Autotools is needed to develop -this patch and test it out. - -Python Distutils is another popular build system that should get -``spack setup`` support. For non-compiled languages like Python, -``spack diy`` may be used. Even better is to put the source directory -directly in the user's ``PYTHONPATH``. Then, edits in source files -are immediately available to run without any install process at all! - -^^^^^^^^^^ -Conclusion -^^^^^^^^^^ - -The ``spack setup`` development workflow provides better automation, -flexibility and safety than workflows relying on environment modules -or filesystem views. However, it has some drawbacks: - -#. It currently works only with projects that use the CMake build - system. Support for other build systems is not hard to build, but - will require a small amount of effort for each build system to be - supported. It might not work well with some IDEs. - -#. It only works with packages that sub-class ``StagedPackage``. - Currently, most Spack packages do not. Converting them is not - hard; but must be done on a package-by-package basis. - -#. It requires that users are comfortable with Spack, as they - integrate Spack explicitly in their workflow. Not all users are - willing to do this. - ------------------------------------- Using Spack to Replace Homebrew/Conda ------------------------------------- diff --git a/lib/spack/spack/cmd/setup.py b/lib/spack/spack/cmd/setup.py index 1d98e3ec4e..45919893cc 100644 --- a/lib/spack/spack/cmd/setup.py +++ b/lib/spack/spack/cmd/setup.py @@ -37,6 +37,7 @@ def setup_parser(subparser): cd_group = subparser.add_mutually_exclusive_group() arguments.add_common_arguments(cd_group, ['clean', 'dirty']) + subparser.epilog = 'DEPRECATED: use `spack dev-build` instead' def write_spconfig(package, dirty): @@ -98,6 +99,8 @@ env = dict(os.environ) def setup(self, args): + tty.warn('DEPRECATED: use `spack dev-build` instead') + if not args.spec: tty.die("spack setup requires a package spec argument.") diff --git a/lib/spack/spack/installer.py b/lib/spack/spack/installer.py index b8bbe7ce3f..401e2428b3 100644 --- a/lib/spack/spack/installer.py +++ b/lib/spack/spack/installer.py @@ -466,7 +466,6 @@ def log(pkg): packages_dir = spack.store.layout.build_packages_path(pkg.spec) # Remove first if we're overwriting another build - # (can happen with spack setup) try: # log and env install paths are inside this shutil.rmtree(packages_dir) |