diff options
Diffstat (limited to 'lib/spack/docs/tutorial_sc16_packaging.rst')
| -rw-r--r-- | lib/spack/docs/tutorial_sc16_packaging.rst | 463 |
1 files changed, 463 insertions, 0 deletions
diff --git a/lib/spack/docs/tutorial_sc16_packaging.rst b/lib/spack/docs/tutorial_sc16_packaging.rst new file mode 100644 index 0000000000..41c938327e --- /dev/null +++ b/lib/spack/docs/tutorial_sc16_packaging.rst @@ -0,0 +1,463 @@ +.. _packaging-tutorial: + +========================= +Package Creation Tutorial +========================= + +This tutorial will walk you through the steps behind building a simple +package installation script. We'll focus building an mpileaks package, +which is a MPI debugging tool. By creating a package file we're +essentially giving Spack a recipe for how to build a particular piece of +software. We're describing some of the software's dependencies, where to +find the package, what commands and options are used to build the package +from source, and more. Once we've specified a package's recipe, we can +ask Spack to build that package in many different ways. + +This tutorial assumes you have a basic familiarity with some of the Spack +commands, and that you have a working version of Spack installed. If +not, we suggest looking at Spack's *Getting Started* guide. This +tutorial also assumes you have at least a beginner's-level familiarity +with Python. + +Also note that this document is a tutorial. It can help you get started +with packaging, but is not intended to be complete. See Spack's +:ref:`packaging-guide` for more complete documentation on this topic. + +--------------- +Getting Started +--------------- + +A few things before we get started: + +- We'll refer to the Spack installation location via the environment + variable ``SPACK_ROOT``. You should point ``SPACK_ROOT`` at wherever + you have Spack installed. +- Add ``$SPACK_ROOT/bin`` to your ``PATH`` before you start. +- Make sure your ``EDITOR`` environment variable is set to some text + editor you like. +- We'll be writting Python code as part of this tutorial. You can find + successive versions of the Python code in + ``$SPACK_ROOT/lib/spack/docs/tutorial/examples``. + +------------------------- +Creating the Package File +------------------------- + +Spack comes with a handy command to create a new package: ``spack create`` + +This command is given the location of a package's source code, downloads +the code, and sets up some basic packaging infrastructure for you. The +mpileaks source code can be found on github, and here's what happens when +we run ``spack create`` on it: + +.. code-block:: console + + % spack create -f https://github.com/hpc/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz + ==> This looks like a URL for mpileaks version 1.0 + ==> Creating template for package mpileaks + ==> Downloading... + ==> Fetching https://github.com/hpc/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz + ###################################################################################### 100.0% + +And Spack should spawn a text editor with this file: + +.. literalinclude:: tutorial/examples/0.package.py + :start-after: # flake8: noqa + :language: python + +Spack has created this file in +``$SPACK_ROOT/var/spack/repos/builtin/packages/mpileaks/package.py``. Take a +moment to look over the file. There's a few placeholders that Spack has +created, which we'll fill in as part of this tutorial: + +- We'll document some information about this package in the comments. +- We'll fill in the dependency list for this package. +- We'll fill in some of the configuration arguments needed to build this + package. + +For the moment, exit your editor and let's see what happens when we try +to build this package: + +.. code-block:: console + + % spack install mpileaks + ==> Installing mpileaks + ==> Using cached archive: /usr/workspace/wsa/legendre/spack/var/spack/cache/mpileaks/mpileaks-1.0.tar.gz + ==> Staging archive: /usr/workspace/wsa/legendre/spack/var/spack/stage/mpileaks-1.0-hufwhwpq5benv3sslie6ryflk5s6nm35/mpileaks-1.0.tar.gz + ==> Created stage in /usr/workspace/wsa/legendre/spack/var/spack/stage/mpileaks-1.0-hufwhwpq5benv3sslie6ryflk5s6nm35 + ==> Ran patch() for mpileaks + ==> Building mpileaks [AutotoolsPackage] + ==> Executing phase : 'autoreconf' + ==> Executing phase : 'configure' + ==> Error: ProcessError: Command exited with status 1: + './configure' '--prefix=/usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/mpileaks-1.0-hufwhwpq5benv3sslie6ryflk5s6nm35' + /usr/workspace/wsa/legendre/spack/lib/spack/spack/build_systems/autotools.py:150, in configure: + 145 def configure(self, spec, prefix): + 146 """Runs configure with the arguments specified in `configure_args` + 147 and an appropriately set prefix + 148 """ + 149 options = ['--prefix={0}'.format(prefix)] + self.configure_args() + >> 150 inspect.getmodule(self).configure(*options) + + See build log for details: + /tmp/legendre/spack-stage/spack-stage-8HVzqu/mpileaks-1.0/spack-build.out + +This obviously didn't work; we need to fill in the package-specific +information. Specifically, Spack didn't try to build any of mpileaks' +dependencies, nor did it use the proper configure arguments. Let's start +fixing things + +--------------------- +Package Documentation +--------------------- + +We can bring the ``package.py`` file back into our ``EDITOR`` with the +``spack edit`` command: + +.. code-block:: console + + % spack edit mpileaks + +Let's remove some of the ``TODO`` comments, and add links to the mpileaks +homepage and document what mpileaks does. I'm also going to cut out the +Copyright clause at this point to keep this tutorial document shorter, +but you shouldn't do that normally. The results of these changes can be +found in ``$SPACK_ROOT/lib/spack/docs/tutorial/examples/1.package.py`` +and are below. Make these changes to your ``package.py``: + +.. literalinclude:: tutorial/examples/1.package.py + :start-after: # flake8: noqa + :language: python + +We've filled in the comment that describes what this package does and +added a link to the web site. That won't help us build yet, but it will +allow Spack to provide some documentation on this package to other users: + +.. code-block:: console + + % spack info mpileaks + AutotoolsPackage: mpileaks + Homepage: https://github.com/hpc/mpileaks + + Safe versions: + 1.0 https://github.com/hpc/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz + + Variants: + None + + Installation Phases: + autoreconf configure build install + + Build Dependencies: + None + + Link Dependencies: + None + + Run Dependencies: + None + + Virtual Packages: + None + + Description: + Tool to detect and report MPI objects like MPI_Requests and + MPI_Datatypes + +As we fill in more information about this package the spack info command +will become more informative. Now let's start making this package build. + +------------ +Dependencies +------------ + +The mpileaks packages depends on three other package: ``MPI``, +``adept-utils``, and ``callpath``. Let's add those via the +``depends_on`` command in our ``package.py`` (this version is in +``$SPACK_ROOT/lib/spack/docs/tutorial/examples/2.package.py``): + +.. literalinclude:: tutorial/examples/2.package.py + :start-after: # flake8: noqa + :language: python + +Now when we go to build mpileaks, Spack will fetch and build these +dependencies before building mpileaks. Note that the mpi dependency is a +different kind of beast than the adept-utils and callpath dependencies; +there is no mpi package available in Spack. Instead mpi is a virtual +dependency. Spack may satisfy that dependency by installing packages +such as ``openmpi`` or ``mvapich``. See the :ref:`packaging-guide` for more +information on virtual dependencies. + +Now when we try to install this package a lot more happens: + +.. code-block:: console + + % spack install mpileaks + ==> Installing mpileaks + ==> openmpi is already installed in /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/openmpi-2.0.1-5ee5j34c2y4kb5c3joygrgahidqnwhnz + ==> callpath is already installed in /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/callpath-1.0.2-zm4pf3gasgxeibyu2y262suktvaazube + ==> adept-utils is already installed in /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/adept-utils-1.0.1-7p7ezxwtajdglj6cmojy2vybjct4j4jz + ==> Using cached archive: /usr/workspace/wsa/legendre/spack/var/spack/cache/mpileaks/mpileaks-1.0.tar.gz + ==> Already staged mpileaks-1.0-eum4hmnlt6ovalwjnciaygfb3beja4gk in /usr/workspace/wsa/legendre/spack/var/spack/stage/mpileaks-1.0-eum4hmnlt6ovalwjnciaygfb3beja4gk + ==> Already patched mpileaks + ==> Building mpileaks [AutotoolsPackage] + ==> Executing phase : 'autoreconf' + ==> Executing phase : 'configure' + ==> Error: ProcessError: Command exited with status 1: + './configure' '--prefix=/usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/mpileaks-1.0-eum4hmnlt6ovalwjnciaygfb3beja4gk' + /usr/workspace/wsa/legendre/spack/lib/spack/spack/build_systems/autotools.py:150, in configure: + 145 def configure(self, spec, prefix): + 146 """Runs configure with the arguments specified in `configure_args` + 147 and an appropriately set prefix + 148 """ + 149 options = ['--prefix={0}'.format(prefix)] + self.configure_args() + >> 150 inspect.getmodule(self).configure(*options) + + See build log for details: + /tmp/legendre/spack-stage/spack-stage-7V5yyk/mpileaks-1.0/spack-build.out + +Note that this command may take a while to run and produce more output if +you don't have an MPI already installed or configured in Spack. + +Now Spack has identified and made sure all of our dependencies have been +built. It found the ``openmpi`` package that will satisfy our ``mpi`` +dependency, and the callpath and ``adept-utils`` package to satisfy our +concrete dependencies. + +------------------------ +Debugging Package Builds +------------------------ + +Our ``mpileaks`` package is still not building. It may be obvious to +many of you that we're still missing the configure options. But let's +pretend we're not all intelligent developers and use this opportunity +spend some time debugging. We a few options that can tell us about +what's going wrong: + +As per the error message, Spack has given us a ``spack-build.out`` debug log: + +.. code-block:: console + + ==> './configure' '--prefix=/usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/mpileaks-1.0-eum4hmnlt6ovalwjnciaygfb3beja4gk' + checking metadata... no + checking installation directory variables... yes + checking for a BSD-compatible install... /usr/bin/install -c + checking whether build environment is sane... yes + checking for a thread-safe mkdir -p... /usr/bin/mkdir -p + checking for gawk... gawk + checking whether make sets $(MAKE)... yes + checking for gcc... /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc + checking for C compiler default output file name... a.out + checking whether the C compiler works... yes + checking whether we are cross compiling... no + checking for suffix of executables... + checking for suffix of object files... o + checking whether we are using the GNU C compiler... yes + checking whether /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc accepts -g... yes + checking for /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc option to accept ISO C89... none needed + checking for style of include used by make... GNU + checking dependency style of /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc... gcc3 + checking whether /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc and cc understand -c and -o together... yes + checking whether we are using the GNU C++ compiler... yes + checking whether /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/g++ accepts -g... yes + checking dependency style of /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/g++... gcc3 + checking for /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/openmpi-2.0.1-5ee5j34c2y4kb5c3joygrgahidqnwhnz/bin/mpicc... /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/openmpi-2.0.1-5ee5j34c2y4kb5c3joygrgahidqnwhnz/bin/mpicc + Checking whether /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/openmpi-2.0.1-5ee5j34c2y4kb5c3joygrgahidqnwhnz/bin/mpicc responds to '-showme:compile'... yes + configure: error: unable to locate ``adept-utils`` installation + +This gives us the output from the build, and it's fairly obvious that +mpileaks isn't finding it's ``adept-utils`` package. Spack has +automatically added the include and library directories of +``adept-utils`` to the compiler's search path, but some packages like +mpileaks can sometimes be picky and still want things spelled out on +their command line. But let's continue to pretend we're not brilliant +developers, and explore some other debugging paths: + +We can also enter the build area and try to manually run the build: + +.. code-block:: console + + % spack env mpileaks tcsh + % spack cd mpileaks + +The ``spack env`` command spawned a new shell that contains the same +environment that Spack used to build the mpileaks package (you can +substitute tcsh for your favorite shell). The ``spack cd`` command +changed our working dirctory to the last attempted build for mpileaks. +From here we can manually re-run the build: + +.. code-block:: console + + % ./configure + checking metadata... no + checking installation directory variables... yes + checking for a BSD-compatible install... /usr/bin/install -c + checking whether build environment is sane... yes + checking for a thread-safe mkdir -p... /usr/bin/mkdir -p + checking for gawk... gawk + checking whether make sets $(MAKE)... yes + checking for gcc... /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc + checking for C compiler default output file name... a.out + checking whether the C compiler works... yes + checking whether we are cross compiling... no + checking for suffix of executables... + checking for suffix of object files... o + checking whether we are using the GNU C compiler... yes + checking whether /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc accepts -g... yes + checking for /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc option to accept ISO C89... none needed + checking for style of include used by make... GNU + checking dependency style of /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc... gcc3 + checking whether /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/gcc and cc understand -c and -o together... yes + checking whether we are using the GNU C++ compiler... yes + checking whether /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/g++ accepts -g... yes + checking dependency style of /usr/workspace/wsa/legendre/spack/lib/spack/env/gcc/g++... gcc3 + checking for /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/openmpi-2.0.1-5ee5j34c2y4kb5c3joygrgahidqnwhnz/bin/mpicc... /usr/workspace/wsa /legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/openmpi-2.0.1-5ee5j34c2y4kb5c3joygrgahidqnwhnz/bin/mpicc + Checking whether /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/openmpi-2.0.1-5ee5j34c2y4kb5c3joygrgahidqnwhnz/bin/mpicc responds to '-showme:compile'... yes + configure: error: unable to locate adept-utils installation + +We're seeing the same error, but now we're in a shell where we can run +the command ourselves and debug as needed. We could, for example, run +``./configure --help`` to see what options we can use to specify +dependencies. + +We can use the ``exit`` command to leave the shell spawned by ``spack +env``. + +------------------------------ +Specifying Configure Arguments +------------------------------ + +Let's add the configure arguments to the mpileaks' ``package.py``. This +version can be found in +``$SPACK_ROOT/lib/spack/docs/tutorial/examples/3.package.py``: + +.. literalinclude:: tutorial/examples/3.package.py + :start-after: # flake8: noqa + :language: python + +This is all we need for working mpileaks! If we install now we'll see: + +.. code-block:: console + + % spack install mpileaks + spack install mpileaks + ==> Installing mpileaks + ==> openmpi is already installed in /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/openmpi-2.0.1-5ee5j34c2y4kb5c3joygrgahidqnwhnz + ==> callpath is already installed in /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/callpath-1.0.2-zm4pf3gasgxeibyu2y262suktvaazube + ==> adept-utils is already installed in /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/adept-utils-1.0.1-7p7ezxwtajdglj6cmojy2vybjct4j4jz + ==> Using cached archive: /usr/workspace/wsa/legendre/spack/var/spack/cache/mpileaks/mpileaks-1.0.tar.gz + ==> Already staged mpileaks-1.0-eum4hmnlt6ovalwjnciaygfb3beja4gk in /usr/workspace/wsa/legendre/spack/var/spack/stage/mpileaks-1.0-eum4hmnlt6ovalwjnciaygfb3beja4gk + ==> Already patched mpileaks + ==> Building mpileaks [AutotoolsPackage] + ==> Executing phase : 'autoreconf' + ==> Executing phase : 'configure' + ==> Executing phase : 'build' + ==> Executing phase : 'install' + ==> Successfully installed mpileaks + Fetch: 0.00s. Build: 14.08s. Total: 14.08s. + [+] /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/mpileaks-1.0-eum4hmnlt6ovalwjnciaygfb3beja4gk + +We took a few shortcuts for this package that are worth highlighting. +Spack automatically detected that mpileaks was an autotools-based package +when we ran spack create. If this had been a CMake-based package we +would have been filling in a ``cmake_args`` function instead of +``configure_args``. If spack hadn't been able to detect the build +system, we'd be filling in a generic install method that would manually +be calling build commands, such as is found in the ``zlib`` package: + +.. code-block:: python + + def install(self, spec, prefix): + configure('--prefix={0}'.format(prefix)) + make() + make('install') + +-------- +Variants +-------- + +We have a successful mpileaks build, but let's take some time to improve +it. ``mpileaks`` has a build-time option to truncate parts of the stack +that it walks. Let's add a variant to allow users to set this when they +build in Spack. + +To do this, we'll add a variant to our package, as per the following (see +``$SPACK_ROOT/lib/spack/docs/tutorial/examples/4.package.py``): + +.. literalinclude:: tutorial/examples/4.package.py + :start-after: # flake8: noqa + :language: python + +We've added the variant ``stackstart``, and given it a default value of +``0``. If we install now we can see the stackstart variant added to the +configure line (output truncated for length): + +.. code-block:: console + + % spack install --verbose mpileaks stackstart=4 + ==> Installing mpileaks + ==> openmpi is already installed in /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/openmpi-2.0.1-5ee5j34c2y4kb5c3joygrgahidqnwhnz + ==> callpath is already installed in /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/callpath-1.0.2-zm4pf3gasgxeibyu2y262suktvaazube + ==> adept-utils is already installed in /usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/adept-utils-1.0.1-7p7ezxwtajdglj6cmojy2vybjct4j4jz + ==> Using cached archive: /usr/workspace/wsa/legendre/spack/var/spack/cache/mpileaks/mpileaks-1.0.tar.gz + ==> Staging archive: /usr/workspace/wsa/legendre/spack/var/spack/stage/mpileaks-1.0-otqo2opkhan5ksujt6tpmdftydrieig7/mpileaks-1.0.tar.gz + ==> Created stage in /usr/workspace/wsa/legendre/spack/var/spack/stage/mpileaks-1.0-otqo2opkhan5ksujt6tpmdftydrieig7 + ==> Ran patch() for mpileaks + ==> Building mpileaks [AutotoolsPackage] + ==> Executing phase : 'autoreconf' + ==> Executing phase : 'configure' + ==> './configure' '--prefix=/usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/mpileaks-1.0-otqo2opkhan5ksujt6tpmdftydrieig7' '--with-adept-utils=/usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/adept-utils-1.0.1-7p7ezxwtajdglj6cmojy2vybjct4j4jz' '--with-callpath=/usr/workspace/wsa/legendre/spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/callpath-1.0.2-zm4pf3gasgxeibyu2y262suktvaazube' '--with-stack-start-c=4' '--with-stack-start-fortran=4' + +--------------- +The Spec Object +--------------- + +This tutorial has glossed over a few important features, which weren't +too relevant for mpleaks but may be useful for other packages. There +were several places we references the ``self.spec`` object. This is a +powerful class for querying information about what we're building. For +example, you could use the spec to query information about how a +package's dependencies were built, or what compiler was being used, or +what version of a package is being installed. Full documentation can be +found in the :ref:`packaging-guide`, but here's some quick snippets with +common queries: + +- Am I building ``mpileaks`` version ``1.1`` or greater? + +.. code-block:: python + + if self.spec.satisfies('@1.1:'): + # Do things needed for 1.1+ + +- Is ``openmpi`` the MPI I'm building with? + +.. code-block:: python + + if self.spec['mpi'].name == 'openmpi': + # Do openmpi things + +- Am I building with ``gcc`` version less than ``5.0.0``: + +.. code-block:: python + + if self.spec.satisfies('%gcc@:5.0.0'): + # Add arguments specific to gcc's earlier than 5.0.0 + +- Am I built with the ``debug`` variant: + +.. code-block:: python + + if self.spec.satisfies('+debug'): + # Add -g option to configure flags + +- Is my ``dyninst`` dependency greater than version ``8.0``? + +.. code-block:: python + + if self.spec['dyninst'].satisfies('@8.0:'): + # Use newest dyninst options + +More examples can be found in the thousands of packages already added to +Spack in ``$SPACK_ROOT/var/spack/repos/builtin/packages``. + +Good Luck! |