1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
|
.. Copyright 2013-2022 Lawrence Livermore National Security, LLC and other
Spack Project Developers. See the top-level COPYRIGHT file for details.
SPDX-License-Identifier: (Apache-2.0 OR MIT)
.. _autotoolspackage:
----------------
AutotoolsPackage
----------------
Autotools is a GNU build system that provides a build-script generator.
By running the platform-independent ``./configure`` script that comes
with the package, you can generate a platform-dependent Makefile.
^^^^^^
Phases
^^^^^^
The ``AutotoolsPackage`` base class comes with the following phases:
#. ``autoreconf`` - generate the configure script
#. ``configure`` - generate the Makefiles
#. ``build`` - build the package
#. ``install`` - install the package
Most of the time, the ``autoreconf`` phase will do nothing, but if the
package is missing a ``configure`` script, ``autoreconf`` will generate
one for you.
The other phases run:
.. code-block:: console
$ ./configure --prefix=/path/to/installation/prefix
$ make
$ make check # optional
$ make install
$ make installcheck # optional
Of course, you may need to add a few arguments to the ``./configure``
line.
^^^^^^^^^^^^^^^
Important files
^^^^^^^^^^^^^^^
The most important file for an Autotools-based package is the ``configure``
script. This script is automatically generated by Autotools and generates
the appropriate Makefile when run.
.. warning::
Watch out for fake Autotools packages!
Autotools is a very popular build system, and many people are used to the
classic steps to install a package:
.. code-block:: console
$ ./configure
$ make
$ make install
For this reason, some developers will write their own ``configure``
scripts that have nothing to do with Autotools. These packages may
not accept the same flags as other Autotools packages, so it is
better to use the ``Package`` base class and create a
:ref:`custom build system <custompackage>`. You can tell if a package
uses Autotools by running ``./configure --help`` and comparing the output
to other known Autotools packages. You should also look for files like:
* ``configure.ac``
* ``configure.in``
* ``Makefile.am``
Packages that don't use Autotools aren't likely to have these files.
^^^^^^^^^^^^^^^^^^^^^^^^^
Build system dependencies
^^^^^^^^^^^^^^^^^^^^^^^^^
Whether or not your package requires Autotools to install depends on
how the source code is distributed. Most of the time, when developers
distribute tarballs, they will already contain the ``configure`` script
necessary for installation. If this is the case, your package does not
require any Autotools dependencies.
However, a basic rule of version control systems is to never commit
code that can be generated. The source code repository itself likely
does not have a ``configure`` script. Developers typically write
(or auto-generate) a ``configure.ac`` script that contains configuration
preferences and a ``Makefile.am`` script that contains build instructions.
Then, ``autoconf`` is used to convert ``configure.ac`` into ``configure``,
while ``automake`` is used to convert ``Makefile.am`` into ``Makefile.in``.
``Makefile.in`` is used by ``configure`` to generate a platform-dependent
``Makefile`` for you. The following diagram provides a high-level overview
of the process:
.. figure:: Autoconf-automake-process.*
:target: https://commons.wikimedia.org/w/index.php?curid=15581407
`GNU autoconf and automake process for generating makefiles <https://commons.wikimedia.org/wiki/File:Autoconf-automake-process.svg>`_
by `Jdthood` under `CC BY-SA 3.0 <https://creativecommons.org/licenses/by-sa/3.0/deed.en>`_
If a ``configure`` script is not present in your tarball, you will
need to generate one yourself. Luckily, Spack already has an ``autoreconf``
phase to do most of the work for you. By default, the ``autoreconf``
phase runs:
.. code-block:: console
$ autoreconf --install --verbose --force -I <aclocal-prefix>/share/aclocal
In case you need to add more arguments, override ``autoreconf_extra_args``
in your ``package.py`` on class scope like this:
.. code-block:: python
autoreconf_extra_args = ["-Im4"]
All you need to do is add a few Autotools dependencies to the package.
Most stable releases will come with a ``configure`` script, but if you
check out a commit from the ``master`` branch, you would want to add:
.. code-block:: python
depends_on('autoconf', type='build', when='@master')
depends_on('automake', type='build', when='@master')
depends_on('libtool', type='build', when='@master')
It is typically redundant to list the ``m4`` macro processor package as a
dependency, since ``autoconf`` already depends on it.
"""""""""""""""""""""""""""""""
Using a custom autoreconf phase
"""""""""""""""""""""""""""""""
In some cases, it might be needed to replace the default implementation
of the autoreconf phase with one running a script interpreter. In this
example, the ``bash`` shell is used to run the ``autogen.sh`` script.
.. code-block:: python
def autoreconf(self, spec, prefix):
which('bash')('autogen.sh')
"""""""""""""""""""""""""""""""""""""""
patching configure or Makefile.in files
"""""""""""""""""""""""""""""""""""""""
In some cases, developers might need to distribute a patch that modifies
one of the files used to generate ``configure`` or ``Makefile.in``.
In this case, these scripts will need to be regenerated. It is
preferable to regenerate these manually using the patch, and then
create a new patch that directly modifies ``configure``. That way,
Spack can use the secondary patch and additional build system
dependencies aren't necessary.
""""""""""""""""""""""""""""
Old Autotools helper scripts
""""""""""""""""""""""""""""
Autotools based tarballs come with helper scripts such as ``config.sub`` and
``config.guess``. It is the responsibility of the developers to keep these files
up to date so that they run on every platform, but for very old software
releases this is impossible. In these cases Spack can help to replace these
files with newer ones, without having to add the heavy dependency on
``automake``.
Automatic helper script replacement is currently enabled by default on
``ppc64le`` and ``aarch64``, as these are the known cases where old scripts fail.
On these targets, ``AutotoolsPackage`` adds a build dependency on ``gnuconfig``,
which is a very light-weight package with newer versions of the helper files.
Spack then tries to run all the helper scripts it can find in the release, and
replaces them on failure with the helper scripts from ``gnuconfig``.
To opt out of this feature, use the following setting:
.. code-block:: python
patch_config_files = False
To enable it conditionally on different architectures, define a property and
make the package depend on ``gnuconfig`` as a build dependency:
.. code-block
depends_on('gnuconfig', when='@1.0:')
@property
def patch_config_files(self):
return self.spec.satisfies("@1.0:")
.. note::
On some exotic architectures it is necessary to use system provided
``config.sub`` and ``config.guess`` files. In this case, the most
transparent solution is to mark the ``gnuconfig`` package as external and
non-buildable, with a prefix set to the directory containing the files:
.. code-block:: yaml
gnuconfig:
buildable: false
externals:
- spec: gnuconfig@master
prefix: /usr/share/configure_files/
""""""""""""""""
force_autoreconf
""""""""""""""""
If for whatever reason you really want to add the original patch
and tell Spack to regenerate ``configure``, you can do so using the
following setting:
.. code-block:: python
force_autoreconf = True
This line tells Spack to wipe away the existing ``configure`` script
and generate a new one. If you only need to do this for a single
version, this can be done like so:
.. code-block:: python
@property
def force_autoreconf(self):
return self.version == Version('1.2.3')
^^^^^^^^^^^^^^^^^^^^^^^
Finding configure flags
^^^^^^^^^^^^^^^^^^^^^^^
Once you have a ``configure`` script present, the next step is to
determine what option flags are available. These flags can be found
by running:
.. code-block:: console
$ ./configure --help
``configure`` will display a list of valid flags separated into
some or all of the following sections:
* Configuration
* Installation directories
* Fine tuning of the installation directories
* Program names
* X features
* System types
* **Optional Features**
* **Optional Packages**
* **Some influential environment variables**
For the most part, you can ignore all but the last 3 sections.
The "Optional Features" sections lists flags that enable/disable
features you may be interested in. The "Optional Packages" section
often lists dependencies and the flags needed to locate them. The
"environment variables" section lists environment variables that the
build system uses to pass flags to the compiler and linker.
^^^^^^^^^^^^^^^^^^^^^^^^^^
Addings flags to configure
^^^^^^^^^^^^^^^^^^^^^^^^^^
For most of the flags you encounter, you will want a variant to
optionally enable/disable them. You can then optionally pass these
flags to the ``configure`` call by overriding the ``configure_args``
function like so:
.. code-block:: python
def configure_args(self):
args = []
if '+mpi' in self.spec:
args.append('--enable-mpi')
else:
args.append('--disable-mpi')
return args
Note that we are explicitly disabling MPI support if it is not
requested. This is important, as many Autotools packages will enable
options by default if the dependencies are found, and disable them
otherwise. We want Spack installations to be as deterministic as possible.
If two users install a package with the same variants, the goal is that
both installations work the same way. See `here <https://www.linux.com/news/best-practices-autotools>`__
and `here <https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Automagic_dependencies>`__
for a rationale as to why these so-called "automagic" dependencies
are a problem.
By default, Autotools installs packages to ``/usr``. We don't want this,
so Spack automatically adds ``--prefix=/path/to/installation/prefix``
to your list of ``configure_args``. You don't need to add this yourself.
^^^^^^^^^^^^^^^^
Helper functions
^^^^^^^^^^^^^^^^
You may have noticed that most of the Autotools flags are of the form
``--enable-foo``, ``--disable-bar``, ``--with-baz=<prefix>``, or
``--without-baz``. Since these flags are so common, Spack provides a
couple of helper functions to make your life easier.
"""""""""""""""""
enable_or_disable
"""""""""""""""""
Autotools flags for simple boolean variants can be automatically
generated by calling the ``enable_or_disable`` method. This is
typically used to enable or disable some feature within the package.
.. code-block:: python
variant(
'memchecker',
default=False,
description='Memchecker support for debugging [degrades performance]'
)
config_args.extend(self.enable_or_disable('memchecker'))
In this example, specifying the variant ``+memchecker`` will generate
the following configuration options:
.. code-block:: console
--enable-memchecker
"""""""""""""""
with_or_without
"""""""""""""""
Autotools flags for more complex variants, including boolean variants
and multi-valued variants, can be automatically generated by calling
the ``with_or_without`` method.
.. code-block:: python
variant(
'schedulers',
values=disjoint_sets(
('auto',), ('alps', 'lsf', 'tm', 'slurm', 'sge', 'loadleveler')
).with_non_feature_values('auto', 'none'),
description="List of schedulers for which support is enabled; "
"'auto' lets openmpi determine",
)
if 'schedulers=auto' not in spec:
config_args.extend(self.with_or_without('schedulers'))
In this example, specifying the variant ``schedulers=slurm,sge`` will
generate the following configuration options:
.. code-block:: console
--with-slurm --with-sge
``enable_or_disable`` is actually functionally equivalent with
``with_or_without``, and accepts the same arguments and variant types;
but idiomatic autotools packages often follow these naming
conventions.
""""""""""""""""
activation_value
""""""""""""""""
Autotools parameters that require an option can still be automatically
generated, using the ``activation_value`` argument to
``with_or_without`` (or, rarely, ``enable_or_disable``).
.. code-block:: python
variant(
'fabrics',
values=disjoint_sets(
('auto',), ('psm', 'psm2', 'verbs', 'mxm', 'ucx', 'libfabric')
).with_non_feature_values('auto', 'none'),
description="List of fabrics that are enabled; "
"'auto' lets openmpi determine",
)
if 'fabrics=auto' not in spec:
config_args.extend(self.with_or_without('fabrics',
activation_value='prefix'))
``activation_value`` accepts a callable that generates the configure
parameter value given the variant value; but the special value
``prefix`` tells Spack to automatically use the dependenency's
installation prefix, which is the most common use for such
parameters. In this example, specifying the variant
``fabrics=libfabric`` will generate the following configuration
options:
.. code-block:: console
--with-libfabric=</path/to/libfabric>
"""""""""""""""""""""""
The ``variant`` keyword
"""""""""""""""""""""""
When Spack variants and configure flags do not correspond one-to-one, the
``variant`` keyword can be passed to ``with_or_without`` and
``enable_or_disable``. For example:
.. code-block:: python
variant('debug_tools', default=False)
config_args += self.enable_or_disable('debug-tools', variant='debug_tools')
Or when one variant controls multiple flags:
.. code-block:: python
variant('debug_tools', default=False)
config_args += self.with_or_without('memchecker', variant='debug_tools')
config_args += self.with_or_without('profiler', variant='debug_tools')
""""""""""""""""""""
Conditional variants
""""""""""""""""""""
When a variant is conditional and its condition is not met on the concrete spec, the
``with_or_without`` and ``enable_or_disable`` methods will simply return an empty list.
For example:
.. code-block:: python
variant('profiler', when='@2.0:')
config_args += self.with_or_without('profiler)
will neither add ``--with-profiler`` nor ``--without-profiler`` when the version is
below ``2.0``.
""""""""""""""""""""
Activation overrides
""""""""""""""""""""
Finally, the behavior of either ``with_or_without`` or
``enable_or_disable`` can be overridden for specific variant
values. This is most useful for multi-values variants where some of
the variant values require atypical behavior.
.. code-block:: python
def with_or_without_verbs(self, activated):
# Up through version 1.6, this option was named --with-openib.
# In version 1.7, it was renamed to be --with-verbs.
opt = 'verbs' if self.spec.satisfies('@1.7:') else 'openib'
if not activated:
return '--without-{0}'.format(opt)
return '--with-{0}={1}'.format(opt, self.spec['rdma-core'].prefix)
Defining ``with_or_without_verbs`` overrides the behavior of a
``fabrics=verbs`` variant, changing the configure-time option to
``--with-openib`` for older versions of the package and specifying an
alternative dependency name:
.. code-block::
--with-openib=</path/to/rdma-core>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Configure script in a sub-directory
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Occasionally, developers will hide their source code and ``configure``
script in a subdirectory like ``src``. If this happens, Spack won't
be able to automatically detect the build system properly when running
``spack create``. You will have to manually change the package base
class and tell Spack where the ``configure`` script resides. You can
do this like so:
.. code-block:: python
configure_directory = 'src'
^^^^^^^^^^^^^^^^^^^^^^
Building out of source
^^^^^^^^^^^^^^^^^^^^^^
Some packages like ``gcc`` recommend building their software in a
different directory than the source code to prevent build pollution.
This can be done using the ``build_directory`` variable:
.. code-block:: python
build_directory = 'spack-build'
By default, Spack will build the package in the same directory that
contains the ``configure`` script
^^^^^^^^^^^^^^^^^^^^^^^^^
Build and install targets
^^^^^^^^^^^^^^^^^^^^^^^^^
For most Autotools packages, the usual:
.. code-block:: console
$ configure
$ make
$ make install
is sufficient to install the package. However, if you need to run
make with any other targets, for example, to build an optional
library or build the documentation, you can add these like so:
.. code-block:: python
build_targets = ['all', 'docs']
install_targets = ['install', 'docs']
^^^^^^^
Testing
^^^^^^^
Autotools-based packages typically provide unit testing via the
``check`` and ``installcheck`` targets. If you build your software
with ``spack install --test=root``, Spack will check for the presence
of a ``check`` or ``test`` target in the Makefile and run
``make check`` for you. After installation, it will check for an
``installcheck`` target and run ``make installcheck`` if it finds one.
^^^^^^^^^^^^^^^^^^^^^^
External documentation
^^^^^^^^^^^^^^^^^^^^^^
For more information on the Autotools build system, see:
https://www.gnu.org/software/automake/manual/html_node/Autotools-Introduction.html
|