1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
|
.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
Spack Project Developers. See the top-level COPYRIGHT file for details.
SPDX-License-Identifier: (Apache-2.0 OR MIT)
.. _repositories:
=============================
Package Repositories
=============================
Spack comes with over 1,000 built-in package recipes in
``var/spack/repos/builtin/``. This is a **package repository** -- a
directory that Spack searches when it needs to find a package by name.
You may need to maintain packages for restricted, proprietary or
experimental software separately from the built-in repository. Spack
allows you to configure local repositories using either the
``repos.yaml`` or the ``spack repo`` command.
A package repository a directory structured like this::
repo/
repo.yaml
packages/
hdf5/
package.py
mpich/
package.py
mpich-1.9-bugfix.patch
trilinos/
package.py
...
The top-level ``repo.yaml`` file contains configuration metadata for the
repository, and the ``packages`` directory contains subdirectories for
each package in the repository. Each package directory contains a
``package.py`` file and any patches or other files needed to build the
package.
Package repositories allow you to:
1. Maintain your own packages separately from Spack;
2. Share your packages (e.g., by hosting them in a shared file system),
without committing them to the built-in Spack package repository; and
3. Override built-in Spack packages with your own implementation.
Packages in a separate repository can also *depend on* built-in Spack
packages. So, you can leverage existing recipes without re-implementing
them in your own repository.
---------------------
``repos.yaml``
---------------------
Spack uses the ``repos.yaml`` file in ``~/.spack`` (and :ref:`elsewhere
<configuration>`) to find repositories. Note that the ``repos.yaml``
configuration file is distinct from the ``repo.yaml`` file in each
repository. For more on the YAML format, and on how configuration file
precedence works in Spack, see :ref:`configuration <configuration>`.
The default ``etc/spack/defaults/repos.yaml`` file looks like this:
.. code-block:: yaml
repos:
- $spack/var/spack/repos/builtin
The file starts with ``repos:`` and contains a single ordered list of
paths to repositories. Each path is on a separate line starting with
``-``. You can add a repository by inserting another path into the list:
.. code-block:: yaml
repos:
- /opt/local-repo
- $spack/var/spack/repos/builtin
When Spack interprets a spec, e.g., ``mpich`` in ``spack install mpich``,
it searches these repositories in order (first to last) to resolve each
package name. In this example, Spack will look for the following
packages and use the first valid file:
1. ``/opt/local-repo/packages/mpich/package.py``
2. ``$spack/var/spack/repos/builtin/packages/mpich/package.py``
.. note::
Currently, Spack can only use repositories in the file system. We plan
to eventually support URLs in ``repos.yaml``, so that you can easily
point to remote package repositories, but that is not yet implemented.
---------------------
Namespaces
---------------------
Every repository in Spack has an associated **namespace** defined in its
top-level ``repo.yaml`` file. If you look at
``var/spack/repos/builtin/repo.yaml`` in the built-in repository, you'll
see that its namespace is ``builtin``:
.. code-block:: console
$ cat var/spack/repos/builtin/repo.yaml
repo:
namespace: builtin
Spack records the repository namespace of each installed package. For
example, if you install the ``mpich`` package from the ``builtin`` repo,
Spack records its fully qualified name as ``builtin.mpich``. This
accomplishes two things:
1. You can have packages with the same name from different namespaces
installed at once.
1. You can easily determine which repository a package came from after it
is installed (more :ref:`below <namespace-example>`).
.. note::
It may seem redundant for a repository to have both a namespace and a
path, but repository *paths* may change over time, or, as mentioned
above, a locally hosted repository path may eventually be hosted at
some remote URL.
Namespaces are designed to allow *package authors* to associate a
unique identifier with their packages, so that the package can be
identified even if the repository moves. This is why the namespace is
determined by the ``repo.yaml`` file in the repository rather than the
local ``repos.yaml`` configuration: the *repository maintainer* sets
the name.
^^^^^^^^^^^^^^^^^^^^^^^^^^^
Uniqueness
^^^^^^^^^^^^^^^^^^^^^^^^^^^
You should choose a namespace that uniquely identifies your package
repository. For example, if you make a repository for packages written
by your organization, you could use your organization's name. You can
also nest namespaces using periods, so you could identify a repository by
a sub-organization. For example, LLNL might use a namespace for its
internal repositories like ``llnl``. Packages from the Physical & Life
Sciences directorate (PLS) might use the ``llnl.pls`` namespace, and
packages created by the Computation directorate might use ``llnl.comp``.
Spack cannot ensure that every repository is named uniquely, but it will
prevent you from registering two repositories with the same namespace at
the same time. If you try to add a repository that has the same name as
an existing one, e.g., ``builtin``, Spack will print a warning message.
.. _namespace-example:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Namespace example
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Suppose that LLNL maintains its own version of ``mpich``, separate from
Spack's built-in ``mpich`` package, and suppose you've installed both
LLNL's and Spack's ``mpich`` packages. If you just use ``spack find``,
you won't see a difference between these two packages:
.. code-block:: console
$ spack find
==> 2 installed packages.
-- linux-rhel6-x86_64 / gcc@4.4.7 -------------
mpich@3.2 mpich@3.2
However, if you use ``spack find -N``, Spack will display the packages
with their namespaces:
.. code-block:: console
$ spack find -N
==> 2 installed packages.
-- linux-rhel6-x86_64 / gcc@4.4.7 -------------
builtin.mpich@3.2 llnl.comp.mpich@3.2
Now you know which one is LLNL's special version, and which one is the
built-in Spack package. As you might guess, packages that are identical
except for their namespace will still have different hashes:
.. code-block:: console
$ spack find -lN
==> 2 installed packages.
-- linux-rhel6-x86_64 / gcc@4.4.7 -------------
c35p3gc builtin.mpich@3.2 itoqmox llnl.comp.mpich@3.2
All Spack commands that take a package :ref:`spec <sec-specs>` can also
accept a fully qualified spec with a namespace. This means you can use
the namespace to be more specific when designating, e.g., which package
you want to uninstall:
.. code-block:: console
spack uninstall llnl.comp.mpich
----------------------------
Overriding built-in packages
----------------------------
Spack's search semantics mean that you can make your own implementation
of a built-in Spack package (like ``mpich``), put it in a repository, and
use it to override the built-in package. As long as the repository
containing your ``mpich`` is earlier any other in ``repos.yaml``, any
built-in package that depends on ``mpich`` will be use the one in your
repository.
Suppose you have three repositories: the builtin Spack repo
(``builtin``), a shared repo for your institution (e.g., ``llnl``), and a
repo containing your own prototype packages (``proto``). Suppose they
contain packages as follows:
+--------------+------------------------------------+-----------------------------+
| Namespace | Path to repo | Packages |
+==============+====================================+=============================+
| ``proto`` | ``~/proto`` | ``mpich`` |
+--------------+------------------------------------+-----------------------------+
| ``llnl`` | ``/usr/local/llnl`` | ``hdf5`` |
+--------------+------------------------------------+-----------------------------+
| ``builtin`` | ``$spack/var/spack/repos/builtin`` | ``mpich``, ``hdf5``, others |
+--------------+------------------------------------+-----------------------------+
Suppose that ``hdf5`` depends on ``mpich``. You can override the
built-in ``hdf5`` by adding the ``llnl`` repo to ``repos.yaml``:
.. code-block:: yaml
repos:
- /usr/local/llnl
- $spack/var/spack/repos/builtin
``spack install hdf5`` will install ``llnl.hdf5 ^builtin.mpich``.
If, instead, ``repos.yaml`` looks like this:
.. code-block:: yaml
repos:
- ~/proto
- /usr/local/llnl
- $spack/var/spack/repos/builtin
``spack install hdf5`` will install ``llnl.hdf5 ^proto.mpich``.
Any unqualified package name will be resolved by searching ``repos.yaml``
from the first entry to the last. You can force a particular
repository's package by using a fully qualified name. For example, if
your ``repos.yaml`` is as above, and you want ``builtin.mpich`` instead
of ``proto.mpich``, you can write::
spack install hdf5 ^builtin.mpich
which will install ``llnl.hdf5 ^builtin.mpich``.
Similarly, you can force the ``builtin.hdf5`` like this::
spack install builtin.hdf5 ^builtin.mpich
This will not search ``repos.yaml`` at all, as the ``builtin`` repo is
specified in both cases. It will install ``builtin.hdf5
^builtin.mpich``.
If you want to see which repositories will be used in a build *before*
you install it, you can use ``spack spec -N``:
.. code-block:: console
$ spack spec -N hdf5
Input spec
--------------------------------
hdf5
Normalized
--------------------------------
hdf5
^zlib@1.1.2:
Concretized
--------------------------------
builtin.hdf5@1.10.0-patch1%clang@7.0.2-apple+cxx~debug+fortran+mpi+shared~szip~threadsafe arch=darwin-elcapitan-x86_64
^builtin.openmpi@2.0.1%clang@7.0.2-apple~mxm~pmi~psm~psm2~slurm~sqlite3~thread_multiple~tm~verbs+vt arch=darwin-elcapitan-x86_64
^builtin.hwloc@1.11.4%clang@7.0.2-apple arch=darwin-elcapitan-x86_64
^builtin.libpciaccess@0.13.4%clang@7.0.2-apple arch=darwin-elcapitan-x86_64
^builtin.libtool@2.4.6%clang@7.0.2-apple arch=darwin-elcapitan-x86_64
^builtin.m4@1.4.17%clang@7.0.2-apple+sigsegv arch=darwin-elcapitan-x86_64
^builtin.libsigsegv@2.10%clang@7.0.2-apple arch=darwin-elcapitan-x86_64
^builtin.pkg-config@0.29.1%clang@7.0.2-apple+internal_glib arch=darwin-elcapitan-x86_64
^builtin.util-macros@1.19.0%clang@7.0.2-apple arch=darwin-elcapitan-x86_64
^builtin.zlib@1.2.8%clang@7.0.2-apple+pic arch=darwin-elcapitan-x86_64
.. warning::
You *can* use a fully qualified package name in a ``depends_on``
directive in a ``package.py`` file, like so::
depends_on('proto.hdf5')
This is *not* recommended, as it makes it very difficult for
multiple repos to be composed and shared. A ``package.py`` like this
will fail if the ``proto`` repository is not registered in
``repos.yaml``.
.. _cmd-spack-repo:
--------------------------
``spack repo``
--------------------------
Spack's :ref:`configuration system <configuration>` allows repository
settings to come from ``repos.yaml`` files in many locations. If you
want to see the repositories registered as a result of all configuration
files, use ``spack repo list``.
^^^^^^^^^^^^^^^^^^^
``spack repo list``
^^^^^^^^^^^^^^^^^^^
.. code-block:: console
$ spack repo list
==> 2 package repositories.
myrepo ~/myrepo
builtin ~/spack/var/spack/repos/builtin
Each repository is listed with its associated namespace. To get the raw,
merged YAML from all configuration files, use ``spack config get repos``:
.. code-block:: console
$ spack config get repos
repos:srepos:
- ~/myrepo
- $spack/var/spack/repos/builtin
mNote that, unlike ``spack repo list``, this does not include the
namespace, which is read from each repo's ``repo.yaml``.
^^^^^^^^^^^^^^^^^^^^^
``spack repo create``
^^^^^^^^^^^^^^^^^^^^^
To make your own repository, you don't need to construct a directory
yourself; you can use the ``spack repo create`` command.
.. code-block:: console
$ spack repo create myrepo
==> Created repo with namespace 'myrepo'.
==> To register it with spack, run this command:
spack repo add ~/myrepo
$ ls myrepo
packages/ repo.yaml
$ cat myrepo/repo.yaml
repo:
namespace: 'myrepo'
By default, the namespace of a new repo matches its directory's name.
You can supply a custom namespace with a second argument, e.g.:
.. code-block:: console
$ spack repo create myrepo llnl.comp
==> Created repo with namespace 'llnl.comp'.
==> To register it with spack, run this command:
spack repo add ~/myrepo
$ cat myrepo/repo.yaml
repo:
namespace: 'llnl.comp'
^^^^^^^^^^^^^^^^^^
``spack repo add``
^^^^^^^^^^^^^^^^^^
Once your repository is created, you can register it with Spack with
``spack repo add``:
.. code-block:: console
$ spack repo add ./myrepo
==> Added repo with namespace 'llnl.comp'.
$ spack repo list
==> 2 package repositories.
llnl.comp ~/myrepo
builtin ~/spack/var/spack/repos/builtin
This simply adds the repo to your ``repos.yaml`` file.
Once a repository is registered like this, you should be able to see its
packages' names in the output of ``spack list``, and you should be able
to build them using ``spack install <name>`` as you would with any
built-in package.
^^^^^^^^^^^^^^^^^^^^^
``spack repo remove``
^^^^^^^^^^^^^^^^^^^^^
You can remove an already-registered repository with ``spack repo rm``.
This will work whether you pass the repository's namespace *or* its
path.
By namespace:
.. code-block:: console
$ spack repo rm llnl.comp
==> Removed repository ~/myrepo with namespace 'llnl.comp'.
$ spack repo list
==> 1 package repository.
builtin ~/spack/var/spack/repos/builtin
By path:
.. code-block:: console
$ spack repo rm ~/myrepo
==> Removed repository ~/myrepo
$ spack repo list
==> 1 package repository.
builtin ~/spack/var/spack/repos/builtin
--------------------------------
Repo namespaces and Python
--------------------------------
You may have noticed that namespace notation for repositories is similar
to the notation for namespaces in Python. As it turns out, you *can*
treat Spack repositories like Python packages; this is how they are
implemented.
You could, for example, extend a ``builtin`` package in your own
repository:
.. code-block:: python
from spack.pkg.builtin.mpich import Mpich
class MyPackage(Mpich):
...
Spack repo namespaces are actually Python namespaces tacked on under
``spack.pkg``. The search semantics of ``repos.yaml`` are actually
implemented using Python's built-in `sys.path
<https://docs.python.org/2/library/sys.html#sys.path>`_ search. The
:py:mod:`spack.repo` module implements a custom `Python importer
<https://docs.python.org/2/library/imp.html>`_.
.. warning::
The mechanism for extending packages is not yet extensively tested,
and extending packages across repositories imposes inter-repo
dependencies, which may be hard to manage. Use this feature at your
own risk, but let us know if you have a use case for it.
|