doc: describe apk2 and apk3 file formats

This change adds two manpages: apk-v2(5) and apk-v3(5). These pages
describe the v2 and v3 file formats respectively, as I currently
understand them.

4 files changed, 207 insertions, 0 deletions

diff --git a/doc/Makefile b/doc/Makefile
index 21d5e03..ffc204c 100644
--- a/doc/Makefile
+++ b/doc/Makefile
@@ -2,6 +2,8 @@ scdocs-y += \
 	apk-cache.5 \
 	apk-keys.5 \
 	apk-repositories.5 \
+	apk-v2.5 \
+	apk-v3.5 \
 	apk-world.5 \
 	apk.8 \
 	apk-add.8 \
diff --git a/doc/apk-v2.5.scd b/doc/apk-v2.5.scd
new file mode 100644
index 0000000..b19904a
--- /dev/null
+++ b/doc/apk-v2.5.scd
@@ -0,0 +1,85 @@
+apk-v2(5)
+
+# NAME
+
+*apk v2* - overview of apk v2 format
+
+# DESCRIPTION
+
+A v2 .apk file contains a single package's contents, some metadata, and
+some signatures. The .apk file contains three concatenated gzip streams,
+which together form a single tar archive. The tar archive contains three
+sections: the signatures, the control section, and the data section.
+
+# THE SIGNATURES
+
+The signatures are a sequence of files whose names start with ".SIGN.",
+which must come before any other data in the tarball. These filenames
+look like:
+
+	*.SIGN.<algorithm>.<keyid>*
+
+where <algorithm> must be one of *DSA*, *RSA*, *RSA256*, and *RSA512*
+and <keyid> must be the name of the key's file in /etc/apk/keys (see
+*apk-keys*(5)).
+
+The signature can be computed over either the metadata (if the metadata
+contains a data hash for the data), or over the metadata and data
+together (if the metadata contains no data hash).
+
+A single signature from a trusted key is sufficient, so an apk can be
+signed by multiple different keys if need be, as long as clients trust
+at least one of them.
+
+# THE CONTROL SECTION
+
+In a v2 apk file, the package metadata is stored in a single file called
+.PKGINFO. That file uses a key-value format, in which keys and values
+are separated by " = " and lines beginning with "#" are comments. There
+are many allowed keys and there is no centralized list of known keys;
+the source of *abuild*(1) is the best reference.
+
+One key is important for understanding the v2 format because it affects
+the interpretation of the signature: if there is a "datahash" key in
+PKGINFO, its value is the sha256 hash of the data part of the apk.
+Packages are supposed to have a datahash, but indexes do not.
+
+The control section is also where pre/post hook scripts for install, deinstall,
+and upgrade live, and where triggers live.
+
+# THE DATA SECTION
+
+The data section is simply a tar archive of the package's contents, as
+produced by the build process. These files are postprocessed by
+*abuild-tar*(1) and use pax extended headers to include per-file
+checksums in a header named APK-TOOLS.checksum.*<hash>*.
+
+# EXAMPLE
+
+As an example, the v2 apk for *scdoc*(1) itself contains these files in
+this order:
+
+	.SIGN.RSA.alpine-devel@lists.alpinelinux.org-6165ee59.rsa.pub
+	.PKGINFO
+	usr/
+	usr/bin/
+	usr/bin/scdoc
+	usr/share/
+	usr/share/pkgconfig/
+	usr/share/pkgconfig/scdoc.pc
+
+Since v2 apk files are simply tarballs (broken into multiple gzip
+streams), they can be inspected and unpacked with *tar*(1), although
+care must be taken when changing them not to reorder the sections or
+invalidate the signature. It is better to use *abuild*(1) to modify
+them. If you want to take them apart into their constituent gzip
+streams, you can use *abuild-gzsplit*(1).
+
+# NOTES
+
+Only the "RSA" (meaning RSA + SHA1) signature scheme is currently used
+by *abuild*(1).
+
+# SEE ALSO
+
+*abuild*(1), *apk*(1), *apk-v3*(5)
diff --git a/doc/apk-v3.5.scd b/doc/apk-v3.5.scd
new file mode 100644
index 0000000..e5113f3
--- /dev/null
+++ b/doc/apk-v3.5.scd
@@ -0,0 +1,118 @@
+apk-v3(5)
+
+# NAME
+
+*apk v3* - overview of apk v3 format
+
+# DECRIPTION
+
+A v3 .apk file contains a single package's contents, some metadata, and
+some signatures. The .apk file contains a tree of objects, represented
+in a custom binary format and conforming overall to a pre-defined
+schema. This file format is referred to inside *apk*(5) as "adb".
+
+# WIRE FORMAT
+
+A v3 apk file is composed of sequences of serialized values, each of
+which begins with a 32-bit little-endian word - the value's tag. The
+high 4 bits of the tag are a type code, and the low 28 bits are used for
+an immediate value. Defined type codes are:
+
+	0x0   Special  (direct)
+	0x1   Int      (direct)
+	0x2   Int32    (indirect)
+	0x3   Int64    (indirect)
+	0x8   Blob8    (indirect)
+	0x9   Blob16   (indirect)
+	0xa   Blob32   (indirect)
+	0xd   Array    (indirect)
+	0xe   Object   (indirect)
+
+A direct value is packed into the low 28 bits of the tag word; an
+indirect value is instead stored elsewhere in the file, and the offset
+of that indirect value is packed into the low 28 bits of the tag word.
+
+Arrays and objects are represented with a sequence of numbered slots;
+the value packed into their tag word is the offset at which this
+sequence starts. The first slot is always the total number of slots, so
+all arrays and objects contain at least one item.
+
+The only real difference between arrays and objects in the wire encoding
+is that arrays are homogenous, whereas objects are heterogenous with a
+separate defined type for each slot.
+
+The special type is used to represent three atoms:
+
+	0x0  NULL
+	0x1  TRUE
+	0x2  FALSE
+
+# FILE SCHEMAS
+
+A schema is a representation of what data elements are expected in an
+adb file. Schemas form a tree, where nodes are either scalar schemas
+(which are leaves in the tree) or array/object schemas, which themselves
+have children. For example, the schema for a package object might
+declare that it contains fields which themselves conform to the string
+array schema, or the pkginfo schema, or similar.
+
+The schemas themselves are not represented in the adb file in any way;
+they exist in the parts of *apk*(1) that read and write such files. A
+full description of all of apk's schemas would be lengthy, but as an
+example, here is the schema for a single file inside a package:
+
+	ADBI_FI_NAME    "name"    string
+	ADBI_FI_ACL     "acl"     acl
+	ADBI_FI_SIZE    "size"    int
+	ADBI_FI_MTIME   "mtime"   int
+	ADBI_FI_HASHES  "hash"    hexblob
+	ADBI_FI_TARGET  "target"  hexblob
+
+Here, all of the fields except for "acl" are scalars, and acl is itself
+a schema looking like:
+
+	ADBI_ACL_MODE   "mode"    oct
+	ADBI_ACL_USER   "user"    string
+	ADBI_ACL_GROUP  "group"   string
+
+# BLOCKS
+
+An actual adb file is composed of a sequence of typed blocks; a block
+also begins with a 32-bit little-endian tag word, which has two bits of
+type and 30 bits of size. The two type bits are:
+
+	0x0  ADB
+	0x1  SIG
+	0x2  DATA
+	0x3  DATAX
+
+The adb file must begin with one ADB block, then optionally one SIG
+block, then one or more DATA blocks. The ADB block must begin with a
+magic number indicating the schema for the entire ADB block's root
+object. The ADB block also contains, outside the root object, some
+metadata describing the version of the adb format in use.
+
+The SIG block contains a signature of the ADB block. Unlike the v2
+format, the key used for the signature is not explicitly specified, so
+verifiers must try all trusted keys until they find one. Also unlike the
+v2 format, the only supported hash algorithm is SHA512, and the
+signature scheme is implied by the signing key in use rather than being
+derived from the signature block.
+
+The DATA blocks are used to store package file data only; all file
+metadata, including content hashes, is stored in the ADB block instead.
+The contents of the DATA blocks are therefore protected by the hashes
+given in the ADB block, which is itself protected by the signature in
+the SIG block.
+
+It is currently illegal for a DATAX block to appear.
+
+# NOTES
+
+The v3 file format is entangled with C struct layout, since it sometimes
+directly writes structs into the adb section, including any
+compiler-added padding and such.
+
+# SEE ALSO
+
+*abuild*(1), *apk*(1), *apk-v2*(5)
diff --git a/doc/meson.build b/doc/meson.build
index 6e2173a..55e51ba 100644
--- a/doc/meson.build
+++ b/doc/meson.build
@@ -19,6 +19,8 @@ man_files = files(
     'apk-stats.8.scd',
     'apk-update.8.scd',
     'apk-upgrade.8.scd',
+    'apk-v2.5.scd',
+    'apk-v3.5.scd',
     'apk-verify.8.scd',
     'apk-version.8.scd',
     'apk-world.5.scd',