path: root/scripts/bootstrap.sh

#!/bin/sh -e

#===============================================================
# Filename : bootstrap.sh
# Purpose  : Bootstraps Adélie from source for any architecture.
# Authors  : Zach van Rijn <me@zv.io>
# License  : MIT
# Revision : 20221207
#===============================================================

#===============================================================
# README
#===============================================================
#
# overview
# --------
#
# Given a basic development environment ("Baseline System") that
# contains standard system utilities, this script bootstraps the
# Adélie Linux distribution for any suported target architecture
# (assuming that musl, gcc, etc. have been ported to it) without
# requiring 'root' privileges. This process is slow* and costly,
# but demonstrates that bootstrapping from source is possible.
#
# Effort has been focused on correctness instead of performance.
# This means, some files can be deleted sooner and some of the
# binaries can run natively on the build machine. We know and do
# appreciate there are different opinions on how to approach the
# bootstrap process. This meets our needs and hopefully offers a
# different perspective on how to simplify cross compilation.
#
# (*) See the "requirements" section for mitigations/discussion.
#
#
# features
# --------
#
#   * One-click bootstrap. Just start the script and walk away.
#
#       $ ./bootstrap.sh ppc64 /path/to/new/scratch/dir
#
#   * Can be done without root privileges: no 'chroot(8)'.
#
#   * This is the real deal. No "seed" binaries are required to
#     go from start to finish. The final output is what we use
#     to set up our official build servers totally* from source.
#
#   * Can be done on a wide range of platforms, almost certainly
#     those supported by mcmtools (want to contribute???).
#
#   * Minimal dependencies. Nothing fancy. Shell scripts only.
#
#   * Can be adapted to assist with porting Adélie to new target
#     architectures. TODO: write a tool to automatically do so.
#
# (*) We don't count binaries in your starting environment, and
#     recommend https://www.gnu.org/software/mes/ for the brave.
#
#
# design
# ------
#
# The goal of this script is to support as wide a range of uses
# as possible. For example, a full cross-platform bootstrap that
# needs to be done without target hardware or virtual machines.
#
# There are a few different strategies, depending on your needs
# and what type of privileges/hardware you have access to.
#
# In this diagram, indentation refers to an output product, and
# moving down vertically refers to an input step. CPU A is the
# build architecture, and CPU B is the target architecture. Your
# mileage may vary, and you can mix/match other approaches..
#
#   * CPU A --> CPU B (slowest: no privilege, no CPU B hardware)
#         --> Bootstrap 1 (Cross Libc + Partial Cross CPU B)
#             --> CPU A + CPU B Mixed Rootfs
#             --> Partial Emulation (PRoot)
#             --> Bootstrap 2 (Full Cross CPU B)
#                 --> CPU B Host Rootfs
#                 --> Full Emulation (PRoot)
#                 --> Adélie Bootstrap
#                     --> Adélie Rootfs for CPU B
#
#   * CPU A --> CPU B (slower:     privilege, no CPU B hardware)
#         --> Bootstrap 1 (Cross Libc + Partial Cross CPU B)
#             --> CPU A + CPU B Mixed Rootfs
#             --> Partial Emulation (registered binfmt_misc)
#             --> Bootstrap 2 (Full Cross CPU B)
#                 --> CPU B Host Rootfs
#                 --> Full Emulation (registered binfmt_misc)
#                 --> Adélie Bootstrap
#                     --> Adélie Rootfs for CPU B
#
#   * CPU A --> CPU B (faster; no privilege,     CPU B hardware)
#         --> Bootstrap 1 (Cross Libc + Partial Cross CPU B)
#             --> CPU A + CPU B Mixed Rootfs
#             --> Partial Emulation (PRoot)
#             --> Bootstrap 2 (Full Cross CPU B)
#                 --> CPU B Host Rootfs
#                 --> Copy to Native CPU B Hardware
#                 --> Adélie Bootstrap
#                     --> Adélie Rootfs for CPU B
#
#   * CPU B --> CPU B (fastest; QEMU-system or native hardware)
#         --> Bootstrap 1 (Cross Libc)
#             --> CPU B Mixed Rootfs
#             --> Bootstrap 2 (Remove Contamination)
#                 --> CPU B Host Rootfs
#                 --> Adélie Bootstrap
#                     --> Adélie Rootfs for CPU B
#
# The resulting Adélie Rootfs is capable of building the entire
# Adélie Linux distribution from source. We use it to seed our
# official build servers. This removes the dependency on using
# previous binary releases of our distribution, which had been
# manually bootstrapped way back from Gentoo and/or on a G5.
#
#
# requirements
# ------------
#
# TL;DR: You must be able to run the mcmtools bootstrap script.
#
# Cross builds take up a lot of TIME and SPACE, and this cannot
# be improved by throwing hardware at it. This is primarily due
# to the tradeoff of not requiring root privileges at any point,
# and the decision to emulate a native environment instead of to
# force explicit cross-compilation at each step in the process.
#
# Targeting an architecture of opposite endianness will be even
# slower; this is a QEMU limitation.
#
# If you wish to bootstrap to the same target CPU architecture,
# or a compatible mode (e.g. i686 on x86_64), it will be faster.
#
# mcmtools is a hard dependency for our bootstrap process now:
#
#     https://git.zv.io/toolchains/bootstrap
#
# It is a simple analog to Buildroot or Yocto. Those tools could
# be used, too, and would provide the "host rootfs" environment.
#
# Internet access is required, but (as an exercise left to the
# reader) it is possible to pre-download all required sources if
# you provide your own rootfs for the static QEMU build process.
#
# If you are in a position to use native hardware, then you are
# able to get away with only the final stages of bootstrapping.
# To do this, you'd essentially comment out the first stages or
# copy the results of the first stages elsewhere and continue.
#
# Other requirements that you should be aware of, estimated:
#
#   * As many CPU cores as you can throw at it;
#
#   * ~15 GB for each rootfs ("seed" and "host"), so ~30GB;
#
#   * ~ 3 GB for toolchains;
#
#   * ~ 2 GB for the "system/" package repository, when built;
#
#   * Please refer to the README in 'bootstrap', linked above,
#     for more information about performance. Most of this can
#     be gained back if you adapt this script to use 'chroot(8)'
#     instead of 'PRoot', at the expense of requiring privilege,
#     with the correct registration of QEMU with 'binfmt_misc'.
#
#     In brief, there is an approximate factor of 13 slowdown on
#     practical workloads when using QEMU user and 'PRoot' to do
#     dynamic binary translation and emulate a foreign machine.
#
#
# process
# -------
#
# The illustration below outlines the complete bootstrap process
# and roughly corresponds to the script layout/organization. Our
# terminology is not perfectly consistent; please excuse this.
#
# The term "build"  is shorthand for "build-native"   CPU, which
# is the machine on which you are performing the bootstrap.
#
# The term "native" is shorthand for "foreign-native" CPU, which
# is the machine to which you are targeting  the bootstrap, and
# relates to the "host" CPU on which the code will run, but that
# with the help of dynamic binary translation, runs "natively".
#
#     - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
#     build              (unstable, everything provided by user)
#
#     +-----------------+       User-provided tools. Dependency
#     | Baseline System |       of 'mcmtools', which will verify
#     +-----------------+       that these tools are available.
#              |
#        +----------+           Script to build pinned versions
#        | mcmtools |           of common system utilities, a
#        +----------+           host-arch host-libc toolchain, a
#              |                host-arch musl-libc toolchain...
#              |
#     - - - - -|- - - - - - - - - - - - - - - - - - - - - - - -
#     mixed    |                (stable versions, unstable libc)
#              |
#       +-------------+         ...and a 'chroot'-able rootfs. A
#       | seed rootfs |         sane, but not clean, environment
#       +-------------+         in which we begin the bootstrap.
#              |
#        +-----------+          Script to build 'PRoot' and its
#        | emulators |          dependencies, as well as static
#        +-----------+          QEMU user binaries. Add to seed.
#              |
#     +-----------------+       Static musl toolchains targeting
#     | musl toolchains |       a given architecture: cross from
#     +-----------------+       the host CPU & foreign "native".
#              |                Output binaries will run via the
#              |                'binfmt_misc' mechanism + QEMU.
#              |
#       +-------------+         Script to build Alpine Package
#       | build tools |         Keeper (APK) and dependencies.
#       +-------------+         All binaries are cross-compiled!
#              |                This step is a sanity check.
#              |
#        +----------+           Script to build pinned versions
#        | mcmtools |           of common system utilities. This
#        +----------+           time, all utilities are target-
#              |                native. Build more toolchains...
#              |
#     - - - - -|- - - - - - - - - - - - - - - - - - - - - - - -
#     host     |                (stable versions,   stable libc)
#              |
#       +-------------+         ...and a 'chroot'-able rootfs. A
#       | host rootfs |         sane, clean, foreign "native"
#       +-------------+         rootfs that requires QEMU, or is
#              |                able to run on native hardware.
#              |
#       +-------------+         Script to build Alpine Package
#       | build tools |         Keeper (APK) and dependencies.
#       +-------------+         These binaries are native built!
#              |                This step is required!
#              |
#       +-------------+         Script to build the Adélie Linux
#       | system repo |         "system/" package repository. It
#       +-------------+         is used to build core packages.
#              |
#      +---------------+        Script to install packages into
#      | image creator |        a clean rootfs. Carryover from
#      +---------------+        the mcmtools process is removed.
#              |
#     - - - - -|- - - - - - - - - - - - - - - - - - - - - - - -
#     verify   |                         (optional verification)
#              |
#             ...               The "host" stage can be repeated
#              |                zero or more times to ensure the
#              |                final image is not contaminated.
#              |
#     - - - - -|- - - - - - - - - - - - - - - - - - - - - - - -
#     output   |                       (Welcome to Adélie Linux)
#              |
#      +---------------+        This is a minimal Adélie rootfs.
#      | adelie rootfs |        Copy it to native hardware, then
#      +---------------+        use it as a builder 'chroot'. It
#                               is used to seed official Adélie
#                               build boxen (e.g. autobuilder).
#
# limitations
# -----------
#
# The builds are not hermetically sealed. That is not the point.
# The build environment should not *need* anything from outside,
# but it is not *prevented* from accessing anything. You should
# run this on a clean, trusted machine.
#
# There is no guarantee of byte-for-byte reproducible builds at
# this time. This is, in part, due to timestamps and tar headers
# but may involve a lack of determinism in parallel builds.
#
# QEMU user emulation may cause subtle incompatibilites with the
# target CPU hardware. For example, CPU feature (mis)detection.
#
# It is not currently possible to safely reuse any of the output
# if the target architecture is changed. You will need to start
# from scratch if you wish to change the target. This is a TODO.
#
#
# todo
# ----
#
#   * Do not inherit the build environment; avoid contamination.
# 

HERE="$(dirname $(readlink -f ${0}))";


#---------------------------------------------------------------
# initialization

##
# Haaaalp!
#
usage ()
{
    printf "Usage: %s ARCH BASE\n\n" "${0}";
    cat <<EOF
    ARCH    { aarch64, armv7, ppc64, ppc, x86_64, pmmx }

    BASE    an absolute path to bootstrap out of.
            ** MAY CAUSE DATA LOSS IF SET INCORRECTLY! **
EOF
    exit 0;
}


##
# argv[1]: ARCH
#
# ARCH is the Adélie Linux target. This is the first step in the
# porting process, so e.g. mips64, sparc64, riscv64 will need to
# be added to this table when the time comes to port to them.
#
# ARCH is translated to canonical GCC and QEMU machine types.
#
case "${1}" in
#   adelie        gcc                 qemu
#   ------        ---                 ----
    aarch64)    m=aarch64:      ;   q=aarch64   ;   ;;
    armv7)      m=armv7l:eabihf ;   q=arm       ;   ;;
    ppc)        m=powerpc:      ;   q=ppc       ;   ;;
    ppc64)      m=powerpc64:    ;   q=ppc64     ;   ;;
    riscv64)    m=riscv64:      ;   q=riscv64   ;   ;;
    x86_64)     m=x86_64:       ;   q=x86_64    ;   ;;
    pmmx)       m=i586:         ;   q=i386      ;   ;;
    *) usage ;;
esac
shift;


##
# argv[2]: BASE
#
# BASE is a semi-permanent scratch directory. It is where all of
# the magic happens, and probably cannot be relocated easily. Be
# careful to not set it incorrectly or to a place you'd regret
# being overwritten, corrupted, or deleted.
#
case "${1}" in
    /*) BASE="${1}";
        ;;
    *)  printf "BASE not set, or not an absolute path!\n";
        exit 1;
        ;;
esac
shift;


##
# Internal variables. Do not modify this section directly.
#
CHAINS=https://git.zv.io/toolchains;
printf "CHAINS=%s\n" "${CHAINS}";

SYSTEM="-adelie-linux-musl";    # we only target musl on Linux
printf "SYSTEM=%s\n" "${SYSTEM}";

NATIVE=$(cc -dumpmachine);      # host arch, host libc
printf "NATIVE=%s\n" "${NATIVE}";

BUILDS="${NATIVE%%-*}${SYSTEM}"; # host arch, musl libc
printf "BUILDS=%s\n" "${BUILDS}";

TARGET="${m%:*}${SYSTEM}${m#*:}"; # ultimate Adélie Linux target
printf "TARGET=%s\n" "${TARGET}";

MTOOLS=${MCMTOOLS:-"${BASE}/mcmtools"}; # CAREFUL! MAY CAUSE DATA LOSS!
printf "MTOOLS=%s\n" "${MTOOLS}";

##
# Default 'PATH' for use inside various rootfs environments.
#
DEF_PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin";

mkdir -p "${BASE}";


#---------------------------------------------------------------
# mcmtools (seed)

##
# Allow the user to supply an existing mcmtools installation. It
# is not ideal, but since mcmtools cannot be (easily) relocated,
# allow the user to save some CPU cycles at the cost of adding a
# bunch of tools to their existing installation. A temporary dir
# is used if this environment variable is omitted. Another case
# for providing a custom value is if '/tmp' is mounted weird.
#
if ! test -d "${MTOOLS}/sys/emus/bin"; then # FIXME: no hard code
    cd "${BASE}";

    test -d bootstrap                                          \
        || git clone ${CHAINS}/bootstrap.git;
    cd bootstrap;
    git checkout 371f5e2624acd89dd05a9bd8c68f369b16dfdde6;

    ## seed rootfs
    #
    # This will build absolutely everything that is needed to be
    # self-reliant, except for some build deps for QEMU.
    #
    # We copy 'config.mak' from mcmtools bootstrap to the rootfs
    # so that when we build "real" toolchain they are the same.
    #
    DEST="${MTOOLS}"                                           \
    ARCH=${BUILDS}                                             \
        ./bootstrap                                            \
        ;
    test -f "${MTOOLS}"/config.mak ||                          \
        cp "${MTOOLS}"/tmp/musl-cross-make/config.mak          \
        "${MTOOLS}"/config.mak                                 \
        ;
    # cleaning
    (
        cd "${MTOOLS}";
        rm -fr tmp; # save 10 GB
    )

    # is any of this actually needed?
    (
        cd "${MTOOLS}"/sys;
        mkdir -p dev;
        mkdir -p proc;
        mkdir -p sys;
        rm -fr usr;
        ln -s . usr;
    )

    ## emulators
    #
    # Dependencies are built with the mcmtools host toolchain; a
    # reason to not force musl here is in the event that these
    # cannot be built statically and the host libc is different.
    # Our priority is to obtain a functioning 'PRoot' above all.
    #
    # QEMU itself is built inside an Alpine Linux rootfs; we do
    # this because we still need Python 3 to build it. You can
    # manually provide your own static QEMU user binaries and be
    # on your way without Alpine, but it is a good 'PRoot' test.
    #
    test -d "${MTOOLS}/sys/emus/bin" ||                        \
    PATH="${MTOOLS}/host/bin:${MTOOLS}/sys/bin"                \
    DEST="${MTOOLS}"                                           \
        ./prootemu                                             \
        ;
fi


#---------------------------------------------------------------
# musl toolchains

##
# We have a musl-targeting toolchain that runs on the host, but
# it is linked to the host libc and may not run in this chroot.
#
# We need to build the same toolchain, but static. There are two
# possible directions to go: (1) a cross toolchain that runs at
# full speed on the build machine, at the cost of having to tell
# downstream build scripts how to cross compile, or (2) a native
# toolchain for the foreign (target) architecture that runs slow
# in QEMU and still requires the cross toolchain to build it.
#
# Build both. We will eventually need both toolchains, anyway.
#
if ! test -d "${MTOOLS}"/sys/tc/native; then # FIXME: no hard code
    cd "${BASE}";

    test -d musl-cross-make                                    \
        || git clone ${CHAINS}/musl-cross-make.git;
    cd musl-cross-make;
    git pull; # always use the latest

    ##
    # Ensure consistent 'config.mak' for all toolchain builds.
    #
    cp "${MTOOLS}"/config.mak config.mak;

    ## musl toolchains
    #
    # Build these toolchains statically using the musl toolchain
    # from the seed rootfs so that it is known to work correctly
    # (the original musl toolchain itself may itself be linked
    # with glibc or be unsafe to use in some contexts).
    #
    # Note: "native" is for the foreign target CPU architecture.
    #
    MCMTOOLS="${MTOOLS}"                                       \
        ./scripts/build ${TARGET}                              \
        ;
    for k in cross native; do
        rm -fr "${MTOOLS}"/sys/tc/${k};
        mkdir  "${MTOOLS}"/sys/tc/${k};
        tar -C "${MTOOLS}"/sys/tc/${k}                         \
            --strip-components=1                               \
            -xzf output/${TARGET}-${k}.tgz                     \
            ;
    done
fi


#---------------------------------------------------------------
# build tools (cross)

##
# Build 'abuild', its dependencies, and other utilities.
#
PROOT_NO_SECCOMP=1                                             \
PATH="/tc/cross/bin:${DEF_PATH}"                               \
SHELL=/bin/sh                                                  \
DEST=/usr/local                                                \
CURL_CA_BUNDLE=/cacert.pem                                     \
CC=${TARGET}-gcc                                               \
CXX=${TARGET}-g++                                              \
AR=${TARGET}-ar                                                \
LD=${TARGET}-ld                                                \
CPP=${TARGET}-cpp                                              \
RANLIB=${TARGET}-ranlib                                        \
${MTOOLS}/sys/emus/bin/proot                                   \
    -S "${MTOOLS}"/sys                                         \
    -q "${MTOOLS}"/sys/emus/bin/qemu-${q}                      \
    -b "${HERE}"                                               \
    "${HERE}"/bootstrap-abuild                                 \
    ;


#---------------------------------------------------------------
# rootfs: build

##
# Create a tarball of the build rootfs. The image creator could
# use this as input if the target architecture matches.
#
cd "${BASE}";
if ! test -f rootfs-${BUILDS}.tgz; then
    tar -C "${BASE}"/mcmtools/sys                              \
        -pczf rootfs-${BUILDS}.tgz                             \
        .                                                      \
        ;
fi


#---------------------------------------------------------------
# mcmtools (host)

##
# NOTE: The 'PATH' order is *really* important. Foreign "native"
# toolchains must be the first ones found; e.g. '/usr/bin/gcc'
# is a symlink to 'ccache' and isn't a functional compiler.
#
# NOTE: 'LD_LIBRARY_PATH' is needed because of nonstandard path
# of toolchain, which otherwise produces these errors:
#
# Error loading shared library libstdc++.so.6:
#     Exec format error (needed by ...)
# Error loading shared library libgcc_s.so.1:
#     Exec format error (needed by ...)
#
# NOTE: We *do* need to build 'musl-cross-make' (even though it
# may seem redundant) so that we can safely install the dynamic
# loader. If we do not do this, we're on the hook to symlink it.
#
# Once 'coreutils' is built, 'uname' will return correctly, then
# future software should(tm) build as if it is built natively.
#
# FIXME: is Linux 3.2.0 really appropriate here?
#
if ! test -d "${BASE}"/mcmtools-${TARGET}/sys/tc/musl/bin; then # FIXME: no hard code
PROOT_NO_SECCOMP=1                                             \
PATH="/tc/native/bin:${DEF_PATH}"                              \
LD_LIBRARY_PATH=/tc/native/lib                                 \
SHELL=/bin/sh                                                  \
BASE="${BASE}"                                                 \
DEST="${BASE}"/mcmtools-${TARGET}                              \
ARCH=${TARGET}                                                 \
CURL_CA_BUNDLE=/cacert.pem                                     \
${MTOOLS}/sys/emus/bin/proot                                   \
    -S "${MTOOLS}"/sys                                         \
    -q "${MTOOLS}"/sys/emus/bin/qemu-${q}                      \
    -b "${HERE}"                                               \
    -k "3.2.0"                                                 \
    "${BASE}"/bootstrap/bootstrap                              \
    ;
fi

# cleaning
(
    cd "${BASE}"/mcmtools-${TARGET};
    rm -fr tmp; # save 10 GB
)

##
# Install the native static toolchain over the native rootfs for
# a quick way to have the dynamic loader. Overwrite the symlink!
#
if test -h   "${BASE}"/mcmtools-${TARGET}/sys/bin/gcc; then # FIXME: no hard code
    tar -C   "${BASE}"/mcmtools-${TARGET}/sys                  \
        --strip-components=1                                   \
        -xzf "${BASE}"/musl-cross-make/output/${TARGET}-native.tgz \
       ;
fi


#---------------------------------------------------------------
# build tools (host)

##
# Build 'abuild', its dependencies, and other utilities.
#
PROOT_NO_SECCOMP=1                                             \
PATH="/tc/musl/bin:${DEF_PATH}"                                \
SHELL=/bin/sh                                                  \
DEST=/usr/local                                                \
CURL_CA_BUNDLE=/cacert.pem                                     \
CC=gcc                                                         \
CXX=g++                                                        \
AR=ar                                                          \
LD=ld                                                          \
CPP=cpp                                                        \
${MTOOLS}/sys/emus/bin/proot                                   \
    -S "${BASE}"/mcmtools-${TARGET}/sys                        \
    -q "${MTOOLS}"/sys/emus/bin/qemu-${q}                      \
    -b "${HERE}"                                               \
    "${HERE}"/bootstrap-abuild                                 \
    ;


#---------------------------------------------------------------
# rootfs: host

##
# Create a tarball of the host rootfs. This is a safety measure.
#
cd "${BASE}";
if ! test -f rootfs-${TARGET}.tgz; then
    tar -C "${BASE}"/mcmtools-${TARGET}/sys                    \
        -pczf rootfs-${TARGET}.tgz                             \
        .                                                      \
        ;
fi


#---------------------------------------------------------------
# finalize pre-adelie rootfs

exit 0;
(
    cd "${BASE}"/mcmtools-${TARGET}/sys;

    # overwrite leftovers from mcmtools bootstrap
    cat > root/.bashrc <<EOF
PATH="${DEF_PATH}";
CURL_CA_BUNDLE=/cacert.pem;
EOF
    mkdir dev proc sys tmp; # will require proper mounting
    cp /etc/resolv.conf etc;
    # need to create the /tmp/tmp.whtever dir as symlink to /

    # handle git backend issue
    # git config --global http.sslCAInfo ...

grep -rl /old/path . | cut -d: -f1 | while read k; do
        perl -pi -e 's@/old/path@/new/path@g' ${k};
    done
)


#---------------------------------------------------------------
# system repository

# TODO


#---------------------------------------------------------------
# image creator

# TODO

# apk --root /foo --arch armv7 --initdb add
# apk --root /foo add adelie-core dash-binsh build-tools