cc: convert compiler wrapper to posix shell

This converts everything in cc to POSIX sh, except for the parts currently
handled with bash arrays. Tests are still passing.

This version tries to be as straightforward as possible. Specifically, most conversions
are kept simple -- convert ifs to ifs, handle indirect expansion the way we do in
`setup-env.sh`, only mess with the logic in `cc`, and don't mess with the python code at
all.

The big refactor is for arrays. We can't rely on bash's nice arrays and be ignorant of
separators anymore. So:

1. To avoid complicated separator logic, there are three types of lists. They are:

    * `$lsep`-separated lists, which end with `_list`. `lsep` is customizable, but we
      picked `^G` (alarm bell) for `$lsep` because it's ASCII and it's unlikely that it
      would actually appear in any arguments. If we need to get fancier (and I will lose
      faith in the world if we do) then we could consider XON or XOFF.
    * `:`-separated directory lists, which end with `_dirs`, `_DIRS`, `PATH`, or `PATHS`
    * Whitespace-separated lists (like flags), which can have any other name.

    Whitespace and colon-separated lists come with the territory with PATHs from env
    vars and lists of flags. `^G` separated lists are what we use for most internal
    variables, b/c it's more likely to work.

2. To avoid subshells, use a bunch of functions that do dirty `eval` stuff instead. This
   adds 3 functions to deal with lists:

    * `append LISTNAME ELEMENT [SEP]` will put `ELEMENT` at the end of the list called
      `LISTNAME`. You can optionally say what separator you expect to use. Note that we
      are taking advantage of everything being global and passing lists by name.

    * `prepend LISTNAME ELEMENT [SEP]` like append, but puts `ELEMENT` at the start of
      `LISTNAME`

    * `extend LISTNAME1 LISTNAME2 [PREFIX]` appends everything in LISTNAME2 to
       LISTNAME1, and optionally prepends `PREFIX` to every element (this is useful for
       things like `-I`, `-isystem `, etc.

    * `preextend LISTNAME1 LISTNAME2 [PREFIX]` prepends everything in LISTNAME2 to
       LISTNAME1 in order, and optionally prepends `PREFIX` to every element.

The routines determine the separator for each argument by its name, so we don't have to
pass around separators everywhere. Amazingly, as long as you do not expand variables'
values within an `eval` environment, you can do all this and still preserve quoting.
When iterating over lists, the user of this API still has to set and unset `IFS`
properly.

We ended up having to ignore shellcheck SC2034 (unused variable), because using evals
all over the place means that shellcheck doesn't notice that our list variables are
actually used.

So far this is looking pretty good. I took the most complex unit test I could find
(which runs a sample link line) and ran the same command line 200 times in a shell
script.  Times are roughly as follows:

For this invocation:

```console
$ bash -c 'time (for i in `seq 1 200`; do ~/test_cc.sh > /dev/null; done)'
```

I get the following performance numbers (the listed shells are what I put in `cc`'s
shebang):

**Original**
* Old version of `cc` with arrays and `bash v3.2.57` (macOS builtin): `4.462s` (`.022s` / call)
* Old version of `cc` with arrays and `bash v5.1.8` (Homebrew): `3.267s` (`.016s` / call)

**Using many subshells (#26408)**
*  with `bash v3.2.57`: `25.302s` (`.127s` / call)
*  with `bash v5.1.8`: `27.801s` (`.139s` / call)
*  with `dash`: `15.302s` (`.077s` / call)

This version didn't seem to work with zsh.

**This PR (no subshells)**
*  with `bash v3.2.57`: `4.973s` (`.025s` / call)
*  with `bash v5.1.8`: `4.984s` (`.025s` / call)
*  with `zsh`: `2.995s` (`.015s` / call)
*  with `dash`: `1.890s` (`.0095s` / call)

Dash, with the new posix design, is easily the winner.

So there are several interesting things to note here:

1. Running the posix version in `bash` is slower than using `bash` arrays. That is to be
   expected because it's doing a bunch of string processing where it likely did not have
   to before, at least in `bash`.

2. `zsh`, at least on macOS, is significantly faster than the ancient `bash` they ship
   with the system. Using `zsh` with the new version also makes the posix wrappers
   faster than `develop`. So it's worth preferring `zsh` if we have it. I suppose we
   should also try this with newer `bash` on Linux.

3. `bash v5.1.8` seems to be significantly faster than the old system `bash v3.2.57` for
   arrays. For straight POSIX stuff, it's a little slower. It did not seem to matter
   whether `--posix` was used.

4. `dash` is way faster than `bash` or `zsh`, so the real payoff just comes from being
   able to use it. I am not sure if that is mostly startup time, but it's significant.
   `dash` is ~2.4x faster than the original `bash` with arrays.

So, doing a lot of string stuff is slower than arrays, but converting to posix seems
worth it to be able to exploit `dash`.

- [x] Convert all but array-related portions to sh
- [x] Fix basic shellcheck issues.
- [x] Convert arrays to use a few convenience functions: `append` and `extend`
- [x] Get `cc` tests passing.
- [x] Add `cc` tests where needed passing.
- [x] Benchmarking.

Co-authored-by: Tom Scogland <scogland1@llnl.gov>
Co-authored-by: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>

1 files changed, 351 insertions, 212 deletions

diff --git a/lib/spack/env/cc b/lib/spack/env/cc
index b603c3bc32..4bef0ddf9e 100755
--- a/lib/spack/env/cc
+++ b/lib/spack/env/cc
@@ -1,4 +1,5 @@
-#!/bin/bash
+#!/bin/sh
+# shellcheck disable=SC2034  # evals in this script fool shellcheck
 #
 # Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
 # Spack Project Developers. See the top-level COPYRIGHT file for details.
@@ -20,25 +21,33 @@
 #      -Wl,-rpath         arguments for dependency /lib directories.
 #
 
+# Reset IFS to the default: whitespace-separated lists. When we use
+# other separators, we set and reset it.
+unset IFS
+
+# Separator for lists whose names end with `_list`.
+# We pick the alarm bell character, which is highly unlikely to
+# conflict with anything. This is a literal bell character (which
+# we have to use since POSIX sh does not convert escape sequences
+# like '\a' outside of the format argument of `printf`).
+# NOTE: Depending on your editor this may look empty, but it is not.
+readonly lsep=''
+
 # This is an array of environment variables that need to be set before
 # the script runs. They are set by routines in spack.build_environment
 # as part of the package installation process.
-parameters=(
-    SPACK_ENV_PATH
-    SPACK_DEBUG_LOG_DIR
-    SPACK_DEBUG_LOG_ID
-    SPACK_COMPILER_SPEC
-    SPACK_CC_RPATH_ARG
-    SPACK_CXX_RPATH_ARG
-    SPACK_F77_RPATH_ARG
-    SPACK_FC_RPATH_ARG
-    SPACK_TARGET_ARGS
-    SPACK_DTAGS_TO_ADD
-    SPACK_DTAGS_TO_STRIP
-    SPACK_LINKER_ARG
-    SPACK_SHORT_SPEC
-    SPACK_SYSTEM_DIRS
-)
+readonly params="\
+SPACK_ENV_PATH
+SPACK_DEBUG_LOG_DIR
+SPACK_DEBUG_LOG_ID
+SPACK_COMPILER_SPEC
+SPACK_CC_RPATH_ARG
+SPACK_CXX_RPATH_ARG
+SPACK_F77_RPATH_ARG
+SPACK_FC_RPATH_ARG
+SPACK_LINKER_ARG
+SPACK_SHORT_SPEC
+SPACK_SYSTEM_DIRS"
 
 # Optional parameters that aren't required to be set
 
@@ -58,60 +67,157 @@ parameters=(
 # Test command is used to unit test the compiler script.
 #   SPACK_TEST_COMMAND
 
-# die()
-# Prints a message and exits with error 1.
-function die {
+# die MESSAGE
+# Print a message and exit with error code 1.
+die() {
     echo "$@"
     exit 1
 }
 
-# read input parameters into proper bash arrays.
-# SYSTEM_DIRS is delimited by :
-IFS=':' read -ra SPACK_SYSTEM_DIRS <<< "${SPACK_SYSTEM_DIRS}"
+# empty VARNAME
+# Return whether the variable VARNAME is unset or set to the empty string.
+empty() {
+    eval "test -z \"\${$1}\""
+}
 
-# SPACK_<LANG>FLAGS and SPACK_LDLIBS are split by ' '
-IFS=' ' read -ra SPACK_FFLAGS   <<< "$SPACK_FFLAGS"
-IFS=' ' read -ra SPACK_CPPFLAGS <<< "$SPACK_CPPFLAGS"
-IFS=' ' read -ra SPACK_CFLAGS   <<< "$SPACK_CFLAGS"
-IFS=' ' read -ra SPACK_CXXFLAGS <<< "$SPACK_CXXFLAGS"
-IFS=' ' read -ra SPACK_LDFLAGS  <<< "$SPACK_LDFLAGS"
-IFS=' ' read -ra SPACK_LDLIBS   <<< "$SPACK_LDLIBS"
+# setsep LISTNAME
+# Set the global variable 'sep' to the separator for a list with name LISTNAME.
+# There are three types of lists:
+#   1. regular lists end with _list and are separated by $lsep
+#   2. directory lists end with _dirs/_DIRS/PATH(S) and are separated by ':'
+#   3. any other list is assumed to be separated by spaces: " "
+setsep() {
+    case "$1" in
+        *_dirs|*_DIRS|*PATH|*PATHS)
+            sep=':'
+            ;;
+        *_list)
+            sep="$lsep"
+            ;;
+        *)
+            sep=" "
+            ;;
+    esac
+}
 
+# prepend LISTNAME ELEMENT [SEP]
+#
+# Prepend ELEMENT to the list stored in the variable LISTNAME,
+# assuming the list is separated by SEP.
+# Handles empty lists and single-element lists.
+prepend() {
+    varname="$1"
+    elt="$2"
+
+    if empty "$varname"; then
+        eval "$varname=\"\${elt}\""
+    else
+        # Get the appropriate separator for the list we're appending to.
+        setsep "$varname"
+        eval "$varname=\"\${elt}${sep}\${$varname}\""
+    fi
+}
+
+# append LISTNAME ELEMENT [SEP]
+#
+# Append ELEMENT to the list stored in the variable LISTNAME,
+# assuming the list is separated by SEP.
+# Handles empty lists and single-element lists.
+append() {
+    varname="$1"
+    elt="$2"
+
+    if empty "$varname"; then
+        eval "$varname=\"\${elt}\""
+    else
+        # Get the appropriate separator for the list we're appending to.
+        setsep "$varname"
+        eval "$varname=\"\${$varname}${sep}\${elt}\""
+    fi
+}
+
+# extend LISTNAME1 LISTNAME2 [PREFIX]
+#
+# Append the elements stored in the variable LISTNAME2
+# to the list stored in LISTNAME1.
+# If PREFIX is provided, prepend it to each element.
+extend() {
+    # Figure out the appropriate IFS for the list we're reading.
+    setsep "$2"
+    if [ "$sep" != " " ]; then
+        IFS="$sep"
+    fi
+    eval "for elt in \${$2}; do append $1 \"$3\${elt}\"; done"
+    unset IFS
+}
+
+# preextend LISTNAME1 LISTNAME2 [PREFIX]
+#
+# Prepend the elements stored in the list at LISTNAME2
+# to the list at LISTNAME1, preserving order.
+# If PREFIX is provided, prepend it to each element.
+preextend() {
+    # Figure out the appropriate IFS for the list we're reading.
+    setsep "$2"
+    if [ "$sep" != " " ]; then
+        IFS="$sep"
+    fi
+
+    # first, reverse the list to prepend
+    _reversed_list=""
+    eval "for elt in \${$2}; do prepend _reversed_list \"$3\${elt}\"; done"
+
+    # prepend reversed list to preextend in order
+    IFS="${lsep}"
+    for elt in $_reversed_list; do prepend "$1" "$3${elt}"; done
+    unset IFS
+}
+
+# system_dir PATH
 # test whether a path is a system directory
-function system_dir {
+system_dir() {
+    IFS=':'  # SPACK_SYSTEM_DIRS is colon-separated
     path="$1"
-    for sd in "${SPACK_SYSTEM_DIRS[@]}"; do
-        if [ "${path}" == "${sd}" ] || [ "${path}" == "${sd}/" ]; then
+    for sd in $SPACK_SYSTEM_DIRS; do
+        if [ "${path}" = "${sd}" ] || [ "${path}" = "${sd}/" ]; then
             # success if path starts with a system prefix
+            unset IFS
             return 0
         fi
     done
+    unset IFS
     return 1  # fail if path starts no system prefix
 }
 
-for param in "${parameters[@]}"; do
-    if [[ -z ${!param+x} ]]; then
+# Fail with a clear message if the input contains any bell characters.
+if eval "[ \"\${*#*${lsep}}\" != \"\$*\" ]"; then
+    die "ERROR: Compiler command line contains our separator ('${lsep}'). Cannot parse."
+fi
+
+# ensure required variables are set
+for param in $params; do
+    if eval "test -z \"\${${param}:-}\""; then
         die "Spack compiler must be run from Spack! Input '$param' is missing."
     fi
 done
 
 # Check if optional parameters are defined
 # If we aren't asking for debug flags, don't add them
-if [[ -z ${SPACK_ADD_DEBUG_FLAGS+x} ]]; then
+if [ -z "${SPACK_ADD_DEBUG_FLAGS:-}" ]; then
     SPACK_ADD_DEBUG_FLAGS="false"
 fi
 
 # SPACK_ADD_DEBUG_FLAGS must be true/false/custom
 is_valid="false"
 for param in "true" "false" "custom"; do
-  if [ "$param" == "$SPACK_ADD_DEBUG_FLAGS" ];  then
+  if [ "$param" = "$SPACK_ADD_DEBUG_FLAGS" ];  then
       is_valid="true"
   fi
 done
 
 # Exit with error if we are given an incorrect value
-if [ "$is_valid" == "false" ]; then
-    die "SPACK_ADD_DEBUG_FLAGS, if defined, must be one of 'true' 'false' or 'custom'"
+if [ "$is_valid" = "false" ]; then
+    die "SPACK_ADD_DEBUG_FLAGS, if defined, must be one of 'true', 'false', or 'custom'."
 fi
 
 # Figure out the type of compiler, the language, and the mode so that
@@ -174,7 +280,7 @@ esac
 # If any of the arguments below are present, then the mode is vcheck.
 # In vcheck mode, nothing is added in terms of extra search paths or
 # libraries.
-if [[ -z $mode ]] || [[ $mode == ld ]]; then
+if [ -z "$mode" ] || [ "$mode" = ld ]; then
     for arg in "$@"; do
         case $arg in
             -v|-V|--version|-dumpversion)
@@ -186,16 +292,16 @@ if [[ -z $mode ]] || [[ $mode == ld ]]; then
 fi
 
 # Finish setting up the mode.
-if [[ -z $mode ]]; then
+if [ -z "$mode" ]; then
     mode=ccld
     for arg in "$@"; do
-        if [[ $arg == -E ]]; then
+        if [ "$arg" = "-E" ]; then
             mode=cpp
             break
-        elif [[ $arg == -S ]]; then
+        elif [ "$arg" = "-S" ]; then
             mode=as
             break
-        elif [[ $arg == -c ]]; then
+        elif [ "$arg" = "-c" ]; then
             mode=cc
             break
         fi
@@ -222,17 +328,18 @@ dtags_to_strip="${SPACK_DTAGS_TO_STRIP}"
 linker_arg="${SPACK_LINKER_ARG}"
 
 # Set up rpath variable according to language.
-eval rpath=\$SPACK_${comp}_RPATH_ARG
+rpath="ERROR: RPATH ARG WAS NOT SET"
+eval "rpath=\${SPACK_${comp}_RPATH_ARG:?${rpath}}"
 
 # Dump the mode and exit if the command is dump-mode.
-if [[ $SPACK_TEST_COMMAND == dump-mode ]]; then
+if [ "$SPACK_TEST_COMMAND" = "dump-mode" ]; then
     echo "$mode"
     exit
 fi
 
 # Check that at least one of the real commands was actually selected,
 # otherwise we don't know what to execute.
-if [[ -z $command ]]; then
+if [ -z "$command" ]; then
     die "ERROR: Compiler '$SPACK_COMPILER_SPEC' does not support compiling $language programs."
 fi
 
@@ -240,24 +347,26 @@ fi
 # Filter '.' and Spack environment directories out of PATH so that
 # this script doesn't just call itself
 #
-IFS=':' read -ra env_path <<< "$PATH"
-IFS=':' read -ra spack_env_dirs <<< "$SPACK_ENV_PATH"
-spack_env_dirs+=("" ".")
-export PATH=""
-for dir in "${env_path[@]}"; do
+new_dirs=""
+IFS=':'
+for dir in $PATH; do
     addpath=true
-    for env_dir in "${spack_env_dirs[@]}"; do
-        if [[ "${dir%%/}" == "$env_dir" ]]; then
-            addpath=false
-            break
-        fi
+    for spack_env_dir in $SPACK_ENV_PATH; do
+        case "${dir%%/}" in
+            "$spack_env_dir"|'.'|'')
+                addpath=false
+                break
+                ;;
+        esac
     done
-    if $addpath; then
-        export PATH="${PATH:+$PATH:}$dir"
+    if [ $addpath = true ]; then
+        append new_dirs "$dir"
     fi
 done
+unset IFS
+export PATH="$new_dirs"
 
-if [[ $mode == vcheck ]]; then
+if [ "$mode" = vcheck ]; then
     exec "${command}" "$@"
 fi
 
@@ -265,16 +374,20 @@ fi
 # It doesn't work with -rpath.
 # This variable controls whether they are added.
 add_rpaths=true
-if [[ ($mode == ld || $mode == ccld) && "$SPACK_SHORT_SPEC" =~ "darwin" ]];
-then
-    for arg in "$@"; do
-        if [[ ($arg == -r && $mode == ld) ||
-              ($arg == -r && $mode == ccld) ||
-              ($arg == -Wl,-r && $mode == ccld) ]]; then
-            add_rpaths=false
-            break
-        fi
-    done
+if [ "$mode" = ld ] || [ "$mode" = ccld ]; then
+    if [ "${SPACK_SHORT_SPEC#*darwin}" != "${SPACK_SHORT_SPEC}" ]; then
+        for arg in "$@"; do
+            if [ "$arg" = "-r" ]; then
+                if [ "$mode" = ld ] || [ "$mode" = ccld ]; then
+                    add_rpaths=false
+                    break
+                fi
+            elif [ "$arg" = "-Wl,-r" ] && [ "$mode" = ccld ]; then
+                add_rpaths=false
+                break
+            fi
+        done
+    fi
 fi
 
 # Save original command for debug logging
@@ -297,17 +410,22 @@ input_command="$*"
 # The libs variable is initialized here for completeness, and it is also
 # used later to inject flags supplied via `ldlibs` on the command
 # line. These come into the wrappers via SPACK_LDLIBS.
-#
-includes=()
-libdirs=()
-rpaths=()
-system_includes=()
-system_libdirs=()
-system_rpaths=()
-libs=()
-other_args=()
-isystem_system_includes=()
-isystem_includes=()
+
+# The loop below breaks up the command line into these lists of components.
+# The lists are all bell-separated to be as flexible as possible, as their
+# contents may come from the command line, from ' '-separated lists,
+# ':'-separated lists, etc.
+include_dirs_list=""
+lib_dirs_list=""
+rpath_dirs_list=""
+system_include_dirs_list=""
+system_lib_dirs_list=""
+system_rpath_dirs_list=""
+isystem_system_include_dirs_list=""
+isystem_include_dirs_list=""
+libs_list=""
+other_args_list=""
+
 
 while [ $# -ne 0 ]; do
 
@@ -327,32 +445,32 @@ while [ $# -ne 0 ]; do
             isystem_was_used=true
             if [ -z "$arg" ]; then shift; arg="$1"; fi
             if system_dir "$arg"; then
-                isystem_system_includes+=("$arg")
+                append isystem_system_include_dirs_list "$arg"
             else
-                isystem_includes+=("$arg")
+                append isystem_include_dirs_list "$arg"
             fi
             ;;
         -I*)
             arg="${1#-I}"
             if [ -z "$arg" ]; then shift; arg="$1"; fi
             if system_dir "$arg"; then
-                system_includes+=("$arg")
+                append system_include_dirs_list "$arg"
             else
-                includes+=("$arg")
+                append include_dirs_list "$arg"
             fi
             ;;
         -L*)
             arg="${1#-L}"
             if [ -z "$arg" ]; then shift; arg="$1"; fi
             if system_dir "$arg"; then
-                system_libdirs+=("$arg")
+                append system_lib_dirs_list "$arg"
             else
-                libdirs+=("$arg")
+                append lib_dirs_list "$arg"
             fi
             ;;
         -l*)
             # -loopopt=0 is generated erroneously in autoconf <= 2.69,
-            # and passed by ifx to the linker, which confuses it with a 
+            # and passed by ifx to the linker, which confuses it with a
             # library. Filter it out.
             # TODO: generalize filtering of args with an env var, so that
             # TODO: we do not have to special case this here.
@@ -363,66 +481,76 @@ while [ $# -ne 0 ]; do
             fi
             arg="${1#-l}"
             if [ -z "$arg" ]; then shift; arg="$1"; fi
-            other_args+=("-l$arg")
+            append other_args_list "-l$arg"
             ;;
         -Wl,*)
             arg="${1#-Wl,}"
             if [ -z "$arg" ]; then shift; arg="$1"; fi
-            if [[ "$arg" = -rpath=* ]]; then
-                rp="${arg#-rpath=}"
-            elif [[ "$arg" = --rpath=* ]]; then
-                rp="${arg#--rpath=}"
-            elif [[ "$arg" = -rpath,* ]]; then
-                rp="${arg#-rpath,}"
-            elif [[ "$arg" = --rpath,* ]]; then
-                rp="${arg#--rpath,}"
-            elif [[ "$arg" =~ ^-?-rpath$ ]]; then
-                shift; arg="$1"
-                if [[ "$arg" != -Wl,* ]]; then
-                    die "-Wl,-rpath was not followed by -Wl,*"
-                fi
-                rp="${arg#-Wl,}"
-            elif [[ "$arg" = "$dtags_to_strip" ]] ; then
-                :  # We want to remove explicitly this flag
-            else
-                other_args+=("-Wl,$arg")
-            fi
+            case "$arg" in
+                -rpath=*)  rp="${arg#-rpath=}"  ;;
+                --rpath=*) rp="${arg#--rpath=}" ;;
+                -rpath,*)  rp="${arg#-rpath,}"  ;;
+                --rpath,*) rp="${arg#--rpath,}" ;;
+                -rpath|--rpath)
+                    shift; arg="$1"
+                    case "$arg" in
+                        -Wl,*)
+                            rp="${arg#-Wl,}"
+                            ;;
+                        *)
+                            die "-Wl,-rpath was not followed by -Wl,*"
+                            ;;
+                    esac
+                    ;;
+                "$dtags_to_strip")
+                    :  # We want to remove explicitly this flag
+                    ;;
+                *)
+                    append other_args_list "-Wl,$arg"
+                    ;;
+            esac
             ;;
         -Xlinker,*)
             arg="${1#-Xlinker,}"
             if [ -z "$arg" ]; then shift; arg="$1"; fi
-            if [[ "$arg" = -rpath=* ]]; then
-                rp="${arg#-rpath=}"
-            elif [[ "$arg" = --rpath=* ]]; then
-                rp="${arg#--rpath=}"
-            elif [[ "$arg" = -rpath  ]] || [[ "$arg" = --rpath ]]; then
-                shift; arg="$1"
-                if [[ "$arg" != -Xlinker,* ]]; then
-                    die "-Xlinker,-rpath was not followed by -Xlinker,*"
-                fi
-                rp="${arg#-Xlinker,}"
-            else
-                other_args+=("-Xlinker,$arg")
-            fi
+
+            case "$arg" in
+                -rpath=*)  rp="${arg#-rpath=}"  ;;
+                --rpath=*) rp="${arg#--rpath=}" ;;
+                -rpath|--rpath)
+                    shift; arg="$1"
+                    case "$arg" in
+                        -Xlinker,*)
+                            rp="${arg#-Xlinker,}"
+                        ;;
+                        *)
+                            die "-Xlinker,-rpath was not followed by -Xlinker,*"
+                            ;;
+                    esac
+                    ;;
+                *)
+                    append other_args_list "-Xlinker,$arg"
+                    ;;
+            esac
             ;;
         -Xlinker)
-            if [[ "$2" == "-rpath" ]]; then
-                if [[ "$3" != "-Xlinker" ]]; then
+            if [ "$2" = "-rpath" ]; then
+                if [ "$3" != "-Xlinker" ]; then
                     die "-Xlinker,-rpath was not followed by -Xlinker,*"
                 fi
                 shift 3;
                 rp="$1"
-            elif [[ "$2" = "$dtags_to_strip" ]] ; then
+            elif [ "$2" = "$dtags_to_strip" ]; then
                 shift  # We want to remove explicitly this flag
             else
-                other_args+=("$1")
+                append other_args_list "$1"
             fi
             ;;
         *)
-            if [[ "$1" = "$dtags_to_strip" ]] ; then
+            if [ "$1" = "$dtags_to_strip" ]; then
                 :  # We want to remove explicitly this flag
             else
-                other_args+=("$1")
+                append other_args_list "$1"
             fi
             ;;
     esac
@@ -430,9 +558,9 @@ while [ $# -ne 0 ]; do
     # test rpaths against system directories in one place.
     if [ -n "$rp" ]; then
         if system_dir "$rp"; then
-            system_rpaths+=("$rp")
+            append system_rpath_dirs_list "$rp"
         else
-            rpaths+=("$rp")
+            append rpath_dirs_list "$rp"
         fi
     fi
     shift
@@ -445,16 +573,15 @@ done
 # See the gmake manual on implicit rules for details:
 # https://www.gnu.org/software/make/manual/html_node/Implicit-Variables.html
 #
-flags=()
+flags_list=""
 
 # Add debug flags
-if [ "${SPACK_ADD_DEBUG_FLAGS}" == "true" ]; then
-    flags=("${flags[@]}" "${debug_flags}")
+if [ "${SPACK_ADD_DEBUG_FLAGS}" = "true" ]; then
+    extend flags_list debug_flags
 
 # If a custom flag is requested, derive from environment
-elif [ "$SPACK_ADD_DEBUG_FLAGS" == "custom" ]; then
-    IFS=' ' read -ra SPACK_DEBUG_FLAGS <<< "$SPACK_DEBUG_FLAGS"
-    flags=("${flags[@]}" "${SPACK_DEBUG_FLAGS[@]}")
+elif [ "$SPACK_ADD_DEBUG_FLAGS" = "custom" ]; then
+    extend flags_list SPACK_DEBUG_FLAGS
 fi
 
 # Fortran flags come before CPPFLAGS
@@ -462,7 +589,8 @@ case "$mode" in
     cc|ccld)
         case $lang_flags in
             F)
-                flags=("${flags[@]}" "${SPACK_FFLAGS[@]}") ;;
+                extend flags_list SPACK_FFLAGS
+                ;;
         esac
         ;;
 esac
@@ -470,7 +598,8 @@ esac
 # C preprocessor flags come before any C/CXX flags
 case "$mode" in
     cpp|as|cc|ccld)
-        flags=("${flags[@]}" "${SPACK_CPPFLAGS[@]}") ;;
+        extend flags_list SPACK_CPPFLAGS
+        ;;
 esac
 
 
@@ -479,67 +608,67 @@ case "$mode" in
     cc|ccld)
         case $lang_flags in
             C)
-                flags=("${flags[@]}" "${SPACK_CFLAGS[@]}") ;;
+                extend flags_list SPACK_CFLAGS
+                ;;
             CXX)
-                flags=("${flags[@]}" "${SPACK_CXXFLAGS[@]}") ;;
+                extend flags_list SPACK_CXXFLAGS
+                ;;
         esac
-        flags=(${SPACK_TARGET_ARGS[@]} "${flags[@]}")
+
+        # prepend target args
+        preextend flags_list SPACK_TARGET_ARGS
         ;;
 esac
 
 # Linker flags
 case "$mode" in
     ld|ccld)
-        flags=("${flags[@]}" "${SPACK_LDFLAGS[@]}") ;;
+        extend flags_list SPACK_LDFLAGS
+        ;;
 esac
 
 # On macOS insert headerpad_max_install_names linker flag
-if [[ ($mode == ld || $mode == ccld) && "$SPACK_SHORT_SPEC" =~ "darwin" ]];
-then
-    case "$mode" in
-        ld)
-            flags=("${flags[@]}" -headerpad_max_install_names) ;;
-        ccld)
-            flags=("${flags[@]}" "-Wl,-headerpad_max_install_names") ;;
-    esac
+if [ "$mode" = ld ] || [ "$mode" = ccld ]; then
+    if [ "${SPACK_SHORT_SPEC#*darwin}" != "${SPACK_SHORT_SPEC}" ]; then
+        case "$mode" in
+            ld)
+                append flags_list "-headerpad_max_install_names" ;;
+            ccld)
+                append flags_list "-Wl,-headerpad_max_install_names" ;;
+        esac
+    fi
 fi
 
-IFS=':' read -ra rpath_dirs <<< "$SPACK_RPATH_DIRS"
-if [[ $mode == ccld || $mode == ld ]]; then
-
-    if [[ "$add_rpaths" != "false" ]] ; then
+if [ "$mode" = ccld ] || [ "$mode" = ld ]; then
+    if [ "$add_rpaths" != "false" ]; then
         # Append RPATH directories. Note that in the case of the
         # top-level package these directories may not exist yet. For dependencies
         # it is assumed that paths have already been confirmed.
-        rpaths=("${rpaths[@]}" "${rpath_dirs[@]}")
+        extend rpath_dirs_list SPACK_RPATH_DIRS
     fi
-
 fi
 
-IFS=':' read -ra link_dirs <<< "$SPACK_LINK_DIRS"
-if [[ $mode == ccld || $mode == ld ]]; then
-    libdirs=("${libdirs[@]}" "${link_dirs[@]}")
+if [ "$mode" = ccld ] || [ "$mode" = ld ]; then
+    extend lib_dirs_list SPACK_LINK_DIRS
 fi
 
 # add RPATHs if we're in in any linking mode
 case "$mode" in
     ld|ccld)
         # Set extra RPATHs
-        IFS=':' read -ra extra_rpaths <<< "$SPACK_COMPILER_EXTRA_RPATHS"
-        libdirs+=("${extra_rpaths[@]}")
-        if [[ "$add_rpaths" != "false" ]] ; then
-            rpaths+=("${extra_rpaths[@]}")
+        extend lib_dirs_list SPACK_COMPILER_EXTRA_RPATHS
+        if [ "$add_rpaths" != "false" ]; then
+            extend rpath_dirs_list SPACK_COMPILER_EXTRA_RPATHS
         fi
 
         # Set implicit RPATHs
-        IFS=':' read -ra implicit_rpaths <<< "$SPACK_COMPILER_IMPLICIT_RPATHS"
-        if [[ "$add_rpaths" != "false" ]] ; then
-            rpaths+=("${implicit_rpaths[@]}")
+        if [ "$add_rpaths" != "false" ]; then
+            extend rpath_dirs_list SPACK_COMPILER_IMPLICIT_RPATHS
         fi
 
         # Add SPACK_LDLIBS to args
-        for lib in "${SPACK_LDLIBS[@]}"; do
-            libs+=("${lib#-l}")
+        for lib in $SPACK_LDLIBS; do
+            append libs_list "${lib#-l}"
         done
         ;;
 esac
@@ -547,63 +676,62 @@ esac
 #
 # Finally, reassemble the command line.
 #
-
-# Includes and system includes first
-args=()
-
-# flags assembled earlier
-args+=("${flags[@]}")
+args_list="$flags_list"
 
 # Insert include directories just prior to any system include directories
+# NOTE: adding ${lsep} to the prefix here turns every added element into two
+extend args_list include_dirs_list "-I"
+extend args_list isystem_include_dirs_list "-isystem${lsep}"
 
-for dir in "${includes[@]}";         do args+=("-I$dir"); done
-for dir in "${isystem_includes[@]}";         do args+=("-isystem" "$dir"); done
-
-IFS=':' read -ra spack_include_dirs <<< "$SPACK_INCLUDE_DIRS"
-if [[ $mode == cpp || $mode == cc || $mode == as || $mode == ccld ]]; then
-    if [[ "$isystem_was_used" == "true" ]] ; then
-        for dir in "${spack_include_dirs[@]}";  do args+=("-isystem" "$dir"); done
-    else
-        for dir in "${spack_include_dirs[@]}";  do args+=("-I$dir"); done
-    fi
-fi
+case "$mode" in
+    cpp|cc|as|ccld)
+        if [ "$isystem_was_used" = "true" ]; then
+            extend args_list SPACK_INCLUDE_DIRS "-isystem${lsep}"
+        else
+            extend args_list SPACK_INCLUDE_DIRS "-I"
+        fi
+        ;;
+esac
 
-for dir in "${system_includes[@]}";  do args+=("-I$dir"); done
-for dir in "${isystem_system_includes[@]}";  do args+=("-isystem" "$dir"); done
+extend args_list system_include_dirs_list -I
+extend args_list isystem_system_include_dirs_list "-isystem${lsep}"
 
 # Library search paths
-for dir in "${libdirs[@]}";          do args+=("-L$dir"); done
-for dir in "${system_libdirs[@]}";   do args+=("-L$dir"); done
+extend args_list lib_dirs_list "-L"
+extend args_list system_lib_dirs_list "-L"
 
 # RPATHs arguments
 case "$mode" in
     ccld)
-        if [ -n "$dtags_to_add" ] ; then args+=("$linker_arg$dtags_to_add") ; fi
-        for dir in "${rpaths[@]}";        do args+=("$rpath$dir"); done
-        for dir in "${system_rpaths[@]}"; do args+=("$rpath$dir"); done
+        if [ -n "$dtags_to_add" ] ; then
+            append args_list "$linker_arg$dtags_to_add"
+        fi
+        extend args_list rpath_dirs_list "$rpath"
+        extend args_list system_rpath_dirs_list "$rpath"
         ;;
     ld)
-        if [ -n "$dtags_to_add" ] ; then args+=("$dtags_to_add") ; fi
-        for dir in "${rpaths[@]}";        do args+=("-rpath" "$dir"); done
-        for dir in "${system_rpaths[@]}"; do args+=("-rpath" "$dir"); done
+        if [ -n "$dtags_to_add" ] ; then
+            append args_list "$dtags_to_add"
+        fi
+        extend args_list rpath_dirs_list "-rpath${lsep}"
+        extend args_list system_rpath_dirs_list "-rpath${lsep}"
         ;;
 esac
 
 # Other arguments from the input command
-args+=("${other_args[@]}")
+extend args_list other_args_list
 
 # Inject SPACK_LDLIBS, if supplied
-for lib in "${libs[@]}"; do
-    args+=("-l$lib");
-done
+extend args_list libs_list "-l"
 
-full_command=("$command" "${args[@]}")
+full_command_list="$command"
+extend full_command_list args_list
 
 # prepend the ccache binary if we're using ccache
 if [ -n "$SPACK_CCACHE_BINARY" ]; then
     case "$lang_flags" in
         C|CXX)  # ccache only supports C languages
-            full_command=("${SPACK_CCACHE_BINARY}" "${full_command[@]}")
+            prepend full_command_list "${SPACK_CCACHE_BINARY}"
             # workaround for stage being a temp folder
             # see #3761#issuecomment-294352232
             export CCACHE_NOHASHDIR=yes
@@ -612,25 +740,36 @@ if [ -n "$SPACK_CCACHE_BINARY" ]; then
 fi
 
 # dump the full command if the caller supplies SPACK_TEST_COMMAND=dump-args
-if [[ $SPACK_TEST_COMMAND == dump-args ]]; then
-    IFS="
-" && echo "${full_command[*]}"
-    exit
-elif [[ $SPACK_TEST_COMMAND =~ dump-env-* ]]; then
-    var=${SPACK_TEST_COMMAND#dump-env-}
-    echo "$0: $var: ${!var}"
-elif [[ -n $SPACK_TEST_COMMAND ]]; then
-    die "ERROR: Unknown test command"
+if [ -n "${SPACK_TEST_COMMAND=}" ]; then
+    case "$SPACK_TEST_COMMAND" in
+        dump-args)
+            IFS="$lsep"
+            for arg in $full_command_list; do
+                echo "$arg"
+            done
+            unset IFS
+            exit
+            ;;
+        dump-env-*)
+            var=${SPACK_TEST_COMMAND#dump-env-}
+            eval "printf '%s\n' \"\$0: \$var: \$$var\""
+            ;;
+        *)
+            die "ERROR: Unknown test command: '$SPACK_TEST_COMMAND'"
+            ;;
+    esac
 fi
 
 #
 # Write the input and output commands to debug logs if it's asked for.
 #
-if [[ $SPACK_DEBUG == TRUE ]]; then
+if [ "$SPACK_DEBUG" = TRUE ]; then
     input_log="$SPACK_DEBUG_LOG_DIR/spack-cc-$SPACK_DEBUG_LOG_ID.in.log"
     output_log="$SPACK_DEBUG_LOG_DIR/spack-cc-$SPACK_DEBUG_LOG_ID.out.log"
     echo "[$mode] $command $input_command" >> "$input_log"
-    echo "[$mode] ${full_command[*]}" >> "$output_log"
+    echo "[$mode] ${full_command_list}" >> "$output_log"
 fi
 
-exec "${full_command[@]}"
+# Execute the full command, preserving spaces with IFS set
+# to the alarm bell separator.
+IFS="$lsep"; exec $full_command_list