config: overrides for caches and system and user scopes (#26735)

Spack's `system` and `user` scopes provide ways for administrators and
users to set global defaults for all Spack instances, but for use cases
where one wants a clean Spack installation, these scopes can be undesirable.
For example, users may want to opt out of global system configuration, or
they may want to ignore their own home directory settings when running in
a continuous integration environment.

Spack also, by default, keeps various caches and user data in `~/.spack`,
but users may want to override these locations.

Spack provides three environment variables that allow you to override or
opt out of configuration locations:

 * `SPACK_USER_CONFIG_PATH`: Override the path to use for the
   `user` (`~/.spack`) scope.

 * `SPACK_SYSTEM_CONFIG_PATH`: Override the path to use for the
   `system` (`/etc/spack`) scope.

 * `SPACK_DISABLE_LOCAL_CONFIG`: set this environment variable to completely
   disable *both* the system and user configuration directories. Spack will
   only consider its own defaults and `site` configuration locations.

And one that allows you to move the default cache location:

 * `SPACK_USER_CACHE_PATH`: Override the default path to use for user data
   (misc_cache, tests, reports, etc.)

With these settings, if you want to isolate Spack in a CI environment, you can do this:

   export SPACK_DISABLE_LOCAL_CONFIG=true
   export SPACK_USER_CACHE_PATH=/tmp/spack

This is a stop-gap approach until we have figured out how to deal with
the system and user config scopes more generally, as there are plans to
potentially / eventually get rid of them.

**User config**

Spack is a bit of a pain when you have:

- a shared $HOME folder across different systems.
- multiple Spack versions on the same system.

**System config**

- On shared systems with a versioned programming environment / toolkit,
  system administrators want to provide config for each version (e.g.
  21.09, 21.10) of the programming environment, and the user Spack
  instance should be able to pick this up without a steep learning
  curve.
- On shared systems the user should be able to opt out of the
  hard-coded config scope in /etc/spack, since it may be incompatible
  with their particular instance. Currently Spack can only opt out of all
  config scopes through overrides with `"config:":`, `"packages:":`, but that
  also drops the defaults config, which would have to be repeated, which
  is undesirable, especially the lengthy packages.yaml.

An example use case is: having config in this folder:

```
/path/to/programming/environment/{version}/{compilers,packages}.yaml
```

and have `module load spack-system-config` set the variable

```
SPACK_SYSTEM_CONFIG_PATH=/path/to/programming/environment/{version}
```

where the user no longer has to worry about what `{version}` they are
on.

**Continuous integration**

Finally, there is the use case of continuous integration, which may
clone an arbitrary Spack version, which optimally should not pick up
system or user config from the previous run (like may happen in
classical bare metal non-containerized filesystem side effect ridden
jenkins pipelines). In fact this is very similar to how spack itself
tries to avoid picking up system dependencies during builds...

**But environments solve this?**

- You could do `include`s in environment files to get similar behavior
  to the spack_system_config_path example, but environments require you
  to:
  1) require paths to individual config files, not directories.
  2) fail if the listed config file does not exist
- They allow you to override config scopes, but this is generally too
  rigurous, as it requires you to repeat the default config, in
  particular packages.yaml, and just defies the point of layered config.

Co-authored-by: Tom Scogland <tscogland@llnl.gov>
Co-authored-by: Tim Fuller <tjfulle@sandia.gov>
Co-authored-by: Steve Leak <sleak@lbl.gov>
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>

2 files changed, 6 insertions, 6 deletions

diff --git a/etc/spack/defaults/bootstrap.yaml b/etc/spack/defaults/bootstrap.yaml
index 392c48b7bb..7a72bdfe6a 100644
--- a/etc/spack/defaults/bootstrap.yaml
+++ b/etc/spack/defaults/bootstrap.yaml
@@ -4,7 +4,7 @@ bootstrap:
   enable: true
   # Root directory for bootstrapping work. The software bootstrapped
   # by Spack is installed in a "store" subfolder of this root directory
-  root: ~/.spack/bootstrap
+  root: $user_config_path/bootstrap
   # Methods that can be used to bootstrap software. Each method may or
   # may not be able to bootstrap all of the software that Spack needs,
   # depending on its type.
diff --git a/etc/spack/defaults/config.yaml b/etc/spack/defaults/config.yaml
index 3487d33162..eb0d4fc409 100644
--- a/etc/spack/defaults/config.yaml
+++ b/etc/spack/defaults/config.yaml
@@ -42,8 +42,8 @@ config:
   # (i.e., ``$TMP` or ``$TMPDIR``).
   #
   # Another option that prevents conflicts and potential permission issues is
-  # to specify `~/.spack/stage`, which ensures each user builds in their home
-  # directory.
+  # to specify `$user_cache_path/stage`, which ensures each user builds in their
+  # home directory.
   #
   # A more traditional path uses the value of `$spack/var/spack/stage`, which
   # builds directly inside Spack's instance without staging them in a
@@ -60,13 +60,13 @@ config:
   # identifies Spack staging to avoid accidentally wiping out non-Spack work.
   build_stage:
     - $tempdir/$user/spack-stage
-    - ~/.spack/stage
+    - $user_cache_path/stage
   # - $spack/var/spack/stage
 
   # Directory in which to run tests and store test results.
   # Tests will be stored in directories named by date/time and package
   # name/hash.
-  test_stage: ~/.spack/test
+  test_stage: $user_cache_path/test
 
   # Cache directory for already downloaded source tarballs and archived
   # repositories. This can be purged with `spack clean --downloads`.
@@ -75,7 +75,7 @@ config:
 
   # Cache directory for miscellaneous files, like the package index.
   # This can be purged with `spack clean --misc-cache`
-  misc_cache: ~/.spack/cache
+  misc_cache: $user_cache_path/cache
 
 
   # Timeout in seconds used for downloading sources etc. This only applies