Add command for reading JSON-based DB description (now with more tests) (#29652)

This is an amended version of https://github.com/spack/spack/pull/24894 (reverted in https://github.com/spack/spack/pull/29603). https://github.com/spack/spack/pull/24894
broke all instances of `spack external find` (namely when it is invoked without arguments/options)
because it was mandating the presence of a file which most systems would not have.
This allows `spack external find` to proceed if that file is not present and adds tests for this.

- [x] Add a test which confirms that `spack external find` successfully reads a manifest file
      if present in the default manifest path

--- Original commit message ---

Adds `spack external read-cray-manifest`, which reads a json file that describes a
set of package DAGs. The parsed results are stored directly in the database. A user
can see these installed specs with `spack find` (like any installed spec). The easiest
way to use them right now as dependencies is to run
`spack spec ... ^/hash-of-external-package`.

Changes include:

* `spack external read-cray-manifest --file <path/to/file>` will add all specs described
  in the file to Spack's installation DB and will also install described compilers to the
  compilers configuration (the expected format of the file is described in this PR as well including examples of the file)
* Database records now may include an "origin" (the command added in this PR
  registers the origin as "external-db"). In the future, it is assumed users may want
  to be able to treat installs registered with this command differently (e.g. they may
  want to uninstall all specs added with this command)
* Hash properties are now always preserved when copying specs if the source spec
  is concrete
  * I don't think the hashes of installed-and-concrete specs should change and this
    was the easiest way to handle that
  * also specs that are concrete preserve their `.normal` property when copied
    (external specs may mention compilers that are not registered, and without this
    change they would fail in `normalize` when calling `validate_or_raise`)
  * it might be this should only be the case if the spec was installed

- [x] Improve testing
- [x] Specifically mark DB records added with this command (so that users can do
      something like "uninstall all packages added with `spack read-external-db`)
  * This is now possible with `spack uninstall --all --origin=external-db` (this will
    remove all specs added from manifest files)
- [x] Strip variants that are listed in json entries but don't actually exist for the package

1 files changed, 10 insertions, 0 deletions

diff --git a/var/spack/repos/builtin.mock/packages/hwloc/package.py b/var/spack/repos/builtin.mock/packages/hwloc/package.py
new file mode 100644
index 0000000000..2cd984f438
--- /dev/null
+++ b/var/spack/repos/builtin.mock/packages/hwloc/package.py
@@ -0,0 +1,10 @@
+# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+from spack import *
+
+
+class Hwloc(Package):
+    version('2.0.3')