Refactor flake8 handling and tool compatibility (#20376)

This PR does three related things to try to improve developer tooling quality of life:

1. Adds new options to `.flake8` so it applies the rules of both `.flake8` and `.flake_package` based on paths in the repository.
2. Adds a re-factoring of the `spack flake8` logic into a flake8 plugin so using flake8 directly, or through editor or language server integration, only reports errors that `spack flake8` would.
3. Allows star import of `spack.pkgkit` in packages, since this is now the thing that needs to be imported for completion to work correctly in package files, it's nice to be able to do that.

I'm sorely tempted to sed over the whole repository and put `from spack.pkgkit import *` in every package, but at least being allowed to do it on a per-package basis helps.

As an example of what the result of this is:

```
~/Workspace/Projects/spack/spack develop* ⇣
❯ flake8 --format=pylint ./var/spack/repos/builtin/packages/kripke/package.py
./var/spack/repos/builtin/packages/kripke/package.py:6: [F403] 'from spack.pkgkit import *' used; unable to detect undefined names
./var/spack/repos/builtin/packages/kripke/package.py:25: [E501] line too long (88 > 79 characters)

~/Workspace/Projects/spack/spack refactor-flake8*
1 ❯ flake8 --format=spack ./var/spack/repos/builtin/packages/kripke/package.py

~/Workspace/Projects/spack/spack refactor-flake8*
❯ flake8 ./var/spack/repos/builtin/packages/kripke/package.py
```

* qa/flake8: update .flake8, spack formatter plugin

Adds:
* Modern flake8 settings for per-path/glob error ignores, allows
  packages to use the same `.flake8` as the rest of spack
* A spack formatter plugin to flake8 that implements the behavior of
  `spack flake8` for direct invocations.  Makes integration with
  developer tooling nicer, linting with flake8 reports only errors that
  `spack flake8` would report.  Using pyls and pyls-flake8, or any other
  non-format-dependent flake8 integration, now works with spack's rules.

* qa/flake8: allow star import of spack.pkgkit

To get working completion of directives and spack components it's
necessary to import the contents of spack.pkgkit.  At the moment doing
this makes flake8 displeased.  For now, allow spack.pkgkit and spack
both, next step is to ban spack * and require spack.pkgkit *.

* first cut at refactoring spack flake8

This version still copies all of the files to be checked as befire, and
some other things that probably aren't necessary, but it relies on the
spack formatter plugin to implement the ignore logic.

* keep flake8 from rejecting itself

* remove separate packages flake8 config

* fix failures from too many files

I ran into this in the PR converting pkgkit to std.  The solution in
that branch does not work in all cases as it turns out, and all the
workarounds I tried to use generated configs to get a single invocation
of flake8 with a filename optoion to work failed.  It's an astonishingly
frustrating config option.

Regardless, this removes all temporary file creation from the command
and relies on the plugin instead.  To work around the huge number of
files in spack and still allow the command to control what gets checked,
it scans files in batches of 100.  This is a completely arbitrary number
but was chosen to be safely under common line-length limits.  One
side-effect of this is that every 100 files the command will produce
output, rather than only at the end, which doesn't seem like a terrible
thing.

1 files changed, 33 insertions, 0 deletions

diff --git a/.flake8 b/.flake8
index 195c783726..04ca460eeb 100644
--- a/.flake8
+++ b/.flake8
@@ -30,3 +30,36 @@
 [flake8]
 ignore = E129,E221,E241,E272,E731,W503,W504,F999,N801,N813,N814
 max-line-length = 79
+
+# F4: Import
+# - F405: `name` may be undefined, or undefined from star imports: `module`
+#
+# F8: Name
+# - F821: undefined name `name`
+#
+per-file-ignores =
+  var/spack/repos/*/package.py:F405,F821
+
+# exclude things we usually do not want linting for.
+# These still get linted when passed explicitly, as when spack flake8 passes
+# them on the command line.
+exclude =
+  .git
+  etc/
+  opt/
+  share/
+  var/spack/cache/
+  var/spack/gpg*/
+  var/spack/junit-report/
+  var/spack/mock-configs/
+  lib/spack/external
+  __pycache__
+  var
+
+format = spack
+
+[flake8:local-plugins]
+report =
+  spack = flake8_formatter:SpackFormatter
+paths =
+  ./share/spack/qa/