summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
-rw-r--r--lib/spack/docs/analyze.rst162
-rw-r--r--lib/spack/docs/developer_guide.rst442
-rw-r--r--lib/spack/docs/index.rst7
-rw-r--r--lib/spack/docs/monitoring.rst94
-rw-r--r--lib/spack/spack/analyzers/__init__.py43
-rw-r--r--lib/spack/spack/analyzers/analyzer_base.py115
-rw-r--r--lib/spack/spack/analyzers/config_args.py32
-rw-r--r--lib/spack/spack/analyzers/environment_variables.py51
-rw-r--r--lib/spack/spack/analyzers/install_files.py30
-rw-r--r--lib/spack/spack/analyzers/libabigail.py116
-rw-r--r--lib/spack/spack/architecture.py20
-rw-r--r--lib/spack/spack/binary_distribution.py74
-rw-r--r--lib/spack/spack/cmd/analyze.py118
-rw-r--r--lib/spack/spack/cmd/install.py25
-rw-r--r--lib/spack/spack/cmd/spec.py11
-rw-r--r--lib/spack/spack/directory_layout.py15
-rw-r--r--lib/spack/spack/environment.py21
-rw-r--r--lib/spack/spack/hooks/__init__.py13
-rw-r--r--lib/spack/spack/hooks/monitor.py73
-rw-r--r--lib/spack/spack/installer.py103
-rw-r--r--lib/spack/spack/monitor.py459
-rw-r--r--lib/spack/spack/package.py9
-rw-r--r--lib/spack/spack/paths.py1
-rw-r--r--lib/spack/spack/spec.py8
-rw-r--r--lib/spack/spack/test/cmd/analyze.py176
-rw-r--r--lib/spack/spack/test/cmd/install.py36
-rw-r--r--lib/spack/spack/test/installer.py26
-rw-r--r--lib/spack/spack/util/classes.py39
-rw-r--r--lib/spack/spack/util/environment.py36
-rwxr-xr-xshare/spack/spack-completion.bash28
-rw-r--r--var/spack/repos/builtin/packages/libabigail/package.py3
31 files changed, 2281 insertions, 105 deletions
diff --git a/lib/spack/docs/analyze.rst b/lib/spack/docs/analyze.rst
new file mode 100644
index 0000000000..38af77cd7f
--- /dev/null
+++ b/lib/spack/docs/analyze.rst
@@ -0,0 +1,162 @@
+.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
+ Spack Project Developers. See the top-level COPYRIGHT file for details.
+
+ SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+.. _analyze:
+
+=======
+Analyze
+=======
+
+
+The analyze command is a front-end to various tools that let us analyze
+package installations. Each analyzer is a module for a different kind
+of analysis that can be done on a package installation, including (but not
+limited to) binary, log, or text analysis. Thus, the analyze command group
+allows you to take an existing package install, choose an analyzer,
+and extract some output for the package using it.
+
+
+-----------------
+Analyzer Metadata
+-----------------
+
+For all analyzers, we write to an ``analyzers`` folder in ``~/.spack``, or the
+value that you specify in your spack config at ``config:analyzers_dir``.
+For example, here we see the results of running an analysis on zlib:
+
+.. code-block:: console
+
+ $ tree ~/.spack/analyzers/
+ └── linux-ubuntu20.04-skylake
+ └── gcc-9.3.0
+ └── zlib-1.2.11-sl7m27mzkbejtkrajigj3a3m37ygv4u2
+ ├── environment_variables
+ │   └── spack-analyzer-environment-variables.json
+ ├── install_files
+ │   └── spack-analyzer-install-files.json
+ └── libabigail
+ └── spack-analyzer-libabigail-libz.so.1.2.11.xml
+
+
+This means that you can always find analyzer output in this folder, and it
+is organized with the same logic as the package install it was run for.
+If you want to customize this top level folder, simply provide the ``--path``
+argument to ``spack analyze run``. The nested organization will be maintained
+within your custom root.
+
+-----------------
+Listing Analyzers
+-----------------
+
+If you aren't familiar with Spack's analyzers, you can quickly list those that
+are available:
+
+.. code-block:: console
+
+ $ spack analyze list-analyzers
+ install_files : install file listing read from install_manifest.json
+ environment_variables : environment variables parsed from spack-build-env.txt
+ config_args : config args loaded from spack-configure-args.txt
+ abigail : Application Binary Interface (ABI) features for objects
+
+
+In the above, the first three are fairly simple - parsing metadata files from
+a package install directory to save
+
+-------------------
+Analyzing a Package
+-------------------
+
+The analyze command, akin to install, will accept a package spec to perform
+an analysis for. The package must be installed. Let's walk through an example
+with zlib. We first ask to analyze it. However, since we have more than one
+install, we are asked to disambiguate:
+
+.. code-block:: console
+
+ $ spack analyze run zlib
+ ==> Error: zlib matches multiple packages.
+ Matching packages:
+ fz2bs56 zlib@1.2.11%gcc@7.5.0 arch=linux-ubuntu18.04-skylake
+ sl7m27m zlib@1.2.11%gcc@9.3.0 arch=linux-ubuntu20.04-skylake
+ Use a more specific spec.
+
+
+We can then specify the spec version that we want to analyze:
+
+.. code-block:: console
+
+ $ spack analyze run zlib/fz2bs56
+
+If you don't provide any specific analyzer names, by default all analyzers
+(shown in the ``list-analyzers`` subcommand list) will be run. If an analyzer does not
+have any result, it will be skipped. For example, here is a result running for
+zlib:
+
+.. code-block:: console
+
+ $ ls ~/.spack/analyzers/linux-ubuntu20.04-skylake/gcc-9.3.0/zlib-1.2.11-sl7m27mzkbejtkrajigj3a3m37ygv4u2/
+ spack-analyzer-environment-variables.json
+ spack-analyzer-install-files.json
+ spack-analyzer-libabigail-libz.so.1.2.11.xml
+
+If you want to run a specific analyzer, ask for it with `--analyzer`. Here we run
+spack analyze on libabigail (already installed) _using_ libabigail1
+
+.. code-block:: console
+
+ $ spack analyze run --analyzer abigail libabigail
+
+
+.. _analyze_monitoring:
+
+----------------------
+Monitoring An Analysis
+----------------------
+
+For any kind of analysis, you can
+use a `spack monitor <https://github.com/spack/spack-monitor>`_ "Spackmon"
+as a server to upload the same run metadata to. You can
+follow the instructions in the `spack monitor documentation <https://spack-monitor.readthedocs.org>`_
+to first create a server along with a username and token for yourself.
+You can then use this guide to interact with the server.
+
+You should first export our spack monitor token and username to the environment:
+
+.. code-block:: console
+
+ $ export SPACKMON_TOKEN=50445263afd8f67e59bd79bff597836ee6c05438
+ $ export SPACKMON_USER=spacky
+
+
+By default, the host for your server is expected to be at ``http://127.0.0.1``
+with a prefix of ``ms1``, and if this is the case, you can simply add the
+``--monitor`` flag to the install command:
+
+.. code-block:: console
+
+ $ spack analyze run --monitor wget
+
+If you need to customize the host or the prefix, you can do that as well:
+
+.. code-block:: console
+
+ $ spack analyze run --monitor --monitor-prefix monitor --monitor-host https://monitor-service.io wget
+
+If your server doesn't have authentication, you can skip it:
+
+.. code-block:: console
+
+ $ spack analyze run --monitor --monitor-disable-auth wget
+
+Regardless of your choice, when you run analyze on an installed package (whether
+it was installed with ``--monitor`` or not, you'll see the results generating as they did
+before, and a message that the monitor server was pinged:
+
+.. code-block:: console
+
+ $ spack analyze --monitor wget
+ ...
+ ==> Sending result for wget bin/wget to monitor.
diff --git a/lib/spack/docs/developer_guide.rst b/lib/spack/docs/developer_guide.rst
index 9964bb6e51..4e349cff45 100644
--- a/lib/spack/docs/developer_guide.rst
+++ b/lib/spack/docs/developer_guide.rst
@@ -106,11 +106,21 @@ with a high level view of Spack's directory structure:
external/ <- external libs included in Spack distro
llnl/ <- some general-use libraries
- spack/ <- spack module; contains Python code
- cmd/ <- each file in here is a spack subcommand
- compilers/ <- compiler description files
- test/ <- unit test modules
- util/ <- common code
+ spack/ <- spack module; contains Python code
+ analyzers/ <- modules to run analysis on installed packages
+ build_systems/ <- modules for different build systems
+ cmd/ <- each file in here is a spack subcommand
+ compilers/ <- compiler description files
+ container/ <- module for spack containerize
+ hooks/ <- hook modules to run at different points
+ modules/ <- modules for lmod, tcl, etc.
+ operating_systems/ <- operating system modules
+ platforms/ <- different spack platforms
+ reporters/ <- reporters like cdash, junit
+ schema/ <- schemas to validate data structures
+ solver/ <- the spack solver
+ test/ <- unit test modules
+ util/ <- common code
Spack is designed so that it could live within a `standard UNIX
directory hierarchy <http://linux.die.net/man/7/hier>`_, so ``lib``,
@@ -251,6 +261,22 @@ Unit tests
This is a fake package hierarchy used to mock up packages for
Spack's test suite.
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Research and Monitoring Modules
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+:mod:`spack.monitor`
+ Contains :class:`SpackMonitor <spack.monitor.SpackMonitor>`. This is accessed
+ from the ``spack install`` and ``spack analyze`` commands to send build
+ and package metadada up to a `Spack Monitor <https://github.com/spack/spack-monitor>`_ server.
+
+
+:mod:`spack.analyzers`
+ A module folder with a :class:`AnalyzerBase <spack.analyzers.analyzer_base.AnalyzerBase>`
+ that provides base functions to run, save, and (optionally) upload analysis
+ results to a `Spack Monitor <https://github.com/spack/spack-monitor>`_ server.
+
+
^^^^^^^^^^^^^
Other Modules
^^^^^^^^^^^^^
@@ -299,6 +325,235 @@ Conceptually, packages are overloaded. They contain:
Stage objects
-------------
+
+.. _writing-analyzers:
+
+-----------------
+Writing analyzers
+-----------------
+
+To write an analyzer, you should add a new python file to the
+analyzers module directory at ``lib/spack/spack/analyzers`` .
+Your analyzer should be a subclass of the :class:`AnalyzerBase <spack.analyzers.analyzer_base.AnalyzerBase>`. For example, if you want
+to add an analyzer class ``Myanalyzer`` you woul write to
+``spack/analyzers/myanalyzer.py`` and import and
+use the base as follows:
+
+.. code-block:: python
+
+ from .analyzer_base import AnalyzerBase
+
+ class Myanalyzer(AnalyzerBase):
+
+
+Note that the class name is your module file name, all lowercase
+except for the first capital letter. You can look at other analyzers in
+that analyzer directory for examples. The guide here will tell you about the basic functions needed.
+
+^^^^^^^^^^^^^^^^^^^^^^^^^
+Analyzer Output Directory
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+By default, when you run ``spack analyze run`` an analyzer output directory will
+be created in your spack user directory in your ``$HOME``. The reason we output here
+is because the install directory might not always be writable.
+
+.. code-block:: console
+
+ ~/.spack/
+ analyzers
+
+Result files will be written here, organized in subfolders in the same structure
+as the package, with each analyzer owning it's own subfolder. for example:
+
+
+.. code-block:: console
+
+ $ tree ~/.spack/analyzers/
+ /home/spackuser/.spack/analyzers/
+ └── linux-ubuntu20.04-skylake
+ └── gcc-9.3.0
+ └── zlib-1.2.11-sl7m27mzkbejtkrajigj3a3m37ygv4u2
+ ├── environment_variables
+ │   └── spack-analyzer-environment-variables.json
+ ├── install_files
+ │   └── spack-analyzer-install-files.json
+ └── libabigail
+ └── lib
+ └── spack-analyzer-libabigail-libz.so.1.2.11.xml
+
+
+Notice that for the libabigail analyzer, since results are generated per object,
+we honor the object's folder in case there are equivalently named files in
+different folders. The result files are typically written as json so they can be easily read and uploaded in a future interaction with a monitor.
+
+
+^^^^^^^^^^^^^^^^^
+Analyzer Metadata
+^^^^^^^^^^^^^^^^^
+
+Your analyzer is required to have the class attributes ``name``, ``outfile``,
+and ``description``. These are printed to the user with they use the subcommand
+``spack analyze list-analyzers``. Here is an example.
+As we mentioned above, note that this analyzer would live in a module named
+``libabigail.py`` in the analyzers folder so that the class can be discovered.
+
+
+.. code-block:: python
+
+ class Libabigail(AnalyzerBase):
+
+ name = "libabigail"
+ outfile = "spack-analyzer-libabigail.json"
+ description = "Application Binary Interface (ABI) features for objects"
+
+
+This means that the name and output file should be unique for your analyzer.
+Note that "all" cannot be the name of an analyzer, as this key is used to indicate
+that the user wants to run all analyzers.
+
+.. _analyzer_run_function:
+
+
+^^^^^^^^^^^^^^^^^^^^^^^^
+An analyzer run Function
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+The core of an analyzer is its ``run()`` function, which should accept no
+arguments. You can assume your analyzer has the package spec of interest at ``self.spec``
+and it's up to the run function to generate whatever analysis data you need,
+and then return the object with a key as the analyzer name. The result data
+should be a list of objects, each with a name, ``analyzer_name``, ``install_file``,
+and one of ``value`` or ``binary_value``. The install file should be for a relative
+path, and not the absolute path. For example, let's say we extract a metric called
+``metric`` for ``bin/wget`` using our analyzer ``thebest-analyzer``.
+We might have data that looks like this:
+
+.. code-block:: python
+
+ result = {"name": "metric", "analyzer_name": "thebest-analyzer", "value": "1", "install_file": "bin/wget"}
+
+
+We'd then return it as follows - note that they key is the analyzer name at ``self.name``.
+
+.. code-block:: python
+
+ return {self.name: result}
+
+This will save the complete result to the analyzer metadata folder, as described
+previously. If you want support for adding a different kind of metadata (e.g.,
+not associated with an install file) then the monitor server would need to be updated
+to support this first.
+
+
+^^^^^^^^^^^^^^^^^^^^^^^^^
+An analyzer init Function
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you don't need any extra dependencies or checks, you can skip defining an analyzer
+init function, as the base class will handle it. Typically, it will accept
+a spec, and an optional output directory (if the user does not want the default
+metadata folder for analyzer results). The analyzer init function should call
+it's parent init, and then do any extra checks or validation that are required to
+work. For example:
+
+.. code-block:: python
+
+ def __init__(self, spec, dirname=None):
+ super(Myanalyzer, self).__init__(spec, dirname)
+
+ # install extra dependencies, do extra preparation and checks here
+
+
+At the end of the init, you will have available to you:
+
+ - **self.spec**: the spec object
+ - **self.dirname**: an optional directory name the user as provided at init to save
+ - **self.output_dir**: the analyzer metadata directory, where we save by default
+ - **self.meta_dir**: the path to the package metadata directory (.spack) if you need it
+
+And can proceed to write your analyzer.
+
+
+^^^^^^^^^^^^^^^^^^^^^^^
+Saving Analyzer Results
+^^^^^^^^^^^^^^^^^^^^^^^
+
+The analyzer will have ``save_result`` called, with the result object generated
+to save it to the filesystem, and if the user has added the ``--monitor`` flag
+to upload it to a monitor server. If your result follows an accepted result
+format and you don't need to parse it further, you don't need to add this
+function to your class. However, if your result data is large or otherwise
+needs additional parsing, you can define it. If you define the function, it
+is useful to know about the ``output_dir`` property, which you can join
+with your output file relative path of choice:
+
+.. code-block:: python
+
+ outfile = os.path.join(self.output_dir, "my-output-file.txt")
+
+
+The directory will be provided by the ``output_dir`` property but it won't exist,
+so you should create it:
+
+
+.. code::block:: python
+
+ # Create the output directory
+ if not os.path.exists(self._output_dir):
+ os.makedirs(self._output_dir)
+
+
+If you are generating results that match to specific files in the package
+install directory, you should try to maintain those paths in the case that
+there are equivalently named files in different directories that would
+overwrite one another. As an example of an analyzer with a custom save,
+the Libabigail analyzer saves ``*.xml`` files to the analyzer metadata
+folder in ``run()``, as they are either binaries, or as xml (text) would
+usually be too big to pass in one request. For this reason, the files
+are saved during ``run()`` and the filenames added to the result object,
+and then when the result object is passed back into ``save_result()``,
+we skip saving to the filesystem, and instead read the file and send
+each one (separately) to the monitor:
+
+
+.. code-block:: python
+
+ def save_result(self, result, monitor=None, overwrite=False):
+ """ABI results are saved to individual files, so each one needs to be
+ read and uploaded. Result here should be the lookup generated in run(),
+ the key is the analyzer name, and each value is the result file.
+ We currently upload the entire xml as text because libabigail can't
+ easily read gzipped xml, but this will be updated when it can.
+ """
+ if not monitor:
+ return
+
+ name = self.spec.package.name
+
+ for obj, filename in result.get(self.name, {}).items():
+
+ # Don't include the prefix
+ rel_path = obj.replace(self.spec.prefix + os.path.sep, "")
+
+ # We've already saved the results to file during run
+ content = spack.monitor.read_file(filename)
+
+ # A result needs an analyzer, value or binary_value, and name
+ data = {"value": content, "install_file": rel_path, "name": "abidw-xml"}
+ tty.info("Sending result for %s %s to monitor." % (name, rel_path))
+ monitor.send_analyze_metadata(self.spec.package, {"libabigail": [data]})
+
+
+
+Notice that this function, if you define it, requires a result object (generated by
+``run()``, a monitor (if you want to send), and a boolean ``overwrite`` to be used
+to check if a result exists first, and not write to it if the result exists and
+overwrite is False. Also notice that since we already saved these files to the analyzer metadata folder, we return early if a monitor isn't defined, because this function serves to send results to the monitor. If you haven't saved anything to the analyzer metadata folder
+yet, you might want to do that here. You should also use ``tty.info`` to give
+the user a message of "Writing result to $DIRNAME."
+
+
.. _writing-commands:
----------------
@@ -345,6 +600,183 @@ Whenever you add/remove/rename a command or flags for an existing command,
make sure to update Spack's `Bash tab completion script
<https://github.com/adamjstewart/spack/blob/develop/share/spack/spack-completion.bash>`_.
+
+-------------
+Writing Hooks
+-------------
+
+A hook is a callback that makes it easy to design functions that run
+for different events. We do this by way of defining hook types, and then
+inserting them at different places in the spack code base. Whenever a hook
+type triggers by way of a function call, we find all the hooks of that type,
+and run them.
+
+Spack defines hooks by way of a module at ``lib/spack/spack/hooks`` where we can define
+types of hooks in the ``__init__.py``, and then python files in that folder
+can use hook functions. The files are automatically parsed, so if you write
+a new file for some integration (e.g., ``lib/spack/spack/hooks/myintegration.py``
+you can then write hook functions in that file that will be automatically detected,
+and run whenever your hook is called. This section will cover the basic kind
+of hooks, and how to write them.
+
+^^^^^^^^^^^^^^
+Types of Hooks
+^^^^^^^^^^^^^^
+
+The following hooks are currently implemented to make it easy for you,
+the developer, to add hooks at different stages of a spack install or similar.
+If there is a hook that you would like and is missing, you can propose to add a new one.
+
+"""""""""""""""""""""
+``pre_install(spec)``
+"""""""""""""""""""""
+
+A ``pre_install`` hook is run within an install subprocess, directly before
+the install starts. It expects a single argument of a spec, and is run in
+a multiprocessing subprocess. Note that if you see ``pre_install`` functions associated with packages these are not hooks
+as we have defined them here, but rather callback functions associated with
+a package install.
+
+
+""""""""""""""""""""""
+``post_install(spec)``
+""""""""""""""""""""""
+
+A ``post_install`` hook is run within an install subprocess, directly after
+the install finishes, but before the build stage is removed. If you
+write one of these hooks, you should expect it to accept a spec as the only
+argument. This is run in a multiprocessing subprocess. This ``post_install`` is
+also seen in packages, but in this context not related to the hooks described
+here.
+
+
+""""""""""""""""""""""""""
+``on_install_start(spec)``
+""""""""""""""""""""""""""
+
+This hook is run at the beginning of ``lib/spack/spack/installer.py``,
+in the install function of a ``PackageInstaller``,
+and importantly is not part of a build process, but before it. This is when
+we have just newly grabbed the task, and are preparing to install. If you
+write a hook of this type, you should provide the spec to it.
+
+.. code-block:: python
+
+ def on_install_start(spec):
+ """On start of an install, we want to...
+ """
+ print('on_install_start')
+
+
+""""""""""""""""""""""""""""
+``on_install_success(spec)``
+""""""""""""""""""""""""""""
+
+This hook is run on a successful install, and is also run inside the build
+process, akin to ``post_install``. The main difference is that this hook
+is run outside of the context of the stage directory, meaning after the
+build stage has been removed and the user is alerted that the install was
+successful. If you need to write a hook that is run on success of a particular
+phase, you should use ``on_phase_success``.
+
+""""""""""""""""""""""""""""
+``on_install_failure(spec)``
+""""""""""""""""""""""""""""
+
+This hook is run given an install failure that happens outside of the build
+subprocess, but somewhere in ``installer.py`` when something else goes wrong.
+If you need to write a hook that is relevant to a failure within a build
+process, you would want to instead use ``on_phase_failure``.
+
+
+"""""""""""""""""""""""""""""""""""""""""""""""
+``on_phase_success(pkg, phase_name, log_file)``
+"""""""""""""""""""""""""""""""""""""""""""""""
+
+This hook is run within the install subprocess, and specifically when a phase
+successfully finishes. Since we are interested in the package, the name of
+the phase, and any output from it, we require:
+
+ - **pkg**: the package variable, which also has the attached spec at ``pkg.spec``
+ - **phase_name**: the name of the phase that was successful (e.g., configure)
+ - **log_file**: the path to the file with output, in case you need to inspect or otherwise interact with it.
+
+"""""""""""""""""""""""""""""""""""""""""""""
+``on_phase_error(pkg, phase_name, log_file)``
+"""""""""""""""""""""""""""""""""""""""""""""
+
+In the case of an error during a phase, we might want to trigger some event
+with a hook, and this is the purpose of this particular hook. Akin to
+``on_phase_success`` we require the same variables - the package that failed,
+the name of the phase, and the log file where we might find errors.
+
+"""""""""""""""""""""""""""""""""
+``on_analyzer_save(pkg, result)``
+"""""""""""""""""""""""""""""""""
+
+After an analyzer has saved some result for a package, this hook is called,
+and it provides the package that we just ran the analysis for, along with
+the loaded result. Typically, a result is structured to have the name
+of the analyzer as key, and the result object that is defined in detail in
+:ref:`analyzer_run_function`.
+
+.. code-block:: python
+
+ def on_analyzer_save(pkg, result):
+ """given a package and a result...
+ """
+ print('Do something extra with a package analysis result here')
+
+
+^^^^^^^^^^^^^^^^^^^^^^
+Adding a New Hook Type
+^^^^^^^^^^^^^^^^^^^^^^
+
+Adding a new hook type is very simple! In ``lib/spack/spack/hooks/__init__.py``
+you can simply create a new ``HookRunner`` that is named to match your new hook.
+For example, let's say you want to add a new hook called ``post_log_write``
+to trigger after anything is written to a logger. You would add it as follows:
+
+.. code-block:: python
+
+ # pre/post install and run by the install subprocess
+ pre_install = HookRunner('pre_install')
+ post_install = HookRunner('post_install')
+
+ # hooks related to logging
+ post_log_write = HookRunner('post_log_write') # <- here is my new hook!
+
+
+You then need to decide what arguments my hook would expect. Since this is
+related to logging, let's say that you want a message and level. That means
+that when you add a python file to the ``lib/spack/spack/hooks``
+folder with one or more callbacks intended to be triggered by this hook. You might
+use my new hook as follows:
+
+.. code-block:: python
+
+ def post_log_write(message, level):
+ """Do something custom with the messsage and level every time we write
+ to the log
+ """
+ print('running post_log_write!')
+
+
+To use the hook, we would call it as follows somewhere in the logic to do logging.
+In this example, we use it outside of a logger that is already defined:
+
+.. code-block:: python
+
+ import spack.hooks
+
+ # We do something here to generate a logger and message
+ spack.hooks.post_log_write(message, logger.level)
+
+
+This is not to say that this would be the best way to implement an integration
+with the logger (you'd probably want to write a custom logger, or you could
+have the hook defined within the logger) but serves as an example of writing a hook.
+
----------
Unit tests
----------
diff --git a/lib/spack/docs/index.rst b/lib/spack/docs/index.rst
index 7a9ac09ec2..fb9c669c12 100644
--- a/lib/spack/docs/index.rst
+++ b/lib/spack/docs/index.rst
@@ -67,6 +67,7 @@ or refer to the full manual below.
build_settings
environments
containers
+ monitoring
mirrors
module_file_support
repositories
@@ -79,6 +80,12 @@ or refer to the full manual below.
.. toctree::
:maxdepth: 2
+ :caption: Research
+
+ analyze
+
+.. toctree::
+ :maxdepth: 2
:caption: Contributing
contribution_guide
diff --git a/lib/spack/docs/monitoring.rst b/lib/spack/docs/monitoring.rst
new file mode 100644
index 0000000000..266f4dfe0a
--- /dev/null
+++ b/lib/spack/docs/monitoring.rst
@@ -0,0 +1,94 @@
+.. Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
+ Spack Project Developers. See the top-level COPYRIGHT file for details.
+
+ SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+.. _monitoring:
+
+==========
+Monitoring
+==========
+
+You can use a `spack monitor <https://github.com/spack/spack-monitor>`_ "Spackmon"
+server to store a database of your packages, builds, and associated metadata
+for provenance, research, or some other kind of development. You should
+follow the instructions in the `spack monitor documentation <https://spack-monitor.readthedocs.org>`_
+to first create a server along with a username and token for yourself.
+You can then use this guide to interact with the server.
+
+-------------------
+Analysis Monitoring
+-------------------
+
+To read about how to monitor an analysis (meaning you want to send analysis results
+to a server) see :ref:`analyze_monitoring`.
+
+---------------------
+Monitoring An Install
+---------------------
+
+Since an install is typically when you build packages, we logically want
+to tell spack to monitor during this step. Let's start with an example
+where we want to monitor the install of hdf5. Unless you have disabled authentication
+for the server, we first want to export our spack monitor token and username to the environment:
+
+.. code-block:: console
+
+ $ export SPACKMON_TOKEN=50445263afd8f67e59bd79bff597836ee6c05438
+ $ export SPACKMON_USER=spacky
+
+
+By default, the host for your server is expected to be at ``http://127.0.0.1``
+with a prefix of ``ms1``, and if this is the case, you can simply add the
+``--monitor`` flag to the install command:
+
+.. code-block:: console
+
+ $ spack install --monitor hdf5
+
+
+If you need to customize the host or the prefix, you can do that as well:
+
+.. code-block:: console
+
+ $ spack install --monitor --monitor-prefix monitor --monitor-host https://monitor-service.io hdf5
+
+
+As a precaution, we cut out early in the spack client if you have not provided
+authentication credentials. For example, if you run the command above without
+exporting your username or token, you'll see:
+
+.. code-block:: console
+
+ ==> Error: You are required to export SPACKMON_TOKEN and SPACKMON_USER
+
+This extra check is to ensure that we don't start any builds,
+and then discover that you forgot to export your token. However, if
+your monitoring server has authentication disabled, you can tell this to
+the client to skip this step:
+
+.. code-block:: console
+
+ $ spack install --monitor --monitor-disable-auth hdf5
+
+If the service is not running, you'll cleanly exit early - the install will
+not continue if you've asked it to monitor and there is no service.
+For example, here is what you'll see if the monitoring service is not running:
+
+.. code-block:: console
+
+ [Errno 111] Connection refused
+
+
+If you want to continue builds (and stop monitoring) you can set the ``--monitor-keep-going``
+flag.
+
+.. code-block:: console
+
+ $ spack install --monitor --monitor-keep-going hdf5
+
+This could mean that if a request fails, you only have partial or no data
+added to your monitoring database. This setting will not be applied to the
+first request to check if the server is running, but to subsequent requests.
+If you don't have a monitor server running and you want to build, simply
+don't provide the ``--monitor`` flag!
diff --git a/lib/spack/spack/analyzers/__init__.py b/lib/spack/spack/analyzers/__init__.py
new file mode 100644
index 0000000000..9e36ed7b3f
--- /dev/null
+++ b/lib/spack/spack/analyzers/__init__.py
@@ -0,0 +1,43 @@
+# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+"""This package contains code for creating analyzers to extract Application
+Binary Interface (ABI) information, along with simple analyses that just load
+existing metadata.
+"""
+
+from __future__ import absolute_import
+
+import spack.util.classes
+import spack.paths
+
+import llnl.util.tty as tty
+
+
+mod_path = spack.paths.analyzers_path
+analyzers = spack.util.classes.list_classes("spack.analyzers", mod_path)
+
+# The base analyzer does not have a name, and cannot do dict comprehension
+analyzer_types = {}
+for a in analyzers:
+ if not hasattr(a, "name"):
+ continue
+ analyzer_types[a.name] = a
+
+
+def list_all():
+ """A helper function to list all analyzers and their descriptions
+ """
+ for name, analyzer in analyzer_types.items():
+ print("%-25s: %-35s" % (name, analyzer.description))
+
+
+def get_analyzer(name):
+ """Courtesy function to retrieve an analyzer, and exit on error if it
+ does not exist.
+ """
+ if name in analyzer_types:
+ return analyzer_types[name]
+ tty.die("Analyzer %s does not exist" % name)
diff --git a/lib/spack/spack/analyzers/analyzer_base.py b/lib/spack/spack/analyzers/analyzer_base.py
new file mode 100644
index 0000000000..41f456b71a
--- /dev/null
+++ b/lib/spack/spack/analyzers/analyzer_base.py
@@ -0,0 +1,115 @@
+# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+"""An analyzer base provides basic functions to run the analysis, save results,
+and (optionally) interact with a Spack Monitor
+"""
+
+import spack.monitor
+import spack.hooks
+import llnl.util.tty as tty
+import spack.util.path
+import spack.config
+
+import os
+
+
+def get_analyzer_dir(spec, analyzer_dir=None):
+ """
+ Given a spec, return the directory to save analyzer results.
+
+ We create the directory if it does not exist. We also check that the
+ spec has an associated package. An analyzer cannot be run if the spec isn't
+ associated with a package. If the user provides a custom analyzer_dir,
+ we use it over checking the config and the default at ~/.spack/analyzers
+ """
+ # An analyzer cannot be run if the spec isn't associated with a package
+ if not hasattr(spec, "package") or not spec.package:
+ tty.die("A spec can only be analyzed with an associated package.")
+
+ # The top level directory is in the user home, or a custom location
+ if not analyzer_dir:
+ analyzer_dir = spack.util.path.canonicalize_path(
+ spack.config.get('config:analyzers_dir', '~/.spack/analyzers'))
+
+ # We follow the same convention as the spec install (this could be better)
+ package_prefix = os.sep.join(spec.package.prefix.split('/')[-3:])
+ meta_dir = os.path.join(analyzer_dir, package_prefix)
+ return meta_dir
+
+
+class AnalyzerBase(object):
+
+ def __init__(self, spec, dirname=None):
+ """
+ Verify that the analyzer has correct metadata.
+
+ An Analyzer is intended to run on one spec install, so the spec
+ with its associated package is required on init. The child analyzer
+ class should define an init function that super's the init here, and
+ also check that the analyzer has all dependencies that it
+ needs. If an analyzer subclass does not have dependencies, it does not
+ need to define an init. An Analyzer should not be allowed to proceed
+ if one or more dependencies are missing. The dirname, if defined,
+ is an optional directory name to save to (instead of the default meta
+ spack directory).
+ """
+ self.spec = spec
+ self.dirname = dirname
+ self.meta_dir = os.path.dirname(spec.package.install_log_path)
+
+ for required in ["name", "outfile", "description"]:
+ if not hasattr(self, required):
+ tty.die("Please add a %s attribute on the analyzer." % required)
+
+ def run(self):
+ """
+ Given a spec with an installed package, run the analyzer on it.
+ """
+ raise NotImplementedError
+
+ @property
+ def output_dir(self):
+ """
+ The full path to the output directory.
+
+ This includes the nested analyzer directory structure. This function
+ does not create anything.
+ """
+ if not hasattr(self, "_output_dir"):
+ output_dir = get_analyzer_dir(self.spec, self.dirname)
+ self._output_dir = os.path.join(output_dir, self.name)
+
+ return self._output_dir
+
+ def save_result(self, result, overwrite=False):
+ """
+ Save a result to the associated spack monitor, if defined.
+
+ This function is on the level of the analyzer because it might be
+ the case that the result is large (appropriate for a single request)
+ or that the data is organized differently (e.g., more than one
+ request per result). If an analyzer subclass needs to over-write
+ this function with a custom save, that is appropriate to do (see abi).
+ """
+ # We maintain the structure in json with the analyzer as key so
+ # that in the future, we could upload to a monitor server
+ if result[self.name]:
+
+ outfile = os.path.join(self.output_dir, self.outfile)
+
+ # Only try to create the results directory if we have a result
+ if not os.path.exists(self._output_dir):
+ os.makedirs(self._output_dir)
+
+ # Don't overwrite an existing result if overwrite is False
+ if os.path.exists(outfile) and not overwrite:
+ tty.info("%s exists and overwrite is False, skipping." % outfile)
+ else:
+ tty.info("Writing result to %s" % outfile)
+ spack.monitor.write_json(result[self.name], outfile)
+
+ # This hook runs after a save result
+ spack.hooks.on_analyzer_save(self.spec.package, result)
diff --git a/lib/spack/spack/analyzers/config_args.py b/lib/spack/spack/analyzers/config_args.py
new file mode 100644
index 0000000000..f29993dcb3
--- /dev/null
+++ b/lib/spack/spack/analyzers/config_args.py
@@ -0,0 +1,32 @@
+# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+"""A configargs analyzer is a class of analyzer that typically just uploads
+already existing metadata about config args from a package spec install
+directory."""
+
+
+import spack.monitor
+from .analyzer_base import AnalyzerBase
+
+import os
+
+
+class ConfigArgs(AnalyzerBase):
+
+ name = "config_args"
+ outfile = "spack-analyzer-config-args.json"
+ description = "config args loaded from spack-configure-args.txt"
+
+ def run(self):
+ """
+ Load the configure-args.txt and save in json.
+
+ The run function will find the spack-config-args.txt file in the
+ package install directory, and read it into a json structure that has
+ the name of the analyzer as the key.
+ """
+ config_file = os.path.join(self.meta_dir, "spack-configure-args.txt")
+ return {self.name: spack.monitor.read_file(config_file)}
diff --git a/lib/spack/spack/analyzers/environment_variables.py b/lib/spack/spack/analyzers/environment_variables.py
new file mode 100644
index 0000000000..33c4034e31
--- /dev/null
+++ b/lib/spack/spack/analyzers/environment_variables.py
@@ -0,0 +1,51 @@
+# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+"""An environment analyzer will read and parse the environment variables
+file in the installed package directory, generating a json file that has
+an index of key, value pairs for environment variables."""
+
+
+from .analyzer_base import AnalyzerBase
+from spack.util.environment import EnvironmentModifications
+
+
+import os
+
+
+class EnvironmentVariables(AnalyzerBase):
+
+ name = "environment_variables"
+ outfile = "spack-analyzer-environment-variables.json"
+ description = "environment variables parsed from spack-build-env.txt"
+
+ def run(self):
+ """
+ Load, parse, and save spack-build-env.txt to analyzers.
+
+ Read in the spack-build-env.txt file from the package install
+ directory and parse the environment variables into key value pairs.
+ The result should have the key for the analyzer, the name.
+ """
+ env_file = os.path.join(self.meta_dir, "spack-build-env.txt")
+ return {self.name: self._read_environment_file(env_file)}
+
+ def _read_environment_file(self, filename):
+ """
+ Read and parse the environment file.
+
+ Given an environment file, we want to read it, split by semicolons
+ and new lines, and then parse down to the subset of SPACK_* variables.
+ We assume that all spack prefix variables are not secrets, and unlike
+ the install_manifest.json, we don't (at least to start) parse the values
+ to remove path prefixes specific to user systems.
+ """
+ if not os.path.exists(filename):
+ return
+
+ mods = EnvironmentModifications.from_sourcing_file(filename)
+ env = {}
+ mods.apply_modifications(env)
+ return env
diff --git a/lib/spack/spack/analyzers/install_files.py b/lib/spack/spack/analyzers/install_files.py
new file mode 100644
index 0000000000..bdebac50fc
--- /dev/null
+++ b/lib/spack/spack/analyzers/install_files.py
@@ -0,0 +1,30 @@
+# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+"""The install files json file (install_manifest.json) already exists in
+the package install folder, so this analyzer simply moves it to the user
+analyzer folder for further processing."""
+
+
+import spack.monitor
+from .analyzer_base import AnalyzerBase
+
+import os
+
+
+class InstallFiles(AnalyzerBase):
+
+ name = "install_files"
+ outfile = "spack-analyzer-install-files.json"
+ description = "install file listing read from install_manifest.json"
+
+ def run(self):
+ """
+ Load in the install_manifest.json and save to analyzers.
+
+ We write it out to the analyzers folder, with key as the analyzer name.
+ """
+ manifest_file = os.path.join(self.meta_dir, "install_manifest.json")
+ return {self.name: spack.monitor.read_json(manifest_file)}
diff --git a/lib/spack/spack/analyzers/libabigail.py b/lib/spack/spack/analyzers/libabigail.py
new file mode 100644
index 0000000000..b53369b75f
--- /dev/null
+++ b/lib/spack/spack/analyzers/libabigail.py
@@ -0,0 +1,116 @@
+# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+
+import spack
+import spack.error
+import spack.bootstrap
+import spack.hooks
+import spack.monitor
+import spack.binary_distribution
+import spack.package
+import spack.repo
+
+import llnl.util.tty as tty
+
+from .analyzer_base import AnalyzerBase
+
+import os
+
+
+class Libabigail(AnalyzerBase):
+
+ name = "libabigail"
+ outfile = "spack-analyzer-libabigail.json"
+ description = "Application Binary Interface (ABI) features for objects"
+
+ def __init__(self, spec, dirname=None):
+ """
+ init for an analyzer ensures we have all needed dependencies.
+
+ For the libabigail analyzer, this means Libabigail.
+ Since the output for libabigail is one file per object, we communicate
+ with the monitor multiple times.
+ """
+ super(Libabigail, self).__init__(spec, dirname)
+
+ # This doesn't seem to work to import on the module level
+ tty.debug("Preparing to use Libabigail, will install if missing.")
+
+ with spack.bootstrap.ensure_bootstrap_configuration():
+
+ # libabigail won't install lib/bin/share without docs
+ spec = spack.spec.Spec("libabigail+docs")
+ spec.concretize()
+
+ self.abidw = spack.bootstrap.get_executable(
+ "abidw", spec=spec, install=True)
+
+ def run(self):
+ """
+ Run libabigail, and save results to filename.
+
+ This run function differs in that we write as we generate and then
+ return a dict with the analyzer name as the key, and the value of a
+ dict of results, where the key is the object name, and the value is
+ the output file written to.
+ """
+ manifest = spack.binary_distribution.get_buildfile_manifest(self.spec)
+
+ # This result will store a path to each file
+ result = {}
+
+ # Generate an output file for each binary or object
+ for obj in manifest.get("binary_to_relocate_fullpath", []):
+
+ # We want to preserve the path in the install directory in case
+ # a library has an equivalenly named lib or executable, for example
+ outdir = os.path.dirname(obj.replace(self.spec.package.prefix,
+ '').strip(os.path.sep))
+ outfile = "spack-analyzer-libabigail-%s.xml" % os.path.basename(obj)
+ outfile = os.path.join(self.output_dir, outdir, outfile)
+ outdir = os.path.dirname(outfile)
+
+ # Create the output directory
+ if not os.path.exists(outdir):
+ os.makedirs(outdir)
+
+ # Sometimes libabigail segfaults and dumps
+ try:
+ self.abidw(obj, "--out-file", outfile)
+ result[obj] = outfile
+ tty.info("Writing result to %s" % outfile)
+ except spack.error.SpackError:
+ tty.warn("Issue running abidw for %s" % obj)
+
+ return {self.name: result}
+
+ def save_result(self, result, overwrite=False):
+ """
+ Read saved ABI results and upload to monitor server.
+
+ ABI results are saved to individual files, so each one needs to be
+ read and uploaded. Result here should be the lookup generated in run(),
+ the key is the analyzer name, and each value is the result file.
+ We currently upload the entire xml as text because libabigail can't
+ easily read gzipped xml, but this will be updated when it can.
+ """
+ if not spack.monitor.cli:
+ return
+
+ name = self.spec.package.name
+
+ for obj, filename in result.get(self.name, {}).items():
+
+ # Don't include the prefix
+ rel_path = obj.replace(self.spec.prefix + os.path.sep, "")
+
+ # We've already saved the results to file during run
+ content = spack.monitor.read_file(filename)
+
+ # A result needs an analyzer, value or binary_value, and name
+ data = {"value": content, "install_file": rel_path, "name": "abidw-xml"}
+ tty.info("Sending result for %s %s to monitor." % (name, rel_path))
+ spack.hooks.on_analyzer_save(self.spec.package, {"libabigail": [data]})
diff --git a/lib/spack/spack/architecture.py b/lib/spack/spack/architecture.py
index ba2d824010..04cd85be49 100644
--- a/lib/spack/spack/architecture.py
+++ b/lib/spack/spack/architecture.py
@@ -58,7 +58,6 @@ will be responsible for compiler detection.
"""
import contextlib
import functools
-import inspect
import warnings
import archspec.cpu
@@ -74,7 +73,7 @@ import spack.paths
import spack.error as serr
import spack.util.executable
import spack.version
-from spack.util.naming import mod_to_class
+import spack.util.classes
from spack.util.spack_yaml import syaml_dict
@@ -502,23 +501,8 @@ def arch_for_spec(arch_spec):
@lang.memoized
def _all_platforms():
- classes = []
mod_path = spack.paths.platform_path
- parent_module = "spack.platforms"
-
- for name in lang.list_modules(mod_path):
- mod_name = '%s.%s' % (parent_module, name)
- class_name = mod_to_class(name)
- mod = __import__(mod_name, fromlist=[class_name])
- if not hasattr(mod, class_name):
- tty.die('No class %s defined in %s' % (class_name, mod_name))
- cls = getattr(mod, class_name)
- if not inspect.isclass(cls):
- tty.die('%s.%s is not a class' % (mod_name, class_name))
-
- classes.append(cls)
-
- return classes
+ return spack.util.classes.list_classes("spack.platforms", mod_path)
@lang.memoized
diff --git a/lib/spack/spack/binary_distribution.py b/lib/spack/spack/binary_distribution.py
index c8477503b4..d23d0ba86a 100644
--- a/lib/spack/spack/binary_distribution.py
+++ b/lib/spack/spack/binary_distribution.py
@@ -550,40 +550,38 @@ def read_buildinfo_file(prefix):
return buildinfo
-def write_buildinfo_file(spec, workdir, rel=False):
+def get_buildfile_manifest(spec):
"""
- Create a cache file containing information
- required for the relocation
+ Return a data structure with information about a build, including
+ text_to_relocate, binary_to_relocate, binary_to_relocate_fullpath
+ link_to_relocate, and other, which means it doesn't fit any of previous
+ checks (and should not be relocated). We blacklist docs (man) and
+ metadata (.spack). This can be used to find a particular kind of file
+ in spack, or to generate the build metadata.
"""
- prefix = spec.prefix
- text_to_relocate = []
- binary_to_relocate = []
- link_to_relocate = []
+ data = {"text_to_relocate": [], "binary_to_relocate": [],
+ "link_to_relocate": [], "other": [],
+ "binary_to_relocate_fullpath": []}
+
blacklist = (".spack", "man")
- prefix_to_hash = dict()
- prefix_to_hash[str(spec.package.prefix)] = spec.dag_hash()
- deps = spack.build_environment.get_rpath_deps(spec.package)
- for d in deps:
- prefix_to_hash[str(d.prefix)] = d.dag_hash()
+
# Do this at during tarball creation to save time when tarball unpacked.
# Used by make_package_relative to determine binaries to change.
- for root, dirs, files in os.walk(prefix, topdown=True):
+ for root, dirs, files in os.walk(spec.prefix, topdown=True):
dirs[:] = [d for d in dirs if d not in blacklist]
for filename in files:
path_name = os.path.join(root, filename)
m_type, m_subtype = relocate.mime_type(path_name)
+ rel_path_name = os.path.relpath(path_name, spec.prefix)
+ added = False
+
if os.path.islink(path_name):
link = os.readlink(path_name)
if os.path.isabs(link):
# Relocate absolute links into the spack tree
if link.startswith(spack.store.layout.root):
- rel_path_name = os.path.relpath(path_name, prefix)
- link_to_relocate.append(rel_path_name)
- else:
- msg = 'Absolute link %s to %s ' % (path_name, link)
- msg += 'outside of prefix %s ' % prefix
- msg += 'should not be relocated.'
- tty.warn(msg)
+ data['link_to_relocate'].append(rel_path_name)
+ added = True
if relocate.needs_binary_relocation(m_type, m_subtype):
if ((m_subtype in ('x-executable', 'x-sharedlib')
@@ -591,11 +589,31 @@ def write_buildinfo_file(spec, workdir, rel=False):
(m_subtype in ('x-mach-binary')
and sys.platform == 'darwin') or
(not filename.endswith('.o'))):
- rel_path_name = os.path.relpath(path_name, prefix)
- binary_to_relocate.append(rel_path_name)
+ data['binary_to_relocate'].append(rel_path_name)
+ data['binary_to_relocate_fullpath'].append(path_name)
+ added = True
+
if relocate.needs_text_relocation(m_type, m_subtype):
- rel_path_name = os.path.relpath(path_name, prefix)
- text_to_relocate.append(rel_path_name)
+ data['text_to_relocate'].append(rel_path_name)
+ added = True
+
+ if not added:
+ data['other'].append(path_name)
+ return data
+
+
+def write_buildinfo_file(spec, workdir, rel=False):
+ """
+ Create a cache file containing information
+ required for the relocation
+ """
+ manifest = get_buildfile_manifest(spec)
+
+ prefix_to_hash = dict()
+ prefix_to_hash[str(spec.package.prefix)] = spec.dag_hash()
+ deps = spack.build_environment.get_rpath_deps(spec.package)
+ for d in deps:
+ prefix_to_hash[str(d.prefix)] = d.dag_hash()
# Create buildinfo data and write it to disk
import spack.hooks.sbang as sbang
@@ -605,10 +623,10 @@ def write_buildinfo_file(spec, workdir, rel=False):
buildinfo['buildpath'] = spack.store.layout.root
buildinfo['spackprefix'] = spack.paths.prefix
buildinfo['relative_prefix'] = os.path.relpath(
- prefix, spack.store.layout.root)
- buildinfo['relocate_textfiles'] = text_to_relocate
- buildinfo['relocate_binaries'] = binary_to_relocate
- buildinfo['relocate_links'] = link_to_relocate
+ spec.prefix, spack.store.layout.root)
+ buildinfo['relocate_textfiles'] = manifest['text_to_relocate']
+ buildinfo['relocate_binaries'] = manifest['binary_to_relocate']
+ buildinfo['relocate_links'] = manifest['link_to_relocate']
buildinfo['prefix_to_hash'] = prefix_to_hash
filename = buildinfo_file_name(workdir)
with open(filename, 'w') as outfile:
diff --git a/lib/spack/spack/cmd/analyze.py b/lib/spack/spack/cmd/analyze.py
new file mode 100644
index 0000000000..318fca1e4a
--- /dev/null
+++ b/lib/spack/spack/cmd/analyze.py
@@ -0,0 +1,118 @@
+# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+import sys
+
+import llnl.util.tty as tty
+
+import spack.analyzers
+import spack.build_environment
+import spack.cmd
+import spack.cmd.common.arguments as arguments
+import spack.environment as ev
+import spack.fetch_strategy
+import spack.monitor
+import spack.paths
+import spack.report
+
+
+description = "analyze installed packages"
+section = "extensions"
+level = "short"
+
+
+def setup_parser(subparser):
+ sp = subparser.add_subparsers(metavar='SUBCOMMAND', dest='analyze_command')
+
+ sp.add_parser('list-analyzers',
+ description="list available analyzers",
+ help="show list of analyzers that are available to run.")
+
+ # This adds the monitor group to the subparser
+ spack.monitor.get_monitor_group(subparser)
+
+ # Run Parser
+ run_parser = sp.add_parser('run', description="run an analyzer",
+ help="provide the name of the analyzer to run.")
+
+ run_parser.add_argument(
+ '--overwrite', action='store_true',
+ help="re-analyze even if the output file already exists.")
+ run_parser.add_argument(
+ '-p', '--path', default=None,
+ dest='path',
+ help="write output to a different directory than ~/.spack/analyzers")
+ run_parser.add_argument(
+ '-a', '--analyzers', default=None,
+ dest="analyzers", action="append",
+ help="add an analyzer (defaults to all available)")
+ arguments.add_common_arguments(run_parser, ['spec'])
+
+
+def analyze_spec(spec, analyzers=None, outdir=None, monitor=None, overwrite=False):
+ """
+ Do an analysis for a spec, optionally adding monitoring.
+
+ We also allow the user to specify a custom output directory.
+ analyze_spec(spec, args.analyzers, args.outdir, monitor)
+
+ Args:
+ spec (Spec): spec object of installed package
+ analyzers (list): list of analyzer (keys) to run
+ monitor (monitor.SpackMonitorClient): a monitor client
+ overwrite (bool): overwrite result if already exists
+ """
+ analyzers = analyzers or list(spack.analyzers.analyzer_types.keys())
+
+ # Load the build environment from the spec install directory, and send
+ # the spec to the monitor if it's not known
+ if monitor:
+ monitor.load_build_environment(spec)
+ monitor.new_configuration([spec])
+
+ for name in analyzers:
+
+ # Instantiate the analyzer with the spec and outdir
+ analyzer = spack.analyzers.get_analyzer(name)(spec, outdir)
+
+ # Run the analyzer to get a json result - results are returned as
+ # a dictionary with a key corresponding to the analyzer type, so
+ # we can just update the data
+ result = analyzer.run()
+
+ # Send the result. We do them separately because:
+ # 1. each analyzer might have differently organized output
+ # 2. the size of a result can be large
+ analyzer.save_result(result, overwrite)
+
+
+def analyze(parser, args, **kwargs):
+
+ # If the user wants to list analyzers, do so and exit
+ if args.analyze_command == "list-analyzers":
+ spack.analyzers.list_all()
+ sys.exit(0)
+
+ # handle active environment, if any
+ env = ev.get_env(args, 'analyze')
+
+ # Get an disambiguate spec (we should only have one)
+ specs = spack.cmd.parse_specs(args.spec)
+ if not specs:
+ tty.die("You must provide one or more specs to analyze.")
+ spec = spack.cmd.disambiguate_spec(specs[0], env)
+
+ # The user wants to monitor builds using github.com/spack/spack-monitor
+ # It is instantianted once here, and then available at spack.monitor.cli
+ monitor = None
+ if args.use_monitor:
+ monitor = spack.monitor.get_client(
+ host=args.monitor_host,
+ prefix=args.monitor_prefix,
+ disable_auth=args.monitor_disable_auth,
+ )
+
+ # Run the analysis
+ analyze_spec(spec, args.analyzers, args.path, monitor, args.overwrite)
diff --git a/lib/spack/spack/cmd/install.py b/lib/spack/spack/cmd/install.py
index 613c4b2422..26d824c128 100644
--- a/lib/spack/spack/cmd/install.py
+++ b/lib/spack/spack/cmd/install.py
@@ -17,6 +17,7 @@ import spack.cmd
import spack.cmd.common.arguments as arguments
import spack.environment as ev
import spack.fetch_strategy
+import spack.monitor
import spack.paths
import spack.report
from spack.error import SpackError
@@ -106,6 +107,8 @@ the dependencies"""
'--cache-only', action='store_true', dest='cache_only', default=False,
help="only install package from binary mirrors")
+ monitor_group = spack.monitor.get_monitor_group(subparser) # noqa
+
subparser.add_argument(
'--include-build-deps', action='store_true', dest='include_build_deps',
default=False, help="""include build deps when installing from cache,
@@ -224,6 +227,7 @@ def install_specs(cli_args, kwargs, specs):
def install(parser, args, **kwargs):
+
if args.help_cdash:
parser = argparse.ArgumentParser(
formatter_class=argparse.RawDescriptionHelpFormatter,
@@ -236,6 +240,14 @@ environment variables:
parser.print_help()
return
+ # The user wants to monitor builds using github.com/spack/spack-monitor
+ if args.use_monitor:
+ monitor = spack.monitor.get_client(
+ host=args.monitor_host,
+ prefix=args.monitor_prefix,
+ disable_auth=args.monitor_disable_auth,
+ )
+
reporter = spack.report.collect_info(
spack.package.PackageInstaller, '_install_task', args.log_format, args)
if args.log_file:
@@ -378,4 +390,17 @@ environment variables:
# overwrite all concrete explicit specs from this build
kwargs['overwrite'] = [spec.dag_hash() for spec in specs]
+ # Update install_args with the monitor args, needed for build task
+ kwargs.update({
+ "monitor_disable_auth": args.monitor_disable_auth,
+ "monitor_keep_going": args.monitor_keep_going,
+ "monitor_host": args.monitor_host,
+ "use_monitor": args.use_monitor,
+ "monitor_prefix": args.monitor_prefix,
+ })
+
+ # If we are using the monitor, we send configs. and create build
+ # The full_hash is the main package id, the build_hash for others
+ if args.use_monitor and specs:
+ monitor.new_configuration(specs)
install_specs(args, kwargs, zip(abstract_specs, specs))
diff --git a/lib/spack/spack/cmd/spec.py b/lib/spack/spack/cmd/spec.py
index 8b4e9e5276..3a66a05f0b 100644
--- a/lib/spack/spack/cmd/spec.py
+++ b/lib/spack/spack/cmd/spec.py
@@ -43,6 +43,10 @@ for further documentation regarding the spec syntax, see:
'-N', '--namespaces', action='store_true', default=False,
help='show fully qualified package names')
subparser.add_argument(
+ '--hash-type', default="build_hash",
+ choices=['build_hash', 'full_hash', 'dag_hash'],
+ help='generate spec with a particular hash type.')
+ subparser.add_argument(
'-t', '--types', action='store_true', default=False,
help='show dependency types')
arguments.add_common_arguments(subparser, ['specs'])
@@ -83,11 +87,14 @@ def spec(parser, args):
if spec.name in spack.repo.path or spec.virtual:
spec.concretize()
+ # The user can specify the hash type to use
+ hash_type = getattr(ht, args.hash_type)
+
if args.format == 'yaml':
# use write because to_yaml already has a newline.
- sys.stdout.write(spec.to_yaml(hash=ht.build_hash))
+ sys.stdout.write(spec.to_yaml(hash=hash_type))
else:
- print(spec.to_json(hash=ht.build_hash))
+ print(spec.to_json(hash=hash_type))
continue
with tree_context():
diff --git a/lib/spack/spack/directory_layout.py b/lib/spack/spack/directory_layout.py
index 41db24bd2c..8a429a850f 100644
--- a/lib/spack/spack/directory_layout.py
+++ b/lib/spack/spack/directory_layout.py
@@ -17,6 +17,7 @@ from llnl.util.filesystem import mkdirp
import spack.config
import spack.hash_types as ht
import spack.spec
+import spack.util.spack_json as sjson
from spack.error import SpackError
@@ -247,6 +248,17 @@ class YamlDirectoryLayout(DirectoryLayout):
# full provenance by full hash so it's availabe if we want it later
spec.to_yaml(f, hash=ht.full_hash)
+ def write_host_environment(self, spec):
+ """The host environment is a json file with os, kernel, and spack
+ versioning. We use it in the case that an analysis later needs to
+ easily access this information.
+ """
+ from spack.util.environment import get_host_environment_metadata
+ env_file = self.env_metadata_path(spec)
+ environ = get_host_environment_metadata()
+ with open(env_file, 'w') as fd:
+ sjson.dump(environ, fd)
+
def read_spec(self, path):
"""Read the contents of a file and parse them as a spec"""
try:
@@ -300,6 +312,9 @@ class YamlDirectoryLayout(DirectoryLayout):
def metadata_path(self, spec):
return os.path.join(spec.prefix, self.metadata_dir)
+ def env_metadata_path(self, spec):
+ return os.path.join(self.metadata_path(spec), "install_environment.json")
+
def build_packages_path(self, spec):
return os.path.join(self.metadata_path(spec), self.packages_dir)
diff --git a/lib/spack/spack/environment.py b/lib/spack/spack/environment.py
index 9e3ad68124..218cd11095 100644
--- a/lib/spack/spack/environment.py
+++ b/lib/spack/spack/environment.py
@@ -9,8 +9,6 @@ import re
import sys
import shutil
import copy
-import socket
-
import six
from ordereddict_backport import OrderedDict
@@ -33,13 +31,11 @@ import spack.config
import spack.user_environment as uenv
from spack.filesystem_view import YamlFilesystemView
import spack.util.environment
-import spack.architecture as architecture
from spack.spec import Spec
from spack.spec_list import SpecList, InvalidSpecConstraintError
from spack.variant import UnknownVariantError
import spack.util.lock as lk
from spack.util.path import substitute_path_variables
-from spack.installer import PackageInstaller
import spack.util.path
#: environment variable used to indicate the active environment
@@ -447,21 +443,11 @@ def _write_yaml(data, str_or_file):
def _eval_conditional(string):
"""Evaluate conditional definitions using restricted variable scope."""
- arch = architecture.Arch(
- architecture.platform(), 'default_os', 'default_target')
- arch_spec = spack.spec.Spec('arch=%s' % arch)
- valid_variables = {
- 'target': str(arch.target),
- 'os': str(arch.os),
- 'platform': str(arch.platform),
- 'arch': arch_spec,
- 'architecture': arch_spec,
- 'arch_str': str(arch),
+ valid_variables = spack.util.environment.get_host_environment()
+ valid_variables.update({
're': re,
'env': os.environ,
- 'hostname': socket.gethostname()
- }
-
+ })
return eval(string, valid_variables)
@@ -1454,6 +1440,7 @@ class Environment(object):
args (Namespace): argparse namespace with command arguments
install_args (dict): keyword install arguments
"""
+ from spack.installer import PackageInstaller
tty.debug('Assessing installation status of environment packages')
# If "spack install" is invoked repeatedly for a large environment
# where all specs are already installed, the operation can take
diff --git a/lib/spack/spack/hooks/__init__.py b/lib/spack/spack/hooks/__init__.py
index 681fec0feb..0faab4a9d6 100644
--- a/lib/spack/spack/hooks/__init__.py
+++ b/lib/spack/spack/hooks/__init__.py
@@ -16,6 +16,7 @@
* post_install(spec)
* pre_uninstall(spec)
* post_uninstall(spec)
+ * on_install_failure(exception)
This can be used to implement support for things like module
systems (e.g. modules, lmod, etc.) or to add other custom
@@ -59,8 +60,20 @@ class HookRunner(object):
hook(*args, **kwargs)
+# pre/post install and run by the install subprocess
pre_install = HookRunner('pre_install')
post_install = HookRunner('post_install')
+# These hooks are run within an install subprocess
pre_uninstall = HookRunner('pre_uninstall')
post_uninstall = HookRunner('post_uninstall')
+on_phase_success = HookRunner('on_phase_success')
+on_phase_error = HookRunner('on_phase_error')
+
+# These are hooks in installer.py, before starting install subprocess
+on_install_start = HookRunner('on_install_start')
+on_install_success = HookRunner('on_install_success')
+on_install_failure = HookRunner('on_install_failure')
+
+# Analyzer hooks
+on_analyzer_save = HookRunner('on_analyzer_save')
diff --git a/lib/spack/spack/hooks/monitor.py b/lib/spack/spack/hooks/monitor.py
new file mode 100644
index 0000000000..8d161b72fa
--- /dev/null
+++ b/lib/spack/spack/hooks/monitor.py
@@ -0,0 +1,73 @@
+# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+import llnl.util.tty as tty
+import spack.monitor
+
+
+def on_install_start(spec):
+ """On start of an install, we want to ping the server if it exists
+ """
+ if not spack.monitor.cli:
+ return
+
+ tty.debug("Running on_install_start for %s" % spec)
+ build_id = spack.monitor.cli.new_build(spec)
+ tty.verbose("Build created with id %s" % build_id)
+
+
+def on_install_success(spec):
+ """On the success of an install (after everything is complete)
+ """
+ if not spack.monitor.cli:
+ return
+
+ tty.debug("Running on_install_success for %s" % spec)
+ result = spack.monitor.cli.update_build(spec, status="SUCCESS")
+ tty.verbose(result.get('message'))
+
+
+def on_install_failure(spec):
+ """Triggered on failure of an install
+ """
+ if not spack.monitor.cli:
+ return
+
+ tty.debug("Running on_install_failure for %s" % spec)
+ result = spack.monitor.cli.fail_task(spec)
+ tty.verbose(result.get('message'))
+
+
+def on_phase_success(pkg, phase_name, log_file):
+ """Triggered on a phase success
+ """
+ if not spack.monitor.cli:
+ return
+
+ tty.debug("Running on_phase_success %s, phase %s" % (pkg.name, phase_name))
+ result = spack.monitor.cli.send_phase(pkg, phase_name, log_file, "SUCCESS")
+ tty.verbose(result.get('message'))
+
+
+def on_phase_error(pkg, phase_name, log_file):
+ """Triggered on a phase error
+ """
+ if not spack.monitor.cli:
+ return
+
+ tty.debug("Running on_phase_error %s, phase %s" % (pkg.name, phase_name))
+ result = spack.monitor.cli.send_phase(pkg, phase_name, log_file, "ERROR")
+ tty.verbose(result.get('message'))
+
+
+def on_analyzer_save(pkg, result):
+ """given a package and a result, if we have a spack monitor, upload
+ the result to it.
+ """
+ if not spack.monitor.cli:
+ return
+
+ # This hook runs after a save result
+ spack.monitor.cli.send_analyze_metadata(pkg, result)
diff --git a/lib/spack/spack/installer.py b/lib/spack/spack/installer.py
index 69474df4a5..c19ee9aff8 100644
--- a/lib/spack/spack/installer.py
+++ b/lib/spack/spack/installer.py
@@ -46,6 +46,7 @@ import spack.binary_distribution as binary_distribution
import spack.compilers
import spack.error
import spack.hooks
+import spack.monitor
import spack.package
import spack.package_prefs as prefs
import spack.repo
@@ -412,6 +413,25 @@ def clear_failures():
spack.store.db.clear_all_failures()
+def combine_phase_logs(phase_log_files, log_path):
+ """
+ Read set or list of logs and combine them into one file.
+
+ Each phase will produce it's own log, so this function aims to cat all the
+ separate phase log output files into the pkg.log_path. It is written
+ generally to accept some list of files, and a log path to combine them to.
+
+ Args:
+ phase_log_files (list): a list or iterator of logs to combine
+ log_path (path): the path to combine them to
+ """
+
+ with open(log_path, 'w') as log_file:
+ for phase_log_file in phase_log_files:
+ with open(phase_log_file, 'r') as phase_log:
+ log_file.write(phase_log.read())
+
+
def dump_packages(spec, path):
"""
Dump all package information for a spec and its dependencies.
@@ -521,6 +541,12 @@ def log(pkg):
# Archive the whole stdout + stderr for the package
fs.install(pkg.log_path, pkg.install_log_path)
+ # Archive all phase log paths
+ for phase_log in pkg.phase_log_files:
+ log_file = os.path.basename(phase_log)
+ log_file = os.path.join(os.path.dirname(packages_dir), log_file)
+ fs.install(phase_log, log_file)
+
# Archive the environment used for the build
fs.install(pkg.env_path, pkg.install_env_path)
@@ -1260,6 +1286,7 @@ class PackageInstaller(object):
def _setup_install_dir(self, pkg):
"""
Create and ensure proper access controls for the install directory.
+ Write a small metadata file with the current spack environment.
Args:
pkg (Package): the package to be built and installed
@@ -1285,6 +1312,9 @@ class PackageInstaller(object):
# Ensure the metadata path exists as well
fs.mkdirp(spack.store.layout.metadata_path(pkg.spec), mode=perms)
+ # Always write host environment - we assume this can change
+ spack.store.layout.write_host_environment(pkg.spec)
+
def _update_failed(self, task, mark=False, exc=None):
"""
Update the task and transitive dependents as failed; optionally mark
@@ -1388,8 +1418,8 @@ class PackageInstaller(object):
Args:
pkg (Package): the package to be built and installed"""
- self._init_queue()
+ self._init_queue()
fail_fast_err = 'Terminating after first install failure'
single_explicit_spec = len(self.build_requests) == 1
failed_explicits = []
@@ -1400,6 +1430,7 @@ class PackageInstaller(object):
if task is None:
continue
+ spack.hooks.on_install_start(task.request.pkg.spec)
install_args = task.request.install_args
keep_prefix = install_args.get('keep_prefix')
@@ -1422,6 +1453,10 @@ class PackageInstaller(object):
tty.warn('{0} does NOT actually have any uninstalled deps'
' left'.format(pkg_id))
dep_str = 'dependencies' if task.priority > 1 else 'dependency'
+
+ # Hook to indicate task failure, but without an exception
+ spack.hooks.on_install_failure(task.request.pkg.spec)
+
raise InstallError(
'Cannot proceed with {0}: {1} uninstalled {2}: {3}'
.format(pkg_id, task.priority, dep_str,
@@ -1441,6 +1476,11 @@ class PackageInstaller(object):
tty.warn('{0} failed to install'.format(pkg_id))
self._update_failed(task)
+ # Mark that the package failed
+ # TODO: this should also be for the task.pkg, but we don't
+ # model transitive yet.
+ spack.hooks.on_install_failure(task.request.pkg.spec)
+
if self.fail_fast:
raise InstallError(fail_fast_err)
@@ -1550,6 +1590,7 @@ class PackageInstaller(object):
# Only terminate at this point if a single build request was
# made.
if task.explicit and single_explicit_spec:
+ spack.hooks.on_install_failure(task.request.pkg.spec)
raise
if task.explicit:
@@ -1561,10 +1602,12 @@ class PackageInstaller(object):
err = 'Failed to install {0} due to {1}: {2}'
tty.error(err.format(pkg.name, exc.__class__.__name__,
str(exc)))
+ spack.hooks.on_install_failure(task.request.pkg.spec)
raise
except (Exception, SystemExit) as exc:
self._update_failed(task, True, exc)
+ spack.hooks.on_install_failure(task.request.pkg.spec)
# Best effort installs suppress the exception and mark the
# package as a failure.
@@ -1662,6 +1705,7 @@ def build_process(pkg, kwargs):
echo = spack.package.PackageBase._verbose
pkg.stage.keep = keep_stage
+
with pkg.stage:
# Run the pre-install hook in the child process after
# the directory is created.
@@ -1679,6 +1723,7 @@ def build_process(pkg, kwargs):
# Do the real install in the source directory.
with fs.working_dir(pkg.stage.source_path):
+
# Save the build environment in a file before building.
dump_environment(pkg.env_path)
@@ -1699,25 +1744,48 @@ def build_process(pkg, kwargs):
debug_level = tty.debug_level()
# Spawn a daemon that reads from a pipe and redirects
- # everything to log_path
- with log_output(pkg.log_path, echo, True,
- env=unmodified_env) as logger:
+ # everything to log_path, and provide the phase for logging
+ for i, (phase_name, phase_attr) in enumerate(zip(
+ pkg.phases, pkg._InstallPhase_phases)):
+
+ # Keep a log file for each phase
+ log_dir = os.path.dirname(pkg.log_path)
+ log_file = "spack-build-%02d-%s-out.txt" % (
+ i + 1, phase_name.lower()
+ )
+ log_file = os.path.join(log_dir, log_file)
- for phase_name, phase_attr in zip(
- pkg.phases, pkg._InstallPhase_phases):
+ try:
+ # DEBUGGING TIP - to debug this section, insert an IPython
+ # embed here, and run the sections below without log capture
+ with log_output(log_file, echo, True,
+ env=unmodified_env) as logger:
+
+ with logger.force_echo():
+ inner_debug_level = tty.debug_level()
+ tty.set_debug(debug_level)
+ tty.msg("{0} Executing phase: '{1}'"
+ .format(pre, phase_name))
+ tty.set_debug(inner_debug_level)
+
+ # Redirect stdout and stderr to daemon pipe
+ phase = getattr(pkg, phase_attr)
- with logger.force_echo():
- inner_debug_level = tty.debug_level()
- tty.set_debug(debug_level)
- tty.msg("{0} Executing phase: '{1}'"
- .format(pre, phase_name))
- tty.set_debug(inner_debug_level)
+ # Catch any errors to report to logging
- # Redirect stdout and stderr to daemon pipe
- phase = getattr(pkg, phase_attr)
- phase(pkg.spec, pkg.prefix)
+ phase(pkg.spec, pkg.prefix)
+ spack.hooks.on_phase_success(pkg, phase_name, log_file)
- echo = logger.echo
+ except BaseException:
+ combine_phase_logs(pkg.phase_log_files, pkg.log_path)
+ spack.hooks.on_phase_error(pkg, phase_name, log_file)
+ raise
+
+ # We assume loggers share echo True/False
+ echo = logger.echo
+
+ # After log, we can get all output/error files from the package stage
+ combine_phase_logs(pkg.phase_log_files, pkg.log_path)
log(pkg)
# Run post install hooks before build stage is removed.
@@ -1733,6 +1801,9 @@ def build_process(pkg, kwargs):
_hms(pkg._total_time)))
_print_installed_pkg(pkg.prefix)
+ # Send final status that install is successful
+ spack.hooks.on_install_success(pkg.spec)
+
# preserve verbosity across runs
return echo
diff --git a/lib/spack/spack/monitor.py b/lib/spack/spack/monitor.py
new file mode 100644
index 0000000000..2d41b62d6a
--- /dev/null
+++ b/lib/spack/spack/monitor.py
@@ -0,0 +1,459 @@
+# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+"""Interact with a Spack Monitor Service. Derived from
+https://github.com/spack/spack-monitor/blob/main/script/spackmoncli.py
+"""
+
+import base64
+import os
+import re
+
+try:
+ from urllib.request import Request, urlopen
+ from urllib.error import URLError
+except ImportError:
+ from urllib2 import urlopen, Request, URLError # type: ignore # novm
+
+import spack
+import spack.hash_types as ht
+import spack.main
+import spack.store
+import spack.util.spack_json as sjson
+import spack.util.spack_yaml as syaml
+import llnl.util.tty as tty
+from copy import deepcopy
+
+
+# A global client to instantiate once
+cli = None
+
+
+def get_client(host, prefix="ms1", disable_auth=False, allow_fail=False):
+ """a common function to get a client for a particular host and prefix.
+ If the client is not running, we exit early, unless allow_fail is set
+ to true, indicating that we should continue the build even if the
+ server is not present. Note that this client is defined globally as "cli"
+ so we can istantiate it once (checking for credentials, etc.) and then
+ always have access to it via spack.monitor.cli. Also note that
+ typically, we call the monitor by way of hooks in spack.hooks.monitor.
+ So if you want the monitor to have a new interaction with some part of
+ the codebase, it's recommended to write a hook first, and then have
+ the monitor use it.
+ """
+ global cli
+ cli = SpackMonitorClient(host=host, prefix=prefix, allow_fail=allow_fail)
+
+ # If we don't disable auth, environment credentials are required
+ if not disable_auth:
+ cli.require_auth()
+
+ # We will exit early if the monitoring service is not running
+ info = cli.service_info()
+
+ # If we allow failure, the response will be done
+ if info:
+ tty.debug("%s v.%s has status %s" % (
+ info['id'],
+ info['version'],
+ info['status'])
+ )
+ return cli
+
+ else:
+ tty.debug("spack-monitor server not found, continuing as allow_fail is True.")
+
+
+def get_monitor_group(subparser):
+ """Since the monitor group is shared between commands, we provide a common
+ function to generate the group for it. The user can pass the subparser, and
+ the group is added, and returned.
+ """
+ # Monitoring via https://github.com/spack/spack-monitor
+ monitor_group = subparser.add_argument_group()
+ monitor_group.add_argument(
+ '--monitor', action='store_true', dest='use_monitor', default=False,
+ help="interact with a montor server during builds.")
+ monitor_group.add_argument(
+ '--monitor-no-auth', action='store_true', dest='monitor_disable_auth',
+ default=False, help="the monitoring server does not require auth.")
+ monitor_group.add_argument(
+ '--monitor-keep-going', action='store_true', dest='monitor_keep_going',
+ default=False, help="continue the build if a request to monitor fails.")
+ monitor_group.add_argument(
+ '--monitor-host', dest='monitor_host', default="http://127.0.0.1",
+ help="If using a monitor, customize the host.")
+ monitor_group.add_argument(
+ '--monitor-prefix', dest='monitor_prefix', default="ms1",
+ help="The API prefix for the monitor service.")
+ return monitor_group
+
+
+class SpackMonitorClient:
+ """The SpackMonitorClient is a handle to interact with a spack monitor
+ server. We require the host url, along with the prefix to discover the
+ service_info endpoint. If allow_fail is set to True, we will not exit
+ on error with tty.fail given that a request is not successful. The spack
+ version is one of the fields to uniquely identify a spec, so we add it
+ to the client on init.
+ """
+
+ def __init__(self, host=None, prefix="ms1", allow_fail=False):
+ self.host = host or "http://127.0.0.1"
+ self.baseurl = "%s/%s" % (self.host, prefix.strip("/"))
+ self.token = os.environ.get("SPACKMON_TOKEN")
+ self.username = os.environ.get("SPACKMON_USER")
+ self.headers = {}
+ self.allow_fail = allow_fail
+ self.spack_version = spack.main.get_version()
+ self.capture_build_environment()
+
+ # We keey lookup of build_id by full_hash
+ self.build_ids = {}
+
+ def load_build_environment(self, spec):
+ """If we are running an analyze command, we will need to load previously
+ used build environment metadata from install_environment.json to capture
+ what was done during the build.
+ """
+ if not hasattr(spec, "package") or not spec.package:
+ tty.die("A spec must have a package to load the environment.")
+
+ pkg_dir = os.path.dirname(spec.package.install_log_path)
+ env_file = os.path.join(pkg_dir, "install_environment.json")
+ build_environment = read_json(env_file)
+ if not build_environment:
+ tty.warning(
+ "install_environment.json not found in package folder. "
+ " This means that the current environment metadata will be used."
+ )
+ else:
+ self.build_environment = build_environment
+
+ def capture_build_environment(self):
+ """Use spack.util.environment.get_host_environment_metadata to capture the
+ environment for the build. This is important because it's a unique
+ identifier, along with the spec, for a Build. It should look something
+ like this:
+
+ {'host_os': 'ubuntu20.04',
+ 'platform': 'linux',
+ 'host_target': 'skylake',
+ 'hostname': 'vanessa-ThinkPad-T490s',
+ 'spack_version': '0.16.1-1455-52d5b55b65',
+ 'kernel_version': '#73-Ubuntu SMP Mon Jan 18 17:25:17 UTC 2021'}
+
+ This is saved to a package install's metadata folder as
+ install_environment.json, and can be loaded by the monitor for uploading
+ data relevant to a later analysis.
+ """
+ from spack.util.environment import get_host_environment_metadata
+ self.build_environment = get_host_environment_metadata()
+
+ def require_auth(self):
+ """Require authentication, meaning that the token and username must
+ not be unset
+ """
+ if not self.token or not self.username:
+ tty.die("You are required to export SPACKMON_TOKEN and SPACKMON_USER")
+
+ def set_header(self, name, value):
+ self.headers.update({name: value})
+
+ def set_basic_auth(self, username, password):
+ """A wrapper to adding basic authentication to the Request"""
+ auth_str = "%s:%s" % (username, password)
+ auth_header = base64.b64encode(auth_str.encode("utf-8"))
+ self.set_header("Authorization", "Basic %s" % auth_header.decode("utf-8"))
+
+ def reset(self):
+ """Reset and prepare for a new request.
+ """
+ if "Authorization" in self.headers:
+ self.headers = {"Authorization": self.headers['Authorization']}
+ else:
+ self.headers = {}
+
+ def prepare_request(self, endpoint, data, headers):
+ """Given an endpoint url and data, prepare the request. If data
+ is provided, urllib makes the request a POST
+ """
+ # Always reset headers for new request.
+ self.reset()
+
+ headers = headers or {}
+
+ # The calling function can provide a full or partial url
+ if not endpoint.startswith("http"):
+ endpoint = "%s/%s" % (self.baseurl, endpoint)
+
+ # If we have data, the request will be POST
+ if data:
+ if not isinstance(data, str):
+ data = sjson.dump(data)
+ data = data.encode('ascii')
+
+ return Request(endpoint, data=data, headers=headers)
+
+ def issue_request(self, request, retry=True):
+ """Given a prepared request, issue it. If we get an error, die. If
+ there are times when we don't want to exit on error (but instead
+ disable using the monitoring service) we could add that here.
+ """
+ try:
+ response = urlopen(request)
+ except URLError as e:
+
+ # If we have an authorization request, retry once with auth
+ if e.code == 401 and retry:
+ if self.authenticate_request(e):
+ request = self.prepare_request(
+ e.url,
+ sjson.load(request.data.decode('utf-8')),
+ self.headers
+ )
+ return self.issue_request(request, False)
+
+ # Otherwise, relay the message and exit on error
+ msg = ""
+ if hasattr(e, 'reason'):
+ msg = e.reason
+ elif hasattr(e, 'code'):
+ msg = e.code
+
+ if self.allow_fail:
+ tty.warning("Request to %s was not successful, but continuing." % e.url)
+ return
+
+ tty.die(msg)
+
+ return response
+
+ def do_request(self, endpoint, data=None, headers=None, url=None):
+ """Do a request. If data is provided, it is POST, otherwise GET.
+ If an entire URL is provided, don't use the endpoint
+ """
+ request = self.prepare_request(endpoint, data, headers)
+
+ # If we have an authorization error, we retry with
+ response = self.issue_request(request)
+
+ # A 200/201 response incidates success
+ if response.code in [200, 201]:
+ return sjson.load(response.read().decode('utf-8'))
+
+ return response
+
+ def authenticate_request(self, originalResponse):
+ """Given a response (an HTTPError 401), look for a Www-Authenticate
+ header to parse. We return True/False to indicate if the request
+ should be retried.
+ """
+ authHeaderRaw = originalResponse.headers.get("Www-Authenticate")
+ if not authHeaderRaw:
+ return False
+
+ # If we have a username and password, set basic auth automatically
+ if self.token and self.username:
+ self.set_basic_auth(self.username, self.token)
+
+ headers = deepcopy(self.headers)
+ if "Authorization" not in headers:
+ tty.error(
+ "This endpoint requires a token. Please set "
+ "client.set_basic_auth(username, password) first "
+ "or export them to the environment."
+ )
+ return False
+
+ # Prepare request to retry
+ h = parse_auth_header(authHeaderRaw)
+ headers.update({
+ "service": h.Service,
+ "Accept": "application/json",
+ "User-Agent": "spackmoncli"}
+ )
+
+ # Currently we don't set a scope (it defaults to build)
+ authResponse = self.do_request(h.Realm, headers=headers)
+
+ # Request the token
+ token = authResponse.get("token")
+ if not token:
+ return False
+
+ # Set the token to the original request and retry
+ self.headers.update({"Authorization": "Bearer %s" % token})
+ return True
+
+ # Functions correspond to endpoints
+ def service_info(self):
+ """get the service information endpoint"""
+ # Base endpoint provides service info
+ return self.do_request("")
+
+ def new_configuration(self, specs):
+ """Given a list of specs, generate a new configuration for each. We
+ return a lookup of specs with their package names. This assumes
+ that we are only installing one version of each package. We aren't
+ starting or creating any builds, so we don't need a build environment.
+ """
+ configs = {}
+
+ # There should only be one spec generally (what cases would have >1?)
+ for spec in specs:
+ # Not sure if this is needed here, but I see it elsewhere
+ if spec.name in spack.repo.path or spec.virtual:
+ spec.concretize()
+ as_dict = {"spec": spec.to_dict(hash=ht.full_hash),
+ "spack_version": self.spack_version}
+ response = self.do_request("specs/new/", data=sjson.dump(as_dict))
+ configs[spec.package.name] = response.get('data', {})
+ return configs
+
+ def new_build(self, spec):
+ """Create a new build, meaning sending the hash of the spec to be built,
+ along with the build environment. These two sets of data uniquely can
+ identify the build, and we will add objects (the binaries produced) to
+ it. We return the build id to the calling client.
+ """
+ return self.get_build_id(spec, return_response=True)
+
+ def get_build_id(self, spec, return_response=False, spec_exists=True):
+ """Retrieve a build id, either in the local cache, or query the server
+ """
+ full_hash = spec.full_hash()
+ if full_hash in self.build_ids:
+ return self.build_ids[full_hash]
+
+ # Prepare build environment data (including spack version)
+ data = self.build_environment.copy()
+ data['full_hash'] = full_hash
+
+ # If we allow the spec to not exist (meaning we create it) we need to
+ # include the full spec.yaml here
+ if not spec_exists:
+ meta_dir = os.path.dirname(spec.package.install_log_path)
+ spec_file = os.path.join(meta_dir, "spec.yaml")
+ data['spec'] = syaml.load(read_file(spec_file))
+
+ response = self.do_request("builds/new/", data=sjson.dump(data))
+
+ # Add the build id to the lookup
+ bid = self.build_ids[full_hash] = response['data']['build']['build_id']
+ self.build_ids[full_hash] = bid
+
+ # If the function is called directly, the user might want output
+ if return_response:
+ return response
+ return bid
+
+ def update_build(self, spec, status="SUCCESS"):
+ """update task will just update the relevant package to indicate a
+ successful install. Unlike cancel_task that sends a cancalled request
+ to the main package, here we don't need to cancel or otherwise update any
+ other statuses. This endpoint can take a general status to update just
+ one
+ """
+ data = {"build_id": self.get_build_id(spec), "status": status}
+ return self.do_request("builds/update/", data=sjson.dump(data))
+
+ def fail_task(self, spec):
+ """Given a spec, mark it as failed. This means that Spack Monitor
+ marks all dependencies as cancelled, unless they are already successful
+ """
+ return self.update_build(spec, status="FAILED")
+
+ def send_analyze_metadata(self, pkg, metadata):
+ """Given a dictionary of analyzers (with key as analyzer type, and
+ value as the data) upload the analyzer output to Spack Monitor.
+ Spack Monitor should either have a known understanding of the analyzer,
+ or if not (the key is not recognized), it's assumed to be a dictionary
+ of objects/files, each with attributes to be updated. E.g.,
+
+ {"analyzer-name": {"object-file-path": {"feature1": "value1"}}}
+ """
+ # Prepare build environment data (including spack version)
+ # Since the build might not have been generated, we include the spec
+ data = {"build_id": self.get_build_id(pkg.spec, spec_exists=False),
+ "metadata": metadata}
+ return self.do_request("analyze/builds/", data=sjson.dump(data))
+
+ def send_phase(self, pkg, phase_name, phase_output_file, status):
+ """Given a package, phase name, and status, update the monitor endpoint
+ to alert of the status of the stage. This includes parsing the package
+ metadata folder for phase output and error files
+ """
+ data = {"build_id": self.get_build_id(pkg.spec)}
+
+ # Send output specific to the phase (does this include error?)
+ data.update({"status": status,
+ "output": read_file(phase_output_file),
+ "phase_name": phase_name})
+
+ return self.do_request("builds/phases/update/", data=sjson.dump(data))
+
+ def upload_specfile(self, filename):
+ """Given a spec file (must be json) upload to the UploadSpec endpoint.
+ This function is not used in the spack to server workflow, but could
+ be useful is Spack Monitor is intended to send an already generated
+ file in some kind of separate analysis. For the environment file, we
+ parse out SPACK_* variables to include.
+ """
+ # We load as json just to validate it
+ spec = read_json(filename)
+ data = {"spec": spec, "spack_verison": self.spack_version}
+ return self.do_request("specs/new/", data=sjson.dump(data))
+
+
+# Helper functions
+
+def parse_auth_header(authHeaderRaw):
+ """parse authentication header into pieces"""
+ regex = re.compile('([a-zA-z]+)="(.+?)"')
+ matches = regex.findall(authHeaderRaw)
+ lookup = dict()
+ for match in matches:
+ lookup[match[0]] = match[1]
+ return authHeader(lookup)
+
+
+class authHeader:
+ def __init__(self, lookup):
+ """Given a dictionary of values, match them to class attributes"""
+ for key in lookup:
+ if key in ["realm", "service", "scope"]:
+ setattr(self, key.capitalize(), lookup[key])
+
+
+def read_file(filename):
+ """Read a file, if it exists. Otherwise return None
+ """
+ if not os.path.exists(filename):
+ return
+ with open(filename, 'r') as fd:
+ content = fd.read()
+ return content
+
+
+def write_file(content, filename):
+ """write content to file"""
+ with open(filename, 'w') as fd:
+ fd.writelines(content)
+ return content
+
+
+def write_json(obj, filename):
+ """Write a json file, if the output directory exists."""
+ if not os.path.exists(os.path.dirname(filename)):
+ return
+ return write_file(sjson.dump(obj), filename)
+
+
+def read_json(filename):
+ """Read a file and load into json, if it exists. Otherwise return None"""
+ if not os.path.exists(filename):
+ return
+ return sjson.load(read_file(filename))
diff --git a/lib/spack/spack/package.py b/lib/spack/spack/package.py
index d87607a8dd..b24f6bf113 100644
--- a/lib/spack/spack/package.py
+++ b/lib/spack/spack/package.py
@@ -15,6 +15,7 @@ import collections
import contextlib
import copy
import functools
+import glob
import hashlib
import inspect
import os
@@ -1067,6 +1068,14 @@ class PackageBase(six.with_metaclass(PackageMeta, PackageViewMixin, object)):
return os.path.join(self.stage.path, _spack_build_logfile)
@property
+ def phase_log_files(self):
+ """Find sorted phase log files written to the staging directory"""
+ logs_dir = os.path.join(self.stage.path, "spack-build-*-out.txt")
+ log_files = glob.glob(logs_dir)
+ log_files.sort()
+ return log_files
+
+ @property
def install_log_path(self):
"""Return the build log file path on successful installation."""
# Backward compatibility: Return the name of an existing install log.
diff --git a/lib/spack/spack/paths.py b/lib/spack/spack/paths.py
index 20c014371a..7a1fee4294 100644
--- a/lib/spack/spack/paths.py
+++ b/lib/spack/spack/paths.py
@@ -33,6 +33,7 @@ external_path = os.path.join(lib_path, "external")
build_env_path = os.path.join(lib_path, "env")
module_path = os.path.join(lib_path, "spack")
command_path = os.path.join(module_path, "cmd")
+analyzers_path = os.path.join(module_path, "analyzers")
platform_path = os.path.join(module_path, 'platforms')
compilers_path = os.path.join(module_path, "compilers")
build_systems_path = os.path.join(module_path, 'build_systems')
diff --git a/lib/spack/spack/spec.py b/lib/spack/spack/spec.py
index f7a78863d4..71971453f2 100644
--- a/lib/spack/spack/spec.py
+++ b/lib/spack/spack/spec.py
@@ -1661,7 +1661,13 @@ class Spec(object):
d['patches'] = variant._patches_in_order_of_appearance
if hash.package_hash:
- d['package_hash'] = self.package.content_hash()
+ package_hash = self.package.content_hash()
+
+ # Full hashes are in bytes
+ if (not isinstance(package_hash, six.text_type)
+ and isinstance(package_hash, six.binary_type)):
+ package_hash = package_hash.decode('utf-8')
+ d['package_hash'] = package_hash
deps = self.dependencies_dict(deptype=hash.deptype)
if deps:
diff --git a/lib/spack/spack/test/cmd/analyze.py b/lib/spack/spack/test/cmd/analyze.py
new file mode 100644
index 0000000000..6d164063ab
--- /dev/null
+++ b/lib/spack/spack/test/cmd/analyze.py
@@ -0,0 +1,176 @@
+# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+import os
+import pytest
+
+import spack.config
+import spack.package
+import spack.cmd.install
+
+from spack.spec import Spec
+import spack.util.spack_json as sjson
+from spack.main import SpackCommand
+
+install = SpackCommand('install')
+analyze = SpackCommand('analyze')
+
+
+def test_test_package_not_installed(mock_fetch, install_mockery_mutable_config):
+ # We cannot run an analysis for a package not installed
+ out = analyze('run', 'libdwarf', fail_on_error=False)
+ assert "==> Error: Spec 'libdwarf' matches no installed packages.\n" in out
+
+
+def test_analyzer_get_install_dir(mock_fetch, install_mockery_mutable_config):
+ """
+ Test that we cannot get an analyzer directory without a spec package.
+ """
+ spec = Spec('libdwarf').concretized()
+ assert 'libdwarf' in spack.analyzers.analyzer_base.get_analyzer_dir(spec)
+
+ # Case 1: spec is missing attribute for package
+ with pytest.raises(SystemExit):
+ spack.analyzers.analyzer_base.get_analyzer_dir(None)
+
+ class Packageless(object):
+ package = None
+
+ # Case 2: spec has package attribute, but it's None
+ with pytest.raises(SystemExit):
+ spack.analyzers.analyzer_base.get_analyzer_dir(Packageless())
+
+
+def test_malformed_analyzer(mock_fetch, install_mockery_mutable_config):
+ """
+ Test that an analyzer missing needed attributes is invalid.
+ """
+ from spack.analyzers.analyzer_base import AnalyzerBase
+
+ # Missing attribute description
+ class MyAnalyzer(AnalyzerBase):
+ name = "my_analyzer"
+ outfile = "my_analyzer_output.txt"
+
+ spec = Spec('libdwarf').concretized()
+ with pytest.raises(SystemExit):
+ MyAnalyzer(spec)
+
+
+def test_analyze_output(tmpdir, mock_fetch, install_mockery_mutable_config):
+ """
+ Test that an analyzer errors if requested name does not exist.
+ """
+ install('libdwarf')
+ install('python@3.8')
+ analyzer_dir = tmpdir.join('analyzers')
+
+ # An analyzer that doesn't exist should not work
+ out = analyze('run', '-a', 'pusheen', 'libdwarf', fail_on_error=False)
+ assert '==> Error: Analyzer pusheen does not exist\n' in out
+
+ # We will output to this analyzer directory
+ analyzer_dir = tmpdir.join('analyzers')
+ out = analyze('run', '-a', 'install_files', '-p', str(analyzer_dir), 'libdwarf')
+
+ # Ensure that if we run again without over write, we don't run
+ out = analyze('run', '-a', 'install_files', '-p', str(analyzer_dir), 'libdwarf')
+ assert "skipping" in out
+
+ # With overwrite it should run
+ out = analyze('run', '-a', 'install_files', '-p', str(analyzer_dir),
+ '--overwrite', 'libdwarf')
+ assert "==> Writing result to" in out
+
+
+def _run_analyzer(name, package, tmpdir):
+ """
+ A shared function to test that an analyzer runs.
+
+ We return the output file for further inspection.
+ """
+ analyzer = spack.analyzers.get_analyzer(name)
+ analyzer_dir = tmpdir.join('analyzers')
+ out = analyze('run', '-a', analyzer.name, '-p', str(analyzer_dir), package)
+
+ assert "==> Writing result to" in out
+ assert "/%s/%s\n" % (analyzer.name, analyzer.outfile) in out
+
+ # The output file should exist
+ output_file = out.strip('\n').split(' ')[-1].strip()
+ assert os.path.exists(output_file)
+ return output_file
+
+
+def test_installfiles_analyzer(tmpdir, mock_fetch, install_mockery_mutable_config):
+ """
+ test the install files analyzer
+ """
+ install('libdwarf')
+ output_file = _run_analyzer("install_files", "libdwarf", tmpdir)
+
+ # Ensure it's the correct content
+ with open(output_file, 'r') as fd:
+ content = sjson.load(fd.read())
+
+ basenames = set()
+ for key, attrs in content.items():
+ basenames.add(os.path.basename(key))
+
+ # Check for a few expected files
+ for key in ['.spack', 'libdwarf', 'packages', 'repo.yaml', 'repos']:
+ assert key in basenames
+
+
+def test_environment_analyzer(tmpdir, mock_fetch, install_mockery_mutable_config):
+ """
+ test the environment variables analyzer.
+ """
+ install('libdwarf')
+ output_file = _run_analyzer("environment_variables", "libdwarf", tmpdir)
+ with open(output_file, 'r') as fd:
+ content = sjson.load(fd.read())
+
+ # Check a few expected keys
+ for key in ['SPACK_CC', 'SPACK_COMPILER_SPEC', 'SPACK_ENV_PATH']:
+ assert key in content
+
+ # The analyzer should return no result if the output file does not exist.
+ spec = Spec('libdwarf').concretized()
+ env_file = os.path.join(spec.package.prefix, '.spack', 'spack-build-env.txt')
+ assert os.path.exists(env_file)
+ os.remove(env_file)
+ analyzer = spack.analyzers.get_analyzer("environment_variables")
+ analyzer_dir = tmpdir.join('analyzers')
+ result = analyzer(spec, analyzer_dir).run()
+ assert "environment_variables" in result
+ assert not result['environment_variables']
+
+
+def test_list_analyzers():
+ """
+ test that listing analyzers shows all the possible analyzers.
+ """
+ from spack.analyzers import analyzer_types
+
+ # all cannot be an analyzer
+ assert "all" not in analyzer_types
+
+ # All types should be present!
+ out = analyze('list-analyzers')
+ for analyzer_type in analyzer_types:
+ assert analyzer_type in out
+
+
+def test_configargs_analyzer(tmpdir, mock_fetch, install_mockery_mutable_config):
+ """
+ test the config args analyzer.
+
+ Since we don't have any, this should return an empty result.
+ """
+ install('libdwarf')
+ analyzer_dir = tmpdir.join('analyzers')
+ out = analyze('run', '-a', 'config_args', '-p', str(analyzer_dir), 'libdwarf')
+ assert out == ''
diff --git a/lib/spack/spack/test/cmd/install.py b/lib/spack/spack/test/cmd/install.py
index ba11c68987..674897b933 100644
--- a/lib/spack/spack/test/cmd/install.py
+++ b/lib/spack/spack/test/cmd/install.py
@@ -119,7 +119,9 @@ def test_install_dirty_flag(arguments, expected):
def test_package_output(tmpdir, capsys, install_mockery, mock_fetch):
- """Ensure output printed from pkgs is captured by output redirection."""
+ """
+ Ensure output printed from pkgs is captured by output redirection.
+ """
# we can't use output capture here because it interferes with Spack's
# logging. TODO: see whether we can get multiple log_outputs to work
# when nested AND in pytest
@@ -140,12 +142,15 @@ def test_package_output(tmpdir, capsys, install_mockery, mock_fetch):
@pytest.mark.disable_clean_stage_check
def test_install_output_on_build_error(mock_packages, mock_archive, mock_fetch,
config, install_mockery, capfd):
+ """
+ This test used to assume receiving full output, but since we've updated
+ spack to generate logs on the level of phases, it will only return the
+ last phase, install.
+ """
# capfd interferes with Spack's capturing
with capfd.disabled():
- out = install('build-error', fail_on_error=False)
- assert 'ProcessError' in out
- assert 'configure: error: in /path/to/some/file:' in out
- assert 'configure: error: cannot run C compiled programs.' in out
+ out = install('-v', 'build-error', fail_on_error=False)
+ assert 'Installing build-error' in out
@pytest.mark.disable_clean_stage_check
@@ -172,20 +177,17 @@ def test_install_with_source(
@pytest.mark.disable_clean_stage_check
def test_show_log_on_error(mock_packages, mock_archive, mock_fetch,
config, install_mockery, capfd):
- """Make sure --show-log-on-error works."""
+ """
+ Make sure --show-log-on-error works.
+ """
with capfd.disabled():
out = install('--show-log-on-error', 'build-error',
fail_on_error=False)
assert isinstance(install.error, spack.build_environment.ChildError)
assert install.error.pkg.name == 'build-error'
- assert 'Full build log:' in out
- print(out)
-
- # Message shows up for ProcessError (1) and output (1)
- errors = [line for line in out.split('\n')
- if 'configure: error: cannot run C compiled programs' in line]
- assert len(errors) == 2
+ assert '==> Installing build-error' in out
+ assert 'See build log for details:' in out
def test_install_overwrite(
@@ -711,7 +713,9 @@ def test_install_only_dependencies_of_all_in_env(
def test_install_help_does_not_show_cdash_options(capsys):
- """Make sure `spack install --help` does not describe CDash arguments"""
+ """
+ Make sure `spack install --help` does not describe CDash arguments
+ """
with pytest.raises(SystemExit):
install('--help')
captured = capsys.readouterr()
@@ -754,7 +758,9 @@ def test_compiler_bootstrap(
def test_compiler_bootstrap_from_binary_mirror(
install_mockery_mutable_config, mock_packages, mock_fetch,
mock_archive, mutable_config, monkeypatch, tmpdir):
- """Make sure installing compiler from buildcache registers compiler"""
+ """
+ Make sure installing compiler from buildcache registers compiler
+ """
# Create a temp mirror directory for buildcache usage
mirror_dir = tmpdir.join('mirror_dir')
diff --git a/lib/spack/spack/test/installer.py b/lib/spack/spack/test/installer.py
index 3d1b72e559..840d4bfa7d 100644
--- a/lib/spack/spack/test/installer.py
+++ b/lib/spack/spack/test/installer.py
@@ -579,6 +579,32 @@ def test_clear_failures_errs(install_mockery, monkeypatch, capsys):
monkeypatch.setattr(os, 'remove', orig_fn)
+def test_combine_phase_logs(tmpdir):
+ """Write temporary files, and assert that combine phase logs works
+ to combine them into one file. We aren't currently using this function,
+ but it's available when the logs are refactored to be written separately.
+ """
+ log_files = ['configure-out.txt', 'install-out.txt', 'build-out.txt']
+ phase_log_files = []
+
+ # Create and write to dummy phase log files
+ for log_file in log_files:
+ phase_log_file = os.path.join(str(tmpdir), log_file)
+ with open(phase_log_file, 'w') as plf:
+ plf.write('Output from %s\n' % log_file)
+ phase_log_files.append(phase_log_file)
+
+ # This is the output log we will combine them into
+ combined_log = os.path.join(str(tmpdir), "combined-out.txt")
+ spack.installer.combine_phase_logs(phase_log_files, combined_log)
+ with open(combined_log, 'r') as log_file:
+ out = log_file.read()
+
+ # Ensure each phase log file is represented
+ for log_file in log_files:
+ assert "Output from %s\n" % log_file in out
+
+
def test_check_deps_status_install_failure(install_mockery, monkeypatch):
const_arg = installer_args(['a'], {})
installer = create_installer(const_arg)
diff --git a/lib/spack/spack/util/classes.py b/lib/spack/spack/util/classes.py
new file mode 100644
index 0000000000..b6465c7581
--- /dev/null
+++ b/lib/spack/spack/util/classes.py
@@ -0,0 +1,39 @@
+# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+# Need this because of spack.util.string
+from __future__ import absolute_import
+from spack.util.naming import mod_to_class
+from llnl.util.lang import memoized, list_modules
+import llnl.util.tty as tty
+
+import inspect
+
+__all__ = [
+ 'list_classes'
+]
+
+
+@memoized
+def list_classes(parent_module, mod_path):
+ """Given a parent path (e.g., spack.platforms or spack.analyzers),
+ use list_modules to derive the module names, and then mod_to_class
+ to derive class names. Import the classes and return them in a list
+ """
+ classes = []
+
+ for name in list_modules(mod_path):
+ mod_name = '%s.%s' % (parent_module, name)
+ class_name = mod_to_class(name)
+ mod = __import__(mod_name, fromlist=[class_name])
+ if not hasattr(mod, class_name):
+ tty.die('No class %s defined in %s' % (class_name, mod_name))
+ cls = getattr(mod, class_name)
+ if not inspect.isclass(cls):
+ tty.die('%s.%s is not a class' % (mod_name, class_name))
+
+ classes.append(cls)
+
+ return classes
diff --git a/lib/spack/spack/util/environment.py b/lib/spack/spack/util/environment.py
index ad32ec82b1..04158c119b 100644
--- a/lib/spack/spack/util/environment.py
+++ b/lib/spack/spack/util/environment.py
@@ -9,7 +9,9 @@ import contextlib
import inspect
import json
import os
+import platform
import re
+import socket
import sys
import os.path
@@ -139,6 +141,40 @@ def pickle_environment(path, environment=None):
open(path, 'wb'), protocol=2)
+def get_host_environment_metadata():
+ """Get the host environment, reduce to a subset that we can store in
+ the install directory, and add the spack version.
+ """
+ import spack.main
+ environ = get_host_environment()
+ return {"host_os": environ['os'],
+ "platform": environ['platform'],
+ "host_target": environ['target'],
+ "hostname": environ['hostname'],
+ "spack_version": spack.main.get_version(),
+ "kernel_version": platform.version()}
+
+
+def get_host_environment():
+ """Return a dictionary (lookup) with host information (not including the
+ os.environ).
+ """
+ import spack.spec
+ import spack.architecture as architecture
+ arch = architecture.Arch(
+ architecture.platform(), 'default_os', 'default_target')
+ arch_spec = spack.spec.Spec('arch=%s' % arch)
+ return {
+ 'target': str(arch.target),
+ 'os': str(arch.os),
+ 'platform': str(arch.platform),
+ 'arch': arch_spec,
+ 'architecture': arch_spec,
+ 'arch_str': str(arch),
+ 'hostname': socket.gethostname()
+ }
+
+
@contextlib.contextmanager
def set_env(**kwargs):
"""Temporarily sets and restores environment variables.
diff --git a/share/spack/spack-completion.bash b/share/spack/spack-completion.bash
index 5829ed653d..0c4bb8eba1 100755
--- a/share/spack/spack-completion.bash
+++ b/share/spack/spack-completion.bash
@@ -333,7 +333,7 @@ _spack() {
then
SPACK_COMPREPLY="-h --help -H --all-help --color -c --config -C --config-scope -d --debug --timestamp --pdb -e --env -D --env-dir -E --no-env --use-env-repo -k --insecure -l --enable-locks -L --disable-locks -m --mock -p --profile --sorted-profile --lines -v --verbose --stacktrace -V --version --print-shell-vars"
else
- SPACK_COMPREPLY="activate add arch blame build-env buildcache cd checksum ci clean clone commands compiler compilers concretize config containerize create deactivate debug dependencies dependents deprecate dev-build develop docs edit env extensions external fetch find flake8 gc gpg graph help info install license list load location log-parse maintainers mark mirror module patch pkg providers pydoc python reindex remove rm repo resource restage solve spec stage style test test-env tutorial undevelop uninstall unit-test unload url verify versions view"
+ SPACK_COMPREPLY="activate add analyze arch blame build-env buildcache cd checksum ci clean clone commands compiler compilers concretize config containerize create deactivate debug dependencies dependents deprecate dev-build develop docs edit env extensions external fetch find flake8 gc gpg graph help info install license list load location log-parse maintainers mark mirror module patch pkg providers pydoc python reindex remove rm repo resource restage solve spec stage style test test-env tutorial undevelop uninstall unit-test unload url verify versions view"
fi
}
@@ -355,6 +355,28 @@ _spack_add() {
fi
}
+_spack_analyze() {
+ if $list_options
+ then
+ SPACK_COMPREPLY="-h --help --monitor --monitor-no-auth --monitor-keep-going --monitor-host --monitor-prefix"
+ else
+ SPACK_COMPREPLY="list-analyzers run"
+ fi
+}
+
+_spack_analyze_list_analyzers() {
+ SPACK_COMPREPLY="-h --help"
+}
+
+_spack_analyze_run() {
+ if $list_options
+ then
+ SPACK_COMPREPLY="-h --help --overwrite -p --path -a --analyzers"
+ else
+ _all_packages
+ fi
+}
+
_spack_arch() {
SPACK_COMPREPLY="-h --help --known-targets -p --platform -o --operating-system -t --target -f --frontend -b --backend"
}
@@ -1041,7 +1063,7 @@ _spack_info() {
_spack_install() {
if $list_options
then
- SPACK_COMPREPLY="-h --help --only -u --until -j --jobs --overwrite --fail-fast --keep-prefix --keep-stage --dont-restage --use-cache --no-cache --cache-only --include-build-deps --no-check-signature --require-full-hash-match --show-log-on-error --source -n --no-checksum --deprecated -v --verbose --fake --only-concrete -f --file --clean --dirty --test --run-tests --log-format --log-file --help-cdash --cdash-upload-url --cdash-build --cdash-site --cdash-track --cdash-buildstamp -y --yes-to-all"
+ SPACK_COMPREPLY="-h --help --only -u --until -j --jobs --overwrite --fail-fast --keep-prefix --keep-stage --dont-restage --use-cache --no-cache --cache-only --monitor --monitor-no-auth --monitor-keep-going --monitor-host --monitor-prefix --include-build-deps --no-check-signature --require-full-hash-match --show-log-on-error --source -n --no-checksum --deprecated -v --verbose --fake --only-concrete -f --file --clean --dirty --test --run-tests --log-format --log-file --help-cdash --cdash-upload-url --cdash-build --cdash-site --cdash-track --cdash-buildstamp -y --yes-to-all"
else
_all_packages
fi
@@ -1505,7 +1527,7 @@ _spack_solve() {
_spack_spec() {
if $list_options
then
- SPACK_COMPREPLY="-h --help -l --long -L --very-long -I --install-status -y --yaml -j --json -c --cover -N --namespaces -t --types"
+ SPACK_COMPREPLY="-h --help -l --long -L --very-long -I --install-status -y --yaml -j --json -c --cover -N --namespaces --hash-type -t --types"
else
_all_packages
fi
diff --git a/var/spack/repos/builtin/packages/libabigail/package.py b/var/spack/repos/builtin/packages/libabigail/package.py
index f59f4324bd..48ea8664bd 100644
--- a/var/spack/repos/builtin/packages/libabigail/package.py
+++ b/var/spack/repos/builtin/packages/libabigail/package.py
@@ -20,6 +20,9 @@ class Libabigail(AutotoolsPackage):
depends_on('libdwarf')
depends_on('libxml2')
+ # Libabigail won't generate it's bin without Python
+ depends_on('python@3.8:')
+
# Documentation dependencies
depends_on('doxygen', type="build", when="+docs")
depends_on('py-sphinx', type='build', when="+docs")