summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorAdam J. Stewart <ajstewart426@gmail.com>2018-01-07 10:28:56 -0600
committerTodd Gamblin <tgamblin@llnl.gov>2018-01-07 08:28:56 -0800
commitffd435bad70bc3005e2d5b81e0119fa67c60db05 (patch)
treeb3f8e6b2230595b380860b3a7cd168988a0ea804
parent38fabce2aabbf5a131b4ca056187bc5afe092983 (diff)
downloadspack-ffd435bad70bc3005e2d5b81e0119fa67c60db05.tar.gz
spack-ffd435bad70bc3005e2d5b81e0119fa67c60db05.tar.bz2
spack-ffd435bad70bc3005e2d5b81e0119fa67c60db05.tar.xz
spack-ffd435bad70bc3005e2d5b81e0119fa67c60db05.zip
Add documentation on Codecov (#6842)
-rw-r--r--lib/spack/docs/contribution_guide.rst72
1 files changed, 60 insertions, 12 deletions
diff --git a/lib/spack/docs/contribution_guide.rst b/lib/spack/docs/contribution_guide.rst
index 6ee55c2f66..50eea422d1 100644
--- a/lib/spack/docs/contribution_guide.rst
+++ b/lib/spack/docs/contribution_guide.rst
@@ -34,13 +34,30 @@ Continuous Integration
Spack uses `Travis CI <https://travis-ci.org/spack/spack>`_ for Continuous Integration
testing. This means that every time you submit a pull request, a series of tests will
-be run to make sure you didn't accidentally introduce any bugs into Spack. Your PR
-will not be accepted until it passes all of these tests. While you can certainly wait
+be run to make sure you didn't accidentally introduce any bugs into Spack. **Your PR
+will not be accepted until it passes all of these tests.** While you can certainly wait
for the results of these tests after submitting a PR, we recommend that you run them
locally to speed up the review process.
+.. note::
+
+ Oftentimes, Travis will fail for reasons other than a problem with your PR.
+ For example, apt-get, pip, or homebrew will fail to download one of the
+ dependencies for the test suite, or a transient bug will cause the unit tests
+ to timeout. If Travis fails, click the "Details" link and click on the test(s)
+ that is failing. If it doesn't look like it is failing for reasons related to
+ your PR, you have two options. If you have write permissions for the Spack
+ repository, you should see a "Restart job" button on the right-hand side. If
+ not, you can close and reopen your PR to rerun all of the tests. If the same
+ test keeps failing, there may be a problem with your PR. If you notice that
+ every recent PR is failing with the same error message, it may be that Travis
+ is down or one of Spack's dependencies put out a new release that is causing
+ problems. If this is the case, please file an issue.
+
+
If you take a look in ``$SPACK_ROOT/.travis.yml``, you'll notice that we test
-against Python 2.6, 2.7, and 3.3-3.6. We currently perform 3 types of tests:
+against Python 2.6, 2.7, and 3.3-3.6 on both macOS and Linux. We currently
+perform 3 types of tests:
^^^^^^^^^^
Unit Tests
@@ -87,7 +104,7 @@ adding new unit tests or strengthening existing tests.
.. note::
There is also a ``run-unit-tests`` script in ``share/spack/qa`` that
- runs the unit tests. Afterwards, it reports back to Coverage with the
+ runs the unit tests. Afterwards, it reports back to Codecov with the
percentage of Spack that is covered by unit tests. This script is
designed for Travis CI. If you want to run the unit tests yourself, we
suggest you use ``spack test``.
@@ -154,14 +171,14 @@ However, if you aren't compliant with PEP 8, flake8 will complain:
Most of the error messages are straightforward, but if you don't understand what
they mean, just ask questions about them when you submit your PR. The line numbers
-will change if you add or delete lines, so simply run ``run-flake8-tests`` again
+will change if you add or delete lines, so simply run ``spack flake8`` again
to update them.
.. tip::
Try fixing flake8 errors in reverse order. This eliminates the need for
- multiple runs of ``flake8`` just to re-compute line numbers and makes it
- much easier to fix errors directly off of the Travis output.
+ multiple runs of ``spack flake8`` just to re-compute line numbers and
+ makes it much easier to fix errors directly off of the Travis output.
.. warning::
@@ -225,11 +242,11 @@ your PR is accepted.
.. note::
- There is also a ``run-doc-tests`` script in the Quality Assurance directory.
- The only difference between running this script and running ``make`` by hand
- is that the script will exit immediately if it encounters an error or warning.
- This is necessary for Travis CI. If you made a lot of documentation changes, it
- is much quicker to run ``make`` by hand so that you can see all of the warnings
+ There is also a ``run-doc-tests`` script in ``share/spack/qa``. The only
+ difference between running this script and running ``make`` by hand is that
+ the script will exit immediately if it encounters an error or warning. This
+ is necessary for Travis CI. If you made a lot of documentation changes, it is
+ much quicker to run ``make`` by hand so that you can see all of the warnings
at once.
If you are editing the documentation, you should obviously be running the
@@ -277,6 +294,37 @@ Documentation changes can result in much more obfuscated warning messages.
If you don't understand what they mean, feel free to ask when you submit
your PR.
+--------
+Coverage
+--------
+
+Spack uses `Codecov <https://codecov.io/>`_ to generate and report unit test
+coverage. This helps us tell what percentage of lines of code in Spack are
+covered by unit tests. Although code covered by unit tests can still contain
+bugs, it is much less error prone than code that is not covered by unit tests.
+
+Codecov provides `browser extensions <https://github.com/codecov/browser-extension>`_
+for Google Chrome, Firefox, and Opera. These extensions integrate with GitHub
+and allow you to see coverage line-by-line when viewing the Spack repository.
+If you are new to Spack, a great way to get started is to write unit tests to
+increase coverage!
+
+Unlike with Travis, Codecov tests are not required to pass in order for your
+PR to be merged. If you modify core Spack libraries, we would greatly
+appreciate unit tests that cover these changed lines. Otherwise, we have no
+way of knowing whether or not your changes introduce a bug. If you make
+substantial changes to the core, we may request unit tests to increase coverage.
+
+.. note::
+
+ If the only files you modified are package files, we do not care about
+ coverage on your PR. You may notice that the Codecov tests fail even though
+ you didn't modify any core files. This means that Spack's overall coverage
+ has increased since you branched off of develop. This is a good thing!
+ If you really want to get the Codecov tests to pass, you can rebase off of
+ the latest develop, but again, this is not required.
+
+
-------------
Git Workflows
-------------