Skip to content

Add a diff coverage action - #422

Open
rwb27 wants to merge 11 commits into
mainfrom
diff-coverage-action
Open

rwb27 wants to merge 11 commits into
mainfrom
diff-coverage-action

Conversation

@rwb27

@rwb27 rwb27 commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

This replaces barecheck with diff-cover-action to show coverage in PRs. It doesn't rely on external services.
I hope it will be better than our current solution, as I think it is able to check coverage of the diff rather than just global coverage. It wraps the Python diff-cover utility, adding some nice reporting to the PR thread.

Rather than have an ever-growing list of reports in pyproject.toml I have removed those options from the pytest section. Instead, coverage reports should now be specified when they're needed. The action works with many formats of report - we now generate XML, JSON and LCOV of which we're currently using XML (for diff-cover) and JSON (to extract overall coverage). I don't think LCOV does any harm so I've left it.

I tried Github's native coverage action as per the github docs but this is restricted to the AI-enhanced security/quality service that we don't have. I'm not aware of an open source/educational tier that would cover it

For monitoring coverage as I develop, I've switched to using the VSCode tests panel as it puts coverage bars in my editor. Other developers are welcome to switch on whatever coverage reports they prefer, but I think it's not massively helpful to have this all turned on by default. Turning off coverage when I run it at the command line takes tests from ~30s to ~20s on my system, and I think most folk will value the ten seconds more than the pre-emptive coverage report.

I've set the test workflow to run on pushes to main and I hope this will fix the coverage badge.

I've also added some manual logic to check that overall coverage hasn't decreased too much (current threshold is -0.2%). While diff-cover enforces that the whole diff is covered, it doesn't stop someone e.g. deleting lots of test code. I think a fraction of a percent is a reasonable threshold (e.g. to allow for fewer import statements as a result of tidying-up, which can drop coverage slightly because the imports are always covered).

This PR contains no AI-generated code.

This PR should have commits squashed before merging!

@rwb27
rwb27 force-pushed the diff-coverage-action branch 2 times, most recently from cd19545 to 66fa048 Compare November 9, 2026 16:24
@github-actions

This comment was marked as outdated.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
100.0%

 passed

████████████████████ 100.0% on changed lines

0 lines changed   0 uncovered   0 files

 📋 Full diff-cover report

Diff Coverage

Diff: origin/main...HEAD, staged and unstaged changes

No lines with coverage information in this diff.

🛡️ diff-cover-action

@rwb27
rwb27 force-pushed the diff-coverage-action branch from 4a8a2bc to 45330e0 Compare November 9, 2026 22:37
@rwb27
rwb27 requested a review from bprobert97 November 10, 2026 11:47
Comment thread .github/workflows/test.yml Resolved
Comment thread .github/workflows/test.yml Resolved
Comment thread .github/workflows/test.yml Resolved
Comment thread .github/workflows/test.yml Resolved
@rwb27
rwb27 force-pushed the diff-coverage-action branch 2 times, most recently from ea3cae2 to e2d02ca Compare November 18, 2026 13:51
rwb27 added 10 commits November 18, 2026 14:17
This uses the popular `5monkeys/coberatura-action` to show coverage in PRs.
It doesn't rely on external services.
I hope it will be better than our current solution, as I think it is able to check coverage of the diff rather than just global coverage.

I added an xml coverage report to make this work.
Rather than have an ever-growing list of reports in `pyproject.toml` I have removed those options from the pytest section.
Instead, coverage reports should now be specified when they're needed.

We only really need the xml report here, but I've left the others in just in case they're useful in the future.

I've switched to using the VSCode tests panel as it puts coverage bars in my editor.
Other developers are welcome to switch on whatever coverage reports they prefer.
Turning off coverage when I run it at the command line takes tests from ~30s to ~20s on my system.
I hope this does a line-by-line diff.
I need to check out the repo in order to generate the git diff, even though
it's not needed to run the tests.
The coverage badge is reinstated.

I removed the code quality upload, as it's reserved for premium tiers.
@rwb27
rwb27 force-pushed the diff-coverage-action branch from e2d02ca to d59d47d Compare November 18, 2026 14:37
I've restored the base coverage job, and added steps to the coverage job that
compare total coverage of this branch to `main`.

The job will now fail if coverage has dropped by more than 0.2%.
I no longer feel the need to set the threshold at 0.

I've also set the test job to run on the `main` branch, so that we can
get the badge updating again.
@rwb27
rwb27 force-pushed the diff-coverage-action branch 2 times, most recently from 645a61f to cbff0ff Compare November 18, 2026 14:57
@rwb27
rwb27 requested a review from bprobert97 November 18, 2026 14:57
echo "delta=$change" >> $GITHUB_OUTPUT

- name: Check coverage tolerance
if: ${{ steps.coverage-delta.outputs.delta < -0.2 }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.totals.percent_covered_display is rounded to coverage.py’s precision setting. That defaults to 0 and .coveragerc doesn’t change it, so both numbers are whole percentages. A real drop of 0.2 points across a rounding boundary (94.6 to 94.4) appears as −1 and fails. A real drop of 0.9 (95.4 to 94.5) can appear as 0 and pass. We can either use .totals.percent_covered (the unrounded value) in both jobs, or update .coveragerc to change the precision

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants