Enhance documentation for coverage baseline in tests #9731
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added details on coverage baseline usage and its impact on test analysis.
Added ideal use cases and current constraints for adaptive testing.
gordonsyme
reviewed
Nov 7, 2025
| * Projects with comprehensive test coverage - The more thorough your tests, the more precisely adaptive testing can identify which tests are impacted by changes | ||
| * Test frameworks with built-in coverage support (Jest, pytest, Go test, Vitest) where generating coverage reports is straightforward | ||
|
|
||
| *Why coverage matters*: In codebases with sparse test coverage, adaptive testing cannot accurately determine which tests cover changed code. This causes the system to default to running all tests, negating the benefits of intelligent test selection. |
Member
There was a problem hiding this comment.
What do you think about
This causes the system to run more tests, reducing the benefits of intelligent test selection.
gordonsyme
reviewed
Nov 7, 2025
| |.circleci/example-baseline.info | ||
| |Path to a baseline coverage file to subtract from test coverage during analysis. + | ||
| Use this to exclude shared setup or initialization code from test impact data. + | ||
| The baseline file should be in the same format as your test coverage output (e.g., LCOV format for `<< outputs.lcov >>`). |
Member
There was a problem hiding this comment.
~~ Not strictly true, the baseline can be in any supported coverage format and we'll deal with it. There's no reason not to use the same format as test coverage output though.
gordonsyme
reviewed
Nov 7, 2025
|
|
||
| === Can I run analysis on branches other than main? | ||
|
|
||
| Yes! The branch behavior is fully customizable through your CircleCI configuration. While analysis typically runs on `main` by default, you can configure it to run on: |
Member
There was a problem hiding this comment.
How about:
While analysis runs on
mainby default, you can configure it to run on:
gordonsyme
reviewed
Nov 7, 2025
| . Any specific branch (for example, `develop` or `staging`). | ||
| . Multiple branches simultaneously. | ||
| . Feature branches if needed for testing. | ||
| . Scheduled pipelines independent of branch. |
Member
There was a problem hiding this comment.
~~ Maybe . Scheduled pipelines is enough?
gordonsyme
reviewed
Nov 7, 2025
| * Vitest | ||
|
|
||
| The key requirement is that your test runner can generate coverage data in a parsable format (typically LCOV or similar). | ||
| The key requirement is that your test runner can generate coverage data in a parsable format (typically LCOV or similar). |
Member
There was a problem hiding this comment.
Right now we can only read LCOV and Go's "legacy coverage" format.
gordonsyme
reviewed
Nov 7, 2025
Comment on lines
24
to
27
| * Distributed test architectures where tests run against external services, separate containers, or multiple isolated contexts | ||
| * Limited coverage tooling for test frameworks that don't provide native instrumentation or coverage reporting | ||
| * Complex test configurations with non-standard test discovery, custom test runners, or unconventional project structures | ||
| * End-to-end tests that span multiple repositories or services, making it difficult to map code changes to specific tests |
Member
There was a problem hiding this comment.
I think we'll want to word these as things you test suite is constrained to do, rather than constrained not to do, e.g.:
=== Current Constraints
* Generating code coverage data is essential for determining how tests are related to code. If tests are run in a way that makes generating and accessing code coverage data tricky then adaptive testing may not be a good fit.
* Adaptive testing needs to be configured with commands to discover all available tests and run a subset of those tests. If you cannot run commands to discover tests and run a subset of tests on the CLI then adaptive testing may not be a good fit.
* Adaptive testing works best when testing a single deployable unit. A monorepo which performs integration tests across many packages at once may not be a good fit.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added details on coverage baseline usage and its impact on test analysis.
Description
What did you change?
Reasons
Why did you make these changes?
Content Checklist
Please follow our style when contributing to CircleCI docs. Our style guide is here: https://circleci.com/docs/style/style-guide-overview.
Please take a moment to check through the following items when submitting your PR (this is just a guide so will not be relevant for all PRs):