From 2cefeb2dd9e4cc256431bdf2f716d02ccfa8e537 Mon Sep 17 00:00:00 2001 From: mseong6251 Date: Tue, 4 Nov 2025 15:55:59 -0500 Subject: [PATCH 01/16] Enhance documentation for coverage baseline in tests Added details on coverage baseline usage and its impact on test analysis. --- .../modules/test/pages/adaptive-testing.adoc | 39 ++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-) diff --git a/docs/guides/modules/test/pages/adaptive-testing.adoc b/docs/guides/modules/test/pages/adaptive-testing.adoc index 62c32f380d..2e96381037 100644 --- a/docs/guides/modules/test/pages/adaptive-testing.adoc +++ b/docs/guides/modules/test/pages/adaptive-testing.adoc @@ -490,6 +490,12 @@ Any remaining tests will be analysed the next time test analysis is run. |true |Whether the tests should be distributed across a shared queue and fetched across multiple dynamic batches. + If a test runner has slow start up time per batch, disabling this can speed up tests. + +| `coverage-baseline` +|.circleci/example-baseline.info +|Path to a baseline coverage file to subtract from test coverage during analysis. + +Use this to exclude shared setup or initialization code from test impact data. + +The baseline file should be in the same format as your test coverage output (e.g., LCOV format for `<< outputs.lcov >>`). |=== The following flags are available to be defined on the `circleci run testsuite` command. @@ -770,6 +776,37 @@ Yes! The branch behavior is fully customizable through your CircleCI configurati See Scenario 3 in the "Flag Usage Scenarios" section for examples of customizing branch behavior. +=== Can I run analysis on branches other than main? + +Yes! The branch behavior is fully customizable through your CircleCI configuration. While analysis typically runs on `main` by default, you can configure it to run on: + +. Any specific branch (for example, `develop` or `staging`). +. Multiple branches simultaneously. +. Feature branches if needed for testing. +. Scheduled pipelines independent of branch. + +See Scenario 3 in the "Flag Usage Scenarios" section for examples of customizing branch behavior. + +=== Why are there so many files impact a test? + +If you see many files impacting each test during analysis (e.g., "...found 150 files impacting test..."), this may be caused by shared setup code like global imports or framework initialization being included in coverage. + +*Solution: Use a coverage baseline to exclude setup code* + +. Create a minimal test that only does imports/setup (no test logic) +. Generate coverage from that test and save it to `.circleci/example-baseline.info` (you can name the file however you'd like) +. Reference it in your test suite: + +[source,yaml] +---- +# .circleci/test-suites.yml +options: + adaptive-testing: true + coverage-baseline: .circleci/example-baseline.info +---- + +The coverage data in the baseline file will be subtracted from each test's coverage during analysis. Rerun analysis and you should see fewer impacting files per test. Note that the baseline file should be in the same format as your test coverage output (e.g., LCOV format for << outputs.lcov >>). + === What test frameworks are supported? Adaptive testing is runner-agnostic. We provide default configurations for the following test frameworks: @@ -782,4 +819,4 @@ Adaptive testing is runner-agnostic. We provide default configurations for the f * Cypress (E2E testing) * Vitest -The key requirement is that your test runner can generate coverage data in a parsable format (typically LCOV or similar). \ No newline at end of file +The key requirement is that your test runner can generate coverage data in a parsable format (typically LCOV or similar). From d8b602d045f8992edccc115b1c2db6a2ec3cc369 Mon Sep 17 00:00:00 2001 From: mseong6251 Date: Thu, 6 Nov 2025 17:33:43 -0500 Subject: [PATCH 02/16] Expand adaptive testing documentation Added ideal use cases and current constraints for adaptive testing. --- .../modules/test/pages/adaptive-testing.adoc | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/guides/modules/test/pages/adaptive-testing.adoc b/docs/guides/modules/test/pages/adaptive-testing.adoc index 2e96381037..bf38f73984 100644 --- a/docs/guides/modules/test/pages/adaptive-testing.adoc +++ b/docs/guides/modules/test/pages/adaptive-testing.adoc @@ -11,6 +11,22 @@ NOTE: This page is currently in development and will be updated as the feature i Use adaptive testing to run only tests that are impacted by code changes and evenly distribute tests across parallel execution nodes. Adaptive testing reduces test execution time while maintaining test confidence. +== Where Adaptive Testing works well: + +=== Ideal use cases +* Unit and integration tests that exercise code within the same repository +* Projects with comprehensive test coverage - The more thorough your tests, the more precisely adaptive testing can identify which tests are impacted by changes +* Test frameworks with built-in coverage support (Jest, pytest, Go test, Vitest) where generating coverage reports is straightforward + +*Why coverage matters*: In codebases with sparse test coverage, adaptive testing cannot accurately determine which tests cover changed code. This causes the system to default to running all tests, negating the benefits of intelligent test selection. + +=== Current Constraints +* Distributed test architectures where tests run against external services, separate containers, or multiple isolated contexts +* Limited coverage tooling for test frameworks that don't provide native instrumentation or coverage reporting +* Complex test configurations with non-standard test discovery, custom test runners, or unconventional project structures +* End-to-end tests that span multiple repositories or services, making it difficult to map code changes to specific tests + + == Key benefits: * Faster CI/CD pipelines through intelligent test selection. From 3909ddb8c6bca13ec04a22fd037b458b00f29768 Mon Sep 17 00:00:00 2001 From: mseong6251 Date: Fri, 7 Nov 2025 09:45:01 -0500 Subject: [PATCH 03/16] Update section title and refine constraints description --- docs/guides/modules/test/pages/adaptive-testing.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/guides/modules/test/pages/adaptive-testing.adoc b/docs/guides/modules/test/pages/adaptive-testing.adoc index bf38f73984..a41d7e611a 100644 --- a/docs/guides/modules/test/pages/adaptive-testing.adoc +++ b/docs/guides/modules/test/pages/adaptive-testing.adoc @@ -20,10 +20,10 @@ Use adaptive testing to run only tests that are impacted by code changes and eve *Why coverage matters*: In codebases with sparse test coverage, adaptive testing cannot accurately determine which tests cover changed code. This causes the system to default to running all tests, negating the benefits of intelligent test selection. -=== Current Constraints -* Distributed test architectures where tests run against external services, separate containers, or multiple isolated contexts +=== Current Friction points +* Cases where it is difficult to instrument and get coverage from the code you want to test * Limited coverage tooling for test frameworks that don't provide native instrumentation or coverage reporting -* Complex test configurations with non-standard test discovery, custom test runners, or unconventional project structures +* Complex test configurations * End-to-end tests that span multiple repositories or services, making it difficult to map code changes to specific tests From 5c5cc5eee942345cdcf11e1e0d7fee32692f8ad6 Mon Sep 17 00:00:00 2001 From: mseong6251 Date: Wed, 12 Nov 2025 16:55:15 -0500 Subject: [PATCH 04/16] Update adaptive-testing.adoc --- .../guides/modules/test/pages/adaptive-testing.adoc | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/docs/guides/modules/test/pages/adaptive-testing.adoc b/docs/guides/modules/test/pages/adaptive-testing.adoc index a41d7e611a..5904a1ac93 100644 --- a/docs/guides/modules/test/pages/adaptive-testing.adoc +++ b/docs/guides/modules/test/pages/adaptive-testing.adoc @@ -18,7 +18,7 @@ Use adaptive testing to run only tests that are impacted by code changes and eve * Projects with comprehensive test coverage - The more thorough your tests, the more precisely adaptive testing can identify which tests are impacted by changes * Test frameworks with built-in coverage support (Jest, pytest, Go test, Vitest) where generating coverage reports is straightforward -*Why coverage matters*: In codebases with sparse test coverage, adaptive testing cannot accurately determine which tests cover changed code. This causes the system to default to running all tests, negating the benefits of intelligent test selection. +*Why coverage matters*: In codebases with sparse test coverage, adaptive testing cannot accurately determine which tests cover changed code. This causes the system to run more tests, reducing the benefits of intelligent test selection. === Current Friction points * Cases where it is difficult to instrument and get coverage from the code you want to test @@ -511,7 +511,7 @@ If a test runner has slow start up time per batch, disabling this can speed up t |.circleci/example-baseline.info |Path to a baseline coverage file to subtract from test coverage during analysis. + Use this to exclude shared setup or initialization code from test impact data. + -The baseline file should be in the same format as your test coverage output (e.g., LCOV format for `<< outputs.lcov >>`). +The baseline file can be in any supported coverage format. While it doesn't need to match your test coverage output format, using the same format (e.g., LCOV format for << outputs.lcov >>) is recommended for consistency. |=== The following flags are available to be defined on the `circleci run testsuite` command. @@ -792,14 +792,13 @@ Yes! The branch behavior is fully customizable through your CircleCI configurati See Scenario 3 in the "Flag Usage Scenarios" section for examples of customizing branch behavior. -=== Can I run analysis on branches other than main? +=== Can I run test analysis and selection on any branch? -Yes! The branch behavior is fully customizable through your CircleCI configuration. While analysis typically runs on `main` by default, you can configure it to run on: +Yes! The branch behavior is fully customizable through your CircleCI configuration. While analysis runs on main by default, you can configure it to run on: . Any specific branch (for example, `develop` or `staging`). -. Multiple branches simultaneously. . Feature branches if needed for testing. -. Scheduled pipelines independent of branch. +. Scheduled pipelines. See Scenario 3 in the "Flag Usage Scenarios" section for examples of customizing branch behavior. @@ -835,4 +834,4 @@ Adaptive testing is runner-agnostic. We provide default configurations for the f * Cypress (E2E testing) * Vitest -The key requirement is that your test runner can generate coverage data in a parsable format (typically LCOV or similar). +The key requirement is that your test runner can generate coverage data in a parsable format (currently, we suppport LCOV and Go's "legacy coverage" format). From 64963c771c534e696e0d46fed14ac73d6b8eba53 Mon Sep 17 00:00:00 2001 From: mseong6251 Date: Wed, 12 Nov 2025 17:00:13 -0500 Subject: [PATCH 05/16] Update Current Constraints --- docs/guides/modules/test/pages/adaptive-testing.adoc | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/docs/guides/modules/test/pages/adaptive-testing.adoc b/docs/guides/modules/test/pages/adaptive-testing.adoc index 5904a1ac93..b05993e6b4 100644 --- a/docs/guides/modules/test/pages/adaptive-testing.adoc +++ b/docs/guides/modules/test/pages/adaptive-testing.adoc @@ -20,12 +20,10 @@ Use adaptive testing to run only tests that are impacted by code changes and eve *Why coverage matters*: In codebases with sparse test coverage, adaptive testing cannot accurately determine which tests cover changed code. This causes the system to run more tests, reducing the benefits of intelligent test selection. -=== Current Friction points -* Cases where it is difficult to instrument and get coverage from the code you want to test -* Limited coverage tooling for test frameworks that don't provide native instrumentation or coverage reporting -* Complex test configurations -* End-to-end tests that span multiple repositories or services, making it difficult to map code changes to specific tests - +=== Current Constraints +* Generating code coverage data is essential for determining how tests are related to code. If tests are run in a way that makes generating and accessing code coverage data tricky then adaptive testing may not be a good fit. +* Adaptive testing needs to be configured with commands to discover all available tests and run a subset of those tests. If you cannot run commands to discover tests and run a subset of tests on the CLI then adaptive testing may not be a good fit. +* Adaptive testing works best when testing a single deployable unit. A monorepo which performs integration tests across many packages at once may not be a good fit. == Key benefits: From 169a7bfa9a7b2a4de007f82957d0ae7590737875 Mon Sep 17 00:00:00 2001 From: mseong6251 Date: Wed, 12 Nov 2025 17:35:55 -0500 Subject: [PATCH 06/16] Link example to baseline coverage feature --- docs/guides/modules/test/pages/adaptive-testing.adoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/guides/modules/test/pages/adaptive-testing.adoc b/docs/guides/modules/test/pages/adaptive-testing.adoc index fa6e970a40..c3d83bbd17 100644 --- a/docs/guides/modules/test/pages/adaptive-testing.adoc +++ b/docs/guides/modules/test/pages/adaptive-testing.adoc @@ -511,7 +511,7 @@ If a test runner has slow start up time per batch, disabling this can speed up t |.circleci/example-baseline.info |Path to a baseline coverage file to subtract from test coverage during analysis. + Use this to exclude shared setup or initialization code from test impact data. + -The baseline file can be in any supported coverage format. While it doesn't need to match your test coverage output format, using the same format (e.g., LCOV format for << outputs.lcov >>) is recommended for consistency. +The baseline file can be in any supported coverage format. While it doesn't need to match your test coverage output format, using the same format (e.g., LCOV format for << outputs.lcov >>) is recommended for consistency. For more information, refer to "<>". |=== -- @@ -806,6 +806,7 @@ Yes! The branch behavior is fully customizable through your CircleCI configurati See Scenario 3 in the "Flag Usage Scenarios" section for examples of customizing branch behavior. +[#baseline-coverage] === Why are there so many files impact a test? If you see many files impacting each test during analysis (e.g., "...found 150 files impacting test..."), this may be caused by shared setup code like global imports or framework initialization being included in coverage. From 4715d781c5137c0abdffd76ecb9912ac8d20005d Mon Sep 17 00:00:00 2001 From: mseong6251 Date: Thu, 13 Nov 2025 09:20:49 -0500 Subject: [PATCH 07/16] Update docs/guides/modules/test/pages/adaptive-testing.adoc Co-authored-by: Rosie Yohannan --- .../modules/test/pages/adaptive-testing.adoc | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/docs/guides/modules/test/pages/adaptive-testing.adoc b/docs/guides/modules/test/pages/adaptive-testing.adoc index c3d83bbd17..c0333901d0 100644 --- a/docs/guides/modules/test/pages/adaptive-testing.adoc +++ b/docs/guides/modules/test/pages/adaptive-testing.adoc @@ -11,20 +11,19 @@ NOTE: This page is currently in development and will be updated as the feature i Use adaptive testing to run only tests that are impacted by code changes and evenly distribute tests across parallel execution nodes. Adaptive testing reduces test execution time while maintaining test confidence. -== Where Adaptive Testing works well: +== Use cases for adaptive testing -=== Ideal use cases -* Unit and integration tests that exercise code within the same repository -* Projects with comprehensive test coverage - The more thorough your tests, the more precisely adaptive testing can identify which tests are impacted by changes -* Test frameworks with built-in coverage support (Jest, pytest, Go test, Vitest) where generating coverage reports is straightforward +* Unit and integration tests that exercise code within the same repository. +* Projects with comprehensive test coverage. The more thorough your tests, the more precisely adaptive testing can identify which tests are impacted by changes. +* Test frameworks with built-in coverage support (Jest, pytest, Go test, Vitest) where generating coverage reports is straightforward. ++ +TIP: In codebases with sparse test coverage, adaptive testing cannot accurately determine which tests cover changed code. This causes the system to run more tests, reducing the benefits of intelligent test selection. -*Why coverage matters*: In codebases with sparse test coverage, adaptive testing cannot accurately determine which tests cover changed code. This causes the system to run more tests, reducing the benefits of intelligent test selection. +== Limitations -=== Current Constraints * Generating code coverage data is essential for determining how tests are related to code. If tests are run in a way that makes generating and accessing code coverage data tricky then adaptive testing may not be a good fit. * Adaptive testing needs to be configured with commands to discover all available tests and run a subset of those tests. If you cannot run commands to discover tests and run a subset of tests on the CLI then adaptive testing may not be a good fit. * Adaptive testing works best when testing a single deployable unit. A monorepo which performs integration tests across many packages at once may not be a good fit. - == Key benefits: * Faster CI/CD pipelines through intelligent test selection. From 7dd656fbc334b97f5612f0ae4dfc7e6f26280aee Mon Sep 17 00:00:00 2001 From: mseong6251 Date: Thu, 13 Nov 2025 09:21:01 -0500 Subject: [PATCH 08/16] Update docs/guides/modules/test/pages/adaptive-testing.adoc Co-authored-by: Rosie Yohannan --- docs/guides/modules/test/pages/adaptive-testing.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/modules/test/pages/adaptive-testing.adoc b/docs/guides/modules/test/pages/adaptive-testing.adoc index c0333901d0..d497f071b9 100644 --- a/docs/guides/modules/test/pages/adaptive-testing.adoc +++ b/docs/guides/modules/test/pages/adaptive-testing.adoc @@ -510,7 +510,7 @@ If a test runner has slow start up time per batch, disabling this can speed up t |.circleci/example-baseline.info |Path to a baseline coverage file to subtract from test coverage during analysis. + Use this to exclude shared setup or initialization code from test impact data. + -The baseline file can be in any supported coverage format. While it doesn't need to match your test coverage output format, using the same format (e.g., LCOV format for << outputs.lcov >>) is recommended for consistency. For more information, refer to "<>". +The baseline file can be in any supported coverage format. While it does not need to match your test coverage output format, using the same format (for example, LCOV format for `<< outputs.lcov >>`) is recommended for consistency. For more information, refer to the <> section. |=== -- From 738b28492c069e7bee23230d464edc412f003788 Mon Sep 17 00:00:00 2001 From: mseong6251 Date: Thu, 13 Nov 2025 09:21:10 -0500 Subject: [PATCH 09/16] Update docs/guides/modules/test/pages/adaptive-testing.adoc Co-authored-by: Rosie Yohannan --- docs/guides/modules/test/pages/adaptive-testing.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/modules/test/pages/adaptive-testing.adoc b/docs/guides/modules/test/pages/adaptive-testing.adoc index d497f071b9..d8fde44185 100644 --- a/docs/guides/modules/test/pages/adaptive-testing.adoc +++ b/docs/guides/modules/test/pages/adaptive-testing.adoc @@ -806,7 +806,7 @@ Yes! The branch behavior is fully customizable through your CircleCI configurati See Scenario 3 in the "Flag Usage Scenarios" section for examples of customizing branch behavior. [#baseline-coverage] -=== Why are there so many files impact a test? +=== Why are there so many files impacting a test? If you see many files impacting each test during analysis (e.g., "...found 150 files impacting test..."), this may be caused by shared setup code like global imports or framework initialization being included in coverage. From 18563608421a5f698d247f3966e6695092744cd0 Mon Sep 17 00:00:00 2001 From: mseong6251 Date: Thu, 13 Nov 2025 09:21:36 -0500 Subject: [PATCH 10/16] Update docs/guides/modules/test/pages/adaptive-testing.adoc Co-authored-by: Rosie Yohannan --- docs/guides/modules/test/pages/adaptive-testing.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/guides/modules/test/pages/adaptive-testing.adoc b/docs/guides/modules/test/pages/adaptive-testing.adoc index d8fde44185..885f26330f 100644 --- a/docs/guides/modules/test/pages/adaptive-testing.adoc +++ b/docs/guides/modules/test/pages/adaptive-testing.adoc @@ -808,7 +808,7 @@ See Scenario 3 in the "Flag Usage Scenarios" section for examples of customizing [#baseline-coverage] === Why are there so many files impacting a test? -If you see many files impacting each test during analysis (e.g., "...found 150 files impacting test..."), this may be caused by shared setup code like global imports or framework initialization being included in coverage. +If you see many files impacting each test during analysis (for example, "...found 150 files impacting test..."), this may be caused by shared setup code like global imports or framework initialization being included in coverage. *Solution: Use a coverage baseline to exclude setup code* From 81ade3c1c414e7aff29344b6707c88a9c7237521 Mon Sep 17 00:00:00 2001 From: mseong6251 Date: Thu, 13 Nov 2025 09:31:14 -0500 Subject: [PATCH 11/16] Update adaptive-testing.adoc --- docs/guides/modules/test/pages/adaptive-testing.adoc | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/guides/modules/test/pages/adaptive-testing.adoc b/docs/guides/modules/test/pages/adaptive-testing.adoc index 885f26330f..cbe906c1b9 100644 --- a/docs/guides/modules/test/pages/adaptive-testing.adoc +++ b/docs/guides/modules/test/pages/adaptive-testing.adoc @@ -24,6 +24,7 @@ TIP: In codebases with sparse test coverage, adaptive testing cannot accurately * Generating code coverage data is essential for determining how tests are related to code. If tests are run in a way that makes generating and accessing code coverage data tricky then adaptive testing may not be a good fit. * Adaptive testing needs to be configured with commands to discover all available tests and run a subset of those tests. If you cannot run commands to discover tests and run a subset of tests on the CLI then adaptive testing may not be a good fit. * Adaptive testing works best when testing a single deployable unit. A monorepo which performs integration tests across many packages at once may not be a good fit. + == Key benefits: * Faster CI/CD pipelines through intelligent test selection. From 0cd9834e3bab97ec87de355c2965891274747115 Mon Sep 17 00:00:00 2001 From: Gordon Syme Date: Fri, 14 Nov 2025 11:10:20 +0000 Subject: [PATCH 12/16] Update baseline docs for baseline command --- .../modules/test/pages/adaptive-testing.adoc | 36 ++++++++++++------- 1 file changed, 23 insertions(+), 13 deletions(-) diff --git a/docs/guides/modules/test/pages/adaptive-testing.adoc b/docs/guides/modules/test/pages/adaptive-testing.adoc index cbe906c1b9..c35bdfd9cc 100644 --- a/docs/guides/modules/test/pages/adaptive-testing.adoc +++ b/docs/guides/modules/test/pages/adaptive-testing.adoc @@ -322,7 +322,6 @@ Supported coverage template variables: * `<< outputs.lcov >>`: Coverage data in LCOV format. * `<< outputs.go-coverage >>`: Coverage data in Go coverage format. -* `<< outputs.gcov >>`: Coverage data in `gcov` coverage format. The coverage location does not need to be set in the outputs map, a temporary file will be created and used during analysis with the template variable from the analysis command. @@ -506,12 +505,6 @@ Any remaining tests will be analysed the next time test analysis is run. |true |Whether the tests should be distributed across a shared queue and fetched across multiple dynamic batches. + If a test runner has slow start up time per batch, disabling this can speed up tests. - -| `coverage-baseline` -|.circleci/example-baseline.info -|Path to a baseline coverage file to subtract from test coverage during analysis. + -Use this to exclude shared setup or initialization code from test impact data. + -The baseline file can be in any supported coverage format. While it does not need to match your test coverage output format, using the same format (for example, LCOV format for `<< outputs.lcov >>`) is recommended for consistency. For more information, refer to the <> section. |=== -- @@ -811,21 +804,38 @@ See Scenario 3 in the "Flag Usage Scenarios" section for examples of customizing If you see many files impacting each test during analysis (for example, "...found 150 files impacting test..."), this may be caused by shared setup code like global imports or framework initialization being included in coverage. -*Solution: Use a coverage baseline to exclude setup code* +This extraneous coverage can be excluded by providing an `analysis-baseline` command to compute the code covered during startup that isn't directly exercised by test code. We call this "baseline coverage data". + +The `analysis-baseline` command must produce coverage output written a coverage template variable. The baseline coverage data can be in any supported coverage format. While it does not need to match your test coverage output format, using the same format (for example, LCOV format for `<< outputs.lcov >>`) is recommended for consistency. -. Create a minimal test that only does imports/setup (no test logic) -. Generate coverage from that test and save it to `.circleci/example-baseline.info` (you can name the file however you'd like) -. Reference it in your test suite: +. Create a minimal test that only does imports/setup (no test logic), in the example below this is called `src/baseline/noop.test.ts`. +. Add an `analysis-baseline` command to your test suite. This command will be broadly similar to your `analysis` command, except that it should only run the minimal test. [source,yaml] ---- # .circleci/test-suites.yml +name: ci tests +discover: jest --listTests --testPathPattern=src/ +run: JEST_JUNIT_OUTPUT_FILE="<< outputs.junit >>" jest --runInBand --reporters=jest-junit --bail << test.atoms >> +analysis: | + jest --runInBand --silent --bail --coverage --coverageProvider=v8 \ + --coverage-directory="$(dirname << outputs.lcov >>)" \ + --coverageReporters=lcovonly \ + << test.atoms >> \ + && cat "$(dirname << outputs.lcov >>)"/*.info > << outputs.lcov >> +analysis-baseline: | + jest --runInBand --silent --bail --coverage --coverageProvider=v8 \ + --coverageReporters=lcovonly \ + --coverage-directory="$(dirname << outputs.lcov >>)" \ + "src/baseline/noop.test.ts" \ + && cat "$(dirname << outputs.lcov >>)"/*.info > << outputs.lcov >> +outputs: + junit: test-reports/tests.xml options: adaptive-testing: true - coverage-baseline: .circleci/example-baseline.info ---- -The coverage data in the baseline file will be subtracted from each test's coverage during analysis. Rerun analysis and you should see fewer impacting files per test. Note that the baseline file should be in the same format as your test coverage output (e.g., LCOV format for << outputs.lcov >>). +The `analysis-baseline` command will be run just before running analysis. The coverage data produced by the `analysis-baseline` command will be subtracted from each test's coverage during analysis. Rerun analysis and you should see fewer impacting files per test. === What test frameworks are supported? From 4f9f8bb192fa5171bf3bffe53148172b60d8891a Mon Sep 17 00:00:00 2001 From: rosie yohannan Date: Fri, 14 Nov 2025 12:23:01 +0000 Subject: [PATCH 13/16] properly format examples and correct links to third example --- .../modules/test/pages/adaptive-testing.adoc | 40 ++++++++++++------- 1 file changed, 25 insertions(+), 15 deletions(-) diff --git a/docs/guides/modules/test/pages/adaptive-testing.adoc b/docs/guides/modules/test/pages/adaptive-testing.adoc index c35bdfd9cc..d070a639f1 100644 --- a/docs/guides/modules/test/pages/adaptive-testing.adoc +++ b/docs/guides/modules/test/pages/adaptive-testing.adoc @@ -9,7 +9,12 @@ CAUTION: *Adaptive testing* is available in closed preview. When the feature is NOTE: This page is currently in development and will be updated as the feature is developed. -Use adaptive testing to run only tests that are impacted by code changes and evenly distribute tests across parallel execution nodes. Adaptive testing reduces test execution time while maintaining test confidence. +Use adaptive testing to optimize test runs as follows: + +* Run only tests that are impacted by code changes. +* Evenly distribute tests across parallel execution nodes. + +Adaptive testing reduces test execution time while maintaining test confidence. == Use cases for adaptive testing @@ -25,7 +30,7 @@ TIP: In codebases with sparse test coverage, adaptive testing cannot accurately * Adaptive testing needs to be configured with commands to discover all available tests and run a subset of those tests. If you cannot run commands to discover tests and run a subset of tests on the CLI then adaptive testing may not be a good fit. * Adaptive testing works best when testing a single deployable unit. A monorepo which performs integration tests across many packages at once may not be a good fit. -== Key benefits: +== Key benefits * Faster CI/CD pipelines through intelligent test selection. * Optimized resource usage and cost efficiency. @@ -342,8 +347,8 @@ options: *Checklist* -. The `analysis` command defines `<< test.atoms >>` to pass in the test, or passes in stdin. -. The `analysis` command defines `<< outputs.lcov|go-coverage|gcov >>` to write coverage data. +* The `analysis` command defines `<< test.atoms >>` to pass in the test, or passes in stdin. +* The `analysis` command defines `<< outputs.lcov|go-coverage|gcov >>` to write coverage data. *Examples of `analysis` commands* @@ -427,8 +432,8 @@ This section will run analysis on a feature branch to seed the initial impact da *Checklist* -. The step output includes prefix Running impact analysis. -. The step output finds files impacting a test (for example, found 12 files impacting test `src/foo.test.ts`). +* The step output includes prefix Running impact analysis. +* The step output finds files impacting a test (for example, found 12 files impacting test `src/foo.test.ts`). [source,yaml] ---- @@ -542,18 +547,19 @@ Now the test suite is set up, test selection is working and the test analysis is *Checklist* -. The `.circleci/config.yml` is set up to run analysis on the default branch. -. The `.circleci/config.yml` is set up to run selection on non-default branch. -. The `.circleci/config.yml` is set up to use high parallelism on the analysis branch. +* The `.circleci/config.yml` is set up to run analysis on the default branch. +* The `.circleci/config.yml` is set up to run selection on non-default branch. +* The `.circleci/config.yml` is set up to use high parallelism on the analysis branch. === Examples -*Running analysis on a branch named `main` and selection on all other branches* +==== Run analysis on a branch named `main` and selection on all other branches No changes required, this is the default setting. -*Running analysis on a branch named `master` and selection on all other branches* +==== Run analysis on a branch named `master` and selection on all other branches +.CircleCI configuration for running analysis on a branch named `master` and selection on all other branches [source,yaml] ---- # .circleci/config.yml @@ -569,8 +575,9 @@ jobs: path: test-reports ---- -*Running higher parallelism on the analysis branch* +==== Run higher parallelism on the analysis branch +.CircleCI configuration for running parallelism of 10 on the main branch and 2 on all other branches [source,yaml] ---- # .circleci/config.yml @@ -586,8 +593,10 @@ jobs: path: test-reports ---- -*Running analysis on a scheduled pipeline and timeboxing some analysis on main* +[#run-analysis-on-scheduled-pipeline] +==== Run analysis on a scheduled pipeline and timeboxing some analysis on main +.CircleCI configuration for running analysis only on scheduled pipelines [source,yaml] ---- # .circleci/config.yml @@ -620,6 +629,7 @@ workflows: - test ---- +.Test suite config. Set time limit of 10 minutes for the analysis on the main branch [source,yaml] ---- # .circleci/test-suites.yml @@ -787,7 +797,7 @@ Yes! The branch behavior is fully customizable through your CircleCI configurati . Feature branches if needed for testing. . Scheduled pipelines independent of branch. -See Scenario 3 in the "Flag Usage Scenarios" section for examples of customizing branch behavior. +See the <> example for an example of customizing branch behavior. === Can I run test analysis and selection on any branch? @@ -797,7 +807,7 @@ Yes! The branch behavior is fully customizable through your CircleCI configurati . Feature branches if needed for testing. . Scheduled pipelines. -See Scenario 3 in the "Flag Usage Scenarios" section for examples of customizing branch behavior. +See the <> example for an example of customizing branch behavior. [#baseline-coverage] === Why are there so many files impacting a test? From 58692237c1796ca14a4bd2d2e971d9042557384f Mon Sep 17 00:00:00 2001 From: rosie yohannan Date: Fri, 14 Nov 2025 12:44:38 +0000 Subject: [PATCH 14/16] split up some long sentences for the linter --- docs/guides/modules/test/pages/adaptive-testing.adoc | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/guides/modules/test/pages/adaptive-testing.adoc b/docs/guides/modules/test/pages/adaptive-testing.adoc index d070a639f1..317069e856 100644 --- a/docs/guides/modules/test/pages/adaptive-testing.adoc +++ b/docs/guides/modules/test/pages/adaptive-testing.adoc @@ -268,7 +268,7 @@ The two most common causes for this: * The tests were run with a different job name, in this case, rerunning the job should find timing data. * The `<< outputs.junit >>` template variable is not set up correctly. Ensure that the run command uses the template variable and the `store_test_results` step provides a path to a directory so that all batches of `<< outputs.junit >>` are stored. -If the tests are still slower, the test runner being used might have initial start up time when running tests, this can cause significant slow down using the dynamic batching as each batch needs to do that initial start up. +If the tests are still slower, the test runner being used might have initial start up time when running tests. Test runner start up time can cause significant slow down using the dynamic batching as each batch needs to do that initial start up. Add the `dynamic-batching: false` option to `.circleci/test-suites.yml` to disable dynamic batching. @@ -294,7 +294,11 @@ The goal of this section is to enable adaptive testing for your test suite. === 2.1 Update the test suites file -When using adaptive testing for test impact analysis, the `discover` command discovers all tests in a test suite, the `run` command runs only impacted tests and a new command, the `analysis` command, analyzes each test impacted. +When using adaptive testing for test impact analysis the following commands are used: + +* The `discover` command discovers all tests in a test suite. +* The `run` command runs only impacted tests and a new command. +* The `analysis` command analyzes each test impacted. . Update the `.circleci/test-suites.yml` file to include a stubbed analysis command. . Update the `.circleci/test-suites.yml` file to include the option `adaptive-testing: true`. @@ -328,7 +332,7 @@ Supported coverage template variables: * `<< outputs.lcov >>`: Coverage data in LCOV format. * `<< outputs.go-coverage >>`: Coverage data in Go coverage format. -The coverage location does not need to be set in the outputs map, a temporary file will be created and used during analysis with the template variable from the analysis command. +The coverage location does not need to be set in the outputs map. A temporary file will be created and used during analysis with the template variable from the analysis command. . Update your `.circleci/test-suites.yml` file with the analysis command. @@ -859,4 +863,4 @@ Adaptive testing is runner-agnostic. We provide default configurations for the f * Cypress (E2E testing) * Vitest -The key requirement is that your test runner can generate coverage data in a parsable format (currently, we suppport LCOV and Go's "legacy coverage" format). +The key requirement is that your test runner can generate coverage data in a parsable format (currently, we support LCOV and Go's "legacy coverage" format). From db7396c109a0e8267129b36ec3a2fd87389a80ff Mon Sep 17 00:00:00 2001 From: rosie yohannan Date: Mon, 17 Nov 2025 11:32:59 +0000 Subject: [PATCH 15/16] remove line that is no longer required --- docs/guides/modules/test/pages/adaptive-testing.adoc | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/guides/modules/test/pages/adaptive-testing.adoc b/docs/guides/modules/test/pages/adaptive-testing.adoc index 317069e856..c92498583a 100644 --- a/docs/guides/modules/test/pages/adaptive-testing.adoc +++ b/docs/guides/modules/test/pages/adaptive-testing.adoc @@ -759,8 +759,6 @@ Yes, you can fully customize commands by defining `discover`, `run`, and `analys . Use the correct template variables (`<< test.atoms >>`, `<< outputs.junit >>`, `<< outputs.lcov >>`) . Output test results in a format CircleCI can parse (typically JUnit XML) -See the "Custom Configuration" section for detailed examples. - === What happens if no tests are impacted by a change? When test selection determines that no existing tests are affected by your changes, the system will run all tests as a safety measure. This ensures: From db4952800913409a8527f6fd7b12278bbf1ad44e Mon Sep 17 00:00:00 2001 From: rosie yohannan Date: Mon, 17 Nov 2025 11:45:44 +0000 Subject: [PATCH 16/16] style fixes --- .../modules/test/pages/adaptive-testing.adoc | 43 +++++++++++-------- 1 file changed, 25 insertions(+), 18 deletions(-) diff --git a/docs/guides/modules/test/pages/adaptive-testing.adoc b/docs/guides/modules/test/pages/adaptive-testing.adoc index c92498583a..a9ae77e951 100644 --- a/docs/guides/modules/test/pages/adaptive-testing.adoc +++ b/docs/guides/modules/test/pages/adaptive-testing.adoc @@ -16,7 +16,9 @@ Use adaptive testing to optimize test runs as follows: Adaptive testing reduces test execution time while maintaining test confidence. -== Use cases for adaptive testing +== Is my project a good fit for adaptive testing? + +The following list shows some examples of where adaptive testing can be most beneficial: * Unit and integration tests that exercise code within the same repository. * Projects with comprehensive test coverage. The more thorough your tests, the more precisely adaptive testing can identify which tests are impacted by changes. @@ -38,7 +40,12 @@ TIP: In codebases with sparse test coverage, adaptive testing cannot accurately * Scale efficiently as test suites grow. == How it works -The adaptive testing feature operates through two main components that work together to optimize your test execution: +Adaptive testing operates through two main components that work together to optimize your test execution: + +* Dynamic test splitting +* Test impact analysis + +Each component is described in more detail below. === Dynamic test splitting Dynamic test splitting distributes your tests across parallel execution nodes. The system maintains a shared queue that each node pulls from to create a balanced workload. @@ -737,11 +744,11 @@ The frequency depends on your test execution speed and development pace: *Consider re-running analysis:* -. After major refactoring or code restructuring -. When test selection seems inaccurate or outdated -. After adding significant new code or tests +* After major refactoring or code restructuring +* When test selection seems inaccurate or outdated +* After adding significant new code or tests -*Remember:* You can customize which branches run analysis through your CircleCI configuration - it doesn't have to be limited to the main branch. +*Remember:* You can customize which branches run analysis through your CircleCI configuration - it does not have to be limited to the main branch. === Can I customize the test-suites.yml commands? @@ -754,10 +761,10 @@ Yes, you can fully customize commands by defining `discover`, `run`, and `analys *Requirements when customizing:* -. Ensure your commands properly handle test execution -. Generate valid coverage data for the analysis phase -. Use the correct template variables (`<< test.atoms >>`, `<< outputs.junit >>`, `<< outputs.lcov >>`) -. Output test results in a format CircleCI can parse (typically JUnit XML) +* Ensure your commands properly handle test execution. +* Generate valid coverage data for the analysis phase. +* Use the correct template variables (`<< test.atoms >>`, `<< outputs.junit >>`, `<< outputs.lcov >>`). +* Output test results in a format CircleCI can parse (typically JUnit XML). === What happens if no tests are impacted by a change? @@ -792,12 +799,12 @@ You can also compare: === Can I run analysis on branches other than main? -Yes! The branch behavior is fully customizable through your CircleCI configuration. While analysis typically runs on `main` by default, you can configure it to run on: +Yes! The branch behavior is fully customizable through your CircleCI configuration. While analysis typically runs on `main` by default, you can configure it to run on any of the following: -. Any specific branch (for example, `develop` or `staging`). -. Multiple branches simultaneously. -. Feature branches if needed for testing. -. Scheduled pipelines independent of branch. +* Any specific branch (for example, `develop` or `staging`). +* Multiple branches simultaneously. +* Feature branches if needed for testing. +* Scheduled pipelines independent of branch. See the <> example for an example of customizing branch behavior. @@ -805,9 +812,9 @@ See the <> example for an example of customizing branch behavior.