Skip to content

Reduce integration test CI time: fewer restarts, one shared database - #2620

Merged
Hazel-Datastax merged 3 commits into
mainfrom
hazel/it-ci-optimization
Oct 7, 2026
Merged

Hazel-Datastax merged 3 commits into
mainfrom
hazel/it-ci-optimization

Conversation

@Hazel-Datastax

@Hazel-Datastax Hazel-Datastax commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

What this PR does:
This PR decreases the time of the integration test jobs. It makes fewer restarts, and most restarts keep the database container. It changes only test code and pom.xml.

Changes

1. Class order: TestResourceGroupingClassOrderer

  • A new JUnit ClassOrderer sorts the integration test classes:
    1. The classes that only use DseTestResource.
    2. The other classes, grouped by their test resources.
  • pom.xml sets the orderer with quarkus.test.class-orderer in the failsafe configuration. Surefire (unit tests) does not use it.
  • QuarkusTestProfileAwareClassOrderer calls this orderer first and keeps its order for integration tests.
  • The orderer finds the test resources on the class, its superclasses and its enclosing classes, as Quarkus does. It ignores global resources, because they apply to all classes.
  • Result: 10 starts instead of 16. There is 1 start for the shared resources and 1 start for each of the 9 other classes.

2. Shared database container: StargateTestResource

  • All test classes in the test JVM use one database container. A static field keeps it.
  • stop() does not stop the shared container. Testcontainers removes it when the JVM stops.
  • At each restart, the resource drops the keyspaces that earlier classes left. It keeps only the keyspaces that the new container had. The app starts on a clean database, as before.
  • A resource can override needsDedicatedContainer() to get a container of its own. These test classes use this:
    • CreateCollectionTooManyIndexesIntegrationTest and CreateCollectionTooManyTablesIntegrationTest. Their limits count all collections and indexes in the database.
    • SessionEvictionIntegrationTest. The test stops and starts the database container.
  • When the app runs in a container (containerNetworkId is set), the behavior does not change. Each start of the app gets a new network, so the resource starts a new database container each time. CI does not use this mode.
  • Result: 4 database starts instead of 16. The other 6 restarts only restart the app.

Results

hcd-it before hcd-it after dse69-it before dse69-it after
Job duration 33.1 min 24.4 min 30.2 min 24.8 min
App starts 16 10 16 10
New database containers 16 4 16 4
Stop and database start (total) 768 s 243 s 479 s 194 s
App start (total) 61 s 47 s 96 s 65 s
Tests (skipped) 1532 (4) 1532 (4) 1524 (66) 1524 (66)
  • hcd-it is 8.7 min faster (−26%). dse69-it is 5.4 min faster (−18%).
  • Compared with the mean of the last 20 successful runs on main, hcd-it decreased from 32.8 to 24.4 min (−26%), and dse69-it from 27.6 to 24.8 min (−10%). 19 of these 20 runs are before test: cover successful findAndRerank through the Nvidia HTTP client #2605. They have fewer tests and only 12 starts.
  • A restart that keeps the database takes 10–11 s on HCD and 13–15 s on DSE, with the cleanup. Before, a restart took approximately 52 s on HCD and 36 s on DSE.
  • Job durations change much from run to run. On main, hcd-it took 30.1–39.7 min in the last 20 runs. The number of starts and the restart times are more stable.

How to check

In the log of an integration test job:

  • Started a new database container occurs 4 times.
  • Reusing the shared database container occurs 6 times.
  • The 78 classes that only use DseTestResource run first. The 9 other classes run after them.

Which issue(s) this PR fixes:
Fixes #2621

Checklist

  • Changes manually tested
  • Automated Tests added/updated
  • Documentation added/updated
  • CLA Signed: DataStax CLA

@Hazel-Datastax
Hazel-Datastax requested a review from a team as a code owner October 7, 2026 04:52
@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

➡️ Unit Test Coverage Delta vs Main Branch

Metric Value
Main Branch 54.71%
This PR 54.71%
Delta ⚪ 0.00%
ℹ️ Coverage unchanged

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Unit Test Coverage Report

Overall Project 54.71% 🍏

There is no coverage information present for the Files changed

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

➡️ Integration Test Coverage Delta vs Main Branch (hcd-it)

Metric Value
Main Branch 74.27%
This PR 74.27%
Delta ⚪ 0.00%
ℹ️ Coverage unchanged

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Integration Test Coverage Report (hcd-it)

Overall Project 74.27% 🍏

There is no coverage information present for the Files changed

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

📉 Integration Test Coverage Delta vs Main Branch (dse69-it)

Metric Value
Main Branch 72.98%
This PR 72.95%
Delta 🔴 -0.04%
⚠️ Coverage decreased

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Integration Test Coverage Report (dse69-it)

Overall Project 72.95% 🍏

There is no coverage information present for the Files changed

@Hazel-Datastax Hazel-Datastax changed the title Optimize CI Reduce integration test CI time: fewer restarts, one shared database Oct 7, 2026

@erichare erichare left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM @Hazel-Datastax ! I left one comment about how we handle the drop keyspaces, but otherwise this looks good and seems to be a huge improvement

@Hazel-Datastax
Hazel-Datastax merged commit 8dff175 into main Oct 7, 2026
3 checks passed
@Hazel-Datastax
Hazel-Datastax deleted the hazel/it-ci-optimization branch October 7, 2026 21:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Integration test jobs spend 30–40% of their time on restarts

2 participants