[DBMON-6882] Add cancellation lifecycle to DatabaseCheck

[eric-weaver]

What does this PR do?

Moves the clean shutdown protocol that Postgres has been carrying into DatabaseCheck, so
every DBM integration can get thread-safe cancellation without reimplementing it. The protocol
was originally added to Postgres in #23728 (deferred teardown when cancel lands mid-run) and
#23640 (dropping dangling references so the check can be reclaimed), and moved onto the shared
async job registry in #24824.

DatabaseCheck gains:

• run() — records whether a run is in flight, returns an empty error report without running
  the check once cancelled, and performs the deferred teardown after check() returns.
• cancel() — signals the registered async jobs and records the cancellation, but does no
  destructive work itself. Teardown is deferred to _finalize(), which runs inline when the
  check is idle and in run() otherwise.
• _finalize() — stops the async jobs, calls shutdown(), then drops the state that keeps
  the check alive. Runs at most once.
• shutdown() — a no-op hook for integrations to release resources they hold for their whole
  lifetime, such as connections, pools and clients. Called once during teardown, after the
  jobs have stopped and never while check() is running.
• is_cancelled — public accessor so check() bodies and collection loops can stop promptly.

run_async_jobs() now no-ops once the check is cancelled, so a run that was in flight when
the cancel arrived does not restart the loops cancel_async_jobs() just stopped.

The implementation carries three fixes over the Postgres original, which Postgres will pick
up when it adopts this:

• _finalize() is guarded so teardown runs at most once. Double cancellation already happens
  in practice (Postgres's run_one_check helper cancels, and the integration_check fixture
  cancels again in teardown) and currently only survives because each individual step
  tolerates it.
• self.log.check = None moved after the last log statement. CheckLoggingAdapter.process
  re-reads self.check.check_id on every call for checks whose check_id never resolved, so
  nulling it before logging raises AttributeError when debug logging is enabled.
• _finalize() clears the async job registry once the jobs have stopped, breaking the
  check -> registry -> job -> check cycle at the base level rather than relying on every job's
  shutdown() to null its own back-reference.

Motivation

Postgres is the only DBM integration with a real shutdown protocol, and the reason is
specific: the Agent can call cancel() from another thread while check() is mid-flight, and
closing a libpq connection at that moment segfaults the Agent rather than raising. MySQL, SQL
Server and ClickHouse only signal their jobs, and ClickHouse additionally blocks on job
futures inside cancel(), which the AgentCheck.cancel() contract warns against. Lifting the
protocol into the base class gives every integration the safe version and lets Postgres delete
its copy, which is what the TODO: move this lock into the base class added in #23728 asks
for.

This is additive and inert until an integration adopts it. Postgres keeps its own run(),
cancel() and _finalize(), which shadow the base versions, so its behavior is unchanged.
MySQL, SQL Server and ClickHouse all override cancel() and none calls super().cancel(), so
the new cancel() is unreachable for them. They do inherit the new run(), but its only
active effect is one uncontended lock acquisition per run: the deferred-finalize branch
requires _cancelled, which only the base cancel() sets.

DatabaseCheck is the right home for now, but probably not forever. Nothing in the state
machine is DBM-specific — only the shutdown_async_jobs() call is — so if we want non-DBM
integrations to adopt the same protocol, AgentCheck is the natural long-term home. Landing
it here first keeps the blast radius to the four DBM checks while the pattern proves out, and
promoting it later would be a move rather than a rewrite. The existing
TODO: move diagnosis cleanup into AgentCheck.cancel() in Postgres points the same direction.

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
• If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

Made with Cursor

[datadog-datadog-prod-us1]

 

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 97.22%
• Overall Coverage: 89.59% (+1.10%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 17348ab | Docs | Datadog PR Page | Give us feedback!}

[cit-pr-commenter-54b7da]

evalya-impact-summary

evalya impact analysis

Impact analysis: 0 selected, 0 skipped (of 0 test tasks)
Publish tasks:   0 (always emitted)
Diff (3 files):
  datadog_checks_base/changelog.d/24844.added
  datadog_checks_base/datadog_checks/base/checks/db.py
  datadog_checks_base/tests/base/checks/test_database_check.py

Debug a specific task: evalya plan impact --path <path> --task <task>

Learn more about CI impact filtering

[dd-octo-sts]

Validation Report

All 21 validations passed.

Show details

Validation	Description	Status
agent-reqs	Verify check versions match the Agent requirements file	✅
ci	Validate CI configuration and code coverage settings	✅
codeowners	Validate every integration has a CODEOWNERS entry	✅
config	Validate default configuration files against spec.yaml	✅
dep	Verify dependency pins are consistent and Agent-compatible	✅
http	Validate integrations use the HTTP wrapper correctly	✅
imports	Validate check imports do not use deprecated modules	✅
integration-style	Validate check code style conventions	✅
jmx-metrics	Validate JMX metrics definition files and config	✅
labeler	Validate PR labeler config matches integration directories	✅
legacy-signature	Validate no integration uses the legacy Agent check signature	✅
license-headers	Validate Python files have proper license headers	✅
licenses	Validate third-party license attribution list	✅
metadata	Validate metadata.csv metric definitions	✅
models	Validate configuration data models match spec.yaml	✅
openmetrics	Validate OpenMetrics integrations disable the metric limit	✅
package	Validate Python package metadata and naming	✅
qa-label	Validate the pull request declares whether it needs QA for the next Agent release	✅
readmes	Validate README files have required sections	✅
saved-views	Validate saved view JSON file structure and fields	✅
version	Validate version consistency between package and changelog	✅

View full run

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 17348abd81

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Avoid double-finalizing Postgres cancellation

When a cancel lands while PostgreSql.run() is in flight, this new DatabaseCheck.run() wrapper now runs inside Postgres's existing wrapper because PostgreSql.run() calls super().run(). The inner finally calls self._finalize() once, then control returns to PostgreSql.run()'s own finally, which still sees _cancelled and calls _finalize() a second time. That re-runs Postgres job shutdown/connection cleanup and can hit the existing self.log.check = None logging hazard, so Postgres cancellation regresses until the subclass is migrated or this path bypasses the new wrapper.

Useful? React with 👍 / 👎.