Add metadata hedging for the Python Cosmos SDK (port of Azure/azure-cosmos-dotnet-v3#5999)

[NaluTripician]

Summary

Ports metadata cache cross-region hedging from the .NET SDK (Azure/azure-cosmos-dotnet-v3#5999) to the Python Cosmos SDK. When a metadata read on the primary region is slow, the SDK dispatches a single hedged request to another region after a short threshold and returns the first acceptable answer, so a single slow region no longer stalls metadata cache population or refresh.

Two metadata reads are hedged: Collection Read (container properties) and PartitionKeyRange ReadFeed (first change-feed page only), covering both initial cache population and warm-path cache refresh. Both the sync and async stacks are covered. The primary is always authoritative — a hedge can only win by being faster and producing a good answer; a misconfigured secondary can never surface a spurious result.

Public API

New keyword on CosmosClient (sync + async), additive and non-breaking:

CosmosClient(url, credential, *, enable_metadata_hedging=None)

• None (default) → follows the account's PPAF (Per-Partition Automatic Failover) state
• True → hedges even when PPAF is off
• False → disabled

The AZURE_COSMOS_METADATA_HEDGING_ENABLED environment variable is an operator kill-switch that overrides the keyword and PPAF state (parity with .NET).

Behavior details

• Threshold: fixed 1.5 s (bounded below the control-plane DBAReadTimeout of 3 s).
• Regional-failure classification (hedge-worthy): transport errors, 503, 500, 403+DatabaseAccountNotFound, and 410+LeaseNotFound. Auth failures (401/plain 403) are not regional.
• PartitionKeyRange continuation integrity: only the first change-feed page is hedged; pages 2..N are pinned to the winning region (via excludedLocations) when a hedge wins page 1.
• Bounded cost: at most one extra in-flight request per eligible read, capped by a per-client concurrency budget; loser is cooperatively cancelled.
• Telemetry surfacing (HedgeFired / HedgeWon / WinningRegion) is a tracked fast-follow pending the cross-region diagnostics surface.

Tests

• Reuses the existing metadata-hedging unit tests plus new regression tests in test_metadata_hedging_gaps.py / test_metadata_hedging_gaps_async.py covering: 410+LeaseNotFound classification, the threshold-safety invariant, the env-var kill-switch, and first-page continuation pinning.
• Full sync + async metadata-hedging and routing-map suites pass; pylint clean for the changed files.

Notes

Reference implementation: Azure/azure-cosmos-dotnet-v3#5999. (A couple of module docstrings that cited the wrong number — Azure/azure-cosmos-dotnet-v3#5923 — were corrected to #5999.)

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[Copilot]

Pull request overview

Ports cold-start metadata-cache cross-region hedging from the .NET Cosmos SDK (Azure/azure-cosmos-dotnet-v3#5999) to the Python SDK. When a metadata read (Collection read or the first PartitionKeyRange change-feed page) is slow on the primary region, the SDK dispatches one hedged request to a second region after a fixed 1.5 s threshold and returns the first acceptable answer, while keeping the primary authoritative. A new additive enable_metadata_hedging_for_cold_start client keyword (tri-state) plus the AZURE_COSMOS_METADATA_HEDGING_ENABLED env kill-switch control the feature, for both sync and async stacks.

Changes:

• New sync/async hedging handlers (_metadata_hedging.py) with bounded concurrency, regional-failure classification, and shared winner-recording logic in the strategy mixin.
• Wiring through client → connection → request dispatch, gated on a new is_metadata_cache_population request flag; PartitionKeyRange drains hedge only page 1 and pin subsequent pages to the winning region for continuation integrity.
• New config (thresholds, env var, tri-state resolver), a LEASE_NOT_FOUND substatus, CHANGELOG entry, and comprehensive sync/async unit + regression tests.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
azure/cosmos/_metadata_hedging.py	New sync handler; executor created here is never shut down on client close.
azure/cosmos/aio/_metadata_hedging.py	New async handler (asyncio tasks, proper cancel/gather).
azure/cosmos/_availability_strategy_config.py	Metadata strategy, env-var kill-switch, tri-state opt-in resolver.
azure/cosmos/_availability_strategy_handler_base.py	Shared _record_winner for continuation pinning.
azure/cosmos/_request_object.py	New metadataCachePopulation internal option + flag.
azure/cosmos/_synchronized_request.py	Sync dispatch + applicability gate.
azure/cosmos/aio/_asynchronous_request.py	Async dispatch + applicability gate (missing a blank line before AsynchronousRequest).
azure/cosmos/_routing/routing_map_provider.py	Sync first-page hedging + later-page region pinning.
azure/cosmos/_routing/aio/routing_map_provider.py	Async first-page hedging + later-page region pinning.
azure/cosmos/_cosmos_client_connection.py	Sync connection wiring; creates handler, flags cache-population reads.
azure/cosmos/aio/_cosmos_client_connection_async.py	Async connection wiring.
azure/cosmos/cosmos_client.py	Sync client keyword + docstring.
azure/cosmos/aio/_cosmos_client.py	Async client keyword + docstring.
azure/cosmos/http_constants.py	Adds LEASE_NOT_FOUND = 1022 (verified correct).
CHANGELOG.md	Feature entry under 4.16.2 (Unreleased).
tests/test_metadata_hedging*.py (+ _gaps, async)	Sync/async unit and regression tests.

[NaluTripician]

Good catch. Added MetadataCrossRegionHedgingHandler.close() (idempotent, calls self._executor.shutdown(wait=False, cancel_futures=True)) and wired it into CosmosClient.__exit__ alongside the routing-map release, so a disposed client no longer leaks its hedge worker threads. Added a regression test. Fixed in 84d0b20.

[NaluTripician]

Self-review pass (parity check vs .NET #5999)

Did a local skeptic-lens review of this PR against the .NET reference (MetadataHedgingStrategy.cs / ResolveWinnerAsync) and pushed one correctness fix (3156640).

Issue found & fixed — the primary was not fully authoritative. The winner resolution accepted a hedge branch for any non-regional, non-auth outcome, so a hedge could:

• win with a definitive error (e.g. a stale-region 404) over a slower primary that would have returned a good answer, and
• override a primary that had already produced a definitive answer.

That violates the .NET invariant: a hedge may win ONLY by settling first with a 2xx success, and even then never over a primary that already settled with a non-regional outcome.

Fix: reworked both the sync and async handlers into a single-exit _resolve_winner helper that classifies each settled branch as SUCCESS / REGIONAL_FAILURE / DEFINITIVE and resolves the winner exactly as .NET's ResolveWinnerAsync does:

• hedge first + success → hedge wins unless the primary already settled non-regional;
• hedge first + non-success → wait for and return the primary's outcome;
• primary first + non-regional → primary wins;
• primary first + regional failure → a successful hedge may win, else return the primary's regional failure.

Tests: added sync + async regression tests in test_metadata_hedging_gaps* (a hedge definitive error must not win over a slow primary success, must defer to a primary definitive error, and must not override a primary regional failure). 5 of them fail against the pre-fix code. Full metadata-hedging + routing-map-provider unit suites pass; pylint clean on the changed modules.

Reviewed and intentionally left as-is: async budget locked()+acquire() (safe — no await between them in single-threaded asyncio); the gateway disableCrossRegionalHedging operator flag (Python uses the AZURE_COSMOS_METADATA_HEDGING_ENABLED env kill-switch); telemetry surfacing (tracked fast-follow).

[NaluTripician]

CI fixes + review comments addressed (84d0b20)

CI failures

• consistency — regenerated api.md + api.metadata.yml with apiview-stub-generator 0.3.28 (matches the committed parserVersion). The PR added public surface that was missing from the snapshot: the async CosmosClient enable_metadata_hedging keyword, SubStatusCodes.LEASE_NOT_FOUND, and the CosmosClientConnection keyword. Diff is exactly those 4 additions.
• Analyze / Run Pylint — two real pylint findings (the local plugin masked them earlier): too-many-locals on CosmosClientConnection.__init__ (the new keyword pushed it 25→26) and do-not-import-asyncio on the from asyncio import ... line in aio/_metadata_hedging.py. Both waived. mypy was already clean.
• cspell — added PPAF to the cosmos word list in .vscode/cspell.json and reworded a falsey comment.

Review comments

• Executor leak — added MetadataCrossRegionHedgingHandler.close() (idempotent) and invoke it from CosmosClient.__exit__; added a regression test.
• Blank line — added the missing second blank line before AsynchronousRequest.

Emulator test failures are pre-existing flakiness, not from this PR. Different cross-region retry / fault-injection tests fail on different Python versions — test_service_retry_policies on 3.10, test_timeout_and_failover_retry_policy[502] on 3.13 — while 3.11/3.12/3.14 pass. A deterministic regression would fail the same tests on every version. These tests exercise document-plane retry, and metadata hedging cannot fire under emulator defaults (no opt-in + PPAF off), so the hedging path is never taken. They should pass on the re-run.

Local validation: 46 metadata-hedging unit tests pass; pylint + mypy clean on the changed modules; cspell clean; api.md diff verified byte-for-byte against a fresh 0.3.28 regeneration.