Skip to content

net-gateway: coalesce concurrent reserved-DNS lookups for the same question - #895

Open
nikhilbirmiwal-pplx wants to merge 1 commit into
mainfrom
nikhilb/netgw-reserved-dns-coalesce-lookups
Open

nikhilbirmiwal-pplx wants to merge 1 commit into
mainfrom
nikhilb/netgw-reserved-dns-coalesce-lookups

Conversation

@nikhilbirmiwal-pplx

Copy link
Copy Markdown
Collaborator

Why

Incident #inc-2026-10-22-brain-high-exit-error-rate: reserved-DNS failures are driven by per-sandbox bursts (Dream preload fans out up to ~80 lookups at once, many for the same host). Today every one of those identical queries becomes its own upstream request on the pipelined DoT connection and holds its own lookup slot (64 total). Google has been observed closing connections after ~46 pipelined queries for repeated names, so duplicate in-flight queries directly amplify the failure.

What

Single-flight coalescing in ReservedDnsCore::resolve_a (new reserved_dns/inflight.rs):

  • Key = (ReservedResolver, canonical Query) (the qname is already lowercased by parse_guest_request). Two concurrent A example.com queries to 8.8.8.8 share one futures::Shared lookup; different resolver / qname / qtype do not coalesce.
  • One lookup slot per shared lookup, acquired inside the shared future, so N identical waiters consume 1 slot instead of N.
  • Per-waiter policy still applies: only the raw upstream answer is shared; each waiter runs sanitize_resolver_answer against its own admitted policy (test covers one waiter allowed / one address-denied on the same shared answer).
  • Per-waiter cancellation: each waiter select!s on its own policy-cancellation token. Cancelling one waiter detaches it without cancelling the lookup for others. The map holds only a WeakShared, so when the last waiter goes away the shared future is dropped, which cancels the upstream Hickory lookup and releases its slot (same as today's single-waiter cancellation). A drop guard removes the map entry on completion or abandonment (id-checked so it never removes a newer entry).
  • Shared lookup is still bounded by TOTAL_LOOKUP_TIMEOUT; late joiners only wait the remaining time. Nothing is cached beyond the in-flight window (Hickory's own cache is unchanged).
  • ResolverFailure is now Clone (NetError is Clone) so failures fan out to all waiters.
  • New metric gateway.reserved_dns.lookup.coalesce{resolver, role=leader|follower} to see the dedup ratio in prod.

Sharing across sandboxes adds no new exposure: Hickory's resolver cache is already shared across sandboxes, the policy qname check runs before joining, and the answer is sanitized per waiter.

Tests

New in reserved_dns/tests.rs:

  • concurrent_identical_queries_share_one_upstream_lookup_and_slot — 8 concurrent queries with lookup capacity 1 → 1 upstream call, all succeed, slot released, map empty.
  • coalescing_is_keyed_by_resolver_and_question
  • cancelling_one_waiter_does_not_cancel_the_shared_lookup
  • abandoned_lookup_is_dropped_and_releases_its_slot — all waiters cancelled → upstream future dropped, next query starts fresh.
  • shared_answer_is_sanitized_against_each_waiters_policy

cargo test -p pplx-space-net-gateway --lib reserved_dns: 29 passed. cargo clippy -p pplx-space-net-gateway -p pplx-space-metrics --all-targets -D warnings: clean. Rebased onto perplexityai/SPACE main.

Independent of #892 (capacity 64 → 256), #893 (per-resolver slot budgets) and #894 (attempt timeout 4s → 500ms); touches different lines, though #893 may need a trivial rebase if it changes how lookup_slots is wired.

…estion

Concurrent guest A queries for the same (reserved resolver, canonical
question) now share one upstream Hickory lookup and one lookup slot via a
single-flight map of futures::Shared. Each waiter still sanitizes the
shared answer against its own admitted policy and observes its own policy
cancellation; the shared lookup is dropped (cancelling upstream I/O and
releasing its slot) once the last waiter goes away.

Adds gateway.reserved_dns.lookup.coalesce{resolver,role=leader|follower}.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant