Repository navigation
net-gateway: coalesce concurrent reserved-DNS lookups for the same question - #895
Open
nikhilbirmiwal-pplx wants to merge 1 commit into
Open
nikhilbirmiwal-pplx wants to merge 1 commit into
nikhilbirmiwal-pplx wants to merge 1 commit into
Conversation
…estion
Concurrent guest A queries for the same (reserved resolver, canonical
question) now share one upstream Hickory lookup and one lookup slot via a
single-flight map of futures::Shared. Each waiter still sanitizes the
shared answer against its own admitted policy and observes its own policy
cancellation; the shared lookup is dropped (cancelling upstream I/O and
releasing its slot) once the last waiter goes away.
Adds gateway.reserved_dns.lookup.coalesce{resolver,role=leader|follower}.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Incident #inc-2026-10-22-brain-high-exit-error-rate: reserved-DNS failures are driven by per-sandbox bursts (Dream preload fans out up to ~80 lookups at once, many for the same host). Today every one of those identical queries becomes its own upstream request on the pipelined DoT connection and holds its own lookup slot (64 total). Google has been observed closing connections after ~46 pipelined queries for repeated names, so duplicate in-flight queries directly amplify the failure.
What
Single-flight coalescing in
ReservedDnsCore::resolve_a(newreserved_dns/inflight.rs):(ReservedResolver, canonical Query)(the qname is already lowercased byparse_guest_request). Two concurrentA example.comqueries to 8.8.8.8 share onefutures::Sharedlookup; different resolver / qname / qtype do not coalesce.sanitize_resolver_answeragainst its own admitted policy (test covers one waiter allowed / one address-denied on the same shared answer).select!s on its own policy-cancellation token. Cancelling one waiter detaches it without cancelling the lookup for others. The map holds only aWeakShared, so when the last waiter goes away the shared future is dropped, which cancels the upstream Hickory lookup and releases its slot (same as today's single-waiter cancellation). A drop guard removes the map entry on completion or abandonment (id-checked so it never removes a newer entry).TOTAL_LOOKUP_TIMEOUT; late joiners only wait the remaining time. Nothing is cached beyond the in-flight window (Hickory's own cache is unchanged).ResolverFailureis nowClone(NetErrorisClone) so failures fan out to all waiters.gateway.reserved_dns.lookup.coalesce{resolver, role=leader|follower}to see the dedup ratio in prod.Sharing across sandboxes adds no new exposure: Hickory's resolver cache is already shared across sandboxes, the policy qname check runs before joining, and the answer is sanitized per waiter.
Tests
New in
reserved_dns/tests.rs:concurrent_identical_queries_share_one_upstream_lookup_and_slot— 8 concurrent queries with lookup capacity 1 → 1 upstream call, all succeed, slot released, map empty.coalescing_is_keyed_by_resolver_and_questioncancelling_one_waiter_does_not_cancel_the_shared_lookupabandoned_lookup_is_dropped_and_releases_its_slot— all waiters cancelled → upstream future dropped, next query starts fresh.shared_answer_is_sanitized_against_each_waiters_policycargo test -p pplx-space-net-gateway --lib reserved_dns: 29 passed.cargo clippy -p pplx-space-net-gateway -p pplx-space-metrics --all-targets -D warnings: clean. Rebased onto perplexityai/SPACE main.Independent of #892 (capacity 64 → 256), #893 (per-resolver slot budgets) and #894 (attempt timeout 4s → 500ms); touches different lines, though #893 may need a trivial rebase if it changes how lookup_slots is wired.