Skip to content

net-gateway: split reserved DNS 50/50 between Google and Cloudflare - #897

Open
nikhilbirmiwal-pplx wants to merge 1 commit into
mainfrom
nikhilb/reserved-dns-split-providers
Open

nikhilbirmiwal-pplx wants to merge 1 commit into
mainfrom
nikhilb/reserved-dns-split-providers

Conversation

@nikhilbirmiwal-pplx

@nikhilbirmiwal-pplx nikhilbirmiwal-pplx commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Why

Guest images write nameserver 8.8.8.8 then nameserver 1.1.1.1. glibc keeps querying the first nameserver until it times out, so nearly all reserved DNS goes to Google, and Cloudflare only sees the retries after a timeout. During incident #866 (brain high exit error rate), Google was dropping pipelined bursts and Cloudflare was mostly idle.

What

The gateway now picks the upstream provider per sandbox instead of always using the address the guest queried:

  • A stable hash of the sandbox id (FNV-1a with the SplitMix64 finalizer, top bit) puts about half of sandboxes on each provider. Those sandboxes are served by the provider they addressed; the other half are served by the other provider.
  • Fallback still works: the mapping is fixed per sandbox and always maps 8.8.8.8 and 1.1.1.1 to different providers. When the guest retries its second nameserver, it still reaches the other provider.
  • Per sandbox, not per query: the bursts come from individual sandboxes, so this spreads them across both providers. Each sandbox's lookups also stay on one provider's cache and connection.
  • Applies everywhere right away: this is gateway-only, so no guest image rebuild or options rotate is needed. It covers existing templates too.
  • Metrics: the resolver tag on reserved-DNS metrics now reports the upstream actually used, not the address the guest queried.

Tests

  • sandboxes_split_evenly_across_upstream_providers: 10k sequential ids split 4,800–5,200 (actual: 4,985). Random UUIDs split about 50.2%.
  • each_sandbox_keeps_a_fixed_provider_per_address_and_falls_back_to_the_other
  • query_to_8_8_8_8_is_served_by_the_sandboxes_assigned_provider
  • Existing tests that check the resolver IP now use a sandbox id that keeps the provider it addressed.

cargo test -p pplx-space-net-gateway --lib: 252 pass. The one failure is interception_exemptions_relay_original_tls_only_after_admission, which also fails on unmodified main in this environment. cargo clippy --all-targets -D warnings is clean.

Interactions

This is independent of #892, #893, #894 and #895. With #895 (coalescing), queries share a lookup per upstream, so mapping before the lookup is correct. #893 (per-resolver slot budgets) may need a small rebase. If its budgets are keyed on the address the guest queried, they should be keyed on upstream_for(...) instead.

Guests list 8.8.8.8 before 1.1.1.1, so glibc sends almost every first
query to Google. Pick the upstream per sandbox with a stable hash of
the sandbox id: half the sandboxes are served by the provider they
addressed, half by the other one. The mapping is fixed per sandbox, so
the guest's retry to its second nameserver still reaches the other
provider.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant