Repository navigation
net-gateway: split reserved DNS 50/50 between Google and Cloudflare - #897
Open
nikhilbirmiwal-pplx wants to merge 1 commit into
Open
nikhilbirmiwal-pplx wants to merge 1 commit into
nikhilbirmiwal-pplx wants to merge 1 commit into
Conversation
Guests list 8.8.8.8 before 1.1.1.1, so glibc sends almost every first query to Google. Pick the upstream per sandbox with a stable hash of the sandbox id: half the sandboxes are served by the provider they addressed, half by the other one. The mapping is fixed per sandbox, so the guest's retry to its second nameserver still reaches the other provider.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Guest images write
nameserver 8.8.8.8thennameserver 1.1.1.1. glibc keeps querying the first nameserver until it times out, so nearly all reserved DNS goes to Google, and Cloudflare only sees the retries after a timeout. During incident #866 (brain high exit error rate), Google was dropping pipelined bursts and Cloudflare was mostly idle.What
The gateway now picks the upstream provider per sandbox instead of always using the address the guest queried:
options rotateis needed. It covers existing templates too.resolvertag on reserved-DNS metrics now reports the upstream actually used, not the address the guest queried.Tests
sandboxes_split_evenly_across_upstream_providers: 10k sequential ids split 4,800–5,200 (actual: 4,985). Random UUIDs split about 50.2%.each_sandbox_keeps_a_fixed_provider_per_address_and_falls_back_to_the_otherquery_to_8_8_8_8_is_served_by_the_sandboxes_assigned_providercargo test -p pplx-space-net-gateway --lib: 252 pass. The one failure isinterception_exemptions_relay_original_tls_only_after_admission, which also fails on unmodified main in this environment.cargo clippy --all-targets -D warningsis clean.Interactions
This is independent of #892, #893, #894 and #895. With #895 (coalescing), queries share a lookup per upstream, so mapping before the lookup is correct. #893 (per-resolver slot budgets) may need a small rebase. If its budgets are keyed on the address the guest queried, they should be keyed on
upstream_for(...)instead.