Skip to content

net-gateway: time reserved DNS upstream requests - #894

Open
nikhilbirmiwal-pplx wants to merge 2 commits into
mainfrom
nikhilb/netgw-reserved-dns-upstream-latency-500ms
Open

nikhilbirmiwal-pplx wants to merge 2 commits into
mainfrom
nikhilb/netgw-reserved-dns-upstream-latency-500ms

Conversation

@nikhilbirmiwal-pplx

@nikhilbirmiwal-pplx nikhilbirmiwal-pplx commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

What

New metric: gateway.reserved_dns.upstream.request.duration_ms (distribution, tags resolver, outcome, git_sha). Recorded in LoggingDnsHandle::send: time from handing one query to an upstream DNS-over-TLS connection to its first response or failure, including any wait for a shared reconnect. outcome is answered, timeout / no_response (Hickory's per-attempt timeout), busy, error, or cancelled (send dropped first, recorded via Drop).

No behavior change: ATTEMPT_TIMEOUT stays at 4s. Tuning it is deferred until this metric shows real upstream latency.

Why

The existing gateway.reserved_dns.request.duration_ms is guest-facing end-to-end time. It includes Hickory cache hits (p50 is ~0.1 ms), local AAAA answers and capacity rejections, so it can't show the actual upstream request/response time. The new metric only counts queries that go upstream, which is what a timeout value should be sized against.

Notes

  • The resolver / outcome tags may need to be allowed in Datadog's tag configuration for this metric, as with the other reserved-DNS metrics.

@nikhilbirmiwal-pplx
nikhilbirmiwal-pplx added this pull request to the merge queue Oct 23, 2026
@nikhilbirmiwal-pplx
nikhilbirmiwal-pplx removed this pull request from the merge queue due to a manual request Oct 23, 2026
@nikhilbirmiwal-pplx nikhilbirmiwal-pplx changed the title net-gateway: time reserved DNS upstream requests; cut attempt timeout 4s -> 500ms net-gateway: time reserved DNS upstream requests Oct 23, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants