adapter: add LaunchDarkly reconnect integration test#37026
Open
jasonhernandez wants to merge 2 commits into
Open
adapter: add LaunchDarkly reconnect integration test#37026jasonhernandez wants to merge 2 commits into
jasonhernandez wants to merge 2 commits into
Conversation
jasonhernandez
force-pushed
the
jason/ld-sync-reconnect-test
branch
from
3a96074 to
006218f
Compare
jasonhernandez
force-pushed
the
jason/ld-sync-reconnect-test
branch
from
006218f to
59a7e8d
Compare
jasonhernandez
force-pushed
the
jason/ld-sync-reconnect-test
branch
2 times, most recently
from
e76f589 to
7fffddc
Compare
jasonhernandez
force-pushed
the
jason/ld-sync-reconnect-test
branch
from
7fffddc to
414a4f7
Compare
def-
reviewed
Jul 2, 2026
def-
left a comment
def-
left a comment
Contributor
There was a problem hiding this comment.
We should actually verify if the test would found the bug.
Contributor
Author
@def- I could stack a PR on this that has an older version of the 3.0 SDK and see if it fails. Would that be a reasonable approach? |
Contributor
|
Sure, works for me! |
Move launchdarkly-server-sdk from the MaterializeInc/rust-server-sdk fork back to upstream crates.io 3.1.1, restoring the launchdarkly-sdk-transport + MetricsTransport setup and dropping the [patch.crates-io] override. The fork existed for launchdarkly/rust-server-sdk#116: a StreamingDataSource /eventsource StreamClosed bug where a non-Eof stream error left the data source stuck with no reconnect, silently breaking LD sync. A prior upgrade to upstream 3.0.1 had to be reverted (incident-984) because that bug was still unfixed upstream. The fixes have since landed, rust-server-sdk#168 and rust-eventsource-client#134/#135, and 3.1.1 resolves eventsource-client to 0.17.5, which carries them. Use the rustls + aws-lc-rs features (hyper-rustls-native-roots, crypto-aws-lc-rs), now the upstream defaults, instead of the prior attempt's native-tls/crypto-openssl, avoiding the OpenSSL path. The transport build_https() call is identical either way. Because the SDK now builds a rustls client, the workspace links both rustls provider features (aws_lc_rs via our features, ring transitively via the hyper-rustls chain). With both enabled rustls cannot select a process-default provider on its own and panics on first client build. Install aws-lc-rs explicitly (idempotently) at the top of the environmentd, clusterd, balancerd, sqllogictest, and testdrive entrypoints. orchestratord already installs it. deny.toml gains skips for the duplicate versions the transport stack pulls (older tower/rustls-native-certs; newer rand/rand_core/getrandom/cpufeatures) and re-adds the launchdarkly-sdk-transport wrapper. Adds MetricsTransport unit tests, including test_metric_frozen_on_midstream_ error, modeling the exact incident-984 failure mode (200 OK then a mid-stream timeout): they assert the last_sse_time_seconds gauge freezes so the staleness alert can detect a stuck data source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
jasonhernandez
force-pushed
the
jason/ld-sync-reconnect-test
branch
from
414a4f7 to
0a4f3d9
Compare
jasonhernandez
force-pushed
the
jason/ld-sync-reconnect-test
branch
from
0a4f3d9 to
6439427
Compare
incident-984 was a runtime failure: the LaunchDarkly data source stopped reconnecting after its streaming connection died, silently wedging flag sync. The existing test/launchdarkly nightly covers value sync, persistence, targeting, and the kill switch, but nothing exercises reconnect after a mid-stream failure. This adds test/launchdarkly-reconnect, which reproduces the incident deterministically against a mock and needs no real LaunchDarkly credentials. The failure mode matters. The incident signature was a mid-body read timeout on a silently-dead connection, hyper::Error(Body, Kind(TimedOut)), which the eventsource client surfaces to the data source as a stream error. A TCP RST or a clean FIN does NOT reproduce it: those are retried inside the eventsource client on every SDK version and never reach the data source (verified empirically, nightly MaterializeInc#17018 stayed green with an RST-based mock even on the exact incident-era dependency stack). The mock serves an initial flag value (2 GiB) on the first TWO streaming connections and then goes silent holding each open, so the transport read timeout ends the stream. Two, because environmentd creates a short-lived bootstrap LD client at boot (load_remote_system_parameters) before the long-lived sync client, and the stall must hit the sync client (verified empirically, nightly MaterializeInc#17020: with only the first connection stalled, the sync client landed on connection two and got the updated value without any reconnect being exercised). Every later (reconnecting) client gets the updated value (3 GiB) plus heartbeats. Production changes, both hidden test knobs: - --launchdarkly-base-uri (env MZ_LAUNCHDARKLY_BASE_URI) overrides the SDK's streaming/polling/events endpoints with a single base URL via the SDK's ServiceEndpointsBuilder::relay_proxy, letting tests point the SDK at the mock. Also generally useful for relay-proxy setups. - MZ_LAUNCHDARKLY_READ_TIMEOUT overrides the transport's streaming read timeout (default 300s) so the test can trigger the timeout path in seconds. mzcompose.py boots environmentd against the mock with a 5s read timeout and asserts SHOW max_result_size reaches 3GB, which can only happen if the sync client's data source reconnected after the timeout. A regressed SDK stays stuck at 2GB and the assertion times out. Wired into the nightly pipeline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
jasonhernandez
force-pushed
the
jason/ld-sync-reconnect-test
branch
from
6439427 to
aaf2f27
Compare
Contributor
Author
@def- @ggevay - this throwaway PR #37460 fails the reconnect test I propose that we merge this PR #37026 if all looks good to both of you. |
def-
approved these changes
Jul 7, 2026
def-
left a comment
def-
left a comment
Contributor
There was a problem hiding this comment.
Thanks for checking, no complaints from QA side
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge this PR to move to LaunchDarkly mainline 3.1.1
Part 2 of 3 in the LaunchDarkly upstream-SDK stack. Stacked on #37025.
What this does
Adds
test/launchdarkly-reconnect, an integration test that reproduces incident-984 deterministically against a mock, with no real LaunchDarkly credentials.The failure mode matters. The incident signature was a mid-body read timeout on a silently-dead connection (
hyper::Error(Body, Kind(TimedOut))), which the eventsource client surfaces to the data source as a stream error. A TCP RST or clean FIN does not reproduce it — those are retried inside the eventsource client on every SDK version and never reach the data source. We verified this empirically: an earlier RST-based version of this mock stayed green (nightly #17018) even on the exact incident-era dependency stack.mock_ld.py— mocks the LD streaming API. The first streaming client gets an initial flag value (2 GiB), then the connection goes silent while held open (no FIN, no RST), so the SDK's transport read timeout ends the stream — the exact incident error class. Every reconnecting client gets an updated value (3 GiB) plus heartbeats.mzcompose.py— boots environmentd against the mock with a 5s read timeout and assertsSHOW max_result_sizereaches3GB, which can only happen if the data source reconnected after the timeout.Production changes, both hidden test knobs:
--launchdarkly-base-uri(envMZ_LAUNCHDARKLY_BASE_URI) — overrides the SDK's endpoints with a single base URL (ServiceEndpointsBuilder::relay_proxy), letting tests point the SDK at the mock.MZ_LAUNCHDARKLY_READ_TIMEOUT— overrides the transport's streaming read timeout (default 300s) so the test can trigger the timeout path in seconds.Against the fixed 3.1.1 SDK (#37025) this should pass green; #37460 proves it goes red on the pre-fix stack.
Stacking note: GitHub can't base a fork PR on another fork branch, so this targets
mainand its diff also contains #37025's migration commit. Review the top commit (adapter: add LaunchDarkly reconnect integration test).