fix(networking): bound in-flight retrying requests in distributed_service - #26263
Draft
samvallad33 wants to merge 1 commit into
Draft
fix(networking): bound in-flight retrying requests in distributed_service#26263samvallad33 wants to merge 1 commit into
samvallad33 wants to merge 1 commit into
Conversation
…vice The single service path puts the concurrency limit outside retry, so a request holds its permit across every attempt and every backoff sleep and poll_ready goes pending once the permits are gone. distributed_service has no limit outside retry. Its only limiter is the per endpoint AdaptiveConcurrencyLimit, which sits inside retry and drops its permit as soon as one attempt resolves, so nothing counts a request that is parked between attempts. During a sustained failure the caller keeps seeing a ready service and retrying requests accumulate without bound. Add a ConcurrencyLimit between the rate limit and the retry, sized at the per endpoint concurrency ceiling times the number of endpoints. That is the most the inner limiters can ever admit at once, so it cannot become the binding constraint on a healthy pipeline. Add a backpressure test module that counts how many requests the service accepts before poll_ready stops returning ready. Against a sustained 429, which the Elasticsearch retry logic retries forever and the health logic never classifies, the service admitted 500 requests before this change, which was the harness cap rather than a settling point.
github-actions
Bot
added
the
domain: sinks
Contributor
|
Thank you for your contribution! Before we can merge this PR, please sign our Contributor License Agreement. To sign, copy and post the phrase below as a new comment on this PR.
I have read the CLA Document and I hereby sign the CLA You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #25212.
The single-service path and the distributed-service path put the concurrency limit on opposite sides of retry, and only one of them creates backpressure. On the single path the limit sits outside retry. A request holds its permit for the whole retry sequence, including every backoff, so when permits run out the service stops accepting work and backpressure reaches the source.
The distributed path has no limit outside retry at all. Its only limit is the per-endpoint AdaptiveConcurrencyLimit, which sits inside retry, and that permit is released as soon as one attempt resolves. So nothing is counting a request that is parked between attempts. The caller keeps seeing a ready service, keeps handing over batches, and retrying futures pile up with no ceiling.
This adds a bound outside retry on the distributed path, so a request in backoff still holds a slot. Both paths now behave the same way under sustained failure.