fix(aws_s3 source): bound S3 GetObject with default client timeouts#25867
Draft
taylorchandleryoung wants to merge 1 commit into
Draft
fix(aws_s3 source): bound S3 GetObject with default client timeouts#25867taylorchandleryoung wants to merge 1 commit into
taylorchandleryoung wants to merge 1 commit into
Conversation
The aws_s3 source created its S3 client with no read or operation timeout (only the SDK-default ~3.1s connect timeout). A half-open or silently dropped connection -- for example one reaped by a NAT gateway idle timeout -- could therefore hang a polling task indefinitely on the GetObject response-body read, stopping SQS polling entirely and stalling S3-based ingestion until the process was restarted. Retries are disabled on this client, so there was no recovery. Apply default S3 client timeouts (connect_timeout_seconds=5, read_timeout_seconds=30) and expose them as source configuration. The read timeout is applied per-read, so a slow but steadily progressing transfer of a large object is not cut off; no operation timeout is set by default to avoid capping legitimately long transfers. This mirrors the existing SQS client timeout support on the same source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
taylorchandleryoung
force-pushed
the
taylor.c.young/aws-s3-read-timeout
branch
from
July 16, 2026 14:20
4386c42 to
212946f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
aws_s3source builds its S3 client with no read or operation timeout (only the AWS SDK-default ~3.1s connect timeout), and retries are disabled on the client. A half-open or silently dropped connection — for example one reaped by a NAT gateway idle timeout — therefore hangs theGetObjectresponse-body read indefinitely. Because the source's polling loop processes each SQS message inline, one hungGetObjectparks the polling task; once allclient_concurrencytasks park this way, SQS polling stops entirely and S3 ingestion stalls until the process is restarted.This was observed in production: with debug logging on, healthy requests show
and after some hours of uptime the source goes completely silent (no further
get_object/reuse idle connectionactivity) while the rest of the process keeps running — the classic stale-pooled-connection hang.Change
AwsTimeout::new(connect, operation, read)constructor.timeout(AwsTimeout) config block to theaws_s3source, mirroring the existing SQS client timeout support on the same source.connect_timeout_seconds = 5andread_timeout_seconds = 30when a dimension is unset (src/sources/aws_s3/mod.rs), instead of passingNone. The read timeout is applied per-read, so a slow but steadily progressing transfer of a large object is not cut off; no operation timeout is set by default to avoid capping legitimately long transfers. All three dimensions can be overridden viaconnect_timeout_seconds/read_timeout_seconds/operation_timeout_seconds.Testing
rustfmt --edition 2024 --checkclean on both edited files.TODO before marking ready
master.make generate-component-docs) sowebsite/cue/reference/components/sources/base/aws_s3.cuereflects the new options.🤖 Generated with Claude Code