Skip to content

fix(aws_s3 source): bound S3 GetObject with default client timeouts#25867

Draft
taylorchandleryoung wants to merge 1 commit into
vectordotdev:masterfrom
taylorchandleryoung:taylor.c.young/aws-s3-read-timeout
Draft

fix(aws_s3 source): bound S3 GetObject with default client timeouts#25867
taylorchandleryoung wants to merge 1 commit into
vectordotdev:masterfrom
taylorchandleryoung:taylor.c.young/aws-s3-read-timeout

Conversation

@taylorchandleryoung

@taylorchandleryoung taylorchandleryoung commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Summary

The aws_s3 source builds its S3 client with no read or operation timeout (only the AWS SDK-default ~3.1s connect timeout), and retries are disabled on the client. A half-open or silently dropped connection — for example one reaped by a NAT gateway idle timeout — therefore hangs the GetObject response-body read indefinitely. Because the source's polling loop processes each SQS message inline, one hung GetObject parks the polling task; once all client_concurrency tasks park this way, SQS polling stops entirely and S3 ingestion stalls until the process is restarted.

This was observed in production: with debug logging on, healthy requests show

timeout settings for this operation:
TimeoutConfig { connect_timeout: Set(3.1s), read_timeout: Unset, operation_timeout: Unset, ... }

and after some hours of uptime the source goes completely silent (no further get_object / reuse idle connection activity) while the rest of the process keeps running — the classic stale-pooled-connection hang.

Change

  • Add a small AwsTimeout::new(connect, operation, read) constructor.
  • Add an optional timeout (AwsTimeout) config block to the aws_s3 source, mirroring the existing SQS client timeout support on the same source.
  • Bound the S3 client by default: apply connect_timeout_seconds = 5 and read_timeout_seconds = 30 when a dimension is unset (src/sources/aws_s3/mod.rs), instead of passing None. The read timeout is applied per-read, so a slow but steadily progressing transfer of a large object is not cut off; no operation timeout is set by default to avoid capping legitimately long transfers. All three dimensions can be overridden via connect_timeout_seconds / read_timeout_seconds / operation_timeout_seconds.

Testing

  • rustfmt --edition 2024 --check clean on both edited files.
  • Full compile/tests to be validated by CI (change is small and mirrors the established SQS timeout pattern on this source).

TODO before marking ready

  • Rebase onto current master.
  • Regenerate component docs (make generate-component-docs) so website/cue/reference/components/sources/base/aws_s3.cue reflects the new options.
  • Consider whether defaulting the S3 client's connect/read timeouts (a behavior change) should instead be opt-in; opted for safe-by-default here to fix the hang out of the box.

🤖 Generated with Claude Code

@github-actions github-actions Bot added the domain: sources Anything related to the Vector's sources label Jul 16, 2026
The aws_s3 source created its S3 client with no read or operation timeout
(only the SDK-default ~3.1s connect timeout). A half-open or silently
dropped connection -- for example one reaped by a NAT gateway idle
timeout -- could therefore hang a polling task indefinitely on the
GetObject response-body read, stopping SQS polling entirely and stalling
S3-based ingestion until the process was restarted. Retries are disabled
on this client, so there was no recovery.

Apply default S3 client timeouts (connect_timeout_seconds=5,
read_timeout_seconds=30) and expose them as source configuration. The
read timeout is applied per-read, so a slow but steadily progressing
transfer of a large object is not cut off; no operation timeout is set by
default to avoid capping legitimately long transfers. This mirrors the
existing SQS client timeout support on the same source.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@datadog-vectordotdev

datadog-vectordotdev Bot commented Jul 16, 2026

Copy link
Copy Markdown

Pipelines

⚠️ Warnings

🚦 2 Pipeline jobs failed

Test Suite | Check Generated Docs   View in Datadog   GitHub Actions

Test Suite | Test Suite   View in Datadog   GitHub Actions

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 212946f | Docs | Give us feedback!

@taylorchandleryoung
taylorchandleryoung force-pushed the taylor.c.young/aws-s3-read-timeout branch from 4386c42 to 212946f Compare July 16, 2026 14:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain: sources Anything related to the Vector's sources

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant