Skip to content

chore(aws_s3 source): add custom endpoint for sqs#25182

Open
serge-r wants to merge 1 commit intovectordotdev:masterfrom
serge-r:aws-separate-sqs-endpoint
Open

chore(aws_s3 source): add custom endpoint for sqs#25182
serge-r wants to merge 1 commit intovectordotdev:masterfrom
serge-r:aws-separate-sqs-endpoint

Conversation

@serge-r
Copy link
Copy Markdown

@serge-r serge-r commented Apr 14, 2026

Summary

Add ability to separate SQS endpoint from s3 configuration. It's useful for working with local environments like MinIO and ElasticMQ or non-AWS clouds.

Vector configuration

[sources.my_s3]
  type = "aws_s3"
  endpoint = "http://minio.cluster.local:9000/" 
  region = "us-east-1"

  [sources.my_s3.sqs]
  queue_url = "http://elasticmq.cluster.local:9324/queue/my-queue"
  endpoint = "http://elasticmq.cluster.local:9324/"

How did you test this PR?

  • cargo check
  • make check-clippy
  • local setup with own S3 and SQS endpoints

Change Type

  • Bug fix
  • New feature
  • Dependencies
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

@serge-r serge-r requested a review from a team as a code owner April 14, 2026 06:23
@github-actions github-actions bot added the domain: sources Anything related to the Vector's sources label Apr 14, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 14, 2026

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bdbe6b6fbf

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

&self.auth,
region.clone(),
endpoint,
sqs_region,
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use S3 region for ingestor region checks

This call now resolves the region from sqs.region, but that returned region is later stored in state.region and compared against s3_event.aws_region in handle_s3_event_record (src/sources/aws_s3/sqs.rs). If a user configures different S3 and SQS regions/endpoints (the new behavior this change enables), events from the S3 bucket region will be rejected as WrongRegion before object fetch, so the override becomes functionally broken for split-region setups.

Useful? React with 👍 / 👎.

@serge-r
Copy link
Copy Markdown
Author

serge-r commented Apr 14, 2026

I have read the CLA Document and I hereby sign the CLA

@serge-r serge-r changed the title chore(aws-sqs): add custom endpoint chore(aws_s3 source): add custom endpoint for sqs Apr 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain: sources Anything related to the Vector's sources

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant