feat(aws_s3 sink): add Parquet encoder with schema_file and auto infer schema support by petere-datadog · Pull Request #25156 · vectordotdev/vector

petere-datadog · 2026-04-09T19:32:42Z

Summary

Added support for parquet encoding for the aws s3 sink
Options supported:
- schema_mode: auto_infer, relaxed, strict
- schema_file: Only needed if schema mode is not auto_infer

This PR was initially started by @szibis: #24706

Vector configuration

auto-infer.yaml

sources:
  demo:
    type: demo_logs
    format: apache_common
    interval: 0.1

sinks:
  s3_parquet:
    type: aws_s3
    inputs:
      - demo
    bucket: obs-pipelines-e2e-tests
    region: us-east-1
    key_prefix: "peter-test/demo_logs/dt=%Y%m%d/hour=%H/"
    filename_time_format: "%s"
    filename_append_uuid: true
    compression: none  # Parquet handles its own compression internally

    # Standard per-event encoding is still required by the field even when
    # batch_encoding takes over. Set it to text as a no-op placeholder.
    encoding:
      codec: text

    batch_encoding:
      codec: parquet
      schema_mode: auto_infer
      compression: 
        algorithm: gzip
        level: 9

    batch:
      max_events: 10000
      timeout_secs: 5

schema-file.yaml

sources:
  demo:
    type: demo_logs
    format: apache_common
    interval: 0.1

sinks:
  s3_parquet:
    type: aws_s3
    inputs:
      - demo
    bucket: obs-pipelines-e2e-tests
    region: us-east-1
    key_prefix: "peter-test/demo_logs/dt=%Y%m%d/hour=%H/"
    filename_time_format: "%s"
    filename_append_uuid: true
    compression: none  # Parquet handles its own compression internally

    # Standard per-event encoding is still required by the field even when
    # batch_encoding takes over. Set it to text as a no-op placeholder.
    encoding:
      codec: text

    batch_encoding:
      codec: parquet
      schema_mode: strict
      schema_file: /Users/peter.ehikhuemen/go/src/github.com/DataDog/vectordotdev/vector/local/apache-common.schema
      compression: 
        algorithm: snappy

    batch:
      max_events: 10000
      timeout_secs: 5

apache-common.schema

message arrow_schema {
  optional binary host (STRING);
  optional binary message (STRING);
  optional binary service (STRING);
  optional binary source_type (STRING);
  optional int64 timestamp (TIMESTAMP(MICROS,true));
}

How did you test this PR?

cargo run --features "codecs-parquet" --  --config local/configs/aws-s3-sink-parquet-encoding.yaml

Change Type

Is this a breaking change?

Yes
No

All the new stuff is added under features: codec-parquet.

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the no-changelog label to this PR.

Notes

Please read our Vector contributor resources.
Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
Some CI checks run only after we manually approve them.
- We recommend adding a pre-push hook, please see this template.
- Alternatively, we recommend running the following locally before pushing to the remote branch:
  - make fmt
  - make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
  - make test
After a review is requested, please avoid force pushes to help us review incrementally.
- Feel free to push as many commits as you want. They will be squashed into one before merging.
- For example, you can run git merge origin master and git push.
If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
run make build-licenses to regenerate the license inventory and commit the changes (if any). More details on the dd-rust-license-tool.

…pport

Keep JSON-based build_record_batch/find_null_non_nullable_fields for Parquet compatibility. Drop unused serde_arrow dep. Regenerate Cargo.lock. Made-with: Cursor

github-actions · 2026-04-09T19:32:55Z

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1eaa921f96

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

lib/codecs/src/encoding/format/arrow.rs

lib/codecs/src/encoding/format/parquet.rs

changelog.d/aws_s3_parquet_encoding.feature.md

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 03a4544905

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

src/sinks/aws_s3/integration_tests.rs

src/sinks/aws_s3/config.rs

pront · 2026-04-10T13:29:20Z

Hey @petere-datadog, while I am taking a look at this PR please see this #25156 (comment) and the codex review comments.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ca201c5078

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

src/sinks/aws_s3/config.rs

changelog.d/aws_s3_parquet_encoding.feature.md

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6dcc854103

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

src/sinks/util/encoding.rs

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 08b9b547f4

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

lib/codecs/src/encoding/format/arrow.rs

datadog-vectordotdev · 2026-04-10T17:38:59Z

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 02e4caa | Docs | Was this helpful? Give us feedback!}

changelog.d/aws_s3_parquet_encoding.feature.md

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0f8dde57d0

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

lib/codecs/src/encoding/format/arrow.rs

petere-datadog · 2026-04-13T15:27:51Z

recheck

petere-datadog · 2026-04-13T15:59:20Z

I have read the CLA Document and I hereby sign the CLA

pront

Some things that stood out. I will take another look.

lib/codecs/src/encoding/format/arrow.rs

lib/codecs/src/encoding/format/parquet.rs

…ting an error

Himan10 · 2026-04-14T09:24:14Z

Hey, we are currently deploying vector on your UAT instances to test few workflows. One of our use-case consists of having logs in the parquet format with gzip compression. I did check some documentation but couldn't find any good resources on it. Could you let me know when this PR is going to be merged or if there's any resource that I can read on how to set-up.

tessneau

nice ! overall seems good to me, just some non-blockers, thanks for all the tests

changelog.d/aws_s3_parquet_encoding.feature.md

lib/codecs/src/internal_events.rs

lib/codecs/src/encoding/format/parquet.rs

src/sinks/aws_s3/config.rs

src/sinks/amqp/config.rs

lib/codecs/src/encoding/format/parquet.rs

petere-datadog · 2026-04-14T14:42:16Z

Hey, we are currently deploying vector on your UAT instances to test few workflows. One of our use-case consists of having logs in the parquet format with gzip compression. I did check some documentation but couldn't find any good resources on it. Could you let me know when this PR is going to be merged or if there's any resource that I can read on how to set-up.

It should be merged later today and yeah this will support encoding batched events with parquet and gzip compression

…ik/aws-s3-parquet-encoding

lib/codecs/Cargo.toml

pront

This PR is now in a reasonable state, thanks!

Due it's large size and number of changes, there are some rough edges and pre-existing issues that can be addressed in follow-up PRs.

Created: #25191 (not introduced by this PR but directly related)

maycmlee

Just some suggestions

website/cue/reference/components/sinks/aws_s3.cue

Co-authored-by: May Lee <may.lee@datadoghq.com>

…rdotdev/vector into peter.ehik/aws-s3-parquet-encoding

petere-datadog added 3 commits April 9, 2026 15:09

feat(codecs): add Parquet encoder with schema_file and schema_mode su…

352e5f6

…pport

Merge master and resolve conflicts

edbed49

Keep JSON-based build_record_batch/find_null_non_nullable_fields for Parquet compatibility. Drop unused serde_arrow dep. Regenerate Cargo.lock. Made-with: Cursor

fix broken tests

1eaa921

petere-datadog requested review from a team as code owners April 9, 2026 19:32

github-actions bot added domain: sinks Anything related to the Vector's sinks domain: ci Anything related to Vector's CI environment domain: external docs Anything related to Vector's external, public documentation labels Apr 9, 2026

maycmlee added the under_review label Apr 9, 2026

chatgpt-codex-connector bot reviewed Apr 9, 2026

View reviewed changes

lib/codecs/src/encoding/format/arrow.rs Outdated Show resolved Hide resolved

lib/codecs/src/encoding/format/parquet.rs Outdated Show resolved Hide resolved

remove benches stuff

03a4544

github-advanced-security AI found potential problems Apr 9, 2026

View reviewed changes

changelog.d/aws_s3_parquet_encoding.feature.md Fixed Show fixed Hide fixed

chatgpt-codex-connector bot reviewed Apr 9, 2026

View reviewed changes

src/sinks/aws_s3/integration_tests.rs Show resolved Hide resolved

src/sinks/aws_s3/config.rs Show resolved Hide resolved

petere-datadog added 3 commits April 10, 2026 10:38

fix the integration tests

2c61e10

Add compression level to compression type

fca54e7

update teh aws_s3.cue file

ca201c5

chatgpt-codex-connector bot reviewed Apr 10, 2026

View reviewed changes

src/sinks/aws_s3/config.rs Outdated Show resolved Hide resolved

Minor updates based on my own PR review of szibis initial PR

6dcc854

github-advanced-security AI found potential problems Apr 10, 2026

View reviewed changes

changelog.d/aws_s3_parquet_encoding.feature.md Fixed Show fixed Hide fixed

changelog.d/aws_s3_parquet_encoding.feature.md Fixed Show fixed Hide fixed

fix minor issue from batch encoder

11aa83b

chatgpt-codex-connector bot reviewed Apr 10, 2026

View reviewed changes

src/sinks/util/encoding.rs Outdated Show resolved Hide resolved

fix tests from szibis PR

08b9b54

chatgpt-codex-connector bot reviewed Apr 10, 2026

View reviewed changes

lib/codecs/src/encoding/format/arrow.rs Outdated Show resolved Hide resolved

Fix the tests

e4235d2

Fix cargo fmt and deny issues

89bdbbd

github-advanced-security AI found potential problems Apr 10, 2026

View reviewed changes

changelog.d/aws_s3_parquet_encoding.feature.md Fixed Show fixed Hide fixed

fix cargo markdown and license errors

f8a4cfd

chatgpt-codex-connector bot reviewed Apr 13, 2026

View reviewed changes

lib/codecs/src/encoding/format/arrow.rs Outdated Show resolved Hide resolved

pront reviewed Apr 13, 2026

View reviewed changes

lib/codecs/src/encoding/format/arrow.rs Outdated Show resolved Hide resolved

lib/codecs/src/encoding/format/arrow.rs Outdated Show resolved Hide resolved

lib/codecs/src/encoding/format/parquet.rs Show resolved Hide resolved

graphcareful approved these changes Apr 13, 2026

View reviewed changes

Short circuit when there's a serialization error instead of just emit…

4781372

…ting an error

tessneau approved these changes Apr 14, 2026

View reviewed changes

petere-datadog added 2 commits April 14, 2026 11:21

PR comments feedback

56f3462

Merge branch 'master' of github.com:vectordotdev/vector into peter.eh…

589f483

…ik/aws-s3-parquet-encoding

petere-datadog enabled auto-merge April 14, 2026 15:54

maycmlee self-assigned this Apr 14, 2026

pront reviewed Apr 14, 2026

View reviewed changes

lib/codecs/Cargo.toml Outdated Show resolved Hide resolved

pront mentioned this pull request Apr 14, 2026

fix(s3 sink): encoding failures after split_input silently ack batches as delivered #25191

Open

petere-datadog added 2 commits April 14, 2026 15:10

emit dropped events metric when there's an error in encoding

dd3b1f3

put JsonSerializationError under parquet feature

c38e5e3

pront approved these changes Apr 14, 2026

View reviewed changes

petere-datadog added 2 commits April 14, 2026 15:46

emit ArrowWriterError and dropped events

860c88d

remove formatting changes in cargo.toml

ced7e55

maycmlee reviewed Apr 14, 2026

View reviewed changes

petere-datadog and others added 3 commits April 14, 2026 16:53

Apply suggestions from May's code review

f49ed2a

Co-authored-by: May Lee <may.lee@datadoghq.com>

remove whitespace updates from lib/codecs/cargo.toml

ae408df

Merge branch 'peter.ehik/aws-s3-parquet-encoding' of github.com:vecto…

02e4caa

…rdotdev/vector into peter.ehik/aws-s3-parquet-encoding

maycmlee approved these changes Apr 14, 2026

View reviewed changes

petere-datadog added this pull request to the merge queue Apr 14, 2026

Merged via the queue into master with commit e32661a Apr 14, 2026
77 checks passed

petere-datadog deleted the peter.ehik/aws-s3-parquet-encoding branch April 14, 2026 23:29

github-actions bot locked and limited conversation to collaborators Apr 14, 2026

Conversation

petere-datadog commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Vector configuration

How did you test this PR?

Change Type

Is this a breaking change?

Does this PR include user facing changes?

Notes

Uh oh!

github-actions bot commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

pront commented Apr 10, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

datadog-vectordotdev bot commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

petere-datadog commented Apr 13, 2026

Uh oh!

petere-datadog commented Apr 13, 2026

Uh oh!

pront left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Himan10 commented Apr 14, 2026

Uh oh!

tessneau left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

petere-datadog commented Apr 14, 2026

Uh oh!

petere-datadog commented Apr 9, 2026 •

edited

Loading

github-actions bot commented Apr 9, 2026 •

edited

Loading

datadog-vectordotdev bot commented Apr 10, 2026 •

edited

Loading