Skip to content

Conversation

@PGBI
Copy link

@PGBI PGBI commented Jan 28, 2026

Summary

Add a new optional max_bytes configuration option to the reduce transform that triggers a flush when the accumulated byte size of a reduced event group would exceed the configured threshold.

This complements the existing max_events option and provides more granular control over memory usage when reducing events. This will also help with sinks that have a max bytes limit, such as Datadog or Kinesis.

Vector configuration

my_transform_id:
    type: reduce
    inputs:
      - my-source-or-transform-id
    expire_after_ms: 30000
    flush_period_ms: 1000
    max_bytes: 1048576  # 1MB - flush when accumulated size would exceed this
    group_by:
      - request_id

How did you test this PR?

By adding unit tests covering the new option.

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

… memory control

Add a new optional `max_bytes` configuration option to the reduce transform
that triggers a flush when the accumulated byte size of a reduced event group
would exceed the configured threshold. This complements the existing `max_events`
option and provides more granular control over memory usage.
@github-actions github-actions bot added the domain: transforms Anything related to Vector's transform components label Jan 28, 2026
@github-actions
Copy link

github-actions bot commented Jan 28, 2026

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@PGBI PGBI changed the title enhancement(reduce transform): add max_bytes configuration option for memory control enhancement(reduce transform): add max_bytes configuration option Jan 28, 2026
@PGBI
Copy link
Author

PGBI commented Jan 28, 2026

I have read the CLA Document and I hereby sign the CLA

@PGBI
Copy link
Author

PGBI commented Jan 28, 2026

recheck

@PGBI PGBI marked this pull request as ready for review January 28, 2026 22:29
@PGBI PGBI requested a review from a team as a code owner January 28, 2026 22:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain: transforms Anything related to Vector's transform components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant