Skip to content

Conversation

@TheSafo
Copy link
Contributor

@TheSafo TheSafo commented Dec 30, 2025

What does this PR do?

Enables configuring byte-based throttlers in generators to be based per block (e.g. 10 blocks per second instead of 100 MB/s).

Impl. details:

  • Adds a wrapper for generators around lading throttle that's based around throttling blocks at a time (which aligns with how the throttler is currently used)
  • Adds some config tests to verify that both the new format and old format is still supported

Motivation

When using payload generators where blocks are intentionally not the same size/close to maximum bytes, it can be helpful to reason about throughput per block. This allows us to do things like:

  • Define a payload where each block is 1 second worth of a system's logs
  • Set the throttle rate at 5 blocks per second
    to simulate generating logs at 5x standard playback speed.

Related issues

A follow up pr (#1683) adds some flushing logic to BufWriter based generators that enables more regular flushing. A separate PR will introduce a new payload format to utilize block-based throttling.

Copy link
Contributor Author

TheSafo commented Dec 30, 2025

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch 3 times, most recently from c4199fc to 6ed7dca Compare December 31, 2025 16:20
@TheSafo TheSafo changed the title blocks [WIP] Update generator throttler to support block-based strategy Dec 31, 2025
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch 11 times, most recently from a5dc28f to 10ba2f9 Compare January 5, 2026 14:50
@TheSafo TheSafo changed the base branch from main to graphite-base/1675 January 5, 2026 16:00
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch from 10ba2f9 to d724a4c Compare January 5, 2026 16:00
@TheSafo TheSafo changed the base branch from graphite-base/1675 to jsaf/throttle-error-change January 5, 2026 16:00
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch 4 times, most recently from 74c2f05 to 9ae8ce6 Compare January 6, 2026 18:11
@TheSafo TheSafo marked this pull request as ready for review January 6, 2026 19:15
@TheSafo TheSafo requested a review from a team as a code owner January 6, 2026 19:15
@TheSafo TheSafo marked this pull request as draft January 6, 2026 19:25
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch 2 times, most recently from 3c1df18 to 01c4cd3 Compare January 6, 2026 21:03
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch from 01c4cd3 to 648642c Compare January 6, 2026 21:16
@TheSafo TheSafo changed the base branch from jsaf/throttle-error-change to graphite-base/1675 January 6, 2026 21:46
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch from 648642c to a636e4c Compare January 6, 2026 21:46
@TheSafo TheSafo changed the base branch from graphite-base/1675 to jsaf/add-config-tests January 6, 2026 21:46
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch from a636e4c to 661d654 Compare January 6, 2026 21:58
@TheSafo TheSafo force-pushed the jsaf/add-config-tests branch from af8d207 to cf2d888 Compare January 6, 2026 22:13
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch 2 times, most recently from 867b097 to e1d8abc Compare January 6, 2026 22:29
@TheSafo TheSafo force-pushed the jsaf/block-based-limiter branch from e1d8abc to e73bb21 Compare January 6, 2026 22:31
@TheSafo TheSafo changed the title [WIP] Update generator throttler to support block-based strategy Update generator throttler to support block-based strategy Jan 7, 2026
@TheSafo TheSafo mentioned this pull request Jan 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants