input_chunk: fix storage limit eviction with 32 or more outputs - #12235
Open
terrorobe wants to merge 3 commits into
Open
input_chunk: fix storage limit eviction with 32 or more outputs#12235terrorobe wants to merge 3 commits into
terrorobe wants to merge 3 commits into
Conversation
The storage limit path used a signed int as a secondary output selection mask. Shifting by output IDs at or above 32 is undefined and can alias a high-ID output to a lower output on common architectures. Check each routed output against its configured limit directly before releasing queue space. This keeps limit enforcement aligned with the dynamically sized routing masks. Signed-off-by: Michael Renner <terrorobe@github.com>
Create 33 outputs and place the constrained output at ID 32. Verify that eviction removes only its route while preserving the unrelated route at ID 0. Signed-off-by: Michael Renner <terrorobe@github.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughFilesystem space reclamation now evaluates routed output limits directly. Over-limit detection returns a boolean. A new integration test validates behavior with 33 outputs and filesystem storage. ChangesInput chunk limit handling
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/internal/input_chunk.c`:
- Around line 1252-1254: In the test setup around flb_input_chunk_get_real_size,
store the result in a signed ssize_t chunk_real_size, assert that it is positive
before converting it to first_chunk_size, and route assertion failure through
the function’s cleanup path. Add the cleanup label before the task teardown loop
so resources are released while failures prevent deriving total_limit_size from
a negative value.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 070dd534-fb16-4c8d-b60a-50e07dcbfca4
📒 Files selected for processing (2)
src/flb_input_chunk.ctests/internal/input_chunk.c
Keep the real chunk size signed until its error result is checked. Avoid deriving the test limit from a failed size lookup. Signed-off-by: Michael Renner <terrorobe@github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In configurations with many outputs,
storage.total_limit_sizeeviction can drop records queued for the wrong destination. In the reproduced case, output ID 32 reached its limit, but Fluent Bit also cleared the route for output ID 0. It then deleted the oldest chunk even though that chunk was still queued for output 0.The storage-limit path represented over-limit outputs in a signed 32-bit integer, even though chunk routing masks support more than 32 outputs. Shifting
1by output ID 32 is undefined; on the tested platform, it reused the bit for output ID 0.This change checks each routed output directly against its configured limit. It keeps the existing two linear output scans and adds neither allocations nor list traversals. The regression test creates 33 outputs and verifies that eviction for ID 32 preserves the route for ID 0.
Before the fix:
After the fix:
The affected code is present in v5.0.9, so this should be backported to the
5.0branch.Summary by CodeRabbit
Bug Fixes
Tests