filter_kubernetes: bound metadata fetch I/O and negative-cache failed lookups - #12169
Open
PovilasV1 wants to merge 1 commit into
Open
filter_kubernetes: bound metadata fetch I/O and negative-cache failed lookups#12169PovilasV1 wants to merge 1 commit into
PovilasV1 wants to merge 1 commit into
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
🚧 Files skipped from review as they are similar to previous changes (4)
📝 WalkthroughWalkthroughThe Kubernetes filter adds configurable metadata I/O timeouts and a TTL-based negative cache for failed pod metadata lookups, including configuration, context storage, initialization, cleanup, and lookup handling. ChangesKubernetes metadata safeguards
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
plugins/filter_kubernetes/kube_meta.c (1)
2437-2462: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueMove the negative-cache locals to the top of the function and add a debug log on suppression.
neg_buf/neg_sizeare declared mid-block; the rest of this file (and the project C style) declares locals at function scope. Also, silently returning un-enriched records makes the negative cache invisible in logs — aflb_plg_debugon the suppression path makes triage of "records missing kube metadata" much easier.♻️ Proposed refactor
static inline int lookup_pod_meta(struct flb_kube *ctx, const char **out_buf, size_t *out_size, struct flb_kube_meta *meta, struct flb_kube_props *props) { int id; int ret; const char *hash_meta_buf; char *tmp_hash_meta_buf; + const char *neg_buf; + size_t neg_size; size_t off = 0; size_t hash_meta_size; msgpack_unpacked result; @@ if (ctx->neg_hash_table) { - const char *neg_buf; - size_t neg_size; if (flb_hash_table_get(ctx->neg_hash_table, meta->cache_key, meta->cache_key_len, (void *) &neg_buf, &neg_size) != -1) { + flb_plg_debug(ctx->ins, "negative cache hit for '%s', " + "skipping metadata request", meta->cache_key); *out_buf = NULL; *out_size = 0; return 0; } }As per coding guidelines: "Declare variables at the start of functions rather than mid-block."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/filter_kubernetes/kube_meta.c` around lines 2437 - 2462, In the function containing the negative-cache check, move neg_buf and neg_size from the if (ctx->neg_hash_table) block to the function-scope declarations at the top. On the suppression path where flb_hash_table_get finds meta->cache_key, add a flb_plg_debug log identifying the negative-cache skip before returning the empty result.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/filter_kubernetes/kube_meta.c`:
- Around line 2455-2462: Update the error handling around
get_and_merge_pod_meta() so neg_hash_table_add() runs only for failures from
network-based pod/API or kubelet lookups. Exclude merge_meta_from_tag()
failures, including the use_tag_for_meta path, from negative caching while
preserving the existing return/error handling.
In `@plugins/filter_kubernetes/kube_meta.c.orig`:
- Around line 1-52: Remove the obsolete tracked kube_meta.c.orig file from the
filter_kubernetes plugin, leaving the active kube_meta.c implementation
unchanged. Optionally add directory-scoped ignore rules for *.orig and *.rej to
prevent regenerated patch artifacts from being reintroduced.
---
Nitpick comments:
In `@plugins/filter_kubernetes/kube_meta.c`:
- Around line 2437-2462: In the function containing the negative-cache check,
move neg_buf and neg_size from the if (ctx->neg_hash_table) block to the
function-scope declarations at the top. On the suppression path where
flb_hash_table_get finds meta->cache_key, add a flb_plg_debug log identifying
the negative-cache skip before returning the empty result.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 67660424-dcac-409f-8032-4beb616b06d7
📒 Files selected for processing (5)
plugins/filter_kubernetes/kube_conf.cplugins/filter_kubernetes/kube_conf.hplugins/filter_kubernetes/kube_meta.cplugins/filter_kubernetes/kube_meta.c.origplugins/filter_kubernetes/kubernetes.c
The pod metadata fetch is synchronous with async disabled and no read timeout, on the pipeline/event-loop thread. A Kubernetes API server / kubelet connection that establishes then stalls blocks recv() indefinitely, so in_tail (same event loop) stops collecting and the tail input flatlines while the pod stays healthy. Failed lookups are not cached, so every record for an unseen pod re-issues the blocking request. Add two safeguards (defaults on, 0 = legacy): - kube_meta_io_timeout (30s): read timeout on the API/kubelet upstreams so a stalled read errors instead of wedging the pipeline forever. - kube_meta_negative_cache_ttl (60s): remember a failed lookup briefly so records for that pod pass through instead of re-blocking on every line. Fixes fluent#12168 Signed-off-by: Povilas Vaitkus <povilas.vaitkus@hostinger.com>
PovilasV1
force-pushed
the
fix/filter_kubernetes-metadata-io-timeout
branch
from
July 28, 2026 16:52
0c06128 to
808e5ee
Compare
Author
|
Pushed `808e5ee` addressing the automated review:
Recompiled locally (v5.0.9 + this change builds clean) and re-checked the reproduction from #12168: behaviour is unchanged for the network path (the new guard only affects the `use_tag_for_meta` path; the added lines are debug-logging only) — the permanent wedge still becomes a bounded stall that recovers. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What / why
filter_kubernetesresolves pod metadata with a synchronous request (async is explicitly disabled on the upstream) that has no read timeout, on the pipeline/event-loop thread. If the API server / kubelet connection establishes and then stalls,recv()blocks that thread indefinitely andin_tail(same event loop) stops collecting — the tail input flatlines while the pod stays healthy (ingestion_paused=0, storage not overlimit). Failed lookups are not cached, so every subsequent record for an unseen pod re-issues the blocking request.Full root-cause analysis, a deterministic reproduction (minikube + Toxiproxy), and a
/proccapture of the pipeline thread parked inread()on the metadata socket are in #12168.Change
Two operator-tunable safeguards (defaults on;
0= legacy behaviour):kube_meta_io_timeout(default30s) — setsnet.io_timeouton the API and kubelet upstreams so a stalled read errors out instead of blocking the pipeline forever.kube_meta_negative_cache_ttl(default60s) — short-TTL negative cache so a failed lookup isn't re-issued (and re-blocked) on every subsequent line for the same pod.This bounds the synchronous fetch; it does not make it async. A fuller fix would be async metadata resolution / a background pod cache — happy to follow up if maintainers prefer that direction.
Testing
Rebuilt
v5.0.9with this change and re-ran the reproduction from #12168 (kube_meta_io_timeout 10):records_total{tail.0}frozen forever; pipeline thread stuck inread()on the metadata socket.Fixes #12168
Summary by CodeRabbit
New Features
Bug Fixes