fix: make the agent panic-safe on unexpected types, and gate it in CI - #332
Merged
Conversation
#329 fixed the nine informer DeleteFunc handlers that panicked on a cache.DeletedFinalStateUnknown tombstone. It fixed the instances, not the class: 25 unchecked type assertions remained, and nothing stopped another being added. An unchecked assertion is not an ordinary bug here. Informer handlers run on client-go's shared goroutine, where a panic reaches apimachinery's runtime handler and terminates the process — the customer that reported this crashlooped 14 times in 12 hours off a single one. Remaining sites, by why they matter: informer AddFunc/UpdateFunc (20) — same fatal goroutine. Add and update never carry a tombstone, but the informers register transform functions (stripPod, stripNode, stripService) that return the object unchanged when their own assertion fails, so an unexpected type can still reach a handler. sync.Map reads in ip_resolver (3) — the same file already guards this pattern in four other places with a "type confusion" log. These were simply inconsistent. cilium.go (2) — CtEntry is decoded from bpffs. A Cilium version whose struct layout differs from the one we build against is exactly the case that yields an unexpected type, and it should degrade to unresolved. tracer.go, pinger.go, container.go (3) — unreachable in practice, but on metrics and connection paths where dying is never the right answer. Every site now skips the event and continues. For an observability agent a degraded resolver beats a crashlooping DaemonSet, and the next informer resync repairs the gap. The lint gate is the part that closes the class. go vet has no unchecked-type-assertion check and the repo had no linter at all; forcetypeassert catches exactly this. Verified by reintroducing the original bug, which fails with: common/ip_resolver.go:707:4: type assertion must be checked (forcetypeassert) Config is deliberately narrow — one linter, tests excluded. A linter that fails on hundreds of pre-existing findings gets disabled rather than fixed. golangci-lint must be built with Go >= the go.mod directive or it refuses to load the config, so v2.13.2 rather than the older v2.1.6.
There was a problem hiding this comment.
Code Review
This pull request configures the forcetypeassert linter and refactors several type assertions across the codebase to use safe type assertions, preventing potential process-killing panics on unexpected types. In common/ip_resolver.go, the reviewer noted that returning an empty Workload{} early upon type confusion in ResolvePodOwner bypasses the API server fallback; they suggest refactoring the logic to allow the function to fall through to the fallback mechanism instead.
blue4209211
approved these changes
Sep 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
#329 fixed the nine informer
DeleteFunchandlers that panicked on acache.DeletedFinalStateUnknowntombstone. That fixed the instances, not the class — 25 unchecked type assertions remained, and nothing stopped another being introduced.An unchecked assertion isn't an ordinary bug here. Informer handlers run on client-go's shared goroutine, where a panic reaches apimachinery's runtime handler and terminates the process. The customer that reported this crashlooped 14 times in 12 hours off a single one.
What was left, and why each matters
AddFunc/UpdateFuncstripPod,stripNode,stripService) that return the object unchanged when their own assertion fails, so an unexpected type can still reach a handler.sync.Mapreads inip_resolver.go"type confusion"log. These were simply inconsistent.cilium.goCtEntryis decoded from bpffs. A Cilium version whose struct layout differs from the one we build against is precisely the case that yields an unexpected type — and it should degrade to unresolved, not kill the agent.tracer.go,pinger.go,container.goEvery site now skips the event and continues. For an observability agent a degraded resolver beats a crashlooping DaemonSet, and the next informer resync repairs the gap.
The lint gate is the actual fix
go vethas no unchecked-type-assertion check, and this repo had no linter at all — so nothing would have caught the original bug, and nothing would catch the next one.forcetypeassertcatches exactly this class. Verified by reintroducing the original bug on top of this branch:On the fixed tree: 0 issues.
Config is deliberately narrow — one linter, tests excluded. A linter that fails on hundreds of pre-existing findings gets disabled rather than fixed, and this needs to survive.
Testing
Note for reviewers
golangci-lintmust be built with Go >= thegodirective in go.mod, or it refuses to load the config:Hence
v2.13.2, not the olderv2.1.6I first tried. CI already installslibsystemd-devbefore this step, which the linter's typecheck needs.Not covered
This makes type assertions safe. It does not make the agent panic-proof generally — nil-map writes, slice bounds, and nil derefs remain possible and are not lintable this way. A broader option is setting
utilruntime.ReallyCrash = falseso informer handler panics are logged instead of fatal; that's a deliberate policy call with its own downside (masking bugs, inconsistent state), so it's left out here rather than slipped in.