Skip to content

feat(controller): reconcile SandboxAgents when referenced header Secrets change - #2438

Draft
voltagebots wants to merge 1 commit into
kagent-dev:mainfrom
voltagebots:ofrank/watch-headersfrom-secret-rotation
Draft

feat(controller): reconcile SandboxAgents when referenced header Secrets change#2438
voltagebots wants to merge 1 commit into
kagent-dev:mainfrom
voltagebots:ofrank/watch-headersfrom-secret-rotation

Conversation

@voltagebots

Copy link
Copy Markdown

What

Adds a Secret watch to the SandboxAgent controller so that changes to a Secret referenced by an agent's request headers trigger reconciliation. The watch resolves referencing agents through two paths:

  • Direct: a tool's HeadersFrom references the Secret.
  • Indirect: a tool points at a RemoteMCPServer whose HeadersFrom references the Secret (namespace-access-checked via AllowedNamespaces).

An UpdateFunc predicate (secretDataChangedPredicate) gates events on actual Data/StringData changes, and remoteMCPServerReferencesSecret is extended to match header-sourced Secret refs.

Why

Before this change, rotating a Secret used only for request headers left running agents with stale credentials until an unrelated reconcile happened to fire. Watching the Secret closes that gap and makes header-credential rotation take effect deterministically.

Fixes #2404.

Accepted tradeoff — rollout on rotation

The agent's rendered config hash includes the resolved header value, so a genuine token rotation changes the hash and rolls the pod. This is intended and accepted: a rotated credential should propagate to a fresh pod. The predicate ensures only real Data/StringData changes qualify, so no-op Secret updates do not churn workloads.

Note — graceful degradation on RemoteMCPServer-list failure

The Secret finder deliberately does not fail closed if the RemoteMCPServer list call errors (documented in-code). Failing closed there would also drop the direct HeadersFrom matches, which never depended on that list. Instead the path degrades to indirect-incomplete — direct matches are always returned; only indirect (via-RemoteMCPServer) matches may be missed on a transient list error, and are recovered on the next reconcile. Covered by an explicit test, including the empty-apiGroup RemoteMCPServer reference form.

Testing

  • go build ./core/internal/controller/... — passes
  • go test -race -skip 'TestE2E.*' -v ./core/internal/controller/ — all pass, including 7 new TestSandboxAgentSecretFinder cases (direct/indirect/cross-namespace/empty-apiGroup/list-failure-degradation) and the data-changed predicate test
  • go vet ./core/internal/controller/... — clean

Opening as draft per the contribution guide's process for changes >100 lines — happy to discuss approach before this is ready for review.

…ets change

Adds a Secret watch to the SandboxAgent controller so that changes to a
Secret referenced by an agent's request headers trigger reconciliation.
Resolves referencing agents through two paths: direct tool.HeadersFrom
references, and indirect references via a RemoteMCPServer whose own
HeadersFrom references the Secret (namespace-access-checked via
AllowedNamespaces). A data-aware predicate gates events on actual
Data/StringData changes so no-op Secret updates don't trigger a re-render.

Fixes kagent-dev#2404

Signed-off-by: Franklin Okpako <franklin.okpako075@gmail.com>
@github-actions github-actions Bot added the enhancement New feature or request label Aug 13, 2026
@voltagebots
voltagebots marked this pull request as ready for review August 13, 2026 20:52
@voltagebots
voltagebots requested a review from a team as a code owner August 13, 2026 20:52
@voltagebots
voltagebots marked this pull request as draft August 13, 2026 20:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RemoteMCPServer headersFrom secret is baked into agent config and never re-rendered on rotation — agents break permanently with short-lived tokens

1 participant