Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
115 changes: 115 additions & 0 deletions docs/adr/0044-per-entity-labels-on-kev-reachable-metric.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
# 0044. Per-entity labels on the KEV-reachable findings metric

- Status: Accepted
- Date: 2026-11-08
- Relates to: [0009](0009-asymmetric-action-bar.md) (the `corroborated` runtime signal this
metric is deliberately NOT keyed on), [0001](0001-async-mitigation-engine.md) /
[0014](0014-behavioral-telemetry-ebpf.md) (the KEV / exploited-in-wild predicate), and the
prior counts-only metric convention this reverses (`engine/src/engine/metrics.rs`)

## Context

Until now every OTLP instrument the engine emits was **counts-only**: a gauge or counter with,
at most, a low-cardinality variant label (`action`, `verdict`, `outcome`, a probe `reason`), and
never a label naming a specific workload, node, or image. The convention was written into the
instrument docstrings as a hard prohibition — a per-node or per-entry series "would be a
cardinality/DoS vector" because node/workload/image names are "attacker-influenceable." The
entity name was pushed into the accompanying structured log instead, and the metric carried only
the count.

That rule guarded two things: unbounded time-series cardinality (one series per distinct label
tuple), and a denial-of-service shape where an attacker who could mint arbitrary workload/node
names could explode the metrics store. Both are real concerns **on a large or multi-tenant
cluster.**

Protector runs on a **bounded, single-operator homelab**. The operator judged the rule
over-cautious there, for two reasons:

1. **The threat it guards presupposes prior compromise.** Workload, node, and image names come
from the operator's own cluster. For an attacker to influence them — to drive cardinality or
forge a misleading label — they must already be creating or renaming Kubernetes objects in
this cluster. At that point the cardinality of a metric series is not the problem to solve;
the attacker is already inside. The bound that makes the guard unnecessary (a small, operator-
controlled namespace of names) is exactly the bound a large/multi-tenant cluster does not
have.

2. **The response value is concrete and immediate.** A CVE that CISA KEV lists as
exploited-in-wild, reachable on an internet-facing (or proven-foothold) entry, is the single
highest-priority finding protector produces. Forcing the operator to cross-reference a bare
count against a separate structured log to learn *which* workload and image to patch costs
minutes at exactly the moment minutes matter. A per-finding labelled series routes the alert
straight to the workload.

Weighed against each other for this deployment, rapid KEV response wins over a cardinality/DoS
guard whose precondition (an attacker who can forge cluster object names) already implies a
worse breach.

## Decision

**We will remove the "no per-entity labels on metrics" invariant and add a KEV-reachable
findings metric that carries per-finding entity labels.**

1. **New counter `protector.engine.kev_reachable_findings`.** Incremented once per pass per
distinct entry workload that is both *reachable* — `is_breach_relevant()` (internet-facing
entry) OR `foothold.is_some()` (a proven exploitable front door) — and *KEV-backed*: its
entry image carries a `Vulnerability` with `exploited_in_wild` set (the CISA KEV catalogue
flag, `observe/exploit_intel.rs`). Emitted from the per-pass chain-fold loop in
`engine/src/engine/mod.rs`, beside the existing `breach_paths` / `corroborations` snapshot.

2. **Labels:** `k8s.namespace.name`, `k8s.workload.kind`, `k8s.workload.name` (all parsed from
the entry `workload/<ns>/<kind>/<name>` key), `image.reference` (the entry's deployed image
reference(s), via a new `SecurityGraph::entry_image_reference` walk over the `RunsImage`
edge), and `cve.ids` (the entry image's exploited-in-wild CVE ids, sorted and joined into one
value). These labels are **attacker-influenceable by design** — that is the protection this
ADR removes, accepted for the homelab bound.

3. **"KEV" is `exploited_in_wild`, not `corroborated`.** This metric's gate is the static CISA
KEV catalogue flag. It is unrelated to [ADR-0009](0009-asymmetric-action-bar.md)'s runtime
`corroborated` signal, which the separate `protector.engine.corroborations` counter tracks.
The two predicates must not be conflated: a finding can be KEV-reachable with no live runtime
corroboration (the latent-foothold case) and vice versa.

4. **Series shape, chosen to avoid gratuitous blowup WITHOUT reinstating the removed guard.**
The counter fires once per distinct entry workload per pass, de-duplicated by entry key. An
entry yields one chain per reachable objective, but every label on this metric is
entry-scoped, so counting each objective-chain would only inflate a single series' value, not
add information — the dedupe keeps the series count equal to the number of KEV-reachable
workloads. Likewise, the several exploited-in-wild CVE ids on one entry are joined into a
single `cve.ids` value rather than fanned out into one series per CVE. This is a
series-hygiene choice, **not** a cardinality cap or an entity-name guard; it does not limit
how many distinct workloads may appear.

## Consequences

- An operator watching `/metrics` (or an alert rule) sees KEV-reachable findings broken down by
the exact workload, image, and CVE — the fast path from "a KEV CVE is reachable" to "patch
*this* image on *this* workload," with no structured-log cross-reference.
- The engine now emits at least one metric whose label values are attacker-influenceable. On
this homelab that is bounded by the operator-controlled name space (see Context); the
structured-log-only discipline no longer applies to new instruments, and the counts-only
docstrings that stated the prohibition are reworded to say per-entity labels are permitted
(the coverage/narrowing-delta gauges stay counts-only because *they* have no use for an entity
label, not because one is forbidden).
- **Revisit trigger — large or multi-tenant deployment.** The entire justification rests on the
single-operator, bounded-name-space bound. If protector is ever deployed on a cluster where
workload/node/image names are not under one trusted operator's control — a shared or
multi-tenant cluster — the cardinality/DoS guard this ADR removed becomes load-bearing again,
and this metric's entity labels (and any that followed it) must be re-gated: bound the series
(e.g. a top-N or a cardinality cap), drop to counts-only with the entity in a log, or
otherwise constrain the attacker-influenceable dimension. This is a deployment-shape decision,
recorded here so the reversal is not silently inherited.
- **Residual worth noting (a consideration, not a control):** even on the homelab the series set
grows with the number of distinct KEV-reachable workloads over the exporter's retention
window. For this cluster's scale that is a handful of series and a non-issue; were it to grow,
a top-N or a cardinality cap would be the place to bound it. It is deliberately NOT added here
— adding it now would reinstate the very protection this ADR removes.

## References

- `engine/src/engine/metrics.rs` — the instrument (`kev_reachable_findings`) and its
`record_kev_reachable_finding` recorder.
- `engine/src/engine/mod.rs` — the per-pass emission in the chain-fold loop.
- `engine/src/engine/graph/mod.rs` — `entry_image_reference` (the `RunsImage` image-ref walk)
and `NodeKey::workload_kind` (the kind-segment accessor) added for the labels.
- [ADR-0009](0009-asymmetric-action-bar.md) — the runtime `corroborated` signal this metric is
explicitly NOT keyed on.
1 change: 1 addition & 0 deletions docs/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,5 +49,6 @@ Copy [`0000-template.md`](0000-template.md) to start one.
| [0041](0041-narrow-blanket-notable-exec-corroboration.md) | Narrow the blanket notable-exec corroboration arm to shapes: a bare interactive-shell/pkg-mgr `ProcessExec` stops blanket-corroborating every objective (`Behavior::ProcessExec → false` in the flat arm — symmetric with PrivilegeChange/PtraceAttach/ModuleLoad, restoring the 0011 on-call-engineer FP guard), and exec-based corroboration re-homes in an entry-scoped `reverse_shell_on_foothold` shape (interactive-shell exec + internet NetworkConnection within a symmetric 60s window, foothold-gated) — unmasking the deferred reverse-shell shape per 0024. Corrects the framing: the model never sees `corroborated` (removed from the prompt by 0034); narrowing buys deterministic auto-apply-gate precision + honest latent/live presentation + 0024 unmasking, NOT model-input precision. Safe **now** because Falco's `Alert` arm still backstops (`falco.enabled: true`), pre-arming in `audit`. One real behavior change (inert under audit): a shell-only + decisive-attack incident becomes a proposal not an auto-cut in enforce (0011-correct). Package managers excluded (always egress); `is_alarming_now`/menu seeding untouched; a narrowing-delta counter feeds the retire-Falco parity bake (0037). Amends 0009's action-bar input | Proposed |
| [0042](0042-flag-remaining-consumer-count-on-consumer-removal.md) | Flag a shared layer's remaining live-consumer count when a PR removes a consumer: a consumer-removal PR states the layer's remaining live-consumer count in one line, and a drop-to-one is flagged in review as a refactor smell rather than merged silently — the discipline PR #189's shared model-completion cache and PR #190's hypothesis-stage deletion needed (their conjunction, invisible to either single-PR review, left a redundant, `Uncertain`-pinning cache that PR #191 had to revert). Extends 0024's single-PR redundant-by-construction rule across a two-PR conjunction | Accepted |
| [0043](0043-internal-serviceaccount-token-auth-on-dashboard-read.md) | Accept an internal Kubernetes ServiceAccount-token on the dashboard read surface, IN ADDITION TO the edge Cloudflare Access path (0030): an in-cluster caller (the cluster repo's `fixer`), authorized by its SA identity, reads `/api/*.json` with its projected SA-token JWT (cluster issuer, RS256, required `aud`, `sub` allowlist) instead of an edge Access JWT it cannot mint in-cluster. Pure addition + read-only: reuses the 0030 `Verifier` (alg-pinned, iss/aud/exp/nbf/sub) pointed at the cluster JWKS via a `ClusterJwksFetcher` (trusts the cluster CA, presents protector's own SA token read fresh per fetch); one gate, two lanes tried in order — a verified+allowlisted token admits read-only at the floor `Redacted` tier, a verified-but-non-allowlisted one is a hard 401 (never a 302 or fall-through), an unverifiable one falls through to the unchanged edge path. FAILS CLOSED (empty allowlist / unreadable CA / JWKS-unreachable ⇒ the lane admits no one; edge untouched). Rides the Oidc-mode Enforcer, so it needs the edge issuer set. Extends 0030; read-only sibling of 0031 | Accepted |
| [0044](0044-per-entity-labels-on-kev-reachable-metric.md) | Remove the "no per-entity labels on metrics" invariant and add `protector.engine.kev_reachable_findings`, a per-pass counter carrying per-finding entity labels (`k8s.namespace.name` / `k8s.workload.kind` / `k8s.workload.name` / `image.reference` / `cve.ids`) for every reachable chain (internet-facing OR proven-foothold) whose entry image runs a CISA-KEV exploited-in-wild CVE. Operator decision: the old counts-only rule (entity names are attacker-influenceable ⇒ a cardinality/DoS vector) is over-cautious for this bounded single-operator homelab — forging those names already implies cluster compromise, and routing a KEV hit straight to the workload/image outweighs the guard. "KEV" is `exploited_in_wild`, NOT the runtime `corroborated` signal (0009). One series per distinct entry workload per pass (labels are entry-scoped) and CVE ids joined into one `cve.ids` value — series hygiene, not a reinstated cap. **Revisit on any large/multi-tenant deployment**: there the bound fails and the removed guard is load-bearing again. Reverses the prior metrics convention | Accepted |

See also [`../VISION.md`](../VISION.md) for the longer-form narrative this ADR realizes.
33 changes: 32 additions & 1 deletion engine/src/engine/graph/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
pub mod attack;
pub mod delta;

use std::collections::BTreeMap;
use std::collections::{BTreeMap, BTreeSet};
use std::time::SystemTime;

use petgraph::stable_graph::{NodeIndex, StableGraph};
Expand Down Expand Up @@ -168,6 +168,17 @@ impl NodeKey {
}
}

/// The Kubernetes workload kind of a workload key (`workload/<ns>/<kind>/<name>`) —
/// the owning kind as observed (`Pod`, `Deployment`, `DaemonSet`, …). `None` for any
/// non-workload or malformed key. Distinct from [`NodeKey::kind`], which names the
/// NODE kind (the literal `workload` discriminant), not the Kubernetes kind.
pub fn workload_kind(&self) -> Option<&str> {
let mut parts = self.0.split('/');
(parts.next()? == "workload")
.then(|| parts.nth(1))
.flatten()
}

/// Everything after the kind prefix — the human-facing remainder of the key with the
/// leading `<kind>/` stripped (e.g. `app/Pod/web` for `workload/app/Pod/web`). Falls
/// back to the whole key when there is no `/`. Used for compact labels where the kind is
Expand Down Expand Up @@ -791,6 +802,26 @@ impl SecurityGraph {
}
(exposed_secrets, misconfigs, rbac_findings)
}

/// The deployed reference(s) of the image(s) an `entry` workload runs, followed across
/// its `RunsImage` edges (the same walk [`entry_evidence`](Self::entry_evidence) uses) —
/// for human narration and the per-finding telemetry label. Distinct references are
/// de-duplicated and joined with `,` (a multi-container pod can run several); `None` when
/// the key is unknown, the node is not a workload, or no image carried a reference
/// (references are human-narration-only and may be absent, see [`Image::reference`]).
pub fn entry_image_reference(&self, entry_key: &NodeKey) -> Option<String> {
let entry = self.index_of(entry_key)?;
let mut refs: BTreeSet<&str> = BTreeSet::new();
for edge in self.graph.edges(entry) {
if matches!(edge.weight().relation, Relation::RunsImage)
&& let Some(Node::Image(image)) = self.graph.node_weight(edge.target())
&& let Some(reference) = image.reference.as_deref()
{
refs.insert(reference);
}
}
(!refs.is_empty()).then(|| refs.into_iter().collect::<Vec<_>>().join(","))
}
}

#[cfg(test)]
Expand Down
Loading
Loading