Skip to content

Fix #86: four AST10 incident records said things their sources do not - #89

Merged
emmanuelgjr merged 11 commits into
mainfrom
fix/incident-source-accuracy
Sep 15, 2026
Merged

Fix #86: four AST10 incident records said things their sources do not#89
emmanuelgjr merged 11 commits into
mainfrom
fix/incident-source-accuracy

Conversation

@emmanuelgjr

@emmanuelgjr emmanuelgjr commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Plan ID: — (data accuracy; follow-up to #29 / T-A10-02) Ticket: — Wave: —
Constraints honored:

  • C4: severity, owasp_entries, MAESTRO layer/role and mitigations are untouched. Only statements a primary source contradicts change.
  • Never fabricate: every correction quotes or follows a fetched page.
  • C2: docs/incidents.js / docs/data.js are regenerated with data only.
  • C1: untouched.

Stacked on #87 (which is stacked on #37). #87 also writes data/incidents.json, and the playbook requires those writers to run one after another. Merge order: #37 → #87 → this, retargeting each to main as its base merges.

Fixes #86.

Corrections

Record Was Source says Source
INC-124 ClawJacked "CVE-2026-28363, CVSS 9.9"; "no rate limiting" NVD: CVE-2026-28363 is "tools.exec.safeBins validation for sort could be bypassed via GNU long-option abbreviations", a different bug. Oasis gives no CVE or CVSS; "The gateway's rate limiter exempts localhost connections entirely"; "The OpenClaw team classified this as High severity and shipped a fix within 24 hours" (2026.2.25) Oasis, NVD API
INC-117 "detected by behavioural analysis rather than by static review — the registry's own checks had cleared them" Caterpillar "statically inspects skill logic and configurations"; nothing about registry checks Alice / PRNewswire
INC-119 "over-permissioned skills … granted broader access than their function required" "283 skills … contain critical security flaws that expose sensitive credentials": instructions that pass secrets "through the LLM's context window and output logs in plaintext". The write-up never describes over-permissioning Snyk
INC-125 "Vidar infostealer variants specifically targeting OpenClaw identity files"; "retargeted" One "live infection"; "not captured by a specialized 'OpenClaw module' … a broad file-grabbing routine". Vidar: Hudson Rock's CTO told THN "likely a variant of Vidar" Hudson Rock (InfoStealers mirror), The Hacker News

Also:

  • References: primary-source URLs added to references[] and external_refs[] for all four. Before this, only the AST10 landing page was cited.
  • Tags: INC-117 behavioural-detection → skill-scanner; INC-124 drops cve.
  • Layer notes: INC-117's two notes and INC-119's L3 note repeated the false claims; their wording now follows the sources.

Human decisions — judgments that rested on a corrected claim (C4, not changed)

  1. INC-124 severity is Critical. OpenClaw classified the issue High, and the 9.9 CVSS that likely justified Critical belonged to another CVE. Re-rate or keep?
  2. INC-119 maps to AST03 (Over-Privileged Skills), with tag over-privilege. With over-permissioning gone, is AST03 still right, or does the plaintext-secrets mechanism belong elsewhere? AST04 Insecure Metadata is one candidate — expert call.
  3. INC-117 L5 role blind-spot was premised on "detection came from runtime behaviour". The new note ("in use by 6,000+ users before a scanner flagged them") still supports a detection gap, but please confirm.
  4. INC-125 impact dropped "context poisoning", which no source mentions. The mitigations are unchanged.

Deliberately NOT changed

Verify

$ node scripts/validate.js      Summary: 0 error(s), 84 warning(s), 313 passed  (before: same)
$ npm run stats:check           ✓ current (incident count unchanged: 131)
$ npm test                      60/60
ajv: data/incidents.json valid against incidents-schema.json
$ grep -rn CVE-2026-28363 data docs src scripts → only the schema's format example

🤖 Generated with Claude Code

https://claude.ai/code/session_014SfR2YzLxRH54DAVzDk8gR

emmanuelgjr and others added 5 commits August 30, 2026 10:10
@emmanuelgjr @claude
T-ENG03: make the webapp data bundles deterministic
1258005 
Every docs/*.js bundle carried a `// Generated: <today>` header. Nothing
else in the generator output varies between runs, so that one line was
the whole reason a regenerate on a different day showed four dirty
files — and, since #30, the reason the Generator reproducibility job and
the `committed entries match a fresh generation` test could only pass on
the day the bundles were last committed.

- generate.js: drop the run-date lines; the Source header now reads the
  version from package.json instead of a hard-coded, stale `v1.5.2`.
- generate.test.mjs: assert no bundle header names a run or carries a date.
- validate.yml: the reproducibility job now diffs every generated
  artefact (backlinks.json, backlinks.js, frameworks-registry.js too).
- CONTRIBUTING.md: document the build contract — generated files, the
  determinism requirement, and why the bundles are committed (Pages
  serves docs/ from main; no deploy workflow exists).

Determinism only. No structural, route, layout or logo change (C2).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0147wBugcuzLkswKPqgofcke
@emmanuelgjr @claude
T-STRAT03: derive evidence_count from incident control failures
cdd09b6 
A mapping says a control addresses a risk; an incident's control_failures[]
says that control failed in the wild. This joins the two and surfaces the
result, without ever counting a claim nobody reviewed.

- scripts/evidence.js — the one place the join is defined. Rules (DRAFT,
  for ratification in docs/EVIDENCE_METHODOLOGY.md): a failure supports a
  mapping only when the incident exemplifies that entry and names the same
  framework + control; only confirmed failures (non-empty confirmed_by)
  count; the unit is the incident. Unabsorbed failures are reported as
  orphans (possible missing mapping — a human call, C4).
- stats.js — new `evidence` block in data/stats.json.
- generate.js — `evidence_count` + `evidence.{confirmed,drafted}` on mapping
  rows that have a linked failure; every other row is byte-identical.
  Currently 6 rows, all drafted, all evidence_count 0.
- validate.js — checkEvidence(): framework and control must resolve in the
  registry, basis must be quotable, a confirmed failure needs a source_url;
  orphans warn. Negative-tested with three planted records (5 errors).
- compliance-report.js — "Controls that failed in the wild" per framework
  and in the summary; evidence in the JSON output.
- src/index.ts — ControlFailure/ExternalRef types, evidenceFor(),
  controlFailures({confirmedOnly}). Reads generated fields, so the package
  cannot disagree with the data it ships.
- data/schema.json — documents the two generated fields.

Also fixes a pre-existing test race this ticket made more likely:
generate.js truncated and rewrote every output even when unchanged, so a
suite reading data/entries in parallel could see an empty file ("Unexpected
end of JSON input", observed 2 in ~15 runs). It now skips identical writes;
12 consecutive full runs after the change were clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014SfR2YzLxRH54DAVzDk8gR
@emmanuelgjr @claude
T-STRAT03: key the evidence index on JSON text, not a NUL separator
1f08139 
The composite-key separator in evidence.js was written to disk as a
literal NUL byte, so git classified the file as binary: no diff, no
review, and a surprise for anyone opening it. Keys are now
JSON.stringify([parts]) — plain text, and no separator a framework name
such as "ISO/IEC 27001:2022" could collide with. No output changes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014SfR2YzLxRH54DAVzDk8gR
@emmanuelgjr @claude
T-STRAT03: pilot batch — 18 drafted control failures across 13 AST in…
164e13d 
…cidents

DRAFT — human confirm. Every confirmed_by is empty; evidence_count stays 0
on every row until a reviewer signs a record off.

Each basis is an exact quotation, cut programmatically from the fetched
source page and re-checked as a substring; 16 of them were re-fetched and
re-checked independently before this commit. Primary disclosures were
preferred (GitHub security advisories, Oasis, Check Point-era GHSAs, Snyk,
Trail of Bits, Air Security, BlueRock, Antiy CERT) over the AST10 summary
pages. Layer and outcome are suggestions for SME review.

Selection rules:
- the source must state a control was missing or defeated; a description
  of the attack alone does not qualify
- the layer must be mapped by the incident's own entries (no orphans)
- hedged quotes ("if any exists") excluded; at most two per incident

The three earlier drafts are replaced, not kept alongside:
- INC-115: its quote was verbatim but described the attacker writing to
  MEMORY.md/SOUL.md, not a missing control — it failed the drafting rule
- INC-123, INC-124: quotes were not verbatim (re-capitalised; truncated
  with a full stop for a comma); now quoted from the primary disclosures

Not drafted: INC-117 and INC-125 (sources state no control failure);
INC-122, INC-130 and one INC-131 candidate would be orphans (MAESTRO L5,
not mapped by their entries) — listed in the PR for a human.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014SfR2YzLxRH54DAVzDk8gR
@emmanuelgjr @claude
Fix #86: four AST10 incident records said things their sources do not
4ea96de 
Each change below was checked against the fetched primary source.

- INC-124 (ClawJacked): drop CVE-2026-28363 and "CVSS 9.9". NVD
  describes that CVE as a tools.exec.safeBins allowlist bypass fixed in
  2026.2.23, a different bug; the Oasis Security disclosure gives no CVE
  or CVSS. Description now follows Oasis: brute-forced gateway password,
  rate limiter exempting localhost, auto-approved localhost pairing,
  fixed in under 24 hours in 2026.2.25, classified High by OpenClaw.
- INC-117: the record said the skills were found by behavioural analysis
  "rather than by static review" and that registry checks had cleared
  them. Alice's release says Caterpillar "statically inspects skill
  logic" and says nothing about registry checks.
- INC-119: Snyk's Leaky Skills write-up describes skills that route
  secrets through the LLM context and logs in plaintext (283 of 3,984),
  not over-permissioning; the record's reference slug was a guess.
- INC-125: Hudson Rock reports one live infection whose files were taken
  by a broad file-grabbing routine, "not ... a specialized OpenClaw
  module"; Vidar is its CTO's "likely" attribution to The Hacker News.

Primary-source URLs added to references/external_refs for all four.
Severity, owasp_entries, MAESTRO layer/role and mitigations unchanged
(C4) — where they rested on a corrected claim, that is flagged in the PR.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014SfR2YzLxRH54DAVzDk8gR
This was referenced Sep 14, 2026
Open
Open
Open
emmanuelgjr and others added 5 commits September 14, 2026 15:44
@emmanuelgjr @claude
Merge main (#91, #92) into T-ENG03
786ee28 
Conflicts were only in the four generated webapp bundles, and only in
their header lines: #92 regenerated them with a `// Generated:` date
that this branch removes. Resolved by re-running scripts/generate.js on
the merged sources; a second run is byte-identical.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014SfR2YzLxRH54DAVzDk8gR
@emmanuelgjr @claude
Merge updated T-ENG03 (main #91, #92) into T-STRAT03
e2458d5 
No conflicts. Generated files re-run on the merged sources; a second
generation is byte-identical. validate 0 errors, 60/60 tests.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014SfR2YzLxRH54DAVzDk8gR
@emmanuelgjr @claude
Merge updated T-STRAT03 (main #91, #92) into incident-source-accuracy
1bd1be3 
Generated files re-run on the merged sources; a second generation is byte-identical.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014SfR2YzLxRH54DAVzDk8gR
@emmanuelgjr @claude
Merge main (#37, #94, #96) into T-STRAT03-evidence-loop
bd14e00 
#37 landed as a squash, so this branch's copies of its commits resolve as identical content. Generated files re-run on the merged sources; a second generation is byte-identical.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014SfR2YzLxRH54DAVzDk8gR
@emmanuelgjr @claude
Merge updated T-STRAT03 (main #37, #94, #96) into incident-source-acc…
08f5f64 
…uracy

Generated files re-run on the merged sources; a second generation is byte-identical.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014SfR2YzLxRH54DAVzDk8gR
Base automatically changed from fix/T-STRAT03-evidence-loop to main September 15, 2026 01:28
@emmanuelgjr @claude
Merge main into incident-source-accuracy before merge
d390d6c 
Generated files re-run on the merged sources; a second generation is byte-identical.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014SfR2YzLxRH54DAVzDk8gR
@emmanuelgjr
emmanuelgjr merged commit 2a7845d into main Sep 15, 2026
6 checks passed
@emmanuelgjr
emmanuelgjr deleted the fix/incident-source-accuracy branch September 15, 2026 01:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Incident records INC-117, INC-119, INC-124, INC-125 state things their sources do not

1 participant

@emmanuelgjr