diff --git a/.claude/skills/policy-author/SKILL.md b/.claude/skills/policy-author/SKILL.md new file mode 100644 index 00000000..88408e28 --- /dev/null +++ b/.claude/skills/policy-author/SKILL.md @@ -0,0 +1,431 @@ +--- +name: policy-author +description: Triage `failproofai audit` findings and author failproofai policies that stop coding agents misbehaving. Use in four cases. (1) An audit just ran and its findings need turning into fixes. (2) The user describes something an agent keeps doing wrong and wants it prevented — "my agent keeps force-pushing", "it deleted my files again", "stop it committing straight to main", "how do I stop it doing X" — they are describing a problem, not asking for a policy, and may not know policies exist. (3) The user explicitly asks to write, create, or add a policy. (4) The user wants the rules in a CLAUDE.md, AGENTS.md, or system-prompt file actually enforced — "agents keep ignoring my CLAUDE.md", "make my rules file strict", "turn my instructions into enforcement". Covers enabling an existing builtin (usually the right answer) as well as writing new custom policy files. +--- + +# Authoring failproofai policies + +Four ways you get here. All converge on the same authoring core (§2). + +**1. Automatically, after an audit.** A `PostToolUse` policy fires when `failproofai audit` +runs and instructs the agent to come here. The findings are already sitting in the cache — +**start at §1** and triage them. This is the primary path: the audit knows what went wrong, +so it should not be on the user to notice and ask. + +**2. The user describes a problem.** *"My agent keeps force-pushing."* *"It deleted my files +again."* *"How do I stop it committing to main?"* They are reporting a failure, not +requesting a policy — most users do not know policies exist. Translate the complaint into a +rule, then **go to §2**. Do not make them specify events or tools; infer from the behavior +and confirm at the end. + +**3. The user asks for a policy outright.** *"Write a policy that blocks X."* Straight to §2. + +**4. The user has a rules file agents keep ignoring.** *"Agents keep skipping what's in my +CLAUDE.md."* Prose rules are advisory — the agent reads them (maybe) and forgets them under +context pressure. Policies are enforcement. Extract the rules, classify what is enforceable, +and turn that subset into policies — **§3** has the procedure and the classification table. + +For 2, 3 and 4, still check the audit cache if one exists — it often shows the behavior is +already happening and gives you real commands to use as test cases (§2.4). + +The single most common mistake is writing a custom policy when a builtin already covers +the case and just needs enabling. Always check coverage before authoring. + +--- + +## 1. Audit-driven triage + +**One finding, or all of them?** If the user names a single finding — by its policy name +(`git-commit-no-verify`), by description ("the one about `--no-verify`"), or by pointing at +a row in the dashboard — do **not** run the full triage. Handle just that one: + +1. Look it up in the cache by name, or by matching its `displayTitle` / `examples[]` + against what they described. +2. Read its `examples[]`. These are **real commands from their machine** — the single most + valuable thing the audit gives you. They become the should-fire cases in §2.4 (should-deny + for a block-mode policy, should-instruct for oversight), which means the finished policy + is proven against what actually happened rather than against invented input. +3. Pick the mode from the finding's severity class (§2's mode table): deny-class findings + get `deny()`, warn-class get `instruct()` oversight. **State the choice in one line and + offer the other mode** — "went with oversight since this has legitimate uses; say the + word for a hard block." The user asked for enforcement; which flavor is their call. +4. Check whether a builtin covers it (§2.1). If yes and it is off, that is a config line, + not a policy. +5. Otherwise author it (§2), test against those examples, report back. + +Mention what else is unaddressed in one line at the end — do not expand into a full triage +they did not ask for. + +Only run the full §1.1–1.4 sweep when they ask broadly: "what should I do about my audit", +"harden this repo", "fix these findings". + +### 1.1 Read the findings + +There is **no machine-readable output flag**. `RunAuditOptions` declares `--json`, `--since`, +`--cli`, `--project`, `--limit` and more, but `runAuditCli` parses only `--help` — every +other argument is rejected outright (`failproofai audit --json` errors). The one programmatic +path is the dashboard cache, which every audit run writes: + +```bash +cat ~/.failproofai/audit-dashboard.json +``` + +**The cache is the only source, and it can be arbitrarily old.** Check `cachedAt` before +trusting anything in it: + +```bash +bun -e 'const j = await Bun.file(process.env.HOME + "/.failproofai/audit-dashboard.json").json(); +const age = (Date.now() - Date.parse(j.cachedAt)) / 86400000; +console.log(`cached ${j.cachedAt} (${age.toFixed(1)} days ago), ${j.result.results.length} findings`);' +``` + +More than a day or two old and the findings describe past behavior, not current — say so +rather than presenting it as the live picture. + +**Refreshing it is awkward.** `failproofai audit` writes the cache and then starts a +dashboard server on port 8020 and opens a browser, so it never exits on its own. Prefer +asking the user to run it. If you must refresh unattended, the cache is written *before* the +server starts, so a timeout gets you fresh data without leaving a server running: + +```bash +timeout 180 failproofai audit >/dev/null 2>&1 || true # exits 124; cache is written +``` + +Do that only when the user has asked for fresh findings — it can take minutes on a large +history (423 transcripts took ~50s here) and it opens a browser tab. + +If the file does not exist at all, the user has never run an audit. Ask them to, rather than +running it for them. + +**The `AuditResult` is nested under a cache envelope.** The file is +`{schemaVersion, cachedAt, params, result}` — the findings are at **`.result.results[]`**, +not `.results[]`: + +```bash +bun -e 'const j = await Bun.file(process.env.HOME + "/.failproofai/audit-dashboard.json").json(); +for (const c of j.result.results.sort((a,b)=>b.hits-a.hits)) + console.log([c.name.replace("failproofai/",""), c.source, c.hits, c.projects, c.enabledInConfig].join(" | "));' +``` + +Each entry in `.result.results[]` is an `AuditCount`. The fields that matter for triage: + +| Field | Use | +|---|---| +| `name` | Builtins are **canonical-prefixed** (`failproofai/block-rm-rf`); detectors are bare. Strip the prefix before matching against config | +| `source` | `"builtin"` = a real policy that *would have* fired; `"audit-detector"` = audit-only pattern | +| `hits`, `projects` | How much this actually happens — prioritize by this | +| `examples[]` | **Real payloads from this machine.** These become your test cases in §2.4 | +| `enabledInConfig` | **Do not trust this** — see below. Always `false` for detectors | + +**`enabledInConfig` is a stale snapshot.** It records the merged config as it was *at audit +time*, and the cache persists indefinitely. Observed on this machine: the audit cached at +20:14 reported all 39 builtins enabled; the global config was emptied at 20:19, five minutes +later. Every finding still claims `enabledInConfig: true`. + +Always re-read the current config before classifying: + +```bash +cat .failproofai/policies-config.json 2>/dev/null # project scope +cat ~/.failproofai/policies-config.json 2>/dev/null # global scope +``` + +`enabledPolicies` is a **union** across project, local and global +(`hooks-config.ts`, grep `enabledSet`) — not precedence. A policy is on if *any* scope lists it. + +**Scope matters more than it looks.** The audit spans every project on the machine, but a +project-scope config protects only one. If findings span many projects and enforcement lives +in a single project config, the honest conclusion is that most of those hits are still +unprotected — and the fix belongs in the **global** config, not a project one. + +### 1.2 Sort every finding into one of three buckets + +**Bucket A — a builtin covers it and is off.** Do not write code. Add the short name to +`enabledPolicies` in `.failproofai/policies-config.json`. This is the cheapest and most +maintainable fix, and it is the right answer for most `source: "builtin"` findings. + +**Bucket B — a builtin covers it and is already on.** No action. Report it so the user knows +the finding is historical, not ongoing. + +**Bucket C — nothing genuinely covers it.** Author a custom policy (§2). + +### 1.3 Do not trust `DETECTOR_TO_POLICY` + +`src/audit/findings.ts`, grep `DETECTOR_TO_POLICY` maps every audit-only detector to some builtin, and its own +header comment explains why: so that "every finding looks like it has a failproofai fix." +Several of those mappings do not actually prevent the behavior. Judge coverage yourself: + +| Detector | Maps to | Real coverage | +|---|---|---| +| `redundant-cd-cwd` | `warn-repeated-tool-calls` | **None** — unrelated heuristic. Bucket C | +| `prefer-edit-over-sed-awk` | `warn-repeated-tool-calls` | **None**. Bucket C | +| `prefer-edit-over-read-cat` | `block-read-outside-cwd` | **None** for in-cwd reads. Bucket C | +| `prefer-write-over-heredoc` | `block-env-files` | Only the `.env` subset. Bucket C for the rest | +| `find-from-root` | `block-read-outside-cwd` | Partial — that policy is about file reads, not `find` | +| `sleep-polling-loop` | `warn-background-process` | Partial | +| `git-commit-no-verify` | `warn-git-amend` | **None** — different command | +| `reread-after-edit` | `warn-repeated-tool-calls` | Partial — only if params are identical | + +So most audit-only detectors are Bucket C. That is the gap this skill exists to fill. + +### 1.4 Present the triage before acting + +Show the user the three buckets with hit counts, then act. For Bucket A, propose the +config diff rather than silently editing — enabling enforcement changes what their agent is +allowed to do. For a single explicit request ("turn on block-rm-rf"), just edit it. + +**Never widen scope on your own initiative.** These three are off-limits without the user +asking for them in the current request: + +| Action | Why it needs asking | +|---|---| +| `failproofai policies --install` at **user scope** | Wires hooks into *every* project on the machine, not the one they are in | +| Editing `~/.failproofai/policies-config.json` | Global config; a deny there fires everywhere | +| Setting `customPoliciesPath` globally | Silently activates policy files across all projects | + +A question — *"what should I do about my findings?"*, *"is this protected?"* — asks for an +answer, not a change. Recommend the machine-wide fix in words and let them decide. Being +right about what should happen is not authorization to make it happen. + +Project-scope edits inside the repo the user is working in are fine when they asked for a +fix. The line is **scope**: their repo, yes; their machine, ask first. + +--- + +## 2. The authoring core + +**Arriving from a complaint?** Translate it into a concrete tool call first — a policy can +only match what actually crosses the wire. Common mappings: + +| What the user says | What to match on | +|---|---| +| "keeps force-pushing" | `Bash`, `command` =~ `git push --force` / `-f` | +| "deleted my files" | `Bash`, `command` =~ `rm -rf` | +| "commits straight to main" | `Bash`, `git commit` + current branch check | +| "reads my secrets / .env" | `Read`/`Bash`, `file_path` or `command` =~ `.env` | +| "installs junk globally" | `Bash`, `command` =~ `npm i -g`, `pip install`, … | +| "edits generated files" | `Write`/`Edit`, `file_path` =~ lockfile / `dist/` | +| "leaks keys into chat" | `sanitize-api-keys` + `additionalPatterns` param first; else a `PostToolUse` sanitizer (blocks the output — `message` is inert, `traps.md` §9) | + +If the mapping is not obvious, ask for the command they actually saw, or pull it from the +audit cache — `examples[]` holds real invocations and doubles as your test cases (§2.4). + +Two judgment calls to make before writing, and to state back to the user at the end: + +- **Which of the three modes?** failproofai does not only block. Match the builtins: + + | Mode | Helper | Name it | When | + |---|---|---|---| + | block | `deny()` | `block-*` | irreversible, no legitimate use | + | **oversight** | `instruct()` | `warn-*` | risky but sometimes right — *"STOP: … Confirm with the user before executing."* | + | sanitize | raw deny object (blocks output; `message` inert — traps.md §9) | `sanitize-*` | secrets in tool output | + + Default to **oversight** when the action has any legitimate use. Blocking those just gets + the policy disabled. See `patterns.md` for the exact voice the builtins use — copy it. +- **Scope.** Project config protects one repo; user scope (`~/.failproofai/policies/`) + applies everywhere. "My agent keeps doing X" usually means *everywhere*, not *here*. + +### 2.1 Check the builtins first + +Read `references/builtins.md`. All 39 builtins with their categories, default state, events +and parameters. If one matches, enabling it beats writing a new file every time. + +Many builtins take `params` (allowlists, thresholds, protected branches) that go in the +`policyParams` map — a parameterized builtin often covers a case that looks custom. + +Then check the project's **existing custom policies** — `ls .failproofai/policies/` and +read their `name`/`description` lines. Coverage is not only builtins: a hand-written policy +may already enforce exactly what you were about to author, and a duplicate means two +policies firing on every matching event. + +### 2.2 Choose the event and tool + +Read `references/api.md` for `PolicyContext`, the decision helpers, and the full event list. + +Rules of thumb: +- Blocking an action before it happens → `PreToolUse` +- Redacting or reacting to output → `PostToolUse` +- Gating the end of a turn → `Stop` (but see `references/traps.md` §6 — unsatisfiable Stop gates — before using this in + this repo) + +### 2.3 Write the file + +Location: `.failproofai/policies/` in the project. + +**The filename must end in `policies.js`, `policies.mjs`, or `policies.ts`.** A file named +`block-foo.mjs` is silently skipped and enforces nothing. Name it `block-foo-policies.mjs`. +This is the highest-frequency failure in the whole system — see `references/traps.md` §1. + +See `references/patterns.md` for worked examples per event type. + +### 2.4 Verify it actually fires + +Loading and execution are both fail-open — a broken policy is indistinguishable from a +working one unless you test it. Never report a policy as done without this step. + +Use the bundled runner: + +```bash +node .claude/skills/policy-author/scripts/test-policy.mjs \ + --policy .failproofai/policies/my-policies.mjs \ + --event PreToolUse --tool Bash \ + --input '{"command":"sudo rm -rf /tmp/x"}' --expect deny +``` + +Or a batch, which is what you want once there is more than one rule — `{{cwd}}` in an input +expands to the sandbox path: + +```json +[{ "name": "blocks sudo", "event": "PreToolUse", "tool": "Bash", + "input": {"command":"sudo ls"}, "expect": "deny" }, + { "name": "allows plain ls", "event": "PreToolUse", "tool": "Bash", + "input": {"command":"ls"}, "expect": "allow" }] +``` + +```bash +node .claude/skills/policy-author/scripts/test-policy.mjs --policy --cases cases.json +``` + +With `--policy`, the file is copied into a throwaway directory that acts as both project and +HOME — so `customPoliciesEnabled: false`, the real project config, and user-scope policies +cannot affect the result. It also **renames the file if it violates the loader convention** +(§2.3) and says so. Omit `--policy` to test the current directory's real config instead. + +Exit code is 1 if any `--expect` fails, so it drops straight into a script. + +Underneath it is just the documented stdin protocol, if you need it by hand: + +```bash +echo '{"hook_event_name":"PreToolUse","tool_name":"Bash","tool_input":{"command":"sudo rm -rf /tmp/x"},"session_id":"test","transcript_path":"/dev/null","cwd":"'"$PWD"'"}' \ + | node scripts/dev-hook.mjs --hook PreToolUse +``` + +**Read stdout, not the exit code.** Both outcomes exit 0: + +| Outcome | stdout | +|---|---| +| Denied (`PreToolUse`) | `{"hookSpecificOutput":{…,"permissionDecision":"deny","permissionDecisionReason":"…"}}` | +| Denied (`PermissionRequest`) | `{"hookSpecificOutput":{…,"decision":{"behavior":"deny","message":"…"}}}` | +| Instructed | `{"hookSpecificOutput":{…,"additionalContext":"Instruction from failproofai: …"}}` | +| Allowed | *(empty)* | + +**Deny is not one shape.** `PreToolUse` uses `permissionDecision`; `PermissionRequest` uses +`decision.behavior`; `instruct` uses `additionalContext`; the non-Claude CLIs use flat +`{decision:"block"}` or `{permission:"deny"}`, and Factory signals deny by exit code 2 with +the reason on stderr. Grepping for a single key will under-report blocks — this exact +mistake produced a false FAIL during testing. `test-policy.mjs` handles all of them. + +To test without touching the project's own config, build a throwaway project directory with +its own `.failproofai/policies/` and `policies-config.json`, and point the payload's `cwd` +at it. Project-scope discovery keys off that `cwd`, so the policy loads in isolation. + +Test both directions: a payload that **should** be denied, and a near-miss that **should** +be allowed. A policy that denies everything passes the first test. + +Two ways this test can lie to you: + +- **Empty output proves nothing on its own.** It is the same result whether your policy + allowed correctly, threw, timed out, or was never loaded. Always pair it with a + should-deny case that actually produces output. +- **A deny does not prove *your* policy denied.** Every enabled policy sees the event, so + another one may have fired — a rule matching nothing can hide behind a green suite. Assert + on text unique to your policy's reason, or test with `enabledPolicies: []`. See + `traps.md` §4; this happens more often than it sounds. + +Use the `examples[]` from the audit finding as your should-deny cases — they are real +commands from this machine, so they prove the policy catches what actually happened. + +### 2.5 Confirm the policy is actually live + +Do **not** rely on `customPoliciesEnabled` in `.failproofai/policies-config.json` — that flag +is a no-op and disables nothing (`traps.md` §2). The only reliable check is the one you +already did in §2.4: run the hook and look at stdout. + +What genuinely stops a policy from running: the filename convention (§2.3), a load-time +throw, or hooks not being installed for the CLI at all. Verify with: + +```bash +failproofai policies --list +``` + +--- + +## 3. Enforcing a rules file (CLAUDE.md / AGENTS.md / system prompts) + +Prose rules are advisory: the agent reads them at session start and drops them under +context pressure. Policies fire on every tool call regardless of what the model remembers. +This path converts the enforceable subset of a rules file into policies — and is honest +about the rest. + +### 3.1 Extract + +Read the file. Pull out every rule stated as *behavior* — quote each verbatim and note its +section heading. Skip background prose, architecture notes, and anything descriptive. + +### 3.2 Classify + +Sort each rule using the table in `references/rules-files.md`. The short version: + +| Rule language | Class | Becomes | +|---|---|---| +| "never X" / "do not X" — tool-shaped | hard rule | `block-*` deny, or a builtin | +| "always X before Y" — ordering | workflow gate | PreToolUse check **on Y** (prefer over Stop gates) | +| "prefer X over Y" / "avoid Y" | preference | `warn-*` `instruct()` nudge | +| "file/config must contain Z" | repo invariant | a **test in the test suite** — not a policy | +| style, judgment, tone | unenforceable | stays prose; say so | + +Two classes here are easy to get wrong. Repo invariants ("configs must use the launcher +form") are about file *states*, and policies see tool *calls* — recommend a test, don't +force a policy. And workflow rules enforce best at the **action they gate** (`gh pr create`, +`git commit`), not as Stop gates — tighter feedback, no loop risk (`traps.md` §6). + +### 3.3 Check coverage — builtins AND existing custom policies + +Rules files are exactly where hand-written policies come from, so the rule you are about to +enforce may already be enforced. Check `references/builtins.md` (with params — §2.1), then +read the project's existing custom policies: + +```bash +ls .failproofai/policies/ && grep -h "name:\|description:" .failproofai/policies/*policies.mjs +``` + +A rule already covered goes in the report as covered — writing a duplicate policy means two +policies fire on every matching event forever. + +### 3.4 Present the extraction before enforcing + +Show the table — rule → class → action — before writing a batch. Turning a page of prose +into enforcement changes what the user's agents are allowed to do everywhere in the +project; that is a decision they confirm, not a side effect (§1.4 discipline). + +The table is **not optional and not summarizable into prose**: even when the user +explicitly asked you to enforce the file, your report opens with the table. It is the one +artifact that lets them audit your classification at a glance — prose bullets hide a +misclassified rule; a table row does not. + +### 3.5 Author and verify + +Route the uncovered enforceable subset through §2. In each generated file, cite provenance +so the policy can be traced back and re-synced when the md changes: + +```js +// derived-from: CLAUDE.md § "Workflow rules" — "Never push a branch that is +// missing commits from main" (extracted 2026-07-24) +``` + +### 3.6 Report the honest split + +End with four lists: **enforced now** (new policies + enabled builtins), **already +covered** (by what), **nudged** (instruct-only), **left as prose** (and why). A rules file +never converts 100%. Claiming it does means something above was misclassified. + +--- + +## 4. Reference files + +| File | Read it when | +|---|---| +| `references/api.md` | Writing any policy — types, helpers, context, events | +| `references/builtins.md` | Triaging coverage, or before authoring anything | +| `references/traps.md` | **Always.** Nine documented silent failures | +| `references/patterns.md` | Worked examples per event type | +| `references/rules-files.md` | Enforcing a CLAUDE.md / AGENTS.md (§3) — classification and worked examples | diff --git a/.claude/skills/policy-author/references/api.md b/.claude/skills/policy-author/references/api.md new file mode 100644 index 00000000..99617341 --- /dev/null +++ b/.claude/skills/policy-author/references/api.md @@ -0,0 +1,168 @@ +# Policy API reference + +Everything here is exported from `src/index.ts` (19 lines — that is the entire public +surface): + +```ts +export { customPolicies, getCustomHooks, clearCustomHooks } from "./hooks/custom-hooks-registry"; +export { allow, deny, instruct } from "./hooks/policy-helpers"; +export type { PolicyContext, PolicyResult, CustomHook, PolicyDecision, PolicyFunction } from "./hooks/policy-types"; +``` + +## The policy object + +`CustomHook` — `src/hooks/policy-types.ts`, grep `interface CustomHook`: + +```ts +export interface CustomHook { + name: string; + description?: string; + match?: { + events?: HookEventType[]; + }; + fn: (ctx: PolicyContext) => PolicyResult | Promise; +} +``` + +Registered with `customPolicies.add(hook)`. That is the whole registration surface — no +remove, no update, and **no validation of any kind** — `custom-hooks-registry.ts` (grep +`getRegistry`) is a bare array push. + +## Context + +`PolicyContext` — `policy-types.ts`, grep `interface PolicyContext`: + +```ts +export interface PolicyContext { + eventType: HookEventType; + payload: Record; + toolName?: string; + toolInput?: Record; + session?: SessionMetadata; + params?: Record; + cli?: IntegrationType; // which agent CLI fired this; mirrors session.cli +} +``` + +- `ctx.toolInput` holds the tool's arguments — `command` for Bash, `file_path`/`content` for + Write, `old_string`/`new_string` for Edit, `pattern` for Grep. These keys are already + canonicalized across all 11 supported CLIs, so you write them once. +- `ctx.session?.cwd` is the working directory. +- `ctx.payload` is the raw hook payload if you need something not surfaced above. + +## Decisions + +`PolicyResult` — `policy-types.ts`, grep `interface PolicyResult`: + +```ts +export interface PolicyResult { + decision: "allow" | "deny" | "instruct"; + reason?: string; + message?: string; +} +``` + +Helpers (`policy-helpers.ts`, the complete file): + +```ts +export function allow(reason?: string): PolicyResult { ... } // reason optional +export function deny(reason: string): PolicyResult { ... } // reason required +export function instruct(reason: string): PolicyResult { ... } // reason required +``` + +- **`allow`** — let it through. Also the correct return when your policy does not apply. +- **`deny`** — block it. `reason` is shown to the agent and the user. +- **`instruct`** — let it through but inject guidance into the agent's next turn. Only + properly supported on Claude Code, Devin and Antigravity; degrades to a stderr note on + Hermes, Goose, OpenClaw and Pi. Fine for a local skill; do not rely on it if the policy + will be distributed. + +### The `message` field is currently inert — sanitize works by blocking, not replacing + +The builtin sanitizers return `deny` plus a `message` that *looks like* a replacement — +`sanitizeJwt` in `builtin-policies.ts`: + +```ts +return { + decision: "deny", + reason: "JWT token detected in tool output", + message: "[REDACTED: JWT token removed by failproofai]", +}; +``` + +**But the evaluator never consumes `PolicyResult.message`** (verified 2026-07-24: the deny +response in `policy-evaluator.ts` is built from `reason` alone; the only `.message` read in +that file is `err.message`). What actually happens on a sanitize deny: the tool output is +**blocked entirely** and the model sees the block reason instead. The secret is still +protected — by omission, not redaction — but the model also loses the rest of that output. +Do not promise users "the output is scrubbed and the rest passes through"; it is not. + +Two practical consequences: +- Put everything useful into `reason` — it is the only channel that surfaces. +- For output scrubbing, check `sanitize-api-keys`'s **`additionalPatterns` param first** + (`{regex, label}` entries) — builtin-first applies to sanitizers too. A custom sanitizer + is only needed when the pattern-per-output-shape doesn't fit that param. + +The `deny()` helper cannot set `message` anyway; if you do set it (future-proofing for when +the evaluator honors it), return the raw object literal. + +## Events + +`HookEventType` — `src/hooks/types.ts`, grep `HOOK_EVENT_TYPES`. The ones worth knowing: + +| Event | Fires | Can block? | +|---|---|---| +| `PreToolUse` | Before a tool runs | **Yes** — the main enforcement point | +| `PostToolUse` | After a tool returns | Yes — a deny blocks the whole output (see the `message` note above) | +| `UserPromptSubmit` | On user input | Yes | +| `Stop` | Agent about to finish its turn | Yes — deny forces another turn | +| `SessionStart` / `SessionEnd` | Session boundaries | Observation | +| `SubagentStop` | Subagent returns | Yes on most CLIs | +| `PermissionRequest` / `PermissionDenied` | Permission flow | Yes | + +Full list also includes `PostToolUseFailure`, `StopFailure`, `Notification`, +`SubagentStart`, `TaskCreated`, `TaskCompleted`, `PreCompact`, `PostCompact`, `FileChanged`, +`CwdChanged`, `ConfigChange`, `WorktreeCreate`, `WorktreeRemove`, `Elicitation`, +`ElicitationResult`, `UserPromptExpansion`, `PostToolBatch`, `InstructionsLoaded`, +`TeammateIdle`, `Setup`. + +## Filtering by tool + +`CustomHook.match` publicly declares **only `events`**. Filter on the tool inside `fn`: + +```js +fn: async (ctx) => { + if (ctx.toolName !== "Bash") return allow(); + ... +} +``` + +An undocumented `match.toolNames` does work at runtime (`handler.ts`, grep `hook.match ??` passes `match` +straight through, and `policy-registry.ts`, grep `getPoliciesForEvent` filters on it), but it is absent from the +public type and could be typed away. Prefer filtering in `fn`. + +## Execution model + +- Custom policies run at **priority -1**, i.e. after all builtins. +- Each `fn` gets a **10-second timeout**. Timeout or throw → `{decision: "allow"}`. +- Namespaced as `custom/`, `.failproofai-project/` or `.failproofai-user/`. + +## Configuration + +`.failproofai/policies-config.json` — `HooksConfig`, `policy-types.ts`, grep `interface HooksConfig`: + +```ts +export interface HooksConfig { + enabledPolicies: string[]; // builtins, by short name + llm?: LlmConfig; + policyParams?: Record>; // keyed by short name + customPoliciesPath?: string; + customPoliciesEnabled?: boolean; // absent = enabled +} +``` + +Builtins are enabled **purely by presence** of the short name in `enabledPolicies` — there +is no per-policy enabled/disabled object, and omission means off. + +Merged across three scopes, in precedence order: project `{cwd}/.failproofai/` → local → +global `~/.failproofai/` (`hooks-config.ts`, grep `readMergedHooksConfig`). diff --git a/.claude/skills/policy-author/references/builtins.md b/.claude/skills/policy-author/references/builtins.md new file mode 100644 index 00000000..bfdf6d6e --- /dev/null +++ b/.claude/skills/policy-author/references/builtins.md @@ -0,0 +1,132 @@ +# Builtin policies (39) + +**Generated — do not hand-edit.** Regenerate with: + +```bash +bun .claude/skills/policy-author/scripts/sync-builtins.mjs +``` + +This snapshot goes stale whenever a builtin is added, renamed, or has its default +flipped. When it matters, ask the CLI instead — it is always current: + +```bash +failproofai policies # every policy, with enabled status and params +``` + +## How to use this for triage + +Enabling a builtin beats writing a custom policy: nothing to maintain, no naming +trap, no fail-open risk, and it ships with tests. + +Before concluding "no builtin covers this", check whether a **parameterized** one +does — several take allowlists or thresholds that widen their scope considerably. +Params go in the `policyParams` map, keyed by short name. + +To enable: add the short name to `enabledPolicies` in `.failproofai/policies-config.json`. + +--- + +### Sanitize + +| Policy | Default | Events | What it catches | +|---|---|---|---| +| `sanitize-jwt` | **on** | PostToolUse | Stop Claude from reading JWTs in tool responses | +| `sanitize-api-keys` | **on** | PostToolUse | Stop Claude from reading API keys (OpenAI, Anthropic, GitHub, AWS, Stripe, Google) in tool responses _(params: additionalPatterns)_ | +| `sanitize-connection-strings` | **on** | PostToolUse | Stop Claude from reading database connection strings with embedded credentials in tool responses | +| `sanitize-private-key-content` | **on** | PostToolUse | Stop Claude from reading PEM private key content in tool responses | +| `sanitize-bearer-tokens` | **on** | PostToolUse | Stop Claude from reading Authorization Bearer tokens in tool responses | + +### Environment + +| Policy | Default | Events | What it catches | +|---|---|---|---| +| `protect-env-vars` | **on** | PreToolUse | Prevent commands that read environment variables | +| `block-env-files` | **on** | PreToolUse | Block reading/writing .env files | +| `block-read-outside-cwd` | off | PreToolUse | Block file reads outside the session working directory _(params: allowPaths)_ | + +### Dangerous Commands + +| Policy | Default | Events | What it catches | +|---|---|---|---| +| `block-sudo` | **on** | PreToolUse, PermissionRequest | Block sudo commands _(params: allowPatterns)_ | +| `block-curl-pipe-sh` | **on** | PreToolUse | Block piping downloads to shell | +| `block-rm-rf` | off | PreToolUse | Prevent catastrophic deletions _(params: allowPaths)_ | +| `block-failproofai-commands` | **on** | PreToolUse | Block failproofai CLI commands and uninstallation | +| `block-secrets-write` | off | PreToolUse | Block writing secret key files _(params: additionalPatterns)_ | + +### Infra Commands + +| Policy | Default | Events | What it catches | +|---|---|---|---| +| `block-kubectl` | off | PreToolUse | Block kubectl commands (Kubernetes cluster mutations) _(params: allowPatterns)_ | +| `block-terraform` | off | PreToolUse | Block terraform and tofu (OpenTofu) commands _(params: allowPatterns)_ | +| `block-aws-cli` | off | PreToolUse | Block aws CLI commands _(params: allowPatterns)_ | +| `block-gcloud` | off | PreToolUse | Block gcloud (Google Cloud) CLI commands _(params: allowPatterns)_ | +| `block-az-cli` | off | PreToolUse | Block az (Azure) CLI commands _(params: allowPatterns)_ | +| `block-helm` | off | PreToolUse | Block helm commands _(params: allowPatterns)_ | +| `block-gh-pipeline` | off | PreToolUse | Block gh CLI pipeline-trigger subcommands (workflow run, run rerun/cancel, pr merge, release create/delete, cache delete, secret set/delete) _(params: allowPatterns)_ | + +### Git + +| Policy | Default | Events | What it catches | +|---|---|---|---| +| `block-push-master` | **on** | PreToolUse | Block pushing to main/master _(params: protectedBranches)_ | +| `block-force-push` | off | PreToolUse | Prevent force-pushing to any branch | +| `block-work-on-main` | off | PreToolUse | Block git commits and merges on main/master branch _(params: protectedBranches)_ | +| `warn-git-amend` | off | PreToolUse | Warns before amending git commits, which rewrites history | +| `warn-git-stash-drop` | off | PreToolUse | Warns before permanently deleting stashed changes | +| `warn-all-files-staged` | off | PreToolUse | Warns before staging all working tree files with git add -A / . / --all | + +### Database + +| Policy | Default | Events | What it catches | +|---|---|---|---| +| `warn-destructive-sql` | off | PreToolUse | Warn before executing destructive SQL (DROP/TRUNCATE/DELETE without WHERE) via database clients | +| `warn-schema-alteration` | off | PreToolUse | Warns before SQL schema changes (ALTER TABLE with column or rename operations) | + +### Packages & System + +| Policy | Default | Events | What it catches | +|---|---|---|---| +| `warn-package-publish` | off | PreToolUse | Warn before publishing packages to public registries (npm, PyPI, crates.io, RubyGems, etc.) | +| `warn-global-package-install` | off | PreToolUse | Warns before installing packages globally (npm -g, cargo install, etc.) | +| `prefer-package-manager` | off | PreToolUse | Blocks non-preferred package managers and tells Claude to use an allowed one (e.g., uv instead of pip) _(params: allowed, blocked)_ | +| `warn-large-file-write` | off | PreToolUse | Warn before writing files larger than 1MB (configurable via thresholdKb param) _(params: thresholdKb)_ | +| `warn-background-process` | off | PreToolUse | Warns before starting detached or background processes | + +### AI Behavior + +| Policy | Default | Events | What it catches | +|---|---|---|---| +| `warn-repeated-tool-calls` | off | PreToolUse | Warn when the same tool is called 3+ times with identical parameters | + +### Workflow + +| Policy | Default | Events | What it catches | +|---|---|---|---| +| `require-commit-before-stop` | off | Stop | Require all changes to be committed before Claude stops | +| `require-push-before-stop` | off | Stop | Require all commits to be pushed to remote before Claude stops _(params: remote, baseBranch)_ | +| `require-pr-before-stop` | off | Stop | Require a pull request to exist for the current branch before Claude stops _(params: baseBranch)_ | +| `require-no-conflicts-before-stop` | off | Stop | Require the current branch to merge cleanly with the base branch before Claude stops _(params: baseBranch)_ | +| `require-ci-green-before-stop` | off | Stop | Require CI checks to pass on the current HEAD commit before Claude stops (ignores stale runs on prior commits) | + +> The five `require-*-before-stop` policies gate the end of a turn. A gate whose +> condition cannot be met in the current project loops forever — see `traps.md` §6 +> before enabling one. + +--- + +## Audit-only detectors + +These have no real-time builtin equivalent, so they are the prime candidates for +custom policies. List them from source with: + +```bash +bun -e 'const {AUDIT_DETECTORS}=await import("./src/audit/detectors/index.ts"); +for (const d of AUDIT_DETECTORS) console.log(d.name, "|", d.category+"/"+d.severity, "|", d.description)' +``` + +All but `reread-after-edit` are Bash-command patterns, so a `PreToolUse` policy +filtering on `ctx.toolName === "Bash"` and matching `ctx.toolInput.command` covers +most of them. `reread-after-edit` needs cross-call session state, which hooks cannot +see — that one needs a builtin, not a custom policy. diff --git a/.claude/skills/policy-author/references/patterns.md b/.claude/skills/policy-author/references/patterns.md new file mode 100644 index 00000000..992eb693 --- /dev/null +++ b/.claude/skills/policy-author/references/patterns.md @@ -0,0 +1,224 @@ +# Worked patterns + +All examples go in `.failproofai/policies/-policies.mjs` — the filename **must** +end in `policies.mjs`. See `traps.md` §1. + +Multiple `customPolicies.add()` calls per file are fine and are the normal way to group +related rules. + +--- + +## Block a Bash command pattern + +The workhorse. Seven of the eight audit detectors are Bash-command patterns, so this shape +covers most Bucket C findings. + +```js +import { customPolicies, allow, deny } from "failproofai"; + +const NO_VERIFY_RE = /\bgit\s+commit\b[^\n]*\s(--no-verify|-n)\b/; + +customPolicies.add({ + name: "block-commit-no-verify", + description: "Block git commit --no-verify, which skips pre-commit hooks", + match: { events: ["PreToolUse"] }, + fn: async (ctx) => { + if (ctx.toolName !== "Bash") return allow(); + const command = String(ctx.toolInput?.command ?? ""); + if (!NO_VERIFY_RE.test(command)) return allow(); + return deny( + "git commit --no-verify skips the pre-commit hooks. Run the checks, or commit without the flag.", + ); + }, +}); +``` + +Points that matter: +- Return `allow()` early for tools you do not handle. A policy that only cares about Bash + still runs on every `PreToolUse`. +- Coerce with `String(... ?? "")`. `toolInput` is `Record`. +- Write the `reason` as an instruction to the agent, not just a complaint. It is what the + agent reads next, so tell it what to do instead. + +## Block by file path + +```js +import { customPolicies, allow, deny } from "failproofai"; + +customPolicies.add({ + name: "block-lockfile-edits", + description: "Lockfiles are generated — block hand-edits", + match: { events: ["PreToolUse"] }, + fn: async (ctx) => { + if (!["Write", "Edit"].includes(ctx.toolName ?? "")) return allow(); + const path = String(ctx.toolInput?.file_path ?? ""); + if (!/\b(bun\.lock|package-lock\.json|yarn\.lock)$/.test(path)) return allow(); + return deny("Lockfiles are generated. Run the package manager instead of editing by hand."); + }, +}); +``` + +`file_path` is canonical across all 11 CLIs — the per-CLI input maps normalize Copilot's +`path`, Hermes's `path`, etc. before your policy sees it. + +## Three modes — pick one deliberately + +failproofai is not only a blocker. The builtins split three ways, and the name prefix +signals which: + +| Prefix | Helper | Effect | Use when | +|---|---|---|---| +| `block-*` (17) | `deny()` | action never runs | irreversible or unsafe, no legitimate case | +| `warn-*` (10) | `instruct()` | action runs; agent told to check with the human first | risky but sometimes correct — needs a human, not a wall | +| `sanitize-*` (5) | raw deny object | output **blocked** before the model sees it (`message` is inert — traps.md §9) | secrets in tool output | + +Name your policy with the matching prefix. A reader should know the mode from the name. + +Most requests that sound like "block X" are really `warn-*`. Blocking something with a +legitimate use just teaches people to disable the policy. + +## Oversight — stop and confirm with the human + +The `warn-*` builtins share one voice, and it is worth copying exactly. They do **not** nudge +toward a better tool — they halt the agent and hand the decision back to the person: + +> **STOP:** This command permanently deletes stashed changes (git stash drop/clear). Stash +> entries cannot be recovered after deletion. Confirm with the user before executing. + +Three parts: **STOP:** + what the command actually does + *Confirm with the user before +executing.* The middle part explains why it matters, so the human can decide in one read. + +```js +import { customPolicies, allow, instruct } from "failproofai"; + +const PROD_DEPLOY_RE = /\b(kubectl\s+apply|helm\s+upgrade|serverless\s+deploy)\b[^\n]*\bprod/; + +customPolicies.add({ + name: "warn-prod-deploy", + description: "Require human confirmation before a production deploy", + match: { events: ["PreToolUse"] }, + fn: async (ctx) => { + if (ctx.toolName !== "Bash") return allow(); + const command = String(ctx.toolInput?.command ?? ""); + if (!PROD_DEPLOY_RE.test(command)) return allow(); + return instruct( + "STOP: This command deploys to production. It affects live traffic and is not " + + "trivially reversible. Confirm with the user before executing.", + ); + }, +}); +``` + +This is the mode to reach for when the answer to "should this ever be allowed?" is +"sometimes, and a human should decide." + +**Caveat:** `instruct` reaches the model properly on Claude Code, Devin and Antigravity. On +Hermes, Goose, OpenClaw and Pi it degrades to a stderr note the agent never sees — so on +those, oversight silently becomes no oversight. If the policy must hold everywhere, use +`deny()` with a reason explaining how to proceed. + +## Nudge toward a better tool + +The other use of `instruct()` — the action is not dangerous, just wasteful. No "STOP", no +human needed, just a better way to do it. + +```js +import { customPolicies, allow, instruct } from "failproofai"; + +customPolicies.add({ + name: "prefer-read-over-cat", + description: "Nudge toward the Read tool instead of shelling out to cat", + match: { events: ["PreToolUse"] }, + fn: async (ctx) => { + if (ctx.toolName !== "Bash") return allow(); + const command = String(ctx.toolInput?.command ?? "").trim(); + if (!/^(cat|head|tail|less|more)\s+\S+$/.test(command)) return allow(); + return instruct("Use the Read tool for single files — it is cheaper and gives line numbers."); + }, +}); +``` + +Caveat: `instruct` only injects properly on Claude Code, Devin and Antigravity. On Hermes, +Goose, OpenClaw and Pi it degrades to a stderr note. Fine for a local skill. + +## Sanitize tool output + +**Check the builtin param first:** `sanitize-api-keys` takes `additionalPatterns` +(`[{regex, label}]`), which covers most "scrub token X from output" requests with a config +edit and no code. Author a custom sanitizer only when that param can't express the need. + +**Know what sanitize actually does today:** the evaluator never consumes +`PolicyResult.message` (`traps.md` §9), so a sanitize deny **blocks the whole output** — +the model sees the block reason, not a redacted version. The secret stays protected; the +rest of that output is lost with it. Setting `message` anyway is harmless future-proofing, +and requires the raw object literal — the `deny()` helper cannot set it. + +```js +import { customPolicies, allow } from "failproofai"; + +const INTERNAL_URL_RE = /https:\/\/[a-z0-9-]+\.internal\.example\.com\/\S*/gi; + +customPolicies.add({ + name: "sanitize-internal-urls", + description: "Strip internal hostnames from tool output", + match: { events: ["PostToolUse"] }, + fn: async (ctx) => { + if (!INTERNAL_URL_RE.test(JSON.stringify(ctx.payload))) return allow(); + return { + decision: "deny", + reason: "Internal URL detected in tool output", + message: "[REDACTED: internal URL removed by failproofai]", + }; + }, +}); +``` + +Note `INTERNAL_URL_RE` has the `g` flag, which makes `.test()` stateful via `lastIndex`. +Either drop `g` or reset `lastIndex` between calls — a classic source of every-other-call +misses. + +## Gate the end of a turn + +```js +import { customPolicies, allow, deny } from "failproofai"; +import { execSync } from "node:child_process"; + +customPolicies.add({ + name: "require-changelog-entry", + description: "Every change must touch CHANGELOG.md before the turn ends", + match: { events: ["Stop"] }, + fn: async (ctx) => { + const cwd = ctx.session?.cwd; + if (!cwd) return allow(); + try { + const changed = execSync("git diff --name-only HEAD", { cwd, encoding: "utf8" }); + if (!changed.trim()) return allow(); + if (changed.includes("CHANGELOG.md")) return allow(); + return deny("Add a CHANGELOG.md entry under the current version heading before finishing."); + } catch { + return allow(); + } + }, +}); +``` + +**Stop policies are the dangerous kind.** A deny forces the agent to take another turn. If +the condition can never be satisfied in the current environment, it loops. Always: + +- wrap external calls in `try/catch` and `return allow()` on failure (fail-open) +- return `allow()` when there is nothing to check +- confirm the condition is actually reachable — this is exactly how the + `require-push-before-stop` loop happened in this repo (`traps.md` §6) + +Also mind the **10-second timeout**. A `Stop` policy shelling out to `gh` or the network can +blow it, and a timeout silently becomes `allow()`. + +## Configuration values + +`ctx.params` is always `{}` for custom policies (`traps.md` §8). Use module constants: + +```js +const PROTECTED = ["main", "master", "release"]; +``` + +Or read your own file inside `fn` if the values need to change without a code edit. diff --git a/.claude/skills/policy-author/references/rules-files.md b/.claude/skills/policy-author/references/rules-files.md new file mode 100644 index 00000000..6a298146 --- /dev/null +++ b/.claude/skills/policy-author/references/rules-files.md @@ -0,0 +1,110 @@ +# Turning a rules file into enforcement + +The full classification guide for SKILL.md §3. The governing idea: **a policy can only +match a tool call.** Every rule must first be translated into "which tool, carrying what +input, in what state" — a rule that cannot be phrased that way cannot be a policy, no +matter how important it is. + +## Classification by language cue + +| The file says | Class | Enforcement | Mode | +|---|---|---|---| +| "never run X", "do not use X" | hard rule | builtin if one matches, else `block-*` | `deny()` | +| "only use X (not Y)" | hard rule, param-shaped | usually a **parameterized builtin** — check before authoring | `deny()` via builtin | +| "always do X before Y" | workflow gate | PreToolUse on **Y**, checking X happened | `deny()` with instructions | +| "X must accompany Y" (changelog with PR) | workflow gate | PreToolUse on Y's command | `deny()` with instructions | +| "prefer X over Y", "avoid Y" | preference | `warn-*` nudge | `instruct()` | +| "confirm with me before X" | oversight | `warn-*`, STOP voice (`patterns.md`) | `instruct()` | +| "file/config must contain Z" | repo invariant | **a test in the test suite** | not a policy | +| "write clear code", tone, style | judgment | stays prose | not enforceable | + +## Worked examples + +**"Use bun. Do not use npm/yarn to install deps."** Param-shaped hard rule — the first +candidate is `prefer-package-manager` with params: + +```json +{ + "enabledPolicies": ["prefer-package-manager"], + "policyParams": { + "prefer-package-manager": { "allowed": ["bun"], "blocked": ["npm", "yarn"] } + } +} +``` + +(Remember: params for a policy absent from `enabledPolicies` do nothing — `traps.md` §7.) + +**But check the builtin's matching breadth before enabling it.** A builtin can be broader +than the rule. `prefer-package-manager`'s npm matcher is a bare `\bnpm\b`, so with +`blocked: ["npm"]` it also denies `npm pack`, `npm view`, `npm ls` — and if the repo's own +docs mandate one of those (this repo's testing protocol requires `npm pack +--ignore-scripts`), the builtin over-blocks a documented workflow. The test is cheap: run +the repo's own legitimate commands through the hook with the builtin enabled in a sandbox. +If a legitimate use gets denied and no param can carve it out, a scoped custom policy +(match the install-family subcommands only) is the right call — say why in the report, +since it is an exception to builtin-first, not the norm. + +**"Every PR must include a CHANGELOG.md update."** Workflow gate. Enforce at the action it +gates — `gh pr create` — not at Stop: + +```js +// derived-from: CLAUDE.md § "Changelog" — "Every PR must include an update to +// CHANGELOG.md" (extracted 2026-07-24) +customPolicies.add({ + name: "require-changelog-in-pr", + description: "gh pr create must not run unless the branch touches CHANGELOG.md", + match: { events: ["PreToolUse"] }, + fn: async (ctx) => { + if (ctx.toolName !== "Bash") return allow(); + const command = String(ctx.toolInput?.command ?? ""); + if (!/\bgh\s+pr\s+create\b/.test(command)) return allow(); + const cwd = ctx.session?.cwd; + if (!cwd) return allow(); + try { + const changed = execSync("git diff --name-only origin/main...HEAD", { + cwd, encoding: "utf8", timeout: 5000, + }); + if (changed.includes("CHANGELOG.md")) return allow(); + return deny("This PR has no CHANGELOG.md entry. Add one under the current version heading, commit it, then create the PR."); + } catch { + return allow(); // can't tell -> fail open, never trap the agent + } + }, +}); +``` + +Why PreToolUse-on-the-action beats a Stop gate for these: the denial lands at the exact +moment of the violation with a precise fix, and there is no retry-loop risk when the +condition is unsatisfiable (`traps.md` §6). + +**"Configs must use the launcher form / `$CLAUDE_PROJECT_DIR` paths."** Repo invariant — +it describes file *contents*, not agent *behavior*. A policy would have to intercept every +Write and re-parse the config; a unit test reads the file directly and fails CI. Recommend +the test. (This repo's `dogfood-configs.test.ts` is exactly that solution.) + +## Extraction discipline + +- Quote each rule **verbatim** and keep its section heading — paraphrases drift, and the + provenance comment needs the exact source. +- One rule can produce two enforcements (a deny AND a nudge for the softer half). One + enforcement can cover several rules. Do not force 1:1. +- Rules about *people* ("ask the team lead before…") are out of scope — policies gate + agents, not humans. + +## Provenance and drift + +Every generated file carries, per rule: + +```js +// derived-from: § "" — "" (extracted ) +``` + +When the rules file changes, the comments say which policies to revisit. There is no +automatic sync — say so in the report rather than implying the enforcement tracks the file. + +## The honest split + +The report ends with four lists — enforced / already covered / nudged / left as prose. +Typical real-world files convert well under half their rules. If your extraction claims +everything became enforceable, the invariants and judgment calls were misclassified, and +the user now believes prose is protection. diff --git a/.claude/skills/policy-author/references/traps.md b/.claude/skills/policy-author/references/traps.md new file mode 100644 index 00000000..329e96a8 --- /dev/null +++ b/.claude/skills/policy-author/references/traps.md @@ -0,0 +1,207 @@ +# Silent failure modes + +Every item here is a documented, already-been-hit failure where a policy looks installed and +enforces nothing. Read this before reporting any policy as working. + +## 1. The filename convention + +`CONVENTION_FILE_RE = /policies\.(js|mjs|ts)$/` — `custom-hooks-loader.ts`, grep `CONVENTION_FILE_RE`. + +A file in `.failproofai/policies/` named `block-foo.mjs` is **silently skipped**. Only names +ending in `policies.js`, `policies.mjs` or `policies.ts` load. + +| Name | Loads? | +|---|---| +| `block-foo-policies.mjs` | yes | +| `security-policies.mjs` | yes | +| `policies.mjs` | yes | +| `block-foo.mjs` | **no** | +| `foo-policy.mjs` | **no** (singular) | + +This is not hypothetical. From the source comment at custom-hooks-loader.ts — grep findSkippedPolicyFiles): + +> This repo shipped `block-version-bumps.mjs` that way, so the guard written after a bad +> version bump had never once run. + +`findSkippedPolicyFiles()` (custom-hooks-loader.ts — grep findSkippedPolicyFiles)) exists solely to catch this, but its warnings go +only to the hook log, which nobody reads. **Always check the filename before anything else.** + +## 2. `customPoliciesEnabled: false` is a silent no-op (does NOT disable anything) + +The config key exists, `configure-wizard` writes it, and the loader checks it — but the +value never reaches the loader, so **setting it to `false` has no effect**. Convention +policies load regardless. + +The break is in `readMergedHooksConfig` (`hooks-config.ts` (grep `return {` inside `readMergedHooksConfig`)), which hand-builds its +return object from four keys and omits this one: + +```ts +return { + enabledPolicies: [...enabledSet], + ...(… ? { policyParams: mergedParams } : {}), + ...(customPoliciesPath !== undefined ? { customPoliciesPath } : {}), + ...(llm !== undefined ? { llm } : {}), +}; // customPoliciesEnabled is never propagated +``` + +So `handler.ts`, grep `customPoliciesEnabled:` passes `undefined`, and `custom-hooks-loader.ts`, grep `conventionEnabled` +(`opts?.customPoliciesEnabled !== false`) evaluates **true**, always. + +Verified empirically: a temp project with `"customPoliciesEnabled": false` plus a canary +policy still fires the canary. Do not tell a user their policies are disabled because this +flag is set — check by running the hook. + +The practical consequence for triage: a repo with this flag set is **still enforcing** its +custom policies. Treat the flag as unreliable in both directions until it is fixed. + +Note the explicit `customPoliciesPath` config key is **not** gated by this flag +(custom-hooks-loader.ts — grep customPoliciesPath)) — only convention discovery is, in intent. + +## 3. Everything is fail-open + +| Failure | Result | +|---|---| +| File not found | zero hooks, logged only | +| Syntax error | zero hooks, logged only | +| Throw at import time | zero hooks, logged only | +| Throw inside `fn` | `{decision: "allow"}` | +| `fn` exceeds 10s | `{decision: "allow"}` | + +Classified for telemetry as `module_not_found` / `syntax_error` / `runtime_error` +(custom-hooks-loader.ts — grep errorType)), then swallowed. Nothing fails loudly. + +The consequence: **silence is not success.** A policy that was never loaded and a policy +that correctly allowed an action produce identical observable behavior. This is why the +verification step in SKILL.md §2.4 is mandatory, not optional. + +## 4. A deny in your test does not prove *your* policy denied + +Every enabled policy sees every matching event. When a test case blocks, some policy blocked +— not necessarily yours. A rule that matches nothing can sit behind a green suite +indefinitely. + +Observed live: a heredoc case written to test a "no hardcoded localhost" policy passed, +but the deny came from the builtin `protect-env-vars` reacting to the word `export` in the +payload. The localhost rule was entirely unproven while reporting green. + +``` +cat < a.js +export const API = "http://localhost:3000"; +EOF +→ deny: "Command exports environment variable" ← protect-env-vars, not your policy +``` + +**Always attribute the deny.** The reason string names the responsible policy's message, so +assert on it rather than on the fact that something blocked: + +```js +if (got.decision !== "deny" || !got.reason.includes("")) + fail("blocked, but not by this policy"); +``` + +Two cheap ways to avoid the trap: + +- Write test payloads that avoid unrelated triggers — no `export`, no `sudo`, no `.env`, no + `curl … | sh` unless that is what you are testing. +- Test against a config with `enabledPolicies: []` so nothing else can fire. This is what + `test-policy.mjs --policy ` does — its sandbox config enables zero builtins, so any + deny is necessarily yours. + +## 5. Validation is essentially nil + +`failproofai p -i -c ` (`manager.ts`, grep `Validated`) executes the file and counts +`customPolicies.add()` calls. It does **not** check: + +- that the hook object has the right shape +- that `name` is unique (duplicates silently coexist) +- that `match.events` are real event names +- that `fn` returns a valid `PolicyResult` + +`customPolicies.add()` is a bare array push with no validation +(`custom-hooks-registry.ts`, grep `add(hook`). "Validated 1 custom hook(s)" means "the file ran and +called add() once" — nothing more. + +## 6. A Stop gate that cannot be satisfied loops forever + +Denying at `Stop` forces another turn. If the condition can never become true in the current +environment, the agent retries, fails, and never exits. Before enabling any +`require-*-before-stop` builtin — or authoring a Stop policy — check the condition is +actually reachable **in the project you are working in**: + +| Gate | Reachable only if | +|---|---| +| `require-commit-before-stop` | always (local git) | +| `require-push-before-stop` | a remote exists **and** credentials work | +| `require-pr-before-stop` | `gh` authenticated, remote is GitHub | +| `require-no-conflicts-before-stop` | base branch fetchable | +| `require-ci-green-before-stop` | CI configured **and** runs on push | + +Cheap check: + +```bash +git remote -v # no output → push/PR/CI gates cannot pass +git push --dry-run 2>&1 | head -2 # auth failure → same +gh auth status 2>&1 | head -3 +``` + +**This does not generalise across projects.** A gate that loops in one repo is correct in +another. Evaluate per project rather than carrying a verdict over. + +Concrete instance: the failproofai repo itself has no push credentials, so its five Stop +gates were deliberately removed from `.failproofai/policies-config.json` after exactly this +loop. That is a fact about that repo, not a rule about Stop gates. + +The same reachability rule applies to custom Stop policies — always `try/catch` external +calls and `return allow()` on failure, so an unavailable tool degrades to letting the turn +end rather than trapping it. + +## 7. Builtins are enabled by presence, not by a flag + +`enabledPolicies` is a `string[]`. Omission means off. There is no +`{"block-rm-rf": false}` form — to disable, remove the string. + +Params live in a **sibling** `policyParams` object keyed by the same short name, not nested +inside the policy entry: + +```json +{ + "enabledPolicies": ["block-read-outside-cwd"], + "policyParams": { + "block-read-outside-cwd": { "allowPaths": ["/tmp"] } + } +} +``` + +A param set for a policy that is not in `enabledPolicies` does nothing. + +## 8. `ctx.params` is always empty for custom policies + +`PolicyContext` exposes `params`, and `policies-config.json` has a `policyParams` map — so it +looks like a custom policy can be configured from config. It cannot. + +`POLICY_PARAMS_MAP` (`policy-evaluator.ts`, grep `POLICY_PARAMS_MAP`) is built **only** from `BUILTIN_POLICIES` +entries that declare a `params` schema. A custom hook has no schema, so it falls to the +else-branch at policy-evaluator.ts — grep without schema get empty params): + +```ts +// Custom hooks and policies without schema get empty params +ctx = { ...baseCtx, params: {} }; +``` + +Adding `policyParams: { "my-custom-policy": {...} }` to config does nothing — silently. + +**Workaround:** hardcode the values as module constants in the policy file, or read your own +config file / env var inside `fn`. The file is real JS, so anything is available. + +## 9. Sanitizers block, they do not redact — and `deny()` cannot set `message` anyway + +Two layered surprises. First, the `message` field of `PolicyResult` is not settable through +the exported `deny()` helper — a sanitizer needs the raw object literal. Second, and bigger: +**the evaluator never consumes `message` at all** (verified live 2026-07-24 — the deny +response in `policy-evaluator.ts` is built from `reason` only). The builtins' own +`[REDACTED: …]` messages are dead code. + +So a sanitize deny on `PostToolUse` blocks the *entire* tool output; the model sees the +block reason, never a redacted version. Protection holds — by omission — but do not tell a +user "the token is scrubbed and the rest passes through." Put anything the agent needs into +`reason`, and prefer `sanitize-api-keys.additionalPatterns` over authoring. See `api.md`. diff --git a/.claude/skills/policy-author/scripts/sync-builtins.mjs b/.claude/skills/policy-author/scripts/sync-builtins.mjs new file mode 100644 index 00000000..cb35ce71 --- /dev/null +++ b/.claude/skills/policy-author/scripts/sync-builtins.mjs @@ -0,0 +1,144 @@ +#!/usr/bin/env node +/** + * Regenerate references/builtins.md from the live BUILTIN_POLICIES registry. + * + * node sync-builtins.mjs rewrite the reference file + * node sync-builtins.mjs --check exit 1 if it is out of date (for CI) + * + * The reference file is a convenience snapshot. It drifts the moment a builtin + * is added, renamed, or has its default flipped — and a stale list is worse than + * no list, because it is quietly authoritative. Run this after any change to + * builtin-policies.ts. + * + * Resolves the registry from the repo checkout first, then from an installed + * failproofai package, so it works inside the repo and from a user's project. + */ +import { readFileSync, writeFileSync, existsSync } from "node:fs"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; + +const HERE = dirname(fileURLToPath(import.meta.url)); +const OUT = resolve(HERE, "..", "references", "builtins.md"); +const check = process.argv.includes("--check"); + +/** Walk up for a repo checkout, then fall back to node_modules. */ +function findRegistry() { + let dir = HERE; + for (let i = 0; i < 8; i++) { + const src = join(dir, "src", "hooks", "builtin-policies.ts"); + if (existsSync(src)) return src; + const dep = join(dir, "node_modules", "failproofai", "src", "hooks", "builtin-policies.ts"); + if (existsSync(dep)) return dep; + const parent = dirname(dir); + if (parent === dir) break; + dir = parent; + } + return null; +} + +const registryPath = findRegistry(); +if (!registryPath) { + console.error("Could not locate builtin-policies.ts (looked in the repo and node_modules)."); + console.error("Run this from inside the failproofai repo, or a project with failproofai installed."); + process.exit(2); +} + +// TypeScript source — needs a TS-capable runtime. Bun handles it directly. +let BUILTIN_POLICIES; +try { + ({ BUILTIN_POLICIES } = await import(pathToFileURL(registryPath).href)); +} catch (err) { + console.error(`Could not import ${registryPath}: ${err.message}`); + console.error("This script needs a TypeScript-capable runtime — try: bun sync-builtins.mjs"); + process.exit(2); +} + +const byCategory = new Map(); +for (const p of BUILTIN_POLICIES) { + if (!byCategory.has(p.category)) byCategory.set(p.category, []); + byCategory.get(p.category).push(p); +} + +const lines = []; +lines.push(`# Builtin policies (${BUILTIN_POLICIES.length})`); +lines.push(""); +lines.push("**Generated — do not hand-edit.** Regenerate with:"); +lines.push(""); +lines.push("```bash"); +lines.push("bun .claude/skills/policy-author/scripts/sync-builtins.mjs"); +lines.push("```"); +lines.push(""); +lines.push("This snapshot goes stale whenever a builtin is added, renamed, or has its default"); +lines.push("flipped. When it matters, ask the CLI instead — it is always current:"); +lines.push(""); +lines.push("```bash"); +lines.push("failproofai policies # every policy, with enabled status and params"); +lines.push("```"); +lines.push(""); +lines.push("## How to use this for triage"); +lines.push(""); +lines.push("Enabling a builtin beats writing a custom policy: nothing to maintain, no naming"); +lines.push("trap, no fail-open risk, and it ships with tests."); +lines.push(""); +lines.push("Before concluding \"no builtin covers this\", check whether a **parameterized** one"); +lines.push("does — several take allowlists or thresholds that widen their scope considerably."); +lines.push("Params go in the `policyParams` map, keyed by short name."); +lines.push(""); +lines.push("To enable: add the short name to `enabledPolicies` in `.failproofai/policies-config.json`."); +lines.push(""); +lines.push("---"); + +for (const [category, policies] of byCategory) { + lines.push(""); + lines.push(`### ${category}`); + lines.push(""); + lines.push("| Policy | Default | Events | What it catches |"); + lines.push("|---|---|---|---|"); + for (const p of policies) { + const events = (p.match?.events ?? []).join(", ") || "—"; + const params = p.params ? ` _(params: ${Object.keys(p.params).join(", ")})_` : ""; + const beta = p.beta ? " _(beta)_" : ""; + lines.push( + `| \`${p.name}\` | ${p.defaultEnabled ? "**on**" : "off"} | ${events} | ${p.description}${params}${beta} |`, + ); + } +} + +lines.push(""); +lines.push("> The five `require-*-before-stop` policies gate the end of a turn. A gate whose"); +lines.push("> condition cannot be met in the current project loops forever — see `traps.md` §6"); +lines.push("> before enabling one."); +lines.push(""); +lines.push("---"); +lines.push(""); +lines.push("## Audit-only detectors"); +lines.push(""); +lines.push("These have no real-time builtin equivalent, so they are the prime candidates for"); +lines.push("custom policies. List them from source with:"); +lines.push(""); +lines.push("```bash"); +lines.push("bun -e 'const {AUDIT_DETECTORS}=await import(\"./src/audit/detectors/index.ts\");"); +lines.push("for (const d of AUDIT_DETECTORS) console.log(d.name, \"|\", d.category+\"/\"+d.severity, \"|\", d.description)'"); +lines.push("```"); +lines.push(""); +lines.push("All but `reread-after-edit` are Bash-command patterns, so a `PreToolUse` policy"); +lines.push("filtering on `ctx.toolName === \"Bash\"` and matching `ctx.toolInput.command` covers"); +lines.push("most of them. `reread-after-edit` needs cross-call session state, which hooks cannot"); +lines.push("see — that one needs a builtin, not a custom policy."); +lines.push(""); + +const generated = lines.join("\n"); + +if (check) { + const current = existsSync(OUT) ? readFileSync(OUT, "utf8") : ""; + if (current !== generated) { + console.error("references/builtins.md is OUT OF DATE."); + console.error(`Registry has ${BUILTIN_POLICIES.length} builtins. Run: bun ${process.argv[1]}`); + process.exit(1); + } + console.log(`references/builtins.md is current (${BUILTIN_POLICIES.length} builtins).`); + process.exit(0); +} + +writeFileSync(OUT, generated); +console.log(`Wrote ${OUT} — ${BUILTIN_POLICIES.length} builtins across ${byCategory.size} categories.`); diff --git a/.claude/skills/policy-author/scripts/test-policy.mjs b/.claude/skills/policy-author/scripts/test-policy.mjs new file mode 100644 index 00000000..0afe7d82 --- /dev/null +++ b/.claude/skills/policy-author/scripts/test-policy.mjs @@ -0,0 +1,207 @@ +#!/usr/bin/env node +/** + * Run a policy against a synthetic hook payload and report the decision. + * + * Exists because loading and execution are both fail-open: a policy that was + * never loaded, threw, or timed out is indistinguishable from one that + * correctly allowed. The only way to know a policy works is to make it fire. + * + * node test-policy.mjs --policy --event PreToolUse --tool Bash \ + * --input '{"command":"cd /x && ls"}' --expect instruct + * + * node test-policy.mjs --cases cases.json + * + * With --policy the policy is copied into a throwaway project (which also acts + * as HOME) so neither the real project config, `customPoliciesEnabled: false`, + * nor user-scope policies can affect the result. Without it, the payload runs + * against the current directory's real config — useful for testing builtins. + */ +import { spawnSync } from "node:child_process"; +import { mkdtempSync, mkdirSync, copyFileSync, writeFileSync, existsSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { resolve, dirname, basename, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +// ---------------------------------------------------------------- arg parsing + +function parseArgs(argv) { + const out = {}; + for (let i = 0; i < argv.length; i++) { + const a = argv[i]; + if (!a.startsWith("--")) continue; + const key = a.slice(2); + const next = argv[i + 1]; + if (next === undefined || next.startsWith("--")) out[key] = true; + else { out[key] = next; i++; } + } + return out; +} + +const args = parseArgs(process.argv.slice(2)); + +if (args.help || (!args.cases && !args.event)) { + console.log(` +Usage: + test-policy.mjs --policy --event [--tool ] \\ + --input '' [--expect deny|allow|instruct] [--cwd ] + test-policy.mjs --cases + + --policy Policy file to test in isolation. Omit to test the current + directory's real config (builtins). + --event Hook event, e.g. PreToolUse, PostToolUse, Stop. + --tool Canonical tool name: Bash, Read, Write, Edit, Grep. + --input Tool input as JSON, e.g. '{"command":"sudo ls"}'. + --expect Assert the outcome. Exits 1 on mismatch. + --cwd cwd reported to the policy. Defaults to the sandbox / process cwd. + +cases.json is an array of objects with the same keys minus --policy: + [{ "name": "blocks sudo", "event": "PreToolUse", "tool": "Bash", + "input": { "command": "sudo ls" }, "expect": "deny" }] +`.trim()); + process.exit(args.help ? 0 : 1); +} + +// ------------------------------------------------------------------- runner + +/** Walk up looking for this repo's dev launcher; fall back to the published CLI. */ +function findRunner() { + let dir = dirname(fileURLToPath(import.meta.url)); + for (let i = 0; i < 8; i++) { + const candidate = join(dir, "scripts", "dev-hook.mjs"); + if (existsSync(candidate)) return { cmd: "node", pre: [candidate] }; + const parent = dirname(dir); + if (parent === dir) break; + dir = parent; + } + return { cmd: "npx", pre: ["-y", "failproofai"] }; +} + +const runner = findRunner(); + +/** + * Build a throwaway project that is also HOME, so project config, global + * config and user-scope policies all resolve inside it and nothing on the + * real machine leaks into the result. + */ +function makeSandbox(policyPath) { + const root = mkdtempSync(join(tmpdir(), "failproofai-policy-test-")); + mkdirSync(join(root, ".failproofai", "policies"), { recursive: true }); + + // The convention regex is /policies\.(js|mjs|ts)$/ — a file that does not + // match is silently skipped, so normalize the name rather than trusting it. + let name = basename(policyPath); + if (!/policies\.(js|mjs|ts)$/.test(name)) { + name = name.replace(/\.(js|mjs|ts)$/, "-policies.$1"); + console.log(` note: renamed to ${name} to satisfy the loader convention`); + } + copyFileSync(policyPath, join(root, ".failproofai", "policies", name)); + writeFileSync( + join(root, ".failproofai", "policies-config.json"), + JSON.stringify({ enabledPolicies: [] }, null, 2), + ); + return root; +} + +/** + * Deny is reported in several different shapes depending on event and CLI. + * Missing one silently downgrades a real deny to "allow", which is the exact + * failure this script exists to catch — so handle all of them. + * + * PreToolUse (Claude) hookSpecificOutput.permissionDecision === "deny" + * PermissionRequest hookSpecificOutput.decision.behavior === "deny" + * instruct hookSpecificOutput.additionalContext + * Copilot/Goose/Devin/… { decision: "block" } + * Cursor/Pi flat { permission: "deny" } + * Factory non-Stop exit code 2, reason on stderr + */ +function classify(stdout, exitCode, stderr) { + const text = stdout.trim(); + if (!text) { + if (exitCode === 2) return { decision: "deny", reason: (stderr ?? "").trim() }; + return { decision: "allow", reason: "" }; + } + let parsed; + try { parsed = JSON.parse(text); } catch { return { decision: "unparseable", reason: text }; } + const h = parsed.hookSpecificOutput ?? {}; + if (h.permissionDecision === "deny") return { decision: "deny", reason: h.permissionDecisionReason ?? "" }; + if (h.decision?.behavior === "deny") return { decision: "deny", reason: h.decision.message ?? "" }; + if (h.additionalContext) return { decision: "instruct", reason: h.additionalContext }; + if (parsed.decision === "block") return { decision: "deny", reason: parsed.reason ?? "" }; + if (parsed.permission === "deny") return { decision: "deny", reason: parsed.reason ?? "" }; + if (exitCode === 2) return { decision: "deny", reason: (stderr ?? "").trim() }; + return { decision: "allow", reason: "" }; +} + +function runCase(c, sandbox) { + const cwd = c.cwd ?? sandbox ?? process.cwd(); + const payload = { + hook_event_name: c.event, + session_id: "policy-test", + transcript_path: "/dev/null", + cwd, + }; + if (c.tool) payload.tool_name = c.tool; + if (c.input) { + // `{{cwd}}` expands to the effective cwd, so cases can reference the + // sandbox path (generated per run) in commands like `cd {{cwd}} && ls`. + const raw = typeof c.input === "string" ? c.input : JSON.stringify(c.input); + payload.tool_input = JSON.parse(raw.split("{{cwd}}").join(cwd)); + } + if (c.event === "Stop") payload.stop_hook_active = false; + + const env = { ...process.env, FAILPROOFAI_TELEMETRY_DISABLED: "1" }; + if (sandbox) env.HOME = sandbox; + + const res = spawnSync(runner.cmd, [...runner.pre, "--hook", c.event], { + input: JSON.stringify(payload), + encoding: "utf8", + cwd, + env, + timeout: 20000, + }); + + if (res.error) return { decision: "error", reason: String(res.error) }; + return classify(res.stdout ?? "", res.status, res.stderr); +} + +// --------------------------------------------------------------------- main + +let cases = args.cases + ? JSON.parse(await import("node:fs").then((fs) => fs.readFileSync(resolve(args.cases), "utf8"))) + : [{ name: `${args.event}${args.tool ? " " + args.tool : ""}`, event: args.event, tool: args.tool, + input: args.input, expect: args.expect, cwd: args.cwd }]; + +// A command-line --cwd is the default for every case that does not set its own. +// Without this, --cwd is silently ignored alongside --cases, and policies that +// inspect real filesystem or git state quietly evaluate against the empty +// sandbox instead — producing allow-everything results that look like failures +// of the policy rather than of the harness. +if (args.cwd) cases = cases.map((c) => ({ ...c, cwd: c.cwd ?? args.cwd })); + +let sandbox = null; +if (args.policy) { + const p = resolve(args.policy); + if (!existsSync(p)) { console.error(`Policy file not found: ${p}`); process.exit(1); } + sandbox = makeSandbox(p); +} + +let failed = 0; +try { + for (const c of cases) { + const got = runCase(c, sandbox); + const label = c.name ?? `${c.event} ${c.tool ?? ""}`.trim(); + if (!c.expect) { + console.log(` ${label}\n → ${got.decision}${got.reason ? ": " + got.reason : ""}`); + continue; + } + const ok = got.decision === c.expect; + if (!ok) failed++; + console.log(` ${ok ? "PASS" : "FAIL"} ${label} (expected ${c.expect}, got ${got.decision})`); + if (got.reason) console.log(` ${got.reason.slice(0, 160)}`); + } +} finally { + if (sandbox) rmSync(sandbox, { recursive: true, force: true }); +} + +if (failed) { console.log(`\n${failed} of ${cases.length} failed.`); process.exit(1); } +if (cases.some((c) => c.expect)) console.log(`\nAll ${cases.length} passed.`); diff --git a/.failproofai/policies/audit-skill-policies.mjs b/.failproofai/policies/audit-skill-policies.mjs new file mode 100644 index 00000000..1f6127dd --- /dev/null +++ b/.failproofai/policies/audit-skill-policies.mjs @@ -0,0 +1,39 @@ +/** + * audit-skill-policies.mjs — point the agent at the policy-author skill after + * an audit run. + * + * A skill cannot invoke itself; it is pulled in when the model matches a + * request against the skill description. That covers "fix these findings", + * but not the case where the user just runs `failproofai audit` and looks at + * the output. This closes that gap using failproofai's own policy engine. + * + * Repo scope on purpose: the policy-author skill lives in this repo's + * .claude/skills/, so advertising it from other projects would point at + * something that is not there. + */ +import { customPolicies, allow, instruct } from "failproofai"; + +customPolicies.add({ + name: "suggest-policy-author-after-audit", + description: "After `failproofai audit`, point the agent at the policy-author skill", + match: { events: ["PostToolUse"] }, + fn: async (ctx) => { + if (ctx.toolName !== "Bash") return allow(); + const command = String(ctx.toolInput?.command ?? ""); + + // Match the audit subcommand only — not `policies`, `config`, or a bare + // `--version`. Tolerates the npx / dev-hook / bare-binary invocations. + if (!/\bfailproofai\b[^\n|;&]*\baudit\b/.test(command)) return allow(); + + // `--help` is not a run, and there are no findings to triage. + if (/\B--help\b|\B-h\b/.test(command)) return allow(); + + return instruct( + "An audit just ran. Findings are cached at ~/.failproofai/audit-dashboard.json " + + "(the AuditResult is nested under `.result.results[]`). Use the policy-author " + + "skill to triage them: enable builtins that already cover a finding, and author " + + "custom policies only for what nothing covers. Re-read the live config rather " + + "than trusting each finding's `enabledInConfig`, which is a stale snapshot.", + ); + }, +}); diff --git a/.gitignore b/.gitignore index e52e93d3..024daa99 100644 --- a/.gitignore +++ b/.gitignore @@ -42,6 +42,7 @@ next-env.d.ts # claude .claude/* !.claude/settings.json +!.claude/skills/ # cursor .cursor/* diff --git a/CHANGELOG.md b/CHANGELOG.md index ef6c44ea..39ca95f8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,14 @@ # Changelog +## 0.0.15-beta.0 — 2026-07-27 + +### Features +- Add a `policy-author` skill (`.claude/skills/policy-author/`) that turns the two things users already have — an audit report and a prose rules file — into enforced policies. It exists because the audit's fix vocabulary is closed over the builtins: `DETECTOR_TO_POLICY` maps every audit-only detector to some builtin, and its own header comment says why ("so every finding looks like it has a failproofai fix"), but several of those mappings prevent nothing (`git-commit-no-verify` → `warn-git-amend` is a different command), and nothing in the audit can say "this one needs a policy you don't have yet" or notice that a hand-written policy already covers it. Four entry paths converge on one authoring core: the audit trigger below, a plain-language complaint ("agents keep force-pushing" — users report problems, they don't request policies), an explicit ask, and a rules file (`CLAUDE.md`/`AGENTS.md`) whose rules keep getting ignored. The load-bearing rule is that checking coverage comes *before* authoring — including **parameterized** builtins and the project's existing custom policies — because most requests are one line in `enabledPolicies`, not a new file; blind agents given two tasks with deliberately opposite correct answers diverged as intended, one enabling `block-sudo` and writing zero files while the other authored a policy and touched zero config. Authoring picks one of three modes matching the builtins' own naming (`block-*`/`deny()`, `warn-*`/`instruct()` carrying the builtins' "STOP: … Confirm with the user before executing" voice, `sanitize-*`), defaulting to oversight whenever an action has any legitimate use, since blocking those just gets the policy disabled. `references/traps.md` documents nine silent failures verified by running the hook rather than reading code — chief among them the loader convention (`block-foo.mjs` is skipped in silence; this repo shipped `block-version-bumps.mjs` that way and it never ran), that loading and execution are both fail-open so an unloaded policy and a correctly-allowing one are indistinguishable, and that a deny in a test does not prove *your* policy denied. `scripts/test-policy.mjs` runs a policy against synthetic payloads in a sandbox that enables zero builtins (so any deny is necessarily yours) and handles all six deny shapes across the CLIs, including `PermissionRequest`'s `decision.behavior` and Factory's exit-2; `scripts/sync-builtins.mjs` regenerates the builtin reference from `BUILTIN_POLICIES` and `--check` exits 1 on drift. Validated across five rounds of blind agents in seeded sandboxes covering sixteen use-case cells — triage, complaint-to-builtin, custom authoring, oversight, sanitize, custom Stop gates, dead-policy diagnosis, and both directions of the scope guardrail (a question about machine-wide findings must not modify the machine; an explicit "do it globally" must not be refused). (#600) +- Point agents at that skill after an audit, via `.failproofai/policies/audit-skill-policies.mjs`. A skill cannot invoke itself — it is pulled in when a request matches its description — which covers "fix these findings" but not the case where someone simply runs `failproofai audit` and reads the output. This dogfood policy closes that gap with failproofai's own enforcement layer: a `PostToolUse` rule matching the audit subcommand returns an `instruct()` naming the cache path and the two things a fresh agent otherwise gets wrong (findings live at `.result.results[]` under an envelope, and `enabledInConfig` is a snapshot taken at audit time that goes stale — observed reporting all 39 builtins enabled five minutes after the global config was emptied). Guarded against the obvious false positives: `npm audit`, `grep -r audit` and `failproofai audit --help` all pass through. + +### Fixes +- Track `.claude/skills/` instead of ignoring it. `.gitignore`'s `.claude/*` rule excluded everything but `settings.json`, so a skill committed to this repo would silently never reach the tree; the carve-out follows the existing `!.claude/settings.json` pattern. + ## 0.0.15-beta.0 — 2026-07-23 ### Dependencies