-
Notifications
You must be signed in to change notification settings - Fork 31
Add a policy-author skill that turns audit findings and prose rules files into enforced policies #600
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a policy-author skill that turns audit findings and prose rules files into enforced policies #600
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,168 @@ | ||
| # Policy API reference | ||
|
|
||
| Everything here is exported from `src/index.ts` (19 lines — that is the entire public | ||
| surface): | ||
|
|
||
| ```ts | ||
| export { customPolicies, getCustomHooks, clearCustomHooks } from "./hooks/custom-hooks-registry"; | ||
| export { allow, deny, instruct } from "./hooks/policy-helpers"; | ||
| export type { PolicyContext, PolicyResult, CustomHook, PolicyDecision, PolicyFunction } from "./hooks/policy-types"; | ||
| ``` | ||
|
|
||
| ## The policy object | ||
|
|
||
| `CustomHook` — `src/hooks/policy-types.ts`, grep `interface CustomHook`: | ||
|
|
||
| ```ts | ||
| export interface CustomHook { | ||
| name: string; | ||
| description?: string; | ||
| match?: { | ||
| events?: HookEventType[]; | ||
| }; | ||
| fn: (ctx: PolicyContext) => PolicyResult | Promise<PolicyResult>; | ||
| } | ||
| ``` | ||
|
|
||
| Registered with `customPolicies.add(hook)`. That is the whole registration surface — no | ||
| remove, no update, and **no validation of any kind** — `custom-hooks-registry.ts` (grep | ||
| `getRegistry`) is a bare array push. | ||
|
|
||
| ## Context | ||
|
|
||
| `PolicyContext` — `policy-types.ts`, grep `interface PolicyContext`: | ||
|
|
||
| ```ts | ||
| export interface PolicyContext { | ||
| eventType: HookEventType; | ||
| payload: Record<string, unknown>; | ||
| toolName?: string; | ||
| toolInput?: Record<string, unknown>; | ||
| session?: SessionMetadata; | ||
| params?: Record<string, unknown>; | ||
| cli?: IntegrationType; // which agent CLI fired this; mirrors session.cli | ||
| } | ||
| ``` | ||
|
|
||
| - `ctx.toolInput` holds the tool's arguments — `command` for Bash, `file_path`/`content` for | ||
| Write, `old_string`/`new_string` for Edit, `pattern` for Grep. These keys are already | ||
| canonicalized across all 11 supported CLIs, so you write them once. | ||
| - `ctx.session?.cwd` is the working directory. | ||
| - `ctx.payload` is the raw hook payload if you need something not surfaced above. | ||
|
|
||
| ## Decisions | ||
|
|
||
| `PolicyResult` — `policy-types.ts`, grep `interface PolicyResult`: | ||
|
|
||
| ```ts | ||
| export interface PolicyResult { | ||
| decision: "allow" | "deny" | "instruct"; | ||
| reason?: string; | ||
| message?: string; | ||
| } | ||
| ``` | ||
|
|
||
| Helpers (`policy-helpers.ts`, the complete file): | ||
|
|
||
| ```ts | ||
| export function allow(reason?: string): PolicyResult { ... } // reason optional | ||
| export function deny(reason: string): PolicyResult { ... } // reason required | ||
| export function instruct(reason: string): PolicyResult { ... } // reason required | ||
| ``` | ||
|
|
||
| - **`allow`** — let it through. Also the correct return when your policy does not apply. | ||
| - **`deny`** — block it. `reason` is shown to the agent and the user. | ||
| - **`instruct`** — let it through but inject guidance into the agent's next turn. Only | ||
| properly supported on Claude Code, Devin and Antigravity; degrades to a stderr note on | ||
| Hermes, Goose, OpenClaw and Pi. Fine for a local skill; do not rely on it if the policy | ||
| will be distributed. | ||
|
|
||
| ### The `message` field is currently inert — sanitize works by blocking, not replacing | ||
|
|
||
| The builtin sanitizers return `deny` plus a `message` that *looks like* a replacement — | ||
| `sanitizeJwt` in `builtin-policies.ts`: | ||
|
|
||
| ```ts | ||
| return { | ||
| decision: "deny", | ||
| reason: "JWT token detected in tool output", | ||
| message: "[REDACTED: JWT token removed by failproofai]", | ||
| }; | ||
| ``` | ||
|
|
||
| **But the evaluator never consumes `PolicyResult.message`** (verified 2026-07-24: the deny | ||
| response in `policy-evaluator.ts` is built from `reason` alone; the only `.message` read in | ||
| that file is `err.message`). What actually happens on a sanitize deny: the tool output is | ||
| **blocked entirely** and the model sees the block reason instead. The secret is still | ||
| protected — by omission, not redaction — but the model also loses the rest of that output. | ||
| Do not promise users "the output is scrubbed and the rest passes through"; it is not. | ||
|
|
||
| Two practical consequences: | ||
| - Put everything useful into `reason` — it is the only channel that surfaces. | ||
| - For output scrubbing, check `sanitize-api-keys`'s **`additionalPatterns` param first** | ||
| (`{regex, label}` entries) — builtin-first applies to sanitizers too. A custom sanitizer | ||
| is only needed when the pattern-per-output-shape doesn't fit that param. | ||
|
|
||
| The `deny()` helper cannot set `message` anyway; if you do set it (future-proofing for when | ||
| the evaluator honors it), return the raw object literal. | ||
|
|
||
| ## Events | ||
|
|
||
| `HookEventType` — `src/hooks/types.ts`, grep `HOOK_EVENT_TYPES`. The ones worth knowing: | ||
|
|
||
| | Event | Fires | Can block? | | ||
| |---|---|---| | ||
| | `PreToolUse` | Before a tool runs | **Yes** — the main enforcement point | | ||
| | `PostToolUse` | After a tool returns | Yes — a deny blocks the whole output (see the `message` note above) | | ||
| | `UserPromptSubmit` | On user input | Yes | | ||
| | `Stop` | Agent about to finish its turn | Yes — deny forces another turn | | ||
| | `SessionStart` / `SessionEnd` | Session boundaries | Observation | | ||
| | `SubagentStop` | Subagent returns | Yes on most CLIs | | ||
| | `PermissionRequest` / `PermissionDenied` | Permission flow | Yes | | ||
|
|
||
| Full list also includes `PostToolUseFailure`, `StopFailure`, `Notification`, | ||
| `SubagentStart`, `TaskCreated`, `TaskCompleted`, `PreCompact`, `PostCompact`, `FileChanged`, | ||
| `CwdChanged`, `ConfigChange`, `WorktreeCreate`, `WorktreeRemove`, `Elicitation`, | ||
| `ElicitationResult`, `UserPromptExpansion`, `PostToolBatch`, `InstructionsLoaded`, | ||
| `TeammateIdle`, `Setup`. | ||
|
|
||
| ## Filtering by tool | ||
|
|
||
| `CustomHook.match` publicly declares **only `events`**. Filter on the tool inside `fn`: | ||
|
|
||
| ```js | ||
| fn: async (ctx) => { | ||
| if (ctx.toolName !== "Bash") return allow(); | ||
| ... | ||
| } | ||
| ``` | ||
|
|
||
| An undocumented `match.toolNames` does work at runtime (`handler.ts`, grep `hook.match ??` passes `match` | ||
| straight through, and `policy-registry.ts`, grep `getPoliciesForEvent` filters on it), but it is absent from the | ||
| public type and could be typed away. Prefer filtering in `fn`. | ||
|
|
||
| ## Execution model | ||
|
|
||
| - Custom policies run at **priority -1**, i.e. after all builtins. | ||
| - Each `fn` gets a **10-second timeout**. Timeout or throw → `{decision: "allow"}`. | ||
| - Namespaced as `custom/<name>`, `.failproofai-project/<name>` or `.failproofai-user/<name>`. | ||
|
|
||
| ## Configuration | ||
|
|
||
| `.failproofai/policies-config.json` — `HooksConfig`, `policy-types.ts`, grep `interface HooksConfig`: | ||
|
|
||
| ```ts | ||
| export interface HooksConfig { | ||
| enabledPolicies: string[]; // builtins, by short name | ||
| llm?: LlmConfig; | ||
| policyParams?: Record<string, Record<string, unknown>>; // keyed by short name | ||
| customPoliciesPath?: string; | ||
| customPoliciesEnabled?: boolean; // absent = enabled | ||
| } | ||
| ``` | ||
|
|
||
| Builtins are enabled **purely by presence** of the short name in `enabledPolicies` — there | ||
| is no per-policy enabled/disabled object, and omission means off. | ||
|
|
||
| Merged across three scopes, in precedence order: project `{cwd}/.failproofai/` → local → | ||
| global `~/.failproofai/` (`hooks-config.ts`, grep `readMergedHooksConfig`). | ||
|
Comment on lines
+164
to
+168
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win Document The skill and traps reference state that enabled policy names are unioned across scopes. “Merged ... in precedence order” suggests a project config can override a global enable, which is false and can leave a builtin active unexpectedly. Clarify merge semantics per field, especially for enforcement settings. 🤖 Prompt for AI Agents |
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,132 @@ | ||
| # Builtin policies (39) | ||
|
|
||
| **Generated — do not hand-edit.** Regenerate with: | ||
|
|
||
| ```bash | ||
| bun .claude/skills/policy-author/scripts/sync-builtins.mjs | ||
| ``` | ||
|
|
||
| This snapshot goes stale whenever a builtin is added, renamed, or has its default | ||
| flipped. When it matters, ask the CLI instead — it is always current: | ||
|
|
||
| ```bash | ||
| failproofai policies # every policy, with enabled status and params | ||
| ``` | ||
|
|
||
| ## How to use this for triage | ||
|
|
||
| Enabling a builtin beats writing a custom policy: nothing to maintain, no naming | ||
| trap, no fail-open risk, and it ships with tests. | ||
|
|
||
| Before concluding "no builtin covers this", check whether a **parameterized** one | ||
| does — several take allowlists or thresholds that widen their scope considerably. | ||
| Params go in the `policyParams` map, keyed by short name. | ||
|
|
||
| To enable: add the short name to `enabledPolicies` in `.failproofai/policies-config.json`. | ||
|
|
||
| --- | ||
|
|
||
| ### Sanitize | ||
|
|
||
| | Policy | Default | Events | What it catches | | ||
| |---|---|---|---| | ||
| | `sanitize-jwt` | **on** | PostToolUse | Stop Claude from reading JWTs in tool responses | | ||
| | `sanitize-api-keys` | **on** | PostToolUse | Stop Claude from reading API keys (OpenAI, Anthropic, GitHub, AWS, Stripe, Google) in tool responses _(params: additionalPatterns)_ | | ||
| | `sanitize-connection-strings` | **on** | PostToolUse | Stop Claude from reading database connection strings with embedded credentials in tool responses | | ||
| | `sanitize-private-key-content` | **on** | PostToolUse | Stop Claude from reading PEM private key content in tool responses | | ||
| | `sanitize-bearer-tokens` | **on** | PostToolUse | Stop Claude from reading Authorization Bearer tokens in tool responses | | ||
|
|
||
| ### Environment | ||
|
|
||
| | Policy | Default | Events | What it catches | | ||
| |---|---|---|---| | ||
| | `protect-env-vars` | **on** | PreToolUse | Prevent commands that read environment variables | | ||
| | `block-env-files` | **on** | PreToolUse | Block reading/writing .env files | | ||
| | `block-read-outside-cwd` | off | PreToolUse | Block file reads outside the session working directory _(params: allowPaths)_ | | ||
|
|
||
| ### Dangerous Commands | ||
|
|
||
| | Policy | Default | Events | What it catches | | ||
| |---|---|---|---| | ||
| | `block-sudo` | **on** | PreToolUse, PermissionRequest | Block sudo commands _(params: allowPatterns)_ | | ||
| | `block-curl-pipe-sh` | **on** | PreToolUse | Block piping downloads to shell | | ||
| | `block-rm-rf` | off | PreToolUse | Prevent catastrophic deletions _(params: allowPaths)_ | | ||
| | `block-failproofai-commands` | **on** | PreToolUse | Block failproofai CLI commands and uninstallation | | ||
| | `block-secrets-write` | off | PreToolUse | Block writing secret key files _(params: additionalPatterns)_ | | ||
|
|
||
| ### Infra Commands | ||
|
|
||
| | Policy | Default | Events | What it catches | | ||
| |---|---|---|---| | ||
| | `block-kubectl` | off | PreToolUse | Block kubectl commands (Kubernetes cluster mutations) _(params: allowPatterns)_ | | ||
| | `block-terraform` | off | PreToolUse | Block terraform and tofu (OpenTofu) commands _(params: allowPatterns)_ | | ||
| | `block-aws-cli` | off | PreToolUse | Block aws CLI commands _(params: allowPatterns)_ | | ||
| | `block-gcloud` | off | PreToolUse | Block gcloud (Google Cloud) CLI commands _(params: allowPatterns)_ | | ||
| | `block-az-cli` | off | PreToolUse | Block az (Azure) CLI commands _(params: allowPatterns)_ | | ||
| | `block-helm` | off | PreToolUse | Block helm commands _(params: allowPatterns)_ | | ||
| | `block-gh-pipeline` | off | PreToolUse | Block gh CLI pipeline-trigger subcommands (workflow run, run rerun/cancel, pr merge, release create/delete, cache delete, secret set/delete) _(params: allowPatterns)_ | | ||
|
|
||
| ### Git | ||
|
|
||
| | Policy | Default | Events | What it catches | | ||
| |---|---|---|---| | ||
| | `block-push-master` | **on** | PreToolUse | Block pushing to main/master _(params: protectedBranches)_ | | ||
| | `block-force-push` | off | PreToolUse | Prevent force-pushing to any branch | | ||
| | `block-work-on-main` | off | PreToolUse | Block git commits and merges on main/master branch _(params: protectedBranches)_ | | ||
| | `warn-git-amend` | off | PreToolUse | Warns before amending git commits, which rewrites history | | ||
| | `warn-git-stash-drop` | off | PreToolUse | Warns before permanently deleting stashed changes | | ||
| | `warn-all-files-staged` | off | PreToolUse | Warns before staging all working tree files with git add -A / . / --all | | ||
|
|
||
| ### Database | ||
|
|
||
| | Policy | Default | Events | What it catches | | ||
| |---|---|---|---| | ||
| | `warn-destructive-sql` | off | PreToolUse | Warn before executing destructive SQL (DROP/TRUNCATE/DELETE without WHERE) via database clients | | ||
| | `warn-schema-alteration` | off | PreToolUse | Warns before SQL schema changes (ALTER TABLE with column or rename operations) | | ||
|
|
||
| ### Packages & System | ||
|
|
||
| | Policy | Default | Events | What it catches | | ||
| |---|---|---|---| | ||
| | `warn-package-publish` | off | PreToolUse | Warn before publishing packages to public registries (npm, PyPI, crates.io, RubyGems, etc.) | | ||
| | `warn-global-package-install` | off | PreToolUse | Warns before installing packages globally (npm -g, cargo install, etc.) | | ||
| | `prefer-package-manager` | off | PreToolUse | Blocks non-preferred package managers and tells Claude to use an allowed one (e.g., uv instead of pip) _(params: allowed, blocked)_ | | ||
| | `warn-large-file-write` | off | PreToolUse | Warn before writing files larger than 1MB (configurable via thresholdKb param) _(params: thresholdKb)_ | | ||
| | `warn-background-process` | off | PreToolUse | Warns before starting detached or background processes | | ||
|
|
||
| ### AI Behavior | ||
|
|
||
| | Policy | Default | Events | What it catches | | ||
| |---|---|---|---| | ||
| | `warn-repeated-tool-calls` | off | PreToolUse | Warn when the same tool is called 3+ times with identical parameters | | ||
|
|
||
| ### Workflow | ||
|
|
||
| | Policy | Default | Events | What it catches | | ||
| |---|---|---|---| | ||
| | `require-commit-before-stop` | off | Stop | Require all changes to be committed before Claude stops | | ||
| | `require-push-before-stop` | off | Stop | Require all commits to be pushed to remote before Claude stops _(params: remote, baseBranch)_ | | ||
| | `require-pr-before-stop` | off | Stop | Require a pull request to exist for the current branch before Claude stops _(params: baseBranch)_ | | ||
| | `require-no-conflicts-before-stop` | off | Stop | Require the current branch to merge cleanly with the base branch before Claude stops _(params: baseBranch)_ | | ||
| | `require-ci-green-before-stop` | off | Stop | Require CI checks to pass on the current HEAD commit before Claude stops (ignores stale runs on prior commits) | | ||
|
|
||
| > The five `require-*-before-stop` policies gate the end of a turn. A gate whose | ||
| > condition cannot be met in the current project loops forever — see `traps.md` §6 | ||
| > before enabling one. | ||
|
|
||
| --- | ||
|
|
||
| ## Audit-only detectors | ||
|
|
||
| These have no real-time builtin equivalent, so they are the prime candidates for | ||
| custom policies. List them from source with: | ||
|
|
||
| ```bash | ||
| bun -e 'const {AUDIT_DETECTORS}=await import("./src/audit/detectors/index.ts"); | ||
| for (const d of AUDIT_DETECTORS) console.log(d.name, "|", d.category+"/"+d.severity, "|", d.description)' | ||
| ``` | ||
|
|
||
| All but `reread-after-edit` are Bash-command patterns, so a `PreToolUse` policy | ||
| filtering on `ctx.toolName === "Bash"` and matching `ctx.toolInput.command` covers | ||
| most of them. `reread-after-edit` needs cross-call session state, which hooks cannot | ||
| see — that one needs a builtin, not a custom policy. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Align all references on the current
customPoliciesEnabledbehavior.The API reference presents
falseas an effective disable switch, while the skill and traps reference document that the field is currently dropped during config merging. Choose the authoritative runtime contract, then make all three locations consistent so users cannot misjudge whether convention policies are active..claude/skills/policy-author/references/api.md#L152-L160: correct the public configuration contract or fix propagation..claude/skills/policy-author/SKILL.md#L337-L341: retain or update the operational verification guidance to match runtime behavior..claude/skills/policy-author/references/traps.md#L29-L55: retain this as the canonical silent-failure warning and cross-reference it from the API documentation.📍 Affects 3 files
.claude/skills/policy-author/references/api.md#L152-L160(this comment).claude/skills/policy-author/SKILL.md#L337-L341.claude/skills/policy-author/references/traps.md#L29-L55🤖 Prompt for AI Agents