Add a policy-author skill that turns audit findings and prose rules files into enforced policies - #600
Add a policy-author skill that turns audit findings and prose rules files into enforced policies#600chhhee10 wants to merge 2 commits into
Conversation
…iles into enforcement
The audit reports what agents did wrong but can only ever point at the 39
builtins: DETECTOR_TO_POLICY maps every audit-only detector to one, and its own
header comment says why ("so every finding looks like it has a failproofai
fix"), yet several of those mappings prevent nothing. Nothing in the audit can
say "this needs a policy you don't have", and nothing checks whether a
hand-written policy already covers it. This repo already worked around that by
hand — workflow-policies.mjs and block-version-bumps-policies.mjs are CLAUDE.md
rules someone converted because prose kept getting ignored.
The skill automates both halves. Four entry paths (audit trigger, plain-language
complaint, explicit ask, rules file) converge on one core whose load-bearing rule
is that coverage checking — builtins including their params, plus the project's
existing custom policies — happens before authoring, since most requests are one
line in enabledPolicies rather than a new file. Authoring then picks one of three
modes matching the builtins' own naming, defaulting to warn-*/instruct() whenever
an action has legitimate uses.
references/traps.md carries nine silent failures, each verified by running the
hook rather than reading code: the loader convention that skips block-foo.mjs in
silence (this repo shipped one that way and it never ran), fail-open loading that
makes an unloaded policy indistinguishable from a correctly-allowing one, and
that a deny in a test does not prove your policy denied. scripts/test-policy.mjs
sandboxes a policy with zero builtins enabled so any deny is necessarily yours,
and handles all six deny shapes across the CLIs; scripts/sync-builtins.mjs keeps
the builtin reference generated, with --check for CI.
Validated across five rounds of blind agents in seeded sandboxes over sixteen
use-case cells, including both directions of the scope guardrail: a question
about machine-wide findings must not modify the machine, and an explicit "do it
globally" must not be refused.
.gitignore needed the carve-out — .claude/* excluded everything but
settings.json, so the skill would have silently never reached the tree.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
📝 WalkthroughWalkthroughAdds a ChangesPolicy authoring workflow
Estimated code review effort: 3 (Moderate) | ~25 minutes Suggested labels: Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
ESLint install timed out. The project may have too many dependencies for the sandbox. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Automated code review started - full review. Results will be posted here. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 10
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.claude/skills/policy-author/references/api.md:
- Around line 164-168: Update the documentation around enabledPolicies and
readMergedHooksConfig to state that policy names are unioned across project,
local, and global scopes, so omission in one scope does not disable an
enablement elsewhere. Clarify that precedence applies only to fields with
override semantics, including enforcement settings, and must not imply generic
precedence for enabledPolicies.
- Around line 152-160: Align the documentation with the current runtime contract
that customPoliciesEnabled is dropped during config merging, so it does not
disable convention policies: update
.claude/skills/policy-author/references/api.md lines 152-160 to remove the
misleading effective-disable behavior and cross-reference the warning; retain or
adjust .claude/skills/policy-author/SKILL.md lines 337-341 to verify behavior
according to this contract; keep
.claude/skills/policy-author/references/traps.md lines 29-55 as the canonical
silent-failure warning.
In @.claude/skills/policy-author/references/patterns.md:
- Around line 159-166: Remove the global flag from INTERNAL_URL_RE in the
sanitize-internal-urls policy example so repeated .test() calls do not mutate
lastIndex or alternate between matches and misses. Keep the sanitizer’s matching
behavior and surrounding policy logic unchanged.
- Around line 184-194: Update the execSync call in the require-changelog-entry
policy’s Stop hook to include an explicit timeout shorter than the documented
10-second evaluation limit. Keep the call inside the existing try/catch so
timeout errors and any partial or ignored diff output follow the current
fail-open handling.
In @.claude/skills/policy-author/references/rules-files.md:
- Around line 64-70: Update the changelog gate around the `execSync`
verification to require an exact root-level `CHANGELOG.md` path rather than any
substring match, and change the `catch` branch to deny when Git verification
fails instead of allowing the PR. Leave heading, subsection, and PR-number
formatting enforcement to CI as specified.
In @.claude/skills/policy-author/scripts/sync-builtins.mjs:
- Around line 1-6: Update the executable shebang and both documented invocations
in the script header to use Bun, so sync-builtins.mjs can import the TypeScript
BUILTIN_POLICIES registry without requiring a Node loader. Keep the existing
rewrite and --check behaviors unchanged.
In @.claude/skills/policy-author/scripts/test-policy.mjs:
- Around line 163-164: Update the result handling around classify in the test
runner so a null res.status is returned as a test error before calling classify.
Preserve the existing res.error handling and normal classify flow for processes
that exit with a status.
In @.claude/skills/policy-author/SKILL.md:
- Around line 387-389: The coverage-check command in SKILL.md must inspect all
supported custom-policy extensions, not only *policies.mjs. Update the policy
listing and grep pattern to include .js, .mjs, and .ts files while preserving
the existing name and description extraction behavior.
In @.failproofai/policies/audit-skill-policies.mjs:
- Around line 24-29: Update the audit detection logic around the command parsing
in the policy so it identifies the actual failproofai invocation and requires
audit to be its subcommand, rather than matching audit anywhere in the raw shell
command. Apply the --help/-h exclusion only to that same parsed invocation,
preserving audit execution when help appears in a separate command segment, and
add regression cases for both false positives.
In `@CHANGELOG.md`:
- Line 7: Update the changelog entries described in the diff, including the
entry around the audit-skill policy and the entry at the corresponding later
location, so each is a single-line description containing the PR marker (`#600`).
Preserve their existing content while adding the marker consistently to every
new entry.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 834ae59c-ed9c-4805-b8df-03f72a9a3f5a
📒 Files selected for processing (11)
.claude/skills/policy-author/SKILL.md.claude/skills/policy-author/references/api.md.claude/skills/policy-author/references/builtins.md.claude/skills/policy-author/references/patterns.md.claude/skills/policy-author/references/rules-files.md.claude/skills/policy-author/references/traps.md.claude/skills/policy-author/scripts/sync-builtins.mjs.claude/skills/policy-author/scripts/test-policy.mjs.failproofai/policies/audit-skill-policies.mjs.gitignoreCHANGELOG.md
| `.failproofai/policies-config.json` — `HooksConfig`, `policy-types.ts`, grep `interface HooksConfig`: | ||
|
|
||
| ```ts | ||
| export interface HooksConfig { | ||
| enabledPolicies: string[]; // builtins, by short name | ||
| llm?: LlmConfig; | ||
| policyParams?: Record<string, Record<string, unknown>>; // keyed by short name | ||
| customPoliciesPath?: string; | ||
| customPoliciesEnabled?: boolean; // absent = enabled |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Align all references on the current customPoliciesEnabled behavior.
The API reference presents false as an effective disable switch, while the skill and traps reference document that the field is currently dropped during config merging. Choose the authoritative runtime contract, then make all three locations consistent so users cannot misjudge whether convention policies are active.
.claude/skills/policy-author/references/api.md#L152-L160: correct the public configuration contract or fix propagation..claude/skills/policy-author/SKILL.md#L337-L341: retain or update the operational verification guidance to match runtime behavior..claude/skills/policy-author/references/traps.md#L29-L55: retain this as the canonical silent-failure warning and cross-reference it from the API documentation.
📍 Affects 3 files
.claude/skills/policy-author/references/api.md#L152-L160(this comment).claude/skills/policy-author/SKILL.md#L337-L341.claude/skills/policy-author/references/traps.md#L29-L55
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.claude/skills/policy-author/references/api.md around lines 152 - 160, Align
the documentation with the current runtime contract that customPoliciesEnabled
is dropped during config merging, so it does not disable convention policies:
update .claude/skills/policy-author/references/api.md lines 152-160 to remove
the misleading effective-disable behavior and cross-reference the warning;
retain or adjust .claude/skills/policy-author/SKILL.md lines 337-341 to verify
behavior according to this contract; keep
.claude/skills/policy-author/references/traps.md lines 29-55 as the canonical
silent-failure warning.
| Builtins are enabled **purely by presence** of the short name in `enabledPolicies` — there | ||
| is no per-policy enabled/disabled object, and omission means off. | ||
|
|
||
| Merged across three scopes, in precedence order: project `{cwd}/.failproofai/` → local → | ||
| global `~/.failproofai/` (`hooks-config.ts`, grep `readMergedHooksConfig`). |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Document enabledPolicies as a union, not generic precedence.
The skill and traps reference state that enabled policy names are unioned across scopes. “Merged ... in precedence order” suggests a project config can override a global enable, which is false and can leave a builtin active unexpectedly. Clarify merge semantics per field, especially for enforcement settings.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.claude/skills/policy-author/references/api.md around lines 164 - 168,
Update the documentation around enabledPolicies and readMergedHooksConfig to
state that policy names are unioned across project, local, and global scopes, so
omission in one scope does not disable an enablement elsewhere. Clarify that
precedence applies only to fields with override semantics, including enforcement
settings, and must not imply generic precedence for enabledPolicies.
| const INTERNAL_URL_RE = /https:\/\/[a-z0-9-]+\.internal\.example\.com\/\S*/gi; | ||
|
|
||
| customPolicies.add({ | ||
| name: "sanitize-internal-urls", | ||
| description: "Strip internal hostnames from tool output", | ||
| match: { events: ["PostToolUse"] }, | ||
| fn: async (ctx) => { | ||
| if (!INTERNAL_URL_RE.test(JSON.stringify(ctx.payload))) return allow(); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Remove the global flag from the sanitizer example.
As written, .test() mutates lastIndex, so repeated matching calls can alternate between matches and misses. The warning below explains the problem, but the copyable example should already be safe.
-const INTERNAL_URL_RE = /https:\/\/[a-z0-9-]+\.internal\.example\.com\/\S*/gi;
+const INTERNAL_URL_RE = /https:\/\/[a-z0-9-]+\.internal\.example\.com\/\S*/i;📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const INTERNAL_URL_RE = /https:\/\/[a-z0-9-]+\.internal\.example\.com\/\S*/gi; | |
| customPolicies.add({ | |
| name: "sanitize-internal-urls", | |
| description: "Strip internal hostnames from tool output", | |
| match: { events: ["PostToolUse"] }, | |
| fn: async (ctx) => { | |
| if (!INTERNAL_URL_RE.test(JSON.stringify(ctx.payload))) return allow(); | |
| const INTERNAL_URL_RE = /https:\/\/[a-z0-9-]+\.internal\.example\.com\/\S*/i; | |
| customPolicies.add({ | |
| name: "sanitize-internal-urls", | |
| description: "Strip internal hostnames from tool output", | |
| match: { events: ["PostToolUse"] }, | |
| fn: async (ctx) => { | |
| if (!INTERNAL_URL_RE.test(JSON.stringify(ctx.payload))) return allow(); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.claude/skills/policy-author/references/patterns.md around lines 159 - 166,
Remove the global flag from INTERNAL_URL_RE in the sanitize-internal-urls policy
example so repeated .test() calls do not mutate lastIndex or alternate between
matches and misses. Keep the sanitizer’s matching behavior and surrounding
policy logic unchanged.
| import { execSync } from "node:child_process"; | ||
|
|
||
| customPolicies.add({ | ||
| name: "require-changelog-entry", | ||
| description: "Every change must touch CHANGELOG.md before the turn ends", | ||
| match: { events: ["Stop"] }, | ||
| fn: async (ctx) => { | ||
| const cwd = ctx.session?.cwd; | ||
| if (!cwd) return allow(); | ||
| try { | ||
| const changed = execSync("git diff --name-only HEAD", { cwd, encoding: "utf8" }); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
rg -n -U 'execSync\([^;]*git diff' .claude/skills/policy-author/references/patterns.mdRepository: FailproofAI/failproofai
Length of output: 253
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the policy snippet and nearby context.
sed -n '150,215p' .claude/skills/policy-author/references/patterns.md
# Search for timeout/Stop hook/docs references in the skill or policy author documentation.
rg -n 'timeout|10 second|Stop|execSync|execFile|child_process' .claude/skills/policy-author -SRepository: FailproofAI/failproofai
Length of output: 8354
🌐 Web query:
Node.js child_process execSync timeout option spawnExecSync timeout documentation
💡 Result:
The Node.js child_process module includes a timeout option for its synchronous methods, including execSync, execFileSync, and spawnSync [1][2][3]. The timeout option is defined as the maximum amount of time, in milliseconds, that the process is allowed to run [1][4]. If the timeout is greater than 0 and the process runs longer than the specified time, the parent process will send a signal to the child process (defined by the killSignal property, which defaults to 'SIGTERM') [1][5][6]. Key characteristics of this timeout functionality include: - Default Value: The default value for the timeout option is undefined (or 0, depending on specific version documentation context) [1][5][7][6]. - Blocking Behavior: Because execSync, execFileSync, and spawnSync are synchronous, the parent process will not return until the child process has fully closed [1][8][2]. If a timeout is triggered and the killSignal is sent, these methods will block until the child process has completely exited [1][2][3]. - Handling SIGTERM: If the child process intercepts and handles the SIGTERM signal and does not exit, the parent process will continue to wait until the child process terminates [1][7][4]. For most use cases, the timeout option provides a built-in mechanism to prevent synchronous child processes from running indefinitely [1][9][10].
Citations:
- 1: https://nodejs.org/api/child_process.html
- 2: https://beta.docs.nodejs.org/child_process.html
- 3: https://nodejs.org/docs/latest-v26.x/api/child_process.html
- 4: https://r2.nodejs.org/dist/v22.5.0/docs/api/child_process.html
- 5: https://nodejs.org/docs/latest-v24.x/api/child_process.html
- 6: https://github.com/nodejs/node/blob/d90d9d55/doc/api/child_process.md
- 7: https://nodejs.org/api/child_process.md
- 8: https://nodejs.org/download/release/latest-v24.x/docs/api/child_process.html
- 9: https://2ality.com/2022/07/nodejs-child-process.html
- 10: https://stackoverflow.com/questions/56627983/how-to-kill-spawnsync-after-a-timeout
Bound the synchronous git diff subprocess in the Stop hook.
execSync has no built-in budget here, so a hanging Git process can block the policy fn and prevent the documented 10-second evaluation timeout from taking effect. Add an explicit timeout and route the child process timeout’s thrown error/ignored diff output through the existing try/catch so this guard fails open.
Suggested fix
-import { execSync } from "node:child_process";
+import { execSync } from "node:child_process";
- const changed = execSync("git diff --name-only HEAD", { cwd, encoding: "utf8" });
+ const changed = execSync("git diff --name-only HEAD", {
+ cwd,
+ encoding: "utf8",
+ timeout: 9_000,
+ });📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| import { execSync } from "node:child_process"; | |
| customPolicies.add({ | |
| name: "require-changelog-entry", | |
| description: "Every change must touch CHANGELOG.md before the turn ends", | |
| match: { events: ["Stop"] }, | |
| fn: async (ctx) => { | |
| const cwd = ctx.session?.cwd; | |
| if (!cwd) return allow(); | |
| try { | |
| const changed = execSync("git diff --name-only HEAD", { cwd, encoding: "utf8" }); | |
| import { execSync } from "node:child_process"; | |
| customPolicies.add({ | |
| name: "require-changelog-entry", | |
| description: "Every change must touch CHANGELOG.md before the turn ends", | |
| match: { events: ["Stop"] }, | |
| fn: async (ctx) => { | |
| const cwd = ctx.session?.cwd; | |
| if (!cwd) return allow(); | |
| try { | |
| const changed = execSync("git diff --name-only HEAD", { | |
| cwd, | |
| encoding: "utf8", | |
| timeout: 9_000, | |
| }); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.claude/skills/policy-author/references/patterns.md around lines 184 - 194,
Update the execSync call in the require-changelog-entry policy’s Stop hook to
include an explicit timeout shorter than the documented 10-second evaluation
limit. Keep the call inside the existing try/catch so timeout errors and any
partial or ignored diff output follow the current fail-open handling.
| const changed = execSync("git diff --name-only origin/main...HEAD", { | ||
| cwd, encoding: "utf8", timeout: 5000, | ||
| }); | ||
| if (changed.includes("CHANGELOG.md")) return allow(); | ||
| return deny("This PR has no CHANGELOG.md entry. Add one under the current version heading, commit it, then create the PR."); | ||
| } catch { | ||
| return allow(); // can't tell -> fail open, never trap the agent |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Do not fail open the changelog gate.
includes("CHANGELOG.md") accepts unrelated paths such as docs/CHANGELOG.md.bak, and any Git failure permits gh pr create. Require the exact root path and deny when verification cannot run; enforce heading/subsection/PR-number formatting separately in CI.
Proposed fix
- if (changed.includes("CHANGELOG.md")) return allow();
+ const files = changed.trim().split(/\r?\n/);
+ if (files.includes("CHANGELOG.md")) return allow();
return deny("This PR has no CHANGELOG.md entry. Add one under the current version heading, commit it, then create the PR.");
} catch {
- return allow(); // can't tell -> fail open, never trap the agent
+ return deny("Cannot verify CHANGELOG.md. Fetch origin/main and retry.");As per coding guidelines, “Every pull request must update CHANGELOG.md under the current dated, versioned heading; entries must use the appropriate subsection and be single-line descriptions including the PR number.”
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const changed = execSync("git diff --name-only origin/main...HEAD", { | |
| cwd, encoding: "utf8", timeout: 5000, | |
| }); | |
| if (changed.includes("CHANGELOG.md")) return allow(); | |
| return deny("This PR has no CHANGELOG.md entry. Add one under the current version heading, commit it, then create the PR."); | |
| } catch { | |
| return allow(); // can't tell -> fail open, never trap the agent | |
| const changed = execSync("git diff --name-only origin/main...HEAD", { | |
| cwd, encoding: "utf8", timeout: 5000, | |
| }); | |
| const files = changed.trim().split(/\r?\n/); | |
| if (files.includes("CHANGELOG.md")) return allow(); | |
| return deny("This PR has no CHANGELOG.md entry. Add one under the current version heading, commit it, then create the PR."); | |
| } catch { | |
| return deny("Cannot verify CHANGELOG.md. Fetch origin/main and retry."); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.claude/skills/policy-author/references/rules-files.md around lines 64 - 70,
Update the changelog gate around the `execSync` verification to require an exact
root-level `CHANGELOG.md` path rather than any substring match, and change the
`catch` branch to deny when Git verification fails instead of allowing the PR.
Leave heading, subsection, and PR-number formatting enforcement to CI as
specified.
Source: Coding guidelines
| #!/usr/bin/env node | ||
| /** | ||
| * Regenerate references/builtins.md from the live BUILTIN_POLICIES registry. | ||
| * | ||
| * node sync-builtins.mjs rewrite the reference file | ||
| * node sync-builtins.mjs --check exit 1 if it is out of date (for CI) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -u
tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' EXIT
printf 'export const marker = 1;\n' > "$tmp/registry.ts"
if node - "$tmp/registry.ts" <<'NODE'
const { pathToFileURL } = require("node:url");
import(pathToFileURL(process.argv[2]).href).then(
() => process.exit(0),
() => process.exit(1),
);
NODE
then
echo "Unexpected: Node imported TypeScript without a loader."
exit 1
else
echo "Confirmed: this Node runtime cannot directly import .ts."
fiRepository: FailproofAI/failproofai
Length of output: 216
🏁 Script executed:
#!/bin/bash
set -u
echo "== locate sync-builtins.mjs =="
fd -a 'sync-builtins\.mjs$' . | sed 's#^\./##'
echo
echo "== file excerpt =="
file="$(fd 'sync-builtins\.mjs$' . 2>/dev/null | head -n 1)"
if [ -n "${file:-}" ]; then
wc -l "$file"
sed -n '1,80p' "$file" | cat -n
fi
echo
echo "== TypeScript loader/babel config references =="
rg -n "ts-node|tsx|`@babel/register`|typescript|\"buildIn|BUILTIN_POLICIES\"|sync-builtins|--check" -S .
echo
echo "== node availability/version =="
if command -v node >/dev/null 2>&1; then
node -v
node --version
else
echo "node not available"
fi
echo
echo "== test dynamic import on this Node runtime with TS =="
tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' EXIT
printf 'export const marker = 1;\n' > "$tmp/registry.ts"
node - "$tmp/registry.ts" <<'NODE' || exit_code=$?
const { pathToFileURL } = require("node:url");
import(pathFileURL = pathToFileURL(process.argv[2]).href)
.then(() => {
console.log("IMPORT_SUCCEEDED", pathFileURL);
process.exit(0);
})
.catch((err) => {
console.log("IMPORT_FAILED", err.code || err.constructor?.name || err);
process.exit(1);
});
NODE
node_ver_exit=$?
printf "Node import dynamic TS exit code: %s\n" "$node_ver_exit"Repository: FailproofAI/failproofai
Length of output: 50379
🌐 Web query:
Node.js v20.9 ESM TypeScript .ts import requires loader documentation
💡 Result:
In Node.js v20.9, executing TypeScript files directly within an ECMAScript module (ESM) environment typically requires a loader because Node.js does not natively execute.ts files [1][2]. While Node.js has since introduced native type-stripping support in later versions (e.g., v22+ and v24 LTS) [2], v20.9 relies on third-party tools to bridge this gap [3][1]. Using a Loader with ts-node To run TypeScript files with ESM support in Node.js v20.9 using ts-node, you must use the --loader flag [3][4]. The loader hooks allow Node.js to intercept import requests, transpile the TypeScript on-the-fly, and provide it to the module system [5][3]. Command: node --loader ts-node/esm./index.ts Alternatively, you can set this via environment variables to ensure all child processes also use the loader: NODE_OPTIONS="--loader ts-node/esm" node./index.ts Important Considerations for Node.js v20.x: 1. Experimental Status: The ESM loader hooks in Node.js v20.x are marked as experimental [3][4]. They are subject to change and may be unstable, making them generally discouraged for production environments [3][4]. 2. Isolated Threads: In Node.js v20, loader hooks run in a separate thread environment from the main application [6][7]. This architectural change caused compatibility issues for some tools, requiring specific workarounds like globalPreload in older versions of ts-node [6]. 3. Modern Alternatives (2026 Context): While you are using v20.9, industry standard practice as of mid-2026 has shifted away from using ts-node for production execution [8][9]. Most modern setups utilize tools like tsx for development [10][8] and standard tsc compilation (or esbuild) to generate JavaScript for production deployment [8][9][2]. This avoids the need for experimental loaders in production [8][2]. Configuration for ESM For any TypeScript ESM setup, ensure your package.json includes "type": "module" [10][2] and your tsconfig.json uses "module": "NodeNext" and "moduleResolution": "NodeNext" to align with Node.js's native ESM resolution logic [10][8][2].
Citations:
- 1: https://nodejs.org/docs/latest/api/typescript.html
- 2: https://blog.openreplay.com/typescript-node-practical-setup/
- 3: https://github.com/TypeStrong/ts-node?tab=readme-ov-file
- 4: https://github.com/typestrong/ts-node
- 5: https://typestrong.org/ts-node/api/index.html
- 6: fix: Add globalPreload to ts-node/esm for node 20 TypeStrong/ts-node#2009
- 7: https://nodejs.org/api/esm.html
- 8: https://dev.to/axiom_agent/typescript-in-nodejs-2026-the-complete-production-guide-2pii
- 9: https://typescript.page/nodejs-typescript-setup-guide
- 10: https://nodewire.net/typescript-nodejs-setup-2026/
🌐 Web query:
Bun supports running .mjs import TypeScript files directly shebang bun
💡 Result:
Yes, Bun supports running TypeScript files directly, and you can use a shebang to specify that a script should be executed by Bun [1][2]. To run a TypeScript file (e.g., script.ts) using a shebang, you can add the following line at the top of your file: #!/usr/bin/env bun When a file has this shebang, you can execute it directly from the terminal (provided it is marked as executable via chmod +x) or by passing it to Bun [1][3]. Important considerations: 1. Handling Existing Shebangs: By default, Bun respects existing shebangs. If a file contains #!/usr/bin/env node, Bun will attempt to run it with Node.js [1][3]. If you want to force such a file to run with Bun instead, you can use the --bun flag [1][4]: bun --bun script.ts 2. Troubleshooting Execution: If you encounter issues where bun run treats your file as a package.json script rather than a standalone file, ensure you are providing a path to the file (e.g.,./script.ts) or use the --bun flag [1][5]. 3. File Extensions: Bun runs.ts,.mts,.tsx,.js,.mjs, and.jsx files natively [1][2]. While you can use a shebang with these extensions, ensure that your environment and file headers are configured correctly to maintain consistent behavior across different runtimes [3].
Citations:
- 1: https://bun.sh/docs/runtime
- 2: https://mintlify.wiki/oven-sh/bun/guides/migrate-from-node
- 3: https://bun.com/docs/pm/bunx
- 4: Ignore hashbang for Node when directly running
bun run file.tsoven-sh/bun#11740 - 5:
bun run script.jsfails when script has node shebang oven-sh/bun#4850
🏁 Script executed:
#!/bin/bash
set -u
echo "== Node loader probe with correct argv handling =="
tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' EXIT
printf 'export const marker = 1;\n' > "$tmp/registry.ts"
node - "$tmp/registry.ts" <<'NODE' || exit_code=$?
const { pathToFileURL } = require("node:url");
const registryPath = process.argv[process.argv.length - 2];
import(pathToFileURL(registryPath).href)
.then(() => {
console.log("IMPORT_SUCCEEDED", registryPath);
process.exit(0);
})
.catch((err) => {
console.log("IMPORT_FAILED", String(err.message || err));
process.exit(1);
});
NODE
printf "Node direct .ts import exit code: %s\n" "${exit_code:-0}"
echo
echo "== Candidate policy-author scripts (repo may have been rebased/renamed) =="
fd -a 'sync-builtins\.(mjs|js|ts)$|builtin-policies\.(ts|js)$' .claude . 2>/dev/null | sed 's#^\./##' | head -n 50Repository: FailproofAI/failproofai
Length of output: 470
Use Bun for the TypeScript registry import.
.claude/skills/policy-author/scripts/sync-builtins.mjs imports src/hooks/builtin-policies.ts, and this Node 20.9 runtime cannot resolve that TypeScript source without a custom loader. Switch the shebang and documented sync-builtins.mjs command to bun, or rewrite the import path to be Node-compatible.
Proposed fix
-#!/usr/bin/env node
+#!/usr/bin/env bun
@@
- * node sync-builtins.mjs rewrite the reference file
- * node sync-builtins.mjs --check exit 1 if it is out of date (for CI)
+ * bun sync-builtins.mjs rewrite the reference file
+ * bun sync-builtins.mjs --check exit 1 if it is out of date (for CI)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| #!/usr/bin/env node | |
| /** | |
| * Regenerate references/builtins.md from the live BUILTIN_POLICIES registry. | |
| * | |
| * node sync-builtins.mjs rewrite the reference file | |
| * node sync-builtins.mjs --check exit 1 if it is out of date (for CI) | |
| #!/usr/bin/env bun | |
| /** | |
| * Regenerate references/builtins.md from the live BUILTIN_POLICIES registry. | |
| * | |
| * bun sync-builtins.mjs rewrite the reference file | |
| * bun sync-builtins.mjs --check exit 1 if it is out of date (for CI) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.claude/skills/policy-author/scripts/sync-builtins.mjs around lines 1 - 6,
Update the executable shebang and both documented invocations in the script
header to use Bun, so sync-builtins.mjs can import the TypeScript
BUILTIN_POLICIES registry without requiring a Node loader. Keep the existing
rewrite and --check behaviors unchanged.
| if (res.error) return { decision: "error", reason: String(res.error) }; | ||
| return classify(res.stdout ?? "", res.status, res.stderr); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
node - <<'NODE'
const { spawnSync } = require("node:child_process");
const result = spawnSync(process.execPath, ["-e", 'process.kill(process.pid, "SIGTERM")']);
console.log({ status: result.status, signal: result.signal, error: result.error?.code });
if (result.status !== null || result.signal !== "SIGTERM") process.exit(1);
NODERepository: FailproofAI/failproofai
Length of output: 215
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate and inspect the referenced file around the reported logic.
if [ -f ".claude/skills/policy-author/scripts/test-policy.mjs"
then
echo "FOUND .claude/skills/policy-author/scripts/test-policy.mjs"
wc -l .claude/skills/policy-author/scripts/test-policy.mjs
sed -n '130,180p' .claude/skills/policy-author/scripts/test-policy.mjs
echo "--- classify/runCase references ---"
rg -n "function classify|const classify|runCase|classify\\(" .claude/skills/policy-author/scripts/test-policy.mjs
else
echo "Referenced file not found; listing nearby policy-author files:"
fd -i "test-policy" . || true
fiRepository: FailproofAI/failproofai
Length of output: 253
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Repository root files around relevant paths:"
git ls-files | grep -E '(^|/)(test-policy\.mjs|policy-author|policy)' | head -200 || true
echo
echo "Locate test-policy.mjs by name:"
fd -a 'test-policy\.mjs$' . || true
echo
echo "Search for classify function definitions/usages:"
rg -n "function classify|const classify|\.classify|classify\\(" . || true
echo
echo "Search for runCase/signaled runner code:"
rg -n "runCase|signal|status === null|spawnSync" . || trueRepository: FailproofAI/failproofai
Length of output: 50379
🏁 Script executed:
#!/bin/bash
set -euo pipefail
FILE=".claude/skills/policy-author/scripts/test-policy.mjs"
echo "Target file size:"
wc -l "$FILE"
echo
echo "Lines 120-175:"
sed -n '120,175p' "$FILE"
echo
echo "Relevant definitions/usages:"
rg -n "function classify|const classify|runCase|classify\\(" "$FILE"Repository: FailproofAI/failproofai
Length of output: 2965
🏁 Script executed:
#!/bin/bash
set -euo pipefail
node - <<'NODE'
function classify(stdout, exitCode, stderr) {
const text = stdout ?? "";
const isEmpty = text.trim().length === 0;
if (isEmpty && exitCode && exitCode !== 2) return { decision: "allow", reason: "" };
if (exitCode === 2) return { decision: "deny", reason: (stderr ?? "").trim() };
let parsed;
try { parsed = JSON.parse(text); } catch { return { decision: "unparseable", reason: text }; }
const h = parsed.hookSpecificOutput ?? {};
if (h.permissionDecision === "deny") return { decision: "deny", reason: h.permissionDecisionReason ?? "" };
if (h.decision?.behavior === "deny") return { decision: "deny", reason: h.decision.message ?? "" };
if (h.additionalContext) return { decision: "instruct", reason: h.additionalContext };
if (parsed.decision === "block") return { decision: "deny", reason: parsed.reason ?? "" };
if (parsed.permission === "deny") return { decision: "deny", reason: parsed.reason ?? "" };
if (exitCode === 2) return { decision: "deny", reason: (stderr ?? "").trim() };
return { decision: "allow", reason: "" };
}
const cases = [
{ name: "signal-null-empty", stdout: "", exitCode: null, stderr: "" },
{ name: "signal-null-json", stdout: JSON.stringify({ decision: "block" }), exitCode: null, stderr: "" },
{ name: "exit0-empty", stdout: "", exitCode: 0, stderr: "" },
{ name: "exit1-empty", stdout: "", exitCode: 1, stderr: "" },
];
for (const c of cases) {
console.log(`${c.name}: ${JSON.stringify(c)}`);
console.log(JSON.stringify(cla$$ssify(c.stdout, c.exitCode, c.stderr)));
}
NODERepository: FailproofAI/failproofai
Length of output: 906
🏁 Script executed:
#!/bin/bash
set -euo pipefail
node - <<'NODE'
function classify(stdout, exitCode, stderr) {
const text = stdout ?? "";
const isEmpty = text.trim().length === 0;
if (isEmpty && exitCode && exitCode !== 2) return { decision: "allow", reason: "" };
if (exitCode === 2) return { decision: "deny", reason: (stderr ?? "").trim() };
let parsed;
try { parsed = JSON.parse(text); } catch { return { decision: "unparseable", reason: text }; }
const h = parsed.hookSpecificOutput ?? {};
if (h.permissionDecision === "deny") return { decision: "deny", reason: h.permissionDecisionReason ?? "" };
if (h.decision?.behavior === "deny") return { decision: "deny", reason: h.decision.message ?? "" };
if (h.additionalContext) return { decision: "instruct", reason: h.additionalContext };
if (parsed.decision === "block") return { decision: "deny", reason: parsed.reason ?? "" };
if (parsed.permission === "deny") return { decision: "deny", reason: parsed.reason ?? "" };
if (exitCode === 2) return { decision: "deny", reason: (stderr ?? "").trim() };
return { decision: "allow", reason: "" };
}
const cases = [
{ name: "signal-null-empty", stdout: "", exitCode: null, stderr: "" },
{ name: "signal-null-json", stdout: JSON.stringify({ decision: "block" }), exitCode: null, stderr: "" },
{ name: "exit0-empty", stdout: "", exitCode: 0, stderr: "" },
{ name: "exit1-empty", stdout: "", exitCode: 1, stderr: "" },
{ name: "exit1-json-deny", stdout: JSON.stringify({ decision: "block" }), exitCode: 1, stderr: "" },
];
for (const c of cases) {
console.log(`${c.name}: ${JSON.stringify(cla$$ssify(c.stdout, c.exitCode, c.stderr))}`);
}
NODERepository: FailproofAI/failproofai
Length of output: 848
🏁 Script executed:
#!/bin/bash
set -euo pipefail
node - <<'NODE'
function classify(stdout, exitCode, stderr) {
const text = stdout ?? '';
const isEmpty = text.trim().length === 0;
if (isEmpty && exitCode && exitCode !== 2) return { decision: 'allow', reason: '' };
if (exitCode === 2) return { decision: 'deny', reason: (stderr ?? '').trim() };
let parsed;
try { parsed = JSON.parse(text); } catch { return { decision: 'unparseable', reason: text }; }
const h = parsed.hookSpecificOutput ?? {};
if (h.permissionDecision === 'deny') return { decision: 'deny', reason: h.permissionDecisionReason ?? '' };
if (h.decision?.behavior === 'deny') return { decision: 'deny', reason: h.decision.message ?? '' };
if (h.additionalContext) return { decision: 'instruct', reason: h.additionalContext };
if (parsed.decision === 'block') return { decision: 'deny', reason: parsed.reason ?? '' };
if (parsed.permission === 'deny') return { decision: 'deny', reason: parsed.reason ?? '' };
if (exitCode === 2) return { decision: 'deny', reason: (stderr ?? '').trim() };
return { decision: 'allow', reason: '' };
}
const cases = [
['signal-null-empty', '', null],
['signal-null-json', JSON.stringify({ decision: 'block' }), null],
['exit0-empty', '', 0],
['exit1-empty', '', 1],
['exit1-json-deny', JSON.stringify({ decision: 'block' }), 1],
];
for (const [name, stdout, exitCode] of cases) {
console.log(JSON.stringify({ [name]: classify(stdout, exitCode, '') }));
}
NODERepository: FailproofAI/failproofai
Length of output: 431
Treat signalled runners as test errors.
A process killed by a signal reports status === null; with empty stdout, classify() falls through unparseable JSON and returns allow. Handle signalled spawn failures before delegating to classify().
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.claude/skills/policy-author/scripts/test-policy.mjs around lines 163 - 164,
Update the result handling around classify in the test runner so a null
res.status is returned as a test error before calling classify. Preserve the
existing res.error handling and normal classify flow for processes that exit
with a status.
| ```bash | ||
| ls .failproofai/policies/ && grep -h "name:\|description:" .failproofai/policies/*policies.mjs | ||
| ``` |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Include all supported custom-policy extensions in the coverage check.
This command searches only *policies.mjs, although the loader and this skill support .js, .mjs, and .ts. Existing JavaScript or TypeScript policies can therefore be missed and duplicated.
Suggested fix
-ls .failproofai/policies/ && grep -h "name:\|description:" .failproofai/policies/*policies.mjs
+for file in .failproofai/policies/*policies.{js,mjs,ts}; do
+ [ -f "$file" ] || continue
+ grep -h "name:\|description:" "$file"
+done📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| ```bash | |
| ls .failproofai/policies/ && grep -h "name:\|description:" .failproofai/policies/*policies.mjs | |
| ``` |
🧰 Tools
🪛 SkillSpector (2.3.11)
[error] 204: [PE3] Credential Access: Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.
Remediation: Remove references to credential paths. Use environment variables or secrets managers. For docs, use placeholder paths (e.g., /path/to/config). Never load .env or token files in production code paths.
(Privilege Escalation (PE3))
[warning] 21: [RA2] Session Persistence: Skill establishes unauthorized persistence across sessions via cron jobs, startup scripts, or state files. Session persistence allows an attacker to maintain access beyond the current interaction.
Remediation: Remove any persistence mechanisms (cron jobs, startup scripts, state files). Skills should not maintain state across sessions without explicit user consent.
(Rogue Agent (RA2))
[error] 39: [TM1] Tool Parameter Abuse: Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).
Remediation: Validate all tool parameters against an allowlist. Reject dangerous parameter values (shell=True, --force, -rf /) and use safe defaults.
(Tool Misuse (TM1))
[error] 201: [TM1] Tool Parameter Abuse: Tool parameters are crafted to achieve unintended or unsafe behavior. Parameter abuse can bypass intended safety checks (e.g. shell=True, --force, dangerous glob patterns).
Remediation: Validate all tool parameters against an allowlist. Reject dangerous parameter values (shell=True, --force, -rf /) and use safe defaults.
(Tool Misuse (TM1))
[error] 202: [YR1] YARA rule 'agent_skill_destructive_autonomous_actions': Autonomous destructive filesystem, shell history, or repository actions in AI agent skills [agent_skills]: YARA rule matched a known malware signature (reverse shell, backdoor, ransomware, C2 framework, or info stealer).
Remediation: Remove the malware payload or compromised file entirely. Investigate how it entered the skill and audit all other artifacts for additional indicators of compromise.
(YARA Match (YR1))
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.claude/skills/policy-author/SKILL.md around lines 387 - 389, The
coverage-check command in SKILL.md must inspect all supported custom-policy
extensions, not only *policies.mjs. Update the policy listing and grep pattern
to include .js, .mjs, and .ts files while preserving the existing name and
description extraction behavior.
| // Match the audit subcommand only — not `policies`, `config`, or a bare | ||
| // `--version`. Tolerates the npx / dev-hook / bare-binary invocations. | ||
| if (!/\bfailproofai\b[^\n|;&]*\baudit\b/.test(command)) return allow(); | ||
|
|
||
| // `--help` is not a run, and there are no findings to triage. | ||
| if (/\B--help\b|\B-h\b/.test(command)) return allow(); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
Bind audit detection and help exclusion to the same parsed invocation.
The raw-text regex does not require audit to be the subcommand (failproofai config get audit matches), while the separate help check scans the entire shell string (failproofai audit && echo --help suppresses the real audit). Parse command segments/argv, or otherwise identify the actual failproofai invocation before applying the help exclusion, and add regression cases.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.failproofai/policies/audit-skill-policies.mjs around lines 24 - 29, Update
the audit detection logic around the command parsing in the policy so it
identifies the actual failproofai invocation and requires audit to be its
subcommand, rather than matching audit anywhere in the raw shell command. Apply
the --help/-h exclusion only to that same parsed invocation, preserving audit
execution when help appears in a separate command segment, and add regression
cases for both false positives.
|
|
||
| ### Features | ||
| - Add a `policy-author` skill (`.claude/skills/policy-author/`) that turns the two things users already have — an audit report and a prose rules file — into enforced policies. It exists because the audit's fix vocabulary is closed over the builtins: `DETECTOR_TO_POLICY` maps every audit-only detector to some builtin, and its own header comment says why ("so every finding looks like it has a failproofai fix"), but several of those mappings prevent nothing (`git-commit-no-verify` → `warn-git-amend` is a different command), and nothing in the audit can say "this one needs a policy you don't have yet" or notice that a hand-written policy already covers it. Four entry paths converge on one authoring core: the audit trigger below, a plain-language complaint ("agents keep force-pushing" — users report problems, they don't request policies), an explicit ask, and a rules file (`CLAUDE.md`/`AGENTS.md`) whose rules keep getting ignored. The load-bearing rule is that checking coverage comes *before* authoring — including **parameterized** builtins and the project's existing custom policies — because most requests are one line in `enabledPolicies`, not a new file; blind agents given two tasks with deliberately opposite correct answers diverged as intended, one enabling `block-sudo` and writing zero files while the other authored a policy and touched zero config. Authoring picks one of three modes matching the builtins' own naming (`block-*`/`deny()`, `warn-*`/`instruct()` carrying the builtins' "STOP: … Confirm with the user before executing" voice, `sanitize-*`), defaulting to oversight whenever an action has any legitimate use, since blocking those just gets the policy disabled. `references/traps.md` documents nine silent failures verified by running the hook rather than reading code — chief among them the loader convention (`block-foo.mjs` is skipped in silence; this repo shipped `block-version-bumps.mjs` that way and it never ran), that loading and execution are both fail-open so an unloaded policy and a correctly-allowing one are indistinguishable, and that a deny in a test does not prove *your* policy denied. `scripts/test-policy.mjs` runs a policy against synthetic payloads in a sandbox that enables zero builtins (so any deny is necessarily yours) and handles all six deny shapes across the CLIs, including `PermissionRequest`'s `decision.behavior` and Factory's exit-2; `scripts/sync-builtins.mjs` regenerates the builtin reference from `BUILTIN_POLICIES` and `--check` exits 1 on drift. Validated across five rounds of blind agents in seeded sandboxes covering sixteen use-case cells — triage, complaint-to-builtin, custom authoring, oversight, sanitize, custom Stop gates, dead-policy diagnosis, and both directions of the scope guardrail (a question about machine-wide findings must not modify the machine; an explicit "do it globally" must not be refused). (#600) | ||
| - Point agents at that skill after an audit, via `.failproofai/policies/audit-skill-policies.mjs`. A skill cannot invoke itself — it is pulled in when a request matches its description — which covers "fix these findings" but not the case where someone simply runs `failproofai audit` and reads the output. This dogfood policy closes that gap with failproofai's own enforcement layer: a `PostToolUse` rule matching the audit subcommand returns an `instruct()` naming the cache path and the two things a fresh agent otherwise gets wrong (findings live at `.result.results[]` under an envelope, and `enabledInConfig` is a snapshot taken at audit time that goes stale — observed reporting all 39 builtins enabled five minutes after the global config was emptied). Guarded against the obvious false positives: `npm audit`, `grep -r audit` and `failproofai audit --help` all pass through. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Add (#600) to every changelog entry.
Line 6 includes the PR number, but the new entries on Lines 7 and 10 do not. As per coding guidelines, every changelog entry must be a single-line description including the PR number.
Also applies to: 10-10
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@CHANGELOG.md` at line 7, Update the changelog entries described in the diff,
including the entry around the audit-skill policy and the entry at the
corresponding later location, so each is a single-line description containing
the PR marker (`#600`). Preserve their existing content while adding the marker
consistently to every new entry.
Source: Coding guidelines
|
Superseded by FailproofAI/skills#12 — the skill belongs in the org's central skills collection, where it ships via Ported rather than copied: the self-referencing script paths, the dev-only Not carried over, and worth deciding separately: the |
2 participants
Why
failproofai audittells you what your agents did wrong, but its fix vocabulary is closed over the 39 builtins.DETECTOR_TO_POLICYmaps every audit-only detector to one of them, and the source comment is candid about the reason — "so every finding looks like it has a failproofai fix" — yet several mappings prevent nothing (git-commit-no-verify→warn-git-amendis a different command entirely). Nothing in the audit can say "this one needs a policy you don't have yet", and nothing reads.failproofai/policies/to notice a rule is already covered.This repo already solved that by hand:
workflow-policies.mjsandblock-version-bumps-policies.mjsare CLAUDE.md rules somebody converted into enforcement because the prose kept getting ignored. The skill automates both halves of that — audit findings and rules files.What it does
Four entry paths converge on one authoring core:
PostToolUsepolicy fires onfailproofai auditand points the agent at the skill.CLAUDE.md/AGENTS.md, classify them, enforce the enforceable subset, and say honestly what stays prose.The load-bearing rule is that coverage checking happens before authoring — builtins including their params, plus the project's existing custom policies — because most requests are one line in
enabledPolicies, not a new file. Authoring then picks one of three modes matching the builtins' own naming (block-*/deny(),warn-*/instruct(),sanitize-*), defaulting to oversight whenever an action has legitimate uses, since blocking those just gets the policy disabled.What's in it
SKILL.mdreferences/traps.mdreferences/api.mdreferences/builtins.mdreferences/patterns.mdreferences/rules-files.mdscripts/test-policy.mjsscripts/sync-builtins.mjs--checkfor drifttraps.mdis the densest part, and every entry was verified by running the hook rather than reading code:block-foo.mjs— this repo shipped one that way and it never ran.test-policy.mjssandboxes a policy into a throwaway project with zero builtins enabled, so any deny is necessarily the policy under test, and it handles all six deny shapes across the CLIs (includingPermissionRequest'sdecision.behaviorand Factory's exit-2 — matching onlypermissionDecisionunder-reports blocks).Validation
Five rounds of blind agents in seeded sandboxes, sixteen use-case cells: triage, complaint→builtin, custom authoring, oversight, sanitize, custom Stop gates, dead-policy diagnosis, and both directions of the scope guardrail — a question about machine-wide findings must not modify the machine, and an explicit "do it globally" must not be refused.
Two moments worth calling out, both cases where a blind agent's judgment beat the test rubric:
prefer-package-manager— the builtin the skill's own worked example recommends — because its matcher is a bare\bnpm\b, which would denynpm pack --ignore-scripts, a step CLAUDE.md's own Docker testing protocol mandates. Verified: it does. That caveat is now inrules-files.md.rm -rf build/,DROP TABLE IF EXISTS, and Makefile writes asinstruct; a later hand-hardening pass flipped everywarntodenyand thereby blocked legitimate task work. The skill's mode judgment was the more correct one.Note on
.gitignore.claude/*excluded everything butsettings.json, so a skill committed here would have silently never reached the tree. The carve-out follows the existing!.claude/settings.jsonpattern.Gates
bun run lint0 errors ·tsc --noEmitclean ·bun run test:run2357/2357 across 131 files.Not included:
.failproofai/policies-config.jsonstays out of this PR — the local copy carries a machine-specificallowPathsentry.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation