From 7c587a192ff8295156666283827dbaf9204f9907 Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Tue, 15 Sep 2026 15:18:14 +0530 Subject: [PATCH 1/6] feat: add native Hermes policy plugin --- CHANGELOG.md | 9 + CLAUDE.md | 92 ++-- .../fixtures/hermes-native-plugin-check.py | 519 ++++++++++++++++++ __tests__/hooks/hermes-native-plugin.test.ts | 19 + __tests__/hooks/integrations.test.ts | 246 +++++++-- __tests__/hooks/scope-attribution.test.ts | 13 +- __tests__/hooks/worker-server.test.ts | 47 ++ bin/failproofai.mjs | 4 +- crates/failproofaid/src/server.rs | 124 ++++- crates/failproofaid/src/worker.rs | 25 + crates/fpai-ipc/src/envelope.rs | 60 ++ docs/reference/harnesses.mdx | 16 +- hermes-plugin/README.md | 133 +++++ hermes-plugin/__init__.py | 248 +++++++++ hermes-plugin/client.py | 134 +++++ hermes-plugin/ledger.py | 187 +++++++ hermes-plugin/plugin.yaml | 37 ++ package.json | 1 + src/hooks/handler.ts | 30 +- src/hooks/integrations.ts | 337 +++++++++--- src/hooks/manager.ts | 22 +- src/hooks/types.ts | 16 +- src/hooks/worker-server.ts | 1 + 23 files changed, 2134 insertions(+), 186 deletions(-) create mode 100644 __tests__/fixtures/hermes-native-plugin-check.py create mode 100644 __tests__/hooks/hermes-native-plugin.test.ts create mode 100644 hermes-plugin/README.md create mode 100644 hermes-plugin/__init__.py create mode 100644 hermes-plugin/client.py create mode 100644 hermes-plugin/ledger.py create mode 100644 hermes-plugin/plugin.yaml diff --git a/CHANGELOG.md b/CHANGELOG.md index 5a6e50b25..98a76f2c9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,15 @@ - Translate the English documentation changes from #788 and #791 into all 14 locales, including the new evaluation pages and SDK event redaction on the custom-agents page, and point translated fragment links at their translated headings (#797) +### Added + +- Add a native, profile-local Hermes plugin that evaluates policies through the existing failproofaid warm worker. Hermes `instruct()` decisions now interrupt the first matching tool attempt with model-visible guidance, then use a persistent bounded retry ledger so advisory policy cannot deadlock the turn; `deny()` remains a hard block. +- Add structured `policyEvaluation` / `policyResult` messages to the versioned local daemon protocol for native integrations, including canonical tool, policy, match, reason, and latency metadata. + +### Changed + +- Hermes installation now copies and enables the managed plugin in every profile, migrates only legacy FailproofAI shell hooks, refuses to overwrite unmanaged plugin directories, and reports incomplete or duplicate profile installations as unhealthy. + ### Dependencies - yaml 2.9.0 → 2.9.1, and rustls 0.23.43 → 0.23.45 (with rustls-webpki 0.103.13 → 0.103.15) in `Cargo.lock`, closing RUSTSEC-2026-0285 (5.3, fixed in 0.23.45). The advisory turned the Supply Chain gate red on `main` itself, not through any PR's change; rustls is transitive-only, via `reqwest` in `failproofaid` and `fpai-collect` (#803) diff --git a/CLAUDE.md b/CLAUDE.md index 4665197a6..a60d2de75 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -321,13 +321,12 @@ we run in-repo. Hermes is a **dual-pillar** integration: an **audit** adapter (`src/audit/cli-adapters/hermes.ts`, reads `~/.hermes/state.db` directly) **and** a **live-hook** integration (`hermes` in `INTEGRATIONS`). -Hermes uses a **Claude/Codex-style external shell-hook system**, but its config is -**YAML** (`~/.hermes/config.yaml`) under a `hooks:` map — the only integration whose -settings file is YAML, so `integrations.ts` has a comment-preserving `readYamlDoc`/ -`writeYamlDoc` layer (the `yaml` package's `Document` API) that rewrites only the -`hooks:` key, preserving the operator's other settings and any comments **outside** -that block. (Comments *inside* the `hooks:` map are not preserved — we rebuild the -key from `doc.toJS()` — but failproofai owns that block, so there's nothing to keep.) +Hermes enforcement uses the shipped **native Python plugin** under each profile's +`plugins/failproofai/` directory. The profile's YAML config enables it through +`plugins.enabled: [failproofai]`. `integrations.ts` copies the plugin atomically, +marks the directory as FailproofAI-managed, refuses to overwrite an unmanaged +directory with the same name, and uses the `yaml` package's comment-preserving +`Document` API for config changes. Settings file paths: @@ -336,21 +335,21 @@ Settings file paths: | user | `~/.hermes/config.yaml` | Hermes is **user-scope only** — there is no project config, so `getSettingsPath` -ignores scope/cwd. Hermes exposes no `$HERMES_PROJECT_DIR`; the installed command uses -the resolved binary path (`"${binaryPath}" --hook --cli hermes`) — since -Hermes is user-scope only, no `npx` project form applies. `timeout` is in **seconds** (30). - -**Consent (headless gateway).** Hermes prompts once per unique `(event, command)` hook -before running it. The gateway has no TTY, so install also writes -`hooks_auto_accept: true` into config.yaml (uninstall removes it). Tradeoff: this -auto-accepts *any* hook the operator adds — a deliberate choice for headless operation. -A more targeted alternative is to pre-seed `~/.hermes/shell-hooks-allowlist.json` with -just our `(event, command)` pairs; deferred as a future refinement. - -**Block contract** (verified live): Hermes reads a `{"decision":"block","reason"}` JSON -object on **stdout** and **ignores exit codes**. So `policy-evaluator.ts` has a -`cli === "hermes"` deny branch (ahead of the Stop block) that emits that shape -unconditionally for every event — one branch covers PreToolUse/PostToolUse/SubagentStop. +ignores scope/cwd. Every default and named profile receives its own plugin copy and +enablement entry. Installed-state detection requires both the complete managed plugin +directory and the config entry; a missing file, disabled plugin, newly-created profile, +or leftover legacy shell hook is reported as unhealthy. + +The plugin talks directly to `failproofaid` over the existing owner-only Unix socket. +The daemon relays a versioned `policyEvaluation` request to the warm TypeScript worker +and returns a structured `policyResult` with `allow`, `deny`, or `instruct`, policy +names, reason, matched policies, canonical tool name, and evaluation duration. The +normal native path does not parse CLI stdout and does not spawn a process per event. + +The contract is pinned to upstream Hermes 0.21.3 / commit +`4d55ca91656ac5f83e1506679b7f81e0238e5e16`: plugin manifest v2, `PluginState.data_dir`, +keyword hook payloads, and `pre_tool_call` correlation IDs are all present there. +Callbacks accept `**kwargs` for additive compatibility. **Platform independence & subagents.** The gateway is one Hermes process and `pre_tool_call` fires on the *tool event*, not the source — so a single install @@ -362,17 +361,36 @@ internal tool calls don't fire Hermes hooks — gate the *spawn* at `pre_tool_ca **Per-event capability matrix:** -| Hermes event | Canonical (`HERMES_EVENT_MAP`) | Veto / mutate? | Notes | -|--------------------|--------------------------------|----------------|-------| -| `pre_tool_call` | `PreToolUse` | ✅ block | The core deny point — stops the tool before it runs. | -| `post_tool_call` | `PostToolUse` | observation | Observe / sanitize. | -| `on_session_start` | `SessionStart` | observation | — | -| `on_session_end` | `SessionEnd` | observation | — | -| `subagent_stop` | `SubagentStop` | observation | **NOT a gate** — see the correction below. | -| `pre_verify` | *(not installed)* | ✅ block | Real turn-end gate upstream — **we deliberately do not install it**; see below. | - -**Corrections (2026-07-29).** Three claims that stood here were wrong, each verified -against upstream `hermes-agent` @ `5771a6e`. `agent/shell_hooks.py:567-621` +| Hermes plugin hook | Canonical (`HERMES_EVENT_MAP`) | Effect | Notes | +|--------------------|--------------------------------|--------|-------| +| `pre_tool_call` | `PreToolUse` | ✅ block | Deny always blocks. Instruct blocks once for model-visible delivery, then the next API iteration may proceed. | +| `post_tool_call` | `PostToolUse` | observation | Return value is ignored by Hermes. | +| `pre_llm_call` | plugin-local | context | Adds the stable instruction protocol hint to the current turn. | +| `on_session_start` | `SessionStart` | observation | Forwarded to the evaluator. | +| `on_session_end` | `SessionEnd` | observation + cleanup | Forwarded, then clears that session's instruction ledger. | +| `on_session_reset` / `on_session_finalize` | plugin-local | cleanup | Clears stale per-session retry state. | +| `subagent_stop` | `SubagentStop` | observation | Not a gate. | +| `pre_verify` | *(not installed)* | none | Hermes exposes a bounded turn-end gate, but FailproofAI does not map it yet. | + +**Bounded `instruct()` delivery.** The first matching instruction in a +profile/session/task/turn/policy/tool scope is persisted to the profile-local SQLite +ledger before `pre_tool_call` returns a block. Re-entry with the same +`api_request_id` remains blocked, which prevents sibling calls from consuming the +permit. A later API request acknowledges the instruction and proceeds. At most two +distinct instruction scopes may interrupt one turn by default. Missing correlation +IDs or ledger failure fail open for advisory instructions so a broken ledger cannot +deadlock Hermes; a real `deny()` remains a hard block. Evaluator or protocol failure +defaults to fail closed and may be changed per profile with +`plugins.entries.failproofai.settings.failure_mode: allow`. + +Install migrates only shell-hook entries carrying the FailproofAI marker or legacy +`failproofai --hook` command. It preserves unrelated hooks and removes the old broad +`hooks_auto_accept` setting only when no hook remains. The native plugin itself needs +no shell-hook consent prompt. + +**Legacy shell-adapter findings (retained for migration context).** Three earlier +claims were corrected against upstream `hermes-agent` @ `5771a6e`. +`agent/shell_hooks.py:567-621` (`_parse_response`) is **event-gated**: it returns a verdict for `pre_tool_call` and `pre_verify` only, and falls through to `return None` for everything else. So: @@ -398,10 +416,10 @@ against upstream `hermes-agent` @ `5771a6e`. `agent/shell_hooks.py:567-621` (`agent/verify_hooks.py:21`, operator-overridable), and on Hermes older than ~2026-06-30 the config key fails `VALID_HOOKS` and is **warn-and-skipped silently** (`agent/shell_hooks.py:325`). -3. **"No additional-context channel" was false.** `pre_llm_call` consumes - `{"context": str}` via the parser's fallthrough (`shell_hooks.py:617-621`). We do - not install it, so `instruct()` still degrades to allow + a stderr note — but that - is now a gap we chose, not a limit of the platform. +3. **"No additional-context channel" was false.** Native `pre_llm_call` callbacks + consume `{"context": str}`. The plugin uses that channel for a stable protocol + hint, while each concrete `instruct()` reason is delivered through the blocked + `pre_tool_call` result that Hermes inserts into model-visible history. Hermes still lacks `UserPromptSubmit` (only per-LLM-call `pre_llm_call`), `PreCompact`/`Notification`, etc. In exchange it has capabilities others lack diff --git a/__tests__/fixtures/hermes-native-plugin-check.py b/__tests__/fixtures/hermes-native-plugin-check.py new file mode 100644 index 000000000..6eb95ce77 --- /dev/null +++ b/__tests__/fixtures/hermes-native-plugin-check.py @@ -0,0 +1,519 @@ +from __future__ import annotations + +import importlib.util +import json +import socket +import struct +import sys +import tempfile +import threading +import unittest +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path +from types import SimpleNamespace +from unittest.mock import patch + + +REPO_ROOT = Path(__file__).resolve().parents[2] +PLUGIN_ROOT = REPO_ROOT / "hermes-plugin" + + +def load_plugin(): + package_name = "failproofai_hermes_plugin_test" + spec = importlib.util.spec_from_file_location( + package_name, + PLUGIN_ROOT / "__init__.py", + submodule_search_locations=[str(PLUGIN_ROOT)], + ) + if spec is None or spec.loader is None: + raise RuntimeError("could not load Hermes plugin") + module = importlib.util.module_from_spec(spec) + sys.modules[package_name] = module + spec.loader.exec_module(module) + return module + + +plugin = load_plugin() +client = sys.modules[f"{plugin.__name__}.client"] +ledger_mod = sys.modules[f"{plugin.__name__}.ledger"] + + +class FakeContext: + def __init__(self, data_dir: Path, settings: dict[str, object] | None = None) -> None: + self.state = SimpleNamespace(data_dir=data_dir) + self.settings = settings or {} + self.hooks: dict[str, object] = {} + + def get_config(self, key: str, default=None): + return self.settings.get(key, default) + + def register_hook(self, name: str, callback) -> None: + self.hooks[name] = callback + + +class LedgerTests(unittest.TestCase): + def test_instruction_blocks_once_then_allows_a_later_api_iteration(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / "instructions.db" + ledger = ledger_mod.InstructionLedger(path, ttl_seconds=60, max_rounds=2) + base = dict( + profile="default", + session_id="session-1", + task_id="task-1", + turn_id="turn-1", + policy_names=("custom/write-route",), + reason="Use the approved write route.", + scope_key="Write", + now_ms=1_000, + ) + self.assertEqual( + ledger.decide(api_request_id="api-1", **base), + ledger_mod.InstructionAction(True, "issued"), + ) + self.assertEqual( + ledger.decide(api_request_id="api-1", **base), + ledger_mod.InstructionAction(True, "same-api-request"), + ) + + # A fresh object proves the permit survives plugin-object restart. + restarted = ledger_mod.InstructionLedger(path, ttl_seconds=60, max_rounds=2) + self.assertEqual( + restarted.decide(api_request_id="api-2", **base), + ledger_mod.InstructionAction(False, "acknowledged"), + ) + self.assertEqual( + restarted.decide(api_request_id="api-3", **base), + ledger_mod.InstructionAction(False, "acknowledged"), + ) + + def test_same_api_request_siblings_block_and_turn_cap_is_bounded(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + ledger = ledger_mod.InstructionLedger( + Path(tmp) / "instructions.db", ttl_seconds=60, max_rounds=1 + ) + common = dict( + profile="default", + session_id="session-1", + task_id="task-1", + turn_id="turn-1", + api_request_id="api-1", + scope_key="Write", + now_ms=1_000, + ) + first = ledger.decide( + policy_names=("custom/first",), reason="first", **common + ) + sibling = ledger.decide( + policy_names=("custom/first",), reason="first", **common + ) + capped = ledger.decide( + policy_names=("custom/second",), reason="second", **common + ) + self.assertEqual(first.status, "issued") + self.assertTrue(sibling.block) + self.assertEqual(capped, ledger_mod.InstructionAction(False, "turn-cap")) + + def test_missing_iteration_identity_fails_open_for_instruct(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + ledger = ledger_mod.InstructionLedger(Path(tmp) / "instructions.db") + action = ledger.decide( + profile="default", + session_id="session-1", + task_id="task-1", + turn_id="", + api_request_id="", + policy_names=("custom/write-route",), + reason="guide", + scope_key="Write", + ) + self.assertEqual(action, ledger_mod.InstructionAction(False, "missing-correlation-id")) + + def test_state_isolated_by_profile_session_task_turn_policy_and_scope(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + ledger = ledger_mod.InstructionLedger( + Path(tmp) / "instructions.db", ttl_seconds=60, max_rounds=10 + ) + base = dict( + profile="default", + session_id="session-1", + task_id="task-1", + turn_id="turn-1", + api_request_id="api-1", + policy_names=("custom/write-route",), + reason="Use the approved write route.", + scope_key="Write", + now_ms=1_000, + ) + self.assertTrue(ledger.decide(**base).block) + for changed in ( + {"profile": "other"}, + {"session_id": "session-2"}, + {"task_id": "task-2"}, + {"turn_id": "turn-2"}, + {"policy_names": ("custom/other",)}, + {"scope_key": "Bash"}, + ): + self.assertTrue(ledger.decide(**{**base, **changed}).block) + + def test_expired_delivery_blocks_again_and_session_cleanup_removes_state(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + ledger = ledger_mod.InstructionLedger( + Path(tmp) / "instructions.db", ttl_seconds=60, max_rounds=2 + ) + base = dict( + profile="default", + session_id="session-1", + task_id="task-1", + turn_id="turn-1", + policy_names=("custom/write-route",), + reason="Use the approved write route.", + scope_key="Write", + ) + self.assertTrue(ledger.decide(api_request_id="api-1", now_ms=1_000, **base).block) + self.assertFalse(ledger.decide(api_request_id="api-2", now_ms=2_000, **base).block) + ledger.clear_session("session-1") + self.assertEqual( + ledger.decide(api_request_id="api-3", now_ms=3_000, **base).status, + "issued", + ) + self.assertEqual( + ledger.decide(api_request_id="api-4", now_ms=63_001, **base).status, + "issued", + ) + + def test_concurrent_siblings_from_one_api_response_both_block(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + ledger = ledger_mod.InstructionLedger( + Path(tmp) / "instructions.db", ttl_seconds=60, max_rounds=2 + ) + ledger._ensure_schema() + barrier = threading.Barrier(2) + + def decide(): + barrier.wait(timeout=2) + return ledger.decide( + profile="default", + session_id="session-1", + task_id="task-1", + turn_id="turn-1", + api_request_id="api-1", + policy_names=("custom/write-route",), + reason="Use the approved write route.", + scope_key="Write", + now_ms=1_000, + ) + + with ThreadPoolExecutor(max_workers=2) as pool: + results = list(pool.map(lambda _: decide(), range(2))) + self.assertTrue(all(result.block for result in results)) + self.assertEqual( + {result.status for result in results}, + {"issued", "same-api-request"}, + ) + + def test_corrupt_database_raises_a_ledger_error(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / "instructions.db" + path.write_text("not sqlite", encoding="utf-8") + ledger = ledger_mod.InstructionLedger(path) + with self.assertRaises(ledger_mod.LedgerError): + ledger.decide( + profile="default", + session_id="session-1", + task_id="task-1", + turn_id="turn-1", + api_request_id="api-1", + policy_names=("custom/write-route",), + reason="guide", + scope_key="Write", + ) + + +class PluginTests(unittest.TestCase): + def test_registers_the_supported_hermes_hooks(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + ctx = FakeContext(Path(tmp)) + plugin.register(ctx) + self.assertEqual( + set(ctx.hooks), + { + "pre_tool_call", + "post_tool_call", + "pre_llm_call", + "on_session_start", + "on_session_end", + "on_session_reset", + "on_session_finalize", + "subagent_stop", + }, + ) + + def test_deny_always_blocks(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + instance = plugin.FailproofAIPlugin(FakeContext(Path(tmp))) + verdict = client.PolicyVerdict( + decision="deny", + policy_names=("failproofai/block-sudo",), + reason="Do not use sudo.", + matched_policies=("failproofai/block-sudo",), + duration_ms=1, + tool_name="Bash", + ) + with patch.object(plugin, "evaluate_policy", return_value=verdict): + result = instance.pre_tool_call( + tool_name="terminal", + args={"command": "sudo whoami"}, + session_id="s", + turn_id="t", + api_request_id="a", + ) + self.assertEqual(result, {"action": "block", "message": "Do not use sudo."}) + + def test_instruct_blocks_once_and_then_allows_the_retry(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + instance = plugin.FailproofAIPlugin(FakeContext(Path(tmp))) + verdict = client.PolicyVerdict( + decision="instruct", + policy_names=("custom/write-route",), + reason="Use the approved write route.", + matched_policies=("custom/write-route",), + duration_ms=1, + tool_name="Write", + ) + with patch.object(plugin, "evaluate_policy", return_value=verdict): + first = instance.pre_tool_call( + tool_name="write_file", + args={"path": "/tmp/a"}, + session_id="s", + task_id="root", + turn_id="t", + api_request_id="a1", + ) + retry = instance.pre_tool_call( + tool_name="write_file", + args={"path": "/tmp/a"}, + session_id="s", + task_id="root", + turn_id="t", + api_request_id="a2", + ) + self.assertEqual(first["action"], "block") + self.assertIn("FAILPROOF INSTRUCTION", first["message"]) + self.assertIn("Do not route around", first["message"]) + self.assertIsNone(retry) + + def test_instruction_state_failure_allows_but_evaluator_failure_blocks(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + instance = plugin.FailproofAIPlugin(FakeContext(Path(tmp))) + verdict = client.PolicyVerdict( + decision="instruct", + policy_names=("custom/write-route",), + reason="Use the approved route.", + matched_policies=("custom/write-route",), + duration_ms=1, + tool_name="Write", + ) + with patch.object(plugin, "evaluate_policy", return_value=verdict), patch.object( + instance.ledger, "decide", side_effect=ledger_mod.LedgerError("broken") + ): + self.assertIsNone( + instance.pre_tool_call( + tool_name="write_file", + session_id="s", + turn_id="t", + api_request_id="a", + ) + ) + + with patch.object( + plugin, "evaluate_policy", side_effect=client.EvaluationError("offline") + ): + result = instance.pre_tool_call(tool_name="terminal") + self.assertEqual(result["action"], "block") + self.assertIn("evaluator is unavailable", result["message"]) + + def test_deny_is_never_weakened_by_an_existing_instruction_permit(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + instance = plugin.FailproofAIPlugin(FakeContext(Path(tmp))) + instruct = client.PolicyVerdict( + decision="instruct", + policy_names=("custom/write-route",), + reason="Use the approved route.", + matched_policies=("custom/write-route",), + duration_ms=1, + tool_name="Write", + ) + deny = client.PolicyVerdict( + decision="deny", + policy_names=("custom/write-route",), + reason="This write is forbidden.", + matched_policies=("custom/write-route",), + duration_ms=1, + tool_name="Write", + ) + with patch.object(plugin, "evaluate_policy", side_effect=(instruct, instruct, deny)): + self.assertIsNotNone( + instance.pre_tool_call( + tool_name="write_file", + session_id="s", + turn_id="t", + api_request_id="a1", + ) + ) + self.assertIsNone( + instance.pre_tool_call( + tool_name="write_file", + session_id="s", + turn_id="t", + api_request_id="a2", + ) + ) + result = instance.pre_tool_call( + tool_name="write_file", + session_id="s", + turn_id="t", + api_request_id="a3", + ) + self.assertEqual( + result, + {"action": "block", "message": "This write is forbidden."}, + ) + + def test_evaluator_failure_can_be_configured_to_allow(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + instance = plugin.FailproofAIPlugin( + FakeContext(Path(tmp), {"failure_mode": "allow"}) + ) + with patch.object( + plugin, "evaluate_policy", side_effect=client.EvaluationError("offline") + ): + self.assertIsNone(instance.pre_tool_call(tool_name="terminal")) + + def test_observation_errors_never_break_the_agent(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + instance = plugin.FailproofAIPlugin(FakeContext(Path(tmp))) + with patch.object(plugin, "evaluate_policy", side_effect=RuntimeError("bug")): + self.assertIsNone( + instance.post_tool_call( + tool_name="terminal", + args={"command": "true"}, + result="ok", + session_id="s", + ) + ) + + def test_pre_llm_context_explains_instruction_results(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + instance = plugin.FailproofAIPlugin(FakeContext(Path(tmp))) + result = instance.pre_llm_call() + self.assertIn("FAILPROOF INSTRUCTION", result["context"]) + self.assertIn("Do not evade or ignore", result["context"]) + + +class ClientTests(unittest.TestCase): + def test_client_speaks_the_framed_policy_protocol(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + socket_path = Path(tmp) / "daemon.sock" + received: dict[str, object] = {} + ready = threading.Event() + + def server() -> None: + with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as listener: + listener.bind(str(socket_path)) + listener.listen(1) + ready.set() + connection, _ = listener.accept() + with connection: + length = struct.unpack(">I", client._read_exact(connection, 4))[0] + received.update(json.loads(client._read_exact(connection, length))) + body = json.dumps( + { + "type": "policyResult", + "protocolVersion": 1, + "decision": "instruct", + "policyNames": ["custom/write-route"], + "reason": "Use the approved route.", + "matchedPolicies": ["custom/write-route"], + "durationMs": 3, + "toolName": "Write", + }, + separators=(",", ":"), + ).encode() + connection.sendall(struct.pack(">I", len(body)) + body) + + thread = threading.Thread(target=server, daemon=True) + thread.start() + self.assertTrue(ready.wait(timeout=2)) + with patch.dict("os.environ", {"FAILPROOFAI_DAEMON_SOCKET": str(socket_path)}): + verdict = client.evaluate_policy( + event="pre_tool_call", + payload={"tool_name": "write_file", "tool_input": {"path": "/tmp/a"}}, + cwd="/tmp", + ) + thread.join(timeout=2) + self.assertEqual(received["type"], "policyEvaluation") + self.assertEqual(received["integration"], "hermes") + self.assertEqual(verdict.decision, "instruct") + self.assertEqual(verdict.tool_name, "Write") + + def test_client_rejects_protocol_mismatch(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + socket_path = Path(tmp) / "daemon.sock" + ready = threading.Event() + + def server() -> None: + with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as listener: + listener.bind(str(socket_path)) + listener.listen(1) + ready.set() + connection, _ = listener.accept() + with connection: + length = struct.unpack(">I", client._read_exact(connection, 4))[0] + client._read_exact(connection, length) + body = json.dumps( + { + "type": "policyResult", + "protocolVersion": 99, + "decision": "allow", + "policyNames": [], + "matchedPolicies": [], + "durationMs": 0, + } + ).encode() + connection.sendall(struct.pack(">I", len(body)) + body) + + thread = threading.Thread(target=server, daemon=True) + thread.start() + self.assertTrue(ready.wait(timeout=2)) + with patch.dict("os.environ", {"FAILPROOFAI_DAEMON_SOCKET": str(socket_path)}): + with self.assertRaisesRegex(client.EvaluationError, "version mismatch"): + client.evaluate_policy(event="pre_tool_call", payload={}, cwd="/tmp") + thread.join(timeout=2) + + def test_client_rejects_an_oversized_response_before_reading_its_body(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + socket_path = Path(tmp) / "daemon.sock" + ready = threading.Event() + + def server() -> None: + with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as listener: + listener.bind(str(socket_path)) + listener.listen(1) + ready.set() + connection, _ = listener.accept() + with connection: + length = struct.unpack(">I", client._read_exact(connection, 4))[0] + client._read_exact(connection, length) + connection.sendall(struct.pack(">I", client.MAX_FRAME_BYTES + 1)) + + thread = threading.Thread(target=server, daemon=True) + thread.start() + self.assertTrue(ready.wait(timeout=2)) + with patch.dict("os.environ", {"FAILPROOFAI_DAEMON_SOCKET": str(socket_path)}): + with self.assertRaisesRegex(client.EvaluationError, "16 MiB limit"): + client.evaluate_policy(event="pre_tool_call", payload={}, cwd="/tmp") + thread.join(timeout=2) + + +if __name__ == "__main__": + unittest.main() diff --git a/__tests__/hooks/hermes-native-plugin.test.ts b/__tests__/hooks/hermes-native-plugin.test.ts new file mode 100644 index 000000000..5879b103b --- /dev/null +++ b/__tests__/hooks/hermes-native-plugin.test.ts @@ -0,0 +1,19 @@ +// @vitest-environment node +import { describe, expect, it } from "vitest"; +import { spawnSync } from "node:child_process"; +import { resolve } from "node:path"; + +describe("Hermes native plugin", () => { + it("passes its Python protocol, state-machine, and hook mapping tests", () => { + const result = spawnSync( + "python3", + [resolve("__tests__/fixtures/hermes-native-plugin-check.py")], + { + cwd: process.cwd(), + encoding: "utf8", + env: { ...process.env, PYTHONDONTWRITEBYTECODE: "1" }, + }, + ); + expect(result.status, `${result.stdout}\n${result.stderr}`).toBe(0); + }); +}); diff --git a/__tests__/hooks/integrations.test.ts b/__tests__/hooks/integrations.test.ts index 71fdc7d02..4344fa7f4 100644 --- a/__tests__/hooks/integrations.test.ts +++ b/__tests__/hooks/integrations.test.ts @@ -10,7 +10,15 @@ * • registry (getIntegration / listIntegrations) */ import { describe, it, expect, beforeEach, afterEach } from "vitest"; -import { mkdtempSync, rmSync, existsSync, readFileSync, writeFileSync, mkdirSync } from "node:fs"; +import { + mkdtempSync, + rmSync, + existsSync, + readFileSync, + writeFileSync, + mkdirSync, + readdirSync, +} from "node:fs"; import { tmpdir } from "node:os"; import { resolve, join } from "node:path"; import { @@ -29,6 +37,8 @@ import { getIntegration, listIntegrations, settingsPathsFor, + hermesProfileHealth, + hermesProfileStatusRows, unhookedHermesProfiles, } from "../../src/hooks/integrations"; import { @@ -554,6 +564,7 @@ describe("Hermes integration", () => { // on a throwaway file instead of the developer's real home. let origHome: string | undefined; let origHermesHome: string | undefined; + let origPackageRoot: string | undefined; beforeEach(() => { origHome = process.env.HOME; process.env.HOME = tempDir; @@ -561,12 +572,16 @@ describe("Hermes integration", () => { // with a profile-scoped shell doesn't get their real config.yaml touched. origHermesHome = process.env.HERMES_HOME; delete process.env.HERMES_HOME; + origPackageRoot = process.env.FAILPROOFAI_PACKAGE_ROOT; + process.env.FAILPROOFAI_PACKAGE_ROOT = ORIG_CWD; }); afterEach(() => { if (origHome === undefined) delete process.env.HOME; else process.env.HOME = origHome; if (origHermesHome === undefined) delete process.env.HERMES_HOME; else process.env.HERMES_HOME = origHermesHome; + if (origPackageRoot === undefined) delete process.env.FAILPROOFAI_PACKAGE_ROOT; + else process.env.FAILPROOFAI_PACKAGE_ROOT = origPackageRoot; }); /** Create `~/.hermes/profiles//` for each name. */ @@ -576,6 +591,17 @@ describe("Hermes integration", () => { } } + function pluginPath(settingsPath: string): string { + return resolve(dirname(settingsPath), "plugins", "failproofai"); + } + + function installAt(settingsPath: string): void { + hermes.prepareInstall!(settingsPath); + const settings = hermes.readSettings(settingsPath); + hermes.writeHookEntries(settings, "/usr/bin/failproofai", "user"); + hermes.writeSettings(settingsPath, settings); + } + it("getSettingsPath is user-scope ~/.hermes/config.yaml regardless of scope/cwd", () => { expect(hermes.getSettingsPath("user")).toBe(resolve(tempDir, ".hermes", "config.yaml")); // scope/cwd are ignored — Hermes has no project config. @@ -597,36 +623,29 @@ describe("Hermes integration", () => { expect(hermes.eventTypes).toContain("subagent_stop"); }); - it("buildHookEntry uses {command,timeout} with --cli hermes, timeout in SECONDS (30)", () => { + it("buildHookEntry identifies the shipped native plugin", () => { const entry = hermes.buildHookEntry("/usr/bin/failproofai", "pre_tool_call", "user") as Record; - expect(entry.command).toBe('"/usr/bin/failproofai" --hook pre_tool_call --cli hermes'); - expect(entry.timeout).toBe(30); expect(entry[FAILPROOFAI_HOOK_MARKER]).toBe(true); + expect(entry._hermesPluginPath).toBe(resolve(ORIG_CWD, "hermes-plugin")); }); - it("project scope uses npx -y failproofai (portable)", () => { - const entry = hermes.buildHookEntry("/usr/bin/failproofai", "pre_tool_call", "project") as Record; - expect(entry.command).toBe("npx -y failproofai --hook pre_tool_call --cli hermes"); - }); - - it("writeHookEntries builds a flat hooks: map keyed by snake_case events + hooks_auto_accept", () => { + it("installs the native plugin and enables it in config", () => { const path = hermes.getSettingsPath("user"); - const settings = hermes.readSettings(path); // empty Document (file absent) - hermes.writeHookEntries(settings, "/usr/bin/failproofai", "user"); - hermes.writeSettings(path, settings); + installAt(path); const parsed = parse(readFileSync(path, "utf-8")) as { - hooks?: Record>>; - hooks_auto_accept?: boolean; + plugins?: { enabled?: string[] }; + hooks?: unknown; }; - for (const eventType of HERMES_HOOK_EVENT_TYPES) { - expect(Array.isArray(parsed.hooks?.[eventType])).toBe(true); - const first = parsed.hooks![eventType][0]; - expect(first.command).toContain("--cli hermes"); - expect(first.timeout).toBe(30); - expect(first[FAILPROOFAI_HOOK_MARKER]).toBe(true); + expect(parsed.plugins?.enabled).toEqual(["failproofai"]); + expect(parsed.hooks).toBeUndefined(); + + const installedPlugin = pluginPath(path); + for (const file of ["plugin.yaml", "__init__.py", "client.py", "ledger.py", ".failproofai-managed"]) { + expect(existsSync(resolve(installedPlugin, file))).toBe(true); } - // Headless-gateway consent: declared hooks auto-accepted. - expect(parsed.hooks_auto_accept).toBe(true); + expect(hermesProfileStatusRows()).toEqual([ + ["hermes/default", "native plugin enabled"], + ]); }); it("readSettings/writeSettings preserve the user's other keys AND comments", () => { @@ -641,61 +660,187 @@ describe("Hermes integration", () => { // Unrelated keys survive... expect(parsed.model).toBe("gpt-5"); expect(parsed.provider).toBe("openai"); - expect((parsed.hooks as Record).pre_tool_call).toBeDefined(); + expect((parsed.plugins as { enabled: string[] }).enabled).toEqual(["failproofai"]); // ...and so do the user's comments (this is why we use the Document API). expect(raw).toContain("# my hermes config"); expect(raw).toContain("# the good one"); }); - it("re-running writeHookEntries is idempotent (replaces, doesn't duplicate)", () => { + it("re-running writeHookEntries is idempotent", () => { const path = hermes.getSettingsPath("user"); const settings = hermes.readSettings(path); hermes.writeHookEntries(settings, "/usr/bin/failproofai", "user"); hermes.writeHookEntries(settings, "/different/failproofai", "user"); hermes.writeSettings(path, settings); - const parsed = parse(readFileSync(path, "utf-8")) as { hooks: Record }; - expect(parsed.hooks.pre_tool_call).toHaveLength(1); - expect((parsed.hooks.pre_tool_call[0] as Record).command).toBe( - '"/different/failproofai" --hook pre_tool_call --cli hermes', - ); + const parsed = parse(readFileSync(path, "utf-8")) as { plugins: { enabled: string[] } }; + expect(parsed.plugins.enabled.filter((name) => name === "failproofai")).toHaveLength(1); }); - it("removeHooksFromFile strips only our entries + auto-accept, preserving user keys", () => { + it("migrates only FailproofAI legacy shell hooks and preserves operator hooks", () => { const path = hermes.getSettingsPath("user"); mkdirSync(dirname(path), { recursive: true }); - writeFileSync(path, "model: gpt-5\n"); + writeFileSync( + path, + [ + "model: gpt-5", + "hooks_auto_accept: true", + "hooks:", + " pre_tool_call:", + " - command: operator-check --tool", + " timeout: 5", + " - command: failproofai --hook pre_tool_call --cli hermes", + " " + FAILPROOFAI_HOOK_MARKER + ": true", + " post_tool_call:", + " - command: failproofai --hook post_tool_call --cli hermes", + " " + FAILPROOFAI_HOOK_MARKER + ": true", + "plugins:", + " enabled: [operator-plugin]", + " disabled: [failproofai, paused-plugin]", + "", + ].join("\n"), + ); const settings = hermes.readSettings(path); hermes.writeHookEntries(settings, "/usr/bin/failproofai", "user"); hermes.writeSettings(path, settings); + const parsed = parse(readFileSync(path, "utf-8")) as { + hooks: Record>; + hooks_auto_accept: boolean; + plugins: { enabled: string[]; disabled: string[] }; + }; + expect(parsed.hooks.pre_tool_call).toEqual([{ command: "operator-check --tool", timeout: 5 }]); + expect(parsed.hooks.post_tool_call).toBeUndefined(); + expect(parsed.hooks_auto_accept).toBe(true); + expect(parsed.plugins.enabled).toEqual(["operator-plugin", "failproofai"]); + expect(parsed.plugins.disabled).toEqual(["paused-plugin"]); + }); + + it("removeHooksFromFile removes plugin registration and managed files, preserving user keys", () => { + const path = hermes.getSettingsPath("user"); + mkdirSync(dirname(path), { recursive: true }); + writeFileSync(path, "model: gpt-5\n"); + installAt(path); + const removed = hermes.removeHooksFromFile(path); - expect(removed).toBe(HERMES_HOOK_EVENT_TYPES.length); + expect(removed).toBe(2); const parsed = parse(readFileSync(path, "utf-8")) as Record; - expect(parsed.hooks).toBeUndefined(); - expect(parsed.hooks_auto_accept).toBeUndefined(); + expect(parsed.plugins).toBeUndefined(); expect(parsed.model).toBe("gpt-5"); // user key survives + expect(existsSync(pluginPath(path))).toBe(false); }); - it("removeHooksFromFile drops hooks_auto_accept even when the hooks were already removed manually", () => { - // Regression (CodeRabbit): the auto-accept flag must not linger and silently - // auto-accept future operator hooks after our hooks are gone. + it("removeHooksFromFile cleans up a managed plugin even when config is missing", () => { + const path = hermes.getSettingsPath("user"); + hermes.prepareInstall!(path); + + expect(existsSync(path)).toBe(false); + expect(existsSync(pluginPath(path))).toBe(true); + expect(hermes.removeHooksFromFile(path)).toBe(1); + expect(existsSync(pluginPath(path))).toBe(false); + expect(existsSync(path)).toBe(false); + }); + + it("removeHooksFromFile does not delete an unowned hooks_auto_accept setting", () => { const path = hermes.getSettingsPath("user"); mkdirSync(dirname(path), { recursive: true }); writeFileSync(path, "model: gpt-5\nhooks_auto_accept: true\n"); // no `hooks:` block const removed = hermes.removeHooksFromFile(path); expect(removed).toBe(0); const parsed = parse(readFileSync(path, "utf-8")) as Record; - expect(parsed.hooks_auto_accept).toBeUndefined(); // dropped despite 0 hooks removed + expect(parsed.hooks_auto_accept).toBe(true); expect(parsed.model).toBe("gpt-5"); }); - it("hooksInstalledInSettings detects installed hooks / false when missing", () => { + it("hooksInstalledInSettings requires config enablement and complete managed files", () => { expect(hermes.hooksInstalledInSettings("user")).toBe(false); const path = hermes.getSettingsPath("user"); const settings = hermes.readSettings(path); hermes.writeHookEntries(settings, "/usr/bin/failproofai", "user"); hermes.writeSettings(path, settings); + expect(hermes.hooksInstalledInSettings("user")).toBe(false); + + hermes.prepareInstall!(path); expect(hermes.hooksInstalledInSettings("user")).toBe(true); + + rmSync(resolve(pluginPath(path), "client.py")); + expect(hermes.hooksInstalledInSettings("user")).toBe(false); + expect(hermesProfileHealth()[0]).toMatchObject({ + name: "default", + pluginEnabled: true, + pluginInstalled: false, + legacyShellHookPresent: false, + healthy: false, + }); + }); + + it("reports a duplicate legacy adapter as unhealthy", () => { + const path = hermes.getSettingsPath("user"); + installAt(path); + const settings = hermes.readSettings(path) as unknown as { + set(key: string, value: unknown): void; + }; + settings.set("hooks", { + pre_tool_call: [ + { + command: "failproofai --hook pre_tool_call --cli hermes", + [FAILPROOFAI_HOOK_MARKER]: true, + }, + ], + }); + hermes.writeSettings(path, settings as unknown as Record); + + expect(hermes.hooksInstalledInSettings("user")).toBe(false); + expect(hermesProfileHealth()[0]).toMatchObject({ + pluginEnabled: true, + pluginInstalled: true, + legacyShellHookPresent: true, + healthy: false, + }); + expect(hermesProfileStatusRows()).toEqual([ + ["hermes/default", "UNHEALTHY — legacy shell hook also present"], + ]); + }); + + it("atomically refreshes a managed plugin directory", () => { + const path = hermes.getSettingsPath("user"); + hermes.prepareInstall!(path); + const destination = pluginPath(path); + writeFileSync(resolve(destination, "client.py"), "stale\n"); + writeFileSync(resolve(destination, "obsolete.py"), "remove me\n"); + + hermes.prepareInstall!(path); + + expect(readFileSync(resolve(destination, "client.py"), "utf8")).not.toBe("stale\n"); + expect(existsSync(resolve(destination, "obsolete.py"))).toBe(false); + expect( + readdirSync(dirname(destination)).filter( + (name) => name.includes(".install-") || name.includes(".backup-"), + ), + ).toEqual([]); + }); + + it("refuses to overwrite an unmanaged plugin directory", () => { + const path = hermes.getSettingsPath("user"); + const destination = pluginPath(path); + mkdirSync(destination, { recursive: true }); + writeFileSync(resolve(destination, "plugin.yaml"), "name: operator-owned\n"); + + expect(() => hermes.prepareInstall!(path)).toThrow(/Refusing to overwrite an unmanaged Hermes plugin/); + expect(readFileSync(resolve(destination, "plugin.yaml"), "utf8")).toBe("name: operator-owned\n"); + }); + + it("leaves an existing managed plugin untouched when source validation fails", () => { + const path = hermes.getSettingsPath("user"); + hermes.prepareInstall!(path); + const destination = pluginPath(path); + const before = readFileSync(resolve(destination, "client.py"), "utf8"); + const brokenPackage = resolve(tempDir, "broken-package", "hermes-plugin"); + mkdirSync(brokenPackage, { recursive: true }); + writeFileSync(resolve(brokenPackage, "plugin.yaml"), "name: failproofai\n"); + process.env.FAILPROOFAI_PACKAGE_ROOT = resolve(tempDir, "broken-package"); + + expect(() => hermes.prepareInstall!(path)).toThrow(/asset is missing/); + expect(readFileSync(resolve(destination, "client.py"), "utf8")).toBe(before); }); // ── Profiles ── @@ -722,33 +867,26 @@ describe("Hermes integration", () => { makeProfiles("work"); const [rootPath, workPath] = settingsPathsFor(hermes, "user"); - const rootSettings = hermes.readSettings(rootPath); - hermes.writeHookEntries(rootSettings, "/usr/bin/failproofai", "user"); - hermes.writeSettings(rootPath, rootSettings); + installAt(rootPath); // Root hooked, `work` still bare → the gateway is only partly enforced. expect(hermes.hooksInstalledInSettings("user")).toBe(false); expect(unhookedHermesProfiles()).toEqual(["work"]); - const workSettings = hermes.readSettings(workPath); - hermes.writeHookEntries(workSettings, "/usr/bin/failproofai", "user"); - hermes.writeSettings(workPath, workSettings); + installAt(workPath); expect(hermes.hooksInstalledInSettings("user")).toBe(true); expect(unhookedHermesProfiles()).toEqual([]); }); - it("writes a real hooks block into a non-default profile's config.yaml", () => { + it("installs and enables the native plugin in a non-default profile", () => { makeProfiles("work"); const workPath = resolve(tempDir, ".hermes", "profiles", "work", "config.yaml"); writeFileSync(workPath, "model: gpt-5\n"); - const settings = hermes.readSettings(workPath); - hermes.writeHookEntries(settings, "/usr/bin/failproofai", "user"); - hermes.writeSettings(workPath, settings); + installAt(workPath); const parsed = parse(readFileSync(workPath, "utf-8")) as Record; - const hooks = parsed.hooks as Record; - expect(Object.keys(hooks).sort()).toEqual([...HERMES_HOOK_EVENT_TYPES].sort()); - expect(hooks.pre_tool_call[0].command).toContain("--cli hermes"); - expect(parsed.hooks_auto_accept).toBe(true); + expect((parsed.plugins as { enabled: string[] }).enabled).toEqual(["failproofai"]); + expect(parsed.hooks).toBeUndefined(); + expect(existsSync(resolve(pluginPath(workPath), "plugin.yaml"))).toBe(true); expect(parsed.model).toBe("gpt-5"); // operator key preserved }); diff --git a/__tests__/hooks/scope-attribution.test.ts b/__tests__/hooks/scope-attribution.test.ts index c83e829d1..6ff6094ba 100644 --- a/__tests__/hooks/scope-attribution.test.ts +++ b/__tests__/hooks/scope-attribution.test.ts @@ -49,15 +49,18 @@ describe("which scopes actually hold hooks", () => { writeFileSync( join(home, ".hermes", "config.yaml"), [ - "hooks:", - " pre_tool_call:", - " - type: command", - ' command: "failproofai --hook PreToolUse --cli hermes"', - " __failproofai_hook__: true", + "plugins:", + " enabled:", + " - failproofai", "", ].join("\n"), "utf8", ); + const pluginDir = join(home, ".hermes", "plugins", "failproofai"); + mkdirSync(pluginDir, { recursive: true }); + for (const file of ["plugin.yaml", "__init__.py", "client.py", "ledger.py", ".failproofai-managed"]) { + writeFileSync(join(pluginDir, file), "", "utf8"); + } const { integrationsInstalledAt } = await import("../../src/hooks/manager"); // Run from a directory that is NOT home, so the project-path collision diff --git a/__tests__/hooks/worker-server.test.ts b/__tests__/hooks/worker-server.test.ts index 1ade3d1da..3cee77064 100644 --- a/__tests__/hooks/worker-server.test.ts +++ b/__tests__/hooks/worker-server.test.ts @@ -129,6 +129,53 @@ describe("hooks/worker-server (real socket, real evaluation)", () => { expect(response.type).toBe("hookResult"); expect(response.exitCode).toBe(0); expect(permissionDecisionOf(response)).toBe("deny"); + expect(response.evaluation).toMatchObject({ + decision: "deny", + policyName: "failproofai/block-sudo", + policyNames: ["failproofai/block-sudo"], + toolName: "Bash", + }); + }); + + it("returns an explicit instruct verdict for native plugin adapters", async () => { + const policyPath = join(projectDir, "instruct-policy.mjs"); + writeFileSync( + policyPath, + `import { customPolicies, instruct } from "failproofai"; +customPolicies.add({ + name: "approved-write-route", + description: "guide writes through the approved route", + match: { events: ["PreToolUse"], tools: ["Write"] }, + fn: async () => instruct("Use the approved write route."), +});\n`, + ); + writeFileSync( + join(projectDir, ".failproofai", "policies-config.json"), + JSON.stringify({ enabledPolicies: [], customPoliciesPaths: [policyPath] }), + ); + + const response = await sendRequest(workerSocketPath, { + type: "hook", + hookEvent: "pre_tool_call", + cli: "hermes", + stdin: JSON.stringify({ + cwd: projectDir, + tool_name: "write_file", + tool_input: { path: join(projectDir, "output.txt"), content: "hello" }, + }), + }); + + expect(response.type).toBe("hookResult"); + expect(response.evaluation).toMatchObject({ + decision: "instruct", + policyName: "custom/approved-write-route", + policyNames: ["custom/approved-write-route"], + reason: "Use the approved write route.", + toolName: "Write", + }); + expect((response.evaluation as { matchedPolicies: string[] }).matchedPolicies).toEqual( + expect.arrayContaining(["custom/approved-write-route"]), + ); }); // The daemon forwards the hook with `json!({ "cwd": cwd })`, and serde writes diff --git a/bin/failproofai.mjs b/bin/failproofai.mjs index 58415f53a..9ce564925 100755 --- a/bin/failproofai.mjs +++ b/bin/failproofai.mjs @@ -2259,6 +2259,8 @@ async function runCli() { ); const opts = optsFor(process.stdout); const report = connectionStatusReport(); + const { hermesProfileStatusRows } = await import("../src/hooks/integrations"); + const hermesRows = hermesProfileStatusRows(); // The version line was written to be "the only place a user can find out // which daemon they are running" and then never called from anywhere. It // is the right thing for the heading to carry, and it retires a heading @@ -2276,7 +2278,7 @@ async function runCli() { // Always printed, including where reports can never work: "why am I // not getting them?" is the question --status exists to answer, and // an omitted line answers it with silence. - rows([...report.rows, ...(result.rows ?? [])], opts), + rows([...report.rows, ...hermesRows, ...(result.rows ?? [])], opts), report.warnings.length > 0 ? warning(report.warnings, opts) : null, // The trailer is not rows — it is the note and the resume command. // Rendering only `rows` dropped the one line that tells a paused diff --git a/crates/failproofaid/src/server.rs b/crates/failproofaid/src/server.rs index ab751eeb4..f5f13be20 100644 --- a/crates/failproofaid/src/server.rs +++ b/crates/failproofaid/src/server.rs @@ -340,6 +340,63 @@ fn dispatch(request: ClientMessage, worker: &Worker) -> ServerMessage { message: format!("worker call failed: {err}"), }, }, + ClientMessage::PolicyEvaluation { + integration, + event, + payload, + cwd, + .. + } => { + let stdin = match serde_json::to_string(&payload) { + Ok(value) => value, + Err(err) => { + return ServerMessage::Error { + protocol_version: PROTOCOL_VERSION, + message: format!("could not encode policy payload: {err}"), + }; + } + }; + match worker.call(&event, &integration, &stdin, cwd.as_deref()) { + Ok(outcome) => match outcome.evaluation { + Some(evaluation) + if matches!( + evaluation.decision.as_str(), + "allow" | "deny" | "instruct" + ) => + { + let policy_names = if evaluation.policy_names.is_empty() { + evaluation.policy_name.into_iter().collect() + } else { + evaluation.policy_names + }; + ServerMessage::PolicyResult { + protocol_version: PROTOCOL_VERSION, + decision: evaluation.decision, + policy_names, + reason: evaluation.reason, + matched_policies: evaluation.matched_policies, + duration_ms: evaluation.duration_ms, + tool_name: evaluation.tool_name, + } + } + Some(evaluation) => ServerMessage::Error { + protocol_version: PROTOCOL_VERSION, + message: format!( + "worker returned an invalid policy decision: {}", + evaluation.decision + ), + }, + None => ServerMessage::Error { + protocol_version: PROTOCOL_VERSION, + message: "worker did not return structured policy metadata".to_string(), + }, + }, + Err(err) => ServerMessage::Error { + protocol_version: PROTOCOL_VERSION, + message: format!("worker call failed: {err}"), + }, + } + } } } @@ -474,6 +531,33 @@ mod tests { } } + #[test] + fn policy_evaluation_gets_an_error_when_the_worker_cannot_be_reached() { + let socket_path = temp_socket_path("policy-broken-worker"); + let _guard = start_test_server(socket_path.clone()); + + let mut stream = UnixStream::connect(&socket_path).unwrap(); + write_message( + &mut stream, + &ClientMessage::PolicyEvaluation { + protocol_version: PROTOCOL_VERSION, + integration: "hermes".to_string(), + event: "pre_tool_call".to_string(), + payload: serde_json::json!({ + "tool_name": "terminal", + "tool_input": {"command": "echo hi"} + }), + cwd: None, + }, + ) + .unwrap(); + let response: ServerMessage = read_message(&mut stream).unwrap(); + match response { + ServerMessage::Error { .. } => {} + other => panic!("expected Error, got {other:?}"), + } + } + /// The real end-to-end path: a real `failproofaid` server relaying a /// real Hook request to the real TypeScript worker (spawned via `bun` /// against this repo's own `bin/failproofai-worker.mjs`), which runs @@ -481,7 +565,7 @@ mod tests { /// sides of the wire protocol actually agree, not just that each side's /// own unit tests pass in isolation. #[test] - fn relays_a_hook_request_to_the_real_typescript_worker_end_to_end() { + fn relays_hook_and_structured_policy_requests_to_the_real_worker() { let repo_root = std::path::Path::new(env!("CARGO_MANIFEST_DIR")) .parent() .and_then(|p| p.parent()) @@ -553,6 +637,44 @@ mod tests { other => panic!("expected HookResult, got {other:?}"), } + let mut stream = UnixStream::connect(&socket_path).unwrap(); + stream + .set_read_timeout(Some(Duration::from_secs(15))) + .unwrap(); + write_message( + &mut stream, + &ClientMessage::PolicyEvaluation { + protocol_version: PROTOCOL_VERSION, + integration: "hermes".to_string(), + event: "pre_tool_call".to_string(), + payload: serde_json::json!({ + "cwd": project_dir.to_string_lossy(), + "tool_name": "terminal", + "tool_input": { "command": "sudo rm -rf /" } + }), + cwd: Some(project_dir.to_string_lossy().to_string()), + }, + ) + .unwrap(); + let response: ServerMessage = read_message(&mut stream).unwrap(); + match response { + ServerMessage::PolicyResult { + decision, + policy_names, + tool_name, + .. + } => { + assert_eq!(decision, "deny"); + assert!( + policy_names + .iter() + .any(|name| name.ends_with("/block-sudo")) + ); + assert_eq!(tool_name.as_deref(), Some("Bash")); + } + other => panic!("expected PolicyResult, got {other:?}"), + } + std::fs::remove_dir_all(&project_dir).ok(); } diff --git a/crates/failproofaid/src/worker.rs b/crates/failproofaid/src/worker.rs index 05edfe65e..686526746 100644 --- a/crates/failproofaid/src/worker.rs +++ b/crates/failproofaid/src/worker.rs @@ -11,6 +11,7 @@ //! `failproofai` CLI version happens to be installed). use fpai_ipc::framing::{read_message, write_message}; +use serde::Deserialize; use serde_json::json; use std::io; use std::os::unix::net::UnixStream; @@ -47,6 +48,21 @@ pub struct HookOutcome { pub exit_code: i32, pub stdout: String, pub stderr: String, + pub evaluation: Option, +} + +#[derive(Debug, Clone, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct PolicyEvaluation { + pub decision: String, + pub policy_name: Option, + #[serde(default)] + pub policy_names: Vec, + pub reason: Option, + #[serde(default)] + pub matched_policies: Vec, + pub duration_ms: u64, + pub tool_name: Option, } /// How to launch the worker process. `FAILPROOFAI_WORKER_CMD` (dev/test @@ -414,10 +430,19 @@ impl Worker { .and_then(|v| v.as_str()) .unwrap_or("") .to_string(); + let evaluation = response + .get("evaluation") + .cloned() + .map(serde_json::from_value::) + .transpose() + .map_err(|err| { + WorkerError::BadResponse(format!("invalid evaluation metadata: {err}")) + })?; Ok(HookOutcome { exit_code, stdout, stderr, + evaluation, }) } Some("error") => { diff --git a/crates/fpai-ipc/src/envelope.rs b/crates/fpai-ipc/src/envelope.rs index 2d80a7f65..789d65517 100644 --- a/crates/fpai-ipc/src/envelope.rs +++ b/crates/fpai-ipc/src/envelope.rs @@ -37,6 +37,17 @@ pub enum ClientMessage { /// resolve project config or custom policies). cwd: Option, }, + /// Structured policy evaluation for native in-process integrations. + /// Unlike `Hook`, this does not expose CLI-specific stdout/stderr shapes: + /// the daemon returns an explicit allow/deny/instruct verdict instead. + #[serde(rename_all = "camelCase")] + PolicyEvaluation { + protocol_version: u32, + integration: String, + event: String, + payload: serde_json::Value, + cwd: Option, + }, } #[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] @@ -51,6 +62,17 @@ pub enum ServerMessage { stdout: String, stderr: String, }, + /// Integration-neutral policy result consumed by native plugins. + #[serde(rename_all = "camelCase")] + PolicyResult { + protocol_version: u32, + decision: String, + policy_names: Vec, + reason: Option, + matched_policies: Vec, + duration_ms: u64, + tool_name: Option, + }, /// The daemon accepted the connection and parsed the request, but could /// not produce a verdict (e.g. the worker is down/hung). Distinct from /// `HookResult` so a client can tell "ran and decided" apart from @@ -70,6 +92,9 @@ impl ClientMessage { ClientMessage::Hook { protocol_version, .. } => *protocol_version, + ClientMessage::PolicyEvaluation { + protocol_version, .. + } => *protocol_version, } } } @@ -81,6 +106,9 @@ impl ServerMessage { ServerMessage::HookResult { protocol_version, .. } => *protocol_version, + ServerMessage::PolicyResult { + protocol_version, .. + } => *protocol_version, ServerMessage::Error { protocol_version, .. } => *protocol_version, @@ -134,6 +162,22 @@ mod tests { } } + #[test] + fn policy_evaluation_uses_structured_camel_case_fields() { + let msg = ClientMessage::PolicyEvaluation { + protocol_version: PROTOCOL_VERSION, + integration: "hermes".to_string(), + event: "pre_tool_call".to_string(), + payload: serde_json::json!({"tool_name": "terminal"}), + cwd: Some("/repo".to_string()), + }; + let json = serde_json::to_value(&msg).unwrap(); + assert_eq!(json["type"], "policyEvaluation"); + assert_eq!(json["integration"], "hermes"); + assert_eq!(json["event"], "pre_tool_call"); + assert_eq!(json["payload"]["tool_name"], "terminal"); + } + #[test] fn unknown_message_type_fails_to_deserialize() { let json = serde_json::json!({ "type": "bogus", "protocolVersion": 1 }); @@ -154,6 +198,22 @@ mod tests { assert_eq!(decoded, msg); } + #[test] + fn structured_policy_result_round_trips() { + let msg = ServerMessage::PolicyResult { + protocol_version: PROTOCOL_VERSION, + decision: "instruct".to_string(), + policy_names: vec!["custom/approved-write-route".to_string()], + reason: Some("Use the approved write route.".to_string()), + matched_policies: vec!["custom/approved-write-route".to_string()], + duration_ms: 4, + tool_name: Some("Write".to_string()), + }; + let json = serde_json::to_string(&msg).unwrap(); + let decoded: ServerMessage = serde_json::from_str(&json).unwrap(); + assert_eq!(decoded, msg); + } + #[test] fn protocol_version_accessor_matches_every_variant() { assert_eq!( diff --git a/docs/reference/harnesses.mdx b/docs/reference/harnesses.mdx index 09e095854..8d52d8952 100644 --- a/docs/reference/harnesses.mdx +++ b/docs/reference/harnesses.mdx @@ -34,7 +34,7 @@ Each integration normalizes its native hook event names, tool names, and tool-in | Cursor | `PreToolUse`, `UserPromptSubmit`, `Stop` | `PostToolUse` and session events are observational. | | OpenCode | `PreToolUse` | Post-tool and lifecycle events are observational; current stop handling is guidance for a later turn rather than a verified gate. | | Pi | `PreToolUse`, `UserPromptSubmit` | Post-tool and lifecycle events are observational; stop guidance applies to a later turn. | -| Hermes | `PreToolUse` | Post-tool, session, and subagent-stop verdicts are not gates. | +| Hermes | `PreToolUse` | A native plugin delivers `instruct()` as one bounded, model-visible interruption before permitting a later API iteration. Post-tool, session, and subagent-stop verdicts are not gates. | | OpenClaw | `PreToolUse`, `UserPromptSubmit`, `Stop` | Post-tool, session, subagent-stop, and compaction events are observational. | | Factory Droid | `PreToolUse`, `UserPromptSubmit`, `Stop`, `PreCompact` | Post-tool and subagent-stop verdicts are observational. | | Devin CLI | `PreToolUse`, `UserPromptSubmit`, `Stop`, conditional `PermissionRequest` | Permission hooks do not run in every permission mode; post-tool and session events are observational. | @@ -43,6 +43,20 @@ Each integration normalizes its native hook event names, tool names, and tool-in Capabilities are version-sensitive. Re-test after upgrading an agent CLI, especially when a policy relies on prompt, stop, permission, or post-tool behavior rather than the common pre-tool gate. +### Hermes native plugin + +Hermes is integrated through a profile-local native plugin rather than a shell +command. Installation copies the plugin into every default and named Hermes +profile, enables it in that profile's `config.yaml`, and migrates only legacy +FailproofAI shell-hook entries. This avoids a process spawn on each hook and +lets `instruct()` reach the model through Hermes' native blocked-tool result. + +The first matching instruction blocks the pending call. The same API request +stays blocked; a later model iteration may retry. A persistent, profile-scoped +ledger and a per-turn cap prevent an advisory instruction from becoming an +unbounded loop. `deny()` remains a hard block. Run `failproofai config --status` +to detect a disabled, incomplete, duplicated, or newly unconfigured profile. + ## Install capture and policy hooks diff --git a/hermes-plugin/README.md b/hermes-plugin/README.md new file mode 100644 index 000000000..3e9766526 --- /dev/null +++ b/hermes-plugin/README.md @@ -0,0 +1,133 @@ +# FailproofAI for Hermes + +This directory is the native Hermes adapter shipped inside the `failproofai` +npm package. It keeps policy evaluation in FailproofAI's TypeScript worker and +translates structured verdicts into Hermes-native hook behavior. + +Validated contract: Hermes 0.21.3 at +`4d55ca91656ac5f83e1506679b7f81e0238e5e16`. The plugin uses only documented +manifest v2 fields, `PluginContext` methods/state, and keyword hook payloads. + +## Installation + +Run: + +```bash +failproofai policies --install --cli hermes --scope user +``` + +FailproofAI copies this directory to every discovered profile at +`/plugins/failproofai/` and adds `failproofai` to +`plugins.enabled` in that profile's `config.yaml`. Reinstall replaces only a +directory carrying `.failproofai-managed`; an unrelated plugin with the same +directory name is never overwritten. + +Legacy FailproofAI shell hooks are removed during migration. Operator-owned +hooks and unrelated plugin settings are preserved. No dashboard deployment or +backtest is part of installation. + +## Runtime path + +```text +Hermes pre_tool_call + -> native Python plugin + -> owner-only failproofaid Unix socket + -> warm TypeScript policy worker + -> structured allow | deny | instruct verdict + -> Hermes-native return value +``` + +There is no cloud request and no new CLI process in the tool-call path. + +## Decision behavior + +- `allow`: return no directive; Hermes runs the tool. +- `deny`: always return `{"action":"block","message":"..."}`. +- `instruct`: persist delivery state, block the first attempt with a + model-visible `FAILPROOF INSTRUCTION`, keep the same API request blocked, then + allow a later API iteration for the same instruction scope. + +The default instruction scope is profile + session + task + turn + policy +fingerprint + canonical tool. The persistent SQLite ledger lives below Hermes' +profile-scoped plugin data directory. Two distinct instruction interruptions +are allowed per turn by default; after that, advisory instructions fail open so +they cannot create an infinite retry loop. A real deny is never bypassed by the +ledger. + +## Configuration + +Settings live under `plugins.entries.failproofai.settings`: + +```yaml +plugins: + enabled: + - failproofai + entries: + failproofai: + settings: + failure_mode: deny + connect_timeout_ms: 250 + evaluation_timeout_ms: 12000 + instruction_ttl_seconds: 3600 + max_instruction_rounds: 2 +``` + +`failure_mode` controls evaluator/protocol failures only. `deny` is the safe +default. `allow` favors availability when the local daemon cannot return a +trusted verdict. Ledger failures always allow only the advisory `instruct` +decision, because otherwise corrupted retry state could block a turn forever. + +## Local protocol + +Requests and responses use the existing length-prefixed failproofaid Unix +socket protocol, version 1. + +```json +{ + "type": "policyEvaluation", + "protocolVersion": 1, + "integration": "hermes", + "event": "pre_tool_call", + "payload": {}, + "cwd": "/workspace/project" +} +``` + +```json +{ + "type": "policyResult", + "protocolVersion": 1, + "decision": "instruct", + "policyNames": ["custom/approved-write-route"], + "reason": "Use the approved write route.", + "matchedPolicies": ["custom/approved-write-route"], + "durationMs": 3, + "toolName": "Write" +} +``` + +Requests are limited to 1 MiB and responses to 16 MiB. Invalid JSON, unknown +message types, version mismatch, timeout, and socket failure are treated as +untrusted evaluation failures. + +## Diagnostics and rollback + +`failproofai config --status` reports each existing Hermes profile as healthy, +disabled, incomplete, or duplicated with a legacy shell hook. Hermes-side load +errors are available through: + +```bash +HERMES_PLUGINS_DEBUG=1 hermes plugins list +hermes plugins doctor ~/.hermes/plugins/failproofai --ci +hermes logs --level WARNING +``` + +Use a temporary `HERMES_HOME` for development and compatibility tests. Remove +the integration with: + +```bash +failproofai policies --uninstall --cli hermes --scope user +``` + +Uninstall removes the config registration and only the plugin directory marked +as FailproofAI-managed. diff --git a/hermes-plugin/__init__.py b/hermes-plugin/__init__.py new file mode 100644 index 000000000..95cb0a0ba --- /dev/null +++ b/hermes-plugin/__init__.py @@ -0,0 +1,248 @@ +"""Native Hermes bridge for FailproofAI policy enforcement.""" + +from __future__ import annotations + +import logging +import os +from pathlib import Path +from typing import Any, Mapping + +from .client import PolicyVerdict, evaluate_policy +from .ledger import InstructionLedger, LedgerError + +logger = logging.getLogger(__name__) + +_PROTOCOL_CONTEXT = ( + "When a tool result starts with FAILPROOF INSTRUCTION, treat it as policy guidance. " + "Reconsider the attempted action before retrying. Do not evade or ignore the instruction; " + "change tools, arguments, targets, or side effects only when the instruction requires it. " + "If you cannot follow the instruction, stop and explain why." +) + + +def _bounded_int(value: object, default: int, minimum: int, maximum: int) -> int: + try: + parsed = int(value) + except (TypeError, ValueError): + return default + return min(max(parsed, minimum), maximum) + + +def _profile_name() -> str: + explicit = os.environ.get("HERMES_PROFILE", "").strip() + if explicit: + return explicit + configured_home = os.environ.get("HERMES_HOME", "").strip() + home = Path(configured_home).expanduser() if configured_home else Path.home() / ".hermes" + if home.parent.name == "profiles": + return home.name + return "default" + + +def _string(value: object) -> str: + return value if isinstance(value, str) else "" + + +class FailproofAIPlugin: + def __init__(self, ctx: Any) -> None: + self.ctx = ctx + self.profile = _profile_name() + self.failure_mode = str(ctx.get_config("failure_mode", "deny")).strip().lower() + self.connect_timeout_ms = _bounded_int( + ctx.get_config("connect_timeout_ms", 250), 250, 25, 5_000 + ) + self.evaluation_timeout_ms = _bounded_int( + ctx.get_config("evaluation_timeout_ms", 12_000), 12_000, 100, 29_000 + ) + ttl_seconds = _bounded_int( + ctx.get_config("instruction_ttl_seconds", 3_600), 3_600, 60, 86_400 + ) + max_rounds = _bounded_int( + ctx.get_config("max_instruction_rounds", 2), 2, 0, 10 + ) + self.ledger = InstructionLedger( + Path(ctx.state.data_dir) / "instructions.db", + ttl_seconds=ttl_seconds, + max_rounds=max_rounds, + ) + + def _fallback(self, error: Exception) -> dict[str, str] | None: + logger.error("FailproofAI evaluator unavailable: %s", error) + if self.failure_mode == "allow": + return None + return { + "action": "block", + "message": ( + "FailproofAI could not verify this tool call because the local policy " + "evaluator is unavailable. Check `failproofai status` before retrying." + ), + } + + def _evaluate(self, event: str, payload: Mapping[str, Any]) -> PolicyVerdict: + cwd = _string(payload.get("cwd")) or os.getcwd() + return evaluate_policy( + event=event, + payload=payload, + cwd=cwd, + connect_timeout_ms=self.connect_timeout_ms, + evaluation_timeout_ms=self.evaluation_timeout_ms, + ) + + def pre_tool_call( + self, + tool_name: str = "", + args: Mapping[str, Any] | None = None, + session_id: str = "", + task_id: str = "", + tool_call_id: str = "", + turn_id: str = "", + api_request_id: str = "", + **kwargs: Any, + ) -> dict[str, str] | None: + payload = { + "hook_event_name": "pre_tool_call", + "tool_name": tool_name, + "tool_input": dict(args or {}), + "session_id": session_id, + "cwd": _string(kwargs.get("cwd")) or os.getcwd(), + "hermes": { + "profile": self.profile, + "task_id": task_id, + "tool_call_id": tool_call_id, + "turn_id": turn_id, + "api_request_id": api_request_id, + }, + } + try: + verdict = self._evaluate("pre_tool_call", payload) + except Exception as exc: + return self._fallback(exc) + + if verdict.decision == "allow": + return None + if verdict.decision == "deny": + return { + "action": "block", + "message": verdict.reason or "Blocked by FailproofAI policy.", + } + + reason = verdict.reason or "Reconsider this tool call before retrying." + try: + action = self.ledger.decide( + profile=self.profile, + session_id=session_id, + task_id=task_id, + turn_id=turn_id, + api_request_id=api_request_id, + policy_names=verdict.policy_names, + reason=reason, + scope_key=verdict.tool_name or tool_name or "unknown-tool", + ) + except LedgerError as exc: + # An advisory instruction must never become an indefinite denial + # because its local retry state is unavailable. + logger.error("FailproofAI instruction state unavailable; allowing: %s", exc) + return None + + if not action.block: + if action.status in {"missing-correlation-id", "turn-cap"}: + logger.warning("FailproofAI instruction allowed in degraded state: %s", action.status) + return None + + policies = ", ".join(verdict.policy_names) or "unknown policy" + return { + "action": "block", + "message": ( + f"FAILPROOF INSTRUCTION ({policies})\n\n{reason}\n\n" + "Reconsider the attempted action, then make a corrected tool call. " + "Do not route around this instruction with an equivalent ungoverned action." + ), + } + + def _observe(self, event: str, payload: Mapping[str, Any]) -> None: + try: + self._evaluate(event, payload) + except Exception as exc: + logger.warning("FailproofAI observation failed for %s: %s", event, exc) + + def post_tool_call( + self, + tool_name: str = "", + args: Mapping[str, Any] | None = None, + result: Any = None, + session_id: str = "", + **kwargs: Any, + ) -> None: + self._observe( + "post_tool_call", + { + "hook_event_name": "post_tool_call", + "tool_name": tool_name, + "tool_input": dict(args or {}), + "tool_response": result, + "session_id": session_id, + "cwd": _string(kwargs.get("cwd")) or os.getcwd(), + "hermes": {"profile": self.profile, **kwargs}, + }, + ) + + def on_session_start(self, session_id: str = "", **kwargs: Any) -> None: + self._observe( + "on_session_start", + { + "hook_event_name": "on_session_start", + "session_id": session_id, + "cwd": _string(kwargs.get("cwd")) or os.getcwd(), + "hermes": {"profile": self.profile, **kwargs}, + }, + ) + + def on_session_end(self, session_id: str = "", **kwargs: Any) -> None: + self._observe( + "on_session_end", + { + "hook_event_name": "on_session_end", + "session_id": session_id, + "cwd": _string(kwargs.get("cwd")) or os.getcwd(), + "hermes": {"profile": self.profile, **kwargs}, + }, + ) + self._clear_session(session_id) + + def subagent_stop(self, session_id: str = "", **kwargs: Any) -> None: + self._observe( + "subagent_stop", + { + "hook_event_name": "subagent_stop", + "session_id": session_id, + "cwd": _string(kwargs.get("cwd")) or os.getcwd(), + "hermes": {"profile": self.profile, **kwargs}, + }, + ) + + def pre_llm_call(self, **kwargs: Any) -> dict[str, str]: + return {"context": _PROTOCOL_CONTEXT} + + def on_session_reset(self, session_id: str = "", **kwargs: Any) -> None: + self._clear_session(session_id) + + def on_session_finalize(self, session_id: str = "", **kwargs: Any) -> None: + self._clear_session(session_id) + + def _clear_session(self, session_id: str) -> None: + try: + self.ledger.clear_session(session_id) + except LedgerError as exc: + logger.warning("FailproofAI instruction cleanup failed: %s", exc) + + +def register(ctx: Any) -> None: + plugin = FailproofAIPlugin(ctx) + ctx.register_hook("pre_tool_call", plugin.pre_tool_call) + ctx.register_hook("post_tool_call", plugin.post_tool_call) + ctx.register_hook("pre_llm_call", plugin.pre_llm_call) + ctx.register_hook("on_session_start", plugin.on_session_start) + ctx.register_hook("on_session_end", plugin.on_session_end) + ctx.register_hook("on_session_reset", plugin.on_session_reset) + ctx.register_hook("on_session_finalize", plugin.on_session_finalize) + ctx.register_hook("subagent_stop", plugin.subagent_stop) diff --git a/hermes-plugin/client.py b/hermes-plugin/client.py new file mode 100644 index 000000000..3de407bbb --- /dev/null +++ b/hermes-plugin/client.py @@ -0,0 +1,134 @@ +"""Versioned Unix-socket client for failproofaid policy evaluation.""" + +from __future__ import annotations + +import json +import os +import socket +import struct +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Mapping, Sequence + +PROTOCOL_VERSION = 1 +MAX_FRAME_BYTES = 16 * 1024 * 1024 +MAX_REQUEST_BYTES = 1024 * 1024 + + +class EvaluationError(RuntimeError): + """The daemon could not return a trustworthy policy verdict.""" + + +@dataclass(frozen=True) +class PolicyVerdict: + decision: str + policy_names: tuple[str, ...] + reason: str | None + matched_policies: tuple[str, ...] + duration_ms: int + tool_name: str | None + + +def daemon_socket_path() -> Path: + override = os.environ.get("FAILPROOFAI_DAEMON_SOCKET", "").strip() + if override: + return Path(override).expanduser() + root = os.environ.get("FAILPROOFAI_HOME", "").strip() + home = Path(root).expanduser() if root else Path.home() / ".failproofai" + return home / "run" / "failproofaid.sock" + + +def _read_exact(sock: socket.socket, length: int) -> bytes: + chunks: list[bytes] = [] + remaining = length + while remaining: + chunk = sock.recv(remaining) + if not chunk: + raise EvaluationError("failproofaid closed the connection before returning a verdict") + chunks.append(chunk) + remaining -= len(chunk) + return b"".join(chunks) + + +def _string_list(value: object, field: str) -> tuple[str, ...]: + if not isinstance(value, Sequence) or isinstance(value, (str, bytes, bytearray)): + raise EvaluationError(f"failproofaid returned an invalid {field}") + if not all(isinstance(item, str) for item in value): + raise EvaluationError(f"failproofaid returned an invalid {field}") + return tuple(value) + + +def evaluate_policy( + *, + event: str, + payload: Mapping[str, Any], + cwd: str | None, + connect_timeout_ms: int = 250, + evaluation_timeout_ms: int = 12_000, +) -> PolicyVerdict: + request = { + "type": "policyEvaluation", + "protocolVersion": PROTOCOL_VERSION, + "integration": "hermes", + "event": event, + "payload": dict(payload), + "cwd": cwd, + } + try: + body = json.dumps( + request, + ensure_ascii=False, + separators=(",", ":"), + default=lambda value: repr(value), + ).encode("utf-8") + except (TypeError, ValueError) as exc: + raise EvaluationError(f"could not encode policy request: {exc}") from exc + if len(body) > MAX_REQUEST_BYTES: + raise EvaluationError("policy request exceeds the 1 MiB limit") + + try: + with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as sock: + sock.settimeout(max(connect_timeout_ms, 1) / 1000) + sock.connect(str(daemon_socket_path())) + sock.settimeout(max(evaluation_timeout_ms, 1) / 1000) + sock.sendall(struct.pack(">I", len(body)) + body) + declared_length = struct.unpack(">I", _read_exact(sock, 4))[0] + if declared_length > MAX_FRAME_BYTES: + raise EvaluationError("failproofaid response exceeds the 16 MiB limit") + response = json.loads(_read_exact(sock, declared_length).decode("utf-8")) + except EvaluationError: + raise + except (OSError, UnicodeError, ValueError, struct.error) as exc: + raise EvaluationError(f"failproofaid evaluation failed: {exc}") from exc + + if not isinstance(response, dict): + raise EvaluationError("failproofaid returned a non-object response") + if response.get("protocolVersion") != PROTOCOL_VERSION: + raise EvaluationError("failproofaid protocol version mismatch") + if response.get("type") == "error": + message = response.get("message") + raise EvaluationError(str(message or "failproofaid could not evaluate the policy")) + if response.get("type") != "policyResult": + raise EvaluationError("failproofaid does not support native policy evaluation") + + decision = response.get("decision") + if decision not in {"allow", "deny", "instruct"}: + raise EvaluationError("failproofaid returned an invalid policy decision") + reason = response.get("reason") + if reason is not None and not isinstance(reason, str): + raise EvaluationError("failproofaid returned an invalid policy reason") + duration_ms = response.get("durationMs", 0) + if not isinstance(duration_ms, int) or duration_ms < 0: + raise EvaluationError("failproofaid returned an invalid evaluation duration") + tool_name = response.get("toolName") + if tool_name is not None and not isinstance(tool_name, str): + raise EvaluationError("failproofaid returned an invalid tool name") + + return PolicyVerdict( + decision=decision, + policy_names=_string_list(response.get("policyNames", []), "policyNames"), + reason=reason, + matched_policies=_string_list(response.get("matchedPolicies", []), "matchedPolicies"), + duration_ms=duration_ms, + tool_name=tool_name, + ) diff --git a/hermes-plugin/ledger.py b/hermes-plugin/ledger.py new file mode 100644 index 000000000..5acd55115 --- /dev/null +++ b/hermes-plugin/ledger.py @@ -0,0 +1,187 @@ +"""Persistent, bounded delivery state for FailproofAI instruct decisions.""" + +from __future__ import annotations + +import hashlib +import json +import sqlite3 +import threading +import time +from dataclasses import dataclass +from pathlib import Path +from typing import Sequence + + +class LedgerError(RuntimeError): + """Instruction state could not be read or updated safely.""" + + +@dataclass(frozen=True) +class InstructionAction: + block: bool + status: str + + +class InstructionLedger: + def __init__(self, path: Path, *, ttl_seconds: int = 3600, max_rounds: int = 2) -> None: + self.path = path + self.ttl_seconds = max(60, int(ttl_seconds)) + self.max_rounds = max(0, int(max_rounds)) + self._init_lock = threading.Lock() + self._initialized = False + + def _connect(self) -> sqlite3.Connection: + self.path.parent.mkdir(parents=True, exist_ok=True) + connection = sqlite3.connect(self.path, timeout=0.25, isolation_level=None) + connection.execute("PRAGMA busy_timeout = 250") + connection.execute("PRAGMA journal_mode = WAL") + connection.execute("PRAGMA synchronous = NORMAL") + return connection + + def _ensure_schema(self) -> None: + if self._initialized: + return + with self._init_lock: + if self._initialized: + return + try: + with self._connect() as connection: + connection.execute( + """ + CREATE TABLE IF NOT EXISTS instruction_deliveries ( + delivery_key TEXT PRIMARY KEY, + profile TEXT NOT NULL, + session_id TEXT NOT NULL, + task_id TEXT NOT NULL, + turn_id TEXT NOT NULL, + policy_fingerprint TEXT NOT NULL, + scope_key TEXT NOT NULL, + blocked_api_request_id TEXT NOT NULL, + acknowledged_api_request_id TEXT, + state TEXT NOT NULL, + created_at_ms INTEGER NOT NULL, + expires_at_ms INTEGER NOT NULL + ) + """ + ) + connection.execute( + """ + CREATE INDEX IF NOT EXISTS instruction_deliveries_turn + ON instruction_deliveries(profile, session_id, task_id, turn_id, expires_at_ms) + """ + ) + except sqlite3.Error as exc: + raise LedgerError(f"could not initialize instruction state: {exc}") from exc + self._initialized = True + + @staticmethod + def _digest(parts: Sequence[str]) -> str: + encoded = json.dumps(list(parts), ensure_ascii=False, separators=(",", ":")).encode("utf-8") + return hashlib.sha256(encoded).hexdigest() + + def decide( + self, + *, + profile: str, + session_id: str, + task_id: str, + turn_id: str, + api_request_id: str, + policy_names: Sequence[str], + reason: str, + scope_key: str, + now_ms: int | None = None, + ) -> InstructionAction: + if not session_id or not turn_id or not api_request_id: + return InstructionAction(False, "missing-correlation-id") + + self._ensure_schema() + now = int(time.time() * 1000) if now_ms is None else int(now_ms) + expires_at = now + self.ttl_seconds * 1000 + task = task_id or "root" + policy_fingerprint = self._digest([*sorted(policy_names), reason]) + delivery_key = self._digest( + [profile, session_id, task, turn_id, policy_fingerprint, scope_key] + ) + + try: + with self._connect() as connection: + connection.execute("BEGIN IMMEDIATE") + connection.execute( + "DELETE FROM instruction_deliveries WHERE expires_at_ms <= ?", (now,) + ) + row = connection.execute( + """ + SELECT blocked_api_request_id, state + FROM instruction_deliveries + WHERE delivery_key = ? + """, + (delivery_key,), + ).fetchone() + if row is not None: + blocked_request, state = row + if state == "issued" and blocked_request == api_request_id: + connection.commit() + return InstructionAction(True, "same-api-request") + if state == "issued": + connection.execute( + """ + UPDATE instruction_deliveries + SET state = 'acknowledged', acknowledged_api_request_id = ? + WHERE delivery_key = ? + """, + (api_request_id, delivery_key), + ) + connection.commit() + return InstructionAction(False, "acknowledged") + + rounds = connection.execute( + """ + SELECT COUNT(*) + FROM instruction_deliveries + WHERE profile = ? AND session_id = ? AND task_id = ? AND turn_id = ? + AND expires_at_ms > ? + """, + (profile, session_id, task, turn_id, now), + ).fetchone()[0] + if rounds >= self.max_rounds: + connection.commit() + return InstructionAction(False, "turn-cap") + + connection.execute( + """ + INSERT INTO instruction_deliveries ( + delivery_key, profile, session_id, task_id, turn_id, + policy_fingerprint, scope_key, blocked_api_request_id, + acknowledged_api_request_id, state, created_at_ms, expires_at_ms + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, NULL, 'issued', ?, ?) + """, + ( + delivery_key, + profile, + session_id, + task, + turn_id, + policy_fingerprint, + scope_key, + api_request_id, + now, + expires_at, + ), + ) + connection.commit() + return InstructionAction(True, "issued") + except (sqlite3.Error, OSError) as exc: + raise LedgerError(f"could not update instruction state: {exc}") from exc + + def clear_session(self, session_id: str) -> None: + if not session_id: + return + self._ensure_schema() + try: + with self._connect() as connection: + connection.execute( + "DELETE FROM instruction_deliveries WHERE session_id = ?", (session_id,) + ) + except (sqlite3.Error, OSError) as exc: + raise LedgerError(f"could not clear instruction state: {exc}") from exc diff --git a/hermes-plugin/plugin.yaml b/hermes-plugin/plugin.yaml new file mode 100644 index 000000000..90f7a7fdf --- /dev/null +++ b/hermes-plugin/plugin.yaml @@ -0,0 +1,37 @@ +name: failproofai +version: 1.0.6-beta.0 +description: Real-time FailproofAI policy enforcement for Hermes +manifest_version: 2 +api_version: 1 +homepage: https://github.com/FailproofAI/failproofai +tags: [security, policy, observability] +provides_hooks: + - pre_tool_call + - post_tool_call + - pre_llm_call + - on_session_start + - on_session_end + - on_session_reset + - on_session_finalize + - subagent_stop +config_schema: + failure_mode: + type: str + default: deny + description: Block or allow when the local FailproofAI evaluator is unavailable. + connect_timeout_ms: + type: int + default: 250 + description: Maximum time to connect to the local FailproofAI daemon. + evaluation_timeout_ms: + type: int + default: 12000 + description: Maximum total time allowed for one policy evaluation. + instruction_ttl_seconds: + type: int + default: 3600 + description: Maximum lifetime of an instruction-delivery record. + max_instruction_rounds: + type: int + default: 2 + description: Maximum distinct instruction interruptions in one Hermes turn. diff --git a/package.json b/package.json index d217f9463..e5b253e53 100644 --- a/package.json +++ b/package.json @@ -12,6 +12,7 @@ "scripts/", "lib/", "pi-extension/", + "hermes-plugin/", "openclaw-plugin/", ".next/standalone/", "dist/", diff --git a/src/hooks/handler.ts b/src/hooks/handler.ts index 9503226d4..d378380e6 100644 --- a/src/hooks/handler.ts +++ b/src/hooks/handler.ts @@ -102,6 +102,21 @@ export interface HookEventOutcome { exitCode: number; stdout: string; stderr: string; + /** + * Structured policy metadata for in-process adapters. Existing shell-hook + * callers intentionally ignore this and keep consuming stdout/stderr. + */ + evaluation?: HookEvaluationSummary; +} + +export interface HookEvaluationSummary { + decision: "allow" | "deny" | "instruct"; + policyName: string | null; + policyNames: string[]; + reason: string | null; + matchedPolicies: string[]; + durationMs: number; + toolName: string | null; } export interface EvaluateHookEventOptions { @@ -757,7 +772,20 @@ export async function evaluateHookEvent( } } - return { exitCode: result.exitCode, stdout: result.stdout, stderr: result.stderr }; + return { + exitCode: result.exitCode, + stdout: result.stdout, + stderr: result.stderr, + evaluation: { + decision: result.decision, + policyName: result.policyName, + policyNames: result.policyNames ?? (result.policyName ? [result.policyName] : []), + reason: result.reason, + matchedPolicies, + durationMs, + toolName: (parsed.tool_name as string) ?? null, + }, + }; } finally { if (opts?.awaitTelemetryFlush ?? true) { // Await any un-awaited (`void trackHookEvent(...)`) events fired during diff --git a/src/hooks/integrations.ts b/src/hooks/integrations.ts index 8e85dd2e2..055b08ab0 100644 --- a/src/hooks/integrations.ts +++ b/src/hooks/integrations.ts @@ -7,7 +7,17 @@ * is agent-agnostic — only install/uninstall plumbing varies. */ import { execSync } from "node:child_process"; -import { readFileSync, writeFileSync, existsSync, mkdirSync, unlinkSync } from "node:fs"; +import { + cpSync, + readFileSync, + writeFileSync, + existsSync, + lstatSync, + mkdirSync, + renameSync, + rmSync, + unlinkSync, +} from "node:fs"; import { resolve, dirname } from "node:path"; import { fileURLToPath } from "node:url"; import { homedir } from "node:os"; @@ -135,6 +145,9 @@ export interface Integration { /** Read the raw settings/hooks file (returns {} when missing). */ readSettings(settingsPath: string): Record; + /** Install integration-owned files before enabling them in settings. */ + prepareInstall?(settingsPath: string): void; + /** Write the settings/hooks file. */ writeSettings(settingsPath: string, settings: Record): void; @@ -1363,14 +1376,14 @@ function makePiProjectRelativeEntry(extPath: string): string { // works for the local user. return extResolved; } -// ── Hermes (hermes-agent) — live hooks (Pillar 1) ──────────────────────────── +// ── Hermes (hermes-agent) — native plugin (Pillar 1) ──────────────────────── // -// External-command CLI like codex/cursor, but its config is YAML -// (`~/.hermes/config.yaml`) under a `hooks:` map, so the I/O layer uses the yaml -// Document API (comment-preserving). Flat per-event arrays like cursor: -// `hooks: { pre_tool_call: [ { command, timeout, } ], … }`. Hermes -// reads a `{"decision":"block",…}` JSON response on stdout (see -// policy-evaluator.ts); exit codes are ignored. User-scope only. +// Hermes loads trusted Python plugins from `/plugins//`. +// The shipped plugin registers Hermes-native hooks and talks directly to the +// local failproofaid socket, avoiding a fresh CLI process per event. Config is +// YAML and profile-scoped; install copies the plugin into every profile and +// adds `failproofai` to `plugins.enabled`. Legacy shell-hook entries are removed +// during migration so one tool call is never evaluated twice. /** One hook entry as stored under a `hooks:` event key in config.yaml. */ interface HermesHookEntry { @@ -1379,6 +1392,200 @@ interface HermesHookEntry { [key: string]: unknown; } +interface HermesPluginsConfig { + enabled?: unknown[]; + disabled?: unknown[]; + entries?: Record; + [key: string]: unknown; +} + +const HERMES_PLUGIN_ID = "failproofai"; +const HERMES_PLUGIN_MARKER = ".failproofai-managed"; +const HERMES_PLUGIN_FILES = ["plugin.yaml", "__init__.py", "client.py", "ledger.py"] as const; + +function getHermesPluginSourcePath(): string { + const fromEnv = process.env.FAILPROOFAI_PACKAGE_ROOT; + if (fromEnv) return resolve(fromEnv, "hermes-plugin"); + return resolve(fileURLToPath(import.meta.url), "..", "..", "..", "hermes-plugin"); +} + +export function hermesPluginPathForSettings(settingsPath: string): string { + return resolve(dirname(settingsPath), "plugins", HERMES_PLUGIN_ID); +} + +function hasManagedHermesPlugin(settingsPath: string): boolean { + const pluginPath = hermesPluginPathForSettings(settingsPath); + try { + if (lstatSync(pluginPath).isSymbolicLink()) return false; + if (!existsSync(resolve(pluginPath, HERMES_PLUGIN_MARKER))) return false; + return HERMES_PLUGIN_FILES.every((file) => existsSync(resolve(pluginPath, file))); + } catch { + return false; + } +} + +function hasHermesPluginMarker(settingsPath: string): boolean { + const pluginPath = hermesPluginPathForSettings(settingsPath); + try { + return !lstatSync(pluginPath).isSymbolicLink() && existsSync(resolve(pluginPath, HERMES_PLUGIN_MARKER)); + } catch { + return false; + } +} + +function installHermesPlugin(settingsPath: string): void { + const source = getHermesPluginSourcePath(); + for (const file of HERMES_PLUGIN_FILES) { + if (!existsSync(resolve(source, file))) { + throw new Error(`Hermes plugin asset is missing from the failproofai package: ${file}`); + } + } + + const destination = hermesPluginPathForSettings(settingsPath); + if (existsSync(destination)) { + if (lstatSync(destination).isSymbolicLink() || !existsSync(resolve(destination, HERMES_PLUGIN_MARKER))) { + throw new Error( + `Refusing to overwrite an unmanaged Hermes plugin at ${destination}. ` + + `Move it aside or install FailproofAI under a different Hermes profile.`, + ); + } + } + + mkdirSync(dirname(destination), { recursive: true }); + const suffix = `${process.pid}-${Date.now()}`; + const temporary = `${destination}.install-${suffix}`; + const backup = `${destination}.backup-${suffix}`; + let movedExisting = false; + try { + mkdirSync(temporary, { recursive: true }); + for (const file of HERMES_PLUGIN_FILES) { + cpSync(resolve(source, file), resolve(temporary, file)); + } + writeFileSync( + resolve(temporary, HERMES_PLUGIN_MARKER), + "Managed by failproofai. Remove with `failproofai policies --uninstall --cli hermes`.\n", + { encoding: "utf8", mode: 0o600 }, + ); + if (existsSync(destination)) { + renameSync(destination, backup); + movedExisting = true; + } + renameSync(temporary, destination); + if (movedExisting) rmSync(backup, { recursive: true, force: true }); + } catch (err) { + rmSync(temporary, { recursive: true, force: true }); + if (movedExisting && !existsSync(destination) && existsSync(backup)) { + renameSync(backup, destination); + } + throw err; + } +} + +function removeManagedHermesPlugin(settingsPath: string): boolean { + const pluginPath = hermesPluginPathForSettings(settingsPath); + // The marker is the ownership boundary. Remove even an incomplete managed + // install so an interrupted copy can always be repaired or uninstalled. + if (!hasHermesPluginMarker(settingsPath)) return false; + rmSync(pluginPath, { recursive: true, force: true }); + return true; +} + +function removeLegacyHermesHooks(doc: Document): number { + const js = (doc.toJS() ?? {}) as { hooks?: Record }; + const hooks = js.hooks; + let removed = 0; + if (hooks && typeof hooks === "object") { + for (const eventType of Object.keys(hooks)) { + const entries = hooks[eventType]; + if (!Array.isArray(entries)) continue; + const before = entries.length; + const filtered = entries.filter((hook) => !isMarkedHook(hook)); + removed += before - filtered.length; + if (filtered.length === 0) delete hooks[eventType]; + else hooks[eventType] = filtered; + } + if (Object.keys(hooks).length === 0) { + doc.delete("hooks"); + // Older FailproofAI installs set this broad Hermes switch so headless + // shell hooks could run. It is safe to remove only when no operator hook + // remains that may rely on it. + if (removed > 0) doc.delete("hooks_auto_accept"); + } else { + doc.set("hooks", hooks); + } + } + return removed; +} + +function hermesConfigState(settingsPath: string): { + pluginEnabled: boolean; + legacyShellHookPresent: boolean; +} { + if (!existsSync(settingsPath)) { + return { pluginEnabled: false, legacyShellHookPresent: false }; + } + const doc = readYamlDoc(settingsPath); + const js = (doc.toJS() ?? {}) as { + hooks?: Record; + plugins?: HermesPluginsConfig; + }; + const enabled = js.plugins?.enabled; + const legacyShellHookPresent = + !!js.hooks && + typeof js.hooks === "object" && + Object.values(js.hooks).some( + (entries) => Array.isArray(entries) && entries.some((entry) => isMarkedHook(entry)), + ); + return { + pluginEnabled: Array.isArray(enabled) && enabled.includes(HERMES_PLUGIN_ID), + legacyShellHookPresent, + }; +} + +export interface HermesProfileHealth { + name: string; + home: string; + settingsPath: string; + pluginInstalled: boolean; + pluginEnabled: boolean; + legacyShellHookPresent: boolean; + healthy: boolean; +} + +/** Read-only, per-profile Hermes installation health for setup/status surfaces. */ +export function hermesProfileHealth(): HermesProfileHealth[] { + return listHermesProfiles().map((profile) => { + const settingsPath = resolve(profile.home, "config.yaml"); + const config = hermesConfigState(settingsPath); + const pluginInstalled = hasManagedHermesPlugin(settingsPath); + return { + name: profile.name, + home: profile.home, + settingsPath, + pluginInstalled, + pluginEnabled: config.pluginEnabled, + legacyShellHookPresent: config.legacyShellHookPresent, + healthy: pluginInstalled && config.pluginEnabled && !config.legacyShellHookPresent, + }; + }); +} + +/** Compact rows for the machine-status display; absent Hermes homes produce no noise. */ +export function hermesProfileStatusRows(): Array<[string, string]> { + return hermesProfileHealth() + .filter((profile) => existsSync(profile.home)) + .map((profile) => { + const problems: string[] = []; + if (!profile.pluginInstalled) problems.push("plugin files missing or incomplete"); + if (!profile.pluginEnabled) problems.push("plugin not enabled"); + if (profile.legacyShellHookPresent) problems.push("legacy shell hook also present"); + return [ + "hermes/" + profile.name, + problems.length === 0 ? "native plugin enabled" : "UNHEALTHY — " + problems.join("; "), + ]; + }); +} + export const hermes: Integration = { id: "hermes", displayName: "Hermes", @@ -1410,75 +1617,66 @@ export const hermes: Integration = { writeYamlDoc(settingsPath, settings as unknown as Document); }, - buildHookEntry(binaryPath, eventType, scope) { - // No matcher → fires for ALL tools / all platforms (slack/telegram/cli/cron) - // and internal subagents. `timeout` is in seconds; Hermes runs the command - // via shlex.split (shell=false). - const command = - scope === "project" - ? `npx -y failproofai --hook ${eventType} --cli hermes` - : `"${binaryPath}" --hook ${eventType} --cli hermes`; + prepareInstall(settingsPath) { + installHermesPlugin(settingsPath); + }, + + buildHookEntry(_binaryPath, _eventType, _scope) { + // Native plugins are package-level registrations. This sentinel preserves + // the common Integration interface; writeHookEntries does the real work. return { - command, - timeout: 30, [FAILPROOFAI_HOOK_MARKER]: true, + _hermesPluginPath: getHermesPluginSourcePath(), }; }, isFailproofaiHook: isMarkedHook, - writeHookEntries(settings, binaryPath, scope) { + writeHookEntries(settings) { const doc = settings as unknown as Document; - // Read the current hooks map as plain JS, then re-set ONLY the `hooks` key — - // preserving comments on every other part of config.yaml. - const js = (doc.toJS() ?? {}) as { hooks?: Record }; - const hooks: Record = - js.hooks && typeof js.hooks === "object" ? js.hooks : {}; - - for (const eventType of HERMES_HOOK_EVENT_TYPES) { - const entry = this.buildHookEntry(binaryPath, eventType, scope) as unknown as HermesHookEntry; - const arr = Array.isArray(hooks[eventType]) ? hooks[eventType] : []; - const idx = arr.findIndex((h) => isMarkedHook(h)); - if (idx >= 0) arr[idx] = entry; - else arr.push(entry); - hooks[eventType] = arr; + // A native plugin and the legacy shell bridge running together would + // evaluate every event twice. Remove only failproofai-owned shell entries; + // unrelated operator hooks remain intact. + removeLegacyHermesHooks(doc); + + const js = (doc.toJS() ?? {}) as { plugins?: HermesPluginsConfig }; + const plugins: HermesPluginsConfig = + js.plugins && typeof js.plugins === "object" ? js.plugins : {}; + const enabled = Array.isArray(plugins.enabled) ? [...plugins.enabled] : []; + if (!enabled.includes(HERMES_PLUGIN_ID)) enabled.push(HERMES_PLUGIN_ID); + plugins.enabled = enabled; + if (Array.isArray(plugins.disabled)) { + plugins.disabled = plugins.disabled.filter((name) => name !== HERMES_PLUGIN_ID); + if (plugins.disabled.length === 0) delete plugins.disabled; } - doc.set("hooks", hooks); - // The headless gateway has no TTY to answer Hermes's first-use hook-consent - // prompt, so auto-accept declared hooks. Tradeoff: also auto-accepts any - // other hook the operator adds; a targeted `shell-hooks-allowlist.json` - // pre-seed is a future refinement. - doc.set("hooks_auto_accept", true); + doc.set("plugins", plugins); }, removeHooksFromFile(settingsPath) { - if (!existsSync(settingsPath)) return 0; const doc = readYamlDoc(settingsPath); - const js = (doc.toJS() ?? {}) as { hooks?: Record }; - const hooks = js.hooks; - - let removed = 0; - if (hooks && typeof hooks === "object") { - for (const eventType of Object.keys(hooks)) { - const entries = hooks[eventType]; - if (!Array.isArray(entries)) continue; - const before = entries.length; - const filtered = entries.filter((h) => !isMarkedHook(h)); - removed += before - filtered.length; - if (filtered.length === 0) delete hooks[eventType]; - else hooks[eventType] = filtered; + let removed = removeLegacyHermesHooks(doc); + const js = (doc.toJS() ?? {}) as { plugins?: HermesPluginsConfig }; + const plugins = js.plugins; + let configChanged = removed > 0; + if (plugins && typeof plugins === "object") { + if (Array.isArray(plugins.enabled)) { + const before = plugins.enabled.length; + plugins.enabled = plugins.enabled.filter((name) => name !== HERMES_PLUGIN_ID); + if (plugins.enabled.length !== before) { + removed += 1; + configChanged = true; + } + if (plugins.enabled.length === 0) delete plugins.enabled; } - if (removed > 0) { - if (Object.keys(hooks).length === 0) doc.delete("hooks"); - else doc.set("hooks", hooks); + if (plugins.entries && HERMES_PLUGIN_ID in plugins.entries) { + delete plugins.entries[HERMES_PLUGIN_ID]; + configChanged = true; } + if (Object.keys(plugins).length === 0) doc.delete("plugins"); + else doc.set("plugins", plugins); } - - // Always drop our headless-consent flag on uninstall — even if the hooks were - // already removed manually — so it can't silently auto-accept future operator - // hooks. `doc.delete` returns true iff the key was present. - const droppedAutoAccept = doc.delete("hooks_auto_accept"); - if (removed > 0 || droppedAutoAccept) writeYamlDoc(settingsPath, doc); + if (configChanged) writeYamlDoc(settingsPath, doc); + if (removeManagedHermesPlugin(settingsPath)) removed += 1; return removed; }, @@ -1497,15 +1695,9 @@ export const hermes: Integration = { }; function hermesConfigHasHooks(settingsPath: string): boolean { - if (!existsSync(settingsPath)) return false; try { - const doc = readYamlDoc(settingsPath); - const js = (doc.toJS() ?? {}) as { hooks?: Record }; - const hooks = js.hooks; - if (!hooks || typeof hooks !== "object") return false; - for (const entries of Object.values(hooks)) { - if (Array.isArray(entries) && entries.some((h) => isMarkedHook(h))) return true; - } + const state = hermesConfigState(settingsPath); + return state.pluginEnabled && hasManagedHermesPlugin(settingsPath) && !state.legacyShellHookPresent; } catch { // Corrupt config — treat as not installed. } @@ -1523,10 +1715,9 @@ function hermesConfigHasHooks(settingsPath: string): boolean { * warning. */ export function unhookedHermesProfiles(): string[] { - return listHermesProfiles() - .filter((p) => existsSync(p.home)) - .filter((p) => !hermesConfigHasHooks(resolve(p.home, "config.yaml"))) - .map((p) => p.name); + return hermesProfileHealth() + .filter((profile) => existsSync(profile.home) && !profile.healthy) + .map((profile) => profile.name); } // ── OpenClaw integration ──────────────────────────────────────────────────── diff --git a/src/hooks/manager.ts b/src/hooks/manager.ts index 13dc34e7b..f77689b2d 100644 --- a/src/hooks/manager.ts +++ b/src/hooks/manager.ts @@ -672,6 +672,7 @@ async function installHooksImpl( const settingsPaths = settingsPathsFor(integration, scope, cwd); try { for (const settingsPath of settingsPaths) { + integration.prepareInstall?.(settingsPath); const settings = integration.readSettings(settingsPath); integration.writeHookEntries(settings, binaryPath, scope); integration.writeSettings(settingsPath, settings); @@ -738,10 +739,17 @@ async function installHooksImpl( for (const { cli: cliId, path } of writtenSettingsPaths) { const integration = getIntegration(cliId); - console.log( - `Failproof AI hooks installed for ${integration.displayName} ` + - `(${integration.eventTypes.length} event types, scope: ${scope}).` - ); + if (cliId === "hermes") { + console.log( + `Failproof AI native plugin installed for ${integration.displayName} ` + + `(8 registered hooks, scope: ${scope}).` + ); + } else { + console.log( + `Failproof AI hooks installed for ${integration.displayName} ` + + `(${integration.eventTypes.length} event types, scope: ${scope}).` + ); + } console.log(`Settings: ${path}`); } if (scope === "project") { @@ -876,7 +884,11 @@ export async function removeHooks(policyNames?: string[], scope: HookScope | "al for (const s of scopesToRemove) { // Usually one path; Hermes returns one per profile. const settingsPaths = settingsPathsFor(integration, s, cwd); - const existing = settingsPaths.filter((p) => existsSync(p)); + // A Hermes install copies its managed plugin before updating config.yaml. + // If the config write is interrupted, uninstall must still call the + // integration so it can remove that orphaned managed directory. + const existing = + cliId === "hermes" ? settingsPaths : settingsPaths.filter((p) => existsSync(p)); if (existing.length === 0) { if (scope !== "all" && selectedClis.length === 1) { diff --git a/src/hooks/types.ts b/src/hooks/types.ts index fd2ff7f3a..5c98a4fb0 100644 --- a/src/hooks/types.ts +++ b/src/hooks/types.ts @@ -113,13 +113,13 @@ export const HERMES_TOOL_INPUT_MAP: Record> = { Edit: { path: "file_path" }, }; -// Hermes live-hook (Pillar 1) events + scopes. Hermes fires these snake_case -// events with a JSON payload on stdin; the command we install runs -// `failproofai --hook --cli hermes`. Config is USER-scope only -// (`~/.hermes/config.yaml`; Hermes has no project scope). `pre_tool_call` is the -// core deny point — it fires for tool calls from every source -// (slack/telegram/cli/cron) and internal subagents, so a single install -// intercepts all platforms. +// Hermes live-hook (Pillar 1) events + scopes. The native Python plugin sends +// these events to failproofaid's warm worker. It also handles `pre_llm_call`, +// `on_session_reset`, and `on_session_finalize` locally for instruction context +// and ledger cleanup, so those do not need canonical policy-event mappings. +// Config is USER-scope only (`~/.hermes/config.yaml`; Hermes has no project +// scope). `pre_tool_call` is the core gate and fires for tool calls from every +// source (Slack/Telegram/CLI/cron) and internal subagents. // // `pre_verify` IS a turn-end gate — the earlier claim here that Hermes has none // was wrong. We deliberately do NOT install it (product decision, 2026-07-29); @@ -143,7 +143,7 @@ export const HERMES_TOOL_INPUT_MAP: Record> = { // 2. Capped at 3 nudges per turn (`DEFAULT_MAX_VERIFY_NUDGES`, // `agent/verify_hooks.py:21`), operator-overridable; resets each turn. // 3. It landed upstream ~2026-06-30. Older Hermes fails the key against -// VALID_HOOKS and warn-and-skips it SILENTLY (`agent/shell_hooks.py:325`). +// VALID_HOOKS and warn-and-skips it silently. // // Until it is installed, `HERMES_EVENT_MAP` emits no `Stop` and those 5 // builtins remain inapplicable on Hermes. diff --git a/src/hooks/worker-server.ts b/src/hooks/worker-server.ts index b01b35170..a2feb837b 100644 --- a/src/hooks/worker-server.ts +++ b/src/hooks/worker-server.ts @@ -192,6 +192,7 @@ function handleConnection(socket: Socket, shutdown: () => void): void { exitCode: result.exitCode, stdout: result.stdout, stderr: result.stderr, + evaluation: result.evaluation, }), ); } catch (err) { From 371de17dc800ec3fd9ecadaa0b4e3cd12e2228d2 Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Tue, 15 Sep 2026 15:36:11 +0530 Subject: [PATCH 2/6] fix hermes plugin daemon safeguards --- CHANGELOG.md | 1 + .../fixtures/hermes-native-plugin-check.py | 53 ++++++++++++++++--- __tests__/hooks/manager.test.ts | 18 +++++++ hermes-plugin/README.md | 7 ++- hermes-plugin/client.py | 28 +++++++--- src/hooks/manager.ts | 15 ++++++ 6 files changed, 109 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 98a76f2c9..d53b108cb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ ### Changed - Hermes installation now copies and enables the managed plugin in every profile, migrates only legacy FailproofAI shell hooks, refuses to overwrite unmanaged plugin directories, and reports incomplete or duplicate profile installations as unhealthy. +- Direct Hermes installation now requires a healthy end-to-end `failproofaid` probe before changing plugin or profile configuration, preventing the default fail-closed mode from locking tool use when no evaluator is available. Hermes policy evaluation also applies `evaluation_timeout_ms` as one total socket deadline instead of resetting it for each partial read. ### Dependencies diff --git a/__tests__/fixtures/hermes-native-plugin-check.py b/__tests__/fixtures/hermes-native-plugin-check.py index 6eb95ce77..76503424e 100644 --- a/__tests__/fixtures/hermes-native-plugin-check.py +++ b/__tests__/fixtures/hermes-native-plugin-check.py @@ -7,6 +7,7 @@ import sys import tempfile import threading +import time import unittest from concurrent.futures import ThreadPoolExecutor from pathlib import Path @@ -424,8 +425,9 @@ def server() -> None: ready.set() connection, _ = listener.accept() with connection: - length = struct.unpack(">I", client._read_exact(connection, 4))[0] - received.update(json.loads(client._read_exact(connection, length))) + deadline = time.monotonic() + 2 + length = struct.unpack(">I", client._read_exact(connection, 4, deadline))[0] + received.update(json.loads(client._read_exact(connection, length, deadline))) body = json.dumps( { "type": "policyResult", @@ -468,8 +470,9 @@ def server() -> None: ready.set() connection, _ = listener.accept() with connection: - length = struct.unpack(">I", client._read_exact(connection, 4))[0] - client._read_exact(connection, length) + deadline = time.monotonic() + 2 + length = struct.unpack(">I", client._read_exact(connection, 4, deadline))[0] + client._read_exact(connection, length, deadline) body = json.dumps( { "type": "policyResult", @@ -502,8 +505,9 @@ def server() -> None: ready.set() connection, _ = listener.accept() with connection: - length = struct.unpack(">I", client._read_exact(connection, 4))[0] - client._read_exact(connection, length) + deadline = time.monotonic() + 2 + length = struct.unpack(">I", client._read_exact(connection, 4, deadline))[0] + client._read_exact(connection, length, deadline) connection.sendall(struct.pack(">I", client.MAX_FRAME_BYTES + 1)) thread = threading.Thread(target=server, daemon=True) @@ -514,6 +518,43 @@ def server() -> None: client.evaluate_policy(event="pre_tool_call", payload={}, cwd="/tmp") thread.join(timeout=2) + def test_evaluation_timeout_is_one_deadline_across_partial_reads(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + socket_path = Path(tmp) / "daemon.sock" + ready = threading.Event() + + def server() -> None: + with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as listener: + listener.bind(str(socket_path)) + listener.listen(1) + ready.set() + connection, _ = listener.accept() + with connection: + deadline = time.monotonic() + 2 + length = struct.unpack(">I", client._read_exact(connection, 4, deadline))[0] + client._read_exact(connection, length, deadline) + try: + for byte in struct.pack(">I", 0): + connection.sendall(bytes((byte,))) + time.sleep(0.04) + except BrokenPipeError: + pass + + thread = threading.Thread(target=server, daemon=True) + thread.start() + self.assertTrue(ready.wait(timeout=2)) + started = time.monotonic() + with patch.dict("os.environ", {"FAILPROOFAI_DAEMON_SOCKET": str(socket_path)}): + with self.assertRaisesRegex(client.EvaluationError, "timed out"): + client.evaluate_policy( + event="pre_tool_call", + payload={}, + cwd="/tmp", + evaluation_timeout_ms=70, + ) + self.assertLess(time.monotonic() - started, 0.14) + thread.join(timeout=2) + if __name__ == "__main__": unittest.main() diff --git a/__tests__/hooks/manager.test.ts b/__tests__/hooks/manager.test.ts index 04806f2ca..0bb2a8367 100644 --- a/__tests__/hooks/manager.test.ts +++ b/__tests__/hooks/manager.test.ts @@ -61,6 +61,10 @@ vi.mock("../../src/hooks/hook-telemetry", () => ({ trackHookEvent: vi.fn(() => Promise.resolve()), })); +vi.mock("../../src/hooks/daemon-service", () => ({ + probeDaemonEndToEnd: vi.fn(() => Promise.resolve(true)), +})); + vi.mock("../../lib/telemetry-id", () => ({ getInstanceId: vi.fn(() => "test-instance-id"), hashToId: vi.fn((raw: string) => `hashed:${raw}`), @@ -95,6 +99,20 @@ describe("hooks/manager", () => { }); describe("installHooks", () => { + it("refuses to enable Hermes before a daemon passes an end-to-end probe", async () => { + const { probeDaemonEndToEnd } = await import("../../src/hooks/daemon-service"); + const { writeScopedHooksConfig } = await import("../../src/hooks/hooks-config"); + vi.mocked(probeDaemonEndToEnd).mockResolvedValue(false); + + const { installHooks } = await import("../../src/hooks/manager"); + await expect( + installHooks(undefined, "user", undefined, false, undefined, undefined, false, ["hermes"]), + ).rejects.toThrow("failproofai config"); + + expect(writeScopedHooksConfig).not.toHaveBeenCalled(); + expect(writeFileSync).not.toHaveBeenCalled(); + }); + // 28, not 29: WorktreeCreate is deliberately not installed — Claude uses it // as a worktree-PATH PROVIDER (first hook's stdout becomes the directory), // and our silent-on-allow contract broke `claude --worktree` for every user. diff --git a/hermes-plugin/README.md b/hermes-plugin/README.md index 3e9766526..d33d39806 100644 --- a/hermes-plugin/README.md +++ b/hermes-plugin/README.md @@ -10,7 +10,12 @@ manifest v2 fields, `PluginContext` methods/state, and keyword hook payloads. ## Installation -Run: +Hermes policy evaluation requires a configured, healthy local `failproofaid` +daemon. If this machine has not been configured yet, run `failproofai config` +first. The standalone installer refuses to enable the fail-closed plugin until +the daemon answers an end-to-end health probe. + +Then run: ```bash failproofai policies --install --cli hermes --scope user diff --git a/hermes-plugin/client.py b/hermes-plugin/client.py index 3de407bbb..374a89ed6 100644 --- a/hermes-plugin/client.py +++ b/hermes-plugin/client.py @@ -6,6 +6,7 @@ import os import socket import struct +import time from dataclasses import dataclass from pathlib import Path from typing import Any, Mapping, Sequence @@ -38,11 +39,22 @@ def daemon_socket_path() -> Path: return home / "run" / "failproofaid.sock" -def _read_exact(sock: socket.socket, length: int) -> bytes: +def _remaining_timeout(deadline: float) -> float: + remaining = deadline - time.monotonic() + if remaining <= 0: + raise EvaluationError("failproofaid evaluation timed out") + return remaining + + +def _read_exact(sock: socket.socket, length: int, deadline: float) -> bytes: chunks: list[bytes] = [] remaining = length while remaining: - chunk = sock.recv(remaining) + sock.settimeout(_remaining_timeout(deadline)) + try: + chunk = sock.recv(remaining) + except TimeoutError as exc: + raise EvaluationError("failproofaid evaluation timed out") from exc if not chunk: raise EvaluationError("failproofaid closed the connection before returning a verdict") chunks.append(chunk) @@ -90,12 +102,16 @@ def evaluate_policy( with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as sock: sock.settimeout(max(connect_timeout_ms, 1) / 1000) sock.connect(str(daemon_socket_path())) - sock.settimeout(max(evaluation_timeout_ms, 1) / 1000) - sock.sendall(struct.pack(">I", len(body)) + body) - declared_length = struct.unpack(">I", _read_exact(sock, 4))[0] + deadline = time.monotonic() + max(evaluation_timeout_ms, 1) / 1000 + sock.settimeout(_remaining_timeout(deadline)) + try: + sock.sendall(struct.pack(">I", len(body)) + body) + except TimeoutError as exc: + raise EvaluationError("failproofaid evaluation timed out") from exc + declared_length = struct.unpack(">I", _read_exact(sock, 4, deadline))[0] if declared_length > MAX_FRAME_BYTES: raise EvaluationError("failproofaid response exceeds the 16 MiB limit") - response = json.loads(_read_exact(sock, declared_length).decode("utf-8")) + response = json.loads(_read_exact(sock, declared_length, deadline).decode("utf-8")) except EvaluationError: raise except (OSError, UnicodeError, ValueError, struct.error) as exc: diff --git a/src/hooks/manager.ts b/src/hooks/manager.ts index f77689b2d..e97cb996b 100644 --- a/src/hooks/manager.ts +++ b/src/hooks/manager.ts @@ -29,6 +29,7 @@ import { CORE_SOURCE, addPack, setPackPolicyEnabled } from "./pack-store"; import type { ResolvedPack } from "./pack-manifest"; import { hasInstalledPacks, readInstalledPacks } from "./pack-manifest"; import { packPolicyParamKey } from "./policy-evaluator"; +import { probeDaemonEndToEnd } from "./daemon-service"; import { chip, note, @@ -464,6 +465,20 @@ async function installHooksImpl( } } + // Hermes runs the FailproofAI plugin in-process and delegates every policy + // decision to failproofaid. Enabling the plugin without a daemon that can + // answer a real evaluation would therefore turn the default fail-closed + // behavior into an immediate lockout. The configure wizard has already + // installed and probed the daemon by the time it reaches this function; this + // guard protects direct `policies --install --cli hermes` invocations. + // Check before writing either the Hermes plugin or its config registration. + if (selectedClis.includes("hermes") && !(await probeDaemonEndToEnd())) { + throw new CliError( + "Hermes requires a healthy failproofaid daemon before its FailproofAI plugin can be enabled.\n" + + "Run `failproofai config` to install and configure the daemon, then retry.", + ); + } + const binaryPath = resolveFailproofaiBinary(); // Capture existing config before overwriting (used for telemetry diff) From 064ab595fdc55d8870e70a115285ea09c0d0f7b5 Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Tue, 15 Sep 2026 15:50:08 +0530 Subject: [PATCH 3/6] fix npm release propagation handling --- .github/workflows/publish.yml | 42 ++++++++++++++------------- CHANGELOG.md | 9 +++++- Cargo.lock | 6 ++-- Cargo.toml | 2 +- __tests__/ci/release-pipeline.test.ts | 4 ++- package.json | 2 +- 6 files changed, 38 insertions(+), 27 deletions(-) diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index d61200262..eb94150d6 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -670,28 +670,30 @@ jobs: done fi - MISSING=() - for NAME in "${NAMES[@]}"; do - # The registry is a read-through cache, so a just-published version - # can take a moment to be visible everywhere. Check immediately, - # then back off 10s / 30s / 1m / 2m before calling it missing — - # long enough that propagation is not mistaken for a failed publish, - # short enough that a genuinely failed publish is still reported in - # the same run rather than hours later by a user. - FOUND="" - for DELAY in 0 10 30 60 120; do - [[ "$DELAY" -gt 0 ]] && sleep "$DELAY" - if npm view "$NAME@$PUBLISH_VERSION" version >/dev/null 2>&1; then - FOUND=1 - break + # npm may accept a publish but keep a large package in asynchronous + # processing for much longer than normal registry propagation. The + # 1.0.6-beta.0 root package took 11m25s to become queryable after npm + # printed `+ failproofai@...`; the previous 3m40s window therefore + # reported a false version split and skipped install verification. + # Check every still-missing name in each round so several genuinely + # missing packages cannot multiply this bounded wait. + MISSING=("${NAMES[@]}") + for DELAY in 0 10 30 60 120 180 300 300; do + [[ "$DELAY" -gt 0 ]] && sleep "$DELAY" + STILL_MISSING=() + for NAME in "${MISSING[@]}"; do + if npm view "$NAME@$PUBLISH_VERSION" version --prefer-online >/dev/null 2>&1; then + echo " ok $NAME@$PUBLISH_VERSION" + else + STILL_MISSING+=("$NAME") fi done - if [[ -n "$FOUND" ]]; then - echo " ok $NAME@$PUBLISH_VERSION" - else - echo " MISSING $NAME@$PUBLISH_VERSION" - MISSING+=("$NAME") - fi + MISSING=("${STILL_MISSING[@]}") + [[ ${#MISSING[@]} -eq 0 ]] && break + done + + for NAME in "${MISSING[@]}"; do + echo " MISSING $NAME@$PUBLISH_VERSION" done if [[ ${#MISSING[@]} -gt 0 ]]; then diff --git a/CHANGELOG.md b/CHANGELOG.md index d53b108cb..c51a5ff76 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,13 @@ # Changelog +## 1.0.6-beta.1 — 2026-09-15 + +### Fixes + +- Require a healthy end-to-end `failproofaid` probe before direct Hermes installation changes plugin or profile configuration, preventing the default fail-closed mode from locking tool use when no evaluator is available. +- Apply Hermes `evaluation_timeout_ms` as one total socket deadline instead of resetting the full timeout for each partial send or receive. +- Allow npm up to 16 minutes 40 seconds to expose an accepted release while checking every package in parallel per retry round. The `1.0.6-beta.0` root package took 11 minutes 25 seconds to become visible after `npm publish` succeeded, causing the workflow to report a false version split while all five packages had actually published. + ## 1.0.6-beta.0 — 2026-09-15 ### Docs @@ -14,7 +22,6 @@ ### Changed - Hermes installation now copies and enables the managed plugin in every profile, migrates only legacy FailproofAI shell hooks, refuses to overwrite unmanaged plugin directories, and reports incomplete or duplicate profile installations as unhealthy. -- Direct Hermes installation now requires a healthy end-to-end `failproofaid` probe before changing plugin or profile configuration, preventing the default fail-closed mode from locking tool use when no evaluator is available. Hermes policy evaluation also applies `evaluation_timeout_ms` as one total socket deadline instead of resetting it for each partial read. ### Dependencies diff --git a/Cargo.lock b/Cargo.lock index bc17f05df..caff20c25 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -202,7 +202,7 @@ dependencies = [ [[package]] name = "failproofaid" -version = "1.0.6-beta.0" +version = "1.0.6-beta.1" dependencies = [ "fpai-collect", "fpai-ipc", @@ -265,7 +265,7 @@ dependencies = [ [[package]] name = "fpai-collect" -version = "1.0.6-beta.0" +version = "1.0.6-beta.1" dependencies = [ "notify", "reqwest", @@ -280,7 +280,7 @@ dependencies = [ [[package]] name = "fpai-ipc" -version = "1.0.6-beta.0" +version = "1.0.6-beta.1" dependencies = [ "libc", "proptest", diff --git a/Cargo.toml b/Cargo.toml index 3bfa308b2..65f46f261 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -3,7 +3,7 @@ resolver = "3" members = ["crates/*"] [workspace.package] -version = "1.0.6-beta.0" +version = "1.0.6-beta.1" edition = "2024" license-file = "LICENSE" repository = "https://github.com/FailproofAI/failproofai" diff --git a/__tests__/ci/release-pipeline.test.ts b/__tests__/ci/release-pipeline.test.ts index 9e0bd44b0..bb3ca0cb2 100644 --- a/__tests__/ci/release-pipeline.test.ts +++ b/__tests__/ci/release-pipeline.test.ts @@ -195,7 +195,9 @@ describe("publish.yml", () => { expect(verify.if).toContain("dry_run != 'true'"); // The registry is a read-through cache — propagation must not read as a // failed publish, and a failed publish must not wait forever. - expect(verify.run).toContain("for DELAY in 0 10 30 60 120"); + expect(verify.run).toContain("for DELAY in 0 10 30 60 120 180 300 300"); + expect(verify.run).toContain('MISSING=("${NAMES[@]}")'); + expect(verify.run).toContain('STILL_MISSING+=("$NAME")'); }); it("installs the published packages from the registry, once per platform", () => { diff --git a/package.json b/package.json index e5b253e53..23e3ec70b 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "failproofai", - "version": "1.0.6-beta.0", + "version": "1.0.6-beta.1", "description": "Observability and enforcement for AI agent harnesses. 39 built-in policies hooked into 12 of them — Claude Code, Codex, Cursor, Hermes, OpenClaw and more — blocking the tool call before it runs. Local dashboard included, no account needed.", "bin": { "failproofai": "./dist/cli.mjs", From 7299f3eea3f7d703a9c81b64edaffe3c95b4893c Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Tue, 15 Sep 2026 16:30:35 +0530 Subject: [PATCH 4/6] harden hermes multi-home installation --- CHANGELOG.md | 2 + __tests__/hooks/integrations.test.ts | 41 +++++++++++++ __tests__/hooks/manager.test.ts | 21 +++++++ __tests__/lib/hermes-profiles.test.ts | 57 ++++++++++++++++++ lib/hermes-profiles.ts | 86 +++++++++++++++++++++------ lib/hermes-sessions.ts | 5 +- src/hooks/integrations.ts | 7 +++ src/hooks/manager.ts | 29 +++++---- 8 files changed, 216 insertions(+), 32 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c51a5ff76..8158118cc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,8 @@ ### Fixes - Require a healthy end-to-end `failproofaid` probe before direct Hermes installation changes plugin or profile configuration, preventing the default fail-closed mode from locking tool use when no evaluator is available. +- Discover config-bearing `~/.hermes-` installations alongside the default and upstream nested profiles, so install, uninstall, health checks, and audit coverage reach every Hermes home without mistaking empty backup directories for active profiles. +- Make daemon readiness an integration capability rather than a Hermes command special-case, so future daemon-only plugins inherit the same pre-install lockout protection while CLI-backed integrations keep their local evaluator fallback. - Apply Hermes `evaluation_timeout_ms` as one total socket deadline instead of resetting the full timeout for each partial send or receive. - Allow npm up to 16 minutes 40 seconds to expose an accepted release while checking every package in parallel per retry round. The `1.0.6-beta.0` root package took 11 minutes 25 seconds to become visible after `npm publish` succeeded, causing the workflow to report a false version split while all five packages had actually published. diff --git a/__tests__/hooks/integrations.test.ts b/__tests__/hooks/integrations.test.ts index 4344fa7f4..32f013d47 100644 --- a/__tests__/hooks/integrations.test.ts +++ b/__tests__/hooks/integrations.test.ts @@ -123,6 +123,12 @@ describe("integrations registry", () => { expect(getIntegration("hermes")).toBe(hermes); }); + it("declares daemon-only installation requirements in registry metadata", () => { + expect(listIntegrations().filter((integration) => integration.requiresHealthyDaemon)).toEqual([ + hermes, + ]); + }); + it("getIntegration('openclaw') returns openclaw", () => { expect(getIntegration("openclaw")).toBe(openclaw); }); @@ -591,6 +597,15 @@ describe("Hermes integration", () => { } } + /** Create legacy/custom `~/.hermes-` homes that Hermes can run independently. */ + function makeSiblingHomes(...names: string[]): void { + for (const name of names) { + const home = resolve(tempDir, `.hermes-${name}`); + mkdirSync(home, { recursive: true }); + writeFileSync(resolve(home, "config.yaml"), `model: ${name}\n`); + } + } + function pluginPath(settingsPath: string): string { return resolve(dirname(settingsPath), "plugins", "failproofai"); } @@ -858,6 +873,19 @@ describe("Hermes integration", () => { ]); }); + it("getSettingsPaths covers valid sibling Hermes installations", () => { + makeProfiles("nested"); + makeSiblingHomes("work", "personal"); + mkdirSync(resolve(tempDir, ".hermes-backup"), { recursive: true }); + + expect(hermes.getSettingsPaths!("user")).toEqual([ + resolve(tempDir, ".hermes", "config.yaml"), + resolve(tempDir, ".hermes", "profiles", "nested", "config.yaml"), + resolve(tempDir, ".hermes-personal", "config.yaml"), + resolve(tempDir, ".hermes-work", "config.yaml"), + ]); + }); + it("settingsPathsFor falls back to the single path for non-profile integrations", () => { expect(settingsPathsFor(claudeCode, "user")).toEqual([claudeCode.getSettingsPath("user")]); expect(settingsPathsFor(hermes, "user")).toEqual(hermes.getSettingsPaths!("user")); @@ -877,6 +905,19 @@ describe("Hermes integration", () => { expect(unhookedHermesProfiles()).toEqual([]); }); + it("health remains false until every sibling installation is hooked", () => { + makeSiblingHomes("work"); + const [rootPath, workPath] = settingsPathsFor(hermes, "user"); + + installAt(rootPath); + expect(hermes.hooksInstalledInSettings("user")).toBe(false); + expect(unhookedHermesProfiles()).toEqual(["work"]); + + installAt(workPath); + expect(hermes.hooksInstalledInSettings("user")).toBe(true); + expect(unhookedHermesProfiles()).toEqual([]); + }); + it("installs and enables the native plugin in a non-default profile", () => { makeProfiles("work"); const workPath = resolve(tempDir, ".hermes", "profiles", "work", "config.yaml"); diff --git a/__tests__/hooks/manager.test.ts b/__tests__/hooks/manager.test.ts index 0bb2a8367..55245ae21 100644 --- a/__tests__/hooks/manager.test.ts +++ b/__tests__/hooks/manager.test.ts @@ -113,6 +113,27 @@ describe("hooks/manager", () => { expect(writeFileSync).not.toHaveBeenCalled(); }); + it("enforces daemon readiness from integration metadata rather than a Hermes ID check", async () => { + const { probeDaemonEndToEnd } = await import("../../src/hooks/daemon-service"); + const { writeScopedHooksConfig } = await import("../../src/hooks/hooks-config"); + const { claudeCode } = await import("../../src/hooks/integrations"); + const previous = claudeCode.requiresHealthyDaemon; + claudeCode.requiresHealthyDaemon = true; + vi.mocked(probeDaemonEndToEnd).mockResolvedValue(false); + + try { + const { installHooks } = await import("../../src/hooks/manager"); + await expect( + installHooks(undefined, "user", undefined, false, undefined, undefined, false, ["claude"]), + ).rejects.toThrow("Claude Code requires a healthy failproofaid daemon"); + } finally { + claudeCode.requiresHealthyDaemon = previous; + } + + expect(writeScopedHooksConfig).not.toHaveBeenCalled(); + expect(writeFileSync).not.toHaveBeenCalled(); + }); + // 28, not 29: WorktreeCreate is deliberately not installed — Claude uses it // as a worktree-PATH PROVIDER (first hook's stdout becomes the directory), // and our silent-on-allow contract broke `claude --worktree` for every user. diff --git a/__tests__/lib/hermes-profiles.test.ts b/__tests__/lib/hermes-profiles.test.ts index 43ed64992..cdc6a3889 100644 --- a/__tests__/lib/hermes-profiles.test.ts +++ b/__tests__/lib/hermes-profiles.test.ts @@ -89,6 +89,63 @@ describe("listHermesProfiles", () => { expect(names.filter((n) => n === "default")).toHaveLength(1); expect(listHermesProfiles()[0].home).toBe(root); // the root wins }); + + it("discovers config-bearing sibling homes and ignores unrelated backups", async () => { + const parent = mkdtempSync(join(tmpdir(), "hermes-homes-")); + dirs.push(parent); + const root = join(parent, ".hermes"); + mkdirSync(join(root, "profiles", "nested"), { recursive: true }); + writeFileSync(join(root, "config.yaml"), "model: default\n"); + mkdirSync(join(parent, ".hermes-work"), { recursive: true }); + writeFileSync(join(parent, ".hermes-work", "config.yaml"), "model: work\n"); + mkdirSync(join(parent, ".hermes-backup"), { recursive: true }); + process.env.HERMES_HOME = root; + + const { listHermesProfiles } = await import("@/lib/hermes-profiles"); + expect(listHermesProfiles()).toEqual([ + { name: "default", home: root }, + { name: "nested", home: join(root, "profiles", "nested") }, + { name: "work", home: join(parent, ".hermes-work") }, + ]); + }); + + it("normalizes HERMES_HOME at a sibling installation and still finds the default", async () => { + const parent = mkdtempSync(join(tmpdir(), "hermes-homes-")); + dirs.push(parent); + const root = join(parent, ".hermes"); + const work = join(parent, ".hermes-work"); + mkdirSync(root, { recursive: true }); + mkdirSync(work, { recursive: true }); + writeFileSync(join(root, "config.yaml"), "model: default\n"); + writeFileSync(join(work, "config.yaml"), "model: work\n"); + process.env.HERMES_HOME = work; + + const { hermesRoot, listHermesProfiles } = await import("@/lib/hermes-profiles"); + expect(hermesRoot()).toBe(root); + expect(listHermesProfiles()).toEqual([ + { name: "default", home: root }, + { name: "work", home: work }, + ]); + }); + + it("keeps both homes when nested and sibling profiles share a name", async () => { + const parent = mkdtempSync(join(tmpdir(), "hermes-homes-")); + dirs.push(parent); + const root = join(parent, ".hermes"); + const sibling = join(parent, ".hermes-work"); + mkdirSync(join(root, "profiles", "work"), { recursive: true }); + mkdirSync(sibling, { recursive: true }); + writeFileSync(join(root, "config.yaml"), "model: default\n"); + writeFileSync(join(sibling, "config.yaml"), "model: sibling\n"); + process.env.HERMES_HOME = root; + + const { listHermesProfiles } = await import("@/lib/hermes-profiles"); + expect(listHermesProfiles()).toEqual([ + { name: "default", home: root }, + { name: "work", home: join(root, "profiles", "work") }, + { name: "work-home", home: sibling }, + ]); + }); }); describe("hermesDbPaths", () => { diff --git a/lib/hermes-profiles.ts b/lib/hermes-profiles.ts index 543e6012b..dbb2e6c67 100644 --- a/lib/hermes-profiles.ts +++ b/lib/hermes-profiles.ts @@ -3,10 +3,11 @@ * * A Hermes "profile" is not a column or a flag: it is a whole separate Hermes * home directory, each with its own `config.yaml`, `.env`, `SOUL.md`, and - * `state.db`. The default profile lives at `~/.hermes`; every other profile at - * `~/.hermes/profiles//`. Selection is `hermes -p `, a generated - * `~/.local/bin/` alias that exports `HERMES_HOME`, or a sticky default - * recorded in `/active_profile`. + * `state.db`. The default profile lives at `~/.hermes`; upstream named profiles + * live at `~/.hermes/profiles//`, while older/custom multi-install setups + * may use sibling homes such as `~/.hermes-work`. Selection is `hermes -p + * `, a generated `~/.local/bin/` alias that exports `HERMES_HOME`, + * or a sticky default recorded in `/active_profile`. * * Upstream's own contributor guide warns that hardcoding `~/.hermes` breaks * profiles — which is exactly what both pillars used to do: @@ -21,12 +22,14 @@ * Home override: set `HERMES_HOME` (Hermes's own env var — respected here so a * profile-scoped shell and failproofai agree on what "all profiles" means). */ -import { readdirSync } from "node:fs"; +import { existsSync, readdirSync } from "node:fs"; import { homedir } from "node:os"; import { basename, dirname, join, resolve } from "node:path"; /** Directory under the Hermes root that holds non-default profiles. */ const PROFILES_DIR = "profiles"; +const STANDARD_HOME = ".hermes"; +const SIBLING_HOME_PREFIX = ".hermes-"; /** Name we give the root home (`~/.hermes`), which Hermes itself leaves unnamed. */ export const HERMES_DEFAULT_PROFILE = "default"; @@ -41,28 +44,45 @@ export interface HermesProfile { /** * The Hermes ROOT home — the directory that owns `profiles/`. * - * `HERMES_HOME` may point AT a profile (`/profiles/`), because - * that's what the per-profile alias wrapper exports. We climb back to `` - * in that case so discovery still sees every sibling profile — mirroring what - * upstream does so `profile list` can see them all. + * `HERMES_HOME` may point AT an upstream profile (`/profiles/`) or + * at a sibling installation (`~/.hermes-`). We normalize either standard + * layout back to `~/.hermes` so discovery covers every installation. A custom + * non-standard path remains authoritative; scanning arbitrary sibling paths + * would risk treating backups and unrelated directories as live agents. */ export function hermesRoot(): string { const env = (process.env.HERMES_HOME || "").trim(); if (env) { - const home = resolve(env); + let home = resolve(env); const parent = dirname(home); - if (basename(parent) === PROFILES_DIR) return dirname(parent); + if (basename(parent) === PROFILES_DIR) home = dirname(parent); + const base = basename(home); + if (base === STANDARD_HOME || base.startsWith(SIBLING_HOME_PREFIX)) { + return join(dirname(home), STANDARD_HOME); + } return home; } - return join(homedir(), ".hermes"); + return join(homedir(), STANDARD_HOME); +} + +function uniqueProfileName(preferred: string, seen: Set): string { + if (!seen.has(preferred)) return preferred; + const base = `${preferred}-home`; + if (!seen.has(base)) return base; + let suffix = 2; + while (seen.has(`${base}-${suffix}`)) suffix += 1; + return `${base}-${suffix}`; } /** * Every Hermes profile on disk: the root home first (as `"default"`), then each - * `/profiles//` in name order. + * `/profiles//`, then valid `~/.hermes-` sibling homes, in + * name order within each layout. * - * Fail-open — a missing or unreadable `profiles/` dir just means "default only", - * which is the single-profile install everyone starts with. + * A sibling home counts only when it contains `config.yaml`, matching OpenClaw + * discovery and avoiding arbitrary `.hermes-*` backup directories. Upstream + * `profiles/` directories are authoritative and remain discoverable before + * their config is first written. * * A profile directory literally named `default` would collide with the root's * reserved name; the root wins and the directory is skipped (dedup by name keeps @@ -73,22 +93,50 @@ export function listHermesProfiles(): HermesProfile[] { const out: HermesProfile[] = [{ name: HERMES_DEFAULT_PROFILE, home: root }]; const seen = new Set([HERMES_DEFAULT_PROFILE]); - let names: string[]; + let names: string[] = []; try { names = readdirSync(join(root, PROFILES_DIR), { withFileTypes: true }) // Symlinked profile dirs are legitimate, and `isDirectory()` is false for them. .filter((e) => (e.isDirectory() || e.isSymbolicLink()) && !e.name.startsWith(".")) .map((e) => e.name) .sort(); - } catch { - return out; - } + } catch {} for (const name of names) { if (seen.has(name)) continue; seen.add(name); out.push({ name, home: join(root, PROFILES_DIR, name) }); } + + // Only the standard ~/.hermes layout defines `.hermes-*` sibling homes. + // A custom HERMES_HOME is intentionally not used as a prefix convention. + if (basename(root) !== STANDARD_HOME) return out; + + let siblings: HermesProfile[] = []; + const parent = dirname(root); + try { + siblings = readdirSync(parent, { withFileTypes: true }) + .filter( + (entry) => + (entry.isDirectory() || entry.isSymbolicLink()) && + entry.name.startsWith(SIBLING_HOME_PREFIX) && + entry.name.length > SIBLING_HOME_PREFIX.length && + existsSync(join(parent, entry.name, "config.yaml")), + ) + .map((entry) => ({ + name: entry.name.slice(SIBLING_HOME_PREFIX.length), + home: join(parent, entry.name), + })) + .sort((a, b) => a.name.localeCompare(b.name)); + } catch { + return out; + } + + for (const sibling of siblings) { + const name = uniqueProfileName(sibling.name, seen); + seen.add(name); + out.push({ name, home: sibling.home }); + } return out; } diff --git a/lib/hermes-sessions.ts b/lib/hermes-sessions.ts index c0fbcc0dd..ae9a53cab 100644 --- a/lib/hermes-sessions.ts +++ b/lib/hermes-sessions.ts @@ -14,8 +14,9 @@ * * PROFILES: Hermes profiles are separate home dirs, each with its OWN state.db * (`~/.hermes/state.db` for the default, `~/.hermes/profiles//state.db` - * otherwise) — see lib/hermes-profiles.ts. So every read here fans out across - * profiles instead of assuming one DB. + * for upstream profiles, and `~/.hermes-/state.db` for sibling homes) — + * see lib/hermes-profiles.ts. So every read here fans out across profiles + * instead of assuming one DB. * * DB path override: set `HERMES_DB_PATH` (used by tests and to point at a copied * or remote state.db) — it collapses discovery to that single file. Set diff --git a/src/hooks/integrations.ts b/src/hooks/integrations.ts index 055b08ab0..e6cbb1311 100644 --- a/src/hooks/integrations.ts +++ b/src/hooks/integrations.ts @@ -122,6 +122,12 @@ function binaryExists(name: string): boolean { export interface Integration { id: IntegrationType; displayName: string; + /** + * This integration has no in-process/CLI fallback and cannot evaluate policy + * unless failproofaid answers an end-to-end probe. Installation must stop + * before writing any hook or plugin when the daemon is unavailable. + */ + requiresHealthyDaemon?: boolean; /** Settings scopes this integration supports (e.g. claude: user/project/local; codex: user/project). */ scopes: readonly HookScope[]; /** Hook events this integration fires (Claude: PascalCase, Codex: snake_case stored as Pascal in settings). */ @@ -1589,6 +1595,7 @@ export function hermesProfileStatusRows(): Array<[string, string]> { export const hermes: Integration = { id: "hermes", displayName: "Hermes", + requiresHealthyDaemon: true, scopes: HERMES_HOOK_SCOPES, eventTypes: HERMES_HOOK_EVENT_TYPES, diff --git a/src/hooks/manager.ts b/src/hooks/manager.ts index e97cb996b..151b3dfac 100644 --- a/src/hooks/manager.ts +++ b/src/hooks/manager.ts @@ -447,9 +447,13 @@ async function installHooksImpl( // the user for multi-CLI selection before reaching here when --cli is omitted. const selectedClis: IntegrationType[] = cli && cli.length > 0 ? [...new Set(cli)] : ["claude"]; + const selectedIntegrations = selectedClis.map((cliId) => ({ + cliId, + integration: getIntegration(cliId), + })); + // Per-CLI scope validation: Codex doesn't have a "local" scope. - for (const cliId of selectedClis) { - const integration = getIntegration(cliId); + for (const { cliId, integration } of selectedIntegrations) { if (!integration.scopes.includes(scope)) { try { await trackHookEvent(getInstanceId(), "scope_validation_failed", { @@ -465,16 +469,19 @@ async function installHooksImpl( } } - // Hermes runs the FailproofAI plugin in-process and delegates every policy - // decision to failproofaid. Enabling the plugin without a daemon that can - // answer a real evaluation would therefore turn the default fail-closed - // behavior into an immediate lockout. The configure wizard has already - // installed and probed the daemon by the time it reaches this function; this - // guard protects direct `policies --install --cli hermes` invocations. - // Check before writing either the Hermes plugin or its config registration. - if (selectedClis.includes("hermes") && !(await probeDaemonEndToEnd())) { + // Daemon-only native integrations fail closed when their evaluator cannot be + // reached. Never enable one until a real policy request succeeds; otherwise + // a direct `policies --install` can lock every tool call. Shell-hook and CLI- + // backed integrations retain their local evaluator fallback and therefore do + // not opt into this requirement. + const daemonRequiredBy = selectedIntegrations + .map(({ integration }) => integration) + .filter((integration) => integration.requiresHealthyDaemon); + if (daemonRequiredBy.length > 0 && !(await probeDaemonEndToEnd())) { + const names = daemonRequiredBy.map((integration) => integration.displayName).join(", "); + const verb = daemonRequiredBy.length === 1 ? "requires" : "require"; throw new CliError( - "Hermes requires a healthy failproofaid daemon before its FailproofAI plugin can be enabled.\n" + + `${names} ${verb} a healthy failproofaid daemon before FailproofAI enforcement can be enabled.\n` + "Run `failproofai config` to install and configure the daemon, then retry.", ); } From f7cadb60209e3b5b530a2049a50d1d88822a1b7a Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Wed, 16 Sep 2026 14:35:06 +0530 Subject: [PATCH 5/6] fix hermes daemon capability check --- CHANGELOG.md | 2 +- __tests__/hooks/daemon-client.test.ts | 58 +++++++ __tests__/hooks/daemon-probe-race.test.ts | 47 +++++- __tests__/hooks/manager.test.ts | 14 +- src/hooks/daemon-client.ts | 178 ++++++++++++++++------ src/hooks/daemon-service.ts | 30 ++++ src/hooks/integrations.ts | 5 +- src/hooks/manager.ts | 8 +- 8 files changed, 276 insertions(+), 66 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8158118cc..82ebee6b9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ ### Fixes -- Require a healthy end-to-end `failproofaid` probe before direct Hermes installation changes plugin or profile configuration, preventing the default fail-closed mode from locking tool use when no evaluator is available. +- Require an exact `policyEvaluation` / `policyResult` capability probe before direct Hermes installation changes plugin or profile configuration. A healthy older daemon that only supports shell-hook requests is now rejected with upgrade instructions instead of enabling a fail-closed plugin it cannot serve. - Discover config-bearing `~/.hermes-` installations alongside the default and upstream nested profiles, so install, uninstall, health checks, and audit coverage reach every Hermes home without mistaking empty backup directories for active profiles. - Make daemon readiness an integration capability rather than a Hermes command special-case, so future daemon-only plugins inherit the same pre-install lockout protection while CLI-backed integrations keep their local evaluator fallback. - Apply Hermes `evaluation_timeout_ms` as one total socket deadline instead of resetting the full timeout for each partial send or receive. diff --git a/__tests__/hooks/daemon-client.test.ts b/__tests__/hooks/daemon-client.test.ts index 4a4c5fe16..e3c78768a 100644 --- a/__tests__/hooks/daemon-client.test.ts +++ b/__tests__/hooks/daemon-client.test.ts @@ -108,6 +108,64 @@ describe("hooks/daemon-client", () => { expect(result).toEqual({ exitCode: 0, stdout: "", stderr: "" }); }); + it("proves native policy evaluation only from a complete policyResult", async () => { + await startServer(async (socket) => { + const req = await readFrame(socket); + expect(req).toMatchObject({ + type: "policyEvaluation", + protocolVersion: 1, + integration: "hermes", + event: "on_session_start", + payload: { hook_event_name: "on_session_start" }, + }); + socket.end( + encodeFrame({ + type: "policyResult", + protocolVersion: 1, + decision: "allow", + policyNames: [], + reason: null, + matchedPolicies: [], + durationMs: 1, + toolName: null, + }), + ); + }); + + const { attemptDaemonPolicyEvaluation } = await import("../../src/hooks/daemon-client"); + await expect( + attemptDaemonPolicyEvaluation({ + integration: "hermes", + event: "on_session_start", + payload: { hook_event_name: "on_session_start" }, + }), + ).resolves.toEqual({ ok: true }); + }); + + it("does not mistake a matching-version hook response for native policy support", async () => { + await startServer(async (socket) => { + await readFrame(socket); + socket.end( + encodeFrame({ + type: "hookResult", + protocolVersion: 1, + exitCode: 0, + stdout: "", + stderr: "", + }), + ); + }); + + const { attemptDaemonPolicyEvaluation } = await import("../../src/hooks/daemon-client"); + await expect( + attemptDaemonPolicyEvaluation({ + integration: "hermes", + event: "on_session_start", + payload: {}, + }), + ).resolves.toEqual({ ok: false, failure: "unreachable" }); + }); + it("round-trips a deny response with real stdout/stderr content", async () => { await startServer(async (socket) => { await readFrame(socket); diff --git a/__tests__/hooks/daemon-probe-race.test.ts b/__tests__/hooks/daemon-probe-race.test.ts index 17d385859..8b05a75ff 100644 --- a/__tests__/hooks/daemon-probe-race.test.ts +++ b/__tests__/hooks/daemon-probe-race.test.ts @@ -21,8 +21,8 @@ let sockPath: string; let server: Server | null = null; const originalSocket = process.env.FAILPROOFAI_DAEMON_SOCKET; -/** A stand-in daemon: answers `ping` with `pong` and `hook` with exit 0. */ -function startDaemon(opts: { answerHooks: boolean }): Promise { +/** A stand-in daemon with independently selectable old and native APIs. */ +function startDaemon(opts: { answerHooks: boolean; answerPolicyEvaluations?: boolean }): Promise { return new Promise((resolve) => { const s = createServer((conn) => { let buf = Buffer.alloc(0); @@ -39,6 +39,29 @@ function startDaemon(opts: { answerHooks: boolean }): Promise { // is exactly the case the taxonomy has to tell apart from a socket that // never came up, so this stub must accept and then stay silent. if (msg.type === "hook" && !opts.answerHooks) return; + if (msg.type === "policyEvaluation") { + const value = opts.answerPolicyEvaluations + ? { + type: "policyResult", + protocolVersion: 1, + decision: "allow", + policyNames: [], + reason: null, + matchedPolicies: [], + durationMs: 0, + toolName: null, + } + : { + type: "error", + protocolVersion: 1, + message: "unknown variant `policyEvaluation`", + }; + const body = Buffer.from(JSON.stringify(value), "utf-8"); + const head = Buffer.alloc(4); + head.writeUInt32BE(body.length, 0); + conn.write(Buffer.concat([head, body])); + return; + } const body = Buffer.from( JSON.stringify( msg.type === "ping" @@ -113,4 +136,24 @@ describe("hooks/daemon-service — health probe startup race", () => { const probe = await probeDaemon(); expect(probe).toEqual({ ok: false, reason: "worker" }); }, 40_000); + + it("rejects a hook-compatible v1 daemon that lacks policyEvaluation", async () => { + server = await startDaemon({ answerHooks: true, answerPolicyEvaluations: false }); + const { probeDaemonEndToEnd, probeDaemonPolicyEvaluation } = await import( + "../../src/hooks/daemon-service" + ); + + // This is the dangerous upgrade state: the old health check passes, but + // enabling the native plugin would make every request hit its fail-closed + // fallback because the daemon cannot answer the request Hermes actually uses. + expect(await probeDaemonEndToEnd()).toBe(true); + expect(await probeDaemonPolicyEvaluation()).toBe(false); + }, 20_000); + + it("accepts a daemon that returns a complete native policy result", async () => { + server = await startDaemon({ answerHooks: true, answerPolicyEvaluations: true }); + const { probeDaemonPolicyEvaluation } = await import("../../src/hooks/daemon-service"); + + expect(await probeDaemonPolicyEvaluation()).toBe(true); + }, 20_000); }); diff --git a/__tests__/hooks/manager.test.ts b/__tests__/hooks/manager.test.ts index 55245ae21..786e3b360 100644 --- a/__tests__/hooks/manager.test.ts +++ b/__tests__/hooks/manager.test.ts @@ -62,7 +62,7 @@ vi.mock("../../src/hooks/hook-telemetry", () => ({ })); vi.mock("../../src/hooks/daemon-service", () => ({ - probeDaemonEndToEnd: vi.fn(() => Promise.resolve(true)), + probeDaemonPolicyEvaluation: vi.fn(() => Promise.resolve(true)), })); vi.mock("../../lib/telemetry-id", () => ({ @@ -99,10 +99,10 @@ describe("hooks/manager", () => { }); describe("installHooks", () => { - it("refuses to enable Hermes before a daemon passes an end-to-end probe", async () => { - const { probeDaemonEndToEnd } = await import("../../src/hooks/daemon-service"); + it("refuses to enable Hermes before a daemon proves native policy evaluation support", async () => { + const { probeDaemonPolicyEvaluation } = await import("../../src/hooks/daemon-service"); const { writeScopedHooksConfig } = await import("../../src/hooks/hooks-config"); - vi.mocked(probeDaemonEndToEnd).mockResolvedValue(false); + vi.mocked(probeDaemonPolicyEvaluation).mockResolvedValue(false); const { installHooks } = await import("../../src/hooks/manager"); await expect( @@ -114,18 +114,18 @@ describe("hooks/manager", () => { }); it("enforces daemon readiness from integration metadata rather than a Hermes ID check", async () => { - const { probeDaemonEndToEnd } = await import("../../src/hooks/daemon-service"); + const { probeDaemonPolicyEvaluation } = await import("../../src/hooks/daemon-service"); const { writeScopedHooksConfig } = await import("../../src/hooks/hooks-config"); const { claudeCode } = await import("../../src/hooks/integrations"); const previous = claudeCode.requiresHealthyDaemon; claudeCode.requiresHealthyDaemon = true; - vi.mocked(probeDaemonEndToEnd).mockResolvedValue(false); + vi.mocked(probeDaemonPolicyEvaluation).mockResolvedValue(false); try { const { installHooks } = await import("../../src/hooks/manager"); await expect( installHooks(undefined, "user", undefined, false, undefined, undefined, false, ["claude"]), - ).rejects.toThrow("Claude Code requires a healthy failproofaid daemon"); + ).rejects.toThrow("Claude Code requires a compatible failproofaid daemon with native policy evaluation"); } finally { claudeCode.requiresHealthyDaemon = previous; } diff --git a/src/hooks/daemon-client.ts b/src/hooks/daemon-client.ts index 394d518c6..6b5683168 100644 --- a/src/hooks/daemon-client.ts +++ b/src/hooks/daemon-client.ts @@ -93,6 +93,26 @@ export type DaemonAttempt = | { ok: true; response: DaemonHookResponse } | { ok: false; failure: DaemonFailure }; +export interface DaemonPolicyEvaluationRequest { + integration: IntegrationType; + event: string; + payload: Record; + cwd?: string; +} + +export type DaemonPolicyEvaluationAttempt = + | { ok: true } + | { ok: false; failure: DaemonFailure }; + +interface DaemonRequestOptions { + /** Override the response budget; the connect budget is never relaxed. */ + responseTimeoutMs?: number; +} + +type DaemonWireAttempt = + | { ok: true; message: Record } + | { ok: false; failure: DaemonFailure }; + /** * Whether a daemon is listening, regardless of how it was started. * @@ -176,29 +196,22 @@ function encodeFrame(value: unknown): Buffer { return Buffer.concat([header, body]); } -/** Attempts a daemon evaluation while preserving the failure category. */ -export async function attemptDaemonHook( - req: DaemonHookRequest, - opts?: { - /** - * Override the RESPONSE budget only — the connect budget is never - * relaxed. Used by the health probe (`probeDaemonEndToEnd`), which runs - * from an interactive command rather than a hook and must not make a - * person wait out the full 30s hook budget to be told their daemon is - * broken. Never set this on the hook path: 30s is matched to the - * daemon's own read timeout so this side never gives up on a request the - * daemon is still honestly working on. - */ - responseTimeoutMs?: number; - }, -): Promise { +/** + * Sends one framed daemon request while preserving the failure category. + * Shape validation belongs to the typed caller: both shell hooks and native + * integrations share the transport, but require different response variants. + */ +async function attemptDaemonRequest( + request: Record, + opts?: DaemonRequestOptions, +): Promise { // Windows never has a daemon in this phase (see the plan's platform // scope) — skip the attempt outright rather than depending on however // Node happens to behave when handed a POSIX socket path on Windows. if (process.platform === "win32") return { ok: false, failure: "unreachable" }; const responseBudget = opts?.responseTimeoutMs ?? DAEMON_RESPONSE_TIMEOUT_MS; - return new Promise((resolvePromise) => { + return new Promise((resolvePromise) => { let settled = false; let timer: ReturnType; const arm = (ms: number) => { @@ -210,7 +223,7 @@ export async function attemptDaemonHook( // process lingers if something upstream forgets to await us. timer.unref?.(); }; - const finish = (result: DaemonAttempt) => { + const finish = (result: DaemonWireAttempt) => { if (settled) return; settled = true; clearTimeout(timer); @@ -230,16 +243,7 @@ export async function attemptDaemonHook( // Connected: the daemon is demonstrably reachable, so the question is // no longer "is it there" but "how long does this evaluation take". arm(responseBudget); - socket.write( - encodeFrame({ - type: "hook", - protocolVersion: PROTOCOL_VERSION, - hookEvent: req.hookEvent, - cli: req.cli, - stdin: req.stdin, - cwd: req.cwd, - }), - ); + socket.write(encodeFrame(request)); }); socket.on("data", (chunk: Buffer) => { @@ -258,13 +262,18 @@ export async function attemptDaemonHook( if (recvBuf.length < declaredLen) return; const body = recvBuf.subarray(0, declaredLen); - let message: Record; + let parsed: unknown; try { - message = JSON.parse(body.toString("utf8")) as Record; + parsed = JSON.parse(body.toString("utf8")); } catch { fail("unreachable"); return; } + if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) { + fail("unreachable"); + return; + } + const message = parsed as Record; if (message.protocolVersion !== PROTOCOL_VERSION) { // Catches BOTH directions. A newer CLI sends v2 and the daemon replies @@ -274,29 +283,98 @@ export async function attemptDaemonHook( fail("protocol-mismatch"); return; } - if ( - message.type === "hookResult" && - typeof message.exitCode === "number" && - typeof message.stdout === "string" && - typeof message.stderr === "string" - ) { - finish({ - ok: true, - response: { - exitCode: message.exitCode, - stdout: message.stdout, - stderr: message.stderr, - }, - }); - return; - } - // Anything else — an explicit `error` message at a MATCHING protocol - // version, a `pong` (protocol confusion), or a well-formed-but-wrong-shape - // body — is treated identically to a connection failure: no partial trust. - fail("unreachable"); + finish({ ok: true, message }); }); socket.on("error", () => fail("unreachable")); socket.on("close", () => fail("unreachable")); }); } + +/** Attempts a shell-hook daemon evaluation while preserving the failure category. */ +export async function attemptDaemonHook( + req: DaemonHookRequest, + opts?: DaemonRequestOptions, +): Promise { + const attempt = await attemptDaemonRequest( + { + type: "hook", + protocolVersion: PROTOCOL_VERSION, + hookEvent: req.hookEvent, + cli: req.cli, + stdin: req.stdin, + cwd: req.cwd, + }, + opts, + ); + if (!attempt.ok) return attempt; + + const message = attempt.message; + if ( + message.type === "hookResult" && + typeof message.exitCode === "number" && + typeof message.stdout === "string" && + typeof message.stderr === "string" + ) { + return { + ok: true, + response: { + exitCode: message.exitCode, + stdout: message.stdout, + stderr: message.stderr, + }, + }; + } + + // An explicit matching-version error, pong, or malformed hook result is not + // an evaluation. Never grant partial trust to a daemon that answered the + // connection but not the request we sent. + return { ok: false, failure: "unreachable" }; +} + +function isStringArray(value: unknown): value is string[] { + return Array.isArray(value) && value.every((item) => typeof item === "string"); +} + +/** + * Attempts the structured request used by native in-process integrations. + * A matching protocol version is not sufficient: pre-native v1 daemons speak + * `hook` but cannot deserialize `policyEvaluation`, so only a complete, + * well-shaped `policyResult` proves this capability exists. + */ +export async function attemptDaemonPolicyEvaluation( + req: DaemonPolicyEvaluationRequest, + opts?: DaemonRequestOptions, +): Promise { + const attempt = await attemptDaemonRequest( + { + type: "policyEvaluation", + protocolVersion: PROTOCOL_VERSION, + integration: req.integration, + event: req.event, + payload: req.payload, + cwd: req.cwd, + }, + opts, + ); + if (!attempt.ok) return attempt; + + const message = attempt.message; + const validOptionalString = (value: unknown) => + value === null || value === undefined || typeof value === "string"; + if ( + message.type === "policyResult" && + (message.decision === "allow" || message.decision === "deny" || message.decision === "instruct") && + isStringArray(message.policyNames) && + validOptionalString(message.reason) && + isStringArray(message.matchedPolicies) && + typeof message.durationMs === "number" && + Number.isInteger(message.durationMs) && + message.durationMs >= 0 && + validOptionalString(message.toolName) + ) { + return { ok: true }; + } + + return { ok: false, failure: "unreachable" }; +} diff --git a/src/hooks/daemon-service.ts b/src/hooks/daemon-service.ts index e447f2ad5..013fee660 100644 --- a/src/hooks/daemon-service.ts +++ b/src/hooks/daemon-service.ts @@ -1070,6 +1070,36 @@ export async function probeDaemonEndToEnd(): Promise { return (await probeDaemon()).ok; } +/** + * Proves that the daemon supports the structured request native plugins use. + * + * This deliberately follows the ordinary end-to-end hook probe. Besides + * retaining its startup-race retry, that makes the failure meaningful: an old + * daemon can be fully healthy for shell hooks while still lacking the + * `policyEvaluation` variant added for Hermes. Protocol v1 predates that + * variant, so version equality alone cannot establish the capability. + */ +export async function probeDaemonPolicyEvaluation(): Promise { + if (!(await probeDaemonEndToEnd())) return false; + try { + const { attemptDaemonPolicyEvaluation } = await import("./daemon-client"); + const attempt = await attemptDaemonPolicyEvaluation( + { + integration: "hermes", + event: "on_session_start", + payload: { + hook_event_name: "on_session_start", + source: "failproofai-native-policy-health-probe", + }, + }, + { responseTimeoutMs: DAEMON_PROBE_TIMEOUT_MS }, + ); + return attempt.ok; + } catch { + return false; + } +} + /** * Waits for the service to report running and to HOLD it — a `Type=simple` unit * is active the moment it forks, so one optimistic reading passes a daemon that diff --git a/src/hooks/integrations.ts b/src/hooks/integrations.ts index e6cbb1311..62e134871 100644 --- a/src/hooks/integrations.ts +++ b/src/hooks/integrations.ts @@ -124,8 +124,9 @@ export interface Integration { displayName: string; /** * This integration has no in-process/CLI fallback and cannot evaluate policy - * unless failproofaid answers an end-to-end probe. Installation must stop - * before writing any hook or plugin when the daemon is unavailable. + * unless failproofaid answers the structured native-policy probe. Installation + * must stop before writing any hook or plugin when the daemon is unavailable + * or predates the `policyEvaluation` capability. */ requiresHealthyDaemon?: boolean; /** Settings scopes this integration supports (e.g. claude: user/project/local; codex: user/project). */ diff --git a/src/hooks/manager.ts b/src/hooks/manager.ts index 151b3dfac..09286eca9 100644 --- a/src/hooks/manager.ts +++ b/src/hooks/manager.ts @@ -29,7 +29,7 @@ import { CORE_SOURCE, addPack, setPackPolicyEnabled } from "./pack-store"; import type { ResolvedPack } from "./pack-manifest"; import { hasInstalledPacks, readInstalledPacks } from "./pack-manifest"; import { packPolicyParamKey } from "./policy-evaluator"; -import { probeDaemonEndToEnd } from "./daemon-service"; +import { probeDaemonPolicyEvaluation } from "./daemon-service"; import { chip, note, @@ -477,12 +477,12 @@ async function installHooksImpl( const daemonRequiredBy = selectedIntegrations .map(({ integration }) => integration) .filter((integration) => integration.requiresHealthyDaemon); - if (daemonRequiredBy.length > 0 && !(await probeDaemonEndToEnd())) { + if (daemonRequiredBy.length > 0 && !(await probeDaemonPolicyEvaluation())) { const names = daemonRequiredBy.map((integration) => integration.displayName).join(", "); const verb = daemonRequiredBy.length === 1 ? "requires" : "require"; throw new CliError( - `${names} ${verb} a healthy failproofaid daemon before FailproofAI enforcement can be enabled.\n` + - "Run `failproofai config` to install and configure the daemon, then retry.", + `${names} ${verb} a compatible failproofaid daemon with native policy evaluation before FailproofAI enforcement can be enabled.\n` + + "Run `failproofai config` to install or update the daemon, then retry.", ); } From 68a2b539b85da6a7aece3352b0e85f594853015d Mon Sep 17 00:00:00 2001 From: chhhee10 Date: Wed, 16 Sep 2026 15:25:04 +0530 Subject: [PATCH 6/6] chore(release): prepare 1.0.6 --- CHANGELOG.md | 18 ++++++++++++++++++ Cargo.lock | 6 +++--- Cargo.toml | 2 +- package.json | 2 +- 4 files changed, 23 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 82ebee6b9..023bb293e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,23 @@ # Changelog +## 1.0.6 — 2026-09-16 + +Stable native Hermes policy-enforcement release, validated across CLI and +Telegram sessions, default and named profiles, instruction delivery, hard +denials, daemon-backed evaluation, and end-to-end event ingestion. + +### Added + +- Add a native Hermes plugin backed by the existing `failproofaid` warm worker. Policies now intercept Hermes tool calls before execution, deliver model-visible `instruct()` guidance with a bounded retry ledger, and preserve hard blocking for `deny()` decisions. +- Add structured `policyEvaluation` / `policyResult` daemon messages for native integrations, including canonical tool, policy, match, reason, and latency metadata. + +### Fixes + +- Install, remove, and report health for every active Hermes home, including the default profile, upstream nested profiles, and config-bearing `~/.hermes-` installations. +- Require an exact native-policy capability check before changing any Hermes plugin or profile configuration. Missing, unhealthy, outdated, or protocol-incompatible daemons are rejected with setup instructions before fail-closed enforcement can be enabled. +- Apply Hermes evaluation timeouts as one total socket deadline and keep advisory instruction retries bounded so an unavailable state store or repeatedly unchanged action cannot deadlock a turn. +- Make npm release verification tolerate registry propagation delay while still requiring the root package and all platform daemon packages to publish at the same version. + ## 1.0.6-beta.1 — 2026-09-15 ### Fixes diff --git a/Cargo.lock b/Cargo.lock index caff20c25..b1526a922 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -202,7 +202,7 @@ dependencies = [ [[package]] name = "failproofaid" -version = "1.0.6-beta.1" +version = "1.0.6" dependencies = [ "fpai-collect", "fpai-ipc", @@ -265,7 +265,7 @@ dependencies = [ [[package]] name = "fpai-collect" -version = "1.0.6-beta.1" +version = "1.0.6" dependencies = [ "notify", "reqwest", @@ -280,7 +280,7 @@ dependencies = [ [[package]] name = "fpai-ipc" -version = "1.0.6-beta.1" +version = "1.0.6" dependencies = [ "libc", "proptest", diff --git a/Cargo.toml b/Cargo.toml index 65f46f261..3720fc6a4 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -3,7 +3,7 @@ resolver = "3" members = ["crates/*"] [workspace.package] -version = "1.0.6-beta.1" +version = "1.0.6" edition = "2024" license-file = "LICENSE" repository = "https://github.com/FailproofAI/failproofai" diff --git a/package.json b/package.json index 23e3ec70b..87c0ff39f 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "failproofai", - "version": "1.0.6-beta.1", + "version": "1.0.6", "description": "Observability and enforcement for AI agent harnesses. 39 built-in policies hooked into 12 of them — Claude Code, Codex, Cursor, Hermes, OpenClaw and more — blocking the tool call before it runs. Local dashboard included, no account needed.", "bin": { "failproofai": "./dist/cli.mjs",