Add privacy policy and tamper-evident audit logging

CHANGELOG.md

@@ -4,6 +4,43 @@ All notable changes to this project are documented in this file.
 
 ## Unreleased
 
+- Privacy, content policy, and an audit trail:
+  - Every `Logger` now has a `content_policy` (`"off"` by default \| `"hash"`
+    \| `"truncate"` \| `"full"`), governing the content fields an LLM call or
+    tool call can carry (`meta.input_messages`/`output_messages`/
+    `system_instructions`/`tool_arguments`/`tool_result`/`state_diff`).
+    Applied before any plugin or transport ever sees the record, so a seeded
+    secret inside a prompt never reaches a transport unless you opt in.
+    `.child()` inherits the parent's policy.
+  - Field classification (`logquill.privacy.FIELD_CLASSES`: `secret` \|
+    `pii` \| `content`) lets a redaction rule target a class instead of
+    listing keys by name: `RedactPlugin(classes=["secret", "content"])`.
+    `RedactPlugin`'s existing default key set is unchanged, now sourced from
+    the same registry.
+  - `PIIRedactPlugin` gained patterns for well-known API-key/secret shapes —
+    OpenAI/Anthropic keys, AWS access key IDs, GitHub/Slack tokens, JWTs,
+    full PEM private-key blocks, generic `Bearer` tokens — alongside its
+    existing email/SSN/credit-card/phone patterns. Also fixed: overlapping
+    matches between two patterns (e.g. a phone-shaped digit run inside a
+    Slack token) used to redact in a sequence of independent substitutions,
+    which could leave a value only half-redacted; matches are now resolved
+    across all patterns in one pass before anything is replaced.
+  - `TamperEvidentPlugin` gained `.head_hash`/`.sign_head(key)` and the
+    module-level `verify_chain_detailed`/`verify_signed_chain`/`sign_head`/
+    `verify_head_signature`. Hash-chaining alone can't catch a log file
+    being truncated — the remaining chain is perfectly self-consistent, just
+    shorter — so signing the chain's current head and checking it
+    separately closes that gap. `logquill verify <file>` (optionally
+    `--sign-key`/`--signature`) wraps this from the command line, reporting
+    where and why a chain broke.
+  - `AuditLogger` bundles `RedactPlugin`, `PIIRedactPlugin`, and
+    `TamperEvidentPlugin` with content capture off by default — the
+    technical controls an audit trail commonly needs, in one call. Its
+    `.child()` shares the parent's hash chain rather than starting a second,
+    interleaved one. Same scope-honesty note as the rest of the plugin
+    pipeline: none of this is a compliance guarantee on its own.
+  - The record contract gained `meta.tool_arguments`/`meta.tool_result`,
+    and documents which fields the content-capture policy governs.
 - Local-first trace viewer:
   - `logquill trace <run_id> --file logs.jsonl` reconstructs and prints one
     agent run's span tree, annotated with each span's own duration and the

MIGRATING.md

@@ -62,7 +62,9 @@ has them unless you use those.
 | `meta.mcp.server`, `meta.mcp.tool`, `meta.mcp.run_id` | `meta` | string |
 | `meta.tool`, `meta.tool_call_id`, `meta.provider`, `meta.agent_name`, `meta.agent_id`, `meta.response_model` | `meta` | string |
 | `meta.operation` | `meta` | `chat`, `text_completion` or `invoke_agent` |
-| `meta.input_messages`, `meta.output_messages` | `meta` | array (opt-in prompt/completion content) |
+| `meta.input_messages`, `meta.output_messages` | `meta` | array |
+| `meta.system_instructions` | `meta` | array |
+| `meta.tool_arguments`, `meta.tool_result` | `meta` | any JSON value |
 
 `llm` is its own block, not part of `meta`, because cost and latency
 dashboards need stable names for it. A record that isn't an LLM call has no
@@ -85,3 +87,20 @@ and `logquill` on npm both test against.
 - If a transport of yours rebuilds a record from its five 1.x fields, copy the
   record instead (`{**record, "meta": new_meta}`) so `schema_version` and `llm`
   aren't dropped.
+
+## 6. Content capture is off by default
+
+The fields above that can carry a prompt, a completion, or a tool's
+arguments/result — `input_messages`, `output_messages`,
+`system_instructions`, `tool_arguments`, `tool_result`, and `state_diff` —
+are now governed by a per-logger content-capture policy, `Logger(
+content_policy="off" | "hash" | "truncate" | "full")`, **off by default**.
+If you were relying on these fields reaching a transport as given, add
+`content_policy="full"` explicitly. `AuditLogger` (new in this release) also
+defaults to `"off"`.
+
+`RedactPlugin` also gained `classes=` (e.g. `RedactPlugin(classes=
+["secret", "content"])`), redacting by field classification instead of only
+by exact key name — see `logquill.privacy.FIELD_CLASSES`. Existing
+`RedactPlugin()` calls with no arguments are unaffected: the default key set
+is unchanged, just now sourced from that same classification.

README.md

@@ -535,10 +535,25 @@ logger.info("support ticket", notes="reach me at jane@example.com, ssn 123-45-67
 ```
 
 Detection is regex-based by default — fast, dependency-free, matched on shape
-rather than meaning. For fuzzier ML-based detection instead, pass
-`use_presidio=True` (`pip install logquill[presidio]`) to route values
-through Microsoft Presidio's analyzer/anonymizer; Presidio stays a real,
-opt-in dependency, never a default one.
+rather than meaning. Beyond PII, the same scan also catches well-known
+API-key/secret shapes wherever they appear — OpenAI/Anthropic keys, AWS
+access key IDs, GitHub/Slack tokens, JWTs, PEM private-key blocks (the whole
+block, not just the `BEGIN` line), and generic `Bearer` tokens — each a
+provider's own fixed prefix or structure, chosen to keep false positives low.
+For fuzzier ML-based detection instead, pass `use_presidio=True` (`pip
+install logquill[presidio]`) to route values through Microsoft Presidio's
+analyzer/anonymizer; Presidio stays a real, opt-in dependency, never a
+default one.
+
+`RedactPlugin` can also target a **field classification**
+(`logquill.privacy.FIELD_CLASSES`: `secret` \| `pii` \| `content`) instead of
+listing key names one by one:
+
+```python
+from logquill import RedactPlugin
+
+RedactPlugin(keys=(), classes=["secret"])  # exactly today's default key set, by class
+```
 
 ### Tamper-evident logs
 
@@ -559,6 +574,89 @@ records[1]["message"] = "tampered"  # simulate an edited log line
 assert TamperEvidentPlugin.verify_chain(records) is False
 ```
 
+`logquill verify <file>` wraps the same check for a JSONL file on disk,
+reporting *where* and *why* it broke:
+
+```bash
+logquill verify app.log
+# logquill verify: OK — 142 record(s) verified in app.log
+# logquill verify: chain head is 9f1c...e03a
+```
+
+Hash-chaining alone can't catch a file being **truncated** — delete the last
+few lines and what remains is still perfectly self-consistent, just shorter.
+Signing the chain's current head, and keeping that signature somewhere other
+than the log file itself, catches that too:
+
+```python
+from logquill import Logger, TamperEvidentPlugin, verify_signed_chain
+
+tamper = TamperEvidentPlugin()
+logger = Logger("app", plugins=[tamper])
+records = [logger.info(f"step {i}") for i in range(3)]
+
+signature = tamper.sign_head(b"a-key-you-keep-somewhere-else")  # not in this file
+
+assert verify_signed_chain(records, key=b"a-key-you-keep-somewhere-else", signature=signature).ok
+
+truncated = records[:-1]  # an attacker deletes the last line
+result = verify_signed_chain(truncated, key=b"a-key-you-keep-somewhere-else", signature=signature)
+assert result.ok is False  # caught — the chain is consistent but its head moved
+```
+
+```bash
+logquill verify app.log --sign-key <hex key> --signature <hex signature>
+```
+
+### Content capture policy
+
+Prompt/completion text, tool call arguments/results, and a span's captured
+state (`meta.input_messages`, `output_messages`, `system_instructions`,
+`tool_arguments`, `tool_result`, `state_diff` — the `content`-classed fields
+in `logquill.privacy.FIELD_CLASSES`) are governed by a per-logger policy,
+**off by default** — applied before any plugin or transport ever sees the
+value, not something a plugin has to remember to attach:
+
+```python
+from logquill import Logger
+
+logger = Logger("app.agent")  # content_policy="off" by default
+
+record = logger.llm_call("chat", model="m", input_messages=["the user's actual prompt"])
+assert "the user's actual prompt" not in str(record["meta"]["input_messages"])
+```
+
+`"hash"` replaces the value with a stable digest (the same content always
+hashes the same, so two records can be confirmed to share one without either
+ever reaching a transport); `"truncate"` keeps the first 200 characters;
+`"full"` passes it through:
+
+```python
+from logquill import Logger
+
+Logger("app.agent", content_policy="full")  # see it all
+Logger("app.agent", content_policy="hash")  # correlate without exposing it
+```
+
+### `AuditLogger`: a preset for audit trails
+
+`AuditLogger` bundles `RedactPlugin`, `PIIRedactPlugin`, and
+`TamperEvidentPlugin`, with content capture off by default — the technical
+controls an audit trail commonly needs, in one call. (Scope-honesty note,
+same as the rest of the plugin pipeline: this doesn't make a deployment
+"compliant" with anything by itself.)
+
+```python
+from logquill import AuditLogger
+
+audit_log = AuditLogger("audit")  # pass transports=[FileTransport(...)] etc. as usual
+record = audit_log.info("user login", user_id=42, password="hunter2")
+assert record["meta"]["password"] == "***"
+# .child() shares the same hash chain, not a second interleaved one
+
+signature = audit_log.sign_head(b"a-key-you-keep-somewhere-else")
+```
+
 ### Alerting on errors
 
 `AlertingPlugin` is a base class for firing an external alert on ERROR/FATAL

logquill/__init__.py

@@ -1,4 +1,5 @@
 from logquill.adapters.base import LogQuillAdapter
+from logquill.audit import AuditLogger
 from logquill.config import load_config, logger_from_env, logger_from_file
 from logquill.context import bind_context, current_context
 from logquill.exceptions import format_exc_info
@@ -20,8 +21,16 @@
 from logquill.plugins.run_plugin import RunPlugin
 from logquill.plugins.sampling_plugin import SamplingPlugin
 from logquill.plugins.slack_alert_plugin import SlackAlertPlugin
-from logquill.plugins.tamper_evident_plugin import TamperEvidentPlugin
+from logquill.plugins.tamper_evident_plugin import (
+    TamperEvidentPlugin,
+    VerificationResult,
+    sign_head,
+    verify_chain_detailed,
+    verify_head_signature,
+    verify_signed_chain,
+)
 from logquill.plugins.trace_context_plugin import TraceContextPlugin
+from logquill.privacy import CONTENT_FIELDS, FIELD_CLASSES, ContentCapturePolicy, FieldClass
 from logquill.records import SCHEMA_VERSION, LLMBlock, LogRecord, parse_record
 from logquill.serverless import with_azure_function, with_cloud_function, with_lambda
 from logquill.toggle import disable, enable, is_enabled
@@ -60,18 +69,23 @@
     "AppriseAlertPlugin",
     "AppInsightsTransport",
     "AsyncWorker",
+    "AuditLogger",
     "BaseQueueTransport",
     "BaseSQLTransport",
     "BatchingTransport",
     "CloudLoggingTransport",
     "CloudWatchTransport",
     "CollectingTransport",
+    "CONTENT_FIELDS",
     "ConsoleTransport",
+    "ContentCapturePolicy",
     "ContextPlugin",
     "DatadogTransport",
     "DynamoDBTransport",
     "ElasticsearchTransport",
     "EmailAlertPlugin",
+    "FIELD_CLASSES",
+    "FieldClass",
     "FileTransport",
     "Formatter",
     "FunctionPlugin",
@@ -114,6 +128,7 @@
     "TextFormatter",
     "TraceContextPlugin",
     "Transport",
+    "VerificationResult",
     "bind_context",
     "current_context",
     "disable",
@@ -127,6 +142,10 @@
     "parse_level",
     "parse_logfmt",
     "parse_record",
+    "sign_head",
+    "verify_chain_detailed",
+    "verify_head_signature",
+    "verify_signed_chain",
     "with_azure_function",
     "with_cloud_function",
     "with_lambda",

logquill/audit.py

@@ -0,0 +1,109 @@
+"""`AuditLogger` — a `Logger` preset bundling LogQuill's privacy and
+integrity controls for an audit trail: content capture off, secret and PII
+redaction on, and hash-chained records so the result can be checked for
+tampering afterward.
+
+Scope honesty, same as the rest of the plugin pipeline: none of this makes
+a deployment "compliant" with anything on its own — it's a technical
+control that commonly supports compliance work (SOC 2, HIPAA, PCI-DSS audit
+trails all expect *something* like this), never a substitute for the
+process and review compliance actually requires.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+
+from logquill.levels import Level
+from logquill.logger import Logger
+from logquill.plugins.context_plugin import ContextPlugin
+from logquill.plugins.pii_redact_plugin import PIIRedactPlugin
+from logquill.plugins.plugin import MiddlewareFunc, Plugin
+from logquill.plugins.redact_plugin import RedactPlugin
+from logquill.plugins.tamper_evident_plugin import TamperEvidentPlugin
+from logquill.privacy import ContentCapturePolicy
+from logquill.records import LogRecord  # noqa: F401 — resolves MiddlewareFunc's forward reference
+from logquill.transports.transport import Transport
+
+
+class AuditLogger(Logger):
+    """A `Logger` preset for an audit trail. Equivalent to:
+
+        Logger(
+            name,
+            content_policy="off",
+            plugins=[RedactPlugin(), PIIRedactPlugin(), TamperEvidentPlugin(), *extra_plugins],
+        )
+
+    plus `.sign_head(key)` and `.head_hash`, forwarded to the
+    `TamperEvidentPlugin` instance this attaches — see that class, and
+    `logquill.plugins.tamper_evident_plugin.verify_signed_chain`, for what
+    to do with a signature and how `logquill verify` checks one.
+
+    `content_policy` defaults to `"off"` here too — pass `"hash"` if an
+    audit trail should at least be able to confirm two entries shared a
+    prompt without storing it. `extra_plugins` run after the three above,
+    in the order given, the same as any other `Logger`.
+    """
+
+    def __init__(
+        self,
+        name: str,
+        *,
+        level: int | str | Level = Level.INFO,
+        transports: list[Transport] | None = None,
+        content_policy: ContentCapturePolicy | str = "off",
+        extra_plugins: list[Plugin | MiddlewareFunc] | None = None,
+        **kwargs: Any,
+    ) -> None:
+        """`**kwargs` are forwarded to `Logger.__init__` (`async_dispatch`,
+        `flush_at_exit`, etc.)."""
+        self._tamper = TamperEvidentPlugin()
+        plugins: list[Plugin | MiddlewareFunc] = [
+            RedactPlugin(),
+            PIIRedactPlugin(),
+            self._tamper,
+            *(extra_plugins or []),
+        ]
+        super().__init__(
+            name,
+            level=level,
+            transports=transports,
+            plugins=plugins,
+            content_policy=content_policy,
+            **kwargs,
+        )
+
+    def child(self, name: str, /, **fixed_meta: Any) -> AuditLogger:
+        """Like `Logger.child()`, but returns another `AuditLogger` with
+        the *same* redaction plugins and, importantly, the **same**
+        `TamperEvidentPlugin` instance as this one — a child shares this
+        logger's transports already (the same output stream), so it
+        continues this logger's hash chain rather than starting a second,
+        interleaved one that would make the combined file unverifiable as
+        a single chain."""
+        child_logger = AuditLogger(
+            f"{self.name}.{name}",
+            level=self._level,
+            transports=self.transports,
+            content_policy=self.content_policy,
+        )
+        child_logger._tamper = self._tamper
+        child_logger.plugins = [RedactPlugin(), PIIRedactPlugin(), self._tamper]
+        if fixed_meta:
+            child_logger.use(ContextPlugin(**fixed_meta))
+        child_logger._worker = self._worker
+        return child_logger
+
+    @property
+    def head_hash(self) -> str:
+        """This logger's current hash-chain head — see
+        `TamperEvidentPlugin.head_hash`."""
+        return self._tamper.head_hash
+
+    def sign_head(self, key: bytes) -> str:
+        """Signs `self.head_hash` with `key` — see
+        `TamperEvidentPlugin.sign_head`/the module-level `sign_head` for
+        what to do with the result and why it has to be kept somewhere
+        other than this logger's own output."""
+        return self._tamper.sign_head(key)