[loveRhythm1990]

The existing self-run benchmark keeps source hashes, metrics and private answer caches, but developers cannot follow one question from the prepared Add batches through ordered Search evidence, the actual answer context, and the local judge result. Add an opt-in local diagnostic export using that execution path and the existing evaluator/cache/checkpoint machinery.

Refs matrixorigin/memoria-website#236. The implementation lives in mb; issue tracking remains in memoria-website.

Result

• --content-diagnostics --diagnostic-source <declaration.json> creates private per-question HTML, shared Add pages, a versioned manifest, and atomic gzip JSON snapshots. Retrieval-only mode explicitly marks Answer/Eval as not run. Model output, selected/excluded ranks, judge parsing/failure state, usage and retry metadata come from actual execution; no explanation or hidden reasoning is invented.
• The declaration binds independently obtained synthetic/public input to its SHA256, source/version, license/local-use basis, permitted uses and 1–30-day retention. Official/unknown sources and mismatched bytes are rejected. These are operator-reviewed declarations, not automatic proof of licensing or data provenance.
• Defaults limit the new diagnostic directory to 100 questions and 64 MiB, including readable pages. Size/selection omissions are disclosed without changing the benchmark/scoring denominator. Existing datasets, output metadata and answer caches are outside this cap.
• Owner-only files, display redaction of known credentials/auth patterns and endpoint URLs, escaped script-free HTML, and a metadata-only CI artifact allowlist protect the new content export. Independent privacy review of input remains required; the redactor is not a general personal-data detector.
• Resume preserves the expiry and rejects missing/incompatible manifests. Pending model work uses the prior Search/cache; cached evidence is hash-checked, missing snapshots are disclosed, and signature-verified answer-cache reuse is labeled. A failed private export cannot discard the final measurement checkpoint.
• A paired HTML/JSON comparison checks dataset/history/Top K/model/evaluator compatibility and keeps missing/unscored coverage visible.

Reproduce without services or model calls

From agent_memory_leaderboard/:

uv run --locked --extra evaluation python -m scripts.content_diagnostics_demo   --output-dir var/content-demo

The printed report path is local. The demo uses independently authored synthetic data, a test byte tokenizer and simulated adapter/model responses, prominently labeled as a tool demonstration rather than a Memoria quality result. Additional scenarios exercise missing evidence, budget exclusion, answer failure and judge failure. Actual-service CLI usage, source review, size/retention boundaries and comparison commands are in benchmarks/EVALUATION.md.

Validation and scope

• Full Python suite: 462 passed, including 33 new diagnostic cases.
• Ruff passed for the existing CI scope and all scripts; whitespace check passed.
• Ran the offline full-chain demo and a baseline/missing-evidence paired comparison; verified scores/signals and all local HTML links, escaping and absence of script dependencies. The small final synthetic demonstration directory was about 20 KiB; this is not a large-dataset size estimate.
• No real-provider quality baseline, QA workload or official evaluation was run for this change.

Capture is disabled by default and implemented in the self-run client only. The deployed Add/Search request handling, Memoria storage and SQLite journal are unchanged; official Smoke/Full requests do not enable this export. No service image, deployment or remote cluster change is included. Keep the tracking issue open pending review and acceptance of the tool.