Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,5 @@ A protocol is not a result. A passing repository check is not independent valida
Commit only reviewed, credential-free summaries and bounded evidence receipts. Keep large JSONL, logs, Sessions, candidate workspaces, caches, credentials, and private settings outside Git under a stable archive identity.

No formal Benchmark result is published by [`Decision 0001`](../decisions/0001-documentation-and-evidence-governance.md). Protocols, templates, and individual results require separate review.

New dated reports start from [`TEMPLATE.md`](TEMPLATE.md). Add each governed report to this index and retain every metadata field: the repository check proves structure, reachability, and link integrity only—not the truth or independent validation of a result.
25 changes: 25 additions & 0 deletions docs/benchmarks/TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
---
status: draft
created: YYYY-MM-DD
last-verified: YYYY-MM-DD
applies-to: OpenPI revision or release
related-issues: "#NNN"
related-prs: none
supersedes: none
source-revision: commit SHA
model: provider/model and immutable version when available
thinking-level: exact setting
task-set: stable task identity
verifier: stable verifier identity
sample-size: exact count
isolation: workspace and scheduling boundary
usage-accounting: receipt or measurement method
failure-classification: explicit taxonomy and counts
limitations: known gaps
evidence-reference: retrievable archive identity and receipt
rerun-entry-point: command or runbook
---

# Benchmark title

Record protocol, results, interpretation, and limitations without embedding private or unbounded raw evidence.
1 change: 1 addition & 0 deletions docs/disciplines.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ reachability, not adoption or evidence-validation state.
| OP-10 | Lint warnings fail the validation round | enforced | yes | `bun run lint` |
| OP-11 | TypeScript is checked without emitting files | enforced | yes | `bun run typecheck` |
| OP-12 | Runtime provenance is verified before diagnosis | manual | no | `bun run provenance` |
| OP-13 | Governed research and Benchmark records are well-formed, indexed, and link-valid | enforced | yes | `bun run check:knowledge-contract` |

`manual` rows are intentional: they document a contributor action that cannot
be proved by a repository-only check without changing the installed Pi state.
2 changes: 2 additions & 0 deletions docs/research/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

Research records preserve sourced investigation and distinguish observations, inferences, recommendations, and unknowns. They are not accepted Decisions or proof of runtime behavior by themselves.

New governed records start from [`TEMPLATE.md`](TEMPLATE.md), retain the required metadata, and link back to their source Issue. Add every governed record to this index; records without frontmatter remain legacy until a scoped migration.

## Legacy records

The following records predate [`Decision 0001`](../decisions/0001-documentation-and-evidence-governance.md). They remain useful historical sources but have not been migrated to the new metadata contract as part of this change:
Expand Down
27 changes: 27 additions & 0 deletions docs/research/TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
---
status: draft
created: YYYY-MM-DD
last-verified: YYYY-MM-DD
applies-to: revision, release, or source boundary
related-issues: "#NNN"
related-prs: none
supersedes: none
---

# Research title

## Verified facts

State sourced observations and their verification boundary.

## Inferences

Separate interpretations from observations.

## Recommendations

Record proposed action without presenting it as an adopted Decision.

## Unknowns

List unresolved questions and evidence that would answer them.
3 changes: 2 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -86,9 +86,10 @@
},
"scripts": {
"prepublishOnly": "bun run check && bun run test",
"check": "bun run check:config-contract && bun run check:discipline && bun run check:web && bun run format:check && bun run lint && bun run typecheck",
"check": "bun run check:config-contract && bun run check:discipline && bun run check:knowledge-contract && bun run check:web && bun run format:check && bun run lint && bun run typecheck",
"check:config-contract": "node scripts/check-config-contract.mjs",
"check:discipline": "node scripts/check-discipline-ledger.mjs",
"check:knowledge-contract": "node scripts/check-knowledge-contract.mjs",
"prepare": "node scripts/prepare-effect-tsgo.mjs",
"format": "biome format --write .",
"format:check": "biome format .",
Expand Down
194 changes: 194 additions & 0 deletions scripts/check-knowledge-contract.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,194 @@
import { existsSync, readdirSync, readFileSync, realpathSync } from "node:fs";
import { dirname, relative, resolve, sep } from "node:path";
import { fileURLToPath, pathToFileURL } from "node:url";

const REPOSITORY_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), "..");
const RECORD_METADATA = [
"status",
"created",
"last-verified",
"applies-to",
"related-issues",
"related-prs",
"supersedes",
];
const BENCHMARK_METADATA = [
"source-revision",
"model",
"thinking-level",
"task-set",
"verifier",
"sample-size",
"isolation",
"usage-accounting",
"failure-classification",
"limitations",
"evidence-reference",
"rerun-entry-point",
];
const RESEARCH_SECTIONS = [
"verified facts",
"inferences",
"recommendations",
"unknowns",
];
const RECORD_STATUSES = new Set(["draft", "validated", "superseded"]);
const DATE_PATTERN = /^\d{4}-\d{2}-\d{2}$/;
const MARKDOWN_LINK_PATTERN =
/!?\[[^\]]*\]\(([^)\s]+)(?:\s+["'][^"']*["'])?\)/g;

function markdownFiles(directory) {
if (!existsSync(directory)) return [];
return readdirSync(directory, { withFileTypes: true }).flatMap((entry) => {
const path = resolve(directory, entry.name);
if (entry.isDirectory()) return markdownFiles(path);
return entry.isFile() && entry.name.endsWith(".md") ? [path] : [];
});
}

export function parseRecordFrontmatter(source) {
const lines = source.split(/\r?\n/);
if (lines[0] !== "---") return undefined;
const end = lines.indexOf("---", 1);
if (end < 0) return undefined;
const metadata = new Map();
for (const line of lines.slice(1, end)) {
const match = /^([a-z][a-z0-9-]*):\s*(.*?)\s*$/.exec(line);
if (!match) continue;
metadata.set(match[1], match[2].replace(/^(?:"(.*)"|'(.*)')$/, "$1$2"));
}
return metadata;
}

function isTemplateOrIndex(path) {
return ["README.md", "TEMPLATE.md"].includes(path.split(sep).at(-1));
}

function relativeRecordPath(root, path) {
return relative(root, path).split(sep).join("/");
}

function validateMetadata({ category, metadata, record, problems }) {
for (const key of RECORD_METADATA) {
if (!metadata.get(key)?.trim()) problems.push(`${record}: missing ${key}`);
}
const status = metadata.get("status");
if (status && !RECORD_STATUSES.has(status)) {
problems.push(`${record}: unsupported status ${status}`);
}
for (const key of ["created", "last-verified"]) {
const value = metadata.get(key);
if (value && !DATE_PATTERN.test(value)) {
problems.push(`${record}: ${key} must use YYYY-MM-DD`);
}
}
if (category === "benchmarks") {
for (const key of BENCHMARK_METADATA) {
if (!metadata.get(key)?.trim())
problems.push(`${record}: missing ${key}`);
}
}
}

function validateResearchSections({ source, record, problems }) {
const headings = new Set(
source
.split(/\r?\n/)
.map((line) => /^##\s+(.+?)\s*$/.exec(line)?.[1].toLowerCase())
.filter(Boolean),
);
for (const section of RESEARCH_SECTIONS) {
if (!headings.has(section))
problems.push(`${record}: missing section ${section}`);
}
}

function validateLinks({ root, path, source, problems }) {
for (const match of source.matchAll(MARKDOWN_LINK_PATTERN)) {
const target = match[1];
if (/^(?:[a-z]+:|#|\/)/i.test(target)) continue;
const decoded = decodeURIComponent(target.split(/[?#]/, 1)[0]);
const resolved = resolve(dirname(path), decoded);
const withinRoot =
resolved === root || resolved.startsWith(`${root}${sep}`);
if (!withinRoot || !existsSync(resolved)) {
problems.push(
`${relativeRecordPath(root, path)}: broken repository link ${target}`,
);
}
}
}

export function checkKnowledgeContract(root = REPOSITORY_ROOT) {
const canonicalRoot = realpathSync(root);
const problems = [];
const records = [];

for (const category of ["research", "benchmarks"]) {
const directory = resolve(canonicalRoot, "docs", category);
const indexPath = resolve(directory, "README.md");
const indexSource = existsSync(indexPath)
? readFileSync(indexPath, "utf8")
: "";
if (!indexSource)
problems.push(`docs/${category}/README.md: missing category index`);

for (const path of markdownFiles(directory)) {
if (isTemplateOrIndex(path)) continue;
const source = readFileSync(path, "utf8");
const metadata = parseRecordFrontmatter(source);
// Decision 0001 is forward-only. A record without frontmatter is legacy
// until a scoped review explicitly migrates it into this contract.
if (!metadata) continue;

const record = relativeRecordPath(canonicalRoot, path);
records.push(record);
validateMetadata({ category, metadata, record, problems });
if (category === "research") {
validateResearchSections({ source, record, problems });
}

const indexTarget = relative(directory, path).split(sep).join("/");
if (!indexSource.includes(`](${indexTarget})`)) {
problems.push(
`${record}: not reachable from docs/${category}/README.md`,
);
}
validateLinks({ root: canonicalRoot, path, source, problems });
}

if (indexSource) {
validateLinks({
root: canonicalRoot,
path: indexPath,
source: indexSource,
problems,
});
}
}

return { records: records.sort(), problems };
}

export function assertKnowledgeContract(root = REPOSITORY_ROOT) {
const result = checkKnowledgeContract(root);
if (result.problems.length > 0) {
throw new Error(
[
"Knowledge contract check failed:",
...result.problems.map((problem) => `- ${problem}`),
].join("\n"),
);
}
return result;
}

if (
process.argv[1] &&
pathToFileURL(resolve(process.argv[1])).href === import.meta.url
) {
const result = assertKnowledgeContract();
process.stdout.write(
`✓ knowledge contract (${result.records.length} governed records)\n`,
);
}
Loading
Loading