feat(core): add portable document ingestion contracts

[phernandez]

Why

Hosted document ingestion needs one portable Basic Memory contract for generated Markdown, provenance, deterministic identity, and safe semantic parsing. Without that boundary, Core and Cloud would disagree about what a raw extraction note means, how retries converge, and which fields an enrichment agent may control.

This PR is the Core contract used by basic-memory-cloud#1650, implements the parser-neutral portion of #1006, and deliberately leaves historical revision indexing to #1177.

Specification

Scope

Core must provide strict, provider-neutral contracts for:

• a current generated Markdown note with type: document;
• a compact, queryable provenance note with type: document_ingestion_run;
• deterministic document and run-note identity derived from trusted inputs;
• canonical raw Markdown whose body is searchable but cannot mint observations or relations from arbitrary extracted syntax;
• a bounded structured enrichment output that cannot replace trusted source, extraction, ingestion, checksum, or identity fields.

Behavioral contract

1. source, extraction, ingestion, and document metadata is strict, nested, extra-forbidden, and serializable through either the YAML schema alias or the Python schema_ref field name.
2. Document and run identities are deterministic. Run identity includes source identity/checksum, extractor engine/version/profile/options, pipeline version, and prompt version.
3. Raw Markdown uses bm_parse_semantics: false; Core preserves body content for search while suppressing observation/relation parsing only for an explicit scalar false.
4. Run-note stage, output revision kind, output identity, and output path must agree with the trusted inputs.
5. Agent-provided body, tags, observations, and relations must round-trip through the real Markdown parser without hidden or additional semantics.
6. Enrichment starts from an exact raw canonical checksum and reconstructs the final trusted envelope server-side.
7. Public schema imports must not pull the heavy Markdown stack into lightweight CLI registration.

Non-goals

• selecting or running a PDF parser;
• Cloud queues, Tigris reads/writes, provider version IDs, or deployment policy;
• enabling production agent enrichment;
• historical revision storage, search, graph reconstruction, retention, or legal hold.

Acceptance Criteria

• document and document_ingestion_run have strict versioned Pydantic contracts.
• Nested provenance frontmatter is canonical, queryable, and round-trips through model serialization.
• Document, run, and sidecar paths are derived from stable trusted inputs.
• OCR page diagnostics reject coerced booleans, floats, and strings.
• Raw generated content remains body-searchable while arbitrary PDF syntax cannot create graph semantics.
• Run lifecycle state cannot contradict its raw/current output revision.
• Agent output cannot smuggle extra tags, observations, relations, context, or trusted provenance.
• Exact raw checksum compare-and-swap is required before enrichment assembly.
• Lightweight CLI imports remain lazy.
• The portable contract is exported for the Cloud implementation without adding a parser/runtime dependency.

What Changed

• Added the document, extraction, ingestion-run, revision, and agent-output models in src/basic_memory/schemas/document.py.
• Added deterministic UUID/path helpers, canonical Markdown parsing/assembly, SHA-256 helpers, lifecycle validation, and protected enrichment assembly.
• Exported the portable contracts from src/basic_memory/schemas/__init__.py.
• Added explicit document_ingestion note-object source support.
• Updated EntityParser to honor the raw semantic opt-out safely without dropping body content.
• Added focused schema, Markdown, runtime, and CLI-import regression coverage.

Implementation Details

• UUID5 run identity is recomputed from the source and pipeline fields instead of trusting a supplied UUID.
• Run-note output identity/path is recomputed from the source entity and source path.
• Canonical Markdown reconstruction protects system-owned metadata and validates the fully assembled semantic body, including fence/comment boundary cases.
• Agent tags use a canonical single-tag grammar; structured observations and relations must parse back to exactly the submitted objects.
• Markdown parser imports remain inside the validators that need them, preserving the lightweight CLI boundary.

Testing

Automated

• just fast-check: passed Ruff fix/check, formatting, and ty type checking.
• uv run pytest -q tests/schemas/test_document.py tests/markdown tests/test_runtime.py::TestRuntimeContracts::test_note_object_metadata_parses_safe_values_only: 128 passed; src/basic_memory/schemas/document.py reached 100% statement coverage.
• uv run pytest -q --no-cov tests/cli/test_cli_exit.py::test_bm_cli_import_does_not_load_heavy_stack: 1 passed.
• GitHub CI on 61d7b0dd55313076a081bad823eb69796d4b8426: unit, SQLite integration, Postgres integration, semantic, Milvus, static, CodeQL, DCO, and CLA jobs passed or were skipped as designed.
• git diff --check: passed.

Manual

• No UI or manual runtime verification applies to this parser-neutral contract PR.

Risks / Follow-ups

• Cloud must continue to validate only the trusted shapes it actually produces; additional future enrichment/history shapes can tighten their own invariants when implemented.
• Agent enrichment remains disabled until Cloud can prove exact raw Tigris materialization (basic-memory-cloud#1649).
• Current search and graph behavior remains head-only; the opt-in “BM data lake” is #1177.
• The commit is SSH-signed and DCO-signed-off. GitHub reports unknown_key, so this description does not claim GitHub Verified status.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e5810e4dca

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5a0ad615fe

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cbd903c254

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fca36b555a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[phernandez]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fa57ffe030

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[phernandez]

@codex review

[phernandez]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bc13b3d444

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 61d7b0dd55

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[phernandez]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 61d7b0dd55

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[phernandez]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: 61d7b0dd55

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 87b60bde48

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Suppress graph semantics for raw extractions

When extracted PDF text contains a line such as - [result] ..., a hashtag, or a wiki link, this serialization emits it as ordinary note content with no stage-specific semantic opt-out, and EntityParser consequently indexes it as an observation or relation. The new test_raw_note_uses_normal_graph_semantics confirms this behavior, so arbitrary raw source text can pollute the knowledge graph before it passes through the bounded enrichment fields; raw-stage documents should remain searchable while semantic parsing is disabled.

AGENTS.md reference: AGENTS.md:L357-L362

Useful? React with 👍 / 👎.

[calebjpicker]

Following this contract with interest — the bm_parse_semantics: false decision especially, keeping arbitrary extracted syntax from minting observations. That seems exactly right.

One question before DocumentAgentObservationV1 freezes at v1, because I want to see if you think it's feasible whether this idea can be extended in the manner described here.

#135 has been open since June 2025 asking for this at the note level — section and observation anchors, so a quote can cite its source precisely. Its first example was a [quote] observation linked to a specific figure in a specific section of another document.

That blocker looks smaller now than it did then. Compared to last year, observations now carry their own permalinks — search_service.py reads obs.permalink and de-duplicates on it, and each observation is its own search_index row with its own permalink and category. The addressable identifiers exist now; what's still missing is that relations target entities only.

#1178 is where it becomes load-bearing. DocumentAgentObservationV1 is category, content, tags, context — no locator. So an extracted quote will be traceable to a document, and to an exact stored version of it, which is awesome! But it still doesn't link to a place inside it. For legal and academic work that's the difference between a source and a cite — "Exhibit 12" versus "Exhibit 12 at 4:17–22." The extractor already knows pages: page_count, extracted_page_count, pages_needing_ocr. The information exists at extraction time and is dropped before it reaches an observation. An optional field now looks easier to implement now rather than a v2 migration later, though you'd know better than I would whether that's actually true.

I don't have the standing to say what fits your architecture, so I asked Claude to sketch options merely as suggestions. Flagging that explicitly — these are prompts, not recommendations, and I'd defer to your read on all of them of coures:

1. Convention only. Put the cite in the context parenthetical that already exists — - [quote] "..." (p. 2). No schema change at all; buys a convention rather than anything resolvable.
2. Optional structured locator on DocumentAgentObservationV1 — say an optional page number — rendered deterministically into context so the line still round-trips through the real parser and satisfies require_exact_parser_semantics.
3. Page map on the document note. Store offset→page ranges once at extraction, so any later feature can resolve a span to a page without re-parsing the PDF.
4. The general Section and Observation Anchor Wikilinks Support #135 fix. Let relations target observation permalinks, with a PDF page anchor as one instance of it.

My own thoughts--once parsed, can it store a page number and line number per page? That nested structure could potentially support direct quote architecture?

EDIT (claude helped me understand the constraints better): Related thought — if a span could carry the source checksum alongside its location, a citation becomes verifiable rather than just locatable: you could confirm a quote still matches the document it came from, and detect drift if the source is ever re-extracted. Given how much this contract already checksums, that might be closer to free than it sounds. - Claude

Entirely possible this belongs after v1 ships, or that one of these cuts against something I can't see from outside. Mostly I wanted to raise it while the contract is still open rather than after it isn't.