Skip to content

fix(crdt): materialize synced deltas into sparse document store (closes #146)#147

Merged
farhan-syah merged 1 commit into
mainfrom
fix/146-crdt-sync-materialization
Jul 2, 2026
Merged

fix(crdt): materialize synced deltas into sparse document store (closes #146)#147
farhan-syah merged 1 commit into
mainfrom
fix/146-crdt-sync-materialization

Conversation

@farhan-syah

Copy link
Copy Markdown
Member

Summary

Fixes the Origin-side root cause of #146: a lite→Origin synced CRDT (Loro) delta was acked and merged into CRDT state (and FTS-indexed via the separate FtsIndex frame) but was never written into the sparse DOCUMENTS btree that DocumentScan / ShapeSnapshot read. As a result a synced collection was catalog-visible but not shape-servable — a plain SELECT id and the ShapeSnapshot both returned 0 rows (doc_count=0).

Change

In execute_crdt_apply (Data Plane), after a delta applies cleanly (both the sync and non-sync arms), the merged Loro row is materialized into the sparse document store:

  • encode_crdt_row — reads the merged row, converts LoroValue → JSON → msgpack → canonicalize_document_for_storage, matching the exact bytes the native schemaless put path stores. Runs while the engine borrow is live and returns owned bytes so the borrow drops before touching self.sparse.
  • materialize_synced_document — writes to the sparse store keyed by surrogate_to_doc_id(surrogate) (identical to the native path); bitemporal collections use versioned_put, others put. FTS is intentionally not re-indexed here (delivered via the separate FtsIndex frame). Write failures are logged and swallowed so a materialization miss never wedges the sync stream.

Idempotent under replay: non-bitemporal put overwrites by key, bitemporal versioned_put appends one version per applied delta, and duplicate deltas are rejected by sync_admit before reaching apply.

Verification

  • 104 crdt/document unit tests green; clippy + fmt clean.
  • Cross-repo lite→Origin E2E (in nodedb-lite): a collection created only on lite, synced with no Origin pre-create, is now returned by pg_class, text_match, and a plain SELECT id (the DocumentScan/ShapeSnapshot path).

Follow-ups (out of scope)

The synced-doc apply path does not yet emit CDC WriteEvents, update column stats, or maintain secondary/vector/spatial indexes the way the native write path does.

CRDT sync applies were merging deltas into the Loro engine but never
writing the resulting document into the sparse document store, so
DocumentScan and ShapeSnapshot could not observe synced writes. Read
the merged row back on a Clean apply, encode it into the canonical
schemaless storage bytes, and put it into the sparse store under the
surrogate-derived key (versioned for bitemporal collections). A
materialization failure is logged and swallowed rather than failing
the apply, since the sync stream must not wedge on it.
@farhan-syah farhan-syah merged commit 34bf32d into main Jul 2, 2026
0 of 2 checks passed
@farhan-syah farhan-syah deleted the fix/146-crdt-sync-materialization branch July 2, 2026 05:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant