fix(crdt): materialize synced deltas into sparse document store (closes #146)#147
Merged
Merged
Conversation
CRDT sync applies were merging deltas into the Loro engine but never writing the resulting document into the sparse document store, so DocumentScan and ShapeSnapshot could not observe synced writes. Read the merged row back on a Clean apply, encode it into the canonical schemaless storage bytes, and put it into the sparse store under the surrogate-derived key (versioned for bitemporal collections). A materialization failure is logged and swallowed rather than failing the apply, since the sync stream must not wedge on it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the Origin-side root cause of #146: a lite→Origin synced CRDT (Loro) delta was acked and merged into CRDT state (and FTS-indexed via the separate FtsIndex frame) but was never written into the sparse
DOCUMENTSbtree thatDocumentScan/ShapeSnapshotread. As a result a synced collection was catalog-visible but not shape-servable — a plainSELECT idand the ShapeSnapshot both returned 0 rows (doc_count=0).Change
In
execute_crdt_apply(Data Plane), after a delta applies cleanly (both the sync and non-sync arms), the merged Loro row is materialized into the sparse document store:encode_crdt_row— reads the merged row, convertsLoroValue→ JSON → msgpack →canonicalize_document_for_storage, matching the exact bytes the native schemaless put path stores. Runs while the engine borrow is live and returns owned bytes so the borrow drops before touchingself.sparse.materialize_synced_document— writes to the sparse store keyed bysurrogate_to_doc_id(surrogate)(identical to the native path); bitemporal collections useversioned_put, othersput. FTS is intentionally not re-indexed here (delivered via the separate FtsIndex frame). Write failures are logged and swallowed so a materialization miss never wedges the sync stream.Idempotent under replay: non-bitemporal
putoverwrites by key, bitemporalversioned_putappends one version per applied delta, and duplicate deltas are rejected bysync_admitbefore reaching apply.Verification
pg_class,text_match, and a plainSELECT id(the DocumentScan/ShapeSnapshot path).Follow-ups (out of scope)
The synced-doc apply path does not yet emit CDC WriteEvents, update column stats, or maintain secondary/vector/spatial indexes the way the native write path does.