feat: v0.11.0 — collaborative state server + CRDT sync stack#30
Open
cuttlefisch wants to merge 93 commits into
Open
feat: v0.11.0 — collaborative state server + CRDT sync stack#30cuttlefisch wants to merge 93 commits into
cuttlefisch wants to merge 93 commits into
Conversation
- Node struct gains `properties: HashMap<String, String>` for arbitrary org property drawer key-value pairs (foundation for activity tracking) - org parser captures ALL properties, not just :ID: - scan_heading_id → scan_heading_properties for heading-level drawers - update_property() utility for rewriting single properties in-place - SQLite schema v4→v5 migration: properties_json column - 6 new tests (parse, round-trip, update_property, migration) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add kb_write_guard: HashSet<PathBuf> to suppress watcher events for paths MAE is currently writing (activity tracking, chain-fill) - drain_kb_watchers() skips Upserted events for guarded paths - file_ops save path guards the path before sync reimport, preventing the duplicate sync+async reimport on every save - KbWatcherStats gains events_suppressed + reimports_total counters - Add kb_dailies_dir + kb_daily_chain_gap_max fields (for Part 4) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New `crates/kb/src/activity.rs`: activity scoring with configurable decay weights, date arithmetic (no chrono), body hash for change detection - `KnowledgeBase::search_sorted_by_activity()` re-sorts results by score - `KnowledgeBase::get_mut()` for in-place property updates - Activity recording: kb_record_access, kb_record_modification, kb_record_link with write-guard protection - Property rewriting via `update_property()` + guarded disk writes - 4 new options: kb_activity_tracking, kb_activity_decay, kb_dailies_dir, kb_daily_chain_gap_max - 10 new tests (date parsing, scoring, hashing, day arithmetic) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Daily note creation with chain-back linking (org-roam-dailies parity): - kb_create_daily_stub, kb_daily_chain_fill (backward chain-linking) - kb_goto_daily_today/yesterday/date, kb_daily_prev/next navigation - show_kb_audit_report with health, watcher stats, instance summary - Commands: daily-goto-today, daily-goto-yesterday, daily-goto-date, daily-prev, daily-next, kb-audit - MiniDialogContext::DailyGotoDate for date input - lookup_key_binding() and query_keybindings() on Editor Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…e (Part 5) - modules/dailies/ with module.toml + autoloads.scm - SPC n d t/y/d/p/n keybindings for daily navigation - kb-delete moved from SPC n d to SPC n D - concept:dailies seed node with chain-fill docs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…n cleanup, metrics (Part 6) - Add `detect_stale_nodes()` and `remove_stale_nodes()` to KnowledgeBase (nodes whose source_file no longer exists on disk) - Add `validate_links()` for on-save broken link detection (advisory warning) - Add `kb-cleanup-orphans` command (SPC n C) — removes orphan user notes while preserving seed nodes (cmd:, concept:, lesson:, scheme:, option: prefixes) - Split KbWatcherStats `events_skipped` into `suppressed_debounce` and `suppressed_timebox` for granular watcher metrics - Add cumulative `drain_us_sum`, `drain_count`, `reimports_total` to watcher stats - Show stale nodes section + watcher metrics in `:kb-health` report - Update introspect AI tool with new watcher stat fields - 4 new tests: stale detection, link validation, orphan cleanup, seed preservation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create keymap-doom module (modules/keymap-doom/) with all SPC leader bindings mirrored from kernel — prepares for flavor extraction in follow-up PRs - Register `keymap_flavor` option (default: "doom") in OptionRegistry - Add docs/KEYBINDINGS.md — full keybinding reference organized by kernel, doom flavor, and module overlays - Add server-client architecture line item to ROADMAP.md near-term section Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… version - Add 5 dailies bindings (SPC n d t/y/d/p/n) to kernel keymaps.rs so they work without the dailies module being declared in (mae!) - Mirror dailies bindings in keymap-doom module (intentional redundancy matching existing pattern for all other SPC bindings) - Implement (set-group-name MAP PREFIX LABEL) Scheme API: - New pending_group_names field in SharedState - Registration in SchemeRuntime::new() - Drain in apply_to_editor alongside keymap bindings - Used by dailies module to set "+dailies" group label - Add version + modules sections to introspect MCP tool output (agents can now verify build version and loaded module state) - Tests: dailies_bindings_registered, spc_sub_prefixes_have_group_names, set_group_name_works, runtime_define_key_updates_keymap Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Which-key popup UX overhaul: - Scrolling: C-j/C-k, C-n/C-p, Up/Down scroll by 1; C-d/C-u by 5. Directional indicators show count above/below visible entries. - Height config: `which-key-max-height-pct` option (10-90%, default 40) replaces hardcoded 2/3 fraction. All 5 which-key options now have config_key for :set-save persistence. - Sort order: `which-key-sort-order` option (key/desc/none). Groups always sort first. - Shared layout: format_keypress() and which_key_column_layout() moved to text_utils.rs — both TUI and GUI renderers use the same function, eliminating height-vs-render mismatch bug. - Group labels: 14 set-group-name definitions in keymap-doom (+ai, +buffer, +code, +eval, +file, +help, +peek, +notes, +open, +project, +quit, +select, +toggle, +window). - Dailies cleanup: removed 5 duplicate define-key calls (bindings already in keymap-doom); module now only provides set-group-name. - Editor helpers: set_which_key_prefix() / clear_which_key_prefix() ensure scroll resets on prefix change. - Fix: UTF-8 safe truncation in AI grading (floor_char_boundary). - 7 new unit tests for format_keypress and which_key_column_layout. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Multi-client MCP server:
- Per-client tokio tasks with ClientSession (session.rs)
- Content-Length framing with line-based fallback for backward compat
- EditorEvent broadcast with per-client bounded channels (broadcast.rs)
- $/ping heartbeat, notifications/subscribe, 5s write timeout
- Initialize handshake extracts clientInfo, reports multiClient capability
File safety (layered defense):
- SHA-256 content-hash verification on buffer load/save/reload
- Advisory file locks (.{name}.mae.lock) with stale PID cleanup
- check_disk_changed_by_hash() catches mtime failures (NFS, sub-second)
KB hardening:
- WAL mode (concurrent readers), busy_timeout 5s, synchronous NORMAL
- New test: wal_mode_enabled
Shared code extraction:
- centered_popup_dims() + truncate_to_width() in text_utils (principle 8)
- TUI and GUI popup renderers deduplicated
File tree focus fix:
- Tree window receives focus on open (was staying on editing window)
- New option: file_tree_focus_on_open (default true, :set-save persistable)
Documentation:
- 4 ADRs in docs/adr/ (protocol, text sync, file safety, KB scaling)
- CLAUDE.md principle 10 (multi-client safety) + server-client section
- ROADMAP KB visibility/scoping/tangle items
- Makefile docs-tangle + docs-tangle-check targets
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two integration tests exercising the full server over Unix sockets: - multi_client_concurrent_connections: two clients connect, initialize, tools/list, ping, tool call, then one disconnects while other continues - multi_client_subscribe_events: subscribe + shutdown lifecycle Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Editor save_buffer_with_hash_check: blocks on mismatch, passes clean - Editor save_buffer_force: overwrites despite mismatch - Editor file lock lifecycle: acquire/release/release_all/contention - MCP $/resync handler: validates response fields - EventBroadcaster sequence monotonicity: validates AtomicU64 increment - CI: widen KB test filter to include changelog+sync tests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements the collaborative sync layer using yrs (Yjs Rust port, YATA algorithm) as specified in ADR-002, ADR-005, and ADR-006. Provides: - TextSync: YText <-> Rope bridge for collaborative text editing - KbNodeDoc: CRDT schema for knowledge base nodes (YMap) - Encoding utilities for state vectors and updates - Stress convergence test (5 clients, 200 random ops each) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Integrates mae-sync into mae-core so that buffers with sync enabled generate yrs updates on every text mutation for broadcast to peers: - Add `sync_doc: Option<TextSync>` + `pending_sync_updates` to Buffer - Hook all 10 mutation methods (insert_char, delete_char_backward/forward, delete_line, delete_word_backward, delete_to_line_start/end, insert_text_at, delete_range, open_line_below/above) - Undo/redo rebuild sync state from rope (correctness over efficiency) - replace_contents/replace_rope recreate sync_doc preserving client_id - Buffer::enable_sync/disable_sync/apply_sync_update public API - MCP: sync/enable, sync/state_vector, sync/update protocol methods - EventBroadcaster: SyncUpdate event variant for peer notification - 6 new unit tests (all passing, 0 regressions in 2000+ test suite) Zero-cost when sync disabled: all hooks are Option::map on None. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wire up executor handlers for sync/enable, sync/state_vector, sync/update, and sync/full_state so MCP clients can enable CRDT sync on buffers, exchange state vectors, and apply remote updates. Pull-based model (clients poll). - New sync_exec.rs dispatcher (4 handlers) in mae-ai executor chain - Buffer lookup by name or index, idempotent enable, base64 transport - sync/full_state added to MCP protocol method match - mae-sync dependency added to mae-ai - 7 unit tests including two-client roundtrip convergence Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wire the EventBroadcaster into the MCP server so subscribed clients receive push notifications when buffers change. This closes the gap where local edits queued yrs updates but nothing delivered them. Key changes: - McpServer gains SharedBroadcaster (Arc<Mutex<EventBroadcaster>>) - handle_client restructured with tokio::select! for simultaneous request handling and event pushing - notifications/subscribe now updates broadcaster filter - write_notification helper sends JSON-RPC notifications (no id field) - New sync_broadcast::drain_and_broadcast() drains pending yrs updates from buffers and broadcasts to subscribed clients - Drain points: McpToolRequest (immediate), IdleTick/frame_timer (~100ms) New tests: - 2 broadcast unit: sync_update delivery + subscription filtering - 4 drain unit: noop, clears pending, multiple buffers, skips non-sync - 5 E2E integration: push after subscribe, no push before subscribe, two clients (one subscribed), survives disconnect, backpressure Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Address 6 findings from the Phase A-D architecture review:
Finding 2 (HIGH — Bug): reload_from_disk() now calls sync_rebuild_from_rope()
to keep yrs doc in sync with rope after file reload.
Finding 3 (MEDIUM): replace_rope() and replace_contents() now use
sync_rebuild_from_rope() instead of inline doc recreation, which
also queues the update for broadcast.
Finding 4 (MEDIUM): remove_buffer_raw() logs a warning when dropping
pending sync updates on buffer close.
Finding 10 (MEDIUM): Push notifications now include per-client `seq`
number in params for event ordering. Notification format changed
from `params: event` to `params: { seq, event }`.
Finding 11 (HIGH — Design): sync_rebuild_from_rope() documented as
creating a fresh Doc (discarding CRDT history). TODO added for
incremental diff approach in Phase E.
Finding 1 (MEDIUM): session.subscriptions documented as informational
copy; broadcaster is the authoritative source for delivery.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Make read_message, write_framed, handle_request pub and generic over AsyncBufRead/AsyncWrite (no longer Unix-socket-only) - Add TCP loopback tests (initialize, write_framed, coexist) - Fix sync/MCP broadcast architecture (audit fixes from Phase D) - Wire TextSync collaborative edits into Buffer properly Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New crate: mae-state-server (TCP binary for multi-client CRDT sync) - CLI: start/doctor/--check-config/--version with manual argv parsing - Config: TOML-based with XDG defaults, validation, CLI overrides - Storage: SQLite WAL-first persistence (StorageBackend trait + SqliteBackend) - DocStore: per-document locking (RwLock<HashMap> + per-doc Mutex) - Handler: JSON-RPC dispatch for sync/update, sync/diff, sync/full_state, sync/state_vector, docs/list, docs/content - Compaction: configurable threshold, snapshot + WAL trim, compact-all on shutdown - Recovery: replay snapshot + WAL tail on startup Integration: - Workspace member + Cargo.lock - Dockerfile: dep cache + runtime binary - Makefile: build/install/docker-network-test targets - CI: state-server tests + check-config in server-client job - Release: builds + ships mae-state-server-linux-x86_64 - Systemd unit, shell completions (bash/zsh/fish) - Docker compose for network E2E tests - ADR-005 (KB CRDT) + ADR-006 (collaborative state engine) - CLAUDE.md + ROADMAP.md updated 5 E2E tests (gated on MAE_STATE_SERVER env), 10 unit tests, all passing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Merge origin/main (v0.10.1, CI fixes, badges workflow) - Resolve CODE_MAP.md conflicts (keep dailies commands) - Update dashboard screenshot with current TUI splash Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove --lib flag from state-server test step (binary-only crate) - Regenerate CODE_MAP with state-server + new pub MCP APIs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ervability Phase 1 — Scalability: - SQLite sharded connection pool (FNV-1a, 4 shards default) in state-server - CRDT-safe undo via reconcile_to() (LCS diff, preserves vector clocks) - DocAddress type (file:/kb:/shared: namespaces) for document addressing - Event sequence tracking (wal_seq on SyncUpdate for gap detection) - Save protocol (SHA-256 content-hash verification) - Background compaction + idle eviction in state-server Phase 2 — UX: - CollabStatus enum + 3 editor fields (status, synced_docs, intent) - 5 collab options (server_address, auto_connect, auto_share, etc.) - 7 commands (collab-start/connect/disconnect/status/share/sync/doctor) - SPC C keybinding group in doom keymap - Status bar segment (priority 4, shows connection state + peer count) - Splash screen quick action (SPC C c) - init.scm section 6 "Collaborative Editing" Phase 2 — AI Tools: - 4 AI tools: collab_status, collab_connect, collab_share, collab_doctor - Tool definitions with parameter schemas and permission tiers - Intent-based dispatch (core sets intent, binary drains each tick) Phase 3 — Observability: - mae doctor: collab section (binary, env, config checks) - audit_configuration: collaboration JSON section - State-server: sync/resync, docs/stats, docs/save_intent, docs/save_committed, $/debug handler methods 30 files changed, 1,130 insertions, 3 new files. 3,336 tests passing, 0 clippy errors. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Phase 3 (observability): - introspect tool: new "collaboration" section with status/server/synced docs - Scheme API: (collab-status) alist, (collab-synced-buffers) list - $/debug method: uptime_secs, connection_count, version fields - handler.rs: server_start_time threading for uptime tracking Phase 5 (documentation): - KB concept nodes: collab-architecture, collab-workflows - KB lesson node: collab-setup (install → configure → connect → share) - docs/COLLABORATION.md: standalone guide (architecture, quickstart, config, debugging) Phase 6 (tests): - E2E network tests: tcp_docs_stats, tcp_save_intent_ok, tcp_resync_protocol, tcp_debug_endpoint - Sync tests: reconcile_to_empty, reconcile_from_empty - Core dispatch tests: 4 collab dispatch unit tests - Status bar format_collab_status tests All 3,324+ tests pass, 0 clippy warnings. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- CLAUDE.md: state-server section expanded with new methods, commands, AI tools, Scheme API, options; API stability counts updated - ADR-006: implementation notes for SQLite pool, reconcile_to, event sequence tracking, save protocol, background compaction, editor UX - CODE_MAP: regenerated (20 crates, 169 public items, 496 commands) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Phase 4 of the collaborative editing plan: - Schema migration v6→v7: adds `crdt_doc BLOB` column to nodes table - Node.crdt_doc field: optional encoded yrs document bytes - Node.to_crdt_doc(): creates KbNodeDoc from text fields or restores from bytes - Node.apply_crdt_doc(): updates title/body/tags from CRDT content - save_to_sqlite/load_from_sqlite/sync_to_sqlite: persist crdt_doc column - Backward-compatible: older DBs auto-migrate, crdt_doc defaults to NULL - mae-kb now depends on mae-sync for KbNodeDoc access Tests: migration v6→v7, CRDT roundtrip via save/load, Node↔KbNodeDoc conversion, apply_crdt_doc field updates (132 KB tests pass). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Document variable naming rules (editor, buf, win, idx), function naming patterns (dispatch_*, handle_*, execute_*), and test helper semantics (editor_with_text vs editor_with_bulk_text vs editor_with_rust). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The collab E2E Docker test is temporarily disabled while the editor struct extraction refactoring is in progress. Will re-enable once the refactor stabilizes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test `compute_cursor_command_mode` does `Editor::default()` then reassigns fields — Clippy flags this but struct-init syntax is impractical with 90+ fields that keep changing during struct extraction. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move all knowledge-base fields into `editor::kb_state::KbContext`:
- Primary KnowledgeBase → `editor.kb.primary`
- Federation (registry, instances, watchers) → `editor.kb.{registry,instances,...}`
- Config (watcher_enabled, search_max_results, etc.) → `editor.kb.*`
- Capture state → `editor.kb.capture_state`
- AI context (visited IDs, write guard) → `editor.kb.{ai_visited_ids,write_guard}`
Editor field count: ~92 → ~71 (21 fields extracted).
Follows same pattern as ViState, AiState, CollabState, ShellIntents.
20 files updated across mae-core, mae-ai, mae-scheme, mae (binary).
All 3,673 tests pass. Clippy clean (workspace + GUI).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
# Conflicts: # Cargo.lock
Move debug_state and pending_dap_intents into `editor::dap_state::DapContext`: - `editor.debug_state` → `editor.dap.state` - `editor.pending_dap_intents` → `editor.dap.pending_intents` Editor field count: ~71 → ~69 (2 fields extracted). Follows same pattern as ViState, AiState, CollabState, ShellIntents, KbContext. 18 files updated across mae-core, mae-ai, mae-gui, mae-renderer, mae (binary). All 3,673 tests pass. Clippy clean (workspace + GUI). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
KbContext (21) and DapContext (2) join the existing 4 sub-structs. Remaining candidate: LspContext (7 fields). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ns, reconnect backoff WU1 — Per-buffer collab status indicators (E8): - Add `collab_is_sharer` field to Buffer (default false) - `format_collab_status` shows role + pending updates: [C:3|sharer], [C:3|synced], [C:3|pending:2], [C:OFFLINE|pending:5] - Set `collab_is_sharer = true` on BufferShared event WU2 — Fix `:w` for non-sharer clients (Bug 2): - Guard `SaveCollab` intent behind `collab_is_sharer` - Joiners save locally only, no `save_intent` broadcast WU3 — Sharer quit notification (Bug 3): - Add `SharerLeft` variant to EditorEvent + CollabEvent - Track `sharer_session_id` on DocEntry in state server - On sharer disconnect, broadcast SharerLeft to peers - Client-side: parse notification, display status message WU4 — Reconnect lifecycle hardening (Bug 4): - Exponential backoff: base * factor^min(attempt, 5), capped at 300s - Max reconnect attempts (0 = infinite, configurable) - ForceSync debounce: skip duplicate requests within 2s per doc 15 new tests (3,688 total, 0 failures). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…DMAP - Bugs 2-4 (save guard, sharer notifications, disconnect lifecycle) fixed in 8de53b8 - E8 (buffer status indicators) complete in 8de53b8 - Bug 1 clarified: reconcile_to() already uses single-txn LCS diff (not full-buffer replacement). Large undos produce proportionally large but correct diffs. Full fix deferred to Phase F (yrs UndoManager). - State server v2 entry updated with completed items - Added Known Limitations section to COLLABORATION.md Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…smoke - Add babel, export, snippets, format, make, lookup, spell crate stubs to the dependency cache layer (Cargo.toml copies + dummy lib.rs) - Remove `(run-tests)` from test_smoke.scm — old pattern incompatible with Rust-side iteration model (causes double-execution) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Per-user undo for collaborative editing — User A's `u` no longer undoes User B's edits. Implementation: **mae-sync (TextSync)**: - `enable_undo()` creates UndoManager scoped to text field with origin-tracked transactions (capture_timeout_millis: 0) - `insert()`/`delete()` use `transact_mut_with(client_id)` when undo active - `undo()`/`redo()` return (success, Vec<update_bytes>) for broadcast - `undo_reset()` for explicit group boundaries, `clear_undo()`, `can_undo/redo()` - `observe_update_v1` captures undo/redo-generated deltas - yrs `sync` feature enabled for Send+Sync on callback types **mae-core (Buffer)**: - `undo()`/`redo()` delegate to CRDT path when sync_doc has active UndoManager - `enable_sync()`/`load_sync_state()` call `sync.enable_undo()` - `end_undo_group()` calls `sync.undo_reset()` for CRDT group boundaries - Updates from CRDT undo/redo pushed to `pending_sync_updates` for broadcast **Tests**: 9 new mae-sync tests (undo_single_insert, redo_after_undo, undo_produces_update_bytes, undo_remote_excluded, undo_group_boundary, two_clients_independent_undo, can_undo_empty, undo_clear, undo_delete_restores) + 6 new mae-core tests + Docker E2E undo test (test_undo.scm) **ROADMAP**: Bug 1 marked complete, 3 refinement items added (cursor positioning, capture_timeout tuning, undo stack size limit) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace single-client test_undo.scm with a proper two-client pair: - test_undo_sharer.scm: Client A shares, inserts "from-A", undoes it after B edits, verifies B's "from-B" is preserved, then redoes - test_undo_joiner.scm: Client B joins, inserts "from-B", verifies A's undo only removed A's text, then B undoes its own edit independently This validates the core per-user undo guarantee: User A's `u` never undoes User B's edits. Docker compose: added undo-sharer + undo-joiner services with /sync volume coordination. Verifier waits for all 4 clients. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Dockerfile now has all 20 crates, per-user CRDT undo is wired, and two-client undo E2E tests are in place. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…timeout - Remove --exclude mae-gui from all CI workflow steps and Makefile ci target - Install GUI deps (clang, libfontconfig, libfreetype) in main matrix and e2e jobs - Skip dep install for fmt step (doesn't compile) - Slim gui job to release binary build only (unit tests/clippy now in main matrix) - Bump collab-e2e timeout to 15 minutes (two-client undo tests need more time) - Local `make ci` now matches remote CI: full --workspace, no exclusions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Steel's `load` function doesn't resolve in the Docker test runner context. Inline connection checks (sleep-ms + collab-status) matching the pattern used by the working test_share.scm/test_join.scm. Also: mount workspace-undo-a/b volumes in verifier container and add undo file verification checks. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix redundant `&` in format! arg (popup_render.rs:1326) - Add info-level structured logging to collab bridge: share, join, response handling, remote updates, buffer content on join - Add info-level logging to state-server: doc request dispatch, sync/share accept, sync/resync state return, sync/update - Upgrade sync_broadcast forwarding log from debug to info These logs will pinpoint the Docker E2E data flow issue. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… CRDT edits never forwarded The test runner's process_side_effects and drain_events_for never called drain_and_broadcast(), so pending_sync_updates accumulated in buffers but were never forwarded to the collab state server via collab_command_tx. In the GUI/TUI event loops this runs on every IdleTick (~100ms). The test runner now mirrors this: a SharedBroadcaster (no-op for MCP) is created at test session start, and drain_and_broadcast is called after every test step and during sleep-ms polling loops. Also fix test scripts to create files before open-file (Buffer::from_file returns Err on non-existent paths, silently leaving [scratch] active). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Named Docker volumes mount as root by default. The runtime container runs as user mae (UID 1000), so write-file to /workspace failed with EPERM. This caused open-file to find no file, leaving [scratch] as the active buffer, which then got shared as "shared:[scratch]" instead of the intended document. Pre-create /workspace and /shared in the Dockerfile with correct ownership, matching the existing pattern for /sync. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…supplied When a client joins with a bare filename (e.g., "test.txt"), the server suffix-matches it to the full doc_id (e.g., "file:no-project/test.txt"). The server returns the resolved name in the response's "doc" field, but the client was ignoring it and using the original request doc_id. This caused the buffer's collab_doc_id to be set to the unresolved name, so subsequent RemoteSyncUpdate broadcasts (which use the server's canonical doc_id) couldn't find the buffer — producing "remote update for unknown buffer — name mismatch?" warnings and lost edits. Also updates shared_docs tracking to replace unresolved names with resolved ones, and enables MAE_LOG=info on all Docker E2E clients for full collab pipeline visibility. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…, diagnostics Three fixes for Docker collab E2E reliability: 1. **read_message partial-peek framing** (mae-mcp): The Content-Length detection required `fill_buf()` to return ≥15 bytes. With small TCP segments or tiny BufReader buffers, the initial peek could return fewer bytes, causing fallthrough to line-based framing and corrupting the message parse. Now uses prefix matching — even a 1-byte peek of 'C' correctly routes to Content-Length parsing. Added 2 regression tests. 2. **Intent drain ordering** (test_runner): drain_collab_intents now runs BEFORE the sleep loop in process_side_effects. The pending_intent single-slot was getting overwritten by GapDetected→ForceSync events during collab event processing in the sleep loop, before the ShareBuffer intent from the test step could be sent. 3. **Diagnostic logging**: Bridge logs write_framed success/failure for share requests. Server handler logs every incoming message at debug level. State-server uses debug log level in Docker E2E tests. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds structured debug logging across the collab pipeline to close
visibility gaps in Docker E2E investigation:
- **Server handler**: debug log for every incoming message (preview),
every broadcast event sent to client, with session ID correlation
- **Bridge task**: debug log for incoming server messages, command
dispatch, and response matching (success/failure/unknown id)
- **Healthcheck**: changed from `echo '{}' | nc` (which created full
server sessions every 3s, flooding clients with PeerJoined/PeerLeft
events) to `nc -z` (TCP connect check only, no session creation)
- **RUST_LOG** for state-server (was incorrectly MAE_LOG)
- **MAE_LOG** filter: `mae::collab_bridge=debug,info` for client debug
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…orchestration Three root-cause fixes for Docker collab E2E test failures: 1. **Cross-client crosstalk** (collab_bridge): Server broadcasts sync_updates to ALL connected clients, not just doc subscribers. Added client-side `shared_docs` filter — bridge ignores updates for docs the client never shared or joined. Prevents buffer focus stealing and false GapDetected events from unrelated docs. 2. **ForceSync destroying undo history** (collab_bridge): GapDetected → ForceSync → BufferJoined used `load_sync_state()` which creates a brand-new TextSync with empty UndoManager. Now uses `apply_sync_update()` (yrs merge) for existing synced buffers, preserving undo/redo stacks. Falls back to full load on merge failure. BufferJoined no longer steals focus for existing buffers. 3. **Docker orchestration** (Makefile): Replaced `--abort-on-container-exit` with `docker compose wait verifier`. The old approach killed slow containers before the verifier's `depends_on: service_completed_successfully` conditions were met. Now all containers exit naturally; verifier starts only after all 4 test containers succeed. Additional fixes: - Integration test `fault_server_drop_mid_session` deadlock: add timeouts to `read_message` calls that blocked forever on duplex stream - Test runner diagnostic logging: active buffer state on failure, event counts during drains - Per-user CRDT undo/redo unit tests at sync and buffer layers - Comprehensive E2E test design documentation (README.md) Docker E2E results: 66 test assertions + 9 verifier checks, all green. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ion cleanup Phase 1 — Fix CRDT Scheme test timing bug: - Replace flag-based two-step drain pattern with always-accumulate approach - `capture_pending_sync_updates()` now always captures updates before `drain_and_broadcast` consumes them (no flag check needed) - `buffer-drain-updates` is now a simple take-and-return from accumulator - Collapse 5 test files from two-step drain to single-step - Remove redundant `pending updates exist` test (covered by drain check) - Result: 141 CRDT tests pass (was 16 failures), 260 editor tests pass Phase 3 — Docker orchestration cleanup: - Fix stale `--abort-on-container-exit` comment in compose file - Fix stale `docker compose wait` command in collab-e2e README - Makefile: robust `docker wait` with verifier container polling Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… collab_heartbeat_interval option - Remove `--exclude mae-test-fixtures` from all 4 cargo commands in Dockerfile (crate is not in workspace, produces warning on every Docker build) - Wire `collab_heartbeat_interval` option through CollabState → CollabSpawn → run_collab_task (resolves TODO at collab_bridge.rs:864) - Add `heartbeat_interval_secs` to config.rs CollaborationSection for config.toml loading - Register get/set handlers in option_ops.rs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…infra - SYNC_PROTOCOL.md Section 7: replace 5 "Deferred" items with current status (4 completed in v0.11.0, 1 still deferred: awareness protocol) - RoamNotes: add Scheme test infrastructure section explaining buffer-drain-updates always-accumulate pattern and test runner cycle - RoamNotes: add local CRDT test catalog (8 files, 141 tests) - RoamNotes: update orchestration to match actual Makefile (docker wait) - RoamNotes: freshness updates to collaborative_state_engine.org and mae_state_server.org noting E2E validation in CI Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add E2E coverage gate entry under server-client architecture section. Every networked feature (sync, save, awareness, auth) requires E2E test coverage before release. Documents current coverage percentages and the methodology proven by the collab E2E suite. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Complete awareness protocol implementation (6 work units):
WU1 — Theme palette: 8-color WCAG AA accessible, colorblind-safe
collaborative cursor palette with dark/light variants. FNV-1a hash
for deterministic color assignment. 17 theme style keys
(ui.collab.cursor.0-7, ui.collab.selection.0-7, ui.collab.label).
WU2 — Protocol wiring: sync/awareness JSON-RPC relay through state
server with echo filtering (broadcast_except). AwarenessState schema,
AwarenessMap with 30s stale timeout, CollabState integration,
EditorEvent::AwarenessUpdate for MCP subscribers. 50ms throttle.
No persistence (ephemeral — no WAL, no SQLite).
WU3 — User identity: auto-derive collab_user_name from git config →
$USER → hostname → "anonymous". Logged at info level on first connect.
WU4 — Renderer integration:
GUI: 2px colored bar cursor + username label (3s auto-hide) +
20% opacity selection fills + ▲/▼ off-screen indicators
TUI: underline + user initial + selection background +
▲/▼ off-screen indicators at viewport edges
Status bar: [C:3|Alice Bob Carol] when awareness active
WU5 — Tests: 12 new awareness tests across 3 tiers:
- 5 sync unit tests (serialize roundtrip, map CRUD, stale cleanup)
- 5 bridge integration tests (roundtrip, echo filter, no persist,
schema validation, color determinism)
- 2 state server E2E tests (relay to peers, not in WAL)
All 3,731 workspace tests pass.
WU6 — Documentation: SYNC_PROTOCOL.md §6.6 awareness spec,
ROADMAP.md updated (awareness + Phase F marked complete).
Bug fix: bridge was sending state_json as a quoted string instead of
nested JSON object — now parsed to serde_json::Value before embedding.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The collab E2E job was failing intermittently with "verifier container never started" because `docker compose ps -q` only shows running containers. The verifier could start and exit between 5s poll intervals, causing the loop to never see it. Replace the fragile poll loop + docker wait with `docker compose wait verifier` (Compose v2.21+), which blocks until the service exits regardless of timing. GitHub ubuntu-latest has Compose v2.27+. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
cuttlefisch
force-pushed
the
feature/v0.11.0-dailies
branch
from
ec13a9f to
ba8b12e
Compare
WU0: Replace `docker compose wait` (races with fast verifier) with foreground `up --build` + `ps -a` exit code inspection. Eliminates the "no containers for project" false failure. WU1-3: Add 13 tests covering protocol gaps: - sync/state_vector, sync/diff, docs/delete, docs/metadata - Concurrent save intents (conflict + retry) - Sharer disconnect notifies peers - Compaction reduces WAL entries - Client connect/disconnect stats tracking - Nonexistent doc full_state behavior - Save epoch lifecycle - Invalid CRDT bytes rejection - Concurrent share convergence Coverage: save 0%→~80%, disconnect ~50%→~80%, errors ~40%→~70%, SQLite ~0%→~40%. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
v0.11.0 delivers collaborative editing (CRDT sync via yrs), org-dailies, and Editor struct decomposition — the three major themes of this release cycle.
Collaborative Editing (Phases A–F)
mae-synccrate: yrs CRDT text bridge + KB node schemamae-state-server: standalone collab server with WAL-based SQLite persistenceOrg-Dailies
Editor Struct Decomposition (197 → ~69 fields)
Infrastructure
Stats
Test Plan
make cipasses (fmt + clippy + check + test)cargo clippy --package mae-gui --all-targets -- -D warningscargo test --workspace— 3,673 tests, 0 failuresmake test-scheme-allMAE_TCP_E2E=1 cargo test -p mae --test collab_tcp_e2e -- --ignored🤖 Generated with Claude Code