Skip to content

Use GRDB for chat.db SQL and schema modeling#73

Open
pmanot wants to merge 33 commits intomainfrom
purav/sql-improvements
Open

Use GRDB for chat.db SQL and schema modeling#73
pmanot wants to merge 33 commits intomainfrom
purav/sql-improvements

Conversation

@pmanot
Copy link
Copy Markdown
Contributor

@pmanot pmanot commented May 6, 2026

This consolidates Messages chat.db access behind GRDB. It models Apple's tables/columns in Swift, keeps runtime guards for OS-specific optional columns, and replaces the custom SQLite statement/cache layer with DatabaseQueue, FetchableRecord, and shared row decoding. The update watcher also does more targeted message/reaction/read-state refreshes, and the hot mapped-message paths batch/dedupe lookups with targeted indexes where available.

How it works:

  • Adds GRDB 6 and opens chat.db through a read-only DatabaseQueue.
  • Represents tables as TableRecords and columns as ColumnExpression enums.
  • Uses TableSchema checks before touching Apple columns that vary across OS versions.
  • Moves row models to GRDB FetchableRecord/DatabaseValueConvertible and central row helpers.

Validation:

  • swift build --target IMDatabase
  • swift test --filter MessageMapperFixtureTests
  • swift test --filter HashingTests
  • yarn test:js src/swift-json.test.ts --runInBand
  • swift test builds, then the local SwiftPM test runner exits with signal 11 after EmojiSPITests.swift:31 reports .nilResponse(method: "initWithLocale:").

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 6, 2026

Caution

Review failed

Failed to post review comments

📝 Walkthrough

Summary by CodeRabbit

  • New Features

    • Message-level update tracking and cursor snapshots for finer sync.
    • New state_sync events for message/reaction upserts, updates, and deletes.
    • Reply-to tracking and account-scoped event subscriptions; per-account initialization.
  • Bug Fixes

    • Deterministic ordering for reaction queries.
  • Chores

    • Migrated to a schema-driven DB access layer and GRDB-backed reads.
    • Chat state keys simplified to string GUIDs.
  • Tests

    • Added comprehensive live SQL and Swift JSON unit tests.

Walkthrough

Migrates IMDatabase from SQLite to GRDB with a schema/type-safe layer; converts many DB paths to GRDB Row APIs; shifts update detection to a message-centric pipeline and GUID-keyed unread state; enriches associated-message/reaction modeling; adds Swift JSON revival utilities, tests, and supporting packaging/utilities.

Changes

Schema, GRDB Migration & Packaging

Layer / File(s) Summary
Schema Definition
src/IMessage/Sources/IMDatabase/Schema/IMDatabaseSchema.swift, src/IMessage/Sources/IMDatabase/Schema/IMDatabaseTables.swift
Adds IMDatabaseColumn/IMDatabaseTable, TableSchema, IMDatabaseSchema and per-table enums (MessageTable, ChatTable, HandleTable, AttachmentTable, join tables, SQLiteSequenceTable).
Package & Dependency
Package.swift, Package.resolved
Introduces GRDB.swift as a package dependency and updates IMDatabase target to depend on GRDB; adds an IMDatabase test target referencing GRDB.
Schema Cache & Index Wiring
src/IMessage/Sources/IMDatabase/Database/IMDatabase.swift
Adds private schemaCache, replaces string-based message index mapping with typed MessageTable.Column tuples, and reworks index creation to inspect TableSchema and use sqlName.
PRAGMA / table info & helpers
src/IMessage/Sources/IMDatabase/Database/IMDatabase+MappedShared.swift
Implements PRAGMA table_info using Row.fetchAll, caches tableColumns, and adds sqlArguments(_:) helper.

Row-based Access & Mapped Rows

Layer / File(s) Summary
GRDB Read Patterns
src/IMessage/Sources/IMDatabase/Database/*
Replaces import SQLite with import GRDB and converts many prepared-statement flows to read { db in Row.fetchAll(...) } across Accounts, Chats, Messages, Attachments, Search, MappedThreads, MappedMessages, MappedShared.
Row Accessors
src/IMessage/Sources/IMDatabase/Support/Column+.swift
Adds Row accessors: optionalString(at:), optionalInt(at:), optionalData(at:), requiredString(at:), requiredInt(at:), imCoreDate(at:), and looseBool(at:).
Mapped Row Protocols & Decoders
src/IMessage/Sources/IMDatabase/Models/MappedDatabaseRows.swift, src/IMessage/Sources/IMDatabase/Models/MappedDatabaseRows+DictionaryBridges.swift
Introduces MappedDatabaseRow protocol with init(row:columns:), adds MappedRowColumnIndexes.index(for:), migrates mapped-row initializers to Row-based decoding, and adds replyToGUID to MappedMessageRow and its serialized object.
GUID DB Convertible
src/IMessage/Sources/IMDatabase/Models/GUID.swift
Switches GUID<Tag> conformance from SQLiteBindable to GRDB's DatabaseValueConvertible for DB interop.
Initializers Updated
multiple src/IMessage/Sources/IMDatabase/Database/*
Various model initializers changed from init(row: borrowing Row) to init(row: Row) and code adapted to GRDB Row APIs (Messages, Attachments, etc.).

Mapped SELECT Builders, Batching & Determinism

Layer / File(s) Summary
Selection SQL Builders
src/IMessage/Sources/IMDatabase/Database/IMDatabase+MappedMessages.swift, src/IMessage/Sources/IMDatabase/Database/IMDatabase+MappedThreads.swift
Replaces ad-hoc column lists with schema-driven builders messageSelectionSQL(messageSchema:) and chatSelectionSQL(chatSchema:); adapts mapped queries to use schema selections.
Message Batching
src/IMessage/Sources/IMDatabase/Database/IMDatabase+MappedMessages.swift
Adds mappedMessageRows(guids:) and mappedMessageRows(rowIDs:) with recursive batching (constant maxMappedMessageRowsBatchSize = 500) and schema-aware dateEdited selection.
Deterministic Reactions
src/IMessage/Sources/IMDatabase/Database/IMDatabase+MappedMessages.swift
Adds ORDER BY m.ROWID ASC to reaction queries for deterministic ordering.

Message-Centric Updates & Event Pipeline

Layer / File(s) Summary
Data Shape & Query API
src/IMessage/Sources/IMDatabase/Database/IMDatabase+Updates.swift
Introduces UpdatedMessageChange and UpdatedMessagesQueryResult; replaces prior chat-centric updates API with package-scoped messages(newerThanRowID:orReadSince:orEditedSince:) and updatedMessagesSinceQuery(dateEditedExpression:).
Cursor Snapshot
src/IMessage/Sources/IMDatabase/Database/IMDatabase+MappedMessages.swift
Adds messageUpdateCursorSnapshot() to compute lastRowID, lastDateRead, and lastDateEdited (schema-aware).
Event Collection & Batching
src/IMessage/Sources/IMessage/EventWatcher/EventWatcher+Updates.swift
Replaces thread-refresh flow with message-centric pipeline: collectMessageUpdateEvents()messageUpdateEvents(for:); introduces PendingMessageKind, PendingMessage, ThreadBatch, and MessageUpdateKind; builds per-thread upsert/update/delete and reaction batches and emits state_sync events.
Hashing & Mapping Wiring
src/IMessage/Sources/IMessage/Hashing/PlatformAPI+Hashing.swift, src/IMessage/Sources/IMessage/PlatformAPI.swift
Adds mapAndHashMessagesByRowID returning [Int: [PlatformSDK.Message]], hashReaction helper; latestThreadMessagesByChatGUID now indexes by message rowID using the new mapping helper.
ServerEvent Shape
src/IMessage/Sources/PlatformSDK/ServerEvent.swift
Adds new state_sync server events (upsert/update/delete messages and reactions, thread stateSync, deleteThreads) and updates jsonObject to emit state_sync payloads; marks old thread refresh as deprecated.

Watcher State, Subscription & Unreads

Layer / File(s) Summary
State Shape & Initialization
src/IMessage/Sources/IMessage/EventWatcher/EventWatcher.swift
chatStates now keyed by String GUIDs; EventWatcher stores currentUserID and accountID; initializer updated to accept accountID.
Lifecycle & Subscription
src/IMessage/Sources/IMessage/EventWatcher/EventWatcherLifecycle.swift
Introduces Subscription wrapper and revised State; subscribeToEvents now accepts accountID; start/watch signatures include lastDateEdited and route callbacks/reporting through Subscription.
Unread Diffing
src/IMessage/Sources/IMessage/EventWatcher/EventWatcher+Unreads.swift, src/IMessage/Sources/IMDatabase/Database/IMDatabase+Unreads.swift
diffChatStates() and chatStates() now operate with GUID-string keys; unread SQL projection reordered to chat_guid, unread_count, last_read_message_timestamp; chatStates() returns [String: ChatState].
ChatRef Removal & Tests
src/IMessage/Sources/IMDatabase/Support/ChatRef.swift, src/IMessage/Sources/IMessage/EventWatcher/ChatRef+Description.swift, src/IMessage/Sources/IMDatabaseTestBench/TestBench.swift
Removes ChatRef type and its description/Hashable extension; TestBench and EventWatcher logic updated to use GUID-keyed chat state maps.

Associated-message & Reaction Model

Layer / File(s) Summary
Type Definitions
src/IMessage/Sources/IMessage/Mappers/MessageMapperTypes.swift
Adds AssociatedMessageTarget, ReactionAction, AssociatedReactionKey, AssociatedReaction, AssociatedMessageType and replaces numeric-to-string mapping with typed associatedMessageTypes: [Int: AssociatedMessageType].
Mapping & Helpers
src/IMessage/Sources/IMessage/Mappers/MessageMapper+Associated.swift
Refactors associated-message parsing to parseAssociatedMessageTarget, pattern-matches on AssociatedMessageType (heading, sticker, reaction), updates assignReactions, introduces reactionStickerAssetURLString, and adds row-based overloads for mapMessageReaction.

Client-side Swift JSON Revival & Tests

Layer / File(s) Summary
Revival Utilities
src/swift-json.ts
Expands SWIFT_DATE_FIELDS, adds isMutableRecord, reviveSwiftDateFields, reviveSwiftEventEntry, and reviveSwiftMessageAPIValue<T> to mutate Swift-bridge payloads (convert numeric timestamps to Date, revive seen maps, etc.).
Tests
src/swift-json.test.ts
Adds unit tests validating parsing, revival of nested structures, and in-place mutation behavior of reviveSwiftMessageAPIValue.
Event Forwarding
src/api.ts
subscribeToEvents now applies reviveSwiftMessageAPIValue to non-TOAST events before forwarding to consumers.

Utilities, Tests & Misc

Layer / File(s) Summary
Array Chunking
src/IMessage/Sources/IMessageCore/Array+Chunks.swift
Adds package-scoped chunks(ofCount:) helper for partitioning arrays into ArraySlice chunks.
Live SQL Tests
src/IMessage/Sources/IMDatabaseTests/LiveSQLTests.swift
Adds a comprehensive live SQL test suite and LocalMessagesDatabase helpers to validate schema and many DB query paths against a local chat.db fixture.
Misc
.gitignore
Adds .swiftpm ignore entry.

Sequence Diagram(s)

sequenceDiagram
    participant DB as IMDatabase
    participant EW as EventWatcher
    participant EU as EventWatcher+Updates
    participant PA as PlatformAPI
    participant PS as PlatformSDK

    EW->>DB: messageUpdateCursorSnapshot()
    DB-->>EW: lastRowID, lastDateRead, lastDateEdited
    EW->>DB: messages(newerThanRowID:orReadSince:orEditedSince:)
    DB-->>EW: UpdatedMessagesQueryResult
    EW->>EU: messageUpdateEvents(for: result)
    EU->>DB: mappedMessageRows / mappedReactionRows / mappedAttachmentRows
    DB-->>EU: msgRows, reactionRows, attachRows
    EU->>PA: mapAndHashMessagesByRowID(msgRows, attachRows, reactionRows, currentUserID, accountID)
    PA-->>EU: [rowID: [PlatformSDK.Message]]
    EU->>EW: [ServerEvent] (upsert/update/delete + reactions)
    EW->>PS: deliver events to connected clients
Loading
sequenceDiagram
    participant Client as Client
    participant API as PlatformAPI
    participant WL as EventWatcherLifecycle
    participant EW as EventWatcher
    participant DB as IMDatabase

    Client->>API: subscribeToEvents(onEvent, accountID)
    API->>WL: subscribeToEvents(onEvent, accountID)
    WL->>EW: attach Subscription(accountID,onEvent,...)
    Client->>API: startEventWatchingFromCurrentState()
    API->>DB: messageUpdateCursorSnapshot()
    DB-->>API: cursor (lastRowID,lastDateRead,lastDateEdited)
    API->>WL: startEventWatchingFromCurrentState(lastRowID,lastDateRead,lastDateEdited)
    WL->>EW: startWatching(subscription,lastRowID,lastDateRead,lastDateEdited)
    EW->>EW: collectMessageUpdateEvents()
    EW-->>Client: push ServerEvent via onEvent
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch purav/sql-improvements

@pmanot pmanot changed the title Consolidate chat.db SQL and message state syncs Use GRDB for chat.db SQL and schema modeling May 6, 2026
@pmanot pmanot force-pushed the purav/sql-improvements branch from fe0c7dd to c0591b6 Compare May 6, 2026 16:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants