Skip to content

Conversation

@oguzkocer
Copy link
Contributor

@oguzkocer oguzkocer commented Dec 18, 2025

Summary

Adds ListMetadataRepository to wp_mobile_cache for tracking paginated list state, enabling efficient cache invalidation and stale detection.

Database Schema

┌─────────────────────────────────────┐
│           list_metadata             │
├─────────────────────────────────────┤
│ rowid          INTEGER PK           │
│ db_site_id     INTEGER FK ──────────┼──► db_sites
│ key            TEXT                 │
│ total_pages    INTEGER?             │
│ total_items    INTEGER?             │
│ current_page   INTEGER              │
│ per_page       INTEGER              │
│ version        INTEGER              │
│ last_first_page_fetched_at  TEXT?   │
│ last_fetched_at             TEXT?   │
└─────────────────────────────────────┘
         │
         │ 1:1                     1:n
         ▼                          │
┌─────────────────────────────┐     │
│    list_metadata_state      │     │
├─────────────────────────────┤     │
│ rowid            INTEGER PK │     │
│ list_metadata_id INTEGER FK─┼──┐  │
│ state            INTEGER    │  │  │
│ error_message    TEXT?      │  │  │
│ updated_at       TEXT       │  │  │
└─────────────────────────────┘  │  │
                                 │  │
┌─────────────────────────────┐  │  │
│    list_metadata_items      │  │  │
├─────────────────────────────┤  │  │
│ rowid            INTEGER PK │  │  │
│ list_metadata_id INTEGER FK─┼──┴──┘
│ entity_id        INTEGER    │
│ modified_gmt     TEXT?      │
│ parent           INTEGER?   │
│ menu_order       INTEGER?   │
└─────────────────────────────┘

Type Hierarchy

ListKey (newtype)
    │
    └──► String

ListState (enum, repr(i32))
    ├── Idle = 0
    ├── FetchingFirstPage = 1
    ├── FetchingNextPage = 2
    └── Error = 3

DbListMetadata ◄─────────────────┐
    ├── row_id: RowId            │
    ├── db_site_id: RowId        │  from_row()
    ├── key: String              │
    ├── total_pages: Option      │
    ├── current_page: i64        │
    ├── per_page: i64            │
    └── version: i64             │
                                 │
DbListMetadataItem ◄─────────────┤
    ├── row_id: RowId            │
    ├── list_metadata_id: RowId  │
    ├── entity_id: i64           │
    ├── modified_gmt: Option     │
    ├── parent: Option           │
    └── menu_order: Option       │
                                 │
DbListMetadataState ◄────────────┤
    ├── row_id: RowId            │
    ├── list_metadata_id: RowId  │
    ├── state: ListState         │
    └── error_message: Option    │
                                 │
DbListHeaderWithState ◄──────────┘  (JOIN query result)
    ├── state: ListState
    ├── error_message: Option
    ├── current_page: i64
    ├── total_pages: Option
    ├── total_items: Option
    └── per_page: i64

Key Features

  • Concurrency control: Version numbers detect stale load-more operations after refresh
  • Batch inserts: Items inserted in batches for performance
  • Type safety: ListKey newtype prevents misuse of arbitrary strings; ListState stored as INTEGER with ToSql/FromSql
  • Race condition handling: get_or_create catches constraint violations and retries
  • Atomic upsert: get_or_create_and_increment_version uses INSERT ... ON CONFLICT DO UPDATE ... RETURNING

Repository Operations

ListMetadataRepository (associated functions)
    │
    ├── Read
    │   ├── get_header(site, key)
    │   ├── get_items_by_list_metadata_id / get_items_by_list_key
    │   ├── get_item_count_by_list_metadata_id / get_item_count_by_list_key
    │   ├── get_state_by_list_metadata_id / get_state_by_list_key
    │   ├── get_version_by_list_metadata_id / get_version_by_list_key
    │   └── get_header_with_state(site, key)
    │
    ├── Write
    │   ├── get_or_create(site, key, per_page)
    │   ├── get_or_create_and_increment_version(site, key, per_page)
    │   ├── set_items_by_list_metadata_id / set_items_by_list_key
    │   ├── append_items_by_list_metadata_id / append_items_by_list_key
    │   ├── update_header_by_list_metadata_id / update_header_by_list_key
    │   ├── update_state_by_list_metadata_id / update_state_by_list_key
    │   └── delete_list(list_metadata_id)
    │
    └── Utility
        └── reset_stale_fetching_states()  ─► runs on migration

Error Handling

  • ConstraintViolation - detected from rusqlite::Error for race condition handling
  • PerPageMismatch { expected, actual } - when existing header has different per_page than requested

Design Decisions

  • Repository is primitive-only: No orchestration/concurrency helpers; these belong in a future service layer
  • per_page is required: Must be passed by caller to match networking config; returns error on mismatch rather than silently using existing value
  • Dual method variants: _by_list_metadata_id for callers with cached rowid, _by_list_key for convenience
  • PRAGMA-based tests: Column enum indices verified against actual database schema

Changes

  • Add migration 0007-create-list-metadata-tables.sql
  • Add ListKey, ListState, and DB types in list_metadata.rs
  • Add column enums in db_types/db_list_metadata.rs
  • Add ListMetadataRepository with associated functions
  • Add ConstraintViolation and PerPageMismatch error variants to SqliteDbError
  • Add log crate dependency for structured logging
  • Call reset_stale_fetching_states after migrations complete

Introduces database infrastructure for tracking list metadata, enabling
efficient cache invalidation and stale detection for paginated lists.

Changes:
- Add migration for list metadata tables
- Add DbListMetadata types for database operations
- Add ListMetadata domain types
- Add ListMetadataRepository with full CRUD operations
- Update PostsRepository with stale detection helpers

Extracted from prototype/metadata-collection branch. Original commits:
- 14b01d80 (Implement stale detection by comparing modified_gmt timestamps)
- e47cec89 (Add database foundation for MetadataService)
- 2440a13f (Add list metadata repository concurrency helpers)
- cc0c8a58 (Reset stale fetching states on app launch)
- d64142fb (Split collection observers for data vs state updates)
- fe7435c9 (make fmt-rust)
- 2918b339 (Add parent and menu_order fields to list metadata items)
- 25f88a49 (Rename last_updated_at to last_fetched_at in list metadata)
- 1d709e70 (Simplify ListMetadataReader trait with combined ListInfo query)

Note: Since we use a rebase/squash merge strategy, these commits may show
"does not belong to any branch" on GitHub but remain accessible via URL.
Changes the database storage for `ListState` from TEXT strings to INTEGER
values for better performance and type safety.

Changes:
- Update migration to use `INTEGER NOT NULL DEFAULT 0` for state column
- Add `#[repr(i32)]` to `ListState` enum with explicit discriminant values
- Implement `ToSql` and `FromSql` traits for direct rusqlite integration
- Remove string-based `as_db_str()` and `From<&str>` implementations
- Update all callers to use the enum directly with rusqlite params
The `FromSql` implementation now returns a proper error when encountering
an unknown integer value, rather than silently defaulting to `Idle`.
This makes data corruption issues visible instead of hiding them.
The repository struct has no state, so methods are converted from
instance methods (&self) to associated functions. This removes the
need to instantiate the struct before calling methods.

Before: ListMetadataRepository.get_header(&conn, &site, key)
After:  ListMetadataRepository::get_header(&conn, &site, key)
Replaces individual INSERT statements with a single batch INSERT for
better performance when inserting multiple items. Uses functional style
with try_for_each and flat_map.

Items are chunked to stay under SQLite's variable limit (999).
Replaces two separate queries (get_header + get_state) with a single
JOIN query via get_header_with_state for better efficiency.
Replaces raw `&str` parameters with `&ListKey` throughout the repository
API. This prevents accidental misuse of arbitrary strings as list keys
and makes the API more self-documenting.

The ListKey type provides:
- Type safety at compile time
- Conversion from &str and String via From trait
- as_str() for SQL parameter usage
- Display impl for debug output
- Condense doc comment, remove references to non-existent MetadataService
- Return Result<usize, SqliteDbError> instead of logging internally
- Ignore errors at call site with explanatory comment
Normalizes the schema by replacing (db_site_id, key) in items table
with a foreign key to list_metadata. This:
- Ensures referential integrity
- Enables cascade delete (simplifies delete_list)
- Reduces storage per item
Adds the `log` facade crate to enable proper logging. Debug logs added
to key list metadata operations:
- begin_refresh, begin_fetch_next_page
- complete_sync, complete_sync_with_error
- set_items, append_items, delete_list

Replaces eprintln! with log::warn! for unknown table warnings.
@oguzkocer oguzkocer added this to the 0.2 milestone Dec 18, 2025
Implement ToSql/FromSql traits for ListKey to simplify repository
code by using the type directly in SQL params instead of .as_str().

Also add logging for reset_stale_fetching_states failures instead
of silently ignoring errors.

Changes:
- Add ToSql/FromSql implementations for ListKey
- Replace key.as_str() with key in all SQL params
- Log warning on reset_stale_fetching_states failure
…st_key variants

Allows callers who already have the list_metadata_id to skip an extra
header lookup query.
Split functions that take site+key to allow callers with an existing
list_metadata_id to skip the header lookup query.

Renamed functions for consistency:
- set_items → set_items_by_list_metadata_id / set_items_by_list_key
- append_items → append_items_by_list_metadata_id / append_items_by_list_key
- update_header → update_header_by_list_metadata_id / update_header_by_list_key
- increment_version → increment_version_by_list_metadata_id / increment_version_by_list_key
- get_state → get_state_by_list_metadata_id
- get_state_by_key → get_state_by_list_key
- update_state → update_state_by_list_metadata_id
- update_state_by_key → update_state_by_list_key
Add `ConstraintViolation` variant to `SqliteDbError` to distinguish
UNIQUE constraint violations from other SQLite errors. Use this in
`ListMetadataRepository::get_or_create` to handle the rare case where
another thread creates the same header between our SELECT and INSERT.

Changes:
- Add `SqliteDbError::ConstraintViolation` variant
- Update `From<rusqlite::Error>` to detect constraint violations
- Update `get_or_create` to catch constraint violations and re-fetch
Add `get_or_create_and_increment_version` that uses a single
`INSERT ... ON CONFLICT DO UPDATE ... RETURNING` query to atomically
create or update a header while incrementing its version.

This reduces `begin_refresh` from 5-6 queries to 2 queries:
- Before: get_or_create (1-2) + increment_version (2) + update_state (1) + get_header (1)
- After: get_or_create_and_increment_version (1) + update_state (1)

Changes:
- Add `HeaderVersionInfo` struct for the return type
- Add `get_or_create_and_increment_version` method using RETURNING clause
- Update `begin_refresh` to use the new optimized method
- Add tests for the new method
Remove `increment_version_by_list_metadata_id` and `increment_version_by_list_key`
since version increment should only happen through `get_or_create_and_increment_version`
as part of a proper refresh flow, not called arbitrarily.
Move concurrency orchestration logic out of the repository layer.
These helpers (`begin_refresh`, `begin_fetch_next_page`, `complete_sync`,
`complete_sync_with_error`) belong in a service layer since they:
- Orchestrate multiple repository operations
- Apply business rules (checking page counts, deciding state transitions)
- Return domain-specific result types

The repository now provides clean primitives that a service layer can compose:
- `get_or_create_and_increment_version` for atomic create/increment
- `update_state_by_list_metadata_id` for state changes
- `get_header`, `get_items`, etc. for reads
Move `reset_stale_fetching_states_internal` from `WpApiCache` to
`ListMetadataRepository` where it belongs. The repository owns
the state table and should provide all operations on it.

`WpApiCache::perform_migrations` still calls this method after
migrations complete, but now delegates to the repository.
@oguzkocer oguzkocer force-pushed the feature/list-metadata-repository branch from ac33332 to 811eb46 Compare December 18, 2025 19:47
The repository layer should not dictate per_page values - this
must be set by the service layer to match networking configuration.

Changes:
- Add PerPageMismatch error variant to SqliteDbError
- Make per_page a required parameter in get_or_create
- Make per_page required in get_or_create_and_increment_version
- Update set_items_by_list_key to require per_page
- Update append_items_by_list_key to require per_page
- Update update_state_by_list_key to require per_page
- Return error when existing header has different per_page
Update schema validation tests to use `get_table_column_names` helper,
which verifies that column enum indices match actual database schema
positions via PRAGMA table_info.

This matches the pattern used by posts repository tests and provides
stronger guarantees that the column enums won't break if migrations
reorder columns.
@oguzkocer oguzkocer changed the title List Metadata Repository Add ListMetadataRepository for paginated list state tracking Dec 18, 2025
@oguzkocer oguzkocer marked this pull request as ready for review December 18, 2025 21:48
@oguzkocer oguzkocer requested a review from jkmassel December 18, 2025 21:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants