-
Notifications
You must be signed in to change notification settings - Fork 3
Add ListMetadataRepository for paginated list state tracking
#1069
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
oguzkocer
wants to merge
25
commits into
trunk
Choose a base branch
from
feature/list-metadata-repository
base: trunk
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduces database infrastructure for tracking list metadata, enabling efficient cache invalidation and stale detection for paginated lists. Changes: - Add migration for list metadata tables - Add DbListMetadata types for database operations - Add ListMetadata domain types - Add ListMetadataRepository with full CRUD operations - Update PostsRepository with stale detection helpers Extracted from prototype/metadata-collection branch. Original commits: - 14b01d80 (Implement stale detection by comparing modified_gmt timestamps) - e47cec89 (Add database foundation for MetadataService) - 2440a13f (Add list metadata repository concurrency helpers) - cc0c8a58 (Reset stale fetching states on app launch) - d64142fb (Split collection observers for data vs state updates) - fe7435c9 (make fmt-rust) - 2918b339 (Add parent and menu_order fields to list metadata items) - 25f88a49 (Rename last_updated_at to last_fetched_at in list metadata) - 1d709e70 (Simplify ListMetadataReader trait with combined ListInfo query) Note: Since we use a rebase/squash merge strategy, these commits may show "does not belong to any branch" on GitHub but remain accessible via URL.
Changes the database storage for `ListState` from TEXT strings to INTEGER values for better performance and type safety. Changes: - Update migration to use `INTEGER NOT NULL DEFAULT 0` for state column - Add `#[repr(i32)]` to `ListState` enum with explicit discriminant values - Implement `ToSql` and `FromSql` traits for direct rusqlite integration - Remove string-based `as_db_str()` and `From<&str>` implementations - Update all callers to use the enum directly with rusqlite params
The `FromSql` implementation now returns a proper error when encountering an unknown integer value, rather than silently defaulting to `Idle`. This makes data corruption issues visible instead of hiding them.
The repository struct has no state, so methods are converted from instance methods (&self) to associated functions. This removes the need to instantiate the struct before calling methods. Before: ListMetadataRepository.get_header(&conn, &site, key) After: ListMetadataRepository::get_header(&conn, &site, key)
Replaces individual INSERT statements with a single batch INSERT for better performance when inserting multiple items. Uses functional style with try_for_each and flat_map. Items are chunked to stay under SQLite's variable limit (999).
Replaces two separate queries (get_header + get_state) with a single JOIN query via get_header_with_state for better efficiency.
Replaces raw `&str` parameters with `&ListKey` throughout the repository API. This prevents accidental misuse of arbitrary strings as list keys and makes the API more self-documenting. The ListKey type provides: - Type safety at compile time - Conversion from &str and String via From trait - as_str() for SQL parameter usage - Display impl for debug output
- Condense doc comment, remove references to non-existent MetadataService - Return Result<usize, SqliteDbError> instead of logging internally - Ignore errors at call site with explanatory comment
Normalizes the schema by replacing (db_site_id, key) in items table with a foreign key to list_metadata. This: - Ensures referential integrity - Enables cascade delete (simplifies delete_list) - Reduces storage per item
Adds the `log` facade crate to enable proper logging. Debug logs added to key list metadata operations: - begin_refresh, begin_fetch_next_page - complete_sync, complete_sync_with_error - set_items, append_items, delete_list Replaces eprintln! with log::warn! for unknown table warnings.
Implement ToSql/FromSql traits for ListKey to simplify repository code by using the type directly in SQL params instead of .as_str(). Also add logging for reset_stale_fetching_states failures instead of silently ignoring errors. Changes: - Add ToSql/FromSql implementations for ListKey - Replace key.as_str() with key in all SQL params - Log warning on reset_stale_fetching_states failure
…st_key variants Allows callers who already have the list_metadata_id to skip an extra header lookup query.
Split functions that take site+key to allow callers with an existing list_metadata_id to skip the header lookup query. Renamed functions for consistency: - set_items → set_items_by_list_metadata_id / set_items_by_list_key - append_items → append_items_by_list_metadata_id / append_items_by_list_key - update_header → update_header_by_list_metadata_id / update_header_by_list_key - increment_version → increment_version_by_list_metadata_id / increment_version_by_list_key - get_state → get_state_by_list_metadata_id - get_state_by_key → get_state_by_list_key - update_state → update_state_by_list_metadata_id - update_state_by_key → update_state_by_list_key
Add `ConstraintViolation` variant to `SqliteDbError` to distinguish UNIQUE constraint violations from other SQLite errors. Use this in `ListMetadataRepository::get_or_create` to handle the rare case where another thread creates the same header between our SELECT and INSERT. Changes: - Add `SqliteDbError::ConstraintViolation` variant - Update `From<rusqlite::Error>` to detect constraint violations - Update `get_or_create` to catch constraint violations and re-fetch
Add `get_or_create_and_increment_version` that uses a single `INSERT ... ON CONFLICT DO UPDATE ... RETURNING` query to atomically create or update a header while incrementing its version. This reduces `begin_refresh` from 5-6 queries to 2 queries: - Before: get_or_create (1-2) + increment_version (2) + update_state (1) + get_header (1) - After: get_or_create_and_increment_version (1) + update_state (1) Changes: - Add `HeaderVersionInfo` struct for the return type - Add `get_or_create_and_increment_version` method using RETURNING clause - Update `begin_refresh` to use the new optimized method - Add tests for the new method
Remove `increment_version_by_list_metadata_id` and `increment_version_by_list_key` since version increment should only happen through `get_or_create_and_increment_version` as part of a proper refresh flow, not called arbitrarily.
Move concurrency orchestration logic out of the repository layer. These helpers (`begin_refresh`, `begin_fetch_next_page`, `complete_sync`, `complete_sync_with_error`) belong in a service layer since they: - Orchestrate multiple repository operations - Apply business rules (checking page counts, deciding state transitions) - Return domain-specific result types The repository now provides clean primitives that a service layer can compose: - `get_or_create_and_increment_version` for atomic create/increment - `update_state_by_list_metadata_id` for state changes - `get_header`, `get_items`, etc. for reads
Move `reset_stale_fetching_states_internal` from `WpApiCache` to `ListMetadataRepository` where it belongs. The repository owns the state table and should provide all operations on it. `WpApiCache::perform_migrations` still calls this method after migrations complete, but now delegates to the repository.
ac33332 to
811eb46
Compare
The repository layer should not dictate per_page values - this must be set by the service layer to match networking configuration. Changes: - Add PerPageMismatch error variant to SqliteDbError - Make per_page a required parameter in get_or_create - Make per_page required in get_or_create_and_increment_version - Update set_items_by_list_key to require per_page - Update append_items_by_list_key to require per_page - Update update_state_by_list_key to require per_page - Return error when existing header has different per_page
Update schema validation tests to use `get_table_column_names` helper, which verifies that column enum indices match actual database schema positions via PRAGMA table_info. This matches the pattern used by posts repository tests and provides stronger guarantees that the column enums won't break if migrations reorder columns.
ListMetadataRepository for paginated list state tracking
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Adds
ListMetadataRepositorytowp_mobile_cachefor tracking paginated list state, enabling efficient cache invalidation and stale detection.Database Schema
Type Hierarchy
Key Features
ListKeynewtype prevents misuse of arbitrary strings;ListStatestored as INTEGER withToSql/FromSqlget_or_createcatches constraint violations and retriesget_or_create_and_increment_versionusesINSERT ... ON CONFLICT DO UPDATE ... RETURNINGRepository Operations
Error Handling
ConstraintViolation- detected fromrusqlite::Errorfor race condition handlingPerPageMismatch { expected, actual }- when existing header has differentper_pagethan requestedDesign Decisions
per_pageis required: Must be passed by caller to match networking config; returns error on mismatch rather than silently using existing value_by_list_metadata_idfor callers with cached rowid,_by_list_keyfor convenienceChanges
0007-create-list-metadata-tables.sqlListKey,ListState, and DB types inlist_metadata.rsdb_types/db_list_metadata.rsListMetadataRepositorywith associated functionsConstraintViolationandPerPageMismatcherror variants toSqliteDbErrorlogcrate dependency for structured loggingreset_stale_fetching_statesafter migrations complete