[ralyodio]

SQLite's one writer caps pitbot's local search mirror at ~63 pages/s; 1M pages/hour needs ~280/s. With CRAWL_INDEX_DATABASE_URL set, the mirror now lives in Postgres (services/crawler/src/pg-index-store.ts) and any number of publish processes write it at once. Unset, the existing SQLite IndexStore is used unchanged (tests, laptops).

Schema (role's default schema, nothing names public)

• pages(id bigint identity pk, url text unique, site, kind, title, description, keywords, body, language, fetched_at text, added_at timestamptz default clock_timestamp(), fts tsvector generated stored)
  • fts = title A, keywords B, description + site (+ site split on dots) C, body D, config 'simple'
  • indexes: pages_fts GIN (fastupdate = on), pages_fetched_at (fetched_at, id)
• page_policy(id pk, site, kind, language, fetched_at, train smallint, reason, source) + (train, id), (site, id), (train, site)
• page_cards(url pk, image, price), policy_site_checks(site pk, checked_at bigint), index_meta(k pk, n bigint) (pages_live, cursors, trim figures)
• Lower autovacuum scale factors on pages/page_policy, set once.

How it maps from SQLite

• Writes: one INSERT ... ON CONFLICT (url) DO UPDATE per batch over unnest() of pgArray() literals (no JS arrays bound). The policy upsert, cards, and pages_live += inserted (xmax = 0) run in the same statement. Rows are sorted by url so concurrent batches lock in one order, and deadlocks are retried. NUL is stripped, and URLs over 2 kB are skipped so they can't fail a batch.
• Search: websearch_to_tsquery('simple') with each token quoted (so or/- stay words). It ranks the first 3,000 GIN matches with ts_rank({0.1,0.4,0.6,1.0}) (bm25 used to score every match). AND first, OR fallback. Each query runs under a ~1.1 s statement timeout, and a timeout returns []. score = -ts_rank, so lower is still better.
• Kind (product) path: 8,000 GIN matches → that kind → top 200 by id → most title words first (same JS sort).
• Full-text plans: enable_seqscan/indexscan off, work_mem 32MB locally. Otherwise a word in most pages made the planner seq-scan and decompress every tsvector (a 2 s timeout at 70k pages, cold). Now it uses a GIN bitmap.
• Corpus: the same id windows, budget, comma lists (a b or c d), filters, and total_matches cap.
• /stats: count()/sizeBytes() are cached and refreshed in the background, and report() caps them at 1 s.
• corpusStats: cached 5 min, stale-while-revalidate.

Behaves differently

• Corpus lag: corpus scans stop at the newest id at least 60 s old (added_at). Concurrent writers commit out of id order, and a cursor must not pass an uncommitted id. Writes have a 30 s statement timeout.
• Byte budget: Postgres files don't shrink, so trim keeps live pages × bytes-per-page within budget. Bytes-per-page is re-measured whenever the files grow. Freed space is reused, so the files plateau near the budget. Lowering the budget below the current file size needs VACUUM FULL or pg_repack to give space back. trim() returns liveBytes. Batches run under a try-advisory lock with FOR UPDATE SKIP LOCKED, never below keepMin.
• Ranking and tokens: ranking is over a bounded candidate set, not every match. There is no diacritic folding (FTS5 had remove_diacritics 2), and node.js/1.3.14 are single tokens.
• Policy mapping: runs on the main thread for PG (async); the Worker stays for SQLite.

migrate-index

pit-crawler migrate-index --from-sqlite <path> [--batch n] [--restart] opens SQLite read-only and pages through keyset batches of pages ⟕ page_policy. Each batch lands in one statement together with its cursor (index_meta.migrate_cursor), so it is resumable. Ids and fetched_at are kept. It first moves the id sequence past SQLite's max id. A URL already in PG (written by the crawl) wins. A page whose id was taken by an earlier PG write gets a fresh id. It then copies cards and site checks, and resync()s the counter.

Archive

Each process writes pages-<hour>-<host>-<pid>.jsonl.gz(.partial). A process seals its own past hours at once and another process's once that hour is two behind. Legacy untagged partials still seal. The pages-*.jsonl.gz glob still matches.

Tests

• services/crawler, SQLite only: 110 pass, 22 skip.
• With CRAWL_TEST_PG against postgres:17-alpine (container stopped after): 121 pass, 10 skip (Redis/Obscura ones). The 10 new PG tests use their own role + schema each. They cover add/upsert/search/kind/cards/odd input/corpus+lag/policies/mapLegacy/stats, 4 concurrent stores counting 300 exactly, trim oldest-first + keepMin, and migrate-index incl. resume, restart, id collision and the sequence moving past.
• Repo root: tsc clean, 643 pass, 22 skip.
• Local bench (synthetic 1,200-word pages, docker PG on this box): 1 writer ~750 pages/s, 6 writers ~1,600 pages/s. At 100k pages, searches run 3-260 ms and corpus 16-260 ms warm (1.7 s right after a bulk insert).

Not deployed; nothing on the live stack changed.

🤖 Generated with Claude Code