Skip to content

perf: count universes and works without materializing every record (#2729)#2732

Merged
atomantic merged 2 commits into
mainfrom
claim/issue-2729
Jul 17, 2026
Merged

perf: count universes and works without materializing every record (#2729)#2732
atomantic merged 2 commits into
mainfrom
claim/issue-2729

Conversation

@atomantic

Copy link
Copy Markdown
Owner

Summary

The Wordsmith skill scored Create/Writers Room engagement with listUniverses().length + listWorks().length — two whole-collection reads for a tally. listUniverses() selects and sanitizes every universe JSONB record and loads the entire universe-run history before sorting; listWorks() runs two queries to rebuild every work's manifest with its drafts[]. That cost is paid on every explicit /character read and every saveCharacter() response, and it grows with the user's library.

Adds countUniverses() / countWorks() across both backends (Postgres + the file escape hatch), threaded through the existing store facades, and points Wordsmith's compute() at them. The skill's value is unchanged for every existing case.

Finding: is universes.deleted authoritative and in sync with the JSONB flag?

Yes — they cannot diverge, so WHERE deleted = FALSE is exact. This was the issue's crux, so stating the verification explicitly:

  • listUniverses() filters !u.deleted on the sanitized record, where the flag comes from sanitizeSoftDeleteFields (server/lib/syncWire.js) — computed as raw?.deleted === true.
  • The universes table has exactly two writers, and each binds the column in the same statement that writes data, from the same record object, with the identical predicate record.deleted === true:
    • writeRaw (server/services/universeBuilder/db.js)
    • the one-time file import (server/scripts/migrateUniversesToDB.js)
  • There is no third writer and no UPDATE that touches deleted without rewriting data (verified by grepping all SQL against the table).

So the column is an exact mirror, and it additionally hits the idx_universes_live partial index that exists for precisely this "live set" scan. No divergence to fix.

Two related correctness points the count had to get right:

  • Tombstones are excluded — the specific bug a naive (await listIds()).length would introduce, since listIds() is documented as returning live + ephemeral + tombstones and leaving filtering to the service. Pinned by tests on both backends.
  • Ephemeral universes are still countedlistUniverses() filters only on deleted, so filtering ephemeral here would undercount. Pinned by a test.

Known boundary (documented in code, deliberately not replicated in SQL): a row whose data can't survive sanitizeTemplate (blank name, non-string id) is dropped by listUniverses's .filter(Boolean) but still counted. It's unreachable through the service — create/update/sync all sanitize before writing — and only a corrupt legacy record imported verbatim could reach it. Replicating the sanitizer's drop rules in SQL would couple the count to sanitizeTemplate's internals and silently drift when they change; a ±1 on an already-corrupt record isn't worth that. Both backends share these semantics, so they agree with each other.

writers_room_works.deleted needed no such analysis: it's a real column (not a JSONB mirror) and it's the same predicate listWorks() already filters on.

Step 5 (saveCharacter(data, { withSkills })) — evaluated, deliberately not shipped

The issue asked to consider it. It has no beneficiary, so the option would be dead weight:

  • Every caller of saveCharacter() (addXP, takeDamage, takeRest, addEvent, syncJiraXP, syncTaskXP, updateCharacterFields, /reset) originates from server/routes/character.js — there are zero background/service callers (verified by grep).
  • Every one of those routes returns the character to client/src/pages/CharacterSheet.jsx, which does setChar(result.character). Passing withSkills: false anywhere would blank the skills card, so every call site would pass true.
  • The performance motivation also largely evaporates with this PR: skill derivation is now a handful of COUNT(*)/stat reads, not a full library scan.

enrichCharacter already accepts { withSkills } (from #2725), so the seam remains if a non-UI caller ever appears.

Test plan

  • server/services/universeBuilder/db.test.js — new: countUniverses counts the live set, excludes the tombstone while listIds() still returns 3, counts ephemeral, agrees with the JSONB-flag filter listRaw() applies, includeDeleted variant, and follows an un-delete. Ran against portos_test via the DB config (verified executed, not skipped).
  • server/services/writersRoom/db.test.js — new: countWorks agrees with listWorks().length and excludes a tombstone that listWorkIds({ includeDeleted: true }) still returns. Same, against portos_test.
  • server/services/universeBuilder.test.js — new (file backend, service level): countUniverses agrees with listUniverses().length before and after a soft-delete, and includeDeleted mirrors listUniverses({ includeDeleted: true }).
  • server/services/writersRoom/local.test.js — new (file backend, service level): countWorks agrees with listWorks().length across a soft-delete.
  • server/services/characterSkills.test.js — mocks updated to the count helpers; all 27 pass, Wordsmith's values unchanged.
  • Full server suite: 20092 passed / 200 skipped. The 2 failures in services/installState.test.js (install-root resolution) are pre-existing on origin/main and untouched by this diff — confirmed by stashing this branch's changes and re-running them red on a clean tree.

No DB guard was weakened; the DB-backed suites ran only via the portos_test config.

Closes #2729

…2729)

The Wordsmith skill scored Create/Writers Room engagement with
`listUniverses().length + listWorks().length`. Both are whole-collection
reads for a tally: listUniverses() selects and sanitizes every universe
JSONB record AND loads the entire universe-run history before sorting,
and listWorks() runs two queries to rebuild every work's manifest with
its drafts[]. That cost is paid on every explicit /character read and on
every saveCharacter() response, and it grows with the user's library.

Adds countUniverses()/countWorks() across both backends and points the
skill at them. The value is unchanged for every existing case.

countUniverses reads the `universes.deleted` mirror column rather than
the JSONB flag listUniverses() filters on. Verified the two cannot
diverge: writeRaw and the one-time file import are the only writers, and
each binds `record.deleted === true` into the column in the same
statement that writes `data` — the same predicate sanitizeSoftDeleteFields
computes for the flag the service filters on. So `WHERE deleted = FALSE`
is exact, and it hits the idx_universes_live partial index that exists
for this scan. Ephemeral universes are deliberately still counted
(listUniverses filters only on `deleted`), and a tombstone is not — the
bug a naive listIds().length would introduce, since listIds() returns
live + ephemeral + tombstones and leaves filtering to the service.

The file backend must read each record to see its `deleted` flag, so its
count has no cheap form; it still skips the run-history load and the
sanitize+sort. Writers Room's file count shares one live-set reader with
listWorks so the listing and the tally can't drift apart.
…#2729)

Reviewer flagged that the column is nullable, so the two forms differ on a
NULL. Keeping deleted = FALSE: it is the form the idx_universes_live partial
index can serve and it matches every sibling live-set query. Neither writer
can produce a NULL, so the distinction is unreachable — noting it so the
choice reads as considered rather than overlooked.
@atomantic
atomantic merged commit 5c39c46 into main Jul 17, 2026
2 checks passed
@atomantic
atomantic deleted the claim/issue-2729 branch July 17, 2026 11:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cheap count helpers for universes and Writers Room works (skill registry materializes every record for a .length)

1 participant