perf(users): sync the user index incrementally, carry character names in index records - #639
Draft
MorquinDevlar wants to merge 2 commits into
Draft
Conversation
Startup previously paid two full passes over every user file: the user index rebuild unmarshaled each file into a complete UserRecord and then appended records one at a time (an fsync per user, ~3.7ms each), and CharacterIndex.Rebuild ran a second full unmarshal of every file on every boot, including copyovers. Both indexes are now fed by a single lightweight scan that decodes only userid, username, and the active character name, and the user index is written in one atomic pass (temp file + rename, single fsync) with the directory checksum folded into the same write. At 1000 users this takes a boot-after-activity rebuild from roughly 4s to roughly 0.1s. The scan also hardens rebuild behavior: unreadable or malformed user files are skipped with a warning instead of silently aborting the walk, and anomalies that usually mean hand-edited data (duplicate userids, duplicate usernames, a filename that disagrees with the userid inside) are logged. Also fixes a silent-failure edge in index loading: loadRecords now logs read failures, and IsUpToDate refuses to trust an index whose records section is shorter than its header claims. Previously a truncated index with an intact header could pass the checksum test and boot with empty maps, which would make GetUniqueUserId hand out userids that already belong to existing user files.
…every file Widens the index record (version 3) to carry the active character name plus the mtime and size the user file had when it was indexed. Startup then syncs instead of rebuilding: one directory listing is compared against the stored records, unchanged files keep their entry without being opened, and only files that are new or changed since the last index write get parsed. Records whose files are gone are dropped. When nothing changed the index file is not rewritten at all. The character index is now rebuilt straight from the index records, so a boot where nothing changed - including every copyover - reads zero user files. Startup cost becomes proportional to how many users were active since the last restart instead of total accounts. Measured at 1000 users (8KB files, warm cache): a no-change sync takes 2.4ms and a 50-file churn sync 18ms, versus roughly 120ms for the full scan this replaces and roughly 4s for the original per-user rebuild. An index in the old format, or one that fails to load cleanly, falls back to one full scan and is rewritten in the new format - the first boot after upgrading pays one full scan, every boot after that syncs. Runtime AddUser writes a stub record (zero mtime), which the next sync re-parses and completes, preserving the register-then-save ordering.
pruuk
added a commit
to pruuk/DOGMud
that referenced
this pull request
Jul 28, 2026
… write Port of upstream GoMud PR GoMudEngine#638 (MorquinDevlar). The old Rebuild fully YAML-decoded every user via SearchOfflineUsers and appended records one-at-a-time (a file open + header rewrite per user). Worse, a single malformed user file returned an error from the walk callback and ABORTED the scan — every user after the bad file silently vanished from the index and could no longer log in. Rebuild now walks the users dir once, decodes only userid/username into a two-field struct, collects records in memory, and writes header + records to a temp file renamed over users.idx (a crash mid-write can never leave a truncated index). Malformed/unreadable files skip with a warning; duplicate userids/usernames skip (first wins) and are logged; filename/content userid mismatches are logged. On-disk V1 format is unchanged. Upstream GoMudEngine#639 (mtime/size incremental sync, index v3) deliberately NOT ported: it is still a draft upstream, our copyover path already skips the rebuild entirely (state rides the pipe), and at ~53 prod users the single minimal scan is already sub-millisecond. Revisit at scale. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Stacked on #638 - contains its commit; will rebase once that lands.
The directory checksum only matches when no user file changed since the last boot. Any login updates
lastloginand any logout saves the file, so on a live server the checksum misses at essentially every boot and the full rebuild runs anyway. Copyovers always pay the full character-name scan regardless.This PR makes startup cost proportional to how many users were active since the last restart instead of total accounts:
Numbers (Apple M5 Pro, 1000 users, 8KB files, warm cache)
10,000 users, nothing changed: 30ms (one readdir + stat pass).
Reproduce with:
Upgrade and compatibility
AddUserwrites a stub record (zero mtime/size). The next sync sees the mismatch, re-parses that one file, and completes the record - the register-then-save ordering is preserved.<userid>.yamlconvention. Old-formatusername.yamlfiles still get indexed correctly but are re-parsed every boot since they never match a record - running the format migration avoids that cost.Behavior changes worth review
IsUpToDateis retained for compatibility but startup no longer calls it - the per-file comparison subsumes the whole-directory checksum.Tests
AddUserstub is completed by the next sync.