Repository navigation
Commit 652da20
fix(metadata-protocol): a seed that stores many seedFile values no longer starves sign-ins (yield per row, 20-row batches) (#26822)
Fixes #26670
Clause-②: no
Metadata-protocol: none
## 维护者速读
- **改了什么**:种子加载器在每一行种子文件的存储与认领之后各让出一次事件循环;携带种子文件的数据集改为每批 20 行插入(原为 200
行)。无种子文件的数据集行为不变。
- **为什么改**:实测证伪了卡上「连接池被占用」的推测——连接池最长持有 44 ms;真正被占住的是事件循环:同步的 sqlite
驱动让每个 await 都成了微任务,一批 200 行的认领连成一条不间断的链,开机种子期间的登录请求要等 3–5 秒才被读到。
- **风险与回滚**:风险低,只改变种子文件工作的调度顺序,不改写入内容与结果;回滚即撤销本 PR 的两个 fix 提交。
-
**先看**:`packages/metadata-protocol/src/seed-files.ts`(`yieldToEventLoop`
及两处调用点,`settleRow` 的
`finally`);`packages/metadata-protocol/src/seed-loader.ts`(`insertBatchSize`)。
- **两分钟验证**:`pnpm --filter @objectstack/metadata-protocol exec vitest
run src/seed-loader-seed-files.test.ts`,预期 19 passed,其中「a seed of many
files yields to waiting requests (#26670)」三条全绿。
- **需要你做的**:无,席位落地。
## What changes
**The reading first: which call holds what.** The card's seam ("the seed
loader holds a pooled connection or a transaction") was marked
unverified, so the first measurement was the pool itself. No call on the
seed-file path opens a transaction: none in `seed-loader.ts`,
`seed-files.ts` or `field-file-service.ts`, nor in the boot runner
(`app-plugin.ts` `runInlineSeed`). Every step is sequential, and there
is no `Promise.all`. Measured on the real composition (`os serve --dev`,
file-backed sqlite, 1,500 rows each holding one 20 KB `seedFile`,
sign-in every 250 ms while the background seed runs). A preload patched
tarn's `Pool` (acquire, release) and sampled event-loop lag with a 100
ms timer:
| reading | before (base `a485b5a3`) | after (fix at `abc4ef3b`; `src`
unchanged since) |
|---|---|---|
| pool acquires during the run | 37,494 | 38,430 |
| longest connection hold | 44 ms | 103 ms |
| longest acquire wait | 21 ms | 19 ms |
| event-loop lag, max per 5 s window, during the seed | 2,827–4,996 ms |
170–317 ms |
| sign-ins during the seed: p50 / slowest | 358 ms / 5,195 ms | 187 ms /
447 ms |
The pool was never starved. The **event loop** was. On a synchronous
driver (better-sqlite3) every `await` of a seed file's roughly 25 engine
calls (the `pending` `sys_file` row, the bytes, then `claimFiles`:
describe, `ownedBySlot`, release, commit, each with its write hooks and
audit rows) resolves as a microtask. The loader buffers inserts into
200-row batches, and after each flush `settleSeedFiles` claims the
batch's rows back to back. Those claims involve no I/O at all, so a
whole batch ran as one unbroken microtask chain, and a sign-in arriving
meanwhile was not even read off the socket until it ended. The stalls
landed once per batch (about 8 for 1,500 rows), and a CPU profile put
the time in better-sqlite3's synchronous `_query` and `prepare`. Control
on the same boot without file values: 1,500 rows and also 30,000 rows
finish inside the 8 s inline budget, with about 2,140 acquires in all,
so the cost is the per-row file work and not row count.
**The fix**, in the producer (`packages/metadata-protocol`, inside the
claim's file surface):
1. `seed-files.ts` gets `yieldToEventLoop()`, one `setImmediate` turn.
This is the cure `@objectstack/core`'s import runner already applies to
the same trap. `prepareRow` ends with it when the row stored or kept
files, and `settleRow` ends with it (in a `finally`) after every row's
claim. A request now waits behind one row's files, never behind a
batch's.
2. `seed-loader.ts` writes a dataset that carries seed files
(`SeedDatasetFiles.carriesFiles`, the bundled `files` map is non-empty)
in insert batches of `SEED_FILE_BATCH_SIZE = 20` instead of 200. With
the settle loop broken up, the remaining stall was the single
`insertMany` of 200 file-bearing rows: about 1.5 s, since each row's
file-reference hooks cost about 7 ms on sqlite. At 20 rows that is about
150 ms. Datasets without seed files keep the 200-row batch.
⛔ No pool size or timeout was raised. The seed's wall time is unchanged
within boot noise (about 25–30 s of seed in each 1,500-file run).
`packages/metadata-protocol/src/protocol.ts` and
`packages/services/service-storage/src/field-file-service.ts` are
untouched: neither holds anything.
## Tests
New `describe` in
`packages/metadata-protocol/src/seed-loader-seed-files.test.ts`. It
seeds 60 files on an object with no self-reference, so the loader takes
the batched-insert path the stall was measured on. The suite's fake
engine and field-file service resolve purely in microtasks, which is the
property of better-sqlite3 that matters, so the pin is deterministic
without a driver:
- `stores at most one row's files and settles at most one row's claim
between two macrotask turns`: samples progress on every `setImmediate`
turn while the load runs.
- `serves an unrelated read issued during the seed within one row's file
work`: a `sys_user` read issued from a macrotask, standing in for the
sign-in's user lookup.
- `writes a dataset that carries seed files in insert batches of at most
20 rows`.
Ablations (each mutation landed and restored by
`scripts/ablation-replace.mjs`, which checks the anchor and blob counts
and proves the restore against the HEAD blob; the test imports source,
so no `dist/` is involved):
- **A**: `yieldToEventLoop` resolves at once. Pins 1 and 2 red: `no
macrotask turn ran while the seed ran: expected 0 to be greater than 1`,
`the read never completed: expected undefined to be defined`. 2 failed,
17 passed.
- **B**: `SEED_FILE_BATCH_SIZE = 200`. Pin 3 red: `expected 60 to be
less than or equal to 20`. 1 failed, 18 passed.
- **C**: only the settle-side yield removed. Pin 1 red: `expected {
maxStored: 1, maxClaims: 20 } to deeply equal { maxStored: 1, maxClaims:
1 }`, which is a whole batch's claims settled with no turn, the measured
shape. The first attempt at C was a no-op (the tool refused: the
replacement was a substring of its anchor, so the test never ran); it
was re-run with a distinct replacement.
- A first version of the pins seeded `clm_version`, whose self-lookup
sends the loader down the sequential per-record path. Leg B stayed green
against it, which exposed the pins as vacuous for the batched path. They
were moved to `clm_doc` before the legs above.
Runs on the final head `4bf7600183` unless noted:
<details><summary>Readings</summary>
- `pnpm --filter @objectstack/metadata-protocol exec vitest run
--maxWorkers=2 src/seed-loader-seed-files.test.ts` at `4bf7600183`:
`Tests 19 passed (19)`, lock VERDICT command-exit 0.
- `pnpm --filter @objectstack/metadata-protocol typecheck` at
`4bf7600183`: exit 0 (same `&&` chain). The test file is in the tsc
program (`--listFiles` names it once).
- `pnpm --filter @objectstack/metadata-protocol test` at `d294c889` (the
merge of `origin/main`; the only later commit types one test literal):
`Test Files 269 passed | 3 skipped (272)`, `Tests 21619 passed | 19
skipped (21638)`, after rebuilding the package's dependency closure
(`pnpm --filter '@objectstack/metadata-protocol^...' build`).
- `pnpm exec eslint` on the three changed source files: exit 0.
- Boot measurement: scratch scripts, not committed; numbers in the table
above.
</details>
## Acceptance notes
- **Not reproduced here:** the card's `Knex: Timeout acquiring a
connection` refusal. The measured pool waits never exceeded 21 ms, so
this run shows the sign-in *slowdown* (5.2 s at worst on 1,500 small
files) and its cause, not the refusal. hotclm's 28 s sign-in and its
refusals are consistent with longer per-batch stalls (bigger files, more
write hooks), but that is inference, not a reading.
- **A residual stall at the end of the seed, outside this diff:** one
3.3 s event-loop stall as the seed settles (a sign-in took 3,548 ms
then). The profile slice shows `emitSeedSettled`, then
`claimSeedOwnershipOnSettle`, then `claimSeedOwnership`
(`@objectstack/plugin-security`, the ADR-0138 hand-over on
`app:seeded`), with a driver `updateMany` and per-row insert and audit
frames. It re-owns the seeded rows in one unbroken chain, scales with
rows rather than files, and runs for any seed that overruns the inline
budget. Reported to the seat, not fixed here.
- The boot measurement used scratch scripts (pool tracer, sign-in probe)
that are not committed.
## Verification
Gate families derived for this diff by `node
scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack` at `4bf7600183` (59 commands), each run on
`4bf7600183` with its exit code captured before any pipe. `--ran`
reconciliation: `59 derived, 59 run, 0 NOT-MEASURED, 0 UNRUN` (a derived
zero: every line records its exit code).
<details><summary>59 gate commands, exit codes</summary>
- `node scripts/check-adr-0087-registration.mjs --base origin/main`:
exit 0
- `node scripts/check-adr-0087-registration.mjs --self-test`: exit 0
- `node scripts/check-changeset-no-major.mjs --base origin/main`: exit 0
- `node scripts/check-closing-keyword-parity.mjs`: exit 0
- `node scripts/check-closing-keyword-parity.mjs --self-test`: exit 0
- `node scripts/check-comment-mask-adoption.mjs`: exit 0
- `node scripts/check-comment-mask-adoption.mjs --self-test`: exit 0
- `node scripts/check-comment-mask-corpus.mjs`: exit 0
- `node scripts/check-dts-emitted.mjs --self-test`: exit 0
- `node scripts/check-empty-changeset.mjs --base origin/main`: exit 0
- `node scripts/check-empty-changeset.mjs --self-test`: exit 0
- `node scripts/check-issue-citations.mjs`: exit 0
- `node scripts/check-plugin-teardown-shape.mjs`: exit 0
- `node scripts/check-plugin-teardown-shape.mjs --self-test`: exit 0
- `node scripts/ci/assert-turbo-remote-cache.mjs --self-test`: exit 0
- `node scripts/ci/plan-test-core-shards.mjs --self-test`: exit 0
- `node scripts/ci/prune-turbo-cache.mjs --self-test`: exit 0
- `node scripts/docs-audit/check-affected-docs.mjs`: exit 0
- `node scripts/docs-audit/check-drift-comment.mjs`: exit 0
- `node scripts/pm/release-rehearsal-clone.mjs --self-test`: exit 0
- `node scripts/release-pending-publish.mjs --self-test`: exit 0
- `pnpm --filter @objectstack/spec run check:duration-unit-keys`: exit 0
- `pnpm check:changeset-gate-self-tests`: exit 0
- `pnpm check:ci-filter-parity`: exit 0
- `pnpm check:closed-ceremony-text`: exit 0
- `pnpm check:cross-package-test-inputs`: exit 0
- `pnpm check:doc-authoring`: exit 0
- `pnpm check:driver-memory-census`: exit 0
- `pnpm check:dts-closure`: exit 0
- `pnpm check:dual-build-cjs-loads`: exit 0
- `pnpm check:durability-log-level`: exit 0
- `pnpm check:engine-double-contract`: exit 0
- `pnpm check:error-status-conformance`: exit 0
- `pnpm check:gitlink-declared`: exit 0
- `pnpm check:issue-citations`: exit 0
- `pnpm check:keyed-text-bounds`: exit 0
- `pnpm check:lean-entry-closure`: exit 0
- `pnpm check:logger-receiver-detach`: exit 0
- `pnpm check:nul-bytes`: exit 0
- `pnpm check:objectql-double-limit`: exit 0
- `pnpm check:org-identifier`: exit 0
- `pnpm check:page-declaration-shape`: exit 0
- `pnpm check:platform-object-tenancy-census`: exit 0
- `pnpm check:pm-changeset-deadline-census`: exit 0
- `pnpm check:published-files`: exit 0
- `pnpm check:query-options-erasure`: exit 0
- `pnpm check:refd-timer-probe`: exit 0
- `pnpm check:registry-log-declared`: exit 0
- `pnpm check:rest-log-spy-declared`: exit 0
- `pnpm check:slot-lookup`: exit 0
- `pnpm check:sourcemap-no-sources-content`: exit 0
- `pnpm check:system-context-census`: exit 0
- `pnpm check:test-source-alias`: exit 0
- `pnpm check:tier-file-adoption`: exit 0
- `pnpm check:type-check-coverage`: exit 0
- `pnpm check:type-check-debt`: exit 0
- `pnpm check:undeclared-dep-imports`: exit 0
- `pnpm check:watch-hint-literal`: exit 0
- `pnpm check:where-matcher`: exit 0
</details>
- The first union (on `d294c889`) caught one real red:
`check:query-options-erasure`, where the test surface grew 236 to 237
through a stand-in read's `as any`. It was fixed in `4bf7600183` by
typing the literal. That union also read `check-plugin-teardown-shape
--self-test` as a prerequisite miss (shallow clone; its pinned fixture
commit was fetched) and `check:dual-build-cjs-loads` as PREREQUISITE NOT
MET. Both are green above on a real measurement.
- Declared to CI, not run locally: `test:repo` (the vitest `repo`
project; the diff touches none of its tests), the repo-wide `pnpm lint`,
the type-check lanes, and the artifact-roster and wide-population
families the derivation lists outside its 59.
- Dogfood battery: not owed (no `!` in the title, `Clause-②: no` with no
arm).
---
_Generated by [Claude
Code](https://claude.ai/code/session_01QSeKYmQXM3rHJcPnrEPk4B)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 8eef9a8 commit 652da20
4 files changed
Lines changed: 164 additions & 3 deletions
File tree
- .changeset
- packages/metadata-protocol/src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
61 | 61 | | |
62 | 62 | | |
63 | 63 | | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
64 | 90 | | |
65 | 91 | | |
66 | 92 | | |
| |||
117 | 143 | | |
118 | 144 | | |
119 | 145 | | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
120 | 151 | | |
121 | 152 | | |
122 | 153 | | |
| |||
297 | 328 | | |
298 | 329 | | |
299 | 330 | | |
| 331 | + | |
| 332 | + | |
300 | 333 | | |
301 | 334 | | |
302 | 335 | | |
| |||
308 | 341 | | |
309 | 342 | | |
310 | 343 | | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
311 | 356 | | |
312 | 357 | | |
313 | 358 | | |
| |||
319 | 364 | | |
320 | 365 | | |
321 | 366 | | |
322 | | - | |
| 367 | + | |
323 | 368 | | |
324 | 369 | | |
325 | 370 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
427 | 427 | | |
428 | 428 | | |
429 | 429 | | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1279 | 1279 | | |
1280 | 1280 | | |
1281 | 1281 | | |
1282 | | - | |
| 1282 | + | |
1283 | 1283 | | |
1284 | 1284 | | |
1285 | 1285 | | |
| |||
1399 | 1399 | | |
1400 | 1400 | | |
1401 | 1401 | | |
| 1402 | + | |
| 1403 | + | |
| 1404 | + | |
| 1405 | + | |
| 1406 | + | |
| 1407 | + | |
| 1408 | + | |
| 1409 | + | |
| 1410 | + | |
| 1411 | + | |
| 1412 | + | |
| 1413 | + | |
1402 | 1414 | | |
1403 | 1415 | | |
1404 | 1416 | | |
| |||
2119 | 2131 | | |
2120 | 2132 | | |
2121 | 2133 | | |
2122 | | - | |
| 2134 | + | |
2123 | 2135 | | |
2124 | 2136 | | |
2125 | 2137 | | |
| |||
2938 | 2950 | | |
2939 | 2951 | | |
2940 | 2952 | | |
| 2953 | + | |
| 2954 | + | |
| 2955 | + | |
2941 | 2956 | | |
2942 | 2957 | | |
2943 | 2958 | | |
| |||
0 commit comments