Skip to content

Commit 652da20

Browse files
fix(metadata-protocol): a seed that stores many seedFile values no longer starves sign-ins (yield per row, 20-row batches) (#26822)
Fixes #26670 Clause-②: no Metadata-protocol: none ## 维护者速读 - **改了什么**:种子加载器在每一行种子文件的存储与认领之后各让出一次事件循环;携带种子文件的数据集改为每批 20 行插入(原为 200 行)。无种子文件的数据集行为不变。 - **为什么改**:实测证伪了卡上「连接池被占用」的推测——连接池最长持有 44 ms;真正被占住的是事件循环:同步的 sqlite 驱动让每个 await 都成了微任务,一批 200 行的认领连成一条不间断的链,开机种子期间的登录请求要等 3–5 秒才被读到。 - **风险与回滚**:风险低,只改变种子文件工作的调度顺序,不改写入内容与结果;回滚即撤销本 PR 的两个 fix 提交。 - **先看**:`packages/metadata-protocol/src/seed-files.ts`(`yieldToEventLoop` 及两处调用点,`settleRow` 的 `finally`);`packages/metadata-protocol/src/seed-loader.ts`(`insertBatchSize`)。 - **两分钟验证**:`pnpm --filter @objectstack/metadata-protocol exec vitest run src/seed-loader-seed-files.test.ts`,预期 19 passed,其中「a seed of many files yields to waiting requests (#26670)」三条全绿。 - **需要你做的**:无,席位落地。 ## What changes **The reading first: which call holds what.** The card's seam ("the seed loader holds a pooled connection or a transaction") was marked unverified, so the first measurement was the pool itself. No call on the seed-file path opens a transaction: none in `seed-loader.ts`, `seed-files.ts` or `field-file-service.ts`, nor in the boot runner (`app-plugin.ts` `runInlineSeed`). Every step is sequential, and there is no `Promise.all`. Measured on the real composition (`os serve --dev`, file-backed sqlite, 1,500 rows each holding one 20 KB `seedFile`, sign-in every 250 ms while the background seed runs). A preload patched tarn's `Pool` (acquire, release) and sampled event-loop lag with a 100 ms timer: | reading | before (base `a485b5a3`) | after (fix at `abc4ef3b`; `src` unchanged since) | |---|---|---| | pool acquires during the run | 37,494 | 38,430 | | longest connection hold | 44 ms | 103 ms | | longest acquire wait | 21 ms | 19 ms | | event-loop lag, max per 5 s window, during the seed | 2,827–4,996 ms | 170–317 ms | | sign-ins during the seed: p50 / slowest | 358 ms / 5,195 ms | 187 ms / 447 ms | The pool was never starved. The **event loop** was. On a synchronous driver (better-sqlite3) every `await` of a seed file's roughly 25 engine calls (the `pending` `sys_file` row, the bytes, then `claimFiles`: describe, `ownedBySlot`, release, commit, each with its write hooks and audit rows) resolves as a microtask. The loader buffers inserts into 200-row batches, and after each flush `settleSeedFiles` claims the batch's rows back to back. Those claims involve no I/O at all, so a whole batch ran as one unbroken microtask chain, and a sign-in arriving meanwhile was not even read off the socket until it ended. The stalls landed once per batch (about 8 for 1,500 rows), and a CPU profile put the time in better-sqlite3's synchronous `_query` and `prepare`. Control on the same boot without file values: 1,500 rows and also 30,000 rows finish inside the 8 s inline budget, with about 2,140 acquires in all, so the cost is the per-row file work and not row count. **The fix**, in the producer (`packages/metadata-protocol`, inside the claim's file surface): 1. `seed-files.ts` gets `yieldToEventLoop()`, one `setImmediate` turn. This is the cure `@objectstack/core`'s import runner already applies to the same trap. `prepareRow` ends with it when the row stored or kept files, and `settleRow` ends with it (in a `finally`) after every row's claim. A request now waits behind one row's files, never behind a batch's. 2. `seed-loader.ts` writes a dataset that carries seed files (`SeedDatasetFiles.carriesFiles`, the bundled `files` map is non-empty) in insert batches of `SEED_FILE_BATCH_SIZE = 20` instead of 200. With the settle loop broken up, the remaining stall was the single `insertMany` of 200 file-bearing rows: about 1.5 s, since each row's file-reference hooks cost about 7 ms on sqlite. At 20 rows that is about 150 ms. Datasets without seed files keep the 200-row batch. ⛔ No pool size or timeout was raised. The seed's wall time is unchanged within boot noise (about 25–30 s of seed in each 1,500-file run). `packages/metadata-protocol/src/protocol.ts` and `packages/services/service-storage/src/field-file-service.ts` are untouched: neither holds anything. ## Tests New `describe` in `packages/metadata-protocol/src/seed-loader-seed-files.test.ts`. It seeds 60 files on an object with no self-reference, so the loader takes the batched-insert path the stall was measured on. The suite's fake engine and field-file service resolve purely in microtasks, which is the property of better-sqlite3 that matters, so the pin is deterministic without a driver: - `stores at most one row's files and settles at most one row's claim between two macrotask turns`: samples progress on every `setImmediate` turn while the load runs. - `serves an unrelated read issued during the seed within one row's file work`: a `sys_user` read issued from a macrotask, standing in for the sign-in's user lookup. - `writes a dataset that carries seed files in insert batches of at most 20 rows`. Ablations (each mutation landed and restored by `scripts/ablation-replace.mjs`, which checks the anchor and blob counts and proves the restore against the HEAD blob; the test imports source, so no `dist/` is involved): - **A**: `yieldToEventLoop` resolves at once. Pins 1 and 2 red: `no macrotask turn ran while the seed ran: expected 0 to be greater than 1`, `the read never completed: expected undefined to be defined`. 2 failed, 17 passed. - **B**: `SEED_FILE_BATCH_SIZE = 200`. Pin 3 red: `expected 60 to be less than or equal to 20`. 1 failed, 18 passed. - **C**: only the settle-side yield removed. Pin 1 red: `expected { maxStored: 1, maxClaims: 20 } to deeply equal { maxStored: 1, maxClaims: 1 }`, which is a whole batch's claims settled with no turn, the measured shape. The first attempt at C was a no-op (the tool refused: the replacement was a substring of its anchor, so the test never ran); it was re-run with a distinct replacement. - A first version of the pins seeded `clm_version`, whose self-lookup sends the loader down the sequential per-record path. Leg B stayed green against it, which exposed the pins as vacuous for the batched path. They were moved to `clm_doc` before the legs above. Runs on the final head `4bf7600183` unless noted: <details><summary>Readings</summary> - `pnpm --filter @objectstack/metadata-protocol exec vitest run --maxWorkers=2 src/seed-loader-seed-files.test.ts` at `4bf7600183`: `Tests 19 passed (19)`, lock VERDICT command-exit 0. - `pnpm --filter @objectstack/metadata-protocol typecheck` at `4bf7600183`: exit 0 (same `&&` chain). The test file is in the tsc program (`--listFiles` names it once). - `pnpm --filter @objectstack/metadata-protocol test` at `d294c889` (the merge of `origin/main`; the only later commit types one test literal): `Test Files 269 passed | 3 skipped (272)`, `Tests 21619 passed | 19 skipped (21638)`, after rebuilding the package's dependency closure (`pnpm --filter '@objectstack/metadata-protocol^...' build`). - `pnpm exec eslint` on the three changed source files: exit 0. - Boot measurement: scratch scripts, not committed; numbers in the table above. </details> ## Acceptance notes - **Not reproduced here:** the card's `Knex: Timeout acquiring a connection` refusal. The measured pool waits never exceeded 21 ms, so this run shows the sign-in *slowdown* (5.2 s at worst on 1,500 small files) and its cause, not the refusal. hotclm's 28 s sign-in and its refusals are consistent with longer per-batch stalls (bigger files, more write hooks), but that is inference, not a reading. - **A residual stall at the end of the seed, outside this diff:** one 3.3 s event-loop stall as the seed settles (a sign-in took 3,548 ms then). The profile slice shows `emitSeedSettled`, then `claimSeedOwnershipOnSettle`, then `claimSeedOwnership` (`@objectstack/plugin-security`, the ADR-0138 hand-over on `app:seeded`), with a driver `updateMany` and per-row insert and audit frames. It re-owns the seeded rows in one unbroken chain, scales with rows rather than files, and runs for any seed that overruns the inline budget. Reported to the seat, not fixed here. - The boot measurement used scratch scripts (pool tracer, sign-in probe) that are not committed. ## Verification Gate families derived for this diff by `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` at `4bf7600183` (59 commands), each run on `4bf7600183` with its exit code captured before any pipe. `--ran` reconciliation: `59 derived, 59 run, 0 NOT-MEASURED, 0 UNRUN` (a derived zero: every line records its exit code). <details><summary>59 gate commands, exit codes</summary> - `node scripts/check-adr-0087-registration.mjs --base origin/main`: exit 0 - `node scripts/check-adr-0087-registration.mjs --self-test`: exit 0 - `node scripts/check-changeset-no-major.mjs --base origin/main`: exit 0 - `node scripts/check-closing-keyword-parity.mjs`: exit 0 - `node scripts/check-closing-keyword-parity.mjs --self-test`: exit 0 - `node scripts/check-comment-mask-adoption.mjs`: exit 0 - `node scripts/check-comment-mask-adoption.mjs --self-test`: exit 0 - `node scripts/check-comment-mask-corpus.mjs`: exit 0 - `node scripts/check-dts-emitted.mjs --self-test`: exit 0 - `node scripts/check-empty-changeset.mjs --base origin/main`: exit 0 - `node scripts/check-empty-changeset.mjs --self-test`: exit 0 - `node scripts/check-issue-citations.mjs`: exit 0 - `node scripts/check-plugin-teardown-shape.mjs`: exit 0 - `node scripts/check-plugin-teardown-shape.mjs --self-test`: exit 0 - `node scripts/ci/assert-turbo-remote-cache.mjs --self-test`: exit 0 - `node scripts/ci/plan-test-core-shards.mjs --self-test`: exit 0 - `node scripts/ci/prune-turbo-cache.mjs --self-test`: exit 0 - `node scripts/docs-audit/check-affected-docs.mjs`: exit 0 - `node scripts/docs-audit/check-drift-comment.mjs`: exit 0 - `node scripts/pm/release-rehearsal-clone.mjs --self-test`: exit 0 - `node scripts/release-pending-publish.mjs --self-test`: exit 0 - `pnpm --filter @objectstack/spec run check:duration-unit-keys`: exit 0 - `pnpm check:changeset-gate-self-tests`: exit 0 - `pnpm check:ci-filter-parity`: exit 0 - `pnpm check:closed-ceremony-text`: exit 0 - `pnpm check:cross-package-test-inputs`: exit 0 - `pnpm check:doc-authoring`: exit 0 - `pnpm check:driver-memory-census`: exit 0 - `pnpm check:dts-closure`: exit 0 - `pnpm check:dual-build-cjs-loads`: exit 0 - `pnpm check:durability-log-level`: exit 0 - `pnpm check:engine-double-contract`: exit 0 - `pnpm check:error-status-conformance`: exit 0 - `pnpm check:gitlink-declared`: exit 0 - `pnpm check:issue-citations`: exit 0 - `pnpm check:keyed-text-bounds`: exit 0 - `pnpm check:lean-entry-closure`: exit 0 - `pnpm check:logger-receiver-detach`: exit 0 - `pnpm check:nul-bytes`: exit 0 - `pnpm check:objectql-double-limit`: exit 0 - `pnpm check:org-identifier`: exit 0 - `pnpm check:page-declaration-shape`: exit 0 - `pnpm check:platform-object-tenancy-census`: exit 0 - `pnpm check:pm-changeset-deadline-census`: exit 0 - `pnpm check:published-files`: exit 0 - `pnpm check:query-options-erasure`: exit 0 - `pnpm check:refd-timer-probe`: exit 0 - `pnpm check:registry-log-declared`: exit 0 - `pnpm check:rest-log-spy-declared`: exit 0 - `pnpm check:slot-lookup`: exit 0 - `pnpm check:sourcemap-no-sources-content`: exit 0 - `pnpm check:system-context-census`: exit 0 - `pnpm check:test-source-alias`: exit 0 - `pnpm check:tier-file-adoption`: exit 0 - `pnpm check:type-check-coverage`: exit 0 - `pnpm check:type-check-debt`: exit 0 - `pnpm check:undeclared-dep-imports`: exit 0 - `pnpm check:watch-hint-literal`: exit 0 - `pnpm check:where-matcher`: exit 0 </details> - The first union (on `d294c889`) caught one real red: `check:query-options-erasure`, where the test surface grew 236 to 237 through a stand-in read's `as any`. It was fixed in `4bf7600183` by typing the literal. That union also read `check-plugin-teardown-shape --self-test` as a prerequisite miss (shallow clone; its pinned fixture commit was fetched) and `check:dual-build-cjs-loads` as PREREQUISITE NOT MET. Both are green above on a real measurement. - Declared to CI, not run locally: `test:repo` (the vitest `repo` project; the diff touches none of its tests), the repo-wide `pnpm lint`, the type-check lanes, and the artifact-roster and wide-population families the derivation lists outside its 59. - Dogfood battery: not owed (no `!` in the title, `Clause-②: no` with no arm). --- _Generated by [Claude Code](https://claude.ai/code/session_01QSeKYmQXM3rHJcPnrEPk4B)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 8eef9a8 commit 652da20

4 files changed

Lines changed: 164 additions & 3 deletions

File tree

‎.changeset/26670-seed-file-pool.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
'@objectstack/metadata-protocol': patch
3+
---
4+
5+
fix(metadata-protocol): a seed that stores many `seedFile` values no longer starves sign-ins and other requests while it runs
6+
7+
Clause-②: no
8+
Metadata-protocol: none
9+
10+
A seed that stores many `seedFile(path)` values no longer starves the requests that arrive while it runs. On a synchronous driver (better-sqlite3, the wasm fallback) every step of a seed file's work — the `sys_file` row, the bytes, the claim that commits it — resolves as a microtask, so the claims the seed loader settled after each 200-row insert batch ran as one unbroken chain and a sign-in during a background boot seed waited seconds unread behind it. The connection pool was never the bottleneck; the event loop was. The loader now hands the event loop one turn after each row's files are stored and after each row's claim, and writes a dataset that carries seed files in insert batches of 20 rows instead of 200. Datasets without seed files are unchanged.

‎packages/metadata-protocol/src/seed-files.ts‎

Lines changed: 46 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,32 @@ const PROVENANCE_SHA256 = 'sha256';
6161
/** How many file ids one `describeFiles` read names. */
6262
const DESCRIBE_CHUNK = 500;
6363

64+
/**
65+
* [#26670] Hand the event loop one macrotask turn.
66+
*
67+
* A seed file costs its row about two dozen sequential engine calls: the
68+
* `sys_file` row written `pending`, the bytes, then the claim (the files read
69+
* again, the slot's held files read and released, the row committed), each
70+
* with the write hooks they run. On a synchronous driver (better-sqlite3,
71+
* the wasm fallback) every one of those `await`s resolves as a microtask, so
72+
* a run of rows never reaches the event loop's I/O phase at all: the claims
73+
* the loader settles after each 200-row insert batch ran as one unbroken
74+
* microtask chain, measured at 3–5 s per batch on a boot seeding 1,500 files,
75+
* and every request that arrived meanwhile — a sign-in — waited unread for
76+
* the whole chain. The connection pool was never the bottleneck (no hold
77+
* over 44 ms); the event loop was.
78+
*
79+
* So each row's file work ends here, and a request waits behind one row's
80+
* files, never behind a dataset's. The same cure as the import runner's
81+
* row loop (`@objectstack/core`, `import-runner.ts`), for the same cause.
82+
*/
83+
function yieldToEventLoop(): Promise<void> {
84+
return new Promise<void>((resolve) => {
85+
if (typeof setImmediate === 'function') setImmediate(resolve);
86+
else setTimeout(resolve, 0);
87+
});
88+
}
89+
6490
/** The file-class fields of an object definition, by name. */
6591
function fileFieldsOf(definition: unknown): Map<string, FileField> {
6692
const found = new Map<string, FileField>();
@@ -117,6 +143,11 @@ export class SeedDatasetFiles {
117143
private readonly service: IFieldFileService | null,
118144
) {}
119145

146+
/** Whether the dataset carries seed-file bytes: some record names `seedFile(path)`. */
147+
get carriesFiles(): boolean {
148+
return Object.keys(this.files).length > 0;
149+
}
150+
120151
/**
121152
* The dataset's file handling, or `null` when the seeded object declares no
122153
* file-class field — nothing below runs for such an object.
@@ -297,6 +328,8 @@ export class SeedDatasetFiles {
297328
args.record[field] = def.multiple ? fileIds : fileIds[0];
298329
claims.push({ field, fileIds });
299330
}
331+
// [#26670] The row's files are stored (or kept): let a waiting request in.
332+
if (planned.length > 0) await yieldToEventLoop();
300333
return { ok: true, claims };
301334
}
302335

@@ -308,6 +341,18 @@ export class SeedDatasetFiles {
308341
*/
309342
async settleRow(recordId: string, claims: SeedFileClaims, organizationId: string | null): Promise<string | null> {
310343
if (!this.service || claims.length === 0) return null;
344+
try {
345+
return await this.settleClaims(recordId, claims, organizationId);
346+
} finally {
347+
// [#26670] The loader settles a whole insert batch's rows back to back;
348+
// each row's claim ends with a turn for whatever request is waiting.
349+
await yieldToEventLoop();
350+
}
351+
}
352+
353+
/** {@link settleRow}'s claims, field by field; the first failure ends it. */
354+
private async settleClaims(recordId: string, claims: SeedFileClaims, organizationId: string | null): Promise<string | null> {
355+
const service = this.service!;
311356
for (const { field, fileIds } of claims) {
312357
const known = await this.describe(fileIds, organizationId);
313358
const settled = fileIds.every((id) => {
@@ -319,7 +364,7 @@ export class SeedDatasetFiles {
319364
});
320365
if (settled) continue;
321366
try {
322-
await this.service.claimFiles({ object: this.objectName, recordId, field }, fileIds, organizationId);
367+
await service.claimFiles({ object: this.objectName, recordId, field }, fileIds, organizationId);
323368
this.forget(fileIds, organizationId);
324369
} catch (err) {
325370
return `${this.objectName}/${recordId}.${field} was written, but its seed file(s) ${fileIds.join(', ')} could ` +

‎packages/metadata-protocol/src/seed-loader-seed-files.test.ts‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -427,3 +427,94 @@ describe('seed files: refusals, by row, before the row is written (#25779)', ()
427427
expect(service.describeFiles).not.toHaveBeenCalled();
428428
});
429429
});
430+
431+
/**
432+
* [#26670] A seed that stores many files must leave the event loop to the
433+
* requests that arrive while it runs.
434+
*
435+
* On a synchronous driver (better-sqlite3) every `await` of the seed's
436+
* per-row file work resolves as a microtask, and the fakes here resolve the
437+
* same way, which is the property that matters: without a macrotask turn the
438+
* whole load runs as one unbroken microtask chain, and a request that
439+
* arrives meanwhile (a sign-in, read on the I/O phase) is not even read until
440+
* the seed is done. Measured on `os serve --dev` over sqlite before the fix:
441+
* the event loop stalled 3–5 s per 200-row insert batch while the connection
442+
* pool never held a connection longer than 44 ms.
443+
*/
444+
describe('a seed of many files yields to waiting requests (#26670)', () => {
445+
const ROWS = 60;
446+
// No self-reference, unlike `clm_version`: the loader buffers this object's
447+
// inserts and flushes them in batches, then settles each landed row's claim
448+
// back to back — the path the stall was measured on.
449+
const manyFiles = () => {
450+
const files: Record<string, unknown> = {};
451+
const records = [];
452+
for (let i = 0; i < ROWS; i++) {
453+
files[`fixtures/f${i}.pdf`] = bundled(`%PDF ${i}`);
454+
records.push({ code: `D${i}`, file: seedFile(`fixtures/f${i}.pdf`) });
455+
}
456+
return dataset(records, files, { object: 'clm_doc' });
457+
};
458+
const batchedLoader = (engine: IDataEngine, service: IFieldFileService) => {
459+
const metadata = createMetadata();
460+
(metadata.getObject as any).mockImplementation(async (name: string) =>
461+
name === 'clm_doc' ? { name: 'clm_doc', fields: { code: { type: 'text' }, file: { type: 'file' } } } : undefined);
462+
return new SeedLoaderService(engine, metadata, createLogger(), { getFieldFiles: () => service });
463+
};
464+
465+
it('stores at most one row\'s files and settles at most one row\'s claim between two macrotask turns', async () => {
466+
const { engine } = createEngine();
467+
const { service, stored, claims } = createFieldFiles();
468+
const turns: Array<{ stored: number; claims: number }> = [];
469+
let done = false;
470+
const sample = () => {
471+
turns.push({ stored: stored.length, claims: claims.length });
472+
if (!done) setImmediate(sample);
473+
};
474+
setImmediate(sample);
475+
const result = await batchedLoader(engine, service).load({ seeds: [manyFiles()], config: CONFIG });
476+
done = true;
477+
478+
expect(result.success).toBe(true);
479+
expect(stored).toHaveLength(ROWS);
480+
expect(claims).toHaveLength(ROWS);
481+
let maxStored = 0;
482+
let maxClaims = 0;
483+
for (let t = 1; t < turns.length; t++) {
484+
maxStored = Math.max(maxStored, turns[t].stored - turns[t - 1].stored);
485+
maxClaims = Math.max(maxClaims, turns[t].claims - turns[t - 1].claims);
486+
}
487+
// A turn ran while the load did, and the first one landed before the last
488+
// file was stored: the load did not run through in one chain.
489+
expect(turns.length, 'no macrotask turn ran while the seed ran').toBeGreaterThan(1);
490+
expect(turns[0].stored,'the first macrotask turn came only after the seed stored every file').toBeLessThan(ROWS);
491+
expect({ maxStored, maxClaims }).toEqual({ maxStored: 1, maxClaims: 1 });
492+
});
493+
494+
it('serves an unrelated read issued during the seed within one row\'s file work', async () => {
495+
const { engine } = createEngine();
496+
const { service, stored, claims } = createFieldFiles();
497+
let readAt: { stored: number; claims: number } | undefined;
498+
// A request read on the I/O phase, as a sign-in's user lookup is.
499+
setImmediate(() => {
500+
const issuedAt = { stored: stored.length, claims: claims.length };
501+
void engine.find('sys_user', { where: { email: 'admin@example.com' } }).then(() => {
502+
readAt = { stored: stored.length - issuedAt.stored, claims: claims.length - issuedAt.claims };
503+
});
504+
});
505+
await batchedLoader(engine, service).load({ seeds: [manyFiles()], config: CONFIG });
506+
expect(readAt, 'the read never completed').toBeDefined();
507+
expect(readAt!.stored + readAt!.claims, 'files stored or claimed while the read waited').toBeLessThanOrEqual(1);
508+
});
509+
510+
it('writes a dataset that carries seed files in insert batches of at most 20 rows', async () => {
511+
const { engine } = createEngine();
512+
const { service } = createFieldFiles();
513+
await batchedLoader(engine, service).load({ seeds: [manyFiles()], config: CONFIG });
514+
const sizes = (engine.insert as any).mock.calls
515+
.filter(([object]: [string]) => object === 'clm_doc')
516+
.map(([, data]: [string, unknown]) => (Array.isArray(data) ? data.length : 1));
517+
expect(sizes.reduce((a: number, b: number) => a + b, 0)).toBe(ROWS);
518+
expect(Math.max(...sizes)).toBeLessThanOrEqual(20);
519+
});
520+
});

‎packages/metadata-protocol/src/seed-loader.ts‎

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1279,7 +1279,7 @@ export class SeedLoaderService implements ISeedLoaderService {
12791279
const writeResults: BulkWriteRowResult[] = await bulkWrite(
12801280
batch.map(b => b.record),
12811281
{
1282-
batchSize: SeedLoaderService.BULK_BATCH_SIZE,
1282+
batchSize: insertBatchSize,
12831283
writeBatchPartial: async (rows, { attempt }) => {
12841284
let toInsert = rows;
12851285
let existing = new Map<string, any>();
@@ -1399,6 +1399,18 @@ export class SeedLoaderService implements ISeedLoaderService {
13991399
organizationId: config.organizationId ?? this.fallbackOrgId ?? null,
14001400
dryRun: config.dryRun === true,
14011401
});
1402+
/**
1403+
* [#26670] Rows per buffered-insert batch. A row holding seed files costs
1404+
* its insert the file-reference hooks, about 7 ms a row on sqlite against
1405+
* well under 1 ms for a plain row, and one `insertMany` is one unbroken
1406+
* run on a synchronous driver: 200 such rows held the event loop for about
1407+
* 1.5 s, and a sign-in waited behind it. A dataset that carries seed files
1408+
* is written in small batches instead, so a request waits behind a few
1409+
* rows; every other dataset keeps the full batch.
1410+
*/
1411+
const insertBatchSize = datasetFiles?.carriesFiles
1412+
? SeedLoaderService.SEED_FILE_BATCH_SIZE
1413+
: SeedLoaderService.BULK_BATCH_SIZE;
14021414
/** Record index → the files that row's slots must own once it lands. */
14031415
const fileClaimsByRecordIndex = new Map<number, { claims: SeedFileClaims; organizationId: string | null }>();
14041416
/**
@@ -2119,7 +2131,7 @@ export class SeedLoaderService implements ISeedLoaderService {
21192131
} else {
21202132
// Insert: buffer for the batched flush rather than writing now.
21212133
pendingInserts.push({ recordIndex: i, externalIdValue, record });
2122-
if (pendingInserts.length >= SeedLoaderService.BULK_BATCH_SIZE) {
2134+
if (pendingInserts.length >= insertBatchSize) {
21232135
await flushPendingInserts();
21242136
}
21252137
}
@@ -2938,6 +2950,9 @@ export class SeedLoaderService implements ISeedLoaderService {
29382950
/** Rows per batch for the buffered-insert flush. See framework#2678. */
29392951
private static readonly BULK_BATCH_SIZE = 200;
29402952

2953+
/** [#26670] Rows per buffered-insert batch for a dataset that carries seed files (see `insertBatchSize`). */
2954+
private static readonly SEED_FILE_BATCH_SIZE = 20;
2955+
29412956
/**
29422957
* Decide what {@link loadDataset}'s non-self-referencing (batched) path
29432958
* should do with a record — mirrors {@link writeRecord}'s mode/existing

0 commit comments

Comments
 (0)