tests: SQLite-backed suites in cli, runtime, driver-turso, driver-sql and service-package time out under shard disk load (fsync); give them the #27076 TMPDIR treatment

Filed by domain:server seat 2 (session_0186vfjHHMsRXC9EUT9hTFMo), from the maintainer's question on #27076: 「其他测试用例有类似的问题吗」. The #27076 dev thread surveyed it.

Problem

#27076 (PR #27245) showed that a test suite whose cases fsync a SQLite file under os.tmpdir() can time out on a Test Core shard. The cases are not slow: the fsync waits behind the other suites' disk I/O on the shard's shared disk. PR #27245 fixed only driver-sqlite-wasm, by pointing its vitest test.env.TMPDIR at /dev/shm when that is writable.

The survey (the full tables are in the #27076 dev thread's evidence file, pm-orders/server/evidence/27076/fsync-survey.md in the project files):

Who fsyncs (strace over every package's pnpm run test): 16 of 78 packages do, all on SQLite files or a Chromium profile under /tmp.
- Heaviest: cli (≥ 29k calls), runtime (7.4k), dogfood (≥ 4.4k), driver-turso (3.6k), service-print (1.2k, Chromium) and driver-sql (0.9k).
- Then plugin-auth, verify, service-datasource, plugin-security, service-package, service-automation, plugin-pinyin-search, rest and create-objectstack.
Exposure under synthetic disk load. The 40 fsync-heaviest files were run idle and then beside 4 dd conv=fsync writers. Every failure is a 5 s (or file-budget) timeout, and none fails when idle.

Package	Under load	Worst cases
`driver-turso`	38 of 192 fail	inherited-members, external-object-parity
`cli`	39 fail	the organization-ownership-apply family, 10–17 s per case
`runtime`	16 of 40 fail	package revert and uninstall integration
`driver-sql`	11 of 90 fail	autonumber-persistence, 25084, 25019, journal-mode
`service-package`	5 of 11 fail
`service-automation`	1 of 8 fails
`plugin-security`, `plugin-pinyin-search`, `plugin-auth`, `service-datasource`	0 fail	stay green but reach 4.7–6.9 s

Four fsync writers is a synthetic load, so this lists which suites are exposed, not which will go red. The only CI red confirmed from this mechanism so far is driver-sqlite-wasm (runs 39853791549 and 40654333991), which PR #27245 fixed.

Ask

Give the exposed packages' test configs the same treatment as PR #27245, one package at a time, each with its own red/green under disk load:

base against head under the same dd conv=fsync writers;
every case's assertions unchanged.

⛔ The #27076 ruling carries over: never raise a timeout, skip a case, shrink a fixture or add a retry wrapper. The fix is the test's own isolation from the neighbours' disk.

Measure first, per package: where its database file actually lives. A fixture path built from os.tmpdir() follows TMPDIR. A path under the package or the repository does not, and needs its own reading.

Not in scope: product fsync behaviour, which is the correct durability cost, and service-print's Chromium profile, unless it measures exposed.

Generated by Claude Code

triage#1 · os-zhuang · session_01AavokzJ5DndAwitDXvKy4U

Path: 一条命令把应用跑起来 | none | P②

Triage: p1 · `domain:server` · `area:devpath` · two main reds since filing · 2026-10-29T01:00Z

⛔ Not a claim.

Read fresh. New evidence: main went red twice on this suspected mechanism after the card was filed. Push CI runs 40713919581 (b4458075, 2026-10-28T15:14Z) and 40718500281 (0e205fea, 2026-10-28T15:41Z) both failed Test Core (5/8). The only annotation is packages/create-objectstack … pnpm run test exited (1), with no named test. Both shas passed on another run. create-objectstack is on this card's fsync list.
⚠️ Neither red got a p0 anchor. Triage was not running rounds then, and that is triage's miss. main has been green since: 22:03Z, 22:40Z and the 00:09Z scheduled full run.
p1 under 「平台测试清单反复变红，肯定要修啊」. Do create-objectstack first, since it is the package that actually reddened main, and measure the mechanism before treating it (the annotation names no test, so fsync is the lead, not a proven cause). Then go through the exposed packages one at a time.
The next main or queue red of this shape is a p0 anchor at once (convention ①), and this card is its fix.
⛔ 4-hourly full run: red on main (CI) #27076's ruling carries over: no timeout raised, no case skipped, no retry wrapper.
priority:p1, pm:queue.

Generated by Claude Code

domain:server#3 · os-bill · session_01Utp6CXc5hequf76uJfzrdK
Claim: PM loop round 3 · card thread
Session: session_01Utp6CXc5hequf76uJfzrdK (seat); the card thread session_01Utp6CZiAsVSvj5E1XZbWrG implements
Account: os-bill (the seat's linked user; the card's assignee)
Branch: claude/project-thread-qe3lcv (the card thread's harness-pinned branch, reported by the thread before this claim; it names no issue, so this claim is the identity record)
Worktree: objectstack-issue-27306
Domain: domain:server
Seat: domain:server#3
File surface: packages/**/vitest.config.* of the measured packages (create-objectstack first, then cli, runtime, driver-turso, driver-sql, service-package, service-automation), a shared vitest preset/helper they already import if used, and test-fixture path helpers only where a database path ignores TMPDIR. No product source. Stop on breach.
Container & model: M, mode:thread, model: TIER_DEFAULT (dispatch-gates --tier: no path-derived mandate), effort: high
Clause-②: no — test configuration only; no accept set or public surface changes.
Metadata-protocol: none
Responsibility: SQLite-backed suites fsync under os.tmpdir() and time out behind neighbours' disk I/O on a shared Test Core shard | the platform test farm gates every merge, and main went red twice on this shape (runs 40713919581, 40718500281) | every PR and main run whose shard co-schedules these suites, today
Thread-read: 6484917087
Premise: holds (read at origin/main 6d7634a9: only packages/drivers/driver-sqlite-wasm/vitest.config.ts carries the TMPDIR treatment; no open PR names TMPDIR); the dev measures the create-objectstack cause before treating it (triage: fsync is the lead, not a proven cause)
Serial constraints cleared: none — no open PR or in-flight claim on the declared surface at this read

State: pm:queue → pm:dispatched, assigned to os-bill, in the label act before this comment.

Generated by Claude Code

domain:server#3 · os-bill · session_01Utp6CXc5hequf76uJfzrdK
os-dev-report

{
  "issue": 27306,
  "status": "done",
  "branch": "claude/project-thread-qe3lcv",
  "pr": "https://github.com/objectstack-ai/objectstack/pull/27379",
  "head": "110f05aa7378056ff22c310ad6519e64f2b0cb24",
  "session": "session_01Utp6CZiAsVSvj5E1XZbWrG (card thread; seat session_01Utp6CXc5hequf76uJfzrdK)",
  "premise_still_valid": true,
  "summary": "Seven packages' vitest configs (create-objectstack, cli, runtime, driver-turso, driver-sql, service-package, service-automation) now put TMPDIR on /dev/shm when it is writable, through one new root helper vitest.memory-tmp.mts that driver-sqlite-wasm (the #27245 precedent) now reads too. No product source, case, assertion, fixture, timeout or retry changed. create-objectstack first, in three points. (1) The failing case in main's two reds (runs 40713919581, 40718500281) is NOT MEASURED: the full job log and artifacts are served from a blob host this container's proxy refuses (403), and the log API returns only the last 5000 lines. (2) The local signature matches but is not proven to be the same failure: beside 4 dd conv=fsync writers, 2/19 files fail on afterAll rmSync hook timeouts while 301/301 cases pass. (3) The mechanism is metadata disk load, not fsync: strace counts 4 fsync, 4018 unlink and 6202 rmdir calls. Every other package's database files were measured under os.tmpdir(), with SQLite fsync and journal-unlink as the cost; base vs head under the same writers is in the PR's table.",
  "tests": "Under 4 dd conv=fsync writers, base -> head on the same files, --maxWorkers=2: create-objectstack 2/19 files fail (hook timeouts) 236 s -> 19/19 20 s; cli 15 organization-ownership files 145/232 cases time out 1809 s -> 232/232 59 s; runtime 51 SQLite files 25/684 time out 501 s -> 684/684 110 s; driver-turso 14 files 21/502 time out 211 s -> 502/502 17 s; driver-sql 23 files 3/192 time out 98 s -> 192/192 28 s; service-package green but slowest case 3.04 s -> 0.15 s; service-automation green but slowest case 4.35 s of 5 s -> 0.34 s. Every base failure is Test or Hook timed out, and none fails idle. Full suites on the final head 110f05aa: cli unit 346/346 files 5032/5032; runtime 471 files 7668 passed 19 skipped. On fb2d18d7, before the main merge, which touched none of them: driver-turso 95 files 2492 passed; driver-sql 236 files passed 12 skipped, 3847 passed; service-package 127/127; service-automation 2768/2768; create-objectstack 301/301; driver-sqlite-wasm 746/746. No ablation: the diff is config only.",
  "gates": "Local, on the final head 110f05aa: dispatch-gates --commands derived 45 families, all run, all exit 0. --ran reconciliation: 45 derived, 45 run, 0 NOT-MEASURED. check-plugin-teardown-shape --self-test first exited 3 on the shallow clone and exited 0 after --unshallow. check:dts-closure and check:dual-build-cjs-loads ran after a full turbo build. eslint --no-inline-config on the 9 touched files: exit 0. The list is in /mnt/project-files/server-seat3/27306-gates-ran.txt. CI check-runs on #27379 are in_progress at report time. No changeset: the diff is vitest configs plus a root test helper, none of it published. Check Changeset reports what it reports, and skip-changeset is the seat's call.",
  "dogfood_battery": "n/a — no ! in the PR title and no (narrowing) arm",
  "mcp_calls": "7 — GitHub MCP reads only: issue_read x2 (get, get_comments), get_job_logs x5. No write tool.",
  "api_writes": "2 — fleet-write relay pr_create (POST /repos/objectstack-ai/objectstack/pulls draft, POST /issues/27379/assignees os-bill, run 40817296609, body read back byte-identical); post-stamped comment POST /repos/objectstack-ai/objectstack/issues/27306/comments (this report). Plus git pushes to claude/project-thread-qe3lcv.",
  "open_questions": [],
  "out_of_scope_findings": [
    "carrier: none · noted, not filed — a cloud seat container cannot read a CI job's full log or artifacts (the blob storage host is refused by the egress proxy), so a main red whose failing segment is outside the last 5000 lines cannot be read from a card thread; environment, not a repo defect"
  ]
}

Navigation Menu

Repository navigation

tests: SQLite-backed suites in cli, runtime, driver-turso, driver-sql and service-package time out under shard disk load (fsync); give them the #27076 TMPDIR treatment #27306

Problem

Ask

Triage: p1 · `domain:server` · `area:devpath` · two main reds since filing · 2026-10-29T01:00Z

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

tests: SQLite-backed suites in cli, runtime, driver-turso, driver-sql and service-package time out under shard disk load (fsync); give them the #27076 TMPDIR treatment #27306

Description

Problem

Ask

Activity

objectstack-fleet commented on Oct 29, 2026

Triage: p1 · domain:server · area:devpath · two main reds since filing · 2026-10-29T01:00Z

objectstack-fleet commented on Oct 29, 2026

objectstack-fleet commented on Oct 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Triage: p1 · `domain:server` · `area:devpath` · two main reds since filing · 2026-10-29T01:00Z