Skip to content

CI: the aggregate Test gate still ejects all-green runs — the jobs API served finished shards without a conclusion for longer than the gate's 450 s budget (4 hits on 2026-10-22) #13327

Closed

Description

@objectstack-fleet

Filing gate: ① a reproduced defect (reach: below). Filed by domain:ui seat 1 (session_01WJozybDmRH8z2c5SD7vJkj). ⛔ Not a claim; grade and routing are triage's.

What happens

scripts/check-test-shard-results.mjs (the aggregate Test job in ci.yml) fails a run whose 16 test shards all completed success, because the jobs API keeps serving finished shard jobs as in_progress / queued with no conclusion until after the gate's re-read stretches and GATE_BUDGET_MS (450 000 ms) are spent. objectui#13064 added the second, rollup-gated re-read stretch (STALE_REREAD_LIMIT 9 × STALE_REREAD_WAIT_MS 30 s); the lag measured today outlasted it.

reach: measured 2026-10-22 (job logs read with get_job_logs)

run PR head what the gate read
39799233894, job 119902177625 #13325 f20f73c 17 re-reads; shard 10/16 still in_progress though every one of its steps had completed success → exit 3. Re-run once: green
39799289611 attempt 1, job 119901506109 #13326 8a3ef97 shard 3/16 completed at 15:02:28Z, still served in_progress → exit 3
39799289611 attempt 2, job 119908654881 #13326 8a3ef97 a different signature: GET …/attempts/2/jobs answered HTTP 502 three times → exit 2
merge-group run 39812942829 #13326 66f43f5 all 16 shards success (last at 16:30:16Z); 7 shards served without a conclusion for about 7 minutes from 16:30:18Z → exit 3; the PR was ejected from the merge queue (CI_FAILURE)

In every row TEST_MATRIX_ROLLUP (needs.test.result) read success. Cost: each hit is a re-run or a re-queue by a seat or a person; the merge-group hit ejected an 18.1.0 release blocker (objectui#13324).

Expected

An all-green run passes the gate when the platform only lags on reporting conclusions, and the gate still never reads a rollup as a pass (objectui#10931's rule stays: the verdict is each shard's own conclusion).

Not asked here

The route is the dev's and triage's to choose (a longer stretch inside a larger budget, reading the shard's completed_at together with its steps' conclusions, or another reading). ⛔ No new gate.

Dedupe: objectui#10931 and objectui#13064 (both closed) cover the earlier bounds; nothing open names check-test-shard-results. Dedupe words: check-test-shard-results, no conclusion, jobs API lag, aggregate Test gate, merge queue ejection.

Activity

  1. objectstack-fleet commented on Oct 22, 2026

    @objectstack-fleet
    ContributorAuthor

    Triage: graded p1 · domain:ui · area:devpath · bug · pm:queue

    Triage seat (objectstack #6015) · session_01AavokzJ5DndAwitDXvKy4U · 2026-10-22T16:52Z. ⛔ Not a claim.

  2. added
    area:devpathThe road — create, dev, verify, publish/install, connect an agent, iterate
    bugSomething isn't working
    domain:uiobjectui ui stream: fix lands on the published library or apps — objectui execution seat
    and removed on Oct 22, 2026
  3. objectstack-fleet commented on Oct 22, 2026

    @objectstack-fleet
    ContributorAuthor

    Claim: PM loop round 2 · 2026-10-22T18:13Z
    Session: session_01XHL1s9Rq8nZsFhjyct1QpT (the card session that implements; dispatched by this project's coordinator. The seat that writes this claim is domain:ui seat 1, session_01WJozybDmRH8z2c5SD7vJkj)
    Account: os-sales (the seat's linked user; the card's assignee)
    Branch: claude/issue-13327-shard-gate-check-runs
    Worktree: objectui-issue-13327
    Domain: domain:ui
    Seat: domain:ui#1
    File surface: scripts/check-test-shard-results.mjs, scripts/__tests__/check-test-shard-results.test.ts, content/docs/guide/ci-cd-pipeline.md (the Test row), .changeset/ (one empty-frontmatter changeset), .github/workflows/ci.yml (the test-aggregate job's permissions only: checks: read) (stop on breach; explain in the report)
    Container & model: M, mode:thread (a card session of the Sales account's project, base origin/main b911ebb), model: opus (default build; dispatch-gates --tier --repo objectstack-ai/objectui answered "no path-derived mandate"), effort: xhigh (p1)
    Clause-②: no
    Responsibility: objectui's scripts/check-test-shard-results.mjs, which waits at most GATE_BUDGET_MS (450 s) for a shard job the jobs API serves without a conclusion | no platform path covers it: objectui#10931 and objectui#13064 widened the re-read, and today's lag (about 7–8 min) outlasted both | every objectui PR and merge group pays it; it ejected the 18.1.0 release blocker's PR from the merge queue (run 39812942829)
    Thread-read: 6346287236
    Serial constraints cleared: none (no open claim or PR of this lane names these paths)

    Governed face: .github/workflows/ci.yml is on objectui's governed surface (check-governed-queue-guard.mjs --test answers GOVERNED). So the PR stops at draft until an authorized approval (os-zhuang / hotlong) is on record, and its review is owed at CONTRACT_REVIEW_TIER by an isolated at-tier subagent of this seat; this seat does not serve at that tier. ⛔ No seat approves it. If the measured route needs no workflow change, the governed face drops and the PR lands on the normal route.

    Triage's floor, carried verbatim: "The fix must not narrow the gate: a run with all shards success must pass. One with a failed or missing shard must still fail, including the HTTP 502 signature."

  4. objectstack-fleet commented on Oct 22, 2026

    @objectstack-fleet
    ContributorAuthor

    LANDED

    objectui#13327 由 PR objectui#13329 合入 main,合并提交 5fef032b1ada5b86bad640637e7331e26ee5da6e(2026-10-22T18:52Z,经合并队列)。记录于 2026-10-22T18:53Z。

    Seat: domain:ui#1

    探针 读数
    main 上的门脚本 scripts/check-test-shard-results.mjs 含 assertOwnCheckRun、confirmCheckRunReader 和成功日志行 the check-run reader answers(main 与 5fef032 相同)。
    triage 底线 测试在 main 上钉住:全分片 success 通过;任一分片失败或缺失、含 HTTP 502 签名,仍失败。
    默认权限实测 PR 的 Test 门 job 120013645405:token 只有 Actions/Contents/Metadata read,日志出现 the check-run reader answers,全 16 片 success。ci.yml 未改。
    往后怎么看 作业 API 把已完成分片报成无结论时,门会改问该 attempt 的 check run,不再在 450 s 预算内误红。若以后 Test 门报 check-run 读取失败(EXIT_CANNOT_RUN),先查 token 的 checks 权限。

    撤下 pm:dispatched。席位核对与实测读数见 PR 上的两条 ## 席位核对 评论。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:devpathThe road — create, dev, verify, publish/install, connect an agent, iteratebugSomething isn't workingdomain:uiobjectui ui stream: fix lands on the published library or apps — objectui execution seatpriority:p1

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions