perf(lanes): measure work-resolve-issue's idle time to a transient re-fire, 7 days before and after #12804 #13073

Activity

1. kyle-sexton

   added

   priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.

   host-runNeeds an agent run or builder session on a host; not a person or the issue lane.Needs an agent run or builder session on a host; not a person or the issue lane.

   on Nov 4, 2026
2. github-actions commented on Nov 4, 2026

   

   github-actionsBot

   on Nov 4, 2026 – with GitHub Actions

   Last edited by melodic-automation-lanes

   Contributor

   More actions

   Proposed work class: work-class: read-only
   Basis: automated intake triage; the lanes App applied this class.

   Size: Small
   Complexity: Low
   Uncertainty: Known
   Flags: none
   Blocked by: none

   Understood: this asks for the median idle time from a transient failure to the re-fire, measured over 7 days before and 7 days after #12804 merged, with the result posted here. It is a measurement over time, so it is host work. The after-window cannot finish before 2026-11-10T16:07Z. The scope is clear and nothing is missing. #13073 is related to #12804's follow-up work but is not a duplicate of any open issue.

   _{Automated triage, applied by the lanes App.}
3. github-actions

   added the

   Task

   issue type on Nov 4, 2026
4. melodic-automation-lanes

   added

   status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.

   work-class: read-onlyAudits, research, reports. No repository mutation; tracker and queue writes only.Audits, research, reports. No repository mutation; tracker and queue writes only.

   on Nov 4, 2026
5. melodic-automation-lanes

   set

   Work status

   to

   ready

   on Nov 4, 2026
6. melodic-automation-lanes

   added

   status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.A maintainer confirmed the item is real and in scope; ready to be worked.

   and removed

   status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.

   on Nov 4, 2026
7. melodic-automation-lanes commented on Nov 4, 2026

   

   melodic-automation-lanesBot

   on Nov 4, 2026

   Contributor

   More actions

   Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 1 (tracker-output: "The median idle time from a transient failure to the re-fire is measured for ..."); no done-when item is left for the lane, so it goes to the host queue (host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).
8. melodic-automation-lanes commented on Nov 4, 2026

   

   melodic-automation-lanesBot

   on Nov 4, 2026

   Contributor

   More actions

   The latest triage verdict holds nothing: admission moved the issue to status: confirmed; its comment says why.
9. melodic-automation-lanes

   changed

   Work status

   to

   confirmed

   on Nov 4, 2026
10. kyle-sexton commented on Nov 4, 2026

    

    kyle-sexton

    on Nov 4, 2026

    MemberAuthor

    More actions

    Claimed by poteto (desk session 789d5794). The 7-day BEFORE and an interim AFTER (from the change's merge until now) get posted here today, with the method and the script path, so the final AFTER is the same command run over the full window once it closes. That final AFTER, and this issue's close, can't come before the 7-day after window ends.
11. kyle-sexton commented on Nov 4, 2026

    

    kyle-sexton

    on Nov 4, 2026

    MemberAuthor

    More actions

    Before measurement and interim after: idle time from a stranded failure to work-unblock's re-fire

    The changed paths almost never fired. In the 7 days before #12804, and in the 23 hours since, the paths whose wait #12804 shortened fired zero times: the no-run first re-fire, the kill-switch backoff after trip 4, and the no-run and failed-mint backoff re-fires. Every transient re-fire measured went through the kill-switch clamp path, and #12804 did not change that path. So far the change has nothing to act on. A before/after delta on the pooled "transient" group would come from other changes, not from #12804.

    Method

    Script: .work/gh-monitor/trends/13073-measure.py <from> <to> (hub-local, run with python3 -I). Line numbers below are on origin/main at e5554b8.

    • Config under test: docs/conventions/pr-pipeline.yaml:1082-1094 (stranded-backoff-hours: no-run 1, no-report 3, failed-again 3, failed-mint 1, kill-switch 1). git log -G shows perf(lanes): configure work-unblock's stranded re-fire backoff per cause and shorten it for transient causes #12804 (ea71ee1) is the only commit since 2026-10-26 to touch stranded-backoff-hours, reset-clamp-min or limit-stops-per-issue.
    • Which re-fires perf(lanes): configure work-unblock's stranded re-fire backoff per cause and shorten it for transient causes #12804 changes:
      • no-run first re-fire: waits hoursFor["no-run"] (.github/scripts/work-unblock/label-unblocked.mjs:1763-1782), 3 h before, 1 h after.
      • kill-switch: hoursFor["kill-switch"] applies only from trip limit-stops-per-issue + 1 = 4. Trips 1-3 wait ended + clamp, where clamp is 0 at the broker check and otherwise reset-clamp-min = 1 h (label-unblocked.mjs:1817-1818, .github/scripts/work-resolve-issue/limits.mjs:11,17). That clamp path is unchanged.
      • failed-mint: the per-cause hours apply only to the backoff after 3 re-fires on one workflow key (label-unblocked.mjs:1836). A first failed mint is clamp-timed, and its record carries the generic failed-run text.
      • backoff re-fires: backoffDue doubles the cause's hours (label-unblocked.mjs:415, 1744-1758).
    • Re-fire event: a bot (melodic-automation-lanes) unlabel then relabel of agent-ready no more than 120 s apart (label-unblocked.mjs:2070-2095 posts or edits the record, removes the label, then adds it back). Each pair is attributed by the bot comment created or edited in the 180 s before it:
      • <!-- work-unblock refired KEY breaker --> is kill-switch (label-unblocked.mjs:1284, report-outcome.mjs:862).
      • <!-- work-unblock refired KEY --> with a strandedCause text is no-run, no-report or failed-again (label-unblocked.mjs:1273-1277). With the generic FAILED_CAUSE text it is a first failed run or a first failed mint; the text cannot tell these two apart.
      • <!-- work-unblock infrastructure KEY -->, created or edited (userContentEdits), is a backoff re-fire, with its cause taken from its text (MINT_CAUSE at label-unblocked.mjs:1279).
      • The workflow-stale text (report-outcome.mjs:866) is the run re-firing itself straight away. It is out of scope and reported apart.
      • A relabel after an outcome= marker (a limit-stop restart) or a recurrence relabel is out of scope.
    • Clock start:
      • kill-switch: the refused run's end, as the record states it ("that run ended at ...").
      • no-run: the last agent-ready add before the pair.
      • every other cause: the end of the issue's newest completed work-resolve-issue run (event=issues, title ending #N) created after that add and ended before the re-fire.
    • Inclusion: a re-fire counts in a window when its clock start is in [from, to) and the re-fire is before to. Re-fires that straddle a window edge are dropped and counted.
    • Incident: re-fires whose clock-to-re-fire interval overlaps the GitHub incident of 2026-11-04T09:54Z to 11:44Z are excluded, and counted.
    • Reads: GraphQL search repo:… is:issue "to start a fresh run" updated:>=<from − 30 h>, then each issue's timeline (comments, labeled and unlabeled events) since from − 30 h. Lane runs are listed per UTC day, only for the days the run-clock rows need.
    • Statistics: median, and p90 by nearest rank, of idle hours.

    BEFORE: 2026-10-27T16:07Z to 2026-11-03T16:07Z

    group	path	median idle h	p90 idle h	n
    transient	no-run, first re-fire (changed 3 h -> 1 h)	-	-	0
    transient	kill-switch, clamp path (trip 1-3, unchanged)	1.27	4.98	206
    transient	kill-switch, backoff path (trip 4+, changed)	-	-	0
    transient	backoff re-fire: no-run (changed)	-	-	0
    transient	backoff re-fire: failed-mint (changed)	-	-	0
    control	no-report, first re-fire (3 h)	3.27	5.25	8
    control	failed-again, first re-fire (3 h)	-	-	0
    control	backoff re-fire: no-report / failed-again (3 h)	-	-	0
    control	generic first failed run or first failed mint (clamp)	1.71	4.67	74
    out of scope	workflow-stale immediate re-fire	0.28	0.64	14
    transient	pooled	1.27	4.98	206
    control	pooled	1.81	4.67	82

    Issues read: 311. Excluded: 31 that straddle a window edge, 36 limit-stop restarts or recurrence relabels, and 4 relabel pairs with no record. Of the counted rows, 8 have no matching lane run, so their clock is the label add.

    Interim AFTER: 2026-11-03T16:07Z to 2026-11-04T15:15Z (interim, about 23 h of 7 days)

    group	path	median idle h	p90 idle h	n
    transient	no-run, first re-fire (changed 3 h -> 1 h)	-	-	0
    transient	kill-switch, clamp path (trip 1-3, unchanged)	0.26	1.16	22
    transient	kill-switch, backoff path (trip 4+, changed)	-	-	0
    transient	backoff re-fire: no-run (changed)	-	-	0
    transient	backoff re-fire: failed-mint (changed)	-	-	0
    control	no-report, first re-fire (3 h)	-	-	0
    control	failed-again, first re-fire (3 h)	-	-	0
    control	backoff re-fire: no-report / failed-again (3 h)	-	-	0
    control	generic first failed run or first failed mint (clamp)	0.29	0.32	3
    out of scope	workflow-stale immediate re-fire	0.72	1.35	23
    transient	pooled	0.26	1.16	22
    control	pooled	0.29	0.32	3

    Interim n = 25 in scope (22 transient, 3 control). Excluded: 15 that straddle a window edge, 1 that overlaps the incident, and 2 limit-stop restarts. The interim window is right-censored: a failure from the last hours that has not been re-fired yet is missing, which biases it low. The 18 rows with no matching lane run are the workflow-stale rows, whose run ends after its own re-fire.

    Reading

    • No re-fire in either window went through a path perf(lanes): configure work-unblock's stranded re-fire backoff per cause and shorten it for transient causes #12804 changed, so its effect is not observable yet. The kill-switch clamp median fell from 1.27 h to 0.26 h, but that path does not read stranded-backoff-hours. Candidate causes, UNVERIFIED: fix(ci): end a check-mode kill-switch refusal as a stop, not a failure #12759 (40b756e, 2026-11-03T07:14Z, a check-mode kill-switch refusal ends as a stop), feat(pr-pipeline): sweep work-unblock when an issue lane run completes #12531 (00a7507, 2026-11-02T23:17Z, a sweep kicked when a lane run completes), and how long each breaker trip lasted in each window. feat(pr-pipeline): sweep work-unblock when an issue lane run completes #12531 also changes the sweep cadence inside BEFORE.
    • No no-run record carried a "1 hour" or "3 hours" text in either window. The config check through the record text has nothing to read yet.
    • If the final AFTER still shows n = 0 on the changed rows, the done-when cannot be met as written. The before/after then shows that the transient causes perf(lanes): configure work-unblock's stranded re-fire backoff per cause and shorten it for transient causes #12804 targets do not occur at a measurable rate, and the pooled transient delta must not be credited to perf(lanes): configure work-unblock's stranded re-fire backoff per cause and shorten it for transient causes #12804.

    Final AFTER (after 2026-11-10T16:07Z)

    python3 -I .work/gh-monitor/trends/13073-measure.py 2026-11-03T16:07:00Z 2026-11-10T16:07:00Z
    

    Run it with no --cache, so it reads fresh. Expected cost: about 20 GraphQL calls and about 30 run-listing calls.

    UNVERIFIED

    • "Listed human" for the kill-switch trip floor is approximated: any non-bot account that added agent-ready, or any outcome= marker, within the 30 h lookback. With no floor inside the lookback, every breaker record in the lookback counts. That can only push a re-fire toward the trip 4+ path, and none landed there.
    • The cause of a backoff re-fire comes from the infrastructure comment's current text, not from the text of the edit made at that re-fire. No backoff re-fires occurred, so this did not affect these numbers.
    • The 4 BEFORE relabel pairs with no record were not inspected one by one.