Repository navigation
Concise skills rollout: dometrain (3 skills, 1 model-invoked) #6605
Description
Part of #6476. Rewrite dometrain's skills to the skill criteria in one draft PR, following the per-plugin runbook (/playbooks:skill-authoring, reference/rewrite-a-plugin.md). The runbook and the rewrite protocol it cites decide what ships; this issue holds only what is specific to the plugin.
The plugin today
As of origin/main on 2026-10-24:
- Skills:
grounding(model-invoked);setup,sync(user-invoked). - Outcome eval case directories under the plugin: 0. Probe files under
probes/: 0. - Synced skills here change only in their house-owned parts (umbrella decision).
Done when
-
/skill-quality:check plugins/dometrainon the PR head reports no FAIL that the run on main does not. -
/evals:validatepasses onplugins/dometrain's eval suite at the PR head. - Each model-invoked skill has a probe file:
plugins/dometrain/probes/grounding.json. - The PR body holds a per-skill decision table, description and body decided separately, with the before and after numbers behind each row.
- The PR body names every skill or case left unmeasured, with the reason.
- The plugin's version or changelog entry is in the PR, per the plugin's release mode.
Eval spend and review
- Eval passes within prepaid API credit are pre-approved (owner decision Q46, 2026-10-24), at the
--max-cost-usd 5per-pass ceiling set on #6476 on 2026-10-18. - No person gates the PR: the owner's 2026-10-24 decision (C12) leaves only an intent mismatch during the build or a security issue at review as stops.
- Eval cases run without a Dometrain account; a case that needs a live third-party call is named unmeasured in the PR, never run against a paid outside service.
Generated by Claude Code
Metadata
Metadata
Assignees
Labels
Type
Fields
No fields configured for Task.
Activity
github-actions commentedon Oct 7, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: none
Understood: rewrite the three
dometrainskills (grounding,setup,sync) to the skill criteria in one draft PR, following the per-plugin runbook. That includes a probe file forgrounding, eval validation, a per-skill decision table in the PR body, and a version or changelog entry. The comment from the owner's session says the earlier hold was lifted. The issue gives the eval spend ceiling and says cases run without a Dometrain account. Nothing is missing, and I found no likely duplicates. Class is proposed for a maintainer to confirm.Automated triage, applied by the lanes App.
kyle-sexton commentedon Oct 19, 2026
This was generated by AI during triage.
Held: per the owner decision on #6476 (2026-10-18), this plugin is in the rollout middle group, which the owner decides after batch 2 reports its numbers. It waits for that decision.
kyle-sexton commentedon Oct 24, 2026
Released by the owner's HITL session (2026-10-24, C12): it waited only on a removed gate. Intake triage re-runs on it to set its fields, then admission takes it.
github-actions commentedon Oct 25, 2026
The triage sweep dispatched intake triage for this issue (attempt 1 of 2).
6 remaining items
kyle-sexton commentedon Nov 1, 2026
test(dometrain): pass a single exit code followed by its meaning and …
kyle-sexton commentedon Nov 1, 2026
This was generated by AI.
Progress: #11833 merged (9e98502). It adds six outcome eval cases (all graders deterministic) and 19 frozen trigger probes for the
dometrainplugin'sgrounding,setupandsyncskills, and widens one grader on review (reports-exit-code). No skill text changed. Its smoke pass was VALID for all six cases, but one smoke pass is a signal, not a measurement.This issue stays open. The concise rewrite and its measured passes wait on the eval validity fixes in #11618 and #11628. The frozen suite digest for those runs is
4ae568d1c3(plugins/dometrain/evalsat 65fe908). The claim on this issue is released.kyle-sexton commentedon Nov 1, 2026
test(dometrain): widen three graders and make routes-grounding with-only
kyle-sexton commentedon Nov 1, 2026
Progress: the first round of the concise rewrite shipped in #12116 (squash commit c2f9bb1).
groundingandsetupbodies are shorter. Measured on Opus 5.5 and Sonnet 5.5 against the frozen suite from test(dometrain): add outcome eval cases and grounding trigger probes #11833, with no loss.syncbody is unchanged. Its Opus result was decided by a grader defect (sy-check-report-only/reports-exit-codemisses a bolded "code 2"). The PR body lists that defect and the other findings.This issue stays open for
sync. It gets a second measured round after the PR that fixes the suite's graders merges.kyle-sexton commentedon Nov 2, 2026
Reopened (lane 3): the #12162 sync body cut failed a retroactive negative control, so it is being reverted.
The owner's rule (2026-11-02) says a cut ships only if its suite scores a deliberately broken copy clearly below the before version. #12162 merged before that rule existed, so lane 3 ran the control afterwards.
The broken copy is main's
syncbody with its check and Output sections emptied. It ran on Sonnet 5.5 against a fresh pass of main's body launched with it, onsy-check-report-only, 5 runs each. Both passes were VALID and no grader missed in any run: 1.000 vs 1.000.FAIL. With one case, the rule is decided on the mean drop, which is 0 against the 0.15 required. The suite cannot tell the body from a copy without those sections, so the round-2 result is no evidence the cut was safe.
A revert PR restores the earlier body. This issue stays open for cases that depend on the check and Output sections. Cost of the control: 0.64 USD (list-price estimate).
fix(dometrain): point at the docs for disable-model-invocation in sync