fix(evals): run invocation probes without subagents and name the run line in run-validity

.changes/evals/fix-probe-suite-validity-probe-tools-8d4b2f61.md

@@ -0,0 +1,7 @@
+---
+bump: patch
+---
+
+### Fixed
+
+- `run-validity.py` adds a `probe tools` check: it warns when an invocation-probe run could call Read, Glob, Grep or Agent, and names the emitted README's `--disallow-tools Read Glob Grep Agent` run line. A probe run whose trace has no init tools list is a warning, and when no probe run's list can be read the check is UNCHECKED, which makes the run INVALID. A probe run without that line was failing on denied searches and subagent timeouts with no pointer to the cause. `reading-results.md` now states how the gate reads invocation probes.

.changes/playbooks/fix-probe-suite-validity-run-line-3e7a0c95.md

@@ -0,0 +1,7 @@
+---
+bump: patch
+---
+
+### Fixed
+
+- `rewrite-a-plugin.md` step 5 runs the emitted trigger probes with the run line from the README beside them, rather than a copy of its flags that went stale.

.changes/skill-quality/fix-probe-suite-validity-disallow-agent-5c1e9a27.md

@@ -0,0 +1,7 @@
+---
+bump: patch
+---
+
+### Fixed
+
+- `measure-invocation.sh emit-plugin-eval` README and `reference/invocation-probes.md` add `Agent` to the probe run line, now `--disallow-tools Read Glob Grep Agent`. After the skill loaded, the agent handed the task to a subagent that outlived the 300-second run timeout, and those timed-out runs made every probe side INVALID.

plugins/evals/skills/plugin-eval/SKILL.md

@@ -297,9 +297,8 @@ Read in this order. Stopping early at any step is the finding.
    for a plugin file it could not read, so the fact belongs in the hub. The exceptions that turn a
    denial into a warning (a denial the environment makes in both arms, such as a user-scope read of
    `~/.claude/standards`; a denial in a case whose denial-free runs scored the same; an invocation
-   probe's denial), how a Bash denial's paths are read, gating an A/B pass, and the 0-turn
-   `context: fork` run the gate does not fail are in
-   [reading-results.md, Denials](reference/reading-results.md#denials).
+   probe's denial), Bash denial paths, A/B gating, the 0-turn `context: fork` run the gate passes,
+   and invocation-probe misses and tools: [reading-results.md, Denials](reference/reading-results.md#denials).
 2. `partial`. `true` (with `partialReason` of `cost_ceiling`, `interrupted`, or `auth_failed`) means
    the suite did not finish: report that and keep the document out of any trend.
 3. Per run, `skippedPaidGraders: true` or a non-null `error`. A skipped judge grader is still

plugins/evals/skills/plugin-eval/reference/reading-results.md

@@ -60,6 +60,13 @@ above the plugin's directory, or with no absolute path or no known plugin direct
 - An invocation probe's denial of a named call other than Skill is a warning when it follows the
   graded Skill call, or the run held a should-not-fire case whose denial-free runs all held; the
   gate prints each probe case's fire counts with and without denials.
+- An invocation probe is a case whose `prompt.md` tags include `invocation-probe`, as
+  `/skill-quality:check measure-invocation` emits it. Its should-fire misses are the trigger rate
+  it measures: `skill fired` reports them as a warning with per-case counts, never a FAIL. Run
+  errors and denials keep their full force, apart from the two probe exceptions above and a
+  max-turns error after the graded Skill call. `probe tools` warns when a probe run could call
+  Read, Glob, Grep or Agent, the tools the emitted README's run line removes, or its trace has no
+  init tools list; when no probe run's list can be read, the check is UNCHECKED.
 
 ## Document fields
 

plugins/evals/skills/plugin-eval/scripts/fixtures/run-validity/traces/clean.jsonl

@@ -1,2 +1,2 @@
-{"type":"system","subtype":"init","model":"claude-opus-5-5","claude_code_version":"2.1.287","permissionMode":"dontAsk"}
+{"type":"system","subtype":"init","model":"claude-opus-5-5","claude_code_version":"2.1.287","permissionMode":"dontAsk","tools":["Skill"]}
 {"type":"result","subtype":"success","is_error":false,"permission_denials":[]}

plugins/evals/skills/plugin-eval/scripts/run-validity.py

@@ -55,7 +55,12 @@
     skill fired       every with-arm run of a should-trigger case fired the skill;
                       in an invocation-probe case a miss is the trigger rate it
                       measures, reported per case and overall as a warning
-    models            the model and judge model, where the result or traces
+    probe tools       no invocation-probe run's session could call Read, Glob,
+                      Grep or Agent, which the emitted README removes with
+                      --disallow-tools; one that could is a warning naming that
+                      run line. A probe run whose init tools list cannot be read
+                      is a warning; none readable is UNCHECKED
+    models           the model and judge model, where the result or traces
                       carry them; runs that disagree are a FAIL
     judge votes       a split llm judge vote is a warning
     ceiling           a case whose without-arm is already 1.00 is a warning and
@@ -180,6 +185,10 @@
     r"(?:exit \d+: )?Reached maximum number of turns \(\d+\)\.?"
 )
 TRACE_ERRORS = "errors"  # on the result line
+TRACE_TOOLS = "tools"  # on the init line: the tools the session could call
+# Tools emit-plugin-eval's README removes from a probe run with --disallow-tools.
+# The init line names the Agent tool Task.
+PROBE_REMOVED_TOOLS = ("Read", "Glob", "Grep", "Agent", "Task")
 
 # case.yaml's context.scaffold_script, in block or flow form, read without a
 # YAML parser. Any uncommented mention of scaffold_script outside these shapes,
@@ -1092,6 +1101,85 @@ def check_fired(cases, runs, root):
     )
 
 
+def check_probe_tools(runs, root):
+    """An invocation-probe run whose session could call a tool the emitted README
+    removes ran without that run line: in its empty workspace those calls are
+    denied, and a subagent it dispatches outlives the run's timeout. Warns, and
+    names the fix, since the denial and error checks already judge the runs. A
+    probe run with no trace, no init line or no tools list on it is unread: none
+    read is UNCHECKED, some unread a warning."""
+    found = {}  # the removable tools a run could call -> the runs that could
+    probes, unread = 0, []
+    for r in runs:
+        if not is_probe(r.case, root):
+            continue
+        probes += 1
+        tools = next(
+            (
+                line.get(TRACE_TOOLS)
+                for line in r.trace or []
+                if (line.get(LINE_TYPE), line.get(LINE_SUBTYPE)) == TRACE_INIT
+            ),
+            None,
+        )
+        if not isinstance(tools, list):
+            unread.append(r)
+            continue
+        extra = ", ".join(t for t in tools if t in PROBE_REMOVED_TOOLS)
+        if extra:
+            found.setdefault(extra, []).append(r)
+    if not probes:
+        return PASS, "no invocation-probe case", [], None
+    unread_details = [
+        "%s: no init tools list (%s)" % (r.where, r.trace_path or "no trace")
+        for r in unread
+    ]
+    if len(unread) == probes:
+        return (
+            UNCHECKED,
+            "no invocation-probe run's init tools list could be read",
+            unread_details,
+            "probe tools unchecked in %s with no readable init tools list"
+            % plural(len(unread), "run"),
+        )
+    summary, reasons = [], []
+    if found:
+        summary.append(
+            "%s of invocation-probe cases could call %s; run the emitted README's"
+            " line, --disallow-tools Read Glob Grep Agent, so a probe run cannot be"
+            " denied a search or outlive its timeout in a subagent"
+            % (
+                plural(sum(len(rs) for rs in found.values()), "run"),
+                "/".join(PROBE_REMOVED_TOOLS),
+            )
+        )
+        reasons.append(
+            "invocation probes ran without --disallow-tools Read Glob Grep Agent"
+        )
+    if unread:
+        summary.append(
+            "%s with no readable init tools list" % plural(len(unread), "probe run")
+        )
+        reasons.append("probe tools unread in %s" % plural(len(unread), "run"))
+    if summary:
+        return (
+            WARN,
+            "; ".join(summary),
+            [
+                "%s: %s, first %s" % (tools, plural(len(rs), "run"), rs[0].where)
+                for tools, rs in found.items()
+            ]
+            + unread_details,
+            "; ".join(reasons),
+        )
+    return (
+        PASS,
+        "no invocation-probe run could call Read, Glob, Grep or Agent",
+        [],
+        None,
+    )
+
+
 def recorded(result, keys):
     for scope in (result, as_dict(result.get(SUITE))):
         for key in keys:
@@ -1284,8 +1372,8 @@ def main(argv=None):
     if unresolved:
         lines.append(
             "traces: %d of %d tracePaths do not resolve (the run needs --keep-temp, and its"
-            " directories must still exist); permission denials and the trace model are"
-            " unchecked for those runs" % (len(unresolved), len(runs))
+            " directories must still exist); permission denials, probe tools and the"
+            " trace model are unchecked for those runs" % (len(unresolved), len(runs))
         )
         lines += [
             "  %s: %s" % (r.where, r.trace_path or "no tracePath") for r in unresolved
@@ -1296,6 +1384,7 @@ def main(argv=None):
         ("session started", check_started(runs)),
         ("permission denials", check_denials(runs, plugin_roots, root)),
         ("skill fired", check_fired(cases, runs, root)),
+        ("probe tools", check_probe_tools(runs, root)),
         ("models", check_models(result, runs)),
         ("judge votes", check_votes(runs)),
         ("ceiling", check_ceiling(cases, arms)),