Skip to content

fix(dojo-tally): 0.1.2 — a structured-output answer counts as returned (2.4.0, unreleased) - #15

Merged
DojoGenesis merged 1 commit into
mainfrom
fix/tally-structured-output
Oct 27, 2026
Merged

DojoGenesis merged 1 commit into
mainfrom
fix/tally-structured-output

Conversation

@DojoGenesis

Copy link
Copy Markdown
Owner

dojo-tally 0.1.2: a workflow agent that answers through StructuredOutput now counts as returned. Until now every such agent was reported as "failed — it returned an empty answer". It's a fix under the unreleased 2.4.0 with a plugin-only version bump, so the marketplace stays at 2.4.0. Do not merge without Cruz's go.

What happened

On 2026-10-27, workflow run wf_73711b2d-734 settled with dispatched 18 · returned 2 · failed 16 · silent 0. Every scout:*, verify:* and synthesize agent was listed as an empty answer. The run's own journal disagreed: it has 16 result lines of 11–29 KB, and the workflow reported agents_empty_result: 0. The only two agents counted as returned were plain Agent-tool builders that ended with text.

Why

  1. A workflow agent() with a schema answers by calling StructuredOutput, and its run ends on that call's result with no text after it. All 16 transcripts end like this.
  2. Its last turn therefore has no visible text, so turn.complete hands the mod answer: "". A run that ends on SubagentHandback doesn't hit this, because the engine puts the handback message into answer. That's why the two builders were counted.
  3. classify() treats a blank answer as failed, so the turn was recorded as "it returned an empty answer".

The fix (plugins/dojo-tally/hooks/register.ts)

  • The tool.call hook keeps the arguments of an agent's last accepted StructuredOutput call. A denied or errored try isn't kept. The map is capped the same way as the early-done map.
  • When a turn ends with no text, the kept arguments become its answer. They're judged like text, and /tally refute hands them to the judge. If the turn has text, the text stays the answer. Either way the kept payload is dropped at the end of the turn.
  • StructuredOutput joins BOOKKEEPING, so handing back an answer doesn't count as looking at anything. Without this, a reviewer that read nothing and handed back "No defects found" would escape silent (rule 9). A mutation check confirmed it.

No other mod is touched.

Checks

  • Two new kit tests in tests/tally.test.ts, built on that run's transcript shape:
    • Three workflow agents and one builder. One of the agents makes a rejected try and then an accepted one. Expected: returned 4, and the refute prompt carries the payload.
    • An idle reviewer with a clean verdict and no reads, which should be silent, and an agent whose only try was rejected, which should be failed.
    • Both fail on 0.1.1 ("returned 1 · failed 3" and "returned 0 · failed 3") and pass on this branch.
  • scripts/release-check.sh --plugin dojo-tally: OVERALL PASS on Claude Code 2.1.308. That covers lint, denylist, 22 unit tests, validate, 53 mod tests, 296 self-tests, face-parity, plugin-lint (WARN-OK) and the always-on budget.
  • Replay of the real run: a kit test generated from the journal and each agent's transcript (every tool call in order, StructuredOutput arguments as given, each agent's real final text):
    • 0.1.1 (9ac21ab): dispatched 18 · returned 2 · failed 16 · silent 0 · unchecked 0, with the same 16 "empty answer" lines as the live row.
    • 0.1.2 (2199f2b): dispatched 18 · returned 18 · failed 0 · silent 0 · unchecked 0.

Limits

  • Not seen live yet: it's still unconfirmed that the engine raises tool.call for a workflow agent's StructuredOutput call. The headless CLI outside the desktop app isn't logged in, so I couldn't run a live check. The first real workflow run with 0.1.2 installed will settle it. If the call turns out not to be raised, the tally stays wrong as it is today; it won't count anything as returned that wasn't.
  • Hot reload: the payload is held in memory between the call and the end of the turn, so a hot reload in that window loses it, and that agent reads as an empty answer. The README says so.

🤖 Generated with Claude Code

…d (2.4.0, unreleased)

A workflow agent run with an output schema answers by calling the
StructuredOutput tool, and its run ends on that call's result: its last
turn has no text, so turn.complete's `answer` is "". classify() fails
any blank answer as "it returned an empty answer", so every such agent
was counted failed. Seen 2026-10-27 after wf_73711b2d-734: the settle
note read "dispatched 18 · returned 2 · failed 16", listing all eight
scout, seven verify and the synthesize agent, while the run's journal
has 16 `result` lines of 11-29 KB and its usage said
agents_empty_result: 0. The two "returned" were Agent-tool builders
that ended with text.

- The tool.call hook keeps the arguments of an agent's last accepted
  StructuredOutput call (a denied or errored try is not kept); the
  turn.complete hook uses them as the answer when the turn has no text.
  Judged like text, and what /tally refute hands the judge.
- StructuredOutput joins the bookkeeping tools: handing back an answer
  looks at nothing, so a clean verdict returned that way after no reads
  is still `silent` (rule 9).
- README: the structured-answer rule, and its limit (a hot reload
  between the call and the turn's end loses the payload).

Two kit tests built from that run's transcript shape (tool_use
StructuredOutput -> "Structured output provided successfully", no
text): both fail on the old register.ts (returned 1 · failed 3;
returned 0 · failed 3) and pass now. Dropping StructuredOutput from
the bookkeeping set alone fails the silent test.

Checks on Claude Code 2.1.308: scripts/release-check.sh --plugin
dojo-tally OVERALL PASS (53 mod tests, 2 new; 22 unittest; validate;
face-parity; always-on). Not seen live: a workflow run with this
version installed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@DojoGenesis
DojoGenesis merged commit be90a0c into main Oct 27, 2026
3 checks passed
@DojoGenesis
DojoGenesis deleted the fix/tally-structured-output branch October 27, 2026 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants