Repository navigation
fix(dojo-tally): 0.1.2 — a structured-output answer counts as returned (2.4.0, unreleased) - #15
Merged
Merged
Conversation
…d (2.4.0, unreleased) A workflow agent run with an output schema answers by calling the StructuredOutput tool, and its run ends on that call's result: its last turn has no text, so turn.complete's `answer` is "". classify() fails any blank answer as "it returned an empty answer", so every such agent was counted failed. Seen 2026-10-27 after wf_73711b2d-734: the settle note read "dispatched 18 · returned 2 · failed 16", listing all eight scout, seven verify and the synthesize agent, while the run's journal has 16 `result` lines of 11-29 KB and its usage said agents_empty_result: 0. The two "returned" were Agent-tool builders that ended with text. - The tool.call hook keeps the arguments of an agent's last accepted StructuredOutput call (a denied or errored try is not kept); the turn.complete hook uses them as the answer when the turn has no text. Judged like text, and what /tally refute hands the judge. - StructuredOutput joins the bookkeeping tools: handing back an answer looks at nothing, so a clean verdict returned that way after no reads is still `silent` (rule 9). - README: the structured-answer rule, and its limit (a hot reload between the call and the turn's end loses the payload). Two kit tests built from that run's transcript shape (tool_use StructuredOutput -> "Structured output provided successfully", no text): both fail on the old register.ts (returned 1 · failed 3; returned 0 · failed 3) and pass now. Dropping StructuredOutput from the bookkeeping set alone fails the silent test. Checks on Claude Code 2.1.308: scripts/release-check.sh --plugin dojo-tally OVERALL PASS (53 mod tests, 2 new; 22 unittest; validate; face-parity; always-on). Not seen live: a workflow run with this version installed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
dojo-tally 0.1.2: a workflow agent that answers through StructuredOutput now counts as returned. Until now every such agent was reported as "failed — it returned an empty answer". It's a fix under the unreleased 2.4.0 with a plugin-only version bump, so the marketplace stays at 2.4.0. Do not merge without Cruz's go.
What happened
On 2026-10-27, workflow run
wf_73711b2d-734settled with dispatched 18 · returned 2 · failed 16 · silent 0. Everyscout:*,verify:*andsynthesizeagent was listed as an empty answer. The run's own journal disagreed: it has 16resultlines of 11–29 KB, and the workflow reportedagents_empty_result: 0. The only two agents counted as returned were plain Agent-tool builders that ended with text.Why
agent()with aschemaanswers by calling StructuredOutput, and its run ends on that call's result with no text after it. All 16 transcripts end like this.turn.completehands the modanswer: "". A run that ends onSubagentHandbackdoesn't hit this, because the engine puts the handback message intoanswer. That's why the two builders were counted.classify()treats a blank answer as failed, so the turn was recorded as "it returned an empty answer".The fix (
plugins/dojo-tally/hooks/register.ts)tool.callhook keeps the arguments of an agent's last accepted StructuredOutput call. A denied or errored try isn't kept. The map is capped the same way as the early-done map./tally refutehands them to the judge. If the turn has text, the text stays the answer. Either way the kept payload is dropped at the end of the turn.BOOKKEEPING, so handing back an answer doesn't count as looking at anything. Without this, a reviewer that read nothing and handed back "No defects found" would escapesilent(rule 9). A mutation check confirmed it.No other mod is touched.
Checks
tests/tally.test.ts, built on that run's transcript shape:scripts/release-check.sh --plugin dojo-tally: OVERALL PASS on Claude Code 2.1.308. That covers lint, denylist, 22 unit tests, validate, 53 mod tests, 296 self-tests, face-parity, plugin-lint (WARN-OK) and the always-on budget.9ac21ab):dispatched 18 · returned 2 · failed 16 · silent 0 · unchecked 0, with the same 16 "empty answer" lines as the live row.2199f2b):dispatched 18 · returned 18 · failed 0 · silent 0 · unchecked 0.Limits
tool.callfor a workflow agent's StructuredOutput call. The headless CLI outside the desktop app isn't logged in, so I couldn't run a live check. The first real workflow run with 0.1.2 installed will settle it. If the call turns out not to be raised, the tally stays wrong as it is today; it won't count anything as returned that wasn't.🤖 Generated with Claude Code