Skip to content

ADFA-7186 AI-Core: ask for an answer instead of stopping silently when the model repeats a successful call - #168

Open
davidschachterADFA wants to merge 8 commits into
mainfrom
fix/ADFA-7186-repeat-after-success
Open

davidschachterADFA wants to merge 8 commits into
mainfrom
fix/ADFA-7186-repeat-after-success

Conversation

@davidschachterADFA

@davidschachterADFA davidschachterADFA commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

ADFA-7186

Problem. When the model repeats its previous tool call exactly, with the same tool and arguments, and that call succeeded, ToolCallProgressGuard returns ASSUME_COMPLETE. AgentLoop then ended the run at once. No tool ran, the model was not asked for an answer, and the user saw nothing, because tool-only turns are hidden from the chat. The guard is for a model that finished its work (typically a write or a build) but forgot to call respond. It misfired on read-only questions not yet answered.

Seen on a phone on 2026-11-16, with Claude Haiku 5.5 and the SQLite debugger's tools (during the ADFA-7147 check): the model called list_databases twice in a row and the run ended with no answer. How the call was approved does not matter; a dialog approval would end the same way.

Fix

  • The first ASSUME_COMPLETE in a run no longer ends it. The model gets one more turn (agent_loop.yml: repeated_after_success): "Your previous tool call(s) already succeeded and were not run again; their results are above. If those results are enough, answer the user now: write the answer, or call the "respond" tool with it. Otherwise make a different call, not the same one again." The repeated batch is not run again. If the repeating turn also carries a respond, that is taken as the answer and the run ends normally, unless the run still owes its required tool (a web_search it had to make first): then it is asked for that tool, as after a plain respond.
  • If that turn answers, the run ends normally with the answer shown. If it makes a different call and that batch succeeds, a later repeat gets the extra turn again; a model that alternates between calls is still ended by the progress guard (CYCLING). The README, the agent_loop.yml comment and the AgentLoop KDoc say the turn comes at most once until a different call succeeds. A second ASSUME_COMPLETE with no successful batch since the extra turn ends the run as before, and now shows a system message in the chat (agent_repeated_step): "The step the Agent repeated had already succeeded and was not run again. The Agent stopped without answering. Try asking again, or rephrasing your question."
  • Unchanged: ToolCallProgressGuard itself, the unapplied-edit warn/outstanding checks that run before the extra turn, the REPEATED/CYCLING limits and the LIVE_READS exemption. The trace keeps assumed-complete=repeat-after-success and adds asked-to-answer=repeat-after-success for the extra turn.

Tests: ten new AgentLoopTest cases:

  • a repeated read gets the instruction, is not run again, and a respond answer ends the run;
  • the same with a prose answer;
  • a respond sent with the repeated call is the answer;
  • but not while the required tool is owed: the run is asked for it instead;
  • a second repeat ends with the repeat event;
  • a repeated write ends after one extra turn instead of looping;
  • with no repeat turn configured, the old behaviour;
  • after a different call succeeds, a later repeat gets the extra turn again;
  • a model alternating between two repeated calls still ends as CYCLING;
  • the shipped prompt renders and names respond.

With the extra-turn branch disabled, four of the six cases at 23394bb failed. ../../gradlew testDebugUnitTest assemblePlugin in plugins/AI-Core passed at 23394bb (1,348 tests), and addons check passed. testDebugUnitTest passes at the head, 1aba2f5 (1,352 tests, 0 failures). No test checks that the notice is shown, only the event that leads to it.

Merge with "Squash and merge". The first commit (db3d313) is missing six files and does not compile on its own; the second (23394bb) adds them. The history was not rewritten, because both were already pushed.

Not verified on a device. Getting a model to repeat itself on demand is not reliable.

Screens: no layout change. A new system message appears in the chat in the second-repeat case.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Vh2DAYT7GPoC2cDzo8LErn

davidschachterADFA and others added 2 commits November 16, 2026 23:36
… to answer

A model that re-issued the call that had just succeeded ended the run at once
with nothing said, which on a read-only question left the user with no answer.
The first such repeat now adds a turn telling the model its earlier result is
above and to answer; the repeated batch is not run again. A second repeat ends
the run as before, now with a notice saying the Agent repeated the same step.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vh2DAYT7GPoC2cDzo8LErn
…nfig wiring

The previous commit was missing these files: the repeated_after_success turn
in agent_loop.yml, its config field, parser line and renderer, the user notice
string, and the README mention.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vh2DAYT7GPoC2cDzo8LErn

@davidschachterADFA davidschachterADFA left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MINOR

  • agent_loop.yml:46 - the extra turn forbids the tool and demands an answer from that one result, unlike after_success and unfinished
  • AgentLoop.kt:377 - a respond sent with the repeated batch is dropped, and the notice then says the Agent did not answer
  • strings.xml:224 - after a repeated write the notice says "Stopped ... try asking again" without saying the step worked

NITPICK - 1 inline, not listed

CRITICAL, IMPORTANT - none

Merge-blocking: none of the findings block merging (no CRITICAL or IMPORTANT). Two of the three MINORs are text changes (one prompt, one string), and the third is a small branch in AgentLoop. All three are worth making before merge, because the PR's whole effect is what the model and the user are told. As the PR asks, merge with "Squash and merge": db3d313 alone does not compile (its ChatViewModel uses ToolResultsPrompt.repeatedAfterSuccess and agent_repeated_step, which 23394bb adds).

Round 1 at 23394bb, effort medium, full review. Not stacked (base main). No earlier reviews or threads, so there was nothing to re-check. Rules read at head: CLAUDE.md, plugins/README.md, plugins/AI-Core/README.md; the plugin-review skill was read for criteria only, not scored. This repo has no written approve/request-changes rule, so the pr-review default applied. The computed verdict is COMMENT (MINOR and NITPICK only), which is also the only event allowed on a self-authored PR.

What checked out in AgentLoop.run:

  • The repeated batch is never run: the new branch continues before executeTools, recordResults and unapplied.record.
  • ToolCallProgressGuard.inspect returns ASSUME_COMPLETE from an early path that changes no state, so a third identical batch is ASSUME_COMPLETE again (not REPEATED or CYCLING) and ends the run with the notice.
  • askedToAnswerAfterRepeat is never reset, so the extra turn is given once per run. Setting askedToFinish stops the unfinished nudge from following a prose answer, and a later real batch resets askedToFinish as before.
  • The unapplied-edit warn/outstanding checks still run first, in the same order, so a failed change is never answered with "answer now".
  • UNPARSABLE, REPEATED, CYCLING, the required tool and the LIVE_READS exemption take the same paths as before.
  • ChatViewModel builds the only AgentLoop and wires toolResultsPrompt::repeatedAfterSuccess. onRepeatAfterSuccess now calls notices.repeatedStep(), which adds agent_repeated_step as a system message like noProgress.
  • The new key is parsed, rendered in ToolResultsPrompt's startup check, and covered by theShippedRepeatTurnNamesTheTerminalToolAndRendersCleanly.

Leads considered and not posted:

  • A repeat on the last allowed turn now ends as MAX_ITERATIONS with that notice, where it ended silently before (edge, and not worse).
  • Two consecutive user turns on the native protocol: the same shape the existing injected turns already produce.
  • askedToAnswerAfterRepeat is never reset after real progress, so a second, separate repeat episode ends with the notice and gets no extra turn: "once per run" is the PR's stated rule, and the user now sees a notice rather than nothing.
  • Having the guard tell read-only repeats from finished writes is a design alternative. The notice wording above covers its user-visible effect.

Not verified: no build, no device, and no model run. Whether a model obeys the new turn, and the notice on screen, are unchecked (the PR says no test covers the notice).

Comment thread plugins/AI-Core/README.md Resolved
davidschachterADFA and others added 3 commits November 17, 2026 08:39
…nt call

The prompt said "answer from that result", which pushes thin answers when a
different call was needed, and "that tool" was ambiguous for a batch. It now
says the call(s) succeeded and were not rerun, to answer if the results are
enough, else make a different call. Also drops a doubled "and" in the README.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vh2DAYT7GPoC2cDzo8LErn
…answer

A turn that repeated a successful call and also carried a terminal respond had
the respond dropped, then the run asked for an answer or ended with the
"did not answer" notice. The respond is now the run's answer; the repeated call
is still not run.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vh2DAYT7GPoC2cDzo8LErn
… stop notice

The notice did not say the step worked, which invites running it twice. It now
says the repeated step had already succeeded and was not run again, then that
the Agent stopped. No test: a string resource with no logic.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vh2DAYT7GPoC2cDzo8LErn

@davidschachterADFA davidschachterADFA left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 5 review, limited to commits 1b5b431, ed17f8d and cb11c7d (23394bb -> cb11c7d). Earlier code was reviewed in earlier rounds.

IMPORTANT

  • AgentLoop.kt:368 - a repeat sent with respond finishes without the required tool (requiredPending is not checked)

MINOR

  • AgentLoop.kt:374 - the ask is not re-armed after the different call the new prompt invites, so a later repeat ends the run without asking

Mixed batches and event order, checked. If the turn holds respond plus calls that differ from the last batch, it gets PROCEED: the calls run and the respond is dropped, as on main. If it holds respond plus exactly the last successful batch, it gets ASSUME_COMPLETE, and the new branch fires after the unapplied-write warning and stop, the same order as the respond-only path. The events are onTurnStarted, onModelTurn, then onFinalAnswer. No onToolResults fires, because the repeated batch is not run. The assistant turn is already in history, so nothing else is appended. An empty respond message becomes "", as in the respond-only path. The texts check out. The prompt names {{TERMINAL_TOOL}} and no longer forbids another call. The notice says the step had succeeded and was not run again. The README row now reads correctly.

Earlier threads these commits address (each checked by reading the code at cb11c7d):

  • Extra turn forbade any further call: fixed in 1b5b431. The prompt now allows a different call, but see the MINOR finding about the loop state that goes with it.
  • respond sent with the repeated batch lost its answer: fixed in ed17f8d, but the fix bypasses the required tool (the IMPORTANT finding).
  • Notice did not say the step had worked: fixed in cb11c7d.
  • README row with a doubled "and": fixed in 1b5b431.

Code-review leads not posted. These concern code outside these commits: ASSUME_COMPLETE for read-only batches in ToolCallProgressGuard, the novelty limit reaching CYCLING before the second repeat, askedToFinish being set without onAskedToFinish, and the stale Verdict.ASSUME_COMPLETE KDoc. Also not posted: a lead that the notice may prompt a user to redo a write. The notice now says the step succeeded and was not run again.

Verdict. The repository has no written approve or request-changes rule. I read CLAUDE.md, plugins/README.md and plugins/AI-Core/README.md, so the default rule applies. The computed verdict is REQUEST_CHANGES because of the IMPORTANT finding. It was not submitted, because this PR is self-authored and the run was non-interactive without --verdict. Not verified: no tests were run and no build was made. Both findings come from reading AgentLoop.kt and ToolCallProgressGuard.kt at cb11c7d.

davidschachterADFA and others added 2 commits November 17, 2026 10:17
… with respond

A repeated call sent together with respond finished the run without asking for the
required tool, so a web_search could be skipped. Redirect it to the tool as a plain respond is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vh2DAYT7GPoC2cDzo8LErn
…ifferent call worked

The repeat prompt invites a different call, but the extra turn was spent for the run. Re-arm it
once a successful batch has run; the progress guard still ends a run that alternates between calls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vh2DAYT7GPoC2cDzo8LErn

@davidschachterADFA davidschachterADFA left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round over cb11c7d..07fe53a (3cc8257, 07fe53a). Not stacked (base main). Verified against 07fe53a. Rules read: CLAUDE.md, plugins/README.md, plugins/AI-Core/README.md; none sets an approve/request-changes rule, so the default applied. Self-authored PR, so COMMENT.

MINOR

  • AgentLoop.kt:16 - the KDoc, README row and agent_loop.yml comment still say the repeat turn is sent once per run (non-blocking)

Re-check of earlier threads (judged on the code at head, not on the replies):

  • F131 (repeat sent with respond skipped the required-tool check): fixed. In the ASSUME_COMPLETE branch, terminal != null && requiredPending now clears the flag and calls askForRequiredTool, the same redirect the plain-respond path uses, ahead of unapplied.warn. If the model repeats again once asked, requiredPending is already false and the run ends, which is the documented "a model that refuses it still stops" behaviour.
  • F140 (extra turn not re-armed): fixed. A fully successful batch clears askedToAnswerAfterRepeat. Bound checked in ToolCallProgressGuard: a batch already in seenSignatures increments turnsWithoutNewSignature and only a new signature (or a moving live read) resets it, so alternating known batches reaches CYCLING; I walked the new alternation test by hand (a, a, b, b, a, a gives 2 retries, then CYCLING at turn 6 with the default limit). A model producing new calls keeps making real progress and is bounded by maxIterations as before.

* @param formatToolResults words each batch's results as the next user turn.
* @param unfinishedTurn words the turn sent when a run that has used tools ends a reply without
* [terminalTool]; null ends the run on that reply instead.
* @param repeatedAfterSuccessTurn words the turn sent once when a run re-issues the batch it has just

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MINOR: the docs still say the repeat turn is sent once, but 07fe53a re-arms it after a different successful batch, so a run can now get it more than once.

This KDoc (words the turn sent once when a run re-issues ...), the agent_loop.yml comment above repeated_after_success (Sent once when ...) and the AI-Core README row at plugins/AI-Core/README.md:164 (sent once when a run re-issues the call that just succeeded) were all written for the one-ask-per-run behaviour. A maintainer reading them will expect at most one such turn per run and may be surprised by a second one, or write a test asserting one. No runtime effect, so it is safe to merge.

Reword the three to sent when ... , again after a different call has succeeded, or at most once per stretch of repeats.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 1aba2f5: the AgentLoop KDoc, the agent_loop.yml comment and README line 164 now say the turn is sent at most once until a different call succeeds, then again for a later repeat.

…nt call worked

07fe53a re-arms the repeated_after_success turn once a different batch has
succeeded, so a run can get it more than once. The AgentLoop KDoc, the
agent_loop.yml comment and the README row still said it is sent once. They
now say it is sent at most once until a different call succeeds. Text only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vh2DAYT7GPoC2cDzo8LErn
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant