fix: detect and abort repetitive reasoning loops (#11337)

[roomote]

Related GitHub Issue

Closes: #11337

Description

This PR attempts to address the infinite looping issue reported in #11337, where some models (particularly Google Gemini) get stuck in a reasoning/thinking loop, repeating the same lines over and over (e.g., "I'll mention that I verified with tests" or "I'll use attempt_completion" repeated indefinitely).

The existing ToolRepetitionDetector only catches repeated tool calls, but does not detect when the model's reasoning/thinking output is looping. This PR fills that gap with a new ReasoningRepetitionDetector that monitors the reasoning stream in real-time.

How it works:

1. A new ReasoningRepetitionDetector class tracks lines as they stream in during the "reasoning" chunk handler in Task.ts.
2. Lines are normalized (trimmed, lowercased, whitespace-collapsed) and counted. Short lines (< 20 chars) are ignored to avoid false positives.
3. When any single line is repeated 5 or more times, the stream is aborted early via cancelCurrentRequest() to save tokens.
4. Instead of retrying (the normal stream-failure path), the task loop continues with a guidance message telling the model its reasoning was repetitive, instructing it to take a different approach.
5. The consecutiveMistakeCount is incremented so if the model keeps looping, the existing mistake limit will eventually pause and ask the user for guidance.

Key design decisions:

• Default threshold of 5 repetitions balances detection sensitivity with avoiding false positives on legitimately repeated reasoning patterns.
• The detector resets at the start of each new API request, so previous reasoning doesn't carry over.
• The abort is handled as a special case in the catch block (via a reasoningRepetitionAborted flag) to avoid the normal retry logic.

Feedback and guidance are welcome.

Test Procedure

• 18 unit tests for ReasoningRepetitionDetector covering:
  • Basic detection, threshold behavior, and the exact pattern from the issue report
  • Multi-line chunks, split-across-chunks, short line filtering
  • Whitespace normalization, case-insensitive comparison
  • isRepetitive() method, getMostRepeatedLine() diagnostics
  • reset() behavior, default threshold, alternating A-B-A-B patterns
  • Varied content below threshold (no false positives)
• Ran existing ToolRepetitionDetector, Task.spec.ts, and grace-retry-errors tests -- all pass.
• All lint and type-check passes across the full monorepo.

Run tests: cd src && npx vitest run core/tools/__tests__/ReasoningRepetitionDetector.spec.ts

Pre-Submission Checklist

• Issue Linked: This PR is linked to an approved GitHub Issue (see "Related GitHub Issue" above).
• Scope: My changes are focused on the linked issue (one major feature/fix per PR).
• Self-Review: I have performed a thorough self-review of my code.
• Testing: New and/or updated tests have been added to cover my changes (if applicable).
• Documentation Impact: No documentation updates needed for this internal behavior change.
• Contribution Guidelines: I have read and agree to the Contributor Guidelines.

Documentation Updates

No documentation updates needed. This is an internal behavior improvement that is transparent to users -- the only visible change is that infinite reasoning loops will now be automatically detected and interrupted with a helpful error message.

---

Important

Introduces ReasoningRepetitionDetector to abort repetitive reasoning loops in models, integrated into Task class with comprehensive testing.

• Behavior:
  • Adds ReasoningRepetitionDetector to detect repetitive reasoning in Task.ts, aborting streams when a line repeats 5+ times.
  • Aborts stream via cancelCurrentRequest() and continues task with guidance message.
  • Increments consecutiveMistakeCount to eventually pause and ask for user guidance.
• Implementation:
  • ReasoningRepetitionDetector tracks lines, normalizes them, and checks repetition threshold.
  • Integrated into Task class to monitor reasoning output in real-time.
• Testing:
  • Adds 18 unit tests for ReasoningRepetitionDetector in ReasoningRepetitionDetector.spec.ts.
  • Tests include detection, threshold behavior, multi-line handling, and normalization.
  • All existing tests pass, including ToolRepetitionDetector and Task.spec.ts.

^{This description was created by for c9b1d5e. You can customize this summary. It will automatically update as commits are pushed.}

[roomote]

Rooviewer   Follow task

Reviewing your PR now. Feedback coming soon!

[roomote]

Rooviewer   See task

Reviewed the ReasoningRepetitionDetector class and its integration into Task.ts. The overall approach is sound. Found one issue:

• isRepetitive() mutates lineCounts on each call, making it non-idempotent -- repeated calls inflate the buffered line's count and can produce false positives

_{Mention @roomote in a comment to request specific changes to this pull request or fix all unresolved issues.}

[roomote]

isRepetitive() mutates lineCounts as a side effect: each call increments the count for whatever is sitting in the buffer. If this method is called more than once without an intervening addChunk() or reset(), the buffered line's count keeps climbing, and can cross the threshold even though the line only appeared once in the actual stream. Although the current Task.ts integration only uses addChunk(), this is a public query method whose semantics suggest it should be idempotent.

One fix: process the buffer into a local variable instead of writing back to lineCounts, or clear the buffer after processing it.

Suggested change

	public isRepetitive(): boolean {
	// Also process any remaining buffer content
	if (this.buffer.length > 0) {
	const line = this.normalizeLine(this.buffer)
	if (line.length >= this.minLineLength) {
	const count = (this.lineCounts.get(line) ?? 0) + 1
	this.lineCounts.set(line, count)
	if (count >= this.repetitionThreshold) {
	return true
	}
	}
	}
	for (const count of this.lineCounts.values()) {
	if (count >= this.repetitionThreshold) {
	return true
	}
	}
	return false
	}
	public isRepetitive(): boolean {
	// Also process any remaining buffer content
	if (this.buffer.length > 0) {
	const line = this.normalizeLine(this.buffer)
	if (line.length >= this.minLineLength) {
	const count = (this.lineCounts.get(line) ?? 0) + 1
	if (count >= this.repetitionThreshold) {
	return true
	}
	}
	}
	for (const count of this.lineCounts.values()) {
	if (count >= this.repetitionThreshold) {
	return true
	}
	}
	return false
	}

_{Fix it with Roo Code or mention @roomote and request a fix.}