answers: give the driver the answer, not the worksheet (#52) - #67
ThinkOffApp wants to merge 1 commit into
Conversation
An /ask on VTA came back opening with the prompt's own scaffold, verbatim:
- KNOWN FACTS: OBD is running but cable NOT U...
before any actual answer. Neither answer path stripped anything: both
agent._think and webchat.answer joined the model output and called .strip(),
which removes whitespace and nothing else.
grounding.strip_scaffold() now removes a leading echo of the prompt's own
headings plus the fact bullets trailing them, and any <think>/<reasoning>
block. Applied in both paths, since voice, /ask and the room all go through
_think while the web chat has its own.
Conservative by construction, because eating a real answer is worse than
printing a scaffolded one:
- it only strips from the START of the reply
- it matches the headings in their prompt casing, so ordinary prose such as
"I only state what appears in KNOWN FACTS" survives untouched
- if stripping would leave nothing, it returns the original
The headings are now constants used BY build_system_prompt, so the prompt and
the stripper cannot drift apart. A test asserts both appear in a built prompt,
and another builds its scaffold from the prompt itself rather than a copy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7a5f48e294
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if any(bare.startswith(m) or bare[:40].find(m) >= 0 | ||
| for m in _SCAFFOLD_MARKERS): |
There was a problem hiding this comment.
When a multiline answer begins with ordinary prose such as According to my KNOWN FACTS, fuel is at 58%., the substring search within the first 40 characters treats that sentence as scaffold and deletes it, leaving only the subsequent lines. The single-line prose test passes only because the empty-output fallback restores the original; require the marker to occupy the heading position rather than appearing anywhere near the start.
Useful? React with 👍 / 👎.
| if dropping and (not bare or _BULLET.match(lines[i])): | ||
| i += 1 # the fact bullets that follow a heading |
There was a problem hiding this comment.
If the model echoes the complete facts section produced by build_system_prompt, stripping stops at the first non-bullet prompt line, such as No manual lookup was performed... or OWNER MANUAL EXCERPTS.... That instruction and any following excerpts are then returned to the driver ahead of the real answer, so a full rather than truncated scaffold echo still leaks the worksheet this change is meant to remove.
Useful? React with 👍 / 👎.
2 participants
Fixes #52.
An
/askon VTA came back opening with the prompt's own scaffold, verbatim:before any actual answer.
Cause
Neither answer path stripped anything.
agent._thinkandwebchat.answerboth join the model output and call.strip()— which removes whitespace and nothing else.Change
grounding.strip_scaffold()removes a leading echo of the prompt's own headings plus the fact bullets trailing them, and any<think>/<reasoning>block. Applied in both paths: voice,/askand the room all go through_think, while the web chat has its own.Conservative by construction, because eating a real answer is worse than printing a scaffolded one:
No drift
The headings are now constants used by
build_system_prompt, so the prompt and the stripper cannot diverge. One test asserts both headings appear in a built prompt; another builds its scaffold input from the prompt itself rather than from a hand-copied string, so rewording the prompt cannot leave the stripper behind.Tests
tests/test_scaffold_leak.py, 6 cases: the exact reported leak, a real echoed heading with its bullets, reasoning blocks, three ordinary-prose strings that must be untouched, the never-return-empty guard, and the shared-headings check.Suite failure on this branch is
test_dash_loop's missing pytest — pre-existing onmain, fixed separately in #64.Not verified on the car
The car is unreachable, so this is verified against the real prompt in tests rather than against a live model's output. The issue also raised a second possible cause —
vta-brain.servicerunning without--reasoning off— which is a server flag, not repo code, and is untouched here.🤖 Generated with Claude Code