feat(evals): add reviewable eval drafts from session logs - #29019
Conversation
…ls from session logs
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a powerful new workflow for improving agent reliability by enabling the direct conversion of behavioral bug reports into regression evaluations. By leveraging existing session logs, the new Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
|
📊 PR Size: size/XL
|
🛑 Action Required: Evaluation ApprovalSteering changes have been detected in this PR. To prevent regressions, a maintainer must approve the evaluation run before this PR can be merged. Maintainers:
Once approved, the evaluation results will be posted here automatically. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces the eval:from-log command and associated utility scripts to automatically generate sanitized and validated regression eval skeletons from Gemini CLI session logs. The review feedback identifies critical security vulnerabilities, including a potential Remote Code Execution (RCE) via code injection from unvalidated tool names in generateAssertBody, and a path traversal vulnerability in sanitizeFileMap where relative paths bypass absolute path checks. Additionally, the reviewer recommends hardening the manual CLI argument parser to prevent flags from incorrectly consuming subsequent flags as values when arguments are missing.
gemini-cli
Bot
added
priority/p3
ved015
changed the title
1 participant
Summary
eval:from-loghelps maintainers and contributors turn a real Gemini CLI interaction into the starting point for a behavioral evaluation.Session logs are valuable because they capture realistic prompts, tool usage, and failure scenarios. However, a log only records what happened; it does not define what should have happened. This feature bridges that gap by extracting one original human turn, combining it with human-defined expectations and an explicitly selected workspace state, and producing a reviewable behavioral eval draft.
The entire workflow is local and deterministic. It does not call an LLM, make network requests, upload the session, infer expected behavior from the observed trajectory, or claim that the generated draft is already a finished regression eval.
Details
End-to-End Workflow
--list-turnspresents the plain-text human turns that can safely form the basis of an eval, along with observed tool calls and possible fixture paths as review evidence.--message-id. Internal tool responses, slash commands, multimodal turns, and persisted compression summaries are not treated as human prompts.--expect-tooland--forbid-tool. Observed calls from the log never become assertions automatically.--fixture, or explicitly confirm that no fixtures are required. Files are copied from the matching original workspace, never reconstructed from logged tool output.--writecan create one new.eval.tsfile directly underevals/. Existing files are never overwritten.Architecture
flowchart TD subgraph Inputs session["Local Gemini CLI session<br/>JSONL or legacy JSON"] workspace["Original workspace<br/>explicit fixture choices"] intent["Human intent<br/>expected and forbidden tools"] end session --> loader["Safe session loader<br/>stable snapshot, size, UTF-8, and format checks"] loader --> turns["Turn analyzer<br/>plain-text human turns"] turns --> selected["Contributor selects one turn"] turns -.-> evidence["Observed tool calls and candidate paths<br/>displayed as evidence, never expectations"] workspace --> fixtures["Workspace and fixture safety gates<br/>project match, containment, portability, and secret checks"] intent --> expectations["Registered tool expectations<br/>canonicalized and conflict-checked"] selected --> generator["Deterministic eval draft generator"] fixtures --> generator expectations --> generator generator --> validator["Existing eval analyzer and validator"] validator --> mode{"Output mode"} mode -->|default| preview["Preview only<br/>nothing written"] mode -->|explicit write| file["New evals/*.eval.ts draft<br/>--write required, no overwrite"] evidence -.-> review["Human review<br/>stronger assertions and fail-before-fix proof"] preview --> review file --> review review --> ready["Remove runtime guard<br/>validated behavioral eval"]What the Draft Contains
The generated file is a normal behavioral eval built around
evalTestand includes:USUALLY_PASSESpolicy for the draft stageThe source is passed through the existing eval analyzer and validator before it is previewed or written. “Structural validation passed” means the file matches the repository's eval structure and registered tool contracts; it does not mean the behavior itself has passed.
Safety and Trust Model
The session log is treated as untrusted input throughout the workflow.
evals/; preview remains the default and writes require--writeplus an explicit path.Secret detection and path redaction are intentionally described as best effort. Contributors must still inspect the complete preview and create a small synthetic reproduction when the original prompt or required files contain sensitive information.
Scope and Limitations
This feature automates the mechanical and safety-sensitive parts of moving from a session to an eval draft. It deliberately does not:
Those decisions remain with the contributor and reviewer because they require product intent, issue context, and fail-before-fix evidence that cannot be inferred reliably from a log alone.
Related Issues
Related to #28696
How to Validate
Inspect the command and locate an eligible human turn:
Preview a draft without changing the repository:
Confirm that the output identifies the selected turn, keeps observed calls as evidence only, copies the fixture from the selected workspace, reports structural validation, prints the runtime guard, and writes nothing.
Then add
--output evals/from-log-review.eval.ts --write. Confirm that exactly one new draft is written and that repeating the same command refuses to overwrite it.Run the production-script typecheck and repository checks:
./node_modules/.bin/tsc --noEmit --pretty false --strict --skipLibCheck \ --esModuleInterop --allowSyntheticDefaultImports --verbatimModuleSyntax \ --module NodeNext --moduleResolution NodeNext --target ES2022 --types node \ scripts/eval-from-log-cli.ts scripts/utils/session-turns.ts \ scripts/utils/log-sanitizer.ts scripts/utils/eval-skeleton-generator.ts \ scripts/utils/eval-from-log.ts npm run test:scripts npm run typecheck npm run build npm run lint:ciPre-Merge Checklist