feat(run_config): Gemini via Google's Antigravity CLI, all three tracks - #18
Open
renxiongscale wants to merge 2 commits into
Open
feat(run_config): Gemini via Google's Antigravity CLI, all three tracks#18renxiongscale wants to merge 2 commits into
renxiongscale wants to merge 2 commits into
Conversation
Adds gemini-3p6-flash_antigravity.sh for qa, tw and rf, following the existing opus-4p6_claude-code.sh pattern. Runs Gemini under Google's own agent scaffold rather than Claude Code or mini-swe-agent, which is the comparison people benchmarking Gemini here are usually after. Each track keeps its own harness settings (keepalive for qa/tw, modal_vm_runtime for rf, the allowlisted agent host for tw). Two caveats documented in the README rather than left to be rediscovered: Harbor's antigravity-cli agent needs harbor-framework/harbor#2780 before it authenticates headlessly or emits a trajectory at all, and agy rejects model ids outside its client-side allowlist before any request is sent, so --model failures are not quota or auth problems. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
renxiongscale
marked this pull request as ready for review
August 18, 2026 17:30
#2780 was superseded before review; the auth/settings fixes live in #2776 and trajectory export in #2687. Also notes the --add-dir finding from #2776, which is the one most likely to affect Test Writing scores. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
gemini-3p6-flash_antigravity.shtorun_config/{qa,tw,rf}/, followingthe existing
opus-4p6_claude-code.shpattern. This runs Gemini under Google'sown agent scaffold (
agy) rather than Claude Code or mini-swe-agent — thecomparison most people benchmarking a Gemini model here actually want.
Each track keeps its own harness settings:
keepalivefor qa/tw,modal_vm_runtimefor rf, and the allowlisted agent host for tw(
generativelanguage.googleapis.com).Two caveats, documented in the README rather than left to be rediscovered
antigravity-cliagent needsharbor#2780 first.
Without it, settings are written to
~/.agy/antigravity-cli/whileagyreads
~/.gemini/antigravity-cli/;GEMINI_API_KEYis forwarded but ignored,so the run falls through to an interactive login that hangs headless; and the
trajectory hook looks for
session-*.jsonlfilesagy1.1.13 no longerwrites, so runs come back with no trajectory at all.
agyrejects model ids outside its client-side allowlist before any requestleaves the process. A
--modelfailure there is not a quota or authproblem, and which ids are accepted depends on your key's entitlement. Hence
gemini-3.6-flashin the committed configs — it works with an ordinary key.Context
I ran all three leaderboards with this scaffold (284/284 tasks). I'm not
proposing those numbers for the leaderboard: our verifier ran inside the Modal
sandbox, where our internal LiteLLM host doesn't resolve, so it fell back to
Gemini grading its own answers. That is not comparable to the Claude Opus 4.5
judge
task.tomldefaults to. Mentioning it only to say the configs areexercised end to end, not written from the docs.
Happy to add a
mswea-style yaml or adjust-n/-kto match your housedefaults — I picked conservative values because
agytreats an upstream 429 asfatal and abandons the run rather than backing off.
Draft pending a maintainer's read on whether you want a Gemini config in-tree at
all.
🤖 Generated with Claude Code
Greptile Summary
Adds Antigravity CLI configurations for running Gemini 3.6 Flash across all three benchmark tracks and documents the required Harbor compatibility patches.
Confidence Score: 5/5
The PR appears safe to merge.
No blocking failure remains.
Important Files Changed
Reviews (2): Last reviewed commit: "docs: point at harbor#2776 and #2687 ins..." | Re-trigger Greptile