Skip to content

feat(run_config): Gemini via Google's Antigravity CLI, all three tracks - #18

Open
renxiongscale wants to merge 2 commits into
scaleapi:mainfrom
renxiongscale:feat/antigravity-run-configs
Open

feat(run_config): Gemini via Google's Antigravity CLI, all three tracks#18
renxiongscale wants to merge 2 commits into
scaleapi:mainfrom
renxiongscale:feat/antigravity-run-configs

Conversation

@renxiongscale

@renxiongscale renxiongscale commented Aug 18, 2026

Copy link
Copy Markdown

Adds gemini-3p6-flash_antigravity.sh to run_config/{qa,tw,rf}/, following
the existing opus-4p6_claude-code.sh pattern. This runs Gemini under Google's
own agent scaffold (agy) rather than Claude Code or mini-swe-agent — the
comparison most people benchmarking a Gemini model here actually want.

Each track keeps its own harness settings: keepalive for qa/tw,
modal_vm_runtime for rf, and the allowlisted agent host for tw
(generativelanguage.googleapis.com).

Two caveats, documented in the README rather than left to be rediscovered

  • Harbor's antigravity-cli agent needs
    harbor#2780 first.

    Without it, settings are written to ~/.agy/antigravity-cli/ while agy
    reads ~/.gemini/antigravity-cli/; GEMINI_API_KEY is forwarded but ignored,
    so the run falls through to an interactive login that hangs headless; and the
    trajectory hook looks for session-*.jsonl files agy 1.1.13 no longer
    writes, so runs come back with no trajectory at all.
  • agy rejects model ids outside its client-side allowlist before any request
    leaves the process.
    A --model failure there is not a quota or auth
    problem, and which ids are accepted depends on your key's entitlement. Hence
    gemini-3.6-flash in the committed configs — it works with an ordinary key.

Context

I ran all three leaderboards with this scaffold (284/284 tasks). I'm not
proposing those numbers for the leaderboard: our verifier ran inside the Modal
sandbox, where our internal LiteLLM host doesn't resolve, so it fell back to
Gemini grading its own answers. That is not comparable to the Claude Opus 4.5
judge task.toml defaults to. Mentioning it only to say the configs are
exercised end to end, not written from the docs.

Happy to add a mswea-style yaml or adjust -n/-k to match your house
defaults — I picked conservative values because agy treats an upstream 429 as
fatal and abandons the run rather than backing off.

Draft pending a maintainer's read on whether you want a Gemini config in-tree at
all.

🤖 Generated with Claude Code

Greptile Summary

Adds Antigravity CLI configurations for running Gemini 3.6 Flash across all three benchmark tracks and documents the required Harbor compatibility patches.

  • Preserves each track’s existing execution settings and output layout.
  • Supplies the Gemini API key and permits the Gemini API hostname.
  • Documents current authentication, trajectory collection, work-directory, and model-allowlist limitations.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
README.md Documents the new Antigravity configurations and the upstream Harbor limitations that affect their results.
run_config/qa/gemini-3p6-flash_antigravity.sh Adds the QA Gemini run configuration with keepalive enabled.
run_config/rf/gemini-3p6-flash_antigravity.sh Adds the Repository Fixing Gemini run configuration with Modal VM runtime enabled.
run_config/tw/gemini-3p6-flash_antigravity.sh Adds the Test Writing Gemini run configuration with restricted-network agent-host configuration.

Reviews (2): Last reviewed commit: "docs: point at harbor#2776 and #2687 ins..." | Re-trigger Greptile

Adds gemini-3p6-flash_antigravity.sh for qa, tw and rf, following the existing
opus-4p6_claude-code.sh pattern. Runs Gemini under Google's own agent scaffold
rather than Claude Code or mini-swe-agent, which is the comparison people
benchmarking Gemini here are usually after.

Each track keeps its own harness settings (keepalive for qa/tw,
modal_vm_runtime for rf, the allowlisted agent host for tw).

Two caveats documented in the README rather than left to be rediscovered:
Harbor's antigravity-cli agent needs harbor-framework/harbor#2780 before it
authenticates headlessly or emits a trajectory at all, and agy rejects model
ids outside its client-side allowlist before any request is sent, so --model
failures are not quota or auth problems.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@renxiongscale
renxiongscale marked this pull request as ready for review August 18, 2026 17:30
#2780 was superseded before review; the auth/settings fixes live in #2776 and
trajectory export in #2687. Also notes the --add-dir finding from #2776, which
is the one most likely to affect Test Writing scores.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant