feat(evalbench): add native Agent Runtime generator support and deployment guide

[arieljassan]

Summary

This PR adds native support for evaluating SQL-generating agents deployed to Agent Runtime (Gemini Enterprise Agent Platform) inside the evalbench pipeline. It allows developers to deploy their ADK agents and run full evaluations (correctness and execution scoring) directly against the remote live endpoints.

Key changes

1. Core Generator:
   • Implemented AgentRuntimeGenerator (evalbench/generators/models/agent_runtime.py) which manages client authentication and queries remote Reasoning Engine endpoints using the AI Platform SDK.
   • Registered the "agent_runtime" model key in the generator factory (evalbench/generators/models/__init__.py).
2. Pipeline Safety Fix:
   • Updated evalbench/work/sqlexecwork.py to use safe dict retrieval (self.experiment_config.get("dialect") instead of self.experiment_config["dialect"]). This prevents KeyError runner crashes on configs using list-based dialects (such as BIRD SQLite configs).
3. Deployment guide and example:
   • Created docs/examples/agent_runtime_deployment.ipynb — a step-by-step Colab-friendly notebook explaining how to define, deploy, test, and evaluate a minimal ADK agent against the BIRD dataset.
4. Configuration templates:
   • Created default configurations datasets/model_configs/agent_runtime.yaml and datasets/bird/test_agent_runtime_run_config.yaml to run pipeline evaluations.
5. Testing:
   • Added unit tests evalbench/test/agent_runtime_test.py covering client authorization setup, remote request payloads, parameters parsing, and response envelope extraction (raw SQL strings vs. JSON envelopes).

[arieljassan]

/gcbrun

[IsmailMehdi]

Nice addition — the generator is clean and factoring the stream parsing into standalone helpers makes it genuinely testable. I ran the new tests locally (make proto + CI PYTHONPATH) and all 8 pass.

Flagging three things I think block a merge. Details inline.

1. _extract_sql doesn't handle ```sql fenced blocks — which is the exact format the accompanying notebook instructs the agent to emit.
2. The committed run config points at a dataset file and a scorer script that don't exist in the repo.
3. The config templates hardcode a personal sandbox project and a live Agent Engine resource ID.

I have a few non-blocking notes too (env var names diverging from util.gcp, the blanket except Exception defeating rate_limit retries, google-cloud-aiplatform not being a declared dependency, and the same ["dialect"] KeyError still present at interactsqlexecwork.py:113) — happy to write those up separately if useful.

[arieljassan]

Thanks for your thorough review and the non-blocking comments. I've addressed all of them in the latest commit:

• GCP Env Vars: Updated agent_runtime.py to use get_gcp_project and get_gcp_region from util.gcp directly, aligning the environment variables with the rest of the codebase.
• Rate Limiting: Replaced the blanket except Exception in generate_internal by catching ResourceExhausted specifically and raising ResourceExhaustedError so that the caller's retry/backoff wrapper behaves correctly.
• Dependencies: Added google-cloud-aiplatform to the dependency list in pyproject.toml.
• KeyError: Safeguarded the dialect check in interactsqlexecwork.py:113 using .get("dialect") to align with the fix in sqlexecwork.py and prevent KeyErrors.

[arieljassan]

/gcbrun

[IsmailMehdi]

Two blockers in the deployment notebook — both make the documented end-to-end flow fail as written. Details inline.