[gjtorikian]

Summary

• Adds a fifth eval suite, discovery, for vendor-neutral tasks whose outcome is categorical (which product the agent chose) rather than pass/fail. It is never a headline suite: HEADLINE_SUITES is unchanged, so validation, framework expansion, planning and the leaderboard aggregate all pick it up from the one constant with no store migration or schema bump.
• Makes the Matrix view's suite tabs show discovery. The view builds its tabs from a hard-coded list, so a suite missing there was invisible even though the report carried it. The list and hover titles are now exported, and the duplicated available-suite filter is one pure helper pinned by a new Matrix.test.ts.
• Adds Runtime.egress: "proxied" | "open". An "open" runtime skips the egress proxy for that experiment's attempts even under --egress-proxy, so an arm that must reach the open internet can share a run with proxied arms instead of needing a second hosted run.
• The opt-out is one condition in resolvePairEnvironments; the Docker layer is untouched because each attempt already gates its internal network and sidecar on its pair's egress.proxy. The choice lands in the pair environment (identity) and can be exempted in a comparison group with treatmentPolicyFields: ["egress"]. Open pairs still record their (inert) allowlist, write no denials log, and report egress as not instrumented. defineExperiment rejects "open" on a host runtime.

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; the previous open-arm gateway failure is fixed.

Summary

Adds the discovery suite and lets container experiments opt out of a run’s egress proxy.

• Discovery evals appear in run results without changing headline scores.
• Container experiments can skip the egress proxy within a run.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Resolve experiment and run settings] --> B{Container with proxy enabled and not open?}
  B -->|Yes| C[Combine runtime, adapter, and endpoint hosts]
  C --> D[Start attempt proxy with resolved allowlist]
  B -->|No| E[Record runtime and adapter hosts without endpoint hosts]
  E --> F[Run without attempt proxy]
  A --> G[Record non-secret endpoint URLs in pair identity]

Loading

_{Reviews (2) · Last reviewed commit: "fix(runtime): keep endpoint-override hos..." · Reviewed by Greptile}