feat: discovery suite and per-experiment egress opt-out

docs/concepts.md

@@ -58,7 +58,10 @@ There is no registry and no special dialect: the scorer is ordinary code, and
 every eval is self-contained. The `suite` field tiers evals: **benchmark**
 and **regression** are the headline suites the leaderboard counts; **smoke**
 holds evals every arm solves, run to catch breakage but never moving the
-headline; **other** is anything else outside it. A framework-expanded eval can
+headline; **discovery** holds vendor-neutral tasks whose outcome is
+categorical (which product the agent chose) rather than pass/fail — each
+attempt records its outcome and is never graded into the headline; **other**
+is anything else outside it. A framework-expanded eval can
 retier one variant with `suites:` (a saturated port goes to smoke while the
 rest stay benchmark). [Writing an eval](./writing-an-eval.md) is the
 authoring reference.
@@ -243,8 +246,8 @@ the matrix banner and the text report's `suspect:` lines display.
 Each entry in `report.json`'s `experiments[]` carries an `aggregate`: one
 leaderboard row per experiment, rendered as a table above the matrix and in
 the text report. Every number in it except SDK recall counts **headline-suite
-attempts only** (benchmark and regression), so smoke and other evals never
-move the ranking. That is why the row's `cost_usd` can be lower than the
+attempts only** (benchmark and regression), so smoke, discovery and other
+evals never move the ranking. That is why the row's `cost_usd` can be lower than the
 run-wide `telemetry.cost_usd`. Its columns and their denominators:
 
 - **Gate**: whether the all-eval pass rate is at or above the report's
@@ -345,7 +348,7 @@ exactly what a negative control should do.
 | **pair**            | One eval × experiment combination — the report's row unit (a *cell* in the UI).              |
 | **run**             | One `quickstudy run`, all pairs and trials, under one run id.                                |
 | **manifest**        | The immutable content-addressed identity record written before attempts start.               |
-| **suite**           | Eval tier: `benchmark` and `regression` (headline), `smoke` (saturated, outside it), `other`. |
+| **suite**           | Eval tier: `benchmark` and `regression` (headline), `smoke` (saturated, outside it), `discovery` (categorical, vendor-neutral, outside it), `other`. |
 
 Next: [Getting started](./getting-started.md) ·
 [Writing an eval](./writing-an-eval.md)

docs/extensions.md

@@ -107,9 +107,18 @@ Unsupported streams remain unavailable/null; zero means an applicable stream was
 observed with no matching event. Offered surfaces and observed cross-arm use remain
 separate in reports and diagnosis.
 
-`--egress-proxy` applies to every container. The allowlist is the pair's runtime
+`--egress-proxy` applies to every container unless its experiment opts out with
+`runtime.egress: "open"`, which skips the proxy for that experiment's attempts
+inside a proxied run (no internal network, no sidecar, no proxy environment), so
+an arm that must reach the open internet can share a run with proxied arms. The
+choice is recorded in each pair environment, so it is identity; exempt it in a
+comparison group with `treatmentPolicyFields: ["egress"]`. An open pair still
+records its (inert) allowlist of runtime and adapter hosts, writes no denials
+log and reports egress as not instrumented; endpoint-override hosts join only
+proxied pairs, so an IP-literal gateway on an open arm never stops the run.
+Container runtimes only: `defineExperiment` rejects `"open"` on a host runtime. The allowlist is the pair's runtime
 hosts plus that adapter's provider endpoints; absent runtime hosts means provider
-endpoints only. Each attempt has its own internal network and egress proxy. A
+endpoints only. Each proxied attempt has its own internal network and egress proxy. A
 shared MCP auth proxy has its own infrastructure traffic, which is not attributed
 to agent attempts. Egress denial artifacts contain sanitized host identifiers and
 attempt IDs. An empty log means zero observed denials; an absent log means unavailable

docs/writing-an-eval.md

@@ -40,7 +40,7 @@ frontmatter schema is deliberately tiny and closed:
 | Key          | Required | Meaning                                                                                                                                                     |
 | ------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
 | `id`         | yes      | Must equal the directory name (prevents silent aliasing).                                                                                                   |
-| `suite`      | yes      | `benchmark` or `regression` (the headline suites the leaderboard counts), `smoke` (evals every arm solves; run, never counted), or `other`.                     |
+| `suite`      | yes      | `benchmark` or `regression` (the headline suites the leaderboard counts), `smoke` (evals every arm solves; run, never counted), `discovery` (vendor-neutral tasks with a categorical outcome; recorded, never counted), or `other`. |
 | `product`    | no       | Free-form product label, used for reporting.                                                                                                                |
 | `framework`  | no       | The app stack the task targets (`nextjs`, `rails`, `django`, …); container runtimes and `--framework` filters read it.                                      |
 | `frameworks` | no       | List of framework tokens; expands the directory into one eval per framework (id `<dirname>-<framework>`). See [Framework variants](#framework-variants-frameworks). |

src/evals/types.ts

@@ -8,10 +8,12 @@
 /**
  * Closed set of eval suites. `benchmark` and `regression` are the headline
  * suites: the leaderboard aggregates count both. `smoke` holds evals every
- * arm solves — run to catch breakage, never moving the headline. `other` is
- * anything else outside the headline.
+ * arm solves — run to catch breakage, never moving the headline. `discovery`
+ * holds vendor-neutral tasks whose outcome is categorical (which product was
+ * chosen) rather than pass/fail; recorded per attempt, never headline.
+ * `other` is anything else outside the headline.
  */
-export const EVAL_SUITES = ["benchmark", "regression", "smoke", "other"] as const;
+export const EVAL_SUITES = ["benchmark", "regression", "smoke", "other", "discovery"] as const;
 export type EvalSuite = (typeof EVAL_SUITES)[number];
 
 /** The suites whose attempts count toward the leaderboard aggregates. */

src/execution-policy.ts

@@ -53,6 +53,13 @@ export function resolvePairEnvironments(options: {
   images?: Record<string, RuntimeImageIdentity>;
   egressProxy?: boolean;
   adapterHosts?: Record<string, readonly string[]>;
+  /**
+   * Per experiment: the hosts of the agent's endpoint overrides. They join
+   * the allowlist only for pairs the proxy fronts, where the list is live and
+   * an IP-literal gateway is refused loudly by design; an open or unproxied
+   * pair never carries them, so such a gateway cannot stop the run.
+   */
+  adapterEndpointHosts?: Record<string, readonly string[]>;
   /** Per experiment: the agent's non-secret endpoint overrides (see PairEnvironment.endpoints). */
   adapterEndpoints?: Record<string, Record<string, string>>;
   budgets?: Partial<LifecycleBudgets>;
@@ -73,16 +80,23 @@ export function resolvePairEnvironments(options: {
         options.imageTags?.[pairKey(loaded.metadata.id, experiment.id)] ??
         runtime.image ??
         runtime.containerImage?.(loaded.metadata);
+      // With the flag, every container is proxied unless its runtime opts
+      // out with egress: "open". No runtime hosts means only the adapter's
+      // provider endpoints are permitted. An open pair still records the
+      // runtime and adapter hosts (inert) so identity holds if the run flag
+      // flips, but endpoint-override hosts join only a live allowlist: the
+      // proxy refuses IP literals, and an IP-literal gateway on an arm the
+      // proxy never fronts must not stop the run.
+      const proxied = runtime.kind === "container" && (options.egressProxy ?? false) && runtime.egress !== "open";
       pairs[pairKey(loaded.metadata.id, experiment.id)] = {
         ...(files ? { workspaceFiles: workspaceFilesIdentity(files) } : {}),
         image: tag ? (options.images?.[tag] ?? { tag, id: "unresolved", repoDigests: [] }) : null,
-        // With the flag, every container is proxied. No runtime hosts means
-        // only the adapter's provider endpoints are permitted.
         egress: {
-          proxy: runtime.kind === "container" && (options.egressProxy ?? false),
+          proxy: proxied,
           allowlist: composeAllowlistFromHosts([
             ...(runtime.egressHosts?.(loaded.metadata) ?? []),
             ...(options.adapterHosts?.[experiment.id] ?? []),
+            ...(proxied ? (options.adapterEndpointHosts?.[experiment.id] ?? []) : []),
           ]),
         },
         webPolicy: runtime.webPolicy?.() ?? "native-web-blocked",

src/experiments/types.ts

@@ -55,6 +55,15 @@ export interface Runtime extends RuntimeCapabilities {
    * declare the same fields. Source/config identity still changes per attempt.
    */
   treatmentPolicyFields?: Array<"egress" | "webPolicy" | "pathTreatment" | "mcp" | "promptPreamble" | "workspaceFiles">;
+  /**
+   * Whether this experiment's attempts go through the run's egress proxy.
+   * "open" skips the proxy sidecar and network for these attempts even when
+   * the run was started with --egress-proxy, so an arm that must reach the
+   * open internet can share a run with proxied arms. The choice is recorded
+   * in each pair environment (identity); exempt it in a comparison group with
+   * treatmentPolicyFields: ["egress"]. Container runtimes only.
+   */
+  egress?: "proxied" | "open";
   /** Attempt-local non-secret files, staged without replacing the normal adapter. */
   workspaceFiles?: WorkspaceFiles;
   /** Container image tag (container runtimes; alternative to `containerImage`). */
@@ -127,6 +136,11 @@ export function defineExperiment(experiment: Experiment): Experiment {
   if (experiment.comparisonGroup !== undefined && experiment.comparisonGroup.trim() === "") {
     throw new Error(`defineExperiment: experiment "${experiment.id}" comparisonGroup must not be empty`);
   }
+  if (kind === "host" && experiment.runtime.egress === "open") {
+    throw new Error(
+      `defineExperiment: experiment "${experiment.id}" runtime.egress "open" is only meaningful on container runtimes`,
+    );
+  }
   validateWorkspaceFilesRuntime(experiment.runtime);
   return deepFreeze(experiment);
 }

src/report/aggregate.ts

@@ -3,8 +3,8 @@
  * the same attempts and (banded) pairs as the rest of the report.
  *
  * Every number is restricted to headline-suite attempts (HEADLINE_SUITES:
- * benchmark and regression), so smoke and other suites never move the
- * headline row. Telemetry follows the report's rule: sums and means cover
+ * benchmark and regression), so the smoke, discovery and other suites never
+ * move the headline row. Telemetry follows the report's rule: sums and means cover
  * only attempts that reported a value, and are null (never 0) when none did.
  */
 import { isHeadlineSuite } from "../evals/types.ts";

src/runner/execute.ts

@@ -471,20 +471,19 @@ export async function executeRunV3(options: ExecuteRunV3Options): Promise<RunSum
       if (scorer.identity) scorerIdentities[loaded.metadata.id] = scorer.identity;
     }
     // Each agent's endpoint overrides set on the host: the non-secret base
-    // URLs join the pair environment (identity), their hosts join the egress
-    // allowlist — but only under the proxy, where the allowlist is live and an
-    // IP-literal gateway is refused loudly by design; without it the list is
-    // inert and an IP-literal override must not stop the run.
+    // URLs join the pair environment (identity); their hosts reach the egress
+    // allowlist only for pairs the proxy fronts (pair resolution decides: the
+    // run flag on and the runtime not opted out), where the list is live and
+    // an IP-literal gateway is refused loudly by design. Elsewhere the list
+    // is inert and an IP-literal override must not stop the run.
     const overridesByExperiment = new Map([...containerAgents].map(([id, adapter]) => [id, collectOptionalHostEnv([adapter])] as const));
     const adapterEndpoints = Object.fromEntries([...overridesByExperiment].flatMap(([id, overrides]) => {
       const endpoints = Object.fromEntries(Object.entries(overrides).filter(([name]) => !isSecretEnvName(name)));
       return Object.keys(endpoints).length > 0 ? [[id, endpoints] as const] : [];
     }));
     const pairs = resolvePairEnvironments({ ...options, workspaceFilesByExperiment, images, imageTags: imageTagsByPair,
-      adapterHosts: Object.fromEntries([...containerAgents].map(([id, adapter]) => [id, [
-        ...(adapter.egressHosts ?? []),
-        ...(options.egressProxy ? endpointHosts(overridesByExperiment.get(id) ?? {}) : []),
-      ]])),
+      adapterHosts: Object.fromEntries([...containerAgents].map(([id, adapter]) => [id, [...(adapter.egressHosts ?? [])]])),
+      adapterEndpointHosts: Object.fromEntries([...overridesByExperiment].map(([id, overrides]) => [id, endpointHosts(overrides)])),
       adapterEndpoints,
       budgets: { ...options.budgets, agentMs: options.attemptTimeoutMs ?? options.budgets?.agentMs ?? attemptTimeoutMs },
       mcpHashes: Object.fromEntries([...declaredByExperiment].map(([id, servers]) => [id, hashString(canonicalJson(servers))])),

tests/evals-discovery.test.ts

@@ -110,7 +110,7 @@ describe("eval discovery", () => {
   test("suite outside the closed set is rejected", () => {
     const root = makeRoot();
     writeEval(root, "bad-suite", { frontmatter: "---\nid: bad-suite\nsuite: production\n---\n" });
-    expect(() => discoverEvals(root)).toThrow(/"suite" must be one of: benchmark, regression, smoke, other/);
+    expect(() => discoverEvals(root)).toThrow(/"suite" must be one of: benchmark, regression, smoke, other, discovery/);
   });
 
   test("an empty task body is rejected", () => {
@@ -279,7 +279,7 @@ describe("eval discovery", () => {
     writeEval(unknown, "multi", {
       frontmatter: `---\nid: multi\nsuite: benchmark\nframeworks:\n  - alpha\nsuites:\n  alpha: headline\n---\n`,
     });
-    expect(() => discoverEvals(unknown)).toThrow(/"suites" entry for "alpha" must be one of: benchmark, regression, smoke, other/);
+    expect(() => discoverEvals(unknown)).toThrow(/"suites" entry for "alpha" must be one of: benchmark, regression, smoke, other, discovery/);
   });
 
   test("smoke is a suite of its own", () => {
@@ -288,6 +288,20 @@ describe("eval discovery", () => {
     expect(discoverEvals(root)[0]!.metadata.suite).toBe("smoke");
   });
 
+  test("the discovery suite validates on its own and through a suites: retier", () => {
+    const root = makeRoot();
+    writeEval(root, "which-vendor", { frontmatter: `---\nid: which-vendor\nsuite: discovery\n---\n` });
+    expect(discoverEvals(root)[0]!.metadata.suite).toBe("discovery");
+    const expanded = makeRoot();
+    writeEval(expanded, "multi", {
+      frontmatter: `---\nid: multi\nsuite: discovery\nframeworks:\n  - alpha\n  - beta\nsuites:\n  beta: benchmark\n---\n`,
+    });
+    expect(discoverEvals(expanded).map((e) => [e.metadata.id, e.metadata.suite])).toEqual([
+      ["multi-alpha", "discovery"],
+      ["multi-beta", "benchmark"],
+    ]);
+  });
+
   test("importEvalScorer validates the default export is a function", async () => {
     const root = makeRoot();
     const good = writeEval(root, "good");