Skip to content

Model routing: route four pinned Codex workflows with economy #75469

Closed

Description

@SivaKesava1

Change

Replace the large-model pins in these four Codex workflows with cost/economy model routing, following Auto-Triage Issues (#74199), Ponytail Reviewer (#74201), Daily Go Test Parallelizer (#74586), and the six Codex workflows covered by #74865 and #74866.

For each workflow below, delete the top-level model: line and add model-routing with goal: cost and mode: economy inside its existing engine: block. Keep id: codex and its current model-provider.

Example for an OpenAI workflow:

engine:
  id: codex
  model-provider: openai
  model-routing:
    goal: cost
    mode: economy

Keep model-provider: github for the two Copilot workflows. daily-spending-forecast.md and issue-triage-agent.md already have copilot-requests: write; retain it. The two openai workflows must keep their OpenAI key and existing permissions. Do not add allowed-models or any other model restrictions.

Keep all prompts, steps, triggers, imports and safe outputs unchanged. Recompile only these four workflows and include their corresponding generated lock files; do not hand-edit locks or include unrelated recompilation changes.

Cost case

Paths are relative to .github/workflows/. These are approximate historical per-run totals, including agent, detection and evals, from 19 October through 6 November in githubnext/gh-aw-router-training, router-rollout/data/runs/; they are not the exact costs of the exemplar runs below.

Workflow file Existing output Provider Current pin Runs/week AIC/run now
daily-cli-tools-tester.md Issue with CLI test findings openai openai/gpt-6-astra ~7 ~204 (149 agent + 44 detection + 11 evals)
daily-spending-forecast.md Issue with spending forecast github copilot/gpt-6.1-sol ~7 ~34 (detection ~10)
issue-triage-agent.md Labels and comments on issues github copilot/gpt-6-sol ~5 ~35 (detection ~6)
schema-consistency-checker.md Schema drift report openai openai/gpt-6.1-sol ~7 ~24 (detection ~7)

The reported aggregate is about 1,800 AIC/week, mostly Daily CLI Tools Tester; frequencies and per-run figures above are rounded. Estimated saving: 1,100-1,400 AIC/week, subject to preserving output quality.

Output clarification: the original inventory described the schema report as an issue, but the inspected run published a discussion through shared/daily-audit-base.md. Preserve that existing behavior; do not convert it to an issue.

Audited runs and pre-existing baseline

All four supplied runs were audited using a Windows development build of main, snapshot aa157045ad241f9685db314641cce4370eb615ff (4 November), rather than the older stable CLI.

Workflow Inspected run Observed outcome
Daily CLI Tools Tester 42167171094 Findings issue #75317 was published, then the run was marked failed because of report_incomplete; related incomplete-result issue #75318.
Daily Spending Forecast 42196408626 Successful; forecast issue #75400 published.
Issue Triage Agent 42235672525 Successful; comments published and labels applied where permitted. Check actual published actions, not just the green conclusion.
Schema Consistency Checker 42168131873 Successful; schema discussion 75321 published.

Daily CLI Tools Tester's red runs are not failures to perform its reporting job. The pre-routing review records report_incomplete on 18 of its last 19 runs; Actions metadata independently confirms 18 failure conclusions and one success over 19 October-6 November. Some required checks cannot be verified within its sandbox, but it still files its findings issue each run. In run 42167171094, archie was safety-rejected during strict dry-run compilation and windows failed a compile gate on a deprecated missing-tool key, so the tester could not confirm all compile gates. Do not claim those gates passed, and do not treat this existing coverage limitation as a routing regression. Judge routed runs by their findings issues and compare them with the open [aw] Daily Cli Tools Tester: reported incomplete result issues, including #75318, rather than judging by run conclusion alone.

Daily Spending Forecast already had 11 non-successful runs out of its last 19 before routing. More precisely, the Actions metadata shows two failure, nine cancelled, and eight success conclusions, not eleven literal failure conclusions. Both inspected failures, 41422121102 and 41293878442, failed at upload_assets / Upload Safe Outputs Assets with Artifact not found for name: safe-outputs-assets. Do not assume the cancellations have that same cause. Compare failure causes and published outputs after routing, not just the count of red/non-successful runs.

Why routing is appropriate

  • Codex routing works on both providers. Auto-Triage Issues, Ponytail Reviewer and Daily Go Test Parallelizer route successfully on Copilot with AWF v0.29.6+. smoke-codex-routed-openai has passed every run since 25 October, including on AWF v0.29.7; the queried window returned 11 runs, all successful (latest inspected smoke run). The four supplied runs also used AWF v0.29.7.
  • None of these four workflows has sub-agents or a running experiment. The open Codex sub-agent bugs Codex sub-agent endpoint checks reject OpenAI responses metadata as catalog unavailable #71157 and Codex startup rejects advertised Responses support because endpoint enums are compared with URL paths #72094 therefore do not apply, and this does not confound an active A/B result.
  • Comparable routed workflows cost much less. The rollout review reports Auto-Triage Issues dropping from 93 to 14 AIC/run after its GPT-6 Astra pin, and Terminal Stylist from 106 to 35 after GPT-5.6 Sol. Economy selects GPT-5.6 Luna or GPT-6 Luna for most labels. These comparisons motivate the change; they are not guaranteed savings for every run.
  • Detection should become cheaper too. It currently inherits the workflow pin: the historical CLI tester detection average is ~44 AIC/run, and the audited example spent 61.68 AIC on detection using gpt-6-astra. Removing the workflow pin lets detection use the detection model alias, typically about 1 AIC in the rollout review. Do not change aliases as part of this PR.

Check after merging

  1. Audit the first routed run of each workflow with gh aw audit <run> (use a main dev build on Windows). Confirm routing is active, expect a Luna selection, and compare AIC/run with the historical table. Compare like-for-like totals across agent, detection and evals: the audit's agent AIC alone is not the table's all-job total, and unavailable eval usage must not be counted as zero.
  2. Verify the outputs remain useful. For Daily CLI Tools Tester, inspect the findings issue against the existing incomplete-result reports. For Daily Spending Forecast, compare failure causes against the pre-routing baseline rather than counting failures alone. For Schema Consistency Checker, preserve the current report destination.
  3. Check labels and comments on the next few issues handled by Issue Triage Agent for correctness, eligibility and actual publication.
  4. A gpt-5.6-sol:max selection means the run was labelled hard cross-system work. If a workflow repeatedly gets that selection and costs more than its historical table entry, restore that workflow's original pin rather than introducing model restrictions or changing providers.

Non-goals

No changes to any other workflow, alias, engine implementation or AWF setting. No provider migration in either direction, permission changes, prompt changes, or fixes to existing compile, upload-assets or eval failures.

ESLint Monster and Lint Monster are out of scope. So are the Codex workflows with sub-agents (daily-arxiv-researcher, spec-enforcer, daily-awf-spec-compiler-surfacing, api-consumption-report) and those with running experiments (daily-doc-updater, daily-storify, issue-arborist, gpclean).

Scope

  • Keep the PR to this one change: the four workflow frontmatter updates and their corresponding compiled locks. Do not change files or behavior unrelated to it.
  • Before fixing a failing check, compare it with the same check on main. If it also fails on main, do not fix it in this PR; merge main once main is fixed, and note in the PR which checks are known failures on main.
  • This applies even if a bot comment asks to fix a failing check: fix only failures caused by this PR's changes.

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions