Repository navigation
Route four pinned Codex workflows with cost/economy model routing - #75470
Conversation
Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅
|
|
✅ Test Quality Sentinel completed test quality analysis. Test Quality SentinelOutcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
✅ PR Specification Reviewer completed successfully! PR Specification ReviewerOutcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft. Design Decision Gate 🏗️Outcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
✅ Ponytail Reviewer completed successfully! Ponytail ReviewerOutcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
✅ PR Code Quality Reviewer completed the code quality review.
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Reviewed the changed workflow routing declarations and associated test update. I found no actionable correctness, security, or maintainability issues in the changed lines. Static review only; repository tests were not run.
🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~3 AIC · ⌖ ~1 AIC · ⊞ 1.6K
Comment /review to run again
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are reported for the economy-routing changes across the four Codex workflows.
No open issues 🎉
What changed in this PR
This PR extends gh-aw’s existing cost/economy model routing to four previously pinned Codex workflows, addressing #75469 while preserving their providers and reporting behavior.
Changes:
- Replaces fixed models with
goal: costandmode: economy, leaving permissions, prompts, triggers, imports, and safe outputs unchanged. - Regenerates four locks with router setup, runtime model metadata, and default threat-detection model resolution.
- Updates the routing regression test to recognize the two additional native OpenAI workflows.
| File | Description |
|---|---|
pkg/workflow/model_routing_native_test.go |
Adds two OpenAI workflows to expected native routing providers. |
.github/workflows/schema-consistency-checker.md |
Replaces the OpenAI model pin with economy routing. |
.github/workflows/schema-consistency-checker.lock.yml |
Adds native OpenAI routing and runtime model reporting. |
.github/workflows/issue-triage-agent.md |
Replaces the Copilot model pin with economy routing. |
.github/workflows/issue-triage-agent.lock.yml |
Adds Copilot routing and runtime model reporting. |
.github/workflows/daily-spending-forecast.md |
Replaces the Copilot model pin with economy routing. |
.github/workflows/daily-spending-forecast.lock.yml |
Adds Copilot routing and runtime model reporting. |
.github/workflows/daily-cli-tools-tester.md |
Replaces the OpenAI model pin with economy routing. |
.github/workflows/daily-cli-tools-tester.lock.yml |
Adds native OpenAI routing and runtime model reporting. |
Review effort: Balanced (auto)
There was a problem hiding this comment.
Skills-Based Review 🧠
Applied /tdd (test coverage of the routing allowlist) and /diagnosing-bugs (checking the generated lock output). Verdict: COMMENT. The config changes look correct. One test gap and one open security alert need a decision before merge.
📋 Key Themes & Highlights
Key Themes
- Test gap:
TestModelRoutingCopilotLocksUnchangedchecks the provider of each routed lock it finds, but never requires the four workflows in this PR to be routed. See the inline comment. - Open security alert (not blocking here): the "Prepare model-routing conversation" step writes the rendered prompt to
/tmp/gh-aw/routing-conversation.json. That prompt can contain issue or PR text fetched through tools. The step is compiler-generated (pkg/workflow/model_routing.go) and already appears in base routed locks such asauto-triage-issues, so this PR does not add the pattern. It does newly apply it todaily-cli-tools-tester, which reads untrusted issue and PR content. Please resolve the alert or record why it is accepted. The pre-fetched alert text was truncated, so this is based on the lock file.
Verification
TestModelRoutingCopilotLocksUnchangedandTestNativeRoutingSmokeEvidencepass at PR head6994471.- Recompiling the four workflows from a clean export of HEAD matches the committed locks, apart from cron scatter minutes.
- The
detectionalias is defined inpkg/workflow/data/model_aliases.json, so threat detection resolves as the PR describes.
Positive Highlights
- ✅ Frontmatter changes are minimal and match the existing routed workflows.
- ✅ Test allowlist is updated in the same commit as the locks.
@copilot please address the review comments above.
🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · routed: haiku55 xhigh · ~7 AIC · ⌖ ~1 AIC · ⊞ 5.4K
Comment /matt to run again
|
Copilot assignment requested for the following outstanding work:
Once you start work, post this work summary as a PR comment without the leading Copilot mention, including the Sous-chef head, work, and state lines below, to record that the assignment is running. Apply the necessary fixes on this PR's existing branch, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved; stop when only human review or CI remains. Do not merge or trigger CI. Sous-chef requested-head: 6994471
|
Work in progress:
Sous-chef requested-head: 6994471 |
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
…-cost-economy Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
…rge main; recompile Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
Done in f86614d. All four locks now have
CI on the new commit has only just started, so no failures yet. The CodeQL check timed out and didn't produce results. |
|
Copilot assignment requested for the following outstanding work:
Once you start work, post this work summary as a PR comment without the leading Copilot mention, including the Sous-chef head, work, and state lines below, to record that the assignment is running. Apply the necessary fixes on this PR's existing branch, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved; stop when only human review or CI remains. Do not merge or trigger CI. Sous-chef requested-head: f86614d
|
Work in progress:
Sous-chef requested-head: f86614d |
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
…-cost-economy Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
| GH_AW_ROUTING_PROMPT: /tmp/gh-aw/aw-prompts/user.txt | ||
| GH_AW_ROUTING_PROMPT_FALLBACK: /tmp/gh-aw/aw-prompts/prompt.txt | ||
| GH_AW_ROUTING_CONVERSATION_FILE: /tmp/gh-aw/routing-conversation.json | ||
| run: node -e "const fs=require('node:fs'); const path=require('node:path'); const readPrompt=file=>{try{return fs.readFileSync(file,'utf8');}catch(error){if(error.code==='ENOENT')return '';throw error;}}; let prompt=readPrompt(process.env.GH_AW_ROUTING_PROMPT); if (!prompt.trim()) prompt=readPrompt(process.env.GH_AW_ROUTING_PROMPT_FALLBACK); if (!prompt.trim()) throw new Error('Rendered workflow prompt is empty; cannot route this task'); const destination=process.env.GH_AW_ROUTING_CONVERSATION_FILE; fs.mkdirSync(path.dirname(destination),{recursive:true,mode:0o700}); fs.writeFileSync(destination,JSON.stringify([{role:'user',parts:[{text:prompt}]}]),{mode:0o600});" |
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
|
The failing Please undo the changes made for it, and keep this PR to the routing change:
Afterwards the PR should change only the four workflows' |
| id: pre_agent_audit | ||
| continue-on-error: true | ||
| run: bash "${RUNNER_TEMP}/gh-aw/actions/audit_pre_agent_workspace.sh" | ||
| - name: Prepare model-routing conversation |
…kflows" This reverts commit 2805816. Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
…-cost-economy Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
Done in b4971e1 (revert of 2805816) and 5b0fd45 (merge of main, then recompile). |
| id: pre_agent_audit | ||
| continue-on-error: true | ||
| run: bash "${RUNNER_TEMP}/gh-aw/actions/audit_pre_agent_workspace.sh" | ||
| - name: Prepare model-routing conversation |
…-cost-economy # Conflicts: # .github/workflows/daily-cli-tools-tester.lock.yml # .github/workflows/daily-spending-forecast.lock.yml # .github/workflows/issue-triage-agent.lock.yml # .github/workflows/schema-consistency-checker.lock.yml Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
Four Codex workflows were pinned to large models and cost about 1,800 AIC/week, most of it Daily CLI Tools Tester. This PR replaces the pins with
goal: cost,mode: economymodel routing through the Copilot provider.daily-cli-tools-tester.mdopenai→githubopenai/gpt-6-astraschema-consistency-checker.mdopenai→githubopenai/gpt-6.1-soldaily-spending-forecast.mdgithub(unchanged)copilot/gpt-6.1-solissue-triage-agent.mdgithub(unchanged)copilot/gpt-6-solWhy Copilot, not the OpenAI key
In AWF today, routing on the OpenAI key can only pick
gpt-5.4: it's the only OpenAI model with effort levels in AWF's model data (github/gh-aw-firewall#10030). The first test runs here pickedopenai/gpt-5.4:xhighfor both OpenAI workflows (Schema Consistency Checker 89 AIC vs ~24 before). Copilot routing can pick from every Copilot model, including Luna.Changes
model:pins are removed; each engine block isid: codex,model-provider: github,model-routingwithgoal: cost,mode: economy.copilot-requests: writeadded to Daily CLI Tools Tester and Schema Consistency Checker, so the run's own token can call Copilot models. This replaces use of the OpenAI key.TestModelRoutingCopilotLocksUnchangednow also checks each of the four locks keeps Copilot routing.Test runs on this branch
upload_assetstep that also fails onmain)gpt-5.6-luna:max, 10.6 AIC vs ~204 before. It published findings issue [cli-tools-test] [exploratory] Strict dry-run compile rejects trusted workflows and logs omit completion timestamps #75504 and endedreport_incomplete, as on most of itsmainruns.gpt-5.6-sol:max, 248 AIC vs ~26 before (about 10× more). Kept on routing on purpose, as evidence for the next router tables.Judge Daily CLI Tools Tester by its findings issue, not the run conclusion.