Skip to content

fix: exclude zero-spend (skipped) runs from token-optimizer AIC ranking#122

Open
mrfelton wants to merge 1 commit into
githubnext:mainfrom
LN-Zap:fix/optimizer-exclude-zero-aic-runs
Open

fix: exclude zero-spend (skipped) runs from token-optimizer AIC ranking#122
mrfelton wants to merge 1 commit into
githubnext:mainfrom
LN-Zap:fix/optimizer-exclude-zero-aic-runs

Conversation

@mrfelton

Copy link
Copy Markdown

Problem

The Aggregate top workflows by AIC usage step in workflows/agentic-token-optimizer.md keeps every completed run, including runs whose agent skipped (activation/preconditions) and therefore consumed 0 AIC. Those zero-spend runs:

  • pad run_count for skip-heavy workflows,
  • dilute avg_ai_credits (the 0s drag the average down), and
  • can surface never-spent workflows into the top-10 that drives target selection.

High-frequency, event-driven workflows (e.g. label/PR-triggered) are exactly this case — many completed runs, agent skipped on most.

Full write-up in #120. Distinct from (but compounding) the self-targeting guard in #119 / #121.

Fix

 [.runs[]
   | select(.status == "completed")
+  | select((.aic // 0) > 0)
   | { workflow_name, ai_credits: (.aic // 0), ... }
 ]

A "top workflows by AIC spend" ranking should only count runs that actually spent AIC. Applied to both workflows/ (template) and .github/workflows/ (active) copies.

⚠️ Compiler-version dependency — please read before merging

This is only safe once the compiler reliably computes AIC. On gh-aw < v0.79.4 — including v0.72.1, which this repo currently pins — the Copilot-engine 0-AIC bug is present (provider github_models not normalized; fixed in v0.79.4 via github/gh-aw#38314, with #38364/#38327/#38353/#38412). There, genuinely-expensive Copilot runs can report aic == 0, so this filter would drop real spend.

Recommendation: pair this with / gate it on bumping these workflows' compiler to ≥ v0.79.4. Once there, aic == 0 reliably means "agent skipped / no consumption" and the filter is correct. We run it in production on v0.79.4.

Notes

Fixes #120

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com

The "Aggregate top workflows by AIC usage" step keeps every completed
run, including runs whose agent skipped via activation/preconditions and
consumed 0 AIC. Those zero-spend runs pad run_count, dilute
avg_ai_credits, and can surface never-spent workflows into the top-10
that drives target selection. High-frequency event-driven workflows
(e.g. label-triggered) are mostly these 0-AIC completed runs.

Add `select((.aic // 0) > 0)` after the status filter so a "top
workflows by AIC spend" ranking only counts runs that actually spent.

NOTE: only correct once the compiler reliably computes AIC. On gh-aw
< v0.79.4 (incl. v0.72.1 used here) the Copilot 0-AIC bug
(github/gh-aw#38314, fixed in v0.79.4) makes real-spend Copilot runs
report 0 AIC, so this filter would drop genuine spend. Pair with /
gate on a compiler bump to >= v0.79.4. Details in githubnext#120.

Source `.md` only — regenerate `.lock.yml` in-repo (fork-built locks
don't match this repo's cron jitter / action-pin resolution).

Fixes githubnext#120

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Token Optimizer ranks zero-spend runs: top-workflows aggregation should exclude runs that consumed 0 AIC (skipped via activation)

1 participant