Copilot: surface AI credit usage for token-based-billing seats - #2647
Open
KSEGIT wants to merge 17 commits into
Open
Copilot: surface AI credit usage for token-based-billing seats#2647KSEGIT wants to merge 17 commits into
KSEGIT wants to merge 17 commits into
Conversation
Copilot Business seats billed by credit report every quota as unlimited with zeroed entitlement/remaining, which previously rendered as a blank Copilot card. Decode credits_used and organization_login_list so a later task can build a usage model and UI on top. QuotaSnapshot gains a creditsUsed field. Because a token-based- billing snapshot is a placeholder by isPlaceholder's existing definition (entitlement == 0 && remaining == 0), QuotaSnapshots now captures premium_interactions' raw decode before the placeholder filter nils it out, storing creditsUsed separately so it survives. CopilotUsageResponse forwards it via premiumInteractionsCreditsUsed and threads the value through the quotaSnapshots rebuild in its own init(from:). Task 1 of the copilot-ai-credits plan; decoding only, no model or UI changes.
Implements CopilotCreditsUsage value type modeling credit consumption with Lane subtype for seat and organization scopes. Includes usedPercent computed property for usage rendering and full Codable/Equatable/Sendable conformance. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nai7wSG88xfy6HiMxYc4uM
Attach CopilotCreditsUsage as an optional field on UsageSnapshot so later tasks can populate and render it: stored property, CodingKeys, memberwise init (parameter + assignment, defaulted nil so no existing call site breaks), Codable persistence in init(from:)/encode(to:), and a Replacement case in replacing(...).
- CopilotUsageFetcher gains seatEntitlement: Double? = nil, placed before transport: so existing call sites keep compiling unchanged. - fetch() builds a CopilotCreditsUsage via a new makeCreditsUsage() helper and passes it through UsageSnapshot(copilotCredits:), riding alongside the existing steipete#1258 guards (unlimited-quota guard and tokenBasedBilling||hasUnlimitedQuota branch) without touching them. - Adds two tests: seat credits surface for token-billing accounts while primary/secondary stay nil (regression guard), and credits are omitted (with the percent bar intact) for metered accounts with no credits_used.
Best-effort fetcher for org-wide AI credit usage from GitHub's billing API. Returns nil on any failure (non-200, network error, malformed JSON, unbuildable URL) since the OAuth device flow only requests read:user and this call may be rejected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nai7wSG88xfy6HiMxYc4uM
.urlPathAllowed deliberately leaves "/" unescaped since it is meant for encoding a full multi-segment path, but org is a single path segment. An org value like "../../repos/x" passed through unchanged, letting an authenticated request be pointed at an arbitrary path on api.github.com. Escape "/" by subtracting it from the allowed set (matches the CopilotDeviceFlow.formEncode / AntigravityLoginRunner.urlQueryValueAllowed convention elsewhere in the repo), and add a regression test that verifies the separator survives on the wire as %2F. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nai7wSG88xfy6HiMxYc4uM
Adds CopilotCreditEntitlementParser (validates a user-entered credit allowance: blank/non-numeric/non-positive -> nil) and three new CopilotProviderSettings fields (orgCreditsEnabled, seatCreditEntitlement, orgCreditEntitlement). GitHub publishes no credit entitlement on any billing endpoint, so the denominator for a usage bar must be user-supplied. Entitlements are stored as raw strings in SettingsStore (copilotSeatCreditEntitlementRaw / copilotOrgCreditEntitlementRaw) and parsed only at the copilotSettingsSnapshot boundary, so a partially-typed value is never destroyed mid-edit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nai7wSG88xfy6HiMxYc4uM
Wires the seat entitlement into CopilotUsageFetcher and, when the user opts into org billing, fetches the org credit lane via CopilotOrgCreditsFetcher and merges it onto the snapshot. The org fetch is strictly best-effort: any missing toggle, missing org login, or failed fetch leaves the snapshot unchanged, so it can never degrade the seat lane, plan label, or budget windows. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nai7wSG88xfy6HiMxYc4uM
Task 7's mergingOrgLane unit test proved the pure merge function was correct, but nothing exercised addOrgCreditsIfNeeded itself, so a swapped entitlement field or a reordered toggle guard would have stayed green. Add three tests that drive the real fetchOutcome/fetch() path through CopilotProviderDescriptor: - the org lane merges with entitlement sourced from orgCreditEntitlement (not seatCreditEntitlement) and resetsAt sourced from the seat lane - the seat lane survives untouched when the org billing call is rejected (the common case: most tokens lack org billing access) - the org billing endpoint is never contacted when the toggle is off Reuses CopilotBudgetWebFetcherTests' existing CopilotBudgetBindingStubURLProtocol (extending its canInit allowlist) instead of registering a second global URLProtocol stub for the same /copilot_internal/user endpoint -- an earlier attempt at a dedicated stub class raced with it under Swift Testing's default parallel execution across suites, since `@Suite(.serialized)` only serializes within a suite, not across suites sharing global URLProtocol state. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nai7wSG88xfy6HiMxYc4uM
Adds copilotCreditMetrics to the general (non-antigravity) metrics path in MenuCardView.swift, appended after extraRateWindowMetrics as a separate, ungated call so credit rows are never hidden by copilotBudgetExtrasEnabled. A lane without a user-entered entitlement renders as a text row (statusText) rather than a bar with a fabricated denominator, per the intent of upstream issue steipete#1258. Settings gains a per-seat/org entitlement field pair and an "Organization AI credits" fetch toggle. Deviates from the task brief in two ways, both intentional: - The metrics builder is registered in MenuCardView.swift's `metrics(input:)` (right after its extraRateWindowMetrics append), not inside MenuCardView+ModelHelpers.swift's antigravityMetrics — that function only runs for .antigravity and never executes for .copilot. - formatCredits pins every NumberFormatter property explicitly rather than just the locale. en_US_POSIX alone still yields "3000" not "3,000" because it sets usesGroupingSeparator to false by default. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nai7wSG88xfy6HiMxYc4uM
Turning off "Organization AI credits" only refreshed the provider, so a failed follow-up refresh (offline, lost org access, 401) kept the last-good snapshot's org row rendering for a disabled feature. Add clearCopilotOrgCredits(), mirroring the sibling clearCopilotBudgetExtras() synchronous strip, and wire it into the toggle's else branch. Also adds UsageSnapshot.with(copilotCredits:), a public wrapper needed because the app-target UsageStore extension cannot reach the module-internal replacing(copilotCredits:) directly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nai7wSG88xfy6HiMxYc4uM
GitHub reports credits_used: 0 on metered (non-credit-billed) snapshots too, so the field's mere presence isn't exclusive to token-billed seats. Without a gate, any Copilot Pro/Individual seat could grow a permanent, unremovable "0 credits used" row. Thread hasUnlimitedQuota (already computed in fetch()) into makeCreditsUsage and only build the lane when it carries actual signal: token/unlimited billing, nonzero credits, or a user-configured entitlement. The steipete#1258 guards in fetch() and makeRateWindow are untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nai7wSG88xfy6HiMxYc4uM
The org-lane fetch swallowed every failure via try? with zero diagnostics, while the sibling budget-extras fetch logs a matching warning. Since a token without org billing access is the common case (device flow only requests read:user), the prior UX was "I enabled it and nothing happened." Split the single guard chain into three explicit failure points (transport error, non-200 status, decode failure), each logging "Copilot org credits unavailable" before returning nil. The best-effort return-nil behavior is unchanged. Also filter usageItems to unitType == "ai-credits" before summing: the endpoint is credit-scoped today, but an unrelated line item would otherwise silently inflate the total. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nai7wSG88xfy6HiMxYc4uM
A user in multiple orgs would silently see org steipete#1's numbers with no indication of which org the row described. Parenthesize the org login in the row title, following the .doubao team-title qualifier pattern already used elsewhere in MenuCardView+ModelHelpers.swift. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nai7wSG88xfy6HiMxYc4uM
The binding -> UserDefaults -> copilotSettingsSnapshot chain for the org-credits toggle and both entitlement fields was untested. Add three tests following the existing "openai exposes project id setting" pattern, including that the org entitlement field's isVisible follows the toggle. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nai7wSG88xfy6HiMxYc4uM
Add an Unreleased entry for steipete#2593 (thanks @KSEGIT), and document the org credits endpoint (headers, opt-in/best-effort framing), the seat lane's snapshot mapping (and why it isn't summed across snapshots), the new AI credit entitlements section explaining the user-entered denominators, and the two new key files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nai7wSG88xfy6HiMxYc4uM
Four small, independent cleanups ahead of the upstream PR: - Fix a stale "below" reference in a code comment; the .doubao team-title pattern it points to is above, not below. - Document the enterprise-host (GHES) API host in docs/copilot.md's org-credits section, matching section 2's phrasing -- the fetcher already routes through the same apiHost(enterpriseHost:) helper. - Add a test isolating the tokenBasedBilling disjunct in the seat-lane signal gate, so a Business account reporting zero credits early in the billing month still gets a seat lane even when no other disjunct is true. - Stop CopilotOrgCreditsFetcher from fabricating a zero when usageItems is non-empty but nothing matches unitType == "ai-credits"; that case now returns nil (unknown) instead of a misleading zero, while a genuinely empty usage list still returns zero.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aaeb632f41
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| else { return nil } | ||
| return CopilotDeviceFlow.makeRequestURL( | ||
| host: CopilotUsageFetcher.apiHost(enterpriseHost: enterpriseHost), | ||
| path: "/orgs/\(encoded)/settings/billing/ai_credit/usage") |
There was a problem hiding this comment.
For the organization AI-credit flow, GitHub's REST docs list this endpoint as GET /organizations/{org}/settings/billing/ai_credit/usage, not /orgs/{org}/... (checked https://docs.github.com/en/rest/billing/usage#get-billing-ai-credit-usage-report-for-an-organization). With the current path, authorized org admins receive a 404 and fetchCreditsUsed returns nil, so the new organization credits row never appears.
Useful? React with 👍 / 👎.
| return 0 | ||
| } | ||
|
|
||
| let creditItems = report.usageItems.filter { $0.unitType == "ai-credits" } |
There was a problem hiding this comment.
For organization AI-credit reports, GitHub's example response uses unitType: "credits", while "ai-credits" is shown for user-level reports (checked https://docs.github.com/en/rest/billing/usage#get-billing-ai-credit-usage-report-for-an-organization). After the path is corrected, real org responses will fall into the “none matched” branch here and return nil, so organization usage still will not render unless this accepts the org unit type.
Useful? React with 👍 / 👎.
clawsweeper
Bot
added
rating: 🦪 silver shellfish
|
Codex review: needs real behavior proof before merge. Reviewed August 4, 2026, 3:59 PM ET / 19:59 UTC. ClawSweeper reviewWhat this changesThis PR adds Copilot AI-credit decoding and menu rows, user-entered credit allowances, and an opt-in organization billing usage fetch. Merge readiness⛔ Blocked until real behavior proof is added - 13 items remain Keep open. The seat-credit portion addresses a source-proven Copilot gap, but the optional organization lane still lacks a supported authorization path, uses the previously flagged REST route and unit assumptions, and has no inspectable after-fix runtime proof. Priority: P2 Review scores
Verification
How this fits togetherCodexBar authenticates a Copilot account, fetches GitHub usage data, stores it in a provider snapshot, and renders that snapshot as menu-bar metrics. This PR adds credit-based metrics for token-billed seats and an optional organization-wide billing enrichment. flowchart TD
A[GitHub device login] --> B[Copilot usage response]
B --> C[Credit usage decoding]
C --> D[Provider usage snapshot]
D --> E[Menu bar credit rows]
A --> F[Optional organization billing request]
F --> D
Decision needed
Why: The standard login only obtains read:user, so adding organization billing is not a mechanical implementation choice; it changes the provider authorization and organization-data access contract. Before merge
Findings
Agent review detailsSecurityNone. Review metrics
Merge-risk optionsMaintainer options:
Technical reviewBest possible solution: Land the no-new-scope seat-credit metric as a focused change, and defer organization billing until maintainers approve an explicit authorization, consent, endpoint, and response-contract design. Do we have a high-confidence way to reproduce the issue? Yes for the underlying gap: current main explicitly suppresses token-billed quota bars and has no alternative credit field, matching the supplied account payload. The after-fix live behavior has not been independently demonstrated with an inspectable artifact. Is this the best way to solve the issue? No. Decoding seat credits from the existing response is a narrow solution, but coupling it to an organization billing fetch without an approved authorization path and verified response contract is not the safest implementation. Full review comments:
Overall correctness: patch is incorrect AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against a82f509ea8e7. LabelsLabel justifications:
EvidenceWhat I checked:
Likely related people:
Rating scale
Overall follows the weaker of proof and patch quality. Workflow
HistoryReview history (3 earlier review cycles)
|
Owner
|
Thanks for this, and for the excellent original report in #2593 — you found both the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2593.
The problem
On a Copilot Business seat with
token_based_billing: true,GET /copilot_internal/userreports every quota snapshot asunlimited: true, entitlement: 0, remaining: 0, percent_remaining: 100. The#1258guards correctly drop those so no misleading "0% used" bar appears — but the result is a Copilot card showing only the plan label and no usage at all.Real consumption is available, in two places CodexBar didn't read:
credits_used, already present on each quota snapshot in the response CodexBar already fetches.GET /orgs/{org}/settings/billing/ai_credit/usage— org-wide, per-model.Scope is larger than the issue described
#2593 said this would be "contained to the model decoder plus the fetcher's token-billing branch". That assumed a
NamedRateWindowwould work. It doesn't: Copilot'sextraRateWindowsare hidden entirely unless Budget extras is enabled (MenuCardView+ModelHelpers.swift:795), andusageKnown: falserenders as the literal string "Unavailable". Credit rows are core data, so they needed their own path. Flagging the change rather than letting it pass unremarked.GitHub publishes no credit entitlement anywhere
This is the finding that shaped the design. I probed all 8 documented billing endpoints plus
budgets,cost-centers,included_credits,ai_credit/entitlement,copilot/metricsandusage/summary. None expose the included-credit ceiling — the "6,000" that the org billing page displays.discountQuantityreveals only what included credits absorbed, so the ceiling is observable only once exceeded.It is derivable as seats × per-seat allowance, but the per-seat figure is currently a promotional 3,000 against a standard 1,900 — a hardcoded table would silently produce a wrong bar when the promo ends. So the denominator is user-entered and never inferred. A lane with no entitlement renders a text row, never a bar with a fabricated ceiling.
If you know of an endpoint I missed, that would simplify this considerably.
What this adds
Seat lane (default on). Numerator from
premium_interactions.credits_used— deliberately not summed across snapshots, sincechat/completionsmay report the same pool. No new endpoint and no new OAuth scope; it reads a field already in the existing response. Created only when it carries real signal (token-billed / unlimited quota / credits > 0 / entitlement set), so metered accounts reportingcredits_used: 0don't gain a permanent empty row.Org lane (opt-in, off by default).
GET /orgs/{org}/settings/billing/ai_credit/usage, org read fromorganization_login_list.first, summingusageItems[].grossQuantityfiltered tounitType == "ai-credits". Strictly best-effort, shaped like the existingaddBudgetWindowsIfNeeded: any failure returns the snapshot untouched. The device flow only requestsread:user, so rejection is the expected common case, not an error — it now logs a warning rather than failing silently.On my account the org endpoint returns 31.13 + 49.97 = 81.10, matching the billing page's numerator exactly.
Testing
swift test --filter Copilot— 125 tests / 13 suites greenmake check— 0 violations across 1,706 filesmake test— full suite greenProviderHTTPTransportStubor the existingURLProtocolstub. No live network, no Keychain access, per AGENTS.md.Regression coverage worth calling out: a test asserts the
#1258behaviour still holds — a Business token-billing payload yields credits andprimary == nil, secondary == nil. No guard line inCopilotUsageFetcher.swiftis modified anywhere in this branch.Two tests came out of review rather than the plan, and both were mutation-checked (the change was reverted to confirm the test actually fails): the
tokenBasedBillingdisjunct in the seat-lane gate, and the org-path percent-encoding.Known gaps
L("AI credits"),L("Org credits"),L("credits used")have noLocalizable.stringsentries.Scripts/check-app-locales.mjsenforces all 23 catalogs together, so a partial addition failsmake check— left out deliberately rather than done badly. Happy to add all 23 here if you'd prefer.CLIRendererdoesn't rendercopilotCredits, socodexbar usagestill shows nothing for these accounts.copilotCreditMetricwould be smaller and instant — happy to rework if you prefer that shape.Commits
17 focused commits, each with its own tests, left unsquashed so they can be reviewed in sequence. Squash on merge if you'd rather.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Nai7wSG88xfy6HiMxYc4uM