Fix Data Explorer dashboard savings formulas and FX fallback

[Michael Flanakin (flanakin)]

Summary

Addresses the remaining Phase 1.3 items from #2093 that were left open after PR #2103 shipped the currency-filtering fix:

1. Savings formulas — replaced inline sum(ListCost) - sum(EffectiveCost) style calculations with sum() of the canonical, row-level clamped columns (x_TotalSavings, x_CommitmentDiscountSavings, x_NegotiatedDiscountSavings) across 21 tiles in dashboard.json.
2. Savings-plan utilization FX distortion — removed the silent 1:1 exchange-rate fallback in the data layer. Reverted — see note below.

Why

Savings formulas (dashboard.json)

Costs/CostsPlus/CostsByMonth/CostsByDay already carry x_TotalSavings, x_CommitmentDiscountSavings, and x_NegotiatedDiscountSavings per row, computed with built-in clamping (e.g. x_NegotiatedDiscountSavings = iff(ListCost < ContractedCost, decimal(0), ListCost - ContractedCost)). Several dashboard tiles instead summed the raw cost columns first and then subtracted (sum(ListCost) - sum(EffectiveCost)), which skips that clamping and can diverge from the canonical values (e.g. when a row's contracted cost temporarily exceeds list cost due to data timing/rounding).

Fixed by summing the canonical x_*Savings columns directly wherever a query aggregates over Costs* and then derives savings. Two tiles (Savings breakdown by month, Effective cost breakdown by month) were already locally re-deriving the same iff() clamp formula redundantly — simplified those to just sum the canonical columns that already exist on the source table, removing the redundant extend.

Left unchanged: the Summary of list, contracted, and effective cost alignment data-quality diagnostic table (Data ingestion page). That tile intentionally computes raw, unclamped deltas (ListCost - ContractedCost, etc.) to detect data-quality anomalies like ListCost too low or ContractedCost should be 0 — using clamped values there would defeat its purpose.

Status: 15 of the 21 tiles were code-reviewed and verified clean as originally shipped — no further changes. The other 6 tiles required a follow-up fix (below) after review feedback.

Follow-up: reconciling on-screen equations on 6 tiles

@rkrummenacher and Brett Wilson (@MSBrett) independently flagged that 6 tiles display an equation or decomposition the user can visually verify on screen (for example, "List − Contracted = Negotiated savings", or Total savings ÷ List cost = Effective Savings Rate). On those tiles, summing the clamped per-row savings columns for the result side while leaving the operand side (ListCost/ContractedCost/EffectiveCost) as raw aggregates broke that on-screen reconciliation — the two sides of the displayed equation no longer matched, because clamping doesn't distribute over sums (sum(clamp(a-b)) + sum(clamp(b-c)) isn't always sum(clamp(a-c))).

Rather than reverting these 6 tiles to the original raw-subtraction formulas (which would silently reintroduce the original clamping bug for just these tiles), the fix derives the displayed ListCost/ContractedCost operands from the same clamped, row-level basis as the results:

• EffectiveCost (raw sum) is the only unclamped anchor — it's never negative-clamped.
• ContractedCost' = EffectiveCost + sum(x_CommitmentDiscountSavings)
• ListCost' = ContractedCost' + sum(x_NegotiatedDiscountSavings)
• TotalSavings' = sum(x_NegotiatedDiscountSavings) + sum(x_CommitmentDiscountSavings) (the sum of the two already-clamped components, not the independently clamped x_TotalSavings)

This makes both sides of each displayed equation reconcile exactly, using clamped math throughout — it does not go back to the raw-subtraction bug the original PR fixed.

Affected tiles:

• Savings breakdown, monthly and daily (full List → Contracted → Effective waterfall)
• Total/Negotiated/Commitment savings breakdown bar
• Cost + savings running total stacked area chart (documented the derivation in a comment; its own stacking already reconciled internally, no formula change needed)
• Effective Savings Rate, monthly and daily

This changes the Effective Savings Rate (ESR) definition on the two ESR tiles, since ESR's TotalSavings and ListCost inputs are now derived the same way. A release note has been added under FinOps hubs v15 in docs-mslearn/toolkit/changelog.md.

Savings-plan utilization FX distortion (data layer) — reverted

This PR originally also changed CommitmentDiscountQuantity derivation (HubSetup_v1_2.kql, IngestionSetup_v1_2.kql) to require isnotempty(x_BillingExchangeRate) before dividing by it, falling through to real(null) otherwise, to avoid silently assuming a 1:1 rate when the exchange rate was missing.

That change has been reverted after a closer look at the actual execution path found the original justification didn't hold:

• The edited branch (CommitmentDiscountCategory == 'Spend', EffectiveCost / x_BillingExchangeRate) sits inside a case() that short-circuits earlier — at isnotempty(CommitmentDiscountQuantity) — for rows that already have a quantity, and the EA/legacy raw-source ingestion paths cited as the motivating scenario also null out CommitmentDiscountQuantity upstream, so they short-circuit at that first branch and never reach the edited line at all.
• x_BillingExchangeRate is an "Azure 1.0-preview(v1)+" FOCUS column that Cost Management commonly leaves null for single-currency billing accounts (no exchange rate needed when PricingCurrency == BillingCurrency) — which is the common case, not the exception. For those accounts, assuming a rate of 1.0 is correct, not a guess.
• Requiring isnotempty(x_BillingExchangeRate) therefore made single-currency accounts (the majority) silently drop rows from CommitmentDiscountQuantity-based aggregates — sum() ignores nulls, so utilization percentages lose rows from their denominator without any visible signal. That trades a bounded, visible inaccuracy (a wrong-but-present number under the old code) for a silent aggregate hole (a confidently wrong percentage) under the new code — a regression for the common case, not an improvement.
• There's also a secondary effect: CommitmentDiscountUnit derives from isempty(CommitmentDiscountQuantity), so nulling the quantity also blanks the unit column for the same rows.

Restored the original coalesce(x_BillingExchangeRate, real(1)) fallback in both files. Only these two lines (plus the comment that justified the now-reverted change) were touched — the savings-formula fix in dashboard.json is untouched and remains in scope.

Out of scope

Per the issue, this PR does not address:

• BillingCurrency visibility in detail tables (only the Purchases table currently shows it).
• Phase 2 USD-normalization toggle (blocked on x_*InUsd column coverage validation across ingestion paths).

Verification

• dashboard.json validated as well-formed JSON before and after edits.
• src/powershell/Tests/Unit/HubsKqlOperators.Tests.ps1 and src/powershell/Tests/Lint/KqlJoinKinds.Tests.ps1 pass (578/578).
• Manually verified the derived-operand fix against the counterexample raised in review — two rows (List, Contracted, Effective) = (100,120,80) and (100,80,60): clamped sums give NegotiatedDiscountSavings=20, CommitmentDiscountSavings=60; derived EffectiveCost=140 → ContractedCost'=200 → ListCost'=220; List' - Contracted' = 20 ✓ matches Negotiated; Contracted' - Effective = 60 ✓ matches Commitment; TotalSavings' = 80 and ESR = 80/220 ≈ 36.4%, both self-consistent with the displayed operands.

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Roland Krummenacher (RolandKrummenacher)]

Reviewed the refactor and validated the assumptions by running the before/after formulas against three production hubs I have read access to (read-only queries, no identifying details below). The mechanical KQL refactor is clean, but the behavior change is materially larger than the description suggests, and it breaks the tiles that render savings as a visible equation.

Summary

The core issue is that x_TotalSavings, x_CommitmentDiscountSavings, and x_NegotiatedDiscountSavings are clamped per row. Summing them is therefore not equivalent to subtracting the aggregates — it is a different metric, one that structurally cannot net out credits, refunds, and corrections. On one hub the last-3-months total savings comes out ~40% higher after this change; on another the divergence is ~0.04%. So the blast radius is entirely tenant-dependent, which makes shipping it silently riskier, not safer.

That may well be the metric we want. But it is a product decision that belongs in #2093 with a release note, not a side effect of a "use the canonical column" refactor.

Details in the inline comments. Ordered by severity:

1. Savings-breakdown tiles now display arithmetic that does not add up — operands stayed raw, results became clamped.
2. Parts no longer sum to the whole — negotiated + commitment overstates total by ~61% on one hub.
3. Effective savings rate silently inflated — clamped numerator over raw denominator.
4. The FX guard likely regresses non-Microsoft FOCUS ingestion — and is a no-op on every Azure hub I checked.
5. The auditability claim in the description only holds for one of the two code paths.

What is right

• The refactor itself is correct KQL. The union/pivot tiles properly re-sum the already-renamed columns in the second summarize.
• Dropping the redundant local extend re-derivations in 3b3f0a58 and 0341b3e4 is a genuine simplification — the latter's were entirely dead code.
• Removing now-unused ListCost/ContractedCost from summarize lists (e.g. 2d7b6447) is a real efficiency win on wide scans.
• Leaving the Summary of list, contracted, and effective cost alignment diagnostic on raw unclamped deltas is the right call, and the reasoning in the description is sound.

Two things not introduced here, but now load-bearing

The two clamp formulas disagree. HubSetup_v1_2.kql:169-171 uses iff(ContractedCost < EffectiveCost, real(0), ...); IngestionSetup_v1_2.kql:706-708 uses iff(isempty(ListCost) or ListCost == 0 or ListCost - EffectiveCost < 0.0001, real(0), ...). Pre-existing, but this PR makes 21 tiles depend on those columns, so a hub with mixed v1.0/v1.2 data will now get two different clamping semantics inside a single displayed number. Worth aligning while we are here.

Dashboard/hub version coupling. I verified via getschema that a hub running a current released version does not yet expose the x_*Savings columns on Costs_v1_2() — they come from this branch. Fine for a normal template deploy, but importing the new dashboard.json against a not-yet-upgraded hub now produces hard query failures on all 21 tiles rather than degraded output. Probably worth a release-note line.

Recommendation

Blocking on 1-3: either keep raw subtraction in the tiles that render savings as a visible equation, or derive the displayed operands from the clamped relationship so the math closes. 4 should be narrowed to a currency-equality guard before merge. 5 is a follow-up.

Happy to take the tile fixes if that is useful.

[Roland Krummenacher (RolandKrummenacher)]

Blocking — the tile now renders arithmetic that does not add up.

This tile (e17346d1) and its CostsByDay twin (f1ef29df) literally display three equations:

Cost without discounts  -  After negotiated discounts  =  Negotiated savings
After negotiated disc.  -  After commitment discounts  =  Commitment savings
Cost without discounts  -  After commitment discounts  =  Total savings

The change leaves the operands as raw sum(ListCost) / sum(ContractedCost) / sum(EffectiveCost) but swaps the results to clamped sum(x_*Savings). Those disagree.

Measured on a production hub, last 3 months, applying this tile's own x_AmortizationClass != 'Principal' filter:

Row	vs. the subtraction shown directly above it
Negotiated savings	+25.9%
Commitment savings	+53.8%
Total savings	+40.2%

So the tile renders something like 2,283,331 - 2,056,289 = 285,782. That is not a rounding artifact — a user will read the two numbers above the result and see it is wrong.

Either keep the raw subtraction here (this tile's whole purpose is showing the derivation), or also derive the displayed ListCost/ContractedCost/EffectiveCost from the clamped relationship so the equation closes.

[Michael Flanakin (flanakin)]

Fixed by deriving the displayed operands from the same clamped, row-level basis as the results, rather than reverting to raw subtraction (which would have silently reintroduced the original clamping bug this PR set out to fix for this tile).

EffectiveCost (raw sum) is the only value that's never negative-clamped, so it anchors the chain:

• ContractedCost' = EffectiveCost + sum(x_CommitmentDiscountSavings)
• ListCost' = ContractedCost' + sum(x_NegotiatedDiscountSavings)
• TotalSavings' = sum(x_NegotiatedDiscountSavings) + sum(x_CommitmentDiscountSavings) (sum of the two clamped components, not the independently-clamped x_TotalSavings)

All three displayed equations now hold exactly: List' - Contracted' = Negotiated, Contracted' - Effective = Commitment, List' - Effective = TotalSavings'. Verified against your counterexample rows (100,120,80) and (100,80,60): Negotiated=20, Commitment=60, Effective=140 → Contracted'=200 → List'=220; 220-200=20 ✓, 200-140=60 ✓, 220-140=80=TotalSavings' ✓.

The other tiles' clamped-sum fix is unaffected by this change.

[Roland Krummenacher (RolandKrummenacher)]

Same defect as e17346d1 above — this is the CostsByDay twin of the savings-breakdown tile, with the same three rendered equations whose operands stayed raw while the results became clamped. Whatever fix lands on the monthly tile needs to land here too.

[Michael Flanakin (flanakin)]

Fixed by deriving the displayed operands from the same clamped, row-level basis as the results, rather than reverting to raw subtraction (which would have silently reintroduced the original clamping bug this PR set out to fix for this tile).

EffectiveCost (raw sum) is the only value that's never negative-clamped, so it anchors the chain:

• ContractedCost' = EffectiveCost + sum(x_CommitmentDiscountSavings)
• ListCost' = ContractedCost' + sum(x_NegotiatedDiscountSavings)
• TotalSavings' = sum(x_NegotiatedDiscountSavings) + sum(x_CommitmentDiscountSavings) (sum of the two clamped components, not the independently-clamped x_TotalSavings)

All three displayed equations now hold exactly: List' - Contracted' = Negotiated, Contracted' - Effective = Commitment, List' - Effective = TotalSavings'. Verified against your counterexample rows (100,120,80) and (100,80,60): Negotiated=20, Commitment=60, Effective=140 → Contracted'=200 → List'=220; 220-200=20 ✓, 200-140=60 ✓, 220-140=80=TotalSavings' ✓.

The other tiles' clamped-sum fix is unaffected by this change.

[Roland Krummenacher (RolandKrummenacher)]

Blocking — the parts no longer sum to the whole.

This tile shows Total / Negotiated / Commitment side by side. Because each x_*Savings column clamps independently at row level, x_NegotiatedDiscountSavings + x_CommitmentDiscountSavings != x_TotalSavings.

On a production hub across the full dataset, negotiated + commitment came out ~61% larger than total. Previously the three were consistent by construction, since all came from the same aggregates.

A user looking at this tile will see two components that visibly exceed the total they are supposed to decompose.

[Michael Flanakin (flanakin)]

Fixed. This tile doesn't display List/Contracted/Effective operands, only the three savings bars (Total, Negotiated, Commitment), so the fix here is narrower than the waterfall tiles: TotalSavings is now derived as sum(x_NegotiatedDiscountSavings) + sum(x_CommitmentDiscountSavings) instead of the independently-clamped sum(x_TotalSavings), so the Total bar always equals Negotiated + Commitment shown alongside it — the parts sum to the whole again, by construction.

Verified against your counterexample rows: Negotiated=20, Commitment=60 → Total'=80, which is internally consistent (no longer the independently-clamped 60 that didn't match 20+60=80).

The other tiles' clamped-sum fix is unaffected by this change.

[Roland Krummenacher (RolandKrummenacher)]

Related to the parts-vs-whole problem: this area chart stacks EffectiveCost + CommitmentDiscountSavings + NegotiatedDiscountSavings. Previously that stack summed to exactly ListCost by construction, so the chart height was meaningful. With independently clamped columns it no longer does, and the stacked total will overshoot list cost by a tenant-dependent amount.

[Michael Flanakin (flanakin)]

Looked at this one closely: the stack itself (EffectiveCost base layer + sum(x_CommitmentDiscountSavings) + sum(x_NegotiatedDiscountSavings), both already clamped sums per this PR) reconciles internally by construction — a stacked area chart's top is always the sum of its layers, so the stack top already equals EffectiveCost + Commitment + Negotiated, the same derived "cost without discounts" quantity used in the waterfall/ESR tiles after their fix. No formula change was needed here.

What I did change: added a comment in the query clarifying that the stack top is this derived figure, not the raw sum(ListCost) KPI shown elsewhere on the page — so nobody cross-checking the stack height against a separate raw List cost tile is confused by a legitimate difference between the two (your 220 vs 200 example). That raw/derived distinction is now called out explicitly at the top of the query.

The other tiles' clamped-sum fix is unaffected by this change.

[Roland Krummenacher (RolandKrummenacher)]

Effective savings rate is silently inflated here.

EffectiveSavingsRate = sum(x_TotalSavings) / sum(ListCost) now mixes a clamped numerator with a raw denominator, so the ratio is no longer internally consistent.

On a production hub, last 3 months, the reported ESR rose by roughly 40% relative to the pre-change value. Broken out by month the instability is worse — one month moved +230% because a single large negative-savings correction got clamped to zero instead of netting out, while adjacent months moved 11-17%.

ESR is a headline FinOps KPI. A step change of that size with no release note will read as a regression to customers, and the month-to-month series becomes non-comparable across the upgrade boundary.

[Michael Flanakin (flanakin)]

Fixed by deriving TotalSavings and ListCost from the same clamped, row-level basis as the other tiles rather than mixing a clamped numerator with a raw denominator:

• EffectiveCost = raw sum (never negative-clamped)
• TotalSavings' = sum(x_NegotiatedDiscountSavings) + sum(x_CommitmentDiscountSavings)
• ListCost' = EffectiveCost + TotalSavings'
• EffectiveSavingsRate = TotalSavings' / ListCost'

This is a genuine ESR definition change, as you and MSBrett both flagged — the numerator and denominator are now internally consistent with each other and with the other savings tiles on the dashboard, but the resulting rate will differ from the pre-PR raw-subtraction ESR whenever any row was clamped, so it isn't just a formula cleanup. Added a release note under FinOps hubs v15 in docs-mslearn/toolkit/changelog.md calling out the definition change explicitly, per your ask.

Verified against the counterexample: Effective=140, TotalSavings'=80 → ListCost'=220 → ESR = 80/220 ≈ 36.4%, self-consistent with the displayed operands.

[Roland Krummenacher (RolandKrummenacher)]

Same ESR issue as e1bc2d51 — clamped numerator over raw denominator, on the CostsByDay variant.

[Michael Flanakin (flanakin)]

Fixed by deriving TotalSavings and ListCost from the same clamped, row-level basis as the other tiles rather than mixing a clamped numerator with a raw denominator:

• EffectiveCost = raw sum (never negative-clamped)
• TotalSavings' = sum(x_NegotiatedDiscountSavings) + sum(x_CommitmentDiscountSavings)
• ListCost' = EffectiveCost + TotalSavings'
• EffectiveSavingsRate = TotalSavings' / ListCost'

This is a genuine ESR definition change, as you and MSBrett both flagged — the numerator and denominator are now internally consistent with each other and with the other savings tiles on the dashboard, but the resulting rate will differ from the pre-PR raw-subtraction ESR whenever any row was clamped, so it isn't just a formula cleanup. Added a release note under FinOps hubs v15 in docs-mslearn/toolkit/changelog.md calling out the definition change explicitly, per your ask.

Verified against the counterexample: Effective=140, TotalSavings'=80 → ListCost'=220 → ESR = 80/220 ≈ 36.4%, self-consistent with the displayed operands.

[Roland Krummenacher (RolandKrummenacher)]

Re-reviewed at 0e05f98. The KQL revert resolves the two threads it touches — replied there, no objection. The PR is now a pure dashboard.json change, +21/-21.

The six dashboard threads are untouched by this commit and all still apply, so keeping changes requested. Consolidating them here, since they're one root cause rather than six independent problems.

Root cause: the change swaps raw subtractions for sum(x_*Savings) on the result side of each tile, while leaving sum(ListCost) / sum(ContractedCost) / sum(EffectiveCost) raw on the operand side. The x_*Savings columns clamp negatives at row level, so the two sides no longer reconcile.

Tile	Symptom, measured on a production hub
Savings breakdown, monthly (e17346d1) + daily (f1ef29df)	the three equations the tile literally displays don't hold: +25.9%, +53.8%, +40.2%
Total / Negotiated / Commitment (line 3074)	components exceed the total they decompose by ~61%
Stacked savings area chart (line 2876)	stack no longer sums to ListCost, overshoots by a tenant-dependent amount
Effective Savings Rate (e1bc2d51 + daily twin)	clamped numerator over raw denominator: ESR +40% relative, one month +230%

One decision settles all four: either keep the raw subtractions on these tiles, or derive the displayed cost operands from the clamped columns so the arithmetic closes. I don't have a strong preference as long as each tile is internally consistent — a user reading two numbers and their stated difference shouldn't see a third number that isn't the difference.

ESR is the one I'd treat as separate from the arithmetic question: it's a headline KPI, and a step change of that size across an upgrade boundary makes the month-over-month series non-comparable. If the clamped definition is the one we want, that needs a release note rather than just a formula change.

Happy to re-measure whichever direction you pick.

[Brett Wilson (MSBrett)]

Requesting changes. I re-reviewed the current head (0e05f98b) from the base diff, traced the savings fields to their source definitions, checked all 21 changed queries, and independently reproduced the arithmetic failures below.

The replacement of aggregate subtraction with sums of row-level clamped savings is not semantics-preserving for visuals that still rely on exact accounting relationships:

1. The monthly and daily savings-summary tiles display false equations (dashboard.json:3086 and :3092). They still show raw aggregate values for List, Contracted, and Effective cost, followed by subtraction and equals signs, but the displayed results now sum independently clamped row values. Whenever a row is clamped, the displayed operands no longer produce the displayed result.

2. The Total / Negotiated / Commitment summary is no longer a valid decomposition (dashboard.json:3074). Independently clamped negotiated and commitment savings are not additive and can sum to more than total savings.

3. The stacked Cost + savings chart no longer reconciles to list cost (dashboard.json:2876). It stacks raw effective cost with both independently clamped savings categories, so the stack can overstate the implied cost without discounts.

A minimal counterexample confirms all three failures: for rows (List, Contracted, Effective) = (100,120,80) and (100,80,60), the raw negotiated difference is 0 while summed clamped negotiated savings is 20; negotiated plus commitment savings is 80 while total savings is 60; and the stacked value is 220 against list cost of 200.

The clamped Effective Savings Rate formula is not inherently invalid—it matches the repository's savings-summary-report.kql precedent—but it is a material KPI-definition change rather than a mechanical refactor. If retained for ESR and standalone savings visuals, the changed semantics and historical discontinuity should be explicitly approved and documented in release notes.

Please either keep aggregate subtraction for visuals that present reconciled equations/components, or redesign those visuals and labels so the independently clamped metrics are not presented as arithmetic identities. Also update the PR title/body now that the FX change has been reverted and remove the claim that the savings portion was verified clean while these blocking threads remain unresolved.