Skip to content

fix(ladder): convert existing coverage to on-demand-equivalent dollars and never claim legacy tranches - #1238

Merged
IAmJSD merged 8 commits into
LeanerCloud:mainfrom
IAmJSD:fix/issue-946
Oct 26, 2026
Merged

IAmJSD merged 8 commits into
LeanerCloud:mainfrom
IAmJSD:fix/issue-946

Conversation

@IAmJSD

@IAmJSD IAmJSD commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The ladder compared an on-demand-equivalent (OD) target with existing Savings Plan coverage measured in commitment dollars. This PR converts existing and expiring coverage to OD dollars before Allocate. Following the maintainer's note on the issue, it also makes legacy NULL-basis and on_demand_equivalent tranches impossible to claim.

When existing coverage cannot be converted because the provider returned no discount rate for its layer, the run is persisted as a Hold that buys nothing and says why (reason_code: missing_discount_rate). An earlier revision failed these runs instead; see "Owner decision" below.

Closes #946

Owner decision (D3, 2026-10-26)

The first revision made any run with existing coverage and no rate Errored on every tick. Both reviews (cristim, rB21) pointed out that this stops ladder planning for most AWS customers. cloud-commitments-go only returns a rate for EC2 Instance SPs, so accounts holding Compute SPs or Convertible RIs had no rate. EC2 Instance SP holders had none either whenever Cost Explorer had no recommendation (ErrNoSPCommitmentRate, an empty rate map).

The acting owner (IAmJSD) decided to replace fail-loud with a persisted Hold-only run that states the reason. The allocator still never gets a guessed rate: no 1:1 netting and no estimate.

Units

  • Commitment $/hr: what is bought and stored. The AWS provider's LayerState.ExistingUSDPerHour and ExpiringUSDPerHour are in this unit: SavingsPlan.Commitment for the SP layers and amortized RI cost for the RI layer.
  • OD $/hr: on-demand-equivalent spend covered. Baseline, target, gap and the netting totals are in this unit.
  • Rate r is in (0, 1] and means commitment dollars per OD dollar. od = commitment / r and commitment = od * r. This is the same rule go's allocate_units.go uses for tranches.

Root cause

executeLadderRun passed GetLayerStates straight to Allocate and assembleLadderPlan. go's LayerState doc says callers must convert these values to OD first. Existing coverage was therefore understated by a factor of r and the gap overstated. Example: a 2.5 commitment $/hr EC2 Instance SP at r = 0.5 really covers 5 OD $/hr. It was netted as 2.5, so against a target of 10 the run placed 7.5 OD (3.75 commitment) instead of 5 OD (2.5 commitment).

Fix

  1. internal/server/ladder_existing_units.go (new): ladderLayerStatesOnDemand(states, rates) returns a converted copy of the states and never mutates the provider's map.
    • It converts each layer with the rate that executeLadderRun already fetched from GetCommitmentRates for the config's term and payment option. This is option 2 on the issue. DescribeSavingsPlans does not expose each plan's own rate, so plans bought at a different discount are approximated by today's rate.
    • The math is exact (big.Rat). When the quotient is not a float64, existing rounds up and expiring rounds down. Net coverage (existing - expiring) is therefore never understated, which errs toward under-buying.
    • A layer with no commitment (nil or 0) needs no rate. Nil stays nil, and Allocate still rejects it.
    • Layers with non-zero existing or expiring commitment and no rate are returned as a list (sorted by layer), and no partial conversion is returned. Their amounts are still validated.
    • Invalid rates (outside (0, 1], NaN, Inf) and negative, NaN or overflowing amounts still fail the run (Errored, nothing persisted). Those are bad provider data, not a missing rate.
  2. Missing-rate Hold (planMissingRateHold): when any layer is missing a rate, executeLadderRun persists a run through the same locked PlanAndSaveLadderRun path, and Allocate is never called.
    • The run has status planned, no tranche rows and total_hourly_commit = 0.
    • The plan JSON has "reason_code": "missing_discount_rate" and one hold action per layer. Each action's rationale names the layer, its commitment amounts and the term and payment option, and starts with "Can't size this yet because". Example: Can't size this yet because layer compute-sp holds existing coverage (2.5 commitment USD/hr, 0.5 expiring) and the provider returned no discount rate for it at 1yr no-upfront, so that coverage cannot be converted to on-demand-equivalent dollars. Nothing is bought this run: the rate is never estimated or assumed 1:1. The ladder sizes the gap once a rate is available.
    • target_usd_hr is reported. existing_usd_hr and gap_usd_hr stay NULL (unmeasured, never 0). A NULL existing snapshot makes the amend ceiling refuse every raise against this run, so a stale commitment-dollar existing from an older run can no longer keep the ceiling loose.
    • Because it is a persisted run, the cadence gate applies, so there is one run per cadence window and no retry storm. It logs one WARN line per run (ladder_run: config X: persisted Hold-only run Y (reason_code=missing_discount_rate): ...) and counts as Planned in LadderRunResult, not Errored.
    • Buffer reshape detection, which runs inside Allocate, is skipped for these runs. Reshapes are display-only today.
  3. Telling the two Holds apart:
    • A target-met Hold (and every other Allocate Hold) is unchanged. It has no reason_code, and its rationale is go's target already met: ....
    • GET /api/ladder/runs now returns the plan's reason_code on each run (omitted when empty).
    • The ladder timeline UI already renders each run's action rationales, both in the run snapshot and as "No planned purchase tranches. ". It already shows a NULL existing as "unavailable". So the UI shows "Can't size this yet because ..." for these runs, and no frontend change was needed.
  4. executeLadderRun converts right after the rate fetch. The converted states feed Allocate and the plan, so ladder_runs.existing_usd_hr and plan JSON existing are now OD. That matches go's LadderPlan.ExistingUSDPerHour doc and the amend ceiling (LEAST(target, baseline*coverage) - existing), which already assumed OD. The ceiling therefore gets stricter, never looser.
  5. Maintainer comment / known issue: ClaimDueLadderTranches (selectDueLadderTranchesTx) now selects only amount_basis = 'commitment' rows. The validated ladder_tranche_amount_provenance constraint guarantees those rows have a commitment_rate.
    • Legacy NULL-basis and on_demand_equivalent rows are never claimed. They stay netted while within LadderFireGrace. After that, the existing overdue sweep cancels them and the next run re-plans their gap in commitment dollars ("cancelled or re-planned", with no migration).
    • The filter is in SQL, before LIMIT ... FOR UPDATE SKIP LOCKED. Older legacy rows therefore cannot use up a claim's limit and starve a due commitment row.
    • Basis is never updated after insert, and the claim still runs under the per-config ladder lock, so it adds no new race.
    • The known_issues entry is marked resolved.

Visible behavior changes

  • AWS accounts whose existing coverage has no rate now get a persisted Hold-only run that buys nothing and explains why. Today that covers Compute SP and Convertible RI holders, and EC2 Instance SP holders whenever Cost Explorer returns no rate. Before this PR, such runs silently netted the commitment dollars 1:1 and produced over-sized plans. The first revision of this PR made them Errored. Once go's provider returns Compute SP and RI rates, or option 3 (Cost Explorer covered OD cost) lands, these runs size the gap again.
  • existing_usd_hr on new sized runs is OD. Historical ladder_runs rows keep their commitment-dollar snapshot because no rate was stored to backfill from. If the latest run predates this change, the amend ceiling uses that older, smaller existing value until the next run writes a new one. A missing-rate Hold writes NULL, which refuses raises.
  • /api/ladder/runs gains an optional reason_code field.
  • No money moves. The ladder executor still has no production caller.

Verification

All evidence comes from unit, fake-capability and local-PostgreSQL tests. Nothing ran against a real AWS account. Tests ran with AWS_CONFIG_FILE=/dev/null AWS_SHARED_CREDENTIALS_FILE=/dev/null AWS_EC2_METADATA_DISABLED=true.

Missing-rate Hold, fail-before (the new ladder_missing_rate_hold_test.go run on the previous fail-loud code, merged with main at 14e41c5):

  • TestExecuteLadderRun_MissingDiscountRatePersistsHoldWithReason/{compute-sp,convertible-ri,ec2-instance-sp} FAIL with an error ("a missing rate is a Hold, not an Errored run").
  • TestExecuteLadderRun_MissingDiscountRateNamesEveryLayer FAILS with an error.
  • TestExecuteLadderRun_InvalidCoverageWithoutRateIsErrored FAILS: the old code reported "no commitment rate" instead of rejecting the negative amount.
  • TestExecuteLadderRun_TargetMetHoldKeepsOrdinaryReason passes before and after. It is the guard that a met target still gives the ordinary Hold.

Pass-after (head 9b19af3):

  • Each layer kind (Compute SP, Convertible RI, EC2 Instance SP with no rate, the other layers rated):
    • the run is persisted with status planned, no tranches, total_hourly_commit 0, NULL existing and gap, and target 10;
    • plan reason_code is missing_discount_rate;
    • one hold names the layer, its amounts and 1yr no-upfront, starts "Can't size this yet because", and does not say "already met".
  • Two rate-less layers give two Holds in layer order.
  • A met target (20 OD existing vs a target of 10) has no reason_code, and its rationale starts target already met:.
  • TestLadderLayerStatesOnDemand_ReportsEveryLayerWithoutARate returns no partial map. ..._RejectsUnconvertibleCoverage now also covers negative or NaN amounts on a rate-less layer.
  • TestLadderTimelineRuns_ProjectsPlanReasonCode (PostgreSQL) checks that /api/ladder/runs returns reason_code for the missing-rate run and omits it for the target-met run.

Unit conversion and claim filter (from the first revision; still pass):

  • Fail-before on origin/main c8bdd46: the executeLadderRun existing/expiring tests showed existing 2.5 (want 5), gap 7.5 (want 5), commit 3.75 (want 2.5), and gap 8 (want 6). TestClaimDueLadderTranches_NeverClaimsOnDemandOrUnknownBasisRows claimed the NULL row and fired the NULL and OD rows.
  • The conversion is pinned by TestLadderLayerStatesOnDemand_PinsCommitmentToOnDemandConversion (5 @ 0.5 → 10, 3 @ 0.25 → 12, 7 @ 1 → 7), ..._RoundsTowardUnderBuying, ..._DoesNotMutateInput and ..._EmptyLayersNeedNoRate.
  • Earlier mutation probes, all killed:
    • expiring rounded up;
    • existing rounded down;
    • a missing rate treated as 1:1;
    • multiply instead of divide;
    • expiring left unconverted;
    • the claim filter dropped.

Merged with main (including #1199): TestLadderTrancheAmount_RoundTripsProviderPrecisionContract claimed 0 of 19 rows on the merged tree, because its NULL-basis fixture is no longer claimable. It now seeds through claimableTranche with the assertions unchanged. It fails before that change ("[]" should have 19 item(s), but has 0) and passes after.

Checks on 9b19af3:

  • go build ./..., go vet ./..., go vet -tags integration ./... and go vet -tags e2e in tests/e2e: clean.
  • golangci-lint v2.10.1 on api, config, purchase and server, with and without --build-tags integration: 0 issues. gocyclo -over 10 on the changed non-test files: silent.
  • go test -race -short -count=1 on api, config, purchase and server: ok.
  • go test -race -tags integration -count=1 on config, api, purchase and server (whole packages, testcontainers PostgreSQL via Docker): ok, with no skips (config 333s, api 351s, purchase 138s, server 166s). The full test runs were on 8d8619b. 9b19af3 only rewraps a comment; build, vet, lint and -race -short on server were re-run on it.
  • The secrets scan over the changed files is clean, and its planted key is flagged. scripts/test-git-secrets-allowlist.sh: 183 PASS, 0 FAIL.

Out of scope / follow-ups

  • Per-plan rates (option 1) or covered-OD from Cost Explorer (option 3) would remove the today's-rate approximation. Both need provider work in cloud-commitments-go.
  • Compute SP and Convertible RI rates from the AWS provider: until they exist, accounts holding those commitments get the missing-rate Hold.
  • go's sumSPHourlyCost sums EC2 Instance SPs from every region, while the EC2 Instance SP rate is regional. This is pre-existing and noted here only.
  • Historical ladder_runs.existing_usd_hr rows are not backfilled.
  • LadderAmountFloat has no production caller now that fix(ladder): floor tranche amounts to the stored scale instead of rounding them up #1199 is in (cristim's note). It is left for a separate cleanup.

Frontend

No change was needed. The timeline already shows run rationales and renders a NULL existing as "unavailable", so a missing-rate Hold reads "Can't size this yet because ..." and a met target reads "target already met: ...". New sized runs now match the chart's OD base label. Only runs from before this PR show a commitment-dollar base.

🤖 Generated with Claude Code

…s and never claim legacy tranches

The AWS provider reports LayerState.ExistingUSDPerHour and
ExpiringUSDPerHour in commitment dollars (SavingsPlan.Commitment for the
SP layers, amortized cost for the RI layer), but Allocate compares them
with an on-demand-equivalent target. The planner passed them through
unconverted, so existing coverage was understated by the commitment rate
and the gap overstated: a 2.5 commitment $/hr EC2 Instance SP at rate 0.5
was netted as 2.5 OD instead of 5.

executeLadderRun now converts each layer with od = commitment / rate,
using the GetCommitmentRates rate the run already fetched (option 2 on
the issue). Existing rounds up and expiring down, so net coverage is never
understated. A layer with non-zero existing or expiring commitment and no
rate fails the run instead of being netted 1:1. ladder_runs.existing_usd_hr
and the plan JSON existing are now on-demand-equivalent.

Per the maintainer's note on the issue, ClaimDueLadderTranches now claims
only amount_basis = 'commitment' rows. Legacy NULL-basis and
on_demand_equivalent rows stay netted until the overdue sweep cancels
them, so an executor can never buy their OD amount as commitment dollars.

Closes LeanerCloud#946

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Caution

Review failed

The pull request is closed.

Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@cristim

cristim commented Oct 25, 2026

Copy link
Copy Markdown
Member

Independent review 2 (claude-fleet, cold): APPROVE 6af6413

Findings (none blocking on their own):

  • [cross-PR, blocking for whichever merges second] internal/config/store_postgres_ladder_lifecycle.go:160 conflicts in meaning with open fix(ladder): floor tranche amounts to the stored scale instead of rounding them up #1199 (head f1bdefa).
  • [non-blocking, wording] store_postgres_ladder_lifecycle.go:104-109, known_issues/05_config_store_postgres.md:139 and the PR body all say NULL-basis rows "stay netted while within LadderFireGrace". That is wrong.
    • Netting fails loudly on a NULL basis (internal/config/ladder_provenance.go:60, "cannot net an unproven amount basis").
    • So a NULL-basis row that is scheduled within grace, or fired and unresolved, makes the run error.
    • The behaviour predates this PR and fails closed. Only the text needs fixing.
  • [non-blocking, track before enabling the executor] internal/server/ladder_existing_units.go:24-28 converts existing coverage at today's rate for the config's term and payment (option 2). That is a known over-buy path.
    • Probe: 36 commitment/hr of 3yr All-Upfront SPs at a true rate of 0.45 covers 80 OD/hr.
    • With a 1yr No-Upfront config at r=0.72, it nets as 50 OD/hr. Against a target of 80, the run buys 21.6 commitment/hr when nothing is needed.
    • The PR's "errs toward under-buying" covers only ULP-level float rounding, not this.
    • The issue accepted this approximation, and it is strictly better than base, which nets 36 here.
  • [non-blocking, disclosed] The AWS provider never returns a Compute SP or RI rate (go providers/aws ladder/commitment_rates.go:18-19, 45).
    • So every AWS account that holds a Compute SP or Convertible RI gets an Errored ladder run.
    • That fails closed, but the maintainer has to choose it over a Hold-only run.

Attacked:

  • Formula, against go at the pinned versions (pkg 541e05e5fd29, providers/aws aeb8a6f7ff51):
    • pkg/ladder/allocate_units.go:9-13 defines od = commitment / r.
    • The AWS rate is spCost/coveredOD (recommendations/sp_commitment_rate.go:74). That puts it in (0, 1], so the direction is right.
    • The SP layer's Existing is SavingsPlan.Commitment (layer_states.go:122, interfaces.go:57), so the input is commitment $/hr.
    • There is no double conversion:
      • Allocate takes LayerStates as given (allocate.go:421-431).
      • It converts only in-flight rows, by their stored rate (allocate_units.go:84).
      • It converts the gap OD to commitment once (:114).
    • The only GetLayerStates/Allocate caller is handler_ladder.go:317/386.
    • The plan's existing and the amend ceiling (store_postgres_ladder_amend_ceiling.go:116) are both OD now.
  • Table probe: run through executeLadderRun, asserting bought commitment ≤ (target - true net OD)·r, measured with big.Rat. All 6 cases pass at equality:
Case Bought
40 @0.72 17.6
30/10 @0.6 28
13.37/2.11 @0.3 9.74
over target 0 tranches
r=1 30
0.05 @0.0001 0.03
  • Rounding:
  • Legacy classification:
    • amount_basis = 'commitment' is a positive match, so NULL fails closed.
    • The provenance CHECK is validated in 000122.
    • The basis is never updated after insert. The only UPDATEs are at store_postgres_ladder.go:341, amend.go:105 and lifecycle.go:77/135.
    • A legacy row that has already fired keeps its execution and is untouched.
    • A scheduled OD row nets 1:1, is never claimed, and is cancelled by the sweep (store_postgres_ladder.go:341-347).
  • Mutation probes on a scratch copy, each restored and confirmed identical with cmp. All were caught:
    • existing rounds down;
    • expiring rounds up;
    • Quo becomes Mul;
    • a missing rate treated as 1:1;
    • the zero/nil guard's && changed to ||;
    • the conversion result discarded in the handler;
    • a rate > 1 allowed;
    • the first overflow check removed;
    • the claim filter changed to IS DISTINCT FROM 'on_demand_equivalent', or deleted. Both fail NeverClaims, and the deletion also fails ProvenanceInsertClaim.

Verified locally, with GOTOOLCHAIN=go1.26.10 and no cloud credentials, under the heavy lock:

  • Head:
    • go build ./... and go vet, with and without -tags integration: clean. gofmt: clean.
    • go test -count=1 on server, config, purchase and api: pass. -race -run Ladder on server: pass.
    • Integration on testcontainers Postgres: the whole config package passes, and -run Ladder passes on purchase, api and server.
  • Fail-before, on base c8bdd46:
    • The new unit file does not compile.
    • The trimmed executeLadderRun tests FAIL: 2.5 vs 5, 7.5 vs 5, 3.75 vs 2.5, 8 vs 6, and no error where one is expected.
    • NeverClaims FAILS, because the NULL and OD rows get claimed.
  • Merged with origin/main (merge-tree clean):
    • Build and vet: clean.
    • Unit tests: pass.
    • Integration -run 'Ladder|Claim' on config, purchase, api and server: pass. This includes main's new immediate and lock ladder tests.
  • No migration.

Reviewed by an independent agent with no author context (second reviewer). Review only: nothing was pushed, and the GitHub review state is unchanged.

@cristim

cristim commented Oct 25, 2026

Copy link
Copy Markdown
Member

VERDICT: APPROVE 6af6413

Merge check: merge-tree vs origin/main 2ded648 is clean (tree 9f4c82b0). go build ./... and go vet -tags integration on config/server/purchase/api pass on the merged tree.

Findings:

  • medium (disclosure, non-blocking) internal/server/ladder_existing_units.go:68-71 with providers/aws@aeb8a6f7 ladder/commitment_rates.go:39-41. AWS returns an EMPTY rate map whenever Cost Explorer has no EC2 Instance SP recommendation (ErrNoSPCommitmentRate), which is common for accounts already well covered by SPs. Any non-zero existing coverage then makes the run Errored, where main persisted a Hold-only run. The body discloses this only for Compute SP and RI. Failing the whole run is still correct, since an unconverted layer understates the total and over-buys other layers, and no money moves. The body should say that EC2-SP-only accounts can also error, and the owner should decide whether Errored or a persisted Hold is wanted.
  • nit process: the single commit 6af6413 has 461 changed lines, over the 400 cap.
  • nit handler_ladder.go:284 has an overlong rewrapped comment line.

Attacked:

  • Units checked against go pkg@541e05e5. ladder/types.go:473-478 says callers must normalize to OD. allocate.go:38-40 and allocate_units.go:9-23 give od = commitment / rate with rate in (0,1]. The PR divides, validates (0,1] the same way, and uses exact big.Rat.
  • Rounding: existing rounds up and expiring rounds down, so net coverage is not understated and expiring <= existing still holds for Allocate's consistency check (allocate.go:431).
  • Double count and drops:
    • Fired, unresolved and completed-after-now tranches still net at their own stored rate. Unchanged.
    • Legacy NULL and OD rows still net within grace. cancelOverdueLadderTranchesTx (store_postgres_ladder.go:336,404) then cancels them.
    • scheduled->fired is only possible via ClaimDueLadderTranches (lifecycle.go:71), and the basis filter sits before LIMIT/SKIP LOCKED.
  • Unit: ./internal/server and ./internal/config passed 1492 tests.
  • Integration on the merged tree (-run Ladder, config/purchase/api/server): all ok. 83 config ladder tests passed, and 1 pre-existing env skip (LadderArming).
  • Mutations, all killed:
    • M1, drop the conversion call: 3 executeLadderRun tests FAIL.
    • M2, expiring rounds up: RoundsTowardUnderBuying FAILS.
    • M3, skip the rate check when only existing is zero: RejectsUnconvertible FAILS.
    • M4, claim filter IS DISTINCT FROM 'on_demand_equivalent' (lets NULL through): NeverClaims integration FAILS.
  • CI at 6af6413: Security Scanning, Build Docker Image and CI Success are red. govulncheck GO-2026-6809 is a stdlib advisory fixed in Go 1.26.10. That is the OLD base c8bdd46, and main is now on 1.26.10, so a rebase or re-run should clear it. Everything else is green.

@IAmJSD CI on this head is red only from the old merge base (govulncheck on go1.26.9, fixed on main by #1236). Please merge or rebase onto current main so CI runs on a fresh merge ref.

@cristim

cristim commented Oct 25, 2026

Copy link
Copy Markdown
Member

Merge-order note from review: when this PR and #1199 are both on one tree, TestLadderTrancheAmount_RoundTripsProviderPrecisionContract claims 0 of 19 rows, because its fixture uses lifecycleTranche (NULL basis). #1199 is queued to merge first, so @IAmJSD, when you next refresh this branch, please switch that fixture to claimableTranche. Separately, LadderAmountFloat has no production caller once both land, so it can go.

Brings in main's CI fixes (LeanerCloud#1236, LeanerCloud#1252) and CI changes since, so this PR gets a fresh CI run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@IAmJSD

IAmJSD commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator Author

Independent review (IAmJSD's agent, fresh context, given only the diff, the issue and the PR text): VERDICT: REQUEST_CHANGES 4fe4f58b23b0cd79d4d2d95860924cd462b191a0

Blocking:

  • internal/config/store_postgres_ladder_lifecycle.go:160, against current main: fix(ladder): floor tranche amounts to the stored scale instead of rounding them up #1199 merged at 19:37Z today (c1d4a4a4) and this branch doesn't contain it. On a local merge of origin/main + 4fe4f58b, the merge is textually clean, but go test -tags integration ./internal/config/ -run TestLadderTrancheAmount FAILS: TestLadderTrancheAmount_RoundTripsProviderPrecisionContract: "[]" should have 19 item(s), but has 0. Its fixture seeds NULL-basis rows through lifecycleTranche, and the new amount_basis = 'commitment' filter correctly stops claiming those. Merging as-is would turn main's integration job red. Fix: merge main, and switch that test's rows (internal/config/store_postgres_ladder_precision_integration_test.go, lines 35/87/88 on the merged tree) to claimableTranche. cristim's earlier review flagged this as a merge-order risk; it's now concrete because fix(ladder): floor tranche amounts to the stored scale instead of rounding them up #1199 has landed.

Non-blocking:

  • Owner decision, flagged correctly in the body: any AWS account holding Compute SPs or Convertible RIs now gets an Errored ladder run every tick (existing coverage: layer compute-sp: no commitment rate ...), because the provider only supplies an EC2 Instance SP rate today. I agree with failing closed: under a shared target there is no safe upper bound for unconverted coverage, and 1:1 over-buys. In practice, though, ladder planning stops for most AWS customers until go supplies those rates. Errored runs persist nothing, so the cadence gate doesn't suppress them and they repeat on every tick (log noise only).
  • From cristim's earlier review, which I confirmed after writing my own: Errored runs aren't limited to Compute SP/RI holders. providers/aws@aeb8a6f ladder/commitment_rates.go returns an EMPTY rate map on ErrNoSPCommitmentRate, which happens when Cost Explorer has no EC2 Instance SP recommendation, common for accounts that are already well covered. So any account with non-zero existing EC2 Instance SP coverage and no current recommendation also goes from main's Hold-only run to Errored. The body's "Visible behavior changes" should say so.
  • Because of that, for those accounts the latest persisted ladder_runs.existing_usd_hr stays the pre-fix commitment-dollar value indefinitely, so the amend ceiling (LEAST(target, baseline*coverage) - existing) stays looser than intended. No executor has a production caller, so no money moves. This should be resolved before executor enablement (feat(ladder): phase 3 - pipeline (schema, scheduled task, approval/auto modes, UI, terraform) #70/fix(ladder): tranche status is never updated, so in-flight netting grows and planning silently stops #118), together with the Compute SP/RI rate work.
  • internal/server/handler_ladder.go:284: the reflowed comment line is over 100 columns. Cosmetic.

Attacked:

  • Units: go pkg/ladder documents CommitmentRates as commitment $ per OD $, so od = commitment / r is the right direction. Rounding keeps expiring <= existing after conversion (existing rounds up, expiring rounds down), so Allocate's consistency check can't newly trip. The provider map isn't mutated, and nil stays nil for Allocate to reject.
  • Executability of legacy tranches: selectDueLadderTranchesTx is the only scheduled->fired path, because TransitionLadderTrancheTx refuses that transition outside the claim. The new amount_basis = 'commitment' filter sits before LIMIT ... FOR UPDATE SKIP LOCKED, so legacy rows can't starve the limit. cancelOverdueLadderTranchesTx ignores basis, so legacy rows still expire after LadderFireGrace. That satisfies cristim's "made non-executable (cancelled or re-planned)" note on fix(ladder): existing Savings Plan coverage is commitment dollars but the target is on-demand-equivalent #946.
  • Fail-before (both reverted and restored): dropping the ladderLayerStatesOnDemand call makes the three TestExecuteLadderRun_(Existing|Expiring)* tests FAIL (existing 2.5 vs 5, gap 7.5 vs 5, commit 3.75 vs 2.5; no error without a rate). Deleting the SQL basis filter makes TestClaimDueLadderTranches_NeverClaimsOnDemandOrUnknownBasisRows FAIL against real Postgres, with the legacy rows fired.
  • Ran on the branch head: go build ./...; go vet (with and without -tags integration) on server/config/purchase/api; go test -race -short -count=1 on those 4 packages (ok); go test -race -tags integration -count=1 on those 4 packages (ok: config 107s, server 30s, purchase 65s, api 117s); golangci-lint v2.10.1 with and without --build-tags integration (0 issues); gocyclo -over 10 on the changed non-test files (clean). On a scratch merge with current origin/main: go build ./... ok, and the fix(ladder): floor tranche amounts to the stored scale instead of rounding them up #1199 integration test fails as described under Blocking.

Body-stale:

  • "Visible behavior changes" names only Compute SP / Convertible RI holders. It should also cover EC2 Instance SP holders whenever the rate map comes back empty (see above).

Reviewer: rB21. Bar while the maintainers are away (set by IAmJSD as acting owner): one approving independent review on the head, two for money, auth and security paths.

IAmJSD and others added 6 commits October 26, 2026 04:27
…p test

After merging main, TestLadderTrancheAmount_RoundTripsProviderPrecisionContract
(LeanerCloud#1199) claimed 0 of its 19 rows: it seeded NULL-basis tranches, which this
branch makes unclaimable. Seed commitment-basis rows at rate 1 through
claimableTranche instead, so the claim path still reads every amount back.
The assertions are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rate

cloud-commitments-go only returns a discount rate for EC2 Instance SPs, so
accounts holding Compute SPs or Convertible RIs (and EC2 Instance SP holders
whenever Cost Explorer returns no rate) cannot have their existing coverage
converted to on-demand-equivalent dollars. Failing those runs made every tick
Errored and stopped ladder planning for most AWS customers.

Owner decision D3: persist the run as a Hold that buys nothing and says why.
ladderLayerStatesOnDemand now reports every layer that holds commitment but
has no rate, and executeLadderRun persists a Hold-only run with plan
reason_code "missing_discount_rate" and one Hold per layer whose rationale
starts "Can't size this yet because". Allocate never runs, so no rate is
guessed or netted 1:1; existing and gap stay NULL, which also keeps the amend
ceiling from allowing raises against that run. The run counts as planned and
logs one WARN line, not an error. Invalid rates and negative or non-finite
amounts still fail the run. Target-met and other Allocate Holds are unchanged
and carry no reason_code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Covers executeLadderRun end to end for each AWS layer kind holding coverage
with no rate (Compute SP, Convertible RI, EC2 Instance SP): the run is
persisted with no tranches, NULL existing and gap, reason_code
missing_discount_rate and a "Can't size this yet" Hold naming the layer. A
genuinely met target must still produce the ordinary "target already met"
Hold with no reason_code, and bad amounts on a rate-less layer must still
fail the run. The first three fail on the previous fail-loud head.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ladder timeline already shows each action's rationale, so the UI tells a
missing-rate Hold ("Can't size this yet because ...") from a target-met Hold
("target already met: ...") without a frontend change. Projecting the plan's
reason_code onto /api/ladder/runs gives API clients the same distinction in
machine-readable form. Runs without one omit the field.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both reviews flagged the reflowed comment line at handler_ladder.go:284 as
over 100 columns. Comment only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@IAmJSD

IAmJSD commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator Author

Updated to owner decision D3. New head: 9b19af309d1389c70bc18004c702f840993b6b09.

What changed since 4fe4f58:

  • Merged origin/main (7816419), which brings in fix(ladder): floor tranche amounts to the stored scale instead of rounding them up #1199. TestLadderTrancheAmount_RoundTripsProviderPrecisionContract now seeds its rows through claimableTranche with the assertions unchanged (14e41c5). Before that change it claimed 0 of 19 rows on the merged tree. After it, the test passes.

  • Missing rate: Hold, not Errored (c73f31a). Existing coverage with no discount rate (Compute SP, Convertible RI, or EC2 Instance SP when Cost Explorer returns none) now persists a Hold-only run:

    • no tranches, and Allocate is never called, so there is no 1:1 netting and no estimate;
    • plan reason_code: "missing_discount_rate", and one Hold per layer starting "Can't size this yet because layer X holds existing coverage ... no discount rate ...";
    • existing and gap stay NULL, so the amend ceiling refuses raises against that run;
    • one WARN log line per run, counted as Planned rather than Errored, and the cadence gate applies, so nothing repeats every tick.

    Invalid rates and bad amounts still fail the run. Target-met Holds are unchanged and have no reason_code.

  • Tests (d72810f): each layer kind gives a Hold with its reason, two rate-less layers give two Holds, a met target still gives the ordinary "target already met" Hold, and bad amounts on a rate-less layer still error. All but the target-met guard fail on the previous head.

  • /api/ladder/runs returns reason_code (8d8619b). The timeline UI already shows run rationales, so no frontend change was needed.

  • Rewrapped the overlong doc line both reviews flagged (9b19af3).

Checks: build and vet (plain, integration and e2e) are clean. golangci-lint, with and without integration tags, reports 0 issues, and gocyclo is clean. -race -short passes on api, config, purchase and server. -race -tags integration passes on those four packages with no skips. The secrets scan is clean, and the allowlist test reports 183 PASS. The PR body has been rewritten for the Hold behavior and records the decision.

@IAmJSD

IAmJSD commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator Author

Independent review (IAmJSD's agent, fresh context, given only the diff, the issue and the PR text): VERDICT: APPROVE 9b19af309d1389c70bc18004c702f840993b6b09

Blocking:

  • none

Non-blocking:

  • internal/server/ladder_existing_units.go:61 (ladderLayerStatesOnDemand): existing coverage is converted at the config's term and payment-option rate, not the rate of the plan being converted. A 3yr or all-upfront plan (r ~0.4) converted at a 1yr no-upfront rate (r ~0.7) has its OD coverage understated by ~40%, so the gap is overstated and the ladder would still over-buy for it. This is strictly better than main, because dividing by r <= 1 never lowers existing, so the PR never over-buys more than today. But the body's "approximated by today's rate" undersells the direction of the error. Before executor enablement (feat(ladder): phase 3 - pipeline (schema, scheduled task, approval/auto modes, UI, terraform) #70/fix(ladder): tranche status is never updated, so in-flight netting grows and planning silently stops #118), this should become a tracked prerequisite: per-plan rate (option 1), Cost Explorer covered-OD (option 3), or at least the deepest-discount rate across terms and payment options as a conservative divisor.

  • Interplay with go#441 (residual baseline): once AWS sets BaselineBasisResidual, the engine no longer subtracts existing at a 100% target. The missing-rate Hold would then block Compute SP and Convertible RI holders even though no rate is needed. Until then (total basis on the current pin), converting existing up to OD makes go#282's double subtraction larger, which means more false "target already met" Holds. That's the safe direction. Whoever bumps the go pin should revisit planMissingRateHold so it applies only when the basis actually nets existing.

  • internal/server/ladder_missing_rate_hold_test.go:16-21: awsLadderLayers says it "mirrors the AWS provider's three layers", but it gives convertible-ri RoleBase. The provider uses RoleBuffer (go providers/aws/ladder/ladder.go:163), and two base layers is a topology Allocate rejects ("at most one base layer allowed, got 2"). The missing-rate tests only pass because that path never reaches Allocate. Please use the real roles so the fixture can't hide a fall-through.

  • Owner decision D3 (persisted Hold rather than Errored) is clearly implemented: NULL existing/gap, the amend ceiling refuses raises on NULL (store_postgres_ladder_amend_ceiling.go:115), and the cadence gate bounds the WARN line to once per window.

  • (Raised in an earlier review, still true at this head) The code comment at internal/config/store_postgres_ladder_lifecycle.go:104-109, known_issues/05_config_store_postgres.md:139 and the PR body say legacy NULL-basis rows "stay netted while within LadderFireGrace". The planner reads in-flight with assumeOD=false (store_postgres_ladder_lock.go:184), and nettingOnDemand fails on a NULL basis ("cannot net an unproven amount basis", ladder_provenance.go:60). So a NULL-basis row inside grace makes the planner run error, until the sweep cancels it. That fails closed and predates this PR, but the docs describe it wrongly. The new test's "legacy rows stay netted within grace" assertion goes through the inFlight helper, which uses the amend path (assumeOD=true), so it checks amend netting, not planning. Please fix the wording, or point the assertion at the planner path.

Attacked:

  • Money paths: scheduled->fired is only reachable through ClaimDueLadderTranches (TransitionLadderTrancheTx refuses that edge), and the new amount_basis = 'commitment' filter sits before LIMIT ... FOR UPDATE SKIP LOCKED, so legacy rows can't starve the claim or be fired. Legacy rows stay netted, which means an under-buy until the overdue sweep, never an over-buy. Rounding goes in the under-buy direction (existing up, expiring down). Invalid rates and amounts still fail the run. The amend ceiling only gets stricter, because existing is now >= the old commitment-dollar value.
  • Fail-before: with the conversion call in executeLadderRun bypassed, the existing and expiring OD tests fail (existing 2.5 vs 5, gap 7.5 vs 5, commit 3.75 vs 2.5, gap 8 vs 6), and so do all four missing-rate tests. A mutant that drops the claim's amount_basis filter fails TestClaimDueLadderTranches_NeverClaimsOnDemandOrUnknownBasisRows (legacy rows fired) and TestLadderTrancheProvenance_InsertClaimAndReconcile. Restored afterwards.
  • Ran: go build ./..., go vet ./... with and without -tags integration, and the e2e vet (all clean). go test -race -short -count=1 on api, config, purchase and server passed. go test -race -tags integration -count=1 on server and purchase (whole packages) passed, and -run Ladder on config and api passed. golangci-lint v2.10.1 with and without integration tags reported 0 issues, and gocyclo -over 10 on the changed non-test files was clean.

Body-stale:

  • "Legacy NULL-basis and on_demand_equivalent rows ... stay netted while within LadderFireGrace": this is true only for on_demand_equivalent rows. NULL-basis rows make a planning run fail within grace (see above).

Reviewer: rC1. Bar while the maintainers are away (set by IAmJSD as acting owner): one approving independent review on the head, two for money, auth and security paths.

@IAmJSD

IAmJSD commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator Author

Independent review (IAmJSD's agent, fresh context, given only the diff, the issue and the PR text): VERDICT: APPROVE 9b19af309d1389c70bc18004c702f840993b6b09

Blocking:

  • none

Non-blocking:

  • internal/server/ladder_missing_rate_hold_test.go awsLadderLayers: the comment says it "mirrors the AWS provider's three layers", but it gives convertible-ri the Base role. The provider (providers/aws/ladder/ladder.go SupportedLayers) has ec2-instance-sp as Base, compute-sp as Flex and convertible-ri as Buffer. With two Base layers, Allocate rejects the fixture ("at most one base layer allowed, got 2"). The Hold tests only pass because Allocate is never reached on that path. Any later test that reuses the fixture on a rated path will fail for the wrong reason. Use the provider's roles.
  • internal/server/ladder_existing_units.go validateCommitments: it checks only that amounts are non-negative and finite. It does not check expiring <= existing, which Allocate treats as an inconsistent provider snapshot. So a rate-less layer with expiring > existing becomes a missing-rate Hold, not an Errored run. TestExecuteLadderRun_MissingDiscountRateNamesEveryLayer pins exactly that case (compute-sp existing 0, expiring 1) as a Hold. This contradicts "bad provider data still fails the run". No money risk, because nothing is bought either way.
  • Today the AWS provider only returns a rate for ec2-instance-sp. Any account holding a Compute SP or a Convertible RI will therefore get Hold-only runs indefinitely, and the reshape detection on the buffer layer stops too. That is owner decision D3, but no follow-up issue in cloud-commitments-go is linked for the Compute SP and RI rates. Without one, the ladder stays dark for most AWS accounts.

Attacked:

  • Unit math: Allocate nets existing - expiring (computeNetExisting). Rounding existing up and expiring down keeps the net at or above the exact value, so it errs toward under-buying and never goes negative.
  • Conversion: the result is a copy. Nil stays nil. A missing rate returns no partial map. Invalid rates still error.
  • Missing-rate Hold: it goes through the locked PlanAndSaveLadderRun. It writes NULL existing and gap, and loadLadderCeiling refuses a raise when existing is NULL. The reason_code NULL to JSON null decodes to "".
  • Claim filter: ClaimDueLadderTranches is the only scheduled->fired path. TransitionLadderTrancheTx forbids that edge. The overdue sweep has no basis filter, so legacy rows still get cancelled after grace.
  • Ran go build ./..., go vet ./..., go vet -tags integration ./... and the e2e vet: clean. go test -race -short -count=1 on api, config, purchase and server: ok. golangci-lint on server and config, with and without integration tags: 0 issues. gocyclo is silent.
  • Ran go test -race -tags integration -count=1 on api, config, purchase and server against testcontainers PostgreSQL: all ok (459s, 425s, 307s, 313s).
  • Fail-before 1: I replaced the ladderLayerStatesOnDemand call with no conversion. The existing/expiring tests fail (existing 2.5 vs 5, gap 7.5 vs 5, commit 3.75 vs 2.5, gap 8 vs 6), and so do the target-met, invalid-coverage and missing-rate tests.
  • Fail-before 2: I dropped the amount_basis = 'commitment' predicate. TestClaimDueLadderTranches_NeverClaimsOnDemandOrUnknownBasisRows fails: the legacy row is claimed under limit 1, NULL and OD rows are fired, and 3 execution rows are created.
  • Both mutations ran in a separate scratch worktree that I then removed.

Body-stale:

  • none

Reviewer: rC4. Bar while the maintainers are away (set by IAmJSD as acting owner): one approving independent review on the head, two for money, auth and security paths.

@IAmJSD

IAmJSD commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator Author

Local verification at 9b19af309d1389c70bc18004c702f840993b6b09 merged onto main (83683307), together with #1238 in that order (combined tree 1435ef56, Linux, Go 1.26.10): go build ./..., go vet with and without -tags integration, the full go test -race -short -count=1 ./..., and go test -race -tags integration -count=1 on every changed package, against Docker Postgres. All pass. CI Success and pre-commit are green on this head, with 2 approving independent review(s) on it. Nothing was run against any cloud.

@IAmJSD
IAmJSD marked this pull request as ready for review October 26, 2026 06:43
@IAmJSD
IAmJSD merged commit d87b96f into LeanerCloud:main Oct 26, 2026
41 checks passed
@IAmJSD

IAmJSD commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator Author

Merged by IAmJSD's agent at IAmJSD's request (IAmJSD is acting owner while the maintainers are away), under the bar IAmJSD set: 2 approving independent review(s) on 9b19af30 (two required: money, auth or security path), green CI, and the combined-tree verification above. Squash-merged at exactly 9b19af309d1389c70bc18004c702f840993b6b09.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/m Days impact/few Limited audience priority/p2 Backlog-worthy severity/medium Moderate harm triaged Item has been triaged type/bug Defect urgency/this-quarter Within the quarter

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(ladder): existing Savings Plan coverage is commitment dollars but the target is on-demand-equivalent

2 participants