Skip to content

fix(recommendations): key stored recs, rec ID and purchase identity on RDS az_config (#408) - #1239

Draft
IAmJSD wants to merge 5 commits into
LeanerCloud:mainfrom
IAmJSD:fix/issue-408
Draft

IAmJSD wants to merge 5 commits into
LeanerCloud:mainfrom
IAmJSD:fix/issue-408

Conversation

@IAmJSD

@IAmJSD IAmJSD commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

#806 and #792 already added the zonal EC2 AZ, platform and tenancy to the recommendations natural key, rec ID, identity key and idempotency key. What #408 still lists as missing is EC2 scope and RDS az_config.

  • RDS az_config (fixed here). Cost Explorer reports Single-AZ and Multi-AZ RDS reservations of the same class, engine, term and payment as two separate recommendations (parseRDSDetails maps DeploymentOption to single-az/multi-az). They are different offerings, but both got the same natural key, rec ID and identity key. That breaks three things on main:
    1. Storage. UpsertRecommendations writes the whole multi-provider batch in one statement. When the batch has both variants it fails with SQLSTATE 21000 ("ON CONFLICT DO UPDATE command cannot affect row a second time"), so the whole collection write is lost, not just one rec.
    2. Purchase pricing. Once both rows are stored, loadStoredRecommendationIndex refuses every purchase for that provider with "more than one row for identity key".
    3. Idempotency. A Multi-AZ submit right after a Single-AZ one dedupes into it.
  • EC2 scope (no change needed). The AWS parser derives scope from the AZ (resolveEC2Scope: zonal exactly when CE gives an AZ). The AZ is already in every identity, so two recs cannot differ only by scope. Azure collects Shared-scope recs only (properties/scope eq 'Shared') and GCP does not set scope. The purchase guard computeDetailMismatch still compares scope as a backstop.
  • Provider offering filters. I checked these against the extended identity. RDS findOfferingID already filters by AZConfig and refuses an empty one, and EC2 filters already use platform, tenancy, scope and AZ, so no provider change is needed.

Fix (four commits, expand/contract like the #792/#806 chains)

  1. feat(db) migration 000140 (expand). Adds:

    • a generated az_config column, COALESCE(payload->'details'->>'az_config', ''), verbatim. The table rewrite backfills it, so no data migration is needed.
    • recommendations_natural_key_az_tp_azc_idx: the 11 columns plus az_config.
    • recommendations_natural_key_source_azc_idx: 000120's source-account key plus az_config. This is needed because any unique index without az_config still rejects the pair. See the feat(insurance): per-recommendation Archera quotes (blocked on an Archera API mapping) #799 note below.

    The old indexes stay, so the current upsert keeps working.

  2. fix(purchase): new RecommendationRecord.DetailsAZConfig(), pinned to the generated column by shared vectors. az_config now goes into:

    • RecIdentityKey. This covers the pricing index, planstep and the ramp-baseline netting.
    • the rec ID, appended as |<az_config> only when set, so EC2, cache and other recs keep their IDs.
    • purchaseIdempotencyKey.
    • describeRec.
  3. fix(config): the upsert ON CONFLICT targets the 12-column index. This needs 000140.

  4. fix(db) migration 000141 (contract). Drops the 11-column index and 000120's recommendations_natural_key_source_idx. Each has a strictly finer replacement from 000140, so no existing row can violate the new indexes. The down migration collapses az_config-only duplicates to the newest row (the table is a regenerable cache) and recreates both indexes exactly.

Prod deploy order (auto_migrate=false)

  1. Dispatch database-migration.yml for 000140 only (steps counted from the live prod version).
  2. Release this code.
  3. Dispatch 000141 only after step 2 is live everywhere. Old code's 11-column ON CONFLICT needs the index that 000141 drops, so never run steps=0 before the release.

Dev auto-migrates at startup, so the window there is one deploy. If a maintainer prefers separate PRs as in #808/#810/#811, the four commits split cleanly.

Behavior changes to know about

Interaction with the #799 source-account chain

Migration 000120 (#881) added recommendations_natural_key_source_idx as a prerequisite and did not switch the upsert to it. That index lacks az_config, so it would keep rejecting the Single-AZ/Multi-AZ pair. This PR replaces it with recommendations_natural_key_source_azc_idx, the same key plus az_config. When #799 later switches to a source-aware upsert, it should target that index. 000120's down-migration guard still works, because rolling back past 000141 recreates both old indexes first.

Verification

All runs were on macOS with Go 1.26 and real Postgres 17 through testcontainers. This is local unit, mock and integration evidence on synthetic fixtures. I did no real-cloud or real-account verification: a live CE response with both RDS deployment options was not exercised. Tests ran with AWS_CONFIG_FILE=/dev/null AWS_SHARED_CREDENTIALS_FILE=/dev/null because this machine's default AWS profile is invalid.

Fail before / pass after

  • Store: go test -tags integration ./internal/config/ -run 'AZConfig(Batch|Concurrent)' on the main schema and store code, with the index precondition disabled, fails in every batch case with failed to insert recommendations batch: ERROR: ON CONFLICT DO UPDATE command cannot affect row a second time (SQLSTATE 21000). It passes after the fix, six repeated runs.
  • Pricing, identity and idempotency: with az_config removed from RecIdentityKey and purchaseIdempotencyKey (pre-fix behavior):
    • TestPriceRecommendationsFromStore_PricesFromRequestedAZConfig fails with stored recommendations for aws contain more than one row for identity key ....
    • TestPriceRecommendationsFromStore_RequestWithoutAZConfigRefusedForRDS, TestPurchaseIdempotencyKey_DiffersByAZConfig and TestRecIdentityKey_DiffersByAZConfig fail.
    • All of them pass after the fix.
  • Rec ID: without the scheduler change, TestConvertRecommendations_RDSAZConfigInID shows both variants getting aws||rds|us-east-1|db.r5.large|mysql|1|all-upfront, and TestRecommendationID_MalformedAZConfigErrors fails. Both pass after the fix.

Migrations (go test -tags integration ./internal/database/postgres/migrations/ -run 'TestMigration_(RecommendationsAZConfig|DropRecommendations11ColNaturalKey|RecommendationsSourceAccount|DropRecommendations9ColNaturalKey|RecommendationsPlatformTenancy)$'): all pass. They cover:

  • the generated-column vectors, which mirror the Go vectors;
  • the old 11-column upsert still working after 000140;
  • the known 23505 between 000140 and 000141;
  • 000141 letting the pair coexist while exact duplicates are still rejected;
  • the down migration's dedupe and exact index recreation;
  • re-applying after rollback.

Concurrency. TestPostgresStore_UpsertRecommendations_AZConfigConcurrent runs 8 concurrent collections re-upserting the stored pair. All succeed and exactly one row per az_config remains.

Suites and checks

  • go build ./...
  • go vet on the touched packages, with and without -tags integration
  • go test ./internal/mocks/ ./internal/config/ ./internal/scheduler/ ./internal/api/ ./internal/purchase/ ./internal/planstep/ ./internal/database/... -count=1
  • go test -tags integration for ./internal/config/, ./internal/scheduler/, ./internal/api/, ./internal/purchase/ and ./internal/database/postgres/migrations/. One test needed a fixture bump: TestApproveAzurePricingRequiresMigrationAndRecollection now migrates to 140, because the upsert needs the new column.
  • golangci-lint v2.10.1 with --allow-parallel-runners, with and without --build-tags integration: 0 issues
  • gocyclo -over 10 on the changed non-test files: silent
  • scripts/gosec-hook.sh: clean
  • git secrets --scan in an isolated temp repo with the changed files tracked: clean. A planted fake key was flagged.
  • scripts/test-git-secrets-allowlist.sh: 137 PASS
  • git diff --check: clean

Out of scope / noted

  • Pre-existing race on main, not caused by this PR. The non-arbiter unique source index from 000120 makes two collections that insert the same new row at the same instant fail with 23505 on that index. Postgres only routes conflicts on the arbiter index to DO UPDATE. I reproduced this on unmodified origin/main with 8 concurrent single-row upserts: it failed 1 run in 4 on recommendations_natural_key_source_idx. This PR keeps an equivalent index (..._source_azc_idx), so the race is unchanged. The concurrency test seeds the rows first so that it covers the steady-state re-collection path. The collection single-flight marker makes this rare. It could be fixed by making the source index non-unique until feat(insurance): per-recommendation Archera quotes (blocked on an Archera API mapping) #799 switches the arbiter, which is the owner's decision.
  • TestMigration_ExecutionsDetachKeepsUpdatedAt (000128, purchase_executions) failed once during a full migrations-suite run and passed 3/3 in isolation. It is time-based and unrelated to this change.
  • The approval email and other displays do not show az_config, which is display-only.
  • Purchase suppression keys are untouched (fix(purchase): scope purchase suppressions to the AZ, platform and tenancy bought #1110 is reworking them for AZ, platform and tenancy).

Frontend follow-up for the UI owner

  • cellKey (frontend/src/recommendations.ts) groups recs by physical cell. It includes the AZ and the platform/tenancy pair, but not az_config. Single-AZ and Multi-AZ RDS rows of the same class therefore render as variants of one cell. Purchases are still correct, because samePurchaseVariantIdentity already compares az_config for RDS, but they should be separate cells, the same way fix(frontend): include platform and tenancy in EC2 recommendation cell key #818 did it for EC2 platform/tenancy.

Closes #408

Maintainer: please mirror the issue's triage labels on this PR: priority/p2, type/chore, triaged, severity/low, urgency/this-quarter, impact/few, effort/m.

🤖 Generated with Claude Code

IAmJSD and others added 4 commits October 22, 2026 01:42
…s index (LeanerCloud#408)

Migration 000140 adds a generated az_config column, read verbatim from
payload->'details'->>'az_config' (RDS single-az/multi-az, Azure SQL zone
redundancy, '' otherwise), and two unique indexes that include it: the
11-column natural key plus az_config (the new upsert arbiter) and the
000120 source-account key plus az_config.

The existing 11-column and 000120 source indexes are kept, so the current
upsert keeps working and this can be applied before the store switch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tency key (LeanerCloud#408)

Single-AZ and Multi-AZ RDS recommendations of the same class, engine,
term and payment shared one rec ID and one identity key. Read az_config
verbatim from Details (RecommendationRecord.DetailsAZConfig, pinned to
the 000140 generated column by shared vectors) and add it to
RecIdentityKey, the rec ID (appended only when set, so every other rec
keeps its ID), the purchase idempotency key and describeRec.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nfig (LeanerCloud#408)

Requires migration 000140.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oud#408)

Migration 000141 drops the 11-column natural key and the 000120 source
index, both superseded by the 000140 indexes that add az_config. While
either exists, a collection batch carrying a Single-AZ and a Multi-AZ
RDS recommendation of the same class fails as a whole. The rollback
collapses az_config-only duplicates to the newest row and recreates both.

Apply only after the 12-column upsert is live everywhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@cristim

cristim commented Oct 25, 2026

Copy link
Copy Markdown
Member

Independent review 2 (claude-fleet, cold): REQUEST_CHANGES a035c5d

Findings:

  • [blocking] Purchase suppression is still keyed on the old identity, without az_config.
    • The sites are internal/config/recommendation_identity.go:100 (SuppressionVariant), internal/config/types.go:754 and internal/scheduler/scheduler.go:1452 on the merged tree. The key is the 6-tuple plus AZ, platform and tenancy.
    • fix(purchase): scope purchase suppressions to the AZ, platform and tenancy bought #1110 (migration 000134) is now on main, so the body's "suppression keys untouched (fix(purchase): scope purchase suppressions to the AZ, platform and tenancy bought #1110 is reworking them)" is out of date.
    • Before this PR, Single-AZ and Multi-AZ rows could never both be stored, so this path was unreachable. With the PR they coexist.
    • So buying N Single-AZ RIs subtracts N from (or hides) the Multi-AZ sibling for the whole grace period. That is an under-buy, or a refused buy.
    • Reproduced on the merged tree: a scratch TestR1239_SuppressionCrossAZConfig FAILS with "multi-az rec was suppressed by a single-az purchase".
    • This is the one identity path left on the old key.
    • Fix: add az_config to SuppressionVariant, and to the purchase_suppressions variant columns and unique key (a new migration). Alternatively, gate 000141 until that lands.
  • [non-blocking, process] Migration numbering: this PR adds 000140/000141, while main is at 000134.
  • [non-blocking, disclosed] Azure SQL rec IDs can change between collections. Azure SQL recs also carry az_config, derived from a cached "dominant" server-inventory value that can flip between "", "none" and "zoneRedundant".
    • So Azure SQL rec IDs and identity keys can change between collections, giving a 409 on a stale tab.
    • Pricing was already refused on a mismatch (computeDetailMismatch, purchase_pricing.go:330), so this adds no new refusal path.
  • [non-blocking] Key matching: DetailsAZConfig uses json.Unmarshal, which matches keys case-insensitively, while the generated column matches exactly. They diverge only on hand-crafted Details.
  • [non-blocking, disclosed follow-up] Dashboard grouping: handler_dashboard.go:285 recCellKey and the frontend cellKey still group Single-AZ and Multi-AZ into one cell. This affects display and rollup only.

Attacked:

  • Identity sites, on the merged tree:
    • RecIdentityKey (the pricing index, planstep resolve.go:237/324/339 and the ramp baseline store_postgres_ramp_baseline.go:159): az_config is included, and both sides are recomputed from persisted Details.
    • The rec ID (scheduler.go:1148): included.
    • The upsert ON CONFLICT: 12 columns, matching the 000140 index.
    • purchaseIdempotencyKey (handler_purchases.go:3378): included, and recomputed on both sides of matchDuplicateInList.
    • Suppression: missing (the blocking finding above).
    • Rec IDs are not persisted anywhere else.
  • Mixed-key window:
    • The generated column's table rewrite backfills stored rows. A legacy multi-az row inserted at 134 reads az_config=multi-az after the up migration, and an EC2 row reads ''.
    • Between the release and 000141, a stored pair still fails with 23505 on the old index. Before this PR the same batch failed with 21000, so this is no regression.
    • Deploying the code before 000140 breaks, because the ON CONFLICT needs the column. The documented deploy order covers this.
  • Non-RDS: rec IDs for these cases are byte-identical between base and merged:
    • EC2 zonal, and EC2 with platform/tenancy;
    • ElastiCache and Savings Plans;
    • RDS without az_config;
    • Azure VM and GCP.
  • Fail-before:
    • The head scheduler identity tests FAIL on base (RDSAZConfigInID, MalformedAZConfigErrors).
    • The AZConfigBatch store test FAILS on base plus 000140 only.
  • Mutation probes on a scratch copy, each restored and confirmed identical with cmp:
    • M1, az_config dropped from RecIdentityKey: caught by RecIdentityKey_DiffersByAZConfig and both PriceRecommendationsFromStore tests.
    • M2, dropped from the idempotency key: caught.
    • M3, the rec ID suffix dropped: caught.
    • M4, ON CONFLICT reverted to 11 columns: caught by Batch and Concurrent.
    • M5, 000141 keeps the 000120 source index: caught by the store and migration tests.
    • M6, COALESCE removed from the generated column: caught by both migration tests and the store test.
    • No test covers the suppression gap.

Verified locally on the merged tree (origin/main 6af022f plus head, with merge-tree clean), with GOTOOLCHAIN=go1.26.10 and no cloud credentials, under the heavy lock, with testcontainers Postgres:

  • Build, vet and unit tests:
    • go build ./...: OK.
    • go vet on config, scheduler, api, purchase, planstep and migrations, with and without -tags integration: OK. gofmt: clean.
    • go test -count=1 on config, scheduler, api, purchase, planstep and mocks: ok.
  • Migration integration tests (-tags integration ./internal/database/postgres/migrations/): PASS.
    • RecommendationsAZConfig and DropRecommendations11ColNaturalKey.
    • The neighbouring tests: RecommendationsPlatformTenancy, DropRecommendations9ColNaturalKey, RecommendationsSourceAccount, PurchaseSuppressionsVariant and DownAllRoundTrip.
  • Up/down/up cycle: two full cycles of 134 -> 141 -> 134 -> 141 in a scratch test.
    • After each down, the index set equals the pre-140 set and the az_config column is gone.
    • The pair collapses to the newest row.
    • An exact EC2 duplicate is still rejected at 141.
  • Other integration tests:
    • ./internal/config -run 'AZConfig|UpsertRecommendations|Suppression': ok.
    • ./internal/purchase -run TestApproveAzurePricingRequiresMigrationAndRecollection: ok.

Reviewed by an independent agent with no author context (second reviewer). Review only: nothing was pushed, and the GitHub review state is unchanged.

Bring the branch up to date with main.

Conflicted files:
- internal/config/store_postgres_recommendations.go: main (LeanerCloud#1209) removed
  ReplaceRecommendations and the onConflict flag, making the natural-key
  ON CONFLICT clause unconditional. The branch had switched that clause to
  the 12-column key with az_config. Kept main's unconditional upsert and
  applied the branch's az_config arbiter to it, plus the branch's doc
  comment on the 12-column index.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@IAmJSD

IAmJSD commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator Author

Merged origin/main (1c5787a) into this branch. New head: 1fa7665.

Conflict resolution:

  • internal/config/store_postgres_recommendations.go: main (fix(db): remove ReplaceRecommendations and its unscoped DELETE #1209) removed ReplaceRecommendations and the onConflict flag, so the natural-key ON CONFLICT clause is now unconditional. This branch had switched that clause to the 12-column key with az_config. I kept main's unconditional upsert, pointed it at the 12-column az_config arbiter, and kept this branch's doc comment.

I found no semantic conflicts. Nothing on the branch calls ReplaceRecommendations. The new tests build alongside #1400's TestMain files. Migrations 000140 and 000141 do not clash with main, which is at 000139.

Checks on the merged head:

  • build, vet (also with -tags integration) and e2e vet pass.
  • golangci-lint v2.10.1 found 0 issues with and without the integration tag, and gocyclo is silent.
  • go test -race -short passes on the changed packages, and so does go test -race -tags integration (api, config, migrations, purchase, scheduler).
  • The secrets scan is clean and the planted key was flagged. The allowlist test passes.

@IAmJSD

IAmJSD commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator Author

Independent review (IAmJSD's agent, fresh context, given only the diff, the issue and the PR text): VERDICT: REQUEST_CHANGES 1fa766514d2abba660a8f04be78207a056d83464

Blocking:

  • internal/config/recommendation_identity.go:100 (SuppressionVariant), internal/config/types.go:754 and internal/scheduler/scheduler.go:1452 (matchingSuppressions): purchase suppression is still keyed on the 6-tuple plus AZ/platform/tenancy, without az_config. I missed this in my own pass. It was first raised in cristim's earlier review on a035c5d5, it is unaddressed on this head (the only change since is a main merge), and I reproduced it independently.
    • Before this PR, a Single-AZ and a Multi-AZ RDS row of the same class could never both be stored. This PR makes them coexist, and that is the point of the PR.
    • So buying N Single-AZ RIs now builds a suppression whose variant ({"" "" ""}) also AppliesTo the Multi-AZ sibling. The sibling's count drops by N, or it is hidden, for the whole grace period. They are different offerings covering different DB instances, so that is a silent under-buy. fix(purchase): suppression key lacks availability zone, platform and tenancy #816/fix(purchase): scope purchase suppressions to the AZ, platform and tenancy bought #1110 fixed exactly this for EC2 platform/tenancy.
    • Repro on this head merged with main: a scratch test builds config.BuildSuppressions from a 2x single-az purchase and runs applySuppressionIndex over a 2x multi-az rec with the same 6-tuple. The multi-az rec is dropped (out=[]).
    • Fix: add az_config to SuppressionVariant, suppressionBucketOf, and the purchase_suppressions variant columns and unique key. That is a new migration, 000149 or above. Alternatively, hold 000141 until it lands.

Non-blocking:

  • Deploy ordering is load-bearing in both directions, and nothing in code enforces it. I reproduced each failure mode:

    • New code before 000140: ON CONFLICT (..., az_config) has no arbiter, so every collection write fails with SQLSTATE 42P10.
    • Old code after 000141: the 11-column arbiter is gone, also 42P10. I hit this by running the merge-base store_postgres_recommendations.go against the migrated schema: TestPostgresStore_UpsertRecommendations_AZConfigBatch fails with 42P10.
    • New code between release and 000141: if a stored RDS row is single-az and Cost Explorer now returns only the multi-az variant of the same class, the 12-column arbiter sees no conflict. The INSERT then violates the still-present 11-column index (23505), and the whole multi-provider batch write is lost. On main that case silently overwrote the row.

    Dev auto-migrates both at once, so there is no window there. For prod, the 140, then release, then 141 runbook in the PR body has to be followed exactly, with 141 dispatched promptly after the release. A schema-version guard in UpsertRecommendations (like azurePricingMigrationVersion) would turn the first mode into a clear error, but that is optional.

  • API contract change: an RDS (or Azure SQL) purchase request without details.az_config no longer matches a stored row that has one, and gets a 409. The web modal echoes Details. Any external or MCP/CLI client that submits RDS purchases without Details will start failing. This is disclosed, and it is consistent with fix(purchase): carry EC2 platform and tenancy in rec ID, identity key and idempotency key (#806) #817's EC2 rule.

  • purchaseIdempotencyKey now appends |<az_config> for every rec, so every submit fingerprint changes on deploy. That is harmless: the fingerprint is only compared against keys recomputed from pending rows (matchDuplicateInList) and is never persisted (the idempotency_key column is the UUID lineage key, see purchase/execution.go). So in-flight dedupe stays consistent across the deploy.

  • From the earlier review: DetailsAZConfig uses json.Unmarshal, which matches keys case-insensitively, while the generated column matches exactly. They diverge only on hand-crafted Details. Also, Azure SQL az_config is derived from a cached dominant server-inventory value, so Azure SQL rec IDs can change between collections, which gives a 409 on a stale tab.

  • The frontend cellKey still groups Single-AZ and Multi-AZ RDS into one cell. The body flags this as a UI follow-up.

Attacked:

  • Migrations. 140/141 are the numbers reserved for fix(recommendations): key stored recs, rec ID and purchase identity on RDS az_config (#408) #1239, and main is at 000139. The 000140 generated column uses the same COALESCE(payload->'details'->>'az_config','') that DetailsAZConfig mirrors. In 000141 down, deduping on the 11-column key also clears duplicates under the finer 000120 source key, and the down migration recreates both indexes byte-for-byte. 000120's down guard still holds because 141-down runs first. The table rewrite is on a regenerable cache table.

  • Identity consumers: RecIdentityKey (pricing index, planstep, ramp-baseline netting), recommendationID, purchaseIdempotencyKey. Each is recomputed from persisted Details on both sides. No rec ID is persisted anywhere (no recommendation_id references in config). ComputeIdentity and DetailsAZConfig share the decode and error shape.

  • Ran: go build ./...; go vet (plain and -tags integration) on api/config/scheduler/purchase/database; go test -race -short -count=1 on api/config/scheduler/planstep/database (ok); go test -race -tags integration on the targeted migration tests (000109/111/120/140/141 and the 9-col drop: all pass), config Recommendation|AZConfig|RampBaseline|Upsert, purchase Azure|AZConfig|Pricing, api Purchase|Recommendation|Pricing, and the whole scheduler package (all ok). The full internal/database/postgres/migrations integration suite passes on a local merge with current main (81 commits ahead of the merge base, clean merge; build, vet and unit also pass there). golangci-lint v2.10.1 with and without the integration tag: 0 issues. gocyclo -over 10 on the changed files: silent.

  • Fail-before: with the merge-base scheduler.go and handler_purchases.go, and az_config blanked out of RecIdentityKey, these fail:

    • TestPriceRecommendationsFromStore_PricesFromRequestedAZConfig, ..._RequestWithoutAZConfigRefusedForRDS
    • TestPurchaseIdempotencyKey_DiffersByAZConfig
    • TestRecIdentityKey_DiffersByAZConfig
    • TestConvertRecommendations_RDSAZConfigInID, TestRecommendationID_MalformedAZConfigErrors

    Restored afterwards.

Body-stale:

Reviewer: rB22. Bar while the maintainers are away (set by IAmJSD as acting owner): one approving independent review on the head, two for money, auth and security paths.

IAmJSD added a commit to IAmJSD/cloud-commitments-platform that referenced this pull request Oct 26, 2026
…ation 000145 -> 000141

golang-migrate only applies versions above the database's current one,
so PRs must merge in migration-number order. 000140-000144 had been
taken by PRs that are now held on owner decisions (LeanerCloud#1239, LeanerCloud#1241,
LeanerCloud#1253). The ready PRs move to 000140-000145 so they can merge ahead of
them, and the held ones will take the next free numbers when decided.
Rename only: the SQL statements are unchanged, and the test version
constants and the comments naming the migration follow the new number.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
IAmJSD added a commit to IAmJSD/cloud-commitments-platform that referenced this pull request Oct 26, 2026
…000140

golang-migrate only applies versions above the database's current one,
so PRs must merge in migration-number order. 000140, 000141, 000143 and
000144 belong to PRs held on owner decisions (LeanerCloud#1239, LeanerCloud#1241, LeanerCloud#1253); this
PR takes 000140 so it can merge ahead of them, and they will take the
next free numbers when decided. Rename only: the SQL is unchanged, and
the test's version constant and the comments naming the migration
follow the new number.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
IAmJSD added a commit to IAmJSD/cloud-commitments-platform that referenced this pull request Oct 26, 2026
…7 -> 000143

golang-migrate only applies versions above the database's current one,
so PRs must merge in migration-number order. 000140-000144 had been
taken by PRs that are now held on owner decisions (LeanerCloud#1239, LeanerCloud#1241,
LeanerCloud#1253). The ready PRs move to 000140-000145 so they can merge ahead of
them, and the held ones will take the next free numbers when decided.
Rename only: the SQL statements are unchanged, and the test version
constants and the comments naming the migration follow the new number.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
IAmJSD added a commit to IAmJSD/cloud-commitments-platform that referenced this pull request Oct 26, 2026
…-> 000145

golang-migrate only applies versions above the database's current one,
so PRs must merge in migration-number order. 000140-000144 had been
taken by PRs that are now held on owner decisions (LeanerCloud#1239, LeanerCloud#1241,
LeanerCloud#1253). The ready PRs move to 000140-000145 so they can merge ahead of
them, and the held ones will take the next free numbers when decided.
Rename only: the SQL statements are unchanged, and the test version
constants and the comments naming the migration follow the new number.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
IAmJSD added a commit to IAmJSD/cloud-commitments-platform that referenced this pull request Oct 26, 2026
…148 -> 000144

golang-migrate only applies versions above the database's current one,
so PRs must merge in migration-number order. 000140-000144 had been
taken by PRs that are now held on owner decisions (LeanerCloud#1239, LeanerCloud#1241,
LeanerCloud#1253). The ready PRs move to 000140-000145 so they can merge ahead of
them, and the held ones will take the next free numbers when decided.
Rename only: the SQL statements are unchanged, and the test version
constants and the comments naming the migration follow the new number.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@IAmJSD

IAmJSD commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator Author

Migration renumbering needed before this merges. The reviewed, ready PRs have taken 000140-000145 so they could merge without waiting on owner decisions: #1240 (000140), #1302 (000141), #1268 (000142), #1287 (000143), #1289 (000144), #1116 (000145). This PR's migration(s) (000140, 000141) must be renumbered to start at 000146, or the next free number at that time, when it's ready to merge (for example 000140 -> 000146 and 000141 -> 000147, keeping the expand/contract pair adjacent and in order). golang-migrate skips a version below the database's current one, so a lower number merged after those would never apply.

Dependency check: none of the six ready migrations depends on this PR's schema. #1240's recommendations_scan_key_idx is a non-unique index on (COALESCE(payload->>'id','') COLLATE "C", id). It doesn't involve az_config or the natural-key indexes, and this PR's generated-column table rewrite just rebuilds it. From reading the SQL, nothing in the ready set gets in the way of this PR applying after them. That ordering hasn't been run, though.

IAmJSD added a commit to IAmJSD/cloud-commitments-platform that referenced this pull request Oct 26, 2026
…n 000146 -> 000142

golang-migrate only applies versions above the database's current one,
so PRs must merge in migration-number order. 000140-000144 had been
taken by PRs that are now held on owner decisions (LeanerCloud#1239, LeanerCloud#1241,
LeanerCloud#1253). The ready PRs move to 000140-000145 so they can merge ahead of
them, and the held ones will take the next free numbers when decided.
Rename only: the SQL statements are unchanged, and the test version
constants and the comments naming the migration follow the new number.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
IAmJSD added a commit that referenced this pull request Oct 26, 2026
…d of the full list (#1240)

* fix(api): add opt-in keyset pagination to GET /api/recommendations

The list endpoint returned the whole filtered recommendations cache in a
single body, so a tenant with a few thousand cached rows pushed the
response past the 6 MB Lambda payload cap and the page failed with 502.
Gzip (#775) raised that ceiling but did not bound it.

Callers can now send limit (1-1000) and the returned next_cursor to read
the list in pages ordered by recommendation ID. The cursor is an opaque
base64url encoding of the last ID seen, so a row deleted between requests
does not shift later pages. summary and regions still describe the full
filtered set on every page, and compute capacity is only decoded for the
returned rows. Requests without limit or cursor keep the existing
unpaginated response, so the current frontend is unaffected until it
adopts the parameters. openapi.yaml documents the parameters and the
actual response shape.

Refs #777

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(api): cap recommendation pages by serialized bytes and reject empty paging params

A page was bounded only by row count, but the opaque Details blob makes a
row's size data-dependent, so 1000 large rows could still pass the 6 MiB
Lambda response cap. Pages now also stop once the rows, measured as they
appear inside the Lambda response envelope (the body re-encoded as a JSON
string, so quotes and backslashes count twice), plus the summary, regions,
keys and next_cursor would pass a 4 MiB budget. The budget is checked
against uncompressed bytes because the #775 gzip only runs when the client
accepts it. A page always holds at least one row, so a row that alone is
over the budget is returned on its own page with a logged warning instead
of stalling the cursor walk.

Legacy unpaginated mode is now selected only when neither limit nor
cursor is present. A present but empty ?limit= or ?cursor= returns 400
instead of falling back to the unbounded response.

Refs #723

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(api): reject duplicate recommendation IDs when paging and document 400s

- Correct the ordering comment: the paging key is the application
  recommendation ID built from the natural-key columns, not the cache
  table's UUID primary key. A paginated request now fails if two rows
  share an ID, since the strict "after" comparison would skip one of
  them at a page boundary.
- Describe the feature as opt-in in-memory cursor slicing, not a SQL
  keyset (store-side paging is #981).
- TestGetRecommendations_UnpaginatedShapeUnchanged copies the expected
  IDs before the call so an in-place reorder in legacy mode fails it.
- Add TestGetRecommendations_PaginationRejectsDuplicateIDs.
- openapi: document the 400 response for bad limit/cursor/filter
  values, and say next_cursor's absence is the only end-of-list signal.

Refs #723

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(db): index recommendations by byte-wise payload id for keyset scans

Migration 000142 adds recommendations_scan_key_idx on
(COALESCE(payload->>'id', '') COLLATE "C", id). The paginated
GET /api/recommendations scan (issue #981) walks the cache in chunks in
that order and resumes each chunk after the last (id, primary key) it
read; without the index every chunk is a full scan plus a sort.

COLLATE "C" is byte order, the same order Go's string comparison uses,
so SQL paging keeps the order the API already returns.

Refs #981

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(config): read stored recommendations in keyset chunks

ListStoredRecommendationsChunk returns at most limit rows matching a
RecommendationFilter, ordered by the byte-wise recommendation ID and then
the table's UUID primary key, resuming strictly after a given position.
The primary key keeps rows that share an ID in a total order, so a chunk
boundary inside such a run neither skips nor repeats a row. The
MinSavingsPct floor stays in-process (shared with
ListStoredRecommendations via decodeStoredRecommendation); rows it drops
still advance the position.

Integration tests pin the order to Go's sort.Strings on IDs that a
linguistic collation orders differently, cover duplicate IDs across every
chunk size, the pct floor emptying a whole chunk, the empty table, an
EXPLAIN check that recommendations_scan_key_idx serves the query without
a Sort node, and walks racing concurrent upserts.

Refs #981

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(scheduler): scan cached recommendations chunk by chunk

ScanRecommendations walks the recommendations cache through
ListStoredRecommendationsChunk in byte-wise ID order and applies the same
suppressions and account overrides as ListRecommendations to each chunk,
handing the survivors to a callback. It holds one chunk at a time instead
of the whole filtered set. Suppressions are read once per scan and each
(account, provider, service) override is resolved once per scan, not once
per chunk. Read failures over-show exactly like ListRecommendations.

The freshness read, cold-start kick and stale-while-revalidate kick are
pulled out of ListRecommendations into startCacheRead/finishCacheRead so
both reads share them unchanged.

Tests pin the scan to ListRecommendations' output (sorted by ID) for chunk
sizes 1 to 31 with suppressions and overrides active.

Refs #981

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(api): page GET /api/recommendations from the chunked scan

A paginated request (limit or cursor sent) no longer loads the whole
filtered set through ListRecommendations and sorts it in memory. It runs
Scheduler.ScanRecommendations, applies the enabled-providers and account
scope filters to each chunk (their inputs read once per request), and
feeds the survivors to recommendationsPageCollector. The collector totals
every row into the summary and keeps only the rows after the cursor up to
limit+1 (the page plus one row proving a next page exists), then cuts the
page with the existing byte budget. Peak memory is one scan chunk plus
one page instead of the whole tenant.

Output is unchanged: rows arrive in byte-wise ID order (Go string order),
duplicate IDs still fail the request, and the summary still covers the
full filtered set. The unpaginated path is untouched.

The in-memory sortRecommendationsAfter/buildBudgetedPage are removed;
the existing #723 pagination tests now run against the scan path, with
buildBudgetedPage kept as a test helper over the collector.

Refs #981

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(api): pin paginated recommendations order, ties and last page

- PaginatedRequestDoesNotLoadFullList: the #981 regression; a paged
  request must use ScanRecommendations, never ListRecommendations.
- PagedWalkMatchesLegacyOrder: for an unrestricted and a scoped reader,
  every chunk size and page size (including the exact-fit last page), the
  walk returns the legacy rows sorted by Go string order in consecutive
  pages, next_cursor on all but the last page, and the legacy summary on
  every page. IDs mix case, "|", "-", "/", spaces and accents.
- PagedDuplicateIDs: ties fail every paged request at any chunk/page
  boundary; a twin removed by the scope filter is not a tie.
- Background-collection scope (#627), the one-page memory bound, the
  out-of-order guard and the empty cache.
- An integration test walks a real Postgres cache spanning several scan
  chunks with the min_savings_pct floor and matches the legacy output.

Refs #981

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(scheduler): scan with main's variant-aware suppression index

Merging main brought #1110 (issue #816), which changed
aggregateSuppressions to return the raw rows grouped by 6-tuple
(map[suppressionKey][]config.PurchaseSuppression) and moved the per-rec
aggregate into applySuppressionIndex, because which rows apply now depends
on the rec's variant (AZ, platform, tenancy). ScanRecommendations still
held the old map[suppressionKey]*suppressionAgg, so the merged tree did not
build.

loadSuppressionIndex now returns main's grouped index and the scan passes
it to the same applySuppressionIndex as ListRecommendations. Matching is
per rec, so applying it chunk by chunk suppresses exactly what the non-scan
path does.

A new test pins the scan to ListRecommendations at every chunk size on an
EC2 fixture with a variant purchase, a fully covered variant, a legacy
wildcard row and a rec with unreadable Details, and checks the expected
per-variant counts. Stripping variants from the scan's index fails it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(db): renumber recommendations_scan_key_idx migration 000142 -> 000140

golang-migrate only applies versions above the database's current one,
so PRs must merge in migration-number order. 000140, 000141, 000143 and
000144 belong to PRs held on owner decisions (#1239, #1241, #1253); this
PR takes 000140 so it can merge ahead of them, and they will take the
next free numbers when decided. Rename only: the SQL is unchanged, and
the test's version constant and the comments naming the migration
follow the new number.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Cristian Magherusan-Stanciu <cristim@users.noreply.github.com>
IAmJSD added a commit that referenced this pull request Oct 26, 2026
…vice (#1302)

* fix(purchase,server): skip the $0 purchase confirmation and report a failed lazy DB init on /health

An execution with nothing selected never reaches a cloud API, but both the
single-account and the per-account fan-out paths still sent a "purchase
confirmed" email listing zero commitments. Gate the confirmation on
anyRecPurchased / committed, so it goes out only when a real commitment
was created (full or partial success).

When reinitializeAfterConnect fails after setting app.Config, ensureDB
closes the pool and clears app.DB but leaves app.Config, and the
config_store health check reported healthy over the closed pool. Snapshot
dbConnected with the store pointers under dbMu and report unhealthy while
lazy PostgreSQL init has not completed.

Audit findings A05-015 and A06-009. A06-019 (commitment_type labels)
follows separately.

Refs #339

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(analytics): label snapshot commitment types from provider and service

commitmentTypeFor matched only a service literally named "SavingsPlans",
which purchase_history never stores, so every commitment was stored as
"RI": AWS Savings Plans ("savingsplans" / "savings-plans-<type>") and GCP
committed-use discounts included. Derive the type from (provider, service)
in the pkg/common CommitmentType vocabulary, and widen the
savings_snapshots CHECK in migration 000145 to match. The CHECK still
admits the legacy 'RI' / 'SavingsPlan' literals so an old replica can
write during a rolling deploy; existing rows are relabeled.

Audit finding A06-019.

Refs #339

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(purchase): make the #339 no-confirmation assertions checkable by the mock guard

TestNoUnfailableMockAssertions could not resolve the receiver of two
AssertNotCalled sites (a mock returned by a helper). Declare the mock
directly in the new test, and in the approve test rely on the missing
SendPurchaseConfirmation expectation, which fails the test on any send.

Refs #339

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(analytics): restore the pkg/common import after #1254

#1254 (on main) replaced the hand-rolled config-store mock in
collector_test.go and dropped the now-unused pkg/common import. This
branch's new commitment-type assertion uses common.CommitmentType, so the
analytics test build broke after main was merged in although the merge
was textually clean. Same assertion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(db): renumber savings_snapshots_commitment_type_vocabulary migration 000145 -> 000141

golang-migrate only applies versions above the database's current one,
so PRs must merge in migration-number order. 000140-000144 had been
taken by PRs that are now held on owner decisions (#1239, #1241,
#1253). The ready PRs move to 000140-000145 so they can merge ahead of
them, and the held ones will take the next free numbers when decided.
Rename only: the SQL statements are unchanged, and the test version
constants and the comments naming the migration follow the new number.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
IAmJSD added a commit that referenced this pull request Oct 26, 2026
…ail-closed plan providers, revocation_in_flight clear (#1268)

* fix(config/db): audit findings A04-010/011/012/013 in migrations and config store

- 000059 down: only remove ...0005 when it really is Purchaser. 000057 gave
  that UUID to Standard Users, so the old down deleted Standard Users and
  stripped it from every member. Refuse with a clear error rather than leave
  a Purchaser-only user with no group. 000096 down gets the same up-front
  refusal instead of a bare users_min_one_group violation.
- 000027: dedupe savings_snapshots on (tableoid, ctid). ctid is per
  partition, so matching on it alone deleted unrelated rows of other
  partitions on a forced re-run.
- DerivePlanProviders: accept the "provider:service" keys 000040 wrote, and
  return an error instead of an empty provider set when no key names a
  provider, so both plan-account guards fail closed.
- MarkPurchaseRevoked clears revocation_in_flight, as 000072 documents;
  migration 000146 clears it on rows already revoked.

Real-Postgres tests for each, using the shared container. Normalizing the
plan-services key delimiter is deferred (see the PR).

Refs #316

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(config): identify legacy-keyed plan services by their provider field

The first #316 commit made DerivePlanProviders fail when no service key
had a provider prefix. That refused supported plans: a legacy alias kept
under its original key (#1123's singleton-alias edit path, exercised by
TestLegacyServiceB1S1*) has a key like "original-key", and so do fixtures
keyed "ec2" with Provider "aws".

Each entry's provider now comes from its key prefix ("provider/service"
or "provider:service"), falling back to the entry's own Provider field,
which is what mergeRequestedServices and plan step resolution use. The
check fails closed only when no entry identifies a provider either way.

Refs #316

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(migrations): drop the duplicate TestMain from helpers_test.go

main's #1400 added main_integration_test.go with a TestMain that stops
the shared PostgreSQL container after the run. This branch added an
identical TestMain to helpers_test.go, so after merging main the
integration build of the package declared TestMain twice and failed to
compile. Keep main's TestMain, which does the same teardown, and
restore helpers_test.go to main's version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(db): renumber purchase_history_clear_revoked_in_flight migration 000146 -> 000142

golang-migrate only applies versions above the database's current one,
so PRs must merge in migration-number order. 000140-000144 had been
taken by PRs that are now held on owner decisions (#1239, #1241,
#1253). The ready PRs move to 000140-000145 so they can merge ahead of
them, and the held ones will take the next free numbers when decided.
Rename only: the SQL statements are unchanged, and the test version
constants and the comments naming the migration follow the new number.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
IAmJSD added a commit that referenced this pull request Oct 26, 2026
…ngs rollups (#1287)

* fix(analytics): keep unknown usage and commitment unknown in the savings rollups

savings_snapshots.total_usage and total_commitment are nullable on purpose:
NULL means the provider gave no figure. monthly_savings_summary and
provider_savings_summary rolled them up with SUM and AVG, which skip NULL,
so "us-east-1 $4000 + eu-west-1 unknown" became a definite $4000.

Migration 000147 recreates both views so a timestamp's total is NULL when
any of its rows is unknown, and the period average is NULL when any of its
timestamps is. A known 0 stays 0. Column names and types, the unique
indexes and the CONCURRENTLY refresh path are unchanged; the down migration
restores the previous definitions.

Integration tests on real Postgres cover unknown, known zero, fully known,
unknown-at-one-instant, down/up, and a refresh_savings_materialized_views()
call queued behind the migration's locks.

Refs #151

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(db): renumber savings_rollups_propagate_unknown migration 000147 -> 000143

golang-migrate only applies versions above the database's current one,
so PRs must merge in migration-number order. 000140-000144 had been
taken by PRs that are now held on owner decisions (#1239, #1241,
#1253). The ready PRs move to 000140-000145 so they can merge ahead of
them, and the held ones will take the next free numbers when decided.
Rename only: the SQL statements are unchanged, and the test version
constants and the comments naming the migration follow the new number.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
IAmJSD added a commit that referenced this pull request Oct 26, 2026
…nd AWS fee (backend slice of #27) (#1289)

* feat(marketplace): add listing settlement columns and guarded poller writes

Migration 000148 adds the purchase_history settlement columns that 000087
deferred (listed_at, listing_proceeds_received, listing_fee_paid).
listing_price_schedule already exists via 000105 (#525) and is left alone,
including on rollback.

The store gains the three statements the RI Marketplace status poller
needs: the scan of listings to poll and two status-guarded UPDATEs. The
guards (same listing id, active or same terminal state, proceeds not yet
recorded) decide whether a poll result applies. Concurrent polls therefore
record proceeds once, and a cancel recorded by the cancel handler is never
revived or turned into a sale. Real-Postgres tests cover the races.

Refs #27

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(marketplace): poll RI Marketplace listings for sales and fees

PollMarketplaceListings reads each listing due for a poll from AWS
(DescribeReservedInstancesListings through the existing listing-finder
interface, signed as the purchase's own account like list and cancel).
While a listing is active it records its CreateDate as listed_at. When
the listing closes, or is canceled after a partial sale, it records the
terminal state with the seller's net proceeds and the AWS fee: the
instances AWS reports sold, times the active price tier, minus the
12% fee, in whole cents. When instances sold but no single active USD
tier exists, the state is recorded and the proceeds are left unset for
a later poll instead of guessed. A failure on one listing is counted
and the run moves on to the next.

Refs #27

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(api): check poller non-writes through MockConfigStore's call log

The vacuous-assertion guard cannot resolve f.store as a receiver. Give
the two tests their own *MockConfigStore and use the name-only
AssertNotCalled, which reads the call log and fails on any call.

Refs #27

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(server): run the marketplace listing poll in the finalize_revocations sweep

The finalize_revocations task now also runs the RI Marketplace status
poller after the revocation repair. Both reconcile purchase_history with
the provider on the same hourly schedule, so no new schedule or
infrastructure is needed. The two steps are independent: one failing
does not skip the other, and the task fails if either did. The task's
result keeps the revocation counts under their existing keys and adds
the poll counts under marketplace_listings.

Refs #27

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(db): renumber purchase_history_listing_settlement migration 000148 -> 000144

golang-migrate only applies versions above the database's current one,
so PRs must merge in migration-number order. 000140-000144 had been
taken by PRs that are now held on owner decisions (#1239, #1241,
#1253). The ready PRs move to 000140-000145 so they can merge ahead of
them, and the held ones will take the next free numbers when decided.
Rename only: the SQL statements are unchanged, and the test version
constants and the comments naming the migration follow the new number.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(config): drop the previous listing's settlement when a relist is recorded

Repricing an RI on the Marketplace means cancel and relist, since AWS has no
price edit. A relisted row kept the earlier listing's listed_at, proceeds and
fee, and only RecordMarketplaceListingActive cleared them, which needs a poll
to see the new listing active first. A relist that sold out (or was canceled)
before the next hourly poll was therefore never settled: the settle guard
refuses rows whose proceeds are set, so the row stayed active with the old
0.00 and was polled and skipped every hour, or showed the old listing's
settlement as the new one's.

UpdatePurchaseHistoryListing and ResolveMarketplaceListingAttempt, the two
statements that write a listing id, now null the three columns in the same
UPDATE when the id changes. Re-recording the id the row already holds (a
cancel or its retry) keeps them. The new real-Postgres test runs the review's
exact sequence plus the canceled-within-the-hour and resolved-attempt
variants.

Refs #27

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
IAmJSD added a commit that referenced this pull request Oct 26, 2026
* fix(auth): bind group account scope entries to account IDs

A group's allowed_accounts could hold an account's display name. The name
was not bound to any account, so once that account was deleted or renamed
the entry matched whichever account was later created or renamed to the
same name. #1032 limited name matching to unique names, which does not
help here: the newcomer is the only account carrying the name.

Scope entries now match by account identifier only:

- AccountScope.Allows never compares an entry with a display name. A
  caller that knows an account only by name (legacy rows keyed by external
  ID) still resolves through WithRegisteredAccounts to the one account
  carrying that name, and that account's ID must be in the scope.
- Group create/update bind allowed_accounts at grant time: "*", account
  IDs and external IDs are kept; a name exactly one registered account
  carries, inside the actor's own scope, is stored as that account's ID;
  anything else is a 400. A failed account or scope load refuses the write.
- Migration 000135 rewrites existing name entries to IDs when exactly one
  account carries the name and it predates the group's last write. Other
  entries are left in place and no longer grant access. Down is a no-op.
- The name-only MatchesAccount and AuthContext.CanAccessAccount helpers,
  which had no production callers, are removed.

Closes #1034

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(api): scope the history-summary fixture by account ID

The history-summary integration tests from #1200 scoped their user with
the tenant's display name. This branch binds allowed_accounts entries to
account IDs and makes a bare name entry match nothing (#1034), so the
scoped user saw no tenant A rows and the totals and listed-rows checks
failed. Grant the scope by tenant A's account ID, the form group writes
now store. The assertions are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(db): renumber the group scope binding migration to 000149

main now ships migrations up to 000139, so 000135 would sit below
versions a deployed database has already applied and golang-migrate
would never run it. Open PRs hold 000140-000148, so this takes 000149
and must merge after that chain. The SQL is unchanged; only the file
names, header comments, test version and test fixture IDs move.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(db): renumber bind_group_account_names_to_ids migration 000149 -> 000145

golang-migrate only applies versions above the database's current one,
so PRs must merge in migration-number order. 000140-000144 had been
taken by PRs that are now held on owner decisions (#1239, #1241,
#1253). The ready PRs move to 000140-000145 so they can merge ahead of
them, and the held ones will take the next free numbers when decided.
Rename only: the SQL statements are unchanged, and the test version
constants and the comments naming the migration follow the new number.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(api): register the wire-format group account for grant-time binding

TestUserGroupWrites_WireFormat came in from main (#1261) and builds a
Handler with no config store. Since #1034, group writes bind every
allowed_accounts entry to a registered account and refuse the write when
the account list cannot be loaded, so POST /api/groups answered 500.
The test now registers its account ID in a store mock, as the binding
tests do; the wire format it pins is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(api): scope the History paging user by account ID, not name

TestIntegration_HistoryPaging_ScopedUserNeverPagesIntoAnotherTenant came
in from main (#1277) and grants the scoped user tenant A by its display
name. Since #1034 an allowed_accounts entry matches by cloud account ID
only, so that user saw none of tenant A's rows and the walk returned
only its own unscoped execution. The grant now names tenant A's ID, as
the other scope tests on this branch do; what the test asserts is
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(analytics): scope account_ids fixtures by account ID

The account_ids analytics tests that arrived with the main merge (#1448)
grant scope by account name ("a", "b", "c"). This branch binds group
scope entries to account IDs and no longer matches an entry against a
display name (#1034), so those name entries authorize nothing and the
scoped cases got 404. List the account IDs in the scope instead, as the
other analytics scope tests on this branch already do. The assertions
are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(analytics): scope the account_ids DB test by account ID

TestAnalyticsAccountIDs_SumsSelectedAccountsOnly_DB (from #1448) grants
the scoped user ["a", "b"], which are account names. Scope entries are
account IDs on this branch (#1034), so the name entries allowed nothing
and the in-scope query returned 404. Use the created accounts' IDs.
The foreign-id refusal assertion is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(history): scope the plan_id filter DB test by account ID

TestIntegration_HistoryPlanFilter_CrossTenant (from #1445) grants the
scoped user tenant A by account name. Scope entries are account IDs on
this branch (#1034), so the name allowed nothing and tenant A's own
plan came back empty. Grant tenant A's ID, as the other history
integration tests on this branch do. The cross-tenant assertions are
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(api): scope #1464's actionable History tests by account ID

#1464 landed on main with tests that scope the user by the display name
"dev" and with an executionScope case built on WithRegisteredNames. This
branch (#1034) makes name entries grant nothing and replaces that
constructor, so after merging main those tests no longer compiled or no
longer exercised the scoped path. Scope them by the account's ID, which
is what a group stores now, and pin that a name entry contributes only
its literal value to the SQL scope.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(auth,api): bind account names to the identifier in scope checks

Review (rE7) found that an ID-scoped user could pass the purchase gate
for an account outside their scope. resolveAccountNamesByID keyed
names by both account ID and the free-text external ID in one map, so
an editor of in-scope account Z who set Z's external ID to victim Y's
ID got Z's name back for Y. AccountScope.Allows then resolved that
name to Z, which is in scope, and allowed a purchase in Y. The same
map fed the dashboard, ladder and RI reshape filters.

Decide those gates on the record's cloud_account_id itself, the ID
the purchase resolves its credentials from, and delete the dual-keyed
map. Bind the name path in Allows as well, so a name only identifies
an account through that account's own external ID and never through a
value that is some registered account's ID. Callers that pass a name
for an external ID already resolve it within the record's provider or
require exactly one owner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@cristim

cristim commented Oct 26, 2026

Copy link
Copy Markdown
Member

VERDICT: REQUEST_CHANGES 1fa7665

Re-review of #1239. The previous verdict was REQUEST_CHANGES at a035c5d. The only change since then is the merge 1fa7665 of origin/main 1c5787a (git log --no-merges a035c5d5..1fa76651 ^origin/main is empty). The PR's own diff against origin/main is unchanged apart from the conflict resolution in store_postgres_recommendations.go, which now uses main's unconditional upsert with the 12-column az_config arbiter. That resolution is correct.

Findings:

  • [blocking] internal/database/postgres/migrations/000140_recommendations_az_config..sql, 000141_drop_recommendations_11col_natural_key..sql: the migration versions collide with main.
    • origin/main (fetched 2026-10-26) now contains 000140_recommendations_scan_key_idx and 000141_savings_snapshots_commitment_type_vocabulary. It goes up to 000147_config_audit_events.
    • So this PR's numbers are not above the highest on main. They duplicate two existing versions.
    • git merge-tree reports no textual conflict because the file names differ, so the merge would go through silently.
    • Reproduced: I loaded the merged tree (429dacb5) through golang-migrate v4.20.1 iofs.New. It fails with duplicate migration file: 000140_recommendations_scan_key_idx.down.sql (source/iofs/iofs.go:68-72). Every migrate run, in every environment and in CI integration tests, would fail at startup.
    • The non-integration migration package tests still pass, so unit CI would not catch it.
    • Fix: renumber 000140 -> 000148 and 000141 -> 000149, or the next free pair at merge time, keeping the expand/contract pair adjacent. Update these with it:
      • the version references in the tests and comments (store_postgres_recommendations.go comment, DetailsAZConfig doc, the migration test names and files);
      • the 140-release-141 runbook in the PR body.
    • This confirms the author's 2026-10-26 comment. Its target of "000146" is itself already taken on main (000146_history_ramp_step_execution_error_not_null, 000147_config_audit_events).
  • [blocking, carried over from a035c5d, NOT fixed] internal/config/recommendation_identity.go:100-105 (SuppressionVariant), internal/config/types.go:754, internal/scheduler/scheduler.go:1451-1452: purchase suppression still has no az_config.
    • The variant is still {AvailabilityZone, Platform, Tenancy}.
    • So with Single-AZ and Multi-AZ RDS rows now coexisting, buying N Single-AZ RIs suppresses N from the Multi-AZ sibling, or hides it, for the grace period. That is an under-buy.
    • The author's own independent review (2026-10-25T23:52) reproduced this as well.
    • Fix: as before, add az_config to SuppressionVariant, suppressionBucketOf, and the purchase_suppressions variant columns and unique key. That needs a new migration, numbered after the renumbered pair. Alternatively, hold the contract migration until that lands.
  • [non-blocking, carried over, unchanged] These are all as described at a035c5d and in the author's review:
    • deploy ordering (expand, then release, then contract) is not enforced in code;
    • RDS purchases without details.az_config now get a 409;
    • Azure SQL az_config can flip between collections;
    • DetailsAZConfig uses case-insensitive json.Unmarshal, while the generated column matches keys exactly;
    • the dashboard cellKey/recCellKey still group Single-AZ and Multi-AZ.

Attacked:

  • Delta a035c5d..1fa7665: no author commits. The merge's resolution of store_postgres_recommendations.go matches main's fix(db): remove ReplaceRecommendations and its unscoped DELETE #1209 shape, with the ON CONFLICT on the 12 columns including az_config. The diff against origin/main is 16 files, +738/-12.
  • Earlier findings re-checked at the head:
    • suppression: still missing (finding 2);
    • migration ordering: now an outright version collision (finding 1);
    • the other non-blocking items: unchanged.
  • Migration numbering:
    • the highest version on origin/main is 000147;
    • this PR adds 000140 and 000141, both duplicates;
    • golang-migrate's duplicate rejection was confirmed by running iofs.New on the merged tree.
  • Merged-tree build: go test -count=1 ./internal/database/postgres/migrations/ (unit) is ok. That shows the collision is invisible without the integration tag or a real migrate run.
  • Not repeated this round: mutation probes M1-M6 and the integration suite from the a035c5d review. The PR code is unchanged since then, and the gating finding above makes the merged tree unmigratable anyway. Re-run them after the renumbering.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/m Days impact/few Limited audience priority/p2 Backlog-worthy severity/low Minor harm triaged Item has been triaged type/chore Maintenance / non-user-visible urgency/this-quarter Within the quarter

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Extend recommendation storage/scheduler identity to include purchase-critical Details (tenancy/platform/scope/az_config)

2 participants