fix(hubs): never lower retention on redeploy

[Michael Flanakin (flanakin)]

Summary

• Copy-FileToAzureBlob.ps1 unconditionally overwrote settings.json's retention.ingestion.months / retention.final.months with whatever value the deploymentScript's Bicep parameters passed
• Bicep always resolves a value for an optional parameter — defaulting to 13 if the caller omitted it — so the script cannot tell an explicit redeploy value from a silently-defaulted one
• A redeploy that dropped a previously-customized retention value silently reset it to the default, and the next purge pipeline run aged out historical data older than the new cutoff (oldest data first) — matches the symptom reported in FinOps Hub upgraded from v13 to v14- two months data is missing in dashboard #2206
• Fix: take the max of stored and incoming retention instead of overwriting. Growing retention is always safe; shrinking it silently purges data and is hard to reverse

Test plan

• Added HubsRetentionGuard.Tests.ps1 asserting the max-guard is present and the old unconditional-overwrite pattern is gone, plus that first-run seeding (no existing settings.json) still honors the requested value
• Full unit suite passes (2283 passed, 0 failed)
• Maintainer review

Related: #2206

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Roland Krummenacher (RolandKrummenacher)]

The [Math]::Max guard is the right call — growing retention is safe, shrinking it silently purges data. Three things before this goes in, one of which means the PR as titled doesn't quite do what it says.

The same failure mode is still open on raw, in a path this guard structurally cannot reach. Details inline — short version is that retention.raw.days is written to settings.json and read by nothing, while the real raw retention is an ADX softdelete policy applied straight from the Bicep param at deploy time. Happy for that to be a follow-up rather than scope creep here, but "never lower retention on redeploy" currently overpromises.

Two of the six new tests can never fail. The Should -Not -Match anchors are wrong, so both regression guards pass whether or not the bug comes back. Inline.

No way to deliberately lower retention. After this, retention is a one-way ratchet — a customer who wants to reduce it to cut storage cost can't do it by redeploying, and there's no documented alternative. On a FinOps toolkit that's a pointed limitation, and the PR has no doc or changelog change. Even a line in the hubs docs saying "edit settings.json in the hub storage account to lower it" would close this.

Requesting changes mainly for the test anchors, since those are load-bearing for the regression this PR exists to prevent.

[Roland Krummenacher (RolandKrummenacher)]

These two negative assertions (this one and the final equivalent on line 44) can never fail, so the regression they're guarding is unprotected.

$ here is end-of-string — -match doesn't set RegexOptions.Multiline — and the trailing \s* can't span the code that follows the assignment. So the pattern only matches if that assignment happens to be the last thing in the file. I checked against a sample with the regression pattern deliberately present mid-file:

PR2288 anchored pattern (\s*$) matches : False
same pattern without the anchor        : True

Both tests pass today, and they'd still pass if someone reverted the fix. Dropping \s*$ from both patterns fixes it.

Minor, while you're here: these are source-text assertions, and the repo's convention for those is Tests/Lint/ (KqlJoinKinds.Tests.ps1, HubsKqlOperators.Tests.ps1) rather than Tests/Unit/.

[Roland Krummenacher (RolandKrummenacher)]

raw is left unguarded here, and it's the one where a silent shrink does the most damage.

I grepped src/ — nothing reads retention.raw.days back out of settings.json. This line is record-only. The retention that actually takes effect is baked into the KQL at deploy time (Analytics/app.bicep:369):

.alter-merge table ActualCosts_raw policy retention softdelete = $$rawRetentionInDays$$d recoverability = disabled

dataExplorerRawRetentionInDays defaults to 0 (main.bicep:154). So a customer who set it to, say, 30 and later redeploys without repeating the param gets softdelete = 0d recoverability = disabled reapplied to every *_raw table — an immediate and explicitly unrecoverable purge. That's the #2206 failure mode with a worse blast radius, and no settings.json guard can catch it because the value never round-trips through settings.json.

Two options: scope the title/description to the settings.json path, or open a follow-up for the raw policy. Either is fine, but it shouldn't go unrecorded.

[Roland Krummenacher (RolandKrummenacher)]

Worth spelling out the consequence in the docs: after this, stored retention can only ever grow. A customer who deliberately wants to lower retention to cut storage cost has no supported path — redeploying with a smaller value is now a no-op, silently.

That's the correct default, but it needs an escape hatch or at least a documented manual one ("edit settings.json in the hub storage account"). Otherwise the next issue is someone asking why their retention change didn't apply.

[microsoft-github-policy-service]

@Michael Flanakin (@flanakin): you have some new feedback!

Please review and resolve all comments and I'll let reviewers know by removing the Needs: Attention label. If I miss anything, just reply with #needs-review and I'll update the status.