Skip to content

fix(db/migrate): don't auto-heal a rolled-back migration as applied; make idempotency test real (adversarial-review) - #1451

Merged
cristim merged 1 commit into
mainfrom
fix/migration-autoheal-safety
Jul 17, 2026
Merged

cristim merged 1 commit into
mainfrom
fix/migration-autoheal-safety

Conversation

@cristim

@cristim cristim commented Jul 17, 2026

Copy link
Copy Markdown
Member

Summary

Two migration-safety defects found by adversarial review of committed origin/main, both in internal/database/postgres/migrations/migrate.go and its tests:

Defect 1 (HIGH): maybeAutoHealDirty silently marks a rolled-back migration as applied

golang-migrate records version=N, dirty=true BEFORE running N's SQL. Every migration in this repo is transactional (TestMigrationsAreTransactional), so an interrupted migration (Lambda timeout, connection drop) rolls back completely -- N's effects are ABSENT -- but the dirty flag persists.

The old Force(N)+Up() auto-heal treated the rolled-back migration as fully applied and continued at N+1, permanently skipping N's DDL while schema_migrations reported success. This is the 000074 divergence class: 42703 "column does not exist" errors at runtime on columns the version record claimed were present.

There is also a narrow race where N commits but the dirty=false update fails (N's effects ARE present). These two cases are indistinguishable without per-migration effect probes, so the conservative fix is:

  • maybeAutoHealDirty now returns a loud error with recovery instructions when dirty is detected, rather than silently clearing the flag.
  • The app still fail-opens (ensureDB swallows the error) and the CloudWatch alarm fires; no crash-loop.
  • Operators use CUDLY_FORCE_MIGRATION_VERSION=N (if effects confirmed) or N-1 (if not) to recover.
  • CUDLY_MIGRATION_AUTOHEAL=false still bypasses the check (surfaces the generic Up() dirty error instead).

Defect 2 (MED): TestMigrations_FullStackIdempotent falsely claimed to guard per-migration idempotency

The test ran RunMigrations twice, but the second call returns ErrNoChange from golang-migrate without executing any SQL -- no migration body is actually re-run. The docstring incorrectly claimed "this is the invariant the opt-in CUDLY_MIGRATION_AUTOHEAL path relies on" and that it "guards the whole directory."

Fix: corrected the docstring to accurately describe the test (verifies a no-op second call on a fully-migrated DB) and removed the false auto-heal dependency claim. The conservative auto-heal no longer relies on per-migration idempotency.

Changes

  • migrate.go: Replace Force(N)+continue in maybeAutoHealDirty with a loud error containing structured recovery instructions.
  • migrate_autoheal_test.go: Update the two "self-heals" test cases to expect errors and verify the dirty flag is left intact. Add regression test dirty-at-N with N-effects-absent (rolls back last migration, injects dirty=true at head, asserts error + dirty flag preserved).
  • migrate_idempotency_test.go: Correct the false docstring; remove the unverified "guards the whole directory" claim.

Test plan

  • go build ./internal/database/... passes
  • go vet ./internal/database/... passes
  • go test ./internal/database/... (non-integration): 181 passed
  • All pre-commit hooks pass (gofmt, go vet, cyclomatic complexity, gosec)
  • Integration tests (require Docker): go test -tags integration ./internal/database/postgres/migrations/... -- CI will run these
  • The three updated auto-heal test cases verify the new fail-loud behavior
  • The new regression test dirty-at-N with N-effects-absent verifies Force(N) is NOT called on a rolled-back migration

…ng effects

maybeAutoHealDirty previously called Force(N) on any dirty
schema_migrations row and let Up() continue at N+1. golang-migrate
records version=N, dirty=true BEFORE running N's SQL; every migration
here is transactional (TestMigrationsAreTransactional), so an
interrupted migration rolls back completely, leaving dirty=true at N
with N's effects ABSENT. The old auto-heal marked the never-applied
migration as complete and skipped its DDL forever while schema_migrations
reported success (the 000074 divergence class: 42703 errors at runtime
on columns the version record claimed were applied).

There is also a narrow race where N commits but the dirty=false update
fails, leaving dirty=true WITH effects present. These two cases are
indistinguishable from schema_migrations alone without per-migration
effect probes, so the conservative fix is to fail loud and let the
operator inspect the actual schema before choosing the recovery target
via CUDLY_FORCE_MIGRATION_VERSION (N if effects confirmed, N-1 if not).
The app still fail-opens (ensureDB swallows the error) and the
CloudWatch alarm fires; no crash-loop.

Test changes:
- Update the two "self-heals" test cases to expect a loud error with
  recovery instructions and verify the dirty flag is left intact.
- Add regression test "dirty-at-N with N-effects-absent": rolls back
  the last migration (effects absent), injects dirty=true at head,
  and asserts RunMigrations errors without Force-clearing the flag.
- Correct TestMigrations_FullStackIdempotent docstring: the second
  RunMigrations call returns ErrNoChange without running any SQL, so
  it does not verify per-migration idempotency; remove the false claim
  that it "guards the whole directory" and that auto-heal relied on it.
@cristim cristim added triaged Item has been triaged priority/p1 Next up; this sprint severity/high Significant harm urgency/this-sprint Within the current sprint impact/many Affects most users effort/m Days type/bug Defect labels Jul 17, 2026
@cristim

cristim commented Jul 17, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 17, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 57 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 81fd0729-5018-4cc1-ad69-77b68de1005f

📥 Commits

Reviewing files that changed from the base of the PR and between 7465a46 and 5c6bf44.

📒 Files selected for processing (3)
  • internal/database/postgres/migrations/migrate.go
  • internal/database/postgres/migrations/migrate_autoheal_test.go
  • internal/database/postgres/migrations/migrate_idempotency_test.go
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/migration-autoheal-safety

Comment @coderabbitai help to get the list of available commands.

@coderabbitai

coderabbitai Bot commented Jul 17, 2026 •

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@cristim

cristim commented Jul 17, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Jul 17, 2026 •

Copy link
Copy Markdown
Contributor
✅ Action performed

Full review finished.


You're currently rate limited under our Fair Usage Limits Policy. Your recent PR review activity is in the 95th percentile or higher among CodeRabbit users, so adaptive limits apply. Your next review will be available in 57 minutes.

@cristim
cristim merged commit 97e6f5c into main Jul 17, 2026
19 checks passed
@cristim
cristim deleted the fix/migration-autoheal-safety branch July 27, 2026 11:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/m Days impact/many Affects most users priority/p1 Next up; this sprint severity/high Significant harm triaged Item has been triaged type/bug Defect urgency/this-sprint Within the current sprint

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant