WIP new migration#5399
Draft
denik wants to merge 11 commits into
Draft
Conversation
Collaborator
|
Commit: 7636ff9
22 interesting tests: 15 SKIP, 7 KNOWN
Top 24 slowest tests (at least 2 minutes):
|
denik
force-pushed
the
denik/new-migration
branch
from
13572c4 to
82112d7
Compare
denik
force-pushed
the
denik/new-migration
branch
from
82112d7 to
969c201
Compare
denik
force-pushed
the
denik/new-migration
branch
from
d9d854f to
f9471b6
Compare
denik
force-pushed
the
denik/new-migration
branch
from
fbe207c to
8a1ad59
Compare
Implement a new `bundle migrate` command that creates the direct state file
from local config + Terraform state attributes, without making any API calls.
Unlike `bundle deployment migrate` which calls DoRead for each resource, this
command reads resolved field values directly from the local TF state file.
Cross-resource references (e.g. ${resources.jobs.src.git_source[0].branch})
are resolved using two independent methods and the results reconciled:
- Method A: look up the field in the TF state of the resource that contains
the reference (e.g. read name from databricks_job.dst TF attributes).
- Method B: evaluate the template by reading each ${resources.*} reference
from the TF state of the referenced resource and interpolating.
If both methods agree, the value is used silently. If only one succeeds, that
value is used. If both succeed but disagree, the longer string is used with a
warning. If both fail, an error is returned.
The bundle/migrate package provides:
- ParseTFStateAttrs: parses the full TF state file (all resource attributes)
- LookupTFField: looks up a field value using DABsPathToTerraform translation
- ResolveFieldRef: reconciles Methods A and B for a single field
Co-authored-by: Isaac
The new migrate command reads only from the local TF state file and never invokes the Terraform engine, so a pre-migration plan check has no place here. The --noplancheck flag is kept but ignored to avoid breaking callers. Co-authored-by: Isaac
…/migrate
MigrateMode was a bool parameter on Apply that forked between "save state
without deploying" and "actually deploy". The two MigrateMode(true) callers
(bundle deployment migrate and upload_state_for_yaml_sync) now use
migrate.BuildStateFromTF directly, reading from the local TF state file
without any API calls.
Changes:
- New bundle/migrate/build_state.go: public BuildStateFromTF extracted from
cmd/bundle/migrate.go, taking *config.Root so callers can pass an
un-interpolated config (needed by upload_state_for_yaml_sync).
- bundle/direct/bundle_apply.go: drop MigrateMode type and parameter; Apply
now only handles real deployments.
- bundle/phases/{deploy,destroy}.go: drop MigrateMode(false) argument.
- upload_state_for_yaml_sync.go: replace CalculatePlan+Apply with
BuildStateFromTF; keep reverseInterpolate since config is TF-interpolated.
- cmd/bundle/deployment/migrate.go: replace CalculatePlan+Apply with
BuildStateFromTF; drop exported RunPlanCheck/GetCommonArgs wrappers.
- cmd/bundle/migrate.go: delegate to migrate.BuildStateFromTF.
Co-authored-by: Isaac
The new approach (BuildStateFromTF, no API calls) has been folded into the existing 'bundle deployment migrate' command. The separate 'bundle migrate' command is removed — it was redundant. Also add resolve_test.go: verify that int and bool cross-resource references work correctly when Method B returns a string value. structaccess.Set already handles string→int and string→bool conversion via strconv, so no bug exists. Co-authored-by: Isaac
The command now reads from the local TF state file without invoking the Terraform binary, so there is nothing for the plan check to verify. Also simplify getCommonArgs to return only the display string (the args slice was only needed to forward to the plan subprocess). Co-authored-by: Isaac
Remove --noplancheck from expected output after dropping that flag. Co-authored-by: Isaac
… update outputs
TF state stores single-block fields (e.g. continuous, deployment) as
single-element arrays [{}], not plain objects. json.Unmarshal into the
generated Go schema structs fails with type mismatch. Switch LookupTFField
to navigate via map[string]any + custom navigateTFState that auto-unwraps
single-element lists when a string-key step follows.
Also:
- Restore --noplancheck as a no-op flag (backward compat; used by invariant
tests for job_with_depends_on config).
- Remove the plan-check lines from acceptance test output.txt files.
- Update help output to include --noplancheck.
Co-authored-by: Isaac
…lancheck The refs map is shared with sv.Refs and gets mutated (entries deleted) during reference resolution. depends_on must be computed before that loop runs. Also restore --noplancheck as a no-op flag kept for backward compatibility (used by the invariant test suite for job_with_depends_on config). Co-authored-by: Isaac
…lias Two categories of field lookup failures in LookupTFField: 1. Postgres resources (postgres_projects, postgres_branches, etc.) use a 'spec' wrapper via DABsToTerraformWrappers, but some fields like 'name' are at the TF state root, not under spec. When the spec-prefixed path fails, retry with the original unwrapped path. 2. Model permissions reference 'model_id' (the numeric model ID) which TF stores as 'registered_model_id'. Add a tfStateFieldAliases map for such state-only field name mismatches. Also update acceptance output files: - default-python: remove musterr/--noplancheck pattern; the plan check that made the first migrate call fail no longer exists, so just call migrate once. - runas: remove plan-check output lines that appeared before 'Success!'. Co-authored-by: Isaac
Co-authored-by: Isaac
denik
force-pushed
the
denik/new-migration
branch
from
6cd9860 to
c8f326f
Compare
Tests cover: - Basic job stored with correct ID - Resource absent from TF state is skipped - Cross-resource string ref: depends_on computed, field resolved from TF state - Cross-resource numeric ref: int value stored as number not string - Dashboard etag stored from etags map Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Why
Tests