acceptance: schema-driven invariant fuzzing for bundle configs#5686
Open
radakam wants to merge 54 commits into
Open
acceptance: schema-driven invariant fuzzing for bundle configs#5686radakam wants to merge 54 commits into
radakam wants to merge 54 commits into
Conversation
Collaborator
Integration test reportCommit: c66e0ef
8 interesting tests: 4 RECOVERED, 4 SKIP
Top 11 slowest tests (at least 2 minutes):
|
Contributor
Approval status: pending
|
denik
reviewed
Jun 26, 2026
Add common.sh with invariant_deploy and _invariant_cleanup so the deploy-based bodies (no_drift, redeploy, update, destroy_recreate) and migrate no longer duplicate the validate/deploy/cleanup prologue. Unify the panic check on '!panic:' across all invariant bodies and fuzz/script so a random generated token containing "panic" can't be a false positive. Also fold in the supporting helpers: check_schema_types.py (fail loud on a schema type the generator can't produce), gen_fuzz_config_check.py plus its selftest (to_yaml contract), util.load_plan shared by the verify_* scripts, and a pass to shorten comments across the harness.
The comment describes connection_name, managed_encryption_settings and custom_max_retention_hours, which are the fields the fix added. It sat above StorageRoot/ProviderName/ShareName, which were already round-tripped. Move it directly above the three added fields so it annotates them.
Replace the shared `<name>.sh` invariant bodies (sourced via FUZZ_INVARIANT and common.sh) with self-contained invariant test directories that double as fuzz targets, selected by FUZZ_TARGET. Each target runs over the curated INPUT_CONFIG matrix and, when FUZZ_SEED is set, against a schema-generated random config. - Inline the deploy/drift logic into no_drift/script and migrate/script - Re-add redeploy, canonical, update, and destroy_recreate as invariant dirs - Drop common.sh and the standalone *.sh bodies - Point fuzz/script at ../$FUZZ_TARGET/script and refresh the README
The origin/main merge added the volume_path_job_ref.yml.tmpl fuzz template to the INPUT_CONFIG matrix, but the canonical/destroy_recreate/redeploy/update out.test.toml goldens were not regenerated, failing the "changed or new files" CI guard.
The generator produced random catalog_name/schema_name values and random or
empty grants. The fake test server accepts them, but real UC rejects them
(CATALOG_DOES_NOT_EXIST, invalid principal/privilege), so such configs deploy
locally yet drift or fail on cloud, masking real invariant coverage.
Pin catalog_name to "main" and schema_name to "default" (the seeded objects
used by the curated invariant configs), and emit one known-good grant per
grant-bearing securable type ("account users" plus a privilege valid for that
type). This removes the spurious grants drift and the reference-rejection class.
Extend the invariant fuzzer to emit more than one resource per config
(--resource-count, matrixed as FUZZ_RESOURCE_COUNT) and link two of them
with a ${resources.*} reference so the interpolation and deploy-ordering
paths are exercised. The reference targets an input identity field
(name/display_name) so it resolves for every resource type and converges
without drift.
Also improve generated-config validity: skip output-only/computed fields
(x-databricks-field-behaviors OUTPUT_ONLY, readOnly, and a name list) to
avoid false drift after migrate, emit valid permissions per resource type,
force prevent_destroy=false so destroy_recreate can run, and add required
fields the schema omits for registered_models.
Gate the resource key/name suffix on the resource index, not the total count, so a given seed produces the same first resource whether --resource-count is 1 or greater. Later resources stay index-suffixed to remain unique.
inject_cross_ref stopped after a single ${resources.*} edge, so a config
with N resources still exercised only one reference. Link each resource to
an earlier one instead, keeping the graph acyclic so deploy can order it,
and add resource-count 3 to the matrix so the multi-link path runs in CI.
test.toml already lists resource count 3; regenerate the recorded matrix to match so the acceptance framework does not flag a mismatch.
Point file_path/source_code_path fields at the staged data/. fixtures and pin typed-string fields so generated configs deploy instead of getting rejected. Also ignore the local .fuzztmp/ driver scratch directory.
Add a mutate engine (mutate_fuzz_config.py) that applies seeded delete/set/dangerous-value mutations to a curated base config, and a fuzz_gen_config.py dispatcher selected by FUZZ_MODE (generate|mutate). Wire the dispatcher into the six invariant target scripts, matrix FUZZ_MODE in fuzz/test.toml (excluding redundant mutate runs at resource-count 2/3), and add a selftest covering the YAML loader round-trip and mutation determinism.
The schema-walk generator pinned every scalar to a known-good value, so dangerous / near-range-end input was only exercised by mutate mode. Inject DANGEROUS_STRINGS/DANGEROUS_INTS into free-form scalars (~15% of the time) so generate mode probes empty/whitespace/over-long/control-char strings and int32/int64 boundaries while pinned fields keep values that still deploy. Centralize the two lists in gen_fuzz_config.py so mutate_fuzz_config.py reuses them unchanged.
The update invariant edits a comment/description and asserts an in-place update, but some resources classify that field as recreate_on_changes (e.g. model_serving_endpoints.description). A fuzz-generated config that set such a field made the correct recreate look like an invariant bug. edit_fuzz_config.py now reads recreate_on_changes from resources.yml, tracks the enclosing resource type while scanning, and skips immutable comment/description fields (picking a mutable one or reporting none). Adds a contract self-check (edit_fuzz_config_check.py) and a selftest.
The schema fuzzer generates resource types the testserver may not model. Hitting an unregistered route made the testserver call t.Errorf, failing the whole fuzz run for a coverage gap rather than a CLI bug. Add a testserver IgnoreUnhandledRequests option (exposed as a test.toml field, plumbed through startLocalServer) that returns 501 and logs the gap instead of failing. The invariant fuzz script already treats a non-zero result before INPUT_CONFIG_OK as a rejection, so such configs are now skipped cleanly. Curated tests leave the flag false so genuine missing-handler bugs stay loud. Covered by a testserver unit test.
Fuzz variants had two time failure modes that both read as bugs: a single seed could hang indefinitely, and a slow-but-progressing variant (mutate + drift at ~30s/seed) could exceed the per-test timeout across a large seed count. Both surfaced only as an opaque "test timed out". Add a per-seed cap (FUZZ_SEED_TIMEOUT, default 180s) enforced via GNU timeout: a seed past the cap is SIGQUIT'd (Go dumps goroutines, so a real deadlock is diagnosable) then SIGKILL'd, and reported as a hang with a reproduce hint, distinct from a drift bug. Where timeout is unavailable (macOS/Windows) seeds run uncapped as before, and FUZZ_SEED_TIMEOUT=0 disables the cap for live inspection. Add an optional overall budget (FUZZ_TIME_BUDGET): the loop stops launching new seeds past it and exits cleanly, so a slow variant tests as many seeds as fit instead of being force-killed. The nightly task sets it under a raised per-variant Timeout; the committed run keeps its defaults and empty output.
Triaging a fuzz run meant hand-parsing every seed's LOG.* across preserved temp dirs to sort deployed / rejected / testserver-gap / hang / invariant bug. Reuse the classification the loop already computes and append one machine-readable line per seed to LOG.summary, plus a per-variant totals block on a clean run. A 501 "No stub found" is tracked as a coverage gap distinct from a genuine config rejection. LOG.summary is a file, not stdout, so the committed run's empty-output assertion is unaffected.
Trim the multi-line explanations added across the fuzz harness changes to short, why-focused notes; no behavior change.
…n config The parent test.toml sets GOOSOnPR.windows/darwin=false and adds job_run.yml.tmpl, but the per-directory out.test.toml snapshots were stale, failing the CI "detected changed or new files" check.
Four improvements to the bundle schema fuzzer: - Oracle: in fuzz mode (SKIP_DRIFT_CHECK) no_drift/redeploy now assert plan determinism (two consecutive `plan` reads must be byte-identical) instead of only no-panic. This is independent of fake-server fidelity, unlike no-drift or no-destructive-drift (an unchanged config's recreate is just a local-vs-remote representation mismatch on an immutable field). redeploy also panic-scans its second deploy while tolerating fidelity-driven redeploy failures. - Coverage: add `task test-fuzz-cover` to run the corpus under -cover and report per-package coverage (0.0% = never exercised), plus opt-in FUZZ_CORPUS_DIR to persist deployable configs as a generator-independent regression corpus. - Rejection waste: emit non-ASCII scalars as literal UTF-8 (ensure_ascii=False) so hostile values reach bundle logic instead of dying at the YAML parser as an invalid surrogate-pair escape (was the single largest rejection bucket). - Truncation: FUZZ_TIME_BUDGET now defaults on (900s, under the 20m Timeout) so a slow variant stops cleanly and passes rather than being force-killed into a false failure; FUZZ_TIME_BUDGET=0 restores an uncapped run.
Add a schema-aware additive op to mutate_fuzz_config: alongside deleting and perturbing existing fields, it now injects a valid optional field the curated base omits, valued by the schema generator. Destructive mutations stay within the base's field set (only reject/panic bugs); adding a valid optional field to a still-deploying config is what reaches reconcile/drift bugs. The no-schema path is unchanged (RNG stream and selftest golden preserved).
… fix, gitignore - Remove the orphaned edit_fuzz_config.py chain (edit script, its check, and selftest); the update invariant it served was removed earlier in this branch. Decouple gen_fuzz_config_check.py from it (the one-line scalar contract is now attributed to mutate_fuzz_config's loader, which actually relies on it). - Revert the libs/testserver/catalogs.go create-payload round-trip; it is a standalone fake-server fidelity change unrelated to the fuzz harness. - Drop the .gitignore /build/ and .fuzztmp/ entries (stale/misleading paths). - Regenerate fuzz/out.test.toml (drops stale GOOSOnPR lines).
no_drift's post-deploy plan already dry-runs what a redeploy would apply, so
it catches the same field-level non-idempotency without the cost of a second
deploy; the only surface redeploy adds (plan/apply divergence) never surfaced
a bug across the nightly runs. Multi-resource generation only added
cross-resource reference injection, which likewise found no distinct defect
(every bug reproduces at a single resource) while lowering acceptance.
Remove both, shrinking the fuzz matrix from 12 leaf variants to 4
({no_drift, migrate} x {generate, mutate}). Regenerate out.test.toml and
update the README accordingly.
# Conflicts: # acceptance/bundle/invariant/no_drift/script
…inism diff The deploy_rc capture in no_drift/script and migrate/script was dead under `bash -euo pipefail`: a failing deploy aborted at the `trace` line before the capture, so the panic check never ran on a failed deploy. Wrap the deploy in `set +e`/`set -e` so the code is captured, the panic check runs even on failure (a panicking-but-rejected config is a bug, not a rejection), and only then do we exit with the deploy's code. Gate the pre-deploy `bundle validate` on FUZZ_SEED: deploy runs the same validate pipeline, so it is redundant for curated configs and only added value as an isolated panic surface for fuzzed configs. This also restores the curated tests to their prior behavior. Guard the fuzz-mode plan-determinism oracle to diff only when both plans succeed: a plan that fails on an unstubbed read is a coverage gap, and its partial output can differ run-to-run, which would false-positive. Reword the load_plan comment to drop the inaccurate fuzzer reference (only the curated drift check reaches it).
Rename fuzz_gen_config.py to emit_fuzz_config.py so it no longer collides with the schema-walk generator gen_fuzz_config.py. Extract the shared config-render, cleanup, and deploy-capture prologue out of the no_drift and migrate scripts into prologue.sh. Shorten the comments added in this PR.
The rename to emit_fuzz_config sorts before envsubst, so ruff I001 flagged the import block in mutate_fuzz_config_check.py.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an acceptance-test fuzzing harness that stresses the bundle engine by generating random configs and asserting deploy invariants. Because the schema comes from the CLI under test, a failure is a real bug: panic, internal error, or drift.
What it does
acceptance/bin/gen_fuzz_config.py): builds a random resource by walking the livedatabricks bundle schema. Pins reference/typed fields (catalog, schema,parent_path, file/notebook paths, durations) and skips output-only/computed fields to keep configs deployable, and ~15% of free-form scalars get a dangerous / near-range-end value (empty, whitespace, over-long, control chars, non-ASCII, path traversal, int32/int64 boundaries).acceptance/bin/mutate_fuzz_config.py): starts from a curated invariant config and applies 1-3 seeded mutations — destructive (delete a field, set a fuzz token, set a dangerous value, or empty a container) plus, with the schema, additive (inject a valid optional field the base omits). Dispatched byacceptance/bin/emit_fuzz_config.pyviaFUZZ_MODE.acceptance/bundle/invariant/prologue.sh(config render, destroy-on-exit trap, and deploy-capture helper):no_drift— deploy, then assert the post-deploy plan has no actions. In fuzz mode, where fake-server fidelity makes exact drift unreliable, it instead asserts plan determinism (two consecutive plans byte-identical).migrate— Terraform deploy → migrate to direct → assert no drift. In fuzz mode it asserts only that no command panics.acceptance/bundle/invariant/fuzz/script): classifies each seed as deployed / rejected / gap / bug / hang, separating genuine bugs from config rejections and testserver coverage gaps. Adds a per-seed cap (FUZZ_SEED_TIMEOUT, SIGQUIT→SIGKILL so a hang dumps goroutines), an overall budget with clean truncation (FUZZ_TIME_BUDGET), an optional replayable corpus (FUZZ_CORPUS_DIR), and a per-seed/per-variant tally inLOG.summary.The matrix is
{no_drift, migrate} × {generate, mutate}— 4 leaf variants.Supporting changes
Taskfile.yml:task test-fuzz(wider window, drift on) andtask test-fuzz-cover(runs the corpus under-cover, reports per-package CLI coverage)..github/workflows/push.yml: nightly (cron) fuzz job over a rotating, non-overlapping seed window (fromGITHUB_RUN_NUMBER) that comments failures on the PR that introduced the commit, with a single-seed repro.libs/testserver: newIgnoreUnhandledRequestsoption (atest.tomlfield, plumbed throughstartLocalServer) that returns 501 and logs unmodeled routes as coverage gaps instead of failing the run. Curated tests leave it off so real missing-handler bugs stay loud.check_schema_types.py(fails loudly on a schema type the generator can't produce),gen_fuzz_config_check.py/mutate_fuzz_config_check.py(+ acceptance selftests), a sharedutil.load_plan, andverify_no_drift.pynow failing cleanly on empty/unparseable plan output.Reproducing a failure
Bugs found by the fuzzer