[ENHANCEMENT](pyspark) Generalize field_path to arbitrary map/array nesting#572
Open
Seth Fitzsimmons (sethfitz) wants to merge 3 commits into
Open
[ENHANCEMENT](pyspark) Generalize field_path to arbitrary map/array nesting#572Seth Fitzsimmons (sethfitz) wants to merge 3 commits into
Seth Fitzsimmons (sethfitz) wants to merge 3 commits into
Conversation
Seth Fitzsimmons (sethfitz)
temporarily deployed
to
staging
July 18, 2026 03:28 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
🗺️ Schema reference docs preview is live!
Note ♻️ This preview updates automatically with each push to this PR. |
Seth Fitzsimmons (sethfitz)
temporarily deployed
to
staging
July 18, 2026 04:18 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Seth Fitzsimmons (sethfitz)
temporarily deployed
to
staging
July 18, 2026 05:38 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Seth Fitzsimmons (sethfitz)
force-pushed
the
generalize-field-path
branch
from
July 20, 2026 20:23
fdefd39 to
9ac022f
Compare
Seth Fitzsimmons (sethfitz)
temporarily deployed
to
staging
July 20, 2026 20:24 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Collapse field_path's ScalarPath/ArrayPath/MapPath taxonomy into Direct
+ Iterated and extend the grammar to interleaved map/array markers. A
container nested directly inside another with no field name between
(list[list[X]], dict[K, list[X]], list[dict], nested maps, a map reached
through an array) becomes an anonymous iterating segment, so every such
shape is now representable and renders, validates, and mutates end to
end.
Replace the four array/map wrap functions with one _wrap_in_iteration
fold over RenderFrames, and add the flattening runtime pair
nested_map_{keys,values}_check. Remove the union-under-multi-list and
newtype-under-list guards -- those shapes render correctly now -- and
add a guard for a union variant field reached through iteration past its
discriminator element, where the ElementGuard would gate the wrong
element.
The generated-conformance-test path handles the new shapes: set_at_path
learns the {value}/{key} grammar and descends trailing containers; model
mutations gain a composite element_path for map-then-array and
array-then-map nesting; shapes with no in-place mutation are loud
capability gates, not silent misroutes.
Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
A model-level constraint declared on a submodel reached through a plain
struct field -- @require_any_of on Details where Feature.details:
Details -- now anchors at the struct prefix instead of collapsing to the
row root. _model_constraint_target is the identity on the prefix; the
renderer qualifies every field and condition reference with that prefix
(F.col("details.foo")) and gates an optional struct on presence
(F.when(F.col("details").isNotNull(), ...)). ModelCheck.read_columns
reads the single top-level struct column.
_guard_struct_nested_anchor narrows to struct-nested discriminated
unions only: their variant ColumnGuards render the discriminator as a
top-level column, which a struct-qualified path cannot express. Plain
struct-nested model constraints are now supported and no longer raise.
Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
Rewrite the Check Builder section of the codegen design doc for the current Direct/Iterated FieldPath taxonomy: anonymous segments, the single _wrap_in_iteration fold, and the mixed map/array nesting cases (dict[K, list], list[dict], nested maps, map-in-array), replacing the retired ScalarPath/ArrayPath/MapPath description. Disambiguate iter_frames (named-only structural folding) from the renderer's per-segment _render_frames walk. Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
Seth Fitzsimmons (sethfitz)
force-pushed
the
generalize-field-path
branch
from
July 20, 2026 20:57
9ac022f to
0d507b8
Compare
Seth Fitzsimmons (sethfitz)
marked this pull request as ready for review
July 20, 2026 21:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #570 — generalize
field_pathto arbitrary map/array nesting.What changed
Collapses
field_path's three-class taxonomy (ScalarPath/ArrayPath/MapPath) intoDirect+Iterated, extends the grammar to interleaved map/array markers, and replaces the four array/map wrap functions with one_wrap_in_iterationfold overRenderFrames. A container nested directly inside another with no field name between them is an anonymous iterating segment, solist[list[X]],dict[K, list],list[dict], nested maps, and maps reached through arrays are now representable and render working validation end-to-end. Adds the flattening runtime pairnested_map_{keys,values}_check.Notes
The generated-conformance-test path now handles mixed map/array shapes: a field check on a map value/key reached through array iteration (
items[].tags{value}) descends the array and mutates the right map side, ground-truthed against real Spark. The remaining mixed shapes with no in-place mutation (subs{value}[],items[].configs{value}) are loud capability gates, not silent misroutes.