[FEATURE](schema) Model axle count as a bounded whole number over VehicleSelector#553
Open
Seth Fitzsimmons (sethfitz) wants to merge 3 commits into
Open
[FEATURE](schema) Model axle count as a bounded whole number over VehicleSelector#553Seth Fitzsimmons (sethfitz) wants to merge 3 commits into
Seth Fitzsimmons (sethfitz) wants to merge 3 commits into
Conversation
Seth Fitzsimmons (sethfitz)
added
the
change type - minor 🤏
Seth Fitzsimmons (sethfitz)
temporarily deployed
to
staging
July 10, 2026 23:02 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
🗺️ Schema reference docs preview is live!
Note ♻️ This preview updates automatically with each push to this PR. |
Seth Fitzsimmons (sethfitz)
temporarily deployed
to
staging
July 10, 2026 23:27 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
…ression Recognize pydantic's Field(multiple_of=n) in the codegen and generate a check_multiple_of(col, n) PySpark check that fires when col % n is non-zero. multiple_of=1 is the integral (whole-number) case; testing the remainder in double space matches pydantic for large integral doubles, NaN, and infinity. Wire it through the pipeline: constraint dispatch, the check_multiple_of runtime expression, conformance invalid-value generation (a non-multiple derived from the divisor), and the markdown constraint description. Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
Seth Fitzsimmons (sethfitz)
force-pushed
the
union-type-compat
branch
from
July 20, 2026 18:46
d029544 to
738a058
Compare
Seth Fitzsimmons (sethfitz)
changed the base branch from
pyspark-codegen
to
pyspark-expression-codegen
July 20, 2026 18:46
Seth Fitzsimmons (sethfitz)
changed the title
Seth Fitzsimmons (sethfitz)
temporarily deployed
to
staging
July 20, 2026 18:47 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
…ounds Field(ge=1, le=100) yields separate Ge and Le constraints, each dispatched to its own check_bounds. They describe one range on one column, so they collapse into a single check_bounds(ge=1, le=100): one violation identity rather than two same-name bounds checks a split union arm cannot label distinctly. This also folds a two-sided bound like ConfidenceScore's ge=0.0, le=1.0 into one check instead of two. Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
…Selector VehicleAxleCountSelector.value was uint8; every other VehicleSelector arm is float64. Flattened to one Spark value column, the arms disagreed on type and the codegen widened uint8 to DoubleType -- an implicit compatibility requirement nothing in the model stated. Model value as float64 with multiple_of=1 (whole number) and a [1, 100] bound. The upper bound anchors it to real axle counts and, with the lower bound, rejects the non-finite doubles a bare float64 would admit -- so pydantic and the generated Spark checks agree without a custom constraint class. Extend the codegen to keep same-name union fields whose per-arm constraints diverge (axle count's bounded whole number vs the other arms' ge=0) as separate arm-gated checks rather than collapsing or widening them, require every arm to agree on Spark type (raising otherwise), and fail generation loudly if two same-name checks ever land in one arm with indistinguishable violation labels. Refs #472 Signed-off-by: Seth Fitzsimmons <seth@mojodna.net>
Seth Fitzsimmons (sethfitz)
force-pushed
the
union-type-compat
branch
from
July 20, 2026 20:10
738a058 to
68bae42
Compare
Seth Fitzsimmons (sethfitz)
marked this pull request as ready for review
July 20, 2026 20:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #472. Rebased onto #569 (
pyspark-expression-codegen).Models
VehicleAxleCountSelector.valueas a bounded whole-numberfloat64so the flattenedvaluecolumn across everyVehicleSelectorarm has one Spark type, and teaches the codegen to carry union arms whose same-name fields diverge only in their constraints.valuebecomesfloat64withField(ge=1, le=100, multiple_of=1)(wasuint8).multiple_of=1rejects fractional counts; the[1, 100]bound anchors it to real axle counts and rejects the non-finite doubles a barefloat64would admit, so pydantic and the generated Spark checks agree — no custom constraint class, and native keywords emit standard JSON Schema (minimum/maximum/multipleOf).multiple_ofto a newcheck_multiple_of(col, n)PySpark expression, with conformance coverage and markdown prose.check_bounds(ge, le)(also fixes a latent two-sided-bound case likeConfidenceScore).The generated Segment
valuecolumn was alreadyDoubleTypevia widening; it is nowDoubleTypenatively, with arm-gated bounds and multiple-of checks.Why native
multiple_ofrather than a custom constraintAn earlier iteration added a bespoke
IntegerConstraintclass to express "whole number." That was unnecessary: pydantic's built-inField(multiple_of=1)says exactly that — we simply hadn't been using it. Nativemultiple_ofis the cleaner foundation:Fieldcarries the bounds and the integrality together, so there is no way to order them wrong (the custom class had to sit after the bound or it leaked a non-standardgeinto the JSON Schema instead ofminimum).multiple_of=ntocheck_multiple_of(col, n), not just the integral case.Whole-number semantics that the custom class handled explicitly (rejecting non-finite doubles) are preserved by the field's own
[1, 100]bound, which excludes inf/NaN on both the pydantic and Spark sides.