Skip to content

[SPARK-59091][SQL] Emit the NaN guard in transpiled equality only for floating-point columns - #58901

Open
jzhan-2026 wants to merge 1 commit into
apache:masterfrom
jzhan-2026:spark-59091-nan-guard
Open

jzhan-2026 wants to merge 1 commit into
apache:masterfrom
jzhan-2026:spark-59091-nan-guard

Conversation

@jzhan-2026

@jzhan-2026 jzhan-2026 commented Sep 17, 2026

Copy link
Copy Markdown

What changes were proposed in this pull request?

Split the transpiler's "numeric" category into "integer" (IntegralType) and "float"
(FractionalType, excluding DecimalType). The NaN guard in _lower_eq is now only emitted
when at least one operand has category "float".

  • transpile.py: _category returns "integer" / "float" instead of "numeric" for
    literals and parameters; _param_category_combos tries ["integer", "float", "string"]
    per parameter; _lower_eq gates the guard on lc == "float" or rc == "float";
    _convert_chunk BinOp dispatch uses _is_numeric_cat() for all three sub-categories.
  • ResolveTranspiledPythonUDFOptions.scala: adds "integer" → IntegralType and "float"
    → FractionalType (excluding Decimal) cases to optionMatchesTypes.

Why are the changes needed?

For integer columns (e.g. BIGINT), the old plan for x == y included isnan(cast(x AS DOUBLE)), which is always false—integers can't be NaN. The NaN guard only makes sense for floating-point types.

Does this PR introduce any user-facing change?

No. The generated Catalyst plan for integer-column equality no longer contains dead isnan guards, but the query results are identical.

How was this patch tested?

New Python tests in test_udf_transpile_unit:

  • test_udf_transpile_integer_eq_no_nan_guard — no isnan in plan for BIGINT ==
  • test_udf_transpile_float_eq_nan_semanticsisnan present for DOUBLE ==; NaN and null semantics verified end-to-end
  • test_udf_transpile_int_float_annotation_categories — unit tests for new category names

New Scala tests in ResolveTranspiledPythonUDFOptionsSuite: "integer" / "float" matching, cross-type rejection, DecimalType exclusion, FloatType matching.

Was this patch authored or co-authored using generative AI tooling?

Co-authored by: Claude Sonnet 4.6

@jzhan-2026
jzhan-2026 marked this pull request as ready for review September 17, 2026 23:11
@jzhan-2026

Copy link
Copy Markdown
Author

@holdenk PTAL when you get a chance! Thanks in advance!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant