[SPARK-58544][SQL] Fix vector distance and norm functions returning wrong results from intermediate float overflow

[SEPURI-SAI-KRISHNA]

What changes were proposed in this pull request?

VectorFunctionImplUtils accumulates dot products and sums of squares in single precision. Those
intermediate quantities are quadratic in the input values, so they overflow to infinity (or
underflow to zero) long before the final result leaves the float range, and the function returns
NaN, Infinity, NULL or an all-zero vector for inputs whose true result is perfectly
representable as a float.

This PR accumulates the intermediate dot products, sums of squares and absolute sums in double
and rounds only the final result back to float. vector_normalize also keeps the norm in
double instead of rounding it to a float before dividing, so a vector whose norm is outside the
float range is still normalized correctly.

Affected functions: vector_cosine_similarity, vector_inner_product, vector_l2_distance,
vector_norm, vector_normalize.

Why are the changes needed?

Cosine similarity and normalization are scale invariant, so rescaling the input must not change
the result -- but today it does:

Query	Before	After (correct)
vector_cosine_similarity(array(3.0e19F, 4.0e19F), array(3.0e19F, 4.0e19F))	NaN	1.0
vector_cosine_similarity(array(1.0e-23F, 0.0F), array(1.0e-23F, 0.0F))	NULL	1.0
vector_l2_distance(array(3.0e19F, 4.0e19F), array(0.0F, 0.0F))	Infinity	5.0E19
vector_norm(array(3.0e19F, 4.0e19F), 2.0F)	Infinity	5.0E19
vector_normalize(array(3.0e19F, 4.0e19F), 2.0F)	[0.0, 0.0]	[0.6, 0.8]
vector_normalize(array(1.0e-23F, 0.0F), 2.0F)	NULL	[1.0, 0.0]

In every row above the correct result is an ordinary float; only the intermediate sum overflows
((3e19)^2 + (4e19)^2 = 2.5e39 > Float.MAX_VALUE) or underflows ((1e-23)^2 = 1e-46 < the smallest positive float). The wrong values are returned silently, so they propagate into
similarity search results and rankings rather than failing loudly.

Double-precision accumulation is also what other vector similarity implementations do
(for example pgvector), and it removes a rounding error in the common case as well: the existing
vector_cosine_similarity(array(1.0F, 2.0F, 3.0F), array(4.0F, 5.0F, 6.0F)) result changes from
0.9746319 to 0.97463185, which is the correctly rounded float of 32 / sqrt(1078)
(0.9746318461970763...).

These functions were added in 4.2.0 by SPARK-54713 and SPARK-55030, so the wrong results are
present in the 4.2.0 release.

Documented behaviours are unchanged: empty vectors, NULL inputs, vectors containing NULL
elements, and genuinely zero-magnitude vectors return exactly what they returned before.

Does this PR introduce any user-facing change?

Yes. The functions listed above now return correct results instead of NaN, Infinity, NULL
or zeros when an intermediate sum overflows or underflows the float range, as shown in the table
above. This is a change relative to 4.2.0, where these functions were introduced.

vector_cosine_similarity also changes by one ULP for ordinary inputs, because the result is now
correctly rounded (0.9746319 -> 0.97463185 for the example above).

How was this patch tested?

• New golden test cases in sql/core/src/test/resources/sql-tests/inputs/vector-distance.sql and
  vector-norm.sql covering large and small magnitude vectors for all five functions; golden
  files regenerated with SPARK_GENERATE_GOLDEN_FILES=1.
• Existing suites: SQLQueryTestSuite (vector files), MiscFunctionsSuite, ExpressionInfoSuite
  (the vector_cosine_similarity example in @ExpressionDescription is updated to the correctly
  rounded value).

On performance: the unrolled loops read through ArrayData.getFloat() / isNullAt(), and those
accesses dominate the loop, so widening the accumulators is not measurable end to end. Timing the
vector_cosine_similarity loop over 1536-dimensional UnsafeArrayData inputs (200k calls, JIT
warmed up) gives 5834 ns/call before and 5891 ns/call after in one run, and 5870 vs 5792 ns/call in
the next -- i.e. within run-to-run noise. The arithmetic in isolation (plain float[], no
ArrayData indirection) is ~1.3x slower, but it accounts for only ~20% of the loop cost.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5)

[uros-b]

Thank you @SEPURI-SAI-KRISHNA! Adding @MaxGekk to also PTAL