Implement SQL CAST in Java filter predicates via SqlRuntimeCast#741
Open
dileepapeiris wants to merge 28 commits intoapache:mainfrom
Open
Implement SQL CAST in Java filter predicates via SqlRuntimeCast#741dileepapeiris wants to merge 28 commits intoapache:mainfrom
CAST in Java filter predicates via SqlRuntimeCast#741dileepapeiris wants to merge 28 commits intoapache:mainfrom
Conversation
Replace the previous ad-hoc CAST handling in FilterPredicateImpl with a call to SqlRuntimeCast.castValue(input.get(0), returnType). This removes the TODO and the previous fallback logic that relied on widenToDouble/ensureComparable, delegating runtime casting to the centralized SqlRuntimeCast implementation for more accurate casting behavior.
Remove the private sqlCast(Object, SqlTypeName) stub from FilterPredicateImpl which threw UnsupportedOperationException and was not used. Cleans up dead code in the SQL conversion functions.
Add a new placeholder source file SqlRuntimeCast.java under wayang-api-sql's calcite converter functions package. The file currently contains only the Apache ASF v2.0 license header; implementation will be added in a follow-up change.
Add package declaration and necessary imports to SqlRuntimeCast.java (BigDecimal, Calendar, Date, Calcite SqlFunctions/SqlTypeName, DateString, NlsString). Prepares the file for implementing runtime cast logic and resolves missing references for date/number/string handling.
Introduce SqlRuntimeCast utility used for runtime SQL CAST evaluation in Wayang Java filters. Adds a private constructor and a castValue(Object, SqlTypeName) method that unwraps inputs, returns null for SQL NULL, and delegates conversions to SqlFunctions for BOOLEAN, numeric types (TINYINT, SMALLINT, INTEGER, BIGINT, DECIMAL), FLOAT/REAL, DOUBLE, and CHAR/VARCHAR. Unsupported target types throw UnsupportedOperationException.
Introduce a private unwrapForCast(Object) method to normalize inputs before runtime casting. It extracts the underlying String from NlsString and converts Character to String, returning the original object otherwise to ensure Cast operations handle Calcite-specific wrappers and char values correctly.
Introduce a private castToFloat(Object) in SqlRuntimeCast that converts DateString, Date, and Calendar to their millisecond instant as a float, falling back to SqlFunctions.toFloat(v). This enables correct runtime casting of date/time values to float in SQL conversions.
Introduce castToDouble in SqlRuntimeCast to convert DateString, java.util.Date, and Calendar values to milliseconds-as-double, falling back to SqlFunctions.toDouble for other types. This ensures runtime casts of date/time inputs to DOUBLE produce consistent epoch-millisecond values.
Introduce a private helper method castToString(Object) in SqlRuntimeCast to support runtime casting to STRING. The new method handles native String and NlsString instances and delegates Boolean, Float and Double conversions to SqlFunctions.toString. This centralizes string casting logic for the SQL calcite converter.
Add runtime-to-string handling for additional value types in SqlRuntimeCast: BigDecimal (via SqlFunctions.toString), general Number, DateString, Character, and java.util.Date. This ensures values of these types are converted to their string representations during SQL runtime casting to avoid unexpected behavior or missing cases.
Add handling for Calendar instances by converting cal.getTime().toString(), and add a generic fallback return String.valueOf(v). This ensures Calendar values and any other runtime objects produce a String representation, improving robustness of SqlRuntimeCast.
Create new test file wayang-api/wayang-api-sql/src/test/java/org/apache/wayang/api/sql/calcite/converter/functions/SqlRuntimeCastTest.java containing the Apache license header. This is a placeholder/skeleton for future unit tests related to SQL runtime cast conversion in the Calcite converter; no test code implemented yet.
Add package declaration and necessary imports to SqlRuntimeCastTest.java, including JUnit assertions and Test, BigDecimal, and Calcite types (SqlTypeName, NlsString). Prepares the file for implementing unit tests for SQL runtime cast conversions.
Introduce SqlRuntimeCastTest#castNullYieldsNull which asserts that SqlRuntimeCast.castValue(null, SqlTypeName.INTEGER) returns null. This verifies that runtime casting preserves null inputs and prevents NPEs during null handling.
Add two unit tests to SqlRuntimeCastTest verifying runtime casting: one ensures an Integer value is cast to VARCHAR (expecting "1"), and the other ensures a numeric String is cast to INTEGER (expecting 42). These tests cover basic int<->varchar conversion behavior.
Add unit tests verifying SqlRuntimeCast.castValue handles string-to-double and NlsString-to-integer conversions. Adds castStringToDouble (asserts "1.5" -> DOUBLE with delta 1e-9) and castNlsStringToInteger (asserts NlsString("7", "UTF-8", null) -> INTEGER).
Add unit test in SqlRuntimeCastTest to verify that casting the string "TRUE" to SqlTypeName.BOOLEAN returns a Boolean instance and evaluates to true. Ensures runtime cast handles string-to-boolean conversion correctly.
Extend SqlRuntimeCastTest with additional unit tests covering error and formatting behavior: verify invalid boolean input throws RuntimeException, ensure BigDecimal -> VARCHAR conversion uses SQL formatting, and assert that casting to DATE is unsupported (throws UnsupportedOperationException).
Add javaFilterWithCastIntColumnToVarchar unit test in SqlToWayangRelTest. The test builds and executes a plan for "SELECT * FROM fs.exampleInt WHERE CAST(NAMEB AS VARCHAR) = '1'", forces the Java platform for execution, and asserts non-empty results and that the filtered NAMEB field equals 1. This verifies casting an INT column to VARCHAR in a WHERE clause works as expected on the Java platform.
Remove the hard-coded switch statement that mapped SQL target types to Java conversion helpers in SqlRuntimeCast. This deletes per-type cases (BOOLEAN, TINYINT, SMALLINT, INTEGER, BIGINT, DECIMAL, FLOAT/REAL, DOUBLE, CHAR/VARCHAR) and the UnsupportedOperationException for unsupported targets — likely part of a refactor to centralize or replace cast handling.
Add a switch on the target SqlTypeName in SqlRuntimeCast to perform runtime conversions using existing SqlFunctions and helper methods. Implements casting for BOOLEAN, TINYINT, SMALLINT, INTEGER, BIGINT, DECIMAL, FLOAT/REAL, DOUBLE and CHAR/VARCHAR, and throws an UnsupportedOperationException for other types. This enables Java-side evaluation of these CAST operations.
Use a traditional instanceof check plus cast for NlsString in unwrapForCast instead of pattern-matching. This avoids relying on newer Java pattern-matching features (or preview syntax) while preserving the original unwrapping behavior.
Modify SqlRuntimeCast to avoid Java pattern-matching instanceof syntax by using traditional instanceof checks and explicit casts. Updated Character, DateString, and Date branches to use classic casts (and o.toString()) to improve compatibility with Java versions that don't support the instanceof variable form; no functional behavior changes.
Replace pattern-matching instanceof usages with explicit casts in SqlRuntimeCast.java (for DateString, Date and Calendar). This is a syntactic change to avoid newer pattern-matching syntax and improve compatibility/consistency; runtime behavior is unchanged.
Replace Java pattern-matching instanceof usages in SqlRuntimeCast with traditional instanceof checks and explicit casts (e.g. String, NlsString, Boolean cases). This removes preview/pattern-matching syntax and improves compatibility with Java versions that do not support pattern-matching; behavior remains unchanged.
Replace pattern-matching instanceof usages that declared pattern variables (e.g. 'if (v instanceof final Float f)') with traditional instanceof checks and explicit casts for Float, Double and BigDecimal in SqlRuntimeCast.java. No behavioral changes — this improves compatibility/readability by avoiding the newer pattern-variable syntax.
Replace Java pattern-matching instanceof (e.g., 'if (v instanceof final Number n)') with classic instanceof checks and explicit casts or direct v.toString() calls for DateString and Character in SqlRuntimeCast.java. This simplifies the code and improves compatibility with Java versions that do not support pattern-matching instanceof.
Replace pattern-matching instanceof uses for Date and Calendar with traditional instanceof checks and explicit casts. The Date binding (previously 'instanceof final Date d') was removed in favor of v.toString(), and the Calendar case now casts to Calendar before calling getTime().toString(). This simplifies the code and improves compatibility with Java versions/compilers that don't support pattern-variable bindings.
dileepapeiris
changed the title
CAST in Java filter predicates via SqlRuntimeCast (fixes #676)CAST in Java filter predicates via SqlRuntimeCast
Author
|
Hi , I am GSOC applicant for (WAYANG-54) : Make Wayang more datalake-friendly . In here , I have resolved this issue by refactoring the casting logic into a more organized and readable standalone utility. For extended testing, I have provided a dedicated repository: https://github.com/dileepapeiris/fix-for-apache-wayang-api-sql-cast-issue . Refer to the README in that repository for detailed execution steps. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Filter pushdown in wayang-api-sql previously evaluated
CASTwith a heuristic (Number→double, elseensureComparable), which broke valid SQL such asCAST(integer_column AS VARCHAR)(it widened todoubleinstead of producing a string). The privatesqlCastmethod was never invoked and always threwUnsupportedOperationException.Fixes (#676)
What changed
SqlRuntimeCast.castValue(Object, SqlTypeName), delegating to Apache CalciteSqlFunctionsfor boolean, integral, decimal, approximate numeric, and character string targets, with unwraps forNlsString/Characterconsistent with the rest of the filter code.FilterPredicateImplnow uses the Rex call’s result type (SqlTypeNamefromCallTreeFactory) forCAST, matching the actual SQL cast target.sqlCaststub fromFilterPredicateImplto avoid duplicate, misleading APIs: behavior lives in one place (SqlRuntimeCast), is testable, and is documented on the class Javadoc.CAST(NAMEB AS VARCHAR) = '1'on integer CSV data, plus this demo module for a runnable sample and enum-wide coverage.Why remove
sqlCastinFilterPredicateImpland addedSqlRuntimeCastas separate file?How to verify
Option A :
./mvnw -pl wayang-api/wayang-api-sql testExpected Output : -
Option B :
For extended testing, I have provided a dedicated repository: https://github.com/dileepapeiris/fix-for-apache-wayang-api-sql-cast-issue . Refer to the README in that repository for detailed execution steps.