Skip to content

Commit ab9cc08

Browse files
Qiegang Longgengliangwang
authored andcommitted
[SPARK-49635][SQL] Remove ANSI config suggestion in CAST error messages
### What changes were proposed in this pull request? This PR improves CAST error messages to provide suggestions based on whether casts work in ANSI mode, non-ANSI mode, or try_cast(): 1. Suggest try_cast() when the cast is valid but fails in ANSI mode and try_cast() supports it 2. Suggest disabling ANSI mode when the cast works in non-ANSI mode but try_cast() doesn't support it 3. Provide no suggestion when the cast is invalid in all modes ### Why are the changes needed? In Spark 4.0.0, ANSI mode is now enabled by default. The previous error messages suggested users disable ANSI mode when encountering cast failures, which goes against the direction of the project. However, completely removing these suggestions would leave users without guidance when migrating code from Spark 3.x that relied on non-ANSI cast behavior. ### Does this PR introduce _any_ user-facing change? Yes. Error messages for CAST operations that fail in ANSI mode now suggest using `try_cast()` instead of disabling ANSI mode, but only when `try_cast()` actually supports that cast operation. **Before:** ``` cannot cast "ARRAY<INT>" to "ARRAY<BINARY>" with ANSI mode on. If you have to cast ARRAY<INT> to ARRAY<BINARY>, you can set "spark.sql.ansi.enabled" as 'false'. ``` **After (for casts supported by try_cast):** ``` cannot cast "ARRAY<INT>" to "ARRAY<BINARY>". To convert values from "ARRAY<INT>" to "ARRAY<BINARY>", you can use the functions `try_cast` instead. ``` **After (suggests disabling ANSI when try_cast doesn't support it):** ``` cannot cast "TIMESTAMP" to "BOOLEAN" with ANSI mode on. This cast is not allowed in ANSI mode but works when "spark.sql.ansi.enabled" is 'false'. ``` **After (for casts not supported by try_cast):** ``` cannot cast "INT" to "BINARY". ``` ### How was this patch tested? - Added new test case for complex type casts (Array[Int] to Array[Binary]) that demonstrates try_cast suggestion - Added test case for TIMESTAMP → BOOLEAN cast that demonstrates config suggestion - Updated existing test expectations in CastWithAnsiOnSuite to reflect the new behavior - All Cast-related unit tests pass - Verified no regression in legacy (ANSI off) mode - Manually tested with spark-shell to verify error messages ### Was this patch authored or co-authored using generative AI tooling? No Closes #53295 from qlong/SPARK-49635-remove-ansi-suggestion. Authored-by: Qiegang Long <qlong@Qiegangs-MacBook-Pro.local> Signed-off-by: Gengliang Wang <gengliang@apache.org>
1 parent 8e75fc9 commit ab9cc08

File tree

2 files changed

+41
-2
lines changed

2 files changed

+41
-2
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -505,6 +505,10 @@ object Cast extends QueryErrorsBase {
505505
"config" -> toSQLConf(fallbackConf.get._1),
506506
"configVal" -> toSQLValue(fallbackConf.get._2, StringType)))
507507

508+
case _ if fallbackConf.isEmpty && Cast.canTryCast(from, to) =>
509+
// Suggest try_cast for valid casts that fail in ANSI mode
510+
withFunSuggest("try_cast")
511+
508512
case _ =>
509513
DataTypeMismatch(
510514
errorSubClass = "CAST_WITHOUT_SUGGESTION",
@@ -588,8 +592,19 @@ case class Cast(
588592
Some(SQLConf.STORE_ASSIGNMENT_POLICY.key ->
589593
SQLConf.StoreAssignmentPolicy.LEGACY.toString))
590594
} else {
591-
Cast.typeCheckFailureMessage(child.dataType, dataType,
592-
Some(SQLConf.ANSI_ENABLED.key -> "false"))
595+
// Check if there's a config workaround for this cast failure:
596+
// - If canTryCast supports this cast, pass None here and let typeCheckFailureMessage
597+
// suggest try_cast (which is more user-friendly than disabling ANSI mode)
598+
// - If canTryCast doesn't support it BUT the cast works in non-ANSI mode,
599+
// suggest disabling ANSI mode as a migration path
600+
// - Otherwise, pass None and let typeCheckFailureMessage decide
601+
val fallbackConf = if (!Cast.canTryCast(child.dataType, dataType) &&
602+
Cast.canCast(child.dataType, dataType)) {
603+
Some(SQLConf.ANSI_ENABLED.key -> "false")
604+
} else {
605+
None
606+
}
607+
Cast.typeCheckFailureMessage(child.dataType, dataType, fallbackConf)
593608
}
594609
case EvalMode.TRY =>
595610
Cast.typeCheckFailureMessage(child.dataType, dataType, None)

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOnSuite.scala

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -211,6 +211,30 @@ class CastWithAnsiOnSuite extends CastSuiteBase with QueryErrorsBase {
211211
}
212212
}
213213

214+
test("SPARK-49635: suggest try_cast for complex type casts") {
215+
// Array[Int] to Array[Binary]: canTryCast=true (uses canCast), canAnsiCast=false
216+
// Should suggest try_cast, not config
217+
val arrayIntType = ArrayType(IntegerType, containsNull = false)
218+
val arrayBinaryType = ArrayType(BinaryType, containsNull = false)
219+
val arrayIntLiteral = Literal.create(Seq(1, 2, 3), arrayIntType)
220+
221+
val arrayResult = cast(arrayIntLiteral, arrayBinaryType).checkInputDataTypes()
222+
evalMode match {
223+
case EvalMode.ANSI =>
224+
assert(arrayResult ==
225+
DataTypeMismatch(
226+
errorSubClass = "CAST_WITH_FUNC_SUGGESTION",
227+
messageParameters = Map(
228+
"srcType" -> toSQLType(arrayIntType),
229+
"targetType" -> toSQLType(arrayBinaryType),
230+
"functionNames" -> "`try_cast`"
231+
)
232+
)
233+
)
234+
case _ =>
235+
}
236+
}
237+
214238
test("ANSI mode: disallow variant cast to non-nullable types") {
215239
// Array
216240
val variantVal = new VariantVal(Array[Byte](12, 3), Array[Byte](1, 0, 0))

0 commit comments

Comments
 (0)