Describe the bug
get_json_object diverges from Spark when a JSON object contains duplicate keys.
For input {"a":1,"a":2} and path $.a:
- Comet returns
2 (the last occurrence)
- Spark returns
1 (the first occurrence)
Spark's GetJsonObjectEvaluator.evaluatePath (sql/catalyst/.../json/JsonExpressionEvalUtils.scala) stops at the first matching field once dirty is set, so it returns the first occurrence. Comet visits every entry and resolves to the last occurrence, matching serde_json's preserve_order overwrite semantics rather than Spark's.
This is a pre-existing divergence, not a regression. It predates the streaming rewrite in #4907 (the old serde_json + preserve_order path had the same last-wins behavior). It was noted during review of #4907.
Steps to reproduce
SELECT get_json_object('{"a":1,"a":2}', '$.a');
-- Spark: 1
-- Comet: 2
Expected behavior
Match Spark and return the first occurrence of a duplicated key.
Additional context
The fix is on the Rust side in native/spark-expr/src/string_funcs/get_json_object.rs: SegmentVisitor::visit_map should stop updating found once a key has matched (while still consuming the remaining entries to validate the document). The test_duplicate_key_last_wins test added in #4907 would need to be flipped to first-wins.
Describe the bug
get_json_objectdiverges from Spark when a JSON object contains duplicate keys.For input
{"a":1,"a":2}and path$.a:2(the last occurrence)1(the first occurrence)Spark's
GetJsonObjectEvaluator.evaluatePath(sql/catalyst/.../json/JsonExpressionEvalUtils.scala) stops at the first matching field oncedirtyis set, so it returns the first occurrence. Comet visits every entry and resolves to the last occurrence, matchingserde_json'spreserve_orderoverwrite semantics rather than Spark's.This is a pre-existing divergence, not a regression. It predates the streaming rewrite in #4907 (the old
serde_json+preserve_orderpath had the same last-wins behavior). It was noted during review of #4907.Steps to reproduce
Expected behavior
Match Spark and return the first occurrence of a duplicated key.
Additional context
The fix is on the Rust side in
native/spark-expr/src/string_funcs/get_json_object.rs:SegmentVisitor::visit_mapshould stop updatingfoundonce a key has matched (while still consuming the remaining entries to validate the document). Thetest_duplicate_key_last_winstest added in #4907 would need to be flipped to first-wins.