Skip to content

Native implementation of get_json_object returns last value for duplicate keys, Spark returns first #4947

Description

@andygrove

Describe the bug

get_json_object diverges from Spark when a JSON object contains duplicate keys.

For input {"a":1,"a":2} and path $.a:

  • Comet returns 2 (the last occurrence)
  • Spark returns 1 (the first occurrence)

Spark's GetJsonObjectEvaluator.evaluatePath (sql/catalyst/.../json/JsonExpressionEvalUtils.scala) stops at the first matching field once dirty is set, so it returns the first occurrence. Comet visits every entry and resolves to the last occurrence, matching serde_json's preserve_order overwrite semantics rather than Spark's.

This is a pre-existing divergence, not a regression. It predates the streaming rewrite in #4907 (the old serde_json + preserve_order path had the same last-wins behavior). It was noted during review of #4907.

Steps to reproduce

SELECT get_json_object('{"a":1,"a":2}', '$.a');
-- Spark: 1
-- Comet: 2

Expected behavior

Match Spark and return the first occurrence of a duplicated key.

Additional context

The fix is on the Rust side in native/spark-expr/src/string_funcs/get_json_object.rs: SegmentVisitor::visit_map should stop updating found once a key has matched (while still consuming the remaining entries to validate the document). The test_duplicate_key_last_wins test added in #4907 would need to be flipped to first-wins.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions