Skip to content

fix: preserve ANY type through schema round-trip#4995

Merged
kunwp1 merged 12 commits into
apache:mainfrom
Ma77Ball:fix/ArrowUtils
May 13, 2026
Merged

fix: preserve ANY type through schema round-trip#4995
kunwp1 merged 12 commits into
apache:mainfrom
Ma77Ball:fix/ArrowUtils

Conversation

@Ma77Ball
Copy link
Copy Markdown
Contributor

@Ma77Ball Ma77Ball commented May 9, 2026

What changes were proposed in this PR?

ArrowUtils.fromTexeraSchema now tags ANY attributes with texera_type=ANY metadata on the Arrow field, and toTexeraSchema reads that tag back. This mirrors the existing LARGE_BINARY mechanism. Without it, ANY round-trips
silently became STRING because both types share the same Arrow representation (Utf8).

Any related issues, documentation, or discussions?

Closes: #4762

How was this PR tested?

Updated ArrowUtilsSpec (in common/workflow-core): replaced the test that pinned the bug ("lose the ANY distinction") with one that asserts ANY is preserved through a round-trip, and added a test that the texera_type=ANY
metadata is attached only to ANY fields. Ran both WorkflowCore (27/27) and WorkflowOperator (14/14) ArrowUtilsSpec suites — all pass.

Was this PR authored or co-authored using generative AI tooling?

Co-Authored with Claude Opus 4.7 in compliance with ASF

@Ma77Ball Ma77Ball changed the title preserve ANY through schema round-trip via texera_type metadata fix: preserve ANY type through schema round-trip May 9, 2026
@Ma77Ball
Copy link
Copy Markdown
Contributor Author

Ma77Ball commented May 9, 2026

/request-review @aicam

@github-actions github-actions Bot requested a review from aicam May 9, 2026 02:08
@aicam aicam requested a review from kunwp1 May 11, 2026 15:49
Copy link
Copy Markdown
Contributor

@aicam aicam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! @kunwp1 could you please review as well and approve

Comment thread common/workflow-core/src/main/scala/org/apache/texera/amber/util/ArrowUtils.scala Outdated
Copy link
Copy Markdown
Contributor

@kunwp1 kunwp1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented May 11, 2026

Codecov Report

❌ Patch coverage is 25.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 42.78%. Comparing base (6b93b12) to head (4692507).

Files with missing lines Patch % Lines
...cala/org/apache/texera/amber/util/ArrowUtils.scala 25.00% 0 Missing and 3 partials ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##               main    #4995   +/-   ##
=========================================
  Coverage     42.78%   42.78%           
- Complexity     2194     2198    +4     
=========================================
  Files          1045     1045           
  Lines         39985    39989    +4     
  Branches       4217     4220    +3     
=========================================
+ Hits          17106    17110    +4     
  Misses        21819    21819           
  Partials       1060     1060           
Flag Coverage Δ *Carryforward flag
access-control-service 39.53% <ø> (ø)
agent-service 33.72% <ø> (ø) Carriedforward from 7879c2a
amber 43.50% <25.00%> (+0.01%) ⬆️
computing-unit-managing-service 0.00% <ø> (ø)
config-service 0.00% <ø> (ø)
file-service 32.18% <ø> (ø)
frontend 34.04% <ø> (ø) Carriedforward from 7879c2a
python 88.90% <ø> (ø) Carriedforward from 7879c2a
workflow-compiling-service 47.72% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@kunwp1 kunwp1 enabled auto-merge (squash) May 13, 2026 03:12
@kunwp1 kunwp1 merged commit ed2f775 into apache:main May 13, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ArrowUtils round-trip loses AttributeType.ANY (becomes STRING)

4 participants