Skip to content

[AURON #2130] Implement native function of weekofyear#2131

Merged
slfan1989 merged 1 commit intoapache:masterfrom
weimingdiit:feat/weekofyear-native-function
Apr 5, 2026
Merged

[AURON #2130] Implement native function of weekofyear#2131
slfan1989 merged 1 commit intoapache:masterfrom
weimingdiit:feat/weekofyear-native-function

Conversation

@weimingdiit
Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes ##2130

Rationale for this change

To improve compatibility with Spark SQL date functions, we should implement weekofyear() with Spark-compatible semantics.

Expected behavior

Function name: weekofyear(expr)

Return type: INT

Week semantics:

  • A week starts on Monday
  • Week 1 is the first week of the year with more than 3 days
  • This matches Spark’s ISO-style week numbering behavior

Examples:

  • weekofyear('2009-07-30') -> 31
  • weekofyear('2016-01-01') -> 53
  • weekofyear('2017-01-01') -> 52

Supports: DATE, TIMESTAMP, and compatible string/date inputs consistent with existing date extraction functions

Additional expectations:

  • Null-safe: returns NULL if input is NULL
  • Array and scalar inputs: consistent with existing native date extraction functions
  • Cross-year boundary behavior should match Spark semantics exactly

What changes are included in this PR?

This PR adds native support for the weekofyear() function with Spark-compatible semantics.
The following changes are included:

  • Added native implementation of spark_weekofyear() in the expression layer
  • Added WeekOfYear expression support in NativeConverters for proper Spark -> native translation
  • Registered Spark_WeekOfYear in native function dispatch
  • Added unit tests to verify correctness for:
    • normal date inputs
    • cross-year boundary cases
    • Spark-compatible ISO week numbering semantics
    • null input handling

Are there any user-facing changes?

NO.

How was this patch tested?

  • Added and ran targeted Rust unit tests for spark_weekofyear()
  • Verified expected results for representative Spark-compatible cases such as:
    • weekofyear('2009-07-30') = 31
    • weekofyear('2016-01-01') = 53
    • weekofyear('2017-01-01') = 52

@weimingdiit weimingdiit marked this pull request as ready for review March 28, 2026 17:40
@slfan1989 slfan1989 self-requested a review March 29, 2026 03:18
@slfan1989 slfan1989 self-assigned this Mar 29, 2026
@slfan1989 slfan1989 requested a review from Copilot March 30, 2026 01:38
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Implements native support for Spark-compatible weekofyear() to reduce Spark fallbacks and improve Spark SQL function parity in Auron’s native execution path.

Changes:

  • Add Spark expression translation for WeekOfYear to a native ext scalar function (Spark_WeekOfYear).
  • Implement spark_weekofyear() in the native Rust function layer and register it in the ext-function dispatcher.
  • Add Spark-side integration coverage plus Rust unit test coverage for core ISO week semantics and null propagation.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
spark-extension/src/main/scala/org/apache/spark/sql/auron/NativeConverters.scala Adds Spark WeekOfYear expression mapping to the native ext function Spark_WeekOfYear.
spark-extension-shims-spark/src/test/scala/org/apache/auron/AuronFunctionSuite.scala Adds an integration test exercising weekofyear() via checkSparkAnswerAndOperator.
native-engine/datafusion-ext-functions/src/spark_dates.rs Adds spark_weekofyear() implementation (ISO week) plus a unit test.
native-engine/datafusion-ext-functions/src/lib.rs Registers Spark_WeekOfYear in the native ext function dispatcher.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@weimingdiit weimingdiit marked this pull request as draft March 30, 2026 13:22
@weimingdiit weimingdiit force-pushed the feat/weekofyear-native-function branch 2 times, most recently from edd72ee to 54bd43c Compare April 1, 2026 03:40
@weimingdiit weimingdiit marked this pull request as ready for review April 1, 2026 03:42
Copy link
Copy Markdown
Contributor

@ShreyeshArangath ShreyeshArangath left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Signed-off-by: weimingdiit <weimingdiit@gmail.com>
@weimingdiit weimingdiit force-pushed the feat/weekofyear-native-function branch from b0855a5 to eefcef5 Compare April 5, 2026 02:15
@slfan1989 slfan1989 merged commit dd242db into apache:master Apr 5, 2026
123 checks passed
@slfan1989
Copy link
Copy Markdown
Contributor

@weimingdiit Thanks for the contribution! @ShreyeshArangath @yew1eb Thanks for the review!

@weimingdiit
Copy link
Copy Markdown
Contributor Author

@slfan1989 Thanks for your merge! @ShreyeshArangath @yew1eb Thanks for the review!

@weimingdiit weimingdiit deleted the feat/weekofyear-native-function branch April 5, 2026 04:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants