Skip to content

fix: from_unixtime should respect datafusion.execution.time_zone - #25161

Open
adriangb wants to merge 6 commits into
apache:mainfrom
pydantic:fix-from-unixtime-session-timezone
Open

fix: from_unixtime should respect datafusion.execution.time_zone#25161
adriangb wants to merge 6 commits into
apache:mainfrom
pydantic:fix-from-unixtime-session-timezone

Conversation

@adriangb

@adriangb adriangb commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

The bug, in SQL

Set a session time zone. Every other date and time function honours it. from_unixtime does not.

SET datafusion.execution.time_zone = 'America/Denver';

-- now() and to_timestamp() honour the session time zone
SELECT arrow_typeof(now()), arrow_typeof(to_timestamp(1704110400));
-- Timestamp(ns, "America/Denver") | Timestamp(ns, "America/Denver")

-- from_unixtime() does not
SELECT arrow_typeof(from_unixtime(1704110400)), from_unixtime(1704110400);
-- Timestamp(s)                    | 2024-01-01T12:00:00

The last row is the defect. The user asks for a Denver session, but the value comes back as a bare wall clock with no zone. 2024-01-01T12:00:00 is the UTC clock, not the Denver clock.

After this PR the same query returns the session time zone:

SET datafusion.execution.time_zone = 'America/Denver';

SELECT arrow_typeof(from_unixtime(1704110400)), from_unixtime(1704110400);
-- Timestamp(s, "America/Denver") | 2024-01-01T05:00:00-07:00

The instant is the same. Only the declared type and the rendered offset change.

What changes for a user

  • from_unixtime(expr) returns a value in datafusion.execution.time_zone.
  • from_unixtime(expr, 'tz') is unchanged. An explicit argument still wins.
  • With the session time zone unset, which is the default, nothing changes at all.
  • A value that no time zone can render is now an error, not a process panic.

Field research: PostgreSQL and DuckDB

Neither engine has a function named from_unixtime. The closest equivalent in both is to_timestamp(<epoch>), so that is the comparison I ran. Both engines are timezone-aware there, and both render in the session time zone.

PostgreSQL 17.11

SHOW TimeZone;                                  -- Etc/UTC
SELECT to_timestamp(1704110400),
       pg_typeof(to_timestamp(1704110400));
-- 2024-01-01 12:00:00+00 | timestamp with time zone

SET TimeZone = 'America/Denver';
SELECT to_timestamp(1704110400);                -- 2024-01-01 05:00:00-07
SELECT extract(epoch from to_timestamp(1704110400));   -- 1704110400.000000

PostgreSQL has no timezone-naive form of to_timestamp. A user must ask for one:

SELECT to_timestamp(1704110400) AT TIME ZONE 'UTC';
-- 2024-01-01 12:00:00 | timestamp without time zone

DuckDB v1.5.2

SELECT current_setting('TimeZone'), to_timestamp(1704110400),
       typeof(to_timestamp(1704110400));
-- America/Chicago | 2024-01-01 06:00:00-06 | TIMESTAMP WITH TIME ZONE

SET TimeZone = 'America/Denver';
SELECT to_timestamp(1704110400);                -- 2024-01-01 05:00:00-07
SELECT epoch(to_timestamp(1704110400));         -- 1704110400.0

DuckDB defaults its session time zone to the operating system zone, so it is never naive. The naive form is make_timestamp(<microseconds>), which returns TIMESTAMP.

What the comparison supports, and what it does not

  • The type and the render rule of this PR agree with both engines. A one-argument epoch conversion is timezone-aware and follows the session time zone.
  • The default differs, and this PR keeps DataFusion's default. DataFusion leaves the session time zone unset, so from_unixtime stays naive out of the box. PostgreSQL and DuckDB always have a session zone.
  • The fixed-offset string form is not a meaningful comparison against PostgreSQL. SET TimeZone = '+08:00' in PostgreSQL means UTC-08:00, because PostgreSQL reads the string POSIX-style. DataFusion reads '+08:00' as UTC+08:00. That divergence is separate and is tracked in AT TIME ZONE '+05:30' uses the opposite sign convention from PostgreSQL #25170.
  • The out-of-range comparison is not meaningful either. The three engines have three different timestamp ranges. PostgreSQL raises ERROR: timestamp out of range at to_timestamp(-8334601211039). DuckDB renders the same value as 262145-12-31 (BC) 23:59:59.000448-04:56. DataFusion sits between the two, and this PR makes it raise an error rather than panic.

What changes are included in this PR?

The fix

  • FromUnixtimeFunc carries an Option<Arc<str>> timezone taken from config.execution.time_zone, and implements ScalarUDFImpl::with_updated_config. NowFunc and the to_timestamp* functions already work this way. The single-argument form reports and produces Timestamp(Second, <session tz>) from both return_field_from_args and invoke_with_args.
  • from_unixtime moves to make_udf_function_with_config!, and its expr_fn gains the @config marker. The session ConfigOptions now reach the function at registration time, and again on every SET or RESET.
  • FromUnixtimeFunc::new() is deprecated since 56.0.0 in favour of FromUnixtimeFunc::new_with_config(). This mirrors the to_timestamp* constructors. Default still yields the previous timezone-naive behaviour.
  • The from_unixtime documentation description was wrong before this PR. It claimed an RFC3339 string with nanosecond precision and a Z suffix, but the function returns an Arrow Timestamp(Second, ...), and the example directly below it shows a -04:00 offset. The description now states what the function returns, and docs/source/user-guide/sql/scalar_functions.md is regenerated with dev/update_function_docs.sh.

The guard

Arrow renders a Timestamp(Second, Some(tz)) through chrono's DateTime::naive_local, which panics when the shifted value leaves NaiveDateTime's range. The panic fires at render time, far from from_unixtime. The two-argument form can always reach it. Once the one-argument form applies a session zone, it can reach it too.

So invoke_with_args now checks the bound and returns a DataFusionError:

Execution error: Cannot convert -8334601211039 to a timestamp in timezone "America/New_York"
for function from_unixtime: the local date and time is outside the supported range

The check has three parts:

  1. A timezone-naive result is never shifted, so the check returns at once. The full NaiveDateTime range stays usable, exactly as on main.
  2. A fast path reads only the array's min and max. Inside [NaiveDateTime::MIN + 86400, NaiveDateTime::MAX - 86400] no zone offset can push a value out of range, so no per-value work happens.
  3. Outside that window the check resolves the zone offset per value.

86400 is a deliberate over-bound. The largest offset in the whole IANA database is Asia/Manila at -15:56:08 local mean time, and the largest positive one is America/Metlakatla at +15:13:42. The largest fixed offset Arrow's Tz parser accepts is +23:59, which is 86340 seconds. +24:00 is rejected by the parser.

What is the testing strategy for this PR?

New sqllogictest file datafusion/sqllogictest/test_files/from_unixtime_timezone.slt, modelled on to_timestamp_timezone.slt. It asserts both arrow_typeof and the value for:

  • the session time zone unset,
  • a fixed offset (+08:00),
  • a named IANA zone (America/Denver),
  • the explicit two-argument form under a set session time zone, which must ignore it,
  • array (not constant folded) input,
  • NULL input,
  • equality of one instant across two time zones,
  • RESET, which must restore the timezone-naive behaviour.

The to_unixtime / from_unixtime round trip from the issue is in the file. So are the out-of-range bounds, in both directions, for a named zone and a fixed offset zone, for the one-argument and the two-argument forms, and for array input.

New unit tests in datafusion/functions/src/datetime/from_unixtime.rs cover return_field_from_args and invoke_with_args with a session time zone set, an explicit time zone that overrides it, and the out-of-range bound.

Every bounds test renders its result. The unit tests go through arrow::util::display::ArrayFormatter, and the sqllogictest cases use query P. This is deliberate: the panic they guard against fires when the value is formatted, not when it is produced, so a test that only computes the value would pass either way.

Verified locally

  • cargo test -p datafusion-sqllogictest --test sqllogictests — 506 files pass
  • cargo test -p datafusion-functions --lib datetime::from_unixtime — 10 tests pass
  • cargo clippy -p datafusion-functions -p datafusion-sql --all-targets -- -D warnings — clean
  • cargo fmt --all --check and ./ci/scripts/doc_prettier_check.sh — clean

Verified against a main build. I built main at the merge base and this branch as two datafusion-cli binaries and diffed their output. With the session time zone unset, eleven probes are byte-identical, including the error paths. The two-argument form is byte-identical across 140 pairs of value and zone. The full detail is in a separate comment on this PR.

Cost of the guard. SELECT count(from_unixtime(v)) FROM generate_series(1, 100000000) t(v), three runs each, ci profile: 3.95 s / 3.65 s / 3.07 s with the zone unset, and 3.09 s / 3.78 s / 3.33 s with America/Denver. The two extra kernel passes sit inside the noise.

Are there any user-facing changes?

Yes, four.

1. The behaviour change, which is the bug fix. When datafusion.execution.time_zone is set, from_unixtime(expr) returns Timestamp(Second, <session tz>) instead of Timestamp(Second, None). The epoch value is unchanged. Only the declared type and the rendered offset differ. With the default unset session time zone nothing changes.

2. An API deprecation. FromUnixtimeFunc::new() is deprecated since 56.0.0 in favour of FromUnixtimeFunc::new_with_config(). Default::default() stays available and behaves like the old new().

3. A public factory signature change. datafusion::functions::datetime::from_unixtime() becomes from_unixtime(&ConfigOptions), because it now uses make_udf_function_with_config!. This breaks callers of that factory. It is the established convention for config-aware functions in this module: now and all five to_timestamp* functions already have this signature on main, with no zero-argument shim. The expr_fn wrapper datafusion::functions::expr_fn::from_unixtime(expr) keeps its signature.

4. A new error in place of a panic. See below.

Out of range values

Arrow renders a Timestamp(Second, Some(tz)) value by a shift of the UTC instant by the zone's offset. It does that with chrono's DateTime::naive_local, which panics when the shifted value leaves NaiveDateTime's range. The panic fires when the value is formatted, not when it is cast, so it surfaces far from where it starts:

SELECT from_unixtime(-8334601211039, 'America/New_York');
-- thread 'main' panicked at chrono-0.4.45/src/datetime/mod.rs:579:14:
-- Local time out of range for `NaiveDateTime`

This PR replaces that panic with an error. The check applies to both forms. Its deliberate limits:

  • Timezone-naive results are unaffected. They are never shifted, so the full NaiveDateTime range stays usable, exactly as on main.
  • A value that is out of range in UTC as well is left to the cast, which already reports it (Cast error: Failed to convert 8210266876800 to datetime for ...). No error message changes.
  • A query that consumed an out-of-range value without a render now returns an error. On main such a value is only fatal at render time, so SELECT to_unixtime(from_unixtime(-8334601211039, 'America/New_York')) returns -8334601211039 today and errors after this PR. Every query that produced a rendered result keeps its behaviour.

Known interaction with issue 16594

This removes the panic reproducer in #16594, which I verified directly. from_unixtime(-8334601211038 - 1, 'America/New_York') errors instead of a panic, and from_unixtime(8210266876799 + 1, 'America/New_York') keeps its existing cast error.

It does not close that issue. The issue asks that all Int64 values convert, and that is not reachable while chrono::NaiveDateTime renders the value.

The underlying defect is an unguarded naive_local() in the Arrow timestamp-with-timezone formatter. It stays reachable with no from_unixtime at all:

SELECT arrow_cast(-8334601211039, 'Timestamp(Second, Some("America/New_York"))');
-- same panic

SET datafusion.execution.time_zone = 'America/New_York';
SELECT to_timestamp_seconds(-8334601211039);
-- same panic, on main and on this branch

A proper guard belongs in Arrow. The check here only stops from_unixtime from handing the formatter a value the formatter cannot render.

Merge order with PR 25175

#25175 pins today's from_unixtime behaviour in a characterization test file. No assertion in it breaks. Its SECTION 10 cases run with the session time zone unset, and this PR keeps Timestamp(Second, None) in that case. I replayed both assertions on this branch and they still pass:

Timestamp(s) 2024-07-01T00:00:00
Timestamp(s, "America/Denver") 2024-06-30T18:00:00-06:00

The comment above them says from_unixtime "ignores datafusion.execution.time_zone". That comment goes stale when this PR lands, so whichever merges second must reword it.

🤖 Generated with Claude Code

`from_unixtime(expr)` (the single argument form) hardcoded
`Timestamp(Second, None)`, so it ignored the session time zone while
`now()` and the `to_timestamp*` family in the same session correctly
returned timestamps in `datafusion.execution.time_zone`:

```sql
SET datafusion.execution.time_zone = 'America/Denver';
SELECT arrow_typeof(from_unixtime(1704110400)), from_unixtime(1704110400);
-- Timestamp(s) | 2024-01-01T12:00:00   <-- timezone naive
```

Give `FromUnixtimeFunc` the same `timezone` field + `new_with_config()` /
`with_updated_config()` treatment that `NowFunc` and the `to_timestamp*`
functions already use, and register it with
`make_udf_function_with_config!` so the session config reaches it both at
registration time and on `SET`/`RESET`.

The two argument form `from_unixtime(expr, 'tz')` is unchanged: an
explicit timezone still wins over the session time zone. With the session
time zone unset (the default) the result is still `Timestamp(Second, None)`,
so existing behaviour is preserved.

`to_unixtime` is not affected: it returns `Int64` epoch seconds, and it
already threads `config_options` into `ToTimestampSecondsFunc` for its
string parsing path.

`FromUnixtimeFunc::new()` is deprecated in favour of `new_with_config()`,
mirroring what was done for the `to_timestamp*` constructors.

Closes apache#12892

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions github-actions Bot added documentation Improvements or additions to documentation sql SQL Planner sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Sep 10, 2026
@adriangb
adriangb requested a balanced review from Copilot September 10, 2026 17:23
}

impl FromUnixtimeFunc {
#[deprecated(since = "55.0.0", note = "use `new_with_config` instead")]

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

55.0.0 is realeased, this should be deprecated in 56.0.0

@github-actions

github-actions Bot commented Sep 10, 2026

Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details
     Cloning apache/main
    Building datafusion-functions v55.0.0 (current)
       Built [  31.246s] (current)
     Parsing datafusion-functions v55.0.0 (current)
      Parsed [   0.090s] (current)
    Building datafusion-functions v55.0.0 (baseline)
       Built [  31.178s] (baseline)
     Parsing datafusion-functions v55.0.0 (baseline)
      Parsed [   0.090s] (baseline)
    Checking datafusion-functions v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.593s] 223 checks: 221 pass, 2 fail, 0 warn, 31 skip

--- failure function_parameter_count_changed: pub fn parameter count changed ---

Description:
A publicly-visible function now takes a different number of parameters.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#fn-change-arity
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/function_parameter_count_changed.ron

Failed in:
  datafusion_functions::datetime::from_unixtime now takes 1 parameters instead of 0, in /home/runner/work/datafusion/datafusion/datafusion/functions/src/datetime/mod.rs:55

--- failure type_method_marked_deprecated: type method #[deprecated] added ---

Description:
A type method is now #[deprecated]. Downstream crates will get a compiler warning when using this method.
        ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/type_method_marked_deprecated.ron

Failed in:
  method datafusion_functions::datetime::from_unixtime::FromUnixtimeFunc::new in /home/runner/work/datafusion/datafusion/datafusion/functions/src/datetime/from_unixtime.rs:94

     Summary semver requires new major version: 1 major and 1 minor checks failed
    Finished [  65.696s] datafusion-functions
    Building datafusion-sql v55.0.0 (current)
       Built [  43.832s] (current)
     Parsing datafusion-sql v55.0.0 (current)
      Parsed [   0.035s] (current)
    Building datafusion-sql v55.0.0 (baseline)
       Built [  43.596s] (baseline)
     Parsing datafusion-sql v55.0.0 (baseline)
      Parsed [   0.036s] (baseline)
    Checking datafusion-sql v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.315s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  89.170s] datafusion-sql
    Building datafusion-sqllogictest v55.0.0 (current)
       Built [ 101.352s] (current)
     Parsing datafusion-sqllogictest v55.0.0 (current)
      Parsed [   0.024s] (current)
    Building datafusion-sqllogictest v55.0.0 (baseline)
       Built [ 105.264s] (baseline)
     Parsing datafusion-sqllogictest v55.0.0 (baseline)
      Parsed [   0.029s] (baseline)
    Checking datafusion-sqllogictest v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.121s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 209.581s] datafusion-sqllogictest

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Sep 10, 2026
@adriangb
adriangb requested a review from kosiew September 10, 2026 17:25
55.0.0 is already tagged and released, so a new deprecation on main
belongs in the next release. Main already carries three other
`since = "56.0.0"` deprecations.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@adriangb

Copy link
Copy Markdown
Contributor Author

@kosiew relatively small timezone support PR for review 🙏🏻

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

It broadens a known process panic and introduces an unannounced breaking Rust factory signature.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Updates from_unixtime to honor the session execution timezone while preserving explicit timezone precedence.

Changes:

  • Adds configuration-aware timezone handling and registration.
  • Adds unit and SQL logic tests for timezone behavior.
  • Updates generated function documentation.
File summaries
File Description
docs/source/user-guide/sql/scalar_functions.md Documents session timezone behavior.
datafusion/sqllogictest/test_files/from_unixtime_timezone.slt Tests timezone and reset scenarios.
datafusion/sql/src/unparser/expr.rs Uses the non-deprecated default constructor.
datafusion/functions/src/datetime/mod.rs Registers and exports the config-aware UDF.
datafusion/functions/src/datetime/from_unixtime.rs Implements timezone-aware results and unit tests.
Review details
  • Files reviewed: 5/5 changed files
  • Comments generated: 3
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.


match len {
1 => args[0].cast_to(&Timestamp(Second, None), None),
1 => args[0].cast_to(&Timestamp(Second, self.timezone.clone()), None),
make_udf_function!(to_local_time::ToLocalTimeFunc, to_local_time);
make_udf_function!(to_time::ToTimeFunc, to_time);
make_udf_function!(to_unixtime::ToUnixtimeFunc, to_unixtime);
make_udf_function_with_config!(from_unixtime::FromUnixtimeFunc, from_unixtime);
Comment on lines +35 to +38
description = r#"
Converts an integer to RFC3339 timestamp format (`YYYY-MM-DDT00:00:00.000000000Z`).
Integers and unsigned integers are interpreted as seconds since the unix epoch
(`1970-01-01T00:00:00Z`) return the corresponding timestamp.
@codecov-commenter

codecov-commenter commented Sep 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.47059% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.92%. Comparing base (da89c7c) to head (fb72121).
⚠️ Report is 142 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/functions/src/datetime/from_unixtime.rs 96.44% 9 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25161      +/-   ##
==========================================
+ Coverage   81.60%   81.92%   +0.32%     
==========================================
  Files        1123     1132       +9     
  Lines      408898   421358   +12460     
  Branches   408898   421358   +12460     
==========================================
+ Hits       333670   345188   +11518     
- Misses      55625    55779     +154     
- Partials    19603    20391     +788     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

adriangb and others added 2 commits September 10, 2026 14:17
The description predates this PR and was wrong in both halves: `from_unixtime`
returns an Arrow `Timestamp(Second, ...)`, not an RFC3339 string with nanosecond
precision, and the trailing `Z` contradicted the example right below it, which
shows a `-04:00` offset.

Describe what the function actually returns and regenerate
`docs/source/user-guide/sql/scalar_functions.md`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…f range

A `Timestamp(Second, Some(tz))` is *rendered* by shifting the UTC instant by the
zone's offset, and Arrow does that with `chrono`'s `DateTime::naive_local`, which
panics when the shifted value falls outside `NaiveDateTime`'s range:

    SELECT from_unixtime(-8334601211039, 'America/New_York');
    -- thread 'main' panicked: Local time out of range for `NaiveDateTime`

The panic fires when the value is *formatted*, not when it is produced, so it
surfaces far away from `from_unixtime`.

The two argument form has always been able to reach it
(apache#16594). The previous commits on this
branch let the one argument form reach it too, because it now applies the session
time zone.

Check the bound in `from_unixtime` and return a `DataFusionError` instead of
producing a value that panics downstream. Values inside the range that is
representable in every time zone (the overwhelmingly common case) take a fast
path that only looks at the array's min and max, so the per-value offset
resolution is confined to inputs near the limits.

Timezone naive results are never shifted and keep the full `NaiveDateTime`
range, exactly as before. Values that are out of range in UTC as well are left
to the cast, which already reports them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@adriangb

Copy link
Copy Markdown
Contributor Author

Thanks — acted on two of the three, pushed as bda12f1 and 7876447.

1. Out of range values (valid, fixed). Correct that this widened a panic's reach: the one-argument form now applies a named-zone offset, and Arrow renders a Timestamp(Second, Some(tz)) with chrono's DateTime::naive_local, which panics when the shifted value leaves NaiveDateTime's range. from_unixtime now bounds checks its input and returns an error instead of producing a value that panics later:

Execution error: Cannot convert -8334601211039 to a timestamp in timezone "America/New_York"
for function from_unixtime: the local date and time is outside the supported range

Applied to both the one- and two-argument forms — the two-argument form panicked on main too, which is #16594. Notes:

  • The panic fires when the value is formatted, not when it is cast, so the new tests render through arrow::util::display::ArrayFormatter; a test that only computes the value passes either way. Removing the check makes all five of them fail.
  • Timezone naive results keep the full NaiveDateTime range (they are never shifted), and values that are out of range in UTC as well are still left to the cast, so no existing error message changes.
  • The common case is free: values inside the range representable in every zone take a fast path over the array's min/max, so resolving the zone offset per value only happens near the limits.

This removes #16594's panic reproducer — from_unixtime(-8334601211038 - 1, 'America/New_York') errors instead of panicking, and its other reproducer (8210266876799 + 1) keeps its existing cast error. It does not close that issue, which asks that all Int64 values be convertible; that is not achievable while the value is rendered through chrono::NaiveDateTime. The root cause is still an unguarded naive_local() in the Arrow formatter, reachable with no from_unixtime involved (SELECT arrow_cast(-8334601211039, 'Timestamp(Second, Some("America/New_York"))')); fixing that properly belongs upstream.

2. Public factory signature (not changed, deliberately). make_udf_function_with_config! is the existing convention for config-aware functions in this module: on main, now and all five to_timestamp* functions already use it, so datafusion::functions::datetime::now(&ConfigOptions) is already the public signature and there is no zero-argument shim for it. Giving from_unixtime a separately-named factory would make it the odd one out among its siblings. It is a user-facing breaking change though, so I've added it explicitly to the "Are there any user-facing changes?" section, which previously only mentioned the FromUnixtimeFunc::new() deprecation.

3. Documentation description (valid, fixed). The description was wrong on both counts and predates this PR: from_unixtime returns an Arrow Timestamp(Second, ...), not an RFC3339 string with nanosecond precision, and the trailing Z contradicted the example directly below it, which shows -04:00. Rewritten to describe a second-resolution timestamp in the selected time zone, keeping the session time zone paragraph. Docs regenerated with dev/update_function_docs.sh.

cargo fmt --all, cargo clippy --all-targets -- -D warnings, cargo test -p datafusion-functions from_unixtime and the full cargo test -p datafusion-sqllogictest --test sqllogictests (506 files) are green.

`unparseable` -> `unparsable`, in a comment added by the previous commit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@adriangb

Copy link
Copy Markdown
Contributor Author

Minor merge-order note: #25175 (timezone characterization tests) has a from_unixtime block in Section 9 of datafusion/sqllogictest/test_files/datetime/timestamps_timezone.slt.

No assertion in it breaks — that block runs with datafusion.execution.time_zone unset, where this PR deliberately keeps Timestamp(Second, None). But the comment above it says from_unixtime "ignores datafusion.execution.time_zone", which stops being true once this lands.

Whichever merges second should update the comment. Flagging it so the stale text does not survive review.

@adriangb

Copy link
Copy Markdown
Contributor Author

Self-review. This is my own PR, so this is a QA pass on my own work, not an independent review. Its value is the list of things I ran, and the risk that remains after them.

I built main (da89c7c85b, the exact merge base) and this branch (e8cb6a69) as two datafusion-cli binaries, and I compared them directly. Everything below comes from a run, not from a read.


Findings

1. The description states "No previously-successful query changes behaviour". That is false.

Two queries succeed on main and fail on this branch:

SELECT to_unixtime(from_unixtime(-8334601211039, 'America/New_York'));
-- main:   -8334601211039
-- branch: Execution error: Cannot convert -8334601211039 to a timestamp in timezone
--         "America/New_York" for function from_unixtime: ...

SELECT from_unixtime(-8334601211039, 'America/New_York')
     > from_unixtime(0, 'America/New_York');
-- main:   false
-- branch: Execution error: ... (same)

The reason is a real asymmetry. On main the value is only fatal when a formatter renders it. A query that computes with the value but never renders it works today. The new guard rejects the value earlier, at production time, so those queries now fail.

The guard is still the right trade: the alternative is a process panic. But the sentence must go. A correct statement is: no query that produced a rendered result changes behaviour; a query that consumed an out-of-range value without rendering it now returns an error.

A count(*) over the same expression still works on both binaries, because the projection is pruned before invoke_with_args runs.

2. to_timestamp_seconds still panics on the same input, on this branch.

SET datafusion.execution.time_zone = 'America/New_York';
SELECT to_timestamp_seconds(-8334601211039);
-- main:   thread 'main' panicked ... Local time out of range for `NaiveDateTime`
-- branch: thread 'main' panicked ... Local time out of range for `NaiveDateTime`

This is a sibling in the same module. It returns Timestamp(Second, <session tz>), which is the exact shape the guard protects in from_unixtime. It is pre-existing and this PR does not make it worse, but the description names only arrow_cast as the residual path. to_timestamp_seconds is a much more likely route for a user, so it belongs in the residual-risk list. The "removes the panic reproducer in #16594" line reads wider than the change is.

3. cargo-semver-checks is red on this PR and the description does not say so.

The bot reports function_parameter_count_changed for datafusion_functions::datetime::from_unixtime and type_method_marked_deprecated for FromUnixtimeFunc::new. The description argues the convention (now and the five to_timestamp* factories already take &ConfigOptions), which is correct, but a reviewer must not discover the red bot on their own.

4. The expr_fn doc string keeps the wrong text that commit bda12f1 fixed elsewhere.

datafusion/functions/src/datetime/mod.rs:82 still reads:

from_unixtime,
"converts an integer to RFC3339 timestamp format string",
@config unixtime

That is the same claim the user_doc change removed. A one-line follow-up.

5. Nit: the #[deprecated] attribute sits above the /// block on new().

Convention in this repo puts the doc comment first and the attribute after it.


What I verified, and how

The 86400 over-bound is safe, with margin

I enumerated all 598 IANA zones and took the UTC offset at year 1 (the local mean time era, before any transition) and at year 9999 (the POSIX footer rule). The two extremes are:

  • Asia/Manila, LMT -15:56:08 = -57368 s
  • America/Metlakatla, LMT +15:13:42 = +54822 s

Both are inside 86400, so the fast path leaves about 29000 s of headroom for named zones. I then confirmed chrono-tz agrees with that data at the exact boundary:

SELECT from_unixtime(-8334601171432, 'Asia/Manila');  -- -262143-01-01T00:00:00-15:56
SELECT from_unixtime(-8334601171433, 'Asia/Manila');  -- Execution error, as intended

Fixed offsets are the tighter case. +23:59 is 86340 s, which passes:

SET datafusion.execution.time_zone = '+23:59';
SELECT from_unixtime(8210266790459);  -- +262142-12-31T23:59:59+23:59

+24:00 and +99:00 never reach the guard, because Arrow's Tz parser rejects them first (Invalid timezone "+24:00": failed to parse timezone). So 86400 is a true over-bound for every zone the engine can hold.

The guard's boundary is exact, and the tests are real

The counterfactual in the description is "removing the bounds check makes all five of them fail". I did not take that on trust. I ran the same values through arrow_cast, which is the unguarded path, in the same binary:

SELECT arrow_cast(8210266844399, 'Timestamp(Second, Some("Asia/Tokyo"))');  -- renders
SELECT arrow_cast(8210266844400, 'Timestamp(Second, Some("Asia/Tokyo"))');  -- panics

The guard errors at exactly 8210266844400 and passes 8210266844399. So the bound is neither loose nor tight by one second.

On the "do the tests render?" question: yes. The unit tests call display(), which builds an arrow::util::display::ArrayFormatter. The sqllogictest cases use query P, which renders the value. A test that only computed the value would prove nothing here, and neither kind of test does that.

The two-argument form is unchanged

I ran 140 pairs — 20 values by 7 zones — through both binaries and diffed the output. The zones include Australia/Lord_Howe (a 30-minute DST step) and Pacific/Chatham (+12:45). The values include 0, ±1, the NaiveDateTime extremes, and 16 random values across ±4e12. Result: byte-identical.

The default (unset) session zone is byte-identical to main

Eleven probes, including the error paths, all byte-identical:

Probe Result
arrow_typeof + value, scalar identical
NULL input identical
Two-argument form identical
Both NaiveDateTime extremes identical
Array (not constant folded) input identical
to_unixtime(from_unixtime(...)) identical
Empty timezone string '' identical
Invalid zone 'Not/AZone' identical
Wrong argument type identical
Equality against an explicit 'UTC' identical
arrow_cast-wrapped input identical

The documentation examples reproduce exactly

Both blocks in scalar_functions.md match the real output character for character, offsets included.

The interaction with #25175 holds

I replayed SECTION 10's two from_unixtime assertions from #25175 on this branch, with the session zone unset:

Timestamp(s) 2024-07-01T00:00:00
Timestamp(s, "America/Denver") 2024-06-30T18:00:00-06:00

Both still pass. Only the comment above them goes stale, exactly as #25175 says.

The guard costs nothing measurable

SELECT count(from_unixtime(v)) FROM generate_series(1, 100000000) t(v), three runs each, ci profile:

  • session zone unset (guard short-circuits): 3.95 s, 3.65 s, 3.07 s
  • session zone America/Denver (guard runs min and max): 3.09 s, 3.78 s, 3.33 s

The two extra kernel passes sit inside the run-to-run noise. The "the common case is free" claim holds at this scale.

Local checks

  • cargo test -p datafusion-functions --lib datetime::from_unixtime — 10 tests pass
  • cargo test -p datafusion-sqllogictest --test sqllogictests — 506 files pass
  • cargo clippy -p datafusion-functions -p datafusion-sql --all-targets -- -D warnings — clean
  • cargo fmt --all --check — clean
  • ./ci/scripts/doc_prettier_check.sh — clean

What I did not verify

  • Serialization. FromUnixtimeFunc now carries state, so a plan that a session with a zone builds, and a session without a zone decodes, resolves the function by name and picks up the second session's zone. now and to_timestamp* behave the same way on main, so this is not new, but I did not run a proto round trip.
  • Behaviour across a SET inside a prepared statement or a view.
  • Any engine other than datafusion-cli.

@adriangb

Copy link
Copy Markdown
Contributor Author

Correction to my own review comment above, item 3: cargo-semver-checks is not red on this PR. The Check semver job reports SUCCESS. I have removed the matching sentence from the description, which claimed the tool flags the deprecation and the factory arity change.

Items 1, 2, 4 and 5 in that comment stand. Item 4 in particular — datafusion/functions/src/datetime/mod.rs still describes from_unixtime as converting "an integer to RFC3339 timestamp format string", the same wrong claim that commit bda12f1 corrected in the user_doc block. I am fixing that in a follow-up commit here.

`mod.rs` still described `from_unixtime` as converting "an integer to RFC3339
timestamp format string". The function returns an Arrow `Timestamp(Second, ...)`,
not a string, and not at nanosecond resolution.

This is the same wrong claim that an earlier commit corrected in the `user_doc`
block. The `expr_fn` copy was missed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adriangb,

Thanks for working on this. The session timezone handling looks good, and I like that the explicit timezone still takes precedence. I only have one non-blocking suggestion to strengthen the DST coverage.

statement ok
SET datafusion.execution.time_zone = 'America/Denver';

query TP

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add an America/Denver summer-instant assertion here, for example using a date in July that renders with -06:00? The named-zone coverage currently only checks January at -07:00, so adding a summer case would make sure we are testing DST behavior rather than something that could also be satisfied by a fixed offset. This is non-blocking since Arrow already provides the timezone rendering behavior, but it would help make the SQL contract a bit more robust.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto detected api change Auto detected API change documentation Improvements or additions to documentation functions Changes to functions implementation sql SQL Planner sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make from_unixtime aware of execution timezone

4 participants