fix: `AT TIME ZONE` on a timezone-aware timestamp returns a naive timestamp by adriangb · Pull Request #25165 · apache/datafusion

adriangb · 2026-09-10T17:43:41Z

Which issue does this PR close?

Closes Casting existing timestamp to timestamp again strips timezone information #12218

Warning

Known limitation, found in review of this PR: the fix does not reach every expression.

The type branch calls Expr::get_type in the SQL planner, and the planner runs before the type
coercion analyzer. For a CASE, get_type reports the first non-null THEN arm and ignores
coercion. So the two queries below have the same coerced input type, Timestamp(ns, "UTC"),
and give different answers:

CASE WHEN b THEN naive ELSE aware END AT TIME ZONE 'America/Denver'
-- Timestamp(ns, "America/Denver")   2024-01-01T05:00:00-07:00   <- old shape, not fixed
CASE WHEN b THEN aware ELSE naive END AT TIME ZONE 'America/Denver'
-- Timestamp(ns)                     2024-01-01T05:00:00         <- fixed

PostgreSQL 17 gives the naive 2024-01-01 05:00:00 for both. coalesce is not affected, because
verify_function_arguments coerces before the planner reads the type.

A correct fix must dispatch after coercion, which is a larger change than this PR. The behaviour is
pinned in datetime/timestamps.slt so it is visible rather than silent. Maintainers: please say
whether you want that fixed here before merge, or tracked separately.

Rationale for this change

Start here: what a user gets today

Two terms run through this description:

an aware timestamp carries a timezone, so it names an instant. Arrow spells it Timestamp(unit, Some(tz)). SQL spells it timestamptz.
a naive timestamp carries no timezone, so it names only a wall clock. Arrow spells it Timestamp(unit, None). SQL spells it timestamp.

This is the case from the issue. Noon UTC is 05:00 in Denver.

SET datafusion.execution.time_zone = 'UTC';

SELECT ('2024-01-01T12:00:00Z'::timestamptz AT TIME ZONE 'America/Denver')::timestamp;

Engine	Result
DataFusion, today	`2024-01-01T12:00:00`
DataFusion, this PR	`2024-01-01T05:00:00`
PostgreSQL 17.11	`2024-01-01 05:00:00`
DuckDB 1.5.2	`2024-01-01 05:00:00`

DataFusion returns the UTC wall clock, so it drops the timezone the user asks for. Drop the final cast and the cause appears:

SELECT
  '2024-01-01T12:00:00Z'::timestamptz AT TIME ZONE 'America/Denver' AS value,
  arrow_typeof('2024-01-01T12:00:00Z'::timestamptz AT TIME ZONE 'America/Denver') AS type;

Engine	Value	Type
DataFusion, today	`2024-01-01T05:00:00-07:00`	`Timestamp(ns, "America/Denver")`
DataFusion, this PR	`2024-01-01T05:00:00`	`Timestamp(ns)`
PostgreSQL 17.11	`2024-01-01 05:00:00`	`timestamp without time zone`
DuckDB 1.5.2	`2024-01-01 05:00:00`	`TIMESTAMP`

The displayed value looks right today, but the type is wrong. The result stays aware, so the next operation on it reads the UTC wall clock again.

Overview

This is a breaking semantic change. It changes the result type of AT TIME ZONE, and so the result value of anything downstream, when the input is an aware timestamp.

AT TIME ZONE is asymmetric in PostgreSQL. It always returns the other kind of timestamp:

a naive input becomes an aware timestamp. The engine reads the value as a wall clock in the target timezone and returns that instant.
an aware input becomes a naive timestamp. The engine reads off the wall clock that the instant has in the target timezone.

DataFusion implements the first half only. This PR adds the second half. The first half does not change.

Why the current lowering is wrong

DataFusion lowers both halves to CAST(expr AS Timestamp(Nanosecond, Some(tz))).

For a naive input that cast is correct. Arrow reads the naive value as local time in tz.
For an aware input that cast only relabels the display timezone. It keeps the instant, so the result stays aware.

What changes are included in this PR?

SqlToRel now types the input of AT TIME ZONE and branches on it. The new function is sql_at_time_zone_to_expr in datafusion/sql/src/expr/mod.rs.

Timestamp(unit, None), or any type that is not a timestamp: no change. The planner emits CAST(expr AS Timestamp(unit, Some(tz))).
Timestamp(unit, Some(_)): the planner emits the same cast, then wraps it in a step that drops the timezone and keeps the wall clock. That step is to_local_time.

The planner hook

datafusion-sql must not depend on datafusion-functions, so the second step goes through a new trait method:

ExprPlanner::plan_at_time_zone in datafusion/expr/src/planner.rs,
implemented by DatetimeFunctionPlanner in datafusion/functions/src/datetime/planner.rs,
with a default PlannerResult::Original body, so out-of-tree implementations still compile.

plan_extract lowers EXTRACT to date_part through the same seam, so this follows an established pattern. The alternative was a name lookup through ContextProvider::get_function_meta("to_local_time"). The hook keeps the function name out of the SQL planner.

The type branch stays in datafusion-sql, because Expr::get_type needs the schema and an ExprPlanner does not have one. The hook receives the expression after the cast, and the trait doc states that contract.

A session that registers no such planner now gets a plan error, where before it got a wrong plan.

The `TimeUnit` change

AT TIME ZONE no longer forces Nanosecond. It keeps the input's TimeUnit when the input is a timestamp.

Input	Type today	Type with this PR
`Timestamp(µs, "UTC")`	`Timestamp(ns, "America/Denver")`	`Timestamp(µs)`
`Timestamp(µs, None)`	`Timestamp(ns, "America/Denver")`	`Timestamp(µs, "America/Denver")`
`Timestamp(s, "UTC")`	`Timestamp(ns, "America/Denver")`	`Timestamp(s)`
`Utf8`, `Date32`, and other non-timestamps	`Timestamp(ns, "America/Denver")`	`Timestamp(ns, "America/Denver")`

PostgreSQL keeps the precision of its input too. timestamptz '2024-01-01 12:00:00.123456Z' AT TIME ZONE 'America/Denver' returns 2024-01-01 05:00:00.123456.

Not changed

Spark. SparkFunctionPlanner implements plan_extract and plan_substring only, and with_spark_features inserts it at index 0 of the list that with_default_features builds. So plan_at_time_zone falls through to DatetimeFunctionPlanner, and no Spark-specific function or planner behaviour changes. A Spark-flavoured session that uses DataFusion's SQL planner does get the new AT TIME ZONE semantics, but Spark SQL has no AT TIME ZONE syntax. It uses from_utc_timestamp and to_utc_timestamp. Nothing under datafusion/spark/ implements, tests or documents the operator.
AT LOCAL. PostgreSQL 17 supports it. sqlparser 0.62.0 has no AtLocal AST node and rejects the syntax, so there is nothing to route.
Fixed-offset strings. See the field research below. Only the result type changes for these.
String-literal inputs. '2000-12-01T04:04:12-05:00' AT TIME ZONE 'x' still takes the naive path, because Utf8 is not a timestamp type. That keeps today's behaviour: DataFusion honours the offset inside the string. PostgreSQL resolves the unknown-typed literal to timestamp and drops the offset. This gap is older than this PR and is separate from Casting existing timestamp to timestamp again strips timezone information #12218.

What is the testing strategy for this PR?

Field research

I measured two reference engines rather than trust the documentation:

PostgreSQL 17.11, docker run --rm -e POSTGRES_PASSWORD=pw -p 55452:5432 postgres:17, with SET TimeZone='UTC',
DuckDB 1.5.2 CLI, with SET TimeZone='UTC'.

The DataFusion columns come from a datafusion-cli build of main and of this branch, with SET datafusion.execution.time_zone = 'UTC'.

The four core cases

Case	DataFusion, today	DataFusion, this PR	PostgreSQL 17.11	DuckDB 1.5.2
naive input: `'2024-01-01 12:00:00'::timestamp AT TIME ZONE 'America/Denver'`	`2024-01-01T12:00:00-07:00` `Timestamp(ns, "America/Denver")`	same as today	`2024-01-01 19:00:00+00` `timestamp with time zone`	`2024-01-01 19:00:00+00` `TIMESTAMP WITH TIME ZONE`
aware input: `'2024-01-01T12:00:00Z'::timestamptz AT TIME ZONE 'America/Denver'`	`2024-01-01T05:00:00-07:00` `Timestamp(ns, "America/Denver")`	`2024-01-01T05:00:00` `Timestamp(ns)`	`2024-01-01 05:00:00` `timestamp without time zone`	`2024-01-01 05:00:00` `TIMESTAMP`
the issue reproducer: `(<aware> AT TIME ZONE 'America/Denver')::timestamp`	`2024-01-01T12:00:00`	`2024-01-01T05:00:00`	`2024-01-01 05:00:00`	`2024-01-01 05:00:00`
chained: `<aware> AT TIME ZONE 'America/Denver' AT TIME ZONE 'Europe/Brussels'`	`2024-01-01T13:00:00+01:00` = 12:00 UTC	`2024-01-01T05:00:00+01:00` = 04:00 UTC	`2024-01-01 04:00:00+00`	`2024-01-01 04:00:00+00`

The naive row agrees across all four columns. 12:00 in Denver is 19:00 UTC. Only the display convention differs.

The chained row is the one that changes value, not just type. Today DataFusion relabels twice and the instant never moves. With this PR the Denver wall clock (05:00) becomes a Brussels wall clock, which is 04:00 UTC. PostgreSQL and DuckDB both return 04:00 UTC.

Both 2024 DST transitions in `America/Denver`, aware input

UTC instant	DataFusion, today	DataFusion, this PR	PostgreSQL 17.11	DuckDB 1.5.2
`2024-01-01T12:00:00Z`	`2024-01-01T05:00:00-07:00`	`2024-01-01T05:00:00`	`2024-01-01 05:00:00`	`2024-01-01 05:00:00`
`2024-03-10T08:59:00Z`	`2024-03-10T01:59:00-07:00`	`2024-03-10T01:59:00`	`2024-03-10 01:59:00`	`2024-03-10 01:59:00`
`2024-03-10T09:00:00Z`	`2024-03-10T03:00:00-06:00`	`2024-03-10T03:00:00`	`2024-03-10 03:00:00`	`2024-03-10 03:00:00`
`2024-07-01T12:00:00Z`	`2024-07-01T06:00:00-06:00`	`2024-07-01T06:00:00`	`2024-07-01 06:00:00`	`2024-07-01 06:00:00`
`2024-11-03T07:59:00Z`	`2024-11-03T01:59:00-06:00`	`2024-11-03T01:59:00`	`2024-11-03 01:59:00`	`2024-11-03 01:59:00`
`2024-11-03T08:00:00Z`	`2024-11-03T01:00:00-07:00`	`2024-11-03T01:00:00`	`2024-11-03 01:00:00`	`2024-11-03 01:00:00`

The rows cover the spring-forward gap (01:59 jumps to 03:00) and the fall-back overlap (01:59 MDT, then 01:00 MST). All four columns agree on the wall clock. Only the DataFusion type changes.

The DST gap and overlap from the other side, naive input

This is the branch that this PR does not touch. It is here so the table stays honest.

Case	DataFusion, today and with this PR	PostgreSQL 17.11	DuckDB 1.5.2
gap: `'2024-03-10 02:30:00'::timestamp AT TIME ZONE 'America/Denver'`	`Arrow error: Cast error: Cannot cast timezone to different timezone`	`2024-03-10 09:30:00+00`	`2024-03-10 09:30:00+00`
overlap: `'2024-11-03 01:30:00'::timestamp AT TIME ZONE 'America/Denver'`	same error	`2024-11-03 08:30:00+00`	`2024-11-03 08:30:00+00`

A bare arrow_cast to Timestamp(Nanosecond, Some("America/Denver")) reproduces the error, so it comes from the arrow cast and not from the SQL planner. This PR does not change it.

The case the two engines cannot arbitrate: a fixed-offset string

Case	DataFusion, today	DataFusion, this PR	PostgreSQL 17.11	DuckDB 1.5.2
`<aware> AT TIME ZONE '+05:30'`	`2024-01-01T17:30:00+05:30` `Timestamp(ns, "+05:30")`	`2024-01-01T17:30:00` `Timestamp(ns)`	`2024-01-01 06:30:00` `timestamp without time zone`	`Not implemented Error: Unknown TimeZone '+05:30'!`

DataFusion follows arrow and ISO 8601, where +05:30 is east of UTC. PostgreSQL applies the POSIX convention to an offset spelled as a string and reads +05:30 as west of UTC. DuckDB rejects the string and accepts IANA names only. So DataFusion's reading matches neither engine.

PostgreSQL does agree with DataFusion when the offset carries a type:

postgres=# SELECT timestamptz '2024-01-01 12:00:00Z' AT TIME ZONE interval '+05:30';
 2024-01-01 17:30:00

This divergence already exists on main. It is separate from #12218 and out of scope here. It is tracked in #25170. This PR changes only the result type for this case, not the value.

Tests

New sqllogictest coverage in datafusion/sqllogictest/test_files/datetime/timestamps.slt. The first commit adds it as a characterization of today's behaviour. The second commit flips it, so the diff shows exactly what changes. Every case asserts arrow_typeof next to the value, and I checked every expected value against PostgreSQL 17 by hand. I did not regenerate them with --complete. The cases are:

a naive column and an aware column,
the exact ::timestamp reproducer from Casting existing timestamp to timestamp again strips timezone information #12218,
two chained AT TIME ZONE applications,
a fixed-offset zone,
microsecond precision in both directions,
now(),
a multi-row aware column that spans both 2024 DST transitions in America/Denver.

Two planner tests in datafusion/sql/tests/sql_integration.rs: the plan shape of the naive lowering, and the error when no ExprPlanner provides to_local_time.

Green locally:

the full sqllogictest suite, with zero changed expectations in the files that main already has,
cargo test -p datafusion-sql,
cargo test -p datafusion-functions to_local_time,
cargo fmt --all and cargo clippy --all-targets -- -D warnings,
the repo's extended workspace suite (--features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption).

One caveat on the zero-change result. AT TIME ZONE appears in exactly one test file on main, and every use there has a Utf8 or naive inner expression. So the corpus had no coverage of the broken case at all, which is why the bug survived from #9647 in DataFusion 37. "Zero changed expectations" is a weak safety signal here, not a strong one. The field research above carries the argument instead.

Are there any user-facing changes?

Yes. This is a breaking change to AT TIME ZONE semantics. The api change label is applied.

The changes, in order of impact:

An aware input now returns a naive timestamp, per the tables above. Any expression downstream of it changes value.
AT TIME ZONE keeps the input's TimeUnit instead of a forced Nanosecond.
An unaliased projection changes its column name for the aware case. SELECT aware AT TIME ZONE 'America/Denver' FROM t now names the column to_local_time(t.aware).
A session that does not register the datetime planner from datafusion-functions now gets a plan error for AT TIME ZONE on an aware input, where before it got a wrong plan.
ExprPlanner gains a plan_at_time_zone method. It has a default body, so out-of-tree implementations still compile.
New AT TIME ZONE section in docs/source/user-guide/sql/operators.md. The to_local_time description says how the two relate.

A naive input does not change. Its value and its timezone label stay the same.

Maintainer input wanted: does this land as a straight behaviour fix, or behind a config flag with a deprecation period? I deliberately added no flag. The current behaviour has no defensible justification, and a flag means two timestamp typings through the planner for a long time. But this changes results for anyone who relies on the old shape, so I would rather you decide than assume.

Merge order: #25175 adds a timezone characterization suite (test_files/datetime/timestamps_timezone.slt) whose SECTION 5b pins the exact behaviour this PR changes, both the arrow_typeof and the value. Whichever of the two merges second must update the other, and CI on it stays red until that happens. After this PR lands, the two queries in SECTION 5b become:

SELECT arrow_typeof(column1 AT TIME ZONE 'Europe/Brussels'), column1 AT TIME ZONE 'Europe/Brussels' FROM c_utc
  Timestamp(ns, "Europe/Brussels") 2024-01-15T13:00:00+01:00  ->  Timestamp(ns) 2024-01-15T13:00:00
  Timestamp(ns, "Europe/Brussels") 2024-07-01T14:00:00+02:00  ->  Timestamp(ns) 2024-07-01T14:00:00

SELECT arrow_typeof(column1 AT TIME ZONE 'America/Denver'), column1 AT TIME ZONE 'America/Denver' FROM c_denver
  Timestamp(ns, "America/Denver") 2024-01-15T12:00:00-07:00  ->  Timestamp(ns) 2024-01-15T12:00:00
  Timestamp(ns, "America/Denver") 2024-07-01T12:00:00-06:00  ->  Timestamp(ns) 2024-07-01T12:00:00

The DIVERGES FROM POSTGRESQL (type, not instant) comment above them can go, because this PR removes that divergence. Nothing else in that file conflicts. Its other AT TIME ZONE cases all apply the operator to a naive input.

🤖 Generated with Claude Code

Records DataFusion's current behaviour for `<tz-aware> AT TIME ZONE zone`, including the reproducer from apache#12218, DST transitions on a real multi-row column, chaining, fixed offsets and precision handling. PostgreSQL returns a timezone-*naive* `timestamp` here, holding the wall clock in `zone`; DataFusion returns a timezone-aware value that merely relabels the display zone. These expectations are flipped in the next commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

`expr AT TIME ZONE 'tz'` was unconditionally lowered to `CAST(expr AS Timestamp(Nanosecond, Some(tz)))`. That is right for a timezone-naive input -- arrow's `Timestamp(_, None) -> Timestamp(_, Some(tz))` cast reads the naive value as local time in `tz` -- but wrong for a timezone-aware one, where it merely relabels the display zone. PostgreSQL (and DuckDB) make the operator asymmetric: `timestamptz AT TIME ZONE zone` returns a `timestamp` (naive) holding the wall clock in `zone`. Because DataFusion kept the value timezone-aware, casting the result to `::timestamp` produced the UTC wall clock instead of the zone's, which is what apache#12218 reports. The SQL planner now types the input and branches on it. The naive case is unchanged. The aware case relabels the instant into `tz` with the same cast and then drops the timezone while keeping the displayed value, which is exactly `to_local_time`. `datafusion-sql` must not depend on `datafusion-functions`, so that second half goes through a new `ExprPlanner::plan_at_time_zone` hook -- the same shape as `plan_extract` lowering `EXTRACT` to `date_part` -- implemented by `DatetimeFunctionPlanner`. Sessions without it get a clear planning error instead of the old silent mislowering. `AT TIME ZONE` also no longer forces `Nanosecond`: it keeps the input's `TimeUnit` when the input is a timestamp. Closes apache#12218 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Adds an `AT TIME ZONE` entry to the SQL operators guide covering both directions (naive -> aware, aware -> naive), precision preservation, chaining and the ISO-vs-POSIX sign convention for fixed-offset strings. `to_local_time`'s own examples all apply `AT TIME ZONE` to timezone-*naive* values, so they are unaffected by the semantic change; its description now says so, and notes that wrapping an already timezone-aware `AT TIME ZONE` in `to_local_time` is redundant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

codecov-commenter · 2026-09-10T18:07:02Z

Codecov Report

❌ Patch coverage is 88.46154% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.91%. Comparing base (da89c7c) to head (3e42f8b).
⚠️ Report is 142 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/sql/src/expr/mod.rs	85.36%	2 Missing and 4 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #25165      +/-   ##
==========================================
+ Coverage   81.60%   81.91%   +0.31%     
==========================================
  Files        1123     1132       +9     
  Lines      408898   421160   +12262     
  Branches   408898   421160   +12262     
==========================================
+ Hits       333670   344992   +11322     
- Misses      55625    55772     +147     
- Partials    19603    20396     +793

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

adriangb · 2026-09-10T19:36:57Z

Heads up on merge order: #25175 adds a timezone characterization suite (test_files/datetime/timestamps_timezone.slt) whose SECTION 5b pins the exact behaviour this PR changes — AT TIME ZONE on an already timezone-aware column, both the arrow_typeof and the value.

Whichever of the two merges second will need to update the other, and CI on it will be red until that happens. Concretely, after this PR lands the two queries in SECTION 5b become:

SELECT arrow_typeof(column1 AT TIME ZONE 'Europe/Brussels'), column1 AT TIME ZONE 'Europe/Brussels' FROM c_utc
  Timestamp(ns, "Europe/Brussels") 2024-01-15T13:00:00+01:00  ->  Timestamp(ns) 2024-01-15T13:00:00
  Timestamp(ns, "Europe/Brussels") 2024-07-01T14:00:00+02:00  ->  Timestamp(ns) 2024-07-01T14:00:00

SELECT arrow_typeof(column1 AT TIME ZONE 'America/Denver'), column1 AT TIME ZONE 'America/Denver' FROM c_denver
  Timestamp(ns, "America/Denver") 2024-01-15T12:00:00-07:00  ->  Timestamp(ns) 2024-01-15T12:00:00
  Timestamp(ns, "America/Denver") 2024-07-01T12:00:00-06:00  ->  Timestamp(ns) 2024-07-01T12:00:00

and the DIVERGES FROM POSTGRESQL (type, not instant) comment above them should be deleted, since this PR removes that divergence.

Nothing else in that file conflicts — its other AT TIME ZONE cases all apply the operator to a timezone-naive input, which this PR leaves alone. The same note is in #25175's description. No action needed here; just so whoever merges either one knows.

adriangb · 2026-09-10T23:09:53Z

Self-review: a QA pass on my own PR. I ran every check below against three sources:

this branch, built as datafusion-cli,
a live PostgreSQL 17.11 in Docker,
a local DuckDB 1.5.2 CLI. One finding is a real gap in the fix. The rest are smaller, plus two claims in the description that I must correct.

Terms: an aware timestamp carries a timezone (Timestamp(unit, Some(tz))). A naive timestamp carries none (Timestamp(unit, None)).

1. The type branch runs before type coercion, so the fix misses some inputs

sql_at_time_zone_to_expr calls Expr::get_type in the SQL planner. That is before the type-coercion analyzer runs. For a CASE expression, Expr::get_type returns the type of the first non-null THEN arm and ignores coercion. So the branch reads "naive" for an expression whose real type is aware.

SET datafusion.execution.time_zone = 'UTC';
CREATE TABLE t AS SELECT '2024-01-01T12:00:00Z'::timestamptz AS aware,
                         '2024-01-01 12:00:00'::timestamp   AS naive, true AS b;

SELECT arrow_typeof(CASE WHEN b THEN naive ELSE aware END) FROM t;
-- Timestamp(ns, "UTC")     <- the real type is aware

SELECT arrow_typeof((CASE WHEN b THEN naive ELSE aware END) AT TIME ZONE 'America/Denver'),
       (CASE WHEN b THEN naive ELSE aware END) AT TIME ZONE 'America/Denver' FROM t;
-- Timestamp(ns, "America/Denver")  2024-01-01T05:00:00-07:00   <- the OLD, wrong shape

SELECT arrow_typeof((CASE WHEN b THEN aware ELSE naive END) AT TIME ZONE 'America/Denver') FROM t;
-- Timestamp(ns)            <- correct, only because the arms swap order

PostgreSQL 17.11 on the same shape:

 case_type                | v                   | t
 timestamp with time zone | 2024-01-01 05:00:00 | timestamp without time zone

So the result depends on the order of the CASE arms. coalesce is safe, because verify_function_arguments coerces the argument fields before it computes the return field. CASE is not.

Options, in my order of preference:

Do the dispatch after type coercion. A rewrite pass sees the coerced type, so the branch is always right.
Keep the planner branch and add the CASE case to the .slt file as a known limitation, with a TODO and an issue link.

I do not want to merge this without at least option 2. A silent, order-dependent hole is worse than the old uniform bug.

2. The description says PostgreSQL and DuckDB agree on every case. They do not

The claim covers "fixed offsets". DuckDB rejects a fixed-offset string outright:

D SELECT (TIMESTAMPTZ '2024-01-01 12:00:00Z' AT TIME ZONE '+05:30');
Not implemented Error: Unknown TimeZone '+05:30'!

PostgreSQL accepts it and reads it POSIX-style (west-positive), so it returns 2024-01-01 06:30:00. DataFusion reads it ISO-style (east-positive) and returns 2024-01-01 17:30:00. PostgreSQL agrees with DataFusion only when the offset is spelled as an interval:

postgres=# SELECT timestamptz '2024-01-01 12:00:00Z' AT TIME ZONE interval '+05:30';
 2024-01-01 17:30:00

The two engines cannot arbitrate this one. The description must say so. It must not claim agreement. The divergence already exists on main and is out of scope here (#25170), but it must not hide behind a blanket claim.

3. The `ExprPlanner` hook is narrower than the description says

The description says a future dialect "gets the same seam for free". It does not.

The hook runs only for an aware input. A dialect cannot change the naive half or the type dispatch.
The hook receives one Expr that is already CAST(input AS Timestamp(unit, Some(tz))). An implementation that wants other semantics must destructure that Cast to recover the input and the target timezone.

Every sibling hook takes the raw operands. plan_extract gets [field, expr]. I suggest plan_at_time_zone(vec![input, lit(tz)]) and let the implementation build its own cast. The trait doc then states a contract instead of a workaround.

Two smaller points on the same code:

The name is wrong. The method plans the second half only. plan_to_local_time says what it does.
Use not_impl_err! for the "no planner" error, to match the EXTRACT arm at datafusion/sql/src/expr/mod.rs:321.

The documented contract itself does hold: the single call site always passes the relabelled CAST.

4. A dictionary-encoded aware timestamp keeps the old behaviour

_ => (TimeUnit::Nanosecond, false) treats Dictionary(_, Timestamp(_, Some(tz))) as naive:

CREATE TABLE d AS SELECT arrow_cast(arrow_cast('2024-01-01T12:00:00Z','Timestamp(Nanosecond, Some("UTC"))'),
                                    'Dictionary(Int32, Timestamp(Nanosecond, Some("UTC")))') AS dts;
SELECT arrow_typeof(dts AT TIME ZONE 'America/Denver'), dts AT TIME ZONE 'America/Denver' FROM d;
-- Timestamp(ns, "America/Denver")  2024-01-01T05:00:00-07:00   <- old shape

Add a Dictionary(_, Timestamp(unit, tz)) arm, or state the limit in the operator doc.

5. The output column name changes, and the new name drops the timezone

SELECT aware AT TIME ZONE 'America/Denver' FROM t;   -- column: to_local_time(t.aware)
SELECT naive AT TIME ZONE 'America/Denver' FROM t;   -- column: t.naive

An unaliased projection changes name for the aware case. The new name never names America/Denver, so it misleads the reader. PostgreSQL names the column timezone. This belongs in the user-facing list.

Claims I verified as correct

Zero changed expectations, and the double edge. I grepped the whole tree. AT TIME ZONE appears in exactly one test file, datafusion/sqllogictest/test_files/datetime/timestamps.slt. Every use on main has a Utf8 or naive inner expression. The two date_bin(interval '1 day', to_local_time(column1)) AT TIME ZONE ... cases qualify because to_local_time returns Timestamp(unit, None). The sqllogictest diff has zero deleted lines across 507 .slt files. (The description says 505.)

The double edge is the part the description must own: the corpus had no coverage of the broken case at all. That is why the bug survived from #9647 in DataFusion 37. "Zero changed expectations" is weak evidence of safety here, not strong. #25175 adds the coverage that main lacks, and whichever PR merges second must update the other.

The to_local_time doc idiom. '2024-04-01T00:00:20Z'::timestamp types as Timestamp(ns, None), so the examples take the unchanged branch. Correct.

But the added sentence is backwards. In that idiom to_local_time is a round trip: to_local_time(<naive> AT TIME ZONE 'X') returns the same wall clock as <naive>. The doc's own first two examples both print 2024-04-01T00:00:20, which shows it. The new text calls the other form redundant and then sends the reader to the round-trip form. Say instead why to_local_time still earns its place: it strips the timezone of a column whose timezone you cannot name in SQL, and AT TIME ZONE needs a literal target.

The TimeUnit change. Safe, and I checked each step:

Coercion::new_exact(TypeSignatureClass::Timestamp) keeps the unit and the timezone. default_casted_type returns origin_type for a timestamp.
to_local_time handles all four units, for both the scalar and the array path.
Measured: Timestamp(µs, "UTC") gives Timestamp(µs), Timestamp(s, "UTC") gives Timestamp(s), and both values are 2024-01-01T05:00:00.
PostgreSQL also keeps the precision of the input, so this closes a real gap.

It is in scope in spirit, but it is a second type change in a PR whose title names one. Give it its own bullet in the user-facing list.

Spark. No Spark behaviour changes. SparkFunctionPlanner implements plan_extract and plan_substring only, so plan_at_time_zone falls through to DatetimeFunctionPlanner. with_spark_features inserts the Spark planner at index 0 of the list that with_default_features builds, and the doc on that trait requires that order. The description's own wording is in tension though: it says a Spark session does get the new semantics, then says no Spark-compat behaviour changes. Say "no Spark-specific function or planner behaviour changes".

Other checks that came back clean:

AT LOCAL: sqlparser 0.62.0 has no AtLocal node. Nothing to route.
No unparser or proto path emits AT TIME ZONE, so there is no round-trip risk.
An untyped placeholder types as DataType::Null and takes the naive branch. No new error.
now() is naive by default and aware only when datafusion.execution.time_zone is set. The .slt comment is right, and the SET is necessary for that case.
DatetimeFunctionPlanner builds the UDF value directly, so an unregistered to_local_time name does not break it.
A bad timezone still errors clearly: Invalid timezone "Not/AZone".
All six DST instants match PostgreSQL 17.11 and DuckDB 1.5.2 exactly, on the value.

One old bug I hit while I measured

A naive input that lands in a DST gap or a DST overlap errors:

SELECT '2024-03-10 02:30:00'::timestamp AT TIME ZONE 'America/Denver';
-- Arrow error: Cast error: Cannot cast timezone to different timezone
SELECT '2024-11-03 01:30:00'::timestamp AT TIME ZONE 'America/Denver';
-- same error

PostgreSQL returns 2024-03-10 09:30:00+00 and 2024-11-03 08:30:00+00. DuckDB returns the same two values. arrow_cast alone reproduces the error, so it comes from the arrow cast and this PR does not change it. It is out of scope, but it deserves its own issue.

🤖 Generated with Claude Code

Review of this PR found that the fix does not reach every expression. The type branch calls `Expr::get_type` in the SQL planner, which runs before the type coercion analyzer. For a `CASE`, `get_type` reports the first non-null `THEN` arm and ignores coercion. So these two have the same coerced input type, `Timestamp(ns, "UTC")`, and give different answers: CASE WHEN b THEN naive ELSE aware END AT TIME ZONE 'America/Denver' -> Timestamp(ns, "America/Denver") 2024-01-01T05:00:00-07:00 CASE WHEN b THEN aware ELSE naive END AT TIME ZONE 'America/Denver' -> Timestamp(ns) 2024-01-01T05:00:00 PostgreSQL gives the naive `2024-01-01 05:00:00` for both. `coalesce` is not affected, because `verify_function_arguments` coerces before the planner sees the type. A correct fix has to dispatch after coercion, which is a larger change than this PR. Pinning the behaviour so it is visible rather than silent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

adriangb and others added 3 commits September 10, 2026 12:10

adriangb added the api change Changes the API exposed to users of the crate label Sep 10, 2026

This was referenced Sep 10, 2026

AT TIME ZONE '+05:30' uses the opposite sign convention from PostgreSQL #25170

Open

test: characterization suite for timestamps with time zone #25175

Open

Navigation Menu

Uh oh!

fix: `AT TIME ZONE` on a timezone-aware timestamp returns a naive timestamp - #25165

fix: `AT TIME ZONE` on a timezone-aware timestamp returns a naive timestamp#25165
adriangb wants to merge 4 commits into
apache:mainfrom
pydantic:fix-at-time-zone-on-tz-aware-timestamps

adriangb commented Sep 10, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Sep 10, 2026 •

edited

Loading

Uh oh!

adriangb commented Sep 10, 2026

Uh oh!

adriangb commented Sep 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

adriangb commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

Start here: what a user gets today

Overview

Why the current lowering is wrong

What changes are included in this PR?

The planner hook

The TimeUnit change

Not changed

What is the testing strategy for this PR?

Field research

The four core cases

Both 2024 DST transitions in America/Denver, aware input

The DST gap and overlap from the other side, naive input

The case the two engines cannot arbitrate: a fixed-offset string

Tests

Are there any user-facing changes?

Uh oh!

codecov-commenter commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

adriangb commented Sep 10, 2026

Uh oh!

adriangb commented Sep 10, 2026

1. The type branch runs before type coercion, so the fix misses some inputs

2. The description says PostgreSQL and DuckDB agree on every case. They do not

3. The ExprPlanner hook is narrower than the description says

4. A dictionary-encoded aware timestamp keeps the old behaviour

5. The output column name changes, and the new name drops the timezone

Claims I verified as correct

One old bug I hit while I measured

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

adriangb commented Sep 10, 2026 •

edited

Loading

The `TimeUnit` change

Both 2024 DST transitions in `America/Denver`, aware input

codecov-commenter commented Sep 10, 2026 •

edited

Loading

3. The `ExprPlanner` hook is narrower than the description says