Describe the bug
Since #19078, the to_timestamp / to_timestamp_seconds / to_timestamp_millis / to_timestamp_micros / to_timestamp_nanos functions override the timezone of an input that is already timezone-aware with the session's datafusion.execution.time_zone.
When that config is unset (the default None), an input of type Timestamp(_, Some(tz)) is returned as Timestamp(_, None). The zone annotation is dropped while the underlying epoch is kept, which silently changes the wall-clock meaning of the value.
Previously, these functions had a dedicated match arm that preserved a zoned input's timezone:
// before #19078
Timestamp(_, Some(tz)) => args[0].cast_to(&Timestamp(Millisecond, Some(tz)), None),
#19078 collapsed that arm into the generic one, which casts to the execution timezone:
// after #19078 (current main)
Null | Int32 | Int64 | Timestamp(_, _) => args[0].cast_to(&Timestamp(Millisecond, self.timezone.clone()), None),
The PR's rationale when I read it, only concerns naive timestamp strings being interpreted in the execution timezone, which is reasonable. Dropping the timezone of inputs that are already zoned appears to be an unintended side effectas it makes a normalisation function silently discard a caller's explicit timezone.
To Reproduce
Run in datafusion-cli with default settings (no SET datafusion.execution.time_zone):
-- input epoch = 2024-01-01T00:00:00Z, annotated +05:00 => local wall clock 05:00
SELECT
arrow_typeof(to_timestamp_millis(arrow_cast(1704067200000, 'Timestamp(Millisecond, Some("+05:00"))'))) AS result_type,
date_part('hour', arrow_cast(1704067200000, 'Timestamp(Millisecond, Some("+05:00"))')) AS hour_direct,
date_part('hour', to_timestamp_millis(arrow_cast(1704067200000, 'Timestamp(Millisecond, Some("+05:00"))'))) AS hour_via_to_timestamp;
Actual
| result_type |
hour_direct |
hour_via_to_timestamp |
Timestamp(Millisecond, None) |
5 |
0 |
Befor the change
| result_type |
hour_direct |
hour_via_to_timestamp |
Timestamp(Millisecond, Some("+05:00")) |
5 |
5 |
Expected behavior
to_timestamp_millis (and the rest of the family) should not silently drop the timezone of an already-zoned input. Passing a Timestamp(_, Some("+05:00")) through the function should either:
- preserve the input zone, or
- if honoring
datafusion.execution.time_zone, convert the timezone but never relabel to a naive type, which changes the wall-clock value.
In particular, when datafusion.execution.time_zone is unset (None), a zoned input should keep its zone rather than becoming naive. As it stands, date_part('hour', to_timestamp_millis(<zoned column>)) returns the UTC hour instead of the local hour, which corrupts any hour-of-day / day-of-month bucketing over zoned columns.
Additional context
Relevant commit: ada0923a3927d16dc5d810637cfbea4146c54f54 (PR #19078 that closed #17998)
Describe the bug
Since #19078, the
to_timestamp/to_timestamp_seconds/to_timestamp_millis/to_timestamp_micros/to_timestamp_nanosfunctions override the timezone of an input that is already timezone-aware with the session'sdatafusion.execution.time_zone.When that config is unset (the default
None), an input of typeTimestamp(_, Some(tz))is returned asTimestamp(_, None). The zone annotation is dropped while the underlying epoch is kept, which silently changes the wall-clock meaning of the value.Previously, these functions had a dedicated match arm that preserved a zoned input's timezone:
#19078 collapsed that arm into the generic one, which casts to the execution timezone:
The PR's rationale when I read it, only concerns naive timestamp strings being interpreted in the execution timezone, which is reasonable. Dropping the timezone of inputs that are already zoned appears to be an unintended side effectas it makes a normalisation function silently discard a caller's explicit timezone.
To Reproduce
Run in
datafusion-cliwith default settings (noSET datafusion.execution.time_zone):Actual
Timestamp(Millisecond, None)Befor the change
Timestamp(Millisecond, Some("+05:00"))Expected behavior
to_timestamp_millis(and the rest of the family) should not silently drop the timezone of an already-zoned input. Passing aTimestamp(_, Some("+05:00"))through the function should either:datafusion.execution.time_zone, convert the timezone but never relabel to a naive type, which changes the wall-clock value.In particular, when
datafusion.execution.time_zoneis unset (None), a zoned input should keep its zone rather than becoming naive. As it stands,date_part('hour', to_timestamp_millis(<zoned column>))returns the UTC hour instead of the local hour, which corrupts any hour-of-day / day-of-month bucketing over zoned columns.Additional context
Relevant commit:
ada0923a3927d16dc5d810637cfbea4146c54f54(PR #19078 that closed #17998)