fix: accept all timestamp precisions in generate_series/range

[adriangb]

Which issue does this PR close?

• Closes generate_series over timestamps accepts only nanosecond precision, with a misleading error #25169

Rationale for this change

What a user hits

Build a timestamp series from second-precision bounds. Second and millisecond precision are common in Parquet, so this is what a series over a column read from storage looks like.

SELECT * FROM generate_series(
  arrow_cast(TIMESTAMP '2024-01-01 00:00:00', 'Timestamp(Second, None)'),
  arrow_cast(TIMESTAMP '2024-01-03 00:00:00', 'Timestamp(Second, None)'),
  INTERVAL '1 day');

On main the call fails, and the message says the argument is not a timestamp while it prints a timestamp:

Error: Error during planning: First argument must be a timestamp or NULL,
got Literal(TimestampSecond(1704067200, None), None)

With this PR the same call works:

+---------------------+
| value               |
+---------------------+
| 2024-01-01T00:00:00 |
| 2024-01-02T00:00:00 |
| 2024-01-03T00:00:00 |
+---------------------+

What changes for a user

generate_series and range accept all four timestamp precisions, not nanoseconds alone. The same query written with a second-precision or millisecond-precision bound now returns a series instead of an error. Queries that worked before keep working, and the result type does not change.

Error messages also name the type they rejected, so a reader can act on them.

The technical detail

generate_series and range over timestamps accepted Timestamp(Nanosecond, _) alone. Any other TimeUnit failed. The dispatch in call_with_args already routed every DataType::Timestamp(_, _) into call_timestamp, and call_timestamp then hard-matched ScalarValue::TimestampNanosecond. So a coarser bound reached a match arm that could not accept it, and fell through to a message about the wrong thing.

There is no design question about the result precision. The output schema is already fixed at Timestamp(Nanosecond, tz), so the function already produced nanoseconds. It refused only coarser inputs.

Field research

Both reference engines accept coarser bounds, so this change moves DataFusion toward them. I measured both rather than quote their documentation.

PostgreSQL 17.11 (postgres:17):

=> SELECT pg_typeof(g), g FROM generate_series('2024-01-01'::timestamp(0), '2024-01-03'::timestamp(0), INTERVAL '1 day') g;
          pg_typeof          |          g
-----------------------------+---------------------
 timestamp without time zone | 2024-01-01 00:00:00
 timestamp without time zone | 2024-01-02 00:00:00
 timestamp without time zone | 2024-01-03 00:00:00
(3 rows)

Mixed precisions also work there:

=> SELECT g FROM generate_series('2024-01-01'::timestamp(0), '2024-01-03'::timestamp(3), INTERVAL '1 day') g;   -- 3 rows

DuckDB 1.5.2:

D SELECT * FROM generate_series(TIMESTAMP_S '2024-01-01', TIMESTAMP_S '2024-01-03', INTERVAL 1 DAY);    -- 3 rows, type TIMESTAMP
D SELECT * FROM generate_series(TIMESTAMP_MS '2024-01-01', TIMESTAMP_MS '2024-01-03', INTERVAL 1 DAY);  -- 3 rows
D SELECT * FROM generate_series(TIMESTAMP_NS '2024-01-01', TIMESTAMP_NS '2024-01-03', INTERVAL 1 DAY);  -- 3 rows
D SELECT * FROM range(TIMESTAMP_S '2024-01-01', TIMESTAMP_S '2024-01-03', INTERVAL 1 DAY);              -- 2 rows
D SELECT * FROM generate_series(TIMESTAMP_S '2024-01-01', TIMESTAMP_MS '2024-01-03', INTERVAL 1 DAY);   -- 3 rows

Two points carry over from both engines:

• Neither engine rejects a coarser bound.
• Neither engine lets the bound's unit reach the result type. PostgreSQL returns timestamp, DuckDB returns TIMESTAMP, and this PR keeps Timestamp(Nanosecond, tz).

Neither engine can arbitrate the timezone question below. Both store a timestamptz as an instant with no per-value zone, so both render the series in the session zone. Arrow puts the zone in the value, so DataFusion must pick one, and no cross-engine comparison is meaningful there.

What changes are included in this PR?

Both timestamp bounds in call_timestamp, which generate_series and range share, go through a new timestamp_arg_to_nanos helper instead of a hard match on ScalarValue::TimestampNanosecond.

• All four TimeUnits are accepted and widened to nanoseconds. The output stays Timestamp(Nanosecond, tz).

• Overflow is checked. Second, Millisecond and Microsecond span far more than an i64 of nanoseconds, which covers roughly 1677 to 2262. The widening is a checked_mul, so an out-of-range bound is a plan error that names the argument, the value and the window:

  First argument for generate_series is out of range of nanosecond timestamps:
  9223372036854775807 (Timestamp(Second, None)) is outside
  1677-09-21T00:12:43.145224192 to 2262-04-11T23:47:16.854775807
  

  An unchecked multiply wraps in silence in a release build and emits a plausible but wrong series. No panic was reachable from SQL through this path before: call_timestamp did no arithmetic on the bounds, and the DATE overload already used checked_mul. The new multiplication is checked so that stays true now that coarser units get through.

• Mixed precisions work. The two bounds are read independently, so generate_series(ts_second, ts_micro, INTERVAL '1 day') is fine. A timestamp denotes an instant whatever unit holds it, and both sides reach the same nanosecond scale before the comparison.

• Timezone handling does not change, but a comment now explains it. The output timezone still comes from the start argument. Two different timezones on the two bounds are deliberately not an error: an Arrow timezone changes how an instant is rendered, not which instant it is. The start's zone is the one kept because it also anchors the calendar arithmetic that advances the series. Month and day steps apply in local time, so they follow that zone's DST rules.

• Error messages name the type they rejected instead of a dump of the whole Expr. This covers the second and third arguments, and the three arguments of the DATE overload. A non-literal argument gets a separate "must be a literal ..." message.

Two limits stay, and both are worth a mention.

• The step argument still accepts Interval(MonthDayNano) alone. That is not reachable as a limit from SQL, because arrow_cast to Interval(DayTime) and Interval(YearMonth) is itself unimplemented. So it is left alone rather than widened on speculation.
• A DATE bound and a TIMESTAMP bound still cannot be mixed. PostgreSQL and DuckDB both accept that mix. It is out of scope for a precision fix, and the new message at least names the type now.

What is the testing strategy for this PR?

Unit tests in datafusion/functions-table/src/generate_series.rs:

• timestamp_arg_accepts_all_time_units — all four units widen to the same instant and keep their timezone
• timestamp_arg_keeps_naive_timestamps_naive
• timestamp_arg_handles_nulls — a NULL of each precision, plus an untyped NULL
• timestamp_arg_overflow_boundary — for Second, Millisecond and Microsecond, the largest and smallest representable values pass, and the first value past each one fails with a message that names the value and the window
• call_timestamp_accepts_mixed_precisions
• call_timestamp_range_accepts_non_nanosecond_precision — covers the range sibling
• call_timestamp_takes_timezone_from_start
• call_timestamp_null_bound_is_empty_series
• call_timestamp_reports_out_of_range_bound
• call_timestamp_rejects_non_timestamp_bound

sqllogictest coverage grows in datafusion/sqllogictest/test_files/table_functions.slt, in a new "Timestamp precision" section beside the existing timestamp-range tests:

• all four units, for generate_series and for range
• a sub-second step over second-precision bounds
• mixed-precision bounds in both directions
• arrow_typeof assertions that the output is Timestamp(Nanosecond, tz) whatever the input unit
• timezone-aware bounds, and bounds with two different timezones
• NULL bounds at non-nanosecond precision
• the overflow boundary in both directions, plus the new message text

Both suites pass:

cargo test -p datafusion-functions-table                     # 13 passed
cargo test -p datafusion-sqllogictest --test sqllogictests   # 505/505 files
cargo clippy --all-targets -- -D warnings                    # clean

Are there any user-facing changes?

Yes. Both of them widen behaviour rather than break it.

• generate_series and range accept Timestamp(Second|Millisecond|Microsecond, _) bounds, which errored before. Queries that worked before keep working, and the result type does not change.
• Several plan error messages are reworded to name the rejected data type. There are no public API changes.

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

❌ Patch coverage is 92.77567% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.91%. Comparing base (da89c7c) to head (aeb4122).
⚠️ Report is 140 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/functions-table/src/generate_series.rs	92.77%	16 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #25173      +/-   ##
==========================================
+ Coverage   81.60%   81.91%   +0.31%     
==========================================
  Files        1123     1132       +9     
  Lines      408898   420823   +11925     
  Branches   408898   420823   +11925     
==========================================
+ Hits       333670   344722   +11052     
- Misses      55625    55781     +156     
- Partials    19603    20320     +717     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[adriangb]

Self-review QA pass on my own PR. I built datafusion-cli from this branch and from main, and I tested the boundaries myself instead of trust in the tests in the diff. The core change is correct. One finding is substantive.

1. The misleading-error family is not closed, and the PR body claims it is

This is the same defect shape that #25169 names: an error that names the wrong type and dumps a raw Expr. On this branch:

> SELECT * FROM generate_series(NULL, TIMESTAMP '2024-01-03', INTERVAL '1 day');
Error: Error during planning: Argument #2 must be an INTEGER or NULL, got Literal(TimestampNanosecond(1704240000000000000, None), None)

The cause is the dispatch in call_with_args, which looks at exprs[0] alone. An untyped NULL first argument routes to call_int64, so call_timestamp never runs and none of the new messages apply.

Both reference engines return an empty series here, so an error is also the wrong answer:

-- PostgreSQL 17.11
=> SELECT * FROM generate_series(NULL, TIMESTAMP '2024-01-03', INTERVAL '1 day');
(0 rows)

-- DuckDB 1.5.2
D SELECT * FROM generate_series(NULL, TIMESTAMP '2024-01-03', INTERVAL 1 DAY);
0 rows

The behaviour is pre-existing, and a precision fix does not have to fix it. But the PR body says error messages now "name the offending type", and this path proves that untrue for the function the PR is about. Pick one:

• widen the dispatch to look at the first argument that is not an untyped NULL, or
• state the gap in the body and file it.

2. The ScalarValue::Null arm is dead for the first argument

timestamp_arg_to_nanos opens with:

ScalarValue::Null => return Ok((None, None)),

Finding 1 explains why no SQL reaches that arm for exprs[0]. It is reachable for exprs[1] only. The unit test timestamp_arg_handles_nulls calls the helper directly with "First argument", so it reports coverage that no query can exercise. Change the test to "Second argument" for that case, or add a comment that says the arm is defensive.

3. The mixed-zone slt case covers one order only

The slt has a start in +05:00 with an end in America/New_York. Swap the two and the row count changes:

> SELECT arrow_typeof(value), value FROM generate_series(
    arrow_cast(TIMESTAMP '2024-01-01T00:00:00Z','Timestamp(Microsecond, Some("America/New_York"))'),
    arrow_cast(TIMESTAMP '2024-01-03T00:00:00Z','Timestamp(Second, Some("+05:00"))'),
    INTERVAL '1 day');
+-----------------------------------+---------------------------+
| Timestamp(ns, "America/New_York") | 2024-01-01T00:00:00-05:00 |
| Timestamp(ns, "America/New_York") | 2024-01-02T00:00:00-05:00 |
+-----------------------------------+---------------------------+

Two rows, against three in the direction the slt covers. That is correct. The two bounds denote different instants once the zones swap, and the comparison is on instants. It is also the single clearest proof of the claim in the new comment, so it earns a place in the file next to the case that is already there.

4. Cross-type bounds still fail, and both reference engines accept them

Adjacent to the change, and worth one line in the body as a stated non-goal:

> SELECT * FROM generate_series(arrow_cast(TIMESTAMP '2024-01-01','Timestamp(Second, None)'), DATE '2024-01-03', INTERVAL '1 day');
Error: Error during planning: Second argument for generate_series must be a TIMESTAMP or NULL, got Date32

PostgreSQL and DuckDB both return the three-row series for the same call. The new message is at least accurate now, which the old one was not.

5. Nit: the comment at the discard site is nine lines

The block that explains _end_tz is good content in a place that splits call_timestamp in half. Two lines plus a pointer to the helper doc would read better in a function a reviewer must follow end to end.

What I checked, and it is correct

The overflow guard is load-bearing, and the bounds are exact. Without checked_mul a release build wraps in silence and emits a plausible but wrong series. I tested each coarse unit at its true boundary in both directions, and one unit past it:

> SELECT * FROM generate_series(arrow_cast(9223372036,'Timestamp(Second, None)'), arrow_cast(9223372036,'Timestamp(Second, None)'), INTERVAL '1 day');
2262-04-11T23:47:16

> SELECT * FROM generate_series(arrow_cast(9223372037,'Timestamp(Second, None)'), ...);
Error: Error during planning: First argument for generate_series is out of range of nanosecond timestamps: 9223372037 (Timestamp(Second, None)) is outside 1677-09-21T00:12:43.145224192 to 2262-04-11T23:47:16.854775807

> SELECT * FROM generate_series(arrow_cast(-9223372036854,'Timestamp(Millisecond, None)'), ...);
1677-09-21T00:12:43.146

> SELECT * FROM generate_series(arrow_cast(-9223372036855,'Timestamp(Millisecond, None)'), ...);
Error: ... -9223372036855 (Timestamp(Millisecond, None)) is outside ...

Rust truncates integer division toward zero, so i64::MIN / nanos_per_unit is the true minimum and not one unit short of it. I checked that trap and the test gets it right.

NANOS_RANGE_MIN and NANOS_RANGE_MAX are hard-coded strings, so I recomputed them from i64::MIN and i64::MAX. Both are exact to the nanosecond. The advertised window is also tight: -9223372036855 milliseconds renders as 1677-09-21T00:12:43.145, which is below .145224192, so the message never rejects a value that renders inside its own range.

The series advance was already safe. TimestampValue::advance and advance_with_end both handle the None from add_month_day_nano, so the new multiplication is the only unchecked arithmetic the wider inputs could reach. The body's claim that no panic was reachable before holds: call_timestamp did no arithmetic on the bounds, and call_date already used checked_mul.

range shares the path. RangeFunc::call_with_args builds a GenerateSeriesFuncImpl { name: "range", include_end: false } and delegates, so it is the same code. Confirmed end to end, including the error text:

> SELECT * FROM range(arrow_cast(TIMESTAMP '2024-01-01','Timestamp(Second, None)'), arrow_cast(TIMESTAMP '2024-01-03','Timestamp(Second, None)'), INTERVAL '1 day');
2024-01-01T00:00:00
2024-01-02T00:00:00

> SELECT * FROM range(arrow_cast(9223372036854775807,'Timestamp(Microsecond, None)'), ...);
Error: Error during planning: First argument for range is out of range of nanosecond timestamps: ...

Mixed precisions work, and the start's zone is the right choice. The two bounds compare as instants, so a unit difference cannot change the answer. The start's zone anchors the month and day arithmetic, and I confirmed it follows DST in that zone:

> SELECT value FROM generate_series(
    arrow_cast(TIMESTAMP '2024-03-09T00:00:00','Timestamp(Second, Some("America/Denver"))'),
    arrow_cast(TIMESTAMP '2024-03-12T00:00:00','Timestamp(Second, Some("America/Denver"))'),
    INTERVAL '1 day');
2024-03-09T00:00:00-07:00
2024-03-10T00:00:00-07:00
2024-03-11T00:00:00-06:00
2024-03-12T00:00:00-06:00

Local midnight holds across the transition. The end's zone carries no information the comparison can use, so the discard is right and no rejection is needed.

The error messages are accurate now. On main the same calls give:

Error: Error during planning: First argument must be a timestamp or NULL, got Literal(TimestampSecond(1704067200, None), None)
Error: Error during planning: Second argument must be a date or NULL, got Literal(Int64(5), None)

On this branch:

Error: Error during planning: Second argument for generate_series must be a TIMESTAMP or NULL, got Int64
Error: Error during planning: Third argument for generate_series must be an INTERVAL or NULL, got Utf8
Error: Error during planning: Second argument for generate_series must be a DATE or NULL, got Int64
Error: Error during planning: Second argument for generate_series must be a literal TIMESTAMP or NULL, got (<subquery>)

The step restriction really is unreachable, as the body claims:

> SELECT arrow_cast(INTERVAL '1 day', 'Interval(DayTime)');
Error: This feature is not implemented: Unsupported CAST from Interval(MonthDayNano) to Interval(DayTime)

Field research supports the change. Both engines accept coarser bounds and produce the series:

-- PostgreSQL 17.11
=> SELECT pg_typeof(g), g FROM generate_series('2024-01-01'::timestamp(0), '2024-01-03'::timestamp(0), INTERVAL '1 day') g;
 timestamp without time zone | 2024-01-01 00:00:00   (3 rows)

-- DuckDB 1.5.2
D SELECT * FROM generate_series(TIMESTAMP_S '2024-01-01', TIMESTAMP_S '2024-01-03', INTERVAL 1 DAY);   -- 3 rows, TIMESTAMP
D SELECT * FROM range(TIMESTAMP_S '2024-01-01', TIMESTAMP_S '2024-01-03', INTERVAL 1 DAY);             -- 2 rows
D SELECT * FROM generate_series(TIMESTAMP_S '2024-01-01', TIMESTAMP_MS '2024-01-03', INTERVAL 1 DAY);  -- 3 rows, mixed units

Neither engine rejects a coarser input, and neither one lets the input unit reach the result type. That is the shape this PR adopts.

Suites are green on this branch: cargo test -p datafusion-functions-table (13 passed), the table_functions sqllogictest file, and cargo clippy --all-targets -- -D warnings.