Skip to content

[Bug] Fix date_bin month bins counting intra-second milliseconds twice - #19267

Open
DaZuiZui wants to merge 2 commits into
apache:masterfrom
DaZuiZui:fix/date-bin-month-subsecond-origin
Open

DaZuiZui wants to merge 2 commits into
apache:masterfrom
DaZuiZui:fix/date-bin-month-subsecond-origin

Conversation

@DaZuiZui

@DaZuiZui DaZuiZui commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Description

Fixes #19264.

DateBinFunctionColumnTransformer.convertToTimestamp computed the result as
toEpochMilli() + getNano(). toEpochMilli() already contains the whole
milliseconds of the second, and getNano() is the full nanosecond-of-second
(0–999_999_999), not the remainder below one millisecond. The intra-second
milliseconds were therefore counted twice, so month/year bins started that many
milliseconds late. For example date_bin(1mo, time, 500) returned 1000 for a
source of 500 and 2678401000 for 2678400500.

The fix converts from Instant.getEpochSecond() and Instant.getNano(), so each
part of the instant is counted exactly once, for ms, us and ns precision.

Hour and day bins do not use this helper and are unaffected. Month bins with a
whole-second origin are also unaffected because getNano() is 0 for them.

How to reproduce

CREATE DATABASE testdb;
USE testdb;
CREATE TABLE t (device_id STRING TAG, s INT32 FIELD);
INSERT INTO t (time, device_id, s) VALUES
  (500, 'd1', 1),
  (2678400500, 'd1', 2);
SELECT time, date_bin(1mo, time, 500) FROM t ORDER BY time;

Before: 1000 and 2678401000. After: 500 and 2678400500.

Tests

Added cases in DateBinFunctionTest for the issue's scenario, a 2024 origin,
dateBinStartEnd, nextDateBin, and us/ns precision with a sub-second
origin.

Tests run: 12, Failures: 0, Errors: 0, Skipped: 0

🤖 Generated with Claude Code

DateBinFunctionColumnTransformer.convertToTimestamp added
Instant.toEpochMilli() and LocalDateTime.getNano() together. The former
already contains the whole milliseconds of the second and the latter is
the full nanosecond-of-second, so the milliseconds were counted twice and
month/year bins started that many milliseconds late. Convert from
getEpochSecond() and getNano() instead so each part is counted once.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@JackieTien97 JackieTien97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix. The root cause is right: toEpochMilli() already includes the milliseconds of the second, so adding getNano() on top counted them twice. Converting from getEpochSecond() + getNano() fixes it for ms, us and ns.

I compared the old and new conversion against an exact BigInteger reference, including pre-1970 instants (where the old code was off by even more), and ran DateBinFunctionTest on 2bca3ad: 12/12 pass. I also agree with your note on #19264: Instant.ofEpochSecond normalizes a negative nano adjustment, so convertToLocalDateTime doesn't throw for negative timestamps.

Two inline comments: an overflow edge case in the rewritten lines, and a suggested pre-1970 test.

One more, not tied to a diff line: the commit message ends with Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>, but the root AGENTS.md says "Do NOT add Co-Authored-By trailer to commit messages." Could you drop it from the commit? It can also be removed when squash-merging.

TimeUnit.SECONDS.toMillis, toMicros and toNanos already saturate, but the
sub-second part was added with an unchecked add, so a month bin past the
long range wrapped to a large negative value. Use saturatedAdd so the bin
end stays at the long bound.

Also cover pre-1970 month bins with a sub-second origin, where
getEpochSecond() rounds down and getNano() stays non-negative.

@JackieTien97 JackieTien97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick update. Both inline comments are addressed in 755a66d: the conversion now saturates at the long bounds instead of wrapping, and the pre-1970 cases are covered.

I re-ran it locally on 755a66d: spotless passes, DateBinFunctionTest passes 13/13, and the probe cases past the long range (ms and ns) now return Long.MAX_VALUE for both the bin end and nextDateBin.

One leftover for the merge: the first commit (2bca3ad) still carries the Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> trailer, which AGENTS.md asks not to add, so please drop it from the squash commit message.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] date_bin month bins add the origin's intra-second milliseconds twice

2 participants