Skip to content

[To dev/1.3] Fix microsecond conversion of datetimes before 1970 with a fractional second (#19284) - #19290

Open
JackieTien97 wants to merge 1 commit into
apache:dev/1.3from
JackieTien97:fix-us-precision-pre-epoch-datetime-dev-1.3
Open

JackieTien97 wants to merge 1 commit into
apache:dev/1.3from
JackieTien97:fix-us-precision-pre-epoch-datetime-dev-1.3

Conversation

@JackieTien97

Copy link
Copy Markdown
Contributor

Description

Backport of #19284 (issue #19283) to dev/1.3.

With timestamp_precision=us, DateTimeUtils.getInstantWithPrecision converted a datetime before 1970-01-01T00:00:00Z that has a fractional second to a timestamp 999,999 us too large. For a negative instant with a fractional part, the us branch follows Instant.toEpochMilli(): it multiplies epochSecond + 1 and must then subtract that extra second, but it subtracted 1 instead of 1_000_000 (one second in microseconds). For example, 1969-12-31T23:59:59.5+00:00 became 499999 instead of -500000.

On dev/1.3, tree-model datetime literals reach this method through ASTVisitor.parseDateTimeFormat and DateTimeUtils.convertDatetimeStrToLong, using the configured timestamp_precision. With us precision this affects time values in INSERT ... VALUES, time filters such as WHERE time = 1969-12-31T23:59:59.5+00:00, and time ranges. #19284 shows the effect on a running 1C1D master server: the row was stored at 1970-01-01T00:00:00.499999Z, and a filter on that literal missed a row stored at -500000.

The fix subtracts 1_000_000L. Whole seconds, instants from 1970 onwards, and the ms/ns branches do not change. The epochSecond + 1 form stays, so the multiplication cannot overflow near Long.MIN_VALUE, and a value outside the long range still throws ArithmeticException.

Source and target

Adaptations for dev/1.3

  • On this branch DateTimeUtils is org.apache.iotdb.db.utils.DateTimeUtils in iotdb-core/datanode, not org.apache.iotdb.commons.queryengine.utils.DateTimeUtils in node-commons. The changed lines are the same as on master: - 1L becomes - 1_000_000L, with the same comment.
  • getInstantWithPrecision on dev/1.3 still wraps the conversion in try/catch (DateTimeParseException), and its us branch checks only "us". Both stay unchanged. The fix is inside that branch, so parse-error handling and the choice of branch for each precision string are the same as before.
  • Tests: getInstantWithPrecisionTest and the two new tests are ported with the same inputs and expected values, except the two assertions that pass "microsecond" as the precision. dev/1.3 has no "microsecond" alias (it came with the table model on master), so those calls take the millisecond branch with or without the fix. With the fix and the master tests unchanged, exactly those two assertions fail (expected:<1546413207689123> but was:<1546413207689> and expected:<-500000> but was:<-500>). The neighbouring "us" assertions use the same inputs.

Validation

mvn -o test -pl iotdb-core/datanode -am -Dtest=DateTimeUtilsTest \
  -Dsurefire.failIfNoSpecifiedTests=false -Dspotless.check.skip=true -Dcheckstyle.skip=true -Drat.skip=true
  • With DateTimeUtils.java reverted to dev/1.3 (tests from this PR): 13 tests, 2 failures, both expected:<-500000> but was:<499999>:

    • getInstantWithPrecisionBeforeEpochWithFractionTest, at getInstantWithPrecision("1969-12-31T23:59:59.5+00:00", "us")
    • convertDatetimeStrToLongBeforeEpochWithFractionTest, at convertDatetimeStrToLong("1969-12-31T23:59:59.5Z", ZoneOffset.UTC, 0, "us")

    getInstantWithPrecisionTest passes; it has no pre-1970 input with a fraction.

  • With this commit: 13 tests, 0 failures, in both surefire executions of iotdb-server (default-test and unit-tests).

  • mvn spotless:apply -pl iotdb-core/datanode changed nothing, and mvn spotless:check -pl iotdb-core/datanode passed.

  • Not run on a dev/1.3 server. The server-level reproduction is in Fix microsecond conversion of datetimes before 1970 with a fractional second #19284 on master.


This PR has:

  • been self-reviewed.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold
    for code coverage.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR
  • iotdb-core/datanode: org.apache.iotdb.db.utils.DateTimeUtils
  • iotdb-core/datanode (test): org.apache.iotdb.db.utils.DateTimeUtilsTest

🤖 Generated with Claude Code

… second

With timestamp_precision=us, DateTimeUtils.getInstantWithPrecision
returned a value 999999 us too large for an instant before the epoch
that has a fractional second. Like Instant#toEpochMilli, the branch for
such instants multiplies (epochSecond + 1), but it then subtracted 1 us
instead of the extra second (1_000_000 us). For example, the literal
1969-12-31T23:59:59.5+00:00 was written and queried as 499999 instead
of -500000.

getInstantWithPrecisionTest did not call getInstantWithPrecision (its
body was a copy of convertDurationStrToLongTest3). It now checks the
ms/us/ns branches, and two new tests cover fractional seconds before
the epoch, directly and through convertDatetimeStrToLong.

Backport of apache#19284 (issue apache#19283) to dev/1.3:
- DateTimeUtils is org.apache.iotdb.db.utils.DateTimeUtils (datanode)
  on this branch, and getInstantWithPrecision keeps its try/catch. The
  fix is the same one-line change with the same comment.
- The two test assertions that pass "microsecond" as the precision are
  not ported. On dev/1.3, getInstantWithPrecision accepts only "us"
  (the "microsecond" alias came with the table model on master), so
  they return milliseconds and fail with or without the fix. The same
  inputs are asserted with "us".

(cherry picked from commit 8e66bf6)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant