Repository navigation
[To dev/1.3] Fix microsecond conversion of datetimes before 1970 with a fractional second (#19284) - #19290
Open
JackieTien97 wants to merge 1 commit into
Conversation
… second With timestamp_precision=us, DateTimeUtils.getInstantWithPrecision returned a value 999999 us too large for an instant before the epoch that has a fractional second. Like Instant#toEpochMilli, the branch for such instants multiplies (epochSecond + 1), but it then subtracted 1 us instead of the extra second (1_000_000 us). For example, the literal 1969-12-31T23:59:59.5+00:00 was written and queried as 499999 instead of -500000. getInstantWithPrecisionTest did not call getInstantWithPrecision (its body was a copy of convertDurationStrToLongTest3). It now checks the ms/us/ns branches, and two new tests cover fractional seconds before the epoch, directly and through convertDatetimeStrToLong. Backport of apache#19284 (issue apache#19283) to dev/1.3: - DateTimeUtils is org.apache.iotdb.db.utils.DateTimeUtils (datanode) on this branch, and getInstantWithPrecision keeps its try/catch. The fix is the same one-line change with the same comment. - The two test assertions that pass "microsecond" as the precision are not ported. On dev/1.3, getInstantWithPrecision accepts only "us" (the "microsecond" alias came with the table model on master), so they return milliseconds and fail with or without the fix. The same inputs are asserted with "us". (cherry picked from commit 8e66bf6)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backport of #19284 (issue #19283) to
dev/1.3.With
timestamp_precision=us,DateTimeUtils.getInstantWithPrecisionconverted a datetime before1970-01-01T00:00:00Zthat has a fractional second to a timestamp 999,999 us too large. For a negative instant with a fractional part, theusbranch followsInstant.toEpochMilli(): it multipliesepochSecond + 1and must then subtract that extra second, but it subtracted1instead of1_000_000(one second in microseconds). For example,1969-12-31T23:59:59.5+00:00became499999instead of-500000.On
dev/1.3, tree-model datetime literals reach this method throughASTVisitor.parseDateTimeFormatandDateTimeUtils.convertDatetimeStrToLong, using the configuredtimestamp_precision. Withusprecision this affects time values inINSERT ... VALUES, time filters such asWHERE time = 1969-12-31T23:59:59.5+00:00, and time ranges. #19284 shows the effect on a running 1C1D master server: the row was stored at1970-01-01T00:00:00.499999Z, and a filter on that literal missed a row stored at-500000.The fix subtracts
1_000_000L. Whole seconds, instants from 1970 onwards, and thems/nsbranches do not change. TheepochSecond + 1form stays, so the multiplication cannot overflow nearLong.MIN_VALUE, and a value outside thelongrange still throwsArithmeticException.Source and target
8e66bf61ae2ee97150255bf742b3cd2e02a0f6ae, cherry-picked with-x.dev/1.3ate986397f6a7bb33cffd81839654e012d8babfc0c.Adaptations for dev/1.3
DateTimeUtilsisorg.apache.iotdb.db.utils.DateTimeUtilsiniotdb-core/datanode, notorg.apache.iotdb.commons.queryengine.utils.DateTimeUtilsinnode-commons. The changed lines are the same as on master:- 1Lbecomes- 1_000_000L, with the same comment.getInstantWithPrecisionondev/1.3still wraps the conversion intry/catch (DateTimeParseException), and itsusbranch checks only"us". Both stay unchanged. The fix is inside that branch, so parse-error handling and the choice of branch for each precision string are the same as before.getInstantWithPrecisionTestand the two new tests are ported with the same inputs and expected values, except the two assertions that pass"microsecond"as the precision.dev/1.3has no"microsecond"alias (it came with the table model on master), so those calls take the millisecond branch with or without the fix. With the fix and the master tests unchanged, exactly those two assertions fail (expected:<1546413207689123> but was:<1546413207689>andexpected:<-500000> but was:<-500>). The neighbouring"us"assertions use the same inputs.Validation
mvn -o test -pl iotdb-core/datanode -am -Dtest=DateTimeUtilsTest \ -Dsurefire.failIfNoSpecifiedTests=false -Dspotless.check.skip=true -Dcheckstyle.skip=true -Drat.skip=trueWith
DateTimeUtils.javareverted todev/1.3(tests from this PR): 13 tests, 2 failures, bothexpected:<-500000> but was:<499999>:getInstantWithPrecisionBeforeEpochWithFractionTest, atgetInstantWithPrecision("1969-12-31T23:59:59.5+00:00", "us")convertDatetimeStrToLongBeforeEpochWithFractionTest, atconvertDatetimeStrToLong("1969-12-31T23:59:59.5Z", ZoneOffset.UTC, 0, "us")getInstantWithPrecisionTestpasses; it has no pre-1970 input with a fraction.With this commit: 13 tests, 0 failures, in both surefire executions of
iotdb-server(default-testandunit-tests).mvn spotless:apply -pl iotdb-core/datanodechanged nothing, andmvn spotless:check -pl iotdb-core/datanodepassed.Not run on a
dev/1.3server. The server-level reproduction is in Fix microsecond conversion of datetimes before 1970 with a fractional second #19284 on master.This PR has:
for code coverage.
Key changed/added classes (or packages if there are too many classes) in this PR
iotdb-core/datanode:org.apache.iotdb.db.utils.DateTimeUtilsiotdb-core/datanode(test):org.apache.iotdb.db.utils.DateTimeUtilsTest🤖 Generated with Claude Code