Skip to content

Fix library UDF edge cases - #17871

Draft
Caideyipi wants to merge 25 commits into
apache:masterfrom
Caideyipi:fix/library-udf-edge-cases
Draft

Caideyipi wants to merge 25 commits into
apache:masterfrom
Caideyipi:fix/library-udf-edge-cases

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Jun 9, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Fix validation, empty-input handling, invalid values, and instance-reuse state leaks across library UDF/UDAF implementations.
  • Preserve MAD aggregation/null semantics and sampled INT64 values when input Row objects are reused.
  • Correct DWT level handling and numerical/timestamp edge cases in Histogram, MinMax, Spline, AR, and moving averages.
  • Fix extreme-value median/MAD calculations and DTW aggregate-state merging, including interleaved partial states.
  • Defer merged DTW match rebuilding until final output and use rolling rows for DTW scratch space.
  • Add regression coverage with test fixtures that model reused input objects and valid access-window sizes.

Examples

  • Spline input timestamps and values 0,1,2,3,4 with points=4 previously emitted timestamps [0,1,3,4] but values approximately [0,1.3333,2.6667,4]. Values now use the emitted timestamps, producing [0,1,3,4]; the terminal value is also preserved when extreme timestamps share a double coordinate. With points=10 over the same timestamp range, it emits each of the five available timestamps once.
  • Histogram with min=0, max=100, and count=100 assigns input 29 to bucket 29. For the subnormal range 0..5*Double.MIN_VALUE with two buckets, inputs 2*Double.MIN_VALUE and 3*Double.MIN_VALUE now produce counts [1,1].
  • A moving average with window=2 and values [Double.MAX_VALUE,Double.MAX_VALUE,0,2,4] skips the overflowing window and recovers with values [Double.MAX_VALUE/2,1,3]. Compensated summation also preserves the average 1 once the large values leave [1e16,1e16,1,1].
  • Two Integer.MAX_VALUE or Double.MAX_VALUE samples return the correct finite median. The median of [Long.MIN_VALUE,Long.MAX_VALUE] is -0.5; adjacent large INT64 values retain MAD 0.5 by subtracting before conversion to double.
  • DTW pattern [1,2,3] over values [1,2,3,1,2,3], split after the first value, returns both windows [1,3] and [4,6]. Matches survive interleaved/reversed/grouped state merges and serialization. Merges accumulate input; final output rebuilds chronological matches when required.

Validation

  • mvn -o spotless:apply -P with-integration-tests -pl library-udf: passed.
  • mvn -o clean test -pl library-udf -am: 139 tests passed, no failures/errors/skips.
  • mvn -o clean test-compile -pl library-udf -am -P with-zh-locale -DskipTests: passed.
  • Lifecycle Checkstyle and git diff --check: passed.
  • Four numerical regressions were reproduced before their production fixes. All newly added tests were run locally.

Both clean reactors rebuilt the required IoTDB source dependency closure with local and remote build caches disabled (-Ddevelocity.cache.local.enabled=false -Ddevelocity.cache.remote.enabled=false). Validation used a local offline fallback because remote Maven plugin downloads were blocked. No full-project build or performance benchmark was run. Remote CI was not waited on or polled.

DTW retains effective input points to support arbitrary temporal merge ordering/interleaving. State memory and serialized state grow with input count plus matches. Merges no longer repeatedly recalculate historical windows; a pending chronological rebuild runs at final output. DTW scratch memory is linear in pattern length, while each window still requires quadratic computation in pattern length.

Caideyipi added 9 commits June 9, 2026 09:46
…rary-udf-edge-cases

# Conflicts:
#	library-udf/src/main/java/org/apache/iotdb/library/anomaly/UDTFKSigma.java
#	library-udf/src/main/java/org/apache/iotdb/library/anomaly/UDTFRange.java
#	library-udf/src/main/java/org/apache/iotdb/library/dprofile/UDAFPercentile.java
#	library-udf/src/main/java/org/apache/iotdb/library/dprofile/UDAFQuantile.java
#	library-udf/src/main/java/org/apache/iotdb/library/dprofile/UDAFSpread.java
#	library-udf/src/main/java/org/apache/iotdb/library/dprofile/UDTFDistinct.java
#	library-udf/src/main/java/org/apache/iotdb/library/dprofile/UDTFSample.java
#	library-udf/src/main/java/org/apache/iotdb/library/drepair/UDTFTimestampRepair.java
#	library-udf/src/main/java/org/apache/iotdb/library/drepair/UDTFValueFill.java
#	library-udf/src/main/java/org/apache/iotdb/library/drepair/UDTFValueRepair.java
#	library-udf/src/main/java/org/apache/iotdb/library/frequency/UDFEnvelopeAnalysis.java
#	library-udf/src/main/java/org/apache/iotdb/library/match/UDAFDTWMatch.java
#	library-udf/src/main/java/org/apache/iotdb/library/match/UDAFPatternMatch.java
@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Self-reviewed this PR. Fixed the quantile floating-point ordering issue involving negative zero; I did not find any other issues.

Resolve UDF validation conflicts while preserving edge-case checks and upstream localized messages.
@Caideyipi
Caideyipi marked this pull request as draft November 13, 2026 03:02
@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Independent re-review completed for 1b94fa945b8ae6c7f71db515177888e3aa664767 against master baseline cca1dbbde6d0dffa38495c92c92a48b2626d2f99, covering the full 84-file diff and relevant context.

Fixed in 063697ab5b89817786a987034b92d718f97e6d18:

  • P1 — Reused rows in isometric sampling: RowIterator.next() reuses one Row. Retaining those objects repeated the final row, or read a final null row. Sampling now retains indexes and reads each selected row immediately; the test iterator now follows the API's reuse contract. UDTFSample.java:182
  • P2 — INT64 precision in triangle sampling: endpoint/all-points branches converted unchanged values through double, e.g. 9007199254740993 became 9007199254740992. These branches now preserve typed values. UDTFSample.java:168
  • P2 — Excess DWT layers: Java's shift-distance masking made layer=33 reconstruct the wrong signal. Inversion now derives the actual forward levels from the signal and filter lengths. DWTUtil.java:247
  • P2 — Spline timestamp precision: converting absolute timestamps to double before subtraction collapsed nearby nanosecond timestamps and caused a non-monotonic-sequence exception. The integer difference is now calculated first. UDTFSpline.java:86
  • P2 — Pattern-test false positives: the result-builder stub fabricated a non-null position even without a write. Tests now use BinaryColumnBuilder and assert exactly one output position. UDAFPatternTest.java:281

The UDAFMad conflict resolution correctly preserves the upstream exception for degenerate sketch results, null/non-finite filtering, and empty-input handling. The UDAFMadTest proxy's isNull response is effective; both MAD tests pass. No further MAD changes were needed.

Validation:

  • All four behavioral regressions were reproduced before their fixes.
  • Offline clean reactor test (library-udf -am, build-cache reuse disabled): 111 passed, 0 failures, 0 errors, 0 skipped. All newly added tests were run locally.
  • Offline clean Chinese-locale reactor test-compile: passed.
  • Spotless apply with with-integration-tests, lifecycle Checkstyle, English compilation, en/zh message-key and format-specifier parity, and PR-diff whitespace checks: passed.

Final local commit, confirmed source-branch head, and confirmed PR head are all 063697ab5b89817786a987034b92d718f97e6d18. The update is complete and the PR remains Draft because this round found behavioral/test-logic issues; a fresh independent re-review is required. No CI waiting or polling was performed.

@Caideyipi

Caideyipi commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator Author

Independent re-review of PR #17871

Reviewed the requested starting head 063697ab5b89817786a987034b92d718f97e6d18 against baseline cca1dbbde6d0dffa38495c92c92a48b2626d2f99. I inspected the complete 84-file diff and relevant context, including MAD state merging and isNull, sampling object reuse and INT64 precision, DWT level calculations, spline timestamp handling, and test doubles.

Confirmed behavior issues found and fixed during this round in commits df4183ead2c54fa27c59420554da7e889f524ab6 and a8e5d9e1f70c240a74c67a7f9135cc5bbc3fb25b:

  • P2 — Triangle sampling lost INT64 precision for selected interior points. The selected value now goes through the typed value writer instead of double. UDTFSample.java:216
  • P2 — Histogram bucket placement was inaccurate for adjacent large bounds and unsafe across extreme or subnormal ranges. The normalized calculation now avoids loss of boundary precision and range overflow/underflow. UDTFHistogram.java:82
  • P2 — MinMax normalization could produce an incorrect result when finite endpoints made max - min overflow. A scaled fallback now preserves normalization across the double range. UDTFMinMax.java:130
  • P2 — Spline timestamp arithmetic could overflow or collapse distinct timestamps into the same double coordinate over a wide range; endpoint evaluation could also produce non-finite output. Coordinates and output timestamps now use overflow-safe arithmetic, and endpoint results are handled explicitly. UDTFSpline.java:94 UDTFSpline.java:134

Validation: the offline clean library-udf source-reactor test passed all 117 tests; Spotless, Checkstyle, clean English and Chinese test-compiles, message-constant/placeholder parity, and git diff --check passed. All added tests were run locally. Remote Maven resolution was blocked while downloading a Central plugin, so verification used the local offline fallback. CI was not polled.

Final local commit, source branch head, and PR head are a8e5d9e1f70c240a74c67a7f9135cc5bbc3fb25b. The PR remains Draft because this round found behavior issues; a fresh independent re-review is required.

@Caideyipi

Caideyipi commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator Author

Independent re-review: fixes completed, Draft retained

Completed one review round of the full diff and relevant context, including MAD merge/isNull, sampling Row reuse and INT64 precision, DWT levels, Histogram/MinMax extremes, Spline timestamps/coordinates/endpoints, and test fixtures.

  • Reviewed head: a8e5d9e1f70c240a74c67a7f9135cc5bbc3fb25b
  • Comparison base: cca1dbbde6d0dffa38495c92c92a48b2626d2f99
  • Final local commit: 033b55d1474c978dd1b4ee07138f337809ede990
  • Confirmed source branch and PR head: 033b55d1474c978dd1b4ee07138f337809ede990

Six behavior issues and one test-fixture issue were found and fixed:

  1. Spline output coordinates: for t=y=0..4, points=4, rounded timestamps [0,1,3,4] previously carried values [0,1.3333,2.6667,4]. Interpolation now evaluates at the emitted timestamps.
  2. Spline terminal coordinate collisions: timestamps near Long.MAX_VALUE can share a double coordinate across the full long range. The final input value is now retained instead of being discarded.
  3. AR small-amplitude inputs: the absolute epsilon cutoff suppressed valid coefficients, e.g. [1e-8,2e-8,3e-8] should yield 6/7. The guard now rejects only zero or non-finite denominators.
  4. DTW pattern validation: comma-only patterns could pass validation because split discarded empty elements. Empty elements, including trailing commas, are now rejected.
  5. Histogram boundaries: with [0,100] and 100 buckets, input 29 could fall into bucket 28. Division by bucket width now preserves this exact boundary, with the existing extreme-range scaling and a fallback when width underflows.
  6. Moving-average recovery: an overflowing sum permanently poisoned later finite windows. A non-finite sum is now rebuilt from the current window; non-finite results are skipped and subsequent finite windows recover.
  7. RowWindow fixture: getRow() returned independent objects despite the intended reuse coverage. It now exposes a shared mutable Row and actual data types, and sampling tests explicitly assert object reuse.

Additional baseline fixes included

The two baseline observations identified during the completed round are now repaired in 033b55d1474c978dd1b4ee07138f337809ede990, following the initial review fixes in c1b81b14a3041e5cdb8cce797ee5cd3e9c835d1f. This implementation follow-up did not start another independent review round.

  • Extreme medians and MAD: two Integer.MAX_VALUE or Long.MAX_VALUE samples previously produced median -1, and two maximal finite floating-point values produced infinity. INT32/FLOAT addition is widened, INT64 values are summed exactly before conversion, and DOUBLE operands are halved only when their sum overflows. Opposite long extremes produce -0.5, and subnormal medians retain their precision.
  • DTW aggregate merging: replaying only the source's final buffer lost historical and boundary matches and could manufacture windows across discarded input. States now retain effective timestamp/value pairs, merge chronologically, and rebuild matching windows without duplicating partial results. Serialization preserves the required input; deserialization initializes/replaces fresh or reused states. Regressions cover historical/boundary matches, false windows, interleaved/reversed/grouped merges, continued input, source immutability, and state round trips.

Resource impact: DTW state memory now grows with accepted input count plus matches, and each combine recalculates windows. This supports arbitrary temporal ordering/interleaving of partial states. No performance benchmark was run.

Validation: all ten additional regressions failed before the production fixes (nine assertion failures and one deserialization NPE), then passed after repair. The earlier six behavior regressions also reproduced before their fixes. Spotless, Checkstyle (0 violations), and git diff --check passed. The final English clean source reactor passed all 133 tests (0 failures/errors/skips); the final Chinese clean source reactor passed test compilation. Both reactors included the required IoTDB source dependencies and disabled local build-cache reuse. A local offline fallback was used because remote Maven plugin downloads were blocked. All newly added tests were run locally. This was focused library-udf verification, not a full-project fresh-dependency build.

The source branch had not changed before either push. The final local commit, source branch, and PR head all match 033b55d1474c978dd1b4ee07138f337809ede990. The PR remains Draft and requires a fresh full-diff review before Ready. No remote CI waiting or polling was performed. This edits the existing single round summary; no additional top-level summary was posted.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Independent re-review: one complete round

Reviewed the complete 86-file diff from baseline cca1dbbde6d0dffa38495c92c92a48b2626d2f99 to 033b55d1474c978dd1b4ee07138f337809ede990, plus relevant implementation/API/test context.

The review covered MAD merge/null semantics, sampling Row reuse and INT64 precision, DWT levels, Histogram/MinMax extremes, Spline timestamps/coordinates/endpoints and test fixtures, AR/DTW validation, moving-average recovery, ExactOrderStatistics extreme-value median/MAD, and DTW history, cross-partition matching, interleaved/reversed/multilevel merges and serialization.

Outcome: six actionable findings were fixed in 111fd493b7a. The final local commit, remote source branch Caideyipi/iotdb:fix/library-udf-edge-cases, and PR head were all confirmed as 111fd493b7a4ab63437394e50beaa09ae72b7c94. The source head was checked immediately before pushing and had no concurrent update. Draft status is retained because this round found behavioral and test-logic issues; a fresh independent re-review is required.

Findings and repairs

Locations below refer to the reviewed commit.

  1. [P2] INT64 MAD loses small deviations at large offsets: ExactOrderStatistics.java:181. Converting samples/median to double before subtraction makes the MAD of adjacent large INT64 values zero instead of 0.5. The fix calculates twice the median and doubled absolute deviations with BigInteger before conversion. Added odd/even and extreme-offset regressions.

  2. [P2] A nonzero subnormal histogram width can still misclassify buckets: UDTFHistogram.java:90. For range 0..5*Double.MIN_VALUE, two buckets, and inputs 2*MIN_VALUE,3*MIN_VALUE, rounded division by the bucket width produces [0,2] instead of [1,1]. The fix uses the normalized ratio for subnormal widths; tests cover positive and negative bounds.

  3. [P2] Spline can emit duplicate timestamps: UDTFSpline.java:119. Input timestamps/values 0..4 with points=10 generate repeated timestamps, violating the PointCollector requirement for strictly increasing output. The fix caps the output count to the available integer timestamps and uses that count for spacing and endpoint selection. The collector regression enforces increasing timestamps and verifies values.

  4. [P2] Moving averages lose finite small values after large samples leave: UDTFMvAvg.java:75. With window=2 and values [1e16,1e16,1,1], the final average is 0.5 rather than 1. Recovery triggered only by nonfinite sums misses this cancellation. The fix uses compensated additions/removals and resets compensation when rebuilding after overflow. Regressions also cover positive/negative maximum doubles and subsequent windows.

  5. [P2] DTW merge repeatedly recalculates all historical windows: UDAFDTWMatch.java:143. Each combine copied the complete accumulated input and recomputed every window. Sequentially combining many small partitions therefore makes matching work quadratic in partition count for a fixed pattern. The fix accumulates incoming points and defers a required chronological rebuild to final output, persists the pending-rebuild flag through serialization, and uses two rolling DTW rows. Tests cover repeated small interleaved merges, serialization before output, continuation, repeated output, source immutability, and nonzero-distance threshold boundaries.

  6. [P2] The LOF series regression used an impossible access window: UDFWindowAndQueueTest.java:279. It configured a two-row window but supplied six rows, so the asserted result could not arise under the configured engine access strategy. The corrected test uses the default access window and embedding size five, with nine input rows including two invalid values, preserving the expected three output timestamps.

Verification

  • Four numerical regressions failed against their pre-fix implementations, then passed after repair.
  • mvn -o spotless:apply -P with-integration-tests -pl library-udf: passed.
  • mvn -o clean test -pl library-udf -am: 139 tests, zero failures/errors/skips.
  • mvn -o clean test-compile -pl library-udf -am -P with-zh-locale -DskipTests: passed.
  • Lifecycle Checkstyle and git diff --check: passed.
  • Both clean source reactors disabled local and remote build caches using -Ddevelocity.cache.local.enabled=false -Ddevelocity.cache.remote.enabled=false. Required IoTDB source dependencies were rebuilt.
  • All newly added tests were run locally. Validation used the local offline fallback because remote Maven plugin downloads were blocked.
  • No full-project build or performance benchmark was run. Remote CI was neither waited on nor polled.

DTW resource limits

The retained ordered input remains O(N + R) in state memory and serialized size, where N is the effective input count and R the match count; arbitrary temporal interleaving still requires the historical input. The merge repair removes repeated historical DTW recalculation during combines. A pending rebuild at final output still costs O(N * L^2) for pattern length L, with O(L) DTW scratch space. This is an algorithmic assessment, not a measured performance result.

This session completed one review round and its repairs; it did not conduct another review after the update.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Independent re-review completed: one full round of the 86-file diff and relevant implementation, API, and test context.

  • Reviewed head: 111fd493b7a4ab63437394e50beaa09ae72b7c94
  • Comparison base: cca1dbbde6d0dffa38495c92c92a48b2626d2f99
  • Final local commit, confirmed source branch head, and confirmed PR head: 32f25d9cedd8af43529c3c0ea26487b2b7344c6a

Two behavioral issues were found and fixed in this round. Locations below refer to the reviewed head.

  1. [P2] DTW raw and continued input did not invalidate cached matches when timestamps went backwards or an earlier value was replaced. DTWState.java:108. With pattern 1,2,3 and threshold 0, feeding timestamps 6,5,4,3,2,1 with corresponding values 3,2,1,3,2,1 missed both chronological matches. Likewise, after merging and observing times 1,3,4, inserting time 2 did not rebuild the historical windows. Updating an earlier value could leave an obsolete match. Non-increasing timestamps now mark the state for the same deferred chronological rebuild used by merges. Regressions cover batches, late continuation after output, historical replacement, serialization before rebuilding, and repeated output.

  2. [P2] Moving-average compensation could itself lose small values permanently. UDTFMvAvg.java:97. For window=3 and input [1e300,1e150,1,1,1], the final window [1,1,1] returned 0 instead of 1; all sums remained finite, so overflow recovery did not help. The fix detects lost compensation bits or overflowing additions, reconstructs the active window sum exactly once, and then updates that exact sum incrementally. The regression covers both signs, subsequent windows, and resetting the function for another run.

The review also covered MAD null handling and INT64 extremes, sampling Row reuse and typed INT64 output, DWT levels, Histogram normal/subnormal/extreme ranges, MinMax extremes, Spline timestamps/coordinates/endpoints/duplicate timestamps and fixtures, AR/DTW validation, LOF valid-window coverage, and DTW history, cross-partition matches, interleaved/reversed/multilevel merges, serialization, continuation, and repeated output.

Validation:

  • All three new regressions failed against the reviewed production implementation, then passed after repair.
  • Local offline mvn -o clean test -pl library-udf -am: 142 tests, zero failures/errors/skips.
  • Local offline Chinese clean test-compile -pl library-udf -am -P with-zh-locale -DskipTests: passed.
  • Spotless with with-integration-tests, lifecycle Checkstyle (zero violations), and git diff --check: passed.
  • Required IoTDB dependencies were rebuilt from clean source; both clean reactors disabled local and remote build caches. All newly added tests were run locally. The local offline fallback retained the previously established remote Maven plugin-download limitation; no new external dependencies were introduced.
  • No full-project build or performance benchmark was run. CI was neither waited on nor polled.

Resource assessment: DTW still retains O(N + R) state and serialized data for N effective input points and R matches. A pending final rebuild costs O(N * L^2) for pattern length L, with O(L) DTW scratch space. Combines insert incoming points without recalculating all historical DTW windows; serialization and multilevel transfer still traverse retained input. This is an algorithmic assessment, not a measured performance result. The moving-average exact fallback retains the active window and an incremental sum, with a one-time window reconstruction.

The source branch was checked immediately before pushing and had no concurrent update. The source branch and PR head were both confirmed at the final SHA above, and the PR remains Draft because this round found behavioral issues. A fresh independent full-diff re-review is required before Ready. No second review round was started after these repairs.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Independent re-review

  • Reviewed SHA: 32f25d9cedd8af43529c3c0ea26487b2b7344c6a.
  • Fixed comparison baseline: cca1dbbde6d0dffa38495c92c92a48b2626d2f99.
  • Final local SHA, confirmed source-branch SHA, and confirmed PR-head SHA: 8c37cb0e5e083759420ae840558064b500db35ab.
  • Source: Caideyipi/iotdb:fix/library-udf-edge-cases. The source and PR heads matched the reviewed SHA immediately before the update; the push used an explicit expected-SHA lease and was a fast-forward. The PR is confirmed Draft.

Finding and fix

P2 — Subnormal Histogram widths misclassify exact bucket boundaries (reviewed UDTFHistogram.java:90). With min=0, max=22*Double.MIN_VALUE, and count=22, an input of 15*Double.MIN_VALUE enters bucket 14 instead of bucket 15: (15.0/22)*22 rounds to 14.999999999999998. This changes the reported histogram counts.

Commit 8c37cb0e5e083759420ae840558064b500db35ab computes the subnormal-width bucket index from exact binary-double values using an integral decimal quotient and clamps endpoints before indexing. The new regression enumerates representable points in positive and negative subnormal ranges, including widths rounded to a nonzero subnormal and widths underflowing to zero; expected bucket counts use an independent integer oracle. The regression reproduced the failure before the production fix.

Review coverage and resource costs

The review covered the complete fixed-base diff and relevant context, including MAD merge/null handling and INT64 extremes; sampled Row reuse and INT64 precision; DWT levels; Histogram/MinMax extremes; Spline timestamps, coordinates, endpoints, duplicates and test doubles; AR/DTW validation; moving-average overflow and compensated-sum recovery; LOF valid-window tests; and DTW history, cross-partition/interleaved/reverse/multilevel merges, deferred rebuild, serialization, continued input and repeated output. No further in-scope actionable issue was confirmed in this round.

For DTW, with N retained input points, R matches and pattern length p, static inspection gives O(N+R+p) retained state and 9 + 16N + 20R serialized bytes. A merge copies retained input, with ordinary map insertion cost O(M log(N+M)) for M source points, and defers DTW replay. Dirty finalization costs approximately O(N*p^2), with O(p) scratch space; multilevel merges repeatedly copy inputs. These are static costs, not measured performance results.

Validation

  • The new regression failed on the reviewed implementation before the fix (one failure, no errors or skips).
  • Local offline clean test -pl library-udf -am: 143 tests, zero failures, errors or skips. Required IoTDB dependencies were rebuilt from clean source, with both local and remote build caches disabled. All newly added tests ran locally.
  • Chinese-profile offline clean source-reactor test-compile, Spotless apply, lifecycle Checkstyle and git diff --check passed.
  • Local verification used the previously established fallback for blocked remote Maven plugin downloads. No full-project build or performance benchmark was run. CI was neither awaited nor polled.

This round found and fixed a behavioral issue, so the PR remains Draft pending a complete independent re-review in a fresh session. No second review round was started in this session.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Completed one independent re-review of the full 86-file diff and relevant context. This round found and fixed one P2 behavioral issue. The PR remains Draft pending a fresh independent re-review.

  • Reviewed SHA: 8c37cb0e5e083759420ae840558064b500db35ab.
  • Fixed comparison baseline: cca1dbbde6d0dffa38495c92c92a48b2626d2f99.
  • Final local SHA, confirmed Caideyipi/iotdb:fix/library-udf-edge-cases head, and confirmed PR head: all 49816e970fe1f5df80f2b762e56b213703272005.
  • Immediately before pushing, both remote heads still matched the reviewed SHA and the PR was Draft. The update was a fast-forward protected by an explicit lease against that SHA; no concurrent update was overwritten.

P2 — fixed: preserve dense Spline knots across a wide timestamp range. At the reviewed revision, UDTFSpline.java:97–113 always centered double coordinates at the time-range midpoint. Valid timestamps [0, 1, 2, 3, 4, 2^60], values t / 2^60, and points=3 collapsed to fewer than five coordinates and produced no output. Expected timestamps were [0, 2^59, 2^60], with values [0, 0.5, 1]. A regression failed against the reviewed production source in a clean source reactor.

The repair chooses among midpoint/start/end origins to retain the most distinct coordinates. The mirrored right-end regression also caught cancellation yielding NaN in Akima's three-point endpoint derivatives during repair validation. The final implementation uses equivalent divided differences for those derivatives, retaining Akima's interior weights and Hermite cubics. Regression coverage asserts timestamps and linear values for dense left/right/interior clusters, plus independently fixed expected values for a non-linear Akima example.

The review also covered MAD merging/isNull/INT64 extrema; sampling with reused Rows and INT64 precision; DWT levels; Histogram/MinMax extrema; Spline rounding/endpoints/duplicate timestamps and test doubles; AR/DTW parameter validation; moving-average overflow recovery and compensated summation; and valid-window LOF tests. No additional actionable PR regression was confirmed. The earlier Histogram subnormal-boundary fix, DTW non-increasing-timestamp cache invalidation, and moving-average exact-sum recovery are present. The pre-existing LOF numerical-semantics audit candidate remains unconfirmed as a PR regression, and its algorithm was left unchanged.

DTW inspection covered historical and cross-shard matching, interleaved/reversed/multi-level merges, deferred reconstruction, serialization, continued input, and repeated output. Static resource assessment: with N retained input points, R matches, and template length L, state grows as O(N + R); serialized state uses 9 + 16N + 20R bytes, excluding temporary allocations and heap-object overhead. Merge generally costs O(M log(N + M)) for M incoming points and defers DTW reconstruction until output. A dirty output can rebuild in O(N * L^2) time with O(L) DTW scratch space; unchanged repeated output reuses cached matches. Repeated merge/output cycles can repeat that reconstruction cost. Complete input retention therefore continues to grow with history; no performance benchmark was run.

Validation used the authorized local offline fallback, with both local and remote Develocity build caches disabled and the required IoTDB dependency closure rebuilt from clean source:

  • mvn -o clean test -pl library-udf -am: 144 tests passed, zero failures/errors/skips, all 56 goals executed.
  • After adding the non-linear expected-value regression, a separate clean source reactor selecting that new test passed: 1 test, all 56 goals executed. Thus all 145 distinct tests, including every newly added test, were run locally and passed across these runs; the final 145-test suite was not rerun as one invocation.
  • mvn -o clean test-compile -pl library-udf -am -P with-zh-locale -DskipTests: passed, all 54 goals executed.
  • Spotless apply/check, Checkstyle, locale message-key parity, and git diff --check: passed.

No full-project build or performance benchmark was run. CI was neither waited for nor polled. This session ends after repair/publication; it does not include a second review round.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Re-review against fixed base cca1dbbde6d0dffa38495c92c92a48b2626d2f99 reviewed PR head 49816e970fe1f5df80f2b762e56b213703272005. I found and fixed four behavioral issues: moving averages lost the finite mean when an intermediate window sum overflowed; spline coordinates could collapse dense timestamp clusters at both long endpoints; DTW matches could remain stale or depend on partial-state merge order, and parseTime rejected the valid Long.MAX_VALUE boundary. Regression tests cover overflow recovery, endpoint-dense spline input, non-monotonic/interleaved pattern input, and exact long bounds.

Validation: clean offline source-reactor build of library-udf and its IoTDB dependency closure passed all 146 tests (0 failures, errors, or skips); Checkstyle, Spotless, and git diff --check passed. DTW retains effective input in O(N) memory and deferred rebuilding costs O(N * W^2); no performance benchmark or full-project build was run. CI was not polled.

Reviewed SHA: 49816e970fe1f5df80f2b762e56b213703272005. Final local and confirmed source-branch/PR-head SHA: 91e48ebf715a2890a2f3116e8066bfebe251f763. The PR remains Draft.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Completed one independent re-review of PR #17871 against fixed base cca1dbbde6d0dffa38495c92c92a48b2626d2f99, reviewing head 91e48ebf715a2890a2f3116e8066bfebe251f763. The round confirmed two actionable issues, both fixed in 223b7cc5981f5ff6ea6cba107c40af923b8a0a0c:

  • UDTFLOF with method=series used window for both embedding width and access-batch size. Explicit values such as 5 could leave too few embeddings to satisfy k, suppressing all output. The fix separates the series dimension from the access batch; the regression checks an explicit width and the configured final partial batch.
  • Resampler treated the absent endTime=-1 as a flush bound for negative timestamps, producing extra output windows, and currentTime + newPeriod could overflow near Long.MAX_VALUE. The fix tracks endpoint presence and guards timestamp advancement, with regressions for negative timestamps and the Long.MAX_VALUE boundary.

Verification: an offline clean-source reactor run rebuilt the parent, iotdb-api, udf-api, and library-udf with local and remote build caches disabled. All 148 library-udf tests passed, including the new regressions; Spotless, Checkstyle, and git diff --check passed. No full-project build or performance benchmark was run. CI was not polled.

Confirmed source branch Caideyipi/iotdb:fix/library-udf-edge-cases and PR head both equal 223b7cc5981f5ff6ea6cba107c40af923b8a0a0c. PR #17871 remains Draft because this round found behavioral and test-logic issues; a fresh independent re-review is required.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Independent re-review

Reviewed PR #17871 against fixed base cca1dbbde6d0dffa38495c92c92a48b2626d2f99 at review SHA 223b7cc5981f5ff6ea6cba107c40af923b8a0a0c.

Confirmed source branch: Caideyipi/iotdb:fix/library-udf-edge-cases. The final source branch SHA and PR head SHA are both a04fa5b8465024f4470e7b4878cf486a5e24aed0. The PR remains Draft.

Findings and fixes:

  • UDTFLOF with method=series, window=9998, and k=3 used a 10,000-row batch, yielding at most three embedding vectors and suppressing all output. The access batch now grows to at least window + k; a regression verifies the four valid outputs.
  • UDAFIntegral and UDAFIntegralAvg overflowed when subtracting timestamps across the signed long range, corrupting area and average results. Both now use exact subtraction with a double fallback for overflowing spans; regressions cover Long.MIN_VALUE through Long.MAX_VALUE.

No other actionable issues were confirmed in the full base-to-head review. The existing Resampler regressions also passed.

Validation: the offline, cache-disabled clean source reactor GITHUB_ACTIONS=true mvn -o -Ddevelocity.off=true clean test -P with-integration-tests -pl library-udf -am passed all 151 tests; reactor Spotless and Checkstyle passed; git diff --check passed. No full-project build or performance benchmark was run, and CI was not queried or polled.

@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Completed one fresh independent re-review of the full 87-file diff and relevant context at a04fa5b8465024f4470e7b4878cf486a5e24aed0, against fixed base cca1dbbde6d0dffa38495c92c92a48b2626d2f99. The previous reviewed revision was 223b7cc5981f5ff6ea6cba107c40af923b8a0a0c.

Three numerical issues were reproduced before their fixes and repaired in 39aed757e38cd3f7cda69aa7c0c3a78bd18df08c:

  1. [P2] Integral and IntegralAvg can overflow before unit scaling or averaging. The trapezoid expressions in Integral and IntegralAvg can produce infinity for finite mathematical results: a constant 1e300 over the full long timestamp range should still average to 1e300; Integral with unit=9223372036854775807ms should return 2e300 for the same constant series at [Long.MIN_VALUE, 0, Long.MAX_VALUE]. Adding two Double.MAX_VALUE endpoints can also overflow on a one-tick interval. Areas now use exact decimal accumulation with exact timestamp differences, followed by scaling before conversion to double; regressions cover positive, negative, maximal and subnormal values.
  2. [P2] Resampler linear interpolation still subtracts long timestamps without overflow protection. In the interpolation expression, points (Long.MIN_VALUE, 0) and (Long.MAX_VALUE, 2) with period Long.MAX_VALUE yield a corrupt middle value instead of 1. The repair uses overflow-safe timestamp differences and computes interpolation weights before multiplying values. The new regression checks all three output times and values.
  3. [P2] Nanosecond hour/day unit multipliers overflow as int expressions. In Util.parseTime, the trailing long operand arrives after intermediate int overflow: 1h is rejected and 1d has the wrong scale. This defect already exists in the fixed baseline and was found within the requested Integral unit audit. The first multiplier now promotes the expression to long; regressions verify second/minute/hour/day scales at all timestamp precisions and Integral with nanosecond hour/day units.

LOF series batch sizing, sliding step, window/k thresholds and tail behavior were checked, including finite-vector counts after invalid rows are removed. No new actionable defect was confirmed there; additional tail-boundary coverage was added. The earlier Resampler end-bound, negative-time and near-Long.MAX_VALUE window-advance fixes were also rechecked.

Validation on the final source tree: Spotless and git diff --check passed; an offline, build-cache-disabled clean test -P with-integration-tests -pl library-udf -am passed 158 tests, zero failures/errors/skips; the corresponding clean test-compile with with-zh-locale also passed. The changing TsFile dependency was rebuilt from pinned source commit 8f665d113dbb5257b901c528c31c8bc24da2a018, and its JAR hashes stayed unchanged across both checks. All newly added tests ran locally. No CI was queried or polled; no full-project build or performance benchmark was run.

Final local commit, confirmed source-branch SHA and confirmed PR head are all 39aed757e38cd3f7cda69aa7c0c3a78bd18df08c. PR/source freshness and Draft state were verified before committing and again before the ordinary fast-forward push; the fixed base is unchanged. The PR remains Draft because this round found behavioral issues; a fresh independent round is required before marking it ready. No update step remains incomplete.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant