Repository navigation
Conversation
…rary-udf-edge-cases
…rary-udf-edge-cases # Conflicts: # library-udf/src/main/java/org/apache/iotdb/library/anomaly/UDTFKSigma.java # library-udf/src/main/java/org/apache/iotdb/library/anomaly/UDTFRange.java # library-udf/src/main/java/org/apache/iotdb/library/dprofile/UDAFPercentile.java # library-udf/src/main/java/org/apache/iotdb/library/dprofile/UDAFQuantile.java # library-udf/src/main/java/org/apache/iotdb/library/dprofile/UDAFSpread.java # library-udf/src/main/java/org/apache/iotdb/library/dprofile/UDTFDistinct.java # library-udf/src/main/java/org/apache/iotdb/library/dprofile/UDTFSample.java # library-udf/src/main/java/org/apache/iotdb/library/drepair/UDTFTimestampRepair.java # library-udf/src/main/java/org/apache/iotdb/library/drepair/UDTFValueFill.java # library-udf/src/main/java/org/apache/iotdb/library/drepair/UDTFValueRepair.java # library-udf/src/main/java/org/apache/iotdb/library/frequency/UDFEnvelopeAnalysis.java # library-udf/src/main/java/org/apache/iotdb/library/match/UDAFDTWMatch.java # library-udf/src/main/java/org/apache/iotdb/library/match/UDAFPatternMatch.java
|
Self-reviewed this PR. Fixed the quantile floating-point ordering issue involving negative zero; I did not find any other issues. |
Resolve UDF validation conflicts while preserving edge-case checks and upstream localized messages.
…r-sync-20261113 # Conflicts: # library-udf/src/main/java/org/apache/iotdb/library/dprofile/UDAFMad.java
|
Independent re-review completed for Fixed in
The Validation:
Final local commit, confirmed source-branch head, and confirmed PR head are all |
|
Independent re-review of PR #17871 Reviewed the requested starting head Confirmed behavior issues found and fixed during this round in commits
Validation: the offline clean library-udf source-reactor test passed all 117 tests; Spotless, Checkstyle, clean English and Chinese test-compiles, message-constant/placeholder parity, and Final local commit, source branch head, and PR head are |
Independent re-review: fixes completed, Draft retainedCompleted one review round of the full diff and relevant context, including MAD merge/isNull, sampling Row reuse and INT64 precision, DWT levels, Histogram/MinMax extremes, Spline timestamps/coordinates/endpoints, and test fixtures.
Six behavior issues and one test-fixture issue were found and fixed:
Additional baseline fixes includedThe two baseline observations identified during the completed round are now repaired in
Resource impact: DTW state memory now grows with accepted input count plus matches, and each combine recalculates windows. This supports arbitrary temporal ordering/interleaving of partial states. No performance benchmark was run. Validation: all ten additional regressions failed before the production fixes (nine assertion failures and one deserialization NPE), then passed after repair. The earlier six behavior regressions also reproduced before their fixes. Spotless, Checkstyle (0 violations), and The source branch had not changed before either push. The final local commit, source branch, and PR head all match |
Independent re-review: one complete roundReviewed the complete 86-file diff from baseline The review covered MAD merge/null semantics, sampling Row reuse and INT64 precision, DWT levels, Histogram/MinMax extremes, Spline timestamps/coordinates/endpoints and test fixtures, AR/DTW validation, moving-average recovery, ExactOrderStatistics extreme-value median/MAD, and DTW history, cross-partition matching, interleaved/reversed/multilevel merges and serialization. Outcome: six actionable findings were fixed in Findings and repairsLocations below refer to the reviewed commit.
Verification
DTW resource limitsThe retained ordered input remains O(N + R) in state memory and serialized size, where N is the effective input count and R the match count; arbitrary temporal interleaving still requires the historical input. The merge repair removes repeated historical DTW recalculation during combines. A pending rebuild at final output still costs O(N * L^2) for pattern length L, with O(L) DTW scratch space. This is an algorithmic assessment, not a measured performance result. This session completed one review round and its repairs; it did not conduct another review after the update. |
|
Independent re-review completed: one full round of the 86-file diff and relevant implementation, API, and test context.
Two behavioral issues were found and fixed in this round. Locations below refer to the reviewed head.
The review also covered MAD null handling and INT64 extremes, sampling Row reuse and typed INT64 output, DWT levels, Histogram normal/subnormal/extreme ranges, MinMax extremes, Spline timestamps/coordinates/endpoints/duplicate timestamps and fixtures, AR/DTW validation, LOF valid-window coverage, and DTW history, cross-partition matches, interleaved/reversed/multilevel merges, serialization, continuation, and repeated output. Validation:
Resource assessment: DTW still retains O(N + R) state and serialized data for N effective input points and R matches. A pending final rebuild costs O(N * L^2) for pattern length L, with O(L) DTW scratch space. Combines insert incoming points without recalculating all historical DTW windows; serialization and multilevel transfer still traverse retained input. This is an algorithmic assessment, not a measured performance result. The moving-average exact fallback retains the active window and an incremental sum, with a one-time window reconstruction. The source branch was checked immediately before pushing and had no concurrent update. The source branch and PR head were both confirmed at the final SHA above, and the PR remains Draft because this round found behavioral issues. A fresh independent full-diff re-review is required before Ready. No second review round was started after these repairs. |
Independent re-review
Finding and fixP2 — Subnormal Histogram widths misclassify exact bucket boundaries (reviewed UDTFHistogram.java:90). With Commit Review coverage and resource costsThe review covered the complete fixed-base diff and relevant context, including MAD merge/null handling and INT64 extremes; sampled Row reuse and INT64 precision; DWT levels; Histogram/MinMax extremes; Spline timestamps, coordinates, endpoints, duplicates and test doubles; AR/DTW validation; moving-average overflow and compensated-sum recovery; LOF valid-window tests; and DTW history, cross-partition/interleaved/reverse/multilevel merges, deferred rebuild, serialization, continued input and repeated output. No further in-scope actionable issue was confirmed in this round. For DTW, with N retained input points, R matches and pattern length p, static inspection gives O(N+R+p) retained state and Validation
This round found and fixed a behavioral issue, so the PR remains Draft pending a complete independent re-review in a fresh session. No second review round was started in this session. |
|
Completed one independent re-review of the full 86-file diff and relevant context. This round found and fixed one P2 behavioral issue. The PR remains Draft pending a fresh independent re-review.
P2 — fixed: preserve dense Spline knots across a wide timestamp range. At the reviewed revision, UDTFSpline.java:97–113 always centered double coordinates at the time-range midpoint. Valid timestamps The repair chooses among midpoint/start/end origins to retain the most distinct coordinates. The mirrored right-end regression also caught cancellation yielding NaN in Akima's three-point endpoint derivatives during repair validation. The final implementation uses equivalent divided differences for those derivatives, retaining Akima's interior weights and Hermite cubics. Regression coverage asserts timestamps and linear values for dense left/right/interior clusters, plus independently fixed expected values for a non-linear Akima example. The review also covered MAD merging/isNull/INT64 extrema; sampling with reused Rows and INT64 precision; DWT levels; Histogram/MinMax extrema; Spline rounding/endpoints/duplicate timestamps and test doubles; AR/DTW parameter validation; moving-average overflow recovery and compensated summation; and valid-window LOF tests. No additional actionable PR regression was confirmed. The earlier Histogram subnormal-boundary fix, DTW non-increasing-timestamp cache invalidation, and moving-average exact-sum recovery are present. The pre-existing LOF numerical-semantics audit candidate remains unconfirmed as a PR regression, and its algorithm was left unchanged. DTW inspection covered historical and cross-shard matching, interleaved/reversed/multi-level merges, deferred reconstruction, serialization, continued input, and repeated output. Static resource assessment: with Validation used the authorized local offline fallback, with both local and remote Develocity build caches disabled and the required IoTDB dependency closure rebuilt from clean source:
No full-project build or performance benchmark was run. CI was neither waited for nor polled. This session ends after repair/publication; it does not include a second review round. |
|
Re-review against fixed base Validation: clean offline source-reactor build of Reviewed SHA: |
|
Completed one independent re-review of PR #17871 against fixed base
Verification: an offline clean-source reactor run rebuilt the parent, Confirmed source branch |
Independent re-reviewReviewed PR #17871 against fixed base Confirmed source branch: Findings and fixes:
No other actionable issues were confirmed in the full base-to-head review. The existing Resampler regressions also passed. Validation: the offline, cache-disabled clean source reactor |
|
Completed one fresh independent re-review of the full 87-file diff and relevant context at Three numerical issues were reproduced before their fixes and repaired in
LOF series batch sizing, sliding step, window/k thresholds and tail behavior were checked, including finite-vector counts after invalid rows are removed. No new actionable defect was confirmed there; additional tail-boundary coverage was added. The earlier Resampler end-bound, negative-time and near-Long.MAX_VALUE window-advance fixes were also rechecked. Validation on the final source tree: Spotless and Final local commit, confirmed source-branch SHA and confirmed PR head are all |
Summary
Examples
0,1,2,3,4withpoints=4previously emitted timestamps[0,1,3,4]but values approximately[0,1.3333,2.6667,4]. Values now use the emitted timestamps, producing[0,1,3,4]; the terminal value is also preserved when extreme timestamps share a double coordinate. Withpoints=10over the same timestamp range, it emits each of the five available timestamps once.min=0,max=100, andcount=100assigns input29to bucket29. For the subnormal range0..5*Double.MIN_VALUEwith two buckets, inputs2*Double.MIN_VALUEand3*Double.MIN_VALUEnow produce counts[1,1].window=2and values[Double.MAX_VALUE,Double.MAX_VALUE,0,2,4]skips the overflowing window and recovers with values[Double.MAX_VALUE/2,1,3]. Compensated summation also preserves the average1once the large values leave[1e16,1e16,1,1].Integer.MAX_VALUEorDouble.MAX_VALUEsamples return the correct finite median. The median of[Long.MIN_VALUE,Long.MAX_VALUE]is-0.5; adjacent large INT64 values retain MAD0.5by subtracting before conversion to double.[1,2,3]over values[1,2,3,1,2,3], split after the first value, returns both windows[1,3]and[4,6]. Matches survive interleaved/reversed/grouped state merges and serialization. Merges accumulate input; final output rebuilds chronological matches when required.Validation
mvn -o spotless:apply -P with-integration-tests -pl library-udf: passed.mvn -o clean test -pl library-udf -am: 139 tests passed, no failures/errors/skips.mvn -o clean test-compile -pl library-udf -am -P with-zh-locale -DskipTests: passed.git diff --check: passed.Both clean reactors rebuilt the required IoTDB source dependency closure with local and remote build caches disabled (
-Ddevelocity.cache.local.enabled=false -Ddevelocity.cache.remote.enabled=false). Validation used a local offline fallback because remote Maven plugin downloads were blocked. No full-project build or performance benchmark was run. Remote CI was not waited on or polled.DTW retains effective input points to support arbitrary temporal merge ordering/interleaving. State memory and serialized state grow with input count plus matches. Merges no longer repeatedly recalculate historical windows; a pending chronological rebuild runs at final output. DTW scratch memory is linear in pattern length, while each window still requires quadratic computation in pattern length.