Fix table UDF result block splitting - #18333
Open
ColinLeeo wants to merge 6 commits into
Open
Conversation
4 tasks
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #18333 +/- ##
=========================================
Coverage ? 43.45%
Complexity ? 374
=========================================
Files ? 5366
Lines ? 383022
Branches ? 49823
=========================================
Hits ? 166428
Misses ? 216594
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR addresses oversized TsBlock outputs from table-model UDFs by enforcing configured TsBlock size/row limits in TableFunctionOperator, preventing large blocks from propagating through the query pipeline and risking RPC frame overflows.
Changes:
- Add final-result TsBlock splitting in
TableFunctionOperator(after pass-through columns are appended) based on configured TsBlock constraints. - Add a unit test covering variable-width outputs with/without pass-through columns.
- Add an integration-test UDF plus an IT validating large UDF results are returned without loss/duplication and with pass-through alignment under a small TsBlock limit.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| iotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/execution/operator/process/tvf/TableFunctionOperatorTest.java | Adds unit coverage for splitting variable-width UDF output blocks with optional pass-through columns. |
| iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/execution/operator/process/function/TableFunctionOperator.java | Implements final-result TsBlock splitting and adjusts operator completion/return-size reporting. |
| integration-test/src/test/java/org/apache/iotdb/relational/it/db/it/udf/IoTDBUserDefinedTableFunctionIT.java | Adds end-to-end validation that large table-UDF results are split without data loss and keep pass-through columns aligned. |
| integration-test/src/main/java/org/apache/iotdb/db/query/udf/example/relational/LargeResultTableFunction.java | Introduces a table UDF used by the integration test to generate large variable-width outputs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
64
to
65
| private static final long INSTANCE_SIZE = | ||
| RamUsageEstimator.shallowSizeOfInstance(AggregationMergeSortOperator.class); |
Comment on lines
423
to
426
| @Override | ||
| public long calculateMaxReturnSize() { | ||
| return Math.max(DEFAULT_MAX_TSBLOCK_SIZE_IN_BYTES, properBlockBuilder.getRetainedSizeInBytes()); | ||
| return maxTsBlockSizeInBytes; | ||
| } |
Comment on lines
+264
to
+277
| /** | ||
| * Splits the final result using the same logical in-memory size accounting as {@link | ||
| * TsBlockBuilder}. | ||
| * | ||
| * <p>Serializing candidate regions to find their exact sizes would write every value into | ||
| * temporary buffers, only for the exchange layer to serialize the selected regions again. | ||
| * Rebuilding the result with a size-tracking {@link TsBlockBuilder} would avoid that temporary | ||
| * serialization, but it would turn the UDF's batched column output into row-by-row, | ||
| * column-by-column copies. | ||
| * | ||
| * <p>Instead, fixed-width values are accounted for directly from their data types, while only the | ||
| * retained sizes of variable-width values are inspected. This deliberately estimates the | ||
| * in-memory TsBlock size rather than its serialized size because the two representations are not | ||
| * equivalent. The resulting regions are views over the original columns and do not copy their |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Problem
A table-model UDF can produce a large result block while processing a partition containing a large device. Previously,
TableFunctionOperatorcould return the generatedTsBlockwithout enforcing the configured row-count and serialized byte-size limits.This could propagate an oversized block through the query pipeline and eventually exceed an RPC frame limit, such as the 64 MB Thrift frame limit.
Design
TableFunctionOperatornow splits each final output block insidebuildTsBlock(), after pass-through columns have been appended. Checking the final block is important because pass-through columns contribute to the serialized result size.For every output fragment, the operator:
max_tsblock_line_number.max_tsblock_size_in_bytes.next()calls.Binary search handles variable-width values such as
TEXT,STRING, andBLOBwithout relying on an average row-size estimate. The size check usesTsBlockSerdebecause the serialized size is the relevant boundary for blocks sent through the query pipeline.The splitting logic is kept in
TableFunctionOperatorinstead of inheriting the retained-block logic fromAbstractOperator. This avoids maintaining two result-fragmentation paths and ensures that proper columns and pass-through columns are always split together.Configuration and corner cases
This PR introduces no new configuration. It uses:
max_tsblock_size_in_bytesmax_tsblock_line_numberRows are indivisible. If a single row is larger than
max_tsblock_size_in_bytes, it is returned alone so execution can continue. Result ordering and empty-output behavior are unchanged.Tests
The unit test covers both output forms:
It verifies the row-count limit, exact serialized byte-size limit, total row count, ordering, values, and pass-through alignment.
The integration test configures a 1 KB TsBlock limit and runs a table UDF that expands four input rows into 256 variable-width output rows. It verifies:
(time, repeat_index)pair is returnedThe following targeted tests passed:
mvn test -pl iotdb-core/datanode -am \ -Dtest=TableFunctionOperatorTest \ -DfailIfNoTests=false \ -Dsurefire.failIfNoSpecifiedTests=false mvn verify -Drat.skip=true -DskipUTs \ -Dit.test=IoTDBUserDefinedTableFunctionIT#testLargeResultIsSplitWithoutDataLoss \ -DfailIfNoTests=false \ -Dfailsafe.failIfNoSpecifiedTests=false \ -pl integration-test -am \ -PTableSimpleIT -P with-integration-testsThis PR has:
Key changed/added classes
TableFunctionOperatorTableFunctionOperatorTestLargeResultTableFunctionIoTDBUserDefinedTableFunctionIT