Skip to content

[Pipe] Reduce allocations in query Field copying and Tablet batch encoding - #19282

Open
Caideyipi wants to merge 1 commit into
masterfrom
codex/pipe-tablet-pr-20261113
Open

Caideyipi wants to merge 1 commit into
masterfrom
codex/pipe-tablet-pr-20261113

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Pipe's TsFile query parser converts each primitive Field to a boxed object before storing it in a Tablet. Raw Tablet batches also encode their payloads through PublicBAOS and DataOutputStream. This change copies primitive fields directly into Tablet columns and writes both tree-model and merged table-model batches into exactly sized ByteBuffers.

The existing null/default conversions, DATE representation compatibility, OBJECT fallback, deletion handling, and request wire format are preserved. For example, if timestamp 0 is fully deleted and timestamps 1 and 2 contain (11, null) and (12, "value"), the output still contains those two rows with the same null bitmap, even though the deleted row's slot is reused. Aligned and non-aligned tree batches and merged table batches retain byte-for-byte equivalence with stream serialization, including NaNs and signed zero.

Related pull requests

Depends on apache/tsfile#1101, specifically commit eaaad099d6a29bff60cc374ad0ff30fad6b40e3d, for Tablet.serialize(ByteBuffer) and matching NaN encoding. The POM selects that source's 2.5.1-SNAPSHOT coordinates. This dependency must be integrated and made available before this PR can merge; the local checks used artifacts built from the specified source revision.

Performance

The following local JMH measurements compare the IoTDB implementation before and after these changes. Both variants use the same TsFile revision above, so these measurements isolate the IoTDB changes and do not attribute the results to changes between TsFile versions. They were collected on November 13 on the earlier development snapshot, before transplanting the same production-file changes onto public master; timings were not rerun on the final publication base.

Operation / input Baseline ms/op Optimized ms/op Mean latency reduction Allocated bytes/op reduction
QueryParser, 64 INT64 columns 336.106 232.381 30.9% 13.1%
QueryParser, 256 INT64 columns 1,439.066 1,140.021 20.8% 12.9%
Tree-model batch, mostly INT64 1.435 0.417 71.0% 0.3%
Tree-model batch, mixed types with nulls 2.239 0.699 68.8% 0.4%
Merged table-model batch, mostly INT64 2.117 1.670 21.1% 0.1%
Merged table-model batch, mixed types with nulls 3.412 2.411 29.4% 0.1%
  • Windows x86_64, Intel Core i9-13900H, JDK 17.0.15, JMH 1.37; one benchmark thread, -Xms1g -Xmx1g -XX:ActiveProcessorCount=4.
  • Each variant used two independent JVM forks, each with three 1-second warmup iterations and five 1-second measurement iterations. Allocation is JMH GCProfiler's normalized allocated bytes per operation.
  • QueryParser read one aligned device with 16,384 rows and either 64 or 256 INT64 columns (1,048,576 or 4,194,304 points per operation). Input generation was excluded, reads used a warm OS cache, and returned values and timestamps were consumed.
  • Batch inputs used 64 columns and 1,024 rows per Tablet. Mostly-INT64 input had one STRING column and 63 INT64 columns. Mixed input covered BOOLEAN, INT32, INT64, FLOAT, DOUBLE, DATE, STRING, and BLOB, with about 14% nulls outside the first column. Tree batches used one Tablet; merged table batches used two. Timing included production batch construction, memory accounting, merge/copy where applicable, final request encoding, and batch reset. Stream-format byte equality was checked outside timed regions.

These are preliminary component measurements on a shared workstation. The JMH 99.9% timing confidence intervals overlap for the 256-column QueryParser and both merged table-model cases, so their percentage changes should not be treated as established, stable speedups. End-to-end sender/network/receiver Pipe throughput and distribution packaging were not tested.

Validation

On public master base ea7b4ffc5d68021f18a18f627218ab625eee7f7a, the required 28-module dependency closure was rebuilt from clean source, and the focused reactor test run completed with 16 tests passed, zero failures, errors, or skips (Windows, JDK 17.0.15, Maven 3.9.12). This includes all five added regression tests. The deletion fixture uses the public branch's existing PatternTreeMap factory; its deletion records and assertions are unchanged.

The original implementation snapshot passed 19 selected local tests. All five newly added tests have been run locally. Verification temporarily disabled build-scan extensions; that configuration change is excluded from the submitted diff.

Checklist

  • Independently self-reviewed once with no actionable findings in the original implementation snapshot; publication preserves the reviewed production-file contents.
  • Added unit tests covering direct Field copy, deleted-row slot reuse, and complete request byte compatibility.
  • Measured local component performance with the limitations stated above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant