Repository navigation
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Pipe's TsFile query parser converts each primitive
Fieldto a boxed object before storing it in a Tablet. Raw Tablet batches also encode their payloads throughPublicBAOSandDataOutputStream. This change copies primitive fields directly into Tablet columns and writes both tree-model and merged table-model batches into exactly sized ByteBuffers.The existing null/default conversions, DATE representation compatibility, OBJECT fallback, deletion handling, and request wire format are preserved. For example, if timestamp 0 is fully deleted and timestamps 1 and 2 contain
(11, null)and(12, "value"), the output still contains those two rows with the same null bitmap, even though the deleted row's slot is reused. Aligned and non-aligned tree batches and merged table batches retain byte-for-byte equivalence with stream serialization, including NaNs and signed zero.Related pull requests
Depends on apache/tsfile#1101, specifically commit
eaaad099d6a29bff60cc374ad0ff30fad6b40e3d, forTablet.serialize(ByteBuffer)and matching NaN encoding. The POM selects that source's2.5.1-SNAPSHOTcoordinates. This dependency must be integrated and made available before this PR can merge; the local checks used artifacts built from the specified source revision.Performance
The following local JMH measurements compare the IoTDB implementation before and after these changes. Both variants use the same TsFile revision above, so these measurements isolate the IoTDB changes and do not attribute the results to changes between TsFile versions. They were collected on November 13 on the earlier development snapshot, before transplanting the same production-file changes onto public
master; timings were not rerun on the final publication base.-Xms1g -Xmx1g -XX:ActiveProcessorCount=4.These are preliminary component measurements on a shared workstation. The JMH 99.9% timing confidence intervals overlap for the 256-column QueryParser and both merged table-model cases, so their percentage changes should not be treated as established, stable speedups. End-to-end sender/network/receiver Pipe throughput and distribution packaging were not tested.
Validation
On public
masterbaseea7b4ffc5d68021f18a18f627218ab625eee7f7a, the required 28-module dependency closure was rebuilt from clean source, and the focused reactor test run completed with 16 tests passed, zero failures, errors, or skips (Windows, JDK 17.0.15, Maven 3.9.12). This includes all five added regression tests. The deletion fixture uses the public branch's existing PatternTreeMap factory; its deletion records and assertions are unchanged.The original implementation snapshot passed 19 selected local tests. All five newly added tests have been run locally. Verification temporarily disabled build-scan extensions; that configuration change is excluded from the submitted diff.
Checklist