Repository navigation
Fragment large TsBlocks of query result responses between java session and server - #19263
Wei-hao-Li wants to merge 4 commits into
Conversation
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #19263 +/- ##
============================================
+ Coverage 49.05% 49.18% +0.13%
- Complexity 689 690 +1
============================================
Files 5590 5594 +4
Lines 413977 414803 +826
Branches 54633 54784 +151
============================================
+ Hits 203061 204036 +975
+ Misses 210916 210767 -149 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
JackieTien97
left a comment
There was a problem hiding this comment.
Please address the memory accounting and read-size quota regressions introduced by fragmentation, and apply the payload limit to the Fast LAST cache-hit paths. The inline comments describe the affected paths.
| @Override | ||
| public ByteBuffer getSerializedTsBlock() throws IoTDBException { | ||
| public ByteBuffer getSerializedTsBlock(boolean retainMemory) throws IoTDBException { | ||
| // the returned bytes are a fresh serialization, not reserved from the memory pool |
There was a problem hiding this comment.
[P1] Account for pending fragments on the local result path
receive() removes the block from SharedTsBlockQueue and releases its reservation. The serialized buffer can now outlive the RPC because PendingResultFragment holds the entire buffer until the client finishes fetching it. On a single-node deployment, several concurrent readers can therefore hold large buffers outside the query memory pool. Please keep the pending buffer accounted for until the last fragment is sent or the query is cleaned up.
| req.fetchSize, | ||
| req.isSetOffset() ? req.getOffset() : 0, | ||
| req.isSetOffset() ? config.getMppDataExchangeMaxPayloadSizeInBytes() : 0); | ||
| finished = fillQueryResult(resp, chunk); |
There was a problem hiding this comment.
[P2] Preserve read-size quota accounting across fragments
The execute path charges only the buffers in its response through quota.addReadResult(). The continuation path here does not charge the remaining bytes. With the default 4 MiB payload limit, a 12 MiB TsBlock that previously fit in the execute response now counts as only 4 MiB against the read-size quota. Please ensure the continuation bytes are accounted for exactly once.
| finished = SELECT_RESULT.apply(resp, queryExecution, req.fetchSize); | ||
| finished = | ||
| SELECT_RESULT.apply( | ||
| resp, queryExecution, req.fetchSize, getMaxResultPayloadSize(clientSession)); |
There was a problem hiding this comment.
[P2] Apply the payload limit to Fast LAST cache hits
This fallback now honors the negotiated limit. The cache-hit branches in executeFastLastDataQueryForOneDeviceV2 and executeFastLastDataQueryForOnePrefixPath still serialize the entire result into one response and set moreData=false. A large cached result can still exceed the Thrift frame limit even though the fallback can stream it in fragments. Please apply the payload limit to both cache-hit paths.




No description provided.