Skip to content

Fragment large TsBlocks of query result responses between java session and server - #19263

Open
Wei-hao-Li wants to merge 4 commits into
masterfrom
fragmentCSRPC
Open

Wei-hao-Li wants to merge 4 commits into
masterfrom
fragmentCSRPC

Conversation

@Wei-hao-Li

Copy link
Copy Markdown
Collaborator

No description provided.

Signed-off-by: Weihao Li <18110526956@163.com>
Signed-off-by: Weihao Li <18110526956@163.com>
Signed-off-by: Weihao Li <18110526956@163.com>
Signed-off-by: Weihao Li <18110526956@163.com>
@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

@codecov

codecov Bot commented Nov 13, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.51471% with 53 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.18%. Comparing base (b92ca10) to head (026fced).
⚠️ Report is 6 commits behind head on master.

Files with missing lines Patch % Lines
.../db/protocol/thrift/impl/ClientRPCServiceImpl.java 27.08% 35 Missing ⚠️
...ain/java/org/apache/iotdb/rpc/IoTDBRpcDataSet.java 91.52% 5 Missing ⚠️
...ache/iotdb/db/protocol/session/IClientSession.java 25.00% 3 Missing ⚠️
...java/org/apache/iotdb/jdbc/IoTDBJDBCResultSet.java 60.00% 2 Missing ⚠️
...va/org/apache/iotdb/session/SessionConnection.java 84.61% 2 Missing ⚠️
.../db/queryengine/plan/execution/QueryExecution.java 66.66% 2 Missing ⚠️
...java/org/apache/iotdb/isession/SessionDataSet.java 83.33% 1 Missing ⚠️
...e/execution/exchange/source/LocalSourceHandle.java 0.00% 1 Missing ⚠️
...yengine/plan/execution/config/ConfigExecution.java 50.00% 1 Missing ⚠️
...a/org/apache/iotdb/db/utils/QueryDataSetUtils.java 98.66% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #19263      +/-   ##
============================================
+ Coverage     49.05%   49.18%   +0.13%     
- Complexity      689      690       +1     
============================================
  Files          5590     5594       +4     
  Lines        413977   414803     +826     
  Branches      54633    54784     +151     
============================================
+ Hits         203061   204036     +975     
+ Misses       210916   210767     -149     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@JackieTien97 JackieTien97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address the memory accounting and read-size quota regressions introduced by fragmentation, and apply the payload limit to the Fast LAST cache-hit paths. The inline comments describe the affected paths.

@Override
public ByteBuffer getSerializedTsBlock() throws IoTDBException {
public ByteBuffer getSerializedTsBlock(boolean retainMemory) throws IoTDBException {
// the returned bytes are a fresh serialization, not reserved from the memory pool

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Account for pending fragments on the local result path

receive() removes the block from SharedTsBlockQueue and releases its reservation. The serialized buffer can now outlive the RPC because PendingResultFragment holds the entire buffer until the client finishes fetching it. On a single-node deployment, several concurrent readers can therefore hold large buffers outside the query memory pool. Please keep the pending buffer accounted for until the last fragment is sent or the query is cleaned up.

req.fetchSize,
req.isSetOffset() ? req.getOffset() : 0,
req.isSetOffset() ? config.getMppDataExchangeMaxPayloadSizeInBytes() : 0);
finished = fillQueryResult(resp, chunk);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve read-size quota accounting across fragments

The execute path charges only the buffers in its response through quota.addReadResult(). The continuation path here does not charge the remaining bytes. With the default 4 MiB payload limit, a 12 MiB TsBlock that previously fit in the execute response now counts as only 4 MiB against the read-size quota. Please ensure the continuation bytes are accounted for exactly once.

finished = SELECT_RESULT.apply(resp, queryExecution, req.fetchSize);
finished =
SELECT_RESULT.apply(
resp, queryExecution, req.fetchSize, getMaxResultPayloadSize(clientSession));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Apply the payload limit to Fast LAST cache hits

This fallback now honors the negotiated limit. The cache-hit branches in executeFastLastDataQueryForOneDeviceV2 and executeFastLastDataQueryForOnePrefixPath still serialize the entire result into one response and set moreData=false. A large cached result can still exceed the Thrift frame limit even though the fallback can stream it in fragments. Please apply the payload limit to both cache-hit paths.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants