Skip to content

fix: skip redundant fetches for complete direct results - #539

Open
vuanhphung wants to merge 7 commits into
mainfrom
fix/cloudfetch-direct-results
Open

vuanhphung wants to merge 7 commits into
mainfrom
fix/cloudfetch-direct-results

Conversation

@vuanhphung

@vuanhphung vuanhphung commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Skip the redundant Thrift FetchResults when direct results explicitly report hasMoreRows=false. Unknown or true flags still fetch normally. For complete CloudFetch and Arrow direct results, this matches Python, JDBC, and Node.

Preserve existing cleanup and error behavior: failed best-effort cleanup after a complete read leaves sql.Rows.Err() clear, while explicit early close still reports its error. Paginated results and Arrow iterator ownership retain their existing behavior.

For large_results_0005mb, timed RPCs change from ExecuteStatement → FetchResults → CloseOperation to ExecuteStatement → CloseOperation.

Client time in milliseconds against main at 95d97b3: medians from four measured pairs after warm-up, subtracting server duration. The paired delta is the median of within-round differences.

Dashboard case Base Fix Paired delta
large_results_0005mb 768.9 683.6 -111.9
large_results_0010mb 846.9 821.7 -88.5
large_results_0025mb 961.0 977.7 -33.6
large_results_0050mb 1201.0 1139.5 -32.3
large_results_0100mb 1781.0 1592.1 -217.5
extended_types_array_inline 192.5 192.6 +0.4
low_latency_select_1 86.6 90.7 +4.3
low_latency_limit_0 91.3 85.5 -3.4

Validation: go test -race ./... and repository lint pass. Public database/sql coverage verifies unchanged EOF, early-close, and fetch-error behavior with successful/failing cleanup. Fresh live checksums match on all eight cases above and the separate 1 GB case; two 1 GB timing rounds return 6,491,176 rows for both versions and retain the needed page fetches.

@vuanhphung
vuanhphung marked this pull request as ready for review November 11, 2026 08:52

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good, with one low-severity API-shape concern. The change is correct: direct-result HasMoreRows=false now skips the extra FetchResults, and rows.Close() with closeOnce closes the operation and runs the telemetry callback exactly once. One related gap is outside this diff: in the GetArrowBatches/IPC path, the iterator's release() still issues CloseOperation itself and drops its error, so in that path Rows.Close() telemetry reports about 0 ms latency and no close error.

@vuanhphung
vuanhphung force-pushed the fix/cloudfetch-direct-results branch from 2333012 to 2405a68 Compare November 12, 2026 04:39

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good overall, with one low concern. Honoring HasMoreRows=false and moving the operation close into a once-only Rows.Close() is correct for database/sql callers, and the new lifecycle and error tests cover that path well. The concern is that GetArrowBatches/GetArrowIPCStreams still close the operation early and drop the error, so close telemetry on those paths is wrong.

Comment thread internal/rows/rows.go Resolved

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good overall, with one low-severity concern. Honoring a direct-results HasMoreRows=false and moving operation cleanup into a sync.Once-guarded rows.Close() is correct, and the new lifecycle and error-reporting tests are thorough. The remaining concern is that the IPC-stream iterator path no longer closes the operation when it is drained.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium · 1 Low

Looks good overall. Honoring HasMoreRows=false from direct results and moving cleanup to Rows.Close() is correct and well tested for row and IPC iteration. One medium concern: the GetArrowBatches iterator still closes the operation itself and drops the error, which defeats the close-error and telemetry handling this PR adds.

Comment thread rows/rows.go Resolved

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium · 1 Low

Mostly sound, with one medium gap: the GetArrowBatches record iterator still closes the operation itself, which bypasses the new Rows.Close() error reporting and telemetry. The sync.Once close, the hasMoreRows short-circuit and the test de-flaking look correct.

Comment thread rows/rows.go Resolved

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good: one low concern about close errors now showing up in sql.Rows.Err() after a clean read. The lifecycle refactor looks correct: closeOnce makes repeated Close() and the telemetry callback idempotent, GetArrowBatches still closes from its own iterator, the HasMoreRows=false short-circuit is limited to direct results, and the new tests cover the paginated, complete and unknown cases plus the database/sql path.

Comment thread internal/rows/rows.go Resolved
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
@vuanhphung vuanhphung changed the title fix: honor CloudFetch end-of-results before closing operations fix: skip redundant fetches for complete direct results Nov 12, 2026
@vuanhphung
vuanhphung force-pushed the fix/cloudfetch-direct-results branch from 735f911 to 9101d3b Compare November 12, 2026 06:10

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant