[python] Defer BLOB payload reads until after filtering#8808
Draft
XiaoHongbo-Hope wants to merge 5 commits into
Draft
[python] Defer BLOB payload reads until after filtering#8808XiaoHongbo-Hope wants to merge 5 commits into
XiaoHongbo-Hope wants to merge 5 commits into
Conversation
…LOB reads Row-level authorization filters were not accounted for in three blob paths: - Keep the auth filter outermost (after inline BLOB resolution) when it references an inline blob-descriptor/blob-view field, so it compares the resolved payload, not descriptor/BlobViewStruct bytes (was a row-level auth bypass). - Skip pushing LIMIT down to the BlobView prescan reader when an auth filter is present, so the prescan row set matches the outer auth-filtered set (was raising "row id ... was not found in upstream table"). - Enable deferred BLOB resolution when only an auth filter is present (no user predicate/limit), so payloads of auth-dropped rows are not read. Add a regression test for the last case.
… inline BLOB When the auth filter referenced an inline (descriptor/view) BLOB, the previous fix ran it outermost, which disabled deferral for all other scalar BLOBs so their payloads were read for auth-dropped rows too. Instead run the auth filter (and LIMIT) inside the split read, after inline BLOB resolution but before deferred scalar resolution: inline resolve -> auth/limit -> deferred scalar resolve. So it still compares resolved inline payloads while non-auth scalar BLOBs are deferred past it.
XiaoHongbo-Hope
marked this pull request as ready for review
July 23, 2026 14:24
XiaoHongbo-Hope
marked this pull request as draft
July 23, 2026 14:27
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Filtered data-evolution scans previously materialized projected BLOB payloads before applying residual filters and limits. This made filtered-out rows still incur payload I/O.
Read eligible BLOB fields as descriptors through merge, filter and limit, then materialize only surviving rows. Predicates referencing the BLOB field keep the eager path.
read.defer-blob-resolve=falserestores the previous behavior.Tests
deferred_blob_resolve_test.pydata_evolution_deletion_vector_test.pytest_limit_pushdown.pyDedicatedFormatWriterTest#test_update_blob_columnDedicatedFormatWriterTest#test_array_blob_column_write_read_and_updateDedicatedFormatWriterTest#test_map_blob_column_write_read_and_update