What is the problem?
Iceberg residual serialization currently requires both sides of an AND to convert. Collection null checks cannot bind in the pinned iceberg-rust version, so a query such as WHERE l IS NOT NULL AND id > 5 drops the entire residual and loses native row-group pruning on id.
The post-scan filter still enforces the complete predicate, so this is a performance limitation, not incorrect results. PR #5732 allows collection-null scans to remain native. Its review also requests skipping unbindable complex-column null predicates before serialization; the conservative whole-conjunction behavior should remain until safe partial pushdown is implemented.
Proposed improvement
Preserve representable conjuncts when safely weakening a residual in positive polarity. Keep the exact post-scan filter. Do not simply change every AND to keep whichever side converted: beneath NOT, that would strengthen the predicate and can incorrectly prune rows. Likewise, dropping an unsupported OR branch is unsafe.
Validation
- A list/map null predicate combined with a primitive predicate retains primitive row-group pruning and matches Spark results.
- Nested
NOT, OR, and mixed conjunctions preserve all qualifying rows.
- Missing or unsupported predicates continue to disable pushdown whenever safe weakening cannot be established.
- Cover null values and files containing both matching and nonmatching primitive values; verify actual pruning metrics, not only serialized predicate shape.
Requested in the review of #5732: #5732 (review)
What is the problem?
Iceberg residual serialization currently requires both sides of an
ANDto convert. Collection null checks cannot bind in the pinned iceberg-rust version, so a query such asWHERE l IS NOT NULL AND id > 5drops the entire residual and loses native row-group pruning onid.The post-scan filter still enforces the complete predicate, so this is a performance limitation, not incorrect results. PR #5732 allows collection-null scans to remain native. Its review also requests skipping unbindable complex-column null predicates before serialization; the conservative whole-conjunction behavior should remain until safe partial pushdown is implemented.
Proposed improvement
Preserve representable conjuncts when safely weakening a residual in positive polarity. Keep the exact post-scan filter. Do not simply change every
ANDto keep whichever side converted: beneathNOT, that would strengthen the predicate and can incorrectly prune rows. Likewise, dropping an unsupportedORbranch is unsafe.Validation
NOT,OR, and mixed conjunctions preserve all qualifying rows.Requested in the review of #5732: #5732 (review)