Skip to content

[python] Refuse a read whose query auth rules cannot be bound to columns - #10018

Open
plusplusjiajia wants to merge 1 commit into
apache:masterfrom
plusplusjiajia:fix/pypaimon-auth-fail-closed
Open

plusplusjiajia wants to merge 1 commit into
apache:masterfrom
plusplusjiajia:fix/pypaimon-auth-fail-closed

Conversation

@plusplusjiajia

Copy link
Copy Markdown
Member

Purpose

pypaimon applies the row filter and the column masks that authTableQuery returns. When a rule could not be matched to a column by name, the masking reader skipped it and returned the column untransformed:

  • a mask whose target was renamed in a later schema;
  • a mask on a ROW column where the query projects a sub-field, since read_type() flattens s.a into a field named s_a — projecting a sub-field was enough to bypass the mask at will;
  • a mask whose transform reads another masked column, which transforms see raw.

Java fails closed on all three. This ports validateAgainstSchema, validateReadableWithoutRename and validateReadType, called where Java calls them, plus two things the port needed:

Binding needs both schemas. Identity comes from the latest one, which is what the rules name, so a re-added column or a name moved between columns refuses rather than masking unrelated data. The type comes from the schema the split was written under, so a column widened since then still reads.

Some names cannot be bound at all. _ROW_ID, _SEQUENCE_NUMBER and _VALUE_KIND may each be the synthetic field or a physical column, and a rename makes the two indistinguishable, so any rule naming one is refused. A physical column carrying such a name therefore cannot carry a rule — a loud refusal rather than a silent skip.

Discovery also had to match execution: _resolve_transform_input reads any dict input as a column, while the collector only counted one carrying both name and index, so an indexless reference escaped every check. Both the scan and the reader now run the same validation, so a split reaching a reader unplanned is held to the same rules.

Tests

TestReaderAppliesTheRules drives a real TableRead with only storage stubbed and asserts the returned values, so a reader that is constructed but masks nothing fails. Each original situation has a case, as do the schema-history ones. Reverting any check fails the suite, including omitting the snapshot schema, swapping the two schemas, restoring the name exemption, and reverting the collector.

@plusplusjiajia
plusplusjiajia marked this pull request as draft September 20, 2026 12:19
@plusplusjiajia
plusplusjiajia force-pushed the fix/pypaimon-auth-fail-closed branch 3 times, most recently from 6bf5939 to a66af3e Compare September 20, 2026 17:22
@plusplusjiajia
plusplusjiajia marked this pull request as ready for review September 20, 2026 17:41
@plusplusjiajia
plusplusjiajia force-pushed the fix/pypaimon-auth-fail-closed branch from a66af3e to cc8c50e Compare September 20, 2026 17:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant