[SPEC] Add finer grained read restrictions as part of loadTable by singhpk234 · Pull Request #13879 · apache/iceberg

singhpk234 · 2025-08-20T21:33:52Z

About the proposal

This aims at returning, policy evaluation result (Access decisions) for fine grained access policies based on the calling user as part of the loadTable response.

This defines a new object called ReadRestrictions which an optional field catalog would use to convey the access decision, The expectation is these rules (projections and the row filters) are correctly applied and enforced by the client, which brings an implicit requirement to have a trusted partner, establishing trust between callers engine and catalog is not scope of this proposal as its totally up to the catalog on how its established via OAuth Delegation Flow or mTLS.

The ReadRestrictions returns back projections which are modeled as Actions and the row filters are modeled as Expression

we have been iterating about the actions in the community sync :)

Future Extension

Expression Expansion in Progress, which can allow expressions to contain udfs

Proposal : https://docs.google.com/document/d/1VthBz0S2I39TeQM8oiF9_gSPQu_gHAjWXvdFpv0QqDk

Additional consideration

clients expression handling capability : lets say we introduce a new expression or a transform in the IRC spec, and the catalog returns this new expression, but the client is running on a lower version so it doesn't understand this newly added expression, IMHO its fine if the client fails in interpreting / parsing the expression as it doesn't understands it, open to broader feedback, (similar discussion in past for spark community (here))
Schema evolution cases, consider a column got renamed, essentially the DDM and RAP should be Binded, having the field-id of the column they are referring to additionally sent to the client to correctly apply the read restrictions consder one had a filter on column A (when the policy was attached) which got renamed to column B later when reading the latest version of the schema reader could essentially apply filter on B essentially adhering to the column projection rule - https://iceberg.apache.org/spec/#column-projection

Acknowledgement

Many thanks to the entire Iceberg community for the extensive discussions and invaluable feedback over the years, which have been instrumental in shaping this proposal into its current form.

Issues

singhpk234 · 2025-08-20T22:12:05Z

cc @rdblue @RussellSpitzer @flyrain @amogh-jahagirdar

singhpk234 · 2025-09-17T18:41:15Z

Re: Schema evolution cases
There was a general alignment to use boundedReference, working on incorporating that in the feedback

singhpk234 · 2025-09-26T03:03:29Z

There was a general alignment to use boundedReference, working on incorporating that in the feedback

Update - actively trying a POC for this in my fork (PR : singhpk234#270 if folks are interested to give some early feedback)

…ULL note Per rdblue review: predicates (not expressions in general) are what cannot produce NULL. Updates the required-row-filter description in both rest-catalog-open-api.yaml and the regenerated .py.

Introduces the Action abstraction from the ReadRestrictions spec (PR apache#13879) as a standalone addition, covering the 10 predefined column projection actions: - MaskAlphanum, ShowFirst4, ShowLast4, MaskToFixedValue, ReplaceWithNull, TruncateToYear, TruncateToMonth, Sha256Global, Sha256QueryLocal, ApplyExpression - Plus an Unknown forward-compat carrier for unrecognized server-side action types so the client fails closed at bind time Each action carries a field id plus any action-specific payload and owns its own bind(Type) returning a SerializableFunction. This mirrors the Transform<S, T> pattern used in partition transforms. Null input produces null output for every action, per spec. No REST wire-format plumbing or engine integration yet; those follow in separate PRs.

… map Extends canBind() and defaultValueFor() to cover the complex types listed in the ReadRestrictions spec for mask-to-fixed-value: - variant -> empty Variant (V1 metadata + empty object value) - geometry -> WKB POINT EMPTY (little-endian, X=Y=NaN) - geography -> WKB POINT EMPTY (same encoding) - list -> empty list - map -> empty map struct is intentionally deferred (TODO) until the recursive type-specific default helper lands; the spec allows excluding nested types per rdblue review (apache#13879).

Per rdblue review: POINT EMPTY's WKB encoding is not defined in Iceberg's format spec (Appendix G defers to OGC SFA, which does not specify empty-geometry WKB), and is not formally standardized in GeoParquet, PostGIS, Apache Parquet's geospatial spec, or any other source that Iceberg cites. The "POINT(NaN, NaN)" convention is widely-used implementation lore but never canonicalized. This also aligns with format/spec.md L333 which states that columns of unknown, variant, geometry, and geography types must default to null and explicitly disallows non-null initial-default / write-default values for those types. Drops: - geometry: POINT EMPTY - geography: POINT EMPTY Tightens Applicable-to clause to exclude unknown, geometry, geography. variant is kept in the fixed-value table because Iceberg's Java API defines a canonical empty Variant byte sequence (EMPTY_V1_BUFFER), giving an implementation-anchorable encoding that geometry/geography lack.

stevenzwu · 2026-06-30T22:51:35Z

+        - float: 0.0
+        - double: 0.0
+        - decimal(p, s): 0 (zero with s digits after the decimal point, e.g. 0.00 for decimal(p,2))
+        - string: "XXXXXXXX"


Reopening to flag a few concerns the thread didn't engage. What other systems use as the default string constant for the equivalent "replace with fixed value" masking:

System Default string constant Source

SQL Server Dynamic Data Masking default() "XXXX" (4 X's, or fewer if column narrower) define-a-dynamic-data-mask

BigQuery Default masking value "" (empty string) column-data-masking-intro#masking_options

Oracle DBMS_REDACT FULL redaction " " (single space) Oracle Data Redaction 3.1

Snowflake masking policy user-defined; docs example uses '********' CREATE MASKING POLICY example

PostgreSQL Anonymizer MASKED WITH VALUE user-defined; docs example uses 'CONFIDENTIAL' Destruction

Of the three systems that ship a mandated default, none use 8 X's: SQL Server picks 4 X's, BigQuery picks empty, Oracle picks a single space. The specific choice of "XXXXXXXX" has no precedent.

Other concerns:

Breaks the spec's own internal pattern. Every other fixed value in this action is the type's empty/minimum form: int: 0, binary: empty byte sequence, list: empty list, map: empty map, variant: {}. The matching minimum for string is "". "XXXXXXXX" is the only entry that violates the convention.

The mask-alphanum analogy isn't quite applicable here. mask-alphanum preserves length and replaces characters in place — that's where the familiar X-pattern comes from. mask-to-fixed-value is the "replace entirely with a constant" mode, a different design. Users who want X-pattern masking already have mask-alphanum.

Suggest string: "" — matches BigQuery, matches this action's per-type empty/collection pattern.

- Use 'reader' consistently in loadTable read-restrictions prose - Rename 'descendant field-id' to 'nested field-id' to match spec terminology - State both actors in the duplicate field-id rule (server must not return, reader must fail) - Forbid mask-to-fixed-value on containers nesting unknown/geometry/geography Regenerate rest-catalog-open-api.py.

Introduces the IcebergFunction<S, T> abstraction from the ReadRestrictions spec (PR apache#13879), covering the 9 predefined column projections used by fine-grained access control: - MaskAlphanum, ShowFirst4, ShowLast4 - MaskToFixedValue, ReplaceWithNull - TruncateToYear, TruncateToMonth - Sha256Global, Sha256QueryLocal (via SaltedFunction, requires salt >= 16 B) Plus UnknownFunction as a forward-compat carrier for unrecognized server-side names so clients fail closed at bind time. Each function carries a field id and any per-function payload and owns bind(Type), returning a SerializableFunction. Mirrors the Transform<S, T> pattern used in partition transforms. Null input produces null output for every function except MaskToFixedValue, which returns the type-specific fixed value per spec. No REST wire-format plumbing or engine integration yet; those follow in separate PRs.

huaxingao · 2026-07-06T23:14:03Z

+            action: mask-alphanum
+        required-row-filter:
+          type: eq
+          term: region


Projections use field-id. Should the row filter use field id too?

required-row-filter is an iceberg expression which refers to one or many field ids

We need new expression to spec to have IDReference, added a pr for this #17138

huaxingao

LGTM

flyrain

LGTM. Thanks a lot for working on it, @singhpk234 !

laurentgo

overall okay with the current proposal, my only possible regret is the lack of header/signal sent by client to advertise support as a sanity check for servers

laurentgo · 2026-07-07T05:10:58Z

+    Replaces the column value with a type-specific fixed value. Readers must use exactly the values listed below to ensure consistency across implementations.
+    Fixed values by type: - boolean: false - int: 0 - long: 0 - float: 0.0 - double: 0.0 - decimal(p, s): 0 (the unscaled value is 0) - string: "XXXXXXXX" - date: 1970-01-01 - time: 00:00:00 - timestamp: 1970-01-01T00:00:00 - timestamptz: 1970-01-01T00:00:00+00:00 - timestamp_ns: 1970-01-01T00:00:00.000000000 - timestamptz_ns: 1970-01-01T00:00:00.000000000+00:00 - uuid: 00000000-0000-0000-0000-000000000000 - fixed(n): n zero bytes - binary: empty byte sequence - variant: {} - list: empty list [] - map: empty map {} - struct: struct with each field set to its type-specific default (applied recursively)
+    NULL input is also replaced with the type-specific fixed value; NULL is not preserved.
+    Applicable to: all data types except unknown, geometry, and geography. Because those three types have no fixed value defined above, a catalog server must not return mask-to-fixed-value for a struct, list, or map that contains a field of type unknown, geometry, or geography at any nesting depth.


(nit) not sure if it is better to come up with a list where this action is supported vs a list where this action is not supported. It seems safer to use the former so that new types would be "blocked" until the spec is updated, vs making the sure this part of the spec is also updated when you type is being addred (if needed).

Or maybe Iceberg spec could define a default value per type when it makes sense and refer to that generically

laurentgo · 2026-07-07T05:16:56Z

+
+        NULL input is also replaced with the type-specific fixed value; NULL is not preserved.
+
+        Applicable to: all data types except unknown, geometry, and geography.


(nit) not sure if it is better to come up with a list where this action is supported vs a list where this action is not supported. It seems safer to use the former so that new types would be "blocked" until the spec is updated, vs making the sure this part of the spec is also updated when you type is being addred (if needed).

Or maybe Iceberg spec could define a default value per type when it makes sense and refer to that generically

I agree allowlist might be better in this scenario because v4 is gonna add more type and its gonna add more confusion let me give this more thought

laurentgo · 2026-07-07T05:20:40Z

+            action: mask-alphanum
+        required-row-filter:
+          type: eq
+          term: region


required-row-filter is an iceberg expression which refers to one or many field ids

laurentgo · 2026-07-07T05:21:02Z

+
+    MaskAlphanum:
+      description: >
+        Redacts the column value using the following rules to transform Unicode code points:


(nit) it seems we are using the word replaces for most actions, I wonder if we should align by saying "Replaces the column value with a redacted value created using the following rules..."

laurentgo · 2026-07-07T05:26:33Z

+        - float: 0.0
+        - double: 0.0
+        - decimal(p, s): 0 (zero with s digits after the decimal point, e.g. 0.00 for decimal(p,2))
+        - string: "XXXXXXXX"


I don't care as well, discussing it more is just bikeshedding at this point.

laurentgo · 2026-07-07T05:27:41Z

+        - geography: POINT EMPTY
+        - list: empty list []
+        - map: empty map {}
+        - struct: struct with each field set to its type-specific default (applied recursively)


danielcweeks · 2026-07-07T15:05:14Z

+            5. The reader must fail the query if a projection references a field-id
+              that is not present in the read schema.


I don't think this is correct. A read restriction should be valid if the referenced id is in any valid schema for the table (not just the current one or the one used in the current read). The way this is worded is confusing because a schema could be used to read that isn't the current schema and the restrictions may reference columns that did not exist in that schema. We should limit this.

We discussed this in sync. My understanding is that we're concerned about what happens when projection schema and read schema differ. Here's a table to illustrate.

# Field-id in projection schema? Field-id in read schema? example When this happens Recommendation

1 ✓ Yes ✓ Yes Normal read Enforce policy

2 ✓ Yes ✗ No Time travel to a snapshot predating the column Fail the query — a policy that favors reads would need to define how to fill in missing columns. Recommend deferring; BigQuery takes the same position and fails this case.

3 ✗ No ✓ Yes Column dropped via ALTER TABLE, reading a historical snapshot where the column still existed Allow read — not selecting data that no longer exists is itself a valid form of protection (@rdblue stated this principle in sync)

4 ✗ No ✗ No n/a included for completeness n/a

I'd like to recommend the following spec text.

(5) The reader MUST fail the query if a projection references a field-id that is not present in the snapshot schema.

If we would like to be more clear about allowing historical reads we can add a clarification, but I think (5) is sufficient.

(6) The reader MUST skip enforcement if a field-id is present in the snapshot schema but not referenced by the projection.

cc: @singhpk234.

Thanks Dan and Nevin
This is what i ended up writing :
A reader must enforce projections on the columns it is actually reading. Projections referencing columns that are not being read do not apply.

We want to say if you are reading a column and if there is read restriction projection for it then you must apply that read restriction, if you can't you must fail and if you are not reading that particular column (doesn't matter if its part of the snapshot schema you are reading vs even if there is a column which is present in the snapshot schema).

Let me know you thouhgts

- Reframe rule 5 around what the reader is actually reading per Ryan's sync framing, dropping the invented "read schema" vocabulary that Dan flagged as confusing. - Replace "authenticated principal" and "global policy" (undefined terms per Dan) with "authentication scope" in the loadTable caching note and drop the phrase from required-row-filter. - Scope the fail rule to conforming readers per huaxingao's nit. - Reword the "missing or empty" clause to drop the overloaded "field" per flyrain's nit. - Move the required/non-nullable note into ReplaceWithNull's "Applicable to" line per flyrain's nit.

Per laurentgo's review nit on yaml:3835 — flip from an exclusion list ("all types except unknown, geometry, geography") to an inclusion by reference to the "Fixed values by type" list above. New Iceberg types are then implicitly blocked until the spec explicitly extends the fixed-values list to cover them. The container-nesting carve-out is dropped: it's redundant with the MUST NOT clause because a struct containing a field with no fixed value isn't a "type with a fixed value defined above" — the recursive definition simply doesn't apply.

github-actions Bot added the OPENAPI label Aug 20, 2025

singhpk234 force-pushed the feature/load-table-return-policy branch from 1234341 to 6422478 Compare August 20, 2025 21:41

rdblue reviewed Aug 20, 2025

View reviewed changes

Comment thread open-api/rest-catalog-open-api.yaml Outdated

rdblue reviewed Aug 20, 2025

View reviewed changes

Comment thread open-api/rest-catalog-open-api.yaml Outdated

rdblue reviewed Aug 20, 2025

View reviewed changes

Comment thread open-api/rest-catalog-open-api.yaml Outdated

rdblue reviewed Aug 20, 2025

View reviewed changes

Comment thread open-api/rest-catalog-open-api.yaml Outdated

rdblue reviewed Aug 20, 2025

View reviewed changes

Comment thread open-api/rest-catalog-open-api.yaml Outdated

rdblue reviewed Aug 20, 2025

View reviewed changes

Comment thread open-api/rest-catalog-open-api.yaml Outdated

amogh-jahagirdar reviewed Aug 20, 2025

View reviewed changes

Comment thread open-api/rest-catalog-open-api.yaml Outdated

Comment thread open-api/rest-catalog-open-api.yaml Outdated

rdblue reviewed Aug 20, 2025

View reviewed changes

Comment thread open-api/rest-catalog-open-api.yaml Outdated

rdblue reviewed Aug 20, 2025

View reviewed changes

Comment thread open-api/rest-catalog-open-api.yaml Outdated

flyrain reviewed Aug 21, 2025

View reviewed changes

Comment thread open-api/rest-catalog-open-api.yaml Outdated

flyrain reviewed Aug 21, 2025

View reviewed changes

Comment thread open-api/rest-catalog-open-api.yaml Outdated

singhpk234 force-pushed the feature/load-table-return-policy branch from 9c1b305 to 6c5b955 Compare August 21, 2025 21:36

singhpk234 changed the title ~~[SPEC] Add FGAC enforcement instructions as part of loadTable~~ [SPEC] Add finer grained read restrictions as part of loadTable Aug 22, 2025

rdblue reviewed Aug 22, 2025

View reviewed changes

Comment thread open-api/rest-catalog-open-api.yaml Outdated

RussellSpitzer reviewed Aug 25, 2025

View reviewed changes

Comment thread open-api/rest-catalog-open-api.py Outdated

RussellSpitzer reviewed Aug 25, 2025

View reviewed changes

Comment thread open-api/rest-catalog-open-api.py Outdated

RussellSpitzer reviewed Aug 25, 2025

View reviewed changes

Comment thread open-api/rest-catalog-open-api.yaml Outdated

RussellSpitzer reviewed Aug 25, 2025

View reviewed changes

Comment thread open-api/rest-catalog-open-api.yaml Outdated

sfc-gh-jxu reviewed Sep 18, 2025

View reviewed changes

Comment thread open-api/rest-catalog-open-api.yaml Outdated

Comment thread open-api/rest-catalog-open-api.yaml Outdated

singhpk234 added the Specification Issues that may introduce spec changes. label Sep 19, 2025

stevenzwu reviewed Sep 19, 2025

View reviewed changes

Comment thread open-api/rest-catalog-open-api.yaml Outdated

Comment thread open-api/rest-catalog-open-api.yaml Outdated

Comment thread open-api/rest-catalog-open-api.yaml Outdated

Comment thread open-api/rest-catalog-open-api.yaml Outdated

pvary reviewed Sep 22, 2025

View reviewed changes

Comment thread open-api/rest-catalog-open-api.yaml Outdated

singhpk234 marked this pull request as ready for review September 26, 2025 23:11

singhpk234 mentioned this pull request Sep 26, 2025

[Proposal] [REST] Fine Grained Access Control #14187

Closed

6 tasks

adutra reviewed Oct 1, 2025

View reviewed changes

Comment thread open-api/rest-catalog-open-api.yaml Outdated

sfc-gh-prsingh force-pushed the feature/load-table-return-policy branch from b1e909b to f5ed94e Compare June 29, 2026 18:01

Spec: Use 'Iceberg predicates' instead of 'Iceberg expressions' for N…

422315a

…ULL note Per rdblue review: predicates (not expressions in general) are what cannot produce NULL. Updates the required-row-filter description in both rest-catalog-open-api.yaml and the regenerated .py.

singhpk234 requested review from RussellSpitzer, c-thiel, danielcweeks, rdblue, stevenzwu, sungwy and talatuyarer June 29, 2026 18:27

stevenzwu reviewed Jul 1, 2026

View reviewed changes

sfc-gh-prsingh added 3 commits July 1, 2026 10:57

Spec: Address Steven's comments on field-id presence in read schema

d67103b

Spec: Regenerate rest-catalog-open-api.py

14125c5

stevenzwu approved these changes Jul 1, 2026

View reviewed changes

huaxingao reviewed Jul 6, 2026

View reviewed changes

Comment thread open-api/rest-catalog-open-api.yaml Outdated

huaxingao approved these changes Jul 6, 2026

View reviewed changes

flyrain approved these changes Jul 7, 2026

View reviewed changes

Comment thread open-api/rest-catalog-open-api.py Outdated

Comment thread open-api/rest-catalog-open-api.yaml

Comment thread open-api/rest-catalog-open-api.yaml Outdated

laurentgo reviewed Jul 7, 2026

View reviewed changes

danielcweeks reviewed Jul 7, 2026

View reviewed changes

Comment thread open-api/rest-catalog-open-api.yaml Outdated

danielcweeks reviewed Jul 7, 2026

View reviewed changes

Comment thread open-api/rest-catalog-open-api.yaml Outdated

sfc-gh-prsingh added 4 commits July 9, 2026 10:18

Spec: Regenerate rest-catalog-open-api.py

92e76c5

Spec: Regenerate rest-catalog-open-api.py

5044d53

System	Default string constant	Source
SQL Server Dynamic Data Masking `default()`	`"XXXX"` (4 X's, or fewer if column narrower)	define-a-dynamic-data-mask
BigQuery Default masking value	`""` (empty string)	column-data-masking-intro#masking_options
Oracle `DBMS_REDACT` FULL redaction	`" "` (single space)	Oracle Data Redaction 3.1
Snowflake masking policy	user-defined; docs example uses `'********'`	CREATE MASKING POLICY example
PostgreSQL Anonymizer `MASKED WITH VALUE`	user-defined; docs example uses `'CONFIDENTIAL'`	Destruction


		NULL input is also replaced with the type-specific fixed value; NULL is not preserved.

		Applicable to: all data types except unknown, geometry, and geography.

		5. The reader must fail the query if a projection references a field-id
		that is not present in the read schema.

#	Field-id in projection schema?	Field-id in read schema?	example When this happens	Recommendation
1	✓ Yes	✓ Yes	Normal read	Enforce policy
2	✓ Yes	✗ No	Time travel to a snapshot predating the column	Fail the query — a policy that favors reads would need to define how to fill in missing columns. Recommend deferring; BigQuery takes the same position and fails this case.
3	✗ No	✓ Yes	Column dropped via `ALTER TABLE`, reading a historical snapshot where the column still existed	Allow read — not selecting data that no longer exists is itself a valid form of protection (@rdblue stated this principle in sync)
4	✗ No	✗ No	n/a included for completeness	n/a

Uh oh!

Conversation

singhpk234 commented Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

About the proposal

Future Extension

Additional consideration

Acknowledgement

Issues

dev lists

docs

syncs

Uh oh!

singhpk234 commented Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

singhpk234 commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

singhpk234 commented Sep 26, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

huaxingao left a comment

Choose a reason for hiding this comment

Uh oh!

flyrain left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

laurentgo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

singhpk234 commented Aug 20, 2025 •

edited

Loading

singhpk234 commented Aug 20, 2025 •

edited

Loading

singhpk234 commented Sep 17, 2025 •

edited

Loading

danielcweeks Jul 7, 2026 •

edited

Loading

nevzheng Jul 7, 2026 •

edited

Loading