Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions platform/data-governance/data/masks-and-filters.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,12 @@ A SQL boolean expression over the table's columns, for example `region = 'us'` o
| **Path** | **Scope**: catalog (required), schema, table, and column for a mask. | A row filter by path needs catalog, schema and table together, and never a column; the dialog shows no column field and blocks saving until the three are set. A line below the fields reads *Applies to demo_warehouse.analytics.sessions*. |
| **Classification** | **Classification**: pick a classification. | The rule applies to every table or column carrying the tag. The tag is stored by its key, shown as *stored as personal-data-pii*. **Change** picks another. A tag rule is catalog-unscoped. |

A mask by **Path** that leaves **Column** empty masks every column below the path: every column of the table, of every
table in the schema, or of every table in the catalog. On Trino and StarRocks it compiles to a masking policy with `*` at
each level left empty (since platform #837). A mask on a whole catalog leaves the catalog's own `information_schema`
alone, so `SHOW TABLES`, `SHOW COLUMNS`, `DESCRIBE` and JDBC metadata calls keep returning real names. The engine's
**Policies** tab lists such a mask as, for example, `wh.!information_schema.*.*` (`!` marks an excluded value).

### Exempt

The principals the rule does not apply to. The picker offers **user** and **group** only, and no **Exclude** list, because Ranger's mask and row-filter items cannot express a role or an exclusion. Egret refuses both at compile time.
Expand All @@ -90,7 +96,7 @@ A StarRocks table has two hidden columns, `_file` (the object-store path of each
mask, egret refuses them to a reader who does not see the real value in every masked column of the table. That reader's
`SELECT *` on the table fails with error 5204 on `_file`. Name the columns you need, or read through a view over the table:
a view shows the masked values and no path. A reader who sees the real value in every masked column can still use
`SELECT *`.
`SELECT *`. This covers every table that a mask on a whole table, schema or catalog reaches.

A change to the masks takes effect on the next compile of the profile. To apply it at once, compile the profile with
`POST /api/control-plane/perm/compile/{profile_id}`.
Expand Down Expand Up @@ -143,7 +149,7 @@ The local platform admin `admin@egret.local` got the same masked values: platfor
- A rule that selects by path is copied to the new name when its table or column is renamed, and the original stays on the old name; a rule that selects by tag follows because classifications do. A rule that selects by a pattern the new name escapes refuses the rename. See [Renames and drops](./grants.md#renames-and-drops).
- A row filter or custom mask expression that names a renamed column is not rewritten (KI-321).
- On official StarRocks images, masks and row filters do not reach a table's Iceberg metadata tables (`orders$files`, `orders$partitions`, …), which StarRocks serves to any login without asking Ranger, so column bounds, partition values and counts are readable regardless of rules (KI-340); they do not apply to prepared statements, so a Data API endpoint with parameters returns rows and values in the clear (KI-345); they do not apply to the HTTP SQL API or Arrow Flight SQL (KI-346), nor to `FILES()`, which reads the data files directly (KI-347); a materialized view stores its rows past its creator's masks and row filters (KI-348); and `EXPLAIN COSTS` shows a filtered person each column's lowest and highest value over the rows the filter hides (KI-350). Egret's patched image `ghcr.io/egret-data/egret-starrocks-fe` `4.1.7-egret.4`, the default for new engines, fixes all of them; on an official image, do not give StarRocks logins to people who must not see the values.
- On Trino, a table's metadata tables (`orders$files`, `orders$partitions`, …) follow its masks, row filters and denies (since platform #821): a person a mask, row filter or deny on the table restricts gets *Access Denied* on every one of them, and a person it exempts or does not restrict reads them as before. A tag rule or clearance does the same for each table that carries the classification. Profiles that egret's permission model does not compile get no such deny (KI-341).
- On Trino, a table's metadata tables (`orders$files`, `orders$partitions`, …) follow its masks, row filters and denies (since platform #821), including a mask on the table's schema or catalog (since platform #837): a person a mask, row filter or deny on the table restricts gets *Access Denied* on every one of them, and a person it exempts or does not restrict reads them as before. A tag rule or clearance does the same for each table that carries the classification. Profiles that egret's permission model does not compile get no such deny (KI-341).
- On StarRocks, `SELECT *` on an Iceberg table shows the hidden `_file` and `_pos` columns to a user with a row filter (KI-337). List the columns instead.

## Further reading
Expand Down
1 change: 1 addition & 0 deletions platform/data-governance/engines-and-policy/migrate.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ If the profiles changed since the preview, a migration reports *Migrating found
- Migration is proven for Trino with plugin 2.5.0 or 2.8.0 and StarRocks with 2.8.0 or 2.9.0. For another plugin version the tab refuses.
- A policy naming a group or role in a way the model cannot express is **Blocked**.
- A policy that names a tag egret has no tag for is **Blocked**: the migration never creates tags. Create a tag whose key and name are both the policy's tag, or keep the policy hand-written (KI-326).
- A hand-written masking policy that names no `column` level, or stops higher, masks nothing on Trino or StarRocks and is **Blocked**. To mask every column below its path, set the missing levels to `*` in the policy editor; that takes effect at once, and the policy can then be migrated. A policy with a gap (for example, a catalog and a table but no schema) is told to name the missing level, or to set only that level to `*`.
- A hand-written tag access policy is **Blocked** and never becomes a clearance; recreate it as a classification clearance (KI-327).
- Policies egret's own plumbing writes (engine delegation, admin access, query limits and the policies **Seed default policies** adds) are **Locked** and are not rewritten (KI-328).

Expand Down
44 changes: 44 additions & 0 deletions platform/deploy-operate/upgrade.md
Original file line number Diff line number Diff line change
Expand Up @@ -501,6 +501,50 @@ There is no migration and no API change.
differs, recompile the profile first ([`POST /api/control-plane/perm/compile/{service_id}`](../reference/rest-api/perm.md)) to bring them in line, then
migrate again.

### Masks on a whole table, schema or catalog take effect (platform #837)

Before platform #837, a mask rule by path that left **Column** empty compiled without the column level, and Trino and
StarRocks applied it to no column. Those columns came back unmasked although the rule was listed as live. From #837 the
rule masks every column below its path, as the rule dialog says.

There is no migration. At its first start the new version recompiles each policy profile that still serves such a mask,
and the plugins download the new policies once. Later starts skip profiles already repaired.

**Before upgrading**, run `GET /api/control-plane/perm/overlaps` (or open each engine's **Overlaps** tab). It lists every
pair of masks or row filters stored in a profile's download that can decide the same request two ways. Resolve every
pair in which one member is a mask by path with an empty **Column**: delete one of the two, or narrow one so they no
longer meet. Such a pair has no effect today, because one member masks nothing. After the upgrade, recompiling the
profile would make the pair real, so egret refuses the recompile:

- the profile keeps serving its previous compile, so its whole-table, schema and catalog masks stay **inert**;
- every write that recompiles the profile is refused with a 409 conflict naming the pair until the pair is resolved.
That includes grants, rules, catalog grants, workspace membership and role changes on the profile's catalogs, and
**any classification (tag) rule write in the installation**, which recompiles every profile;
- the server logs the refusal at error level at every start. On the Governance overview, **Needs attention** lists a
**Critical** finding, *The last compile of 1 profile failed; engines keep the previous compile. The refusal names rules
A and B.*, with the profile's line reading *Refused at boot: masks with no column stay inert (previous compile
served); writes recompiling this profile are refused until these rules are resolved.* and links to both rules.

The pair's coordinate on the **Overlaps** tab does not say which member has an empty Column; open each rule to see. If
the recompile fails because of a database error or a lock timeout, the line reads *Boot recompile failed: masks with no
column stay inert (previous compile served); the next boot retries.*, and writes are not blocked by it. Any other
refusal, such as an expansion ceiling, shows the same line but keeps refusing writes until it is fixed, as it did
before the upgrade.

**What readers notice after the upgrade:**

- columns under a whole-table, schema or catalog mask are masked for everyone the rule does not exempt;
- on Trino, the metadata tables (`$files`, `$partitions`, `$snapshots`, …) of every table under a schema or catalog mask
are denied to the same people;
- on StarRocks, `SELECT *` on those tables is refused to them (error 5204);
- a rule migrated earlier from a hand-written masking policy that named no column now masks; the original never did.
Review migrated mask rules with an empty Column.

A mask on a whole catalog does not apply to the catalog's `information_schema`. The engine's **Policies** tab lists it
as, for example, `wh.!information_schema.*.*` (`!` marks an excluded value). A mask on a whole table lists as
`wh.sales.orders.*`. A hand-written masking policy that names no column is left as it is and still masks nothing;
migrating it is refused until its missing levels are set (see [Migrate policies](../data-governance/engines-and-policy/migrate.md)).

## Rolling back

There is no supported in-place rollback. Restore the database from the backup taken in step 2, then start the previous
Expand Down
Loading