diff --git a/platform/data-governance/data/masks-and-filters.md b/platform/data-governance/data/masks-and-filters.md index dfd8c7d..85e05eb 100644 --- a/platform/data-governance/data/masks-and-filters.md +++ b/platform/data-governance/data/masks-and-filters.md @@ -76,6 +76,12 @@ A SQL boolean expression over the table's columns, for example `region = 'us'` o | **Path** | **Scope**: catalog (required), schema, table, and column for a mask. | A row filter by path needs catalog, schema and table together, and never a column; the dialog shows no column field and blocks saving until the three are set. A line below the fields reads *Applies to demo_warehouse.analytics.sessions*. | | **Classification** | **Classification**: pick a classification. | The rule applies to every table or column carrying the tag. The tag is stored by its key, shown as *stored as personal-data-pii*. **Change** picks another. A tag rule is catalog-unscoped. | +A mask by **Path** that leaves **Column** empty masks every column below the path: every column of the table, of every +table in the schema, or of every table in the catalog. On Trino and StarRocks it compiles to a masking policy with `*` at +each level left empty (since platform #837). A mask on a whole catalog leaves the catalog's own `information_schema` +alone, so `SHOW TABLES`, `SHOW COLUMNS`, `DESCRIBE` and JDBC metadata calls keep returning real names. The engine's +**Policies** tab lists such a mask as, for example, `wh.!information_schema.*.*` (`!` marks an excluded value). + ### Exempt The principals the rule does not apply to. The picker offers **user** and **group** only, and no **Exclude** list, because Ranger's mask and row-filter items cannot express a role or an exclusion. Egret refuses both at compile time. @@ -90,7 +96,7 @@ A StarRocks table has two hidden columns, `_file` (the object-store path of each mask, egret refuses them to a reader who does not see the real value in every masked column of the table. That reader's `SELECT *` on the table fails with error 5204 on `_file`. Name the columns you need, or read through a view over the table: a view shows the masked values and no path. A reader who sees the real value in every masked column can still use -`SELECT *`. +`SELECT *`. This covers every table that a mask on a whole table, schema or catalog reaches. A change to the masks takes effect on the next compile of the profile. To apply it at once, compile the profile with `POST /api/control-plane/perm/compile/{profile_id}`. @@ -143,7 +149,7 @@ The local platform admin `admin@egret.local` got the same masked values: platfor - A rule that selects by path is copied to the new name when its table or column is renamed, and the original stays on the old name; a rule that selects by tag follows because classifications do. A rule that selects by a pattern the new name escapes refuses the rename. See [Renames and drops](./grants.md#renames-and-drops). - A row filter or custom mask expression that names a renamed column is not rewritten (KI-321). - On official StarRocks images, masks and row filters do not reach a table's Iceberg metadata tables (`orders$files`, `orders$partitions`, …), which StarRocks serves to any login without asking Ranger, so column bounds, partition values and counts are readable regardless of rules (KI-340); they do not apply to prepared statements, so a Data API endpoint with parameters returns rows and values in the clear (KI-345); they do not apply to the HTTP SQL API or Arrow Flight SQL (KI-346), nor to `FILES()`, which reads the data files directly (KI-347); a materialized view stores its rows past its creator's masks and row filters (KI-348); and `EXPLAIN COSTS` shows a filtered person each column's lowest and highest value over the rows the filter hides (KI-350). Egret's patched image `ghcr.io/egret-data/egret-starrocks-fe` `4.1.7-egret.4`, the default for new engines, fixes all of them; on an official image, do not give StarRocks logins to people who must not see the values. -- On Trino, a table's metadata tables (`orders$files`, `orders$partitions`, …) follow its masks, row filters and denies (since platform #821): a person a mask, row filter or deny on the table restricts gets *Access Denied* on every one of them, and a person it exempts or does not restrict reads them as before. A tag rule or clearance does the same for each table that carries the classification. Profiles that egret's permission model does not compile get no such deny (KI-341). +- On Trino, a table's metadata tables (`orders$files`, `orders$partitions`, …) follow its masks, row filters and denies (since platform #821), including a mask on the table's schema or catalog (since platform #837): a person a mask, row filter or deny on the table restricts gets *Access Denied* on every one of them, and a person it exempts or does not restrict reads them as before. A tag rule or clearance does the same for each table that carries the classification. Profiles that egret's permission model does not compile get no such deny (KI-341). - On StarRocks, `SELECT *` on an Iceberg table shows the hidden `_file` and `_pos` columns to a user with a row filter (KI-337). List the columns instead. ## Further reading diff --git a/platform/data-governance/engines-and-policy/migrate.md b/platform/data-governance/engines-and-policy/migrate.md index d41adce..0398044 100644 --- a/platform/data-governance/engines-and-policy/migrate.md +++ b/platform/data-governance/engines-and-policy/migrate.md @@ -81,6 +81,7 @@ If the profiles changed since the preview, a migration reports *Migrating found - Migration is proven for Trino with plugin 2.5.0 or 2.8.0 and StarRocks with 2.8.0 or 2.9.0. For another plugin version the tab refuses. - A policy naming a group or role in a way the model cannot express is **Blocked**. - A policy that names a tag egret has no tag for is **Blocked**: the migration never creates tags. Create a tag whose key and name are both the policy's tag, or keep the policy hand-written (KI-326). +- A hand-written masking policy that names no `column` level, or stops higher, masks nothing on Trino or StarRocks and is **Blocked**. To mask every column below its path, set the missing levels to `*` in the policy editor; that takes effect at once, and the policy can then be migrated. A policy with a gap (for example, a catalog and a table but no schema) is told to name the missing level, or to set only that level to `*`. - A hand-written tag access policy is **Blocked** and never becomes a clearance; recreate it as a classification clearance (KI-327). - Policies egret's own plumbing writes (engine delegation, admin access, query limits and the policies **Seed default policies** adds) are **Locked** and are not rewritten (KI-328). diff --git a/platform/deploy-operate/upgrade.md b/platform/deploy-operate/upgrade.md index e287b40..5dda4fc 100644 --- a/platform/deploy-operate/upgrade.md +++ b/platform/deploy-operate/upgrade.md @@ -501,6 +501,50 @@ There is no migration and no API change. differs, recompile the profile first ([`POST /api/control-plane/perm/compile/{service_id}`](../reference/rest-api/perm.md)) to bring them in line, then migrate again. +### Masks on a whole table, schema or catalog take effect (platform #837) + +Before platform #837, a mask rule by path that left **Column** empty compiled without the column level, and Trino and +StarRocks applied it to no column. Those columns came back unmasked although the rule was listed as live. From #837 the +rule masks every column below its path, as the rule dialog says. + +There is no migration. At its first start the new version recompiles each policy profile that still serves such a mask, +and the plugins download the new policies once. Later starts skip profiles already repaired. + +**Before upgrading**, run `GET /api/control-plane/perm/overlaps` (or open each engine's **Overlaps** tab). It lists every +pair of masks or row filters stored in a profile's download that can decide the same request two ways. Resolve every +pair in which one member is a mask by path with an empty **Column**: delete one of the two, or narrow one so they no +longer meet. Such a pair has no effect today, because one member masks nothing. After the upgrade, recompiling the +profile would make the pair real, so egret refuses the recompile: + +- the profile keeps serving its previous compile, so its whole-table, schema and catalog masks stay **inert**; +- every write that recompiles the profile is refused with a 409 conflict naming the pair until the pair is resolved. + That includes grants, rules, catalog grants, workspace membership and role changes on the profile's catalogs, and + **any classification (tag) rule write in the installation**, which recompiles every profile; +- the server logs the refusal at error level at every start. On the Governance overview, **Needs attention** lists a + **Critical** finding, *The last compile of 1 profile failed; engines keep the previous compile. The refusal names rules + A and B.*, with the profile's line reading *Refused at boot: masks with no column stay inert (previous compile + served); writes recompiling this profile are refused until these rules are resolved.* and links to both rules. + +The pair's coordinate on the **Overlaps** tab does not say which member has an empty Column; open each rule to see. If +the recompile fails because of a database error or a lock timeout, the line reads *Boot recompile failed: masks with no +column stay inert (previous compile served); the next boot retries.*, and writes are not blocked by it. Any other +refusal, such as an expansion ceiling, shows the same line but keeps refusing writes until it is fixed, as it did +before the upgrade. + +**What readers notice after the upgrade:** + +- columns under a whole-table, schema or catalog mask are masked for everyone the rule does not exempt; +- on Trino, the metadata tables (`$files`, `$partitions`, `$snapshots`, …) of every table under a schema or catalog mask + are denied to the same people; +- on StarRocks, `SELECT *` on those tables is refused to them (error 5204); +- a rule migrated earlier from a hand-written masking policy that named no column now masks; the original never did. + Review migrated mask rules with an empty Column. + +A mask on a whole catalog does not apply to the catalog's `information_schema`. The engine's **Policies** tab lists it +as, for example, `wh.!information_schema.*.*` (`!` marks an excluded value). A mask on a whole table lists as +`wh.sales.orders.*`. A hand-written masking policy that names no column is left as it is and still masks nothing; +migrating it is refused until its missing levels are set (see [Migrate policies](../data-governance/engines-and-policy/migrate.md)). + ## Rolling back There is no supported in-place rollback. Restore the database from the backup taken in step 2, then start the previous