diff --git a/platform/control-plane/query-engines/engine-detail.md b/platform/control-plane/query-engines/engine-detail.md index f85613d..1a20de5 100644 --- a/platform/control-plane/query-engines/engine-detail.md +++ b/platform/control-plane/query-engines/engine-detail.md @@ -12,7 +12,7 @@ Select an engine in **Control Plane → Engines** to open its page. The tabs dep The header shows the engine name, its namespace (`egret-engines`), the image, the endpoint, the authentication (*JWT auth*) and the Ranger plugin version, with the type and status badges. The endpoint is the `host:port` a workspace connection to the engine dials, for example `docs-trino-trino.egret-engines.svc:8443` (port 8443 is the HTTPS listener of an engine with JWT delegation). **Open in Engine & policy** opens the engine in Governance, **Restart** restarts the engine, and the **More engine actions** menu (**⋯**) holds **Delete engine**. The tabs are **Overview**, **Catalogs**, **Workspaces**, **Settings** and **Monitor**. -A provisioned StarRocks engine that a policy profile names shows, above the tabs, the warning that official StarRocks images serve the Iceberg metadata tables without asking Ranger and apply no mask or row filter to prepared statements (since platform #822; see [External engine](#overview-1), KI-340 and KI-345). It is not shown while the profile list loads, or to a viewer who cannot read it. It is not shown for a provisioned engine on Egret's patched image `ghcr.io/egret-data/egret-starrocks-fe` (the default for new engines since platform #826), which fixes both holes; an external engine always shows it, because Egret does not know its image. +A provisioned StarRocks engine that a policy profile names shows, above the tabs, the warning that unless the engine runs `egret-starrocks-fe` `4.1.7-egret.3` or later its StarRocks lets users past the profile: official images serve the Iceberg metadata tables without asking Ranger and apply no mask or row filter to prepared statements, and official and earlier patched images apply none through the HTTP SQL API or Arrow Flight SQL, let anyone read object storage through `FILES()` and let a masked or filtered user store a table's rows in the clear in a materialized view (since platform #822, widened in #826 and #827; see [External engine](#overview-1), KI-340, KI-345 to KI-348). It is not shown while the profile list loads, or to a viewer who cannot read it. It is not shown for a provisioned engine on Egret's patched image `ghcr.io/egret-data/egret-starrocks-fe` `4.1.7-egret.3` or later (the default for new engines since platform #827), which fixes all of them; it is shown for an engine still on `4.1.7-egret.1`; an external engine always shows it, because Egret does not know its image. ### Overview @@ -119,7 +119,7 @@ The header shows the engine name, its type, an **External** badge and **Open in | **Ranger plugin version** | The version you declared. | | **Policy profile** | The bound profile, or *None bound. Bind one on the Settings tab.* | -For a StarRocks engine with a bound profile the tab also shows a warning (since platform #822; widened in platform #826): official StarRocks images serve the metadata tables of Iceberg tables (`$files`, `$partitions`, `$snapshots` and the others) without asking Ranger, so anyone who can log in to the engine can read row counts, partition values, data file paths and each column's lowest and highest values for every table of an attached Iceberg catalog, whatever the profile's grants, masks, row filters and classifications say, and apply no mask or row filter to prepared statements such as Data API endpoints with parameters (KI-340, KI-345). Egret's patched image `ghcr.io/egret-data/egret-starrocks-fe` fixes both; run the engine on it. Give StarRocks logins only to people who may see these values, and serve masked or filtered data through Trino. A Trino engine shows no such warning. +For a StarRocks engine with a bound profile the tab also shows a warning (since platform #822; widened in platform #826 and #827): official StarRocks images serve the metadata tables of Iceberg tables (`$files`, `$partitions`, `$snapshots` and the others) without asking Ranger, so anyone who can log in to the engine can read row counts, partition values, data file paths and each column's lowest and highest values for every table of an attached Iceberg catalog, whatever the profile's grants, masks, row filters and classifications say, and apply no mask or row filter to prepared statements such as Data API endpoints with parameters; official images and Egret's earlier patched images also apply none through the HTTP SQL API or Arrow Flight SQL, let anyone read object storage through `FILES()` and let a masked or filtered user store a table's rows in the clear in a materialized view (KI-340, KI-345 to KI-348). Egret's patched image `ghcr.io/egret-data/egret-starrocks-fe` `4.1.7-egret.3` or later fixes all of them; run the engine on it. Give StarRocks logins only to people who may see these values, and serve masked or filtered data through Trino. A Trino engine shows no such warning. ![The Overview tab of the external StarRocks engine docs-starrocks with the StarRocks security warning](/img/platform/admin/cp-engine-ext-sr-overview.png) diff --git a/platform/control-plane/query-engines/register-external.md b/platform/control-plane/query-engines/register-external.md index 23c05dd..1d1556d 100644 --- a/platform/control-plane/query-engines/register-external.md +++ b/platform/control-plane/query-engines/register-external.md @@ -44,7 +44,7 @@ Egret proves migrations and explains access only for Trino with plugin 2.5.0 or *Bind a Trino profile now, or leave it and bind one later from the engine's Settings tab. A profile can belong to one engine.* The list shows **No profile yet**, then each Trino profile, marking those already taken (for example *docs-trino — bound to docs-trino*). A Trino profile with no tag service is listed but cannot be chosen, marked *… — no tag service: give it one on its Advanced tab*, because egret would refuse to bind it. A profile already bound to another engine cannot be chosen again. If binding would make two mask or row-filter rules decide one column differently, egret refuses and changes nothing, and the form says *Nothing was changed. Resolve one of the two, then try again.* -For a StarRocks engine, picking a profile shows a warning under the picker (since platform #822): official StarRocks images serve the metadata tables of Iceberg tables without asking Ranger, so anyone who can log in to the engine can read row counts, partition values, data file paths and column bounds of every table in its Iceberg catalogs, whatever the profile says, and apply no mask or row filter to prepared statements (KI-340, KI-345). Run the external engine on Egret's patched image `ghcr.io/egret-data/egret-starrocks-fe`, which fixes both; the warning stays, because Egret cannot see which image an external engine runs. +For a StarRocks engine, picking a profile shows a warning under the picker (since platform #822): official StarRocks images serve the metadata tables of Iceberg tables without asking Ranger, so anyone who can log in to the engine can read row counts, partition values, data file paths and column bounds of every table in its Iceberg catalogs, whatever the profile says, and apply no mask or row filter to prepared statements; official images and Egret's earlier patched images also apply none through the HTTP SQL API or Arrow Flight SQL, let anyone read object storage through `FILES()` and let a masked or filtered user store a table's rows in the clear in a materialized view (KI-340, KI-345 to KI-348; widened in platform #827). Run the external engine on Egret's patched image `ghcr.io/egret-data/egret-starrocks-fe` `4.1.7-egret.3` or later, which fixes all of them; the warning stays, because Egret cannot see which image an external engine runs. ![Register external engine with type StarRocks and a profile picked: the StarRocks security warning under Policy profile](/img/platform/admin/cp-register-sr-warning.png) diff --git a/platform/data-governance/data/masks-and-filters.md b/platform/data-governance/data/masks-and-filters.md index 5013784..542188f 100644 --- a/platform/data-governance/data/masks-and-filters.md +++ b/platform/data-governance/data/masks-and-filters.md @@ -142,7 +142,7 @@ The local platform admin `admin@egret.local` got the same masked values: platfor - A mask on a column whose type the mask cannot apply to can return an error from the engine rather than a masked value. - A rule that selects by path is copied to the new name when its table or column is renamed, and the original stays on the old name; a rule that selects by tag follows because classifications do. A rule that selects by a pattern the new name escapes refuses the rename. See [Renames and drops](./grants.md#renames-and-drops). - A row filter or custom mask expression that names a renamed column is not rewritten (KI-321). -- On official StarRocks images, masks and row filters do not reach a table's Iceberg metadata tables (`orders$files`, `orders$partitions`, …), which StarRocks serves to any login without asking Ranger, so column bounds, partition values and counts are readable regardless of rules (KI-340); and they do not apply to prepared statements, so a Data API endpoint with parameters returns rows and values in the clear (KI-345). Egret's patched image `ghcr.io/egret-data/egret-starrocks-fe`, the default for new engines, fixes both; on an official image, do not give StarRocks logins to people who must not see the values. +- On official StarRocks images, masks and row filters do not reach a table's Iceberg metadata tables (`orders$files`, `orders$partitions`, …), which StarRocks serves to any login without asking Ranger, so column bounds, partition values and counts are readable regardless of rules (KI-340); they do not apply to prepared statements, so a Data API endpoint with parameters returns rows and values in the clear (KI-345); they do not apply to the HTTP SQL API or Arrow Flight SQL (KI-346), nor to `FILES()`, which reads the data files directly (KI-347); and a materialized view stores its rows past its creator's masks and row filters (KI-348). Egret's patched image `ghcr.io/egret-data/egret-starrocks-fe` `4.1.7-egret.3`, the default for new engines, fixes all of them; on an official image, do not give StarRocks logins to people who must not see the values. - On Trino, a table's metadata tables (`orders$files`, `orders$partitions`, …) follow its masks, row filters and denies (since platform #821): a person a mask, row filter or deny on the table restricts gets *Access Denied* on every one of them, and a person it exempts or does not restrict reads them as before. A tag rule or clearance does the same for each table that carries the classification. Profiles that egret's permission model does not compile get no such deny (KI-341). - On StarRocks, `SELECT *` on an Iceberg table shows the hidden `_file` and `_pos` columns to a user with a row filter (KI-337). List the columns instead. diff --git a/platform/data-governance/engines-and-policy/index.md b/platform/data-governance/engines-and-policy/index.md index f5a5b46..4b7d93e 100644 --- a/platform/data-governance/engines-and-policy/index.md +++ b/platform/data-governance/engines-and-policy/index.md @@ -60,7 +60,7 @@ A Trino profile with no tag service cannot be bound to an engine: egret refuses Select an engine to open its page. The header shows the engine name, its **Type** badge, **Provisioned** or **External**, the bound **Profile**, the **Ranger plugin** version, and **Open in Control plane**. For an external engine a note reads *egret compiles this engine's policies but cannot see whether the external engine downloads them.* -For a StarRocks engine bound to a profile, a warning under the header says that official StarRocks images serve the metadata tables of Iceberg tables (`$files`, `$partitions`, `$snapshots` and the others) without asking Ranger, so anyone who can log in to the engine can read row counts, partition values, data file paths and each column's lowest and highest values for every table of an attached Iceberg catalog, whatever the profile's grants, masks, row filters and classifications say, and apply no mask or row filter to prepared statements (KI-340, KI-345). It advises running the engine on Egret's patched image `ghcr.io/egret-data/egret-starrocks-fe`, or giving StarRocks logins only to people who may see these values and serving masked or filtered data through Trino (since platform #822). With **engine:read**, it is not shown for a provisioned engine on Egret's patched image `ghcr.io/egret-data/egret-starrocks-fe` (the default for new engines since platform #826), which fixes both holes; an external engine always shows it, because Egret does not know its image. +For a StarRocks engine bound to a profile, a warning under the header says that official StarRocks images serve the metadata tables of Iceberg tables (`$files`, `$partitions`, `$snapshots` and the others) without asking Ranger, so anyone who can log in to the engine can read row counts, partition values, data file paths and each column's lowest and highest values for every table of an attached Iceberg catalog, whatever the profile's grants, masks, row filters and classifications say, and apply no mask or row filter to prepared statements; official images and Egret's earlier patched images also apply none through the HTTP SQL API or Arrow Flight SQL, let anyone read object storage through `FILES()` and let a masked or filtered user store a table's rows in the clear in a materialized view (KI-340, KI-345 to KI-348). It advises running the engine on Egret's patched image `ghcr.io/egret-data/egret-starrocks-fe` `4.1.7-egret.3` or later, or giving StarRocks logins only to people who may see these values and serving masked or filtered data through Trino (since platform #822, widened in #827). With **engine:read**, it is not shown for a provisioned engine on Egret's patched image `ghcr.io/egret-data/egret-starrocks-fe` `4.1.7-egret.3` or later (the default for new engines since platform #827), which fixes all of them; it is shown for an engine still on `4.1.7-egret.1`; an external engine always shows it, because Egret does not know its image. ![The Governance page of the external StarRocks engine docs-starrocks: the warning that official StarRocks images serve Iceberg metadata tables without asking Ranger and apply no mask or row filter to prepared statements, above the tabs](/img/platform/admin/gov-engine-starrocks-warning.png) diff --git a/platform/deploy-operate/engines-and-ranger.md b/platform/deploy-operate/engines-and-ranger.md index 484eddd..7f95c74 100644 --- a/platform/deploy-operate/engines-and-ranger.md +++ b/platform/deploy-operate/engines-and-ranger.md @@ -28,7 +28,8 @@ jar out of each. | Trino | `trinodb/trino:483` | 2.8.0 | | StarRocks | `starrocks/fe-ubuntu:4.1.4` | 2.8.0 | | StarRocks | `starrocks/fe-ubuntu:4.1.6` | 2.9.0 | -| StarRocks | `ghcr.io/egret-data/egret-starrocks-fe:4.1.7-egret.1` (Egret's patched 4.1.7) | 2.9.0 | +| StarRocks | `ghcr.io/egret-data/egret-starrocks-fe:4.1.7-egret.1` (Egret's patched 4.1.7, first fixes; superseded) | 2.9.0 | +| StarRocks | `ghcr.io/egret-data/egret-starrocks-fe:4.1.7-egret.3` (Egret's patched 4.1.7) | 2.9.0 | Plugin 2.5.0 cannot present the engine credential the policy service requires (see [Authentication of the plugin](#authentication-of-the-plugin)), so Egret refuses to create an engine on `trinodb/trino:470` or update one onto it (400), and refuses to register an external engine that declares a plugin older than 2.8.0. An existing Trino 470 engine must move to Trino 483. @@ -45,21 +46,30 @@ get `ERROR 5204`. Grant the view, or the base columns for an `INVOKER` view, to ## Egret's patched StarRocks image -The official StarRocks images have two holes that no policy Egret compiles can close (KI-340, KI-345): +The official StarRocks images have holes that no policy Egret compiles can close: - they serve the metadata tables of Iceberg tables (`orders$files`, `$partitions`, `$snapshots`, …) to any login without - asking Ranger, which exposes row counts, partition values, data file paths and each column's lowest and highest value; + asking Ranger, which exposes row counts, partition values, data file paths and each column's lowest and highest value + (KI-340); - they apply no mask and no row filter to prepared statements (`PREPARE`/`EXECUTE`, or the MySQL binary protocol), and Egret runs Data API endpoints with parameters on StarRocks that way, so such an endpoint returns every row and column - in the clear. + in the clear (KI-345); +- they apply no mask and no row filter to the front end's HTTP SQL API (port 8030, which the StarRocks Ingress exposes) + or to Arrow Flight SQL (KI-346); +- `FILES()` reads and writes object storage with no privilege at all (KI-347); +- a materialized view stores its base tables' rows past its creator's masks and row filters (KI-348); +- `SELECT *` under a mask or row filter returns the hidden `_file` and `_pos` columns (KI-337), and `SHOW CREATE CATALOG` + shows every catalog's properties to anyone (KI-349). `ghcr.io/egret-data/egret-starrocks-fe` is the official `starrocks/fe-ubuntu` image of the same release with two front -end jars rebuilt with the fixes; it ships the official Ranger plugin. The source, the patches and the security test every +end jars rebuilt with the fixes; it ships the official Ranger plugin. `4.1.7-egret.3` fixes all of the above; +`4.1.7-egret.1` fixed only the first two. On it, `FILES()` needs the OPERATE privilege on SYSTEM, and creating a +materialized view over a table where a mask or row filter applies to the creator is refused. The source, the patches and the security test every image passes before it is published are at [github.com/egret-data/egret-starrocks](https://github.com/egret-data/egret-starrocks). New provisioned engines use it. To move an existing engine onto it, update the engine's `feImage` to -`ghcr.io/egret-data/egret-starrocks-fe:4.1.7-egret.1@sha256:da948dad19e98a5a43b0073ec48f87c9e7429246ce4581a000c0955c742af250` +`ghcr.io/egret-data/egret-starrocks-fe:4.1.7-egret.3@sha256:5c74c820f9b741bc8054247f47f3d54dd9e526710b6fc2b5a7873cae6bb2d2b0` and its `cnImage` to `starrocks/cn-ubuntu:4.1.7@sha256:39f7320127c9924abb1f1ce0ba0583ab11989cbe5bd1bb6274ada6f89ba5740d` -(the pods restart; from 4.1.4 or 4.1.6 this is an upgrade). Run an external StarRocks engine on the same image. +(the pods restart; from 4.1.4 or 4.1.6 this is an upgrade; an engine on `4.1.7-egret.1` moves the same way). Run an external StarRocks engine on the same image. ## Connect an engine diff --git a/platform/deploy-operate/local.md b/platform/deploy-operate/local.md index ab510b4..73f6587 100644 --- a/platform/deploy-operate/local.md +++ b/platform/deploy-operate/local.md @@ -32,7 +32,7 @@ named `egret-dev`. | `valkey` | `valkey/valkey:8` | 6399 (to 6379 in the container) | The shared limiter and response cache for the Data API (`data_api.shared_store_url`). The dev server does not need it; `just check` does, for the store tests. No volume: its contents can be rebuilt. | | `rustfs` | `rustfs/rustfs:1.0.1` | 9000 (S3 API), 9001 (console) | S3-compatible object storage. Access key `egret`, secret `egretsecret`. It replaced MinIO, whose public images were withdrawn. It must support STS `AssumeRole`, because the embedded catalog vends credentials through it. | | `trino` | `trinodb/trino:483` (pinned by digest) | 8090 (HTTP), 8453 (HTTPS) | A development Trino with JWT authentication against Keycloak and a file-based group access control. JWT is only enforced over TLS, so the HTTPS listener uses a self-signed keystore (`deploy/trino/etc/gen-certs.sh` makes it). It ships no catalogs: mount property files into `deploy/trino/catalog`. | -| `starrocks` | `ghcr.io/egret-data/egret-starrocks-allin1:4.1.7-egret.1` (Egret's patched StarRocks, pinned by digest) | 9030 (MySQL protocol), 8030 (FE HTTP) | A single-container StarRocks. The Ranger plugin configuration (`ranger-starrocks-security.xml`, `ranger-starrocks-audit.xml`) is mounted into the front end. | +| `starrocks` | `ghcr.io/egret-data/egret-starrocks-allin1:4.1.7-egret.3` (Egret's patched StarRocks, pinned by digest) | 9030 (MySQL protocol), 8030 (FE HTTP) | A single-container StarRocks. The Ranger plugin configuration (`ranger-starrocks-security.xml`, `ranger-starrocks-audit.xml`) is mounted into the front end. | | `keycloak` | `quay.io/keycloak/keycloak:26.0` | 8081 | The development identity provider, started with `start-dev --import-realm`. The `egret` realm is imported from `deploy/keycloak/egret-realm.json`. The Keycloak administrator is `admin` / `admin`. | | `frontend` | built from `frontend/` (nginx) | 5173 | The production build of the web app, for static smoke tests only. The backend is not containerized in this file, so `/api` does not resolve from it. Use `npm run dev` in `frontend/` for development. | diff --git a/platform/deploy-operate/upgrade.md b/platform/deploy-operate/upgrade.md index 884053f..1108534 100644 --- a/platform/deploy-operate/upgrade.md +++ b/platform/deploy-operate/upgrade.md @@ -422,6 +422,26 @@ From platform #823 there is no migration and no API shape change. grant principal or rule exemption already uses is left without one, and the server logs a warning; allocate theirs by hand from a grant dialog, which answers 409 naming those rows until they are resolved. +### StarRocks engines default to `egret-starrocks-fe` `4.1.7-egret.3` (platform #827) + +From platform #827 there is no migration and no API change. + +- **Behaviour change:** a new provisioned StarRocks engine runs `ghcr.io/egret-data/egret-starrocks-fe:4.1.7-egret.3` + (pinned by digest) instead of `4.1.7-egret.1`; the compute nodes stay on the official `starrocks/cn-ubuntu:4.1.7`. It + also fixes KI-337 (`_file` and `_pos` in `SELECT *` under a row filter), KI-346 (no masks or row filters through the + HTTP SQL API and Arrow Flight SQL), KI-347 (`FILES()` with no privilege), KI-348 (materialized views stored past their + creator's masks and row filters) and KI-349 (`SHOW CREATE CATALOG` with no privilege). The local development stack's + `starrocks` service runs `egret-starrocks-allin1:4.1.7-egret.3`. +- **New refusals on the patched image:** `FILES()` needs the OPERATE privilege on SYSTEM (error 5211), and creating a + materialized view over a table where a mask or row filter applies to the creator is refused (error 5212), whichever + columns the view reads; an existing materialized view of such a creator stops refreshing. `SHOW CREATE CATALOG` needs + a privilege on the catalog. +- **Existing engines keep their image**, including those on `4.1.7-egret.1`. Move each one with + `PUT /api/control-plane/engines/{id}` as described under *Egret's patched StarRocks image* on **Engines and Ranger**; + its pods restart. +- **The warning** now names every hole, and is shown for every StarRocks engine bound to a profile unless it runs + `egret-starrocks-fe` `4.1.7-egret.3` or later, so it comes back for an engine on `4.1.7-egret.1`. + ## Rolling back There is no supported in-place rollback. Restore the database from the backup taken in step 2, then start the previous diff --git a/platform/known-issues.md b/platform/known-issues.md index 72301ea..a0577a0 100644 --- a/platform/known-issues.md +++ b/platform/known-issues.md @@ -354,7 +354,7 @@ Collected while writing the detailed pages for each area (October 2026), checked | KI-301 | Alerts / print page | Consistency | S3 | The print page that PNG and PDF reports are made from draws each chart with the chart's own settings only, so the dataset's **Display names** and column formats do not apply: a report can say `revenue` where the dashboard and the chart page say **Revenue** (since platform #751, KI-290 and KI-251). A **Column title** set in the chart itself does apply. *Steps:* a PNG report of a dashboard whose chart relies on a dataset display name. (Found by platform #751, which left the print page unchanged.) *Status:* fixed in platform #754 (October 2026): the render payload carries each card's dataset display names, formats and semantic types, and the print page layers them under the chart's own settings with the same function the dashboard uses, so a report says **Revenue** where the dashboard does. A **Column title** set in the chart still wins. | | KI-302 | Naming | Consistency | S3 | The alert editor's SQL condition, in a workspace with no connection, warns *This workspace has no database connection yet. Add one under Workspace settings → Engines before creating an alert.*, but the menu entry is **Connections** since platform #748 (KI-007) and the rest of the UI says *connection* since platform #751 (KI-298). *Steps:* a workspace with no connection → **+ Alert** → **Condition**. (Found while writing the docs for platform #751; read from code.) *Status:* fixed in platform #754 (October 2026): it reads *This workspace has no connection yet. Add one under Workspace settings → Connections before creating an alert.* The Data API **Preview** note and the governed-connection refusal (*This connection is governed: …*) say *connection* too. | | KI-303 | Governance / Classifications | Bug | S2 | Renaming or dropping a table leaves its classifications, on the table and on its columns, on the old name, on Trino and StarRocks alike. After a rename the table carries no classification under its new name, so a clearance or a rule by classification no longer applies to it, and on StarRocks the old name's whole-table select deny stays compiled for a table that no longer exists. A table created later under the old name gets the old classifications. *Workaround:* attach the classifications to the new name and remove them from the old one. (Found while building platform #755; not fixed there.) *Status:* fixed in platform #757 (October 2026): classifications follow a table rename and a column rename made through egret's Iceberg catalog (a column by its Iceberg field id). egret holds the rename until every bound Trino plugin serves the new tags and every StarRocks plugin the new policies, then lets the catalog commit it, so the renamed table is protected from its first read under the new name; a rename egret cannot hold safely is refused with a 409 ending *Nothing was renamed.* (KI-310). A drop keeps the classifications on the name, so an undrop restores them; a recursive namespace drop removes them. Classifications a rename left on an old name before #757 are reported in the Governance overview, and so is the swap pattern (a new table at a renamed table's old name). Path-selector grants and rules, hand-written Ranger policies and catalog grants still stay on the old name (KI-308, KI-309). See [Renames and drops](./data-governance/classifications/tagging#renames-and-drops). | -| KI-304 | Governance / Classifications | Security | S3 | On StarRocks, a reader who gets a real mask (not *Unmasked*) on any column of an Iceberg table, such as a rule's **Hash** or a NULL mask, sees the table's hidden `_file` column (the full object-store path of the data file) and `_pos` in `SELECT *`. An uncleared reader of a whole-table classification is refused instead. Older than platform #755: column masks cause it. *Steps:* a mask rule on one column of an Iceberg table; `SELECT *` on StarRocks as a non-exempt reader. (Measured in the live check of platform #755.) *Status:* fixed in platform #766 (October 2026): every StarRocks table with a mask gets one deny on `_file` and `_pos` for each reader who does not see the real value in every masked column. That reader's `SELECT *` on the table is refused with error 5204 on `_file`; named columns and views work as before. A mask on a table selected by a wildcard or an excluded name gets no deny, so the leak stays there. *Status:* fixed in platform #798 (October 2026): a mask whose table selector is a wildcard or an exclusion now gets a hidden-column deny for each table it matches in the registered catalogs, with the same exemption rule as an exact table. A wildcard that matches no registered catalog refuses the write. Not covered: a catalog that egret does not register (for example `default_catalog`), and a table created after the last compile, until the next compile. *Note (November 2026):* the deny closes only `SELECT *`. The `_file` path itself is readable by any StarRocks login from the table's `$files` metadata table, which StarRocks serves without asking Ranger (KI-340), and with `SELECT _file` by any reader of the table. A row filter shows the two columns in `SELECT *` too (KI-337). | +| KI-304 | Governance / Classifications | Security | S3 | On StarRocks, a reader who gets a real mask (not *Unmasked*) on any column of an Iceberg table, such as a rule's **Hash** or a NULL mask, sees the table's hidden `_file` column (the full object-store path of the data file) and `_pos` in `SELECT *`. An uncleared reader of a whole-table classification is refused instead. Older than platform #755: column masks cause it. *Steps:* a mask rule on one column of an Iceberg table; `SELECT *` on StarRocks as a non-exempt reader. (Measured in the live check of platform #755.) *Status:* fixed in platform #766 (October 2026): every StarRocks table with a mask gets one deny on `_file` and `_pos` for each reader who does not see the real value in every masked column. That reader's `SELECT *` on the table is refused with error 5204 on `_file`; named columns and views work as before. A mask on a table selected by a wildcard or an excluded name gets no deny, so the leak stays there. *Status:* fixed in platform #798 (October 2026): a mask whose table selector is a wildcard or an exclusion now gets a hidden-column deny for each table it matches in the registered catalogs, with the same exemption rule as an exact table. A wildcard that matches no registered catalog refuses the write. Not covered: a catalog that egret does not register (for example `default_catalog`), and a table created after the last compile, until the next compile. *Note (November 2026):* the deny closes only `SELECT *`. The `_file` path itself is readable by any StarRocks login from the table's `$files` metadata table, which StarRocks serves without asking Ranger (KI-340), and with `SELECT _file` by any reader of the table. A row filter shows the two columns in `SELECT *` too (KI-337). *Status (November 2026):* on Egret's patched StarRocks image `4.1.7-egret.3` (platform #827) `SELECT *` under a mask no longer reaches `_file` and `_pos`, so the masked reader's `SELECT *` works; the hidden-column deny still keeps `SELECT _file` from that reader. | | KI-305 | Governance / Classifications | Limitation | S3 | On StarRocks, a view over a table with a whole-table clearance can show an uncleared reader real values for a short time after the table changes: egret writes the NULL masks before an undrop or a column add returns, but the plugin applies them only at its next poll (30 seconds by default). A direct read is refused throughout. If the recompile that follows a table change fails (another permission write held the profile longer than 10 seconds), a view that reads a new column shows its real values until the profile's next recompile or the next start of egret; the Governance overview shows **Profile compile failed** meanwhile. (Measured in the live check of platform #755; the failed recompile is from its design.) The live check of platform #757 measured the undrop window at 23 seconds. #757 holds a classified rename until the plugins serve the new policies, but not an undrop or a column add. During such a hold, an unrelated permission write that recompiles the StarRocks profile drops the renamed table's NULL masks from the compile until the hold finishes, so a view over the new name can show real values for that time; the direct read stays refused. *Status:* kept as a limitation (October 2026). | | KI-306 | Control plane / Catalogs | Limitation | S3 | A table register that egret refuses because the recompile would cross a clearance ceiling (409 *egret refused the table change: …; the table was not registered*) is dropped again, but softly: the table then sits in the catalog's **Recycle bin** for the warehouse's soft-delete retention (7 days by default), and undropping it is refused with *egret refused the undrop: …*. Nothing leaks, but the entry looks recoverable. *Steps:* register a table whose metadata has more columns than the ceiling allows, under a name that carries a whole-table clearance; open the Recycle bin. (Found in the live check of platform #755.) *Status:* kept as a limitation (October 2026). | | KI-307 | Governance / Classifications | UX | S3 | Attaching a classification with a clearance to a table that is not in the catalog yet (unknown, a view, or not created) compiles the whole-table select deny but no NULL masks on StarRocks, so a view over it is not protected until the table exists. The attach itself answers only success: the skip is reported by **Who can read?** on the table's columns (*No NULL mask for clearance … is served here: egret found no table … in the catalog, so a view over it shows the values*) and by a profile recompile, not to the steward who attached it. The masks are written when the table is created. (Found in the live check of platform #755.) *Status:* fixed in platform #818 (November 2026): the attach answers `notices` with `clearance_masks_pending_table` when the attachment is to a whole table, the classification has a clearance with roles, a StarRocks profile serves the warehouse and the catalog has no active table of that name. Governance **Data** → **Add classification** and the catalog **Browse** tag control show it as a warning toast in place of the success toast. The attach still succeeds and the masks are still written once the table exists. A clearance added later to a classification already on such a table gives no notice. *Status:* refined by platform #820 (November 2026): when the name is a view in the catalog, the attach answers the notice `clearance_on_view` instead, which says that StarRocks gets only a select deny on the view's name and never column masks, and to classify the table the view reads; a name the catalog does not have keeps `clearance_masks_pending_table`. | @@ -387,15 +387,19 @@ Collected while writing the detailed pages for each area (October 2026), checked | KI-334 | Governance / Data | Bug | S3 | The engines in the Governance **Data** page's object header can show a running engine as `provisioning`: in the live check of platform #818 the Trino engine `p7jwt` showed `provisioning` there, while the engine API said it was running. The header shows the `status` the governance browse API answers for each engine attached to the catalog, read from the stored engine row; why that row still said `provisioning` is not confirmed (seen with the Docker provisioner). Nothing else depends on the label. (Found in the live check of platform #818.) | | KI-335 | Engines / StarRocks | Bug | S2 | On Kubernetes, a provisioned StarRocks engine can come up with its compute node (CN) in a crash loop. The CN spec carries no root password, while the `{name}-initpwd` Job sets the FE's root password first: when the CN starts after that, for example on a node that does not have the CN image cached yet and has to pull it, it cannot log in to the FE (`ERROR 1045`), queries fail with `ERROR 5904` *Warehouse default_warehouse is not available*, the engine stays `provisioning`, and the catalog attachment still says *applied*. *Steps:* provision a StarRocks engine on a fresh node without `starrocks/cn-ubuntu:4.1.6` cached. *Workaround:* give the CN the root password from the `{name}-initpwd-secret` Secret as `MYSQL_PWD` (`cnEnvVars` of the StarRocksCluster resource, `valueFrom.secretKeyRef`, key `password`), or pull the CN image onto the node before provisioning. (Found in the Wave 4 re-check on kind, StarRocks 4.1.6.) | | KI-336 | Engines / StarRocks | Gap | S2 | A StarRocks engine provisioned without a policy profile (no `authzServiceId`) stays without one, even when Ranger is on: egret creates a profile automatically only for a Trino engine, and **Add engine** has no profile picker for StarRocks. Such an engine is not governed by egret's policies, and in the re-check the admin saw no egret catalog on it and nothing said why. *Workaround:* create a StarRocks profile (**Engine & policy → New profile**, type **StarRocks**), then pass its id as `authzServiceId` when you create the engine through `POST /api/control-plane/engines`, or bind it afterwards with `PUT /api/control-plane/engines/{id}`. (Found in the Wave 4 re-check on kind.) | -| KI-337 | Governance / Masks and row filters | Bug | S2 | On StarRocks (4.1.6 and later, an upstream StarRocks bug), when a row filter applies to a user, `SELECT *` on an Iceberg table returns two extra columns: `_file` (the object-store path of the data file) and `_pos` (the row's position). Users without a filter get the table's own columns; Trino is not affected. So the result has a different shape per user: `INSERT … SELECT *`, and datasets or charts built on `SELECT *`, can break for a filtered user, and an Iceberg view whose stored SQL is a literal `SELECT * FROM t` fails for that user with *Index N out of bounds*. With a mask on the table the query is refused instead (KI-304). The path is not a new disclosure: any reader can already get it with `SELECT _file` or from the table's `$files` metadata table (KI-340). *Workaround:* list the columns (`SELECT id, region, email …`); a view created from StarRocks stores the expanded column list and is not affected. Severity S2: wrong result shape with a workaround, no value the user could not read otherwise. (Found in the Wave 4 re-check; the cause is StarRocks' row-policy rewrite, which turns the hidden columns into visible ones.) | +| KI-337 | Governance / Masks and row filters | Bug | S2 | **Fixed in Egret's patched StarRocks image `4.1.7-egret.3` (platform #827); open upstream.** On StarRocks (4.1.6 and later, an upstream StarRocks bug), when a row filter applies to a user, `SELECT *` on an Iceberg table returns two extra columns: `_file` (the object-store path of the data file) and `_pos` (the row's position). Users without a filter get the table's own columns; Trino is not affected. So the result has a different shape per user: `INSERT … SELECT *`, and datasets or charts built on `SELECT *`, can break for a filtered user, and an Iceberg view whose stored SQL is a literal `SELECT * FROM t` fails for that user with *Index N out of bounds*. With a mask on the table the query is refused instead (KI-304). The path is not a new disclosure: any reader can already get it with `SELECT _file` or from the table's `$files` metadata table (KI-340). *Workaround:* list the columns (`SELECT id, region, email …`); a view created from StarRocks stores the expanded column list and is not affected. Severity S2: wrong result shape with a workaround, no value the user could not read otherwise. (Found in the Wave 4 re-check; the cause is StarRocks' row-policy rewrite, which turns the hidden columns into visible ones.) | | KI-338 | SQL Lab / StarRocks | Bug | S2 | In SQL Lab, the schema explorer of a StarRocks connection lists StarRocks' default catalog (`_statistics_`, `information_schema`, `sys`), not the egret catalog the engine serves, so the tables you can query are not shown there. Queries work. *Workaround:* write fully qualified names (`..`). Severity S2: the explorer shows the wrong catalog, and a workaround exists. (Found in the Wave 4 re-check on kind.) | | KI-339 | Workspaces | UX | S3 | A member whose only grant in a workspace is a deny gets the 403 page there (platform #811), and on that page the workspace switcher is empty, so it offers no way to another workspace. *Workaround:* open another workspace from its address or from the home page. (Found in the Wave 4 re-check.) | -| KI-340 | Engines / StarRocks | Security | S1 | **Fixed in Egret's patched StarRocks image (platform #826); open upstream.** Official StarRocks images serve the Iceberg metadata tables of a table (`orders$files`, `$partitions`, `$manifests`, `$snapshots`, `$history`, `$refs`, `$metadata_log_entries`, …) to any login without asking Ranger. So a StarRocks user reads, regardless of grants, masks and row filters: per-column minimum and maximum values (for example the first and last e-mail address of a masked column), partition values, row counts and data file paths, even when the same user is refused `SELECT` on the table. No egret policy can stop it: StarRocks 4.1.x never checks a metadata table against Ranger and has no setting that turns them off. egret does not filter SQL text, since users connect to StarRocks directly. *Workaround:* do not give StarRocks logins to people who must not see the values; serve masked or filtered data to them through Trino. Since platform #822 egret says so with a warning wherever a StarRocks engine is bound to a policy profile: the Governance engine page, the Control plane engine page (provisioned, and an external engine's **Overview** and **Settings**) and **Register external engine** once a profile is picked. (Measured in the Wave 4 re-check and in the work for platform #821, StarRocks 4.1.6.) *Status:* from platform #826 new provisioned StarRocks engines run Egret's patched front end image `ghcr.io/egret-data/egret-starrocks-fe:4.1.7-egret.1` (source and patches at github.com/egret-data/egret-starrocks), on which reading a metadata table needs `SELECT` on its table and is refused to a person a mask or row filter on that table restricts. Existing engines keep their image until moved (see *Egret's patched StarRocks image* on **Engines and Ranger**); an external engine is fixed only when its operator runs the same image. The warning is no longer shown for a provisioned engine on the patched image. Reported to the StarRocks security team on 2026-11-02. | +| KI-340 | Engines / StarRocks | Security | S1 | **Fixed in Egret's patched StarRocks image (platform #826; `4.1.7-egret.3` from platform #827 also closes KI-346 to KI-349); open upstream.** Official StarRocks images serve the Iceberg metadata tables of a table (`orders$files`, `$partitions`, `$manifests`, `$snapshots`, `$history`, `$refs`, `$metadata_log_entries`, …) to any login without asking Ranger. So a StarRocks user reads, regardless of grants, masks and row filters: per-column minimum and maximum values (for example the first and last e-mail address of a masked column), partition values, row counts and data file paths, even when the same user is refused `SELECT` on the table. No egret policy can stop it: StarRocks 4.1.x never checks a metadata table against Ranger and has no setting that turns them off. egret does not filter SQL text, since users connect to StarRocks directly. *Workaround:* do not give StarRocks logins to people who must not see the values; serve masked or filtered data to them through Trino. Since platform #822 egret says so with a warning wherever a StarRocks engine is bound to a policy profile: the Governance engine page, the Control plane engine page (provisioned, and an external engine's **Overview** and **Settings**) and **Register external engine** once a profile is picked. (Measured in the Wave 4 re-check and in the work for platform #821, StarRocks 4.1.6.) *Status:* from platform #826 new provisioned StarRocks engines run Egret's patched front end image `ghcr.io/egret-data/egret-starrocks-fe:4.1.7-egret.1` (source and patches at github.com/egret-data/egret-starrocks), on which reading a metadata table needs `SELECT` on its table and is refused to a person a mask or row filter on that table restricts. Existing engines keep their image until moved (see *Egret's patched StarRocks image* on **Engines and Ranger**); an external engine is fixed only when its operator runs the same image. The warning is no longer shown for a provisioned engine on the patched image. Reported to the StarRocks security team on 2026-11-02. | | KI-341 | Governance / Masks and row filters | Security | S2 | On Trino, the Iceberg metadata tables of a table (`orders$files`, `$entries`, `$manifests`, `$partitions`, `$history`, …) did not follow masks, row filters or denies: the Ranger plugin checks `orders$files` as a table named `orders$files`, which a grant on `table: *` matches and a mask, row filter or deny on `table: orders` does not. A masked user read the masked column's minimum and maximum from `$files`, a row-filtered user read the bounds, partition values and counts of the rows filtered away, and a user denied `orders` read its metadata tables through a broader schema grant. Unlike StarRocks (KI-340), Trino does check `SELECT` on the metadata table. *Status:* fixed in platform #821 (November 2026): for each mask, row filter, select deny or deny-all-else of a Trino profile that egret compiles (from egret's own rules and grants, and from enabled hand-written policies in such a profile), egret also compiles a deny on that table's metadata tables (`
$*`, every one of them) for the people the policy restricts, except the people it exempts; a tag mask, tag row filter or clearance gets one per table that carries the classification. A restricted user now gets *Access Denied* on every `t$…` table of that table; users with no restriction on it read them as before. Not covered: profiles that egret's permission model does not compile. (Found in the Wave 4 work on StarRocks row filters; measured on Trino 483.) | | KI-342 | Governance / Grants | UX | S2 | A local user created in **Control plane → Users** (or through `POST /api/control-plane/users`) has no engine username (`username` is `null`) until their first query, so a grant or rule exemption that names them through the API is refused with 400: by email or id, *this is a user's email, and that user has no username yet (one is minted the first time they use an engine), so no engine can name them*; by the username they will get, *no user has this username*. So an administrator cannot prepare a new person's access through the API before that person has used an engine. The Governance grant and rule dialogs list such a user apart with **Allocate username and add** (see [Grants](./data-governance/data/grants.md)). *Workaround:* in the API, allocate the username first with `POST /api/control-plane/perm/people/users/{user_id}/username`, then name the user by the returned username; or grant a group the user belongs to. Severity S2: a task is refused for a reason the API caller does not expect, and a workaround exists. (Seen while preparing the Wave 4 checks; reproduced on platform main fb168a8f.) *Status:* fixed in platform #823 (November 2026): a user gets an engine username when they are created (**New user**, `POST /api/control-plane/users`, `egret create-admin`, first SSO sign-in), returned in `username`, so a grant can name them at once. At start egret gives a username to every existing person who has none, except one whose would-be name a stored grant principal or rule exemption already uses (left without one and logged). Service accounts still get theirs at their first token. | | KI-343 | Engines / Policy profiles | Bug | S3 | When **Add engine** fails after egret created the new Trino engine's policy profile (for example the Docker provisioner refused it because a container of that name already exists), egret deletes the tag profile it created but can leave the Trino profile behind: the delete is refused because the first compile already wrote a grant's policy into it (*profile … still holds 1 policy compiled by the SP4a permission model*; the server logs *create compensation: policy profile delete failed*). The leftover profile is listed under **Profiles without an engine** and names a tag service that no longer exists. A later create of an engine with the same name was not tried. *Workaround:* recompile the leftover profile (`POST /api/control-plane/perm/compile/{service_id}` reaps the compiled policy) and delete it. (Seen on platform main fb168a8f, Docker provisioner, while preparing the docs screenshots.) | | KI-344 | Governance / Data | UX | S3 | The Data page of a name the catalog does not have (opened by its address, for example to attach a classification before the table exists) shows **Kind unknown** and, under **Columns**, the error *The columns could not be loaded. Retry, or reload the page.*, although nothing failed: there are no columns to read. **Retry** gives the same error. The page should say that the catalog has no such table. Attaching a classification there works and warns as described in [Tagging](./data-governance/classifications/tagging.md#tag-a-table-or-a-column). (Seen while capturing the attach-notice screenshots on platform main fb168a8f.) | -| KI-345 | Engines / StarRocks | Security | S1 | **Fixed in Egret's patched StarRocks image (platform #826); open upstream (StarRocks #78176, fix #78368 not merged).** Official StarRocks images apply no column mask and no row filter to a statement run through `PREPARE`/`EXECUTE` or the MySQL binary protocol (`COM_STMT_PREPARE`/`COM_STMT_EXECUTE`). Egret's StarRocks connector runs queries with parameters that way, so on an official image a **Data API endpoint with parameters** on a StarRocks database returns every row and every column in the clear to a person whose masks and row filters should apply; so does any client that prepares statements on the server (for example JDBC with `useServerPrepStmts=true`). Grants are still enforced. Measured on StarRocks 4.1.6 and 4.1.7 with `access_control = ranger`: a person with a NULL mask on `phone` and the row filter `id = 1` got `[(1, NULL)]` from a plain query and all three rows with the real phone numbers through `PREPARE`, the binary protocol, and Egret's own path (the service user switching to the person with `EXECUTE AS`, then the binary protocol). *Workaround on an official image:* do not publish Data API endpoints with parameters on StarRocks databases whose readers have masks or row filters; serve them through Trino. *Status:* fixed on `ghcr.io/egret-data/egret-starrocks-fe:4.1.7-egret.1`, the default for new provisioned engines from platform #826; existing and external engines are fixed when moved to it. | +| KI-345 | Engines / StarRocks | Security | S1 | **Fixed in Egret's patched StarRocks image (platform #826; `4.1.7-egret.3` from platform #827 also closes KI-346 to KI-349); open upstream (StarRocks #78176, fix #78368 not merged).** Official StarRocks images apply no column mask and no row filter to a statement run through `PREPARE`/`EXECUTE` or the MySQL binary protocol (`COM_STMT_PREPARE`/`COM_STMT_EXECUTE`). Egret's StarRocks connector runs queries with parameters that way, so on an official image a **Data API endpoint with parameters** on a StarRocks database returns every row and every column in the clear to a person whose masks and row filters should apply; so does any client that prepares statements on the server (for example JDBC with `useServerPrepStmts=true`). Grants are still enforced. Measured on StarRocks 4.1.6 and 4.1.7 with `access_control = ranger`: a person with a NULL mask on `phone` and the row filter `id = 1` got `[(1, NULL)]` from a plain query and all three rows with the real phone numbers through `PREPARE`, the binary protocol, and Egret's own path (the service user switching to the person with `EXECUTE AS`, then the binary protocol). *Workaround on an official image:* do not publish Data API endpoints with parameters on StarRocks databases whose readers have masks or row filters; serve them through Trino. *Status:* fixed on `ghcr.io/egret-data/egret-starrocks-fe:4.1.7-egret.1`, the default for new provisioned engines from platform #826; existing and external engines are fixed when moved to it. | +| KI-346 | Engines / StarRocks | Security | S1 | **Fixed in Egret's patched StarRocks image `4.1.7-egret.3` (platform #827); open upstream (StarRocks #78175, fix #78049 not merged).** StarRocks applies no column mask and no row filter to a query sent to the front end's HTTP SQL API (`POST /api/v1/catalogs//databases//sql`, port 8030) or to Arrow Flight SQL. A person under a mask and a row filter who can log in to the engine reads every row and column in the clear that way; the StarRocks Ingress that Egret can create exposes port 8030. Measured on `4.1.7-egret.2`: the masked, filtered reader got all four rows with clear emails over HTTP and over Flight, and two masked rows over MySQL. Arrow Flight SQL is off unless `arrow_flight_port` is set. *Workaround on other images:* keep port 8030 and the Flight port unreachable for people with a StarRocks login, or do not create the Ingress. | +| KI-347 | Engines / StarRocks | Security | S1 | **Fixed in Egret's patched StarRocks image `4.1.7-egret.3` (platform #827); open upstream (StarRocks #79980, #80233).** `FILES()` has no privilege of its own on StarRocks: anyone who can log in reads and writes object storage through it, past every grant, mask and row filter, given storage keys in the statement, or with `aws.s3.use_instance_profile` through the identity of the engine's nodes. Measured: a person with no grant read every row of a table's Parquet files. On the patched image every use of `FILES()` needs the OPERATE privilege on SYSTEM (error 5211). | +| KI-348 | Engines / StarRocks | Security | S1 | **Fixed in Egret's patched StarRocks image `4.1.7-egret.3` (platform #827).** A materialized view on StarRocks is filled by a refresh that reads its base tables without its creator's masks and row filters. A person under a mask and a row filter who may create a materialized view (`create materialized view` on a `default_catalog` database) builds one over the table and reads every row and column in the clear through it. Measured: such a person's view held all three rows with clear phone numbers; a plain view, `CREATE TABLE … AS SELECT` and `INSERT … SELECT` kept the policy. Egret's clearance compiler already denies `create materialized view` on profiles with clearances; path masks and row filters had no such deny. On the patched image creating a materialized view over a table where a mask or row filter applies to the creator is refused (error 5212, whichever columns the view reads), and a refresh stops once its user comes under one. | +| KI-349 | Engines / StarRocks | Security | S3 | **Fixed in Egret's patched StarRocks image `4.1.7-egret.3` (platform #827).** `SHOW CREATE CATALOG` needs no privilege on StarRocks: anyone who can log in reads every external catalog's properties, its endpoints and warehouse, and the first and last two characters of its keys and OAuth credential, although `SHOW CATALOGS` lists only catalogs the person has a privilege on. On the patched image it needs a privilege on the catalog. | ## Not verified diff --git a/static/img/platform/admin/cp-engine-ext-sr-overview.png b/static/img/platform/admin/cp-engine-ext-sr-overview.png index f26a495..f28a725 100644 Binary files a/static/img/platform/admin/cp-engine-ext-sr-overview.png and b/static/img/platform/admin/cp-engine-ext-sr-overview.png differ diff --git a/static/img/platform/admin/cp-engine-ext-sr-settings.png b/static/img/platform/admin/cp-engine-ext-sr-settings.png index d92a166..170edd0 100644 Binary files a/static/img/platform/admin/cp-engine-ext-sr-settings.png and b/static/img/platform/admin/cp-engine-ext-sr-settings.png differ diff --git a/static/img/platform/admin/cp-register-sr-warning.png b/static/img/platform/admin/cp-register-sr-warning.png index 5f0a72b..8eebe92 100644 Binary files a/static/img/platform/admin/cp-register-sr-warning.png and b/static/img/platform/admin/cp-register-sr-warning.png differ diff --git a/static/img/platform/admin/gov-engine-starrocks-warning.png b/static/img/platform/admin/gov-engine-starrocks-warning.png index 0f35c20..d2537b3 100644 Binary files a/static/img/platform/admin/gov-engine-starrocks-warning.png and b/static/img/platform/admin/gov-engine-starrocks-warning.png differ