Skip to content

docs: add M1a implementation plan and split M1 into M1a-M1c #19

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
viethqb wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs: add M1a implementation plan and split M1 into M1a-M1c #19

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 9 additions & 5 deletions docs/superpowers/plans/2026-10-17-p0-roadmap.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,16 +19,20 @@ Every milestone follows the repository's open-source workflow (see `CONTRIBUTING
| Milestone | Deliverable | Spec sections | Depends on | Exit criteria |
|---|---|---|---|---|
| **M0 Foundation** | Cargo workspace, CI, configuration loading, `egret-engine` binary with `--role all`, pgwire simple-query frontend over in-memory DataFusion, one DataFusion session per connection | §1.1, §1.3, §1.4, §4.1 (simple query only), §4.4 (`42601`, `XX000`) | — | `psql -h 127.0.0.1 -c "SELECT 1"` works against the binary; CI green |
| M1 Iceberg single-node | `CREATE CATALOG ... USING iceberg_rest`, PostgreSQL metadata store + migrations, Iceberg scan with delete files, table/manifest caches, snapshot diff API, Spark-generated fixtures | §2.1, §2.2, §2.3 (single-node), §4.4 (`42P01`) | M0 | Fixture tables (equality deletes, deletion vectors, schema/partition evolution) return the same results as Spark |
| M2 Distribution | `egret-data/datafusion-distributed` fork (per-query memory pool, `TaskKey.attempt`), worker role, `WorkerResolver` (static + Kubernetes), Iceberg work unit feed, credentials in work units | §1.2, §2.3, §4.3 (memory), §5.1 | M1 | TPC-H SF1 distributed results equal single-node results |
| M1a Iceberg catalogs and single-node reads ([plan](2026-10-24-p0-m1a-catalogs.md), #16) | PostgreSQL metadata store + migrations, `CREATE/DROP/SHOW CATALOG ... USING iceberg_rest` with secret references, Iceberg reads through iceberg-rust's `TableScan` (delete files, schema evolution), engine + per-connection sessions built after startup, Docker test environment with Spark-generated fixtures | §1.3, §2.1, §2.3 (vended credentials from `loadTable`), §4.1 (namespaces), §4.4 (`42P01`) | M0 | Positional-delete, deletion-vector, schema- and partition-evolution fixtures return the same rows as Spark over pgwire; catalogs survive a restart |
| M1b Scan planning and metadata caches (#17) | Engine-side `plan_files` on iceberg-rust's evaluators (thin `egret-data/iceberg-rust` fork), global manifest list / manifest caches, table cache (staleness, single-flight, `strong`), statistics, filter pushdown, equality-delete fixtures | §2.1 (scan pushdown), §2.2 | M1a | All fixtures, including equality deletes, match Spark; repeated queries do not re-read manifests |
| M1c Snapshot diff and credential refresh (#18) | Snapshot diff API, vended-credential refresh, Lakekeeper in the test environment | §2.2 (snapshot diff), §2.3 | M1b | Diffs match Spark-created snapshot histories; queries keep working across credential expiry |
| M2 Distribution | `egret-data/datafusion-distributed` fork (per-query memory pool, `TaskKey.attempt`), worker role, `WorkerResolver` (static + Kubernetes), Iceberg work unit feed, credentials in work units | §1.2, §2.3, §4.3 (memory), §5.1 | M1c | TPC-H SF1 distributed results equal single-node results |
| M3 Caching | foyer `CachingStorage`, rendezvous affinity + `RouteTaskHandler`, population policy, `CACHE SELECT`, cache metrics and `system.cache_stats` | §2.4, §2.5, §2.6 | M2 | Warm-cache rerun reads ≥ 90% of bytes from cache; cache survives worker restart |
| M4 Security | Authentication rules (JWT, OAUTHBEARER, SCRAM, TLS), roles/grants, RLS, masking, `SecurityBarrier`, OPA (embedded + remote), audit log | §3 | M1 (M3 for cache-safety tests) | RLS/masking differential tests match PostgreSQL; leak tests pass |
| M4 Security | Authentication rules (JWT, OAUTHBEARER, SCRAM, TLS), roles/grants, RLS, masking, `SecurityBarrier`, OPA (embedded + remote), audit log | §3 | M1a (M3 for cache-safety tests) | RLS/masking differential tests match PostgreSQL; leak tests pass |
| M5 Lifecycle & compatibility | Extended query protocol, portals, `CancelRequest`, admission control, `QUERY` retry, SQLSTATE mapping, `pg_catalog`/`information_schema`, job framework, `system.running_queries` | §4 (all), §1.3 `jobs` | M2, M4 | Client smoke tests (psql, psycopg, pgjdbc, SQLAlchemy, DBeaver, Metabase) pass; worker kill during a query < 32 MB succeeds |
| M6 Packaging | Helm chart, distroless image, `deploy/docker-compose.yml`, operations docs, TPC-H SF10/SF100 benchmark | §5.3, §5.4 (performance), §5.6 | M5 | Definition of done (§5.6) satisfied |

## Known pre-work per milestone

- **M1:** confirm that iceberg-rust's `ManifestEvaluator`, `ExpressionEvaluator` and `InclusiveMetricsEvaluator` are
public at revision `5b82ab99`; otherwise create a thin fork first.
- **M1 (done in M1a):** at rev `5b82ab99`, iceberg-rust's `expr::visitors` (`ManifestEvaluator`,
`ExpressionEvaluator`, `InclusiveMetricsEvaluator`) and `delete_file_index` are `pub(crate)`, and upstream `main`
is the same, so M1b needs a thin fork. `iceberg::cache::ObjectCacheProvide` exists but is not wired into `Table`
yet; M1b evaluates wiring it in as the smaller patch.
- **M2:** create the `egret-data/datafusion-distributed` fork and decide the base revision.
- **M4:** confirm `regorus` covers the Rego built-ins used by the policy library.
Loading
Loading