Skip to content

feat(catalog-rest): client support for server-side REST scan planning#2656

Draft
huan233usc wants to merge 2 commits into
apache:mainfrom
huan233usc:scan-planning/option-b-scanplanner
Draft

feat(catalog-rest): client support for server-side REST scan planning#2656
huan233usc wants to merge 2 commits into
apache:mainfrom
huan233usc:scan-planning/option-b-scanplanner

Conversation

@huan233usc

@huan233usc huan233usc commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Part of #1690 — client support for REST server-side scan planning.

Stacked on #2651 (vended storage credentials). The credential commit shown
in this diff comes from #2651; once that merges this branch will be rebased and
the diff will narrow to just the scan-planning changes. This PR reuses
#2651's credential support rather than duplicating it.

What changes are included in this PR?

A client implementation of the REST scan-planning protocol (planTableScan /
fetchPlanningResult / fetchScanTasks). When a catalog advertises the planning
endpoints, a table scan delegates planning to the server and consumes the
returned FileScanTasks; otherwise it transparently falls back to native
client-side planning.

Design

The feature hangs off a single seam — TableScan::plan_files() — so execution
(to_arrow, the Arrow reader) is untouched and DataFusion needs no changes.

Table::scan() ─► TableScan::plan_files()
   │ injected ScanPlanner present?
   ├─ no  ─► native manifest planning (unchanged)
   └─ yes ─► ScanPlanner::plan_table_scan(ScanPlanRequest)
                │ endpoint negotiation: gate on GET /v1/config `endpoints`
                ▼
   POST .../plan ─► COMPLETED ─────────────────────────────┐
                 └─► SUBMITTED ─► poll GET .../plan/{id}     │ (exp backoff)
   plan-tasks? ─► POST .../tasks (tokens may recurse) ───────┤
                                                             ▼
   convert wire content-files ─► FileScanTask (public builders)
   + build a plan-scoped FileIO from vended `storage-credentials`
     via FileIOBuilder::with_prefixed_props (from #2651)
                                                             ▼
   ServerScanPlan { tasks, file_io } ─► to_arrow() reads tasks through file_io
   (on Drop before completion: best-effort DELETE .../plan/{id})

Components

  1. Injection seam — a narrow ScanPlanner capability trait
    (crates/iceberg/src/scan/planner.rs); Table carries an optional
    Arc<dyn ScanPlanner>; plan_files() delegates and falls back on
    FeatureUnsupported. The core Catalog trait is untouched.
  2. Endpoint negotiationCatalogConfig parses the endpoints field of
    GET /v1/config; scan-plan calls are gated by Endpoint::check.
  3. Wire DTOs — plan / fetch-planning-result / fetch-scan-tasks types plus a
    lean content-file shape.
  4. State machine — submit → poll-with-backoff → recursive fetchScanTasks,
    with best-effort cancel on drop.
  5. Conversion — wire content-files → FileScanTask via the public builders;
    the scan's own bound filter is the per-task predicate, pushed down as Iceberg
    expression JSON when losslessly encodable.
  6. Credential reuseScanPlanner::plan_table_scan returns
    ServerScanPlan { tasks, file_io }; the planner builds the plan-scoped
    FileIO from the vended storage-credentials using feat(rest): support vended storage credentials with per-prefix FileIO #2651's
    FileIOBuilder::with_prefixed_props (no duplicate credential machinery).

Alternative injection design

The same feature with planning placed on the Catalog trait
(Catalog::plan_table_scan, Table holding Arc<dyn Catalog>) is in
huan233usc#2 for comparison. This PR uses
the narrow-trait design because it keeps the central Catalog trait minimal and
avoids giving every Table a back-reference to the full catalog.

Future work (design feedback welcome)

Native planning stays inline in plan_files() here; a follow-up could generalize
ScanPlanner into the single planning abstraction (a NativeScanPlanner peer +
a FallbackScanPlanner that composes server→native), making plan_files() a
one-liner with no branch. Deferred to keep this PR focused.

Are these changes tested?

Yes — unit tests for the wire DTOs, endpoint codec, and expression-JSON
serialization, conversion tests, and end-to-end mockito tests covering the
completed-inline, submitted-then-polled, and recursive plan-task fan-out paths.

…anPlanner injection)

Implements the client side of the Iceberg REST scan-planning protocol
(planTableScan / fetchPlanningResult / fetchScanTasks) so that, when a REST
catalog advertises the planning endpoints, table scans delegate planning to
the server instead of reading manifests locally.

Injection design — Variant B (narrow capability trait):
- New `iceberg::scan::ScanPlanner` trait + `ScanPlanRequest`; `Table`/
  `TableScanBuilder` gain an optional `Arc<dyn ScanPlanner>`; `TableScan::
  plan_files()` delegates to it and falls back to native planning on
  `FeatureUnsupported`. The core `Catalog` trait is left untouched.

REST crate:
- `scan_planning` module: wire DTOs, endpoint negotiation (parses the
  `endpoints` field of /v1/config and gates the scan-plan calls), the async
  submit/poll/fetch state machine with exponential backoff and best-effort
  Drop-based cancel, and conversion of wire content-files into FileScanTasks
  via their public builders (no DataFile internals needed).
- `RestScanPlanner` is attached to every table the catalog returns.
- Per-task row predicate is the client's own bound scan filter (correct), and
  the scan filter is pushed down as Iceberg expression JSON when encodable.

Tested: DTO/endpoint/expr unit tests, conversion, and end-to-end mockito tests
for completed-inline, submitted-then-polled, and plan-task fan-out paths.
DataFusion needs no changes.
…ning

Builds on the Variant B server-side scan planning so a server-planned scan can
actually read its data files end-to-end against Unity Catalog FGAC tables
(verified live: server applies column masks, client reads the masked rows).

- Vended storage credentials: REST `storage-credentials` (from load-table and
  from scan-plan responses) are attached to a `FileIO` via the new
  `FileIOBuilder::with_storage_credentials` / `StorageConfig` credential support,
  resolved per object path (longest-prefix) by `OpenDalResolvingStorageFactory`.
- R10 (plan-scoped credentials): `ScanPlanner::plan_table_scan` now returns
  `ServerScanPlan { tasks, file_io }`. The REST planner builds a plan-scoped
  `FileIO` from the credentials the server vends in the plan/poll responses, and
  `TableScan::to_arrow` reads through it. Server-planned tables typically vend
  credentials only in the plan response, not at load-table time.
- PlanStatus / content type now also accept the SCREAMING_CASE forms UC emits
  (e.g. `COMPLETED`, `DATA`) in addition to the kebab-case spec forms.

Note: UC's scan planning currently rejects `case-sensitive=true`; callers must
build scans with `with_case_sensitive(false)` against such servers.
@huan233usc huan233usc force-pushed the scan-planning/option-b-scanplanner branch 2 times, most recently from 90ca4a2 to 551d64a Compare June 16, 2026 20:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant