feat(rest): support vended storage credentials with per-prefix FileIO by plusplusjiajia · Pull Request #2651 · apache/iceberg-rust

plusplusjiajia · 2026-06-15T10:33:56Z

What

load_table / create_table now use the storage_credentials returned in
LoadTableResult to build a per-prefix FileIO. The field was exposed (its doc
says clients should prefer it over config) but never consumed.

Changes

Build one storage per credential prefix instead of merging all credentials into
a single props map: FileIO routes each path to the longest-matching prefix
(new FileIOBuilder::with_prefixed_props), so multi-location tables use the
right credentials. Paths matching no prefix fall back to the default storage —
same behavior as Java's S3FileIO.clientForStoragePath. delete_stream groups
paths by routed storage.
Requesting delegation is opt-in via the existing header.* catalog config
(e.g. header.X-Iceberg-Access-Delegation=vended-credentials), applied to every
request like Java's RESTUtil.configHeaders — no dedicated prop, no hardcoded
default.
update_table no longer reloads after commit; since CommitTableResponse
carries no credentials, the transaction layer reuses the credentialed FileIO
it already holds (new Table::with_file_io).

xanderbailey

Thanks for the PR, I had a quick look and left some comments / questions.

xanderbailey · 2026-06-15T12:20:42Z

            .client
            .request(Method::GET, context.config.table_endpoint(table_ident))
+            // Opt in to vended storage credentials.
+            .header("X-Iceberg-Access-Delegation", "vended-credentials")


Java will read these headers off the catalog config, should we do the same?

header.X-Iceberg-Access-Delegation

Link to Java

@xanderbailey Thanks for the catch. Reworked to match the Java client: dropped the dedicated prop and rely on the existing generic header.* config (extra_headers). Vended credentials are opt-in via header.X-Iceberg-Access-Delegation=vended-credentials, applied to all requests like Java's RESTUtil.configHeaders — no hardcoded default.

xanderbailey · 2026-06-15T12:48:28Z

        let file_io = self
-            .load_file_io(Some(&response.metadata_location), None)
-            .await?;
+            .load_table(commit.identifier())


A full load table seems heavy to me. Can we not wire in the pre-credentialed FileIO into this method somehow?

@xanderbailey Good point — update_table no longer reloads. It builds FileIO from the commit response, and the transaction layer reuses the credentialed FileIO it already holds (do_commit + new Table::with_file_io). Back to one GET per commit.

xanderbailey · 2026-06-15T13:23:03Z

 }

+/// FileIO props: server `config`, then vended `storage_credentials` (longest prefix wins), then user props.
+fn table_file_io_config(


I also notice that Java constructs clientByPrefix Link which keeps credentials separate per prefix. So we can't support multi-location tables here I think? Also it would silently fail in those cases which I don't think is ideal? WDYT?

@xanderbailey Thanks, agreed. Each vended credential now maps to its own storage instead of merging all into one props map. FileIO routes each path to the longest-matching prefix(with_prefixed_props/get_storage); removed table_file_io_config.
On silent failures: this matches Java's S3FileIO.clientForStoragePath — a path matching no credential prefix falls back to the default storage from base properties (longest-prefix-match), so a real access problem surfaces as a normal I/O error, not a silent no-op.

…redentials apache#2651) Client implementation of the REST scan-planning protocol, reworked to build on the vended-credential support from apache#2651 instead of duplicating it. - Reuses `FileIOBuilder::with_prefixed_props` (from apache#2651) to build a plan-scoped FileIO from the `storage-credentials` the server vends in the plan response; no separate credential-resolution machinery. - Narrow `ScanPlanner` capability trait + `Table` handle; `plan_files()` delegates and falls back to native planning on `FeatureUnsupported`. - Endpoint negotiation, submit/poll/fetch state machine, wire→FileScanTask conversion. DataFusion unchanged. Stacked on apache#2651 (vended storage credentials); will rebase to drop those commits once it merges.

huan233usc · 2026-06-16T19:56:05Z

+        }
+
+        // Otherwise group paths by routed storage, then batch-delete each group.
+        let mut groups: HashMap<String, Vec<String>> = HashMap::new();


This collects the entire stream before deleting. Since delete_stream is used by purge/cleanup paths, large tables can make this unbounded in memory.

Can we keep routing per prefix but flush bounded batches as we iterate, instead of materializing all paths first?

@huan233usc Good catch, thanks. Reworked to flush bounded per-prefix batches as we iterate instead of materializing everything — memory's now bounded regardless of stream size, mirroring Java's S3FileIO.deleteFiles. Added a test crossing the batch boundary.

huan233usc · 2026-06-16T19:58:17Z

+            for cred in creds {
+                let mut prefixed = props.clone();
+                prefixed.extend(cred.config.clone());
+                builder = builder.with_prefixed_props(cred.prefix.clone(), prefixed);


qq: do we plan to handle credential refresh as follow up?

@huan233usc Good question！— not in this PR. FileIO is built once from the LoadTable creds, so no refresh yet. Good follow-up: the opendal S3 backend already has a ProvideCredential hook (reqsign re-invokes on expires_in), so a provider re-fetching from loadCredentials before s3.session-token-expires-at-ms — like Java's VendedCredentialsProvider — fits there. Will track as a separate PR.

huan233usc · 2026-06-19T17:43:57Z


-        let file_io = self.load_file_io(Some(metadata_location), None).await?;
+        let file_io = self
+            .load_file_io(Some(metadata_location), None, None)


Worth check if e2e for register table requires plumbing credentials. But it could be a separate pr if needed.

huan233usc

LGTM, thanks

feat(rest): use vended storage credentials from LoadTable response

894d5cf

plusplusjiajia force-pushed the feat/rest-vended-credentials branch from ac46430 to 894d5cf Compare June 15, 2026 12:01

xanderbailey reviewed Jun 15, 2026

View reviewed changes

plusplusjiajia force-pushed the feat/rest-vended-credentials branch from 61ed31d to 9591c53 Compare June 16, 2026 02:04

plusplusjiajia changed the title ~~feat(rest): use vended storage credentials from LoadTable response~~ feat(rest): support vended storage credentials with per-prefix FileIO Jun 16, 2026

feat(rest): per-prefix vended credentials with opt-in delegation header

d5720b7

plusplusjiajia force-pushed the feat/rest-vended-credentials branch from 9591c53 to d5720b7 Compare June 16, 2026 02:33

huan233usc mentioned this pull request Jun 16, 2026

feat(catalog-rest): client support for server-side REST scan planning #2656

Draft

huan233usc reviewed Jun 16, 2026

View reviewed changes

fix(io): bound delete_stream memory by flushing per-prefix batches

47748e3

plusplusjiajia force-pushed the feat/rest-vended-credentials branch from 00a51e4 to 47748e3 Compare June 17, 2026 09:46

huan233usc reviewed Jun 19, 2026

View reviewed changes

huan233usc approved these changes Jun 19, 2026

View reviewed changes

Conversation

plusplusjiajia commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Changes

Uh oh!

xanderbailey left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

plusplusjiajia Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

plusplusjiajia Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

plusplusjiajia Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

huan233usc left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

plusplusjiajia commented Jun 15, 2026 •

edited

Loading

plusplusjiajia Jun 15, 2026 •

edited

Loading

plusplusjiajia Jun 15, 2026 •

edited

Loading

plusplusjiajia Jun 15, 2026 •

edited

Loading