[fix](lance) Pass namespace-vended storage options through to the BE

[FANNG1]

What problem does this PR solve?

Fixes #66772 (problem 1).

A Lance REST catalog discarded the storage_options a namespace vended for a table, so any scan relying on credential vending failed:

open Lance dataset failed: LanceError(IO): ...
Failed to get AWS credentials: CredentialsNotLoaded("no providers in chain provided credentials")

The options were re-encoded twice on the way to the BE, through one five-entry S3-only table used in both directions:

namespace vends  access_key_id
  -> FE  LanceStorageOptions.forBackend   aws_access_key_id -> AWS_ACCESS_KEY
  -> TFileScanRangeParams.properties
  -> BE  kStorageKeys                     AWS_ACCESS_KEY -> aws_access_key_id
  -> lance-c

That table was written to emit one canonical spelling, which is correct outbound. Reading it backwards turns it into a parser that accepts only the spelling it happens to emit, so everything else was dropped — credentials under any other accepted alias, and every non-S3 provider's keys, which left the catalog unable to use credential vending outside S3 at all.

The failure was also split across the two halves, which made it hard to read: the FE passes the vended map to the Lance Java SDK untouched, so SHOW TABLES and DESC succeeded and only the scan failed.

The Lance Namespace specification describes storage_options as configuration "passed directly to Lance", so the protocol defines no key vocabulary of its own and a client cannot assume one.

Release note

Fixed a Lance REST catalog failing to scan with CredentialsNotLoaded when it relied on credentials vended by the namespace rather than static s3.access_key / s3.secret_key.

What is changed and how it works?

Stop re-encoding server-supplied options.

• TFileScanRangeParams.lance_storage_options (new, id 38) carries the options in Lance's own vocabulary. Set at ScanNode level — the same pattern as paimon_options — so credentials are not serialized once per fragment split. properties is deliberately not reused: it is copied wholesale into io::FileSystemProperties for the shared filesystem layer.
• FE builds a single option map, used both by the Java SDK when it opens the dataset and by the BE. The two can no longer disagree about how a dataset is reached. forBackend is deleted.
• BE _storage_options hands that map to lance-c as it arrives; kStorageKeys and the allow_http / use_path_style derivations are gone.
• The s3() TVF path populates the same field, including the schema-fetch RPC.

Two details worth review attention:

Vended aliases are renamed before merging. object_store resolves an alias and its canonical name to the same config key and keeps only one of the two values, chosen by hash order:

// lance-io/src/object_store/providers/aws.rs
let s3_key = AmazonS3ConfigKey::from_str(&key.to_ascii_lowercase()).ok()?;
...collect()   // HashMap — last write wins, iteration order randomized

So a namespace vending aws_endpoint while the catalog contributes endpoint would not override it; both would survive and Lance would pick between them unpredictably. Only the aws_-prefixed aliases are renamed — bare names such as token mean different things across providers (S3 session token vs. Azure bearer token) and this layer does not know the provider.

Catalog options now use the unprefixed spelling. object_store accepts both, but the unprefixed name is also the field name used by the OpenDAL backend, which performs no alias normalization at all.

allow_http is derived after merging, from whichever endpoint ends up in use, since a namespace can replace the catalog's endpoint or supply the only one there is.

Options that would change which data is read (bucket and its aliases, root) are dropped from the vended map, mirroring the keys Lance itself protects.

Rolling upgrade

An FE upgraded ahead of the BEs no longer puts vended credentials into TFileScanRangeParams.properties, and an older BE reads only that. A REST catalog with no static credentials therefore cannot be scanned until the BEs are upgraded too. The Lance catalog is not in a release yet, so this only affects development clusters.

How was this patch tested?

Unit tests. LanceStorageOptionsTest is new — LanceStorageOptions had no test of its own, and forBackend had zero call sites in any test, which is why this went unnoticed. It covers the emitted spelling, alias renaming, allow_http derivation, protected keys, non-S3 options, and empty/absent input. LanceThriftContractTest gains round-trip coverage of the new field.

Regression. The docker stub only ever vended aws_-prefixed keys, so the alias path had no coverage anywhere. It now serves a second table, all_types_unprefixed, backed by the same dataset but vending the unprefixed spelling, and test_lance_rest_catalog scans it with no static credentials configured.

End to end, against Apache Gravitino 1.3.0's lance-rest service, which vends the unprefixed spelling. Catalog with no s3.access_key / s3.secret_key:

CREATE CATALOG lance_novend PROPERTIES (
    "type" = "lance",
    "lance.catalog.type" = "rest",
    "lance.rest.uri" = "http://127.0.0.1:9101/lance",
    "lance.namespace.parent" = "lance_catalog"
);

SELECT count(*), min(row_id), max(row_id), sum(row_id) FROM lance_novend.doris_probe.rest_probe;
-- 1034  1  1034  535095

against a 1034-row, 2-fragment dataset with an IVF_FLAT index. Predicate pushdown and vector search both return correct results on the same catalog, and a catalog with static credentials is unaffected. Before this change the same statement failed with CredentialsNotLoaded.

https://claude.ai/code/session_01M3mYXBKShBonG6Lg3br4Ld

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?