[Vedin]

Why

On S3, a data-range GET of an object that was requested less than about a second earlier gets its response headers roughly twice as fast as one to an object left untouched for ten seconds or more. Measured on the deployed SPCS service (TPC-H SF100, one run, one image):

• with the footer cache off, every data GET follows a footer GET of the same file, and the data GET header p50 is 28–42 ms (Q4, Q6, Q12, Q14) against 85–87 ms with the cache on;
• with the cache on, data GETs to objects touched < 100 ms earlier have p50 42–46 ms, against 79–89 ms when the last touch was 1–300 s earlier.

A footer cache hit removes exactly the footer read that made the data read fast, so warm scans pay the slow first touch on every data file.

What

With ICEBERG_PARQUET_PRE_TOUCH_MAX_IN_FLIGHT above 0 (read once per process; default 0, off), a footer cache hit issues one 8-byte ranged GET of the file's tail. Under file_stream_open_ahead the hit happens while the previous file is still being read, so the touch lands before that file's data reads start.

• The touch never delays the hit: it is a SpawnedTask owned by the reader (aborted when the reader drops; no detached task).
• It takes a slot of a process-wide semaphore without waiting and is skipped when every slot is taken.
• Its outcome is ignored: the data reads that follow report any real error.
• New scan metrics parquet_pre_touches and parquet_pre_touch_skipped; the cache init log gains pre_touch_max_in_flight.
• Cost: at most one extra GET per file whose footer was served from cache.

MetadataCache::load becomes load_reporting_hit, which also returns whether the footer came from the cache.

Tests

cargo test -p datafusion_iceberg --lib parquet_metadata_cache: 27 passed (3 ignored benches), including four new tests:

• the env value parses off by default and is capped;
• a hit issues exactly one GET, of size-8..size, and counts it;
• with no free slot the touch is skipped and counted;
• dropping the reader aborts a slow touch and frees its slot.

cargo clippy -p datafusion_iceberg --all-targets: no warnings in the changed file.

Stacked on #140. The engine measurement on SPCS follows with the knob on in a separate arm.

🤖 Generated with Claude Code