Skip to content

Do not record the table handle in the table scan span - #139

Open
Vedin wants to merge 1 commit into
fix/snowflake-zero-precision-decimalfrom
fix/scan-span-credentials
Open

Vedin wants to merge 1 commit into
fix/snowflake-zero-precision-decimalfrom
fix/scan-span-credentials

Conversation

@Vedin

@Vedin Vedin commented Oct 26, 2026

Copy link
Copy Markdown

Problem

The #[instrument] attribute on table_scan skipped only arrow_schema, session and filters. As a result, table: &Table and config: Option<&DataFusionTableConfig> were recorded with their derived Debug on every scan.

Table's Debug includes its catalog. For a RestCatalog, that is:

  • the REST Configuration, which prints bearer_access_token in clear;
  • the default ObjectStoreBuilder, whose object_store AmazonS3Builder Debug prints secret_access_key and token.

The full table metadata follows. Subscribers that keep span fields print all of it. A JSON fmt layer prints the span list in every event raised inside the span, which includes the INFO "Iceberg Parquet data cache initialized" line. An OpenTelemetry layer exports the fields as span attributes.

Recording the handle also costs a Debug format of tens of KB per scan. config can carry a large set of excluded file paths.

Change

table_scan now skips table and config. It keeps table_identifier and the other explicit fields, and adds has_config = config.is_some(). A comment says why the table is not recorded.

Test

table::tests::table_scan_span_does_not_record_the_table:

  • creates a table in an in-memory SQL catalog, with a fake s3.secret-access-key table property that stands in for a credential reachable from the table's Debug (the test asserts that the table's Debug contains it);
  • builds a scan under a minimal capturing subscriber written with tracing only, so no new dev-dependency;
  • asserts that the scan span has table_identifier and has_config=false, has no table or config field, and that no span contains the fake value.

Without the change, the test fails because the scan span records table=Table { … }.

Notes

  • Independent of Validate declared string and binary lengths at Iceberg writes #137 (already merged into this branch) and of Remove uncommitted Iceberg data files after failed writes #138. The branch starts at the current head of fix/snowflake-zero-precision-decimal and merges cleanly with fix/cleanup-failed-writes.
  • Possible follow-ups, not in this PR:
    • credential-free Debug impls for ObjectStoreBuilder (S3 and GCS builders print their secrets), the REST Configuration (bearer_access_token, basic_auth, api_key, aws_v4_key.access_key), Table and the REST catalogs;
    • skip_all on the transaction and view Operation::execute spans and on the arrow::write spans, which record table metadata or properties.

🤖 Generated with Claude Code

The table_scan span recorded `table` and `config` with their Debug.
A Table's Debug includes its catalog: for a REST catalog that is the
configuration, which prints the bearer token in clear, and the default
object-store builder, which prints the S3 secret access key and session
token, followed by the whole table metadata. Span fields are printed in
every log line raised inside the span (for example the parquet data
cache INFO line) and exported as span attributes, and formatting them
costs tens of KB of Debug output per scan.

Skip both arguments, keep table_identifier and record has_config. A
test captures the scan span's fields and checks that the table is not
recorded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant