Skip to content

[spark] Externalize oversized input partition metadata #10041

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
JustinBinber wants to merge 1 commit into from
Closed

[spark] Externalize oversized input partition metadata #10041

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions docs/docs/spark/configuration.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,20 @@ SELECT * FROM default.T1 JOIN default.T2 ON xxxx;
Streaming admission limits operate on whole splits. See [Triggers and Read Limits](./structured-streaming#triggers-and-read-limits)
for how byte, row, and file thresholds are applied.

For scans whose individual Spark input partitions contain very large split metadata, set
`spark.paimon.source.split.metadata.externalization.path` to a seekable file-system directory
shared by every executor. Paimon then stores input-partition metadata larger than
`spark.paimon.source.split.metadata.inline-threshold` in a scan-level container and each task reads
only its versioned byte range. Split frames carry length and checksum metadata so truncated or
corrupted ranges fail the task instead of returning partial data. The option reduces
task-serialization pressure; it does not bound the memory needed to plan or decode one input
partition. Enabled scans perform an extra serialization pass to measure each partition. Temporary
files and broadcasts are normally removed when the Spark application ends. An abruptly terminated
driver can leave files behind, so use a dedicated directory and clean orphaned application
directories only after confirming that no application still references them. Restrict write access
to trusted Spark applications; frame checksums detect accidental corruption but do not authenticate
the metadata writer.

<ConfigTable html={sparkConnectorConfigurationHtml} />

## Shared Paimon Options
Expand Down
12 changes: 12 additions & 0 deletions docs/generated/spark_connector_configuration.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,18 @@
<td>Boolean</td>
<td>Whether to verify SparkSession is initialized with required configurations.</td>
</tr>
<tr>
<td><h5>source.split.metadata.externalization.path</h5></td>
<td style="word-wrap: break-word;">(none)</td>
<td>String</td>
<td>A seekable shared file-system directory used to externalize oversized Spark batch input-partition metadata. The feature is disabled when this option is not set. Configure it as a Spark session option with the 'spark.paimon.' prefix.</td>
</tr>
<tr>
<td><h5>source.split.metadata.inline-threshold</h5></td>
<td style="word-wrap: break-word;">128 mb</td>
<td>MemorySize</td>
<td>Maximum serialized size of one Spark batch input partition kept inline. Larger metadata is stored under 'source.split.metadata.externalization.path'.</td>
</tr>
<tr>
<td><h5>source.split.target-size-with-column-pruning</h5></td>
<td style="word-wrap: break-word;">false</td>
Expand Down
Loading
Loading