Repository navigation
fix: disable Fabric fast optimize during compaction - #111
Merged
Miles Cole (mwc360) merged 3 commits intoSep 24, 2026
Merged
Conversation
## Summary Temporarily disable Fabric Spark fast optimize while LakeBench runs an explicit OPTIMIZE command, then restore the exact prior session configuration even when compaction fails. ## Prompting Intent The engineer asked LakeBench's Fabric Spark engine to prevent fast optimize from skipping compaction work during benchmark OPTIMIZE phases while preserving the caller's pre-existing Spark configuration afterward. The change should remain narrowly scoped to Fabric Spark and avoid persistent session changes. ## Linked Sources - Microsoft Fabric table compaction and fast optimize documentation: https://learn.microsoft.com/en-us/fabric/data-engineering/table-compaction ## Rationale The configuration is changed around each OPTIMIZE call rather than during engine initialization. Constructor-scoped mutation would remain active when optimization is not requested and restoring after the first table would re-enable fast optimize for later tables in multi-table benchmarks. A try/finally block preserves the original value on both successful and failed compaction. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Temporarily disable Fabric Spark fast optimize while LakeBench runs an explicit OPTIMIZE command, then restore the exact prior session configuration even when compaction fails.
Rationale
Fast Optimize optimistically reduces scope of compaction operations to reduce write amplification but use cases that prioritize absolute best query performance, it is recommended to be disabled and let OPTIMIZE do a full scoped compaction job of all small files.