Skip to content

fix: preserve typed DataFusion errors through Shared and ReplayExec - #8677

Draft
wjones127 with Copilot wants to merge 2 commits into
mainfrom
copilot/preserve-typed-datafusion-errors
Draft

fix: preserve typed DataFusion errors through Shared and ReplayExec#8677
wjones127 with Copilot wants to merge 2 commits into
mainfrom
copilot/preserve-typed-datafusion-errors

Conversation

Copilot AI commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Two error paths erased the concrete type of a DataFusionError, retaining only its display string—making it impossible for callers to downcast_ref or walk the source chain to recover structured error information.

Changes

lance-core/src/error.rsDataFusionError::Shared with multiple owners

When Arc::try_unwrap fails (the Arc has >1 owner), the previous fallback constructed Error::Execution(shared.to_string()), dropping the typed inner error. Now it re-wraps the Arc as DataFusionError::Shared(shared) and stores it via Error::External, so the original type remains reachable through Error::source() / downcast_ref.

lance/src/io/exec/utils.rsReplayExec shared stream

ShareableRecordBatchStream previously routed errors through lance_core::Error + CloneableError. On clone, CloneableError fell through to Error::cloned(to_string()) for Error::External variants, discarding the type before the second consumer ever saw it.

The stream now stores errors as Arc<DataFusionError> (which is Clone) and the adapter emits them as DataFusionError::Shared(arc). Both consumers share the same Arc and retain full access to the typed source chain.

// Before — second consumer receives a display-only string error
.map_err(|e| DataFusionError::External(e.0.to_string().into()))

// After — both consumers get DataFusionError::Shared(arc) pointing at the original typed error
.map_err(DataFusionError::Shared)

Tests

  • test_datafusion_shared_multi_owner_preserves_type (lance-core): verifies a marker error carried as DataFusionError::External survives conversion from a multiply-owned DataFusionError::Shared.
  • test_replay_preserves_typed_error (lance): verifies both ReplayExec partitions can downcast_ref to the original typed error.

- In error.rs: when Arc::try_unwrap fails for DataFusionError::Shared,
  re-wrap in DataFusionError::Shared and store as Error::External so the
  typed source chain is reachable via Error::source/downcast_ref.

- In utils.rs: replace CloneableResult/CloneableError in the replay
  stream with Result<RecordBatch, Arc<DataFusionError>>. Errors are now
  wrapped with Arc::new and emitted as DataFusionError::Shared(arc),
  preserving the typed source for both ReplayExec consumers.

Closes #8676

Co-authored-by: wjones127 <5488879+wjones127@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix error handling to preserve typed DataFusion errors fix: preserve typed DataFusion errors through Shared and ReplayExec Aug 20, 2026
Copilot AI requested a review from wjones127 August 20, 2026 22:29
@github-actions github-actions Bot added the bug Something isn't working label Aug 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Preserve typed DataFusion errors through Shared and ReplayExec

2 participants