Skip to content

Batch builder API#98

Draft
gbrgr wants to merge 13 commits into
mainfrom
gb/batch-builder
Draft

Batch builder API#98
gbrgr wants to merge 13 commits into
mainfrom
gb/batch-builder

Conversation

@gbrgr
Copy link
Copy Markdown
Contributor

@gbrgr gbrgr commented May 11, 2026

Summary

  • Retire GatheredColumn/GatheredBatch API — removed ~350 lines of Julia and Rust code. The old gather API accumulated slices in Julia memory before writing; all callers now use ColumnBatchBuilder directly.
  • Route ColumnBatch writes through ColumnBatchBuilder internallywrite_columns_inner in Rust now uses the builder instead of the retired build_arrow_array_gathered path, making ColumnBatchBuilder the single canonical copy path.
  • New SliceBatch Julia API — high-level wrapper that constructs SliceRef structs from plain Julia arrays, so users don't need to manage raw pointers.
  • New writer tests — three integration tests covering multi-slice coalescing, builder reuse across windows, and Julia→Unix epoch conversion for date/timestamp columns.
  • batch_builder.rs cleanup — extracted append_primitive! and append_transform! macros, reducing append_numeric from 147 to 35 lines; merged duplicate JULIA_DATE/JULIA_TIMESTAMP/JULIA_TIMESTAMPTZ finalize arms with their non-Julia counterparts (−67 lines total).

gbrgr and others added 13 commits May 11, 2026 10:33
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Arrow.jl returns Arrow.Date/Arrow.Timestamp wrappers, not Dates.Date/DateTime.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…rge Julia finalize arms

Removes ~130 lines of duplicated identity/scatter dispatch in append_numeric by
introducing two declarative macros. Merges JULIA_DATE/TIMESTAMP/TIMESTAMPTZ finalize
arms with their non-Julia counterparts since the Arrow output type is identical.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This reverts commit dfdb437.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant