Skip to content

Conversation

@linguoxuan
Copy link
Contributor

@linguoxuan linguoxuan commented Jan 23, 2026

Purpose

Linked issue: close #199

Brief change log

  1. Add it test for produce & scan log for all supported datatypes.
  2. Fix incorrect Arrow array type reading for Timestamp, Date, Time in Log table deserialization.

Tests

API and Format

Documentation

@linguoxuan
Copy link
Contributor Author

linguoxuan commented Jan 23, 2026

@luoyuxia Hi, can you take a look?

@luoyuxia luoyuxia requested a review from Copilot January 24, 2026 02:37
@luoyuxia
Copy link
Contributor

@fresh-borzoni Could you please also help review this?

Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds an end-to-end integration test for producing and scanning log tables across all supported data types and fixes how Arrow temporal types and CHAR are read during log table deserialization.

Changes:

  • Add all_supported_datatypes integration test to validate append/scan behavior across integers, floats, booleans, strings, decimals, dates, times (various precisions), timestamps (NTZ/LTZ with multiple precisions), bytes, and nulls.
  • Update ColumnarRow to read Arrow Timestamp, Date, and Time columns via their concrete Arrow array types, correctly converting to Fluss temporal representations.
  • Enhance get_char to handle both FixedSizeBinary (KV table) and Utf8 (log table) storage formats.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
crates/fluss/tests/integration/table.rs Adds all_supported_datatypes IT that creates a log table with all supported data types, appends one fully-populated row plus a fully-null row, scans via the log scanner, and asserts correct round-trip values for each column and null handling.
crates/fluss/src/row/column.rs Fixes deserialization of Arrow Timestamp, Date32, and Time32/Time64 into Fluss TimestampNtz, TimestampLtz, Date, and Time, and extends get_char to support both FixedSizeBinary and Utf8 representations, improving compatibility with log tables.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Contributor

@fresh-borzoni fresh-borzoni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@linguoxuan Thank you for the PR.
LGTM

Copy link
Contributor

@luoyuxia luoyuxia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@linguoxuan Thanks for the pr. LGTM. I only have one question. PTAL

std::str::from_utf8(bytes).expect("Invalid UTF-8 in char field")
match arrow_field.data_type() {
ArrowDataType::FixedSizeBinary(_) => {
// KV table format: char stored as FixedSizeBinary
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hi, what does it mean? Why said KV table format stored as FixedSizeBinary. Do you mean change log for kv table format?
Shouldn't KV table format and Log table format use same tye for arrow change?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@luoyuxia Hi, sorry for the misleading in my previous code and comment!

You're absolutely right to question this. The truth is: Looking at the to_arrow_type function in arrow.rs, DataType::Char(_) is consistently mapped to ArrowDataType::Utf8 for all table types. So The get_char() method should read data from StringArray, not FixedSizeBinaryArray. Thanks for pointing this out! I have pushed a new commit.

@luoyuxia
Copy link
Contributor

@linguoxuan Thanks for update. But the ci fails

@linguoxuan
Copy link
Contributor Author

linguoxuan commented Jan 26, 2026

@linguoxuan Thanks for update. But the ci fails

Hi @luoyuxia. The ci fails because there's new commit in the main branch which change the interface of the GenericRow::new() method. I have rebased the main branch and fixed the compilation failure issue. By the way, please ignore my accidental action of closing the pull request.

@linguoxuan linguoxuan closed this Jan 26, 2026
@linguoxuan linguoxuan reopened this Jan 26, 2026
@luoyuxia luoyuxia merged commit 1bc399b into apache:main Jan 26, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

add it for produce & scan log for all supported datatypes

3 participants