-
Notifications
You must be signed in to change notification settings - Fork 23
test: add it for produce & scan log for all supported datatypes #205
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
@luoyuxia Hi, can you take a look? |
|
@fresh-borzoni Could you please also help review this? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR adds an end-to-end integration test for producing and scanning log tables across all supported data types and fixes how Arrow temporal types and CHAR are read during log table deserialization.
Changes:
- Add
all_supported_datatypesintegration test to validate append/scan behavior across integers, floats, booleans, strings, decimals, dates, times (various precisions), timestamps (NTZ/LTZ with multiple precisions), bytes, and nulls. - Update
ColumnarRowto read ArrowTimestamp,Date, andTimecolumns via their concrete Arrow array types, correctly converting to Fluss temporal representations. - Enhance
get_charto handle bothFixedSizeBinary(KV table) andUtf8(log table) storage formats.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| crates/fluss/tests/integration/table.rs | Adds all_supported_datatypes IT that creates a log table with all supported data types, appends one fully-populated row plus a fully-null row, scans via the log scanner, and asserts correct round-trip values for each column and null handling. |
| crates/fluss/src/row/column.rs | Fixes deserialization of Arrow Timestamp, Date32, and Time32/Time64 into Fluss TimestampNtz, TimestampLtz, Date, and Time, and extends get_char to support both FixedSizeBinary and Utf8 representations, improving compatibility with log tables. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
fresh-borzoni
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@linguoxuan Thank you for the PR.
LGTM
luoyuxia
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@linguoxuan Thanks for the pr. LGTM. I only have one question. PTAL
crates/fluss/src/row/column.rs
Outdated
| std::str::from_utf8(bytes).expect("Invalid UTF-8 in char field") | ||
| match arrow_field.data_type() { | ||
| ArrowDataType::FixedSizeBinary(_) => { | ||
| // KV table format: char stored as FixedSizeBinary |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hi, what does it mean? Why said KV table format stored as FixedSizeBinary. Do you mean change log for kv table format?
Shouldn't KV table format and Log table format use same tye for arrow change?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@luoyuxia Hi, sorry for the misleading in my previous code and comment!
You're absolutely right to question this. The truth is: Looking at the to_arrow_type function in arrow.rs, DataType::Char(_) is consistently mapped to ArrowDataType::Utf8 for all table types. So The get_char() method should read data from StringArray, not FixedSizeBinaryArray. Thanks for pointing this out! I have pushed a new commit.
|
@linguoxuan Thanks for update. But the ci fails |
Hi @luoyuxia. The ci fails because there's new commit in the main branch which change the interface of the GenericRow::new() method. I have rebased the main branch and fixed the compilation failure issue. By the way, please ignore my accidental action of closing the pull request. |
Purpose
Linked issue: close #199
Brief change log
Tests
API and Format
Documentation