Skip to content

Preserve null maps when projecting a map of structs - #4118

Open
syahra712 wants to merge 2 commits into
apache:mainfrom
syahra712:fix/null-map-of-structs
Open

syahra712 wants to merge 2 commits into
apache:mainfrom
syahra712:fix/null-map-of-structs

Conversation

@syahra712

Copy link
Copy Markdown

Rationale for this change

Closes #4114.

When ArrowProjectionVisitor.map() rebuilds a map whose values are structs, it calls pa.MapArray.from_arrays(offsets, keys, values). That drops the validity bitmap of the original array, so null maps come back as empty maps ({}) after projection. This is the map counterpart of the list issue handled in #4065.

The fix keeps the validity and offsets buffers of the original map array and only swaps in the projected child array, so nulls (and slicing offsets) are preserved.

Are these changes tested?

Yes. Added test__to_requested_schema_null_map_of_structs in tests/io/test_pyarrow.py, covering a map with struct values, a map with primitive values, and a sliced batch. The test fails on main (null rows become []) and passes with the fix.

Are there any user-facing changes?

Null map columns with struct values are now read back as null instead of an empty map.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Null map<K, struct<…>> is read back as an empty map

1 participant