Skip to content

[Bug] Arrow sub field writers inherit the parent field's nullability #8757

Description

@thswlsqls

Search before asking

  • I searched in the issues and found nothing similar.

Paimon version
master @ 345526e (2.0-SNAPSHOT)

Compute Engine
Engine-agnostic (paimon-arrow)

Minimal reproduce step
Write a null array element into a column declared ARRAY<INT> NOT NULL:

RowType rowType = RowType.of(DataTypes.ARRAY(DataTypes.INT()).notNull());
try (ArrowFormatWriter writer = new ArrowFormatWriter(rowType, 16, true)) {
    writer.write(GenericRow.of(new GenericArray(new Object[] {1, null, 3})));
}

This throws IllegalArgumentException: Field 'element' expected not null but found null value.

What doesn't meet your expectations?
The array is NOT NULL but its element type is nullable, so a null element is valid and should be written.

ArrowFieldWriterFactoryVisitor.visit(ArrayType/VectorType/MapType/RowType) passes the parent field's isNullable down to the element/key/value/child writers instead of the nested type's own isNullable(), and ArrowFieldWriter.write(int, DataGetters, int) rejects nulls when that flag is false. The reverse case (ARRAY<INT NOT NULL> under a nullable parent) silently writes nulls, contradicting the non-nullable child field that ArrowUtils.toArrowField() declares.

Anything else?
Follow-up of #5538, which applied the same rule to ArrowFormatWriter only and left the sub writer creation sites unchanged.

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions