Search before asking
Paimon version
master @ 345526e (2.0-SNAPSHOT)
Compute Engine
Engine-agnostic (paimon-arrow)
Minimal reproduce step
Write a null array element into a column declared ARRAY<INT> NOT NULL:
RowType rowType = RowType.of(DataTypes.ARRAY(DataTypes.INT()).notNull());
try (ArrowFormatWriter writer = new ArrowFormatWriter(rowType, 16, true)) {
writer.write(GenericRow.of(new GenericArray(new Object[] {1, null, 3})));
}
This throws IllegalArgumentException: Field 'element' expected not null but found null value.
What doesn't meet your expectations?
The array is NOT NULL but its element type is nullable, so a null element is valid and should be written.
ArrowFieldWriterFactoryVisitor.visit(ArrayType/VectorType/MapType/RowType) passes the parent field's isNullable down to the element/key/value/child writers instead of the nested type's own isNullable(), and ArrowFieldWriter.write(int, DataGetters, int) rejects nulls when that flag is false. The reverse case (ARRAY<INT NOT NULL> under a nullable parent) silently writes nulls, contradicting the non-nullable child field that ArrowUtils.toArrowField() declares.
Anything else?
Follow-up of #5538, which applied the same rule to ArrowFormatWriter only and left the sub writer creation sites unchanged.
Are you willing to submit a PR?
Search before asking
Paimon version
master @ 345526e (2.0-SNAPSHOT)
Compute Engine
Engine-agnostic (
paimon-arrow)Minimal reproduce step
Write a null array element into a column declared
ARRAY<INT> NOT NULL:This throws
IllegalArgumentException: Field 'element' expected not null but found null value.What doesn't meet your expectations?
The array is NOT NULL but its element type is nullable, so a null element is valid and should be written.
ArrowFieldWriterFactoryVisitor.visit(ArrayType/VectorType/MapType/RowType)passes the parent field'sisNullabledown to the element/key/value/child writers instead of the nested type's ownisNullable(), andArrowFieldWriter.write(int, DataGetters, int)rejects nulls when that flag is false. The reverse case (ARRAY<INT NOT NULL>under a nullable parent) silently writes nulls, contradicting the non-nullable child field thatArrowUtils.toArrowField()declares.Anything else?
Follow-up of #5538, which applied the same rule to
ArrowFormatWriteronly and left the sub writer creation sites unchanged.Are you willing to submit a PR?