[FLINK-39415][postgres] Fix TIMESTAMPTZ type mapping in pipeline connector#4371
Open
avichaym wants to merge 1 commit intoapache:masterfrom
Open
[FLINK-39415][postgres] Fix TIMESTAMPTZ type mapping in pipeline connector#4371avichaym wants to merge 1 commit intoapache:masterfrom
avichaym wants to merge 1 commit intoapache:masterfrom
Conversation
avichaym
force-pushed
the
fix/timestamptz-iceberg-sink
branch
from
bdc434b to
23ca01d
Compare
avichaym
force-pushed
the
fix/timestamptz-iceberg-sink
branch
3 times, most recently
from
5336d69 to
e8d1f25
Compare
…ector Root cause: PostgresTypeUtils (pipeline connector) mapped TIMESTAMPTZ to ZonedTimestampType (TIMESTAMP_WITH_TIME_ZONE), but the Debezium deserializer only has a converter for TIMESTAMP_WITH_LOCAL_TIME_ZONE. This mismatch causes the deserializer to produce LocalZonedTimestampData while BinaryWriter.write() casts to ZonedTimestampData, resulting in binary data corruption and NumberFormatException when the Iceberg sink reads the field via BinaryRecordData.getZonedTimestamp(). The existing test PostgresFullTypesITCase already expects LocalZonedTimestampData for TIMESTAMPTZ (line 1211: DataTypes.TIMESTAMP_LTZ(0)), confirming the correct type mapping is TIMESTAMP_WITH_LOCAL_TIME_ZONE. Changes: - PostgresTypeUtils: TIMESTAMPTZ -> TIMESTAMP_LTZ(scale) (was ZonedTimestampType) - DebeziumEventDeserializationSchema: add convertToZonedTimestamp() for future TIMESTAMP_WITH_TIME_ZONE support - Add PostgresTypeUtilsTimestamptzTest: validates TIMESTAMPTZ maps to TIMESTAMP_WITH_LOCAL_TIME_ZONE type root - Add IcebergTypeUtilsTest: validates TIMESTAMP_LTZ type conversion and field getter creation for Iceberg sink
avichaym
force-pushed
the
fix/timestamptz-iceberg-sink
branch
from
e8d1f25 to
f48c4e4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
Fix TIMESTAMPTZ type mapping in the PostgreSQL pipeline connector that causes
NumberFormatExceptionwhen writing to the Iceberg sink.PostgresTypeUtilsmappedTIMESTAMPTZtoZonedTimestampType(
TIMESTAMP_WITH_TIME_ZONE), but the Debezium deserializer only producesLocalZonedTimestampData(TIMESTAMP_WITH_LOCAL_TIME_ZONE). This typemismatch causes binary data corruption in
BinaryRecordData, crashing theIceberg sink's
IcebergTypeUtils.createFieldGetter()during both snapshotand CDC phases.
The existing test
PostgresFullTypesITCase(line 1211) already expectsTIMESTAMP_LTZ(0)for TIMESTAMPTZ, confirming the correct mapping.Brief change log
PostgresTypeUtils: MapTIMESTAMPTZ→TIMESTAMP_LTZ(scale)(wasZonedTimestampType)DebeziumEventDeserializationSchema: AddconvertToZonedTimestamp()for futureTIMESTAMP_WITH_TIME_ZONEsupportPostgresTypeUtilsTimestamptzTest— validates type mappingIcebergTypeUtilsTest: Add assertions forTIMESTAMP_LTZtype conversion and field getterVerifying this change
Does this pull request potentially affect one of the following parts
Documentation