[FLINK-38413] Adding support for append only tables to MySQL Pipeline connector #4137
Conversation
|
@lvyanquan Can I get a review on the above PR? Thanks. |
|
Please create a jira issue for this. |
|
|
||
| @Experimental | ||
| public static final ConfigOption<Boolean> SCAN_READ_CHANGELOG_AS_APPEND_ONLY_ENABLED = | ||
| ConfigOptions.key("scan.read-changelog-as-append-only.enabled") |
There was a problem hiding this comment.
I think this change would affect too many tables. Could we set a regular expression (e.g., scan.read-changelog-as-append-only.tables: adb.*) to modify the primary key only for tables that match this pattern?
There was a problem hiding this comment.
That's a good suggestion, I will try incorporating it.
There was a problem hiding this comment.
@lvyanquan On a second thought, one can have a dedicated Flink instance for creating append only tables. It seems having a common configuration scan.read-changelog-as-append-only.enabled seems to better as it brings consistency with the mysql-cdc-connector.
...java/org/apache/flink/cdc/connectors/mysql/source/parser/CustomAlterTableParserListener.java
Outdated
Show resolved
Hide resolved
There is already a ticket created mentioned in the description. FLINK-38413 for reference. |
lvyanquan
changed the title
junmuz
force-pushed
the
master
branch
2 times, most recently
from
d6c2958 to
36bbbae
Compare
|
@lvyanquan can you trigger the tests again and review the changes? I have tried responding to the comments raised from your side. Thanks in advance. |
2 participants
Description
In some cases, it is nice to have an append only table created for keeping infinite history of the MySQL CDC changes. With FLINK-38413, I am proposing the capability to create append only tables. This PR allows enabling append only table creation from mysql pipeline connector. I have an added an integration test as well that creates out an Append only Paimon table from MySQL. In order to keep it consistent with the mysql-cdc-source connector, I have used the same property name here.