[fix](fe) Fix timezone-sensitive MV rewrite and TIMESTAMPTZ MTMV partition issues #66795
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,133 @@ | ||
| // Licensed to the Apache Software Foundation (ASF) under one | ||
| // or more contributor license agreements. See the NOTICE file | ||
| // distributed with this work for additional information | ||
| // regarding copyright ownership. The ASF licenses this file | ||
| // to you under the Apache License, Version 2.0 (the | ||
| // "License"); you may not use this file except in compliance | ||
| // with the License. You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, | ||
| // software distributed under the License is distributed on an | ||
| // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| // KIND, either express or implied. See the License for the | ||
| // specific language governing permissions and limitations | ||
| // under the License. | ||
|
|
||
| import org.junit.Assert; | ||
|
|
||
| /** | ||
| * For a TIMESTAMPTZ(6) bounded-range partition table, the MTMV physical partition name auto-generated by | ||
| * Doris is longer than the internal threshold and used to be appended with System.currentTimeMillis(). | ||
| * That made the name generated when validating a partition refresh differ from the real stored name, so | ||
| * refreshing by the real partition name returned by SHOW PARTITIONS failed with "partition not exist". | ||
| * The generated name must be deterministic and the refresh validation must accept the real stored name. | ||
| */ | ||
| suite("test_timestamptz_mtmv_partition_refresh_name","mtmv") { | ||
| def dbName = "timestamptz_mtmv_partition_refresh_name" | ||
| def tableName = "timestamptz_mtmv_partition_refresh_name_table" | ||
| def mvName = "timestamptz_mtmv_partition_refresh_name_mv" | ||
|
|
||
| sql "DROP DATABASE IF EXISTS ${dbName}" | ||
| sql "CREATE DATABASE ${dbName}" | ||
| sql "USE ${dbName}" | ||
|
|
||
| sql "SET enable_nereids_planner = true" | ||
| sql "SET enable_fallback_to_original_planner = false" | ||
| sql "SET time_zone = '+00:00'" | ||
|
|
||
| // TIMESTAMPTZ(6) with two bounded range partitions: the generated MTMV partition name is long. | ||
| sql "DROP TABLE IF EXISTS ${tableName}" | ||
| sql """ | ||
| CREATE TABLE ${tableName} ( | ||
| id INT, | ||
| ts TIMESTAMPTZ(6) NOT NULL | ||
| ) | ||
| DUPLICATE KEY(id) | ||
| PARTITION BY RANGE(ts) ( | ||
| PARTITION p1 VALUES [('2024-01-01 00:00:00+00:00'), ('2024-01-02 00:00:00+00:00')), | ||
| PARTITION p2 VALUES [('2024-01-02 00:00:00+00:00'), ('2024-01-03 00:00:00+00:00')) | ||
| ) | ||
| DISTRIBUTED BY HASH(id) BUCKETS 1 | ||
| PROPERTIES('replication_num' = '1') | ||
| """ | ||
| sql """ | ||
| INSERT INTO ${tableName} VALUES | ||
| (1, '2024-01-01 00:30:00+00:00'), | ||
| (2, '2024-01-02 00:30:00+00:00') | ||
| """ | ||
| sql "sync" | ||
|
|
||
| sql "DROP MATERIALIZED VIEW IF EXISTS ${mvName}" | ||
| sql """ | ||
| CREATE MATERIALIZED VIEW ${mvName} | ||
| BUILD DEFERRED REFRESH AUTO ON MANUAL | ||
| PARTITION BY(ts) | ||
| DISTRIBUTED BY RANDOM BUCKETS 1 | ||
| PROPERTIES('replication_num' = '1') | ||
| AS SELECT id, ts FROM ${tableName} | ||
| """ | ||
|
|
||
| // The real physical partition names returned by SHOW PARTITIONS. | ||
| def partitions = sql "SHOW PARTITIONS FROM ${mvName}" | ||
| Assert.assertTrue("expected at least one partition, got " + partitions.size(), partitions.size() >= 1) | ||
| def realName = partitions[0][1].toString() | ||
| Assert.assertFalse(realName.isEmpty()) | ||
|
|
||
| // Refreshing by the real partition name must be accepted (previously "partition not exist"). | ||
| sql "REFRESH MATERIALIZED VIEW ${mvName} PARTITIONS(`" + realName + "`)" | ||
| waitingMTMVTaskFinishedByMvName(mvName, dbName) | ||
|
|
||
| // Only the refreshed partition carries data (BUILD DEFERRED + partial refresh). | ||
| def mvRes = sql "SELECT id, CAST(ts AS STRING) FROM ${mvName} ORDER BY id" | ||
| Assert.assertEquals(1, mvRes.size()) | ||
| Assert.assertTrue(mvRes[0][1].toString().contains("2024-01-01 00:30:00.000000+00:00")) | ||
|
|
||
| // AUTO refresh fills the remaining partition(s). | ||
| sql "REFRESH MATERIALIZED VIEW ${mvName} AUTO" | ||
| waitingMTMVTaskFinishedByMvName(mvName, dbName) | ||
| def mvResAuto = sql "SELECT id, CAST(ts AS STRING) FROM ${mvName} ORDER BY id" | ||
| Assert.assertEquals(2, mvResAuto.size()) | ||
| Assert.assertTrue(mvResAuto[0][1].toString().contains("2024-01-01 00:30:00.000000+00:00")) | ||
| Assert.assertTrue(mvResAuto[1][1].toString().contains("2024-01-02 00:30:00.000000+00:00")) | ||
|
|
||
| // DATETIME control group: refresh by real partition name also works. | ||
| def dtTable = "timestamptz_mtmv_partition_refresh_name_dt_table" | ||
| def dtMv = "timestamptz_mtmv_partition_refresh_name_dt_mv" | ||
| sql "DROP TABLE IF EXISTS ${dtTable}" | ||
| sql """ | ||
| CREATE TABLE ${dtTable} ( | ||
| id INT, | ||
| ts DATETIME(6) NOT NULL | ||
| ) | ||
| DUPLICATE KEY(id) | ||
| PARTITION BY RANGE(ts) ( | ||
| PARTITION p1 VALUES [('2024-01-01 00:00:00'), ('2024-01-02 00:00:00')), | ||
| PARTITION p2 VALUES [('2024-01-02 00:00:00'), ('2024-01-03 00:00:00')) | ||
| ) | ||
| DISTRIBUTED BY HASH(id) BUCKETS 1 | ||
| PROPERTIES('replication_num' = '1') | ||
| """ | ||
| sql """ | ||
| INSERT INTO ${dtTable} VALUES | ||
| (1, '2024-01-01 00:30:00'), | ||
| (2, '2024-01-02 00:30:00') | ||
| """ | ||
| sql "sync" | ||
| sql "DROP MATERIALIZED VIEW IF EXISTS ${dtMv}" | ||
| sql """ | ||
| CREATE MATERIALIZED VIEW ${dtMv} | ||
| BUILD DEFERRED REFRESH AUTO ON MANUAL | ||
| PARTITION BY(ts) | ||
| DISTRIBUTED BY RANDOM BUCKETS 1 | ||
| PROPERTIES('replication_num' = '1') | ||
| AS SELECT id, ts FROM ${dtTable} | ||
| """ | ||
| def dtPartitions = sql "SHOW PARTITIONS FROM ${dtMv}" | ||
| def dtRealName = dtPartitions[0][1].toString() | ||
| sql "REFRESH MATERIALIZED VIEW ${dtMv} PARTITIONS(`" + dtRealName + "`)" | ||
| waitingMTMVTaskFinishedByMvName(dtMv, dbName) | ||
| def dtRes = sql "SELECT COUNT(*) FROM ${dtMv}" | ||
| Assert.assertEquals(1, dtRes[0][0]) | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[P1] Make the deterministic partition identity collision-safe
The suffix has only 32 bits. Two valid long values with the same first 30 sanitized characters and the standard Java
Aa/BBhash collision generate the same physical name. Initial MTMV creation then fails with a duplicate partition name; if the second base partition appears later, theIF NOT EXISTSadd silently no-ops and leaves it permanently unrepresented. Please use a collision-resistant identity and explicitly reject a same-name/different-description add.