Export partition - allow non matching partition expressions in case we can prove the destination expression does not split the data by arthurpassos · Pull Request #2074 · Altinity/ClickHouse

arthurpassos · 2026-07-17T20:39:56Z

Changelog category (leave one):

Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Allow export partition through different partition expressions as long as the destination expression does not repartition the data. This is validated at schedule time through two mechanisms:

Structural match: in case the source and destination are identical, the destination expression is a subset of the source expression or the destination expression can be entirely computed using only constants and the exact values guaranteed (pinned) by the source.
Dynamic proof: the destination expression is monotonic over the source partition min/max range.

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

Regression jobs to run:

github-actions · 2026-07-17T20:41:02Z

Workflow [PR], commit [9a1cb91]

arthurpassos · 2026-07-18T17:35:56Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3f3af7c4fc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…n-monotonic-compat

ilejn · 2026-07-21T14:55:32Z

Consider mentioning 'export' in the title and/or description

arthurpassos · 2026-07-21T14:59:02Z

Consider mentioning 'export' in the title and/or description

Done, I'll soon add the description

…n-monotonic-compat

arthurpassos · 2026-07-24T13:04:32Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ca43cf1f6d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

arthurpassos · 2026-08-05T17:34:07Z

Hey @arthurpassos - separate from the positional/name column-matching issue. I want to confirm the intended behaviour here before I pin it in a test.

What happens

A destination with an extra column is rejected on column count:
CREATE TABLE src (id Int64, a Int32)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/shard0/src', '{replica}')
ORDER BY tuple() PARTITION BY a;

CREATE TABLE dst (id Int64, a Int32, b Int32)
ENGINE = S3(..., format='Parquet', partition_strategy='hive') PARTITION BY a;

INSERT INTO src VALUES (1, 42);

ALTER TABLE src EXPORT PARTITION ID '42' TO TABLE dst;
-- Code: 20. Number of columns doesn't match (source: 2 and result: 3).
That one seems right to me.

But giving the extra column a DEFAULT changes nothing:
CREATE TABLE dst (id Int64, a Int32, b Int32 DEFAULT 42)
ENGINE = S3(..., format='Parquet', partition_strategy='hive') PARTITION BY a;

ALTER TABLE src EXPORT PARTITION ID '42' TO TABLE dst;
-- Code: 20. Number of columns doesn't match (source: 2 and result: 3).
Same error, same counts - the DEFAULT is not taken into account.

What I'd expect

I'd expect the second case to succeed, with b filled from its default:
SELECT id, a, b FROM dst;
-- 1  42  42
which is what
INSERT INTO dst (id, a)
SELECT id, a
FROM src;
would produce.

A DEFAULT column is exactly the case where the destination can supply the value itself, so requiring the source to provide it seems off.

Is the current behaviour intended? There's no data-loss risk either way since it's a loud rejection - I just want the test to assert the intended behaviour rather than whichever one I guess at.

I think this deserves a separate issue

arthurpassos · 2026-08-05T17:34:58Z

I suggest adding setting to turn on/off this functionality.

Export partition is already experimental and back by a setting the user must opt in

arthurpassos · 2026-08-05T20:18:32Z

Can you check #2138 - this does seem like we should not be allowing the export here or have some other fix in this case.

it is not related to this PR, but I did push a fix: 3e60b54

arthurpassos · 2026-08-05T20:24:30Z

@Selfeer I would appreciate if you could "humanize" a bit more those AI reports. For example, the following issue is very hard to understand and doesn't explain what is happening: 1. 🔴 High — Argument order lost, monotonicity proof evaluates the wrong expression.

It says "anchor" pointing to some code location. Then it says the impact. Then the trigger. It is too much to read to understand the real problem. Brain energy required to process this is very high.

Instead, if you could understand the issue yourself first and then give me a few SQL instructions that repro the case and a human explanation of what's going on, that would be 100 times better.

Selfeer · 2026-08-06T10:38:21Z

@Selfeer I would appreciate if you could "humanize" a bit more those AI reports. For example, the following issue is very hard to understand and doesn't explain what is happening: 1. 🔴 High — Argument order lost, monotonicity proof evaluates the wrong expression.

It says "anchor" pointing to some code location. Then it says the impact. Then the trigger. It is too much to read to understand the real problem. Brain energy required to process this is very high.

Instead, if you could understand the issue yourself first and then give me a few SQL instructions that repro the case and a human explanation of what's going on, that would be 100 times better.

I agree. I'd much rather do it the way you described: perform my own investigation on the findings and raise issues as needed-and we actually do that. But even in that case, I would still have to post this exact message first.

The purpose of the audit review has always been to perform a quick review of the PR without running any tests first, share the findings with the developer, and let you decide whether they are actual issues. If they are, we then raise separate issues afterward.

I can update the skills we use for the audit review to make the output easier to read, but overall, the audit review has always been a separate part of the verification process, separate from our actual testing.

…n-monotonic-compat

arthurpassos · 2026-08-20T13:07:46Z

@k-morozov hi, I have made some refactorings, fixed conflicts and updated the docs. Could you please re-review it?

arthurpassos · 2026-08-20T13:09:20Z

@Selfeer regarding the 5 AI findings you posted

Should be fixed
Seems like AI gibberish. At the very least, not related to this pr
Should be fixed
Outdated, that code path does not exist anymore
Not related to this pr

k-morozov · 2026-08-21T08:26:11Z

+            "To allow its usage, enable the setting `allow_insert_into_iceberg`.");
+    }
+
+    ExportPartitionUtils::verifyExportSchemaCastable(


Calls ExportPartitionUtils::verifyExportSchemaCastable twice here and below.

Thanks, bad rebase

k-morozov · 2026-08-21T08:39:24Z

    )


-def test_export_part_unnamed_tuple_partition_key_owner_matching_named_destination_is_allowed(cluster):


What's the reason?

k-morozov · 2026-08-21T08:42:36Z

+    /// so the same expression over the two types can produce different partitions.
+    bool isSameTypeForPartitioning(const DataTypePtr & lhs, const DataTypePtr & rhs)
+    {
+        return lhs->getName() == rhs->getName();


What about unnamed Tuple, Array, etc?
Does it related with test test_export_part_unnamed_tuple_partition_key_owner_matching_named_destination_is_allowed that was removed?
up: maybe haveSameTupleElementLayout will be usefull.

Answering #2074 (comment) here as it relates to this question.

The main reason I removed that test, and yes, it is affected by this name check, is to draw a line of "good enough" and move on. This PR needs to get merged so that we fix the broader and more common use cases, we can worry about named tuples later.

k-morozov · 2026-08-21T08:54:44Z

+
+        /// A NULL value forms its own destination partition, so a Nullable column may split the source
+        /// partition; min/max cannot rule that out. Require a structural match for such columns.
+        if (isNullableOrLowCardinalityNullable(source_type))


Seems that this behauvor should be in documentation (and motivation maybe?) Also I need to find tests for this condition..

k-morozov · 2026-08-21T09:02:23Z

+
+        /// Compute the global min/max index of the parts
+        IMergeTreeDataPart::MinMaxIndex minmax;
+        for (const auto & part : parts)


As I can see, we prepare mimax every time even if we have an equal partition key. But we really need it only for verifyOutputMapsToSinglePartition.

Right, but verifyOutputMapsToSinglePartition is called within a for loop. What is your suggestion?

For example, create 2 loops:
1 for happy path.
2. for verifyOutputMapsToSinglePartition.

So we could create minmax between 1 and 2 steps.

Another way - to fill minmax within the current loop only once before verifyOutputMapsToSinglePartition.

Imo, that's somewhat of an unecessary "optimization" that will not give us any speed up with the cost of more code. I vote for keeping it as is unless you strongly object.

k-morozov · 2026-08-21T09:22:20Z

        if (parquet_row_group_size_bytes)
            json.set("parquet_row_group_size_bytes", *parquet_row_group_size_bytes);
+        if (iceberg_partition_timezone)
+            json.set("iceberg_partition_timezone", *iceberg_partition_timezone);


We try to set this setting for export. But if we have several columns with different timezones, we would set the value from the setting for all of them, right?

I think that's how this setting was designed, @ianton-ru can confirm. AFAIK, this setting was created due to a misconception on the Apache Iceberg specs (the spec itself is poorly written) when it was believed we could tweak the timezone of an apache iceberg column. When in reality, according to 3rd party docs, the types are always UTC.

arthurpassos added 2 commits July 17, 2026 16:59

vibe coded monotonicity check for mt -> iceberg exports

3f47047

add some more tests

343c2d9

make tests better

3f3af7c

chatgpt-codex-connector Bot reviewed Jul 18, 2026

View reviewed changes

Comment thread src/Storages/MergeTree/ExportPartitionUtils.cpp Outdated

Comment thread src/Storages/MergeTree/ExportPartitionUtils.cpp Outdated

arthurpassos added 3 commits July 20, 2026 09:48

more vibe coded changes

d385268

recompute dst key with correct set of parts, include tz

454e9fc

Merge branch 'antalya-26.3' into feature/antalya-26.3/export-partitio…

abcba15

…n-monotonic-compat

arthurpassos changed the title ~~Allow non matching partition expressions iff monotonicity is proved and data is not repartitioned~~ Export partition - allow non matching partition expressions iff monotonicity is proved and data is not repartitioned Jul 21, 2026

arthurpassos added 6 commits July 22, 2026 09:31

Merge branch 'antalya-26.3' into feature/antalya-26.3/export-partitio…

4e10ade

…n-monotonic-compat

some more vibe coded style changes

e4a69a0

add missing tests

c60c97d

code simplification

a9f51e1

some docs

12c5e5b

handle iceberg_timezone

ca43cf1

chatgpt-codex-connector Bot reviewed Jul 24, 2026

View reviewed changes

Comment thread src/Storages/MergeTree/ExportPartitionUtils.cpp

arthurpassos added 10 commits July 24, 2026 10:23

missing arg restore

56fde58

docs

ee1993a

opsy

41fbe8f

possible simplification

450eddc

some more simplifications

6df47c7

add some comments

1501cc7

simplifications and docs

a89d2f0

fix more vibe coded issues

776be0b

further simplifications

7495ab6

one more simplification

e75220d

arthurpassos added port-antalya PRs to be ported to all new Antalya releases antalya-26.3 labels Aug 5, 2026

function extraction

c529fbe

arthurpassos added 2 commits August 5, 2026 15:33

simplification

55304f7

piggy back some bug fixes

3e60b54

arthurpassos added 7 commits August 19, 2026 10:18

re-use existing code and make the structural match a bit more flexible

1822382

Merge branch 'antalya-26.3' into feature/antalya-26.3/export-partitio…

378d9ca

…n-monotonic-compat

one fix for sure

9a6a4db

fix lc(nullable)

834a50b

fix tests

0743a3d

fix a few tests

7caa8d4

rmv corner case

35605c1

arthurpassos changed the title ~~Export partition - allow non matching partition expressions iff monotonicity is proved and data is not repartitioned~~ Export partition - allow non matching partition expressions in case we can prove the destination expression does not split the data Aug 20, 2026

k-morozov reviewed Aug 21, 2026

View reviewed changes

Comment thread src/Storages/MergeTree/ExportPartitionUtils.cpp

rmv duplicated call

9a1cb91

k-morozov previously approved these changes Aug 21, 2026

View reviewed changes

arthurpassos dismissed k-morozov’s stale review via 9a1cb91 August 21, 2026 19:36

		)


		def test_export_part_unnamed_tuple_partition_key_owner_matching_named_destination_is_allowed(cluster):

Uh oh!

Conversation

arthurpassos commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

CI/CD Options

Exclude tests:

Regression jobs to run:

Uh oh!

github-actions Bot commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arthurpassos commented Jul 18, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

ilejn commented Jul 21, 2026

Uh oh!

arthurpassos commented Jul 21, 2026

Uh oh!

arthurpassos commented Jul 24, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

arthurpassos commented Aug 5, 2026

What happens

What I'd expect

Uh oh!

arthurpassos commented Aug 5, 2026

Uh oh!

arthurpassos commented Aug 5, 2026

Uh oh!

arthurpassos commented Aug 5, 2026

Uh oh!

Selfeer commented Aug 6, 2026

Uh oh!

arthurpassos commented Aug 20, 2026

Uh oh!

arthurpassos commented Aug 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

k-morozov Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

k-morozov Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

k-morozov Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

arthurpassos commented Jul 17, 2026 •

edited

Loading

github-actions Bot commented Jul 17, 2026 •

edited

Loading

k-morozov Aug 21, 2026 •

edited

Loading

k-morozov Aug 21, 2026 •

edited

Loading

k-morozov Aug 21, 2026 •

edited

Loading