Skip to content

fix(clickhouse): write values too long for the URL into the statement - #3781

Merged
datlechin merged 2 commits into
mainfrom
fix/3777-clickhouse-long-values
Oct 22, 2026
Merged

datlechin merged 2 commits into
mainfrom
fix/3777-clickhouse-long-values

Conversation

@datlechin

@datlechin datlechin commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

Summary

  • ClickHouse bound values stay param_ URL items while each is at most 131,072 bytes percent-encoded and the items fit 1 MiB less 4 KiB, the limits the server enforced. Past that, every value of the statement is written into it and travels in the POST body: a plain literal, as the server substitutes a String parameter, and CAST('…' AS String) inside INSERT data, where a plain literal goes to the column parser (a braced UUID was stored as another UUID).
  • A request body longer than the user's max_query_size, read from system.settings, carries max_query_size sized to it when the profile lets the user set it. Covers grid saves, imports and their key probes, object copy, MCP inserts, data sync, compare lookups and the iOS row editor.
  • Every URL item is percent-encoded outside RFC 3986 unreserved. URLComponents left + alone and the server decoded it as a space, so a+b was stored as a b, and a database named a+b was looked up as a b.
  • The ? scanner skips -- and /* */ comments and $$ / $tag$ bodies, as SQLParameterExtractor does, so an apostrophe or ? in a comment no longer breaks a parameterized query.
  • The import budget charges ClickHouse values as literals, like Databend.
  • escapeStringLiteral walks scalars and drops MySQL's \Z, which ClickHouse reads as a backslash and a Z.
  • Data sync writes ClickHouse text through the driver's escaper. The PluginKit default left backslashes single there, so C:\new arrived with a newline and a value ending in \' closed the literal.
  • PluginKit escapedParameterValue walks Unicode scalars. A quote with a combining mark after it was one Character no case matched, so it went out unescaped and ended the literal (data sync on every engine, the default binder Trino uses). Body change only, no ABI change.
  • apple/scripts/probes/check-clickhouse-long-values.sh checks the server facts this rests on.

Test plan

  • ClickHouseParameterBindingTests: long values written in, the URL budget in encoded bytes and item names, INSERT data cast and nowhere else, escapes, comments and dollar quotes.
  • ClickHouseRequestTests: max_query_size against the user's limit, readonly user and constraint max; a stubbed 200 KB INSERT and 300 KB UPDATE send no param_ item; + percent-encoded.
  • ClickHouseServerProfileTests, SQLWriteBatchBudgetTests, CompareSQLLiteralTextTests, EscapedParameterValueTests, SqlStandardEscapeTests.
  • clickhouse-server 25.8.33.6: the probe passes; the issue's 1,000 rows of 1,100 characters (1.1 MB body) and a 2 MB value insert; a 300 KB ALTER TABLE … UPDATE fails at 262,144 and runs with max_query_size; plain literals (WHERE, UPDATE) and casts (INSERT, one and two rows) match String parameters on 24 column types; column type corpus clickhouse holds.
  • verify: build, test (ClickHouse suites, import, object copy, transfer, data sync), package TableProCore, plugins, ios ClickHouseOnIOSTests PluginBackedDriverTests RowWriteRoutingTests, abi, lint.
  • Docs: docs(databases): how ClickHouse sends long values docs#120, to merge after this one.

Fixes #3777

Values stay param_ URL items within the server's 128 KiB field and 1 MiB URL limits, every URL item is percent-encoded (a + arrived as a space), and a body over max_query_size raises it for that request.
@datlechin
datlechin enabled auto-merge (squash) October 22, 2026 11:22
@datlechin datlechin added backport release/0.83 Merged into main; pick onto release/0.83 backport release/0.82 Merged into main; pick onto release/0.82 labels Oct 22, 2026
@datlechin
datlechin merged commit 18bfd4c into main Oct 22, 2026
9 checks passed
@datlechin
datlechin deleted the fix/3777-clickhouse-long-values branch October 22, 2026 15:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport release/0.82 Merged into main; pick onto release/0.82 backport release/0.83 Merged into main; pick onto release/0.83

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ClickHouse: text values over 128 KiB, or batches over 1 MiB of URL, fail to write

1 participant