Skip to content

Create Nested(...) columns as Array(Tuple(...)) - #575

Open
Hakaou wants to merge 1 commit into
ClickHouse:masterfrom
Hakaou:nested-as-array-of-tuple
Open

Hakaou wants to merge 1 commit into
ClickHouse:masterfrom
Hakaou:nested-as-array-of-tuple

Conversation

@Hakaou

@Hakaou Hakaou commented Oct 17, 2026 •

Copy link
Copy Markdown

Closes #571.

TypeParser had no entry for Nested, so the name became a Void terminal. At the root ValidateAST rejects it ("unsupported column type"), but inside a container nothing checks it: Array(Nested(key String, value String)) was created as Array(Void), ColumnNothing reads one byte per row and the block stream desynchronises.

Change

  • TypeParser recognises Nested as its own AST node, next to Map and SimpleAggregateFunction (TypeAst::Meta gains a value; the AST keeps the server's type name).
  • CreateColumnFromAst builds an Array of the named Tuple for it, the same way the Map case builds its Array(Tuple). The Tuple case body moves to CreateTupleColumn so both cases share it; a bare Nested without fields gives nullptr like any unsupported type.
  • This mirrors the server, where DataTypeNested.cpp builds DataTypeArray(DataTypeTuple(types, names)) under a custom name.

Trade-off, named: the created column reports the structural type name, Array(Tuple(key String, value UInt8)), not Nested(...). The server accepts that name on INSERT, named or unnamed tuple alike (checked with clickhouse local 26.11 and covered by ArrayOfNested_InsertFromClient). A dedicated Nested column type would add public API for no behavioural gain, so I did not go that way; happy to discuss.

Tests

  • type_parser_ut: Nested(...) parses to a Nested node with named elements, inside Array, with a backtick-quoted field name, and with a field itself named Nested.
  • CreateColumnByType_ut: Nested, Array(Nested), Map(String, Nested), Nested(a Nested(...)) give the expected column types; a bare Nested gives nullptr.
  • client_ut (both fixtures, no compression and LZ4): select from a server-typed Array(Nested(...)) column with empty inner and outer arrays; insert from the client, rendered back by the server with toString; a top-level Nested column with flatten_nested = 0.

Before this change, on master fc5d98f against ClickHouse 26.11.1:

Case Before After
CreateColumnByType("Array(Nested(key String, value UInt8))") Array(Void) Array(Array(Tuple(key String, value UInt8)))
Select on that column, no compression Can't cast from Void 3 rows, values match
Select on that column, LZ4 some data was not read 3 rows, values match
Select on a Nested column, flatten_nested = 0 unsupported column type: Nested(key String, value UInt8) 2 rows, values match
Insert from the client into Array(Nested(...)) works works (kept as a guard)

Full suite: 1 138 tests pass locally (macOS arm64, Apple clang 21, server 26.11.1 master build). Parser, factory, client and roundtrip suites also pass under AddressSanitizer and UBSan.

Out of scope, two notes

  • ValidateAST only checks the root, so any other unknown type nested in a container still decodes as Void silently. SELECT [] relies on that path today: Nothing is not registered and decodes as Void by accident, one byte per row on both sides. Would you take a follow-up that registers Nothing and validates recursively, so an unknown type fails loudly instead of desynchronising the stream?
  • UBSan reports three out-of-bounds reads in the existing InsertData test (client_ut.cpp 625 to 627) when the server returns the eight rows in one block: the expectation indexes TEST_DATA past its four rows. Test code only, fixed separately in Compare InsertData rows against both datasets regardless of block boundaries #576.

The type parser had no entry for Nested, so a column typed
Array(Nested(key String, value String)) was created as Array(Void)
and the native block stream fell out of sync ("Can't cast from Void",
"some data was not read"). A top-level Nested column, which the server
sends with flatten_nested = 0, was rejected as unsupported.

The server defines Nested as a custom name over Array(Tuple(...))
(DataTypeNested.cpp). The parser now recognises Nested as its own
node, like Map and SimpleAggregateFunction, and the column factory
builds an Array of the named Tuple from it, sharing the Tuple code.

Closes ClickHouse#571
} else {
return nullptr;
}
case TypeAst::Nested: {

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same shape as the Map case just below: the surface type becomes its Array of Tuple here, so the parser and the AST stay faithful to the type name the server sends. The Tuple body moved to CreateTupleColumn unchanged so both cases share it.

@CLAassistant

CLAassistant commented Oct 17, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Array(Nested(...)) columns silently decode as Array(Nothing) and desynchronize the native block stream

2 participants