Skip to content

[UUID 3/8] UUID result rendering (DataSchema, Arrow/JSON encoders) - #18871

Merged
xiangfu0 merged 1 commit into
apache:masterfrom
xiangfu0:uuid-split/03-result-rendering
Jul 26, 2026
Merged

[UUID 3/8] UUID result rendering (DataSchema, Arrow/JSON encoders)#18871
xiangfu0 merged 1 commit into
apache:masterfrom
xiangfu0:uuid-split/03-result-rendering

Conversation

@xiangfu0

@xiangfu0 xiangfu0 commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Parent tracking issue: #16619

What

Renders UUID result columns as canonical lowercase RFC-4122 strings.

Changes

  • DataSchema.ColumnDataType.UUID / UUID_ARRAY + conversion and format helpers
  • Arrow and JSON broker response encoders
  • UUID null placeholder: ColumnDataType.UUID overrides getNullPlaceholder() to return the nil UUID, matching the default null sentinel FieldSpec already uses for UUID columns. Its stored type BYTES supplies a zero-length placeholder, which is not a valid 16-byte UUID and fails to render, so the three call sites that build null-aware blocks (DataBlockBuilder, GroupByResultsBlock, GroupByDataTableReducer) now resolve the placeholder on the logical type rather than the stored type. For every other logical type the two are identical — an invariant pinned by a new DataSchemaTest case.
  • DataSchema.fromBytes reports an unrecognized ColumnDataType token with a message naming the mixed-version cause instead of a bare IllegalArgumentException.

Enables

SELECT, GROUP BY, DISTINCT, and join outputs render UUID strings instead of hex.

⚠️ Rolling-upgrade note: DataSchema serializes ColumnDataType by name; an old broker/server that receives a UUID column token from a new node will fail to parse it. Treat as an atomic-upgrade feature — UUID columns should only be queried once the whole cluster is upgraded. No existing (non-UUID) column is affected.

About this PR / how to review

This is part 3 of 8 splitting #18140 (first-class logical UUID type) into layered PRs, as requested there.

The split is enabled by the v1 design: DataType.UUID has stored type BYTES, so most paths handle it automatically; each PR adds explicit UUID semantics to one subsystem. Head branch lives on xiangfu0/pinot.

Both parents (#18869, #18870) have merged, so this PR is now rebased directly on master and the Files-changed tab shows only this layer.

Full stack (merge bottom → top)

  1. [UUID 1/8] Add logical UUID type foundation (pinot-spi) #18869 — [UUID 1/8] logical UUID type foundation (pinot-spi) ✅ merged
  2. [UUID 2/8] UUID ingest and segment storage #18870 — [UUID 2/8] UUID ingest and segment storage ✅ merged
  3. [UUID 3/8] UUID result rendering (DataSchema, Arrow/JSON encoders) #18871 — [UUID 3/8] UUID result rendering (DataSchema, Arrow/JSON encoders) ← this PR
  4. [UUID 4/8] Server-side predicate evaluation for the logical UUID type #18872 — [UUID 4/8] UUID server-side predicates, CAST and transforms
  5. [UUID 5/8] UUID aggregation, group-by and distinct #18873 — [UUID 5/8] UUID aggregation, group-by and distinct
  6. [UUID 6/8] UUID multi-stage engine (planner + runtime) #18874 — [UUID 6/8] UUID multi-stage engine (planner + runtime)
  7. [UUID 7/8] UUID partitioning #18875 — [UUID 7/8] UUID scalar UDFs and partitioning
  8. [UUID 8/8] UUID integration tests, benchmarks and docs #18876 — [UUID 8/8] UUID integration tests, benchmarks and docs

PRs 4 and 7 (#18872, #18875) depend only on the merged foundation and can be reviewed in parallel; PRs 5/6 (#18873, #18874) also need this one; PR 8 (#18876) needs #18874.

Full feature description, v1 design contract, scope exclusions, and benchmark numbers: #18140.

@codecov-commenter

codecov-commenter commented Jun 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 61.11111% with 35 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.51%. Comparing base (ab3fd2b) to head (69c8a42).

Files with missing lines Patch % Lines
...java/org/apache/pinot/common/utils/DataSchema.java 51.42% 28 Missing and 6 partials ⚠️
...not/core/query/reduce/GroupByDataTableReducer.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #18871      +/-   ##
============================================
+ Coverage     65.49%   65.51%   +0.02%     
  Complexity     1421     1421              
============================================
  Files          3428     3428              
  Lines        217518   217587      +69     
  Branches      34558    34570      +12     
============================================
+ Hits         142453   142544      +91     
+ Misses        63527    63508      -19     
+ Partials      11538    11535       -3     
Flag Coverage Δ
custom-integration1 ?
integration 100.00% <ø> (ø)
integration1 100.00% <ø> (ø)
java-25 65.51% <61.11%> (+0.02%) ⬆️
temurin 65.51% <61.11%> (+0.02%) ⬆️
unittests 65.50% <61.11%> (+0.02%) ⬆️
unittests1 56.80% <61.11%> (+0.04%) ⬆️
unittests2 37.93% <2.22%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@xiangfu0
xiangfu0 force-pushed the uuid-split/03-result-rendering branch 7 times, most recently from 10b0acb to 16d0757 Compare July 7, 2026 07:08
@xiangfu0
xiangfu0 force-pushed the uuid-split/03-result-rendering branch 6 times, most recently from 9667e39 to b40ea56 Compare July 8, 2026 08:05
Comment thread pinot-common/src/main/java/org/apache/pinot/common/utils/DataSchema.java Outdated
@xiangfu0
xiangfu0 force-pushed the uuid-split/03-result-rendering branch 4 times, most recently from 09bec51 to 04a5acb Compare July 13, 2026 08:02
@xiangfu0
xiangfu0 force-pushed the uuid-split/03-result-rendering branch 10 times, most recently from aa6a09d to b3fbfa1 Compare July 21, 2026 08:10
@xiangfu0
xiangfu0 force-pushed the uuid-split/03-result-rendering branch 4 times, most recently from e976bdf to faa0149 Compare July 25, 2026 08:01
@xiangfu0
xiangfu0 requested a review from Jackie-Jiang July 25, 2026 19:31
Part 3/8 of splitting apache#18140 (logical UUID type). Rebased onto
master, which now includes the merged apache#18869 type foundation and apache#18870
ingest/storage layer.

Renders UUID result columns as canonical lowercase RFC-4122 strings:
- DataSchema.ColumnDataType.UUID / UUID_ARRAY plus the conversion and format
  helpers, backed by the UuidUtils class merged in apache#18869.
- Arrow and JSON broker response encoders.

UUID null placeholder: ColumnDataType.UUID overrides getNullPlaceholder() to
return the nil UUID, matching the default null sentinel FieldSpec already uses
for UUID columns. Its stored type BYTES supplies a zero-length placeholder,
which is not a valid 16-byte UUID and fails to render, so the three call sites
that build null-aware blocks (DataBlockBuilder, GroupByResultsBlock,
GroupByDataTableReducer) now resolve the placeholder on the logical type rather
than the stored type. For every other logical type the two are identical, an
invariant pinned by a new DataSchemaTest case.

DataSchema.fromBytes now reports an unrecognized ColumnDataType token with a
message naming the mixed-version cause instead of a bare IllegalArgumentException.

Rolling-upgrade note: DataSchema serializes ColumnDataType by name, so an old
broker/server that receives a UUID token from a new node cannot parse it. Treat
as an atomic-upgrade feature - UUID columns should only be queried once the
whole cluster is upgraded. No existing (non-UUID) column is affected.
@xiangfu0
xiangfu0 force-pushed the uuid-split/03-result-rendering branch from faa0149 to 69c8a42 Compare July 26, 2026 08:02
@xiangfu0
xiangfu0 merged commit 2050fce into apache:master Jul 26, 2026
11 checks passed
@xiangfu0
xiangfu0 deleted the uuid-split/03-result-rendering branch July 26, 2026 20:29
@xiangfu0

Copy link
Copy Markdown
Contributor Author

Documentation follow-up: pinot-contrib/pinot-docs#940

xiangfu0 added a commit to pinot-contrib/pinot-docs that referenced this pull request Jul 26, 2026
xiangfu0 added a commit to xiangfu0/pinot that referenced this pull request Jul 28, 2026
Adds the UUID scalar function surface on top of the logical UUID type:

- IS_UUID / TO_UUID: polymorphic over STRING and BYTES, sharing dispatch via
  AbstractStringOrBytesUuidFunction
- UUID_TO_BYTES, BYTES_TO_UUID, UUID_TO_STRING, UUID_VERSION, UUID_TIMESTAMP
- UUID_V4 / UUID_V7 generators, marked isDeterministic = false so the broker's
  CompileTimeFunctionsInvoker does not fold one value across every row
- Matching multi-stage Udf wrappers so the functions are registered with the
  MSE UDF registry

Depends only on UuidUtils (apache#18869) and ColumnDataType.UUID (apache#18871), both
already on master. Split out of apache#18875 so it can be reviewed independently of
the UUID partitioning work.
xiangfu0 added a commit to xiangfu0/pinot that referenced this pull request Jul 28, 2026
Adds the UUID scalar function surface on top of the logical UUID type:

- IS_UUID / TO_UUID: polymorphic over STRING and BYTES, sharing dispatch via
  AbstractStringOrBytesUuidFunction
- UUID_TO_BYTES, BYTES_TO_UUID, UUID_TO_STRING, UUID_VERSION, UUID_TIMESTAMP
- UUID_V4 / UUID_V7 generators, marked isDeterministic = false so the broker's
  CompileTimeFunctionsInvoker does not fold one value across every row
- Matching multi-stage Udf wrappers so the functions are registered with the
  MSE UDF registry

Depends only on UuidUtils (apache#18869) and ColumnDataType.UUID (apache#18871), both
already on master. Split out of apache#18875 so it can be reviewed independently of
the UUID partitioning work.
@Jackie-Jiang Jackie-Jiang added the feature New functionality label Jul 28, 2026
xiangfu0 added a commit to xiangfu0/pinot that referenced this pull request Jul 29, 2026
Adds the UUID scalar function surface on top of the logical UUID type:

- IS_UUID / TO_UUID: polymorphic over STRING and BYTES, sharing dispatch via
  AbstractStringOrBytesUuidFunction
- UUID_TO_BYTES, BYTES_TO_UUID, UUID_TO_STRING, UUID_VERSION, UUID_TIMESTAMP
- UUID_V4 / UUID_V7 generators, marked isDeterministic = false so the broker's
  CompileTimeFunctionsInvoker does not fold one value across every row
- Matching multi-stage Udf wrappers so the functions are registered with the
  MSE UDF registry

Depends only on UuidUtils (apache#18869) and ColumnDataType.UUID (apache#18871), both
already on master. Split out of apache#18875 so it can be reviewed independently of
the UUID partitioning work.
xiangfu0 added a commit to xiangfu0/pinot that referenced this pull request Jul 30, 2026
Adds the UUID scalar function surface on top of the logical UUID type:

- IS_UUID / TO_UUID: polymorphic over STRING and BYTES, sharing dispatch via
  AbstractStringOrBytesUuidFunction
- UUID_TO_BYTES, BYTES_TO_UUID, UUID_TO_STRING, UUID_VERSION, UUID_TIMESTAMP
- UUID_V4 / UUID_V7 generators, marked isDeterministic = false so the broker's
  CompileTimeFunctionsInvoker does not fold one value across every row
- Matching multi-stage Udf wrappers so the functions are registered with the
  MSE UDF registry

Depends only on UuidUtils (apache#18869) and ColumnDataType.UUID (apache#18871), both
already on master. Split out of apache#18875 so it can be reviewed independently of
the UUID partitioning work.
xiangfu0 added a commit that referenced this pull request Jul 30, 2026
* Add UUID scalar functions and multi-stage UDF wrappers

Adds the UUID scalar function surface on top of the logical UUID type:

- IS_UUID / TO_UUID: polymorphic over STRING and BYTES, sharing dispatch via
  AbstractStringOrBytesUuidFunction
- UUID_TO_BYTES, BYTES_TO_UUID, UUID_TO_STRING, UUID_VERSION, UUID_TIMESTAMP
- UUID_V4 / UUID_V7 generators, marked isDeterministic = false so the broker's
  CompileTimeFunctionsInvoker does not fold one value across every row
- Matching multi-stage Udf wrappers so the functions are registered with the
  MSE UDF registry

Depends only on UuidUtils (#18869) and ColumnDataType.UUID (#18871), both
already on master. Split out of #18875 so it can be reviewed independently of
the UUID partitioning work.

* Fix UUID-to-string scalar evaluation

* Support polymorphic UUID scalar inputs

* Harden polymorphic UUID scalar support
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New functionality

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants