[UUID 8/8] UUID integration tests, benchmarks and docs - #18876
Open
xiangfu0 wants to merge 6 commits into
Open
Conversation
This was referenced Jun 29, 2026
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #18876 +/- ##
============================================
- Coverage 66.65% 66.61% -0.05%
Complexity 1423 1423
============================================
Files 3443 3447 +4
Lines 218632 218995 +363
Branches 34793 34888 +95
============================================
+ Hits 145726 145874 +148
- Misses 61192 61382 +190
- Partials 11714 11739 +25
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
xiangfu0
force-pushed
the
uuid-split/08-it-bench-docs
branch
from
June 29, 2026 12:05
93272e3 to
4856ac5
Compare
xiangfu0
commented
Jun 29, 2026
xiangfu0
left a comment
xiangfu0
left a comment
Contributor
Author
There was a problem hiding this comment.
Found critical mixed-version compatibility issues inherited in this stack head; see inline comments.
| // does not contain this enum constant. This is safe because DataSchema serialization uses enum names (not ordinals) | ||
| // via ColumnDataType.name() / ColumnDataType.valueOf(). If ordinal-based serialization is ever added for | ||
| // ColumnDataType, UUID must be moved to the end of the enum (as was done for FieldSpec.DataType.UUID). | ||
| UUID(BYTES, null) { |
Contributor
Author
There was a problem hiding this comment.
This makes UUID a new broker/server wire-visible ColumnDataType. DataSchema.toBytes() emits the enum name, and older peers still throw in parseColumnDataType(...) once they see UUID, so rolling upgrades and rollback are unsafe as soon as UUID-typed results are in flight. Please keep the wire representation on an existing type until all peers are upgraded, or add an explicit mixed-version compatibility path plus coverage.
| // does not know UUID = 22 / UUID_ARRAY = 23 will fail planning with UnknownEnumValueException when receiving a plan | ||
| // that includes a UUID literal. Avoid issuing UUID queries until all brokers and servers are upgraded. See the | ||
| // matching note on DataSchema.toBytes and ProtoExpressionToRexExpression#convertColumnDataType. | ||
| UUID = 22; |
Contributor
Author
There was a problem hiding this comment.
This introduces new UUID / UUID_ARRAY proto enum values with no compatibility path for older MSQ peers. Older brokers/servers decode them as UNRECOGNIZED and throw in convertColumnDataType(...), so a UUID literal can break mixed-version planning before execution even starts. Please encode UUID literals using an existing wire type until the cluster is homogeneous, or add version-gated dual-read/dual-write behavior with mixed-version tests.
xiangfu0
force-pushed
the
uuid-split/08-it-bench-docs
branch
7 times, most recently
from
July 5, 2026 19:24
1eb0c13 to
ce33dae
Compare
xiangfu0
force-pushed
the
uuid-split/08-it-bench-docs
branch
12 times, most recently
from
July 13, 2026 08:02
5284a05 to
e4a51de
Compare
xiangfu0
force-pushed
the
uuid-split/08-it-bench-docs
branch
10 times, most recently
from
July 20, 2026 08:08
0f57332 to
b977430
Compare
xiangfu0
force-pushed
the
uuid-split/08-it-bench-docs
branch
4 times, most recently
from
July 28, 2026 08:02
6dcfae2 to
2d38a93
Compare
28 tasks
xiangfu0
force-pushed
the
uuid-split/08-it-bench-docs
branch
6 times, most recently
from
August 7, 2026 09:28
f8ee606 to
51eb86e
Compare
Adds UUID handling to the predicate evaluators, so =, !=, IN, NOT IN and range predicates work against a UUID column on both the raw and the dictionary path. UUID follows the pattern TIMESTAMP already uses: a logical type whose stored type does the work. The literal is parsed to its 16-byte stored form once, when the evaluator is built, and from there the existing BYTES evaluators apply -- no per-value conversion in the scan loop. The dictionary path needs no UUID branch: Dictionary#getStoredValue returns hex for a UUID column and indexOf(String) hex-decodes, so the existing String-keyed lookup is already correct. PredicateUtils renders the literal to that hex form for those String-typed lookup APIs. Split into apache#19181 (CAST), apache#19182 (bloom filter pruning) and apache#19183 (transform functions); this PR is now just the predicate evaluators.
The UUID branch accepted several input forms via UuidUtils.toBytes(Object), which was inconsistent with every other branch -- they each cast to one type. The input is in fact deterministic. GroupByDataTableReducer runs every column through ColumnDataType#convert immediately before calling isMatch, and convert returns UuidUtils.toUUID(...) for UUID, so the value is always a java.util.UUID. The other reduce path cannot deliver a UUID here at all: getConvertedKey has no UUID case and throws for it. Adds testHavingFilterOnUuidColumn, which pins the contract -- this line was previously untested in either direction. Verified it reaches the UUID branch rather than passing vacuously.
Part 5/8 of splitting apache#18140 (logical UUID type). Rebased onto latest master; stacked on uuid-split/04-sse-predicates-cast. Downstream references use the UuidKey class merged in apache#18869.
Part 6/8 of splitting apache#18140 (logical UUID type). Rebased onto latest master; stacked on uuid-split/05-agg-groupby-distinct. Downstream references use the UuidKey class merged in apache#18869.
Part 7/8 of splitting apache#18140 (logical UUID type). - UuidPartitionFunction: hashes the 16-byte UUID form via Murmur2, matching what an external producer keyed on raw UUID bytes computes - PartitionerFactory / TableConfigPartitioner: thread the column's logical DataType through so UUID columns render canonically instead of as bare hex - UUID_ARRAY entries for the array scalar functions The UUID scalar functions and multi-stage UDF wrappers that were previously part of this layer now live in their own PR (apache#19091) so they can be reviewed and merged in parallel.
Part 8/8 of splitting apache#18140 (logical UUID type). Rebased onto latest master; stacked on uuid-split/07-udfs-partitioning. Downstream references use the UuidKey class merged in apache#18869.
xiangfu0
force-pushed
the
uuid-split/08-it-bench-docs
branch
from
August 8, 2026 09:36
51eb86e to
e848ec1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Parent tracking issue: #16619
What
End-to-end verification and documentation.
Changes
UuidTypeTest,UuidTypeRealtimeTest,UuidUpsertRealtimeTest(extendCustomDataQueryClusterIntegrationTest)BenchmarkUuidGroupingAndLookup,BenchmarkUuidQueryExecutionREADMEusage / limitations / migrationEnables
Offline + realtime + upsert integration coverage and performance benchmarks. The top of the stack — its cumulative diff equals the original #18140.
Depends on
#18874 (benchmarks import multi-stage runtime classes). #18869 and #18871 have already merged.
About this PR / how to review
This is part 8 of 8 splitting #18140 (first-class logical
UUIDtype) into layered PRs, as requested there.The split is enabled by the v1 design:
DataType.UUIDhas stored typeBYTES, so most paths handle it automatically; each PR adds explicit UUID semantics to one subsystem. Head branch lives onxiangfu0/pinot.This PR is stacked on #18875 (branch
uuid-split/07-udfs-partitioning). Because GitHub PRs to apache must base onmaster, the Files-changed tab is cumulative — it now includes layers 4–8 (layers 1–3 have merged) until the remaining PRs below it merge. Review the commit titled[UUID 8/8] UUID integration tests, benchmarks and docs— that is this layer's change. Each parent merge shrinks this diff after a rebase.Full stack (merge bottom → top)
[UUID 1/8] Add logical UUID type foundation (pinot-spi) #18869 — [UUID 1/8] logical UUID type foundation (pinot-spi)merged[UUID 2/8] UUID ingest and segment storage #18870 — [UUID 2/8] UUID ingest and segment storagemerged[UUID 3/8] UUID result rendering (DataSchema, Arrow/JSON encoders) #18871 — [UUID 3/8] UUID result rendering (DataSchema, Arrow/JSON encoders)mergedWith #18869 and #18871 landed, PRs 4/7 (#18872, #18875) and 5/6 (#18873, #18874) can be reviewed in parallel; PR 8 (#18876) needs #18874.
Full feature description, v1 design contract, scope exclusions, and benchmark numbers: #18140.