Skip to content

[phase-31 1/4] Sort schema foundation#6242

Merged
g-talbot merged 16 commits intomainfrom
gtt/phase-31-sort-schema
Apr 8, 2026
Merged

[phase-31 1/4] Sort schema foundation#6242
g-talbot merged 16 commits intomainfrom
gtt/phase-31-sort-schema

Conversation

@g-talbot
Copy link
Copy Markdown
Contributor

Summary

Sort schema types, parsing, and time-window arithmetic for metrics compaction (Phase 31 Metadata Foundation, PR 1 of 4).

Stacks on matthew.kim/metrics-wide-schema.

What's included

quickwit-proto:

  • Vendored event_store_sortschema.proto (SortSchema, SortColumn, RowKeys, ColumnValue types)
  • SortFieldsError error type for parser/validation errors
  • From<SortFieldsError> impl for MetastoreError

quickwit-parquet-engine:

  • sort_fields/ module:
    • column_type.rs — ColumnTypeId enum matching Go iota values, suffix/name resolution
    • parser.rs — Direct port of Go StringToSchema with V2-only enforcement
    • display.rsschema_to_string / schema_to_string_short (port of Go SchemaToString)
    • equivalence.rs — Schema comparison for compaction (port of Go EquivalentSchemas)
    • validation.rs — Schema validation rules (port of Go ValidateSchema)
    • window.rswindow_start() with rem_euclid for negative timestamp correctness, validate_window_duration() with hour-divisibility constraint
    • tests.rs — 97 tests including Go test port, proptests for window invariants
  • table_config.rsTableConfig with per-product-type default sort fields, serde support

Verification

  • cargo build -p quickwit-parquet-engine
  • cargo test -p quickwit-parquet-engine -- sort_fields table_config ✅ (97 tests)
  • cargo clippy -p quickwit-parquet-engine --all-features --tests ✅ (no new warnings)

Test plan

  • All 97 sort_fields and table_config tests pass
  • Proptest properties verified (window alignment, containment, determinism)
  • Round-trip tests for parse → display → parse
  • Go test suite fully ported
  • Clippy clean

🤖 Generated with Claude Code

@mattmkim mattmkim force-pushed the matthew.kim/metrics-wide-schema branch 3 times, most recently from 7621e22 to 1b50f21 Compare March 30, 2026 18:12
@mattmkim mattmkim force-pushed the matthew.kim/metrics-wide-schema branch from 1b50f21 to 0b642de Compare March 30, 2026 18:26
@mattmkim mattmkim force-pushed the matthew.kim/metrics-wide-schema branch from 0b642de to 7a5979f Compare March 30, 2026 18:45
@fulmicoton-dd
Copy link
Copy Markdown
Collaborator

@mattmkim I only reviewed the changes in the global scope. Can you complete the review?

@g-talbot g-talbot force-pushed the gtt/phase-31-sort-schema branch 5 times, most recently from 366dd33 to a64ddaf Compare March 31, 2026 21:26
@g-talbot g-talbot force-pushed the gtt/phase-31-sort-schema branch 3 times, most recently from 8fce718 to 018a265 Compare March 31, 2026 21:40
@g-talbot g-talbot force-pushed the gtt/phase-31-sort-schema branch from 018a265 to 3696bb4 Compare March 31, 2026 21:50
@g-talbot g-talbot force-pushed the gtt/phase-31-sort-schema branch 4 times, most recently from cf9484d to bb42b42 Compare April 1, 2026 12:25
…, window, TableConfig

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@g-talbot g-talbot force-pushed the gtt/phase-31-sort-schema branch from bb42b42 to 5b2304c Compare April 1, 2026 16:59
@g-talbot g-talbot force-pushed the gtt/phase-31-sort-schema branch from 39f875f to 4d42fd9 Compare April 1, 2026 20:17
Copy link
Copy Markdown
Contributor

@mattmkim mattmkim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - just a few Qs for myself, 1 small change since we've removed the tag_ prefix from tag columns

Base automatically changed from matthew.kim/metrics-wide-schema to main April 6, 2026 19:48
g-talbot and others added 4 commits April 6, 2026 21:11
Co-authored-by: Matthew Kim <matthew.kim@datadoghq.com>
Co-authored-by: Matthew Kim <matthew.kim@datadoghq.com>
Resolve merge conflicts by taking main's versions of otel_metrics.rs
and arrow_metrics.rs (the PR didn't modify these files — conflicts
came from the base branch divergence). Kept PR's table_config module
export in quickwit-parquet-engine/src/lib.rs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@g-talbot g-talbot merged commit ba9c817 into main Apr 8, 2026
8 checks passed
@g-talbot g-talbot deleted the gtt/phase-31-sort-schema branch April 8, 2026 11:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants