From bad9e20871daf81240eab0fa24c6609bc6961485 Mon Sep 17 00:00:00 2001 From: Davide Angelocola Date: Mon, 27 Jul 2026 07:57:27 +0200 Subject: [PATCH] fix(editions): address PR #320 review findings - Drop Edition.minVortexVersion: it names the Rust reference's own release train, not vortex-java's, so it was never meaningful here. - Drop WriteOptions#withoutEditionGuard(): the only escape hatch was guard-wide and untracked. Reaching an unstable-family encoding now requires enabling that family's edition explicitly via withEdition(Editions.UNSTABLE_...), which is both narrower and self-documenting. Updates the 9 test call sites that used the old escape hatch to reach fastlanes.delta/vortex.patched. - Implement ReadRegistry's edition-aware "no decoder registered" message, which was documented in the ADR/CHANGELOG/docs but never actually wired into decode()/decodeAsSegment() in the original PR. Add coverage for the core-edition, unstable-edition, and unknown-to-all-editions message branches, plus decodeAsSegment's error path (previously untested). - java.util.Map.of() -> Map.of() (redundant qualification) across 11 files. - Update ADR 0023 and CHANGELOG.md to match the final design; sweep docs/reference.md and docs/compatibility.md for stale references. Addresses the 6 review comments left on #320 (issue #301). Co-Authored-By: Claude Sonnet 5 --- CHANGELOG.md | 2 +- adr/0023-vortex-editions-adoption.md | 54 ++++++++++------ .../vortex/calcite/AggregateSumNullTest.java | 2 +- .../calcite/AggregateWhereBoundaryTest.java | 4 +- .../AggregateWhereCleanPartitionTest.java | 4 +- .../dfa1/vortex/calcite/OhlcGenerator.java | 2 +- .../vortex/calcite/UnsignedColumnTest.java | 2 +- .../calcite/VortexAdapterCoverageTest.java | 2 +- .../dfa1/vortex/core/model/Edition.java | 29 ++++----- .../dfa1/vortex/core/model/EditionFamily.java | 4 +- .../dfa1/vortex/core/model/Editions.java | 10 +-- .../dfa1/vortex/core/model/EditionsTest.java | 6 +- docs/compatibility.md | 31 +++++---- docs/reference.md | 23 ++++--- .../JavaRoundTripIntegrationTest.java | 3 +- .../JavaWritesRustReadsIntegrationTest.java | 5 +- .../CalciteBoundaryAggregateBenchmark.java | 2 +- .../dfa1/vortex/reader/ReadRegistry.java | 29 ++++++++- .../dfa1/vortex/reader/ReadRegistryTest.java | 64 +++++++++++++++++++ .../dfa1/vortex/writer/VortexWriter.java | 9 +-- .../dfa1/vortex/writer/WriteOptions.java | 16 ++--- .../dfa1/vortex/writer/DeltaEncodingTest.java | 15 +++-- .../vortex/writer/NullCountPruningTest.java | 2 +- .../dfa1/vortex/writer/VortexWriterTest.java | 2 +- .../vortex/writer/WriterEditionGuardTest.java | 29 ++------- .../dfa1/vortex/writer/WriterZoneMapTest.java | 22 +++---- 26 files changed, 222 insertions(+), 151 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1b057c51..152f3465 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- Adopted the Vortex [editions](https://github.com/vortex-data/vortex/blob/develop/docs/specs/editions.md) model: `WriteOptions` gates which encodings a write may emit (defaulting to the latest frozen `core` edition) and fails immediately if it would emit one outside it; `ReadRegistry`'s unknown-encoding error now names the edition an id belongs to and the minimum Vortex version needed. ([#301](https://github.com/dfa1/vortex-java/issues/301)) +- Adopted the Vortex [editions](https://github.com/vortex-data/vortex/blob/develop/docs/specs/editions.md) model: `WriteOptions` gates which encodings a write may emit (defaulting to the latest frozen `core` edition) and fails immediately if it would emit one outside it; `ReadRegistry`'s unknown-encoding error now names the edition an id belongs to. ([#301](https://github.com/dfa1/vortex-java/issues/301)) ## [0.12.5] — 2026-07-26 diff --git a/adr/0023-vortex-editions-adoption.md b/adr/0023-vortex-editions-adoption.md index cd4fcb45..d5aa8854 100644 --- a/adr/0023-vortex-editions-adoption.md +++ b/adr/0023-vortex-editions-adoption.md @@ -12,8 +12,8 @@ Upstream Vortex (`vortex-data/vortex` [PR #8871](https://github.com/vortex-data/ formalized **editions**: frozen, named, additive sets of encoding IDs, each carrying a forever read-compatibility guarantee. Editions come in independently-versioned **families** (`core`, `unstable` today); a writer targets at most one edition per family and may emit any encoding in -the union of its enabled editions' cumulative members. An edition with no recorded -`min_vortex_version` is a **draft** — every `unstable` edition is a draft. +the union of its enabled editions' cumulative members. Every `unstable`-family edition is a draft, +with no compatibility guarantee. vortex-java already implements every encoding in every frozen `core` edition through `core2026.07.0` (issue #301). This ADR is not about chasing new encodings — it is about adopting @@ -45,13 +45,16 @@ faithfully rather than being truncated to what is implemented today. `Comparable`, simplifying `isAtOrBefore`); `toString()` matches Rust's `Display` exactly (`"core2025.05.0"`, month zero-padded, version not); `isAtOrBefore(EditionId)` orders editions within a family (cross-family is always `false`). - - `Edition(EditionId id, Optional minVortexVersion, Set added)` — folds - Rust's separate `Edition`/`EditionDeclaration` into one type; `added` is only what joins *at - this exact edition*, matching Rust's field exactly (not cumulative). Populated with the real - `EncodingId.WellKnown` constants wherever one exists ("strings at the boundary, types - inside" — these come from a hardcoded catalog, not wire input, so there is no boundary to - parse a string at), falling back to `EncodingId.Custom(rawId)` only for the handful of - `unstable` ids with no `WellKnown` constant yet. + - `Edition(EditionId id, Set added)` — folds Rust's separate + `Edition`/`EditionDeclaration` into one type; `added` is only what joins *at this exact + edition*, matching Rust's field exactly (not cumulative). Deliberately drops Rust's + `min_vortex_version`: that field records the minimum *Rust* Vortex reader release a frozen + `core` edition requires — it refers to a different implementation's own release train, has no + relationship to vortex-java's versioning, and isn't meaningful in any vortex-java API. + Populated with the real `EncodingId.WellKnown` constants wherever one exists ("strings at the + boundary, types inside" — these come from a hardcoded catalog, not wire input, so there is no + boundary to parse a string at), falling back to `EncodingId.Custom(rawId)` only for the + handful of `unstable` ids with no `WellKnown` constant yet. - `Editions` — a final utility class holding the 8 catalog constants, `ALL` (declaration order), `cumulativeMembers(Edition)` (union of `added()` for every same-family edition at or before the given one, seeded with the argument's own `added()` first), and @@ -68,8 +71,9 @@ faithfully rather than being truncated to what is implemented today. ("any reader supporting `core2026.07.0` can read this file"); a private, single-writer family would carry none of that benefit, since no other reader in the ecosystem would recognize it. There is no legitimate use case for a custom family, so the type is closed. A caller who wants - to write encodings outside any known family uses `WriteOptions#withoutEditionGuard()`, not a - fabricated family. + to write an out-of-edition encoding reaches it by explicitly enabling that encoding's own + family via `WriteOptions#withEdition(Edition)` (e.g. `withEdition(Editions.UNSTABLE_2025_05_0)`) + — not a fabricated family, and (per Decision #7) not a guard-wide opt-out either. - `Edition` itself still cannot be made construction-restricted the same way: Java forbids a record's canonical constructor from being *more* restrictive than the record type's own visibility, and `Edition` must stay `public` for `WriteOptions`'s public API @@ -86,8 +90,8 @@ faithfully rather than being truncated to what is implemented today. against readers not having caught up to encodings *it* just shipped) — so "latest frozen `core`" carries no analogous risk here and avoids an arbitrary "how many versions to lag" choice. `withEdition(Edition)` replaces any edition already enabled for that edition's family - (a writer may target at most one edition per family, but multiple families at once); - `withoutEditionGuard()` clears every enabled edition — the explicit escape hatch. + (a writer may target at most one edition per family, but multiple families at once); see + Decision #7 for why there is deliberately no guard-wide opt-out. 4. **The guard filters candidates during selection, not just checks after encoding completes — a mid-implementation course correction.** The original design only checked the @@ -134,9 +138,9 @@ faithfully rather than being truncated to what is implemented today. 5. **Reader UX**: `ReadRegistry`'s two "no decoder registered" throw sites (`decode`, `decodeAsSegment`) now consult `Editions.owningEdition(id)` before throwing — naming the - edition and its `minVortexVersion` (or "draft, no compatibility guarantee" if none) when the - id is known to some edition, or saying it is unknown to all editions and pointing at - `allowUnknown()` otherwise. Both sites also switch to `VortexException`'s attributed + edition it joined (adding ", unstable — no compatibility guarantee" when that edition's family + is `UNSTABLE`) when the id is known to some edition, or saying it is unknown to all editions and + pointing at `allowUnknown()` otherwise. Both sites also switch to `VortexException`'s attributed `(EncodingId, String)` constructor (which prefixes the id), dropping the now-redundant id from the message body — a small, unrelated bug the same lines already had. @@ -145,6 +149,20 @@ faithfully rather than being truncated to what is implemented today. Resolves the issue's own open question ("persist edition into file metadata, or write-time only?"): don't. No `.fbs`/`.proto` schema touched anywhere in this feature. +7. **Follow-up correction (post-merge PR review): drop `WriteOptions#withoutEditionGuard()`, + no guard-wide escape hatch.** The version described in Decisions #2–#3 above shipped with a + `withoutEditionGuard()` method that cleared every enabled edition, disabling the guard + entirely — modeled loosely on upstream's "opt out entirely" case. Review on the merged PR + flagged this as too broad: it lets a caller silently emit encodings from *any* family, + including ones with no relationship to the edition the caller actually meant to enable, with no + record of which one they intended. The fix removes the method outright; a caller who needs an + `unstable`-family encoding now has exactly one path — `withEdition(Editions.UNSTABLE_2025_05_0)` + (or whichever `unstable` edition covers the encoding) — which both enables the guard-compatible + path and self-documents which unstable surface the caller opted into. All call sites that used + `withoutEditionGuard()` to reach `fastlanes.delta`/`vortex.patched` in tests were updated to the + corresponding `withEdition(Editions.UNSTABLE_...)` call. There is now no way to disable the + guard globally, only to widen it deliberately, one family at a time. + ## Consequences ### Positive @@ -154,7 +172,7 @@ faithfully rather than being truncated to what is implemented today. of surfacing as a confusing read error in someone else's environment later. - The default (`core2026.07.0`) is a zero-cost guardrail today: it changes no default write's output, verified by full reactor build + unit + integration test runs before and after. -- The reader's unknown-encoding error becomes actionable ("this needs vortex >= 0.70.0") instead +- The reader's unknown-encoding error becomes actionable ("joined edition core2025.06.0") instead of a bare, unhelpful id string. - `EncodeContext.excluded()`'s reuse for edition filtering required zero changes to `CascadingCompressor`, `SparseEncodingEncoder`, or `MaskedEncodingEncoder` — the mechanism @@ -165,7 +183,7 @@ faithfully rather than being truncated to what is implemented today. - `WriteOptions` gaining an 8th record component required mechanical (behavior-preserving) updates at 29 direct-constructor call sites across 12 files, plus 9 existing test call sites that deliberately exercise `unstable`-family encoders through the explicit-encoder-list - overload and needed `.withoutEditionGuard()` added. + overload and needed `.withEdition(Editions.UNSTABLE_...)` added (Decision #7). - Two guard mechanisms (graceful filtering + after-the-fact backstop) is more moving parts than a single check, and the split is only obvious from having traced the actual reachability gap — a future contributor extending the guard needs to understand both halves. diff --git a/calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateSumNullTest.java b/calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateSumNullTest.java index 7abd3dc3..025132d6 100644 --- a/calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateSumNullTest.java +++ b/calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateSumNullTest.java @@ -49,7 +49,7 @@ private SchemaPlus tableOf(long[] values, boolean[] valid) throws IOException { List.of(ColumnName.of("v")), List.of(new DType.Primitive(PType.I64, true)), false); Path file = tmp.resolve("sum-nulls.vortex"); // Large chunk so the whole column is one chunk; zone maps on so the SUM stat is emitted. - WriteOptions opts = new WriteOptions(1024, true, 0.90, 0, false, false, 256L * 1024 * 1024, java.util.Map.of()); + WriteOptions opts = new WriteOptions(1024, true, 0.90, 0, false, false, 256L * 1024 * 1024, Map.of()); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var writer = VortexWriter.create(ch, schema, opts)) { writer.writeChunk(Map.of(ColumnName.of("v"), new NullableData(values, valid))); diff --git a/calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateWhereBoundaryTest.java b/calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateWhereBoundaryTest.java index 44c19298..50fa5de9 100644 --- a/calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateWhereBoundaryTest.java +++ b/calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateWhereBoundaryTest.java @@ -57,7 +57,7 @@ static void write() throws Exception { .build(); // enableZoneMaps=true emits the per-chunk min/max/sum/null-count the tier-1 fold reads and the // classify() step uses to find the boundary zones. - WriteOptions opts = new WriteOptions(CHUNK, true, 0.90, 0, true, false, 256L * 1024 * 1024, java.util.Map.of()); + WriteOptions opts = new WriteOptions(CHUNK, true, 0.90, 0, true, false, 256L * 1024 * 1024, Map.of()); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); VortexWriter writer = VortexWriter.create(ch, schema, opts)) { for (int c = 0; c < CHUNKS; c++) { @@ -456,7 +456,7 @@ private static Ground reduce(java.util.function.LongPredicate predicate) { private static void writeChunks(Path file, DType.Struct schema, Map chunk0, Map chunk1) throws Exception { // chunkSize large so each writeChunk is exactly one chunk (one zone). - WriteOptions opts = new WriteOptions(1024, true, 0.90, 0, false, false, 256L * 1024 * 1024, java.util.Map.of()); + WriteOptions opts = new WriteOptions(1024, true, 0.90, 0, false, false, 256L * 1024 * 1024, Map.of()); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); VortexWriter writer = VortexWriter.create(ch, schema, opts)) { writer.writeChunk(chunk0); diff --git a/calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateWhereCleanPartitionTest.java b/calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateWhereCleanPartitionTest.java index 9fb21012..b03c5d8f 100644 --- a/calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateWhereCleanPartitionTest.java +++ b/calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateWhereCleanPartitionTest.java @@ -56,7 +56,7 @@ static void write() throws Exception { .field("val", DType.I64) .build(); // enableZoneMaps=true emits the per-chunk min/max/sum/null-count the fold reads. - WriteOptions opts = new WriteOptions(CHUNK, true, 0.90, 0, true, false, 256L * 1024 * 1024, java.util.Map.of()); + WriteOptions opts = new WriteOptions(CHUNK, true, 0.90, 0, true, false, 256L * 1024 * 1024, Map.of()); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); VortexWriter writer = VortexWriter.create(ch, schema, opts)) { for (int c = 0; c < CHUNKS; c++) { @@ -429,7 +429,7 @@ private static Path nullPartitionedFile(String name) throws Exception { private static void writeChunks(Path file, DType.Struct schema, Map chunk0, Map chunk1) throws Exception { // chunkSize large so each writeChunk is exactly one chunk (one zone). - WriteOptions opts = new WriteOptions(1024, true, 0.90, 0, false, false, 256L * 1024 * 1024, java.util.Map.of()); + WriteOptions opts = new WriteOptions(1024, true, 0.90, 0, false, false, 256L * 1024 * 1024, Map.of()); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); VortexWriter writer = VortexWriter.create(ch, schema, opts)) { writer.writeChunk(chunk0); diff --git a/calcite/src/test/java/io/github/dfa1/vortex/calcite/OhlcGenerator.java b/calcite/src/test/java/io/github/dfa1/vortex/calcite/OhlcGenerator.java index 34f0a979..ad0cb84b 100644 --- a/calcite/src/test/java/io/github/dfa1/vortex/calcite/OhlcGenerator.java +++ b/calcite/src/test/java/io/github/dfa1/vortex/calcite/OhlcGenerator.java @@ -20,7 +20,7 @@ private OhlcGenerator() { } static void write(Path file, int totalRows, int chunkSize) throws IOException { - WriteOptions opts = new WriteOptions(chunkSize, true, 0.90, 0, true, false, 256L * 1024 * 1024, java.util.Map.of()); + WriteOptions opts = new WriteOptions(chunkSize, true, 0.90, 0, true, false, 256L * 1024 * 1024, Map.of()); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var writer = VortexWriter.create(ch, OhlcData.SCHEMA, opts)) { for (OhlcData.Batch batch : OhlcData.generate(totalRows, chunkSize)) { diff --git a/calcite/src/test/java/io/github/dfa1/vortex/calcite/UnsignedColumnTest.java b/calcite/src/test/java/io/github/dfa1/vortex/calcite/UnsignedColumnTest.java index d40dfacd..46a62d82 100644 --- a/calcite/src/test/java/io/github/dfa1/vortex/calcite/UnsignedColumnTest.java +++ b/calcite/src/test/java/io/github/dfa1/vortex/calcite/UnsignedColumnTest.java @@ -110,7 +110,7 @@ private Path write(String name, WriteOptions opts, int u32, long u64) throws Exc private static WriteOptions noZoneMaps() { // Same shape as the adapter coverage test's zone-maps-off options: the second flag disables // zone maps so no per-zone SUM exists and VortexAggregates falls back to scanSum. - return new WriteOptions(65_536, false, 0.90, 0, true, false, 256L * 1024 * 1024, java.util.Map.of()); + return new WriteOptions(65_536, false, 0.90, 0, true, false, 256L * 1024 * 1024, Map.of()); } private static ReadRegistry registry() { diff --git a/calcite/src/test/java/io/github/dfa1/vortex/calcite/VortexAdapterCoverageTest.java b/calcite/src/test/java/io/github/dfa1/vortex/calcite/VortexAdapterCoverageTest.java index c9bd95e4..7884f918 100644 --- a/calcite/src/test/java/io/github/dfa1/vortex/calcite/VortexAdapterCoverageTest.java +++ b/calcite/src/test/java/io/github/dfa1/vortex/calcite/VortexAdapterCoverageTest.java @@ -278,7 +278,7 @@ void nonNumericColumn_throws() throws Exception { void noZoneMap_sumFallsBackToFullScan(@TempDir Path noStats) throws Exception { // Given — a file written with zone maps off, so no per-zone SUM exists to fold Path bare = noStats.resolve("nostats.vortex"); - WriteOptions noZoneMaps = new WriteOptions(65_536, false, 0.90, 0, true, false, 256L * 1024 * 1024, java.util.Map.of()); + WriteOptions noZoneMaps = new WriteOptions(65_536, false, 0.90, 0, true, false, 256L * 1024 * 1024, Map.of()); try (var ch = FileChannel.open(bare, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var w = VortexWriter.create(ch, SCHEMA, noZoneMaps)) { w.writeChunk(Map.ofEntries( diff --git a/core/src/main/java/io/github/dfa1/vortex/core/model/Edition.java b/core/src/main/java/io/github/dfa1/vortex/core/model/Edition.java index 1b6c2cb8..03c58846 100644 --- a/core/src/main/java/io/github/dfa1/vortex/core/model/Edition.java +++ b/core/src/main/java/io/github/dfa1/vortex/core/model/Edition.java @@ -1,14 +1,20 @@ package io.github.dfa1.vortex.core.model; -import java.util.Optional; import java.util.Set; -/// A Vortex edition: a named, frozen (or draft) set of encodings that join its family at this -/// point, carrying a forever read-compatibility guarantee once frozen. +/// A Vortex edition: a named set of encodings that join its family at this point. A `core`-family +/// edition is frozen and carries a forever read-compatibility guarantee; an `unstable`-family +/// edition is a draft and carries none — see [EditionFamily]. /// /// `added` is only the encodings that join *at this exact edition*, not the family's cumulative /// set — [Editions#cumulativeMembers(Edition)] computes that. /// +/// Deliberately does not carry the upstream Rust spec's `min_vortex_version` (the earliest +/// *Rust* Vortex release whose reader supports the edition): that version number refers to a +/// different implementation's release train and has no relationship to vortex-java's own +/// versioning, so surfacing it in a vortex-java message or doc would read as (and be) meaningless +/// for vortex-java's own users. +/// /// The only `Edition` instances a caller should ever construct or pass to the writer module's /// `WriteOptions#withEdition(Edition)` are [Editions]'s 8 catalog constants, mirroring the real, /// frozen Vortex spec — an `Edition` fabricated with an invented member set would carry no actual @@ -18,23 +24,12 @@ /// module — which depends on `core` but not vice versa — references it in its own public API), so /// this is an API contract enforced by convention and documentation, not the compiler. /// -/// @param id the edition identifier -/// @param minVortexVersion the minimum Vortex release whose reader supports every encoding in -/// this edition, or empty if this edition is a draft (see [#isDraft()]) -/// @param added the encodings that join the family at this edition -public record Edition(EditionId id, Optional minVortexVersion, Set added) { +/// @param id the edition identifier +/// @param added the encodings that join the family at this edition +public record Edition(EditionId id, Set added) { /// Defensively copies `added` into an immutable set. public Edition { added = Set.copyOf(added); } - - /// An edition is a **draft** until its [#minVortexVersion] has been recorded — recording it is - /// the act of freezing. A draft carries no read-compatibility guarantee and is never a default - /// write target. - /// - /// @return `true` if this edition has no recorded `minVortexVersion` - public boolean isDraft() { - return minVortexVersion.isEmpty(); - } } diff --git a/core/src/main/java/io/github/dfa1/vortex/core/model/EditionFamily.java b/core/src/main/java/io/github/dfa1/vortex/core/model/EditionFamily.java index 49675570..9b480a7a 100644 --- a/core/src/main/java/io/github/dfa1/vortex/core/model/EditionFamily.java +++ b/core/src/main/java/io/github/dfa1/vortex/core/model/EditionFamily.java @@ -6,8 +6,8 @@ /// interface with a `Custom` fallback: an edition family is a cross-implementation /// compatibility promise ("any reader supporting `core2026.07.0` can read this file"), so a /// private, single-writer "family" would carry no real guarantee — nothing else in the Vortex -/// ecosystem would recognize it. A caller who wants to write encodings outside any known family -/// uses the writer module's `WriteOptions#withoutEditionGuard()`, not a fabricated family. +/// ecosystem would recognize it. There is no legitimate use case for a fabricated family, so +/// none is offered. /// /// Deliberately does not override `toString()`: an enum's `toString()` returning anything other /// than its declared constant name is surprising (it breaks the usual assumption that diff --git a/core/src/main/java/io/github/dfa1/vortex/core/model/Editions.java b/core/src/main/java/io/github/dfa1/vortex/core/model/Editions.java index 725a3db4..09598401 100644 --- a/core/src/main/java/io/github/dfa1/vortex/core/model/Editions.java +++ b/core/src/main/java/io/github/dfa1/vortex/core/model/Editions.java @@ -26,10 +26,9 @@ /// faithfully rather than being truncated to what is implemented today. public final class Editions { - /// The baseline `core` edition: stable encodings writable by Vortex 0.36.0. + /// The baseline `core` edition: stable encodings writable by Vortex (Rust reference) 0.36.0. public static final Edition CORE_2025_05_0 = new Edition( new EditionId(EditionFamily.CORE, YearMonth.of(2025, 5), 0), - Optional.of("0.36.0"), Set.of( EncodingId.FASTLANES_BITPACKED, EncodingId.FASTLANES_FOR, EncodingId.VORTEX_ALP, EncodingId.VORTEX_ALPRD, EncodingId.VORTEX_BOOL, @@ -44,38 +43,32 @@ public final class Editions { /// The `core` edition adding stable encodings released through June 2025. public static final Edition CORE_2025_06_0 = new Edition( new EditionId(EditionFamily.CORE, YearMonth.of(2025, 6), 0), - Optional.of("0.40.0"), Set.of(EncodingId.VORTEX_PCO, EncodingId.VORTEX_SEQUENCE, EncodingId.VORTEX_ZSTD)); /// The `core` edition adding stable encodings released through October 2025. public static final Edition CORE_2025_10_0 = new Edition( new EditionId(EditionFamily.CORE, YearMonth.of(2025, 10), 0), - Optional.of("0.54.0"), Set.of(EncodingId.FASTLANES_RLE, EncodingId.VORTEX_FIXED_SIZE_LIST, EncodingId.VORTEX_LISTVIEW, EncodingId.VORTEX_MASKED)); /// The `core` edition adding stable encodings released through July 2026. public static final Edition CORE_2026_07_0 = new Edition( new EditionId(EditionFamily.CORE, YearMonth.of(2026, 7), 0), - Optional.of("0.65.0"), Set.of(EncodingId.VORTEX_VARIANT)); /// The May 2025 draft edition of the `unstable` family. public static final Edition UNSTABLE_2025_05_0 = new Edition( new EditionId(EditionFamily.UNSTABLE, YearMonth.of(2025, 5), 0), - Optional.empty(), Set.of(EncodingId.FASTLANES_DELTA)); /// The February 2026 draft edition of the `unstable` family. public static final Edition UNSTABLE_2026_02_0 = new Edition( new EditionId(EditionFamily.UNSTABLE, YearMonth.of(2026, 2), 0), - Optional.empty(), Set.of(new EncodingId.Custom("vortex.zstd_buffers"))); /// The April 2026 draft edition of the `unstable` family. public static final Edition UNSTABLE_2026_04_0 = new Edition( new EditionId(EditionFamily.UNSTABLE, YearMonth.of(2026, 4), 0), - Optional.empty(), Set.of( new EncodingId.Custom("vortex.parquet.variant"), EncodingId.VORTEX_PATCHED, new EncodingId.Custom("vortex.tensor.cosine_similarity"), @@ -86,7 +79,6 @@ public final class Editions { /// The June 2026 draft edition of the `unstable` family. public static final Edition UNSTABLE_2026_06_0 = new Edition( new EditionId(EditionFamily.UNSTABLE, YearMonth.of(2026, 6), 0), - Optional.empty(), Set.of(new EncodingId.Custom("vortex.onpair"))); /// Every declared edition, in the order above. Order matters: [#owningEdition(EncodingId)] diff --git a/core/src/test/java/io/github/dfa1/vortex/core/model/EditionsTest.java b/core/src/test/java/io/github/dfa1/vortex/core/model/EditionsTest.java index 2236c463..04552f7e 100644 --- a/core/src/test/java/io/github/dfa1/vortex/core/model/EditionsTest.java +++ b/core/src/test/java/io/github/dfa1/vortex/core/model/EditionsTest.java @@ -148,7 +148,7 @@ void editionNotInAll_cumulativeSeedsFromItsOwnAddedRatherThanRequiringIdentityIn // in production), so this exercises cumulativeMembers' seeding logic directly rather // than a reachable real-world scenario. Edition hypothetical = new Edition( - new EditionId(EditionFamily.CORE, YearMonth.of(2099, 1), 0), Optional.of("9.0.0"), + new EditionId(EditionFamily.CORE, YearMonth.of(2099, 1), 0), Set.of(new EncodingId.Custom("vortex.future"))); // When @@ -178,14 +178,14 @@ void owningEdition_coreFamilyId_returnsTheEditionItFirstJoined() { } @Test - void owningEdition_unstableFamilyId_returnsTheDraftEdition() { + void owningEdition_unstableFamilyId_returnsTheUnstableEdition() { // Given // When Optional result = Editions.owningEdition(EncodingId.VORTEX_PATCHED); // Then assertThat(result).contains(Editions.UNSTABLE_2026_04_0); - assertThat(result.get().isDraft()).isTrue(); + assertThat(result.get().id().family()).isEqualTo(EditionFamily.UNSTABLE); } @Test diff --git a/docs/compatibility.md b/docs/compatibility.md index cf89709e..1df2910f 100644 --- a/docs/compatibility.md +++ b/docs/compatibility.md @@ -174,9 +174,9 @@ Bitpacked produces `LazyAlp(MaterializedXxx)`). ### Unknown encodings Files containing unrecognized encoding IDs throw `VortexException` by default. The message names -which [edition](#editions) the id belongs to and the minimum Vortex version needed (or that it's -an `unstable` draft with no compatibility guarantee), or says the id is unknown to every edition -if it belongs to none. Opt in to passthrough mode to read such files without failing: +which [edition](#editions) the id belongs to (and whether it's an `unstable` draft with no +compatibility guarantee), or says the id is unknown to every edition if it belongs to none. Opt +in to passthrough mode to read such files without failing: ```java ReadRegistry registry = ReadRegistry.builder() @@ -195,16 +195,21 @@ guarantee once frozen (ADR 0023) — a write-time/read-time policy, not part of `WriteOptions.defaults()` targets the latest frozen `core` edition; see [the reference doc](reference.md#editions) for the writer/reader integration. -| Edition | Family | Min Vortex | Adds | -|---|---|---|---| -| `core2025.05.0` | `core` | 0.36.0 | 23 baseline: `fastlanes.bitpacked`/`for`, `vortex.alp`/`alprd`/`bool`/`bytebool`/`chunked`/`constant`/`datetimeparts`/`decimal`/`decimal_byte_parts`/`dict`/`ext`/`fsst`/`list`/`null`/`primitive`/`runend`/`sparse`/`struct`/`varbin`/`varbinview`/`zigzag` | -| `core2025.06.0` | `core` | 0.40.0 | `vortex.pco`, `vortex.sequence`, `vortex.zstd` | -| `core2025.10.0` | `core` | 0.54.0 | `fastlanes.rle`, `vortex.fixed_size_list`, `vortex.listview`, `vortex.masked` | -| `core2026.07.0` | `core` | 0.65.0 | `vortex.variant` | -| `unstable2025.05.0` | `unstable` | draft | `fastlanes.delta` ✅ implemented | -| `unstable2026.02.0` | `unstable` | draft | `vortex.zstd_buffers` ❌ not implemented | -| `unstable2026.04.0` | `unstable` | draft | `vortex.parquet.variant` ❌, `vortex.patched` ✅, `vortex.tensor.*` (4 ids) ❌ | -| `unstable2026.06.0` | `unstable` | draft | `vortex.onpair` ❌ not implemented | +| Edition | Family | Adds | +|---|---|---| +| `core2025.05.0` | `core` | 23 baseline: `fastlanes.bitpacked`/`for`, `vortex.alp`/`alprd`/`bool`/`bytebool`/`chunked`/`constant`/`datetimeparts`/`decimal`/`decimal_byte_parts`/`dict`/`ext`/`fsst`/`list`/`null`/`primitive`/`runend`/`sparse`/`struct`/`varbin`/`varbinview`/`zigzag` | +| `core2025.06.0` | `core` | `vortex.pco`, `vortex.sequence`, `vortex.zstd` | +| `core2025.10.0` | `core` | `fastlanes.rle`, `vortex.fixed_size_list`, `vortex.listview`, `vortex.masked` | +| `core2026.07.0` | `core` | `vortex.variant` | +| `unstable2025.05.0` | `unstable` | `fastlanes.delta` ✅ implemented | +| `unstable2026.02.0` | `unstable` | `vortex.zstd_buffers` ❌ not implemented | +| `unstable2026.04.0` | `unstable` | `vortex.parquet.variant` ❌, `vortex.patched` ✅, `vortex.tensor.*` (4 ids) ❌ | +| `unstable2026.06.0` | `unstable` | `vortex.onpair` ❌ not implemented | + +`core` editions are frozen with a forever read-compatibility guarantee; `unstable` editions are +drafts with none. (Upstream additionally records each frozen `core` edition's minimum *Rust* +Vortex reader release — that number refers to a different implementation's release train, has no +relationship to vortex-java's own versioning, and isn't surfaced here or in any vortex-java API.) ## Extension types diff --git a/docs/reference.md b/docs/reference.md index 0faa352c..11872145 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -65,7 +65,7 @@ shortcut returning a nullable copy), `withNullable(boolean)`, `DType.Struct.fiel | `ColumnName` | `record ColumnName(String value)` | Validated column name: non-blank, no control characters. `ColumnName.violation(String)` is the policy chokepoint shared by builder, writer, and file parser | | `EditionFamily` | `enum` — `CORE`, `UNSTABLE` | Closed (unlike `EncodingId`/`LayoutId`): an edition family is a cross-implementation compatibility promise, so a custom family carries no real guarantee — see [Editions](#editions) | | `EditionId` | `record EditionId(EditionFamily family, YearMonth cutMonth, int version)` | e.g. `core2025.05.0`; `isAtOrBefore` orders editions within a family only | -| `Edition` | `record Edition(EditionId id, Optional minVortexVersion, Set added)` | Only `Editions`'s 8 catalog constants should be constructed (API contract, not compiler-enforced — see ADR 0023) | +| `Edition` | `record Edition(EditionId id, Set added)` | Only `Editions`'s 8 catalog constants should be constructed (API contract, not compiler-enforced — see ADR 0023) | ## Reader API @@ -142,8 +142,7 @@ Record: `(int chunkSize, boolean enableZoneMaps, double compressionRatioThreshol | `withGlobalDict(boolean)` | Toggle the shared cross-chunk dictionary | | `withZstd(boolean)` | Add Zstandard to the cascade codec competition | | `withGlobalDictMaxRetainedBytes(long)` | Aggregate heap budget for buffered global-dict candidate columns | -| `withEdition(Edition)` | Enable an [edition](#editions) for its family, replacing any edition already enabled for that family | -| `withoutEditionGuard()` | Clear every enabled edition — the writer may then emit any registered encoding, at the cost of the edition compatibility guarantee | +| `withEdition(Edition)` | Enable an [edition](#editions) for its family, replacing any edition already enabled for that family. No method disables the guard entirely — an `unstable`-family encoding is reached by enabling its edition explicitly, e.g. `withEdition(Editions.UNSTABLE_2025_05_0)` | --- @@ -256,7 +255,7 @@ the wire format** — nothing about a targeted edition is ever persisted into a | Member | Notes | |--------------------------------------|------------------------------------------------------------------------------| -| `CORE_2025_05_0` … `CORE_2026_07_0` | The 4 frozen `core` editions (min Vortex 0.36.0 → 0.65.0) | +| `CORE_2025_05_0` … `CORE_2026_07_0` | The 4 frozen `core` editions (forever read-compatibility guarantee) | | `UNSTABLE_2025_05_0` … `UNSTABLE_2026_06_0` | The 4 draft `unstable` editions (no compatibility guarantee) | | `ALL` | Every declared edition, in declaration order | | `cumulativeMembers(Edition)` | The edition's own additions plus every earlier same-family edition's | @@ -279,18 +278,18 @@ Where possible (any selection routed through `CascadingCompressor`, including ne like a masked column's validity-bitmap cascade) the guard instead steers selection away from the ineligible candidate ahead of time, falling back to the best remaining one — see ADR 0023. -- `WriteOptions.withEdition(Edition)` enables an edition, replacing any edition already enabled - for that edition's family. Multiple families can be enabled at once (e.g. `core` and `unstable` - simultaneously); at most one edition per family. -- `WriteOptions.withoutEditionGuard()` clears every enabled edition — the escape hatch for custom - or experimental encodings, at the cost of the compatibility guarantee. +`WriteOptions.withEdition(Edition)` enables an edition, replacing any edition already enabled for +that edition's family. Multiple families can be enabled at once (e.g. `core` and `unstable` +simultaneously); at most one edition per family. There is deliberately no "disable the guard" +method — an `unstable`-family encoding is reached by enabling its edition explicitly, not by +opting out of the guard altogether. ### Reader integration When `ReadRegistry` hits an unregistered encoding id (and `allowUnknown()` is off), the thrown -`VortexException` names the edition the id belongs to and its minimum required Vortex version -(or that it is a draft with no guarantee), or says the id is unknown to every edition and points -at `allowUnknown()`. +`VortexException` names the edition the id belongs to (and whether it's an `unstable` draft with +no compatibility guarantee), or says the id is unknown to every edition and points at +`allowUnknown()`. --- diff --git a/integration/src/test/java/io/github/dfa1/vortex/integration/JavaRoundTripIntegrationTest.java b/integration/src/test/java/io/github/dfa1/vortex/integration/JavaRoundTripIntegrationTest.java index 55d317c8..66988e77 100644 --- a/integration/src/test/java/io/github/dfa1/vortex/integration/JavaRoundTripIntegrationTest.java +++ b/integration/src/test/java/io/github/dfa1/vortex/integration/JavaRoundTripIntegrationTest.java @@ -2,6 +2,7 @@ import io.github.dfa1.vortex.core.model.ColumnName; import io.github.dfa1.vortex.core.model.DType; +import io.github.dfa1.vortex.core.model.Editions; import io.github.dfa1.vortex.reader.ReadRegistry; import io.github.dfa1.vortex.reader.ScanOptions; import io.github.dfa1.vortex.reader.VortexReader; @@ -50,7 +51,7 @@ void patched_i32_javaWriteJavaRead(@TempDir Path tmp) throws IOException { data[61] = 7_000_000; data[88] = 8_000_000; try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); - var sut = VortexWriter.create(ch, I32_SCHEMA, WriteOptions.defaults().withoutEditionGuard(), + var sut = VortexWriter.create(ch, I32_SCHEMA, WriteOptions.defaults().withEdition(Editions.UNSTABLE_2026_04_0), List.of(new PatchedEncodingEncoder()))) { // When sut.writeChunk(Map.of(ColumnName.of("v"), data)); diff --git a/integration/src/test/java/io/github/dfa1/vortex/integration/JavaWritesRustReadsIntegrationTest.java b/integration/src/test/java/io/github/dfa1/vortex/integration/JavaWritesRustReadsIntegrationTest.java index c7998ff1..10f837b5 100644 --- a/integration/src/test/java/io/github/dfa1/vortex/integration/JavaWritesRustReadsIntegrationTest.java +++ b/integration/src/test/java/io/github/dfa1/vortex/integration/JavaWritesRustReadsIntegrationTest.java @@ -10,6 +10,7 @@ import dev.vortex.arrow.ArrowAllocation; import dev.vortex.jni.NativeLoader; import io.github.dfa1.vortex.core.model.DType; +import io.github.dfa1.vortex.core.model.Editions; import io.github.dfa1.vortex.core.model.PType; import io.github.dfa1.vortex.core.testing.OhlcData; import io.github.dfa1.vortex.writer.encode.BoolEncodingEncoder; @@ -509,7 +510,7 @@ void javaWriter_jniReader_zoneMapped_multipleZones(@TempDir Path tmp) throws IOE // zone-map with one zone per chunk. The Rust reader must parse that layout and still // return every value (zones are a transparent pruning aux). Path file = tmp.resolve("java_zoned.vtx"); - WriteOptions zoneMapped = new WriteOptions(4, true, 0.90, 0, true, false, 256L * 1024 * 1024, java.util.Map.of()); + WriteOptions zoneMapped = new WriteOptions(4, true, 0.90, 0, true, false, 256L * 1024 * 1024, Map.of()); long[] ids = new long[20]; double[] vals = new double[20]; for (int i = 0; i < 20; i++) { @@ -1046,7 +1047,7 @@ void javaWriter_rustReader_delta_i64(@TempDir Path tmp) throws IOException { data[i] = acc; } try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); - var sut = VortexWriter.create(ch, TS_SCHEMA, WriteOptions.defaults().withoutEditionGuard(), + var sut = VortexWriter.create(ch, TS_SCHEMA, WriteOptions.defaults().withEdition(Editions.UNSTABLE_2025_05_0), List.of(new DeltaEncodingEncoder()))) { // When sut.writeChunk(Map.of(ColumnName.of("ts"), data)); diff --git a/performance/src/main/java/io/github/dfa1/vortex/performance/CalciteBoundaryAggregateBenchmark.java b/performance/src/main/java/io/github/dfa1/vortex/performance/CalciteBoundaryAggregateBenchmark.java index 5896d06d..970ad455 100644 --- a/performance/src/main/java/io/github/dfa1/vortex/performance/CalciteBoundaryAggregateBenchmark.java +++ b/performance/src/main/java/io/github/dfa1/vortex/performance/CalciteBoundaryAggregateBenchmark.java @@ -194,7 +194,7 @@ private static void writeFixture(Path file) throws IOException { .field("val", DType.I64) .build(); // enableZoneMaps=true emits the per-chunk min/max/sum/null-count the interior-zone fold reads. - WriteOptions opts = new WriteOptions(CHUNK_SIZE, true, 0.90, 0, true, false, 256L * 1024 * 1024, java.util.Map.of()); + WriteOptions opts = new WriteOptions(CHUNK_SIZE, true, 0.90, 0, true, false, 256L * 1024 * 1024, Map.of()); java.util.Random rng = new java.util.Random(SEED); try (FileChannel ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE, StandardOpenOption.TRUNCATE_EXISTING); diff --git a/reader/src/main/java/io/github/dfa1/vortex/reader/ReadRegistry.java b/reader/src/main/java/io/github/dfa1/vortex/reader/ReadRegistry.java index 6a63b851..40f2ef0e 100644 --- a/reader/src/main/java/io/github/dfa1/vortex/reader/ReadRegistry.java +++ b/reader/src/main/java/io/github/dfa1/vortex/reader/ReadRegistry.java @@ -3,6 +3,9 @@ import io.github.dfa1.vortex.core.error.VortexException; import io.github.dfa1.vortex.reader.array.Array; import io.github.dfa1.vortex.reader.array.UnknownArray; +import io.github.dfa1.vortex.core.model.Edition; +import io.github.dfa1.vortex.core.model.EditionFamily; +import io.github.dfa1.vortex.core.model.Editions; import io.github.dfa1.vortex.core.model.EncodingId; import io.github.dfa1.vortex.reader.decode.AlpEncodingDecoder; import io.github.dfa1.vortex.reader.decode.AlpRdEncodingDecoder; @@ -45,6 +48,7 @@ import java.util.Collections; import java.util.Comparator; import java.util.Map; +import java.util.Optional; import java.util.TreeMap; /// Read-side registry: maps [EncodingId] to [EncodingDecoder] implementations. @@ -124,7 +128,7 @@ public Array decode(DecodeContext ctx) { if (allowUnknown) { return decodeUnknown(ctx, node); } - throw new VortexException("no decoder registered for " + node.encodingId().id()); + throw new VortexException(node.encodingId(), noDecoderMessage(node.encodingId())); } /// Decodes the array described by `ctx` and returns its primary backing segment. @@ -137,8 +141,27 @@ public MemorySegment decodeAsSegment(DecodeContext ctx) { if (decoder != null) { return decoder.decode(ctx).materialize(ctx.arena()); } - throw new VortexException("no decoder registered for " + node.encodingId().id() - + " (or encoding has no primary segment)"); + throw new VortexException(node.encodingId(), + noDecoderMessage(node.encodingId()) + " (or encoding has no primary segment)"); + } + + /// Builds the "no decoder registered" message, enriched with which [Edition] `id` belongs to + /// when known — an actionable pointer ("this file is newer than your build, or the encoding is + /// unstable/experimental") instead of a bare id string. Issue #301. + /// + /// @param id the unrecognized encoding id + /// @return the message body (the caller's [VortexException] constructor prefixes the id itself) + private static String noDecoderMessage(EncodingId id) { + Optional owning = Editions.owningEdition(id); + if (owning.isEmpty()) { + return "no decoder registered; unknown to all editions (custom/experimental encoding)" + + " — register a decoder or enable ReadRegistry.Builder#allowUnknown()"; + } + Edition edition = owning.get(); + String draftNote = edition.id().family() == EditionFamily.UNSTABLE + ? ", unstable — no compatibility guarantee" : ""; + return "no decoder registered (joined edition " + edition.id() + draftNote + + "); register a decoder or enable ReadRegistry.Builder#allowUnknown()"; } private static UnknownArray decodeUnknown(DecodeContext ctx, ArrayNode node) { diff --git a/reader/src/test/java/io/github/dfa1/vortex/reader/ReadRegistryTest.java b/reader/src/test/java/io/github/dfa1/vortex/reader/ReadRegistryTest.java index 3b213628..e99ab84d 100644 --- a/reader/src/test/java/io/github/dfa1/vortex/reader/ReadRegistryTest.java +++ b/reader/src/test/java/io/github/dfa1/vortex/reader/ReadRegistryTest.java @@ -50,6 +50,70 @@ void decodeKnownEncodingWithoutDecoderThrowsByDefault() { .hasMessageContaining("vortex.primitive"); } + @Test + void decodeCoreEditionEncodingWithoutDecoder_namesTheEditionInTheMessage() { + // Given — vortex.zstd first joins core2025.06.0, no decoder registered for it (issue #301) + ReadRegistry sut = ReadRegistry.empty(); + ArrayNode node = new ArrayNode(EncodingId.VORTEX_ZSTD, + MemorySegment.ofArray(new byte[0]), new ArrayNode[0], new int[0]); + DecodeContext ctx = new DecodeContext(node, DTypes.I32, 0L, + new MemorySegment[0], sut, Arena.ofAuto()); + + // When / Then + assertThatThrownBy(() -> sut.decode(ctx)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("joined edition core2025.06.0") + .hasMessageContaining("allowUnknown") + .hasMessageNotContaining("unstable"); + } + + @Test + void decodeUnstableEditionEncodingWithoutDecoder_notesNoCompatibilityGuarantee() { + // Given — fastlanes.delta belongs to the draft unstable2025.05.0 edition + ReadRegistry sut = ReadRegistry.empty(); + ArrayNode node = new ArrayNode(EncodingId.FASTLANES_DELTA, + MemorySegment.ofArray(new byte[0]), new ArrayNode[0], new int[0]); + DecodeContext ctx = new DecodeContext(node, DTypes.I32, 0L, + new MemorySegment[0], sut, Arena.ofAuto()); + + // When / Then + assertThatThrownBy(() -> sut.decode(ctx)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("joined edition unstable2025.05.0") + .hasMessageContaining("unstable — no compatibility guarantee"); + } + + @Test + void decodeUnknownEncoding_saysUnknownToAllEditions() { + // Given — not a WellKnown id, and no edition declares it either + ReadRegistry sut = ReadRegistry.empty(); + ArrayNode node = new ArrayNode(EncodingId.parse("some.unknown"), + MemorySegment.ofArray(new byte[0]), new ArrayNode[0], new int[0]); + DecodeContext ctx = new DecodeContext(node, DTypes.I32, 0L, + new MemorySegment[0], sut, Arena.ofAuto()); + + // When / Then + assertThatThrownBy(() -> sut.decode(ctx)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("unknown to all editions"); + } + + @Test + void decodeAsSegmentKnownEditionEncodingWithoutDecoder_alsoNamesTheEdition() { + // Given — decodeAsSegment shares the same enriched message as decode + ReadRegistry sut = ReadRegistry.empty(); + ArrayNode node = new ArrayNode(EncodingId.VORTEX_ZSTD, + MemorySegment.ofArray(new byte[0]), new ArrayNode[0], new int[0]); + DecodeContext ctx = new DecodeContext(node, DTypes.I32, 0L, + new MemorySegment[0], sut, Arena.ofAuto()); + + // When / Then + assertThatThrownBy(() -> sut.decodeAsSegment(ctx)) + .isInstanceOf(VortexException.class) + .hasMessageContaining("joined edition core2025.06.0") + .hasMessageContaining("no primary segment"); + } + @Test void decodeKnownEncodingWithoutDecoderReturnsUnknownArrayWhenAllowed() { // Given diff --git a/writer/src/main/java/io/github/dfa1/vortex/writer/VortexWriter.java b/writer/src/main/java/io/github/dfa1/vortex/writer/VortexWriter.java index b0a7f3d9..7e5e9866 100644 --- a/writer/src/main/java/io/github/dfa1/vortex/writer/VortexWriter.java +++ b/writer/src/main/java/io/github/dfa1/vortex/writer/VortexWriter.java @@ -782,14 +782,9 @@ private VortexException editionViolation(EncodingId id) { .collect(Collectors.joining(", ")); Optional owning = Editions.owningEdition(id); String hint = owning.isPresent() - ? "; it joins " + owning.get().id() - + (owning.get().isDraft() - ? " (draft, no compatibility guarantee)" - : " (requires reader >= " + owning.get().minVortexVersion().get() + ")") - + " — enable it via WriteOptions.withEdition(...)" + ? "; it joins " + owning.get().id() + " — enable that edition via WriteOptions.withEdition(...)" : "; it is not part of any known edition"; - return new VortexException(id, "outside the configured edition(s) [" + configured + "]" + hint - + ", or call WriteOptions.withoutEditionGuard() to opt out"); + return new VortexException(id, "outside the configured edition(s) [" + configured + "]" + hint); } private EncodingEncoder findEncoder(DType dtype) { diff --git a/writer/src/main/java/io/github/dfa1/vortex/writer/WriteOptions.java b/writer/src/main/java/io/github/dfa1/vortex/writer/WriteOptions.java index bb346f02..2b55d0e3 100644 --- a/writer/src/main/java/io/github/dfa1/vortex/writer/WriteOptions.java +++ b/writer/src/main/java/io/github/dfa1/vortex/writer/WriteOptions.java @@ -40,8 +40,10 @@ /// the latest frozen `core` edition ([Editions#CORE_2026_07_0]): vortex-java /// implements every `core`-family encoding, and the default cascade never /// selects an `unstable`-family one, so this is a zero-cost guardrail against -/// ever silently widening a file's minimum required reader. Cleared entirely by -/// [#withoutEditionGuard()]. +/// ever silently widening a file's minimum required reader. Deliberately has +/// no "disable the guard" method: an `unstable`-family encoding is reached by +/// enabling its edition explicitly via [#withEdition(Edition)], not by opting +/// out of the guard altogether. public record WriteOptions( int chunkSize, boolean enableZoneMaps, @@ -157,14 +159,4 @@ public WriteOptions withEdition(Edition edition) { return new WriteOptions(chunkSize, enableZoneMaps, compressionRatioThreshold, allowedCascading, globalDict, enableZstd, globalDictMaxRetainedBytes, updated); } - - /// Returns a copy of these options with every enabled edition cleared — the explicit escape - /// hatch. With no edition enabled, the writer may emit any registered encoding, including - /// third-party or experimental ones, at the cost of the edition read-compatibility guarantee. - /// - /// @return a new `WriteOptions` with no edition guard at all - public WriteOptions withoutEditionGuard() { - return new WriteOptions(chunkSize, enableZoneMaps, compressionRatioThreshold, allowedCascading, globalDict, - enableZstd, globalDictMaxRetainedBytes, Map.of()); - } } diff --git a/writer/src/test/java/io/github/dfa1/vortex/writer/DeltaEncodingTest.java b/writer/src/test/java/io/github/dfa1/vortex/writer/DeltaEncodingTest.java index 9799c8b9..cdc59dbf 100644 --- a/writer/src/test/java/io/github/dfa1/vortex/writer/DeltaEncodingTest.java +++ b/writer/src/test/java/io/github/dfa1/vortex/writer/DeltaEncodingTest.java @@ -2,6 +2,7 @@ import io.github.dfa1.vortex.core.model.ColumnName; import io.github.dfa1.vortex.core.model.DType; +import io.github.dfa1.vortex.core.model.Editions; import io.github.dfa1.vortex.writer.encode.DeltaEncodingEncoder; import io.github.dfa1.vortex.reader.ReadRegistry; import io.github.dfa1.vortex.reader.VortexReader; @@ -41,7 +42,7 @@ void roundTrip_monotonicIncreasing(@TempDir Path tmp) throws IOException { long[] data = {100L, 105L, 110L, 115L, 120L}; try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); - var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults().withoutEditionGuard(), + var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults().withEdition(Editions.UNSTABLE_2025_05_0), List.of(new DeltaEncodingEncoder()))) { // When sut.writeChunk(Map.of(ColumnName.of("ts"), data)); @@ -60,7 +61,7 @@ void roundTrip_monotonicDecreasing(@TempDir Path tmp) throws IOException { long[] data = {1000L, 990L, 975L, 950L}; try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); - var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults().withoutEditionGuard(), + var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults().withEdition(Editions.UNSTABLE_2025_05_0), List.of(new DeltaEncodingEncoder()))) { // When sut.writeChunk(Map.of(ColumnName.of("ts"), data)); @@ -79,7 +80,7 @@ void roundTrip_allSameValue(@TempDir Path tmp) throws IOException { long[] data = {42L, 42L, 42L, 42L}; try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); - var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults().withoutEditionGuard(), + var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults().withEdition(Editions.UNSTABLE_2025_05_0), List.of(new DeltaEncodingEncoder()))) { // When sut.writeChunk(Map.of(ColumnName.of("ts"), data)); @@ -98,7 +99,7 @@ void roundTrip_singleElement(@TempDir Path tmp) throws IOException { long[] data = {99L}; try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); - var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults().withoutEditionGuard(), + var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults().withEdition(Editions.UNSTABLE_2025_05_0), List.of(new DeltaEncodingEncoder()))) { // When sut.writeChunk(Map.of(ColumnName.of("ts"), data)); @@ -117,7 +118,7 @@ void roundTrip_mixedDeltas(@TempDir Path tmp) throws IOException { long[] data = {0L, 5L, 7L, 12L, 11L, 15L}; try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); - var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults().withoutEditionGuard(), + var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults().withEdition(Editions.UNSTABLE_2025_05_0), List.of(new DeltaEncodingEncoder()))) { // When sut.writeChunk(Map.of(ColumnName.of("ts"), data)); @@ -139,7 +140,7 @@ void roundTrip_multipleChunks(@TempDir Path tmp) throws IOException { long[] chunk2 = {2000L, 2005L, 2010L}; try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); - var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults().withoutEditionGuard(), + var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults().withEdition(Editions.UNSTABLE_2025_05_0), List.of(new DeltaEncodingEncoder()))) { // When sut.writeChunk(Map.of(ColumnName.of("ts"), chunk1)); @@ -164,7 +165,7 @@ void roundTrip_sequentialTimestamps(int n, @TempDir Path tmp) throws IOException } try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); - var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults().withoutEditionGuard(), + var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults().withEdition(Editions.UNSTABLE_2025_05_0), List.of(new DeltaEncodingEncoder()))) { // When sut.writeChunk(Map.of(ColumnName.of("ts"), data)); diff --git a/writer/src/test/java/io/github/dfa1/vortex/writer/NullCountPruningTest.java b/writer/src/test/java/io/github/dfa1/vortex/writer/NullCountPruningTest.java index 2bb3685e..5f9cbd03 100644 --- a/writer/src/test/java/io/github/dfa1/vortex/writer/NullCountPruningTest.java +++ b/writer/src/test/java/io/github/dfa1/vortex/writer/NullCountPruningTest.java @@ -33,7 +33,7 @@ class NullCountPruningTest { // sizes and null patterns: 3 rows / 0 nulls, 2 rows / 1 null, 4 rows / all null. private Path write() throws IOException { Path file = tmp.resolve("nulls.vtx"); - WriteOptions opts = new WriteOptions(1024, true, 0.90, 0, false, false, 256L * 1024 * 1024, java.util.Map.of()); + WriteOptions opts = new WriteOptions(1024, true, 0.90, 0, false, false, 256L * 1024 * 1024, Map.of()); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var sut = VortexWriter.create(ch, SCHEMA, opts)) { sut.writeChunk(Map.of(ColumnName.of("v"), new NullableData( diff --git a/writer/src/test/java/io/github/dfa1/vortex/writer/VortexWriterTest.java b/writer/src/test/java/io/github/dfa1/vortex/writer/VortexWriterTest.java index b7d961cc..b4973be8 100644 --- a/writer/src/test/java/io/github/dfa1/vortex/writer/VortexWriterTest.java +++ b/writer/src/test/java/io/github/dfa1/vortex/writer/VortexWriterTest.java @@ -60,7 +60,7 @@ void writeSegments_are64ByteAligned(@TempDir Path tmp) throws IOException { // VortexWriter pads before each segment so every buffer starts 64-aligned (Arrow-compatible); // a broken pad — wrong modulus arithmetic or a skipped writePadding — leaves a segment offset // off a 64-byte boundary. - WriteOptions opts = new WriteOptions(3, false, 0.90, 0, false, false, 256L * 1024 * 1024, java.util.Map.of()); + WriteOptions opts = new WriteOptions(3, false, 0.90, 0, false, false, 256L * 1024 * 1024, Map.of()); Path file = tmp.resolve("aligned.vtx"); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var sut = VortexWriter.create(ch, SCHEMA, opts)) { diff --git a/writer/src/test/java/io/github/dfa1/vortex/writer/WriterEditionGuardTest.java b/writer/src/test/java/io/github/dfa1/vortex/writer/WriterEditionGuardTest.java index c482738b..3c9d69d1 100644 --- a/writer/src/test/java/io/github/dfa1/vortex/writer/WriterEditionGuardTest.java +++ b/writer/src/test/java/io/github/dfa1/vortex/writer/WriterEditionGuardTest.java @@ -27,6 +27,10 @@ /// single-candidate explicit encoder list, as used here. The graceful-fallback filtering behavior /// for cost-based competitions is covered separately by /// `io.github.dfa1.vortex.writer.encode.MaskedValidityCascadeEditionExclusionTest`. +/// +/// There is deliberately no "disable the guard" escape hatch — an `unstable`-family encoding +/// must be reached by enabling its edition explicitly (see +/// [#withEdition_enablingUnstable_allowsTheForcedEncoder]). class WriterEditionGuardTest { private static final DType.Struct I64_SCHEMA = new DType.Struct( @@ -57,33 +61,14 @@ void defaultGuard_forcedOutOfEditionEncoder_throwsNamingIdAndEdition(@TempDir Pa .isInstanceOf(VortexException.class) .hasMessageContaining("fastlanes.delta") .hasMessageContaining("core2026.07.0") - .hasMessageContaining("withoutEditionGuard"); - } - } - - @Test - void withoutEditionGuard_allowsTheSameForcedEncoder(@TempDir Path tmp) throws IOException { - // Given — the identical setup, but with the guard explicitly cleared - Path file = tmp.resolve("delta_unguarded.vtx"); - long[] data = {100L, 105L, 110L, 115L, 120L}; - - // When - try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); - var sut = VortexWriter.create(ch, I64_SCHEMA, WriteOptions.defaults().withoutEditionGuard(), - List.of(new DeltaEncodingEncoder()))) { - sut.writeChunk(Map.of(ColumnName.of("ts"), data)); - } - - // Then - try (var vf = VortexReader.open(file, deltaRegistry())) { - assertThat(readAllLongs(vf, "ts")).containsExactly(data); + .hasMessageContaining("unstable2025.05.0") + .hasMessageContaining("withEdition"); } } @Test void withEdition_enablingUnstable_allowsTheForcedEncoder(@TempDir Path tmp) throws IOException { - // Given — instead of clearing the guard entirely, explicitly opt into the unstable - // edition that covers fastlanes.delta + // Given — explicitly opt into the unstable edition that covers fastlanes.delta Path file = tmp.resolve("delta_unstable.vtx"); long[] data = {100L, 105L, 110L, 115L, 120L}; WriteOptions options = WriteOptions.defaults().withEdition(Editions.UNSTABLE_2025_05_0); diff --git a/writer/src/test/java/io/github/dfa1/vortex/writer/WriterZoneMapTest.java b/writer/src/test/java/io/github/dfa1/vortex/writer/WriterZoneMapTest.java index aa3bf7fd..b50142b1 100644 --- a/writer/src/test/java/io/github/dfa1/vortex/writer/WriterZoneMapTest.java +++ b/writer/src/test/java/io/github/dfa1/vortex/writer/WriterZoneMapTest.java @@ -40,7 +40,7 @@ class WriterZoneMapTest { // Three zones of four rows: [0..3], [4..7], [8..11]. private static Path write(Path tmp, boolean zoneMaps) throws IOException { - WriteOptions opts = new WriteOptions(4, zoneMaps, 0.90, 0, true, false, 256L * 1024 * 1024, java.util.Map.of()); + WriteOptions opts = new WriteOptions(4, zoneMaps, 0.90, 0, true, false, 256L * 1024 * 1024, Map.of()); Path file = tmp.resolve("zoned-" + zoneMaps + ".vtx"); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var sut = VortexWriter.create(ch, SCHEMA, opts)) { @@ -139,7 +139,7 @@ void zoneMaps_nullableColumn_recordsPerZoneNullCount(@TempDir Path tmp) throws I // zone 1 = [null, null] DType.Struct schema = new DType.Struct( List.of(ColumnName.of("v")), List.of(new DType.Primitive(PType.I64, true)), false); - WriteOptions opts = new WriteOptions(2, true, 0.90, 0, true, false, 256L * 1024 * 1024, java.util.Map.of()); + WriteOptions opts = new WriteOptions(2, true, 0.90, 0, true, false, 256L * 1024 * 1024, Map.of()); Path file = tmp.resolve("nullable.vtx"); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var sut = VortexWriter.create(ch, schema, opts)) { @@ -176,7 +176,7 @@ void everyPrimitiveType_emitsZonedLayout(PType ptype, @TempDir Path tmp) throws // every one gets a vortex.stats layout — exercising each per-ptype stat-column arm. DType.Struct schema = new DType.Struct( List.of(ColumnName.of("v")), List.of(new DType.Primitive(ptype, false)), false); - WriteOptions opts = new WriteOptions(2, true, 0.90, 0, false, false, 256L * 1024 * 1024, java.util.Map.of()); + WriteOptions opts = new WriteOptions(2, true, 0.90, 0, false, false, 256L * 1024 * 1024, Map.of()); Path file = tmp.resolve("ptype-" + ptype + ".vtx"); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var sut = VortexWriter.create(ch, schema, opts)) { @@ -196,7 +196,7 @@ void everyPrimitiveType_emitsZonedLayout(PType ptype, @TempDir Path tmp) throws void noChunks_emitsNoZoneMap(@TempDir Path tmp) throws IOException { // Given a file closed without any writeChunk: the column has no chunks, so flushZoneMaps // skips it (the empty-chunks guard) and emits no zone-map. - WriteOptions opts = new WriteOptions(4, true, 0.90, 0, false, false, 256L * 1024 * 1024, java.util.Map.of()); + WriteOptions opts = new WriteOptions(4, true, 0.90, 0, false, false, 256L * 1024 * 1024, Map.of()); Path file = tmp.resolve("empty.vtx"); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var sut = VortexWriter.create(ch, SCHEMA, opts)) { @@ -217,7 +217,7 @@ void chunkWithoutStats_emitsNullCountOnlyZoneMap(@TempDir Path tmp) throws IOExc // — SUM is independent (the empty zone's sum is simply null). DType.Struct schema = new DType.Struct( List.of(ColumnName.of("v")), List.of(DType.I64), false); - WriteOptions opts = new WriteOptions(2, true, 0.90, 0, false, false, 256L * 1024 * 1024, java.util.Map.of()); + WriteOptions opts = new WriteOptions(2, true, 0.90, 0, false, false, 256L * 1024 * 1024, Map.of()); Path file = tmp.resolve("partial.vtx"); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var sut = VortexWriter.create(ch, schema, opts)) { @@ -241,7 +241,7 @@ void utf8Column_emitsStringMinMaxZoneMap(@TempDir Path tmp) throws IOException { // the zone-map carries MAX+MIN+NULL_COUNT (string min/max), not null_count alone. DType.Struct schema = new DType.Struct( List.of(ColumnName.of("s")), List.of(DType.UTF8), false); - WriteOptions opts = new WriteOptions(2, true, 0.90, 0, false, false, 256L * 1024 * 1024, java.util.Map.of()); + WriteOptions opts = new WriteOptions(2, true, 0.90, 0, false, false, 256L * 1024 * 1024, Map.of()); Path file = tmp.resolve("utf8.vtx"); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var sut = VortexWriter.create(ch, schema, opts)) { @@ -279,7 +279,7 @@ void extensionColumn_emitsStorageMinMaxZoneMap(@TempDir Path tmp) throws IOExcep DType ext = new DType.Extension( "test.ext", DType.I64, null, false); DType.Struct schema = new DType.Struct(List.of(ColumnName.of("t")), List.of(ext), false); - WriteOptions opts = new WriteOptions(2, true, 0.90, 0, false, false, 256L * 1024 * 1024, java.util.Map.of()); + WriteOptions opts = new WriteOptions(2, true, 0.90, 0, false, false, 256L * 1024 * 1024, Map.of()); Path file = tmp.resolve("ext.vtx"); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var sut = VortexWriter.create(ch, schema, opts)) { @@ -316,7 +316,7 @@ void dictColumn_emitsStringMinMaxZoneMapWrappingDict(@TempDir Path tmp) throws I // zone 1 = a..c → MAX+MIN+NULL_COUNT, with the column wrapped as vortex.stats over the dict. DType.Struct schema = new DType.Struct( List.of(ColumnName.of("s")), List.of(DType.UTF8), false); - WriteOptions opts = new WriteOptions(6, true, 0.90, 0, true, false, 256L * 1024 * 1024, java.util.Map.of()); + WriteOptions opts = new WriteOptions(6, true, 0.90, 0, true, false, 256L * 1024 * 1024, Map.of()); Path file = tmp.resolve("dict.vtx"); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var sut = VortexWriter.create(ch, schema, opts)) { @@ -353,7 +353,7 @@ void primitiveDictColumn_emitsNumericMinMaxZoneMapWrappingDict(@TempDir Path tmp // Zone-map min/max are computed on the logical I64 values: zone 0 = 1..2, zone 1 = 1..3. DType.Struct schema = new DType.Struct( List.of(ColumnName.of("v")), List.of(DType.I64), false); - WriteOptions opts = new WriteOptions(6, true, 0.90, 0, true, false, 256L * 1024 * 1024, java.util.Map.of()); + WriteOptions opts = new WriteOptions(6, true, 0.90, 0, true, false, 256L * 1024 * 1024, Map.of()); Path file = tmp.resolve("primdict.vtx"); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var sut = VortexWriter.create(ch, schema, opts)) { @@ -393,7 +393,7 @@ void zoneMaps_f64StatsPayloadDecodesPerZoneMinMaxSum(@TempDir Path tmp) throws I // a decode that returned 0.0 or read the wrong scalar field would slip through. Fractional // .5 values also make a truncating (int) decode observable. DType.Struct schema = new DType.Struct(List.of(ColumnName.of("v")), List.of(DType.F64), false); - WriteOptions opts = new WriteOptions(4, true, 0.90, 0, true, false, 256L * 1024 * 1024, java.util.Map.of()); + WriteOptions opts = new WriteOptions(4, true, 0.90, 0, true, false, 256L * 1024 * 1024, Map.of()); Path file = tmp.resolve("zoned-f64.vtx"); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var sut = VortexWriter.create(ch, schema, opts)) { @@ -434,7 +434,7 @@ void zoneMaps_perTypeStatsDecodePerZoneMinMax(PType ptype, @TempDir Path tmp) th // F32 statColumn arms (and the f32 scalar field read in scalarDouble), which the I64/F64 // value tests never reach. DType.Struct schema = new DType.Struct(List.of(ColumnName.of("v")), List.of(new DType.Primitive(ptype, false)), false); - WriteOptions opts = new WriteOptions(2, true, 0.90, 0, false, false, 256L * 1024 * 1024, java.util.Map.of()); + WriteOptions opts = new WriteOptions(2, true, 0.90, 0, false, false, 256L * 1024 * 1024, Map.of()); Path file = tmp.resolve("ptype-stats-" + ptype + ".vtx"); try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE); var sut = VortexWriter.create(ch, schema, opts)) {