From 5eb24330e721534d52f8d3c2c2260c54429daf5f Mon Sep 17 00:00:00 2001 From: "Claude (on behalf of Steven Schlansker)" Date: Thu, 28 May 2026 18:44:19 +0000 Subject: [PATCH 01/13] feat(format): schema evolution for the Java row codec Opt in with `.withSchemaEvolution()` on any row, array, or map codec builder. Fields carry `@ForyVersion(since, until)`; removed fields are listed on a nested interface referenced from `@ForySchema(removedFields = ...)`, which preserves parameterized types like `List`. Older payloads are dispatched at read time; nothing changes when the flag is off. Standard and compact formats supported; interface-typed beans included. --- docs/guide/java/row-format.md | 70 ++ .../fory/format/annotation/ForySchema.java | 69 ++ .../fory/format/annotation/ForyVersion.java | 44 ++ .../format/encoder/ArrayCodecBuilder.java | 119 ++- .../format/encoder/ArrayEncoderBuilder.java | 14 +- .../encoder/BaseBinaryEncoderBuilder.java | 69 +- .../fory/format/encoder/BaseCodecBuilder.java | 17 + .../format/encoder/BinaryArrayEncoder.java | 75 +- .../fory/format/encoder/BinaryMapEncoder.java | 80 +- .../fory/format/encoder/BinaryRowEncoder.java | 63 +- .../encoder/CompactArrayEncoderBuilder.java | 5 + .../format/encoder/CompactCodecFormat.java | 26 + .../encoder/CompactMapEncoderBuilder.java | 5 + .../encoder/CompactRowEncoderBuilder.java | 8 + .../format/encoder/DefaultCodecFormat.java | 26 + .../apache/fory/format/encoder/Encoders.java | 77 ++ .../apache/fory/format/encoder/Encoding.java | 27 + .../fory/format/encoder/MapCodecBuilder.java | 122 ++- .../format/encoder/MapEncoderBuilder.java | 9 +- .../fory/format/encoder/RowCodecBuilder.java | 118 ++- .../format/encoder/RowEncoderBuilder.java | 168 +++- .../fory/format/type/SchemaHistory.java | 393 ++++++++++ .../fory/format/type/TypeInference.java | 10 + .../encoder/SchemaEvolutionStressTest.java | 736 ++++++++++++++++++ .../format/encoder/SchemaEvolutionTest.java | 496 ++++++++++++ 25 files changed, 2704 insertions(+), 142 deletions(-) create mode 100644 java/fory-format/src/main/java/org/apache/fory/format/annotation/ForySchema.java create mode 100644 java/fory-format/src/main/java/org/apache/fory/format/annotation/ForyVersion.java create mode 100644 java/fory-format/src/main/java/org/apache/fory/format/type/SchemaHistory.java create mode 100644 java/fory-format/src/test/java/org/apache/fory/format/encoder/SchemaEvolutionStressTest.java create mode 100644 java/fory-format/src/test/java/org/apache/fory/format/encoder/SchemaEvolutionTest.java diff --git a/docs/guide/java/row-format.md b/docs/guide/java/row-format.md index 477f9ec136..48ba35872c 100644 --- a/docs/guide/java/row-format.md +++ b/docs/guide/java/row-format.md @@ -113,6 +113,76 @@ Row format is ideal for: - **Data pipelines**: Processing data without full object reconstruction - **Cross-language data sharing**: When data needs to be accessed from multiple languages +## Schema evolution + +Enable `.withSchemaEvolution()` on a row, array, or map codec builder to read payloads written +by older versions of the same bean. Writing always uses the current version; reading detects +the payload's version from a strict hash at the head of the payload. Java only. + +Annotate fields added after v1 with `@ForyVersion(since = N)`: + +```java +@Data +public class Person { + private String name; + private int age; + + @ForyVersion(since = 2) + private String email; +} +``` + +A v1 payload (with `name` and `age` only) decodes to a `Person` whose `email` is `null`. +Primitive fields added later default to `0` / `false`. If a class adopts versioning after its +v1 is already in the wild, set `@ForySchema(baseVersion = N)` so unannotated fields are +treated as present since version `N`. + +Remove a field by deleting the Java member and listing it on a nested history interface. The +interface's methods carry the original field's name, return type, and `[since, until)` window. +Parameterized types are expressed naturally because the methods are real Java declarations. + +```java +@Data +@ForySchema(removedFields = Person.History.class) +public class Person { + private String name; + + @ForyVersion(since = 2) + private String email; + + interface History { + @ForyVersion(until = 3) + int age(); + + @ForyVersion(until = 5) + List tags(); + } +} +``` + +Each history method must carry a `@ForyVersion` with `until` set. The method name matches the +original live descriptor name: the field name for Lombok `@Data` or record-style classes +(`age`, `tags`), or the full accessor name for JavaBeans-style classes and interfaces +(`getAge`). + +### Wire format and limitations + +Producers and consumers must agree on the `withSchemaEvolution()` flag — they are not +wire-compatible otherwise. Row payloads already carry an 8-byte hash slot whose value changes +under evolution (the strict hash includes field name and nullability). For arrays and maps +whose element bean opts into evolution, an 8-byte hash prefix is prepended; arrays and maps +whose element is not a versioned bean carry no prefix. + +Cross-language consumers (Python, C++) cannot read evolution-enabled payloads. + +Map keys do not carry a per-payload hash; a versioned bean used as a map key is read with the +current schema only, not dispatched to a projection codec. + +A versioned bean nested as a struct field inside another versioned bean is read with its +current schema regardless of what the wire bytes were written from — the row format does not +carry a per-nested-struct hash. Evolve either the outer or the nested bean, but expect the +nested-bean schema to remain stable while the outer evolves (or vice versa). + ## Cross-Language Compatibility Row format works seamlessly across languages. The same binary data can be accessed from: diff --git a/java/fory-format/src/main/java/org/apache/fory/format/annotation/ForySchema.java b/java/fory-format/src/main/java/org/apache/fory/format/annotation/ForySchema.java new file mode 100644 index 0000000000..c1807b905a --- /dev/null +++ b/java/fory-format/src/main/java/org/apache/fory/format/annotation/ForySchema.java @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.fory.format.annotation; + +import java.lang.annotation.ElementType; +import java.lang.annotation.Retention; +import java.lang.annotation.RetentionPolicy; +import java.lang.annotation.Target; + +/** + * Class-level row-codec schema metadata used when the codec builder enables schema evolution. + * + *

{@link #baseVersion()} sets the implicit {@code since} for live fields that lack a + * {@link ForyVersion} annotation; this lets a class adopt versioning mid-history without having + * to annotate every existing member. + * + *

{@link #removedFields()} points at a class (conventionally a nested {@code interface}) whose + * accessor methods describe fields that have been removed from this bean but still appear on the + * wire in older payloads. Each method's return type is the original Java type of the removed + * field; each method must carry a {@link ForyVersion} annotation with {@code until} set, since + * removed fields have a known end-of-life version. + * + *

Example: + * + * 

{@code
+ * @Data
+ * @ForySchema(removedFields = MyBean.History.class)
+ * public class MyBean {
+ *   private String name;
+ *
+ *   interface History {
+ *     @ForyVersion(until = 3)
+ *     List tags();
+ *
+ *     @ForyVersion(since = 2, until = 5)
+ *     Map counters();
+ *   }
+ * }
+ * }
+ */ +@Retention(RetentionPolicy.RUNTIME) +@Target(ElementType.TYPE) +public @interface ForySchema { + int baseVersion() default 1; + + /** + * A class whose accessor methods describe historically-present-but-now-removed fields. Default + * {@code void.class} means there are no removed fields. The class is never instantiated; the + * codec reads its method signatures and annotations. + */ + Class removedFields() default void.class; +} diff --git a/java/fory-format/src/main/java/org/apache/fory/format/annotation/ForyVersion.java b/java/fory-format/src/main/java/org/apache/fory/format/annotation/ForyVersion.java new file mode 100644 index 0000000000..feb2af8913 --- /dev/null +++ b/java/fory-format/src/main/java/org/apache/fory/format/annotation/ForyVersion.java @@ -0,0 +1,44 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.fory.format.annotation; + +import java.lang.annotation.ElementType; +import java.lang.annotation.Retention; +import java.lang.annotation.RetentionPolicy; +import java.lang.annotation.Target; + +/** + * Declares the version window in which a row-codec field is logically present. The window is + * inclusive on the left and exclusive on the right, so {@code since=2, until=5} means versions 2, + * 3, and 4. + * + *

Only effective when the codec builder is configured with + * {@code withSchemaEvolution()}; otherwise the annotation is ignored and the field is treated as + * always present. + */ +@Retention(RetentionPolicy.RUNTIME) +@Target({ElementType.FIELD, ElementType.METHOD, ElementType.RECORD_COMPONENT}) +public @interface ForyVersion { + /** First version (inclusive) that contains this field. Defaults to the class base version. */ + int since() default 1; + + /** First version (exclusive) that no longer contains this field. */ + int until() default Integer.MAX_VALUE; +} diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/ArrayCodecBuilder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/ArrayCodecBuilder.java index 6e6c6d3645..fb464082f7 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/ArrayCodecBuilder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/ArrayCodecBuilder.java @@ -22,18 +22,25 @@ import static org.apache.fory.type.TypeUtils.getRawType; import java.lang.invoke.MethodHandle; -import java.lang.invoke.MethodHandles; -import java.lang.invoke.MethodType; import java.util.Collection; +import java.util.HashMap; import java.util.HashSet; +import java.util.Map; import java.util.Set; import java.util.function.Function; import java.util.function.Supplier; +import java.util.function.UnaryOperator; +import org.apache.fory.Fory; import org.apache.fory.format.row.binary.writer.BinaryArrayWriter; +import org.apache.fory.format.row.binary.writer.CompactBinaryRowWriter; +import org.apache.fory.format.type.CustomTypeEncoderRegistry; import org.apache.fory.format.type.DataTypes; import org.apache.fory.format.type.Field; +import org.apache.fory.format.type.Schema; +import org.apache.fory.format.type.SchemaHistory; import org.apache.fory.format.type.TypeInference; import org.apache.fory.reflect.TypeRef; +import org.apache.fory.type.TypeResolutionContext; import org.apache.fory.type.TypeUtils; import org.apache.fory.util.ExceptionUtils; @@ -63,17 +70,100 @@ public ArrayEncoder get() { Function> buildWithWriter() { loadArrayInnerCodecs(); - final Function generatedEncoderFactory = + if (!schemaEvolution || !isVersionedBeanElement()) { + final Function generatedEncoderFactory = + generatedEncoderFactory(); + return new Function>() { + @Override + public ArrayEncoder apply(final BinaryArrayWriter writer) { + return new BinaryArrayEncoder<>( + writer, generatedEncoderFactory.apply(writer), sizeEmbedded); + } + }; + } + return buildVersionedWithWriter(); + } + + private boolean isVersionedBeanElement() { + Class elementClass = getRawType(TypeUtils.getElementType(collectionType)); + // Use the same resolution context as the row-format type inference, which synthesizes + // interface-typed bean fields. Without this, classes that contain interface members + // would not be recognized as beans even though the row codec can encode them. + return TypeUtils.isBean( + TypeRef.of(elementClass), + new TypeResolutionContext(CustomTypeEncoderRegistry.customTypeHandler(), true)); + } + + private Function> buildVersionedWithWriter() { + Class elementClass = getRawType(TypeUtils.getElementType(collectionType)); + UnaryOperator schemaTransform = + codecFormat == CompactCodecFormat.INSTANCE + ? CompactBinaryRowWriter::sortSchema + : UnaryOperator.identity(); + SchemaHistory history = SchemaHistory.build(elementClass, schemaTransform); + SchemaHistory.VersionedSchema current = history.current(); + + // Make sure the current-version row codec class is generated. + Encoders.loadOrGenRowCodecClass(elementClass, codecFormat); + // Generate per-version row codec classes and per-version array codec classes. + Map projectionFactories = new HashMap<>(); + for (SchemaHistory.VersionedSchema vs : history.versions()) { + if (vs == current) { + continue; + } + String suffix = "_V" + vs.version(); + Encoders.loadOrGenProjectionRowCodecClass( + elementClass, codecFormat, vs.schema(), vs.liveFieldNames(), suffix); + Class arrayClass = + Encoders.loadOrGenProjectionArrayCodecClass( + collectionType, TypeRef.of(elementClass), codecFormat, suffix); + MethodHandle ctor = Encoders.constructorHandleFor(arrayClass, GeneratedArrayEncoder.class); + // The array's "elementField" is a ListType whose valueField is the element struct. Build + // a parallel ListType for this historical version so the projection codec can produce a + // BinaryArray with the right element width. + Field histValueField = + DataTypes.field( + DataTypes.ARRAY_ITEM_NAME, new DataTypes.StructType(vs.schema().fields()), true); + Field histListField = DataTypes.arrayField(elementField.name(), histValueField); + projectionFactories.put(vs.strictHash(), new ProjectionArrayFactory(histListField, ctor)); + } + final Function currentFactory = generatedEncoderFactory(); + long currentHash = current.strictHash(); return new Function>() { @Override public ArrayEncoder apply(final BinaryArrayWriter writer) { + Map proj = new HashMap<>(); + for (Map.Entry entry : projectionFactories.entrySet()) { + proj.put(entry.getKey(), entry.getValue().instantiate(fory)); + } return new BinaryArrayEncoder<>( - writer, generatedEncoderFactory.apply(writer), sizeEmbedded); + writer, currentFactory.apply(writer), sizeEmbedded, currentHash, proj); } }; } + private final class ProjectionArrayFactory { + private final Field elementField; + private final MethodHandle ctor; + + ProjectionArrayFactory(Field elementField, MethodHandle ctor) { + this.elementField = elementField; + this.ctor = ctor; + } + + BinaryArrayEncoder.ProjectionArrayCodec instantiate(Fory fory) { + try { + BinaryArrayWriter projWriter = codecFormat.newArrayWriter(elementField); + Object[] references = {elementField, projWriter, fory}; + GeneratedArrayEncoder codec = (GeneratedArrayEncoder) ctor.invokeExact(references); + return new BinaryArrayEncoder.ProjectionArrayCodec(projWriter, codec); + } catch (Throwable e) { + throw ExceptionUtils.throwException(e); + } + } + } + private void loadArrayInnerCodecs() { final Set> set = new HashSet<>(); Encoders.findBeanToken(collectionType, set); @@ -90,30 +180,15 @@ Function generatedEncoderFactory() { final TypeRef elementType = TypeUtils.getElementType(collectionType); final Class arrayCodecClass = Encoders.loadOrGenArrayCodecClass(collectionType, elementType, codecFormat); - - final MethodHandle constructorHandle; - try { - final var constructor = - arrayCodecClass.asSubclass(GeneratedArrayEncoder.class).getConstructor(Object[].class); - constructorHandle = - MethodHandles.lookup() - .unreflectConstructor(constructor) - .asType(MethodType.methodType(GeneratedArrayEncoder.class, Object[].class)); - } catch (final NoSuchMethodException | IllegalAccessException e) { - throw new EncoderException( - "Failed to construct array codec for " - + collectionType - + " with element class " - + elementType, - e); - } + final MethodHandle constructorHandle = + Encoders.constructorHandleFor(arrayCodecClass, GeneratedArrayEncoder.class); return new Function() { @Override public GeneratedArrayEncoder apply(final BinaryArrayWriter writer) { final Object[] references = {writer.getField(), writer, fory}; try { return (GeneratedArrayEncoder) constructorHandle.invokeExact(references); - } catch (final Throwable t) { + } catch (Throwable t) { throw ExceptionUtils.throwException(t); } } diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/ArrayEncoderBuilder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/ArrayEncoderBuilder.java index c24611cd82..3ff8139c80 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/ArrayEncoderBuilder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/ArrayEncoderBuilder.java @@ -54,7 +54,17 @@ public ArrayEncoderBuilder(Class arrayCls, Class beanClass) { } public ArrayEncoderBuilder(TypeRef clsType, TypeRef beanType) { + this(clsType, beanType, null); + } + + /** + * Construct an array codec builder that embeds row codec class references for its element bean + * with the supplied suffix. Used by schema-evolution code to point per-version array codecs at + * per-version row codecs. + */ + ArrayEncoderBuilder(TypeRef clsType, TypeRef beanType, String rowCodecSuffix) { super(new CodegenContext(), beanType); + this.rowCodecSuffixForBeans = rowCodecSuffix; arrayToken = clsType; ctx.reserveName(ROOT_ARRAY_WRITER_NAME); ctx.reserveName(ROOT_ARRAY_NAME); @@ -83,7 +93,9 @@ public ArrayEncoderBuilder(TypeRef clsType, TypeRef beanType) { @Override public String genCode() { ctx.setPackage(CodeGenerator.getPackage(beanClass)); - String className = codecClassName(beanClass, TypeInference.inferTypeName(arrayToken)); + String className = + codecClassName(beanClass, TypeInference.inferTypeName(arrayToken)) + + (rowCodecSuffixForBeans == null ? "" : rowCodecSuffixForBeans); ctx.setClassName(className); // don't addImport(arrayClass), because user class may name collide. // janino don't support generics, so GeneratedCodec has no generics diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/BaseBinaryEncoderBuilder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/BaseBinaryEncoderBuilder.java index 2d214e287e..111d55a192 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/BaseBinaryEncoderBuilder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/BaseBinaryEncoderBuilder.java @@ -96,6 +96,12 @@ public abstract class BaseBinaryEncoderBuilder extends CodecBuilder { protected final Map, Reference> arrayWriterMap = new HashMap<>(); protected final Map, Reference> beanEncoderMap = new HashMap<>(); + /** + * When non-null, nested bean codec class references generated by this builder will be suffixed + * with this string. Used by schema-evolution code paths to direct generated array/map codecs + * to the projection variant of an element bean's row codec. + */ + protected String rowCodecSuffixForBeans; // We need to call beanEncoder's rowWriter.reset() before write a corresponding nested bean every // time. // Outermost beanEncoder's rowWriter.reset() should be called outside generated code before @@ -481,34 +487,9 @@ protected Expression serializeForBean( Field fieldIfKnown, TypeRef typeRef, Expression structField) { - Class rawType = getRawType(typeRef); - Reference rowWriter; + registerBeanCodec(writer, typeRef, structField); + Reference rowWriter = rowWriterMap.get(typeRef); Reference beanEncoder = beanEncoderMap.get(typeRef); - if (beanEncoder == null) { - // janino generics don't add cast, so this `<${type}>` is only for generated code readability - Expression schema = createSchemaFromStructField(structField); - String rowWriterName = - ctx.newName(StringUtils.uncapitalize(rawType.getSimpleName() + "RowWriter")); - NewInstance newRowWriter = new NewInstance(rowWriterType(), schema, writer); - ctx.addField(ctx.type(rowWriterType()), rowWriterName, newRowWriter); - - Preconditions.checkArgument(!codecClassName(rawType).contains(".")); - String encoderName = ctx.newName(StringUtils.uncapitalize(codecClassName(rawType))); - String encoderClass = codecQualifiedClassName(rawType); - TypeRef codecTypeRef = TypeRef.of(GeneratedRowEncoder.class); - NewInstance newEncoder = - new NewInstance( - codecTypeRef, - encoderClass, - ExpressionUtils.newObjectArray(schema, newRowWriter, foryRef)); - ctx.addField(encoderClass, encoderName, newEncoder); - - rowWriter = new Reference(rowWriterName, rowWriterType()); - rowWriterMap.put(typeRef, rowWriter); - beanEncoder = new Reference(encoderName, codecTypeRef); - beanEncoderMap.put(typeRef, beanEncoder); - } - rowWriter = rowWriterMap.get(typeRef); Expression expression = serializeForNotNullBean(ordinal, writer, inputObject, fieldIfKnown, rowWriter, beanEncoder); @@ -517,6 +498,40 @@ protected Expression serializeForBean( new Expression.IsNull(inputObject), new Invoke(writer, "setNullAt", ordinal), expression); } + /** + * Idempotently add the nested-bean row writer and row encoder as fields on the generated codec + * class and register them in {@link #beanEncoderMap} and {@link #rowWriterMap}. Used both by + * {@link #serializeForBean} and by decode-only projection codegen, where the encode pass is + * skipped but the decode pass still needs the bean encoder reference. + */ + protected void registerBeanCodec(Expression writer, TypeRef typeRef, Expression structField) { + if (beanEncoderMap.containsKey(typeRef)) { + return; + } + Class rawType = getRawType(typeRef); + Expression schema = createSchemaFromStructField(structField); + String rowWriterName = + ctx.newName(StringUtils.uncapitalize(rawType.getSimpleName() + "RowWriter")); + NewInstance newRowWriter = new NewInstance(rowWriterType(), schema, writer); + ctx.addField(ctx.type(rowWriterType()), rowWriterName, newRowWriter); + + Preconditions.checkArgument(!codecClassName(rawType).contains(".")); + String encoderName = ctx.newName(StringUtils.uncapitalize(codecClassName(rawType))); + String encoderClass = + codecQualifiedClassName(rawType) + + (rowCodecSuffixForBeans == null ? "" : rowCodecSuffixForBeans); + TypeRef codecTypeRef = TypeRef.of(GeneratedRowEncoder.class); + NewInstance newEncoder = + new NewInstance( + codecTypeRef, + encoderClass, + ExpressionUtils.newObjectArray(schema, newRowWriter, foryRef)); + ctx.addField(encoderClass, encoderName, newEncoder); + + rowWriterMap.put(typeRef, new Reference(rowWriterName, rowWriterType())); + beanEncoderMap.put(typeRef, new Reference(encoderName, codecTypeRef)); + } + protected Expression createSchemaFromStructField(Expression structField) { return new StaticInvoke( DataTypes.class, "schemaFromStructField", "schema", SCHEMA_TYPE, false, structField); diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/BaseCodecBuilder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/BaseCodecBuilder.java index 81f78ca247..72463c8a21 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/BaseCodecBuilder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/BaseCodecBuilder.java @@ -30,6 +30,7 @@ public class BaseCodecBuilder> { protected boolean sizeEmbedded = true; protected Fory fory; protected Encoding codecFormat = DefaultCodecFormat.INSTANCE; + protected boolean schemaEvolution = false; BaseCodecBuilder(final Schema schema) { this.schema = schema; @@ -58,6 +59,22 @@ public B withSizeEmbedded(final boolean sizeEmbedded) { return castThis(); } + /** + * Enable schema evolution. The codec accepts payloads written by older versions of the same + * bean, using the {@link org.apache.fory.format.annotation.ForyVersion} and + * {@link org.apache.fory.format.annotation.ForySchema} annotations to reconstruct historical + * schemas. Writing always uses the current version. + * + *

For array and map codecs, this changes the wire format by adding an 8-byte strict-hash + * prefix to the payload, so producers and consumers must agree on the flag. Row payloads + * already carry an 8-byte hash slot; under schema evolution that slot is computed with a + * stricter hash that also distinguishes field names and nullability. + */ + public B withSchemaEvolution() { + this.schemaEvolution = true; + return castThis(); + } + /** * Configure compact encoding, which is more space efficient than the default encoding, but is not * yet stable. See {@link CompactBinaryRow} for details. diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryArrayEncoder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryArrayEncoder.java index f40a968aa8..c42b6c3215 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryArrayEncoder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryArrayEncoder.java @@ -19,6 +19,8 @@ package org.apache.fory.format.encoder; +import java.util.Map; +import org.apache.fory.exception.ClassNotCompatibleException; import org.apache.fory.format.row.binary.BinaryArray; import org.apache.fory.format.row.binary.writer.BinaryArrayWriter; import org.apache.fory.format.type.Field; @@ -29,14 +31,43 @@ class BinaryArrayEncoder implements ArrayEncoder { private final BinaryArrayWriter writer; private final GeneratedArrayEncoder codec; private final boolean sizeEmbedded; + /** Strict hash of the element bean's current schema; written before the array payload when {@code schemaEvolution} is on. */ + private final long currentHash; + /** Per-version projection codecs and their element fields. {@code null} disables versioning. */ + private final Map projections; + + /** + * A projection variant of the array codec along with the writer used to materialize an array + * instance of the right physical type (standard vs. compact) for the historical element field. + */ + static final class ProjectionArrayCodec { + final BinaryArrayWriter writer; + final GeneratedArrayEncoder codec; + + ProjectionArrayCodec(BinaryArrayWriter writer, GeneratedArrayEncoder codec) { + this.writer = writer; + this.codec = codec; + } + } BinaryArrayEncoder( final BinaryArrayWriter writer, final GeneratedArrayEncoder codec, final boolean sizeEmbedded) { + this(writer, codec, sizeEmbedded, 0L, null); + } + + BinaryArrayEncoder( + final BinaryArrayWriter writer, + final GeneratedArrayEncoder codec, + final boolean sizeEmbedded, + final long currentHash, + final Map projections) { this.writer = writer; this.codec = codec; this.sizeEmbedded = sizeEmbedded; + this.currentHash = currentHash; + this.projections = projections; } @Override @@ -65,18 +96,49 @@ public T decode(final byte[] bytes) { return decode(MemoryUtils.wrap(bytes), bytes.length); } + @SuppressWarnings("unchecked") T decode(final MemoryBuffer buffer, final int size) { - final BinaryArray array = writer.newArray(); + if (projections == null) { + final BinaryArray array = writer.newArray(); + final int readerIndex = buffer.readerIndex(); + array.pointTo(buffer, readerIndex, size); + buffer.readerIndex(readerIndex + size); + return fromArray(array); + } + final long peerHash = buffer.readInt64(); + final int payloadSize = size - 8; + if (peerHash == currentHash) { + final BinaryArray array = writer.newArray(); + final int readerIndex = buffer.readerIndex(); + array.pointTo(buffer, readerIndex, payloadSize); + buffer.readerIndex(readerIndex + payloadSize); + return fromArray(array); + } + ProjectionArrayCodec projection = projections.get(peerHash); + if (projection == null) { + throw new ClassNotCompatibleException( + String.format( + "Array element schema is not consistent. self/peer hash are %s/%s.", + currentHash, peerHash)); + } + BinaryArray array = projection.writer.newArray(); final int readerIndex = buffer.readerIndex(); - array.pointTo(buffer, readerIndex, size); - buffer.readerIndex(readerIndex + size); - return fromArray(array); + array.pointTo(buffer, readerIndex, payloadSize); + buffer.readerIndex(readerIndex + payloadSize); + return (T) projection.codec.fromArray(array); } @Override public byte[] encode(final T obj) { final BinaryArray array = toArray(obj); - return writer.getBuffer().getBytes(0, array.getSizeInBytes()); + if (projections == null) { + return writer.getBuffer().getBytes(0, array.getSizeInBytes()); + } + int n = array.getSizeInBytes(); + MemoryBuffer out = MemoryUtils.buffer(8 + n); + out.writeInt64(currentHash); + out.writeBytes(writer.getBuffer().getBytes(0, n)); + return out.getBytes(0, 8 + n); } @Override @@ -86,6 +148,9 @@ public int encode(final MemoryBuffer buffer, final T obj) { if (sizeEmbedded) { buffer.writeInt32(-1); } + if (projections != null) { + buffer.writeInt64(currentHash); + } try { writer.setBuffer(buffer); toArray(obj); diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryMapEncoder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryMapEncoder.java index 925bdc332b..5f3b5a88b7 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryMapEncoder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryMapEncoder.java @@ -19,6 +19,8 @@ package org.apache.fory.format.encoder; +import java.util.Map; +import org.apache.fory.exception.ClassNotCompatibleException; import org.apache.fory.format.row.binary.BinaryArray; import org.apache.fory.format.row.binary.BinaryMap; import org.apache.fory.format.row.binary.writer.BinaryArrayWriter; @@ -33,6 +35,24 @@ class BinaryMapEncoder implements MapEncoder { private final BinaryArrayWriter keyWriter; private final GeneratedMapEncoder codec; private final boolean sizeEmbedded; + private final long currentHash; + private final Map projections; + + /** + * Per-version projection codec; the {@code Encoding} and historical {@code mapField} together + * materialize an empty map shaped for the historical layout (standard vs. compact). + */ + static final class ProjectionMapCodec { + final Encoding format; + final Field mapField; + final GeneratedMapEncoder codec; + + ProjectionMapCodec(Encoding format, Field mapField, GeneratedMapEncoder codec) { + this.format = format; + this.mapField = mapField; + this.codec = codec; + } + } BinaryMapEncoder( final Encoding format, @@ -41,12 +61,26 @@ class BinaryMapEncoder implements MapEncoder { final BinaryArrayWriter keyWriter, final GeneratedMapEncoder codec, final boolean sizeEmbedded) { + this(format, mapField, valWriter, keyWriter, codec, sizeEmbedded, 0L, null); + } + + BinaryMapEncoder( + final Encoding format, + final Field mapField, + final BinaryArrayWriter valWriter, + final BinaryArrayWriter keyWriter, + final GeneratedMapEncoder codec, + final boolean sizeEmbedded, + final long currentHash, + final Map projections) { this.format = format; this.mapField = mapField; this.valWriter = valWriter; this.keyWriter = keyWriter; this.codec = codec; this.sizeEmbedded = sizeEmbedded; + this.currentHash = currentHash; + this.projections = projections; } @Override @@ -75,12 +109,36 @@ public M decode(final MemoryBuffer buffer) { return decode(buffer, sizeEmbedded ? buffer.readInt32() : buffer.remaining()); } + @SuppressWarnings("unchecked") M decode(final MemoryBuffer buffer, final int size) { - final BinaryMap map = format.newMap(mapField); - final int readerIndex = buffer.readerIndex(); - map.pointTo(buffer, readerIndex, size); - buffer.readerIndex(readerIndex + size); - return fromMap(map); + if (projections == null) { + final BinaryMap map = format.newMap(mapField); + final int readerIndex = buffer.readerIndex(); + map.pointTo(buffer, readerIndex, size); + buffer.readerIndex(readerIndex + size); + return fromMap(map); + } + long peerHash = buffer.readInt64(); + int payloadSize = size - 8; + if (peerHash == currentHash) { + final BinaryMap map = format.newMap(mapField); + int readerIndex = buffer.readerIndex(); + map.pointTo(buffer, readerIndex, payloadSize); + buffer.readerIndex(readerIndex + payloadSize); + return fromMap(map); + } + ProjectionMapCodec projection = projections.get(peerHash); + if (projection == null) { + throw new ClassNotCompatibleException( + String.format( + "Map bean schema is not consistent. self/peer hash are %s/%s.", + currentHash, peerHash)); + } + BinaryMap map = projection.format.newMap(projection.mapField); + int readerIndex = buffer.readerIndex(); + map.pointTo(buffer, readerIndex, payloadSize); + buffer.readerIndex(readerIndex + payloadSize); + return (M) projection.codec.fromMap(map); } @Override @@ -91,7 +149,14 @@ public M decode(final byte[] bytes) { @Override public byte[] encode(final M obj) { final BinaryMap map = toMap(obj); - return map.getBuf().getBytes(map.getBaseOffset(), map.getSizeInBytes()); + if (projections == null) { + return map.getBuf().getBytes(map.getBaseOffset(), map.getSizeInBytes()); + } + int n = map.getSizeInBytes(); + MemoryBuffer out = MemoryUtils.buffer(8 + n); + out.writeInt64(currentHash); + out.writeBytes(map.getBuf().getBytes(map.getBaseOffset(), n)); + return out.getBytes(0, 8 + n); } @Override @@ -101,6 +166,9 @@ public int encode(final MemoryBuffer buffer, final M obj) { if (sizeEmbedded) { buffer.writeInt32(-1); } + if (projections != null) { + buffer.writeInt64(currentHash); + } try { keyWriter.setBuffer(buffer); valWriter.setBuffer(buffer); diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryRowEncoder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryRowEncoder.java index b21bff49e9..83d09069d4 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryRowEncoder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryRowEncoder.java @@ -19,6 +19,7 @@ package org.apache.fory.format.encoder; +import java.util.Map; import org.apache.fory.exception.ClassNotCompatibleException; import org.apache.fory.format.row.binary.BinaryRow; import org.apache.fory.format.row.binary.writer.BaseBinaryRowWriter; @@ -34,20 +35,48 @@ class BinaryRowEncoder implements RowEncoder { private final BaseBinaryRowWriter writer; private final boolean sizeEmbedded; private final long schemaHash; + /** + * Hash → (historical schema, projection codec) for older versions. {@code null} when schema + * evolution is disabled; in that case a hash mismatch is a hard error. + */ + private final Map projections; private final MemoryBuffer buffer = MemoryUtils.buffer(16); + /** Pairing of a historical schema with the projection codec that reads it. */ + static final class ProjectionCodec { + final Schema schema; + final GeneratedRowEncoder codec; + + ProjectionCodec(Schema schema, GeneratedRowEncoder codec) { + this.schema = schema; + this.codec = codec; + } + } + BinaryRowEncoder( final Schema schema, final Encoding codecFactory, final GeneratedRowEncoder codec, final BaseBinaryRowWriter writer, final boolean sizeEmbedded) { + this(schema, codecFactory, codec, writer, sizeEmbedded, DataTypes.computeSchemaHash(schema), null); + } + + BinaryRowEncoder( + final Schema schema, + final Encoding codecFactory, + final GeneratedRowEncoder codec, + final BaseBinaryRowWriter writer, + final boolean sizeEmbedded, + final long schemaHash, + final Map projections) { this.schema = schema; this.codecFactory = codecFactory; this.codec = codec; this.writer = writer; this.sizeEmbedded = sizeEmbedded; - this.schemaHash = DataTypes.computeSchemaHash(schema); + this.schemaHash = schemaHash; + this.projections = projections; } @Override @@ -71,20 +100,30 @@ public T decode(final MemoryBuffer buffer) { return decode(buffer, sizeEmbedded ? buffer.readInt32() : buffer.remaining()); } + @SuppressWarnings("unchecked") T decode(final MemoryBuffer buffer, final int size) { final long peerSchemaHash = buffer.readInt64(); - if (peerSchemaHash != schemaHash) { - throw new ClassNotCompatibleException( - String.format( - "Schema is not consistent, encoder schema is %s. " - + "self/peer schema hash are %s/%s. " - + "Please check writer schema.", - schema, schemaHash, peerSchemaHash)); + if (peerSchemaHash == schemaHash) { + final BinaryRow row = codecFactory.newRow(schema); + row.pointTo(buffer, buffer.readerIndex(), size); + buffer.increaseReaderIndex(size - 8); + return fromRow(row); + } + if (projections != null) { + ProjectionCodec projection = projections.get(peerSchemaHash); + if (projection != null) { + final BinaryRow row = codecFactory.newRow(projection.schema); + row.pointTo(buffer, buffer.readerIndex(), size); + buffer.increaseReaderIndex(size - 8); + return (T) projection.codec.fromRow(row); + } } - final BinaryRow row = codecFactory.newRow(schema); - row.pointTo(buffer, buffer.readerIndex(), size); - buffer.increaseReaderIndex(size - 8); - return fromRow(row); + throw new ClassNotCompatibleException( + String.format( + "Schema is not consistent, encoder schema is %s. " + + "self/peer schema hash are %s/%s. " + + "Please check writer schema.", + schema, schemaHash, peerSchemaHash)); } @Override diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/CompactArrayEncoderBuilder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/CompactArrayEncoderBuilder.java index 65f8508e35..b6a659c00e 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/CompactArrayEncoderBuilder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/CompactArrayEncoderBuilder.java @@ -33,6 +33,11 @@ public CompactArrayEncoderBuilder(final TypeRef clsType, final TypeRef bea super(clsType, beanType); } + CompactArrayEncoderBuilder( + final TypeRef clsType, final TypeRef beanType, final String rowCodecSuffix) { + super(clsType, beanType, rowCodecSuffix); + } + @Override protected Invoke beanWriterReset( final Expression writer, final Reference rowWriter, final Expression ordinal) { diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/CompactCodecFormat.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/CompactCodecFormat.java index 4c18e96798..435db5acb4 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/CompactCodecFormat.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/CompactCodecFormat.java @@ -21,6 +21,7 @@ import java.util.Collection; import java.util.Map; +import java.util.Set; import org.apache.fory.format.row.binary.BinaryArray; import org.apache.fory.format.row.binary.BinaryMap; import org.apache.fory.format.row.binary.BinaryRow; @@ -64,18 +65,43 @@ public RowEncoderBuilder newRowEncoder(final TypeRef beanType) { return new CompactRowEncoderBuilder(beanType); } + @Override + public RowEncoderBuilder newProjectionRowEncoder( + final TypeRef beanType, + final Schema historicalSchema, + final Set liveNames, + final String classSuffix) { + return new CompactRowEncoderBuilder(beanType, historicalSchema, liveNames, classSuffix); + } + @Override public ArrayEncoderBuilder newArrayEncoder( final TypeRef> collectionType, final TypeRef elementType) { return new CompactArrayEncoderBuilder(collectionType, elementType); } + @Override + public ArrayEncoderBuilder newProjectionArrayEncoder( + final TypeRef> collectionType, + final TypeRef elementType, + final String rowCodecSuffix) { + return new CompactArrayEncoderBuilder(collectionType, elementType, rowCodecSuffix); + } + @Override public MapEncoderBuilder newMapEncoder( final TypeRef> mapType, final TypeRef beanToken) { return new CompactMapEncoderBuilder(mapType, beanToken); } + @Override + public MapEncoderBuilder newProjectionMapEncoder( + final TypeRef> mapType, + final TypeRef beanToken, + final String rowCodecSuffix) { + return new CompactMapEncoderBuilder(mapType, beanToken, rowCodecSuffix); + } + @Override public BinaryRow newRow(final Schema schema) { return new CompactBinaryRow(schema); diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/CompactMapEncoderBuilder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/CompactMapEncoderBuilder.java index be3d206d59..7a55f54881 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/CompactMapEncoderBuilder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/CompactMapEncoderBuilder.java @@ -36,6 +36,11 @@ public CompactMapEncoderBuilder(final TypeRef clsType, final TypeRef beanT super(clsType, beanType); } + CompactMapEncoderBuilder( + final TypeRef clsType, final TypeRef beanType, final String rowCodecSuffix) { + super(clsType, beanType, rowCodecSuffix); + } + @Override protected Invoke beanWriterReset( final Expression writer, final Reference rowWriter, final Expression ordinal) { diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/CompactRowEncoderBuilder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/CompactRowEncoderBuilder.java index 79ccc53391..828bdc9e43 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/CompactRowEncoderBuilder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/CompactRowEncoderBuilder.java @@ -41,6 +41,14 @@ public CompactRowEncoderBuilder(final TypeRef beanType) { super(beanType); } + CompactRowEncoderBuilder( + final TypeRef beanType, + final Schema historicalSchema, + final java.util.Set liveNames, + final String classSuffix) { + super(beanType, historicalSchema, liveNames, classSuffix); + } + @Override protected Schema inferSchema(final TypeRef beanType) { return CompactBinaryRowWriter.sortSchema(super.inferSchema(beanType)); diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/DefaultCodecFormat.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/DefaultCodecFormat.java index 78f42e69b7..f4ab67fe6f 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/DefaultCodecFormat.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/DefaultCodecFormat.java @@ -21,6 +21,7 @@ import java.util.Collection; import java.util.Map; +import java.util.Set; import org.apache.fory.format.row.binary.BinaryArray; import org.apache.fory.format.row.binary.BinaryMap; import org.apache.fory.format.row.binary.BinaryRow; @@ -60,18 +61,43 @@ public RowEncoderBuilder newRowEncoder(final TypeRef beanClass) { return new RowEncoderBuilder(beanClass); } + @Override + public RowEncoderBuilder newProjectionRowEncoder( + final TypeRef beanType, + final Schema historicalSchema, + final Set liveNames, + final String classSuffix) { + return new RowEncoderBuilder(beanType, historicalSchema, liveNames, classSuffix); + } + @Override public ArrayEncoderBuilder newArrayEncoder( final TypeRef> collectionType, final TypeRef elementType) { return new ArrayEncoderBuilder(collectionType, elementType); } + @Override + public ArrayEncoderBuilder newProjectionArrayEncoder( + final TypeRef> collectionType, + final TypeRef elementType, + final String rowCodecSuffix) { + return new ArrayEncoderBuilder(collectionType, elementType, rowCodecSuffix); + } + @Override public MapEncoderBuilder newMapEncoder( final TypeRef> mapType, final TypeRef beanToken) { return new MapEncoderBuilder(mapType, beanToken); } + @Override + public MapEncoderBuilder newProjectionMapEncoder( + final TypeRef> mapType, + final TypeRef beanToken, + final String rowCodecSuffix) { + return new MapEncoderBuilder(mapType, beanToken, rowCodecSuffix); + } + @Override public BinaryRow newRow(final Schema schema) { return new BinaryRow(schema); diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/Encoders.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/Encoders.java index 13403c15f3..ba0d781a22 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/Encoders.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/Encoders.java @@ -22,6 +22,10 @@ import static org.apache.fory.type.TypeUtils.OBJECT_TYPE; import static org.apache.fory.type.TypeUtils.getRawType; +import java.lang.invoke.MethodHandle; +import java.lang.invoke.MethodHandles; +import java.lang.invoke.MethodType; +import java.lang.reflect.Constructor; import java.util.Collection; import java.util.HashSet; import java.util.LinkedHashSet; @@ -320,6 +324,28 @@ static Class loadOrGenRowCodecClass(Class beanClass, Encoding codecFactory return loadCls(compileUnits); } + /** + * Compile and load a projection codec class for one historical version of {@code beanClass}. + * The current-version codec class is loaded separately by {@link #loadOrGenRowCodecClass}; this + * is used by schema-evolution code paths to materialize a decoder for each older version. + */ + static Class loadOrGenProjectionRowCodecClass( + Class beanClass, + Encoding codecFactory, + org.apache.fory.format.type.Schema historicalSchema, + Set liveNames, + String classSuffix) { + final RowEncoderBuilder codecBuilder = + codecFactory.newProjectionRowEncoder( + TypeRef.of(beanClass), historicalSchema, liveNames, classSuffix); + CompileUnit compileUnit = + new CompileUnit( + CodeGenerator.getPackage(beanClass), + codecBuilder.codecClassName(beanClass) + classSuffix, + codecBuilder::genCode); + return loadCls(compileUnit); + } + static Class loadOrGenArrayCodecClass( TypeRef> arrayCls, TypeRef elementType, Encoding codecFactory) { LOG.info("Create ArrayCodec for classes {}", elementType); @@ -337,6 +363,23 @@ static Class loadOrGenArrayCodecClass( return loadCls(compileUnit); } + static Class loadOrGenProjectionArrayCodecClass( + TypeRef> arrayCls, + TypeRef elementType, + Encoding codecFactory, + String rowCodecSuffix) { + Class cls = getRawType(elementType); + String prefix = TypeInference.inferTypeName(arrayCls); + ArrayEncoderBuilder codecBuilder = + codecFactory.newProjectionArrayEncoder(arrayCls, elementType, rowCodecSuffix); + CompileUnit compileUnit = + new CompileUnit( + CodeGenerator.getPackage(cls), + codecBuilder.codecClassName(cls, prefix) + rowCodecSuffix, + codecBuilder::genCode); + return loadCls(compileUnit); + } + static Class loadOrGenMapCodecClass( TypeRef> mapCls, TypeRef keyToken, @@ -370,6 +413,23 @@ static Class loadOrGenMapCodecClass( return loadCls(compileUnit); } + static Class loadOrGenProjectionMapCodecClass( + TypeRef> mapCls, + TypeRef beanToken, + Encoding codecFactory, + String rowCodecSuffix) { + Class cls = getRawType(beanToken); + String prefix = TypeInference.inferTypeName(mapCls); + MapEncoderBuilder codecBuilder = + codecFactory.newProjectionMapEncoder(mapCls, beanToken, rowCodecSuffix); + CompileUnit compileUnit = + new CompileUnit( + CodeGenerator.getPackage(cls), + codecBuilder.codecClassName(cls, prefix) + rowCodecSuffix, + codecBuilder::genCode); + return loadCls(compileUnit); + } + private static Class loadCls(CompileUnit... compileUnit) { CodeGenerator codeGenerator = CodeGenerator.getSharedCodeGenerator(Thread.currentThread().getContextClassLoader()); @@ -381,4 +441,21 @@ private static Class loadCls(CompileUnit... compileUnit) { throw new IllegalStateException("Impossible because we just compiled class", e); } } + + /** + * Build a {@link MethodHandle} bound to {@code generatedClass}'s {@code (Object[])} constructor, + * adapted so it returns {@code generatedType}. All generated row/array/map codec classes share + * this constructor shape; this helper centralises the reflection and exception wrapping. + */ + static MethodHandle constructorHandleFor(Class generatedClass, Class generatedType) { + try { + Constructor constructor = + generatedClass.asSubclass(generatedType).getConstructor(Object[].class); + return MethodHandles.lookup() + .unreflectConstructor(constructor) + .asType(MethodType.methodType(generatedType, Object[].class)); + } catch (NoSuchMethodException | IllegalAccessException e) { + throw new EncoderException("Failed to resolve constructor for " + generatedClass, e); + } + } } diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/Encoding.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/Encoding.java index 56217ef5bb..2f994b2000 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/Encoding.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/Encoding.java @@ -21,6 +21,7 @@ import java.util.Collection; import java.util.Map; +import java.util.Set; import org.apache.fory.format.row.binary.BinaryArray; import org.apache.fory.format.row.binary.BinaryMap; import org.apache.fory.format.row.binary.BinaryRow; @@ -42,11 +43,37 @@ interface Encoding { RowEncoderBuilder newRowEncoder(TypeRef beanType); + /** + * Construct a projection codec builder for an older version of {@code beanType}, reading the + * supplied historical schema and producing instances of the current bean class. Used only by + * the schema-evolution code path. + */ + RowEncoderBuilder newProjectionRowEncoder( + TypeRef beanType, Schema historicalSchema, Set liveNames, String classSuffix); + ArrayEncoderBuilder newArrayEncoder( TypeRef> collectionType, TypeRef elementType); + /** + * Construct an array encoder builder whose generated code references the row codec class for + * the element bean with the supplied suffix. Used by schema-evolution paths to generate one + * array codec per historical version of the element bean. + */ + ArrayEncoderBuilder newProjectionArrayEncoder( + TypeRef> collectionType, + TypeRef elementType, + String rowCodecSuffix); + MapEncoderBuilder newMapEncoder(TypeRef> mapType, TypeRef beanToken); + /** + * Construct a map encoder builder whose generated code references the bean row codec class + * with the supplied suffix. Used by schema-evolution paths to generate one map codec per + * historical version of the bean. + */ + MapEncoderBuilder newProjectionMapEncoder( + TypeRef> mapType, TypeRef beanToken, String rowCodecSuffix); + BinaryRow newRow(Schema schema); BinaryArray newArray(Field field); diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/MapCodecBuilder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/MapCodecBuilder.java index 44ad87e6de..f27baf2d13 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/MapCodecBuilder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/MapCodecBuilder.java @@ -20,16 +20,22 @@ package org.apache.fory.format.encoder; import java.lang.invoke.MethodHandle; -import java.lang.invoke.MethodHandles; -import java.lang.invoke.MethodType; +import java.util.HashMap; import java.util.Map; import java.util.function.BiFunction; import java.util.function.Supplier; +import java.util.function.UnaryOperator; +import org.apache.fory.Fory; import org.apache.fory.format.row.binary.writer.BinaryArrayWriter; +import org.apache.fory.format.row.binary.writer.CompactBinaryRowWriter; +import org.apache.fory.format.type.CustomTypeEncoderRegistry; import org.apache.fory.format.type.DataTypes; import org.apache.fory.format.type.Field; +import org.apache.fory.format.type.Schema; +import org.apache.fory.format.type.SchemaHistory; import org.apache.fory.format.type.TypeInference; import org.apache.fory.reflect.TypeRef; +import org.apache.fory.type.TypeResolutionContext; import org.apache.fory.type.TypeUtils; import org.apache.fory.util.ExceptionUtils; @@ -55,23 +61,108 @@ public class MapCodecBuilder> extends BaseCodecBuilder> build() { loadMapInnerCodecs(); - final var mapEncoderFactory = generatedMapEncoder(); + if (!schemaEvolution || !isVersionedBeanValue()) { + final var mapEncoderFactory = generatedMapEncoder(); + return new Supplier>() { + @Override + public MapEncoder get() { + final BinaryArrayWriter keyWriter = codecFormat.newArrayWriter(keyField); + final BinaryArrayWriter valWriter = + codecFormat.newArrayWriter(valField, keyWriter.getBuffer()); + final var codec = mapEncoderFactory.apply(keyWriter, valWriter); + return new BufferResettingMapEncoder<>( + initialBufferSize, + keyWriter, + valWriter, + new BinaryMapEncoder(codecFormat, field, valWriter, keyWriter, codec, sizeEmbedded)); + } + }; + } + return buildVersioned(); + } + + private boolean isVersionedBeanValue() { + return TypeUtils.isBean( + valType, + new TypeResolutionContext(CustomTypeEncoderRegistry.customTypeHandler(), true)); + } + + private Supplier> buildVersioned() { + Class valClass = TypeUtils.getRawType(valType); + UnaryOperator schemaTransform = + codecFormat == CompactCodecFormat.INSTANCE + ? CompactBinaryRowWriter::sortSchema + : UnaryOperator.identity(); + SchemaHistory history = SchemaHistory.build(valClass, schemaTransform); + SchemaHistory.VersionedSchema current = history.current(); + + Encoders.loadOrGenRowCodecClass(valClass, codecFormat); + Map projectionFactories = new HashMap<>(); + for (SchemaHistory.VersionedSchema vs : history.versions()) { + if (vs == current) { + continue; + } + String suffix = "_V" + vs.version(); + Encoders.loadOrGenProjectionRowCodecClass( + valClass, codecFormat, vs.schema(), vs.liveFieldNames(), suffix); + Class mapClass = + Encoders.loadOrGenProjectionMapCodecClass( + mapType, TypeRef.of(valClass), codecFormat, suffix); + MethodHandle ctor = Encoders.constructorHandleFor(mapClass, GeneratedMapEncoder.class); + // Build a MapType whose value is the historical element struct, keeping the same key. + Field individualKey = DataTypes.keyFieldForMap(field); + Field histIndividualVal = + DataTypes.field( + DataTypes.MAP_VALUE_NAME, new DataTypes.StructType(vs.schema().fields()), true); + Field histMapField = DataTypes.mapField(field.name(), individualKey, histIndividualVal); + projectionFactories.put(vs.strictHash(), new ProjectionMapFactory(histMapField, ctor)); + } + final var currentFactory = generatedMapEncoder(); + long currentHash = current.strictHash(); return new Supplier>() { @Override public MapEncoder get() { - final BinaryArrayWriter keyWriter = codecFormat.newArrayWriter(keyField); - final BinaryArrayWriter valWriter = - codecFormat.newArrayWriter(valField, keyWriter.getBuffer()); - final var codec = mapEncoderFactory.apply(keyWriter, valWriter); + BinaryArrayWriter keyWriter = codecFormat.newArrayWriter(keyField); + BinaryArrayWriter valWriter = codecFormat.newArrayWriter(valField, keyWriter.getBuffer()); + var codec = currentFactory.apply(keyWriter, valWriter); + Map proj = new HashMap<>(); + for (Map.Entry entry : projectionFactories.entrySet()) { + proj.put(entry.getKey(), entry.getValue().instantiate(codecFormat, fory)); + } return new BufferResettingMapEncoder<>( initialBufferSize, keyWriter, valWriter, - new BinaryMapEncoder(codecFormat, field, valWriter, keyWriter, codec, sizeEmbedded)); + new BinaryMapEncoder( + codecFormat, field, valWriter, keyWriter, codec, sizeEmbedded, currentHash, proj)); } }; } + private final class ProjectionMapFactory { + private final Field histMapField; + private final MethodHandle ctor; + + ProjectionMapFactory(Field histMapField, MethodHandle ctor) { + this.histMapField = histMapField; + this.ctor = ctor; + } + + BinaryMapEncoder.ProjectionMapCodec instantiate(Encoding format, Fory fory) { + try { + Field histKeyField = DataTypes.keyArrayFieldForMap(histMapField); + Field histValField = DataTypes.itemArrayFieldForMap(histMapField); + BinaryArrayWriter projKey = format.newArrayWriter(histKeyField); + BinaryArrayWriter projVal = format.newArrayWriter(histValField, projKey.getBuffer()); + Object[] references = {histKeyField, histValField, projKey, projVal, fory, histMapField}; + GeneratedMapEncoder codec = (GeneratedMapEncoder) ctor.invokeExact(references); + return new BinaryMapEncoder.ProjectionMapCodec(format, histMapField, codec); + } catch (Throwable e) { + throw ExceptionUtils.throwException(e); + } + } + } + private void loadMapInnerCodecs() { Encoders.loadMapCodecs(keyType, codecFormat); Encoders.loadMapCodecs(valType, codecFormat); @@ -81,17 +172,8 @@ BiFunction generatedM final Class arrayCodecClass = Encoders.loadOrGenMapCodecClass(mapType, keyType, valType, codecFormat); - final MethodHandle constructorHandle; - try { - final var constructor = - arrayCodecClass.asSubclass(GeneratedMapEncoder.class).getConstructor(Object[].class); - constructorHandle = - MethodHandles.lookup() - .unreflectConstructor(constructor) - .asType(MethodType.methodType(GeneratedMapEncoder.class, Object[].class)); - } catch (final NoSuchMethodException | IllegalAccessException e) { - throw new EncoderException("Failed to construct array codec for " + mapType, e); - } + final MethodHandle constructorHandle = + Encoders.constructorHandleFor(arrayCodecClass, GeneratedMapEncoder.class); return new BiFunction() { @Override public GeneratedMapEncoder apply( @@ -99,7 +181,7 @@ public GeneratedMapEncoder apply( final Object[] references = {keyField, valField, keyWriter, valWriter, fory, field}; try { return (GeneratedMapEncoder) constructorHandle.invokeExact(references); - } catch (final Throwable t) { + } catch (Throwable t) { throw ExceptionUtils.throwException(t); } } diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/MapEncoderBuilder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/MapEncoderBuilder.java index fa84944188..975c10bb83 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/MapEncoderBuilder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/MapEncoderBuilder.java @@ -58,7 +58,12 @@ public MapEncoderBuilder(Class mapCls, Class keyClass) { } public MapEncoderBuilder(TypeRef clsType, TypeRef beanType) { + this(clsType, beanType, null); + } + + MapEncoderBuilder(TypeRef clsType, TypeRef beanType, String rowCodecSuffix) { super(new CodegenContext(), beanType); + this.rowCodecSuffixForBeans = rowCodecSuffix; mapToken = clsType; ctx.reserveName(ROOT_KEY_WRITER_NAME); ctx.reserveName(ROOT_VALUE_WRITER_NAME); @@ -72,7 +77,9 @@ public MapEncoderBuilder(TypeRef clsType, TypeRef beanType) { @Override public String genCode() { ctx.setPackage(CodeGenerator.getPackage(beanClass)); - String className = codecClassName(beanClass, TypeInference.inferTypeName(mapToken)); + String className = + codecClassName(beanClass, TypeInference.inferTypeName(mapToken)) + + (rowCodecSuffixForBeans == null ? "" : rowCodecSuffixForBeans); ctx.setClassName(className); // don't addImport(arrayClass), because user class may name collide. // janino don't support generics, so GeneratedCodec has no generics diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/RowCodecBuilder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/RowCodecBuilder.java index b2f6b65384..2d90029666 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/RowCodecBuilder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/RowCodecBuilder.java @@ -20,11 +20,16 @@ package org.apache.fory.format.encoder; import java.lang.invoke.MethodHandle; -import java.lang.invoke.MethodHandles; -import java.lang.invoke.MethodType; +import java.util.HashMap; +import java.util.Map; import java.util.function.Function; import java.util.function.Supplier; +import java.util.function.UnaryOperator; +import org.apache.fory.Fory; import org.apache.fory.format.row.binary.writer.BaseBinaryRowWriter; +import org.apache.fory.format.row.binary.writer.CompactBinaryRowWriter; +import org.apache.fory.format.type.Schema; +import org.apache.fory.format.type.SchemaHistory; import org.apache.fory.format.type.TypeInference; import org.apache.fory.util.ExceptionUtils; @@ -46,10 +51,13 @@ public class RowCodecBuilder extends BaseCodecBuilder> { */ public Supplier> build() { final Function> rowEncoderFactory = buildForWriter(); + // Snapshot schema at build time so a supplier remains pinned to the schema in effect when + // it was constructed, even if the builder is mutated afterwards. + final Schema currentSchema = schema; return new Supplier>() { @Override public RowEncoder get() { - final BaseBinaryRowWriter writer = codecFormat.newWriter(schema); + final BaseBinaryRowWriter writer = codecFormat.newWriter(currentSchema); return new BufferResettingRowEncoder( initialBufferSize, writer, rowEncoderFactory.apply(writer)); } @@ -57,39 +65,107 @@ public RowEncoder get() { } Function> buildForWriter() { + if (!schemaEvolution) { + return defaultBuildForWriter(); + } + return evolvingBuildForWriter(); + } + + private Function> defaultBuildForWriter() { + final Schema currentSchema = schema; final Function rowEncoderFactory = - rowEncoderFactory(); + rowEncoderFactory(currentSchema); return new Function>() { @Override public RowEncoder apply(final BaseBinaryRowWriter writer) { return new BinaryRowEncoder( - schema, codecFormat, rowEncoderFactory.apply(writer), writer, sizeEmbedded); + currentSchema, codecFormat, rowEncoderFactory.apply(writer), writer, sizeEmbedded); } }; } - Function rowEncoderFactory() { - final Class rowCodecClass = Encoders.loadOrGenRowCodecClass(beanClass, codecFormat); - MethodHandle constructorHandle; - try { - final var constructor = - rowCodecClass.asSubclass(GeneratedRowEncoder.class).getConstructor(Object[].class); - constructorHandle = - MethodHandles.lookup() - .unreflectConstructor(constructor) - .asType(MethodType.methodType(GeneratedRowEncoder.class, Object[].class)); - } catch (final NoSuchMethodException | IllegalAccessException e) { - throw new EncoderException("Failed to construct codec for " + beanClass, e); + private Function> evolvingBuildForWriter() { + UnaryOperator schemaTransform = + codecFormat == CompactCodecFormat.INSTANCE + ? CompactBinaryRowWriter::sortSchema + : UnaryOperator.identity(); + SchemaHistory history = SchemaHistory.build(beanClass, schemaTransform); + SchemaHistory.VersionedSchema currentVersion = history.current(); + // The history-derived schema is the one writers, generated codec, and decode dispatch must + // agree on. Pin it on the builder so build() picks up the rotated schema; pass it into the + // current-version codec factory locally so a later mutation of the field cannot affect + // already-constructed encoders. + final Schema currentSchema = currentVersion.schema(); + schema = currentSchema; + + final Function currentFactory = + rowEncoderFactory(currentSchema); + // Projection codecs for each older version; classes are loaded eagerly. + final Map projectionFactories = new HashMap<>(); + for (SchemaHistory.VersionedSchema vs : history.versions()) { + if (vs == currentVersion) { + continue; + } + String suffix = "_V" + vs.version(); + Class projectionClass = + Encoders.loadOrGenProjectionRowCodecClass( + beanClass, codecFormat, vs.schema(), vs.liveFieldNames(), suffix); + MethodHandle ctor = + Encoders.constructorHandleFor(projectionClass, GeneratedRowEncoder.class); + projectionFactories.put(vs.strictHash(), new ProjectionCodecFactory(vs.schema(), ctor)); + } + + final long currentHash = currentVersion.strictHash(); + return new Function>() { + @Override + public RowEncoder apply(final BaseBinaryRowWriter writer) { + Map projections = new HashMap<>(); + for (Map.Entry entry : projectionFactories.entrySet()) { + projections.put(entry.getKey(), entry.getValue().instantiate(writer, fory)); + } + return new BinaryRowEncoder( + currentSchema, + codecFormat, + currentFactory.apply(writer), + writer, + sizeEmbedded, + currentHash, + projections); + } + }; + } + + private static final class ProjectionCodecFactory { + private final Schema historicalSchema; + private final MethodHandle ctor; + + ProjectionCodecFactory(Schema historicalSchema, MethodHandle ctor) { + this.historicalSchema = historicalSchema; + this.ctor = ctor; } + + BinaryRowEncoder.ProjectionCodec instantiate(BaseBinaryRowWriter writer, Fory fory) { + try { + Object[] references = {historicalSchema, writer, fory}; + GeneratedRowEncoder codec = (GeneratedRowEncoder) ctor.invokeExact(references); + return new BinaryRowEncoder.ProjectionCodec(historicalSchema, codec); + } catch (Throwable e) { + throw ExceptionUtils.throwException(e); + } + } + } + + Function rowEncoderFactory(final Schema codecSchema) { + final Class rowCodecClass = Encoders.loadOrGenRowCodecClass(beanClass, codecFormat); + final MethodHandle constructorHandle = + Encoders.constructorHandleFor(rowCodecClass, GeneratedRowEncoder.class); return new Function() { @Override public GeneratedRowEncoder apply(final BaseBinaryRowWriter writer) { try { - final Object[] references = {schema, writer, fory}; + final Object[] references = {codecSchema, writer, fory}; return (GeneratedRowEncoder) constructorHandle.invokeExact(references); - } catch (final ReflectiveOperationException e) { - throw new EncoderException("Failed to construct codec for " + beanClass, e); - } catch (final Throwable e) { + } catch (Throwable e) { throw ExceptionUtils.throwException(e); } } diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/RowEncoderBuilder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/RowEncoderBuilder.java index ea7dc25ece..7a2b73cbc4 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/RowEncoderBuilder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/RowEncoderBuilder.java @@ -31,6 +31,7 @@ import java.util.List; import java.util.Map; import java.util.Optional; +import java.util.Set; import java.util.SortedMap; import org.apache.fory.Fory; import org.apache.fory.builder.CodecBuilder; @@ -76,16 +77,42 @@ class RowEncoderBuilder extends BaseBinaryEncoderBuilder { protected Reference beanClassRef = new Reference(BEAN_CLASS_NAME, CLASS_TYPE); private final CodegenContext generatedBeanImpl; private final String generatedBeanImplName; + /** + * When non-null, this builder produces a decode-only projection codec: schema fields whose + * name is in {@code projectionLiveNames} are assigned to the bean as usual; others are decoded + * for offset arithmetic only and discarded. {@code toRow} on a projection codec throws. + */ + private final Set projectionLiveNames; + private final String projectionClassSuffix; public RowEncoderBuilder(Class beanClass) { this(TypeRef.of(beanClass)); } public RowEncoderBuilder(TypeRef beanType) { + this(beanType, null, null, null); + } + + /** + * Construct a decode-only projection builder for an older version of {@code beanType}. The + * supplied {@code historicalSchema} is used as the layout to decode; only fields whose name is + * in {@code liveNames} are written into the resulting bean. {@code classSuffix} distinguishes + * this codec from the current-version codec and from other historical projections. + */ + RowEncoderBuilder( + TypeRef beanType, + Schema historicalSchema, + Set liveNames, + String classSuffix) { super(new CodegenContext(), beanType); Preconditions.checkArgument(beanClass.isInterface() || TypeUtils.isBean(beanType, typeCtx)); - className = codecClassName(beanClass); - this.schema = inferSchema(beanType); + this.projectionLiveNames = liveNames; + this.projectionClassSuffix = classSuffix; + className = + projectionClassSuffix == null + ? codecClassName(beanClass) + : codecClassName(beanClass) + projectionClassSuffix; + this.schema = historicalSchema != null ? historicalSchema : inferSchema(beanType); this.descriptorsMap = Descriptor.getDescriptorsMap(beanClass); ctx.reserveName(ROOT_ROW_WRITER_NAME); ctx.reserveName(SCHEMA_NAME); @@ -105,7 +132,13 @@ public RowEncoderBuilder(TypeRef beanType) { ctx.addImports(Row.class, ArrayData.class, MapData.class); ctx.addImports(BinaryRow.class, BinaryArray.class, BinaryMap.class); if (beanClass.isInterface()) { - generatedBeanImplName = beanClass.getSimpleName() + "GeneratedImpl"; + // Append the projection suffix so each historical version of an interface bean gets its + // own impl class; the impl classes are inner classes of the codec and would collide on + // the simple name otherwise. + generatedBeanImplName = + beanClass.getSimpleName() + + "GeneratedImpl" + + (projectionClassSuffix == null ? "" : projectionClassSuffix); generatedBeanImpl = buildImplClass(); } else { generatedBeanImplName = null; @@ -203,8 +236,14 @@ public Expression buildEncodeExpression() { // schema field's name must correspond to descriptor's name. for (int i = 0; i < numFields; i++) { Field field = schema.field(i); + if (projectionLiveNames != null && !projectionLiveNames.contains(field.name())) { + // Removed wire field — no Java accessor to read from, so we cannot emit encode + // code. The projection codec's encode body is unreachable anyway because + // BinaryRowEncoder never dispatches a projection codec on write. + continue; + } Descriptor d = getDescriptorByFieldName(field.name()); - Preconditions.checkNotNull(d); + Preconditions.checkNotNull(d, "missing descriptor for schema field " + field.name()); TypeRef fieldType = d.getTypeRef(); Expression fieldValue = getFieldValue(bean, d); Literal ordinal = Literal.ofInt(i); @@ -215,6 +254,12 @@ public Expression buildEncodeExpression() { serializeFor(ordinal, fieldValue, writer, fieldType, field, foryField, new HashSet<>()); expressions.add(fieldExpr); } + if (projectionLiveNames != null) { + // Decode-only: never run the writer logic. The expressions above were generated only for + // their side effects on the codegen context (registering nested-bean encoder fields). + return new Expression.Block( + "throw new UnsupportedOperationException(\"projection codec is decode-only\");\n"); + } expressions.add( new Expression.Return( new Expression.Invoke(writer, "getRow", TypeRef.of(BinaryRow.class)))); @@ -237,19 +282,27 @@ public Expression buildDecodeExpression() { bean = new Expression.Reference("new " + generatedBeanImplName + "(row)"); } else { int numFields = schema.numFields(); - List fieldNames = new ArrayList<>(numFields); - Expression[] values = new Expression[numFields]; - Descriptor[] descriptors = new Descriptor[numFields]; - // schema field's name must correspond to descriptor's name. + // Build, in schema order, the per-slot bean-side info for live fields only. Discarded + // slots are part of the row layout but have no Java target; we skip emitting any code + // for them because BinaryRow's offset arithmetic is keyed on slot index, not on prior + // reads. + List liveFieldDescriptorNames = new ArrayList<>(); + List liveDescriptors = new ArrayList<>(); + List liveValues = new ArrayList<>(); for (int i = 0; i < numFields; i++) { Literal ordinal = Literal.ofInt(i); - Descriptor d = getDescriptorByFieldName(schema.field(i).name()); - fieldNames.add(d.getName()); - descriptors[i] = d; + String wireName = schema.field(i).name(); + if (projectionLiveNames != null && !projectionLiveNames.contains(wireName)) { + continue; + } + Descriptor d = getDescriptorByFieldName(wireName); + Preconditions.checkNotNull(d, "missing descriptor for wire field " + wireName); TypeRef fieldType = d.getTypeRef(); Expression.Variable value = new Expression.Variable("value_" + d.getName(), nullValue(fieldType)); - values[i] = value; + liveFieldDescriptorNames.add(d.getName()); + liveDescriptors.add(d); + liveValues.add(value); expressions.add(value); Expression.Invoke isNullAt = new Expression.Invoke( @@ -267,17 +320,12 @@ public Expression buildDecodeExpression() { expressions.add(decode); } if (RecordUtils.isRecord(beanClass)) { - int[] map = RecordUtils.buildRecordComponentMapping(beanClass, fieldNames); - Expression[] args = new Expression[numFields]; - for (int i = 0; i < numFields; i++) { - args[i] = values[map[i]]; - } - bean = new Expression.NewInstance(beanType, beanType.getRawType().getName(), args); + bean = buildRecordInstance(liveFieldDescriptorNames, liveValues); } else { bean = newBean(); expressions.add(bean); - for (int i = 0; i < values.length; i++) { - expressions.add(setFieldValue(bean, descriptors[i], values[i])); + for (int i = 0; i < liveDescriptors.size(); i++) { + expressions.add(setFieldValue(bean, liveDescriptors.get(i), liveValues.get(i))); } } } @@ -290,6 +338,30 @@ public Expression buildDecodeExpression() { return expressions; } + /** + * Build a record instance, supplying defaults for components not contributed by the wire. The + * non-projection path always supplies every component; the projection path may supply a + * subset. + */ + private Expression buildRecordInstance(List liveDescriptorNames, List liveValues) { + Map byName = new HashMap<>(liveDescriptorNames.size() * 2); + for (int i = 0; i < liveDescriptorNames.size(); i++) { + byName.put(liveDescriptorNames.get(i), liveValues.get(i)); + } + java.lang.reflect.RecordComponent[] components = beanClass.getRecordComponents(); + Expression[] args = new Expression[components.length]; + for (int i = 0; i < components.length; i++) { + String compName = components[i].getName(); + Expression value = byName.get(compName); + if (value == null) { + TypeRef compType = TypeRef.of(components[i].getGenericType()); + value = nullValue(compType); + } + args[i] = value; + } + return new Expression.NewInstance(beanType, beanType.getRawType().getName(), args); + } + private static Expression nullValue(TypeRef fieldType) { Class rawType = fieldType.getRawType(); if (TypeUtils.isOptionalType(rawType)) { @@ -303,7 +375,11 @@ private void addDecoderMethods() { int numFields = schema.numFields(); for (int i = 0; i < numFields; i++) { Literal ordinal = Literal.ofInt(i); - Descriptor d = getDescriptorByFieldName(schema.field(i).name()); + String wireName = schema.field(i).name(); + if (projectionLiveNames != null && !projectionLiveNames.contains(wireName)) { + continue; + } + Descriptor d = getDescriptorByFieldName(wireName); TypeRef fieldType = d.getTypeRef(); Class rawFieldType = fieldType.getRawType(); TypeRef columnAccessType = fieldType; @@ -355,7 +431,14 @@ private CodegenContext buildImplClass() { int numFields = schema.numFields(); for (int i = 0; i < numFields; i++) { Literal ordinal = Literal.ofInt(i); - Descriptor d = getDescriptorByFieldName(schema.field(i).name()); + String wireName = schema.field(i).name(); + if (projectionLiveNames != null && !projectionLiveNames.contains(wireName)) { + // Removed wire field — no Java member to back this slot. The other interface methods + // can still be served lazily from the row; the row's offset arithmetic does not need + // us to read this slot. + continue; + } + Descriptor d = getDescriptorByFieldName(wireName); TypeRef fieldType = d.getTypeRef(); Class rawFieldType = fieldType.getRawType(); @@ -407,6 +490,7 @@ private CodegenContext buildImplClass() { // Note: adding constructor captures init code, so must happen after all fields are collected implClass.addConstructor("this.row = row;", BinaryRow.class, "row"); + final boolean projecting = projectionLiveNames != null; methodsNeedingImpl.forEach( (methodName, signatures) -> signatures.forEach( @@ -419,16 +503,46 @@ private CodegenContext buildImplClass() { params[i * 2] = methodType.parameterType(i); params[i * 2 + 1] = "unused" + i; } - implClass.addMethod( - methodName, - "throw new UnsupportedOperationException();", - methodType.returnType(), - params); + String body; + if (projecting && isAccessorOfAbsentField(methodName, methodType)) { + body = + "return " + defaultValueExpression(methodType.returnType(), implClass) + ";"; + } else { + body = "throw new UnsupportedOperationException();"; + } + implClass.addMethod(methodName, body, methodType.returnType(), params); })); return implClass; } + /** + * True when {@code methodName(returnType)} on the current bean class names a property whose + * field is not in the historical schema this projection codec is generating. Such a method + * gets a default-value body instead of {@code throw} so the interface proxy can serve callers + * that don't know the field is missing in this version. + */ + private boolean isAccessorOfAbsentField(String methodName, MethodType methodType) { + Descriptor d = descriptorsMap.get(methodName); + if (d == null) { + return false; + } + if (d.getTypeRef().getRawType() != methodType.returnType()) { + return false; + } + // The main loop above emits getters for every wire field that is also a live Java member. + // Anything left in methodsNeedingImpl that matches a descriptor by name and type must + // correspond to a Java member whose wire field is not in this version. + return true; + } + + private static String defaultValueExpression(Class returnType, CodegenContext ctx) { + if (TypeUtils.isOptionalType(returnType)) { + return ctx.type(returnType) + ".empty()"; + } + return TypeUtils.defaultValue(returnType); + } + private Descriptor getDescriptorByFieldName(String fieldName) { String name = StringUtils.lowerUnderscoreToLowerCamelCase(fieldName); return descriptorsMap.get(name); diff --git a/java/fory-format/src/main/java/org/apache/fory/format/type/SchemaHistory.java b/java/fory-format/src/main/java/org/apache/fory/format/type/SchemaHistory.java new file mode 100644 index 0000000000..331d470754 --- /dev/null +++ b/java/fory-format/src/main/java/org/apache/fory/format/type/SchemaHistory.java @@ -0,0 +1,393 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.fory.format.type; + +import java.lang.reflect.AnnotatedElement; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.TreeSet; +import java.util.function.UnaryOperator; +import org.apache.fory.annotation.Internal; +import org.apache.fory.format.annotation.ForySchema; +import org.apache.fory.format.annotation.ForyVersion; +import org.apache.fory.reflect.TypeRef; +import org.apache.fory.type.Descriptor; +import org.apache.fory.util.StringUtils; + +/** + * Resolves the version history of a row-codec bean. Each entry exposes the schema as it appeared + * at a particular version, along with a strict hash that uniquely identifies the historical + * layout. Only used when {@code withSchemaEvolution()} is configured on the codec builder. + * + *

The hash mixes field names and nullability in addition to types, so that two schemas that + * differ only in field order or naming are distinguishable. This is intentionally a different + * hash from {@link DataTypes#computeSchemaHash} and is used only by versioning code paths. + */ +@Internal +public final class SchemaHistory { + + /** One entry in a {@link SchemaHistory}. */ + public static final class VersionedSchema { + private final int version; + private final Schema schema; + private final long strictHash; + private final Set liveFieldNames; + + VersionedSchema(int version, Schema schema, long strictHash, Set liveFieldNames) { + this.version = version; + this.schema = schema; + this.strictHash = strictHash; + this.liveFieldNames = liveFieldNames; + } + + public int version() { + return version; + } + + public Schema schema() { + return schema; + } + + public long strictHash() { + return strictHash; + } + + /** + * Names of fields in this version that still have a Java member on the current bean class. + * Other fields are read-and-discarded during projection. + */ + public Set liveFieldNames() { + return liveFieldNames; + } + } + + private final List versions; + private final VersionedSchema current; + + private SchemaHistory(List versions, VersionedSchema current) { + this.versions = versions; + this.current = current; + } + + public VersionedSchema current() { + return current; + } + + /** All known versions, ordered by version number ascending. */ + public List versions() { + return versions; + } + + /** + * Build a history from the bean's annotations. The schema for each version is transformed by + * {@code schemaTransform} after filtering; pass an identity for standard format, or + * {@code CompactBinaryRowWriter::sortSchema} for compact format. + */ + public static SchemaHistory build(Class beanClass, UnaryOperator schemaTransform) { + ForySchema schemaAnn = beanClass.getAnnotation(ForySchema.class); + int baseVersion = schemaAnn == null ? 1 : schemaAnn.baseVersion(); + Class removedFieldsClass = schemaAnn == null ? void.class : schemaAnn.removedFields(); + + List all = collectLiveFields(beanClass, baseVersion); + if (removedFieldsClass != void.class) { + all.addAll(collectRemovedFields(removedFieldsClass)); + } + + // Materialize a schema at every version V where the field set changes — both "since" and + // "until" boundaries qualify, because either adds or removes a field from the active set. + TreeSet schemaVersions = new TreeSet<>(); + schemaVersions.add(baseVersion); + for (FieldEntry fe : all) { + schemaVersions.add(fe.since); + if (fe.until != Integer.MAX_VALUE) { + schemaVersions.add(fe.until); + } + } + + validateNoNameCollision(all); + + // Sort by Java member name so the per-version schema matches the order + // TypeInference.inferSchema produces (which iterates Descriptor.getDescriptors, alphabetical + // by Java member name). Removed fields synthesize a Java name from their wire name. + all.sort((a, b) -> a.javaName.compareTo(b.javaName)); + // A field with finite [since, until) can leave two boundaries with identical field sets + // (e.g. v1 and v4 both lack a field that lived in [v2, v4)). Collapse boundaries that + // produce the same field set into one VersionedSchema, since they round-trip identically. + // A real strict-hash collision — two distinct field sets producing the same hash — is + // caught by comparing canonical signatures on insertion. + int latestVersion = schemaVersions.last(); + Map bySignature = new LinkedHashMap<>(); + Map hashToSignature = new HashMap<>(); + for (int v : schemaVersions) { + List fields = new ArrayList<>(); + Set liveNames = new HashSet<>(); + for (FieldEntry fe : all) { + if (fe.since <= v && v < fe.until) { + fields.add(TypeInference.inferNamedField(fe.name, fe.typeRef)); + if (fe.live) { + liveNames.add(fe.name); + } + } + } + Schema schema = schemaTransform.apply(new Schema(fields)); + long hash = computeStrictSchemaHash(schema); + String signature = schemaSignature(schema); + String previousSig = hashToSignature.putIfAbsent(hash, signature); + if (previousSig != null && !previousSig.equals(signature)) { + throw new IllegalStateException( + "Strict hash collision for bean " + + beanClass.getName() + + " at version " + + v + + ": two distinct historical schemas hashed to the same value. Please file an " + + "issue with the bean definition."); + } + // Record the highest version at which this signature first appears. The latest boundary + // is the writer's "current" version; preferring it over earlier first-appearances keeps + // current().version() aligned with what writers emit. + bySignature.put( + signature, + new VersionedSchema(v, schema, hash, Collections.unmodifiableSet(liveNames))); + } + // current is the schema in effect at latestVersion. + VersionedSchema current = null; + for (VersionedSchema vs : bySignature.values()) { + if (vs.version() == latestVersion) { + current = vs; + break; + } + } + return new SchemaHistory( + Collections.unmodifiableList(new ArrayList<>(bySignature.values())), current); + } + + /** + * Canonical textual signature of a schema, used to distinguish a real strict-hash collision + * (two genuinely different schemas with the same hash) from the benign case where two version + * boundaries produce the same field set. + */ + private static String schemaSignature(Schema schema) { + StringBuilder sb = new StringBuilder(64); + for (Field field : schema.fields()) { + sb.append(field.name()) + .append(':') + .append(field.type()) + .append(field.nullable() ? "?" : "!") + .append(';'); + } + return sb.toString(); + } + + private static List collectRemovedFields(Class historyClass) { + List descriptors = Descriptor.getDescriptors(historyClass); + List out = new ArrayList<>(descriptors.size()); + for (Descriptor d : descriptors) { + ForyVersion ann = lookupForyVersion(d); + if (ann == null) { + throw new IllegalStateException( + "Removed-field declaration " + + historyClass.getName() + + "." + + d.getName() + + " requires a @ForyVersion(until = ...) annotation"); + } + if (ann.until() == Integer.MAX_VALUE) { + throw new IllegalStateException( + "Removed-field declaration " + + historyClass.getName() + + "." + + d.getName() + + " must specify @ForyVersion.until (no upper bound makes no sense for a field " + + "that has been removed)"); + } + if (ann.since() >= ann.until()) { + throw new IllegalStateException( + "Invalid @ForyVersion on " + + historyClass.getName() + + "." + + d.getName() + + ": since (" + + ann.since() + + ") must be strictly less than until (" + + ann.until() + + ")"); + } + // The history method's name must mirror the live field/method name. Wire names are + // derived the same way the live path derives them: descriptor name -> lower_underscore. + // For Lombok @Data or record-style beans the descriptor name is the field name + // ("tags"); for interface beans or JavaBean-style classes it is the method name + // ("getTags"). The user writes the history method to match. + String wireName = StringUtils.lowerCamelToLowerUnderscore(d.getName()); + out.add(new FieldEntry(wireName, d.getName(), d.getTypeRef(), ann.since(), ann.until(), /*live*/ false)); + } + return out; + } + + private static List collectLiveFields(Class beanClass, int baseVersion) { + List descriptors = Descriptor.getDescriptors(beanClass); + List out = new ArrayList<>(descriptors.size()); + for (Descriptor d : descriptors) { + ForyVersion ann = lookupForyVersion(d); + int since = ann == null ? baseVersion : ann.since(); + int until = ann == null ? Integer.MAX_VALUE : ann.until(); + if (since >= until) { + throw new IllegalStateException( + "Invalid @ForyVersion on " + beanClass.getName() + "." + d.getName() + + ": since (" + since + ") must be strictly less than until (" + until + ")"); + } + String wireName = StringUtils.lowerCamelToLowerUnderscore(d.getName()); + out.add(new FieldEntry(wireName, d.getName(), d.getTypeRef(), since, until, /*live*/ true)); + } + return out; + } + + private static ForyVersion lookupForyVersion(Descriptor d) { + ForyVersion ann = readAnnotation(d.getField()); + if (ann != null) { + return ann; + } + return readAnnotation(d.getReadMethod()); + } + + private static ForyVersion readAnnotation(AnnotatedElement element) { + return element == null ? null : element.getAnnotation(ForyVersion.class); + } + + private static void validateNoNameCollision(List entries) { + // For each pair with the same name, their [since, until) windows must not overlap. + Map> byName = new HashMap<>(); + for (FieldEntry fe : entries) { + byName.computeIfAbsent(fe.name, k -> new ArrayList<>()).add(fe); + } + for (Map.Entry> e : byName.entrySet()) { + List group = e.getValue(); + if (group.size() < 2) { + continue; + } + group.sort((a, b) -> Integer.compare(a.since, b.since)); + for (int i = 1; i < group.size(); i++) { + FieldEntry prev = group.get(i - 1); + FieldEntry curr = group.get(i); + if (curr.since < prev.until) { + throw new IllegalStateException( + "Field name '" + + e.getKey() + + "' is declared with overlapping version windows [" + + prev.since + + "," + + prev.until + + ") and [" + + curr.since + + "," + + curr.until + + "); each version must have one definition per name. Adjust the @ForyVersion " + + "annotations on the live field or in the removed-fields class to make the " + + "windows disjoint."); + } + } + } + } + + /** + * Strict schema hash, used only by versioning code paths. Distinguishes schemas that differ in + * field name or nullability, unlike {@link DataTypes#computeSchemaHash}. + */ + private static long computeStrictSchemaHash(Schema schema) { + long hash = 1469598103934665603L; // FNV offset basis + Set seen = new HashSet<>(); + for (Field field : schema.fields()) { + if (!seen.add(field.name())) { + throw new IllegalStateException( + "Duplicate field name in schema: " + field.name()); + } + hash = hashField(hash, field); + } + return hash; + } + + private static long hashField(long hash, Field field) { + hash = mix(hash, field.name()); + DataType type = field.type(); + // The type's name() carries its identity including any inline width (e.g. + // fixedSizeBinary(N)), which is enough for every type except DecimalType, whose + // precision and scale are stored separately. Mix those in explicitly so two decimals of + // different shape don't collide. + hash = mix(hash, type.name()); + if (type instanceof DataTypes.DecimalType) { + hash = mix(hash, ((DataTypes.DecimalType) type).precision()); + hash = mix(hash, ((DataTypes.DecimalType) type).scale()); + } + hash = mix(hash, field.nullable() ? 1 : 0); + if (type instanceof DataTypes.ListType) { + hash = hashField(hash, DataTypes.arrayElementField(field)); + } else if (type instanceof DataTypes.MapType) { + hash = hashField(hash, DataTypes.keyFieldForMap(field)); + hash = hashField(hash, DataTypes.itemFieldForMap(field)); + } else if (type instanceof DataTypes.StructType) { + for (Field child : type.fields()) { + hash = hashField(hash, child); + } + } + return hash; + } + + private static long mix(long hash, long value) { + hash ^= value; + hash *= 1099511628211L; // FNV prime + return hash; + } + + private static long mix(long hash, String value) { + for (int i = 0; i < value.length(); i++) { + hash = mix(hash, value.charAt(i)); + } + return mix(hash, 0); + } + + private static final class FieldEntry { + final String name; + /** + * Java member name used for canonical ordering. Matches {@link Descriptor#getName} so live + * fields and removed fields (declared on the history class) sort into the same order as + * {@link TypeInference#inferSchema} produces. + */ + final String javaName; + final TypeRef typeRef; + final int since; + final int until; + final boolean live; + + FieldEntry( + String name, String javaName, TypeRef typeRef, int since, int until, boolean live) { + this.name = name; + this.javaName = javaName; + this.typeRef = typeRef; + this.since = since; + this.until = until; + this.live = live; + } + } +} diff --git a/java/fory-format/src/main/java/org/apache/fory/format/type/TypeInference.java b/java/fory-format/src/main/java/org/apache/fory/format/type/TypeInference.java index 4617f04faa..dafc34c17c 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/type/TypeInference.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/type/TypeInference.java @@ -117,6 +117,16 @@ private static Field inferField(TypeRef typeRef) { return inferField(null, typeRef); } + /** + * Infer a single named field from its Java type, used by schema-evolution code paths that need + * to reconstruct historical fields by name and type without going through a Java member. + */ + static Field inferNamedField(String name, TypeRef typeRef) { + TypeResolutionContext ctx = + new TypeResolutionContext(CustomTypeEncoderRegistry.customTypeHandler(), true); + return inferField(name, typeRef, ctx); + } + private static Field inferField(TypeRef arrayTypeRef, TypeRef typeRef) { TypeResolutionContext ctx = new TypeResolutionContext(CustomTypeEncoderRegistry.customTypeHandler(), true); diff --git a/java/fory-format/src/test/java/org/apache/fory/format/encoder/SchemaEvolutionStressTest.java b/java/fory-format/src/test/java/org/apache/fory/format/encoder/SchemaEvolutionStressTest.java new file mode 100644 index 0000000000..e56d7b46e8 --- /dev/null +++ b/java/fory-format/src/test/java/org/apache/fory/format/encoder/SchemaEvolutionStressTest.java @@ -0,0 +1,736 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.fory.format.encoder; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import lombok.Data; +import org.apache.fory.exception.ClassNotCompatibleException; +import org.apache.fory.format.annotation.ForySchema; +import org.apache.fory.format.annotation.ForyVersion; +import org.testng.Assert; +import org.testng.annotations.Test; + +/** + * Stress tests for row-codec schema evolution. Each test probes a specific edge case; the names + * say what is being stressed. Tests that surfaced real bugs are kept with a note pointing at the + * fix; tests kept for coverage are short. + */ +public class SchemaEvolutionStressTest { + + // --------------------------------------------------------------------------- + // Long version chain: a field added at each version 1..5, plus a removal at v3. + // Verifies projection codecs are built and dispatched for every historical version. + // --------------------------------------------------------------------------- + + @Data + public static class ChainV1 { + private int a; // since 1 + } + + @Data + public static class ChainV2 { + private int a; + + @ForyVersion(since = 2) + private String b; + } + + @Data + public static class ChainV3 { + private int a; + + @ForyVersion(since = 2) + private String b; + + @ForyVersion(since = 3) + private long c; + } + + @Data + public static class ChainV4 { + private int a; + + @ForyVersion(since = 2) + private String b; + + @ForyVersion(since = 3) + private long c; + + @ForyVersion(since = 4) + private double d; + } + + /** + * v5 also removes the v1 'a' field starting at v5. The reader must therefore know about three + * different historical schemas: v1, v2-3, and v4 (since 'a' is removed and a new field 'e' + * shows up in v5; 'a' removal makes v5 differ from v4). + */ + @Data + @ForySchema(removedFields = ChainV5.History.class) + public static class ChainV5 { + @ForyVersion(since = 2) + private String b; + + @ForyVersion(since = 3) + private long c; + + @ForyVersion(since = 4) + private double d; + + @ForyVersion(since = 5) + private boolean e; + + interface History { + @ForyVersion(until = 5) + int a(); + } + } + + @Test + public void longChainAllVersionsReadable() { + RowEncoder w1 = + Encoders.buildBeanCodec(ChainV1.class).withSchemaEvolution().build().get(); + RowEncoder w2 = + Encoders.buildBeanCodec(ChainV2.class).withSchemaEvolution().build().get(); + RowEncoder w3 = + Encoders.buildBeanCodec(ChainV3.class).withSchemaEvolution().build().get(); + RowEncoder w4 = + Encoders.buildBeanCodec(ChainV4.class).withSchemaEvolution().build().get(); + RowEncoder reader = + Encoders.buildBeanCodec(ChainV5.class).withSchemaEvolution().build().get(); + + ChainV1 v1 = new ChainV1(); + v1.setA(11); + ChainV2 v2 = new ChainV2(); + v2.setA(21); + v2.setB("two"); + ChainV3 v3 = new ChainV3(); + v3.setA(31); + v3.setB("three"); + v3.setC(333L); + ChainV4 v4 = new ChainV4(); + v4.setA(41); + v4.setB("four"); + v4.setC(444L); + v4.setD(4.4); + + ChainV5 out1 = reader.decode(w1.encode(v1)); + Assert.assertNull(out1.getB()); + Assert.assertEquals(out1.getC(), 0L); + Assert.assertEquals(out1.getD(), 0.0); + Assert.assertFalse(out1.isE()); + + ChainV5 out2 = reader.decode(w2.encode(v2)); + Assert.assertEquals(out2.getB(), "two"); + Assert.assertEquals(out2.getC(), 0L); + + ChainV5 out3 = reader.decode(w3.encode(v3)); + Assert.assertEquals(out3.getC(), 333L); + Assert.assertEquals(out3.getD(), 0.0); + + ChainV5 out4 = reader.decode(w4.encode(v4)); + Assert.assertEquals(out4.getB(), "four"); + Assert.assertEquals(out4.getC(), 444L); + Assert.assertEquals(out4.getD(), 4.4); + Assert.assertFalse(out4.isE()); + } + + // --------------------------------------------------------------------------- + // Compact format with alignment shuffle: v1 has only longs; v2 adds a byte. + // Compact sorts fields by alignment width so the v1 and v2 schemas have + // different physical orders, even though their logical field sets differ by + // only the added byte. + // --------------------------------------------------------------------------- + + @Data + public static class AlignV1 { + private long x; + private long y; + } + + @Data + public static class AlignV2 { + private long x; + private long y; + + @ForyVersion(since = 2) + private byte flag; + } + + @Test + public void compactAlignmentReshuffleAcrossVersions() { + RowEncoder writer = + Encoders.buildBeanCodec(AlignV1.class) + .compactEncoding() + .withSchemaEvolution() + .build() + .get(); + RowEncoder reader = + Encoders.buildBeanCodec(AlignV2.class) + .compactEncoding() + .withSchemaEvolution() + .build() + .get(); + AlignV1 in = new AlignV1(); + in.setX(11); + in.setY(22); + AlignV2 out = reader.decode(writer.encode(in)); + Assert.assertEquals(out.getX(), 11); + Assert.assertEquals(out.getY(), 22); + Assert.assertEquals(out.getFlag(), (byte) 0); // primitive default + } + + // --------------------------------------------------------------------------- + // Boxed vs primitive default for an absent field. + // --------------------------------------------------------------------------- + + @Data + public static class DefaultsV1 { + private String name; + } + + @Data + public static class DefaultsV2 { + private String name; + + @ForyVersion(since = 2) + private int primitiveCount; // default 0 + + @ForyVersion(since = 2) + private Integer boxedCount; // default null + } + + @Test + public void primitiveAndBoxedDefaults() { + RowEncoder writer = + Encoders.buildBeanCodec(DefaultsV1.class).withSchemaEvolution().build().get(); + RowEncoder reader = + Encoders.buildBeanCodec(DefaultsV2.class).withSchemaEvolution().build().get(); + DefaultsV1 in = new DefaultsV1(); + in.setName("n"); + DefaultsV2 out = reader.decode(writer.encode(in)); + Assert.assertEquals(out.getName(), "n"); + Assert.assertEquals(out.getPrimitiveCount(), 0); + Assert.assertNull(out.getBoxedCount()); + } + + // --------------------------------------------------------------------------- + // Disjoint-window false collision (regression). A field whose [since, until) + // window starts above the base version and ends below infinity leaves the + // pre-since and post-until boundaries with identical field sets. SchemaHistory + // must collapse those into one entry rather than flagging a false collision. + // --------------------------------------------------------------------------- + + @Data + @ForySchema(removedFields = GappedWindow.History.class) + public static class GappedWindow { + private String name; + + interface History { + @ForyVersion(since = 2, until = 4) + int oldField(); + } + } + + @Test + public void disjointWindowDoesNotFalseCollide() { + // Build alone is the assertion: the bug was an IllegalStateException at build time. + RowEncoder codec = + Encoders.buildBeanCodec(GappedWindow.class).withSchemaEvolution().build().get(); + GappedWindow in = new GappedWindow(); + in.setName("hi"); + Assert.assertEquals(codec.decode(codec.encode(in)).getName(), "hi"); + } + + // --------------------------------------------------------------------------- + // Removed field whose original type was a nested struct. The projection + // codec must skip the slot without trying to read or decode it. + // --------------------------------------------------------------------------- + + @Data + public static class StructRefV1 { + private String id; + private DefaultsV1 detail; // removed at v2 + } + + @Data + @ForySchema(removedFields = StructRefV2.History.class) + public static class StructRefV2 { + private String id; + + interface History { + @ForyVersion(until = 2) + DefaultsV1 detail(); + } + } + + @Test + public void removedNestedStructField() { + RowEncoder writer = + Encoders.buildBeanCodec(StructRefV1.class).withSchemaEvolution().build().get(); + RowEncoder reader = + Encoders.buildBeanCodec(StructRefV2.class).withSchemaEvolution().build().get(); + StructRefV1 in = new StructRefV1(); + in.setId("x"); + DefaultsV1 d = new DefaultsV1(); + d.setName("inner"); + in.setDetail(d); + StructRefV2 out = reader.decode(writer.encode(in)); + Assert.assertEquals(out.getId(), "x"); + } + + // --------------------------------------------------------------------------- + // Removed collection-typed field. The history interface preserves the full + // parameterized type, so List and Map round-trip + // through the projection without losing element-type information. + // --------------------------------------------------------------------------- + + @Data + public static class CollectionsV1 { + private String id; + private List tags; // removed at v2 + private java.util.Map counters; // removed at v2 + } + + @Data + @ForySchema(removedFields = CollectionsV2.History.class) + public static class CollectionsV2 { + private String id; + + interface History { + @ForyVersion(until = 2) + List tags(); + + @ForyVersion(until = 2) + java.util.Map counters(); + } + } + + @Test + public void removedParameterizedCollectionFields() { + RowEncoder writer = + Encoders.buildBeanCodec(CollectionsV1.class).withSchemaEvolution().build().get(); + RowEncoder reader = + Encoders.buildBeanCodec(CollectionsV2.class).withSchemaEvolution().build().get(); + CollectionsV1 in = new CollectionsV1(); + in.setId("c"); + in.setTags(Arrays.asList("alpha", "beta")); + java.util.Map counters = new java.util.HashMap<>(); + counters.put("k1", 1L); + counters.put("k2", 2L); + in.setCounters(counters); + CollectionsV2 out = reader.decode(writer.encode(in)); + Assert.assertEquals(out.getId(), "c"); + } + + // --------------------------------------------------------------------------- + // Same wire-name retyped across versions: 'tag' was int [1,3), then String [3,inf). + // --------------------------------------------------------------------------- + + @Data + public static class RetypeV1 { + private int tag; // present in v1, v2 + } + + @Data + @ForySchema(removedFields = RetypeV3.History.class) + public static class RetypeV3 { + @ForyVersion(since = 3) + private String tag; + + interface History { + @ForyVersion(until = 3) + int tag(); + } + } + + @Test + public void retypedSameNameAcrossVersions() { + RowEncoder writer = + Encoders.buildBeanCodec(RetypeV1.class).withSchemaEvolution().build().get(); + RowEncoder reader = + Encoders.buildBeanCodec(RetypeV3.class).withSchemaEvolution().build().get(); + RetypeV1 in = new RetypeV1(); + in.setTag(7); + RetypeV3 out = reader.decode(writer.encode(in)); + // The 'tag' on the wire was int and is dropped during projection; the v3 String 'tag' has + // no source in this payload so defaults to null. + Assert.assertNull(out.getTag()); + } + + // --------------------------------------------------------------------------- + // Wide schema (more than 64 fields) crossing the null-bitmap word boundary. + // --------------------------------------------------------------------------- + + @Data + public static class WideV1 { + private int f00, f01, f02, f03, f04, f05, f06, f07, f08, f09; + private int f10, f11, f12, f13, f14, f15, f16, f17, f18, f19; + private int f20, f21, f22, f23, f24, f25, f26, f27, f28, f29; + private int f30, f31, f32, f33, f34, f35, f36, f37, f38, f39; + private int f40, f41, f42, f43, f44, f45, f46, f47, f48, f49; + private int f50, f51, f52, f53, f54, f55, f56, f57, f58, f59; + private int f60, f61, f62, f63, f64, f65, f66, f67; + } + + @Data + public static class WideV2 { + private int f00, f01, f02, f03, f04, f05, f06, f07, f08, f09; + private int f10, f11, f12, f13, f14, f15, f16, f17, f18, f19; + private int f20, f21, f22, f23, f24, f25, f26, f27, f28, f29; + private int f30, f31, f32, f33, f34, f35, f36, f37, f38, f39; + private int f40, f41, f42, f43, f44, f45, f46, f47, f48, f49; + private int f50, f51, f52, f53, f54, f55, f56, f57, f58, f59; + private int f60, f61, f62, f63, f64, f65, f66, f67; + + @ForyVersion(since = 2) + private String extra; + } + + @Test + public void wideSchemaAcrossBitmapWord() { + RowEncoder writer = + Encoders.buildBeanCodec(WideV1.class).withSchemaEvolution().build().get(); + RowEncoder reader = + Encoders.buildBeanCodec(WideV2.class).withSchemaEvolution().build().get(); + WideV1 in = new WideV1(); + in.setF00(100); + in.setF63(163); + in.setF67(167); // past the first 64-bit bitmap word + WideV2 out = reader.decode(writer.encode(in)); + Assert.assertEquals(out.getF00(), 100); + Assert.assertEquals(out.getF63(), 163); + Assert.assertEquals(out.getF67(), 167); + Assert.assertNull(out.getExtra()); + } + + // --------------------------------------------------------------------------- + // Many elements through a single projection codec: 100 elements written by the + // same older version must all decode correctly via the same projection codec, + // with each element's data preserved and no carry-over of state across slots. + // --------------------------------------------------------------------------- + + @Test + public void arrayManyElementsThroughOneProjection() { + ArrayEncoder> writer = + Encoders.buildArrayCodec(new org.apache.fory.reflect.TypeRef>() {}) + .withSchemaEvolution() + .build() + .get(); + ArrayEncoder> reader = + Encoders.buildArrayCodec(new org.apache.fory.reflect.TypeRef>() {}) + .withSchemaEvolution() + .build() + .get(); + List in = new ArrayList<>(); + for (int i = 0; i < 100; i++) { + ChainV2 e = new ChainV2(); + e.setA(i); + e.setB("elem-" + i); + in.add(e); + } + List out = reader.decode(writer.encode(in)); + Assert.assertEquals(out.size(), 100); + for (int i = 0; i < 100; i++) { + Assert.assertEquals(out.get(i).getB(), "elem-" + i); + Assert.assertEquals(out.get(i).getC(), 0L); + Assert.assertFalse(out.get(i).isE()); + } + } + + // --------------------------------------------------------------------------- + // Sanity: two readers for the same (class, history) co-exist without + // interfering. The two readers share the cached generated codec class (by + // design of the codec cache), so the test exercises whether + // BinaryRowEncoder's per-instance projection map and current-codec instance + // are correctly per-reader rather than accidentally shared. + // --------------------------------------------------------------------------- + + @Test + public void twoIndependentReadersForSameClass() { + RowEncoder writer = + Encoders.buildBeanCodec(DefaultsV1.class).withSchemaEvolution().build().get(); + RowEncoder r1 = + Encoders.buildBeanCodec(DefaultsV2.class).withSchemaEvolution().build().get(); + RowEncoder r2 = + Encoders.buildBeanCodec(DefaultsV2.class).withSchemaEvolution().build().get(); + DefaultsV1 in1 = new DefaultsV1(); + in1.setName("first"); + DefaultsV1 in2 = new DefaultsV1(); + in2.setName("second"); + byte[] b1 = writer.encode(in1); + byte[] b2 = writer.encode(in2); + Assert.assertEquals(r1.decode(b1).getName(), "first"); + Assert.assertEquals(r2.decode(b2).getName(), "second"); + Assert.assertEquals(r1.decode(b2).getName(), "second"); + Assert.assertEquals(r2.decode(b1).getName(), "first"); + } + + // --------------------------------------------------------------------------- + // Schema-history misconfiguration: overlapping windows for the same name + // must fail builder construction, not at first bad payload. + // --------------------------------------------------------------------------- + + @Data + @ForySchema(removedFields = OverlapMisconfig.History.class) + public static class OverlapMisconfig { + // Live field 'x' since 1 (default) collides with the removed window [1, 5). + private int x; + + interface History { + @ForyVersion(since = 1, until = 5) + int x(); + } + } + + @Test(expectedExceptions = IllegalStateException.class) + public void overlappingWindowFailsAtBuild() { + Encoders.buildBeanCodec(OverlapMisconfig.class).withSchemaEvolution().build().get(); + } + + // --------------------------------------------------------------------------- + // Roundtrip a List field nested inside a versioned outer record. + // Verifies the projection codec generated for the outer correctly handles + // an inline list of plain beans whose layout is fixed. + // --------------------------------------------------------------------------- + + @Data + public static class NestedListV1 { + private List items; + } + + @Data + public static class NestedListV2 { + private List items; + + @ForyVersion(since = 2) + private String tag; + } + + // --------------------------------------------------------------------------- + // Evolution flag asymmetry: same class, one side opt-in, the other not. + // Documented as wire-incompatible. Verify the failure mode is a clear + // ClassNotCompatibleException, not silent garbage. + // --------------------------------------------------------------------------- + + @Test + public void evolutionFlagAsymmetryFailsLoud() { + RowEncoder withFlag = + Encoders.buildBeanCodec(DefaultsV1.class).withSchemaEvolution().build().get(); + RowEncoder noFlag = Encoders.buildBeanCodec(DefaultsV1.class).build().get(); + DefaultsV1 in = new DefaultsV1(); + in.setName("hi"); + byte[] withFlagBytes = withFlag.encode(in); + try { + noFlag.decode(withFlagBytes); + Assert.fail("expected ClassNotCompatibleException"); + } catch (ClassNotCompatibleException expected) { + // ok + } + byte[] noFlagBytes = noFlag.encode(in); + try { + withFlag.decode(noFlagBytes); + Assert.fail("expected ClassNotCompatibleException"); + } catch (ClassNotCompatibleException expected) { + // ok + } + } + + // --------------------------------------------------------------------------- + // Map with a versioned bean as the KEY (rare; documented as not dispatched). + // Verify the codec at least builds and the current-version round-trip works, + // confirming the documented behavior doesn't crash. + // --------------------------------------------------------------------------- + + @Test + public void mapWithVersionedKey() { + MapEncoder> codec = + Encoders.buildMapCodec( + new org.apache.fory.reflect.TypeRef>() {}) + .withSchemaEvolution() + .build() + .get(); + DefaultsV2 k = new DefaultsV2(); + k.setName("k"); + k.setPrimitiveCount(1); + k.setBoxedCount(2); + java.util.Map in = new java.util.HashMap<>(); + in.put(k, "v"); + java.util.Map out = codec.decode(codec.encode(in)); + Assert.assertEquals(out.size(), 1); + DefaultsV2 outKey = out.keySet().iterator().next(); + Assert.assertEquals(outKey.getName(), "k"); + Assert.assertEquals(outKey.getPrimitiveCount(), 1); + Assert.assertEquals(outKey.getBoxedCount(), Integer.valueOf(2)); + } + + // --------------------------------------------------------------------------- + // Removed nullable struct that was null on the wire: the v1 writer leaves + // the slot's null bit set; the v2 reader skips the slot during projection. + // --------------------------------------------------------------------------- + + @Data + public static class NullableStructV1 { + private String id; + private DefaultsV1 detail; // nullable, removed at v2 + } + + @Data + @ForySchema(removedFields = NullableStructV2.History.class) + public static class NullableStructV2 { + private String id; + + interface History { + @ForyVersion(until = 2) + DefaultsV1 detail(); + } + } + + @Test + public void removedNullableStructWasNullOnWire() { + RowEncoder writer = + Encoders.buildBeanCodec(NullableStructV1.class).withSchemaEvolution().build().get(); + RowEncoder reader = + Encoders.buildBeanCodec(NullableStructV2.class).withSchemaEvolution().build().get(); + NullableStructV1 in = new NullableStructV1(); + in.setId("only-id"); + // detail intentionally left null + NullableStructV2 out = reader.decode(writer.encode(in)); + Assert.assertEquals(out.getId(), "only-id"); + } + + // --------------------------------------------------------------------------- + // Builder method ordering: compactEncoding() before vs after withSchemaEvolution() + // must produce equivalent codecs. + // --------------------------------------------------------------------------- + + @Test + public void builderMethodOrderingIsCommutative() { + RowEncoder w = + Encoders.buildBeanCodec(DefaultsV1.class) + .compactEncoding() + .withSchemaEvolution() + .build() + .get(); + RowEncoder rOrderA = + Encoders.buildBeanCodec(DefaultsV2.class) + .compactEncoding() + .withSchemaEvolution() + .build() + .get(); + RowEncoder rOrderB = + Encoders.buildBeanCodec(DefaultsV2.class) + .withSchemaEvolution() + .compactEncoding() + .build() + .get(); + DefaultsV1 in = new DefaultsV1(); + in.setName("commute"); + byte[] bytes = w.encode(in); + Assert.assertEquals(rOrderA.decode(bytes).getName(), "commute"); + Assert.assertEquals(rOrderB.decode(bytes).getName(), "commute"); + } + + @Test + public void nestedListSurvivesOuterProjection() { + RowEncoder writer = + Encoders.buildBeanCodec(NestedListV1.class).withSchemaEvolution().build().get(); + RowEncoder reader = + Encoders.buildBeanCodec(NestedListV2.class).withSchemaEvolution().build().get(); + DefaultsV1 a = new DefaultsV1(); + a.setName("a"); + DefaultsV1 b = new DefaultsV1(); + b.setName("b"); + NestedListV1 in = new NestedListV1(); + in.setItems(Arrays.asList(a, b)); + NestedListV2 out = reader.decode(writer.encode(in)); + Assert.assertEquals(out.getItems().size(), 2); + Assert.assertEquals(out.getItems().get(0).getName(), "a"); + Assert.assertEquals(out.getItems().get(1).getName(), "b"); + Assert.assertNull(out.getTag()); + } + + // --------------------------------------------------------------------------- + // Nested versioned bean: a parent bean with a struct field whose own type is + // versioned independently. The wire layout for the inner struct is inline in + // the parent's bytes with no per-inner hash. The reader, dispatching on the + // parent's strict hash, needs to choose an inner schema consistent with what + // the writer used. + // --------------------------------------------------------------------------- + + /** Stand-in for "older code that wrote the inner struct without field x". */ + @Data + public static class NestedInnerWriter { + private String name; + } + + /** Stand-in for "older code that wrote the outer containing NestedInnerWriter". */ + @Data + public static class NestedOuterWriter { + private long id; + private NestedInnerWriter inner; + } + + /** Newer inner with an added field at v2. */ + @Data + public static class NestedInnerV2 { + private String name; + + @ForyVersion(since = 2) + private String addedField; + } + + /** Newer outer that still has just (id, inner) but its inner type evolved. */ + @Data + public static class NestedOuterV2 { + private long id; + private NestedInnerV2 inner; + } + + // TODO: nested versioned beans inside another versioned bean are not yet dispatched. The + // strict hash naturally encodes inner-struct shape, but SchemaHistory.build does not + // currently cross-product over nested-bean versions, so no projection codec is generated for + // the older inner shape. Re-enable when implemented. + @Test(enabled = false) + public void nestedInnerEvolution_readerInnerNewerThanWriter() { + // Writer uses the "older shape" inner. Both writer and reader are evolution-on so they + // agree on strict-hash framing. + RowEncoder writer = + Encoders.buildBeanCodec(NestedOuterWriter.class).withSchemaEvolution().build().get(); + RowEncoder reader = + Encoders.buildBeanCodec(NestedOuterV2.class).withSchemaEvolution().build().get(); + + NestedOuterWriter in = new NestedOuterWriter(); + in.setId(42); + NestedInnerWriter inn = new NestedInnerWriter(); + inn.setName("hello"); + in.setInner(inn); + + byte[] bytes = writer.encode(in); + NestedOuterV2 out = reader.decode(bytes); + Assert.assertEquals(out.getId(), 42); + Assert.assertNotNull(out.getInner()); + Assert.assertEquals(out.getInner().getName(), "hello"); + Assert.assertNull(out.getInner().getAddedField()); + } +} + diff --git a/java/fory-format/src/test/java/org/apache/fory/format/encoder/SchemaEvolutionTest.java b/java/fory-format/src/test/java/org/apache/fory/format/encoder/SchemaEvolutionTest.java new file mode 100644 index 0000000000..29eb1e7488 --- /dev/null +++ b/java/fory-format/src/test/java/org/apache/fory/format/encoder/SchemaEvolutionTest.java @@ -0,0 +1,496 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.fory.format.encoder; + +import java.util.Arrays; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import lombok.Data; +import org.apache.fory.format.annotation.ForySchema; +import org.apache.fory.format.annotation.ForyVersion; +import org.apache.fory.reflect.TypeRef; +import org.testng.Assert; +import org.testng.annotations.Test; + +public class SchemaEvolutionTest { + + /** Original v1 bean: just a name and an age. */ + @Data + public static class PersonV1 { + private String name; + private int age; + } + + /** + * v2: added an email. The codec built against this class must still be able to read v1 payloads + * (email will default to null). + */ + @Data + public static class PersonV2 { + private String name; + private int age; + + @ForyVersion(since = 2) + private String email; + } + + /** + * v3: same as v2 with the age field removed. The codec built against this class must read v1 + * payloads (with age) and v2 payloads (with age + email). + */ + @Data + @ForySchema(removedFields = PersonV3.History.class) + public static class PersonV3 { + private String name; + + @ForyVersion(since = 2) + private String email; + + interface History { + @ForyVersion(until = 3) + int age(); + } + } + + /** Round-trip at the current version: writing PersonV2, reading PersonV2 with evolution on. */ + @Test + public void currentVersionRoundTrip() { + RowEncoder codec = + Encoders.buildBeanCodec(PersonV2.class).withSchemaEvolution().build().get(); + PersonV2 in = new PersonV2(); + in.setName("alice"); + in.setAge(30); + in.setEmail("alice@example.com"); + byte[] bytes = codec.encode(in); + PersonV2 out = codec.decode(bytes); + Assert.assertEquals(out.getName(), "alice"); + Assert.assertEquals(out.getAge(), 30); + Assert.assertEquals(out.getEmail(), "alice@example.com"); + } + + /** + * The crux: a payload produced by PersonV1 (literally a different Java class with the + * v1-shaped schema) decoded by PersonV2's evolution-enabled codec. We use PersonV1 as a + * stand-in for "what older code wrote." Both classes are encoded with schema evolution on so + * they share the strict-hash format; PersonV1's history is a single entry, and PersonV2's + * history contains both v1 (without email) and v2 (with email) entries that match PersonV1's + * single entry by hash. + */ + @Test + public void olderPayloadReadByNewerCodec() { + RowEncoder oldWriter = + Encoders.buildBeanCodec(PersonV1.class).withSchemaEvolution().build().get(); + RowEncoder newReader = + Encoders.buildBeanCodec(PersonV2.class).withSchemaEvolution().build().get(); + + PersonV1 in = new PersonV1(); + in.setName("alice"); + in.setAge(30); + byte[] bytes = oldWriter.encode(in); + + PersonV2 out = newReader.decode(bytes); + Assert.assertEquals(out.getName(), "alice"); + Assert.assertEquals(out.getAge(), 30); + Assert.assertNull(out.getEmail()); + } + + // --- Compact row format --- + + @Test + public void compactRowOlderPayloadReadByNewerCodec() { + RowEncoder oldWriter = + Encoders.buildBeanCodec(PersonV1.class) + .compactEncoding() + .withSchemaEvolution() + .build() + .get(); + RowEncoder newReader = + Encoders.buildBeanCodec(PersonV2.class) + .compactEncoding() + .withSchemaEvolution() + .build() + .get(); + PersonV1 in = new PersonV1(); + in.setName("bob"); + in.setAge(42); + byte[] bytes = oldWriter.encode(in); + PersonV2 out = newReader.decode(bytes); + Assert.assertEquals(out.getName(), "bob"); + Assert.assertEquals(out.getAge(), 42); + Assert.assertNull(out.getEmail()); + } + + // --- Array of versioned beans --- + + @Test + public void arrayStandardOlderPayloadReadByNewerCodec() { + ArrayEncoder> oldWriter = + Encoders.buildArrayCodec(new TypeRef>() {}) + .withSchemaEvolution() + .build() + .get(); + ArrayEncoder> newReader = + Encoders.buildArrayCodec(new TypeRef>() {}) + .withSchemaEvolution() + .build() + .get(); + PersonV1 a = new PersonV1(); + a.setName("alice"); + a.setAge(30); + PersonV1 b = new PersonV1(); + b.setName("bob"); + b.setAge(42); + byte[] bytes = oldWriter.encode(Arrays.asList(a, b)); + List out = newReader.decode(bytes); + Assert.assertEquals(out.size(), 2); + Assert.assertEquals(out.get(0).getName(), "alice"); + Assert.assertEquals(out.get(0).getAge(), 30); + Assert.assertNull(out.get(0).getEmail()); + Assert.assertEquals(out.get(1).getName(), "bob"); + } + + @Test + public void arrayCompactOlderPayloadReadByNewerCodec() { + ArrayEncoder> oldWriter = + Encoders.buildArrayCodec(new TypeRef>() {}) + .compactEncoding() + .withSchemaEvolution() + .build() + .get(); + ArrayEncoder> newReader = + Encoders.buildArrayCodec(new TypeRef>() {}) + .compactEncoding() + .withSchemaEvolution() + .build() + .get(); + PersonV1 p = new PersonV1(); + p.setName("carol"); + p.setAge(25); + byte[] bytes = oldWriter.encode(Arrays.asList(p)); + List out = newReader.decode(bytes); + Assert.assertEquals(out.size(), 1); + Assert.assertEquals(out.get(0).getName(), "carol"); + Assert.assertEquals(out.get(0).getAge(), 25); + Assert.assertNull(out.get(0).getEmail()); + } + + // --- Map with versioned bean values --- + + @Test + public void mapStandardOlderPayloadReadByNewerCodec() { + MapEncoder> oldWriter = + Encoders.buildMapCodec(new TypeRef>() {}) + .withSchemaEvolution() + .build() + .get(); + MapEncoder> newReader = + Encoders.buildMapCodec(new TypeRef>() {}) + .withSchemaEvolution() + .build() + .get(); + Map in = new HashMap<>(); + PersonV1 p = new PersonV1(); + p.setName("dave"); + p.setAge(40); + in.put("k1", p); + byte[] bytes = oldWriter.encode(in); + Map out = newReader.decode(bytes); + Assert.assertEquals(out.size(), 1); + Assert.assertEquals(out.get("k1").getName(), "dave"); + Assert.assertEquals(out.get("k1").getAge(), 40); + Assert.assertNull(out.get("k1").getEmail()); + } + + @Test + public void mapCompactOlderPayloadReadByNewerCodec() { + MapEncoder> oldWriter = + Encoders.buildMapCodec(new TypeRef>() {}) + .compactEncoding() + .withSchemaEvolution() + .build() + .get(); + MapEncoder> newReader = + Encoders.buildMapCodec(new TypeRef>() {}) + .compactEncoding() + .withSchemaEvolution() + .build() + .get(); + Map in = new HashMap<>(); + PersonV1 p = new PersonV1(); + p.setName("eve"); + p.setAge(28); + in.put("k1", p); + byte[] bytes = oldWriter.encode(in); + Map out = newReader.decode(bytes); + Assert.assertEquals(out.get("k1").getName(), "eve"); + Assert.assertEquals(out.get("k1").getAge(), 28); + Assert.assertNull(out.get("k1").getEmail()); + } + + // --- Interface-typed beans --- + // + // The wire field name is derived from each interface's accessor method name (via + // lowerCamelToLowerUnderscore), so two interfaces that share the same accessor names produce + // the same wire layout. Use accessor-style getters consistently across versions. + + /** v1 interface: just name and age. */ + public interface PersonIfaceV1 { + String getName(); + + int getAge(); + } + + /** v2 interface: adds email. Same accessor naming so the wire field names match. */ + public interface PersonIfaceV2 { + String getName(); + + int getAge(); + + @ForyVersion(since = 2) + String getEmail(); + } + + @Test + public void interfaceOlderPayloadReadByNewerCodec() { + RowEncoder oldWriter = + Encoders.buildBeanCodec(PersonIfaceV1.class).withSchemaEvolution().build().get(); + RowEncoder newReader = + Encoders.buildBeanCodec(PersonIfaceV2.class).withSchemaEvolution().build().get(); + PersonIfaceV1 in = + new PersonIfaceV1() { + public String getName() { + return "alice"; + } + + public int getAge() { + return 30; + } + }; + byte[] bytes = oldWriter.encode(in); + PersonIfaceV2 out = newReader.decode(bytes); + Assert.assertEquals(out.getName(), "alice"); + Assert.assertEquals(out.getAge(), 30); + // email was added in v2; v1 payload has none. The interface proxy returns the default. + Assert.assertNull(out.getEmail()); + } + + /** + * v3 interface: name and email; age removed (only present in v1 and v2). The history interface + * declares the removed field's original signature; its method name follows the same JavaBeans + * accessor convention as the live interface, so {@code getAge()} maps to wire name {@code age}. + */ + @ForySchema(removedFields = PersonIfaceV3.History.class) + public interface PersonIfaceV3 { + String getName(); + + @ForyVersion(since = 2) + String getEmail(); + + interface History { + @ForyVersion(until = 3) + int getAge(); + } + } + + @Test + public void interfaceRemovedFieldReadByNewerCodec() { + RowEncoder v2Writer = + Encoders.buildBeanCodec(PersonIfaceV2.class).withSchemaEvolution().build().get(); + RowEncoder v3Reader = + Encoders.buildBeanCodec(PersonIfaceV3.class).withSchemaEvolution().build().get(); + PersonIfaceV2 in = + new PersonIfaceV2() { + public String getName() { + return "alice"; + } + + public int getAge() { + return 30; + } + + public String getEmail() { + return "alice@example.com"; + } + }; + byte[] bytes = v2Writer.encode(in); + PersonIfaceV3 out = v3Reader.decode(bytes); + Assert.assertEquals(out.getName(), "alice"); + Assert.assertEquals(out.getEmail(), "alice@example.com"); + } + + /** Removed-field test: v3 codec reads v2 payload, dropping the no-longer-present 'age'. */ + @Test + public void removedFieldReadByNewerCodec() { + RowEncoder v2Writer = + Encoders.buildBeanCodec(PersonV2.class).withSchemaEvolution().build().get(); + RowEncoder v3Reader = + Encoders.buildBeanCodec(PersonV3.class).withSchemaEvolution().build().get(); + + PersonV2 in = new PersonV2(); + in.setName("alice"); + in.setAge(30); + in.setEmail("alice@example.com"); + byte[] bytes = v2Writer.encode(in); + + PersonV3 out = v3Reader.decode(bytes); + Assert.assertEquals(out.getName(), "alice"); + Assert.assertEquals(out.getEmail(), "alice@example.com"); + } + + // --------------------------------------------------------------------------- + // Compositional test + // + // Outer mutable bean evolves v1 -> v2 (adds displayName, removes legacyName). + // The bean carries diverse nested data shapes that themselves do not evolve: + // a concrete struct, an interface-typed struct (lazy proxy), an inline list + // of structs, and an inline map. The test exercises one + // dispatch boundary (the outer codec, or the outer list codec) and verifies + // that the projected outer correctly carries every nested shape through. + // --------------------------------------------------------------------------- + + @Data + public static class Profile { + private String bio; + private int rating; + } + + /** Address is interface-typed; the row codec generates a lazy proxy for reads. */ + public interface Address { + String getStreet(); + + String getCity(); + } + + @Data + public static class Item { + private String name; + private long quantity; + } + + @Data + public static class OuterV1 { + private long id; + private String legacyName; + private Profile profile; + private Address address; + private List items; + private Map properties; + } + + /** + * OuterV2 adds {@code displayName} at version 2 and removes {@code legacyName} at version 2. + * Everything else carries forward unchanged. The compositional test writes an OuterV1 and + * reads as OuterV2. + */ + @Data + @ForySchema(removedFields = OuterV2.History.class) + public static class OuterV2 { + private long id; + + @ForyVersion(since = 2) + private String displayName; + + private Profile profile; + private Address address; + private List items; + private Map properties; + + interface History { + @ForyVersion(until = 2) + String legacyName(); + } + } + + private static OuterV1 sampleV1() { + OuterV1 in = new OuterV1(); + in.setId(7); + in.setLegacyName("retired"); + Profile p = new Profile(); + p.setBio("hello"); + p.setRating(5); + in.setProfile(p); + in.setAddress( + new Address() { + public String getStreet() { + return "1 Main"; + } + + public String getCity() { + return "Springfield"; + } + }); + Item a = new Item(); + a.setName("a"); + a.setQuantity(1); + Item b = new Item(); + b.setName("b"); + b.setQuantity(2); + in.setItems(Arrays.asList(a, b)); + Map props = new HashMap<>(); + props.put("k1", a); + props.put("k2", b); + in.setProperties(props); + return in; + } + + private static void assertProjectedToV2(OuterV2 out) { + Assert.assertEquals(out.getId(), 7); + Assert.assertNull(out.getDisplayName()); // added in v2, absent in v1 wire + Assert.assertEquals(out.getProfile().getBio(), "hello"); + Assert.assertEquals(out.getProfile().getRating(), 5); + Assert.assertEquals(out.getAddress().getStreet(), "1 Main"); + Assert.assertEquals(out.getAddress().getCity(), "Springfield"); + Assert.assertEquals(out.getItems().size(), 2); + Assert.assertEquals(out.getItems().get(0).getName(), "a"); + Assert.assertEquals(out.getItems().get(1).getQuantity(), 2); + Assert.assertEquals(out.getProperties().get("k1").getName(), "a"); + Assert.assertEquals(out.getProperties().get("k2").getQuantity(), 2); + } + + @Test + public void compositionalRowEvolution() { + RowEncoder writer = + Encoders.buildBeanCodec(OuterV1.class).withSchemaEvolution().build().get(); + RowEncoder reader = + Encoders.buildBeanCodec(OuterV2.class).withSchemaEvolution().build().get(); + byte[] bytes = writer.encode(sampleV1()); + assertProjectedToV2(reader.decode(bytes)); + } + + @Test + public void compositionalArrayEvolution() { + ArrayEncoder> writer = + Encoders.buildArrayCodec(new TypeRef>() {}) + .withSchemaEvolution() + .build() + .get(); + ArrayEncoder> reader = + Encoders.buildArrayCodec(new TypeRef>() {}) + .withSchemaEvolution() + .build() + .get(); + byte[] bytes = writer.encode(Arrays.asList(sampleV1(), sampleV1())); + List out = reader.decode(bytes); + Assert.assertEquals(out.size(), 2); + assertProjectedToV2(out.get(0)); + assertProjectedToV2(out.get(1)); + } +} From b221d0eaaf54441f1ac06860a660bee4b33079f6 Mon Sep 17 00:00:00 2001 From: "Claude (on behalf of Steven Schlansker)" Date: Thu, 28 May 2026 19:32:01 +0000 Subject: [PATCH 02/13] fix(format): pass row body size, not full payload size, to BinaryRow.pointTo BinaryRowEncoder.decode consumes an 8-byte schema hash before pointing the row at the rest of the buffer, but passed the full payload size to BinaryRow.pointTo. The row's recorded sizeInBytes was therefore 8 too large, so any subsequent copy(), toBytes(), or getSizeInBytes() call on a row obtained through this path would include 8 trailing bytes of unrelated buffer content. The bug was latent because BinaryRowEncoder.decode does not expose the row to callers; the buffer's reader index was already advanced correctly by size - 8. Fix is local to BinaryRowEncoder.decode. Adds a regression test that injects a size-recording BinaryRow and asserts the encoder passes the row body size. --- .../fory/format/encoder/BinaryRowEncoder.java | 10 +- .../encoder/BinaryRowEncoderPointToTest.java | 203 ++++++++++++++++++ 2 files changed, 209 insertions(+), 4 deletions(-) create mode 100644 java/fory-format/src/test/java/org/apache/fory/format/encoder/BinaryRowEncoderPointToTest.java diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryRowEncoder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryRowEncoder.java index 83d09069d4..600b3acbf6 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryRowEncoder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryRowEncoder.java @@ -103,18 +103,20 @@ public T decode(final MemoryBuffer buffer) { @SuppressWarnings("unchecked") T decode(final MemoryBuffer buffer, final int size) { final long peerSchemaHash = buffer.readInt64(); + // The 8-byte hash has just been consumed; the row body occupies the remaining bytes. + final int rowSize = size - 8; if (peerSchemaHash == schemaHash) { final BinaryRow row = codecFactory.newRow(schema); - row.pointTo(buffer, buffer.readerIndex(), size); - buffer.increaseReaderIndex(size - 8); + row.pointTo(buffer, buffer.readerIndex(), rowSize); + buffer.increaseReaderIndex(rowSize); return fromRow(row); } if (projections != null) { ProjectionCodec projection = projections.get(peerSchemaHash); if (projection != null) { final BinaryRow row = codecFactory.newRow(projection.schema); - row.pointTo(buffer, buffer.readerIndex(), size); - buffer.increaseReaderIndex(size - 8); + row.pointTo(buffer, buffer.readerIndex(), rowSize); + buffer.increaseReaderIndex(rowSize); return (T) projection.codec.fromRow(row); } } diff --git a/java/fory-format/src/test/java/org/apache/fory/format/encoder/BinaryRowEncoderPointToTest.java b/java/fory-format/src/test/java/org/apache/fory/format/encoder/BinaryRowEncoderPointToTest.java new file mode 100644 index 0000000000..7d13f9be00 --- /dev/null +++ b/java/fory-format/src/test/java/org/apache/fory/format/encoder/BinaryRowEncoderPointToTest.java @@ -0,0 +1,203 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.fory.format.encoder; + +import java.util.concurrent.atomic.AtomicInteger; +import lombok.Data; +import org.apache.fory.format.row.binary.BinaryRow; +import org.apache.fory.format.row.binary.writer.BaseBinaryRowWriter; +import org.apache.fory.format.row.binary.writer.BinaryRowWriter; +import org.apache.fory.format.type.Schema; +import org.apache.fory.format.type.TypeInference; +import org.apache.fory.memory.MemoryBuffer; +import org.testng.Assert; +import org.testng.annotations.Test; + +/** + * Regression test for BinaryRowEncoder.decode. The old code passed the full payload size + * (including the 8-byte schema hash) to BinaryRow.pointTo, so the decoded row's sizeInBytes was + * 8 too large. The bug is latent because BinaryRowEncoder.decode does not expose the row, but + * any code that later reads getSizeInBytes(), copy(), or toBytes() on a row constructed via this + * code path would see 8 trailing bytes of unrelated buffer content. + */ +public class BinaryRowEncoderPointToTest { + + @Data + public static class Tiny { + private int x; + } + + /** + * Build a minimal BinaryRowEncoder wired with a hooking Encoding so we can observe the size + * argument passed to BinaryRow.pointTo during decode. + */ + @Test + public void decodePassesRowBodySizeNotPayloadSize() { + Schema schema = TypeInference.inferSchema(Tiny.class); + + // Capture the size argument the encoder passes to BinaryRow.pointTo during decode. + AtomicInteger capturedSize = new AtomicInteger(-1); + Encoding hookingFormat = new HookingEncoding(capturedSize::set); + + // Encode through the normal path to get a valid byte payload. + RowEncoder normal = Encoders.bean(Tiny.class); + Tiny in = new Tiny(); + in.setX(42); + byte[] payload = normal.encode(in); + final int payloadSize = payload.length; + + // Manually build a BinaryRowEncoder using the hooking format. The constructor needs a + // generated codec, which we get by going through the normal builder and then swapping + // codecs is impractical — instead reuse the existing encoder's machinery by giving it the + // hooking format. The simplest path: construct a small encoder directly. + BinaryRowWriter writer = new BinaryRowWriter(schema); + BinaryRowEncoder encoder = newEncoder(schema, hookingFormat, normal, writer); + + encoder.decode(payload); + int observed = capturedSize.get(); + Assert.assertEquals( + observed, + payloadSize - 8, + "BinaryRowEncoder.decode must pass the row body size (payload minus 8-byte hash) to " + + "BinaryRow.pointTo, not the full payload size."); + } + + private static BinaryRowEncoder newEncoder( + Schema schema, + Encoding hookingFormat, + RowEncoder realEncoder, + BaseBinaryRowWriter writer) { + // Wrap the real encoder's generated codec so fromRow still works while the Encoding's + // newRow comes from our hooking factory. + GeneratedRowEncoder wrappedCodec = + new GeneratedRowEncoder() { + @Override + public BinaryRow toRow(Object o) { + return realEncoder.toRow((Tiny) o); + } + + @Override + public Object fromRow(BinaryRow row) { + return realEncoder.fromRow(row); + } + }; + return new BinaryRowEncoder<>(schema, hookingFormat, wrappedCodec, writer, false); + } + + /** Delegates everything to DefaultCodecFormat except newRow, which records the size. */ + private static final class HookingEncoding implements Encoding { + private final java.util.function.IntConsumer sizeRecorder; + + HookingEncoding(java.util.function.IntConsumer sizeRecorder) { + this.sizeRecorder = sizeRecorder; + } + + @Override + public BinaryRow newRow(Schema schema) { + return new BinaryRow(schema) { + @Override + public void pointTo(MemoryBuffer buffer, int offset, int sizeInBytes) { + sizeRecorder.accept(sizeInBytes); + super.pointTo(buffer, offset, sizeInBytes); + } + }; + } + + // The rest is uninteresting — delegate to the standard format. + private final Encoding delegate = DefaultCodecFormat.INSTANCE; + + @Override + public BaseBinaryRowWriter newWriter(Schema schema) { + return delegate.newWriter(schema); + } + + @Override + public BaseBinaryRowWriter newWriter(Schema schema, MemoryBuffer buffer) { + return delegate.newWriter(schema, buffer); + } + + @Override + public org.apache.fory.format.row.binary.writer.BinaryArrayWriter newArrayWriter( + org.apache.fory.format.type.Field field) { + return delegate.newArrayWriter(field); + } + + @Override + public org.apache.fory.format.row.binary.writer.BinaryArrayWriter newArrayWriter( + org.apache.fory.format.type.Field field, MemoryBuffer buffer) { + return delegate.newArrayWriter(field, buffer); + } + + @Override + public RowEncoderBuilder newRowEncoder(org.apache.fory.reflect.TypeRef beanType) { + return delegate.newRowEncoder(beanType); + } + + @Override + public RowEncoderBuilder newProjectionRowEncoder( + org.apache.fory.reflect.TypeRef beanType, + Schema historicalSchema, + java.util.Set liveNames, + String classSuffix) { + return delegate.newProjectionRowEncoder(beanType, historicalSchema, liveNames, classSuffix); + } + + @Override + public ArrayEncoderBuilder newArrayEncoder( + org.apache.fory.reflect.TypeRef> collectionType, + org.apache.fory.reflect.TypeRef elementType) { + return delegate.newArrayEncoder(collectionType, elementType); + } + + @Override + public ArrayEncoderBuilder newProjectionArrayEncoder( + org.apache.fory.reflect.TypeRef> collectionType, + org.apache.fory.reflect.TypeRef elementType, + String rowCodecSuffix) { + return delegate.newProjectionArrayEncoder(collectionType, elementType, rowCodecSuffix); + } + + @Override + public MapEncoderBuilder newMapEncoder( + org.apache.fory.reflect.TypeRef> mapType, + org.apache.fory.reflect.TypeRef beanToken) { + return delegate.newMapEncoder(mapType, beanToken); + } + + @Override + public MapEncoderBuilder newProjectionMapEncoder( + org.apache.fory.reflect.TypeRef> mapType, + org.apache.fory.reflect.TypeRef beanToken, + String rowCodecSuffix) { + return delegate.newProjectionMapEncoder(mapType, beanToken, rowCodecSuffix); + } + + @Override + public org.apache.fory.format.row.binary.BinaryArray newArray( + org.apache.fory.format.type.Field field) { + return delegate.newArray(field); + } + + @Override + public org.apache.fory.format.row.binary.BinaryMap newMap( + org.apache.fory.format.type.Field field) { + return delegate.newMap(field); + } + } +} From 9bef6713e8f2091705ea6f8f93faaef5097e48c4 Mon Sep 17 00:00:00 2001 From: "Claude (on behalf of Steven Schlansker)" Date: Thu, 28 May 2026 19:34:57 +0000 Subject: [PATCH 03/13] perf(format): one allocation per encode in evolution-enabled array/map codecs BinaryArrayEncoder.encode(T) and BinaryMapEncoder.encode(T) previously composed the hash-prefixed payload through MemoryUtils.buffer + writeInt64 + writeBytes + getBytes, allocating three byte[] copies and a MemoryBuffer per call. Build the result directly into a single byte[]: wrap it to write the 8-byte hash header, then System.arraycopy the body in. The non-evolution paths are unchanged. --- .../fory/format/encoder/BinaryArrayEncoder.java | 13 ++++++++----- .../fory/format/encoder/BinaryMapEncoder.java | 13 ++++++++----- 2 files changed, 16 insertions(+), 10 deletions(-) diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryArrayEncoder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryArrayEncoder.java index c42b6c3215..20aa7d8444 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryArrayEncoder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryArrayEncoder.java @@ -24,6 +24,7 @@ import org.apache.fory.format.row.binary.BinaryArray; import org.apache.fory.format.row.binary.writer.BinaryArrayWriter; import org.apache.fory.format.type.Field; +import org.apache.fory.memory.LittleEndian; import org.apache.fory.memory.MemoryBuffer; import org.apache.fory.memory.MemoryUtils; @@ -134,11 +135,13 @@ public byte[] encode(final T obj) { if (projections == null) { return writer.getBuffer().getBytes(0, array.getSizeInBytes()); } - int n = array.getSizeInBytes(); - MemoryBuffer out = MemoryUtils.buffer(8 + n); - out.writeInt64(currentHash); - out.writeBytes(writer.getBuffer().getBytes(0, n)); - return out.getBytes(0, 8 + n); + // Build the result with a single allocation: the result byte[]. The hash header is poked + // in via LittleEndian (no buffer wrapper) and the body is copied in via System.arraycopy. + final int n = array.getSizeInBytes(); + final byte[] result = new byte[8 + n]; + LittleEndian.putInt64(result, 0, currentHash); + writer.getBuffer().get(0, result, 8, n); + return result; } @Override diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryMapEncoder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryMapEncoder.java index 5f3b5a88b7..9bec2a170c 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryMapEncoder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryMapEncoder.java @@ -25,6 +25,7 @@ import org.apache.fory.format.row.binary.BinaryMap; import org.apache.fory.format.row.binary.writer.BinaryArrayWriter; import org.apache.fory.format.type.Field; +import org.apache.fory.memory.LittleEndian; import org.apache.fory.memory.MemoryBuffer; import org.apache.fory.memory.MemoryUtils; @@ -152,11 +153,13 @@ public byte[] encode(final M obj) { if (projections == null) { return map.getBuf().getBytes(map.getBaseOffset(), map.getSizeInBytes()); } - int n = map.getSizeInBytes(); - MemoryBuffer out = MemoryUtils.buffer(8 + n); - out.writeInt64(currentHash); - out.writeBytes(map.getBuf().getBytes(map.getBaseOffset(), n)); - return out.getBytes(0, 8 + n); + // Build the result with a single allocation: the result byte[]. The hash header is poked + // in via LittleEndian (no buffer wrapper) and the body is copied in via System.arraycopy. + final int n = map.getSizeInBytes(); + final byte[] result = new byte[8 + n]; + LittleEndian.putInt64(result, 0, currentHash); + map.getBuf().get(map.getBaseOffset(), result, 8, n); + return result; } @Override From cee2fe2fe9d32505141f5e863263ca1dd566075b Mon Sep 17 00:00:00 2001 From: "Claude (on behalf of Steven Schlansker)" Date: Thu, 28 May 2026 20:11:50 +0000 Subject: [PATCH 04/13] perf(format): hoist compact row layout to avoid per-element schema recomputation CompactBinaryArray.getStruct and CompactBinaryRow.getStruct previously rebuilt the element schema's fixedOffsets array and walked allNotNullable twice on every call, even though the schema itself was already cached at the slot. For a list of 32 structs that meant 32 int[] allocations plus duplicate field-list iterations. Move the per-schema work into a new CompactRowLayout holder. CompactBinaryArray computes its element's layout once at construction (the element type is fixed for the lifetime of the array); CompactBinaryRow.getStruct caches a layout per slot in extData. Add a package-private CompactBinaryRow constructor that takes precomputed offsets / nullability / bitmap width so the row allocation skips the two redundant iterations. Measured allocation reduction with org.apache.fory.format.perf.RowFormatAllocationProbe (included for repeatability). Bytes allocated per decode op: scenario standard compact (before -> after) root 72868 110170 -> 87516 (-21%) array 6639 9982 -> 7856 (-21%) matrix 52704 79664 -> 62968 (-21%) map 6080 8168 -> 7216 (-12%) Standard format unchanged (control). Per-element saving ~65 bytes, consistent with the eliminated int[fields+1] alloc plus avoided iteration work. --- .../fory/format/row/binary/BinaryRow.java | 13 + .../format/row/binary/CompactBinaryArray.java | 35 ++- .../format/row/binary/CompactBinaryRow.java | 54 +++- .../format/row/binary/CompactRowLayout.java | 61 +++++ .../format/perf/RowFormatAllocationProbe.java | 236 ++++++++++++++++++ 5 files changed, 383 insertions(+), 16 deletions(-) create mode 100644 java/fory-format/src/main/java/org/apache/fory/format/row/binary/CompactRowLayout.java create mode 100644 java/fory-format/src/test/java/org/apache/fory/format/perf/RowFormatAllocationProbe.java diff --git a/java/fory-format/src/main/java/org/apache/fory/format/row/binary/BinaryRow.java b/java/fory-format/src/main/java/org/apache/fory/format/row/binary/BinaryRow.java index dc099913b7..628c5de490 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/row/binary/BinaryRow.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/row/binary/BinaryRow.java @@ -76,6 +76,19 @@ public BinaryRow(Schema schema) { initializeExtData(numFields); } + /** + * Allocate a BinaryRow with a precomputed bitmap width. Subclasses that already know the + * width (e.g. {@link CompactBinaryRow} constructed from a cached schema) use this to skip + * the per-call iteration in {@link #computeBitmapWidthInBytes}. + */ + protected BinaryRow(Schema schema, int bitmapWidthInBytes) { + this.schema = schema; + this.numFields = schema.numFields(); + Preconditions.checkArgument(numFields > 0); + this.bitmapWidthInBytes = bitmapWidthInBytes; + initializeExtData(numFields); + } + protected int computeBitmapWidthInBytes() { return BitUtils.calculateBitmapWidthInBytes(numFields); } diff --git a/java/fory-format/src/main/java/org/apache/fory/format/row/binary/CompactBinaryArray.java b/java/fory-format/src/main/java/org/apache/fory/format/row/binary/CompactBinaryArray.java index 40d10de9c2..4bb58b3217 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/row/binary/CompactBinaryArray.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/row/binary/CompactBinaryArray.java @@ -33,12 +33,26 @@ public class CompactBinaryArray extends BinaryArray { private final boolean elementNullable; private int headerInBytes; + /** + * Hoisted element-struct metadata. Computed once at construction so each {@link #getStruct} + * call can build a row without re-deriving the schema layout. {@code null} when the element + * is not a fixed-width struct. + */ + private final CompactRowLayout elementLayout; + public CompactBinaryArray(final Field field) { super(field, CompactBinaryArrayWriter.elementWidth(field)); DataTypes.ListType listType = (DataTypes.ListType) field.type(); elementField = listType.valueField(); fixedWidth = CompactBinaryRowWriter.fixedWidthFor(elementField); elementNullable = elementField.nullable(); + if (elementField.type() instanceof DataTypes.StructType) { + elementLayout = + new CompactRowLayout( + CompactBinaryRowWriter.sortSchema(DataTypes.schemaFromStructField(elementField))); + } else { + elementLayout = null; + } } @Override @@ -105,14 +119,16 @@ protected BinaryRow getStruct(final int ordinal, final Field field, final int ex return null; } assert field == elementField; + final CompactBinaryRow row = elementLayout.newRow(); if (fixedWidth == -1) { - return super.getStruct(ordinal, field, extDataSlot); + // Variable-width element: read the (offset, size) slot and point at it. + final long offsetAndSize = getInt64(ordinal); + final int relativeOffset = (int) (offsetAndSize >> 32); + final int size = (int) offsetAndSize; + row.pointTo(getBuffer(), getBaseOffset() + relativeOffset, size); + } else { + row.pointTo(getBuffer(), getOffset(ordinal), fixedWidth); } - if (extData[extDataSlot] == null) { - extData[extDataSlot] = newSchema(field); - } - final BinaryRow row = newRow((Schema) extData[extDataSlot]); - row.pointTo(getBuffer(), getOffset(ordinal), fixedWidth); return row; } @@ -123,7 +139,12 @@ protected Schema newSchema(final Field field) { @Override protected BinaryRow newRow(final Schema schema) { - // TODO: don't re-compute fixed offsets + // The variable-width getStruct path lands here. The element schema is fixed for this + // array, so reuse the hoisted layout when the schema matches; fall back to a one-off + // computation when called with an unexpected schema. + if (elementLayout != null && elementLayout.schema == schema) { + return elementLayout.newRow(); + } return new CompactBinaryRow(schema, CompactBinaryRowWriter.fixedOffsets(schema)); } diff --git a/java/fory-format/src/main/java/org/apache/fory/format/row/binary/CompactBinaryRow.java b/java/fory-format/src/main/java/org/apache/fory/format/row/binary/CompactBinaryRow.java index 946418b2b3..34b79749b4 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/row/binary/CompactBinaryRow.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/row/binary/CompactBinaryRow.java @@ -43,6 +43,13 @@ public class CompactBinaryRow extends BinaryRow { private final boolean allFieldsNotNullable; private final int[] fixedOffsets; + /** + * Per-field fixed-width cache, or {@code null} when the row was constructed without a hoisted + * layout. {@code -1} entries mean the field is variable-width. Read by {@link #getBuffer}, + * {@link #getBinary}, and {@link #getStruct} to avoid recursing through the schema. + */ + private final int[] fixedWidths; + private final int bitmapOffset; public CompactBinaryRow(final Schema schema) { @@ -52,10 +59,29 @@ public CompactBinaryRow(final Schema schema) { public CompactBinaryRow(final Schema schema, final int[] fixedOffsets) { super(schema); this.fixedOffsets = fixedOffsets; + this.fixedWidths = null; bitmapOffset = fixedOffsets[fixedOffsets.length - 1]; allFieldsNotNullable = CompactBinaryRowWriter.allNotNullable(schema.fields()); } + /** + * Allocate a CompactBinaryRow from a {@link CompactRowLayout} that has precomputed every + * schema-derived quantity. Used on hot read paths (see {@link CompactBinaryArray#getStruct}). + */ + CompactBinaryRow(CompactRowLayout layout) { + super(layout.schema, layout.bitmapWidthInBytes); + this.fixedOffsets = layout.fixedOffsets; + this.fixedWidths = layout.fixedWidths; + this.bitmapOffset = layout.fixedOffsets[layout.fixedOffsets.length - 1]; + this.allFieldsNotNullable = layout.allFieldsNotNullable; + } + + private int fixedWidthFor(int ordinal) { + return fixedWidths != null + ? fixedWidths[ordinal] + : CompactBinaryRowWriter.fixedWidthFor(schema, ordinal); + } + @Override protected int computeBitmapWidthInBytes() { // cannot use field due to initialization order @@ -81,7 +107,7 @@ public int getOffset(final int ordinal) { @Override public MemoryBuffer getBuffer(final int ordinal) { - final int fixedWidthBinary = CompactBinaryRowWriter.fixedWidthFor(schema, ordinal); + final int fixedWidthBinary = fixedWidthFor(ordinal); if (fixedWidthBinary >= 0) { if (isNullAt(ordinal)) { return null; @@ -94,7 +120,7 @@ public MemoryBuffer getBuffer(final int ordinal) { @Override public byte[] getBinary(final int ordinal) { - final int fixedWidthBinary = CompactBinaryRowWriter.fixedWidthFor(schema, ordinal); + final int fixedWidthBinary = fixedWidthFor(ordinal); if (fixedWidthBinary >= 0) { if (isNullAt(ordinal)) { return null; @@ -112,15 +138,25 @@ protected BinaryRow getStruct(final int ordinal, final Field field, final int ex if (isNullAt(ordinal)) { return null; } - final int fixedWidthBinary = CompactBinaryRowWriter.fixedWidthFor(schema, ordinal); - if (fixedWidthBinary == -1) { - return super.getStruct(ordinal, field, extDataSlot); + CompactRowLayout layout = (CompactRowLayout) extData[extDataSlot]; + if (layout == null) { + // The parent's schema was already sorted at construction, so field.type().fields() is in + // compact order; wrap without re-sorting. + layout = new CompactRowLayout(DataTypes.createSchema(field)); + extData[extDataSlot] = layout; } - if (extData[extDataSlot] == null) { - extData[extDataSlot] = DataTypes.createSchema(field); + final CompactBinaryRow row = layout.newRow(); + final int fixedWidthBinary = fixedWidthFor(ordinal); + if (fixedWidthBinary == -1) { + // Variable-width nested struct: the slot stores (offset, size). + final long offsetAndSize = getInt64(ordinal); + final int relativeOffset = (int) (offsetAndSize >> 32); + final int size = (int) offsetAndSize; + row.pointTo(getBuffer(), getBaseOffset() + relativeOffset, size); + } else { + // Fixed-width nested struct: inline at getOffset(ordinal). + row.pointTo(getBuffer().slice(getOffset(ordinal), fixedWidthBinary), 0, fixedWidthBinary); } - final BinaryRow row = newRow((Schema) extData[extDataSlot]); - row.pointTo(getBuffer().slice(getOffset(ordinal), fixedWidthBinary), 0, fixedWidthBinary); return row; } diff --git a/java/fory-format/src/main/java/org/apache/fory/format/row/binary/CompactRowLayout.java b/java/fory-format/src/main/java/org/apache/fory/format/row/binary/CompactRowLayout.java new file mode 100644 index 0000000000..a86699be06 --- /dev/null +++ b/java/fory-format/src/main/java/org/apache/fory/format/row/binary/CompactRowLayout.java @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.fory.format.row.binary; + +import java.util.List; +import org.apache.fory.annotation.Internal; +import org.apache.fory.format.row.binary.writer.CompactBinaryRowWriter; +import org.apache.fory.format.type.Field; +import org.apache.fory.format.type.Schema; + +/** + * Cached compact-row layout for a single nested-struct slot. Holding this in a parent row's + * ext-data slot lets repeated {@code getStruct} calls construct fresh {@link CompactBinaryRow} + * instances without re-deriving the layout — all of which are pure functions of the schema. + * + *

Caches {@code fixedOffsets}, per-field {@code fixedWidths} (used by {@code getBuffer}, + * {@code getBinary}, and the variable-vs-fixed branch in {@code getStruct}), {@code + * allFieldsNotNullable}, and the bitmap width. + */ +@Internal +final class CompactRowLayout { + final Schema schema; + final int[] fixedOffsets; + final int[] fixedWidths; + final boolean allFieldsNotNullable; + final int bitmapWidthInBytes; + + CompactRowLayout(Schema schema) { + this.schema = schema; + this.fixedOffsets = CompactBinaryRowWriter.fixedOffsets(schema); + final List fields = schema.fields(); + this.fixedWidths = new int[fields.size()]; + for (int i = 0; i < fields.size(); i++) { + fixedWidths[i] = CompactBinaryRowWriter.fixedWidthFor(fields.get(i)); + } + this.allFieldsNotNullable = CompactBinaryRowWriter.allNotNullable(fields); + this.bitmapWidthInBytes = + allFieldsNotNullable ? 0 : CompactBinaryRowWriter.headerBytes(schema); + } + + CompactBinaryRow newRow() { + return new CompactBinaryRow(this); + } +} diff --git a/java/fory-format/src/test/java/org/apache/fory/format/perf/RowFormatAllocationProbe.java b/java/fory-format/src/test/java/org/apache/fory/format/perf/RowFormatAllocationProbe.java new file mode 100644 index 0000000000..7ec7225847 --- /dev/null +++ b/java/fory-format/src/test/java/org/apache/fory/format/perf/RowFormatAllocationProbe.java @@ -0,0 +1,236 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.fory.format.perf; + +import com.sun.management.ThreadMXBean; +import java.lang.management.ManagementFactory; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import lombok.Data; +import org.apache.fory.format.encoder.ArrayEncoder; +import org.apache.fory.format.encoder.BaseCodecBuilder; +import org.apache.fory.format.encoder.Encoders; +import org.apache.fory.format.encoder.MapEncoder; +import org.apache.fory.format.encoder.RowEncoder; +import org.apache.fory.reflect.TypeRef; + +/** + * Standalone allocation probe for nested row-format read paths. Uses + * {@link com.sun.management.ThreadMXBean#getCurrentThreadAllocatedBytes()} to measure bytes + * allocated per decode op, isolating the per-element waste hidden inside nested struct/array/map + * paths. + * + *

Run with: {@code java -cp org.apache.fory.format.perf.RowFormatAllocationProbe} + * + *

Output columns: scenario, format, bytes/op (mean over {@link #ITERATIONS} iterations), + * bytes/op (post-warmup). + */ +public final class RowFormatAllocationProbe { + + private static final int LEAF_COUNT = 32; + private static final int MAP_ENTRIES = 16; + private static final int MATRIX_ROWS = 8; + private static final int WARMUP = 1_000; + private static final int ITERATIONS = 10_000; + + // -------------------- Beans -------------------- + + @Data + public static class Leaf { + private long a; + private long b; + private int c; + private String d; + } + + @Data + public static class Branch { + private Leaf leaf; + private List leaves; + } + + @Data + public static class Root { + private long id; + private Branch branch; + private List leaves; + private Map table; + private List> matrix; + } + + // -------------------- Test data -------------------- + + private static Leaf leaf(int seed) { + Leaf l = new Leaf(); + l.setA(seed); + l.setB(seed * 31L); + l.setC(seed); + l.setD("leaf-" + seed); + return l; + } + + private static List leaves(int n, int seed) { + List out = new ArrayList<>(n); + for (int i = 0; i < n; i++) { + out.add(leaf(seed + i)); + } + return out; + } + + private static Branch branch(int seed) { + Branch b = new Branch(); + b.setLeaf(leaf(seed)); + b.setLeaves(leaves(LEAF_COUNT, seed)); + return b; + } + + private static Root buildRoot() { + Root r = new Root(); + r.setId(7); + r.setBranch(branch(100)); + r.setLeaves(leaves(LEAF_COUNT, 200)); + Map table = new HashMap<>(); + for (int i = 0; i < MAP_ENTRIES; i++) { + table.put("k" + i, leaf(300 + i)); + } + r.setTable(table); + List> matrix = new ArrayList<>(); + for (int i = 0; i < MATRIX_ROWS; i++) { + matrix.add(leaves(LEAF_COUNT, 400 + i * LEAF_COUNT)); + } + r.setMatrix(matrix); + return r; + } + + // -------------------- Probe -------------------- + + private static final ThreadMXBean BEAN = (ThreadMXBean) ManagementFactory.getThreadMXBean(); + + private static long measure(Runnable op) { + // Warm up. + for (int i = 0; i < WARMUP; i++) { + op.run(); + } + // Measure: average bytes per iteration. + long before = BEAN.getCurrentThreadAllocatedBytes(); + for (int i = 0; i < ITERATIONS; i++) { + op.run(); + } + long after = BEAN.getCurrentThreadAllocatedBytes(); + return (after - before) / ITERATIONS; + } + + // -------------------- Scenarios -------------------- + + private static > B configure(B b, boolean compact) { + if (compact) { + b.compactEncoding(); + } + return b; + } + + private static void run(String label, boolean compact) { + RowEncoder rootCodec = configure(Encoders.buildBeanCodec(Root.class), compact).build().get(); + ArrayEncoder> arrayCodec = + configure(Encoders.buildArrayCodec(new TypeRef>() {}), compact).build().get(); + ArrayEncoder>> matrixCodec = + configure(Encoders.buildArrayCodec(new TypeRef>>() {}), compact) + .build() + .get(); + MapEncoder> mapCodec = + configure(Encoders.buildMapCodec(new TypeRef>() {}), compact) + .build() + .get(); + + Root r = buildRoot(); + byte[] rootBytes = rootCodec.encode(r); + byte[] arrayBytes = arrayCodec.encode(r.getLeaves()); + byte[] matrixBytes = matrixCodec.encode(r.getMatrix()); + byte[] mapBytes = mapCodec.encode(r.getTable()); + + // For each scenario, also fully traverse the result so lazy paths actually fire. + long rootAlloc = + measure( + () -> { + Root out = rootCodec.decode(rootBytes); + touchRoot(out); + }); + long arrayAlloc = + measure( + () -> { + List out = arrayCodec.decode(arrayBytes); + touchLeaves(out); + }); + long matrixAlloc = + measure( + () -> { + List> out = matrixCodec.decode(matrixBytes); + for (List row : out) { + touchLeaves(row); + } + }); + long mapAlloc = + measure( + () -> { + Map out = mapCodec.decode(mapBytes); + for (Leaf leaf : out.values()) { + touch(leaf); + } + }); + + System.out.printf( + "%-9s root=%-7d array=%-7d matrix=%-7d map=%-7d (bytes/op)%n", + label, rootAlloc, arrayAlloc, matrixAlloc, mapAlloc); + } + + private static long sink; + + private static void touch(Leaf l) { + sink += l.getA() + l.getB() + l.getC() + l.getD().length(); + } + + private static void touchLeaves(List ls) { + for (Leaf l : ls) { + touch(l); + } + } + + private static void touchRoot(Root r) { + sink += r.getId(); + if (r.getBranch() != null) { + touch(r.getBranch().getLeaf()); + touchLeaves(r.getBranch().getLeaves()); + } + touchLeaves(r.getLeaves()); + for (Leaf l : r.getTable().values()) { + touch(l); + } + for (List row : r.getMatrix()) { + touchLeaves(row); + } + } + + public static void main(String[] args) { + run("standard", false); + run("compact ", true); + } +} From f70fc1f8cdb9577ad9292035b17b9cd2e0a939f1 Mon Sep 17 00:00:00 2001 From: "Claude (on behalf of Steven Schlansker)" Date: Thu, 28 May 2026 22:25:17 +0000 Subject: [PATCH 05/13] feat(format): dispatch nested versioned beans by recursive strict hash MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The strict schema hash already recurses through StructType, so two payloads whose inner-struct shapes differ produce different outer hashes. The implementation gap was in SchemaHistory.build, which only enumerated the outer bean's own version boundaries — projection codecs for "outer V=K with inner V=L" weren't generated, so older inner shapes failed to deserialize even though the hash distinguished them. Implementation: - SchemaHistory.build now recurses into nested-bean fields whose type carries schema-evolution annotations, builds each inner's history, and cross-products over inner versions when enumerating outer versions. Each VersionedSchema now carries a map of (nested bean class -> chosen inner version) so the codec builder can wire the right inner projection codec. - RowCodecBuilder.evolvingBuildForWriter emits one projection codec class per cross-product combination, using a per-nested-bean-type suffix map passed down through Encoding/RowEncoderBuilder. BaseBinaryEncoderBuilder exposes a `nestedBeanSuffix(TypeRef)` hook that the projection builder overrides to look up each nested bean's right suffix. - Inner projection classes are generated recursively from nestedSuffixesFor(), so a deeply-nested versioned bean produces the required class tree at outer-build time. Class-count complexity is O(product of versions across nesting), but each projection class is small (decode-only) and only those reachable from the outer's enumeration are generated. Regression test nestedInnerEvolution_readerInnerNewerThanWriter and the two-axis crossOuterAndInnerEvolution both pass. 138 tests in fory-format green. --- docs/guide/java/row-format.md | 5 - .../encoder/BaseBinaryEncoderBuilder.java | 11 +- .../format/encoder/CompactCodecFormat.java | 6 +- .../encoder/CompactRowEncoderBuilder.java | 5 +- .../format/encoder/DefaultCodecFormat.java | 6 +- .../apache/fory/format/encoder/Encoders.java | 18 +- .../apache/fory/format/encoder/Encoding.java | 12 +- .../fory/format/encoder/RowCodecBuilder.java | 83 ++++++++- .../format/encoder/RowEncoderBuilder.java | 21 ++- .../fory/format/type/SchemaHistory.java | 170 ++++++++++++++---- .../encoder/BinaryRowEncoderPointToTest.java | 6 +- .../encoder/SchemaEvolutionStressTest.java | 54 +++++- 12 files changed, 339 insertions(+), 58 deletions(-) diff --git a/docs/guide/java/row-format.md b/docs/guide/java/row-format.md index 48ba35872c..b3d06ab166 100644 --- a/docs/guide/java/row-format.md +++ b/docs/guide/java/row-format.md @@ -178,11 +178,6 @@ Cross-language consumers (Python, C++) cannot read evolution-enabled payloads. Map keys do not carry a per-payload hash; a versioned bean used as a map key is read with the current schema only, not dispatched to a projection codec. -A versioned bean nested as a struct field inside another versioned bean is read with its -current schema regardless of what the wire bytes were written from — the row format does not -carry a per-nested-struct hash. Evolve either the outer or the nested bean, but expect the -nested-bean schema to remain stable while the outer evolves (or vice versa). - ## Cross-Language Compatibility Row format works seamlessly across languages. The same binary data can be accessed from: diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/BaseBinaryEncoderBuilder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/BaseBinaryEncoderBuilder.java index 111d55a192..694a9c09b6 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/BaseBinaryEncoderBuilder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/BaseBinaryEncoderBuilder.java @@ -519,7 +519,7 @@ protected void registerBeanCodec(Expression writer, TypeRef typeRef, Expressi String encoderName = ctx.newName(StringUtils.uncapitalize(codecClassName(rawType))); String encoderClass = codecQualifiedClassName(rawType) - + (rowCodecSuffixForBeans == null ? "" : rowCodecSuffixForBeans); + + nestedBeanSuffix(typeRef); TypeRef codecTypeRef = TypeRef.of(GeneratedRowEncoder.class); NewInstance newEncoder = new NewInstance( @@ -532,6 +532,15 @@ protected void registerBeanCodec(Expression writer, TypeRef typeRef, Expressi beanEncoderMap.put(typeRef, new Reference(encoderName, codecTypeRef)); } + /** + * Suffix to append to a nested bean's codec class name when emitting a reference. Defaults to + * the single uniform suffix (or empty); subclasses with per-type version routing can override + * to return a per-typeRef suffix from a map. + */ + protected String nestedBeanSuffix(TypeRef typeRef) { + return rowCodecSuffixForBeans == null ? "" : rowCodecSuffixForBeans; + } + protected Expression createSchemaFromStructField(Expression structField) { return new StaticInvoke( DataTypes.class, "schemaFromStructField", "schema", SCHEMA_TYPE, false, structField); diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/CompactCodecFormat.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/CompactCodecFormat.java index 435db5acb4..0e1d4a1334 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/CompactCodecFormat.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/CompactCodecFormat.java @@ -70,8 +70,10 @@ public RowEncoderBuilder newProjectionRowEncoder( final TypeRef beanType, final Schema historicalSchema, final Set liveNames, - final String classSuffix) { - return new CompactRowEncoderBuilder(beanType, historicalSchema, liveNames, classSuffix); + final String classSuffix, + final Map, String> nestedSuffixes) { + return new CompactRowEncoderBuilder( + beanType, historicalSchema, liveNames, classSuffix, nestedSuffixes); } @Override diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/CompactRowEncoderBuilder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/CompactRowEncoderBuilder.java index 828bdc9e43..e19fd3b2a1 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/CompactRowEncoderBuilder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/CompactRowEncoderBuilder.java @@ -45,8 +45,9 @@ public CompactRowEncoderBuilder(final TypeRef beanType) { final TypeRef beanType, final Schema historicalSchema, final java.util.Set liveNames, - final String classSuffix) { - super(beanType, historicalSchema, liveNames, classSuffix); + final String classSuffix, + final java.util.Map, String> nestedSuffixes) { + super(beanType, historicalSchema, liveNames, classSuffix, nestedSuffixes); } @Override diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/DefaultCodecFormat.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/DefaultCodecFormat.java index f4ab67fe6f..c432738930 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/DefaultCodecFormat.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/DefaultCodecFormat.java @@ -66,8 +66,10 @@ public RowEncoderBuilder newProjectionRowEncoder( final TypeRef beanType, final Schema historicalSchema, final Set liveNames, - final String classSuffix) { - return new RowEncoderBuilder(beanType, historicalSchema, liveNames, classSuffix); + final String classSuffix, + final Map, String> nestedSuffixes) { + return new RowEncoderBuilder( + beanType, historicalSchema, liveNames, classSuffix, nestedSuffixes); } @Override diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/Encoders.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/Encoders.java index ba0d781a22..d6cd609cd8 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/Encoders.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/Encoders.java @@ -335,9 +335,25 @@ static Class loadOrGenProjectionRowCodecClass( org.apache.fory.format.type.Schema historicalSchema, Set liveNames, String classSuffix) { + return loadOrGenProjectionRowCodecClass( + beanClass, + codecFactory, + historicalSchema, + liveNames, + classSuffix, + java.util.Collections.emptyMap()); + } + + static Class loadOrGenProjectionRowCodecClass( + Class beanClass, + Encoding codecFactory, + org.apache.fory.format.type.Schema historicalSchema, + Set liveNames, + String classSuffix, + Map, String> nestedSuffixes) { final RowEncoderBuilder codecBuilder = codecFactory.newProjectionRowEncoder( - TypeRef.of(beanClass), historicalSchema, liveNames, classSuffix); + TypeRef.of(beanClass), historicalSchema, liveNames, classSuffix, nestedSuffixes); CompileUnit compileUnit = new CompileUnit( CodeGenerator.getPackage(beanClass), diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/Encoding.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/Encoding.java index 2f994b2000..a7c07e36c3 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/Encoding.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/Encoding.java @@ -45,11 +45,17 @@ interface Encoding { /** * Construct a projection codec builder for an older version of {@code beanType}, reading the - * supplied historical schema and producing instances of the current bean class. Used only by - * the schema-evolution code path. + * supplied historical schema and producing instances of the current bean class. The + * {@code nestedSuffixes} map directs codegen to embed a specific projection codec class for + * each nested-bean type (used when a nested versioned bean was on the wire at an older + * version). An empty map means all nested beans use their current-version codecs. */ RowEncoderBuilder newProjectionRowEncoder( - TypeRef beanType, Schema historicalSchema, Set liveNames, String classSuffix); + TypeRef beanType, + Schema historicalSchema, + Set liveNames, + String classSuffix, + Map, String> nestedSuffixes); ArrayEncoderBuilder newArrayEncoder( TypeRef> collectionType, TypeRef elementType); diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/RowCodecBuilder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/RowCodecBuilder.java index 2d90029666..839ee142de 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/RowCodecBuilder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/RowCodecBuilder.java @@ -20,7 +20,9 @@ package org.apache.fory.format.encoder; import java.lang.invoke.MethodHandle; +import java.util.ArrayList; import java.util.HashMap; +import java.util.List; import java.util.Map; import java.util.function.Function; import java.util.function.Supplier; @@ -100,16 +102,25 @@ private Function> evolvingBuildForWriter() { final Function currentFactory = rowEncoderFactory(currentSchema); - // Projection codecs for each older version; classes are loaded eagerly. + // Projection codecs for each non-current combination of (outer-version, inner-versions). + // The suffix encodes the combination so different cross-product entries get distinct + // generated classes; the nested-bean version map directs the projection codec to embed + // the right inner projection class for each nested-bean type. final Map projectionFactories = new HashMap<>(); for (SchemaHistory.VersionedSchema vs : history.versions()) { if (vs == currentVersion) { continue; } - String suffix = "_V" + vs.version(); + String suffix = projectionSuffix(vs); + Map, String> nestedSuffixes = nestedSuffixesFor(vs); Class projectionClass = Encoders.loadOrGenProjectionRowCodecClass( - beanClass, codecFormat, vs.schema(), vs.liveFieldNames(), suffix); + beanClass, + codecFormat, + vs.schema(), + vs.liveFieldNames(), + suffix, + nestedSuffixes); MethodHandle ctor = Encoders.constructorHandleFor(projectionClass, GeneratedRowEncoder.class); projectionFactories.put(vs.strictHash(), new ProjectionCodecFactory(vs.schema(), ctor)); @@ -135,6 +146,72 @@ public RowEncoder apply(final BaseBinaryRowWriter writer) { }; } + /** + * Build a unique suffix for a projection codec class, encoding the outer version plus each + * nested-bean version. Two entries in the cross-product differ in at least one of these, so + * the resulting class names don't collide. + */ + private static String projectionSuffix(SchemaHistory.VersionedSchema vs) { + StringBuilder sb = new StringBuilder("_V").append(vs.version()); + if (!vs.nestedBeanVersions().isEmpty()) { + // Sort by class name for determinism across JVM invocations. + List, Integer>> entries = + new ArrayList<>(vs.nestedBeanVersions().entrySet()); + entries.sort((a, b) -> a.getKey().getName().compareTo(b.getKey().getName())); + for (Map.Entry, Integer> e : entries) { + sb.append("_").append(e.getKey().getSimpleName()).append(e.getValue()); + } + } + return sb.toString(); + } + + /** + * Per-nested-bean-type suffix map for codegen. The projection codec uses this to look up + * which inner codec class to embed for each nested bean type (the inner's own projection + * suffix at this combination's version). + */ + private Map, String> nestedSuffixesFor(SchemaHistory.VersionedSchema vs) { + Map, String> out = new HashMap<>(); + for (Map.Entry, Integer> e : vs.nestedBeanVersions().entrySet()) { + // The inner codec for class C at version v has its own suffix; we mirror the inner + // SchemaHistory.build's suffix scheme. Compute by recursively building the inner + // history and finding its VersionedSchema whose version matches; use its suffix. + Class innerClass = e.getKey(); + int innerVersion = e.getValue(); + UnaryOperator innerTransform = + codecFormat == CompactCodecFormat.INSTANCE + ? CompactBinaryRowWriter::sortSchema + : UnaryOperator.identity(); + SchemaHistory innerHistory = SchemaHistory.build(innerClass, innerTransform); + SchemaHistory.VersionedSchema innerVs = null; + for (SchemaHistory.VersionedSchema cand : innerHistory.versions()) { + if (cand.version() == innerVersion) { + innerVs = cand; + break; + } + } + if (innerVs == null) { + throw new IllegalStateException( + "No inner VersionedSchema for " + innerClass.getName() + " at v" + innerVersion); + } + if (innerVs == innerHistory.current()) { + out.put(innerClass, ""); + } else { + out.put(innerClass, projectionSuffix(innerVs)); + // Also generate the inner's projection class so the outer projection's `new InnerCodec_VN` + // resolves at class load. + Encoders.loadOrGenProjectionRowCodecClass( + innerClass, + codecFormat, + innerVs.schema(), + innerVs.liveFieldNames(), + projectionSuffix(innerVs), + nestedSuffixesFor(innerVs)); + } + } + return out; + } + private static final class ProjectionCodecFactory { private final Schema historicalSchema; private final MethodHandle ctor; diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/RowEncoderBuilder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/RowEncoderBuilder.java index 7a2b73cbc4..c52b7a0978 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/RowEncoderBuilder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/RowEncoderBuilder.java @@ -84,13 +84,14 @@ class RowEncoderBuilder extends BaseBinaryEncoderBuilder { */ private final Set projectionLiveNames; private final String projectionClassSuffix; + private final java.util.Map, String> nestedSuffixes; public RowEncoderBuilder(Class beanClass) { this(TypeRef.of(beanClass)); } public RowEncoderBuilder(TypeRef beanType) { - this(beanType, null, null, null); + this(beanType, null, null, null, java.util.Collections.emptyMap()); } /** @@ -98,16 +99,21 @@ public RowEncoderBuilder(TypeRef beanType) { * supplied {@code historicalSchema} is used as the layout to decode; only fields whose name is * in {@code liveNames} are written into the resulting bean. {@code classSuffix} distinguishes * this codec from the current-version codec and from other historical projections. + * {@code nestedSuffixes} routes each nested-bean type to a specific projection codec class + * (used when an inner versioned bean was on the wire at an older version). */ RowEncoderBuilder( TypeRef beanType, Schema historicalSchema, Set liveNames, - String classSuffix) { + String classSuffix, + java.util.Map, String> nestedSuffixes) { super(new CodegenContext(), beanType); Preconditions.checkArgument(beanClass.isInterface() || TypeUtils.isBean(beanType, typeCtx)); this.projectionLiveNames = liveNames; this.projectionClassSuffix = classSuffix; + this.nestedSuffixes = + nestedSuffixes == null ? java.util.Collections.emptyMap() : nestedSuffixes; className = projectionClassSuffix == null ? codecClassName(beanClass) @@ -150,6 +156,17 @@ protected Schema inferSchema(TypeRef beanType) { return TypeInference.inferSchema(getRawType(beanType)); } + @Override + protected String nestedBeanSuffix(TypeRef typeRef) { + if (nestedSuffixes != null) { + String s = nestedSuffixes.get(getRawType(typeRef)); + if (s != null) { + return s; + } + } + return super.nestedBeanSuffix(typeRef); + } + @Override protected String codecSuffix() { return "RowCodec"; diff --git a/java/fory-format/src/main/java/org/apache/fory/format/type/SchemaHistory.java b/java/fory-format/src/main/java/org/apache/fory/format/type/SchemaHistory.java index 331d470754..953eddd8b7 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/type/SchemaHistory.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/type/SchemaHistory.java @@ -35,6 +35,7 @@ import org.apache.fory.format.annotation.ForyVersion; import org.apache.fory.reflect.TypeRef; import org.apache.fory.type.Descriptor; +import org.apache.fory.type.TypeUtils; import org.apache.fory.util.StringUtils; /** @@ -55,12 +56,19 @@ public static final class VersionedSchema { private final Schema schema; private final long strictHash; private final Set liveFieldNames; + private final Map, Integer> nestedBeanVersions; - VersionedSchema(int version, Schema schema, long strictHash, Set liveFieldNames) { + VersionedSchema( + int version, + Schema schema, + long strictHash, + Set liveFieldNames, + Map, Integer> nestedBeanVersions) { this.version = version; this.schema = schema; this.strictHash = strictHash; this.liveFieldNames = liveFieldNames; + this.nestedBeanVersions = nestedBeanVersions; } public int version() { @@ -82,6 +90,16 @@ public long strictHash() { public Set liveFieldNames() { return liveFieldNames; } + + /** + * For each nested versioned bean type referenced by this schema, the version of that + * inner bean represented in this combination. Empty when the schema has no nested + * versioned beans. Used by the codec builder to choose which inner projection codec class + * to embed for each nested-bean slot. + */ + public Map, Integer> nestedBeanVersions() { + return nestedBeanVersions; + } } private final List versions; @@ -116,6 +134,17 @@ public static SchemaHistory build(Class beanClass, UnaryOperator sche all.addAll(collectRemovedFields(removedFieldsClass)); } + // Recursively expand any nested versioned bean field's own history. For each entry whose + // type is a versioned bean (has @ForyVersion-annotated descriptors or @ForySchema), we + // attach its SchemaHistory so the outer's enumeration can cross-product over inner + // versions. The inner schema substitutes into the outer at materialization time. + for (FieldEntry fe : all) { + Class raw = TypeUtils.getRawType(fe.typeRef); + if (raw != null && isBeanWithVersioning(raw)) { + fe.innerHistory = build(raw, schemaTransform); + } + } + // Materialize a schema at every version V where the field set changes — both "since" and // "until" boundaries qualify, because either adds or removes a field from the active set. TreeSet schemaVersions = new TreeSet<>(); @@ -141,49 +170,128 @@ public static SchemaHistory build(Class beanClass, UnaryOperator sche int latestVersion = schemaVersions.last(); Map bySignature = new LinkedHashMap<>(); Map hashToSignature = new HashMap<>(); + String currentSignature = null; for (int v : schemaVersions) { - List fields = new ArrayList<>(); - Set liveNames = new HashSet<>(); + List activeEntries = new ArrayList<>(); for (FieldEntry fe : all) { if (fe.since <= v && v < fe.until) { - fields.add(TypeInference.inferNamedField(fe.name, fe.typeRef)); + activeEntries.add(fe); + } + } + // Cross-product over each nested versioned bean's history. If no entries have nested + // histories, this yields a single combination. + List> innerChoices = new ArrayList<>(activeEntries.size()); + List innerEntries = new ArrayList<>(activeEntries.size()); + for (FieldEntry fe : activeEntries) { + if (fe.innerHistory != null) { + innerEntries.add(fe); + innerChoices.add(fe.innerHistory.versions()); + } + } + for (Map combination : cartesian(innerEntries, innerChoices)) { + List fields = new ArrayList<>(activeEntries.size()); + Set liveNames = new HashSet<>(); + Map, Integer> nestedBeanVersionsMap = new HashMap<>(); + for (FieldEntry fe : activeEntries) { + Field field; + if (combination.containsKey(fe)) { + // Substitute the chosen inner version's struct fields. + VersionedSchema innerVs = combination.get(fe); + field = + DataTypes.field( + fe.name, + new DataTypes.StructType(innerVs.schema().fields()), + fe.typeRef.getRawType() == null + || !fe.typeRef.getRawType().isPrimitive()); + nestedBeanVersionsMap.put(TypeUtils.getRawType(fe.typeRef), innerVs.version()); + } else { + field = TypeInference.inferNamedField(fe.name, fe.typeRef); + } + fields.add(field); if (fe.live) { liveNames.add(fe.name); } } + Schema schema = schemaTransform.apply(new Schema(fields)); + long hash = computeStrictSchemaHash(schema); + String signature = schemaSignature(schema); + String previousSig = hashToSignature.putIfAbsent(hash, signature); + if (previousSig != null && !previousSig.equals(signature)) { + throw new IllegalStateException( + "Strict hash collision for bean " + + beanClass.getName() + + " at version " + + v + + ": two distinct historical schemas hashed to the same value. Please file an " + + "issue with the bean definition."); + } + // Determine whether this combination's nested-versions are all "current" for their + // inner. If so, this combination represents the writer-side configuration at outer + // version v. + boolean innerAllCurrent = + combination.entrySet().stream() + .allMatch(e -> e.getValue() == e.getKey().innerHistory.current()); + VersionedSchema vs = + new VersionedSchema( + v, + schema, + hash, + Collections.unmodifiableSet(liveNames), + Collections.unmodifiableMap(nestedBeanVersionsMap)); + bySignature.putIfAbsent(signature, vs); + if (v == latestVersion && innerAllCurrent) { + currentSignature = signature; + } } - Schema schema = schemaTransform.apply(new Schema(fields)); - long hash = computeStrictSchemaHash(schema); - String signature = schemaSignature(schema); - String previousSig = hashToSignature.putIfAbsent(hash, signature); - if (previousSig != null && !previousSig.equals(signature)) { - throw new IllegalStateException( - "Strict hash collision for bean " - + beanClass.getName() - + " at version " - + v - + ": two distinct historical schemas hashed to the same value. Please file an " - + "issue with the bean definition."); - } - // Record the highest version at which this signature first appears. The latest boundary - // is the writer's "current" version; preferring it over earlier first-appearances keeps - // current().version() aligned with what writers emit. - bySignature.put( - signature, - new VersionedSchema(v, schema, hash, Collections.unmodifiableSet(liveNames))); } - // current is the schema in effect at latestVersion. - VersionedSchema current = null; - for (VersionedSchema vs : bySignature.values()) { - if (vs.version() == latestVersion) { - current = vs; - break; - } + VersionedSchema current = bySignature.get(currentSignature); + if (current == null) { + // Fallback: pick whatever the last-inserted entry is. This is reachable only when the + // latest-version outer schema has no nested versioned beans. + current = bySignature.values().stream().reduce((a, b) -> b).orElseThrow(); } return new SchemaHistory( Collections.unmodifiableList(new ArrayList<>(bySignature.values())), current); } + /** Cartesian product over (FieldEntry, list-of-inner-VersionedSchema). */ + private static List> cartesian( + List entries, List> choices) { + List> out = new ArrayList<>(); + out.add(new HashMap<>()); + for (int i = 0; i < entries.size(); i++) { + FieldEntry fe = entries.get(i); + List options = choices.get(i); + List> next = new ArrayList<>(out.size() * options.size()); + for (Map prefix : out) { + for (VersionedSchema opt : options) { + Map extended = new HashMap<>(prefix); + extended.put(fe, opt); + next.add(extended); + } + } + out = next; + } + return out; + } + + /** True if the class is a row-codec bean and carries any schema-evolution annotations. */ + private static boolean isBeanWithVersioning(Class cls) { + if (cls.isAnnotationPresent(ForySchema.class)) { + return true; + } + try { + for (Descriptor d : Descriptor.getDescriptors(cls)) { + if (lookupForyVersion(d) != null) { + return true; + } + } + } catch (Exception ignored) { + // Not a bean we can introspect (e.g. enum, primitive wrapper) — treat as not versioned. + } + return false; + } + /** * Canonical textual signature of a schema, used to distinguish a real strict-hash collision * (two genuinely different schemas with the same hash) from the benign case where two version @@ -379,6 +487,8 @@ private static final class FieldEntry { final int since; final int until; final boolean live; + /** SchemaHistory of this entry's bean type, when the type is itself versioned. */ + SchemaHistory innerHistory; FieldEntry( String name, String javaName, TypeRef typeRef, int since, int until, boolean live) { diff --git a/java/fory-format/src/test/java/org/apache/fory/format/encoder/BinaryRowEncoderPointToTest.java b/java/fory-format/src/test/java/org/apache/fory/format/encoder/BinaryRowEncoderPointToTest.java index 7d13f9be00..ce55619054 100644 --- a/java/fory-format/src/test/java/org/apache/fory/format/encoder/BinaryRowEncoderPointToTest.java +++ b/java/fory-format/src/test/java/org/apache/fory/format/encoder/BinaryRowEncoderPointToTest.java @@ -154,8 +154,10 @@ public RowEncoderBuilder newProjectionRowEncoder( org.apache.fory.reflect.TypeRef beanType, Schema historicalSchema, java.util.Set liveNames, - String classSuffix) { - return delegate.newProjectionRowEncoder(beanType, historicalSchema, liveNames, classSuffix); + String classSuffix, + java.util.Map, String> nestedSuffixes) { + return delegate.newProjectionRowEncoder( + beanType, historicalSchema, liveNames, classSuffix, nestedSuffixes); } @Override diff --git a/java/fory-format/src/test/java/org/apache/fory/format/encoder/SchemaEvolutionStressTest.java b/java/fory-format/src/test/java/org/apache/fory/format/encoder/SchemaEvolutionStressTest.java index e56d7b46e8..2914cc343e 100644 --- a/java/fory-format/src/test/java/org/apache/fory/format/encoder/SchemaEvolutionStressTest.java +++ b/java/fory-format/src/test/java/org/apache/fory/format/encoder/SchemaEvolutionStressTest.java @@ -706,11 +706,7 @@ public static class NestedOuterV2 { private NestedInnerV2 inner; } - // TODO: nested versioned beans inside another versioned bean are not yet dispatched. The - // strict hash naturally encodes inner-struct shape, but SchemaHistory.build does not - // currently cross-product over nested-bean versions, so no projection codec is generated for - // the older inner shape. Re-enable when implemented. - @Test(enabled = false) + @Test public void nestedInnerEvolution_readerInnerNewerThanWriter() { // Writer uses the "older shape" inner. Both writer and reader are evolution-on so they // agree on strict-hash framing. @@ -732,5 +728,53 @@ public void nestedInnerEvolution_readerInnerNewerThanWriter() { Assert.assertEquals(out.getInner().getName(), "hello"); Assert.assertNull(out.getInner().getAddedField()); } + + // --------------------------------------------------------------------------- + // Outer + inner versioned independently. The cross-product enumeration must + // generate a projection codec for each (outer-version, inner-version) pair + // that isn't the current combination. + // --------------------------------------------------------------------------- + + /** Outer with its own added field at v2; inner stays at v1. */ + @Data + public static class CrossOuterV2_InnerV1 { + private long id; + private NestedInnerWriter inner; + + @ForyVersion(since = 2) + private String label; + } + + /** Outer v2 reader with inner evolved to v2. Both dimensions evolve independently. */ + @Data + public static class CrossOuterV2_InnerV2 { + private long id; + private NestedInnerV2 inner; + + @ForyVersion(since = 2) + private String label; + } + + @Test + public void crossOuterAndInnerEvolution() { + // Writer writes outer V1 + inner V1 (no label, no addedField). + RowEncoder writer = + Encoders.buildBeanCodec(NestedOuterWriter.class).withSchemaEvolution().build().get(); + RowEncoder reader = + Encoders.buildBeanCodec(CrossOuterV2_InnerV2.class).withSchemaEvolution().build().get(); + + NestedOuterWriter in = new NestedOuterWriter(); + in.setId(100); + NestedInnerWriter inn = new NestedInnerWriter(); + inn.setName("legacy-inner"); + in.setInner(inn); + + byte[] bytes = writer.encode(in); + CrossOuterV2_InnerV2 out = reader.decode(bytes); + Assert.assertEquals(out.getId(), 100); + Assert.assertEquals(out.getInner().getName(), "legacy-inner"); + Assert.assertNull(out.getInner().getAddedField()); + Assert.assertNull(out.getLabel()); + } } From 5f877afad16ecbee8d8b012d1c189ea4ee4c7e05 Mon Sep 17 00:00:00 2001 From: "Claude (on behalf of Steven Schlansker)" Date: Thu, 28 May 2026 23:19:34 +0000 Subject: [PATCH 06/13] fix(format): route inner-bean version through array/map projection codecs Array and map evolution paths were generating per-outer-version projection classes named with only the outer version suffix and instantiated without an inner-version routing map. When the element bean contained a versioned nested bean, multiple cross-product entries collided on the codegen cache: the projection always read inner beans at whichever version was compiled first. The row codec already did this correctly; lift its suffix and nested- suffix logic into a shared ProjectionRouting helper and reuse it from ArrayCodecBuilder and MapCodecBuilder. Add array/map regression tests that fail before the fix and pass after. --- .../format/encoder/ArrayCodecBuilder.java | 9 +- .../fory/format/encoder/MapCodecBuilder.java | 8 +- .../format/encoder/ProjectionRouting.java | 104 ++++++++++++++++++ .../fory/format/encoder/RowCodecBuilder.java | 72 +----------- .../encoder/SchemaEvolutionStressTest.java | 79 +++++++++++++ 5 files changed, 197 insertions(+), 75 deletions(-) create mode 100644 java/fory-format/src/main/java/org/apache/fory/format/encoder/ProjectionRouting.java diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/ArrayCodecBuilder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/ArrayCodecBuilder.java index fb464082f7..c6ceb4e764 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/ArrayCodecBuilder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/ArrayCodecBuilder.java @@ -105,15 +105,18 @@ private Function> buildVersionedWithWriter() // Make sure the current-version row codec class is generated. Encoders.loadOrGenRowCodecClass(elementClass, codecFormat); - // Generate per-version row codec classes and per-version array codec classes. + // Generate per-combination row codec classes and per-combination array codec classes. The + // suffix encodes the outer version plus each chosen inner-bean version so that distinct + // cross-product entries do not collide on a single generated class. Map projectionFactories = new HashMap<>(); for (SchemaHistory.VersionedSchema vs : history.versions()) { if (vs == current) { continue; } - String suffix = "_V" + vs.version(); + String suffix = ProjectionRouting.projectionSuffix(vs); + Map, String> nestedSuffixes = ProjectionRouting.nestedSuffixesFor(vs, codecFormat); Encoders.loadOrGenProjectionRowCodecClass( - elementClass, codecFormat, vs.schema(), vs.liveFieldNames(), suffix); + elementClass, codecFormat, vs.schema(), vs.liveFieldNames(), suffix, nestedSuffixes); Class arrayClass = Encoders.loadOrGenProjectionArrayCodecClass( collectionType, TypeRef.of(elementClass), codecFormat, suffix); diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/MapCodecBuilder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/MapCodecBuilder.java index f27baf2d13..c5ace01c3b 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/MapCodecBuilder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/MapCodecBuilder.java @@ -97,14 +97,18 @@ private Supplier> buildVersioned() { SchemaHistory.VersionedSchema current = history.current(); Encoders.loadOrGenRowCodecClass(valClass, codecFormat); + // Generate per-combination row codec classes and per-combination map codec classes. The + // suffix encodes the outer version plus each chosen inner-bean version so that distinct + // cross-product entries do not collide on a single generated class. Map projectionFactories = new HashMap<>(); for (SchemaHistory.VersionedSchema vs : history.versions()) { if (vs == current) { continue; } - String suffix = "_V" + vs.version(); + String suffix = ProjectionRouting.projectionSuffix(vs); + Map, String> nestedSuffixes = ProjectionRouting.nestedSuffixesFor(vs, codecFormat); Encoders.loadOrGenProjectionRowCodecClass( - valClass, codecFormat, vs.schema(), vs.liveFieldNames(), suffix); + valClass, codecFormat, vs.schema(), vs.liveFieldNames(), suffix, nestedSuffixes); Class mapClass = Encoders.loadOrGenProjectionMapCodecClass( mapType, TypeRef.of(valClass), codecFormat, suffix); diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/ProjectionRouting.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/ProjectionRouting.java new file mode 100644 index 0000000000..c7ae209984 --- /dev/null +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/ProjectionRouting.java @@ -0,0 +1,104 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.fory.format.encoder; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.function.UnaryOperator; +import org.apache.fory.format.row.binary.writer.CompactBinaryRowWriter; +import org.apache.fory.format.type.Schema; +import org.apache.fory.format.type.SchemaHistory; + +/** + * Suffix routing shared by row/array/map projection codec generation. Each cross-product entry + * gets a unique class-name suffix encoding the outer version and each chosen inner version, and + * the per-nested-bean suffix map directs codegen to embed the right inner projection class for + * each nested-bean type at this combination's versions. + */ +final class ProjectionRouting { + private ProjectionRouting() {} + + /** + * Build a unique suffix for a projection codec class, encoding the outer version plus each + * nested-bean version. Two entries in the cross-product differ in at least one of these, so + * the resulting class names don't collide. + */ + static String projectionSuffix(SchemaHistory.VersionedSchema vs) { + StringBuilder sb = new StringBuilder("_V").append(vs.version()); + if (!vs.nestedBeanVersions().isEmpty()) { + // Sort by class name for determinism across JVM invocations. + List, Integer>> entries = + new ArrayList<>(vs.nestedBeanVersions().entrySet()); + entries.sort((a, b) -> a.getKey().getName().compareTo(b.getKey().getName())); + for (Map.Entry, Integer> e : entries) { + sb.append("_").append(e.getKey().getSimpleName()).append(e.getValue()); + } + } + return sb.toString(); + } + + /** + * Per-nested-bean-type suffix map for codegen, recursively materializing every inner + * projection class implied by {@code vs}. Empty string means the inner bean uses its + * current-version codec class. + */ + static Map, String> nestedSuffixesFor( + SchemaHistory.VersionedSchema vs, Encoding codecFormat) { + Map, String> out = new HashMap<>(); + UnaryOperator innerTransform = + codecFormat == CompactCodecFormat.INSTANCE + ? CompactBinaryRowWriter::sortSchema + : UnaryOperator.identity(); + for (Map.Entry, Integer> e : vs.nestedBeanVersions().entrySet()) { + Class innerClass = e.getKey(); + int innerVersion = e.getValue(); + SchemaHistory innerHistory = SchemaHistory.build(innerClass, innerTransform); + SchemaHistory.VersionedSchema innerVs = null; + for (SchemaHistory.VersionedSchema cand : innerHistory.versions()) { + if (cand.version() == innerVersion) { + innerVs = cand; + break; + } + } + if (innerVs == null) { + throw new IllegalStateException( + "No inner VersionedSchema for " + innerClass.getName() + " at v" + innerVersion); + } + if (innerVs == innerHistory.current()) { + out.put(innerClass, ""); + } else { + String innerSuffix = projectionSuffix(innerVs); + out.put(innerClass, innerSuffix); + // Eagerly generate the inner's projection class so the outer's `new InnerCodec_VN` + // resolves at class load. + Encoders.loadOrGenProjectionRowCodecClass( + innerClass, + codecFormat, + innerVs.schema(), + innerVs.liveFieldNames(), + innerSuffix, + nestedSuffixesFor(innerVs, codecFormat)); + } + } + return out; + } +} diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/RowCodecBuilder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/RowCodecBuilder.java index 839ee142de..cfc8e229af 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/RowCodecBuilder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/RowCodecBuilder.java @@ -20,9 +20,7 @@ package org.apache.fory.format.encoder; import java.lang.invoke.MethodHandle; -import java.util.ArrayList; import java.util.HashMap; -import java.util.List; import java.util.Map; import java.util.function.Function; import java.util.function.Supplier; @@ -111,8 +109,8 @@ private Function> evolvingBuildForWriter() { if (vs == currentVersion) { continue; } - String suffix = projectionSuffix(vs); - Map, String> nestedSuffixes = nestedSuffixesFor(vs); + String suffix = ProjectionRouting.projectionSuffix(vs); + Map, String> nestedSuffixes = ProjectionRouting.nestedSuffixesFor(vs, codecFormat); Class projectionClass = Encoders.loadOrGenProjectionRowCodecClass( beanClass, @@ -146,72 +144,6 @@ public RowEncoder apply(final BaseBinaryRowWriter writer) { }; } - /** - * Build a unique suffix for a projection codec class, encoding the outer version plus each - * nested-bean version. Two entries in the cross-product differ in at least one of these, so - * the resulting class names don't collide. - */ - private static String projectionSuffix(SchemaHistory.VersionedSchema vs) { - StringBuilder sb = new StringBuilder("_V").append(vs.version()); - if (!vs.nestedBeanVersions().isEmpty()) { - // Sort by class name for determinism across JVM invocations. - List, Integer>> entries = - new ArrayList<>(vs.nestedBeanVersions().entrySet()); - entries.sort((a, b) -> a.getKey().getName().compareTo(b.getKey().getName())); - for (Map.Entry, Integer> e : entries) { - sb.append("_").append(e.getKey().getSimpleName()).append(e.getValue()); - } - } - return sb.toString(); - } - - /** - * Per-nested-bean-type suffix map for codegen. The projection codec uses this to look up - * which inner codec class to embed for each nested bean type (the inner's own projection - * suffix at this combination's version). - */ - private Map, String> nestedSuffixesFor(SchemaHistory.VersionedSchema vs) { - Map, String> out = new HashMap<>(); - for (Map.Entry, Integer> e : vs.nestedBeanVersions().entrySet()) { - // The inner codec for class C at version v has its own suffix; we mirror the inner - // SchemaHistory.build's suffix scheme. Compute by recursively building the inner - // history and finding its VersionedSchema whose version matches; use its suffix. - Class innerClass = e.getKey(); - int innerVersion = e.getValue(); - UnaryOperator innerTransform = - codecFormat == CompactCodecFormat.INSTANCE - ? CompactBinaryRowWriter::sortSchema - : UnaryOperator.identity(); - SchemaHistory innerHistory = SchemaHistory.build(innerClass, innerTransform); - SchemaHistory.VersionedSchema innerVs = null; - for (SchemaHistory.VersionedSchema cand : innerHistory.versions()) { - if (cand.version() == innerVersion) { - innerVs = cand; - break; - } - } - if (innerVs == null) { - throw new IllegalStateException( - "No inner VersionedSchema for " + innerClass.getName() + " at v" + innerVersion); - } - if (innerVs == innerHistory.current()) { - out.put(innerClass, ""); - } else { - out.put(innerClass, projectionSuffix(innerVs)); - // Also generate the inner's projection class so the outer projection's `new InnerCodec_VN` - // resolves at class load. - Encoders.loadOrGenProjectionRowCodecClass( - innerClass, - codecFormat, - innerVs.schema(), - innerVs.liveFieldNames(), - projectionSuffix(innerVs), - nestedSuffixesFor(innerVs)); - } - } - return out; - } - private static final class ProjectionCodecFactory { private final Schema historicalSchema; private final MethodHandle ctor; diff --git a/java/fory-format/src/test/java/org/apache/fory/format/encoder/SchemaEvolutionStressTest.java b/java/fory-format/src/test/java/org/apache/fory/format/encoder/SchemaEvolutionStressTest.java index 2914cc343e..6479df530f 100644 --- a/java/fory-format/src/test/java/org/apache/fory/format/encoder/SchemaEvolutionStressTest.java +++ b/java/fory-format/src/test/java/org/apache/fory/format/encoder/SchemaEvolutionStressTest.java @@ -776,5 +776,84 @@ public void crossOuterAndInnerEvolution() { Assert.assertNull(out.getInner().getAddedField()); Assert.assertNull(out.getLabel()); } + + // --------------------------------------------------------------------------- + // Cross-product enumeration must route inner-bean versions through array and + // map projection codecs, not just through the row codec. The reader's outer + // type has N outer versions x M inner versions; multiple cross-product entries + // share an outer version number, so the per-class suffix must encode the + // inner version to keep them from colliding on the codegen cache. + // --------------------------------------------------------------------------- + + @Test + public void crossOuterAndInnerEvolution_array() { + ArrayEncoder> writer = + Encoders.buildArrayCodec(new org.apache.fory.reflect.TypeRef>() {}) + .withSchemaEvolution() + .build() + .get(); + ArrayEncoder> reader = + Encoders.buildArrayCodec(new org.apache.fory.reflect.TypeRef>() {}) + .withSchemaEvolution() + .build() + .get(); + + List in = new ArrayList<>(); + for (int i = 0; i < 3; i++) { + NestedOuterWriter e = new NestedOuterWriter(); + e.setId(i); + NestedInnerWriter inn = new NestedInnerWriter(); + inn.setName("legacy-" + i); + e.setInner(inn); + in.add(e); + } + + List out = reader.decode(writer.encode(in)); + Assert.assertEquals(out.size(), 3); + for (int i = 0; i < 3; i++) { + Assert.assertEquals(out.get(i).getId(), i); + Assert.assertEquals(out.get(i).getInner().getName(), "legacy-" + i); + Assert.assertNull(out.get(i).getInner().getAddedField()); + Assert.assertNull(out.get(i).getLabel()); + } + } + + @Test + public void crossOuterAndInnerEvolution_map() { + MapEncoder> writer = + Encoders.buildMapCodec( + new org.apache.fory.reflect.TypeRef>() {}) + .withSchemaEvolution() + .build() + .get(); + MapEncoder> reader = + Encoders.buildMapCodec( + new org.apache.fory.reflect.TypeRef< + java.util.Map>() {}) + .withSchemaEvolution() + .build() + .get(); + + java.util.LinkedHashMap in = new java.util.LinkedHashMap<>(); + for (int i = 0; i < 3; i++) { + NestedOuterWriter e = new NestedOuterWriter(); + e.setId(i); + NestedInnerWriter inn = new NestedInnerWriter(); + inn.setName("legacy-" + i); + e.setInner(inn); + in.put("k" + i, e); + } + + java.util.Map out = reader.decode(writer.encode(in)); + Assert.assertEquals(out.size(), 3); + for (int i = 0; i < 3; i++) { + CrossOuterV2_InnerV2 v = out.get("k" + i); + Assert.assertNotNull(v, "missing key k" + i); + Assert.assertEquals(v.getId(), i); + Assert.assertEquals(v.getInner().getName(), "legacy-" + i); + Assert.assertNull(v.getInner().getAddedField()); + Assert.assertNull(v.getLabel()); + } + } } From 82fc5d196237c9b1056450a55d7aadbdfb53d07c Mon Sep 17 00:00:00 2001 From: "Claude (on behalf of Steven Schlansker)" Date: Thu, 28 May 2026 23:27:39 +0000 Subject: [PATCH 07/13] test(format): cover producer/consumer flag asymmetry on array and map codecs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The existing row test (evolutionFlagAsymmetryFailsLoud) had no array or map equivalent. Add both. The evolution-on consumer reading evolution-off bytes direction is loud (ClassNotCompatibleException); the reverse direction is undefined per the wire format but must not silently return a structurally plausible value. Rename isVersionedBeanElement/Value to isBeanElement/Value with a doc comment, since the predicate is just isBean — calling it "versioned" suggested the unversioned-bean case was excluded. --- .../format/encoder/ArrayCodecBuilder.java | 9 +- .../fory/format/encoder/MapCodecBuilder.java | 9 +- .../encoder/SchemaEvolutionStressTest.java | 88 +++++++++++++++++++ 3 files changed, 102 insertions(+), 4 deletions(-) diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/ArrayCodecBuilder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/ArrayCodecBuilder.java index c6ceb4e764..d8637389ba 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/ArrayCodecBuilder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/ArrayCodecBuilder.java @@ -70,7 +70,7 @@ public ArrayEncoder get() { Function> buildWithWriter() { loadArrayInnerCodecs(); - if (!schemaEvolution || !isVersionedBeanElement()) { + if (!schemaEvolution || !isBeanElement()) { final Function generatedEncoderFactory = generatedEncoderFactory(); return new Function>() { @@ -84,7 +84,12 @@ public ArrayEncoder apply(final BinaryArrayWriter writer) { return buildVersionedWithWriter(); } - private boolean isVersionedBeanElement() { + /** + * True if the element is a bean — the only case where schema evolution affects the wire + * format. Unversioned beans still take the evolution path so the strict-hash prefix is always + * present and an evolution-on consumer can detect a flag-mismatched producer cleanly. + */ + private boolean isBeanElement() { Class elementClass = getRawType(TypeUtils.getElementType(collectionType)); // Use the same resolution context as the row-format type inference, which synthesizes // interface-typed bean fields. Without this, classes that contain interface members diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/MapCodecBuilder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/MapCodecBuilder.java index c5ace01c3b..8d58deab68 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/MapCodecBuilder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/MapCodecBuilder.java @@ -61,7 +61,7 @@ public class MapCodecBuilder> extends BaseCodecBuilder> build() { loadMapInnerCodecs(); - if (!schemaEvolution || !isVersionedBeanValue()) { + if (!schemaEvolution || !isBeanValue()) { final var mapEncoderFactory = generatedMapEncoder(); return new Supplier>() { @Override @@ -81,7 +81,12 @@ public MapEncoder get() { return buildVersioned(); } - private boolean isVersionedBeanValue() { + /** + * True if the value is a bean — the only case where schema evolution affects the wire format. + * Unversioned beans still take the evolution path so the strict-hash prefix is always present + * and an evolution-on consumer can detect a flag-mismatched producer cleanly. + */ + private boolean isBeanValue() { return TypeUtils.isBean( valType, new TypeResolutionContext(CustomTypeEncoderRegistry.customTypeHandler(), true)); diff --git a/java/fory-format/src/test/java/org/apache/fory/format/encoder/SchemaEvolutionStressTest.java b/java/fory-format/src/test/java/org/apache/fory/format/encoder/SchemaEvolutionStressTest.java index 6479df530f..6980ab72a7 100644 --- a/java/fory-format/src/test/java/org/apache/fory/format/encoder/SchemaEvolutionStressTest.java +++ b/java/fory-format/src/test/java/org/apache/fory/format/encoder/SchemaEvolutionStressTest.java @@ -555,6 +555,94 @@ public void evolutionFlagAsymmetryFailsLoud() { } } + @Test + public void evolutionFlagAsymmetryFailsLoud_array() { + ArrayEncoder> withFlag = + Encoders.buildArrayCodec(new org.apache.fory.reflect.TypeRef>() {}) + .withSchemaEvolution() + .build() + .get(); + ArrayEncoder> noFlag = + Encoders.buildArrayCodec(new org.apache.fory.reflect.TypeRef>() {}) + .build() + .get(); + DefaultsV1 v = new DefaultsV1(); + v.setName("hi"); + List in = Arrays.asList(v); + // Evolution-on consumer reading evolution-off bytes: the absent strict-hash prefix is read + // out of the array header and produces a hash mismatch. + byte[] noFlagBytes = noFlag.encode(in); + try { + withFlag.decode(noFlagBytes); + Assert.fail("expected ClassNotCompatibleException"); + } catch (ClassNotCompatibleException expected) { + // ok + } + // Evolution-off consumer reading evolution-on bytes: the 8-byte hash prefix bleeds into the + // array header. We cannot guarantee a clean failure mode without a wire-format-level flag, + // but we at least require the decode to throw rather than silently return a plausible-looking + // array. Documented as wire-incompatible in the user guide; mismatched producers/consumers + // must use the same flag. + byte[] withFlagBytes = withFlag.encode(in); + try { + List out = noFlag.decode(withFlagBytes); + // If decode returned, sanity-check it didn't silently produce a "correct" result. The + // array length and the recovered string must not both look right. + boolean lengthLooksRight = out != null && out.size() == in.size(); + boolean stringLooksRight = + lengthLooksRight && !out.isEmpty() && "hi".equals(out.get(0).getName()); + Assert.assertFalse( + lengthLooksRight && stringLooksRight, + "evolution-off decoder silently accepted evolution-on bytes as a valid array"); + } catch (RuntimeException | AssertionError expected) { + // ok — undefined behavior, but a thrown exception is a tolerable failure mode. + } + } + + @Test + public void evolutionFlagAsymmetryFailsLoud_map() { + MapEncoder> withFlag = + Encoders.buildMapCodec( + new org.apache.fory.reflect.TypeRef>() {}) + .withSchemaEvolution() + .build() + .get(); + MapEncoder> noFlag = + Encoders.buildMapCodec( + new org.apache.fory.reflect.TypeRef>() {}) + .build() + .get(); + DefaultsV1 v = new DefaultsV1(); + v.setName("hi"); + java.util.LinkedHashMap in = new java.util.LinkedHashMap<>(); + in.put("k", v); + // Evolution-on consumer reading evolution-off bytes: clean hash mismatch. + byte[] noFlagBytes = noFlag.encode(in); + try { + withFlag.decode(noFlagBytes); + Assert.fail("expected ClassNotCompatibleException"); + } catch (ClassNotCompatibleException expected) { + // ok + } + // Reverse direction: see the array test above for the rationale. Require a thrown exception + // or a value that is observably wrong. + byte[] withFlagBytes = withFlag.encode(in); + try { + java.util.Map out = noFlag.decode(withFlagBytes); + boolean sizeLooksRight = out != null && out.size() == in.size(); + boolean valueLooksRight = + sizeLooksRight + && out.containsKey("k") + && out.get("k") != null + && "hi".equals(out.get("k").getName()); + Assert.assertFalse( + sizeLooksRight && valueLooksRight, + "evolution-off decoder silently accepted evolution-on bytes as a valid map"); + } catch (RuntimeException | AssertionError expected) { + // ok — undefined behavior, but a thrown exception is a tolerable failure mode. + } + } + // --------------------------------------------------------------------------- // Map with a versioned bean as the KEY (rare; documented as not dispatched). // Verify the codec at least builds and the current-version round-trip works, From 82acf9471093bfe4bb2f361c0bee406f9a276b9c Mon Sep 17 00:00:00 2001 From: "Claude (on behalf of Steven Schlansker)" Date: Thu, 28 May 2026 23:30:16 +0000 Subject: [PATCH 08/13] fix(format): prefer all-current combination when SchemaHistory signatures collapse bySignature.putIfAbsent could store a non-all-current cross-product combination under the signature that build() later marks as the writer-side current. The stored VS's nestedBeanVersions would then misreport at least one inner bean as living at a non-current version, violating the documented contract on current().nestedBeanVersions(). Reachable only if two combinations canonicalize to the same outer signature, which today's inner-bySignature collapse prevents, but the contract should not depend on that. Add a contract test that asserts the invariant for a deeply nested versioned bean. --- .../fory/format/type/SchemaHistory.java | 17 +++++++++++++- .../encoder/SchemaEvolutionStressTest.java | 23 +++++++++++++++++++ 2 files changed, 39 insertions(+), 1 deletion(-) diff --git a/java/fory-format/src/main/java/org/apache/fory/format/type/SchemaHistory.java b/java/fory-format/src/main/java/org/apache/fory/format/type/SchemaHistory.java index 953eddd8b7..a53252fef2 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/type/SchemaHistory.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/type/SchemaHistory.java @@ -180,6 +180,12 @@ public static SchemaHistory build(Class beanClass, UnaryOperator sche } // Cross-product over each nested versioned bean's history. If no entries have nested // histories, this yields a single combination. + // + // The class count generated downstream is the product of the per-bean version counts. If + // that growth becomes a concern, drop entries from each bean's History interface once you + // no longer need to read payloads from that range — that removes the corresponding + // VersionedSchema from this enumeration. Retiring history entries is purely a read-side + // concern; the writer always uses the current schema. List> innerChoices = new ArrayList<>(activeEntries.size()); List innerEntries = new ArrayList<>(activeEntries.size()); for (FieldEntry fe : activeEntries) { @@ -238,7 +244,16 @@ public static SchemaHistory build(Class beanClass, UnaryOperator sche hash, Collections.unmodifiableSet(liveNames), Collections.unmodifiableMap(nestedBeanVersionsMap)); - bySignature.putIfAbsent(signature, vs); + // Prefer the all-current combination on collapse so the stored VS's nestedBeanVersions + // map reflects the writer-side state at this outer version. This guards a contract on + // current().nestedBeanVersions() in case two combinations ever canonicalize to the + // same signature; today's inner-bySignature collapse means inner.versions() has no + // wire-equal duplicates, but the guard preserves the invariant for future callers. + if (innerAllCurrent) { + bySignature.put(signature, vs); + } else { + bySignature.putIfAbsent(signature, vs); + } if (v == latestVersion && innerAllCurrent) { currentSignature = signature; } diff --git a/java/fory-format/src/test/java/org/apache/fory/format/encoder/SchemaEvolutionStressTest.java b/java/fory-format/src/test/java/org/apache/fory/format/encoder/SchemaEvolutionStressTest.java index 6980ab72a7..c27b110bf5 100644 --- a/java/fory-format/src/test/java/org/apache/fory/format/encoder/SchemaEvolutionStressTest.java +++ b/java/fory-format/src/test/java/org/apache/fory/format/encoder/SchemaEvolutionStressTest.java @@ -865,6 +865,29 @@ public void crossOuterAndInnerEvolution() { Assert.assertNull(out.getLabel()); } + /** + * Contract: {@code SchemaHistory.current().nestedBeanVersions()} must report each nested bean + * at its current version. Two cross-product combinations canonicalizing to the same signature + * is rare today (the inner's own bySignature collapses wire-equal schemas before the outer + * sees them) but the contract is documented and future callers may rely on it. + */ + @Test + public void schemaHistoryCurrentReflectsCurrentInnerVersions() { + org.apache.fory.format.type.SchemaHistory history = + org.apache.fory.format.type.SchemaHistory.build( + CrossOuterV2_InnerV2.class, java.util.function.UnaryOperator.identity()); + org.apache.fory.format.type.SchemaHistory.VersionedSchema current = history.current(); + for (java.util.Map.Entry, Integer> e : current.nestedBeanVersions().entrySet()) { + org.apache.fory.format.type.SchemaHistory innerHistory = + org.apache.fory.format.type.SchemaHistory.build( + e.getKey(), java.util.function.UnaryOperator.identity()); + Assert.assertEquals( + (int) e.getValue(), + innerHistory.current().version(), + "current().nestedBeanVersions() must report inner " + e.getKey() + " at its current"); + } + } + // --------------------------------------------------------------------------- // Cross-product enumeration must route inner-bean versions through array and // map projection codecs, not just through the row codec. The reader's outer From 685a46a3246b48a628692fdb0b3df0cf324c0aca Mon Sep 17 00:00:00 2001 From: "Claude (on behalf of Steven Schlansker)" Date: Thu, 28 May 2026 23:32:55 +0000 Subject: [PATCH 09/13] test(format): cover @ForyVersion on record components @ForyVersion declares RECORD_COMPONENT as a valid target but no test exercised the record path. Add three cases in fory-latest-jdk-tests: a record with a String field added at v2, a record with the @ForySchema-removed-field History interface, and a record with a primitive int field added at v2 (verifying the 0 default). --- .../fory/integration_tests/RecordRowTest.java | 54 +++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/java/fory-latest-jdk-tests/src/test/java/org/apache/fory/integration_tests/RecordRowTest.java b/java/fory-latest-jdk-tests/src/test/java/org/apache/fory/integration_tests/RecordRowTest.java index 99c61c64ce..117d2a112d 100644 --- a/java/fory-latest-jdk-tests/src/test/java/org/apache/fory/integration_tests/RecordRowTest.java +++ b/java/fory-latest-jdk-tests/src/test/java/org/apache/fory/integration_tests/RecordRowTest.java @@ -21,6 +21,8 @@ import java.time.Instant; import java.time.LocalDate; +import org.apache.fory.format.annotation.ForySchema; +import org.apache.fory.format.annotation.ForyVersion; import org.apache.fory.format.encoder.Encoders; import org.apache.fory.format.encoder.RowEncoder; import org.apache.fory.format.row.binary.BinaryRow; @@ -86,4 +88,56 @@ public void testRecordNestedInterface() { final TestRecordNestedInterface deserializedBean = encoder.fromRow(row); Assert.assertEquals(deserializedBean.f1().f1(), bean.f1().f1()); } + + // --------------------------------------------------------------------------- + // Records with schema evolution. @ForyVersion targets RECORD_COMPONENT, so a + // newer reader record can pick up older payloads, defaulting components added + // later. The history interface still works because the bean is a record: live + // component names match the wire field names (record short-style naming). + // --------------------------------------------------------------------------- + + public record PersonV1(String name, int age) {} + + @ForySchema(removedFields = PersonV2.History.class) + public record PersonV2(String name, @ForyVersion(since = 2) String email) { + interface History { + @ForyVersion(until = 2) + int age(); + } + } + + @Test + public void recordSchemaEvolution_readsOlderPayloads() { + RowEncoder writer = + Encoders.buildBeanCodec(PersonV1.class).withSchemaEvolution().build().get(); + RowEncoder reader = + Encoders.buildBeanCodec(PersonV2.class).withSchemaEvolution().build().get(); + PersonV2 out = reader.decode(writer.encode(new PersonV1("Luna", 7))); + Assert.assertEquals(out.name(), "Luna"); + Assert.assertNull(out.email()); + } + + @Test + public void recordSchemaEvolution_currentRoundTrip() { + RowEncoder codec = + Encoders.buildBeanCodec(PersonV2.class).withSchemaEvolution().build().get(); + PersonV2 in = new PersonV2("Mars", "mars@example.com"); + Assert.assertEquals(codec.decode(codec.encode(in)), in); + } + + /** Record with a primitive added at v2: an older payload must produce the primitive default. */ + public record CounterV1(String name) {} + + public record CounterV2(String name, @ForyVersion(since = 2) int count) {} + + @Test + public void recordSchemaEvolution_primitiveDefault() { + RowEncoder writer = + Encoders.buildBeanCodec(CounterV1.class).withSchemaEvolution().build().get(); + RowEncoder reader = + Encoders.buildBeanCodec(CounterV2.class).withSchemaEvolution().build().get(); + CounterV2 out = reader.decode(writer.encode(new CounterV1("Luna"))); + Assert.assertEquals(out.name(), "Luna"); + Assert.assertEquals(out.count(), 0); + } } From c462410f81804021d189d6af1e821d72aa40ebeb Mon Sep 17 00:00:00 2001 From: "Claude (on behalf of Steven Schlansker)" Date: Thu, 28 May 2026 23:33:41 +0000 Subject: [PATCH 10/13] docs(format): clarify wire format and cross-product growth Tighten the row-format schema-evolution doc to reflect the actual flag-mismatch behavior (loud in one direction, undefined in the reverse for array/map) and add a note that the projection codec class count grows as the product of per-bean version counts in a composition, with retiring history entries as the way to bound it. --- docs/guide/java/row-format.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/docs/guide/java/row-format.md b/docs/guide/java/row-format.md index b3d06ab166..f446491e8f 100644 --- a/docs/guide/java/row-format.md +++ b/docs/guide/java/row-format.md @@ -168,16 +168,24 @@ original live descriptor name: the field name for Lombok `@Data` or record-style ### Wire format and limitations Producers and consumers must agree on the `withSchemaEvolution()` flag — they are not -wire-compatible otherwise. Row payloads already carry an 8-byte hash slot whose value changes -under evolution (the strict hash includes field name and nullability). For arrays and maps -whose element bean opts into evolution, an 8-byte hash prefix is prepended; arrays and maps -whose element is not a versioned bean carry no prefix. +wire-compatible otherwise. Row payloads always carry an 8-byte hash slot; under evolution its +value is the strict hash (which includes field name and nullability), so a flag-mismatched +peer fails loudly with `ClassNotCompatibleException`. Arrays and maps of bean elements prepend +an 8-byte strict-hash prefix under evolution and no prefix otherwise; an evolution-on consumer +reading evolution-off bytes also fails with `ClassNotCompatibleException`, but the reverse +direction (evolution-off consumer, evolution-on bytes) is undefined. Cross-language consumers (Python, C++) cannot read evolution-enabled payloads. Map keys do not carry a per-payload hash; a versioned bean used as a map key is read with the current schema only, not dispatched to a projection codec. +When a versioned bean contains other versioned beans, the reader generates one projection codec +class per combination of versions across the composition. The count grows as the product of the +per-bean version counts. If that becomes a concern, drop entries from each bean's `History` +interface once you no longer need to read payloads from that range. Retiring a history entry is +purely a read-side decision; the writer always uses the current schema. + ## Cross-Language Compatibility Row format works seamlessly across languages. The same binary data can be accessed from: From f59e96df141ff1f0c84ec3685a6c0d5c09054d28 Mon Sep 17 00:00:00 2001 From: "Claude (on behalf of Steven Schlansker)" Date: Thu, 28 May 2026 23:34:21 +0000 Subject: [PATCH 11/13] docs(format): drop stray javadoc fragment on Encoders The class-level javadoc had a trailing "

, ganrunsheng" line, evidently a truncated tag. Reduce the class doc to its one useful sentence. --- .../main/java/org/apache/fory/format/encoder/Encoders.java | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/Encoders.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/Encoders.java index d6cd609cd8..b2ff3b42b8 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/Encoders.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/Encoders.java @@ -47,11 +47,7 @@ import org.apache.fory.type.TypeUtils; import org.apache.fory.util.Preconditions; -/** - * Factory to create {@link Encoder}. - * - *

, ganrunsheng - */ +/** Factory to create {@link Encoder}. */ public class Encoders { private static final Logger LOG = LoggerFactory.getLogger(Encoders.class); From c57abc079bd667e29a29e60fcfb69e452a818e72 Mon Sep 17 00:00:00 2001 From: "Claude (on behalf of Steven Schlansker)" Date: Thu, 28 May 2026 23:35:36 +0000 Subject: [PATCH 12/13] docs(format): note byte[] decode skips sizeEmbedded by design decode(byte[]) takes bytes.length as the payload size unconditionally, paired with encode(T) which writes no size prefix either. The MemoryBuffer overloads respect sizeEmbedded for stream framing; the byte[] overloads do not, because they handle a single message. Comment the three byte[] decode methods so the asymmetry isn't read as a bug. --- .../java/org/apache/fory/format/encoder/BinaryArrayEncoder.java | 2 ++ .../java/org/apache/fory/format/encoder/BinaryMapEncoder.java | 2 ++ .../java/org/apache/fory/format/encoder/BinaryRowEncoder.java | 2 ++ 3 files changed, 6 insertions(+) diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryArrayEncoder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryArrayEncoder.java index 20aa7d8444..b69ae854ce 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryArrayEncoder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryArrayEncoder.java @@ -94,6 +94,8 @@ public T decode(final MemoryBuffer buffer) { @Override public T decode(final byte[] bytes) { + // byte[] is treated as a single-message payload — no size prefix, regardless of + // sizeEmbedded, paired with encode(T) which writes no size prefix either. return decode(MemoryUtils.wrap(bytes), bytes.length); } diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryMapEncoder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryMapEncoder.java index 9bec2a170c..a2b8d85622 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryMapEncoder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryMapEncoder.java @@ -144,6 +144,8 @@ M decode(final MemoryBuffer buffer, final int size) { @Override public M decode(final byte[] bytes) { + // byte[] is treated as a single-message payload — no size prefix, regardless of + // sizeEmbedded, paired with encode(M) which writes no size prefix either. return decode(MemoryUtils.wrap(bytes), bytes.length); } diff --git a/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryRowEncoder.java b/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryRowEncoder.java index 600b3acbf6..3ecf3e286c 100644 --- a/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryRowEncoder.java +++ b/java/fory-format/src/main/java/org/apache/fory/format/encoder/BinaryRowEncoder.java @@ -130,6 +130,8 @@ T decode(final MemoryBuffer buffer, final int size) { @Override public T decode(final byte[] bytes) { + // byte[] is treated as a single-message payload — no size prefix, regardless of + // sizeEmbedded, paired with encode(T) which writes no size prefix either. return decode(MemoryUtils.wrap(bytes), bytes.length); } From eabbce4f28b53df7e1fb538bab3d4b8e31509c03 Mon Sep 17 00:00:00 2001 From: "Claude (on behalf of Steven Schlansker)" Date: Thu, 28 May 2026 23:58:42 +0000 Subject: [PATCH 13/13] docs(format): tighten row-format schema-evolution prose Three small edits in the row-format schema-evolution section: name all primitive defaults (0, 0.0, false), fold the "parameterized types are expressed naturally" assertion into the lead-in to the removed-field example, and drop the trailing sentence that restated what the example already showed. --- docs/guide/java/row-format.md | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/docs/guide/java/row-format.md b/docs/guide/java/row-format.md index f446491e8f..7da4067753 100644 --- a/docs/guide/java/row-format.md +++ b/docs/guide/java/row-format.md @@ -133,13 +133,13 @@ public class Person { ``` A v1 payload (with `name` and `age` only) decodes to a `Person` whose `email` is `null`. -Primitive fields added later default to `0` / `false`. If a class adopts versioning after its -v1 is already in the wild, set `@ForySchema(baseVersion = N)` so unannotated fields are -treated as present since version `N`. +Primitive fields added later default to `0`, `0.0`, or `false`. If a class adopts versioning +after its v1 is already in the wild, set `@ForySchema(baseVersion = N)` so unannotated fields +are treated as present since version `N`. -Remove a field by deleting the Java member and listing it on a nested history interface. The -interface's methods carry the original field's name, return type, and `[since, until)` window. -Parameterized types are expressed naturally because the methods are real Java declarations. +Remove a field by deleting the Java member and declaring it on a nested history interface as a +method with a `@ForyVersion(until = N)`. The method's return type carries any parameterized +type information from the original field. ```java @Data @@ -160,10 +160,9 @@ public class Person { } ``` -Each history method must carry a `@ForyVersion` with `until` set. The method name matches the -original live descriptor name: the field name for Lombok `@Data` or record-style classes -(`age`, `tags`), or the full accessor name for JavaBeans-style classes and interfaces -(`getAge`). +The history method name matches the original live descriptor name: the field name for Lombok +`@Data` or records (`age`, `tags`), or the full accessor name for JavaBeans-style classes and +interfaces (`getAge`). ### Wire format and limitations