diff --git a/CHANGELOG.md b/CHANGELOG.md index a4d48df..c8c8183 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,8 +3,30 @@ v0.3.1 (Unreleased) ### Query translation * **`toStartOf*` date-time functions** via `EF.Functions`: `ToStartOfYear`, `ToStartOfQuarter`, `ToStartOfMonth`, `ToStartOfWeek` (with optional week `mode`), `ToStartOfDay`, `ToStartOfHour`, `ToStartOfMinute`, `ToStartOfSecond`, the fixed buckets `ToStartOfFiveMinutes` / `ToStartOfTenMinutes` / `ToStartOfFifteenMinutes`, and the general `ToStartOfInterval(source, value, unit)`. Each maps to the matching ClickHouse function and works in `GROUP BY`. Return types follow ClickHouse: the calendar buckets (`Year`/`Quarter`/`Month`/`Week`) return `Date`, the day/hour/minute buckets return `DateTime`, and `ToStartOfSecond` returns `DateTime64`. All accept `DateTime`/`DateTime64` columns, and the plain truncation functions also accept `DateOnly`; `ToStartOfInterval` requires a `DateTime`/`DateTime64` column on older ClickHouse, which rejects a `DateOnly` (Date/Date32) source with `Illegal type Date32 of 1st argument` for every unit; recent versions accept it. `ToStartOfInterval` takes a `ClickHouseInterval` unit (`Second`…`Year`) and emits `toStartOfInterval(source, toInterval(value))`; the unit must be a constant so it can be translated. The default `Date`/`DateTime` result types only span 1970–2149/2106, so ClickHouse narrows out-of-range values — enable `enable_extended_results_for_datetime_functions` (e.g. `set_enable_extended_results_for_datetime_functions=1` in the connection string) for range-preserving `Date32`/`DateTime64` results. +### Types +* **`DateTimeOffset` support.** A `DateTimeOffset` property now maps to `DateTime64(7, 'UTC')` through the new `ClickHouseDateTimeOffsetTypeMapping`. Previously the provider had no mapping for the type, so EF Core fell back to `DateTimeOffsetToStringConverter` and silently produced a `String` column. That fallback broke queries against real `DateTime64` columns (`TYPE_MISMATCH`, because the parameter was declared `String`) and `SaveChanges` could not write the value at all. ([#53](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/53)) + * The store type is UTC-pinned on purpose. For a timezone-less parameter type such as `DateTime64(7)`, the driver sends a UTC wall clock and the server then reads it in `session_timezone`, which moves the instant when that setting is not UTC. + * Precision 7 matches one .NET tick (100 ns), so the round trip is exact and no value is silently truncated. + * ClickHouse has no type that stores a UTC offset, so the instant is kept and the offset is not. A value read back carries the offset of the column's declared timezone, and a non-UTC timezone such as `DateTime64(6, 'Asia/Tokyo')` is read correctly. One exception: in a timezone with daylight saving, the repeated hour when clocks go back is ambiguous, because the driver gives a wall clock and drops the offset. The provider recovers the instant when the zone's standard offset is zero (Europe/London); where both candidate offsets are non-zero (Europe/Paris) the reading can be one hour early. The default `'UTC'` store type has no daylight saving and is not affected. + * A timezone that the host operating system cannot resolve now throws a clear error instead of silently reading the value as UTC, which would have been wrong by the zone's offset. Minimal Linux images may need the `tzdata` package. + * A column may declare a fixed UTC offset rather than a named zone, which ClickHouse spells `Fixed/UTC±HH:MM:SS` — for example `DateTime64(7, 'Fixed/UTC+05:30:00')`. No .NET timezone has such a name, so these are read by parsing the offset out of the name, and they are never ambiguous because a fixed offset does not change. Two kinds of such a column cannot be read, and each reports the timezone and the reason: an offset that `DateTimeOffset` cannot hold, because it caps the magnitude at 14 hours and requires whole minutes (`Fixed/UTC+15:00:00`, `Fixed/UTC+00:00:42`); and a name whose minutes or seconds field is above 59, which ClickHouse carries (`Fixed/UTC+05:60:00` means `+06:00`) but the driver does not read — the message gives the plain spelling to use instead. + * **Known limit:** `DateTimeOffset.MinValue` and `MaxValue` only round trip through a column whose timezone offset is zero. Both sit at the edge of the `DateTime` range, and the driver builds a wall clock in the column's timezone to return a value, so a non-zero offset pushes one end outside `DateTime` and the read throws. A named zone is worse than a fixed offset: zones carry a Local Mean Time offset for year 1 (`+09:18:59` for `Asia/Tokyo`), so a value near `MinValue` reads back quietly shifted instead. + * `HasPrecision(n)` is honoured for a property with no `HasColumnType`, and keeps the UTC pin — for example `HasPrecision(3)` gives `DateTime64(3, 'UTC')`. + * SQL literals carry the offset (`'2026-01-15 10:00:00.1234567+05:00'`), which lands on the same instant whatever timezone the target column declares. + * No value converter is used, so the driver receives the `DateTimeOffset` directly on both the query parameter path and the bulk insert path. + * **Behaviour change:** a `DateTimeOffset` property that relied on the old `String` column now resolves to `DateTime64(7, 'UTC')`. Add `HasConversion()` to keep the previous shape. Note that `HasColumnType("String")` on its own is not enough — it resolves the plain string mapping with no converter, so the CLR type no longer agrees with the property. Keeping the old shape also makes the property read-only until [#54](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/54) is fixed, because `SaveChanges` does not apply a value converter on the insert path. + ### Bug fixes * `ToStartOfWeek` now rejects row-dependent week modes during query translation, and `ToStartOfInterval` likewise rejects row-dependent interval sizes. ClickHouse requires these operands to be constant for the query; literals and captured query parameters remain supported. +* **Composite columns now convert their components on read.** `Array(T)`, `Map(K, V)`, and `Tuple(...)` read the whole column through `GetValue`, so a component mapping's own read pipeline never ran. Any component whose CLR type differs from the type the driver produces therefore threw `InvalidCastException` — both `DateTimeOffset` and `DateOnly` arrive from the driver as `DateTime`. `DateOnly[]`, `Dictionary` and `Tuple` were affected before `DateTimeOffset` existed as a mapped type. + * The composite is now rebuilt component by component, applying both steps EF Core applies to a scalar column: the mapping's data-reader conversion, then its `ValueConverter`. Components that convert through a converter therefore work too — a C# `enum` component (`EnumToStringConverter`), a `List` component (`ListToArrayConverter`), and the collection interfaces (`IList`, `IReadOnlyList`). + * A component that needs no conversion keeps the direct cast. Note that the integer mappings do convert, because they widen with `Convert.ToInt32` for aggregates; a runtime fast path returns the driver's value untouched when it is already the target type, so those columns cost nothing beyond the check. That fast path is skipped for a component carrying a `ValueConverter`, which may change the value while keeping the CLR type, so a matching type is no proof that the read is done. An element converter set with `ElementType(el => el.HasConversion(...))` is the reachable case. + * Nested composites compose: the element mapping of `Array(Array(DateTime64))` is itself an array mapping, so its conversion runs per element. + * `Array(Nullable(T))` also converts, with nulls passing straight through. A NULL in a non-nullable tuple slot now reports which component and type disagree instead of failing with a `NullReferenceException`. + * A component that cannot be read as the property's type now reports the store type and both CLR types, rather than EF Core's bare `No coercion operator is defined between types ...`. + * **Known limit:** *writing* a component that needs a `ValueConverter` still does not work, because the bulk insert path passes model values to the driver without applying converters ([#54](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/54)). An `enum` inside a composite is written as its raw ordinal. Reading such a column works; populate it outside EF until #54 is fixed. +* **`Array(Nullable(T))` DDL is no longer double-wrapped.** For a value-type element, the store type came out as `Array(Nullable(Nullable(T)))`, so `EnsureCreated` and migrations produced DDL that ClickHouse rejects with `Nested type Nullable(T) cannot be inside Nullable type`. The component mapping is resolved from a store type that already carries the wrapper, and `HasColumnType(...)` text is kept verbatim, so the nullable-element wrapper added a second one. It now adds the wrapper only when the inner store type does not already have it, including through `LowCardinality(Nullable(T))`. Reference-type elements were never affected. +* **Composite component mappings now honour the CLR component type.** A single ClickHouse store type can serve more than one CLR type — `DateTime64` serves both `DateTime` and `DateTimeOffset`, and `Date32` serves both `DateTime` and `DateOnly`. Resolving a component from an explicit store type such as `HasColumnType("Array(Nullable(DateTime64(7, 'UTC')))")` always picked the default CLR type, which gave the composite the wrong element type and broke change tracking. Array, Map, and Tuple now pass the component CLR type from the model when they have one. * `Sum`/`SumAsync` over a `double` or `float` column no longer throws `InvalidCastException`. EF Core wraps a top-level aggregate so the empty case returns `0`, supplying that fallback as a boxed `Int32` carrying the `Float64`/`Float32` mapping; the literal generators now convert rather than unbox. The `Float32` read path also converts, since ClickHouse widens `sum(Float32)` to `Float64` (which the driver's `GetFloat()` refuses to downcast). ([#46](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/46)) * **SummingMergeTree with multiple sum columns**: `HasSummingMergeTreeEngine("A", "B")` now generates valid DDL (`SummingMergeTree((A, B))`). Previously it emitted a comma-separated argument list (`SummingMergeTree(A, B)`), which ClickHouse rejects with `NUMBER_OF_ARGUMENTS_DOESNT_MATCH`. Single-column and no-column usage are unaffected. diff --git a/README.md b/README.md index dc7d9c8..6d365aa 100644 --- a/README.md +++ b/README.md @@ -67,7 +67,7 @@ public class PageView | **Bool** | `Bool` | `bool` | | **Strings** | `String`, `FixedString(N)` | `string` | | **Enums** | `Enum8(...)`, `Enum16(...)` | `string` or C# `enum` | -| **Date/time** | `Date`, `Date32`, `DateTime`, `DateTime64(P, 'TZ')` | `DateOnly`, `DateTime` | +| **Date/time** | `Date`, `Date32`, `DateTime`, `DateTime64(P, 'TZ')` | `DateOnly`, `DateTime`, `DateTimeOffset` (see [below](#datetimeoffset)) | | **Time** | `Time`, `Time64(N)` | `TimeSpan` | | **UUID** | `UUID` | `Guid` | | **Network** | `IPv4`, `IPv6` | `IPAddress` | @@ -80,6 +80,101 @@ public class PageView | **Geographic** | `Point`, `Ring`, `LineString`, `Polygon`, `MultiLineString`, `MultiPolygon`, `Geometry` | `Tuple` and arrays thereof; `object` for Geometry | | **Wrappers** | `Nullable(T)`, `LowCardinality(T)` | Unwrapped automatically | +### DateTimeOffset + +A `DateTimeOffset` property maps to `DateTime64(7, 'UTC')` by default: + +```csharp +public class Reading +{ + public long Id { get; set; } + public DateTimeOffset RecordedAt { get; set; } // DateTime64(7, 'UTC') +} +``` + +Three things to know: + +**The offset is not kept.** ClickHouse has no type that stores a UTC offset. `DateTime64` holds an +instant, and a declared timezone only decides how that instant is rendered. A value written with +any offset is stored as the correct instant, and a value read back carries the offset of the +column's timezone — `+00:00` for the default store type. With that default store type, comparisons +and ordering are instant-correct, so these two values match the same row: + +```csharp +// The same instant, written two ways. +var a = new DateTimeOffset(2026, 1, 15, 10, 0, 0, TimeSpan.FromHours(5)); +var b = new DateTimeOffset(2026, 1, 15, 5, 0, 0, TimeSpan.Zero); +``` + +If you must keep the offset, store it yourself in a second column alongside a `DateTime`. To keep +the whole value as text, ask for the conversion explicitly with `HasConversion()` — note +that `HasColumnType("String")` on its own is not enough, because it adds no converter. Such a +column is read-only for now: `SaveChanges` cannot write any property that has a value converter +([#54](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/54)). + +**Precision 7 makes the round trip exact.** One .NET tick is 100 ns, which is precision 7, so a +stored value never comes back truncated. Precision 7 also covers the full `DateTimeOffset` range, +which lets you use `DateTimeOffset.MinValue` and `MaxValue` as open-ended range limits on the +default store type. Keep those two sentinels to a column whose timezone offset is zero, such as the +default `'UTC'`. Both sit at the edge of the `DateTime` range, and the driver has to build a wall +clock in the column's timezone to return a value, so any non-zero offset pushes one end outside +`DateTime`: reading `MaxValue` from a `DateTime64(7, 'Asia/Tokyo')` column throws. Choose a smaller +precision if you prefer, but be aware that it discards the digits below it: + +```csharp +b.Property(e => e.RecordedAt).HasColumnType("DateTime64(3, 'UTC')"); // milliseconds +b.Property(e => e.RecordedAt).HasPrecision(3); // the same thing +``` + +Do not go above precision 7. .NET cannot represent more than 7 fractional digits, and the driver +overflows an `Int64` at precision 8 or 9 for dates far from the epoch, which corrupts the value +without an error. + +**Keep `'UTC'` in the store type** unless you have a reason to change it. For a timezone-less type +such as `DateTime64(7)`, the server reads the query parameter in its `session_timezone`, which moves +the instant when that setting is not UTC. + +A column that declares a different timezone, for example `DateTime64(6, 'Asia/Tokyo')`, is read +correctly and returns that zone's offset. Two limits apply to such a column: + +- The host operating system must know the timezone, or the read throws. Minimal Linux images may + need the `tzdata` package. +- In a zone with daylight saving, the repeated hour when clocks go back is ambiguous, because the + driver gives a wall clock and drops the offset. The provider recovers the instant where the zone's + standard offset is zero, such as `Europe/London`. Where both candidate offsets are non-zero, such + as `Europe/Paris`, the value can read back one hour early. +- Dates at the far ends of the `DateTimeOffset` range do not survive, as described above. A named + zone is worse than a fixed offset here: zones carry a Local Mean Time offset for year 1 + (`+09:18:59` for `Asia/Tokyo`), so a value near `DateTimeOffset.MinValue` reads back quietly + shifted rather than reporting an error. + +A column can also declare a fixed UTC offset instead of a named zone. ClickHouse spells this +`Fixed/UTC±HH:MM:SS`, with two digits in every field — the server rejects `Fixed/UTC+5:30:00` and +`Fixed/UTC+05:30`: + +```csharp +b.Property(e => e.RecordedAt).HasColumnType("DateTime64(7, 'Fixed/UTC+05:30:00')"); +``` + +Neither limit above applies here: the host needs no timezone data, and a fixed offset is never +ambiguous. Two limits of its own do, and each reports the timezone and the reason rather than +failing obscurely: + +- `DateTimeOffset` holds an offset only within plus or minus 14 hours, and only in whole minutes. + ClickHouse accepts more, so `Fixed/UTC+15:00:00` and `Fixed/UTC+00:00:42` cannot be read into one. +- ClickHouse does not hold the minutes and seconds fields to 59 — it carries the excess, so + `Fixed/UTC+05:60:00` is a legal name for `+06:00`. The driver reads only the plain spelling, so + declare that offset as `Fixed/UTC+06:00:00`. + +Map such a column as `DateTime` if you cannot change how it is declared. + +`DateTimeOffset` also composes into the collection types, so `DateTimeOffset[]`, +`List`, `Dictionary` and `Tuple` all +round trip. + +`DateTimeOffset` members such as `.Year` and `.UtcDateTime` do not translate to SQL yet. This +applies to `DateTime` as well — see [#55](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/55). + ## Current Status This provider is in active development. It supports **LINQ queries**, **inserts**, **table engine configuration**, and **migrations** — you can define ClickHouse tables with engine-specific settings, create them via `dotnet ef migrations` or `EnsureCreated`, query with LINQ, and write data via `SaveChanges`. @@ -272,7 +367,7 @@ Configure ClickHouse table engines, ordering, partitioning, and more via EF Core ```csharp modelBuilder.Entity(b => { - b.HasKey(e => e.Id); + b.HasKey(e => e.Id); // becomes ORDER BY (the ClickHouse primary key) when no explicit ORDER BY is set b.Property(e => e.Temperature).HasCodec("Delta, ZSTD"); b.Property(e => e.Location).HasColumnComment("Installation site"); b.HasIndex(e => e.Timestamp) @@ -298,6 +393,8 @@ modelBuilder.Entity(b => **Default behavior:** If no engine is configured, the provider defaults to `MergeTree` with the EF primary key as `ORDER BY`. +**Primary key vs sorting key:** In ClickHouse the `ORDER BY` (sorting key) *is* the primary key, so `HasKey` alone is sufficient — it becomes `ORDER BY`. Only use `.WithPrimaryKey(...)` when you need the primary index to differ from the sort order (e.g. a `SummingMergeTree`/`AggregatingMergeTree` rollup with a long `ORDER BY` but a narrow index). ClickHouse requires the primary key to be a prefix of the `ORDER BY` columns. + ### Migrations The provider supports `dotnet ef migrations` for creating and applying migrations: diff --git a/src/EFCore.ClickHouse/Metadata/Builders/ClickHouseEngineBuilder.cs b/src/EFCore.ClickHouse/Metadata/Builders/ClickHouseEngineBuilder.cs index 5b57490..85d90f3 100644 --- a/src/EFCore.ClickHouse/Metadata/Builders/ClickHouseEngineBuilder.cs +++ b/src/EFCore.ClickHouse/Metadata/Builders/ClickHouseEngineBuilder.cs @@ -16,6 +16,10 @@ protected ClickHouseEngineBuilder(IMutableEntityType entityType, string engineNa entityType.SetEngine(engineName); } + /// + /// Sets the table's sorting key (ORDER BY). In ClickHouse the sorting key also serves as the + /// primary key unless an explicit one is set via . + /// public ClickHouseEngineBuilder WithOrderBy(params string[] columns) { ArgumentNullException.ThrowIfNull(columns); @@ -30,6 +34,11 @@ public ClickHouseEngineBuilder WithPartitionBy(params string[] columns) return this; } + /// + /// Sets an explicit primary key (PRIMARY KEY) distinct from the sorting key. Only needed when the + /// primary index should differ from ORDER BY; otherwise the sorting key is used as the primary key. + /// ClickHouse requires these columns to be a prefix of the columns. + /// public ClickHouseEngineBuilder WithPrimaryKey(params string[] columns) { ArgumentNullException.ThrowIfNull(columns); diff --git a/src/EFCore.ClickHouse/Storage/Internal/ClickHouseTypeMappingSource.cs b/src/EFCore.ClickHouse/Storage/Internal/ClickHouseTypeMappingSource.cs index 22e9e08..eab8f8a 100644 --- a/src/EFCore.ClickHouse/Storage/Internal/ClickHouseTypeMappingSource.cs +++ b/src/EFCore.ClickHouse/Storage/Internal/ClickHouseTypeMappingSource.cs @@ -28,6 +28,7 @@ public class ClickHouseTypeMappingSource : RelationalTypeMappingSource private static readonly RelationalTypeMapping Float64Mapping = new ClickHouseDoubleTypeMapping(); private static readonly RelationalTypeMapping DateTimeMapping = new ClickHouseDateTimeTypeMapping(); private static readonly RelationalTypeMapping DateTime64Mapping = new ClickHouseDateTime64TypeMapping(); + private static readonly RelationalTypeMapping DateTimeOffsetMapping = new ClickHouseDateTimeOffsetTypeMapping(); private static readonly RelationalTypeMapping DateOnlyMapping = new ClickHouseDateOnlyTypeMapping(); private static readonly RelationalTypeMapping GuidMapping = new ClickHouseGuidTypeMapping(); private static readonly RelationalTypeMapping IPv4Mapping = new ClickHouseIPAddressTypeMapping("IPv4"); @@ -80,6 +81,7 @@ public class ClickHouseTypeMappingSource : RelationalTypeMappingSource { typeof(float), Float32Mapping }, { typeof(double), Float64Mapping }, { typeof(DateTime), DateTimeMapping }, + { typeof(DateTimeOffset), DateTimeOffsetMapping }, { typeof(DateOnly), DateOnlyMapping }, { typeof(Guid), GuidMapping }, { typeof(char), StringMapping }, @@ -144,6 +146,9 @@ public class ClickHouseTypeMappingSource : RelationalTypeMappingSource // Matches a single-quoted string like 'UTC' or 'Asia/Tokyo' private static readonly Regex TimezoneRegex = new(@"'([^']+)'", RegexOptions.Compiled); + /// ClickHouse reads a bare DateTime64 with no argument as precision 3. + private const int BareDateTime64Precision = 3; + public ClickHouseTypeMappingSource( TypeMappingSourceDependencies dependencies, RelationalTypeMappingSourceDependencies relationalDependencies) @@ -283,6 +288,7 @@ public ClickHouseTypeMappingSource( // Call base so plugin/extension type mappings can intercept before our defaults. var mapping = base.FindMapping(in mappingInfo) + ?? FindDateTimeOffsetMapping(mappingInfo) ?? FindDateTime64Mapping(mappingInfo) ?? FindDateTimeMapping(mappingInfo) ?? FindFixedStringMapping(mappingInfo) @@ -340,6 +346,56 @@ private static bool IsCollectionClrType(Type? clrType) || def == typeof(IReadOnlyCollection<>); } + /// + /// Resolves properties. This runs before the + /// DateTime64/DateTime resolvers and before the store-type aliases, because those + /// all produce a CLR type. Without it, EF Core would find no mapping and + /// fall back to DateTimeOffsetToStringConverter, which silently makes a + /// String column (issue #53). + /// + private static RelationalTypeMapping? FindDateTimeOffsetMapping(in RelationalTypeMappingInfo mappingInfo) + { + if (mappingInfo.ClrType != typeof(DateTimeOffset)) + return null; + + var baseName = mappingInfo.StoreTypeNameBase; + var storeTypeName = mappingInfo.StoreTypeName; + + // No store type configured — use the UTC-pinned default, but respect HasPrecision(n). + if (string.IsNullOrWhiteSpace(baseName) && string.IsNullOrWhiteSpace(storeTypeName)) + { + return mappingInfo.Precision is null + ? DateTimeOffsetMapping + : new ClickHouseDateTimeOffsetTypeMapping( + mappingInfo.Precision, + ClickHouseDateTimeOffsetTypeMapping.DefaultTimezone); + } + + if (string.Equals(baseName, "DateTime64", StringComparison.OrdinalIgnoreCase)) + { + // A bare DateTime64 with no argument is precision 3 in ClickHouse, which is what + // FindDateTime64Mapping assumes as well. Our own default of 7 applies only when the + // model configures no store type at all. + return new ClickHouseDateTimeOffsetTypeMapping( + mappingInfo.Precision ?? BareDateTime64Precision, + storeTypeName is null ? null : ExtractTimezone(storeTypeName)); + } + + if (string.Equals(baseName, "DateTime", StringComparison.OrdinalIgnoreCase)) + { + return new ClickHouseDateTimeOffsetTypeMapping( + precision: null, + storeTypeName is null ? null : ExtractTimezone(storeTypeName)); + } + + // Any other explicit store type falls through to the resolvers below, which key off the + // store type rather than the CLR type. Pointing a DateTimeOffset property at an unrelated + // store type such as String therefore gives that store type's mapping with no converter, + // and the CLR type will not agree with the property. Use HasConversion() to store + // the value as text. + return null; + } + private RelationalTypeMapping? FindDateTime64Mapping(in RelationalTypeMappingInfo mappingInfo) { if (!string.Equals(mappingInfo.StoreTypeNameBase, "DateTime64", StringComparison.OrdinalIgnoreCase)) @@ -350,7 +406,7 @@ private static bool IsCollectionClrType(Type? clrType) if (storeTypeName is null || !storeTypeName.Contains('(')) return null; - var precision = mappingInfo.Precision ?? 3; + var precision = mappingInfo.Precision ?? BareDateTime64Precision; var timezone = ExtractTimezone(storeTypeName); return new ClickHouseDateTime64TypeMapping(precision, timezone); } @@ -391,6 +447,7 @@ private static bool IsCollectionClrType(Type? clrType) private RelationalTypeMapping? FindArrayMapping(in RelationalTypeMappingInfo mappingInfo) { RelationalTypeMapping? elementMapping = null; + var elementClrTypeHint = GetCollectionElementType(mappingInfo.ClrType); // Resolve element mapping from store type: Array(X). When the user wrote // HasColumnType("Array(...)"), prefer parsing the inner type from the store @@ -406,7 +463,7 @@ private static bool IsCollectionClrType(Type? clrType) if (innerType is null) return null; - elementMapping = FindComponentMapping(innerType); + elementMapping = FindComponentMapping(innerType, elementClrTypeHint); } // Fall back to the pre-resolved element type mapping from EF Core (used by @@ -414,7 +471,7 @@ private static bool IsCollectionClrType(Type? clrType) elementMapping ??= mappingInfo.ElementTypeMapping as RelationalTypeMapping; var clrType = mappingInfo.ClrType; - var elementClrType = GetCollectionElementType(clrType); + var elementClrType = elementClrTypeHint; // Resolve element mapping from CLR type if not already resolved if (elementMapping is null && elementClrType is not null) @@ -461,14 +518,34 @@ private static bool IsCollectionClrType(Type? clrType) /// /// EF Core's scalar nullability lives on , /// which is why strips Nullable(...) and - /// FindMapping returns the unwrapped scalar mapping — correct for scalar columns - /// where the property/column annotation carries the nullability separately, but - /// insufficient for composites whose element nullability has no annotation channel. + /// FindMapping returns the unwrapped scalar mapping. That is correct for a scalar column, + /// where the property annotation carries nullability separately, but a composite needs the + /// element nullability in the element mapping's CLR type. + /// + /// Note that EF Core does model this for a primitive collection, on + /// , which this + /// resolver does not yet consult. It has no equivalent for a Map value or one Tuple + /// position, so the store type stays the only channel for those. + /// /// /// - private RelationalTypeMapping? FindComponentMapping(string innerStoreType) + /// + /// The component's CLR type, where the model supplies one. Several CLR types share a single + /// ClickHouse store type — DateTime64 serves both and + /// , and Date32 serves both and + /// — so resolving from the store type alone would always pick the + /// default CLR type and give the composite the wrong element type. + /// + private RelationalTypeMapping? FindComponentMapping(string innerStoreType, Type? clrTypeHint = null) { - var inner = FindMapping(innerStoreType); + // Element nullability rides on the store type, so strip Nullable<> from the hint and let + // the wrapper below re-apply it. + var hint = clrTypeHint is null ? null : Nullable.GetUnderlyingType(clrTypeHint) ?? clrTypeHint; + + var inner = hint is null + ? FindMapping(innerStoreType) + : FindMapping(hint, innerStoreType) ?? FindMapping(innerStoreType); + if (inner is null) return null; @@ -546,8 +623,14 @@ private static bool HasNullableElementWrapper(string storeType) if (innerTypes is null) return null; - var keyMapping = FindComponentMapping(innerTypes[0]); - var valueMapping = FindComponentMapping(innerTypes[1]); + // Dictionary supplies the component CLR types when the model has one. + var dictionaryArgs = mappingInfo.ClrType is { IsGenericType: true } dictType + && dictType.GetGenericTypeDefinition() == typeof(Dictionary<,>) + ? dictType.GetGenericArguments() + : null; + + var keyMapping = FindComponentMapping(innerTypes[0], dictionaryArgs?[0]); + var valueMapping = FindComponentMapping(innerTypes[1], dictionaryArgs?[1]); if (keyMapping is null || valueMapping is null) return null; @@ -581,10 +664,18 @@ private static bool HasNullableElementWrapper(string storeType) if (innerTypes is null || innerTypes.Count == 0) return null; + // A tuple CLR type supplies the component CLR types, provided the arity agrees. + var tupleArgs = mappingInfo.ClrType is { IsGenericType: true } tupleType + && ClassifyTupleType(tupleType).IsTuple + && tupleType.GetGenericArguments() is { } args + && args.Length == innerTypes.Count + ? args + : null; + var elementMappings = new List(); - foreach (var innerType in innerTypes) + for (var i = 0; i < innerTypes.Count; i++) { - var mapping = FindComponentMapping(innerType); + var mapping = FindComponentMapping(innerTypes[i], tupleArgs?[i]); if (mapping is null) return null; elementMappings.Add(mapping); diff --git a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseArrayTypeMapping.cs b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseArrayTypeMapping.cs index 99d8294..84c2172 100644 --- a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseArrayTypeMapping.cs +++ b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseArrayTypeMapping.cs @@ -14,6 +14,9 @@ public class ClickHouseArrayTypeMapping : RelationalTypeMapping private static readonly MethodInfo GetValueMethod = typeof(DbDataReader).GetRuntimeMethod(nameof(DbDataReader.GetValue), [typeof(int)])!; + private static readonly MethodInfo ConvertArrayMethod = + typeof(ClickHouseArrayTypeMapping).GetMethod(nameof(ConvertArray), BindingFlags.Static | BindingFlags.NonPublic)!; + public RelationalTypeMapping ElementMapping { get; } /// @@ -81,7 +84,49 @@ public override Expression CustomizeDataReaderExpression(Expression expression) // When there's a ValueConverter (e.g. List ↔ T[]), the data reader must produce // the provider type (T[]). EF Core applies the converter afterward. var targetType = Converter?.ProviderClrType ?? ClrType; - return Expression.Convert(expression, targetType); + + // An element whose CLR type differs from what the driver produces (DateTimeOffset and + // DateOnly both arrive as DateTime) needs the array rebuilt element by element. Casting + // the whole array would throw InvalidCastException. Otherwise cast directly, which is + // both correct and cheaper. + if (!ClickHouseComponentConversion.NeedsConversion(ElementMapping)) + return Expression.Convert(expression, targetType); + + var elementType = ElementMapping.ClrType; + Expression converted = Expression.Call( + ConvertArrayMethod.MakeGenericMethod(elementType), + expression, + ClickHouseComponentConversion.CreateConverter(ElementMapping, elementType), + Expression.Constant(ClickHouseComponentConversion.CanPassThrough(ElementMapping))); + + return converted.Type == targetType ? converted : Expression.Convert(converted, targetType); + } + + /// + /// Rebuilds the driver's array as TElement[], converting each element. Nested composites + /// compose through this: an Array(Array(DateTime64)) element mapping is itself a + /// , so its own conversion runs per element. + /// + private static TElement[] ConvertArray( + object value, + Func convertElement, + bool canPassThrough) + { + // The driver often already produces the target type, for example Array(Int32) -> int[]. + // See ClickHouseComponentConversion.CanPassThrough for when that proves there is no work + // left to do. + if (canPassThrough && value is TElement[] alreadyTyped) + return alreadyTyped; + + var source = (Array)value; + var result = new TElement[source.Length]; + for (var i = 0; i < source.Length; i++) + { + var element = source.GetValue(i); + result[i] = element is null or DBNull ? default! : convertElement(element); + } + + return result; } protected override string GenerateNonNullSqlLiteral(object value) diff --git a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseComponentConversion.cs b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseComponentConversion.cs new file mode 100644 index 0000000..4e62a37 --- /dev/null +++ b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseComponentConversion.cs @@ -0,0 +1,156 @@ +using System.Collections.Concurrent; +using System.Linq.Expressions; +using Microsoft.EntityFrameworkCore.Storage; + +namespace ClickHouse.EntityFrameworkCore.Storage.Internal.Mapping; + +/// +/// Lets a composite mapping (Array, Map, Tuple) apply its component mappings' own read conversions +/// to each component of a value that the driver returned. +/// +/// +/// +/// A composite mapping reads its whole column through GetValue, so the component mappings' +/// read pipeline never runs. Where a component's CLR type differs from the type the driver produces, +/// casting the whole composite throws . The composite must instead +/// be rebuilt component by component, reusing each component mapping's existing conversion rather +/// than repeating it here. +/// +/// +/// A component read has the same two steps EF Core applies to a scalar column: +/// +/// +/// turns the raw driver value +/// into the provider CLR type — this is where and +/// are built from the the driver returns. +/// +/// +/// The mapping's turns the provider CLR type into +/// the model CLR type — this is where an Enum8 component becomes a C# enum and an +/// Array(T) component becomes a List<T>. +/// +/// +/// Both steps are needed. Applying only the first would leave a component mapping that carries a +/// converter readable as its provider type but not as the type the property declares. +/// +/// +internal static class ClickHouseComponentConversion +{ + // Keyed by (mapping, target type). The compiled converter is embedded in the materializer as a + // constant, so it is built once per mapping rather than once per row. + private static readonly ConcurrentDictionary<(RelationalTypeMapping Mapping, Type Target), Delegate> ConverterCache = new(); + + private static readonly ConcurrentDictionary NeedsConversionCache = new(); + + /// + /// Reports whether reading changes the value the driver produced. + /// + /// + /// Cached, because a nested composite's answer depends on its own components' answers and the + /// probe below builds a throw-away sub-tree to find out. + /// + public static bool NeedsConversion(RelationalTypeMapping mapping) + => NeedsConversionCache.GetOrAdd(mapping, static m => + { + // A value converter always changes the value. + if (m.Converter is not null) + return true; + + // CustomizeDataReaderExpression returns its argument unchanged when a mapping needs no + // conversion, so reference equality against a probe is a reliable test. + var probe = Expression.Parameter(typeof(object), "component"); + return !ReferenceEquals(m.CustomizeDataReaderExpression(probe), probe); + }); + + /// + /// Reports whether a composite that the driver already produced at the target CLR type may be + /// returned unchanged, instead of being rebuilt component by component. + /// + /// + /// + /// Rebuilding is only needed where reading a component changes its value. The composite + /// mappings therefore keep a fast path for a driver value that already has the target type — + /// which earns its keep for a nested composite such as Array(Array(Int32)), where the + /// inner mapping's read is a cast and nothing more. + /// + /// + /// That fast path is only sound where matching CLR types prove there is nothing left to do. + /// It holds for : every + /// component mapping that uses it either changes the CLR type — + /// and are both built from a — or coerces a + /// numeric type the driver may return too wide, which is a no-op once the type already matches. + /// It does not hold for a , which is free to + /// change the value while keeping the CLR type. So a component that carries one is always + /// rebuilt. + /// + /// + public static bool CanPassThrough(RelationalTypeMapping mapping) + => mapping.Converter is null; + + /// + /// Returns an expression of type Func<object, TComponent> that reads one component, + /// where TComponent is . + /// + /// + /// The converter is compiled once per mapping and embedded as a constant. An inline + /// would instead be rebuilt on + /// every materialization, allocating a delegate per row for every composite column. The trade-off + /// is that a constant holding a delegate cannot be quoted, so composite columns whose components + /// convert are not usable from precompiled queries. + /// + public static Expression CreateConverter(RelationalTypeMapping mapping, Type componentType) + { + var converter = ConverterCache.GetOrAdd( + (mapping, componentType), + static key => Compile(key.Mapping, key.Target)); + + return Expression.Constant(converter, typeof(Func<,>).MakeGenericType(typeof(object), componentType)); + } + + private static Delegate Compile(RelationalTypeMapping mapping, Type componentType) + { + var parameter = Expression.Parameter(typeof(object), "component"); + var body = mapping.CustomizeDataReaderExpression(parameter); + + if (mapping.Converter is { } valueConverter) + { + // Step 1 produces the provider CLR type, which is what the converter accepts. + body = Coerce(body, valueConverter.ProviderClrType, mapping, componentType); + body = Expression.Invoke(valueConverter.ConvertFromProviderExpression, body); + } + + body = Coerce(body, componentType, mapping, componentType); + + return Expression.Lambda( + typeof(Func<,>).MakeGenericType(typeof(object), componentType), + body, + parameter) + .Compile(); + } + + private static Expression Coerce( + Expression expression, + Type targetType, + RelationalTypeMapping mapping, + Type componentType) + { + if (expression.Type == targetType) + return expression; + + try + { + return Expression.Convert(expression, targetType); + } + catch (InvalidOperationException ex) + { + // Without this the user sees EF Core's bare "No coercion operator is defined between + // types ...", which names two CLR types they never wrote. + throw new NotSupportedException( + $"The ClickHouse provider cannot read a component of store type '{mapping.StoreType}' " + + $"as '{componentType}'. Reading it produces '{expression.Type}', and no conversion " + + $"to '{targetType}' exists. Change the property type to match the column, or set an " + + $"explicit column type with HasColumnType(...).", + ex); + } + } +} diff --git a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseDateTimeOffsetTypeMapping.cs b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseDateTimeOffsetTypeMapping.cs new file mode 100644 index 0000000..4005279 --- /dev/null +++ b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseDateTimeOffsetTypeMapping.cs @@ -0,0 +1,297 @@ +using System.Collections.Concurrent; +using System.Data.Common; +using System.Globalization; +using System.Linq; +using System.Linq.Expressions; +using System.Reflection; +using System.Text.RegularExpressions; +using Microsoft.EntityFrameworkCore.Storage; + +namespace ClickHouse.EntityFrameworkCore.Storage.Internal.Mapping; + +/// +/// Maps to DateTime64 (or DateTime). +/// +/// ClickHouse has no type that stores a UTC offset: DateTime64 holds an instant, and any +/// declared timezone only decides how that instant is rendered. So the instant is preserved and +/// the original offset is not. A value read back carries the offset of the column's declared +/// timezone, which is +00:00 for the default store type. +/// +/// The default store type pins the timezone to 'UTC' on purpose. For a timezone-less +/// parameter type such as DateTime64(7), the driver sends a UTC wall clock and the server +/// then reads it in session_timezone, which moves the instant when that setting is not UTC. +/// +/// No value converter is used. The driver accepts a directly on both +/// the query parameter path and the bulk insert path, and converts it to the correct instant. +/// +/// A column may declare a fixed UTC offset rather than a named zone, which ClickHouse spells +/// Fixed/UTC±HH:MM:SS. reads those, because .NET has no +/// timezone of that name. +/// +/// One limit applies to a column that declares a timezone with daylight saving. The driver gives a +/// wall clock in that timezone and drops the offset, so the repeated hour when clocks go back is +/// ambiguous. recovers it when the zone's standard offset is zero, for +/// example Europe/London. In a zone where both candidate offsets are not zero, such as +/// Europe/Paris, the reading falls back to standard time and can be one hour early. The default +/// 'UTC' store type has no daylight saving and is not affected, and neither is a fixed +/// offset, which by definition never changes. +/// +public class ClickHouseDateTimeOffsetTypeMapping : RelationalTypeMapping +{ + /// + /// One .NET tick is 100 ns, which is precision 7. This makes the round trip exact, so a + /// stored value never comes back truncated. + /// + public const int DefaultPrecision = 7; + + public const string DefaultTimezone = "UTC"; + + /// .NET cannot render more than 7 fractional digits, because a tick is its smallest unit. + private const int MaxFractionalDigits = 7; + + /// + /// ClickHouse spells a fixed-offset timezone Fixed/UTC±HH:MM:SS. See + /// for why the pattern is this strict. + /// + private static readonly Regex FixedOffsetRegex = new( + @"^Fixed/UTC([+-])(\d{2}):(\d{2}):(\d{2})$", + RegexOptions.Compiled | RegexOptions.CultureInvariant); + + // DateTimeOffset holds an offset only within ±14 hours, and only in whole minutes. ClickHouse + // accepts both a larger magnitude and a finer granularity, for example 'Fixed/UTC+00:00:42'. + private static readonly TimeSpan MaxRepresentableOffset = TimeSpan.FromHours(14); + private static readonly TimeSpan MinRepresentableOffset = TimeSpan.FromHours(-14); + + private static readonly MethodInfo GetValueMethod = + typeof(DbDataReader).GetRuntimeMethod(nameof(DbDataReader.GetValue), [typeof(int)])!; + + private static readonly MethodInfo ConvertToDateTimeOffsetMethod = + typeof(ClickHouseDateTimeOffsetTypeMapping).GetMethod( + nameof(ConvertToDateTimeOffset), + BindingFlags.Public | BindingFlags.Static)!; + + // Resolved per row during materialization, so the lookup is cached. + private static readonly ConcurrentDictionary TimeZoneCache = new(); + + /// + /// The timezone declared by the store type, or when the store type + /// declares none. A timezone-less column is read as a UTC wall clock by the driver. + /// + public string? Timezone { get; } + + public ClickHouseDateTimeOffsetTypeMapping() + : this(DefaultPrecision, DefaultTimezone) + { + } + + /// + /// The DateTime64 precision, or for the second-precision + /// DateTime store type. + /// + /// The declared timezone, or for none. + public ClickHouseDateTimeOffsetTypeMapping(int? precision, string? timezone) + : base( + new RelationalTypeMappingParameters( + new CoreTypeMappingParameters(typeof(DateTimeOffset)), + FormatStoreType(precision, timezone), + StoreTypePostfix.None, + System.Data.DbType.DateTimeOffset, + precision: precision)) + { + Timezone = timezone; + } + + protected ClickHouseDateTimeOffsetTypeMapping(RelationalTypeMappingParameters parameters, string? timezone) + : base(parameters) + { + Timezone = timezone; + } + + protected override RelationalTypeMapping Clone(RelationalTypeMappingParameters parameters) + => new ClickHouseDateTimeOffsetTypeMapping(parameters, Timezone); + + // The driver returns DateTime for DateTime64 columns, never DateTimeOffset — + // GetFieldValue throws InvalidCastException. Read the raw value + // and attach the offset of the column's declared timezone. + public override MethodInfo GetDataReaderMethod() + => GetValueMethod; + + public override Expression CustomizeDataReaderExpression(Expression expression) + => Expression.Call( + ConvertToDateTimeOffsetMethod, + expression, + Expression.Constant(Timezone, typeof(string))); + + /// + /// Converts a value read from a ClickHouse date/time column into a . + /// + /// + /// The driver gives when the column's timezone is offset zero at + /// that instant, and a wall clock in the column's + /// timezone in all other cases. A column with no declared timezone is read as a UTC wall clock. + /// + public static DateTimeOffset ConvertToDateTimeOffset(object value, string? timezone) + { + if (value is DateTimeOffset dateTimeOffset) + return dateTimeOffset; + + var dateTime = (DateTime)value; + + // The driver already resolved the instant for us. + if (dateTime.Kind == DateTimeKind.Utc) + return new DateTimeOffset(dateTime); + + // The column declares no timezone, so the driver's wall clock is already UTC. + if (timezone is null) + return new DateTimeOffset(DateTime.SpecifyKind(dateTime, DateTimeKind.Utc)); + + // A fixed offset resolves without the host's timezone data. This must come before the + // lookup below, which cannot resolve such a name. + if (TryParseFixedOffset(timezone, out var fixedOffset)) + return new DateTimeOffset(DateTime.SpecifyKind(dateTime, DateTimeKind.Unspecified), fixedOffset); + + var zone = FindTimeZone(timezone) + ?? throw new InvalidOperationException( + $"Cannot read the DateTimeOffset column because this machine does not know the " + + $"timezone '{timezone}' that the column declares. The driver gives a wall clock in " + + $"that timezone, so the offset cannot be found without it. Install the operating " + + $"system timezone data (the 'tzdata' package on a minimal Linux image), or declare " + + $"the column as DateTime64(P, 'UTC')."); + + var wallClock = DateTime.SpecifyKind(dateTime, DateTimeKind.Unspecified); + return new DateTimeOffset(wallClock, ResolveOffset(zone, wallClock)); + } + + /// + /// Reads a ClickHouse fixed-offset timezone name into its offset. + /// + /// + /// + /// ClickHouse lets a column declare a fixed UTC offset instead of a named zone, and spells it + /// Fixed/UTC±HH:MM:SS — for example DateTime64(7, 'Fixed/UTC+05:30:00'). Each + /// field must have exactly two digits, and the server rejects Fixed/UTC+5:30:00, + /// Fixed/UTC+05:30 and any change of case. Such a name is not in the IANA database, so + /// cannot resolve it however complete the + /// host's timezone data is. It needs no daylight-saving logic either, because the offset is + /// fixed by definition, so the reading is never ambiguous. + /// + /// + /// The minutes and seconds fields are not held to 59. ClickHouse carries the excess, so + /// Fixed/UTC+05:60:00 is a legal name for the offset +06:00, and the server + /// accepts any name up to a total of 24 hours. The whole shape is matched here so that such a + /// name is diagnosed rather than left to the unresolvable-timezone error, but the offset is + /// only returned for the spelling the driver also reads. See the throw below. + /// + /// + private static bool TryParseFixedOffset(string timezone, out TimeSpan offset) + { + offset = default; + + var match = FixedOffsetRegex.Match(timezone); + if (!match.Success) + return false; + + var magnitude = new TimeSpan( + int.Parse(match.Groups[2].Value, CultureInfo.InvariantCulture), + int.Parse(match.Groups[3].Value, CultureInfo.InvariantCulture), + int.Parse(match.Groups[4].Value, CultureInfo.InvariantCulture)); + + var sign = match.Groups[1].Value == "-" ? -1 : 1; + var candidate = sign * magnitude; + + // ClickHouse accepts offsets that DateTimeOffset cannot hold. Its constructor would throw + // an ArgumentException naming only the rule, so report the timezone that broke it instead. + var limit = candidate < MinRepresentableOffset || candidate > MaxRepresentableOffset + ? "DateTimeOffset holds an offset only within plus or minus 14 hours" + : candidate.Ticks % TimeSpan.TicksPerMinute != 0 + ? "DateTimeOffset holds an offset only in whole minutes" + : null; + + if (limit is not null) + { + throw new InvalidOperationException( + $"Cannot read the DateTimeOffset column because its declared timezone '{timezone}' " + + $"is an offset of {candidate}, and {limit}. Declare the column with an offset that " + + $"a DateTimeOffset can hold, or map the property as DateTime."); + } + + // ClickHouse carries minutes and seconds above 59, so 'Fixed/UTC+05:60:00' is a legal name + // for the offset +06:00. The driver does not read those, and returns a UTC wall clock + // instead of one in the column's timezone, so the offset here cannot be attached to it — + // that would move the instant by the whole offset and report nothing. Only the spelling the + // driver agrees with can be read. + if (magnitude.Minutes != int.Parse(match.Groups[3].Value, CultureInfo.InvariantCulture) + || magnitude.Seconds != int.Parse(match.Groups[4].Value, CultureInfo.InvariantCulture)) + { + throw new InvalidOperationException( + $"Cannot read the DateTimeOffset column because the ClickHouse driver does not " + + $"support the timezone '{timezone}' that the column declares. ClickHouse reads it " + + $"as the offset {candidate}, but only spells that offset in a form the driver " + + $"accepts when the minutes and seconds are below 60. Declare the column as " + + $"DateTime64(P, '{FormatFixedOffset(candidate)}') instead."); + } + + offset = candidate; + return true; + } + + /// Spells an offset the way ClickHouse names a fixed-offset timezone. + private static string FormatFixedOffset(TimeSpan offset) + => string.Create( + CultureInfo.InvariantCulture, + $"Fixed/UTC{(offset < TimeSpan.Zero ? '-' : '+')}{offset.Duration():hh\\:mm\\:ss}"); + + private static TimeSpan ResolveOffset(TimeZoneInfo zone, DateTime wallClock) + { + // GetUtcOffset reads an Unspecified value as a local time in the given zone. + if (!zone.IsAmbiguousTime(wallClock)) + return zone.GetUtcOffset(wallClock); + + // An ambiguous wall clock — the hour that repeats when clocks go back — has two candidate + // offsets, and GetUtcOffset would pick the standard-time one. We know more than it does: + // the driver only gives Kind=Unspecified when the true offset is not zero, so a zero + // candidate can be discarded. That recovers the exact instant for every zone whose + // standard offset is zero, such as Europe/London. + // + // Where both candidates are non-zero (Europe/Paris, America/New_York) the offset the + // driver dropped cannot be recovered, so keep the standard-time reading. + var standardOffset = zone.GetUtcOffset(wallClock); + if (standardOffset != TimeSpan.Zero) + return standardOffset; + + return zone.GetAmbiguousTimeOffsets(wallClock) + .FirstOrDefault(candidate => candidate != TimeSpan.Zero, standardOffset); + } + + private static TimeZoneInfo? FindTimeZone(string timezone) + => TimeZoneCache.GetOrAdd(timezone, static id => + { + try + { + return TimeZoneInfo.FindSystemTimeZoneById(id); + } + catch (Exception ex) when (ex is TimeZoneNotFoundException or InvalidTimeZoneException) + { + return null; + } + }); + + // An ISO-8601 literal that carries the offset is instant-exact whatever timezone the target + // column declares. A bare wall clock is not: the server reads it in the column's timezone. + protected override string GenerateNonNullSqlLiteral(object value) + { + var dateTimeOffset = (DateTimeOffset)value; + var digits = Math.Min(Precision ?? 0, MaxFractionalDigits); + var fraction = digits == 0 ? string.Empty : "." + new string('f', digits); + return $"'{dateTimeOffset.ToString($"yyyy-MM-dd HH:mm:ss{fraction}zzz", CultureInfo.InvariantCulture)}'"; + } + + private static string FormatStoreType(int? precision, string? timezone) + => (precision, timezone) switch + { + (null, null) => "DateTime", + (null, _) => $"DateTime('{timezone}')", + (_, null) => $"DateTime64({precision})", + _ => $"DateTime64({precision}, '{timezone}')" + }; +} diff --git a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseMapTypeMapping.cs b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseMapTypeMapping.cs index 980b3a3..c7a2c92 100644 --- a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseMapTypeMapping.cs +++ b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseMapTypeMapping.cs @@ -13,6 +13,9 @@ public class ClickHouseMapTypeMapping : RelationalTypeMapping private static readonly MethodInfo GetValueMethod = typeof(DbDataReader).GetRuntimeMethod(nameof(DbDataReader.GetValue), [typeof(int)])!; + private static readonly MethodInfo ConvertMapMethod = + typeof(ClickHouseMapTypeMapping).GetMethod(nameof(ConvertMap), BindingFlags.Static | BindingFlags.NonPublic)!; + public RelationalTypeMapping KeyMapping { get; } public RelationalTypeMapping ValueMapping { get; } @@ -46,7 +49,50 @@ public override MethodInfo GetDataReaderMethod() => GetValueMethod; public override Expression CustomizeDataReaderExpression(Expression expression) - => Expression.Convert(expression, ClrType); + { + // A key or value whose CLR type differs from what the driver produces (DateTimeOffset and + // DateOnly both arrive as DateTime) needs the dictionary rebuilt entry by entry. Casting + // the whole dictionary would throw InvalidCastException. + if (!ClickHouseComponentConversion.NeedsConversion(KeyMapping) + && !ClickHouseComponentConversion.NeedsConversion(ValueMapping)) + { + return Expression.Convert(expression, ClrType); + } + + Expression converted = Expression.Call( + ConvertMapMethod.MakeGenericMethod(KeyMapping.ClrType, ValueMapping.ClrType), + expression, + ClickHouseComponentConversion.CreateConverter(KeyMapping, KeyMapping.ClrType), + ClickHouseComponentConversion.CreateConverter(ValueMapping, ValueMapping.ClrType), + Expression.Constant( + ClickHouseComponentConversion.CanPassThrough(KeyMapping) + && ClickHouseComponentConversion.CanPassThrough(ValueMapping))); + + return converted.Type == ClrType ? converted : Expression.Convert(converted, ClrType); + } + + private static Dictionary ConvertMap( + object value, + Func convertKey, + Func convertValue, + bool canPassThrough) + where TKey : notnull + { + // See ClickHouseComponentConversion.CanPassThrough for when a dictionary the driver already + // typed needs no rebuilding. + if (canPassThrough && value is Dictionary alreadyTyped) + return alreadyTyped; + + var source = (IDictionary)value; + var result = new Dictionary(source.Count); + foreach (DictionaryEntry entry in source) + { + result[convertKey(entry.Key)] = + entry.Value is null or DBNull ? default! : convertValue(entry.Value); + } + + return result; + } protected override string GenerateNonNullSqlLiteral(object value) { diff --git a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseNullableElementMapping.cs b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseNullableElementMapping.cs index 2ed5ffa..6c6e640 100644 --- a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseNullableElementMapping.cs +++ b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseNullableElementMapping.cs @@ -1,3 +1,4 @@ +using System.Linq.Expressions; using System.Reflection; using Microsoft.EntityFrameworkCore.Storage; @@ -14,10 +15,14 @@ namespace ClickHouse.EntityFrameworkCore.Storage.Internal.Mapping; /// so 's FindMapping strips /// Nullable(...) wrappers in ParseStoreTypeName and returns the unwrapped scalar /// mapping. That convention works for scalar columns but breaks composites: an -/// Array(Nullable(Int32)) property is int?[] at the CLR level, and there is no -/// per-element IsNullable annotation channel for the composite to consult. The only -/// way to surface element-level nullability is through the element mapping's -/// . +/// Array(Nullable(Int32)) property is int?[] at the CLR level, so the composite must +/// take element nullability from the element mapping's . +/// +/// For a primitive collection, EF Core does model this on +/// , and the +/// resolver should prefer that channel. It has no equivalent for a Map value or a single +/// Tuple position, so this wrapper stays necessary for those. +/// /// /// /// This wrapper exists for that single purpose: report Nullable<T> as the CLR @@ -60,7 +65,7 @@ private static RelationalTypeMappingParameters BuildParameters(RelationalTypeMap valueGeneratorFactory: null, elementMapping: inner.ElementTypeMapping, jsonValueReaderWriter: inner.JsonValueReaderWriter), - $"Nullable({inner.StoreType})", + FormatStoreType(inner.StoreType), inner.StoreTypePostfix, inner.DbType, inner.IsUnicode, @@ -70,10 +75,39 @@ private static RelationalTypeMappingParameters BuildParameters(RelationalTypeMap inner.Scale); } + /// + /// Adds the Nullable(...) wrapper unless the inner store type already carries one. + /// + /// + /// The inner mapping is resolved from the component store type, which still holds the + /// Nullable(...) text, and PreserveExplicitStoreType keeps that text verbatim. + /// Wrapping it again would give Nullable(Nullable(T)), which ClickHouse rejects with + /// Nested type Nullable(T) cannot be inside Nullable type. + /// + private static string FormatStoreType(string innerStoreType) + => DenotesNullable(innerStoreType) ? innerStoreType : $"Nullable({innerStoreType})"; + + private static bool DenotesNullable(string storeType) + { + var s = storeType.AsSpan().Trim(); + + // LowCardinality(Nullable(T)) is already nullable; the wrapper must not be added again. + if (s.StartsWith("LowCardinality(", StringComparison.OrdinalIgnoreCase) && s.EndsWith(")")) + s = s["LowCardinality(".Length..^1].Trim(); + + return s.StartsWith("Nullable(", StringComparison.OrdinalIgnoreCase); + } + protected override RelationalTypeMapping Clone(RelationalTypeMappingParameters parameters) => new ClickHouseNullableElementMapping(parameters, Inner); public override MethodInfo GetDataReaderMethod() => Inner.GetDataReaderMethod(); + // Delegate the read conversion as well, so a composite over Nullable(DateTime64) or + // Nullable(Date32) still converts each element. Callers handle the null case before this + // runs, so the inner non-nullable conversion is safe here. + public override Expression CustomizeDataReaderExpression(Expression expression) + => Inner.CustomizeDataReaderExpression(expression); + protected override string GenerateNonNullSqlLiteral(object value) => Inner.GenerateSqlLiteral(value); } diff --git a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseTupleTypeMapping.cs b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseTupleTypeMapping.cs index b91cd96..11b0c74 100644 --- a/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseTupleTypeMapping.cs +++ b/src/EFCore.ClickHouse/Storage/Internal/Mapping/ClickHouseTupleTypeMapping.cs @@ -14,10 +14,11 @@ public class ClickHouseTupleTypeMapping : RelationalTypeMapping typeof(DbDataReader).GetRuntimeMethod(nameof(DbDataReader.GetValue), [typeof(int)])!; private static readonly MethodInfo ConvertMethod = - typeof(ClickHouseTupleTypeMapping).GetMethod(nameof(ConvertToValueTuple), BindingFlags.Static | BindingFlags.NonPublic)!; + typeof(ClickHouseTupleTypeMapping).GetMethod(nameof(ConvertTuple), BindingFlags.Static | BindingFlags.NonPublic)!; - // Cache compiled constructors per ValueTuple type to avoid Activator.CreateInstance per row - private static readonly ConcurrentDictionary ConstructorCache = new(); + // Cache the compiled constructor and its component types per tuple type, to avoid + // Activator.CreateInstance and reflection per row. + private static readonly ConcurrentDictionary ConstructorCache = new(); public IReadOnlyList ElementMappings { get; } @@ -48,48 +49,93 @@ public override MethodInfo GetDataReaderMethod() public override Expression CustomizeDataReaderExpression(Expression expression) { - // The driver returns System.Tuple<>, but C# value tuples are ValueTuple<>. - // Use a conversion helper that handles both cases. - if (ClrType.IsValueType) - return Expression.Call(ConvertMethod.MakeGenericMethod(ClrType), expression); - - return Expression.Convert(expression, ClrType); + // Two reasons to rebuild: the driver returns System.Tuple<> even where the CLR type is a + // ValueTuple<>, and a component whose CLR type differs from what the driver produces + // (DateTimeOffset and DateOnly both arrive as DateTime) cannot be cast in place. + var needsComponentConversion = ElementMappings.Any(ClickHouseComponentConversion.NeedsConversion); + + if (!ClrType.IsValueType && !needsComponentConversion) + return Expression.Convert(expression, ClrType); + + var componentConverters = Expression.NewArrayInit( + typeof(Func), + ElementMappings.Select( + mapping => (Expression)ClickHouseComponentConversion.CreateConverter(mapping, typeof(object)))); + + return Expression.Call( + ConvertMethod.MakeGenericMethod(ClrType), + expression, + componentConverters, + Expression.Constant(ElementMappings.All(ClickHouseComponentConversion.CanPassThrough))); } - // Converts the driver's Tuple<> to ValueTuple<> (or passes through if already correct type) - private static T ConvertToValueTuple(object value) where T : struct + // Rebuilds the driver's tuple as T, converting each component. Handles both ValueTuple<> and + // System.Tuple<> targets, since both expose a constructor taking every component. + private static T ConvertTuple( + object value, + Func[] convertComponents, + bool canPassThrough) { - if (value is T t) - return t; + // A ValueTuple target never takes this path, because the driver returns System.Tuple<>. + // See ClickHouseComponentConversion.CanPassThrough for the component condition. + if (canPassThrough && value is T alreadyTyped) + return alreadyTyped; + + if (value is not ITuple tuple) + throw new InvalidCastException($"Cannot convert {value.GetType()} to {typeof(T)}"); + + if (tuple.Length != convertComponents.Length) + throw new InvalidCastException( + $"Cannot convert {value.GetType()} to {typeof(T)}: the value has {tuple.Length} " + + $"components but the mapping expects {convertComponents.Length}."); - // Driver returns System.Tuple<>, need to create ValueTuple<> from its elements - if (value is ITuple tuple) + var (factory, componentTypes) = ConstructorCache.GetOrAdd(typeof(T), static type => { - var args = new object?[tuple.Length]; - for (var i = 0; i < tuple.Length; i++) - args[i] = tuple[i]; + var constructor = type.GetConstructors()[0]; + var ctorParams = constructor.GetParameters(); + var argsParam = Expression.Parameter(typeof(object[]), "args"); + var bodyArgs = new Expression[ctorParams.Length]; - var factory = ConstructorCache.GetOrAdd(typeof(T), static type => + for (var j = 0; j < ctorParams.Length; j++) { - var ctorParams = type.GetConstructors()[0].GetParameters(); - var argsParam = Expression.Parameter(typeof(object[]), "args"); - var bodyArgs = new Expression[ctorParams.Length]; + bodyArgs[j] = Expression.Convert( + Expression.ArrayIndex(argsParam, Expression.Constant(j)), + ctorParams[j].ParameterType); + } + + return ( + (Delegate)Expression.Lambda>( + Expression.New(constructor, bodyArgs), argsParam).Compile(), + Array.ConvertAll(ctorParams, p => p.ParameterType)); + }); - for (var j = 0; j < ctorParams.Length; j++) + var args = new object?[tuple.Length]; + for (var i = 0; i < tuple.Length; i++) + { + var component = tuple[i]; + if (component is null or DBNull) + { + // The array and map helpers substitute default(T) for a null component. A tuple slot + // cannot: the constructor takes it positionally, so a null on a non-nullable + // value-type slot would fail inside the compiled factory as a NullReferenceException. + // ClickHouse only returns NULL for a Nullable(...) slot, so reaching this means the + // column and the CLR tuple disagree — say so plainly. + if (componentTypes[i].IsValueType && Nullable.GetUnderlyingType(componentTypes[i]) is null) { - bodyArgs[j] = Expression.Convert( - Expression.ArrayIndex(argsParam, Expression.Constant(j)), - ctorParams[j].ParameterType); + throw new InvalidCastException( + $"Cannot convert {value.GetType()} to {typeof(T)}: component {i} is NULL but " + + $"'{componentTypes[i]}' is not nullable. Declare that tuple component as " + + $"'{componentTypes[i]}?', or make the column non-nullable."); } - var body = Expression.New(type.GetConstructors()[0], bodyArgs); - return Expression.Lambda>(body, argsParam).Compile(); - }); + args[i] = null; + continue; + } - return ((Func)factory)(args!); + args[i] = convertComponents[i](component); } - throw new InvalidCastException($"Cannot convert {value.GetType()} to {typeof(T)}"); + return ((Func)factory)(args!); } protected override string GenerateNonNullSqlLiteral(object value) diff --git a/test/EFCore.ClickHouse.Tests/CompositeElementConversionTests.cs b/test/EFCore.ClickHouse.Tests/CompositeElementConversionTests.cs new file mode 100644 index 0000000..2901c12 --- /dev/null +++ b/test/EFCore.ClickHouse.Tests/CompositeElementConversionTests.cs @@ -0,0 +1,663 @@ +using System.Linq.Expressions; +using ClickHouse.EntityFrameworkCore.Storage.Internal.Mapping; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Storage; +using Microsoft.EntityFrameworkCore.Storage.ValueConversion; +using Xunit; + +namespace EFCore.ClickHouse.Tests; + +/// +/// A composite mapping reads its whole column with GetValue, so the component mappings' +/// read conversions have to be applied per component. Both and +/// arrive from the driver as , so without that the +/// whole composite fails to cast. +/// +public class CompositeElementEntity +{ + public long Id { get; set; } + public DateTimeOffset[] Offsets { get; set; } = []; + public DateOnly[] Dates { get; set; } = []; + public List OffsetList { get; set; } = []; + public DateTimeOffset[][] NestedOffsets { get; set; } = []; + public Dictionary OffsetsByName { get; set; } = []; + public Dictionary NamesByDate { get; set; } = []; + public Tuple? OffsetTuple { get; set; } + public (DateTimeOffset When, int Count) OffsetValueTuple { get; set; } + public int[] Ints { get; set; } = []; + public Dictionary Counts { get; set; } = []; +} + +public class NullableCompositeElementEntity +{ + public long Id { get; set; } + public DateTimeOffset?[] Offsets { get; set; } = []; + public DateOnly?[] Dates { get; set; } = []; +} + +public enum CompositeColour +{ + Red, + Green, + Blue +} + +/// +/// Components that carry a ValueConverter rather than a data-reader conversion: an enum +/// converts through EnumToStringConverter, and a List<T> component through +/// ListToArrayConverter. The composite has to apply both, or the column becomes writable but +/// unreadable. +/// +public class ConvertedCompositeEntity +{ + public long Id { get; set; } + public CompositeColour[] Colours { get; set; } = []; + public Tuple? ColourTuple { get; set; } + public Dictionary ColourByName { get; set; } = []; + public Dictionary> Buckets { get; set; } = []; + public List> Nested { get; set; } = []; + public List[] ListArray { get; set; } = []; + public IList Interfaced { get; set; } = new List(); + public IReadOnlyList ReadOnlyDates { get; set; } = []; + + // Components that need no conversion at all — these must keep the direct cast. + public string[] Names { get; set; } = []; + public double[] Ratios { get; set; } = []; + public Dictionary Labels { get; set; } = []; +} + +public class CompositeElementDbContext : DbContext +{ + private readonly string _connectionString; + + public CompositeElementDbContext(string connectionString) => _connectionString = connectionString; + + public DbSet Entities => Set(); + public DbSet NullableEntities => Set(); + public DbSet ConvertedEntities => Set(); + + protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder) + => optionsBuilder.UseClickHouse(_connectionString); + + protected override void OnModelCreating(ModelBuilder modelBuilder) + { + modelBuilder.Entity(e => + { + e.ToTable("composite_elements"); + e.HasKey(x => x.Id); + e.Property(x => x.Id).HasColumnName("id"); + e.Property(x => x.Offsets).HasColumnName("offsets"); + e.Property(x => x.Dates).HasColumnName("dates"); + e.Property(x => x.OffsetList).HasColumnName("offset_list"); + e.Property(x => x.NestedOffsets).HasColumnName("nested_offsets"); + e.Property(x => x.OffsetsByName).HasColumnName("offsets_by_name"); + e.Property(x => x.NamesByDate).HasColumnName("names_by_date"); + e.Property(x => x.OffsetTuple).HasColumnName("offset_tuple"); + e.Property(x => x.OffsetValueTuple).HasColumnName("offset_value_tuple"); + e.Property(x => x.Ints).HasColumnName("ints"); + e.Property(x => x.Counts).HasColumnName("counts"); + }); + + modelBuilder.Entity(e => + { + e.ToTable("nullable_composite_elements"); + e.HasKey(x => x.Id); + e.Property(x => x.Id).HasColumnName("id"); + // The resolver takes element nullability from the store type. EF Core also models it on + // IElementType.IsNullable for a primitive collection, which the resolver does not use yet. + e.Property(x => x.Offsets).HasColumnName("offsets") + .HasColumnType("Array(Nullable(DateTime64(7, 'UTC')))"); + e.Property(x => x.Dates).HasColumnName("dates") + .HasColumnType("Array(Nullable(Date32))"); + }); + + modelBuilder.Entity(e => + { + e.ToTable("converted_composites"); + e.HasKey(x => x.Id); + e.Property(x => x.Id).HasColumnName("id"); + e.Property(x => x.Colours).HasColumnName("colours") + .HasColumnType("Array(Enum8('Red' = 1, 'Green' = 2, 'Blue' = 3))"); + e.Property(x => x.ColourTuple).HasColumnName("colour_tuple") + .HasColumnType("Tuple(Enum8('Red' = 1, 'Green' = 2, 'Blue' = 3), Int32)"); + e.Property(x => x.ColourByName).HasColumnName("colour_by_name") + .HasColumnType("Map(String, Enum8('Red' = 1, 'Green' = 2, 'Blue' = 3))"); + e.Property(x => x.Buckets).HasColumnName("buckets"); + e.Property(x => x.Nested).HasColumnName("nested"); + e.Property(x => x.ListArray).HasColumnName("list_array"); + e.Property(x => x.Interfaced).HasColumnName("interfaced"); + e.Property(x => x.ReadOnlyDates).HasColumnName("readonly_dates"); + e.Property(x => x.Names).HasColumnName("names"); + e.Property(x => x.Ratios).HasColumnName("ratios"); + e.Property(x => x.Labels).HasColumnName("labels"); + }); + } +} + +public class ElementConverterEntity +{ + public long Id { get; set; } + public string[] Tags { get; set; } = []; +} + +/// +/// An element converter that keeps the CLR type, set through EF Core's public +/// ElementType().HasConversion(...) API. +/// +public class ElementConverterDbContext : DbContext +{ + private readonly string _connectionString; + + public ElementConverterDbContext(string connectionString) => _connectionString = connectionString; + + public DbSet Entities => Set(); + + protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder) + => optionsBuilder.UseClickHouse(_connectionString); + + protected override void OnModelCreating(ModelBuilder modelBuilder) + => modelBuilder.Entity(e => + { + e.ToTable("element_converted"); + e.HasKey(x => x.Id); + e.Property(x => x.Id).HasColumnName("id"); + e.PrimitiveCollection(x => x.Tags).HasColumnName("tags") + .ElementType(el => el.HasConversion( + new ValueConverter(v => v, v => v + "!"))); + }); +} + +public class CompositeElementConversionFixture : IAsyncLifetime +{ + public string ConnectionString { get; private set; } = string.Empty; + + public async Task InitializeAsync() + { + ConnectionString = await SharedContainer.GetConnectionStringAsync(); + using var ctx = new CompositeElementDbContext(ConnectionString); + await ctx.Database.EnsureCreatedAsync(); + } + + public Task DisposeAsync() => Task.CompletedTask; +} + +public class CompositeElementConversionTests : IClassFixture +{ + private readonly CompositeElementConversionFixture _fixture; + + public CompositeElementConversionTests(CompositeElementConversionFixture fixture) + => _fixture = fixture; + + private static readonly DateTimeOffset Instant = new(2026, 1, 15, 5, 0, 0, TimeSpan.Zero); + + private static CompositeElementEntity NewRow(long id) => new() + { + Id = id, + Offsets = [Instant, Instant.AddDays(1)], + Dates = [new DateOnly(2026, 1, 15), new DateOnly(2026, 2, 20)], + OffsetList = [Instant, Instant.AddHours(3)], + NestedOffsets = [[Instant], [Instant.AddDays(2), Instant.AddDays(3)]], + OffsetsByName = new Dictionary { ["start"] = Instant, ["end"] = Instant.AddDays(5) }, + NamesByDate = new Dictionary { [new DateOnly(2026, 3, 1)] = "march" }, + OffsetTuple = Tuple.Create(Instant, "reference"), + OffsetValueTuple = (Instant.AddDays(7), 42), + Ints = [1, 2, 3], + Counts = new Dictionary { ["a"] = 1, ["b"] = 2 }, + }; + + [Fact] + public async Task Store_types_are_the_expected_composites() + { + using var connection = new global::ClickHouse.Driver.ADO.ClickHouseConnection(_fixture.ConnectionString); + await connection.OpenAsync(); + using var command = connection.CreateCommand(); + command.CommandText = "DESCRIBE TABLE composite_elements"; + + var columns = new Dictionary(); + using var reader = await command.ExecuteReaderAsync(); + while (await reader.ReadAsync()) + columns[(string)reader.GetValue(0)] = (string)reader.GetValue(1); + + Assert.Equal("Array(DateTime64(7, 'UTC'))", columns["offsets"]); + Assert.Equal("Array(Date32)", columns["dates"]); + Assert.Equal("Array(Array(DateTime64(7, 'UTC')))", columns["nested_offsets"]); + Assert.Equal("Map(String, DateTime64(7, 'UTC'))", columns["offsets_by_name"]); + Assert.Equal("Map(Date32, String)", columns["names_by_date"]); + Assert.Equal("Tuple(DateTime64(7, 'UTC'), String)", columns["offset_tuple"]); + } + + [Fact] + public async Task Every_composite_shape_round_trips() + { + using var writeContext = new CompositeElementDbContext(_fixture.ConnectionString); + writeContext.Entities.Add(NewRow(1)); + await writeContext.SaveChangesAsync(); + + using var readContext = new CompositeElementDbContext(_fixture.ConnectionString); + var row = await readContext.Entities.SingleAsync(e => e.Id == 1); + var expected = NewRow(1); + + Assert.Equal(expected.Offsets, row.Offsets); + Assert.Equal(expected.Dates, row.Dates); + Assert.Equal(expected.OffsetList, row.OffsetList); + Assert.Equal(expected.OffsetsByName, row.OffsetsByName); + Assert.Equal(expected.NamesByDate, row.NamesByDate); + Assert.Equal(expected.OffsetTuple, row.OffsetTuple); + Assert.Equal(expected.OffsetValueTuple, row.OffsetValueTuple); + Assert.Equal(expected.Ints, row.Ints); + Assert.Equal(expected.Counts, row.Counts); + + // Nested arrays compose: the element mapping is itself an array mapping. + Assert.Equal(expected.NestedOffsets.Length, row.NestedOffsets.Length); + for (var i = 0; i < expected.NestedOffsets.Length; i++) + Assert.Equal(expected.NestedOffsets[i], row.NestedOffsets[i]); + } + + /// Projecting the column alone exercises the mapping without entity materialization. + [Fact] + public async Task Projecting_a_composite_column_converts_its_elements() + { + using var writeContext = new CompositeElementDbContext(_fixture.ConnectionString); + writeContext.Entities.Add(NewRow(2)); + await writeContext.SaveChangesAsync(); + + using var ctx = new CompositeElementDbContext(_fixture.ConnectionString); + + Assert.Equal([Instant, Instant.AddDays(1)], await ctx.Entities.Where(e => e.Id == 2).Select(e => e.Offsets).SingleAsync()); + Assert.Equal([new DateOnly(2026, 1, 15), new DateOnly(2026, 2, 20)], await ctx.Entities.Where(e => e.Id == 2).Select(e => e.Dates).SingleAsync()); + Assert.Equal(Tuple.Create(Instant, "reference"), await ctx.Entities.Where(e => e.Id == 2).Select(e => e.OffsetTuple).SingleAsync()); + } + + [Fact] + public async Task Empty_composites_round_trip() + { + using var writeContext = new CompositeElementDbContext(_fixture.ConnectionString); + writeContext.Entities.Add(new CompositeElementEntity + { + Id = 3, + Offsets = [], + Dates = [], + OffsetList = [], + NestedOffsets = [], + OffsetsByName = [], + NamesByDate = [], + OffsetTuple = Tuple.Create(Instant, "only"), + OffsetValueTuple = (Instant, 0), + Ints = [], + Counts = [], + }); + await writeContext.SaveChangesAsync(); + + using var readContext = new CompositeElementDbContext(_fixture.ConnectionString); + var row = await readContext.Entities.SingleAsync(e => e.Id == 3); + + Assert.Empty(row.Offsets); + Assert.Empty(row.Dates); + Assert.Empty(row.OffsetList); + Assert.Empty(row.NestedOffsets); + Assert.Empty(row.OffsetsByName); + } + + /// + /// The nullable wrapper must add exactly one Nullable(...). The inner mapping is resolved + /// from a store type that already carries the wrapper, and that text is preserved verbatim, so + /// wrapping again gave Array(Nullable(Nullable(T))) — DDL that ClickHouse rejects. + /// + [Fact] + public async Task Nullable_element_store_type_is_not_double_wrapped() + { + using var ctx = new CompositeElementDbContext(_fixture.ConnectionString); + var entityType = ctx.Model.FindEntityType(typeof(NullableCompositeElementEntity))!; + + Assert.Equal( + "Array(Nullable(DateTime64(7, 'UTC')))", + entityType.FindProperty(nameof(NullableCompositeElementEntity.Offsets))!.GetColumnType()); + Assert.Equal( + "Array(Nullable(Date32))", + entityType.FindProperty(nameof(NullableCompositeElementEntity.Dates))!.GetColumnType()); + + // And the table really exists with that shape, so EnsureCreated accepted the DDL. + using var connection = new global::ClickHouse.Driver.ADO.ClickHouseConnection(_fixture.ConnectionString); + await connection.OpenAsync(); + using var command = connection.CreateCommand(); + command.CommandText = "DESCRIBE TABLE nullable_composite_elements"; + + var columns = new Dictionary(); + using var reader = await command.ExecuteReaderAsync(); + while (await reader.ReadAsync()) + columns[(string)reader.GetValue(0)] = (string)reader.GetValue(1); + + Assert.Equal("Array(Nullable(DateTime64(7, 'UTC')))", columns["offsets"]); + Assert.Equal("Array(Nullable(Date32))", columns["dates"]); + } + + /// + /// Array(Nullable(T)) goes through ClickHouseNullableElementMapping, which must + /// delegate the read conversion to the inner mapping while nulls pass straight through. + /// + [Fact] + public async Task Nullable_elements_round_trip_with_nulls() + { + using var writeContext = new CompositeElementDbContext(_fixture.ConnectionString); + writeContext.NullableEntities.Add(new NullableCompositeElementEntity + { + Id = 1, + Offsets = [Instant, null, Instant.AddDays(1)], + Dates = [new DateOnly(2026, 1, 15), null], + }); + await writeContext.SaveChangesAsync(); + + using var readContext = new CompositeElementDbContext(_fixture.ConnectionString); + var row = await readContext.NullableEntities.SingleAsync(e => e.Id == 1); + + Assert.Equal([Instant, null, Instant.AddDays(1)], row.Offsets); + Assert.Equal([new DateOnly(2026, 1, 15), null], row.Dates); + } + + /// + /// A component that needs no conversion must keep the direct cast, so the common case pays + /// nothing. Asserted on the shape of the read expression rather than on a round trip, because a + /// round trip passes either way. String and Float64 convert nothing; note that the + /// integer mappings do (they widen with Convert.ToInt32 for aggregates), so + /// int[] is not a valid example here. + /// + [Theory] + [InlineData(nameof(ConvertedCompositeEntity.Names))] + [InlineData(nameof(ConvertedCompositeEntity.Ratios))] + [InlineData(nameof(ConvertedCompositeEntity.Labels))] + public void Components_needing_no_conversion_keep_the_direct_cast(string propertyName) + { + using var ctx = new CompositeElementDbContext(_fixture.ConnectionString); + var mapping = ctx.Model + .FindEntityType(typeof(ConvertedCompositeEntity))! + .FindProperty(propertyName)! + .GetRelationalTypeMapping(); + + var read = mapping.CustomizeDataReaderExpression(Expression.Parameter(typeof(object), "v")); + + // A direct cast is a UnaryExpression; the rebuild path emits a Call to a Convert* helper. + Assert.IsAssignableFrom(read); + } + + /// The counterpart: a converting component must take the rebuild path. + [Theory] + [InlineData(nameof(CompositeElementEntity.Offsets))] + [InlineData(nameof(CompositeElementEntity.Dates))] + [InlineData(nameof(CompositeElementEntity.OffsetsByName))] + [InlineData(nameof(CompositeElementEntity.Ints))] + public void Converting_components_take_the_rebuild_path(string propertyName) + { + using var ctx = new CompositeElementDbContext(_fixture.ConnectionString); + var mapping = ctx.Model + .FindEntityType(typeof(CompositeElementEntity))! + .FindProperty(propertyName)! + .GetRelationalTypeMapping(); + + var read = mapping.CustomizeDataReaderExpression(Expression.Parameter(typeof(object), "v")); + + Assert.IsAssignableFrom(read); + } + + /// + /// The per-component converter must be embedded as a constant, not as an inline lambda. An inline + /// lambda is rebuilt on every materialization, which allocates a delegate per row for every + /// composite column that converts — including ones where the runtime fast path then returns the + /// driver's array untouched. + /// + [Theory] + [InlineData(nameof(CompositeElementEntity.Offsets))] + [InlineData(nameof(CompositeElementEntity.Ints))] + [InlineData(nameof(CompositeElementEntity.OffsetsByName))] + public void Component_converters_are_embedded_as_constants(string propertyName) + { + using var ctx = new CompositeElementDbContext(_fixture.ConnectionString); + var mapping = ctx.Model + .FindEntityType(typeof(CompositeElementEntity))! + .FindProperty(propertyName)! + .GetRelationalTypeMapping(); + + var read = (MethodCallExpression)mapping.CustomizeDataReaderExpression( + Expression.Parameter(typeof(object), "v")); + + // Argument 0 is the raw value; every argument after it is a component converter. + Assert.All( + read.Arguments.Skip(1), + argument => Assert.Equal(ExpressionType.Constant, argument.NodeType)); + } + + // --- components carrying a ValueConverter ------------------------------- + + private static ConvertedCompositeEntity NewConvertedRow(long id) => new() + { + Id = id, + Colours = [CompositeColour.Red, CompositeColour.Blue], + ColourTuple = Tuple.Create(CompositeColour.Blue, 7), + ColourByName = new Dictionary { ["primary"] = CompositeColour.Green }, + Buckets = new Dictionary> { ["low"] = [1, 2], ["high"] = [9] }, + Nested = [[1, 2], [3]], + ListArray = [[4, 5], []], + Interfaced = new List { Instant, Instant.AddDays(1) }, + ReadOnlyDates = [new DateOnly(2026, 4, 1)], + Names = ["a", "b"], + Ratios = [1.5, 2.5], + Labels = new Dictionary { ["k"] = "v" }, + }; + + /// + /// A component mapping can convert through a ValueConverter instead of a data-reader + /// conversion. The composite must apply that too — otherwise the column is writable but not + /// readable, which is worse than refusing the model up front. + /// + /// + /// Seeded with raw SQL on purpose. Writing a converter-bearing component through + /// SaveChanges does not work yet: the bulk insert path passes model values straight to the + /// driver without applying the converter, so an enum component is written as its raw ordinal + /// (issue #54). This test covers the read direction, which is what the composite conversion + /// fixes. + /// + [Fact] + public async Task Components_with_a_value_converter_read_correctly() + { + using var connection = new global::ClickHouse.Driver.ADO.ClickHouseConnection(_fixture.ConnectionString); + await connection.OpenAsync(); + using var insert = connection.CreateCommand(); + insert.CommandText = """ + INSERT INTO converted_composites + (id, colours, colour_tuple, colour_by_name, buckets, nested, list_array, + interfaced, readonly_dates, names, ratios, labels) + VALUES + (1, ['Red', 'Blue'], ('Blue', 7), {'primary': 'Green'}, + {'low': [1, 2], 'high': [9]}, [[1, 2], [3]], [[4, 5], []], + ['2026-01-15 05:00:00.0000000', '2026-01-16 05:00:00.0000000'], + ['2026-04-01'], ['a', 'b'], [1.5, 2.5], {'k': 'v'}) + """; + await insert.ExecuteNonQueryAsync(); + + using var readContext = new CompositeElementDbContext(_fixture.ConnectionString); + var row = await readContext.ConvertedEntities.SingleAsync(e => e.Id == 1); + var expected = NewConvertedRow(1); + + // EnumToStringConverter components. + Assert.Equal(expected.Colours, row.Colours); + Assert.Equal(expected.ColourTuple, row.ColourTuple); + Assert.Equal(expected.ColourByName, row.ColourByName); + + // ListToArrayConverter components, including inside a Map and nested one level. + Assert.Equal(expected.Buckets, row.Buckets); + Assert.Equal(expected.Nested, row.Nested); + Assert.Equal(expected.ListArray.Length, row.ListArray.Length); + for (var i = 0; i < expected.ListArray.Length; i++) + Assert.Equal(expected.ListArray[i], row.ListArray[i]); + + // Collection-interface components go through EnumerableToArrayConverter. + Assert.Equal(expected.Interfaced, row.Interfaced); + Assert.Equal(expected.ReadOnlyDates, row.ReadOnlyDates); + + // And the no-conversion components still work. + Assert.Equal(expected.Names, row.Names); + Assert.Equal(expected.Ratios, row.Ratios); + Assert.Equal(expected.Labels, row.Labels); + } + + [Fact] + public async Task Enum_component_store_types_are_as_configured() + { + using var connection = new global::ClickHouse.Driver.ADO.ClickHouseConnection(_fixture.ConnectionString); + await connection.OpenAsync(); + using var command = connection.CreateCommand(); + command.CommandText = "DESCRIBE TABLE converted_composites"; + + var columns = new Dictionary(); + using var reader = await command.ExecuteReaderAsync(); + while (await reader.ReadAsync()) + columns[(string)reader.GetValue(0)] = (string)reader.GetValue(1); + + Assert.Equal("Array(Enum8('Red' = 1, 'Green' = 2, 'Blue' = 3))", columns["colours"]); + Assert.Equal("Map(String, Enum8('Red' = 1, 'Green' = 2, 'Blue' = 3))", columns["colour_by_name"]); + Assert.Equal("Map(String, Array(Int32))", columns["buckets"]); + Assert.Equal("Array(Array(Int32))", columns["nested"]); + } + + // --- the already-typed pass-through ------------------------------------- + + /// + /// Reading a composite keeps a fast path for a driver value that already has the target CLR + /// type. That is sound only where matching types prove there is no work left, which a + /// ValueConverter can break: it may change the value and keep the CLR type, so the + /// driver's array is already string[] while ConvertFromProvider still has to run. + /// + /// + /// No mapping the provider resolves on its own is shaped this way — every converter it uses also + /// changes the CLR type — but the shape is reachable from the public API through + /// ElementType().HasConversion(...), which the model test below covers. This one drives + /// the mapping directly so the fast path is exercised without a model. + /// + [Fact] + public void A_same_clr_type_component_converter_is_not_skipped_by_the_fast_path() + { + using var ctx = new CompositeElementDbContext(_fixture.ConnectionString); + var source = ctx.GetService(); + var stringMapping = source.FindMapping(typeof(string), "String")!; + + // A converter that keeps the CLR type but changes the value. + var elementMapping = (RelationalTypeMapping)stringMapping.WithComposedConverter( + new ValueConverter(v => v, v => v + "!")); + var arrayMapping = new ClickHouseArrayTypeMapping(elementMapping); + + Assert.Equal(typeof(string[]), arrayMapping.ClrType); + Assert.NotNull(elementMapping.Converter); + + // The driver hands back string[], which is already the target type. + var read = Read(arrayMapping, new[] { "a", "b" }); + + Assert.Equal(["a!", "b!"], read); + } + + /// + /// The fast path must survive for the case it exists to serve: a component whose read is a cast + /// and nothing more, as in a nested array. Here the driver's value is returned as it stands. + /// + [Fact] + public void An_already_typed_component_with_no_converter_passes_straight_through() + { + using var ctx = new CompositeElementDbContext(_fixture.ConnectionString); + var source = ctx.GetService(); + var innerArray = source.FindMapping(typeof(int[]), "Array(Int32)")!; + var outerArray = new ClickHouseArrayTypeMapping(innerArray); + + // No converter, so a matching CLR type is proof enough that nothing is left to do. + Assert.Null(innerArray.Converter); + + var driverValue = new[] { new[] { 1, 2 }, new[] { 3 } }; + var read = Read(outerArray, driverValue); + + Assert.Same(driverValue, read); + } + + /// + /// The same hazard for a Map. Both the key and the value mapping must be checked, so this + /// puts the converter on the value and leaves the key alone. + /// + [Fact] + public void A_same_clr_type_map_value_converter_is_not_skipped_by_the_fast_path() + { + using var ctx = new CompositeElementDbContext(_fixture.ConnectionString); + var source = ctx.GetService(); + var stringMapping = source.FindMapping(typeof(string), "String")!; + + var valueMapping = (RelationalTypeMapping)stringMapping.WithComposedConverter( + new ValueConverter(v => v, v => v + "!")); + var mapMapping = new ClickHouseMapTypeMapping(stringMapping, valueMapping); + + // The driver hands back the target dictionary type already. + var read = Read>( + mapMapping, + new Dictionary { ["k"] = "a" }); + + Assert.Equal("a!", read["k"]); + } + + /// + /// The same hazard for a Tuple. A reference tuple is used, because a ValueTuple + /// target never reaches the fast path — the driver returns System.Tuple<>. + /// + [Fact] + public void A_same_clr_type_tuple_component_converter_is_not_skipped_by_the_fast_path() + { + using var ctx = new CompositeElementDbContext(_fixture.ConnectionString); + var source = ctx.GetService(); + var stringMapping = source.FindMapping(typeof(string), "String")!; + + var componentMapping = (RelationalTypeMapping)stringMapping.WithComposedConverter( + new ValueConverter(v => v, v => v + "!")); + var tupleMapping = new ClickHouseTupleTypeMapping( + [componentMapping, componentMapping], + useValueTuple: false); + + Assert.Equal(typeof(Tuple), tupleMapping.ClrType); + + var read = Read>(tupleMapping, Tuple.Create("a", "b")); + + Assert.Equal(Tuple.Create("a!", "b!"), read); + } + + /// + /// The reachable route to the same shape: an element converter set through the public API. This + /// is why the gate matters rather than being defence against a shape nobody can build. + /// + [Fact] + public async Task An_element_converter_set_on_the_model_is_applied_on_read() + { + using var connection = new global::ClickHouse.Driver.ADO.ClickHouseConnection(_fixture.ConnectionString); + await connection.OpenAsync(); + + // The fixture already created the database, so EnsureCreated would add nothing. Written + // outside EF anyway, because SaveChanges does not apply converters yet (#54). + using var create = connection.CreateCommand(); + create.CommandText = + "CREATE TABLE IF NOT EXISTS element_converted (id Int64, tags Array(String)) " + + "ENGINE = MergeTree ORDER BY id"; + await create.ExecuteNonQueryAsync(); + + using var command = connection.CreateCommand(); + command.CommandText = "INSERT INTO element_converted VALUES (1, ['a', 'b'])"; + await command.ExecuteNonQueryAsync(); + + using var readContext = new ElementConverterDbContext(_fixture.ConnectionString); + var row = await readContext.Entities.SingleAsync(e => e.Id == 1); + + // Without the gate the driver's raw string[] would come straight through as "a", "b". + Assert.Equal(["a!", "b!"], row.Tags); + } + + /// Compiles and runs a mapping's data-reader expression over one driver value. + private static T Read(RelationalTypeMapping mapping, object driverValue) + { + var parameter = Expression.Parameter(typeof(object), "value"); + var body = mapping.CustomizeDataReaderExpression(parameter); + + return Expression.Lambda>(Expression.Convert(body, typeof(T)), parameter) + .Compile()(driverValue); + } +} diff --git a/test/EFCore.ClickHouse.Tests/DateTimeOffsetMappingTests.cs b/test/EFCore.ClickHouse.Tests/DateTimeOffsetMappingTests.cs new file mode 100644 index 0000000..32a5e81 --- /dev/null +++ b/test/EFCore.ClickHouse.Tests/DateTimeOffsetMappingTests.cs @@ -0,0 +1,1017 @@ +using ClickHouse.EntityFrameworkCore.Storage.Internal.Mapping; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Xunit; + +namespace EFCore.ClickHouse.Tests; + +public class DateTimeOffsetEntity +{ + public long Id { get; set; } + public DateTimeOffset Default { get; set; } + public DateTimeOffset? Nullable { get; set; } + public double Value { get; set; } +} + +/// Points DateTimeOffset properties at columns with differing declared timezones. +public class DateTimeOffsetTzEntity +{ + public long Id { get; set; } + public DateTimeOffset Utc { get; set; } + public DateTimeOffset Naive { get; set; } + public DateTimeOffset Tokyo { get; set; } + public DateTimeOffset Seconds { get; set; } +} + +/// A zone with daylight saving, for the ambiguous-wall-clock case. +public class DateTimeOffsetDstEntity +{ + public long Id { get; set; } + public DateTimeOffset London { get; set; } +} + +/// Precision set through HasPrecision rather than HasColumnType. +public class DateTimeOffsetPrecisionEntity +{ + public long Id { get; set; } + public DateTimeOffset Millis { get; set; } +} + +/// +/// Columns that declare a fixed UTC offset instead of a named zone. ClickHouse spells these +/// Fixed/UTC±HH:MM:SS, and .NET has no timezone of that name. +/// +public class DateTimeOffsetFixedEntity +{ + public long Id { get; set; } + public DateTimeOffset Half { get; set; } + public DateTimeOffset Negative { get; set; } + public DateTimeOffset Quarter { get; set; } + public DateTimeOffset Zero { get; set; } + public DateTimeOffset Seconds { get; set; } +} + +/// +/// A column whose fixed-offset name carries minutes above 59. ClickHouse accepts the name, the +/// driver cannot read it, so the read must report that rather than guess an offset. +/// +public class DateTimeOffsetCarriedEntity +{ + public long Id { get; set; } + public DateTimeOffset Carried { get; set; } +} + +public class DateTimeOffsetDbContext : DbContext +{ + private readonly string _connectionString; + + public DateTimeOffsetDbContext(string connectionString) => _connectionString = connectionString; + + public DbSet Entities => Set(); + public DbSet TzEntities => Set(); + public DbSet DstEntities => Set(); + public DbSet PrecisionEntities => Set(); + public DbSet FixedEntities => Set(); + public DbSet CarriedEntities => Set(); + + protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder) + => optionsBuilder.UseClickHouse(_connectionString); + + protected override void OnModelCreating(ModelBuilder modelBuilder) + { + modelBuilder.Entity(e => + { + e.ToTable("dto_default"); + e.HasKey(x => x.Id); + e.Property(x => x.Id).HasColumnName("id"); + e.Property(x => x.Default).HasColumnName("dt"); + e.Property(x => x.Nullable).HasColumnName("dt_null"); + e.Property(x => x.Value).HasColumnName("value"); + }); + + modelBuilder.Entity(e => + { + e.ToTable("dto_tz"); + e.HasKey(x => x.Id); + e.Property(x => x.Id).HasColumnName("id"); + e.Property(x => x.Utc).HasColumnName("utc").HasColumnType("DateTime64(6, 'UTC')"); + // A timezone-less column only round trips here because the test container's + // session_timezone is UTC. This is the hazard the default UTC pin avoids. + e.Property(x => x.Naive).HasColumnName("naive").HasColumnType("DateTime64(6)"); + e.Property(x => x.Tokyo).HasColumnName("tokyo").HasColumnType("DateTime64(6, 'Asia/Tokyo')"); + e.Property(x => x.Seconds).HasColumnName("seconds").HasColumnType("DateTime('UTC')"); + }); + + modelBuilder.Entity(e => + { + e.ToTable("dto_dst"); + e.HasKey(x => x.Id); + e.Property(x => x.Id).HasColumnName("id"); + e.Property(x => x.London).HasColumnName("london").HasColumnType("DateTime64(7, 'Europe/London')"); + }); + + modelBuilder.Entity(e => + { + e.ToTable("dto_precision"); + e.HasKey(x => x.Id); + e.Property(x => x.Id).HasColumnName("id"); + // Precision only — no HasColumnType, so the UTC pin must be kept. + e.Property(x => x.Millis).HasColumnName("millis").HasPrecision(3); + }); + + modelBuilder.Entity(e => + { + e.ToTable("dto_fixed"); + e.HasKey(x => x.Id); + e.Property(x => x.Id).HasColumnName("id"); + e.Property(x => x.Half).HasColumnName("half") + .HasColumnType("DateTime64(7, 'Fixed/UTC+05:30:00')"); + e.Property(x => x.Negative).HasColumnName("negative") + .HasColumnType("DateTime64(7, 'Fixed/UTC-07:00:00')"); + // A quarter-hour offset, which no whole-hour shortcut would handle. + e.Property(x => x.Quarter).HasColumnName("quarter") + .HasColumnType("DateTime64(7, 'Fixed/UTC+05:45:00')"); + // Offset zero: the driver reports Kind=Utc here rather than a wall clock. + e.Property(x => x.Zero).HasColumnName("zero") + .HasColumnType("DateTime64(7, 'Fixed/UTC+00:00:00')"); + // A whole-minute offset written with the seconds field the name always carries. + e.Property(x => x.Seconds).HasColumnName("seconds") + .HasColumnType("DateTime64(7, 'Fixed/UTC+00:01:00')"); + }); + + modelBuilder.Entity(e => + { + e.ToTable("dto_carried"); + e.HasKey(x => x.Id); + e.Property(x => x.Id).HasColumnName("id"); + // ClickHouse reads this as +06:00. The driver does not read it at all. + e.Property(x => x.Carried).HasColumnName("carried") + .HasColumnType("DateTime64(7, 'Fixed/UTC+05:60:00')"); + }); + } +} + +public class StringDateTimeOffsetEntity +{ + public long Id { get; set; } + public DateTimeOffset ViaConversion { get; set; } + public DateTimeOffset ViaColumnType { get; set; } +} + +public class StringDateTimeOffsetDbContext : DbContext +{ + private readonly string _connectionString; + + public StringDateTimeOffsetDbContext(string connectionString) => _connectionString = connectionString; + + public DbSet Entities => Set(); + + protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder) + => optionsBuilder.UseClickHouse(_connectionString); + + protected override void OnModelCreating(ModelBuilder modelBuilder) + { + modelBuilder.Entity(e => + { + e.ToTable("dto_string"); + e.HasKey(x => x.Id); + e.Property(x => x.Id).HasColumnName("id"); + e.Property(x => x.ViaConversion).HasColumnName("via_conversion").HasConversion(); + e.Property(x => x.ViaColumnType).HasColumnName("via_column_type").HasColumnType("String"); + }); + } +} + +public class DateTimeOffsetMappingFixture : IAsyncLifetime +{ + public string ConnectionString { get; private set; } = string.Empty; + + public async Task InitializeAsync() + { + ConnectionString = await SharedContainer.GetConnectionStringAsync(); + using var ctx = new DateTimeOffsetDbContext(ConnectionString); + await ctx.Database.EnsureCreatedAsync(); + } + + public Task DisposeAsync() => Task.CompletedTask; +} + +public class DateTimeOffsetMappingTests : IClassFixture +{ + private readonly DateTimeOffsetMappingFixture _fixture; + + public DateTimeOffsetMappingTests(DateTimeOffsetMappingFixture fixture) => _fixture = fixture; + + private static long ToMicros(DateTimeOffset value) => value.ToUnixTimeMilliseconds() * 1000L; + + // --- mapping resolution ------------------------------------------------- + + [Fact] + public void Default_mapping_is_utc_pinned_datetime64() + { + using var ctx = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var property = ctx.Model + .FindEntityType(typeof(DateTimeOffsetEntity))! + .FindProperty(nameof(DateTimeOffsetEntity.Default))!; + + var mapping = property.GetRelationalTypeMapping(); + + Assert.IsType(mapping); + Assert.Equal(typeof(DateTimeOffset), mapping.ClrType); + Assert.Equal("DateTime64(7, 'UTC')", mapping.StoreType); + // The old behaviour resolved a String column through DateTimeOffsetToStringConverter. + Assert.Null(mapping.Converter); + } + + [Fact] + public void Nullable_property_resolves_the_same_mapping() + { + using var ctx = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var property = ctx.Model + .FindEntityType(typeof(DateTimeOffsetEntity))! + .FindProperty(nameof(DateTimeOffsetEntity.Nullable))!; + + var mapping = property.GetRelationalTypeMapping(); + + Assert.IsType(mapping); + Assert.Equal("DateTime64(7, 'UTC')", mapping.StoreType); + } + + [Theory] + [InlineData(nameof(DateTimeOffsetTzEntity.Utc), "DateTime64(6, 'UTC')")] + [InlineData(nameof(DateTimeOffsetTzEntity.Naive), "DateTime64(6)")] + [InlineData(nameof(DateTimeOffsetTzEntity.Tokyo), "DateTime64(6, 'Asia/Tokyo')")] + [InlineData(nameof(DateTimeOffsetTzEntity.Seconds), "DateTime('UTC')")] + public void Explicit_store_type_is_preserved(string propertyName, string expectedStoreType) + { + using var ctx = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var property = ctx.Model + .FindEntityType(typeof(DateTimeOffsetTzEntity))! + .FindProperty(propertyName)!; + + var mapping = property.GetRelationalTypeMapping(); + + Assert.IsType(mapping); + Assert.Equal(typeof(DateTimeOffset), mapping.ClrType); + Assert.Equal(expectedStoreType, property.GetColumnType()); + } + + [Fact] + public async Task EnsureCreated_makes_a_datetime64_column_not_a_string() + { + using var connection = new global::ClickHouse.Driver.ADO.ClickHouseConnection(_fixture.ConnectionString); + await connection.OpenAsync(); + using var command = connection.CreateCommand(); + command.CommandText = "DESCRIBE TABLE dto_default"; + + var columns = new Dictionary(); + using var reader = await command.ExecuteReaderAsync(); + while (await reader.ReadAsync()) + columns[(string)reader.GetValue(0)] = (string)reader.GetValue(1); + + Assert.Equal("DateTime64(7, 'UTC')", columns["dt"]); + Assert.Equal("Nullable(DateTime64(7, 'UTC'))", columns["dt_null"]); + } + + /// + /// HasPrecision(n) with no HasColumnType must change the precision and keep the + /// UTC pin. Dropping the facet would silently give the default precision 7 instead. + /// + [Fact] + public void HasPrecision_is_honoured_and_keeps_the_utc_pin() + { + using var ctx = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var property = ctx.Model + .FindEntityType(typeof(DateTimeOffsetPrecisionEntity))! + .FindProperty(nameof(DateTimeOffsetPrecisionEntity.Millis))!; + + var mapping = property.GetRelationalTypeMapping(); + + Assert.Equal("DateTime64(3, 'UTC')", mapping.StoreType); + Assert.Equal("DateTime64(3, 'UTC')", property.GetColumnType()); + Assert.Equal(3, mapping.Precision); + } + + /// A bare DateTime64 with no argument is precision 3 in ClickHouse, not 7. + [Fact] + public void Bare_datetime64_store_type_uses_clickhouse_default_precision() + { + using var ctx = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var source = ctx.GetService(); + + var mapping = source.FindMapping(typeof(DateTimeOffset), "DateTime64"); + + Assert.Equal(3, mapping!.Precision); + } + + [Fact] + public void Bare_datetime_store_type_maps_to_seconds_precision() + { + var mapping = new ClickHouseDateTimeOffsetTypeMapping(precision: null, timezone: null); + + Assert.Equal("DateTime", mapping.StoreType); + Assert.Null(mapping.Precision); + } + + /// + /// When the configured text differs from the mapping's canonical store type, + /// PreserveExplicitStoreType clones the mapping to keep that text. The clone must carry + /// Timezone over, or the read path would lose the offset of the declared zone. + /// + [Theory] + [InlineData("DateTime64(6,'Asia/Tokyo')", "Asia/Tokyo")] + [InlineData("Nullable(DateTime64(6, 'Asia/Tokyo'))", "Asia/Tokyo")] + [InlineData("LowCardinality(DateTime64(6, 'Asia/Tokyo'))", "Asia/Tokyo")] + [InlineData("datetime64(6, 'Asia/Tokyo')", "Asia/Tokyo")] + public void Clone_for_a_preserved_store_type_keeps_the_timezone(string columnType, string expectedTimezone) + { + using var ctx = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var source = ctx.GetService(); + + var mapping = source.FindMapping(typeof(DateTimeOffset), columnType); + + var typed = Assert.IsType(mapping); + Assert.Equal(typeof(DateTimeOffset), typed.ClrType); + // The user's text survives verbatim... + Assert.Equal(columnType, typed.StoreType); + // ...and the timezone needed by the read path survives the clone. + Assert.Equal(expectedTimezone, typed.Timezone); + } + + // --- SQL literals ------------------------------------------------------- + + [Theory] + [InlineData(7, "UTC", "DateTime64(7, 'UTC')", "'2026-01-15 10:00:00.1234567+05:00'")] + [InlineData(6, "UTC", "DateTime64(6, 'UTC')", "'2026-01-15 10:00:00.123456+05:00'")] + [InlineData(3, null, "DateTime64(3)", "'2026-01-15 10:00:00.123+05:00'")] + [InlineData(0, "UTC", "DateTime64(0, 'UTC')", "'2026-01-15 10:00:00+05:00'")] + public void Literal_carries_the_offset(int precision, string? timezone, string expectedStoreType, string expectedLiteral) + { + var mapping = new ClickHouseDateTimeOffsetTypeMapping(precision, timezone); + var value = new DateTimeOffset(2026, 1, 15, 10, 0, 0, TimeSpan.FromHours(5)).AddTicks(1234567); + + Assert.Equal(expectedStoreType, mapping.StoreType); + Assert.Equal(expectedLiteral, mapping.GenerateSqlLiteral(value)); + } + + [Fact] + public void Seconds_precision_literal_has_no_fractional_digits() + { + var mapping = new ClickHouseDateTimeOffsetTypeMapping(precision: null, timezone: "UTC"); + var value = new DateTimeOffset(2026, 1, 15, 10, 0, 0, TimeSpan.FromHours(5)); + + Assert.Equal("DateTime('UTC')", mapping.StoreType); + Assert.Equal("'2026-01-15 10:00:00+05:00'", mapping.GenerateSqlLiteral(value)); + } + + /// + /// A literal that carries its offset must land on the same instant whatever timezone the + /// target column declares. A bare wall clock does not, which is why the offset is emitted. + /// + [Theory] + [InlineData("DateTime64(6)")] + [InlineData("DateTime64(6, 'UTC')")] + [InlineData("DateTime64(6, 'Asia/Tokyo')")] + [InlineData("DateTime64(6, 'Fixed/UTC+05:30:00')")] + [InlineData("DateTime")] + [InlineData("DateTime('Asia/Tokyo')")] + public async Task Literal_is_instant_exact_for_any_column_timezone(string columnType) + { + var mapping = new ClickHouseDateTimeOffsetTypeMapping(6, "UTC"); + var value = new DateTimeOffset(2026, 1, 15, 10, 0, 0, TimeSpan.FromHours(5)); + + using var connection = new global::ClickHouse.Driver.ADO.ClickHouseConnection(_fixture.ConnectionString); + await connection.OpenAsync(); + using var command = connection.CreateCommand(); + command.CommandText = + $"SELECT toUnixTimestamp64Micro(toDateTime64(CAST({mapping.GenerateSqlLiteral(value)} AS {columnType}), 6))"; + + Assert.Equal(ToMicros(value), Convert.ToInt64(await command.ExecuteScalarAsync())); + } + + // --- round trip --------------------------------------------------------- + + [Fact] + public async Task Insert_and_read_back_preserves_the_instant_at_tick_precision() + { + var value = new DateTimeOffset(2026, 1, 15, 10, 0, 0, TimeSpan.FromHours(5)).AddTicks(1234567); + + using var writeContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + writeContext.Entities.Add(new DateTimeOffsetEntity { Id = 1, Default = value, Nullable = value, Value = 1.5 }); + await writeContext.SaveChangesAsync(); + + using var readContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var row = await readContext.Entities.SingleAsync(e => e.Id == 1); + + // The instant survives exactly; the offset becomes the column's, which is UTC. + Assert.Equal(value.ToUniversalTime(), row.Default); + Assert.Equal(TimeSpan.Zero, row.Default.Offset); + Assert.Equal(value.ToUniversalTime(), row.Nullable); + } + + /// + /// The pattern in issue #53 uses DateTimeOffset.MinValue/MaxValue as open-ended + /// range sentinels, so both ends of the CLR range must survive. Precision 7 keeps this working: + /// 100 ns units in an Int64 span about 29,000 years, which covers all of + /// . A higher precision would not — DateTime64(9) overflows + /// well before year 9999. + /// + [Fact] + public async Task Min_and_max_values_round_trip_and_work_as_range_sentinels() + { + using var writeContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + writeContext.Entities.AddRange( + new DateTimeOffsetEntity { Id = 40, Default = DateTimeOffset.MinValue, Value = 1 }, + new DateTimeOffsetEntity { Id = 41, Default = DateTimeOffset.MaxValue, Value = 2 }); + await writeContext.SaveChangesAsync(); + + using var readContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var min = await readContext.Entities.SingleAsync(e => e.Id == 40); + var max = await readContext.Entities.SingleAsync(e => e.Id == 41); + + Assert.Equal(DateTimeOffset.MinValue, min.Default); + Assert.Equal(DateTimeOffset.MaxValue, max.Default); + + // Used as sentinels, they must bracket everything. + DateTimeOffset? startDate = null; + DateTimeOffset? endDate = null; + startDate ??= DateTimeOffset.MinValue; + endDate ??= DateTimeOffset.MaxValue; + + var total = await readContext.Entities + .Where(e => e.Id == 40 || e.Id == 41) + .Where(e => e.Default >= startDate) + .Where(e => e.Default <= endDate) + .CountAsync(); + + Assert.Equal(2, total); + } + + /// + /// The sentinel pattern above only works on a column whose timezone offset is zero. Both ends of + /// the range sit at the edge of 's range, and + /// the driver has to build a wall clock in the column's timezone to return one. Any non-zero + /// offset pushes one end outside , and the driver throws while doing so — + /// before the provider sees the value, so this cannot be reported any better from here. + /// + /// + /// This is not specific to a fixed offset; a named zone such as Asia/Tokyo behaves the + /// same way. Reading the low end of the range from a named zone is worse than an error: zones + /// carry a Local Mean Time offset for year 1 (+09:18:59 for Tokyo), so the value comes + /// back quietly shifted. + /// + [Fact] + public async Task Max_value_does_not_read_back_from_a_non_zero_offset_column() + { + using var writeContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + writeContext.TzEntities.Add(new DateTimeOffsetTzEntity + { + Id = 50, + Utc = DateTimeOffset.MaxValue, + Naive = DateTimeOffset.MaxValue, + Tokyo = DateTimeOffset.MaxValue, + Seconds = new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero) + }); + + // The write itself is fine: an instant needs no wall clock. + await writeContext.SaveChangesAsync(); + + using var readContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + await Assert.ThrowsAsync( + () => readContext.TzEntities.SingleAsync(e => e.Id == 50)); + + // Reading the same row through the zero-offset column alone is fine, which is why the + // sentinel pattern still works on the default store type. (These columns are precision 6, + // so the value truncates to microseconds; Min_and_max_values_round_trip_and_work_as_range + // _sentinels covers the exact round trip on a precision 7 column.) + using var projectingContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var utcOnly = await projectingContext.TzEntities + .Where(e => e.Id == 50) + .Select(e => e.Utc) + .SingleAsync(); + + Assert.Equal(DateTimeOffset.MaxValue.ToUnixTimeMilliseconds(), utcOnly.ToUnixTimeMilliseconds()); + } + + [Fact] + public async Task Null_round_trips() + { + using var writeContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + writeContext.Entities.Add(new DateTimeOffsetEntity + { + Id = 2, + Default = new DateTimeOffset(2026, 2, 1, 0, 0, 0, TimeSpan.Zero), + Nullable = null, + Value = 1 + }); + await writeContext.SaveChangesAsync(); + + using var readContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var row = await readContext.Entities.SingleAsync(e => e.Id == 2); + + Assert.Null(row.Nullable); + } + + /// + /// Reading a column that declares a non-UTC timezone must still give the right instant. The + /// driver returns a wall clock in the column's timezone, so the mapping attaches that offset. + /// + [Fact] + public async Task Read_attaches_the_offset_of_the_declared_timezone() + { + var value = new DateTimeOffset(2026, 1, 15, 5, 0, 0, TimeSpan.Zero); + + using var writeContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + writeContext.TzEntities.Add(new DateTimeOffsetTzEntity + { + Id = 1, + Utc = value, + Naive = value, + Tokyo = value, + Seconds = value + }); + await writeContext.SaveChangesAsync(); + + using var readContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var row = await readContext.TzEntities.SingleAsync(e => e.Id == 1); + + Assert.Equal(value, row.Utc); + Assert.Equal(value, row.Naive); + Assert.Equal(value, row.Tokyo); + Assert.Equal(value, row.Seconds); + + // Same instant, rendered in the column's zone. + Assert.Equal(TimeSpan.Zero, row.Utc.Offset); + Assert.Equal(TimeSpan.FromHours(9), row.Tokyo.Offset); + } + + // --- queries ------------------------------------------------------------ + + /// Reproduces issue #53 directly: a range filter plus an aggregate. + [Fact] + public async Task Range_filter_with_parameters_and_aggregate() + { + using var seedContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + seedContext.Entities.AddRange( + new DateTimeOffsetEntity { Id = 10, Default = new DateTimeOffset(2026, 1, 15, 0, 0, 0, TimeSpan.Zero), Value = 1.5 }, + new DateTimeOffsetEntity { Id = 11, Default = new DateTimeOffset(2026, 2, 15, 0, 0, 0, TimeSpan.Zero), Value = 2.5 }, + new DateTimeOffsetEntity { Id = 12, Default = new DateTimeOffset(2026, 5, 15, 0, 0, 0, TimeSpan.Zero), Value = 4.0 }); + await seedContext.SaveChangesAsync(); + + using var ctx = new DateTimeOffsetDbContext(_fixture.ConnectionString); + DateTimeOffset? startDate = null; + DateTimeOffset? endDate = null; + startDate ??= new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero); + endDate ??= new DateTimeOffset(2026, 3, 1, 0, 0, 0, TimeSpan.Zero); + + var query = ctx.Entities + .Where(e => e.Id >= 10 && e.Id <= 12) + .Where(e => e.Default >= startDate) + .Where(e => e.Default < endDate); + + // The parameter must be declared as the column type, not String. + Assert.Contains("{startDate:DateTime64(7, 'UTC')}", query.ToQueryString()); + + Assert.Equal(4.0, await query.SumAsync(e => e.Value)); + } + + [Fact] + public async Task Filter_with_a_non_zero_offset_parameter_matches_the_same_instant() + { + using var seedContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + seedContext.Entities.Add(new DateTimeOffsetEntity + { + Id = 20, + Default = new DateTimeOffset(2026, 6, 15, 5, 0, 0, TimeSpan.Zero), + Value = 7.0 + }); + await seedContext.SaveChangesAsync(); + + using var ctx = new DateTimeOffsetDbContext(_fixture.ConnectionString); + // The same instant written with a +05:00 offset. + var equivalent = new DateTimeOffset(2026, 6, 15, 10, 0, 0, TimeSpan.FromHours(5)); + + var found = await ctx.Entities.SingleOrDefaultAsync(e => e.Id == 20 && e.Default == equivalent); + + Assert.NotNull(found); + } + + [Fact] + public async Task Ordering_and_projection_work() + { + using var seedContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + seedContext.Entities.AddRange( + new DateTimeOffsetEntity { Id = 30, Default = new DateTimeOffset(2026, 9, 3, 0, 0, 0, TimeSpan.Zero), Value = 1 }, + new DateTimeOffsetEntity { Id = 31, Default = new DateTimeOffset(2026, 9, 1, 0, 0, 0, TimeSpan.Zero), Value = 2 }); + await seedContext.SaveChangesAsync(); + + using var ctx = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var ordered = await ctx.Entities + .Where(e => e.Id == 30 || e.Id == 31) + .OrderBy(e => e.Default) + .Select(e => e.Default) + .ToListAsync(); + + Assert.Equal([ + new DateTimeOffset(2026, 9, 1, 0, 0, 0, TimeSpan.Zero), + new DateTimeOffset(2026, 9, 3, 0, 0, 0, TimeSpan.Zero) + ], ordered); + } + + // --- opt out of the new default ----------------------------------------- + + /// + /// HasConversion<string>() is the documented way to keep the old String shape. + /// It composes a converter, so the CLR type stays . + /// Note that writing any converted property is a separate defect (#54). + /// + [Fact] + public void HasConversion_string_keeps_the_old_string_shape() + { + using var ctx = new StringDateTimeOffsetDbContext(_fixture.ConnectionString); + var property = ctx.Model + .FindEntityType(typeof(StringDateTimeOffsetEntity))! + .FindProperty(nameof(StringDateTimeOffsetEntity.ViaConversion))!; + + var mapping = property.GetRelationalTypeMapping(); + + Assert.Equal("String", mapping.StoreType); + Assert.Equal(typeof(DateTimeOffset), mapping.ClrType); + Assert.NotNull(mapping.Converter); + } + + /// + /// By contrast, a bare HasColumnType("String") gives the plain string mapping with no + /// converter, so the CLR type does not agree with the property. This is pre-existing behaviour + /// for any CLR type pointed at an unrelated store type, and is why the README tells users to + /// use HasConversion<string>() instead. + /// + [Fact] + public void HasColumnType_string_alone_does_not_compose_a_converter() + { + using var ctx = new StringDateTimeOffsetDbContext(_fixture.ConnectionString); + var property = ctx.Model + .FindEntityType(typeof(StringDateTimeOffsetEntity))! + .FindProperty(nameof(StringDateTimeOffsetEntity.ViaColumnType))!; + + var mapping = property.GetRelationalTypeMapping(); + + Assert.Equal("String", mapping.StoreType); + Assert.Equal(typeof(string), mapping.ClrType); + Assert.Null(mapping.Converter); + } + + // --- daylight saving ---------------------------------------------------- + + /// + /// The hour that repeats when clocks go back is ambiguous, because the driver gives a wall + /// clock and drops the offset. For a zone whose standard offset is zero the instant is still + /// recoverable: the driver only returns when the true + /// offset is not zero, so the zero candidate can be discarded. + /// + [Fact] + public async Task Ambiguous_wall_clock_round_trips_in_a_zero_standard_offset_zone() + { + // On 2026-10-25 the UK goes back from +01:00 to +00:00 at 02:00 local. + // These two distinct instants share the wall clock 01:30 in Europe/London. + var duringDaylightSaving = new DateTimeOffset(2026, 10, 25, 0, 30, 0, TimeSpan.Zero); + var afterDaylightSaving = new DateTimeOffset(2026, 10, 25, 1, 30, 0, TimeSpan.Zero); + + using var writeContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + writeContext.DstEntities.AddRange( + new DateTimeOffsetDstEntity { Id = 1, London = duringDaylightSaving }, + new DateTimeOffsetDstEntity { Id = 2, London = afterDaylightSaving }); + await writeContext.SaveChangesAsync(); + + using var readContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var first = await readContext.DstEntities.SingleAsync(e => e.Id == 1); + var second = await readContext.DstEntities.SingleAsync(e => e.Id == 2); + + // Both instants survive, and they stay distinct. + Assert.Equal(duringDaylightSaving, first.London.ToUniversalTime()); + Assert.Equal(afterDaylightSaving, second.London.ToUniversalTime()); + Assert.NotEqual(first.London.ToUniversalTime(), second.London.ToUniversalTime()); + + // The offsets show which side of the change each one is on. + Assert.Equal(TimeSpan.FromHours(1), first.London.Offset); + Assert.Equal(TimeSpan.Zero, second.London.Offset); + } + + [Fact] + public void Ambiguous_wall_clock_picks_the_non_zero_offset_when_standard_is_zero() + { + // 01:30 on 2026-10-25 is ambiguous in Europe/London: +01:00 or +00:00. + var wallClock = new DateTime(2026, 10, 25, 1, 30, 0, DateTimeKind.Unspecified); + + var result = ClickHouseDateTimeOffsetTypeMapping.ConvertToDateTimeOffset(wallClock, "Europe/London"); + + // Kind=Unspecified means the driver found a non-zero offset, so it must be the daylight one. + Assert.Equal(TimeSpan.FromHours(1), result.Offset); + Assert.Equal(new DateTimeOffset(2026, 10, 25, 0, 30, 0, TimeSpan.Zero), result.ToUniversalTime()); + } + + /// + /// Documents the known limit. In a zone where both candidate offsets are non-zero the offset + /// that the driver dropped cannot be recovered, so the reading keeps standard time. Europe/Paris + /// goes back from +02:00 to +01:00, so an ambiguous wall clock reads as +01:00 either way. + /// + [Fact] + public void Ambiguous_wall_clock_keeps_standard_time_when_both_offsets_are_non_zero() + { + // 02:30 on 2026-10-25 is ambiguous in Europe/Paris: +02:00 or +01:00. + var wallClock = new DateTime(2026, 10, 25, 2, 30, 0, DateTimeKind.Unspecified); + + var result = ClickHouseDateTimeOffsetTypeMapping.ConvertToDateTimeOffset(wallClock, "Europe/Paris"); + + Assert.Equal(TimeSpan.FromHours(1), result.Offset); + } + + [Fact] + public void Unresolvable_declared_timezone_throws_rather_than_guessing() + { + var wallClock = new DateTime(2026, 1, 15, 5, 0, 0, DateTimeKind.Unspecified); + + var ex = Assert.Throws( + () => ClickHouseDateTimeOffsetTypeMapping.ConvertToDateTimeOffset(wallClock, "Not/AZone")); + + Assert.Contains("Not/AZone", ex.Message); + } + + // --- fixed-offset timezones --------------------------------------------- + + /// + /// ClickHouse lets a column declare a fixed UTC offset, spelled Fixed/UTC±HH:MM:SS. + /// No such .NET timezone exists, so TimeZoneInfo.FindSystemTimeZoneById cannot resolve + /// the name however complete the host's timezone data is. Reading it must still work. + /// + [Fact] + public void A_fixed_offset_name_is_not_a_dotnet_timezone() + => Assert.Throws( + () => TimeZoneInfo.FindSystemTimeZoneById("Fixed/UTC+05:30:00")); + + [Theory] + [InlineData("Fixed/UTC+05:30:00", 5, 30)] + [InlineData("Fixed/UTC-07:00:00", -7, 0)] + [InlineData("Fixed/UTC+05:45:00", 5, 45)] + [InlineData("Fixed/UTC-09:30:00", -9, -30)] + [InlineData("Fixed/UTC+00:01:00", 0, 1)] + [InlineData("Fixed/UTC+14:00:00", 14, 0)] + [InlineData("Fixed/UTC-14:00:00", -14, 0)] + public void ConvertToDateTimeOffset_reads_a_fixed_offset_timezone( + string timezone, int expectedHours, int expectedMinutes) + { + var wallClock = new DateTime(2026, 1, 15, 10, 0, 0, DateTimeKind.Unspecified); + + var result = ClickHouseDateTimeOffsetTypeMapping.ConvertToDateTimeOffset(wallClock, timezone); + + var expectedOffset = new TimeSpan(expectedHours, expectedMinutes, 0); + Assert.Equal(expectedOffset, result.Offset); + // The wall clock is kept as given; only the offset is attached. + Assert.Equal(wallClock, result.DateTime); + Assert.Equal(wallClock - expectedOffset, result.UtcDateTime); + } + + /// + /// A fixed offset never changes, so the daylight-saving ambiguity that affects a named zone + /// cannot arise. The same wall clock therefore always gives the same instant. + /// + [Fact] + public void A_fixed_offset_is_never_ambiguous() + { + // In Europe/London this wall clock is the repeated hour when clocks go back. + var ambiguousElsewhere = new DateTime(2026, 10, 25, 1, 30, 0, DateTimeKind.Unspecified); + + var result = ClickHouseDateTimeOffsetTypeMapping.ConvertToDateTimeOffset( + ambiguousElsewhere, "Fixed/UTC+01:00:00"); + + Assert.Equal(TimeSpan.FromHours(1), result.Offset); + Assert.Equal(new DateTimeOffset(2026, 10, 25, 0, 30, 0, TimeSpan.Zero), result.ToUniversalTime()); + } + + /// + /// ClickHouse accepts offsets that cannot hold: it caps the + /// magnitude at 14 hours and requires whole minutes. Both must report the column rather than + /// let the constructor throw naming only the rule it enforces. + /// + [Theory] + [InlineData("Fixed/UTC+15:00:00", "plus or minus 14 hours")] + [InlineData("Fixed/UTC-15:00:00", "plus or minus 14 hours")] + [InlineData("Fixed/UTC+24:00:00", "plus or minus 14 hours")] + [InlineData("Fixed/UTC+00:00:42", "whole minutes")] + [InlineData("Fixed/UTC+05:30:30", "whole minutes")] + [InlineData("Fixed/UTC+09:99:99", "whole minutes")] + public void An_unrepresentable_fixed_offset_reports_the_timezone_and_the_limit( + string timezone, string expectedReason) + { + var wallClock = new DateTime(2026, 1, 15, 10, 0, 0, DateTimeKind.Unspecified); + + var ex = Assert.Throws( + () => ClickHouseDateTimeOffsetTypeMapping.ConvertToDateTimeOffset(wallClock, timezone)); + + Assert.Contains(timezone, ex.Message); + Assert.Contains(expectedReason, ex.Message); + } + + /// + /// A name that only looks like a fixed offset is not one. ClickHouse rejects each of these, so + /// no such column can exist, and the unresolvable-timezone error is the right answer. + /// + [Theory] + [InlineData("Fixed/UTC+05:30")] + [InlineData("fixed/utc+05:30:00")] + [InlineData("Fixed/UTC05:30:00")] + [InlineData("Fixed/UTC+5:30:00")] + public void A_name_that_only_looks_like_a_fixed_offset_is_not_guessed_at(string timezone) + { + var wallClock = new DateTime(2026, 1, 15, 10, 0, 0, DateTimeKind.Unspecified); + + var ex = Assert.Throws( + () => ClickHouseDateTimeOffsetTypeMapping.ConvertToDateTimeOffset(wallClock, timezone)); + + Assert.Contains(timezone, ex.Message); + Assert.Contains("does not know the timezone", ex.Message); + } + + /// + /// ClickHouse does not hold the minutes and seconds fields to 59 — it carries the excess, so + /// Fixed/UTC+05:60:00 is a legal name for +06:00. The driver does not read those + /// names, and gives a UTC wall clock rather than one in the column's timezone, so attaching the + /// offset would move the instant by the whole offset and report nothing. Such a column must + /// report the driver's limit and the spelling to use instead. + /// + [Theory] + [InlineData("Fixed/UTC+05:60:00", "Fixed/UTC+06:00:00")] + [InlineData("Fixed/UTC+05:00:60", "Fixed/UTC+05:01:00")] + [InlineData("Fixed/UTC+00:99:00", "Fixed/UTC+01:39:00")] + [InlineData("Fixed/UTC-05:60:00", "Fixed/UTC-06:00:00")] + public void A_carried_fixed_offset_name_reports_the_driver_limit(string timezone, string suggested) + { + var wallClock = new DateTime(2026, 1, 15, 10, 0, 0, DateTimeKind.Unspecified); + + var ex = Assert.Throws( + () => ClickHouseDateTimeOffsetTypeMapping.ConvertToDateTimeOffset(wallClock, timezone)); + + Assert.Contains(timezone, ex.Message); + Assert.Contains("driver does not support", ex.Message); + // The canonical spelling of the same offset, which the driver does read. + Assert.Contains(suggested, ex.Message); + // It must not be reported as missing host timezone data, which was the old wrong answer. + Assert.DoesNotContain("tzdata", ex.Message); + } + + [Theory] + [InlineData(nameof(DateTimeOffsetFixedEntity.Half), "DateTime64(7, 'Fixed/UTC+05:30:00')", "Fixed/UTC+05:30:00")] + [InlineData(nameof(DateTimeOffsetFixedEntity.Negative), "DateTime64(7, 'Fixed/UTC-07:00:00')", "Fixed/UTC-07:00:00")] + [InlineData(nameof(DateTimeOffsetFixedEntity.Zero), "DateTime64(7, 'Fixed/UTC+00:00:00')", "Fixed/UTC+00:00:00")] + public void A_fixed_offset_store_type_resolves_and_keeps_its_timezone( + string propertyName, string expectedStoreType, string expectedTimezone) + { + using var ctx = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var property = ctx.Model + .FindEntityType(typeof(DateTimeOffsetFixedEntity))! + .FindProperty(propertyName)!; + + var mapping = property.GetRelationalTypeMapping(); + + var typed = Assert.IsType(mapping); + Assert.Equal(expectedStoreType, property.GetColumnType()); + Assert.Equal(expectedTimezone, typed.Timezone); + } + + /// + /// The end-to-end case from the review. Each column declares a different fixed offset, and + /// every instant must survive with the offset the column declares. + /// + [Fact] + public async Task Fixed_offset_columns_round_trip_the_instant_and_report_their_offset() + { + var value = new DateTimeOffset(2026, 3, 21, 14, 25, 36, TimeSpan.Zero).AddTicks(1234567); + + using var writeContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + writeContext.FixedEntities.Add(new DateTimeOffsetFixedEntity + { + Id = 1, + Half = value, + Negative = value, + Quarter = value, + Zero = value, + Seconds = value + }); + await writeContext.SaveChangesAsync(); + + using var readContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var row = await readContext.FixedEntities.SingleAsync(e => e.Id == 1); + + // The instant survives exactly at tick precision through every declared offset. + Assert.Equal(value, row.Half.ToUniversalTime()); + Assert.Equal(value, row.Negative.ToUniversalTime()); + Assert.Equal(value, row.Quarter.ToUniversalTime()); + Assert.Equal(value, row.Zero.ToUniversalTime()); + Assert.Equal(value, row.Seconds.ToUniversalTime()); + + // Each value is rendered at the offset its column declares. + Assert.Equal(new TimeSpan(5, 30, 0), row.Half.Offset); + Assert.Equal(new TimeSpan(-7, 0, 0), row.Negative.Offset); + Assert.Equal(new TimeSpan(5, 45, 0), row.Quarter.Offset); + Assert.Equal(TimeSpan.Zero, row.Zero.Offset); + Assert.Equal(new TimeSpan(0, 1, 0), row.Seconds.Offset); + } + + /// + /// The same case end to end. ClickHouse accepts Fixed/UTC+05:60:00 and creates the + /// column, so the write succeeds; the read must then say what is actually wrong. Reporting + /// missing host timezone data, as it did before, sends the user to install tzdata that + /// would never help. + /// + [Fact] + public async Task A_carried_fixed_offset_column_reports_the_driver_limit_on_read() + { + using var writeContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + writeContext.CarriedEntities.Add(new DateTimeOffsetCarriedEntity + { + Id = 1, + Carried = new DateTimeOffset(2026, 3, 21, 14, 25, 36, TimeSpan.Zero) + }); + await writeContext.SaveChangesAsync(); + + using var readContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + var ex = await Assert.ThrowsAsync( + () => readContext.CarriedEntities.SingleAsync(e => e.Id == 1)); + + Assert.Contains("Fixed/UTC+05:60:00", ex.Message); + Assert.Contains("Fixed/UTC+06:00:00", ex.Message); + Assert.DoesNotContain("tzdata", ex.Message); + } + + /// A filter must still match on the instant, whatever offset the column declares. + [Fact] + public async Task Fixed_offset_column_filters_on_the_instant() + { + var value = new DateTimeOffset(2026, 4, 2, 8, 15, 0, TimeSpan.Zero); + + using var writeContext = new DateTimeOffsetDbContext(_fixture.ConnectionString); + writeContext.FixedEntities.Add(new DateTimeOffsetFixedEntity + { + Id = 2, + Half = value, + Negative = value, + Quarter = value, + Zero = value, + Seconds = value + }); + await writeContext.SaveChangesAsync(); + + using var ctx = new DateTimeOffsetDbContext(_fixture.ConnectionString); + // The same instant written at a third offset again. + var equivalent = new DateTimeOffset(2026, 4, 2, 13, 45, 0, TimeSpan.FromHours(5.5)); + + var found = await ctx.FixedEntities.SingleOrDefaultAsync(e => e.Id == 2 && e.Half == equivalent); + + Assert.NotNull(found); + Assert.Contains("Fixed/UTC+05:30:00", ctx.FixedEntities.Where(e => e.Half == equivalent).ToQueryString()); + } + + // --- conversion helper -------------------------------------------------- + + [Fact] + public void ConvertToDateTimeOffset_reads_utc_kind_as_offset_zero() + { + var value = new DateTime(2026, 1, 15, 5, 0, 0, DateTimeKind.Utc); + + var result = ClickHouseDateTimeOffsetTypeMapping.ConvertToDateTimeOffset(value, "UTC"); + + Assert.Equal(new DateTimeOffset(2026, 1, 15, 5, 0, 0, TimeSpan.Zero), result); + } + + [Fact] + public void ConvertToDateTimeOffset_reads_a_timezone_less_wall_clock_as_utc() + { + var value = new DateTime(2026, 1, 15, 5, 0, 0, DateTimeKind.Unspecified); + + var result = ClickHouseDateTimeOffsetTypeMapping.ConvertToDateTimeOffset(value, null); + + Assert.Equal(new DateTimeOffset(2026, 1, 15, 5, 0, 0, TimeSpan.Zero), result); + } + + [Fact] + public void ConvertToDateTimeOffset_attaches_a_declared_zone_offset() + { + // A Tokyo wall clock of 14:00 is the instant 05:00Z. + var value = new DateTime(2026, 1, 15, 14, 0, 0, DateTimeKind.Unspecified); + + var result = ClickHouseDateTimeOffsetTypeMapping.ConvertToDateTimeOffset(value, "Asia/Tokyo"); + + Assert.Equal(TimeSpan.FromHours(9), result.Offset); + Assert.Equal(new DateTimeOffset(2026, 1, 15, 5, 0, 0, TimeSpan.Zero), result.ToUniversalTime()); + } + + [Fact] + public void ConvertToDateTimeOffset_passes_through_a_datetimeoffset() + { + var value = new DateTimeOffset(2026, 1, 15, 10, 0, 0, TimeSpan.FromHours(5)); + + var result = ClickHouseDateTimeOffsetTypeMapping.ConvertToDateTimeOffset(value, "UTC"); + + Assert.Equal(value, result); + } +}