Map DateTimeOffset to DateTime64 and fix composite component reads (#53)

CHANGELOG.md

@@ -3,8 +3,30 @@ v0.3.1 (Unreleased)
 ### Query translation
 * **`toStartOf*` date-time functions** via `EF.Functions`: `ToStartOfYear`, `ToStartOfQuarter`, `ToStartOfMonth`, `ToStartOfWeek` (with optional week `mode`), `ToStartOfDay`, `ToStartOfHour`, `ToStartOfMinute`, `ToStartOfSecond`, the fixed buckets `ToStartOfFiveMinutes` / `ToStartOfTenMinutes` / `ToStartOfFifteenMinutes`, and the general `ToStartOfInterval(source, value, unit)`. Each maps to the matching ClickHouse function and works in `GROUP BY`. Return types follow ClickHouse: the calendar buckets (`Year`/`Quarter`/`Month`/`Week`) return `Date`, the day/hour/minute buckets return `DateTime`, and `ToStartOfSecond` returns `DateTime64`. All accept `DateTime`/`DateTime64` columns, and the plain truncation functions also accept `DateOnly`; `ToStartOfInterval` requires a `DateTime`/`DateTime64` column on older ClickHouse, which rejects a `DateOnly` (Date/Date32) source with `Illegal type Date32 of 1st argument` for every unit; recent versions accept it. `ToStartOfInterval` takes a `ClickHouseInterval` unit (`Second`…`Year`) and emits `toStartOfInterval(source, toInterval<unit>(value))`; the unit must be a constant so it can be translated. The default `Date`/`DateTime` result types only span 1970–2149/2106, so ClickHouse narrows out-of-range values — enable `enable_extended_results_for_datetime_functions` (e.g. `set_enable_extended_results_for_datetime_functions=1` in the connection string) for range-preserving `Date32`/`DateTime64` results.
 
+### Types
+* **`DateTimeOffset` support.** A `DateTimeOffset` property now maps to `DateTime64(7, 'UTC')` through the new `ClickHouseDateTimeOffsetTypeMapping`. Previously the provider had no mapping for the type, so EF Core fell back to `DateTimeOffsetToStringConverter` and silently produced a `String` column. That fallback broke queries against real `DateTime64` columns (`TYPE_MISMATCH`, because the parameter was declared `String`) and `SaveChanges` could not write the value at all. ([#53](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/53))
+  * The store type is UTC-pinned on purpose. For a timezone-less parameter type such as `DateTime64(7)`, the driver sends a UTC wall clock and the server then reads it in `session_timezone`, which moves the instant when that setting is not UTC.
+  * Precision 7 matches one .NET tick (100 ns), so the round trip is exact and no value is silently truncated.
+  * ClickHouse has no type that stores a UTC offset, so the instant is kept and the offset is not. A value read back carries the offset of the column's declared timezone, and a non-UTC timezone such as `DateTime64(6, 'Asia/Tokyo')` is read correctly. One exception: in a timezone with daylight saving, the repeated hour when clocks go back is ambiguous, because the driver gives a wall clock and drops the offset. The provider recovers the instant when the zone's standard offset is zero (Europe/London); where both candidate offsets are non-zero (Europe/Paris) the reading can be one hour early. The default `'UTC'` store type has no daylight saving and is not affected.
+  * A timezone that the host operating system cannot resolve now throws a clear error instead of silently reading the value as UTC, which would have been wrong by the zone's offset. Minimal Linux images may need the `tzdata` package.
+  * A column may declare a fixed UTC offset rather than a named zone, which ClickHouse spells `Fixed/UTC±HH:MM:SS` — for example `DateTime64(7, 'Fixed/UTC+05:30:00')`. No .NET timezone has such a name, so these are read by parsing the offset out of the name, and they are never ambiguous because a fixed offset does not change. Two kinds of such a column cannot be read, and each reports the timezone and the reason: an offset that `DateTimeOffset` cannot hold, because it caps the magnitude at 14 hours and requires whole minutes (`Fixed/UTC+15:00:00`, `Fixed/UTC+00:00:42`); and a name whose minutes or seconds field is above 59, which ClickHouse carries (`Fixed/UTC+05:60:00` means `+06:00`) but the driver does not read — the message gives the plain spelling to use instead.
+  * **Known limit:** `DateTimeOffset.MinValue` and `MaxValue` only round trip through a column whose timezone offset is zero. Both sit at the edge of the `DateTime` range, and the driver builds a wall clock in the column's timezone to return a value, so a non-zero offset pushes one end outside `DateTime` and the read throws. A named zone is worse than a fixed offset: zones carry a Local Mean Time offset for year 1 (`+09:18:59` for `Asia/Tokyo`), so a value near `MinValue` reads back quietly shifted instead.
+  * `HasPrecision(n)` is honoured for a property with no `HasColumnType`, and keeps the UTC pin — for example `HasPrecision(3)` gives `DateTime64(3, 'UTC')`.
+  * SQL literals carry the offset (`'2026-01-15 10:00:00.1234567+05:00'`), which lands on the same instant whatever timezone the target column declares.
+  * No value converter is used, so the driver receives the `DateTimeOffset` directly on both the query parameter path and the bulk insert path.
+  * **Behaviour change:** a `DateTimeOffset` property that relied on the old `String` column now resolves to `DateTime64(7, 'UTC')`. Add `HasConversion<string>()` to keep the previous shape. Note that `HasColumnType("String")` on its own is not enough — it resolves the plain string mapping with no converter, so the CLR type no longer agrees with the property. Keeping the old shape also makes the property read-only until [#54](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/54) is fixed, because `SaveChanges` does not apply a value converter on the insert path.
+
 ### Bug fixes
 * `ToStartOfWeek` now rejects row-dependent week modes during query translation, and `ToStartOfInterval` likewise rejects row-dependent interval sizes. ClickHouse requires these operands to be constant for the query; literals and captured query parameters remain supported.
+* **Composite columns now convert their components on read.** `Array(T)`, `Map(K, V)`, and `Tuple(...)` read the whole column through `GetValue`, so a component mapping's own read pipeline never ran. Any component whose CLR type differs from the type the driver produces therefore threw `InvalidCastException` — both `DateTimeOffset` and `DateOnly` arrive from the driver as `DateTime`. `DateOnly[]`, `Dictionary<string, DateOnly>` and `Tuple<DateOnly, …>` were affected before `DateTimeOffset` existed as a mapped type.
+  * The composite is now rebuilt component by component, applying both steps EF Core applies to a scalar column: the mapping's data-reader conversion, then its `ValueConverter`. Components that convert through a converter therefore work too — a C# `enum` component (`EnumToStringConverter`), a `List<T>` component (`ListToArrayConverter`), and the collection interfaces (`IList<T>`, `IReadOnlyList<T>`).
+  * A component that needs no conversion keeps the direct cast. Note that the integer mappings do convert, because they widen with `Convert.ToInt32` for aggregates; a runtime fast path returns the driver's value untouched when it is already the target type, so those columns cost nothing beyond the check. That fast path is skipped for a component carrying a `ValueConverter`, which may change the value while keeping the CLR type, so a matching type is no proof that the read is done. An element converter set with `ElementType(el => el.HasConversion(...))` is the reachable case.
+  * Nested composites compose: the element mapping of `Array(Array(DateTime64))` is itself an array mapping, so its conversion runs per element.
+  * `Array(Nullable(T))` also converts, with nulls passing straight through. A NULL in a non-nullable tuple slot now reports which component and type disagree instead of failing with a `NullReferenceException`.
+  * A component that cannot be read as the property's type now reports the store type and both CLR types, rather than EF Core's bare `No coercion operator is defined between types ...`.
+  * **Known limit:** *writing* a component that needs a `ValueConverter` still does not work, because the bulk insert path passes model values to the driver without applying converters ([#54](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/54)). An `enum` inside a composite is written as its raw ordinal. Reading such a column works; populate it outside EF until #54 is fixed.
+* **`Array(Nullable(T))` DDL is no longer double-wrapped.** For a value-type element, the store type came out as `Array(Nullable(Nullable(T)))`, so `EnsureCreated` and migrations produced DDL that ClickHouse rejects with `Nested type Nullable(T) cannot be inside Nullable type`. The component mapping is resolved from a store type that already carries the wrapper, and `HasColumnType(...)` text is kept verbatim, so the nullable-element wrapper added a second one. It now adds the wrapper only when the inner store type does not already have it, including through `LowCardinality(Nullable(T))`. Reference-type elements were never affected.
+* **Composite component mappings now honour the CLR component type.** A single ClickHouse store type can serve more than one CLR type — `DateTime64` serves both `DateTime` and `DateTimeOffset`, and `Date32` serves both `DateTime` and `DateOnly`. Resolving a component from an explicit store type such as `HasColumnType("Array(Nullable(DateTime64(7, 'UTC')))")` always picked the default CLR type, which gave the composite the wrong element type and broke change tracking. Array, Map, and Tuple now pass the component CLR type from the model when they have one.
 * `Sum`/`SumAsync` over a `double` or `float` column no longer throws `InvalidCastException`. EF Core wraps a top-level aggregate so the empty case returns `0`, supplying that fallback as a boxed `Int32` carrying the `Float64`/`Float32` mapping; the literal generators now convert rather than unbox. The `Float32` read path also converts, since ClickHouse widens `sum(Float32)` to `Float64` (which the driver's `GetFloat()` refuses to downcast). ([#46](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/46))
 * **SummingMergeTree with multiple sum columns**: `HasSummingMergeTreeEngine("A", "B")` now generates valid DDL (`SummingMergeTree((A, B))`). Previously it emitted a comma-separated argument list (`SummingMergeTree(A, B)`), which ClickHouse rejects with `NUMBER_OF_ARGUMENTS_DOESNT_MATCH`. Single-column and no-column usage are unaffected.
 

README.md

@@ -67,7 +67,7 @@ public class PageView
 | **Bool** | `Bool` | `bool` |
 | **Strings** | `String`, `FixedString(N)` | `string` |
 | **Enums** | `Enum8(...)`, `Enum16(...)` | `string` or C# `enum` |
-| **Date/time** | `Date`, `Date32`, `DateTime`, `DateTime64(P, 'TZ')` | `DateOnly`, `DateTime` |
+| **Date/time** | `Date`, `Date32`, `DateTime`, `DateTime64(P, 'TZ')` | `DateOnly`, `DateTime`, `DateTimeOffset` (see [below](#datetimeoffset)) |
 | **Time** | `Time`, `Time64(N)` | `TimeSpan` |
 | **UUID** | `UUID` | `Guid` |
 | **Network** | `IPv4`, `IPv6` | `IPAddress` |
@@ -80,6 +80,101 @@ public class PageView
 | **Geographic** | `Point`, `Ring`, `LineString`, `Polygon`, `MultiLineString`, `MultiPolygon`, `Geometry` | `Tuple<double,double>` and arrays thereof; `object` for Geometry |
 | **Wrappers** | `Nullable(T)`, `LowCardinality(T)` | Unwrapped automatically |
 
+### DateTimeOffset
+
+A `DateTimeOffset` property maps to `DateTime64(7, 'UTC')` by default:
+
+```csharp
+public class Reading
+{
+    public long Id { get; set; }
+    public DateTimeOffset RecordedAt { get; set; }   // DateTime64(7, 'UTC')
+}
+```
+
+Three things to know:
+
+**The offset is not kept.** ClickHouse has no type that stores a UTC offset. `DateTime64` holds an
+instant, and a declared timezone only decides how that instant is rendered. A value written with
+any offset is stored as the correct instant, and a value read back carries the offset of the
+column's timezone — `+00:00` for the default store type. With that default store type, comparisons
+and ordering are instant-correct, so these two values match the same row:
+
+```csharp
+// The same instant, written two ways.
+var a = new DateTimeOffset(2026, 1, 15, 10, 0, 0, TimeSpan.FromHours(5));
+var b = new DateTimeOffset(2026, 1, 15,  5, 0, 0, TimeSpan.Zero);
+```
+
+If you must keep the offset, store it yourself in a second column alongside a `DateTime`. To keep
+the whole value as text, ask for the conversion explicitly with `HasConversion<string>()` — note
+that `HasColumnType("String")` on its own is not enough, because it adds no converter. Such a
+column is read-only for now: `SaveChanges` cannot write any property that has a value converter
+([#54](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/54)).
+
+**Precision 7 makes the round trip exact.** One .NET tick is 100 ns, which is precision 7, so a
+stored value never comes back truncated. Precision 7 also covers the full `DateTimeOffset` range,
+which lets you use `DateTimeOffset.MinValue` and `MaxValue` as open-ended range limits on the
+default store type. Keep those two sentinels to a column whose timezone offset is zero, such as the
+default `'UTC'`. Both sit at the edge of the `DateTime` range, and the driver has to build a wall
+clock in the column's timezone to return a value, so any non-zero offset pushes one end outside
+`DateTime`: reading `MaxValue` from a `DateTime64(7, 'Asia/Tokyo')` column throws. Choose a smaller
+precision if you prefer, but be aware that it discards the digits below it:
+
+```csharp
+b.Property(e => e.RecordedAt).HasColumnType("DateTime64(3, 'UTC')");   // milliseconds
+b.Property(e => e.RecordedAt).HasPrecision(3);                         // the same thing
+```
+
+Do not go above precision 7. .NET cannot represent more than 7 fractional digits, and the driver
+overflows an `Int64` at precision 8 or 9 for dates far from the epoch, which corrupts the value
+without an error.
+
+**Keep `'UTC'` in the store type** unless you have a reason to change it. For a timezone-less type
+such as `DateTime64(7)`, the server reads the query parameter in its `session_timezone`, which moves
+the instant when that setting is not UTC.
+
+A column that declares a different timezone, for example `DateTime64(6, 'Asia/Tokyo')`, is read
+correctly and returns that zone's offset. Two limits apply to such a column:
+
+- The host operating system must know the timezone, or the read throws. Minimal Linux images may
+  need the `tzdata` package.
+- In a zone with daylight saving, the repeated hour when clocks go back is ambiguous, because the
+  driver gives a wall clock and drops the offset. The provider recovers the instant where the zone's
+  standard offset is zero, such as `Europe/London`. Where both candidate offsets are non-zero, such
+  as `Europe/Paris`, the value can read back one hour early.
+- Dates at the far ends of the `DateTimeOffset` range do not survive, as described above. A named
+  zone is worse than a fixed offset here: zones carry a Local Mean Time offset for year 1
+  (`+09:18:59` for `Asia/Tokyo`), so a value near `DateTimeOffset.MinValue` reads back quietly
+  shifted rather than reporting an error.
+
+A column can also declare a fixed UTC offset instead of a named zone. ClickHouse spells this
+`Fixed/UTC±HH:MM:SS`, with two digits in every field — the server rejects `Fixed/UTC+5:30:00` and
+`Fixed/UTC+05:30`:
+
+```csharp
+b.Property(e => e.RecordedAt).HasColumnType("DateTime64(7, 'Fixed/UTC+05:30:00')");
+```
+
+Neither limit above applies here: the host needs no timezone data, and a fixed offset is never
+ambiguous. Two limits of its own do, and each reports the timezone and the reason rather than
+failing obscurely:
+
+- `DateTimeOffset` holds an offset only within plus or minus 14 hours, and only in whole minutes.
+  ClickHouse accepts more, so `Fixed/UTC+15:00:00` and `Fixed/UTC+00:00:42` cannot be read into one.
+- ClickHouse does not hold the minutes and seconds fields to 59 — it carries the excess, so
+  `Fixed/UTC+05:60:00` is a legal name for `+06:00`. The driver reads only the plain spelling, so
+  declare that offset as `Fixed/UTC+06:00:00`.
+
+Map such a column as `DateTime` if you cannot change how it is declared.
+
+`DateTimeOffset` also composes into the collection types, so `DateTimeOffset[]`,
+`List<DateTimeOffset>`, `Dictionary<string, DateTimeOffset>` and `Tuple<DateTimeOffset, …>` all
+round trip.
+
+`DateTimeOffset` members such as `.Year` and `.UtcDateTime` do not translate to SQL yet. This
+applies to `DateTime` as well — see [#55](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/55).
+
 ## Current Status
 
 This provider is in active development. It supports **LINQ queries**, **inserts**, **table engine configuration**, and **migrations** — you can define ClickHouse tables with engine-specific settings, create them via `dotnet ef migrations` or `EnsureCreated`, query with LINQ, and write data via `SaveChanges`.
@@ -272,7 +367,7 @@ Configure ClickHouse table engines, ordering, partitioning, and more via EF Core
 ```csharp
 modelBuilder.Entity<SensorReading>(b =>
 {
-    b.HasKey(e => e.Id);
+    b.HasKey(e => e.Id); // becomes ORDER BY (the ClickHouse primary key) when no explicit ORDER BY is set
     b.Property(e => e.Temperature).HasCodec("Delta, ZSTD");
     b.Property(e => e.Location).HasColumnComment("Installation site");
     b.HasIndex(e => e.Timestamp)
@@ -298,6 +393,8 @@ modelBuilder.Entity<SensorReading>(b =>
 
 **Default behavior:** If no engine is configured, the provider defaults to `MergeTree` with the EF primary key as `ORDER BY`.
 
+**Primary key vs sorting key:** In ClickHouse the `ORDER BY` (sorting key) *is* the primary key, so `HasKey` alone is sufficient — it becomes `ORDER BY`. Only use `.WithPrimaryKey(...)` when you need the primary index to differ from the sort order (e.g. a `SummingMergeTree`/`AggregatingMergeTree` rollup with a long `ORDER BY` but a narrow index). ClickHouse requires the primary key to be a prefix of the `ORDER BY` columns.
+
 ### Migrations
 
 The provider supports `dotnet ef migrations` for creating and applying migrations:

src/EFCore.ClickHouse/Metadata/Builders/ClickHouseEngineBuilder.cs

@@ -16,6 +16,10 @@ protected ClickHouseEngineBuilder(IMutableEntityType entityType, string engineNa
         entityType.SetEngine(engineName);
     }
 
+    /// <summary>
+    /// Sets the table's sorting key (<c>ORDER BY</c>). In ClickHouse the sorting key also serves as the
+    /// primary key unless an explicit one is set via <see cref="WithPrimaryKey"/>.
+    /// </summary>
     public ClickHouseEngineBuilder WithOrderBy(params string[] columns)
     {
         ArgumentNullException.ThrowIfNull(columns);
@@ -30,6 +34,11 @@ public ClickHouseEngineBuilder WithPartitionBy(params string[] columns)
         return this;
     }
 
+    /// <summary>
+    /// Sets an explicit primary key (<c>PRIMARY KEY</c>) distinct from the sorting key. Only needed when the
+    /// primary index should differ from <c>ORDER BY</c>; otherwise the sorting key is used as the primary key.
+    /// ClickHouse requires these columns to be a prefix of the <see cref="WithOrderBy"/> columns.
+    /// </summary>
     public ClickHouseEngineBuilder WithPrimaryKey(params string[] columns)
     {
         ArgumentNullException.ThrowIfNull(columns);