From 6ca769be21e061e1d92b3477226bb2d7c8aae49b Mon Sep 17 00:00:00 2001 From: Alex Soffronow-Pagonidis Date: Fri, 14 Aug 2026 15:53:19 +0200 Subject: [PATCH 1/2] Translate standard DateTime members and methods (#55) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The provider registered no date/time member translator, so only a direct comparison worked: every member and method threw "The LINQ expression ... could not be translated". Add one shared translator, which serves DateTime and DateOnly because the ClickHouse function is the same for each. Components map to the to* extraction functions. Those return UInt8/UInt16, which the provider's integer mappings already widen on read. .DayOfWeek maps to toDayOfWeek(x, 2). Week mode 2 agrees with System.DayOfWeek exactly, so no arithmetic correction is applied. The result carries a number-backed enum mapping, because this provider maps a C# enum to a ClickHouse string and that mapping would otherwise render x.DayOfWeek == DayOfWeek.Sunday as a comparison against 'Sunday'. .TimeOfDay maps to toTime64(x, 7); precision 7 is one .NET tick, so the fraction survives, which toTime would drop. AddYears and AddMonths take an int, so they map straight onto addYears and addMonths, which clamp the day of month the way .NET does. The other Add* methods take a double, which .NET scales to whole ticks. The matching ClickHouse function takes a whole number of its own unit and discards the rest, so addDays(x, 1.5) would add only one day. A constant is therefore folded to ticks and expressed in the coarsest unit that holds it exactly: the natural function when possible, otherwise addMilliseconds. Preferring the natural function keeps the source's store type and keeps Date/Date32 columns working, which addMilliseconds rejects. A sub-millisecond offset, a non-constant offset, and a value outside the DateTime range are left untranslated rather than rounded to fit. addNanoseconds would express a tick exactly, but promotes the result to DateTime64(9), whose Int64 nanosecond count cannot span the DateTime64 range — that would trade a rounding error for a silently wrong date. Server-side rounding was rejected too: ClickHouse round() is banker's rounding, so it disagrees with .NET. An untranslated call still gives the correct value through client evaluation in a projection, and reports a reason in a predicate. Also report a clear reason for arithmetic on two date/time values. ClickHouse has no operator that matches the .NET result: one date minus another gives a TimeSpan while dateDiff counts whole units, Time64 subtraction gives a Decimal of seconds, and a date plus a TimeSpan is rejected outright. These used to fail with an internal cast or coercion error naming CLR types the user never wrote. Reporting the reason also restores client evaluation in a projection, where the .NET result is correct. DateTimeOffset follows in the next commit, once the store mapping this branch is stacked on is in place. The Northwind GroupJoin_aggregate_anonymous_key_selectors2 query now translates, so its "not translatable" override is removed. Co-Authored-By: Claude --- CHANGELOG.md | 14 + README.md | 58 ++ .../ClickHouseDateTimeMemberTranslator.cs | 243 +++++++ .../ClickHouseDateTimeMethodTranslator.cs | 166 ++++- .../ClickHouseMemberTranslatorProvider.cs | 1 + ...ickHouseSqlTranslatingExpressionVisitor.cs | 55 ++ .../Query/NorthwindJoinQueryClickHouseTest.cs | 5 - .../DateTimeMemberTranslationTests.cs | 602 ++++++++++++++++++ 8 files changed, 1137 insertions(+), 7 deletions(-) create mode 100644 src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMemberTranslator.cs create mode 100644 test/EFCore.ClickHouse.Tests/DateTimeMemberTranslationTests.cs diff --git a/CHANGELOG.md b/CHANGELOG.md index c8c8183..9e6ccb3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,19 @@ v0.3.1 (Unreleased) ### Query translation * **`toStartOf*` date-time functions** via `EF.Functions`: `ToStartOfYear`, `ToStartOfQuarter`, `ToStartOfMonth`, `ToStartOfWeek` (with optional week `mode`), `ToStartOfDay`, `ToStartOfHour`, `ToStartOfMinute`, `ToStartOfSecond`, the fixed buckets `ToStartOfFiveMinutes` / `ToStartOfTenMinutes` / `ToStartOfFifteenMinutes`, and the general `ToStartOfInterval(source, value, unit)`. Each maps to the matching ClickHouse function and works in `GROUP BY`. Return types follow ClickHouse: the calendar buckets (`Year`/`Quarter`/`Month`/`Week`) return `Date`, the day/hour/minute buckets return `DateTime`, and `ToStartOfSecond` returns `DateTime64`. All accept `DateTime`/`DateTime64` columns, and the plain truncation functions also accept `DateOnly`; `ToStartOfInterval` requires a `DateTime`/`DateTime64` column on older ClickHouse, which rejects a `DateOnly` (Date/Date32) source with `Illegal type Date32 of 1st argument` for every unit; recent versions accept it. `ToStartOfInterval` takes a `ClickHouseInterval` unit (`Second`…`Year`) and emits `toStartOfInterval(source, toInterval(value))`; the unit must be a constant so it can be translated. The default `Date`/`DateTime` result types only span 1970–2149/2106, so ClickHouse narrows out-of-range values — enable `enable_extended_results_for_datetime_functions` (e.g. `set_enable_extended_results_for_datetime_functions=1` in the connection string) for range-preserving `Date32`/`DateTime64` results. +* **Standard `DateTime` members and methods now translate to SQL.** Previously the provider registered no date/time member translator, so only a direct comparison worked and every member threw `The LINQ expression ... could not be translated`. One shared translator serves `DateTime` and `DateOnly`, because the ClickHouse function is the same for each. ([#55](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/55)) + * **Components** — `.Year` → `toYear`, `.Month` → `toMonth`, `.Day` → `toDayOfMonth`, `.Hour` → `toHour`, `.Minute` → `toMinute`, `.Second` → `toSecond`, `.Millisecond` → `toMillisecond`, `.DayOfYear` → `toDayOfYear`. These ClickHouse functions return `UInt8`/`UInt16`, which the provider's integer mappings widen to `int` on read. `DateOnly` gets the date components only, matching the members it declares. + * **`.DayOfWeek`** → `toDayOfWeek(x, 2)`. Week mode 2 agrees with `System.DayOfWeek` exactly (Sunday 0 … Saturday 6), so no arithmetic correction is applied — the default mode 0 starts the week on Monday, which is why the mode argument is always sent. The result carries a number-backed enum mapping, because this provider maps a C# `enum` to a ClickHouse string and that mapping would otherwise render `x.DayOfWeek == DayOfWeek.Sunday` as a comparison against `'Sunday'`. + * **`.Date`** → `toStartOfDay`, which keeps the timezone of the source. Note that `toStartOfDay` returns a `DateTime`, whose range is 1970–2106, and ClickHouse **wraps** a value outside that window rather than reporting it — so `.Date` on a `DateTime64` column holding a pre-1970 date reads back wrong. Enable `enable_extended_results_for_datetime_functions` (for example `set_enable_extended_results_for_datetime_functions=1` in the connection string) to get a range-preserving `DateTime64` result. This is the same caveat that already applies to `EF.Functions.ToStartOfDay`. + * **`.TimeOfDay`** → `toTime64(x, 7)`; precision 7 is one .NET tick, so no part of the value is lost (`toTime` would drop the fraction). + * **`DateTime.UtcNow`** → `now64(7, 'UTC')`, **`DateTime.Now`** → `now64(7)` and **`DateTime.Today`** → `toStartOfDay(now())`. `today()` is not used for `.Today` because it returns a `Date`, whereas the member's type is `DateTime`. + * **`.AddYears(n)`** → `addYears` and **`.AddMonths(n)`** → `addMonths`. Both take an `int` in .NET, and ClickHouse clamps the day of month the same way .NET does, so `2026-01-31` plus one month gives `2026-02-28` in both. + * **`.AddDays`/`.AddHours`/`.AddMinutes`/`.AddSeconds`/`.AddMilliseconds`** take a `double` in .NET, which .NET scales to whole **ticks** (100 ns), rounding half away from zero — so `AddSeconds(0.1234567)` adds exactly 1 234 567 ticks. The matching ClickHouse function takes a whole number of its own unit and discards the rest, so `addDays(x, 1.5)` would add only one day. A constant argument is therefore folded to ticks during translation and then expressed in the coarsest unit that holds it exactly: a whole number of the unit emits the natural function (`addDays(x, 1)`), and otherwise `addMilliseconds` carries the exact count (`AddDays(1.5)` → `addMilliseconds(x, 129600000)`). Preferring the natural function keeps the store type of the source and keeps `Date`/`Date32` columns working, since `addMilliseconds` rejects those outright. + * A **sub-millisecond** offset, a **non-constant** offset, and a value **outside the `DateTime` range** are deliberately left untranslated rather than rounded to fit. Milliseconds are as fine as the translation goes, because `addNanoseconds` would express a tick exactly but promotes the result to `DateTime64(9)`, whose Int64 nanosecond count cannot span the `DateTime64` range — that would trade a rounding error for a silently wrong date. An untranslated call still gives the correct .NET value through client evaluation in a projection, and reports a clear reason in a predicate. `DateOnly.AddDays` takes an `int`, so it always emits `addDays`. + * A previously-unsupported Northwind query, `GroupJoin_aggregate_anonymous_key_selectors2`, now passes as a result of these translations; its provider-specific "not translatable" override is removed. + * Not yet translated: `.Ticks`, `.AddTicks`, and the `.Microsecond`/`.Nanosecond` members. `DateTimeOffset` is not covered yet either — it has no store mapping until [#53](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/53), so translating its members would silently drop the offset. The translator is shaped to take the CLR type as a parameter, so it gains `DateTimeOffset` with that mapping. The ClickHouse-specific functions such as `dateDiff` and `dateTrunc` are tracked in [#58](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/58). + * **Behaviour change:** `DateTime.Now` and `DateTime.Today` in a *projection* used to be evaluated on the client; they now read the **server** clock. The value therefore follows the server's timezone rather than the client's, and comes back with `DateTimeKind.Unspecified` instead of `Local`. Use `DateTime.UtcNow` for an instant that does not depend on server configuration. In a predicate all three were untranslatable before, so nothing changes there. + ### Types * **`DateTimeOffset` support.** A `DateTimeOffset` property now maps to `DateTime64(7, 'UTC')` through the new `ClickHouseDateTimeOffsetTypeMapping`. Previously the provider had no mapping for the type, so EF Core fell back to `DateTimeOffsetToStringConverter` and silently produced a `String` column. That fallback broke queries against real `DateTime64` columns (`TYPE_MISMATCH`, because the parameter was declared `String`) and `SaveChanges` could not write the value at all. ([#53](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/53)) * The store type is UTC-pinned on purpose. For a timezone-less parameter type such as `DateTime64(7)`, the driver sends a UTC wall clock and the server then reads it in `session_timezone`, which moves the instant when that setting is not UTC. @@ -17,6 +30,7 @@ v0.3.1 (Unreleased) * **Behaviour change:** a `DateTimeOffset` property that relied on the old `String` column now resolves to `DateTime64(7, 'UTC')`. Add `HasConversion()` to keep the previous shape. Note that `HasColumnType("String")` on its own is not enough — it resolves the plain string mapping with no converter, so the CLR type no longer agrees with the property. Keeping the old shape also makes the property read-only until [#54](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/54) is fixed, because `SaveChanges` does not apply a value converter on the insert path. ### Bug fixes +* **Subtracting one date/time value from another no longer fails with an internal error.** `dt1 - dt2` gives a `TimeSpan`, which ClickHouse has no operator for — `dateDiff` returns a count of whole units instead. The expression used to reach type-mapping inference and fail with an `InvalidCastException` or a bare `No coercion operator is defined between types ...`, both of which name CLR types the user never wrote. The subtraction is now reported as not translatable, with the reason attached. In a projection EF Core can therefore fall back to the client and return the correct `TimeSpan`; in a predicate, where no fallback exists, the message explains why and what to do instead. ([#55](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/55)) * `ToStartOfWeek` now rejects row-dependent week modes during query translation, and `ToStartOfInterval` likewise rejects row-dependent interval sizes. ClickHouse requires these operands to be constant for the query; literals and captured query parameters remain supported. * **Composite columns now convert their components on read.** `Array(T)`, `Map(K, V)`, and `Tuple(...)` read the whole column through `GetValue`, so a component mapping's own read pipeline never ran. Any component whose CLR type differs from the type the driver produces therefore threw `InvalidCastException` — both `DateTimeOffset` and `DateOnly` arrive from the driver as `DateTime`. `DateOnly[]`, `Dictionary` and `Tuple` were affected before `DateTimeOffset` existed as a mapped type. * The composite is now rebuilt component by component, applying both steps EF Core applies to a scalar column: the mapping's data-reader conversion, then its `ValueConverter`. Components that convert through a converter therefore work too — a C# `enum` component (`EnumToStringConverter`), a `List` component (`ListToArrayConverter`), and the collection interfaces (`IList`, `IReadOnlyList`). diff --git a/README.md b/README.md index 6d365aa..7bd83d1 100644 --- a/README.md +++ b/README.md @@ -217,6 +217,64 @@ ClickHouse returns `NULL` from a scalar subquery that matches no rows, where sta ### Date/Time Functions +#### Standard members and methods + +The standard .NET date/time members translate to ClickHouse functions, for both `DateTime` and `DateOnly`: + +| .NET | ClickHouse | +| --- | --- | +| `.Year` `.Month` `.Day` | `toYear` `toMonth` `toDayOfMonth` | +| `.Hour` `.Minute` `.Second` `.Millisecond` | `toHour` `toMinute` `toSecond` `toMillisecond` | +| `.DayOfYear` | `toDayOfYear` | +| `.DayOfWeek` | `toDayOfWeek(x, 2)` | +| `.Date` | `toStartOfDay` | +| `.TimeOfDay` | `toTime64(x, 7)` | +| `.AddYears(n)` `.AddMonths(n)` | `addYears` `addMonths` | +| `.AddDays(n)` `.AddHours(n)` `.AddMinutes(n)` `.AddSeconds(n)` `.AddMilliseconds(n)` | `addDays` `addHours` … (see below) | +| `DateTime.UtcNow` | `now64(7, 'UTC')` | +| `DateTime.Now` | `now64(7)` | +| `DateTime.Today` | `toStartOfDay(now())` | + +```csharp +// Runs entirely on the server +var busyHours = await ctx.Events + .Where(e => e.Timestamp.Year == 2026 && e.Timestamp.DayOfWeek == DayOfWeek.Sunday) + .GroupBy(e => e.Timestamp.Hour) + .Select(g => new { Hour = g.Key, Count = g.Count() }) + .ToListAsync(); + +var recent = await ctx.Events + .Where(e => e.Timestamp > DateTime.UtcNow.AddDays(-7)) + .ToListAsync(); +``` + +`DateOnly` gets the date components only, which are the members it declares. `DateTimeOffset` is not covered yet, because it has no store mapping — see [#53](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/53). + +Five points are worth knowing: + +**`.DayOfWeek` needs no correction.** ClickHouse week mode 2 agrees with `System.DayOfWeek` exactly — Sunday is 0 through to Saturday 6 — so the value is used as it comes back. The mode argument is always sent, because the default mode starts the week on Monday. + +**`.Now` and `.Today` read the server clock**, so they follow the *server's* timezone, not the client's, and they come back with `DateTimeKind.Unspecified`. Use `DateTime.UtcNow` when you need an instant that does not depend on server configuration. + +**`.Date` narrows outside 1970–2106.** `toStartOfDay` returns a `DateTime`, and ClickHouse *wraps* a value outside that window instead of reporting it — so `.Date` on a `DateTime64` column holding a pre-1970 date reads back wrong. Enable [`enable_extended_results_for_datetime_functions`](https://clickhouse.com/docs/operations/settings/settings#enable_extended_results_for_datetime_functions) — for example `set_enable_extended_results_for_datetime_functions=1` in the connection string — to get a range-preserving `DateTime64` result. + +**A fractional `Add*` argument is exact or is not translated.** `AddDays` and the other time-based methods take a `double`, which .NET scales to whole *ticks* (100 ns), so `AddSeconds(0.1234567)` adds exactly 1 234 567 ticks. The ClickHouse `addDays` function takes a whole number of days and discards the rest, so it cannot be used directly. A constant argument is folded to ticks and then expressed in the coarsest unit that holds it exactly: + +```csharp +e.Timestamp.AddDays(1) // addDays(ts, 1) +e.Timestamp.AddDays(1.5) // addMilliseconds(ts, 129600000) +e.Timestamp.AddMilliseconds(0.5) // not translated — 5 000 ticks is below millisecond resolution +e.Timestamp.AddDays(offsetVariable) // not translated — cannot be checked for exactness +``` + +The natural function keeps the column's store type, and it is the only form that works on a `Date`/`Date32` column — ClickHouse rejects `addMilliseconds` on those. Anything the provider cannot express exactly is left untranslated rather than rounded to fit, so a projection still gives the correct .NET value through client evaluation, while a predicate reports why. `DateOnly.AddDays` takes an `int`, so it always emits `addDays`. + +**Arithmetic on two date/time values is not translated.** `dt1 - dt2` and `time1 - time2` give a `TimeSpan`, and `date + timeSpan` mixes types ClickHouse rejects; `dateDiff` returns a count of whole units, and `Time64` subtraction returns a decimal number of seconds. In a projection EF Core reads the columns and does the arithmetic on the client, which gives the correct result. In a predicate there is no client fallback, so the query fails with an explanation. + +Not yet translated: `.Ticks`, `.AddTicks`, and the `.Microsecond`/`.Nanosecond` members. + +#### `toStartOf*` bucketing + The ClickHouse `toStartOf*` family is exposed through `EF.Functions`, so you can bucket and truncate timestamps directly in queries, including in `GROUP BY`: `ToStartOfYear`, `ToStartOfQuarter`, `ToStartOfMonth`, `ToStartOfWeek` (with an optional ClickHouse week `mode`), `ToStartOfDay`, `ToStartOfHour`, `ToStartOfMinute`, `ToStartOfSecond`, `ToStartOfFiveMinutes`, `ToStartOfTenMinutes`, `ToStartOfFifteenMinutes`, and the general `ToStartOfInterval(source, value, unit)`. diff --git a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMemberTranslator.cs b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMemberTranslator.cs new file mode 100644 index 0000000..4c5f01b --- /dev/null +++ b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMemberTranslator.cs @@ -0,0 +1,243 @@ +using System.Reflection; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Diagnostics; +using Microsoft.EntityFrameworkCore.Query; +using Microsoft.EntityFrameworkCore.Query.SqlExpressions; +using Microsoft.EntityFrameworkCore.Storage; +using Microsoft.EntityFrameworkCore.Storage.ValueConversion; + +namespace ClickHouse.EntityFrameworkCore.Query.ExpressionTranslators.Internal; + +/// +/// Translates the standard date/time members of , +/// and to ClickHouse functions. +/// +/// +/// One class serves all three CLR types, because the ClickHouse function is the same for each: the +/// to* extraction functions accept Date, Date32, DateTime and +/// DateTime64 alike. registers the members that the +/// given type declares, so gets the date components only. +/// +public class ClickHouseDateTimeMemberTranslator : IMemberTranslator +{ + /// + /// Week mode 2 makes toDayOfWeek agree with exactly: Sunday is 0 + /// through to Saturday is 6. The default mode 0 starts the week on Monday, so the argument is + /// required and no arithmetic correction is needed. + /// + private const byte SundayFirstWeekMode = 2; + + /// + /// One tick is 100 ns, which is Time64 precision 7, and one .NET + /// tick is also the resolution of now64(7). Asking for that precision keeps + /// exact, whereas toTime drops the fraction. + /// + private const int TickPrecision = 7; + + /// Members that map to a ClickHouse function taking the source value alone. + private static readonly Dictionary ComponentFunctions = []; + + private static readonly HashSet DayOfWeekMembers = []; + private static readonly HashSet DateMembers = []; + private static readonly HashSet TimeOfDayMembers = []; + + /// Static members that read the server clock. + private static readonly Dictionary ServerClockMembers = []; + + private readonly ISqlExpressionFactory _sqlExpressionFactory; + private readonly IRelationalTypeMappingSource _typeMappingSource; + + /// How a server-clock member is built: a function name, and whether it pins UTC. + private enum ServerClock + { + /// now64(7, 'UTC') — an exact instant, independent of server settings. + UtcNow, + + /// now64(7) — the server's local clock, in its configured timezone. + LocalNow, + + /// toStartOfDay(now()) — midnight today on the server's local clock. + LocalToday + } + + /// + /// The mapping given to a result. Built once, because it composes a + /// converter over the Int32 mapping and nothing about it varies per translation. + /// + private readonly RelationalTypeMapping? _dayOfWeekMapping; + + static ClickHouseDateTimeMemberTranslator() + { + RegisterInstanceMembers(typeof(DateTime), hasTimeComponents: true); + RegisterInstanceMembers(typeof(DateOnly), hasTimeComponents: false); + + ServerClockMembers.Add(Property(typeof(DateTime), nameof(DateTime.UtcNow)), ServerClock.UtcNow); + ServerClockMembers.Add(Property(typeof(DateTime), nameof(DateTime.Now)), ServerClock.LocalNow); + ServerClockMembers.Add(Property(typeof(DateTime), nameof(DateTime.Today)), ServerClock.LocalToday); + + // DateTimeOffset is deliberately absent, even though every function here would serve it. The + // provider has no DateTimeOffset store mapping yet, so such a property resolves to String: the + // extraction functions then fail on the server, and addDays silently drops the offset and the + // sub-second part. Register it here together with the mapping (issue #53). + } + + public ClickHouseDateTimeMemberTranslator( + ISqlExpressionFactory sqlExpressionFactory, + IRelationalTypeMappingSource typeMappingSource) + { + _sqlExpressionFactory = sqlExpressionFactory; + _typeMappingSource = typeMappingSource; + + // DayOfWeek is an enum, and this provider maps a C# enum to a ClickHouse string. That mapping + // would render the other side of a comparison as 'Sunday' against a number, so the result + // carries a number-backed enum mapping instead. + _dayOfWeekMapping = typeMappingSource.FindMapping(typeof(int)) is { } intMapping + ? (RelationalTypeMapping)intMapping.WithComposedConverter(new EnumToNumberConverter()) + : null; + } + + public SqlExpression? Translate( + SqlExpression? instance, + MemberInfo member, + Type returnType, + IDiagnosticsLogger logger) + { + if (instance is null) + { + return TranslateServerClock(member, returnType); + } + + // toYear and friends return UInt8/UInt16, which the provider's integer mappings widen on read. + if (ComponentFunctions.TryGetValue(member, out var function)) + { + return _sqlExpressionFactory.Function( + name: function, + arguments: [instance], + nullable: true, + argumentsPropagateNullability: [true], + returnType: returnType, + typeMapping: _typeMappingSource.FindMapping(returnType)); + } + + if (DayOfWeekMembers.Contains(member)) + { + return _sqlExpressionFactory.Function( + name: "toDayOfWeek", + arguments: [instance, _sqlExpressionFactory.Constant(SundayFirstWeekMode)], + nullable: true, + // Only the source propagates nullability; the week mode is a constant. + argumentsPropagateNullability: [true, false], + returnType: returnType, + typeMapping: _dayOfWeekMapping); + } + + // toStartOfDay keeps the timezone of the source, which is what DateTime.Date means for a + // column: midnight on the same calendar day that the column renders. + // + // Note that toStartOfDay returns a DateTime, whose range is 1970-2106. ClickHouse wraps a value + // outside that window rather than reporting it, so a DateTime64 column holding a date before + // 1970 reads back wrong unless the session enables + // enable_extended_results_for_datetime_functions, which widens the result to DateTime64. This + // matches EF.Functions.ToStartOfDay and is documented alongside it. + if (DateMembers.Contains(member)) + { + return _sqlExpressionFactory.Function( + name: "toStartOfDay", + arguments: [instance], + nullable: true, + argumentsPropagateNullability: [true], + returnType: returnType, + // Reuse the source's mapping only when it describes the member's own CLR type. A mapping + // for a different type cannot be coerced during materialization. + typeMapping: instance.TypeMapping?.ClrType == returnType + ? instance.TypeMapping + : _typeMappingSource.FindMapping(returnType)); + } + + if (TimeOfDayMembers.Contains(member)) + { + return _sqlExpressionFactory.Function( + name: "toTime64", + arguments: [instance, _sqlExpressionFactory.Constant(TickPrecision)], + nullable: true, + argumentsPropagateNullability: [true, false], + returnType: returnType, + typeMapping: _typeMappingSource.FindMapping($"Time64({TickPrecision})")); + } + + return null; + } + + private SqlExpression? TranslateServerClock(MemberInfo member, Type returnType) + { + if (!ServerClockMembers.TryGetValue(member, out var clock)) + { + return null; + } + + var mapping = _typeMappingSource.FindMapping(returnType); + + // DateTime.Today is midnight today. today() returns a Date, whereas the member's type is + // DateTime, so truncate the clock value instead and keep a DateTime store type. + if (clock == ServerClock.LocalToday) + { + return _sqlExpressionFactory.Function( + name: "toStartOfDay", + arguments: [Niladic("now", returnType, mapping)], + nullable: false, + argumentsPropagateNullability: [false], + returnType: returnType, + typeMapping: mapping); + } + + List arguments = [_sqlExpressionFactory.Constant(TickPrecision)]; + if (clock == ServerClock.UtcNow) + { + arguments.Add(_sqlExpressionFactory.Constant("UTC")); + } + + return _sqlExpressionFactory.Function( + name: "now64", + arguments: arguments, + nullable: false, + argumentsPropagateNullability: arguments.Select(_ => false), + returnType: returnType, + typeMapping: mapping); + } + + private SqlExpression Niladic(string name, Type returnType, RelationalTypeMapping? typeMapping) + => _sqlExpressionFactory.Function( + name: name, + arguments: [], + nullable: false, + argumentsPropagateNullability: [], + returnType: returnType, + typeMapping: typeMapping); + + private static void RegisterInstanceMembers(Type type, bool hasTimeComponents) + { + ComponentFunctions.Add(Property(type, nameof(DateTime.Year)), "toYear"); + ComponentFunctions.Add(Property(type, nameof(DateTime.Month)), "toMonth"); + ComponentFunctions.Add(Property(type, nameof(DateTime.Day)), "toDayOfMonth"); + ComponentFunctions.Add(Property(type, nameof(DateTime.DayOfYear)), "toDayOfYear"); + + DayOfWeekMembers.Add(Property(type, nameof(DateTime.DayOfWeek))); + + if (!hasTimeComponents) + { + return; + } + + ComponentFunctions.Add(Property(type, nameof(DateTime.Hour)), "toHour"); + ComponentFunctions.Add(Property(type, nameof(DateTime.Minute)), "toMinute"); + ComponentFunctions.Add(Property(type, nameof(DateTime.Second)), "toSecond"); + ComponentFunctions.Add(Property(type, nameof(DateTime.Millisecond)), "toMillisecond"); + + DateMembers.Add(Property(type, nameof(DateTime.Date))); + TimeOfDayMembers.Add(Property(type, nameof(DateTime.TimeOfDay))); + } + + private static MemberInfo Property(Type type, string name) + => type.GetProperty(name) + ?? throw new InvalidOperationException($"Property {type.Name}.{name} was not found."); +} diff --git a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs index fa31725..fb88ea0 100644 --- a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs +++ b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs @@ -9,13 +9,28 @@ namespace ClickHouse.EntityFrameworkCore.Query.ExpressionTranslators.Internal; /// -/// Translates the EF.Functions.ToStartOf* extension methods -/// () to their ClickHouse SQL functions. +/// Translates date/time method calls to ClickHouse SQL functions: the +/// EF.Functions.ToStartOf* extension methods +/// (), and the standard Add* methods of +/// , and . /// public class ClickHouseDateTimeMethodTranslator : IMethodCallTranslator { private readonly ISqlExpressionFactory _sqlExpressionFactory; + /// + /// Add* methods that take an , keyed to their ClickHouse function. An + /// integer count needs no rounding, so these always translate. + /// + private static readonly Dictionary IntegralAddMethods = []; + + /// + /// Add* methods that take a , keyed to their ClickHouse function and + /// the tick length of the method's own unit. Keeping the two families in separate dictionaries + /// makes a unit with no fixed tick length (a month, a year) unrepresentable here. + /// + private static readonly Dictionary FractionalAddMethods = []; + /// /// Maps the generic method definitions that take only the source value (and, for /// ToStartOfWeek(source, mode), an extra scalar argument) directly to a ClickHouse function name. @@ -105,8 +120,49 @@ void RegisterSourceOnly(string methodName, string sqlFunction) && parameters[2].ParameterType == typeof(int) && parameters[3].ParameterType == typeof(ClickHouseInterval); }) ?? throw new InvalidOperationException("Method ToStartOfInterval with strict signature not found."); + + RegisterAddMethods(typeof(DateTime), hasTimeComponents: true); + + // DateOnly declares no time-based Add* method, and its AddDays takes an int. + RegisterAddMethods(typeof(DateOnly), hasTimeComponents: false); + + // DateTimeOffset is deliberately absent: the provider has no DateTimeOffset store mapping yet, + // so such a property resolves to String and these functions would either fail on the server or + // silently drop the offset. Add it here together with the mapping (issue #53). } + /// + /// Registers the Add* methods that declares. + /// + /// + /// AddYears and AddMonths take an on every supported type, so they + /// map straight onto addYears/addMonths. The time-based methods take a + /// on , which needs the exactness check that + /// applies. On , AddDays takes an + /// instead, so it is registered as integral. + /// + private static void RegisterAddMethods(Type type, bool hasTimeComponents) + { + IntegralAddMethods.Add(Method(type, nameof(DateTime.AddYears), typeof(int)), "addYears"); + IntegralAddMethods.Add(Method(type, nameof(DateTime.AddMonths), typeof(int)), "addMonths"); + + if (!hasTimeComponents) + { + IntegralAddMethods.Add(Method(type, nameof(DateOnly.AddDays), typeof(int)), "addDays"); + return; + } + + FractionalAddMethods.Add(Method(type, nameof(DateTime.AddDays), typeof(double)), ("addDays", TimeSpan.TicksPerDay)); + FractionalAddMethods.Add(Method(type, nameof(DateTime.AddHours), typeof(double)), ("addHours", TimeSpan.TicksPerHour)); + FractionalAddMethods.Add(Method(type, nameof(DateTime.AddMinutes), typeof(double)), ("addMinutes", TimeSpan.TicksPerMinute)); + FractionalAddMethods.Add(Method(type, nameof(DateTime.AddSeconds), typeof(double)), ("addSeconds", TimeSpan.TicksPerSecond)); + FractionalAddMethods.Add(Method(type, nameof(DateTime.AddMilliseconds), typeof(double)), ("addMilliseconds", TimeSpan.TicksPerMillisecond)); + } + + private static MethodInfo Method(Type type, string name, Type argumentType) + => type.GetRuntimeMethod(name, [argumentType]) + ?? throw new InvalidOperationException($"Method {type.Name}.{name}({argumentType.Name}) was not found."); + public ClickHouseDateTimeMethodTranslator(ISqlExpressionFactory sqlExpressionFactory) { _sqlExpressionFactory = sqlExpressionFactory; @@ -118,6 +174,20 @@ public ClickHouseDateTimeMethodTranslator(ISqlExpressionFactory sqlExpressionFac IReadOnlyList arguments, IDiagnosticsLogger logger) { + if (instance is not null) + { + if (IntegralAddMethods.TryGetValue(method, out var integralFunction)) + { + return AddFunction(integralFunction, instance, arguments[0], method.ReturnType); + } + + if (FractionalAddMethods.TryGetValue(method, out var fractional)) + { + return TranslateFractionalAdd( + instance, fractional.Function, fractional.TicksPerUnit, arguments[0], method.ReturnType); + } + } + var genericMethod = method.IsGenericMethod ? method.GetGenericMethodDefinition() : method; if (SupportedMethods.TryGetValue(genericMethod, out var function)) @@ -180,6 +250,98 @@ public ClickHouseDateTimeMethodTranslator(ISqlExpressionFactory sqlExpressionFac return null; } + /// + /// Translates one Add* call whose .NET argument is a , or returns + /// when no exact translation exists. + /// + /// + /// + /// .NET scales the argument to whole ticks and rounds half away from zero, so + /// AddSeconds(0.1234567) adds exactly 1 234 567 ticks. The matching ClickHouse function takes + /// a whole number of its own unit and discards the rest, so addDays(x, 1.5) would add one day. + /// The two agree only when the tick count divides exactly into the function's unit. + /// + /// + /// A constant is therefore folded to ticks here and then expressed in the coarsest unit that holds + /// it exactly. The natural unit is preferred, and not only for readability: it keeps the store type + /// of the source, and addMilliseconds rejects a Date or Date32 source outright + /// (ILLEGAL_TYPE_OF_ARGUMENT). + /// + /// + /// Everything else is left untranslated on purpose, rather than rounded to fit. This covers a + /// sub-millisecond offset and any value that is not a constant. Milliseconds are as fine as this + /// goes: addNanoseconds would express a tick exactly but promotes the result to + /// DateTime64(9), whose Int64 nanosecond count cannot span the DateTime64 range, which + /// would trade a rounding error for a silently wrong date. An untranslated call still gives the + /// correct .NET value through client evaluation in a projection, and reports a clear reason in a + /// predicate. + /// + /// + private SqlExpression? TranslateFractionalAdd( + SqlExpression instance, + string function, + long ticksPerUnit, + SqlExpression value, + Type returnType) + { + if (value is not SqlConstantExpression { Value: double constantValue }) + { + return null; + } + + if (TicksFor(constantValue, ticksPerUnit) is not { } ticks) + { + return null; + } + + if (ticks % ticksPerUnit == 0) + { + return AddFunction(function, instance, _sqlExpressionFactory.Constant(ticks / ticksPerUnit), returnType); + } + + if (ticks % TimeSpan.TicksPerMillisecond != 0) + { + return null; + } + + return AddFunction( + "addMilliseconds", + instance, + _sqlExpressionFactory.Constant(ticks / TimeSpan.TicksPerMillisecond), + returnType); + } + + /// + /// The tick count that .NET would add, or when .NET would not produce one. + /// + /// + /// Mirrors 's scaling: multiply by the unit's tick length, then + /// round half away from zero. A value that cannot represent is rejected, so + /// the comes from .NET during client evaluation instead of + /// from a wrapped Int64 on the server, which ClickHouse reports as a decimal overflow — or, for the + /// larger magnitudes, does not report at all. + /// + private static long? TicksFor(double value, long ticksPerUnit) + { + if (double.IsNaN(value) || double.IsInfinity(value)) + { + return null; + } + + var scaled = value * ticksPerUnit + (value >= 0 ? 0.5 : -0.5); + + return double.Abs(scaled) > DateTime.MaxValue.Ticks ? null : (long)scaled; + } + + private SqlExpression AddFunction(string function, SqlExpression instance, SqlExpression value, Type returnType) + => _sqlExpressionFactory.Function( + name: function, + arguments: [instance, value], + nullable: true, + argumentsPropagateNullability: [true, true], + returnType: returnType, + typeMapping: instance.TypeMapping); + private static bool IsQueryConstant(SqlExpression expression) => expression is SqlConstantExpression or SqlParameterExpression; diff --git a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseMemberTranslatorProvider.cs b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseMemberTranslatorProvider.cs index c97a393..aab320b 100644 --- a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseMemberTranslatorProvider.cs +++ b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseMemberTranslatorProvider.cs @@ -16,6 +16,7 @@ public ClickHouseMemberTranslatorProvider( [ new ClickHouseArrayMethodTranslator(sqlExpressionFactory, typeMappingSource), new ClickHouseStringMethodTranslator(sqlExpressionFactory), + new ClickHouseDateTimeMemberTranslator(sqlExpressionFactory, typeMappingSource), ]); } } diff --git a/src/EFCore.ClickHouse/Query/Internal/ClickHouseSqlTranslatingExpressionVisitor.cs b/src/EFCore.ClickHouse/Query/Internal/ClickHouseSqlTranslatingExpressionVisitor.cs index dc66ad3..d1d04a9 100644 --- a/src/EFCore.ClickHouse/Query/Internal/ClickHouseSqlTranslatingExpressionVisitor.cs +++ b/src/EFCore.ClickHouse/Query/Internal/ClickHouseSqlTranslatingExpressionVisitor.cs @@ -44,4 +44,59 @@ protected override Expression VisitMethodCall(MethodCallExpression methodCallExp => _arrayLinqTranslator.TryTranslate(methodCallExpression, out var translated) ? translated : base.VisitMethodCall(methodCallExpression); + + /// + /// Reports a clear reason when two date/time values are added or subtracted. + /// + /// + /// + /// ClickHouse has no operator for any of these shapes, and each one fails differently: + /// + /// + /// + /// One date minus another gives a in .NET, whereas dateDiff + /// returns a count of whole units. + /// + /// + /// One time of day minus another gives a in .NET, whereas ClickHouse + /// Time64 subtraction gives a Decimal number of seconds. + /// + /// + /// A date plus or minus a keeps the date type in .NET, whereas ClickHouse + /// rejects the mixed operands outright (Illegal types ... of arguments of function plus). + /// + /// + /// + /// Left alone, each of these reaches type-mapping inference or the server and fails with an internal + /// cast error or raw SQL error that names types the user never wrote. Reporting the reason here turns + /// that into EF Core's normal "could not be translated" message with an explanation attached — which + /// also restores client evaluation in a projection, where the .NET result is correct. + /// + /// + protected override Expression VisitBinary(BinaryExpression binaryExpression) + { + if (binaryExpression.NodeType is ExpressionType.Add or ExpressionType.Subtract + && IsDateOrTimeType(binaryExpression.Left.Type) + && IsDateOrTimeType(binaryExpression.Right.Type)) + { + AddTranslationErrorDetails( + "Arithmetic on two date or time values is not supported, because ClickHouse has no " + + "operator that matches the .NET result. Compare the two values directly, or project " + + "them and do the arithmetic on the client."); + + return QueryCompilationContext.NotTranslatedExpression; + } + + return base.VisitBinary(binaryExpression); + } + + private static bool IsDateOrTimeType(Type type) + { + var unwrapped = Nullable.GetUnderlyingType(type) ?? type; + return unwrapped == typeof(DateTime) + || unwrapped == typeof(DateTimeOffset) + || unwrapped == typeof(DateOnly) + || unwrapped == typeof(TimeSpan) + || unwrapped == typeof(TimeOnly); + } } diff --git a/test/EFCore.ClickHouse.FunctionalTests/Query/NorthwindJoinQueryClickHouseTest.cs b/test/EFCore.ClickHouse.FunctionalTests/Query/NorthwindJoinQueryClickHouseTest.cs index 6aaffeb..7c26242 100644 --- a/test/EFCore.ClickHouse.FunctionalTests/Query/NorthwindJoinQueryClickHouseTest.cs +++ b/test/EFCore.ClickHouse.FunctionalTests/Query/NorthwindJoinQueryClickHouseTest.cs @@ -36,11 +36,6 @@ public override Task Take_in_collection_projection_with_FirstOrDefault_on_top_le public override Task SelectMany_with_client_eval_with_constructor(bool async) => AssertUnsupported(() => base.SelectMany_with_client_eval_with_constructor(async)); - // Complex LINQ pattern not translatable - public override Task GroupJoin_aggregate_anonymous_key_selectors2(bool async) - => Assert.ThrowsAsync( - () => base.GroupJoin_aggregate_anonymous_key_selectors2(async)); - private static async Task AssertUnsupported(Func test) => await Assert.ThrowsAsync(test); } diff --git a/test/EFCore.ClickHouse.Tests/DateTimeMemberTranslationTests.cs b/test/EFCore.ClickHouse.Tests/DateTimeMemberTranslationTests.cs new file mode 100644 index 0000000..8b5be82 --- /dev/null +++ b/test/EFCore.ClickHouse.Tests/DateTimeMemberTranslationTests.cs @@ -0,0 +1,602 @@ +using Microsoft.EntityFrameworkCore; +using Xunit; + +namespace EFCore.ClickHouse.Tests; + +public class DateTimeMemberEntity +{ + public long Id { get; set; } + + /// Mapped to ClickHouse DateTime, which holds whole seconds only. + public DateTime Timestamp { get; set; } + + /// Mapped to ClickHouse DateTime64(7). Precision 7 is one .NET tick. + public DateTime Timestamp64 { get; set; } + + /// Mapped to ClickHouse Date32. + public DateOnly Date { get; set; } +} + +public class DateTimeMemberDbContext : DbContext +{ + public DbSet Events => Set(); + + private readonly string _connectionString; + + public DateTimeMemberDbContext(string connectionString) + { + _connectionString = connectionString; + } + + protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder) + { + optionsBuilder.UseClickHouse(_connectionString); + } + + protected override void OnModelCreating(ModelBuilder modelBuilder) + { + modelBuilder.Entity(entity => + { + entity.ToTable("datetime_member_test"); + entity.HasKey(e => e.Id); + entity.Property(e => e.Id).HasColumnName("id"); + entity.Property(e => e.Timestamp).HasColumnName("ts"); + entity.Property(e => e.Timestamp64).HasColumnName("ts64").HasColumnType("DateTime64(7)"); + entity.Property(e => e.Date).HasColumnName("d"); + }); + } +} + +public class DateTimeMemberFixture : IAsyncLifetime +{ + public string ConnectionString { get; private set; } = string.Empty; + + /// + /// Row 1's instant: Sunday 2026-08-16 13:47:32.1234567. A Sunday on purpose — ClickHouse + /// toDayOfWeek gives Sunday 7 in its default mode and 0 in mode 2, so a Sunday is the day + /// that proves the mode argument reaches the server. + /// + public static readonly DateTime Instant = new DateTime(2026, 8, 16, 13, 47, 32).AddTicks(1_234_567); + + /// Row 1's time of day, to one tick. + public static readonly TimeSpan InstantTimeOfDay = TimeSpan.FromTicks(496_521_234_567); + + public async Task InitializeAsync() + { + ConnectionString = await SharedContainer.GetConnectionStringAsync(); + + using var connection = new global::ClickHouse.Driver.ADO.ClickHouseConnection(ConnectionString); + await connection.OpenAsync(); + + using var createCmd = connection.CreateCommand(); + createCmd.CommandText = """ + CREATE TABLE datetime_member_test ( + id Int64, + ts DateTime, + ts64 DateTime64(7), + d Date32 + ) ENGINE = MergeTree() + ORDER BY id + """; + await createCmd.ExecuteNonQueryAsync(); + + using var insertCmd = connection.CreateCommand(); + // Row 2 is the last day of a month, so AddMonths and AddYears have a day to clamp. + insertCmd.CommandText = """ + INSERT INTO datetime_member_test (id, ts, ts64, d) VALUES + (1, '2026-08-16 13:47:32', '2026-08-16 13:47:32.1234567', '2026-08-16'), + (2, '2026-01-31 00:00:00', '2026-01-31 00:00:00.0000000', '2026-01-31') + """; + await insertCmd.ExecuteNonQueryAsync(); + } + + public Task DisposeAsync() => Task.CompletedTask; +} + +public class DateTimeMemberTranslationTest : IClassFixture +{ + private readonly DateTimeMemberFixture _fixture; + + public DateTimeMemberTranslationTest(DateTimeMemberFixture fixture) + { + _fixture = fixture; + } + + private async Task SelectSingleAsync( + Func, IQueryable> selector, + long id = 1) + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + return await selector(context.Events.AsNoTracking().Where(e => e.Id == id)).SingleAsync(); + } + + private async Task> WhereIdsAsync( + System.Linq.Expressions.Expression> predicate) + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + return await context.Events.AsNoTracking().Where(predicate) + .OrderBy(e => e.Id).Select(e => e.Id).ToListAsync(); + } + + // ---------------------------------------------------------------- components + + [Fact] + public async Task Year_translates_to_toYear() + => Assert.Equal(2026, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.Year))); + + [Fact] + public async Task Month_translates_to_toMonth() + => Assert.Equal(8, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.Month))); + + [Fact] + public async Task Day_translates_to_toDayOfMonth() + => Assert.Equal(16, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.Day))); + + [Fact] + public async Task Hour_translates_to_toHour() + => Assert.Equal(13, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.Hour))); + + [Fact] + public async Task Minute_translates_to_toMinute() + => Assert.Equal(47, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.Minute))); + + [Fact] + public async Task Second_translates_to_toSecond() + => Assert.Equal(32, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.Second))); + + [Fact] + public async Task Millisecond_translates_to_toMillisecond() + => Assert.Equal(123, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.Millisecond))); + + [Fact] + public async Task DayOfYear_translates_to_toDayOfYear() + => Assert.Equal(228, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.DayOfYear))); + + [Fact] + public async Task Components_agree_with_dotnet_on_a_second_precision_column() + { + var result = await SelectSingleAsync(q => q.Select(e => new { e.Timestamp.Year, e.Timestamp.Hour, e.Timestamp.Second })); + + Assert.Equal(DateTimeMemberFixture.Instant.Year, result.Year); + Assert.Equal(DateTimeMemberFixture.Instant.Hour, result.Hour); + Assert.Equal(DateTimeMemberFixture.Instant.Second, result.Second); + } + + // ---------------------------------------------------------------- DayOfWeek + + [Fact] + public async Task DayOfWeek_projects_the_dotnet_value() + { + var result = await SelectSingleAsync(q => q.Select(e => e.Timestamp64.DayOfWeek)); + + // .NET DayOfWeek.Sunday is 0. ClickHouse mode 2 agrees; the default mode would give 7. + Assert.Equal(DayOfWeek.Sunday, result); + } + + [Fact] + public async Task DayOfWeek_compares_against_a_dotnet_constant() + { + // The provider maps a C# enum to a ClickHouse string, so this is the test that proves the + // constant renders as a number rather than as 'Sunday'. + Assert.Equal([1L], await WhereIdsAsync(e => e.Timestamp64.DayOfWeek == DayOfWeek.Sunday)); + } + + [Fact] + public async Task DayOfWeek_of_a_Saturday_is_six() + { + // Row 2 is 2026-01-31, a Saturday. + Assert.Equal(DayOfWeek.Saturday, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.DayOfWeek), id: 2)); + } + + // ---------------------------------------------------------------- Date / TimeOfDay + + [Fact] + public async Task Date_translates_to_toStartOfDay() + => Assert.Equal(new DateTime(2026, 8, 16), await SelectSingleAsync(q => q.Select(e => e.Timestamp64.Date))); + + [Fact] + public async Task TimeOfDay_keeps_tick_precision() + { + var result = await SelectSingleAsync(q => q.Select(e => e.Timestamp64.TimeOfDay)); + + // toTime would drop the fraction; toTime64(x, 7) keeps every tick. + Assert.Equal(DateTimeMemberFixture.InstantTimeOfDay, result); + } + + // ---------------------------------------------------------------- DateOnly + + [Fact] + public async Task DateOnly_components_translate() + { + var result = await SelectSingleAsync(q => q.Select(e => new + { + e.Date.Year, + e.Date.Month, + e.Date.Day, + e.Date.DayOfYear, + e.Date.DayOfWeek + })); + + Assert.Equal(2026, result.Year); + Assert.Equal(8, result.Month); + Assert.Equal(16, result.Day); + Assert.Equal(228, result.DayOfYear); + Assert.Equal(DayOfWeek.Sunday, result.DayOfWeek); + } + + // ---------------------------------------------------------------- Add* + + [Fact] + public async Task AddYears_translates_to_addYears() + => Assert.Equal( + DateTimeMemberFixture.Instant.AddYears(1), + await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddYears(1)))); + + [Fact] + public async Task AddMonths_translates_to_addMonths() + => Assert.Equal( + DateTimeMemberFixture.Instant.AddMonths(1), + await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddMonths(1)))); + + [Fact] + public async Task AddMonths_clamps_the_day_like_dotnet() + { + // Row 2 is 2026-01-31, so one month lands on 2026-02-28 in both systems. + var result = await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddMonths(1)), id: 2); + + Assert.Equal(new DateTime(2026, 2, 28), result); + Assert.Equal(new DateTime(2026, 1, 31).AddMonths(1), result); + } + + [Fact] + public async Task AddDays_with_a_whole_number_translates_to_addDays() + => Assert.Equal( + DateTimeMemberFixture.Instant.AddDays(1), + await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddDays(1)))); + + [Fact] + public async Task AddDays_with_a_negative_whole_number_translates_to_addDays() + => Assert.Equal( + DateTimeMemberFixture.Instant.AddDays(-1), + await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddDays(-1)))); + + [Fact] + public async Task AddDays_with_a_fraction_keeps_dotnet_semantics() + { + // addDays(x, 1.5) would discard the fraction and add one day. The translation must not. + var result = await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddDays(1.5))); + + Assert.Equal(DateTimeMemberFixture.Instant.AddDays(1.5), result); + Assert.Equal(new DateTime(2026, 8, 18, 1, 47, 32).AddTicks(1_234_567), result); + } + + [Fact] + public async Task AddHours_translates() + => Assert.Equal( + DateTimeMemberFixture.Instant.AddHours(2), + await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddHours(2)))); + + [Fact] + public async Task AddMinutes_with_a_fraction_keeps_dotnet_semantics() + => Assert.Equal( + DateTimeMemberFixture.Instant.AddMinutes(0.5), + await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddMinutes(0.5)))); + + [Fact] + public async Task AddSeconds_with_a_fraction_keeps_dotnet_semantics() + => Assert.Equal( + DateTimeMemberFixture.Instant.AddSeconds(1.5), + await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddSeconds(1.5)))); + + [Fact] + public async Task AddMilliseconds_translates() + => Assert.Equal( + DateTimeMemberFixture.Instant.AddMilliseconds(1), + await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddMilliseconds(1)))); + + [Fact] + public async Task AddDays_on_a_Date32_column_keeps_the_date_store_type() + { + // DateOnly.AddDays takes an int, so this must use addDays — addMilliseconds rejects a Date32. + var result = await SelectSingleAsync(q => q.Select(e => e.Date.AddDays(1))); + + Assert.Equal(new DateOnly(2026, 8, 17), result); + } + + [Fact] + public async Task AddMonths_on_a_Date32_column_clamps_like_dotnet() + => Assert.Equal( + new DateOnly(2026, 2, 28), + await SelectSingleAsync(q => q.Select(e => e.Date.AddMonths(1)), id: 2)); + + [Fact] + public async Task AddSeconds_below_millisecond_resolution_keeps_dotnet_semantics() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64.AddSeconds(0.1234567)); + + // .NET adds exactly 1 234 567 ticks. No ClickHouse unit holds that without promoting the result + // to DateTime64(9), so the call is left untranslated and the client supplies the exact value. + Assert.DoesNotContain("addSeconds", query.ToQueryString()); + Assert.DoesNotContain("addMilliseconds", query.ToQueryString()); + + var result = await query.SingleAsync(); + Assert.Equal(DateTimeMemberFixture.Instant.AddSeconds(0.1234567), result); + Assert.Equal(DateTimeMemberFixture.Instant.AddTicks(1_234_567), result); + } + + [Fact] + public async Task AddMilliseconds_below_millisecond_resolution_keeps_dotnet_semantics() + { + // .NET rounds to the nearest tick, so this adds 5 000 ticks — not 0 ms and not 1 ms. + var result = await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddMilliseconds(0.5))); + + Assert.Equal(DateTimeMemberFixture.Instant.AddMilliseconds(0.5), result); + Assert.Equal(DateTimeMemberFixture.Instant.AddTicks(5_000), result); + } + + [Fact] + public async Task AddDays_with_a_parameter_keeps_dotnet_semantics() + { + var days = 1.5; + + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64.AddDays(days)); + + // A parameter cannot be checked for exactness, so it is not translated. Rounding it on the server + // would disagree with .NET, because ClickHouse round() is banker's rounding. + Assert.DoesNotContain("addDays", query.ToQueryString()); + Assert.Equal(DateTimeMemberFixture.Instant.AddDays(1.5), await query.SingleAsync()); + } + + [Fact] + public async Task AddDays_with_a_parameter_in_a_predicate_reports_a_reason() + { + var days = 1.5; + + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Timestamp64.AddDays(days) > e.Timestamp); + + await Assert.ThrowsAsync(() => query.ToListAsync()); + } + + [Fact] + public async Task AddDays_beyond_the_dotnet_range_throws_the_dotnet_exception() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64.AddDays(1e30)); + + // Folding this would wrap to Int64.MaxValue and give a server decimal-overflow error, or worse a + // silently wrong date. Left untranslated, .NET raises its own exception on the client. + await Assert.ThrowsAsync(() => query.SingleAsync()); + } + + [Fact] + public async Task Add_composes_with_a_component_member() + => Assert.Equal(2027, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddYears(1).Year))); + + // ---------------------------------------------------------------- server clock + + [Fact] + public async Task UtcNow_runs_on_the_server() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + + // Both bounds are offset by a century so the outcome does not depend on the day the suite runs. + var query = context.Events.AsNoTracking().Where(e => e.Timestamp64 < DateTime.UtcNow.AddYears(100)) + .Select(e => e.Id); + + // If EF Core evaluated DateTime.UtcNow on the client, the SQL would carry a literal instead. + Assert.Contains("now64", query.ToQueryString()); + Assert.Equal([1L, 2L], await query.OrderBy(id => id).ToListAsync()); + + var none = await context.Events.AsNoTracking() + .Where(e => e.Timestamp64 < DateTime.UtcNow.AddYears(-100)) + .Select(e => e.Id).ToListAsync(); + + Assert.Empty(none); + } + + [Fact] + public async Task Today_runs_on_the_server() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Timestamp64 < DateTime.Today) + .Select(e => e.Id); + + Assert.Contains("toStartOfDay", query.ToQueryString()); + Assert.Contains("now()", query.ToQueryString()); + + // Execute it too: a SQL-shape assertion alone would not catch the server rejecting the call. + var beforeToday = await query.ToListAsync(); + + // Row 2 is 2026-01-31, which is before today on any day this suite can run. + Assert.Contains(2L, beforeToday); + } + + // ---------------------------------------------------------------- subtraction + + [Fact] + public async Task Subtracting_two_date_times_in_a_projection_now_works() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + + // Previously this failed with an opaque cast/coercion error from type-mapping inference. The + // subtraction is now reported as not translatable, so EF Core reads both columns and subtracts + // on the client, which is the correct .NET result. + var result = await context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64 - e.Timestamp).SingleAsync(); + + Assert.Equal(TimeSpan.FromTicks(1_234_567), result); + } + + [Fact] + public async Task Subtracting_two_date_times_in_a_predicate_reports_a_clear_reason() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + + // A predicate cannot fall back to the client, so this is where the reason must surface. + var query = context.Events.AsNoTracking() + .Where(e => e.Timestamp64 - e.Timestamp > TimeSpan.Zero); + + var exception = await Assert.ThrowsAsync(() => query.ToListAsync()); + + Assert.Contains("Arithmetic on two date or time values", exception.Message); + } + + [Fact] + public async Task Subtracting_two_times_of_day_in_a_projection_works() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + + // ClickHouse Time64 subtraction gives a Decimal of seconds, not a TimeSpan, so this must stay + // on the client rather than emit SQL that materializes into the wrong type. + var result = await context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64.TimeOfDay - e.Timestamp.TimeOfDay).SingleAsync(); + + Assert.Equal(TimeSpan.FromTicks(1_234_567), result); + } + + [Fact] + public async Task Adding_a_time_of_day_to_a_date_in_a_projection_works() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + + // ClickHouse rejects DateTime + Time64 outright, so this must stay on the client too. + var result = await context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64.Date + e.Timestamp64.TimeOfDay).SingleAsync(); + + Assert.Equal(DateTimeMemberFixture.Instant, result); + } + + [Fact] + public async Task Adding_a_TimeSpan_to_a_date_in_a_projection_works() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + + var result = await context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64 + TimeSpan.FromHours(1)).SingleAsync(); + + Assert.Equal(DateTimeMemberFixture.Instant.AddHours(1), result); + } +} + +public class DateTimeMemberTranslationOfflineTest +{ + private sealed class OfflineContext : DbContext + { + public DbSet Events => Set(); + + protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder) + { + optionsBuilder.UseClickHouse("Host=localhost;Protocol=http;Port=8123;Database=test"); + } + + protected override void OnModelCreating(ModelBuilder modelBuilder) + { + modelBuilder.Entity(entity => + { + entity.ToTable("datetime_member_test"); + entity.HasKey(e => e.Id); + entity.Property(e => e.Timestamp64).HasColumnType("DateTime64(7)"); + }); + } + } + + private static string Sql(Func, IQueryable> selector) + { + using var context = new OfflineContext(); + return selector(context.Events).ToQueryString(); + } + + [Fact] + public void Year_emits_toYear() + => Assert.Contains("toYear(", Sql(q => q.Select(e => e.Timestamp64.Year))); + + [Fact] + public void Day_emits_toDayOfMonth() + => Assert.Contains("toDayOfMonth(", Sql(q => q.Select(e => e.Timestamp64.Day))); + + [Fact] + public void DayOfWeek_emits_week_mode_two() + { + var sql = Sql(q => q.Select(e => e.Timestamp64.DayOfWeek)); + + // Assert the mode argument itself — a bare "2" would also match toDayOfWeek(x, 12). + Assert.Contains(", 2)", sql); + Assert.Contains("toDayOfWeek(", sql); + } + + [Fact] + public void DayOfWeek_comparison_emits_a_number_not_a_string() + { + var sql = Sql(q => q.Where(e => e.Timestamp64.DayOfWeek == DayOfWeek.Sunday).Select(e => e.Id)); + + Assert.DoesNotContain("'Sunday'", sql); + Assert.Contains("= 0", sql); + } + + [Fact] + public void TimeOfDay_emits_toTime64_with_tick_precision() + => Assert.Contains("toTime64(", Sql(q => q.Select(e => e.Timestamp64.TimeOfDay))); + + [Fact] + public void AddDays_with_a_whole_number_emits_addDays() + { + var sql = Sql(q => q.Select(e => e.Timestamp64.AddDays(1))); + + Assert.Contains("addDays(", sql); + Assert.DoesNotContain("addMilliseconds", sql); + } + + [Fact] + public void AddDays_with_a_fraction_emits_addMilliseconds() + { + var sql = Sql(q => q.Select(e => e.Timestamp64.AddDays(1.5))); + + // 1.5 days is exactly 129 600 000 ms, folded at translation time. + Assert.Contains("addMilliseconds(", sql); + Assert.Contains("129600000", sql); + } + + [Fact] + public void AddMilliseconds_below_millisecond_resolution_emits_no_add_function() + { + // 0.5 ms is 5 000 ticks, which no ClickHouse unit holds exactly without promoting to + // DateTime64(9), so the call must not be translated. + var sql = Sql(q => q.Select(e => e.Timestamp64.AddMilliseconds(0.5))); + + Assert.DoesNotContain("addMilliseconds", sql); + Assert.DoesNotContain("addNanoseconds", sql); + } + + [Fact] + public void AddSeconds_with_a_whole_number_of_milliseconds_emits_addMilliseconds() + { + // 1.5 s is 1 500 ms exactly, so it is still translatable — just not in seconds. + var sql = Sql(q => q.Select(e => e.Timestamp64.AddSeconds(1.5))); + + Assert.Contains("addMilliseconds(", sql); + Assert.Contains("1500", sql); + } + + [Fact] + public void AddHours_with_a_whole_number_emits_addHours() + => Assert.Contains("addHours(", Sql(q => q.Select(e => e.Timestamp64.AddHours(3)))); + + [Fact] + public void UtcNow_emits_a_utc_pinned_now64() + => Assert.Contains("now64(7, 'UTC')", Sql(q => q.Where(e => e.Timestamp64 < DateTime.UtcNow).Select(e => e.Id))); + + [Fact] + public void Now_emits_a_timezone_less_now64() + { + var sql = Sql(q => q.Where(e => e.Timestamp64 < DateTime.Now).Select(e => e.Id)); + + Assert.Contains("now64(7)", sql); + Assert.DoesNotContain("'UTC'", sql); + } +} From 0389e05cdc3c7d9ce6ab86335c6edaf60db97db6 Mon Sep 17 00:00:00 2001 From: Alex Soffronow-Pagonidis Date: Fri, 14 Aug 2026 15:59:01 +0200 Subject: [PATCH 2/2] Extend the date/time translators to DateTimeOffset (#55) The shared translator already takes the CLR type as a parameter, so serving DateTimeOffset is a registration. It was held back only because the type had no store mapping: such a property resolved to String, where the extraction functions fail on the server and addDays silently drops both the offset and the sub-second part. The mapping from #53, which this branch is stacked on, removes that obstacle. The result is in the timezone the column declares, which the store type pins to UTC. That agrees with .NET, because a value read back from such a column carries the +00:00 offset, so .Hour and .Date describe the same instant on both sides. DateTimeOffset.Now translates to the same UTC-pinned now64 as UtcNow, since a DateTimeOffset is an instant. Co-Authored-By: Claude --- CHANGELOG.md | 5 +- README.md | 9 +- .../ClickHouseDateTimeMemberTranslator.cs | 16 ++- .../ClickHouseDateTimeMethodTranslator.cs | 5 +- .../DateTimeMemberTranslationTests.cs | 98 ++++++++++++++++++- 5 files changed, 115 insertions(+), 18 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9e6ccb3..ab46f63 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,7 +3,7 @@ v0.3.1 (Unreleased) ### Query translation * **`toStartOf*` date-time functions** via `EF.Functions`: `ToStartOfYear`, `ToStartOfQuarter`, `ToStartOfMonth`, `ToStartOfWeek` (with optional week `mode`), `ToStartOfDay`, `ToStartOfHour`, `ToStartOfMinute`, `ToStartOfSecond`, the fixed buckets `ToStartOfFiveMinutes` / `ToStartOfTenMinutes` / `ToStartOfFifteenMinutes`, and the general `ToStartOfInterval(source, value, unit)`. Each maps to the matching ClickHouse function and works in `GROUP BY`. Return types follow ClickHouse: the calendar buckets (`Year`/`Quarter`/`Month`/`Week`) return `Date`, the day/hour/minute buckets return `DateTime`, and `ToStartOfSecond` returns `DateTime64`. All accept `DateTime`/`DateTime64` columns, and the plain truncation functions also accept `DateOnly`; `ToStartOfInterval` requires a `DateTime`/`DateTime64` column on older ClickHouse, which rejects a `DateOnly` (Date/Date32) source with `Illegal type Date32 of 1st argument` for every unit; recent versions accept it. `ToStartOfInterval` takes a `ClickHouseInterval` unit (`Second`…`Year`) and emits `toStartOfInterval(source, toInterval(value))`; the unit must be a constant so it can be translated. The default `Date`/`DateTime` result types only span 1970–2149/2106, so ClickHouse narrows out-of-range values — enable `enable_extended_results_for_datetime_functions` (e.g. `set_enable_extended_results_for_datetime_functions=1` in the connection string) for range-preserving `Date32`/`DateTime64` results. -* **Standard `DateTime` members and methods now translate to SQL.** Previously the provider registered no date/time member translator, so only a direct comparison worked and every member threw `The LINQ expression ... could not be translated`. One shared translator serves `DateTime` and `DateOnly`, because the ClickHouse function is the same for each. ([#55](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/55)) +* **Standard `DateTime` members and methods now translate to SQL.** Previously the provider registered no date/time member translator, so only a direct comparison worked and every member threw `The LINQ expression ... could not be translated`. One shared translator serves `DateTime`, `DateTimeOffset` and `DateOnly`, because the ClickHouse function is the same for each. ([#55](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/55)) * **Components** — `.Year` → `toYear`, `.Month` → `toMonth`, `.Day` → `toDayOfMonth`, `.Hour` → `toHour`, `.Minute` → `toMinute`, `.Second` → `toSecond`, `.Millisecond` → `toMillisecond`, `.DayOfYear` → `toDayOfYear`. These ClickHouse functions return `UInt8`/`UInt16`, which the provider's integer mappings widen to `int` on read. `DateOnly` gets the date components only, matching the members it declares. * **`.DayOfWeek`** → `toDayOfWeek(x, 2)`. Week mode 2 agrees with `System.DayOfWeek` exactly (Sunday 0 … Saturday 6), so no arithmetic correction is applied — the default mode 0 starts the week on Monday, which is why the mode argument is always sent. The result carries a number-backed enum mapping, because this provider maps a C# `enum` to a ClickHouse string and that mapping would otherwise render `x.DayOfWeek == DayOfWeek.Sunday` as a comparison against `'Sunday'`. * **`.Date`** → `toStartOfDay`, which keeps the timezone of the source. Note that `toStartOfDay` returns a `DateTime`, whose range is 1970–2106, and ClickHouse **wraps** a value outside that window rather than reporting it — so `.Date` on a `DateTime64` column holding a pre-1970 date reads back wrong. Enable `enable_extended_results_for_datetime_functions` (for example `set_enable_extended_results_for_datetime_functions=1` in the connection string) to get a range-preserving `DateTime64` result. This is the same caveat that already applies to `EF.Functions.ToStartOfDay`. @@ -13,7 +13,8 @@ v0.3.1 (Unreleased) * **`.AddDays`/`.AddHours`/`.AddMinutes`/`.AddSeconds`/`.AddMilliseconds`** take a `double` in .NET, which .NET scales to whole **ticks** (100 ns), rounding half away from zero — so `AddSeconds(0.1234567)` adds exactly 1 234 567 ticks. The matching ClickHouse function takes a whole number of its own unit and discards the rest, so `addDays(x, 1.5)` would add only one day. A constant argument is therefore folded to ticks during translation and then expressed in the coarsest unit that holds it exactly: a whole number of the unit emits the natural function (`addDays(x, 1)`), and otherwise `addMilliseconds` carries the exact count (`AddDays(1.5)` → `addMilliseconds(x, 129600000)`). Preferring the natural function keeps the store type of the source and keeps `Date`/`Date32` columns working, since `addMilliseconds` rejects those outright. * A **sub-millisecond** offset, a **non-constant** offset, and a value **outside the `DateTime` range** are deliberately left untranslated rather than rounded to fit. Milliseconds are as fine as the translation goes, because `addNanoseconds` would express a tick exactly but promotes the result to `DateTime64(9)`, whose Int64 nanosecond count cannot span the `DateTime64` range — that would trade a rounding error for a silently wrong date. An untranslated call still gives the correct .NET value through client evaluation in a projection, and reports a clear reason in a predicate. `DateOnly.AddDays` takes an `int`, so it always emits `addDays`. * A previously-unsupported Northwind query, `GroupJoin_aggregate_anonymous_key_selectors2`, now passes as a result of these translations; its provider-specific "not translatable" override is removed. - * Not yet translated: `.Ticks`, `.AddTicks`, and the `.Microsecond`/`.Nanosecond` members. `DateTimeOffset` is not covered yet either — it has no store mapping until [#53](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/53), so translating its members would silently drop the offset. The translator is shaped to take the CLR type as a parameter, so it gains `DateTimeOffset` with that mapping. The ClickHouse-specific functions such as `dateDiff` and `dateTrunc` are tracked in [#58](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/58). + * For a `DateTimeOffset` property the result is in the timezone the column declares, which the store type pins to UTC. That agrees with .NET, because a value read back from such a column carries the `+00:00` offset, so `.Hour` and `.Date` describe the same instant on both sides. + * Not yet translated: `.Ticks`, `.AddTicks`, and the `.Microsecond`/`.Nanosecond` members. The ClickHouse-specific functions such as `dateDiff` and `dateTrunc` are tracked in [#58](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/58). * **Behaviour change:** `DateTime.Now` and `DateTime.Today` in a *projection* used to be evaluated on the client; they now read the **server** clock. The value therefore follows the server's timezone rather than the client's, and comes back with `DateTimeKind.Unspecified` instead of `Local`. Use `DateTime.UtcNow` for an instant that does not depend on server configuration. In a predicate all three were untranslatable before, so nothing changes there. ### Types diff --git a/README.md b/README.md index 7bd83d1..23f7d58 100644 --- a/README.md +++ b/README.md @@ -172,8 +172,9 @@ Map such a column as `DateTime` if you cannot change how it is declared. `List`, `Dictionary` and `Tuple` all round trip. -`DateTimeOffset` members such as `.Year` and `.UtcDateTime` do not translate to SQL yet. This -applies to `DateTime` as well — see [#55](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/55). +The standard members and methods — `.Year`, `.DayOfWeek`, `.AddDays(n)` and the rest — translate to +SQL; see [Date/Time Functions](#datetime-functions). `.UtcDateTime`, `.LocalDateTime` and `.Offset` +do not. ## Current Status @@ -219,7 +220,7 @@ ClickHouse returns `NULL` from a scalar subquery that matches no rows, where sta #### Standard members and methods -The standard .NET date/time members translate to ClickHouse functions, for both `DateTime` and `DateOnly`: +The standard .NET date/time members translate to ClickHouse functions, for `DateTime`, `DateTimeOffset` and `DateOnly` alike: | .NET | ClickHouse | | --- | --- | @@ -248,7 +249,7 @@ var recent = await ctx.Events .ToListAsync(); ``` -`DateOnly` gets the date components only, which are the members it declares. `DateTimeOffset` is not covered yet, because it has no store mapping — see [#53](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/53). +`DateOnly` gets the date components only, which are the members it declares. For a `DateTimeOffset` property the result is in the timezone the column declares, which the store type pins to UTC — that agrees with .NET, because a value read back carries the `+00:00` offset. Five points are worth knowing: diff --git a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMemberTranslator.cs b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMemberTranslator.cs index 4c5f01b..2ae1ba7 100644 --- a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMemberTranslator.cs +++ b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMemberTranslator.cs @@ -13,10 +13,17 @@ namespace ClickHouse.EntityFrameworkCore.Query.ExpressionTranslators.Internal; /// and to ClickHouse functions. /// /// +/// /// One class serves all three CLR types, because the ClickHouse function is the same for each: the /// to* extraction functions accept Date, Date32, DateTime and /// DateTime64 alike. registers the members that the /// given type declares, so gets the date components only. +/// +/// +/// For the result is in the timezone the column declares, which the +/// provider pins to UTC. That agrees with .NET, because a value read back from such a column carries +/// the +00:00 offset, so .Hour and .Date describe the same instant on both sides. +/// /// public class ClickHouseDateTimeMemberTranslator : IMemberTranslator { @@ -69,16 +76,17 @@ private enum ServerClock static ClickHouseDateTimeMemberTranslator() { RegisterInstanceMembers(typeof(DateTime), hasTimeComponents: true); + RegisterInstanceMembers(typeof(DateTimeOffset), hasTimeComponents: true); RegisterInstanceMembers(typeof(DateOnly), hasTimeComponents: false); ServerClockMembers.Add(Property(typeof(DateTime), nameof(DateTime.UtcNow)), ServerClock.UtcNow); ServerClockMembers.Add(Property(typeof(DateTime), nameof(DateTime.Now)), ServerClock.LocalNow); ServerClockMembers.Add(Property(typeof(DateTime), nameof(DateTime.Today)), ServerClock.LocalToday); - // DateTimeOffset is deliberately absent, even though every function here would serve it. The - // provider has no DateTimeOffset store mapping yet, so such a property resolves to String: the - // extraction functions then fail on the server, and addDays silently drops the offset and the - // sub-second part. Register it here together with the mapping (issue #53). + // A DateTimeOffset is an instant, and its store type is UTC-pinned, so both of its clock + // members read the same UTC value. + ServerClockMembers.Add(Property(typeof(DateTimeOffset), nameof(DateTimeOffset.UtcNow)), ServerClock.UtcNow); + ServerClockMembers.Add(Property(typeof(DateTimeOffset), nameof(DateTimeOffset.Now)), ServerClock.UtcNow); } public ClickHouseDateTimeMemberTranslator( diff --git a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs index fb88ea0..aa3f29e 100644 --- a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs +++ b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs @@ -122,13 +122,10 @@ void RegisterSourceOnly(string methodName, string sqlFunction) }) ?? throw new InvalidOperationException("Method ToStartOfInterval with strict signature not found."); RegisterAddMethods(typeof(DateTime), hasTimeComponents: true); + RegisterAddMethods(typeof(DateTimeOffset), hasTimeComponents: true); // DateOnly declares no time-based Add* method, and its AddDays takes an int. RegisterAddMethods(typeof(DateOnly), hasTimeComponents: false); - - // DateTimeOffset is deliberately absent: the provider has no DateTimeOffset store mapping yet, - // so such a property resolves to String and these functions would either fail on the server or - // silently drop the offset. Add it here together with the mapping (issue #53). } /// diff --git a/test/EFCore.ClickHouse.Tests/DateTimeMemberTranslationTests.cs b/test/EFCore.ClickHouse.Tests/DateTimeMemberTranslationTests.cs index 8b5be82..c62c13c 100644 --- a/test/EFCore.ClickHouse.Tests/DateTimeMemberTranslationTests.cs +++ b/test/EFCore.ClickHouse.Tests/DateTimeMemberTranslationTests.cs @@ -15,6 +15,9 @@ public class DateTimeMemberEntity /// Mapped to ClickHouse Date32. public DateOnly Date { get; set; } + + /// Mapped to ClickHouse DateTime64(7, 'UTC') by the DateTimeOffset mapping. + public DateTimeOffset Offset { get; set; } } public class DateTimeMemberDbContext : DbContext @@ -43,6 +46,7 @@ protected override void OnModelCreating(ModelBuilder modelBuilder) entity.Property(e => e.Timestamp).HasColumnName("ts"); entity.Property(e => e.Timestamp64).HasColumnName("ts64").HasColumnType("DateTime64(7)"); entity.Property(e => e.Date).HasColumnName("d"); + entity.Property(e => e.Offset).HasColumnName("off"); }); } } @@ -74,7 +78,8 @@ CREATE TABLE datetime_member_test ( id Int64, ts DateTime, ts64 DateTime64(7), - d Date32 + d Date32, + off DateTime64(7, 'UTC') ) ENGINE = MergeTree() ORDER BY id """; @@ -83,9 +88,9 @@ ORDER BY id using var insertCmd = connection.CreateCommand(); // Row 2 is the last day of a month, so AddMonths and AddYears have a day to clamp. insertCmd.CommandText = """ - INSERT INTO datetime_member_test (id, ts, ts64, d) VALUES - (1, '2026-08-16 13:47:32', '2026-08-16 13:47:32.1234567', '2026-08-16'), - (2, '2026-01-31 00:00:00', '2026-01-31 00:00:00.0000000', '2026-01-31') + INSERT INTO datetime_member_test (id, ts, ts64, d, off) VALUES + (1, '2026-08-16 13:47:32', '2026-08-16 13:47:32.1234567', '2026-08-16', '2026-08-16 13:47:32.1234567'), + (2, '2026-01-31 00:00:00', '2026-01-31 00:00:00.0000000', '2026-01-31', '2026-01-31 00:00:00.0000000') """; await insertCmd.ExecuteNonQueryAsync(); } @@ -224,6 +229,91 @@ public async Task DateOnly_components_translate() Assert.Equal(DayOfWeek.Sunday, result.DayOfWeek); } + // ---------------------------------------------------------------- DateTimeOffset + + [Fact] + public async Task DateTimeOffset_components_translate() + { + var result = await SelectSingleAsync(q => q.Select(e => new + { + e.Offset.Year, + e.Offset.Month, + e.Offset.Day, + e.Offset.Hour, + e.Offset.Minute, + e.Offset.Second, + e.Offset.DayOfYear, + e.Offset.DayOfWeek + })); + + // The store type is UTC-pinned, and a value read back carries +00:00, so every component + // describes the same instant on both sides. + Assert.Equal(2026, result.Year); + Assert.Equal(8, result.Month); + Assert.Equal(16, result.Day); + Assert.Equal(13, result.Hour); + Assert.Equal(47, result.Minute); + Assert.Equal(32, result.Second); + Assert.Equal(228, result.DayOfYear); + Assert.Equal(DayOfWeek.Sunday, result.DayOfWeek); + } + + [Fact] + public async Task DateTimeOffset_components_agree_with_dotnet() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + + var expected = await context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Offset).SingleAsync(); + var actual = await SelectSingleAsync(q => q.Select(e => new { e.Offset.Year, e.Offset.Hour, e.Offset.Second })); + + Assert.Equal(expected.Year, actual.Year); + Assert.Equal(expected.Hour, actual.Hour); + Assert.Equal(expected.Second, actual.Second); + } + + [Fact] + public async Task DateTimeOffset_Date_translates() + => Assert.Equal( + new DateTime(2026, 8, 16), + await SelectSingleAsync(q => q.Select(e => e.Offset.Date))); + + [Fact] + public async Task DateTimeOffset_TimeOfDay_keeps_tick_precision() + => Assert.Equal( + DateTimeMemberFixture.InstantTimeOfDay, + await SelectSingleAsync(q => q.Select(e => e.Offset.TimeOfDay))); + + [Fact] + public async Task DateTimeOffset_AddDays_translates() + { + var result = await SelectSingleAsync(q => q.Select(e => e.Offset.AddDays(1))); + + Assert.Equal(new DateTimeOffset(2026, 8, 17, 13, 47, 32, TimeSpan.Zero).AddTicks(1_234_567), result); + } + + [Fact] + public async Task DateTimeOffset_AddMonths_clamps_like_dotnet() + => Assert.Equal( + new DateTimeOffset(2026, 2, 28, 0, 0, 0, TimeSpan.Zero), + await SelectSingleAsync(q => q.Select(e => e.Offset.AddMonths(1)), id: 2)); + + [Fact] + public async Task DateTimeOffset_DayOfWeek_compares_against_a_dotnet_constant() + => Assert.Equal([1L], await WhereIdsAsync(e => e.Offset.DayOfWeek == DayOfWeek.Sunday)); + + [Fact] + public async Task DateTimeOffset_UtcNow_runs_on_the_server() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking() + .Where(e => e.Offset < DateTimeOffset.UtcNow.AddYears(100)) + .Select(e => e.Id); + + Assert.Contains("now64", query.ToQueryString()); + Assert.Equal([1L, 2L], await query.OrderBy(id => id).ToListAsync()); + } + // ---------------------------------------------------------------- Add* [Fact]