diff --git a/CHANGELOG.md b/CHANGELOG.md index c8c8183..ab46f63 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,20 @@ v0.3.1 (Unreleased) ### Query translation * **`toStartOf*` date-time functions** via `EF.Functions`: `ToStartOfYear`, `ToStartOfQuarter`, `ToStartOfMonth`, `ToStartOfWeek` (with optional week `mode`), `ToStartOfDay`, `ToStartOfHour`, `ToStartOfMinute`, `ToStartOfSecond`, the fixed buckets `ToStartOfFiveMinutes` / `ToStartOfTenMinutes` / `ToStartOfFifteenMinutes`, and the general `ToStartOfInterval(source, value, unit)`. Each maps to the matching ClickHouse function and works in `GROUP BY`. Return types follow ClickHouse: the calendar buckets (`Year`/`Quarter`/`Month`/`Week`) return `Date`, the day/hour/minute buckets return `DateTime`, and `ToStartOfSecond` returns `DateTime64`. All accept `DateTime`/`DateTime64` columns, and the plain truncation functions also accept `DateOnly`; `ToStartOfInterval` requires a `DateTime`/`DateTime64` column on older ClickHouse, which rejects a `DateOnly` (Date/Date32) source with `Illegal type Date32 of 1st argument` for every unit; recent versions accept it. `ToStartOfInterval` takes a `ClickHouseInterval` unit (`Second`…`Year`) and emits `toStartOfInterval(source, toInterval(value))`; the unit must be a constant so it can be translated. The default `Date`/`DateTime` result types only span 1970–2149/2106, so ClickHouse narrows out-of-range values — enable `enable_extended_results_for_datetime_functions` (e.g. `set_enable_extended_results_for_datetime_functions=1` in the connection string) for range-preserving `Date32`/`DateTime64` results. +* **Standard `DateTime` members and methods now translate to SQL.** Previously the provider registered no date/time member translator, so only a direct comparison worked and every member threw `The LINQ expression ... could not be translated`. One shared translator serves `DateTime`, `DateTimeOffset` and `DateOnly`, because the ClickHouse function is the same for each. ([#55](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/55)) + * **Components** — `.Year` → `toYear`, `.Month` → `toMonth`, `.Day` → `toDayOfMonth`, `.Hour` → `toHour`, `.Minute` → `toMinute`, `.Second` → `toSecond`, `.Millisecond` → `toMillisecond`, `.DayOfYear` → `toDayOfYear`. These ClickHouse functions return `UInt8`/`UInt16`, which the provider's integer mappings widen to `int` on read. `DateOnly` gets the date components only, matching the members it declares. + * **`.DayOfWeek`** → `toDayOfWeek(x, 2)`. Week mode 2 agrees with `System.DayOfWeek` exactly (Sunday 0 … Saturday 6), so no arithmetic correction is applied — the default mode 0 starts the week on Monday, which is why the mode argument is always sent. The result carries a number-backed enum mapping, because this provider maps a C# `enum` to a ClickHouse string and that mapping would otherwise render `x.DayOfWeek == DayOfWeek.Sunday` as a comparison against `'Sunday'`. + * **`.Date`** → `toStartOfDay`, which keeps the timezone of the source. Note that `toStartOfDay` returns a `DateTime`, whose range is 1970–2106, and ClickHouse **wraps** a value outside that window rather than reporting it — so `.Date` on a `DateTime64` column holding a pre-1970 date reads back wrong. Enable `enable_extended_results_for_datetime_functions` (for example `set_enable_extended_results_for_datetime_functions=1` in the connection string) to get a range-preserving `DateTime64` result. This is the same caveat that already applies to `EF.Functions.ToStartOfDay`. + * **`.TimeOfDay`** → `toTime64(x, 7)`; precision 7 is one .NET tick, so no part of the value is lost (`toTime` would drop the fraction). + * **`DateTime.UtcNow`** → `now64(7, 'UTC')`, **`DateTime.Now`** → `now64(7)` and **`DateTime.Today`** → `toStartOfDay(now())`. `today()` is not used for `.Today` because it returns a `Date`, whereas the member's type is `DateTime`. + * **`.AddYears(n)`** → `addYears` and **`.AddMonths(n)`** → `addMonths`. Both take an `int` in .NET, and ClickHouse clamps the day of month the same way .NET does, so `2026-01-31` plus one month gives `2026-02-28` in both. + * **`.AddDays`/`.AddHours`/`.AddMinutes`/`.AddSeconds`/`.AddMilliseconds`** take a `double` in .NET, which .NET scales to whole **ticks** (100 ns), rounding half away from zero — so `AddSeconds(0.1234567)` adds exactly 1 234 567 ticks. The matching ClickHouse function takes a whole number of its own unit and discards the rest, so `addDays(x, 1.5)` would add only one day. A constant argument is therefore folded to ticks during translation and then expressed in the coarsest unit that holds it exactly: a whole number of the unit emits the natural function (`addDays(x, 1)`), and otherwise `addMilliseconds` carries the exact count (`AddDays(1.5)` → `addMilliseconds(x, 129600000)`). Preferring the natural function keeps the store type of the source and keeps `Date`/`Date32` columns working, since `addMilliseconds` rejects those outright. + * A **sub-millisecond** offset, a **non-constant** offset, and a value **outside the `DateTime` range** are deliberately left untranslated rather than rounded to fit. Milliseconds are as fine as the translation goes, because `addNanoseconds` would express a tick exactly but promotes the result to `DateTime64(9)`, whose Int64 nanosecond count cannot span the `DateTime64` range — that would trade a rounding error for a silently wrong date. An untranslated call still gives the correct .NET value through client evaluation in a projection, and reports a clear reason in a predicate. `DateOnly.AddDays` takes an `int`, so it always emits `addDays`. + * A previously-unsupported Northwind query, `GroupJoin_aggregate_anonymous_key_selectors2`, now passes as a result of these translations; its provider-specific "not translatable" override is removed. + * For a `DateTimeOffset` property the result is in the timezone the column declares, which the store type pins to UTC. That agrees with .NET, because a value read back from such a column carries the `+00:00` offset, so `.Hour` and `.Date` describe the same instant on both sides. + * Not yet translated: `.Ticks`, `.AddTicks`, and the `.Microsecond`/`.Nanosecond` members. The ClickHouse-specific functions such as `dateDiff` and `dateTrunc` are tracked in [#58](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/58). + * **Behaviour change:** `DateTime.Now` and `DateTime.Today` in a *projection* used to be evaluated on the client; they now read the **server** clock. The value therefore follows the server's timezone rather than the client's, and comes back with `DateTimeKind.Unspecified` instead of `Local`. Use `DateTime.UtcNow` for an instant that does not depend on server configuration. In a predicate all three were untranslatable before, so nothing changes there. + ### Types * **`DateTimeOffset` support.** A `DateTimeOffset` property now maps to `DateTime64(7, 'UTC')` through the new `ClickHouseDateTimeOffsetTypeMapping`. Previously the provider had no mapping for the type, so EF Core fell back to `DateTimeOffsetToStringConverter` and silently produced a `String` column. That fallback broke queries against real `DateTime64` columns (`TYPE_MISMATCH`, because the parameter was declared `String`) and `SaveChanges` could not write the value at all. ([#53](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/53)) * The store type is UTC-pinned on purpose. For a timezone-less parameter type such as `DateTime64(7)`, the driver sends a UTC wall clock and the server then reads it in `session_timezone`, which moves the instant when that setting is not UTC. @@ -17,6 +31,7 @@ v0.3.1 (Unreleased) * **Behaviour change:** a `DateTimeOffset` property that relied on the old `String` column now resolves to `DateTime64(7, 'UTC')`. Add `HasConversion()` to keep the previous shape. Note that `HasColumnType("String")` on its own is not enough — it resolves the plain string mapping with no converter, so the CLR type no longer agrees with the property. Keeping the old shape also makes the property read-only until [#54](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/54) is fixed, because `SaveChanges` does not apply a value converter on the insert path. ### Bug fixes +* **Subtracting one date/time value from another no longer fails with an internal error.** `dt1 - dt2` gives a `TimeSpan`, which ClickHouse has no operator for — `dateDiff` returns a count of whole units instead. The expression used to reach type-mapping inference and fail with an `InvalidCastException` or a bare `No coercion operator is defined between types ...`, both of which name CLR types the user never wrote. The subtraction is now reported as not translatable, with the reason attached. In a projection EF Core can therefore fall back to the client and return the correct `TimeSpan`; in a predicate, where no fallback exists, the message explains why and what to do instead. ([#55](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/55)) * `ToStartOfWeek` now rejects row-dependent week modes during query translation, and `ToStartOfInterval` likewise rejects row-dependent interval sizes. ClickHouse requires these operands to be constant for the query; literals and captured query parameters remain supported. * **Composite columns now convert their components on read.** `Array(T)`, `Map(K, V)`, and `Tuple(...)` read the whole column through `GetValue`, so a component mapping's own read pipeline never ran. Any component whose CLR type differs from the type the driver produces therefore threw `InvalidCastException` — both `DateTimeOffset` and `DateOnly` arrive from the driver as `DateTime`. `DateOnly[]`, `Dictionary` and `Tuple` were affected before `DateTimeOffset` existed as a mapped type. * The composite is now rebuilt component by component, applying both steps EF Core applies to a scalar column: the mapping's data-reader conversion, then its `ValueConverter`. Components that convert through a converter therefore work too — a C# `enum` component (`EnumToStringConverter`), a `List` component (`ListToArrayConverter`), and the collection interfaces (`IList`, `IReadOnlyList`). diff --git a/README.md b/README.md index 6d365aa..23f7d58 100644 --- a/README.md +++ b/README.md @@ -172,8 +172,9 @@ Map such a column as `DateTime` if you cannot change how it is declared. `List`, `Dictionary` and `Tuple` all round trip. -`DateTimeOffset` members such as `.Year` and `.UtcDateTime` do not translate to SQL yet. This -applies to `DateTime` as well — see [#55](https://github.com/ClickHouse/ClickHouse.EntityFrameworkCore/issues/55). +The standard members and methods — `.Year`, `.DayOfWeek`, `.AddDays(n)` and the rest — translate to +SQL; see [Date/Time Functions](#datetime-functions). `.UtcDateTime`, `.LocalDateTime` and `.Offset` +do not. ## Current Status @@ -217,6 +218,64 @@ ClickHouse returns `NULL` from a scalar subquery that matches no rows, where sta ### Date/Time Functions +#### Standard members and methods + +The standard .NET date/time members translate to ClickHouse functions, for `DateTime`, `DateTimeOffset` and `DateOnly` alike: + +| .NET | ClickHouse | +| --- | --- | +| `.Year` `.Month` `.Day` | `toYear` `toMonth` `toDayOfMonth` | +| `.Hour` `.Minute` `.Second` `.Millisecond` | `toHour` `toMinute` `toSecond` `toMillisecond` | +| `.DayOfYear` | `toDayOfYear` | +| `.DayOfWeek` | `toDayOfWeek(x, 2)` | +| `.Date` | `toStartOfDay` | +| `.TimeOfDay` | `toTime64(x, 7)` | +| `.AddYears(n)` `.AddMonths(n)` | `addYears` `addMonths` | +| `.AddDays(n)` `.AddHours(n)` `.AddMinutes(n)` `.AddSeconds(n)` `.AddMilliseconds(n)` | `addDays` `addHours` … (see below) | +| `DateTime.UtcNow` | `now64(7, 'UTC')` | +| `DateTime.Now` | `now64(7)` | +| `DateTime.Today` | `toStartOfDay(now())` | + +```csharp +// Runs entirely on the server +var busyHours = await ctx.Events + .Where(e => e.Timestamp.Year == 2026 && e.Timestamp.DayOfWeek == DayOfWeek.Sunday) + .GroupBy(e => e.Timestamp.Hour) + .Select(g => new { Hour = g.Key, Count = g.Count() }) + .ToListAsync(); + +var recent = await ctx.Events + .Where(e => e.Timestamp > DateTime.UtcNow.AddDays(-7)) + .ToListAsync(); +``` + +`DateOnly` gets the date components only, which are the members it declares. For a `DateTimeOffset` property the result is in the timezone the column declares, which the store type pins to UTC — that agrees with .NET, because a value read back carries the `+00:00` offset. + +Five points are worth knowing: + +**`.DayOfWeek` needs no correction.** ClickHouse week mode 2 agrees with `System.DayOfWeek` exactly — Sunday is 0 through to Saturday 6 — so the value is used as it comes back. The mode argument is always sent, because the default mode starts the week on Monday. + +**`.Now` and `.Today` read the server clock**, so they follow the *server's* timezone, not the client's, and they come back with `DateTimeKind.Unspecified`. Use `DateTime.UtcNow` when you need an instant that does not depend on server configuration. + +**`.Date` narrows outside 1970–2106.** `toStartOfDay` returns a `DateTime`, and ClickHouse *wraps* a value outside that window instead of reporting it — so `.Date` on a `DateTime64` column holding a pre-1970 date reads back wrong. Enable [`enable_extended_results_for_datetime_functions`](https://clickhouse.com/docs/operations/settings/settings#enable_extended_results_for_datetime_functions) — for example `set_enable_extended_results_for_datetime_functions=1` in the connection string — to get a range-preserving `DateTime64` result. + +**A fractional `Add*` argument is exact or is not translated.** `AddDays` and the other time-based methods take a `double`, which .NET scales to whole *ticks* (100 ns), so `AddSeconds(0.1234567)` adds exactly 1 234 567 ticks. The ClickHouse `addDays` function takes a whole number of days and discards the rest, so it cannot be used directly. A constant argument is folded to ticks and then expressed in the coarsest unit that holds it exactly: + +```csharp +e.Timestamp.AddDays(1) // addDays(ts, 1) +e.Timestamp.AddDays(1.5) // addMilliseconds(ts, 129600000) +e.Timestamp.AddMilliseconds(0.5) // not translated — 5 000 ticks is below millisecond resolution +e.Timestamp.AddDays(offsetVariable) // not translated — cannot be checked for exactness +``` + +The natural function keeps the column's store type, and it is the only form that works on a `Date`/`Date32` column — ClickHouse rejects `addMilliseconds` on those. Anything the provider cannot express exactly is left untranslated rather than rounded to fit, so a projection still gives the correct .NET value through client evaluation, while a predicate reports why. `DateOnly.AddDays` takes an `int`, so it always emits `addDays`. + +**Arithmetic on two date/time values is not translated.** `dt1 - dt2` and `time1 - time2` give a `TimeSpan`, and `date + timeSpan` mixes types ClickHouse rejects; `dateDiff` returns a count of whole units, and `Time64` subtraction returns a decimal number of seconds. In a projection EF Core reads the columns and does the arithmetic on the client, which gives the correct result. In a predicate there is no client fallback, so the query fails with an explanation. + +Not yet translated: `.Ticks`, `.AddTicks`, and the `.Microsecond`/`.Nanosecond` members. + +#### `toStartOf*` bucketing + The ClickHouse `toStartOf*` family is exposed through `EF.Functions`, so you can bucket and truncate timestamps directly in queries, including in `GROUP BY`: `ToStartOfYear`, `ToStartOfQuarter`, `ToStartOfMonth`, `ToStartOfWeek` (with an optional ClickHouse week `mode`), `ToStartOfDay`, `ToStartOfHour`, `ToStartOfMinute`, `ToStartOfSecond`, `ToStartOfFiveMinutes`, `ToStartOfTenMinutes`, `ToStartOfFifteenMinutes`, and the general `ToStartOfInterval(source, value, unit)`. diff --git a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMemberTranslator.cs b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMemberTranslator.cs new file mode 100644 index 0000000..2ae1ba7 --- /dev/null +++ b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMemberTranslator.cs @@ -0,0 +1,251 @@ +using System.Reflection; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Diagnostics; +using Microsoft.EntityFrameworkCore.Query; +using Microsoft.EntityFrameworkCore.Query.SqlExpressions; +using Microsoft.EntityFrameworkCore.Storage; +using Microsoft.EntityFrameworkCore.Storage.ValueConversion; + +namespace ClickHouse.EntityFrameworkCore.Query.ExpressionTranslators.Internal; + +/// +/// Translates the standard date/time members of , +/// and to ClickHouse functions. +/// +/// +/// +/// One class serves all three CLR types, because the ClickHouse function is the same for each: the +/// to* extraction functions accept Date, Date32, DateTime and +/// DateTime64 alike. registers the members that the +/// given type declares, so gets the date components only. +/// +/// +/// For the result is in the timezone the column declares, which the +/// provider pins to UTC. That agrees with .NET, because a value read back from such a column carries +/// the +00:00 offset, so .Hour and .Date describe the same instant on both sides. +/// +/// +public class ClickHouseDateTimeMemberTranslator : IMemberTranslator +{ + /// + /// Week mode 2 makes toDayOfWeek agree with exactly: Sunday is 0 + /// through to Saturday is 6. The default mode 0 starts the week on Monday, so the argument is + /// required and no arithmetic correction is needed. + /// + private const byte SundayFirstWeekMode = 2; + + /// + /// One tick is 100 ns, which is Time64 precision 7, and one .NET + /// tick is also the resolution of now64(7). Asking for that precision keeps + /// exact, whereas toTime drops the fraction. + /// + private const int TickPrecision = 7; + + /// Members that map to a ClickHouse function taking the source value alone. + private static readonly Dictionary ComponentFunctions = []; + + private static readonly HashSet DayOfWeekMembers = []; + private static readonly HashSet DateMembers = []; + private static readonly HashSet TimeOfDayMembers = []; + + /// Static members that read the server clock. + private static readonly Dictionary ServerClockMembers = []; + + private readonly ISqlExpressionFactory _sqlExpressionFactory; + private readonly IRelationalTypeMappingSource _typeMappingSource; + + /// How a server-clock member is built: a function name, and whether it pins UTC. + private enum ServerClock + { + /// now64(7, 'UTC') — an exact instant, independent of server settings. + UtcNow, + + /// now64(7) — the server's local clock, in its configured timezone. + LocalNow, + + /// toStartOfDay(now()) — midnight today on the server's local clock. + LocalToday + } + + /// + /// The mapping given to a result. Built once, because it composes a + /// converter over the Int32 mapping and nothing about it varies per translation. + /// + private readonly RelationalTypeMapping? _dayOfWeekMapping; + + static ClickHouseDateTimeMemberTranslator() + { + RegisterInstanceMembers(typeof(DateTime), hasTimeComponents: true); + RegisterInstanceMembers(typeof(DateTimeOffset), hasTimeComponents: true); + RegisterInstanceMembers(typeof(DateOnly), hasTimeComponents: false); + + ServerClockMembers.Add(Property(typeof(DateTime), nameof(DateTime.UtcNow)), ServerClock.UtcNow); + ServerClockMembers.Add(Property(typeof(DateTime), nameof(DateTime.Now)), ServerClock.LocalNow); + ServerClockMembers.Add(Property(typeof(DateTime), nameof(DateTime.Today)), ServerClock.LocalToday); + + // A DateTimeOffset is an instant, and its store type is UTC-pinned, so both of its clock + // members read the same UTC value. + ServerClockMembers.Add(Property(typeof(DateTimeOffset), nameof(DateTimeOffset.UtcNow)), ServerClock.UtcNow); + ServerClockMembers.Add(Property(typeof(DateTimeOffset), nameof(DateTimeOffset.Now)), ServerClock.UtcNow); + } + + public ClickHouseDateTimeMemberTranslator( + ISqlExpressionFactory sqlExpressionFactory, + IRelationalTypeMappingSource typeMappingSource) + { + _sqlExpressionFactory = sqlExpressionFactory; + _typeMappingSource = typeMappingSource; + + // DayOfWeek is an enum, and this provider maps a C# enum to a ClickHouse string. That mapping + // would render the other side of a comparison as 'Sunday' against a number, so the result + // carries a number-backed enum mapping instead. + _dayOfWeekMapping = typeMappingSource.FindMapping(typeof(int)) is { } intMapping + ? (RelationalTypeMapping)intMapping.WithComposedConverter(new EnumToNumberConverter()) + : null; + } + + public SqlExpression? Translate( + SqlExpression? instance, + MemberInfo member, + Type returnType, + IDiagnosticsLogger logger) + { + if (instance is null) + { + return TranslateServerClock(member, returnType); + } + + // toYear and friends return UInt8/UInt16, which the provider's integer mappings widen on read. + if (ComponentFunctions.TryGetValue(member, out var function)) + { + return _sqlExpressionFactory.Function( + name: function, + arguments: [instance], + nullable: true, + argumentsPropagateNullability: [true], + returnType: returnType, + typeMapping: _typeMappingSource.FindMapping(returnType)); + } + + if (DayOfWeekMembers.Contains(member)) + { + return _sqlExpressionFactory.Function( + name: "toDayOfWeek", + arguments: [instance, _sqlExpressionFactory.Constant(SundayFirstWeekMode)], + nullable: true, + // Only the source propagates nullability; the week mode is a constant. + argumentsPropagateNullability: [true, false], + returnType: returnType, + typeMapping: _dayOfWeekMapping); + } + + // toStartOfDay keeps the timezone of the source, which is what DateTime.Date means for a + // column: midnight on the same calendar day that the column renders. + // + // Note that toStartOfDay returns a DateTime, whose range is 1970-2106. ClickHouse wraps a value + // outside that window rather than reporting it, so a DateTime64 column holding a date before + // 1970 reads back wrong unless the session enables + // enable_extended_results_for_datetime_functions, which widens the result to DateTime64. This + // matches EF.Functions.ToStartOfDay and is documented alongside it. + if (DateMembers.Contains(member)) + { + return _sqlExpressionFactory.Function( + name: "toStartOfDay", + arguments: [instance], + nullable: true, + argumentsPropagateNullability: [true], + returnType: returnType, + // Reuse the source's mapping only when it describes the member's own CLR type. A mapping + // for a different type cannot be coerced during materialization. + typeMapping: instance.TypeMapping?.ClrType == returnType + ? instance.TypeMapping + : _typeMappingSource.FindMapping(returnType)); + } + + if (TimeOfDayMembers.Contains(member)) + { + return _sqlExpressionFactory.Function( + name: "toTime64", + arguments: [instance, _sqlExpressionFactory.Constant(TickPrecision)], + nullable: true, + argumentsPropagateNullability: [true, false], + returnType: returnType, + typeMapping: _typeMappingSource.FindMapping($"Time64({TickPrecision})")); + } + + return null; + } + + private SqlExpression? TranslateServerClock(MemberInfo member, Type returnType) + { + if (!ServerClockMembers.TryGetValue(member, out var clock)) + { + return null; + } + + var mapping = _typeMappingSource.FindMapping(returnType); + + // DateTime.Today is midnight today. today() returns a Date, whereas the member's type is + // DateTime, so truncate the clock value instead and keep a DateTime store type. + if (clock == ServerClock.LocalToday) + { + return _sqlExpressionFactory.Function( + name: "toStartOfDay", + arguments: [Niladic("now", returnType, mapping)], + nullable: false, + argumentsPropagateNullability: [false], + returnType: returnType, + typeMapping: mapping); + } + + List arguments = [_sqlExpressionFactory.Constant(TickPrecision)]; + if (clock == ServerClock.UtcNow) + { + arguments.Add(_sqlExpressionFactory.Constant("UTC")); + } + + return _sqlExpressionFactory.Function( + name: "now64", + arguments: arguments, + nullable: false, + argumentsPropagateNullability: arguments.Select(_ => false), + returnType: returnType, + typeMapping: mapping); + } + + private SqlExpression Niladic(string name, Type returnType, RelationalTypeMapping? typeMapping) + => _sqlExpressionFactory.Function( + name: name, + arguments: [], + nullable: false, + argumentsPropagateNullability: [], + returnType: returnType, + typeMapping: typeMapping); + + private static void RegisterInstanceMembers(Type type, bool hasTimeComponents) + { + ComponentFunctions.Add(Property(type, nameof(DateTime.Year)), "toYear"); + ComponentFunctions.Add(Property(type, nameof(DateTime.Month)), "toMonth"); + ComponentFunctions.Add(Property(type, nameof(DateTime.Day)), "toDayOfMonth"); + ComponentFunctions.Add(Property(type, nameof(DateTime.DayOfYear)), "toDayOfYear"); + + DayOfWeekMembers.Add(Property(type, nameof(DateTime.DayOfWeek))); + + if (!hasTimeComponents) + { + return; + } + + ComponentFunctions.Add(Property(type, nameof(DateTime.Hour)), "toHour"); + ComponentFunctions.Add(Property(type, nameof(DateTime.Minute)), "toMinute"); + ComponentFunctions.Add(Property(type, nameof(DateTime.Second)), "toSecond"); + ComponentFunctions.Add(Property(type, nameof(DateTime.Millisecond)), "toMillisecond"); + + DateMembers.Add(Property(type, nameof(DateTime.Date))); + TimeOfDayMembers.Add(Property(type, nameof(DateTime.TimeOfDay))); + } + + private static MemberInfo Property(Type type, string name) + => type.GetProperty(name) + ?? throw new InvalidOperationException($"Property {type.Name}.{name} was not found."); +} diff --git a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs index fa31725..aa3f29e 100644 --- a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs +++ b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseDateTimeMethodTranslator.cs @@ -9,13 +9,28 @@ namespace ClickHouse.EntityFrameworkCore.Query.ExpressionTranslators.Internal; /// -/// Translates the EF.Functions.ToStartOf* extension methods -/// () to their ClickHouse SQL functions. +/// Translates date/time method calls to ClickHouse SQL functions: the +/// EF.Functions.ToStartOf* extension methods +/// (), and the standard Add* methods of +/// , and . /// public class ClickHouseDateTimeMethodTranslator : IMethodCallTranslator { private readonly ISqlExpressionFactory _sqlExpressionFactory; + /// + /// Add* methods that take an , keyed to their ClickHouse function. An + /// integer count needs no rounding, so these always translate. + /// + private static readonly Dictionary IntegralAddMethods = []; + + /// + /// Add* methods that take a , keyed to their ClickHouse function and + /// the tick length of the method's own unit. Keeping the two families in separate dictionaries + /// makes a unit with no fixed tick length (a month, a year) unrepresentable here. + /// + private static readonly Dictionary FractionalAddMethods = []; + /// /// Maps the generic method definitions that take only the source value (and, for /// ToStartOfWeek(source, mode), an extra scalar argument) directly to a ClickHouse function name. @@ -105,8 +120,46 @@ void RegisterSourceOnly(string methodName, string sqlFunction) && parameters[2].ParameterType == typeof(int) && parameters[3].ParameterType == typeof(ClickHouseInterval); }) ?? throw new InvalidOperationException("Method ToStartOfInterval with strict signature not found."); + + RegisterAddMethods(typeof(DateTime), hasTimeComponents: true); + RegisterAddMethods(typeof(DateTimeOffset), hasTimeComponents: true); + + // DateOnly declares no time-based Add* method, and its AddDays takes an int. + RegisterAddMethods(typeof(DateOnly), hasTimeComponents: false); } + /// + /// Registers the Add* methods that declares. + /// + /// + /// AddYears and AddMonths take an on every supported type, so they + /// map straight onto addYears/addMonths. The time-based methods take a + /// on , which needs the exactness check that + /// applies. On , AddDays takes an + /// instead, so it is registered as integral. + /// + private static void RegisterAddMethods(Type type, bool hasTimeComponents) + { + IntegralAddMethods.Add(Method(type, nameof(DateTime.AddYears), typeof(int)), "addYears"); + IntegralAddMethods.Add(Method(type, nameof(DateTime.AddMonths), typeof(int)), "addMonths"); + + if (!hasTimeComponents) + { + IntegralAddMethods.Add(Method(type, nameof(DateOnly.AddDays), typeof(int)), "addDays"); + return; + } + + FractionalAddMethods.Add(Method(type, nameof(DateTime.AddDays), typeof(double)), ("addDays", TimeSpan.TicksPerDay)); + FractionalAddMethods.Add(Method(type, nameof(DateTime.AddHours), typeof(double)), ("addHours", TimeSpan.TicksPerHour)); + FractionalAddMethods.Add(Method(type, nameof(DateTime.AddMinutes), typeof(double)), ("addMinutes", TimeSpan.TicksPerMinute)); + FractionalAddMethods.Add(Method(type, nameof(DateTime.AddSeconds), typeof(double)), ("addSeconds", TimeSpan.TicksPerSecond)); + FractionalAddMethods.Add(Method(type, nameof(DateTime.AddMilliseconds), typeof(double)), ("addMilliseconds", TimeSpan.TicksPerMillisecond)); + } + + private static MethodInfo Method(Type type, string name, Type argumentType) + => type.GetRuntimeMethod(name, [argumentType]) + ?? throw new InvalidOperationException($"Method {type.Name}.{name}({argumentType.Name}) was not found."); + public ClickHouseDateTimeMethodTranslator(ISqlExpressionFactory sqlExpressionFactory) { _sqlExpressionFactory = sqlExpressionFactory; @@ -118,6 +171,20 @@ public ClickHouseDateTimeMethodTranslator(ISqlExpressionFactory sqlExpressionFac IReadOnlyList arguments, IDiagnosticsLogger logger) { + if (instance is not null) + { + if (IntegralAddMethods.TryGetValue(method, out var integralFunction)) + { + return AddFunction(integralFunction, instance, arguments[0], method.ReturnType); + } + + if (FractionalAddMethods.TryGetValue(method, out var fractional)) + { + return TranslateFractionalAdd( + instance, fractional.Function, fractional.TicksPerUnit, arguments[0], method.ReturnType); + } + } + var genericMethod = method.IsGenericMethod ? method.GetGenericMethodDefinition() : method; if (SupportedMethods.TryGetValue(genericMethod, out var function)) @@ -180,6 +247,98 @@ public ClickHouseDateTimeMethodTranslator(ISqlExpressionFactory sqlExpressionFac return null; } + /// + /// Translates one Add* call whose .NET argument is a , or returns + /// when no exact translation exists. + /// + /// + /// + /// .NET scales the argument to whole ticks and rounds half away from zero, so + /// AddSeconds(0.1234567) adds exactly 1 234 567 ticks. The matching ClickHouse function takes + /// a whole number of its own unit and discards the rest, so addDays(x, 1.5) would add one day. + /// The two agree only when the tick count divides exactly into the function's unit. + /// + /// + /// A constant is therefore folded to ticks here and then expressed in the coarsest unit that holds + /// it exactly. The natural unit is preferred, and not only for readability: it keeps the store type + /// of the source, and addMilliseconds rejects a Date or Date32 source outright + /// (ILLEGAL_TYPE_OF_ARGUMENT). + /// + /// + /// Everything else is left untranslated on purpose, rather than rounded to fit. This covers a + /// sub-millisecond offset and any value that is not a constant. Milliseconds are as fine as this + /// goes: addNanoseconds would express a tick exactly but promotes the result to + /// DateTime64(9), whose Int64 nanosecond count cannot span the DateTime64 range, which + /// would trade a rounding error for a silently wrong date. An untranslated call still gives the + /// correct .NET value through client evaluation in a projection, and reports a clear reason in a + /// predicate. + /// + /// + private SqlExpression? TranslateFractionalAdd( + SqlExpression instance, + string function, + long ticksPerUnit, + SqlExpression value, + Type returnType) + { + if (value is not SqlConstantExpression { Value: double constantValue }) + { + return null; + } + + if (TicksFor(constantValue, ticksPerUnit) is not { } ticks) + { + return null; + } + + if (ticks % ticksPerUnit == 0) + { + return AddFunction(function, instance, _sqlExpressionFactory.Constant(ticks / ticksPerUnit), returnType); + } + + if (ticks % TimeSpan.TicksPerMillisecond != 0) + { + return null; + } + + return AddFunction( + "addMilliseconds", + instance, + _sqlExpressionFactory.Constant(ticks / TimeSpan.TicksPerMillisecond), + returnType); + } + + /// + /// The tick count that .NET would add, or when .NET would not produce one. + /// + /// + /// Mirrors 's scaling: multiply by the unit's tick length, then + /// round half away from zero. A value that cannot represent is rejected, so + /// the comes from .NET during client evaluation instead of + /// from a wrapped Int64 on the server, which ClickHouse reports as a decimal overflow — or, for the + /// larger magnitudes, does not report at all. + /// + private static long? TicksFor(double value, long ticksPerUnit) + { + if (double.IsNaN(value) || double.IsInfinity(value)) + { + return null; + } + + var scaled = value * ticksPerUnit + (value >= 0 ? 0.5 : -0.5); + + return double.Abs(scaled) > DateTime.MaxValue.Ticks ? null : (long)scaled; + } + + private SqlExpression AddFunction(string function, SqlExpression instance, SqlExpression value, Type returnType) + => _sqlExpressionFactory.Function( + name: function, + arguments: [instance, value], + nullable: true, + argumentsPropagateNullability: [true, true], + returnType: returnType, + typeMapping: instance.TypeMapping); + private static bool IsQueryConstant(SqlExpression expression) => expression is SqlConstantExpression or SqlParameterExpression; diff --git a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseMemberTranslatorProvider.cs b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseMemberTranslatorProvider.cs index c97a393..aab320b 100644 --- a/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseMemberTranslatorProvider.cs +++ b/src/EFCore.ClickHouse/Query/ExpressionTranslators/Internal/ClickHouseMemberTranslatorProvider.cs @@ -16,6 +16,7 @@ public ClickHouseMemberTranslatorProvider( [ new ClickHouseArrayMethodTranslator(sqlExpressionFactory, typeMappingSource), new ClickHouseStringMethodTranslator(sqlExpressionFactory), + new ClickHouseDateTimeMemberTranslator(sqlExpressionFactory, typeMappingSource), ]); } } diff --git a/src/EFCore.ClickHouse/Query/Internal/ClickHouseSqlTranslatingExpressionVisitor.cs b/src/EFCore.ClickHouse/Query/Internal/ClickHouseSqlTranslatingExpressionVisitor.cs index dc66ad3..d1d04a9 100644 --- a/src/EFCore.ClickHouse/Query/Internal/ClickHouseSqlTranslatingExpressionVisitor.cs +++ b/src/EFCore.ClickHouse/Query/Internal/ClickHouseSqlTranslatingExpressionVisitor.cs @@ -44,4 +44,59 @@ protected override Expression VisitMethodCall(MethodCallExpression methodCallExp => _arrayLinqTranslator.TryTranslate(methodCallExpression, out var translated) ? translated : base.VisitMethodCall(methodCallExpression); + + /// + /// Reports a clear reason when two date/time values are added or subtracted. + /// + /// + /// + /// ClickHouse has no operator for any of these shapes, and each one fails differently: + /// + /// + /// + /// One date minus another gives a in .NET, whereas dateDiff + /// returns a count of whole units. + /// + /// + /// One time of day minus another gives a in .NET, whereas ClickHouse + /// Time64 subtraction gives a Decimal number of seconds. + /// + /// + /// A date plus or minus a keeps the date type in .NET, whereas ClickHouse + /// rejects the mixed operands outright (Illegal types ... of arguments of function plus). + /// + /// + /// + /// Left alone, each of these reaches type-mapping inference or the server and fails with an internal + /// cast error or raw SQL error that names types the user never wrote. Reporting the reason here turns + /// that into EF Core's normal "could not be translated" message with an explanation attached — which + /// also restores client evaluation in a projection, where the .NET result is correct. + /// + /// + protected override Expression VisitBinary(BinaryExpression binaryExpression) + { + if (binaryExpression.NodeType is ExpressionType.Add or ExpressionType.Subtract + && IsDateOrTimeType(binaryExpression.Left.Type) + && IsDateOrTimeType(binaryExpression.Right.Type)) + { + AddTranslationErrorDetails( + "Arithmetic on two date or time values is not supported, because ClickHouse has no " + + "operator that matches the .NET result. Compare the two values directly, or project " + + "them and do the arithmetic on the client."); + + return QueryCompilationContext.NotTranslatedExpression; + } + + return base.VisitBinary(binaryExpression); + } + + private static bool IsDateOrTimeType(Type type) + { + var unwrapped = Nullable.GetUnderlyingType(type) ?? type; + return unwrapped == typeof(DateTime) + || unwrapped == typeof(DateTimeOffset) + || unwrapped == typeof(DateOnly) + || unwrapped == typeof(TimeSpan) + || unwrapped == typeof(TimeOnly); + } } diff --git a/test/EFCore.ClickHouse.FunctionalTests/Query/NorthwindJoinQueryClickHouseTest.cs b/test/EFCore.ClickHouse.FunctionalTests/Query/NorthwindJoinQueryClickHouseTest.cs index 6aaffeb..7c26242 100644 --- a/test/EFCore.ClickHouse.FunctionalTests/Query/NorthwindJoinQueryClickHouseTest.cs +++ b/test/EFCore.ClickHouse.FunctionalTests/Query/NorthwindJoinQueryClickHouseTest.cs @@ -36,11 +36,6 @@ public override Task Take_in_collection_projection_with_FirstOrDefault_on_top_le public override Task SelectMany_with_client_eval_with_constructor(bool async) => AssertUnsupported(() => base.SelectMany_with_client_eval_with_constructor(async)); - // Complex LINQ pattern not translatable - public override Task GroupJoin_aggregate_anonymous_key_selectors2(bool async) - => Assert.ThrowsAsync( - () => base.GroupJoin_aggregate_anonymous_key_selectors2(async)); - private static async Task AssertUnsupported(Func test) => await Assert.ThrowsAsync(test); } diff --git a/test/EFCore.ClickHouse.Tests/DateTimeMemberTranslationTests.cs b/test/EFCore.ClickHouse.Tests/DateTimeMemberTranslationTests.cs new file mode 100644 index 0000000..c62c13c --- /dev/null +++ b/test/EFCore.ClickHouse.Tests/DateTimeMemberTranslationTests.cs @@ -0,0 +1,692 @@ +using Microsoft.EntityFrameworkCore; +using Xunit; + +namespace EFCore.ClickHouse.Tests; + +public class DateTimeMemberEntity +{ + public long Id { get; set; } + + /// Mapped to ClickHouse DateTime, which holds whole seconds only. + public DateTime Timestamp { get; set; } + + /// Mapped to ClickHouse DateTime64(7). Precision 7 is one .NET tick. + public DateTime Timestamp64 { get; set; } + + /// Mapped to ClickHouse Date32. + public DateOnly Date { get; set; } + + /// Mapped to ClickHouse DateTime64(7, 'UTC') by the DateTimeOffset mapping. + public DateTimeOffset Offset { get; set; } +} + +public class DateTimeMemberDbContext : DbContext +{ + public DbSet Events => Set(); + + private readonly string _connectionString; + + public DateTimeMemberDbContext(string connectionString) + { + _connectionString = connectionString; + } + + protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder) + { + optionsBuilder.UseClickHouse(_connectionString); + } + + protected override void OnModelCreating(ModelBuilder modelBuilder) + { + modelBuilder.Entity(entity => + { + entity.ToTable("datetime_member_test"); + entity.HasKey(e => e.Id); + entity.Property(e => e.Id).HasColumnName("id"); + entity.Property(e => e.Timestamp).HasColumnName("ts"); + entity.Property(e => e.Timestamp64).HasColumnName("ts64").HasColumnType("DateTime64(7)"); + entity.Property(e => e.Date).HasColumnName("d"); + entity.Property(e => e.Offset).HasColumnName("off"); + }); + } +} + +public class DateTimeMemberFixture : IAsyncLifetime +{ + public string ConnectionString { get; private set; } = string.Empty; + + /// + /// Row 1's instant: Sunday 2026-08-16 13:47:32.1234567. A Sunday on purpose — ClickHouse + /// toDayOfWeek gives Sunday 7 in its default mode and 0 in mode 2, so a Sunday is the day + /// that proves the mode argument reaches the server. + /// + public static readonly DateTime Instant = new DateTime(2026, 8, 16, 13, 47, 32).AddTicks(1_234_567); + + /// Row 1's time of day, to one tick. + public static readonly TimeSpan InstantTimeOfDay = TimeSpan.FromTicks(496_521_234_567); + + public async Task InitializeAsync() + { + ConnectionString = await SharedContainer.GetConnectionStringAsync(); + + using var connection = new global::ClickHouse.Driver.ADO.ClickHouseConnection(ConnectionString); + await connection.OpenAsync(); + + using var createCmd = connection.CreateCommand(); + createCmd.CommandText = """ + CREATE TABLE datetime_member_test ( + id Int64, + ts DateTime, + ts64 DateTime64(7), + d Date32, + off DateTime64(7, 'UTC') + ) ENGINE = MergeTree() + ORDER BY id + """; + await createCmd.ExecuteNonQueryAsync(); + + using var insertCmd = connection.CreateCommand(); + // Row 2 is the last day of a month, so AddMonths and AddYears have a day to clamp. + insertCmd.CommandText = """ + INSERT INTO datetime_member_test (id, ts, ts64, d, off) VALUES + (1, '2026-08-16 13:47:32', '2026-08-16 13:47:32.1234567', '2026-08-16', '2026-08-16 13:47:32.1234567'), + (2, '2026-01-31 00:00:00', '2026-01-31 00:00:00.0000000', '2026-01-31', '2026-01-31 00:00:00.0000000') + """; + await insertCmd.ExecuteNonQueryAsync(); + } + + public Task DisposeAsync() => Task.CompletedTask; +} + +public class DateTimeMemberTranslationTest : IClassFixture +{ + private readonly DateTimeMemberFixture _fixture; + + public DateTimeMemberTranslationTest(DateTimeMemberFixture fixture) + { + _fixture = fixture; + } + + private async Task SelectSingleAsync( + Func, IQueryable> selector, + long id = 1) + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + return await selector(context.Events.AsNoTracking().Where(e => e.Id == id)).SingleAsync(); + } + + private async Task> WhereIdsAsync( + System.Linq.Expressions.Expression> predicate) + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + return await context.Events.AsNoTracking().Where(predicate) + .OrderBy(e => e.Id).Select(e => e.Id).ToListAsync(); + } + + // ---------------------------------------------------------------- components + + [Fact] + public async Task Year_translates_to_toYear() + => Assert.Equal(2026, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.Year))); + + [Fact] + public async Task Month_translates_to_toMonth() + => Assert.Equal(8, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.Month))); + + [Fact] + public async Task Day_translates_to_toDayOfMonth() + => Assert.Equal(16, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.Day))); + + [Fact] + public async Task Hour_translates_to_toHour() + => Assert.Equal(13, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.Hour))); + + [Fact] + public async Task Minute_translates_to_toMinute() + => Assert.Equal(47, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.Minute))); + + [Fact] + public async Task Second_translates_to_toSecond() + => Assert.Equal(32, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.Second))); + + [Fact] + public async Task Millisecond_translates_to_toMillisecond() + => Assert.Equal(123, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.Millisecond))); + + [Fact] + public async Task DayOfYear_translates_to_toDayOfYear() + => Assert.Equal(228, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.DayOfYear))); + + [Fact] + public async Task Components_agree_with_dotnet_on_a_second_precision_column() + { + var result = await SelectSingleAsync(q => q.Select(e => new { e.Timestamp.Year, e.Timestamp.Hour, e.Timestamp.Second })); + + Assert.Equal(DateTimeMemberFixture.Instant.Year, result.Year); + Assert.Equal(DateTimeMemberFixture.Instant.Hour, result.Hour); + Assert.Equal(DateTimeMemberFixture.Instant.Second, result.Second); + } + + // ---------------------------------------------------------------- DayOfWeek + + [Fact] + public async Task DayOfWeek_projects_the_dotnet_value() + { + var result = await SelectSingleAsync(q => q.Select(e => e.Timestamp64.DayOfWeek)); + + // .NET DayOfWeek.Sunday is 0. ClickHouse mode 2 agrees; the default mode would give 7. + Assert.Equal(DayOfWeek.Sunday, result); + } + + [Fact] + public async Task DayOfWeek_compares_against_a_dotnet_constant() + { + // The provider maps a C# enum to a ClickHouse string, so this is the test that proves the + // constant renders as a number rather than as 'Sunday'. + Assert.Equal([1L], await WhereIdsAsync(e => e.Timestamp64.DayOfWeek == DayOfWeek.Sunday)); + } + + [Fact] + public async Task DayOfWeek_of_a_Saturday_is_six() + { + // Row 2 is 2026-01-31, a Saturday. + Assert.Equal(DayOfWeek.Saturday, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.DayOfWeek), id: 2)); + } + + // ---------------------------------------------------------------- Date / TimeOfDay + + [Fact] + public async Task Date_translates_to_toStartOfDay() + => Assert.Equal(new DateTime(2026, 8, 16), await SelectSingleAsync(q => q.Select(e => e.Timestamp64.Date))); + + [Fact] + public async Task TimeOfDay_keeps_tick_precision() + { + var result = await SelectSingleAsync(q => q.Select(e => e.Timestamp64.TimeOfDay)); + + // toTime would drop the fraction; toTime64(x, 7) keeps every tick. + Assert.Equal(DateTimeMemberFixture.InstantTimeOfDay, result); + } + + // ---------------------------------------------------------------- DateOnly + + [Fact] + public async Task DateOnly_components_translate() + { + var result = await SelectSingleAsync(q => q.Select(e => new + { + e.Date.Year, + e.Date.Month, + e.Date.Day, + e.Date.DayOfYear, + e.Date.DayOfWeek + })); + + Assert.Equal(2026, result.Year); + Assert.Equal(8, result.Month); + Assert.Equal(16, result.Day); + Assert.Equal(228, result.DayOfYear); + Assert.Equal(DayOfWeek.Sunday, result.DayOfWeek); + } + + // ---------------------------------------------------------------- DateTimeOffset + + [Fact] + public async Task DateTimeOffset_components_translate() + { + var result = await SelectSingleAsync(q => q.Select(e => new + { + e.Offset.Year, + e.Offset.Month, + e.Offset.Day, + e.Offset.Hour, + e.Offset.Minute, + e.Offset.Second, + e.Offset.DayOfYear, + e.Offset.DayOfWeek + })); + + // The store type is UTC-pinned, and a value read back carries +00:00, so every component + // describes the same instant on both sides. + Assert.Equal(2026, result.Year); + Assert.Equal(8, result.Month); + Assert.Equal(16, result.Day); + Assert.Equal(13, result.Hour); + Assert.Equal(47, result.Minute); + Assert.Equal(32, result.Second); + Assert.Equal(228, result.DayOfYear); + Assert.Equal(DayOfWeek.Sunday, result.DayOfWeek); + } + + [Fact] + public async Task DateTimeOffset_components_agree_with_dotnet() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + + var expected = await context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Offset).SingleAsync(); + var actual = await SelectSingleAsync(q => q.Select(e => new { e.Offset.Year, e.Offset.Hour, e.Offset.Second })); + + Assert.Equal(expected.Year, actual.Year); + Assert.Equal(expected.Hour, actual.Hour); + Assert.Equal(expected.Second, actual.Second); + } + + [Fact] + public async Task DateTimeOffset_Date_translates() + => Assert.Equal( + new DateTime(2026, 8, 16), + await SelectSingleAsync(q => q.Select(e => e.Offset.Date))); + + [Fact] + public async Task DateTimeOffset_TimeOfDay_keeps_tick_precision() + => Assert.Equal( + DateTimeMemberFixture.InstantTimeOfDay, + await SelectSingleAsync(q => q.Select(e => e.Offset.TimeOfDay))); + + [Fact] + public async Task DateTimeOffset_AddDays_translates() + { + var result = await SelectSingleAsync(q => q.Select(e => e.Offset.AddDays(1))); + + Assert.Equal(new DateTimeOffset(2026, 8, 17, 13, 47, 32, TimeSpan.Zero).AddTicks(1_234_567), result); + } + + [Fact] + public async Task DateTimeOffset_AddMonths_clamps_like_dotnet() + => Assert.Equal( + new DateTimeOffset(2026, 2, 28, 0, 0, 0, TimeSpan.Zero), + await SelectSingleAsync(q => q.Select(e => e.Offset.AddMonths(1)), id: 2)); + + [Fact] + public async Task DateTimeOffset_DayOfWeek_compares_against_a_dotnet_constant() + => Assert.Equal([1L], await WhereIdsAsync(e => e.Offset.DayOfWeek == DayOfWeek.Sunday)); + + [Fact] + public async Task DateTimeOffset_UtcNow_runs_on_the_server() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking() + .Where(e => e.Offset < DateTimeOffset.UtcNow.AddYears(100)) + .Select(e => e.Id); + + Assert.Contains("now64", query.ToQueryString()); + Assert.Equal([1L, 2L], await query.OrderBy(id => id).ToListAsync()); + } + + // ---------------------------------------------------------------- Add* + + [Fact] + public async Task AddYears_translates_to_addYears() + => Assert.Equal( + DateTimeMemberFixture.Instant.AddYears(1), + await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddYears(1)))); + + [Fact] + public async Task AddMonths_translates_to_addMonths() + => Assert.Equal( + DateTimeMemberFixture.Instant.AddMonths(1), + await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddMonths(1)))); + + [Fact] + public async Task AddMonths_clamps_the_day_like_dotnet() + { + // Row 2 is 2026-01-31, so one month lands on 2026-02-28 in both systems. + var result = await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddMonths(1)), id: 2); + + Assert.Equal(new DateTime(2026, 2, 28), result); + Assert.Equal(new DateTime(2026, 1, 31).AddMonths(1), result); + } + + [Fact] + public async Task AddDays_with_a_whole_number_translates_to_addDays() + => Assert.Equal( + DateTimeMemberFixture.Instant.AddDays(1), + await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddDays(1)))); + + [Fact] + public async Task AddDays_with_a_negative_whole_number_translates_to_addDays() + => Assert.Equal( + DateTimeMemberFixture.Instant.AddDays(-1), + await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddDays(-1)))); + + [Fact] + public async Task AddDays_with_a_fraction_keeps_dotnet_semantics() + { + // addDays(x, 1.5) would discard the fraction and add one day. The translation must not. + var result = await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddDays(1.5))); + + Assert.Equal(DateTimeMemberFixture.Instant.AddDays(1.5), result); + Assert.Equal(new DateTime(2026, 8, 18, 1, 47, 32).AddTicks(1_234_567), result); + } + + [Fact] + public async Task AddHours_translates() + => Assert.Equal( + DateTimeMemberFixture.Instant.AddHours(2), + await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddHours(2)))); + + [Fact] + public async Task AddMinutes_with_a_fraction_keeps_dotnet_semantics() + => Assert.Equal( + DateTimeMemberFixture.Instant.AddMinutes(0.5), + await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddMinutes(0.5)))); + + [Fact] + public async Task AddSeconds_with_a_fraction_keeps_dotnet_semantics() + => Assert.Equal( + DateTimeMemberFixture.Instant.AddSeconds(1.5), + await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddSeconds(1.5)))); + + [Fact] + public async Task AddMilliseconds_translates() + => Assert.Equal( + DateTimeMemberFixture.Instant.AddMilliseconds(1), + await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddMilliseconds(1)))); + + [Fact] + public async Task AddDays_on_a_Date32_column_keeps_the_date_store_type() + { + // DateOnly.AddDays takes an int, so this must use addDays — addMilliseconds rejects a Date32. + var result = await SelectSingleAsync(q => q.Select(e => e.Date.AddDays(1))); + + Assert.Equal(new DateOnly(2026, 8, 17), result); + } + + [Fact] + public async Task AddMonths_on_a_Date32_column_clamps_like_dotnet() + => Assert.Equal( + new DateOnly(2026, 2, 28), + await SelectSingleAsync(q => q.Select(e => e.Date.AddMonths(1)), id: 2)); + + [Fact] + public async Task AddSeconds_below_millisecond_resolution_keeps_dotnet_semantics() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64.AddSeconds(0.1234567)); + + // .NET adds exactly 1 234 567 ticks. No ClickHouse unit holds that without promoting the result + // to DateTime64(9), so the call is left untranslated and the client supplies the exact value. + Assert.DoesNotContain("addSeconds", query.ToQueryString()); + Assert.DoesNotContain("addMilliseconds", query.ToQueryString()); + + var result = await query.SingleAsync(); + Assert.Equal(DateTimeMemberFixture.Instant.AddSeconds(0.1234567), result); + Assert.Equal(DateTimeMemberFixture.Instant.AddTicks(1_234_567), result); + } + + [Fact] + public async Task AddMilliseconds_below_millisecond_resolution_keeps_dotnet_semantics() + { + // .NET rounds to the nearest tick, so this adds 5 000 ticks — not 0 ms and not 1 ms. + var result = await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddMilliseconds(0.5))); + + Assert.Equal(DateTimeMemberFixture.Instant.AddMilliseconds(0.5), result); + Assert.Equal(DateTimeMemberFixture.Instant.AddTicks(5_000), result); + } + + [Fact] + public async Task AddDays_with_a_parameter_keeps_dotnet_semantics() + { + var days = 1.5; + + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64.AddDays(days)); + + // A parameter cannot be checked for exactness, so it is not translated. Rounding it on the server + // would disagree with .NET, because ClickHouse round() is banker's rounding. + Assert.DoesNotContain("addDays", query.ToQueryString()); + Assert.Equal(DateTimeMemberFixture.Instant.AddDays(1.5), await query.SingleAsync()); + } + + [Fact] + public async Task AddDays_with_a_parameter_in_a_predicate_reports_a_reason() + { + var days = 1.5; + + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Timestamp64.AddDays(days) > e.Timestamp); + + await Assert.ThrowsAsync(() => query.ToListAsync()); + } + + [Fact] + public async Task AddDays_beyond_the_dotnet_range_throws_the_dotnet_exception() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64.AddDays(1e30)); + + // Folding this would wrap to Int64.MaxValue and give a server decimal-overflow error, or worse a + // silently wrong date. Left untranslated, .NET raises its own exception on the client. + await Assert.ThrowsAsync(() => query.SingleAsync()); + } + + [Fact] + public async Task Add_composes_with_a_component_member() + => Assert.Equal(2027, await SelectSingleAsync(q => q.Select(e => e.Timestamp64.AddYears(1).Year))); + + // ---------------------------------------------------------------- server clock + + [Fact] + public async Task UtcNow_runs_on_the_server() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + + // Both bounds are offset by a century so the outcome does not depend on the day the suite runs. + var query = context.Events.AsNoTracking().Where(e => e.Timestamp64 < DateTime.UtcNow.AddYears(100)) + .Select(e => e.Id); + + // If EF Core evaluated DateTime.UtcNow on the client, the SQL would carry a literal instead. + Assert.Contains("now64", query.ToQueryString()); + Assert.Equal([1L, 2L], await query.OrderBy(id => id).ToListAsync()); + + var none = await context.Events.AsNoTracking() + .Where(e => e.Timestamp64 < DateTime.UtcNow.AddYears(-100)) + .Select(e => e.Id).ToListAsync(); + + Assert.Empty(none); + } + + [Fact] + public async Task Today_runs_on_the_server() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + var query = context.Events.AsNoTracking().Where(e => e.Timestamp64 < DateTime.Today) + .Select(e => e.Id); + + Assert.Contains("toStartOfDay", query.ToQueryString()); + Assert.Contains("now()", query.ToQueryString()); + + // Execute it too: a SQL-shape assertion alone would not catch the server rejecting the call. + var beforeToday = await query.ToListAsync(); + + // Row 2 is 2026-01-31, which is before today on any day this suite can run. + Assert.Contains(2L, beforeToday); + } + + // ---------------------------------------------------------------- subtraction + + [Fact] + public async Task Subtracting_two_date_times_in_a_projection_now_works() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + + // Previously this failed with an opaque cast/coercion error from type-mapping inference. The + // subtraction is now reported as not translatable, so EF Core reads both columns and subtracts + // on the client, which is the correct .NET result. + var result = await context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64 - e.Timestamp).SingleAsync(); + + Assert.Equal(TimeSpan.FromTicks(1_234_567), result); + } + + [Fact] + public async Task Subtracting_two_date_times_in_a_predicate_reports_a_clear_reason() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + + // A predicate cannot fall back to the client, so this is where the reason must surface. + var query = context.Events.AsNoTracking() + .Where(e => e.Timestamp64 - e.Timestamp > TimeSpan.Zero); + + var exception = await Assert.ThrowsAsync(() => query.ToListAsync()); + + Assert.Contains("Arithmetic on two date or time values", exception.Message); + } + + [Fact] + public async Task Subtracting_two_times_of_day_in_a_projection_works() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + + // ClickHouse Time64 subtraction gives a Decimal of seconds, not a TimeSpan, so this must stay + // on the client rather than emit SQL that materializes into the wrong type. + var result = await context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64.TimeOfDay - e.Timestamp.TimeOfDay).SingleAsync(); + + Assert.Equal(TimeSpan.FromTicks(1_234_567), result); + } + + [Fact] + public async Task Adding_a_time_of_day_to_a_date_in_a_projection_works() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + + // ClickHouse rejects DateTime + Time64 outright, so this must stay on the client too. + var result = await context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64.Date + e.Timestamp64.TimeOfDay).SingleAsync(); + + Assert.Equal(DateTimeMemberFixture.Instant, result); + } + + [Fact] + public async Task Adding_a_TimeSpan_to_a_date_in_a_projection_works() + { + await using var context = new DateTimeMemberDbContext(_fixture.ConnectionString); + + var result = await context.Events.AsNoTracking().Where(e => e.Id == 1) + .Select(e => e.Timestamp64 + TimeSpan.FromHours(1)).SingleAsync(); + + Assert.Equal(DateTimeMemberFixture.Instant.AddHours(1), result); + } +} + +public class DateTimeMemberTranslationOfflineTest +{ + private sealed class OfflineContext : DbContext + { + public DbSet Events => Set(); + + protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder) + { + optionsBuilder.UseClickHouse("Host=localhost;Protocol=http;Port=8123;Database=test"); + } + + protected override void OnModelCreating(ModelBuilder modelBuilder) + { + modelBuilder.Entity(entity => + { + entity.ToTable("datetime_member_test"); + entity.HasKey(e => e.Id); + entity.Property(e => e.Timestamp64).HasColumnType("DateTime64(7)"); + }); + } + } + + private static string Sql(Func, IQueryable> selector) + { + using var context = new OfflineContext(); + return selector(context.Events).ToQueryString(); + } + + [Fact] + public void Year_emits_toYear() + => Assert.Contains("toYear(", Sql(q => q.Select(e => e.Timestamp64.Year))); + + [Fact] + public void Day_emits_toDayOfMonth() + => Assert.Contains("toDayOfMonth(", Sql(q => q.Select(e => e.Timestamp64.Day))); + + [Fact] + public void DayOfWeek_emits_week_mode_two() + { + var sql = Sql(q => q.Select(e => e.Timestamp64.DayOfWeek)); + + // Assert the mode argument itself — a bare "2" would also match toDayOfWeek(x, 12). + Assert.Contains(", 2)", sql); + Assert.Contains("toDayOfWeek(", sql); + } + + [Fact] + public void DayOfWeek_comparison_emits_a_number_not_a_string() + { + var sql = Sql(q => q.Where(e => e.Timestamp64.DayOfWeek == DayOfWeek.Sunday).Select(e => e.Id)); + + Assert.DoesNotContain("'Sunday'", sql); + Assert.Contains("= 0", sql); + } + + [Fact] + public void TimeOfDay_emits_toTime64_with_tick_precision() + => Assert.Contains("toTime64(", Sql(q => q.Select(e => e.Timestamp64.TimeOfDay))); + + [Fact] + public void AddDays_with_a_whole_number_emits_addDays() + { + var sql = Sql(q => q.Select(e => e.Timestamp64.AddDays(1))); + + Assert.Contains("addDays(", sql); + Assert.DoesNotContain("addMilliseconds", sql); + } + + [Fact] + public void AddDays_with_a_fraction_emits_addMilliseconds() + { + var sql = Sql(q => q.Select(e => e.Timestamp64.AddDays(1.5))); + + // 1.5 days is exactly 129 600 000 ms, folded at translation time. + Assert.Contains("addMilliseconds(", sql); + Assert.Contains("129600000", sql); + } + + [Fact] + public void AddMilliseconds_below_millisecond_resolution_emits_no_add_function() + { + // 0.5 ms is 5 000 ticks, which no ClickHouse unit holds exactly without promoting to + // DateTime64(9), so the call must not be translated. + var sql = Sql(q => q.Select(e => e.Timestamp64.AddMilliseconds(0.5))); + + Assert.DoesNotContain("addMilliseconds", sql); + Assert.DoesNotContain("addNanoseconds", sql); + } + + [Fact] + public void AddSeconds_with_a_whole_number_of_milliseconds_emits_addMilliseconds() + { + // 1.5 s is 1 500 ms exactly, so it is still translatable — just not in seconds. + var sql = Sql(q => q.Select(e => e.Timestamp64.AddSeconds(1.5))); + + Assert.Contains("addMilliseconds(", sql); + Assert.Contains("1500", sql); + } + + [Fact] + public void AddHours_with_a_whole_number_emits_addHours() + => Assert.Contains("addHours(", Sql(q => q.Select(e => e.Timestamp64.AddHours(3)))); + + [Fact] + public void UtcNow_emits_a_utc_pinned_now64() + => Assert.Contains("now64(7, 'UTC')", Sql(q => q.Where(e => e.Timestamp64 < DateTime.UtcNow).Select(e => e.Id))); + + [Fact] + public void Now_emits_a_timezone_less_now64() + { + var sql = Sql(q => q.Where(e => e.Timestamp64 < DateTime.Now).Select(e => e.Id)); + + Assert.Contains("now64(7)", sql); + Assert.DoesNotContain("'UTC'", sql); + } +}