Skip to content

Map DateTimeOffset to DateTime64 and fix composite component reads (#53) - #63

Draft
alex-clickhouse wants to merge 1 commit into
mainfrom
fix/issue-53-datetimeoffset
Draft

Map DateTimeOffset to DateTime64 and fix composite component reads (#53)#63
alex-clickhouse wants to merge 1 commit into
mainfrom
fix/issue-53-datetimeoffset

Conversation

@alex-clickhouse

Copy link
Copy Markdown
Collaborator

Fixes #53.

Problem

The provider had no mapping for DateTimeOffset. EF Core therefore fell back to
DateTimeOffsetToStringConverter through the value-converter selector, and made a String column
without a warning. Two failures came from this:

  • A query against a real DateTime64 column failed with TYPE_MISMATCH, because the provider
    declared the parameter as String.
  • SaveChanges could not write the value at all.

Solution

A DateTimeOffset property now maps to DateTime64(7, 'UTC') through the new
ClickHouseDateTimeOffsetTypeMapping. No value converter is used, so the driver receives the
DateTimeOffset directly on the query parameter path and on the bulk insert path.

Three design decisions are important:

The store type pins the timezone to 'UTC'. For a timezone-less parameter type such as
DateTime64(7), the driver sends a UTC wall clock, and the server then reads it in
session_timezone. The instant moves when that setting is not UTC. A UTC-pinned store type removes
this dependency on server configuration.

Precision 7 is one .NET tick (100 ns). The round trip is therefore exact, and no value comes
back truncated. Precision 7 also covers the full DateTimeOffset range, so MinValue and MaxValue
work as open-ended range limits.

The offset is not kept. ClickHouse has no type that stores a UTC offset. DateTime64 holds an
instant, and a declared timezone only decides how that instant is rendered. The instant is preserved
and the offset is not, so a value read back carries the offset of the column's declared timezone.
The README records this, together with the alternative for a caller who must keep the offset.

Also in this PR: three composite-mapping defects

Work on #53 exposed these. They are in the same PR because DateTimeOffset inside a composite type
does not work without them, and because they touch the same resolution path in
ClickHouseTypeMappingSource.

  1. Composite columns did not convert their components on read. Array(T), Map(K, V) and
    Tuple(...) read the whole column through GetValue, so a component mapping's own read pipeline
    never ran. Any component whose CLR type differs from the driver's type threw
    InvalidCastException. This was not new with DateTimeOffsetDateOnly[],
    Dictionary<string, DateOnly> and Tuple<DateOnly, …> were already affected, because DateOnly
    also arrives from the driver as a DateTime.

    The new ClickHouseComponentConversion helper rebuilds the composite component by component. It
    applies the same two steps EF Core applies to a scalar column: the mapping's data-reader
    conversion, then its ValueConverter. A component that needs no conversion keeps the direct
    cast, and a runtime fast path returns the driver's array untouched when it is already the target
    type.

  2. Array(Nullable(T)) DDL was double-wrapped for a value-type element, which gave
    Array(Nullable(Nullable(T))). ClickHouse rejects this with Nested type Nullable(T) cannot be inside Nullable type, so EnsureCreated and migrations both failed.

  3. A component mapping ignored the CLR component type. One store type can serve more than one
    CLR type: DateTime64 serves DateTime and DateTimeOffset, and Date32 serves DateTime and
    DateOnly. Resolving a component from an explicit store type always picked the default CLR type,
    which gave the composite the wrong element type and broke change tracking.

Tests

DateTimeOffsetMappingTests (680 lines) and CompositeElementConversionTests (484 lines), 59 tests
in total. All run against a real ClickHouse server through Testcontainers, as the guidance in
AGENTS.md prefers.

They cover the round trip at each precision, a non-UTC column timezone, the ambiguous
daylight-saving hour, ordering and comparison, MinValue/MaxValue, SQL literal generation,
HasPrecision, the bulk insert path, and DateTimeOffset and DateOnly inside Array, Map,
Tuple and nested composites.

Known limits

  • Writing a component that needs a ValueConverter still does not work, because the bulk insert
    path passes model values to the driver without applying converters (SaveChanges fails for value-converted properties: bulk insert path does not apply the converter #54). An enum inside a
    composite is written as its raw ordinal. Reading such a column works.
  • DateTimeOffset members such as .Year do not translate to SQL yet (No LINQ translation for DateTime members and methods #55).
  • In a column timezone with daylight saving, the repeated hour when clocks go back is ambiguous,
    because the driver gives a wall clock and drops the offset. The provider recovers the instant
    where the zone's standard offset is zero, such as Europe/London. Where both candidate offsets
    are non-zero, such as Europe/Paris, the value can read back one hour early. The default 'UTC'
    store type is not affected.

Behaviour change

A DateTimeOffset property that relied on the old String column now resolves to
DateTime64(7, 'UTC'). Add HasConversion<string>() to keep the previous shape. Note that
HasColumnType("String") on its own is not enough, because it adds no converter.

Note for the reviewer

ClickHouseEngineBuilder.cs has two XML doc comments only, which explain the relationship between
WithOrderBy and WithPrimaryKey. This is unrelated to #53. Say the word and I will take it out.

🤖 Generated with Claude Code

Map DateTimeOffset to DateTime64(7, 'UTC'). The provider had no mapping
for the type, so EF Core fell back to DateTimeOffsetToStringConverter and
silently made a String column. That broke queries against real DateTime64
columns with TYPE_MISMATCH, and SaveChanges could not write the value.

The store type pins the timezone to 'UTC'. For a timezone-less parameter
type the driver sends a UTC wall clock, which the server then reads in
session_timezone. Precision 7 is one .NET tick, so the round trip is
exact. No value converter is used, so the driver gets the DateTimeOffset
directly on the query parameter path and the bulk insert path.

ClickHouse stores no UTC offset, so the instant is kept and the offset is
not. A value read back carries the offset of the column's declared
timezone.

Also fix three defects in the composite mappings, which #53 exposed:

* Array(T), Map(K, V) and Tuple(...) read the whole column through
  GetValue, so a component mapping's read pipeline never ran. Any
  component whose CLR type differs from the driver's type therefore threw
  InvalidCastException. DateOnly components were already affected before
  DateTimeOffset existed as a mapped type. Composites are now rebuilt
  component by component, applying the data-reader conversion and then
  the ValueConverter.
* Array(Nullable(T)) DDL was double-wrapped for a value-type element,
  which ClickHouse rejects.
* A component resolved from an explicit store type always picked the
  default CLR type, because one store type can serve more than one CLR
  type. Array, Map and Tuple now pass the component CLR type.

Known limit: writing a component that needs a ValueConverter still does
not work, because the bulk insert path skips converters (#54).

Co-Authored-By: Claude <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds native DateTimeOffset support and repairs composite component materialization.

Changes:

  • Maps DateTimeOffset to UTC-pinned DateTime64(7).
  • Converts Array, Map, and Tuple components during reads.
  • Adds integration coverage and user documentation.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
ClickHouseDateTimeOffsetTypeMapping.cs Implements mapping, literals, and timezone-aware reads.
ClickHouseComponentConversion.cs Centralizes component conversion.
ClickHouseArrayTypeMapping.cs Rebuilds arrays requiring conversion.
ClickHouseMapTypeMapping.cs Rebuilds converted map entries.
ClickHouseTupleTypeMapping.cs Converts tuple components.
ClickHouseNullableElementMapping.cs Handles nullable component reads and DDL.
ClickHouseTypeMappingSource.cs Registers and resolves new mappings.
ClickHouseEngineBuilder.cs Documents sorting and primary keys.
DateTimeOffsetMappingTests.cs Tests mapping and round trips.
CompositeElementConversionTests.cs Tests composite materialization.
README.md Documents DateTimeOffset behavior.
CHANGELOG.md Records feature and fixes.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +111 to +113
// The driver often already produces the target type, for example Array(Int32) -> int[].
if (value is TElement[] alreadyTyped)
return alreadyTyped;
Comment on lines +72 to +73
if (value is T alreadyTyped)
return alreadyTyped;
Comment thread README.md
Comment on lines +109 to +111
If you must keep the offset, store it yourself in a second column alongside a `DateTime`. To keep
the whole value as text, ask for the conversion explicitly with `HasConversion<string>()` — note
that `HasColumnType("String")` on its own is not enough, because it adds no converter.
Comment thread CHANGELOG.md
* `HasPrecision(n)` is honoured for a property with no `HasColumnType`, and keeps the UTC pin — for example `HasPrecision(3)` gives `DateTime64(3, 'UTC')`.
* SQL literals carry the offset (`'2026-01-15 10:00:00.1234567+05:00'`), which lands on the same instant whatever timezone the target column declares.
* No value converter is used, so the driver receives the `DateTimeOffset` directly on both the query parameter path and the bulk insert path.
* **Behaviour change:** a `DateTimeOffset` property that relied on the old `String` column now resolves to `DateTime64(7, 'UTC')`. Add `HasConversion<string>()` to keep the previous shape. Note that `HasColumnType("String")` on its own is not enough — it resolves the plain string mapping with no converter, so the CLR type no longer agrees with the property.
Comment on lines +77 to +78
if (value is Dictionary<TKey, TValue> alreadyTyped)
return alreadyTyped;
@alex-clickhouse

Copy link
Copy Markdown
Collaborator Author

Let's make sure we have tests with Fixed/UTC±HH:MM:SS timezone (which clickhouse supports and .NET doesn't natively...there is support for them in the client lib).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DateTimeOffset not supported

2 participants