Skip to content

fix(mysql,starrocks)!: correct DATEDIFF, LIKE, VARIANCE and TO_DAYS semantics [CLAUDE] - #7967

Closed
UCHPUCHMAK wants to merge 1 commit into
tobymao:mainfrom
UCHPUCHMAK:starrocks-datediff-like-variance
Closed

fix(mysql,starrocks)!: correct DATEDIFF, LIKE, VARIANCE and TO_DAYS semantics [CLAUDE]#7967
UCHPUCHMAK wants to merge 1 commit into
tobymao:mainfrom
UCHPUCHMAK:starrocks-datediff-like-variance

Conversation

@UCHPUCHMAK

Copy link
Copy Markdown
Contributor

Fixes four rewrites in the StarRocks/MySQL dialects that silently produced different numbers rather than an error:

# Input Generated before Result before → after
1 DATEDIFF(a, b) (StarRocks) DATE_DIFF('DAY', a, b) 01 on DATETIMEs crossing midnight
2 like(a, 'x%') (StarRocks) 'x%' LIKE a predicate flipped: 01
3 VAR_SAMP(x) / VAR_POP(x) (MySQL family) VARIANCE(x) / VARIANCE_POP(x) sample silently became population; VARIANCE_POP isn't valid MySQL
4 TO_DAYS(x) (StarRocks) (DATE_DIFF('DAY', DATE(x), DATE('0000-01-01')) + 1) off by one: 740051740050

How this was verified

Every claim below was checked by executing both the original and the generated SQL on a live StarRocks 3.5.18 cluster and comparing results, not just parse round-trips. The MySQL-layer changes are additionally backed by the MySQL 8.4 Reference Manual, and the two other dialects inheriting the MySQL layer were checked explicitly: Doris (vendor docs confirm the same semantics) and SingleStore (already carried identical overrides, so this is a no-op there).

1. DATEDIFF ↔ DATE_DIFF

StarRocks DATEDIFF counts crossed day boundaries ("Only the date parts of the values are used"), while DATE_DIFF counts whole units, so rewriting one into the other is lossy on DATETIME inputs:

SELECT DATEDIFF('2026-03-11 01:00:00', '2026-03-10 23:00:00');         -- 1
SELECT DATE_DIFF('DAY', '2026-03-11 01:00:00', '2026-03-10 23:00:00'); -- 0  (what sqlglot generated)

DATEDIFF now parses into exp.DateDiff(..., date_part_boundary=True) — the flag BigQuery, MySQL and Snowflake parsers already use for boundary-crossing semantics. Generation handles the full unit matrix (each row executed on the cluster):

Boundary diff arriving at StarRocks Generated Notes
DAY, or no unit (MySQL DATEDIFF) DATEDIFF(a, b) native boundary semantics
MONTH, QUARTER, YEAR, HOUR, MINUTE, SECOND DATE_DIFF('U', DATE_TRUNC('U', a), DATE_TRUNC('U', b)) truncating the operands turns whole-unit counting into boundary counting, mirroring the Presto generator (#7911)
WEEK / WEEK(<DAY>) / ISOWEEK DATE_TRUNC('WEEK', x + INTERVAL 'n' DAY) on both operands StarRocks' DATE_TRUNC('WEEK', ...) is Monday-based; the shared week_unit_to_dow helper computes the shift (+1 for Sunday, 0 for Monday/ISOWEEK, ...)
MILLISECOND DATE_DIFF('MILLISECOND', a, b) untruncated the analyzer rejects sub-second DATE_TRUNC over DATETIME-typed arguments (verified on 3.5.18 even with NOW(6), despite the docs advertising support since 3.1.7); for second-precision values whole-ms and boundary counting coincide anyway
ISOYEAR and anything else DATE_TRUNC can't handle DATE_DIFF('ISOYEAR', a, b) untruncated fails loudly on the server instead of silently returning whole-unit counts
diffs from whole-unit dialects (StarRocks itself, DuckDB) plain DATE_DIFF unchanged

Sample cluster evidence for the truncation rows (raw DATE_DIFF returns 0 for all of these):

SELECT DATE_DIFF('MONTH', DATE_TRUNC('MONTH', CAST('2026-03-01 10:00:00' AS DATETIME)),
                          DATE_TRUNC('MONTH', CAST('2026-02-28 09:00:00' AS DATETIME)));  -- 1 (and -1 reversed)
SELECT DATE_DIFF('WEEK', DATE_TRUNC('WEEK', CAST('2026-03-15' AS DATE) + INTERVAL '1' DAY),
                         DATE_TRUNC('WEEK', CAST('2026-03-14' AS DATE) + INTERVAL '1' DAY)); -- 1 (Sunday-start week)

As a side effect, DATEDIFF from StarRocks now fans out correctly to other dialects too, e.g. postgres now gets (CAST(a AS DATE) - CAST(b AS DATE)) instead of an epoch division that counted whole days.

2. LIKE()

StarRocks like(expr, pattern) takes its arguments in the natural order, but the base builder reverses them, which flipped the predicate:

SELECT like('xyz', 'x%');  -- 1
SELECT 'x%' LIKE 'xyz';    -- 0  (what sqlglot generated)

The parser now uses build_like(exp.Like), same as snowflake/spark/clickhouse. The 3-arg ESCAPE form keeps working through the same builder.

3. VARIANCE family (MySQL layer)

Per the MySQL docs, VARIANCE() is a synonym for VAR_POP() ("Returns the population standard variance ... provided as a MySQL extension"), VAR_SAMP() is the sample variance, and VARIANCE_POP does not exist. sqlglot treated VARIANCE as the sample variance (exp.Variance._sql_names includes it) and generated VARIANCE_POP for exp.VariancePop, so:

  • VAR_SAMP(x)VARIANCE(x): a sample variance silently became a population one — 2.3333 vs 1.5556 on (1, 2, 4)
  • VAR_POP(x)VARIANCE_POP(x): invalid MySQL

The MySQL parser now maps VARIANCE to exp.VariancePop, and the generator emits VAR_SAMP/VAR_POP. Inheritance checked: StarRocks documents the same population semantics (variance,var_pop,variance_pop with a separate var_samp page), and so does Doris ("Unlike VARIANCE (population variance), VAR_SAMP uses n-1"); SingleStore already carried identical parser/generator overrides, which this makes redundant rather than conflicting.

4. TO_DAYS

StarRocks supports TO_DAYS natively, and MySQL's expansion is off by one there:

SELECT to_days('2026-03-10');                                        -- 740050
SELECT DATE_DIFF('DAY', DATE('2026-03-10'), DATE('0000-01-01')) + 1; -- 740051  (what sqlglot generated)

StarRocks now parses TO_DAYS into the existing exp.ToDays; MySQL keeps its expansion, which is correct for MySQL.

Breaking changes (hence the !)

  • MySQL/Doris round trips: VARIANCE(x)VAR_POP(x), VAR_SAMP(x) stays VAR_SAMP(x) (previously VARIANCE(x))
  • StarRocks round trips: DATEDIFF(a, b) stays DATEDIFF(a, b) (previously DATE_DIFF('DAY', a, b)), TO_DAYS(x) stays TO_DAYS(x)
  • StarRocks TO_DAYS no longer lowers into day arithmetic for targets without the function — round-trip fidelity on StarRocks is the priority, matching how the base dialect treats TO_DAYS

Deliberately out of scope

  • STDDEV: MySQL's STDDEV/STD/STDDEV_POP are population functions with the same naming quirk, but their round trip is already stable and changing exp.Stddev would affect every dialect
  • The reverse direction (whole-unit sources generating into boundary-counting dialects, e.g. StarRocks DATE_DIFF('MONTH', ...) → BigQuery) is a target-generator concern shared by all whole-unit dialects
  • TIMESTAMPDIFF flows through exp.TimestampDiff and is untouched

Tests: identity/cross-dialect coverage for every rewrite in test_starrocks.py/test_mysql.py, including pins for the MONTH/HOUR/WEEK/ISOWEEK/ISOYEAR unit shapes and the Doris/DuckDB/Postgres fan-outs.

🤖 Generated with Claude Code

…emantics [CLAUDE]

All four rewrites below silently produced different numbers rather than an
error. Verified by executing both the original and the generated form on
StarRocks 3.5.18.

StarRocks DATEDIFF counts crossed day boundaries while DATE_DIFF counts whole
units, so rewriting one into the other is lossy on DATETIME inputs:

    DATEDIFF('2026-03-11 01:00:00', '2026-03-10 23:00:00')         -> 1
    DATE_DIFF('DAY', '2026-03-11 01:00:00', '2026-03-10 23:00:00') -> 0

DATEDIFF now sets date_part_boundary, the flag already used for exactly these
semantics by BigQuery, MySQL and Snowflake. On generation, a boundary diff
over DAY (or with no unit, i.e. MySQL's DATEDIFF) lands on DATEDIFF, and any
other unit truncates its operands down to the unit, mirroring the Presto
generator, since whole-unit counting over truncated operands equals boundary
counting:

    DATE_DIFF('MONTH', '2026-03-01', '2026-02-28')                 -> 0
    DATE_DIFF('MONTH', DATE_TRUNC('MONTH', ...), DATE_TRUNC(...))  -> 1

Week units realign StarRocks' Monday-based DATE_TRUNC('WEEK', ...) to the
requested week start by shifting both operands. Units DATE_TRUNC cannot
handle, e.g. ISOYEAR, stay untruncated and fail loudly on the server;
MILLISECOND is also left alone: the analyzer rejects sub-second DATE_TRUNC
over DATETIME-typed arguments, and second-precision values coincide anyway. Diffs parsed from dialects that count
whole units keep generating plain DATE_DIFF.

StarRocks' LIKE(expr, pattern) takes its arguments in the natural order, but
the base builder reverses them, which flipped the predicate:

    like('xyz', 'x%') -> 1,  'x%' LIKE 'xyz' -> 0

MySQL's VARIANCE is a synonym of VAR_POP, not of VAR_SAMP, and MySQL has no
VARIANCE_POP at all, so exp.Variance -> VARIANCE turned a sample variance into
a population one and exp.VariancePop -> VARIANCE_POP generated invalid MySQL:

    var_samp(x) -> 2.3333,  variance(x) = var_pop(x) -> 1.5556

Doris documents the same population semantics for VARIANCE and inherits this
fix; SingleStore already carried identical overrides, which are now redundant
with the base MySQL behavior.

StarRocks supports TO_DAYS natively, so MySQL's expansion into DATE_DIFF + 1
is both unnecessary and off by one there:

    to_days('2026-03-10')                                        -> 740050
    DATE_DIFF('DAY', DATE('2026-03-10'), DATE('0000-01-01')) + 1  -> 740051

As a consequence StarRocks TO_DAYS no longer lowers into day arithmetic for
targets without the function; round-trip fidelity on StarRocks itself is the
priority here, matching how the base dialect treats TO_DAYS.

STDDEV is left alone: MySQL's STDDEV/STD/STDDEV_POP are population functions
too, but the round trip is already stable and changing exp.Stddev would affect
every dialect.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@geooo109

Copy link
Copy Markdown
Collaborator

@UCHPUCHMAK The PR makes a lot of changes all at once. Moreover, the description is AI generated and not very helpful for reviewing. I suggest you read the following: https://github.com/tobymao/sqlglot/blob/main/CONTRIBUTING.md . After that you can break the PR into smaller ones.

@geooo109 geooo109 closed this Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants