Skip to content

Read strict query rows without dispatching on each column type - #391

Merged
quinnj-bot[bot] merged 2 commits into
masterfrom
claude/strict-row-reads
Nov 19, 2026
Merged

quinnj-bot[bot] merged 2 commits into
masterfrom
claude/strict-row-reads

Conversation

@quinnj-bot

@quinnj-bot quinnj-bot Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Why

People who loop over a strict query (DBInterface.execute(db, sql; strict=true)) and read row.x or Tables.getcolumn(row, i) paid for a method-cache search on every value. Each read dispatched on the column's runtime Union type, which Julia's fast dispatch cache can't hold. Strict row loops ran 3-5x slower than non-strict ones.

What changed

  • For strict rows, Tables.getcolumn(row, i) calls getvalue statically for Int64, Float64, String, Vector{UInt8}, and Any columns. Those are the types that declared column types and raw BLOBs give. Other column types, such as deserialized Julia values, still dispatch at run time.
  • One testset reads every column type from strict rows by index and by name, plus an exact conversion and the errors for values of another type. It passes on master too.

Scope

  • Row reads by index and by name. columntable, DataFrame, and other schema-based sinks already read statically, so they don't change.
  • Non-strict reads don't change.

Blast Radius

Only strict row reads. The static calls reach the same getvalue methods the dynamic call found, so values and errors don't change.

Verification

  • A strict row loop over 1M rows of one INTEGER column went from 850 ms to 197 ms (2923 to 668 million cycles). The non-strict loop over the same rows takes 160 ms. Over a mixed 10-column table it went from 6193 ms to 1596 ms. Each number is the median of 7 fresh-process runs per side, interleaved with master, on a shared 14-core Apple silicon machine (load average 10 to 50).
  • How big the win is depends on what the process has already read. An independent review measured 766 ms to 169 ms on the INTEGER table after reading a mixed table, and 3.3x on a 6-column table. In a fresh process that reads only the INTEGER table it measured 207 ms to 169 ms (1.2x). Once a strict read has seen several column types, every later read pays the slow dispatch, and that is the case this fixes.
  • None of the other 29 read, write, and latency cases is slower at rank-sum p < 0.01.
  • Pkg.test() passes on Julia 1.12 and 1.10.
  • Profile on master: the strict loop spent its time in ijl_apply_generic and jl_typemap_assoc_exact under getcolumn(r::Row, i).

🤖 Generated with Claude Code

Reading a strict query row by index or by name (`row.x`,
`Tables.getcolumn(row, i)`) called `getvalue` with the column's runtime
`Union` type. Julia's fast dispatch cache can't hold such calls, so every
value took about 0.5 µs to find its method. Iterating 1M rows of one
INTEGER column with `strict=true` took about 860 ms, against 160 ms
without `strict`.

`getcolumn` now compares the column type with the types that declared
column types and raw BLOBs give (Int64, Float64, String, Vector{UInt8},
and Any) and calls `getvalue` statically for those. Other types, such as
deserialized Julia values, still dispatch at run time. Column tables and
other schema-based sinks already called `getvalue` statically.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.33%. Comparing base (511b83a) to head (27c4ffc).

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #391      +/-   ##
==========================================
+ Coverage   97.30%   97.33%   +0.02%     
==========================================
  Files           5        5              
  Lines         854      862       +8     
==========================================
+ Hits          831      839       +8     
  Misses         23       23              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

A Date column takes the runtime-dispatched read in getcolumn, the path
for column types other than the ones it reads statically.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@quinnj-bot
quinnj-bot Bot merged commit a86b16d into master Nov 19, 2026
9 checks passed
@quinnj-bot
quinnj-bot Bot deleted the claude/strict-row-reads branch November 19, 2026 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant