Repository navigation
Stop boxing the row and iteration state in load!'s insert loop - #392
Merged
Merged
Conversation
`load!` ran its insert loop inside the `DBInterface.transaction` closure and reassigned `row` and `st` there. Julia stores captured variables that a closure reassigns in a `Core.Box`, so every row was boxed and every `iterate`, `eachcolumn`, and bind call in the loop dispatched at run time. Loading 1M rows of one integer column took about 370 ms and allocated 150 MB. The loop now lives in `insertrows!`, which the closure calls with the first row and state as arguments, so they are ordinary typed locals. Behavior is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #392 +/- ##
=======================================
Coverage 97.30% 97.31%
=======================================
Files 5 5
Lines 854 857 +3
=======================================
+ Hits 831 834 +3
Misses 23 23 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
SQLite.load!spent as much time on its own loop as SQLite spent inserting, for narrow tables. The insert loop ran inside the transaction closure and reassignedrowandstthere, so Julia boxed both. Every row then allocated, and everyiterate,eachcolumn, and bind call dispatched at run time.What changed
insertrows!. The closure passes it the first row and iteration state, so they are ordinary typed locals.Scope
load!only. Its behavior, errors, and transaction handling don't change.executemanygoes through DBInterface and doesn't change.Blast Radius
Every
load!call. The loop body is the same code. The existingload!tests cover the known-schema, unknown-schema, empty, append, and conflict paths.Verification
load!of 1M rows of one integer column into an in-memory database went from 379 ms to 199 ms, and its allocations from 153 MB to 15 MB. A mixed 10-column table went from 658 ms to 547 ms (2237 to 1851 million cycles). Each number is the median of 7 fresh-process runs per side, interleaved with master, on a shared 14-core Apple silicon machine (load average 10 to 50).Pkg.test()passes on Julia 1.12 and 1.10. Lowered code of theload!keyword body no longer containsCore.Box.🤖 Generated with Claude Code