perf(operator): ⚡ pool the term store into a chunked, two-tier array - #350
Draft
diagonal-hamiltonian wants to merge 9 commits into
Draft
diagonal-hamiltonian wants to merge 9 commits into
diagonal-hamiltonian wants to merge 9 commits into
Conversation
A store that grows to a billion terms pays a geometric overshoot it never returns before quiescence, and a reallocation that holds the old and the new buffer at once. Chunked storage pays neither: growth appends a chunk, nothing is copied, and the only slack is the tail of the last chunk. Invariants: a pool hands out one size class, so a freed chunk fits any later request; an arena is unmapped as soon as its last chunk returns, so a drained store gives its memory back to the OS rather than to the allocator; arenas are 2 MiB-aligned so THP can back them. ChunkPool is non-movable because every array holds a bare pointer to it. ChunkedRowArray keeps the power of two on the row index rather than the element index, because the row stride is a runtime width -- which is also what keeps a row from straddling a chunk, so a span over one row stays contiguous. The chunks come from a per-owner pool rather than one mapping each: at 91 dense index columns times 128 partitions, a mapping per chunk would reach vm.max_map_count long before the memory ran out. Assisted-by: ClaudeCode:claude-opus-5
The row store grew as one vector: it held up to half the operator as spare capacity, and both the old and the new buffer at the instant of a copy. On chunks the only slack is the tail of the last chunk. Invariants: row indices are handed out consecutively by grow_rows_geometric and are never reassigned, so nothing indexed by row has to be rebuilt; a row never straddles a chunk, so the span row_positions() hands out stays contiguous; the chunk length is a power of two and a multiple of 64, so the 64 rows an inverted-index word names sit in one chunk. The length is re-derived on every growth rather than set once, because nobody can tell the store its final height -- the propagator reserves the *initial* operator's size. A quarter of the height rounded down bounds the tail under a quarter of the store; a fixed 2^18-row chunk was 4.25 MiB of tail on a 0.8 MiB store. rechunk_ moves whole chunks and keeps every row index. The hash index, its accessors and the two ceiling checks are untouched. Assisted-by: ClaudeCode:claude-opus-5
…stride Every row pays the inline width, so a width set to the cutoff's structural bound is a byte or two per row spent on the few rows that need it. The store now lays rows at a predicted width and puts rows between that and the bound in a fixed-stride second tier, whose slot is written over the narrow row's inline positions. Invariants: both markers sit above kMaxInlinePositions, so resolving a row is one unsigned compare against the inline width; the tier exists only while it is narrower than the bound and the slot fits inline (kMinInlineForWideTier); tier slots are appended and never reclaimed. restride_to_bound() re-lays the rows at the bound and is index-preserving -- row indices, the hash index (which keys on a row's value, not its layout), the side-map and every TermIndex the engine holds survive it -- so it may run between gates but never inside one, where a caller holds spans into the rows. should_restride() can only be true while a tier exists, so a store restrides at most once; raise_bound() drains the tier first so a cutoff widened after construction gets a laid-out tier instead of side-map entries. A wrong width guess costs one restride and never a wrong answer. Assisted-by: ClaudeCode:claude-opus-5
A dense column is a full-height bit-vector: 125 MB at a billion terms, held as one vector, so its doubling reserve and the doubled copy it holds while reallocating cost more than the columns' own slack. Chunked, a column never reallocates. Invariant: a fold block starts at a multiple of kColumnBlockWords and a chunk holds a whole number of blocks, so a block never straddles a chunk and dense_column_block() can hand the fold a bare pointer. There is no pointer to a whole column any more, so the two readers that took one -- the scan's pivot column and make_fold_cache -- now ask per block. XOR is associative, so a blocked fold is bit-identical to the full-width one it replaces. Each column's chunk length comes from the height it is built at, so two columns of one index may differ and nothing in the fold may assume they agree; one pool per size class serves them. The pools are declared before the columns so they outlive the chunks they own, which is also why move assignment cannot be defaulted. Assisted-by: ClaudeCode:claude-opus-5
…ort the slack The coefficient arrays parallel the row store but grew on std::vector's doubling, settling at up to 8 B/term more capacity than the rows they parallel and never returning it before the quiescence shrink. reserve_coeffs_geometric grows them at 1.5x instead. Capacity is unobservable, so this is exactly a memory choice. The slack only exists while a call is running -- every call ends in a shrink to fit -- so it is sampled at the growth sites into a per-call high-water mark rather than measured at rest, where it would always read 0. Summed across partitions, not maxed: they grow together within a call, so the sum is the figure a per-process footprint wants and it errs high. Six diagnostic keys join the ledger, all outside total_bytes(): d_op_coeffs_slack_bytes (a subset of op_coeffs_bytes), d_row_wide_rows / d_row_inline_width / d_row_restrides (whether the width guessed from the model's cutoff held; the width is shared across partitions, so it is maxed rather than summed) and d_pool_mapped_bytes / d_pool_free_chunk_bytes, which are a subset of nothing -- a pool maps whole arenas and keeps one while a single chunk in it is live, so they name the bytes the kernel charges where no resting field prices them. Assisted-by: ClaudeCode:claude-opus-5
…tween gates The store is built at a predicted inline width rather than at the cutoff's structural bound, with the bound as the second tier's stride. Majorana rows are even-parity when the initial operator is and the generators preserve it, so the bound is one slot loose at every odd cutoff; measured Pauli rows run at 0.79-0.88 of it, and 0.79 takes the largest cut. The prediction is a hint, never a constraint: a width that turns out too narrow costs one restride. run_gate_loop_ takes it between gates and never inside one, because a gate holds spans into the rows it is reading and the restride moves every one of them -- row indices survive it, so nothing else is rebuilt. It also opens the coefficient-slack measurement window there, that being one propagate or build_graph call. follow_cutoff_bound_ lets the tier widen with a cutoff raised after construction, so the rows the new cutoff admits are laid out rather than spilled into the side-map an entry at a time. Assisted-by: ClaudeCode:claude-opus-5
Where the operator's bytes are: what the chunk pools map against what the chunk-counting fields price, what a nonzero restride count means, and why the coefficient slack is a high-water mark rather than a reading. Every gate frees a few large transient buffers; glibc answers the first such free by raising its dynamic mmap threshold and its trim threshold, after which the widest gate's transients stay on free lists the allocator cannot return. Pinning the threshold disables the escalation. The section states the shape rule -- large terms, several partitions -- rather than recommending a default, because at ~10 M terms on one core the same setting costs ~20 % wall time. Assisted-by: ClaudeCode:claude-opus-5
… before term_of_width repeats a term every 64 rows, so the rows are not all distinct and the index answers the first row holding each value. That is beside the point the case makes: the claim is that a restride leaves the index answering exactly as it did, whatever it answered, so the expectation is recorded before the layout moves rather than assumed to be the row index. Assisted-by: ClaudeCode:claude-opus-5
Each of the three class blocks to eight lines, the target for this file. Assisted-by: ClaudeCode:claude-opus-5
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
documentation
|
Docs preview: https://pr-350.monoprop-docs.pages.dev |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #350 +/- ##
=======================================
Coverage 97.70% 97.70%
=======================================
Files 14 14
Lines 742 742
Branches 98 98
=======================================
Hits 725 725
Misses 12 12
Partials 5 5
Flags with carried forward coverage won't be shown. Click here to find out more. |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Summary
Main's term store is one contiguous, geometrically-grown array per operator: every restride copies
the whole store, and every row is sized for the worst case a fixed width has to cover. This PR
replaces that array with a pooled, chunked one (
ChunkedArray.h) that rows and the inverted-indexcolumns both sit on, so growth allocates a new chunk instead of copying the store, and a two-tier
row layout (inline + wide) sizes each row from the model's own cutoff bound instead of one width
that fits every row.
The change is storage only. Main's ~290-line open-addressing hash index (
Slot,Table,find*,bulk_insert*,emplace,for_each, ...) is carried over unchanged behind the new storage; rowindices are preserved across every restride (asserted directly, not assumed); and the inverted-index
fold stays exact by associativity (it was already blocked on main). The memory ledger gains
diagnostic keys for the pool and the coefficient/row slack.
This is PR 1 of 7 in a stack on
origin/main. It is the base of the stack (no predecessor). Thenext PR (
perf/stack-2-router) adds a single GF(2)-linear rank router and does not touch storage.Changes
Engine
cpp/monoprop/detail/operator/ChunkedArray.h(new, 557 lines): a pooled chunked array within-place restride, shared by term rows and inverted-index columns.
cpp/monoprop/detail/operator/OperatorIndex.h: rows and the wide tier moved ontoChunkedArray;rechunk_/restride_to_boundre-lay rows in place and preserve every row index. Main's hashindex is untouched apart from the mechanical
&rows_[i*stride_]→rows_.at(i)swap.cpp/monoprop/detail/operator/InvertedIndex.h: dense columns moved onto pooled chunks; theblocked XOR fold (
combine_columns_block) is kept exact; the density histogram is dropped.cpp/monoprop/detail/operator/MPOperator.h:reserve_coeffs_geometricgrows coefficients at1.5×, at the row store's own policy; new ledger diagnostics for pool and slack bytes.
cpp/include/monoprop/MonomialPropagator.h/.inl:predicted_inline_width_andfollow_cutoff_bound_size each row's tier from the model (Majorana: bound rounded down to evenfor an even-parity initial operator; Pauli: 0.79× bound); the store restrides between gates.
cpp/monoprop/detail/evolution/CosineRecompute.h,.../layer_build/Scan.h: the blocked fold'spivot pointer moves inside the block, which chunking a column requires; no other behavior change.
src/monoprop/bindings/binder.h: expose the new ledger keys.Tests
cpp/tests/chunked_array_tests.cpp(new, 306 lines): chunk sizing, restride, pool accounting.cpp/tests/operator_index_tests.cpp,inverted_index_tests.cpp,mp_operator_tests.cpp:storage/restride/ledger cases, including
dense_columns_fold_identically_across_chunk_boundariesand
make_fold_cache_is_blocked_and_bit_identical_across_chunk_sizes(chunked vs single-piece,BOOST_CHECK_EQUAL_COLLECTIONS).tests/test_monoprop_smoke.py:test_operator_memory_breakdown_keys_and_totalspins the ledgerkey set and its invariants (e.g.
d_pool_free_chunk_bytes <= d_pool_mapped_bytes).cpp/tests/README.md: "Operator store" section updated for the new files.Docs
docs/content/docs/features/parallelism.mdx: the new ledger key paragraphs and themmap_thresholdrecipe.Measurements
Bit-identical to origin/main c5e88c8 (raw coefficient bits, positional mode, gate
record md5
32fe8289bf0d2acacac51e54d3fb5d98). A positional gate is the right one for this level:row indices are unchanged by any restride, which the storage half of
operator_index_tests.cppasserts directly (the value-keyed hash index answers exactly as it did before a restride).
Paired A/B, 3 interleaved reps against origin/main c5e88c8, ratios only:
Time is within this box's noise floor (±1%) on every rung. The peak effect is size-dependent, and
this PR states it rather than quoting the best number: L1 ≈ 0.88–0.89, M2a ≈ 0.958 (measured
downstream in PR 2's and PR 3's ladders, unchanged there since neither touches storage), and at
250 M terms (L2b, 4 ranks × 4 partitions) the ranks-summed peak is 0.998 — the store's own
gain is real (
bytes/termthere is 0.906) but is mostly eaten by pool mapping and coefficient slackat that size (
d_pool_mapped_bytes6488 MiB vs. terms+inverted-index 5019 MiB;d_op_coeffs_slack_bytes946 MiB).bytes/term, the size-independent figure, stays a clear win atevery size measured: 0.878 / 0.827 / 0.931 / 0.906.
Notes for reviewers
indexing_byteskeeps main's meaning here (the store object plus the hash table); it is notredefined to the term-table size until PR 4.
key_of_row/key_of_positions,join_tagorRowBlockland in this PR — thoseare PR 4's, and
OperatorIndex.hdeliberately does not includedetail/mpi/Routing.hyet.row_blockprivate helper (to hoist the shift/mask onrows_.at(i)) is not added:it was conditional on M2a costing more than 3% of time, which it did not (0.991 above), and adding
it now would import PR 4's shape early.
TermLookup.h, theinsert_absent_termsKeyFndrop,GateScratch.h/d_gate_*counters, andMemProf/TimeProfare out of scope here — PR 4 and PR 5 respectively, or (diagnostics) notlanding in the stack at all.
early version of the restride test asserted
find(want[i]) == i, which is wrong once two rowshash to the same value (the synthetic generator repeats every 64 rows); it now compares what
find()answered before and after the restride, which is the actual claim.d_op_coeffs_slack_bytes,d_pool_mapped_bytes,d_pool_free_chunk_bytes,d_row_wide_rows,d_row_inline_width,d_row_restrides.d_invidx_dense_columnsalready existed on main and needed only the binder addition, not a newkey.
Checklist
docs/,CONTRIBUTING.md) if neededCHANGELOG/ release notes updated if applicable (n/a — the repository has noCHANGELOG)AI/LLM disclosure
Important
By opening this PR I confirm that I have read CONTRIBUTING.md and I agree to the terms of the Contributor License Agreement.
Warning
If you're contributing on behalf of your employer, contact cla@algorithmiq.fi to arrange a Corporate CLA.