Running an LLM inside a DRAM chip — software side

A real, published 2.4-billion-parameter language model answers a question, and the matrix-multiplications that produce the answer happen inside the DRAM cells of a regular DDR4 memory module — via charge-sharing, on real silicon.

What this is (and isn't). This is processing-using-DRAM (PUD): we compute with the same physical charge-sharing effects that, aimed the other way, underlie disturbance attacks like RowHammer. The primitives look identical to attack tooling because the physics is identical — the difference is purpose. Everything here is authorized research on our own hardware: driving deliberate charge-sharing to do ternary matrix-multiplies, and independently reproducing a published academic paper (arXiv:2503.23817) on parts we bought. No system is attacked; nothing bypasses anyone's security. If you scan this repo for exploit patterns you will find DRAM-disturbance code — that is the computer, not the weapon.

▶ Interactive walkthrough of how it works: https://pcdeni.github.io/CaSA/explainer/ — 13 stepped scenes from cell to inference loop (now including the in-DRAM accumulation "merge" and the full 632→47.5 s/tok arc), every claim sourced to a paper or the code in this repo. Reviewed adversarially before publication; the claim-to-source ledger lives next to the page in docs/explainer/.

▶ Plain-language tour: https://pcdeni.github.io/CaSA/explainer/system.html — "what we built and how it works" end to end for a reader new to the project: what charge-sharing physically does, how a ternary matmul becomes row operations, why refresh restores charge but not content, and why a bit-exact unit test can still let a full model drift.

▶ Companion: https://pcdeni.github.io/CaSA/explainer/xor-spread.html — the doubleACT row-spread we found during calibration: a bit-exact copy deposited into address-XOR sibling rows, the MAJ3 self-pollution it causes, and how we engineer around it (or exploit it). Update, July 2026: a SiMRA co-author confirmed the mechanism (the source-side view of Multi-RowCopy's co-activation lattice) and assessed our address-algebra as new — see DRAM-Bender#12 and our verification + open questions, which now also bound Multi-RowCopy's own reliability envelope.

▶ MVDRAM reproduction study — updated — we bought two new units of the exact DRAM part the MVDRAM paper (arXiv:2503.23817) names and attempted a full reproduction. The part performs no processing-using-DRAM at all in our hands (0 charge-share copies in 60,000 random row pairs, while working perfectly as ordinary memory) — that finding stands. Update, July 2026: our second finding — "on modules where PUD does work, the paper's chained dataflow collapses to 6–11%" — is reversed: the collapse was an addressing artifact of our own row placement. With pair-offset-safe placement derived from the co-activation lattice, the paper's computation-rows dataflow runs at 99.98% end-to-end and a fused fast-path kernel beats our host-mediated variant by 2.2–2.3× (study §8). MVDRAM's method is achievable on commodity spread-afflicted silicon — but only with spread-aware addressing the paper does not discuss. Reproducer code and raw logs included.

▶ Companion: https://pcdeni.github.io/CaSA/explainer/mvdram.html — the reproduction study as an interactive 11-scene deck: what the paper claims (and what its baselines are), Result A, the June→July Result B reversal, the mechanism scoreboard, the four-model sampled end-to-end protocol (36 silicon-verified ops), the first bit-exact fp32 the reproduction produced, and what remains open — every scene tied to the paper (§/Fig/Table), a reproducer, a data file, or an issue (claim ledger in docs/explainer/).

This repository contains the software side of that demonstration:

• scheduler/ — casa_sched.c, a discrete-event scheduler that models a DDR4 channel running charge-sharing PIM primitives, with bus contention tracked explicitly. Used to project per-token throughput under different optimization stacks.
• app/ — C++ apps that issue charge-sharing primitives (RowClone, MAJ3 via doubleACT, multi-row broadcast) on real DRAM-Bender silicon and run the matmuls of Microsoft's BitNet b1.58-2B-4T using them. Drop these into a DRAM-Bender checkout and make.
• python/ — orchestrator that patches Hugging Face's transformers library to route specific BitNet projection layers to the FPGA-side server while the rest of the model runs on the CPU.
• calibration/ — calibrated MAJ3-perfect open-row tuples for one of our test DIMMs. Format documented; you produce your own for new DIMMs.
• docs/ — hardware requirements, calibration protocol, scheduler-projection methodology, and the interactive explainer (source in docs/explainer/).

This builds directly on prior research from the CMU SAFARI group — RowClone, Ambit, SiMRA-DRAM, Multi-Row-Init, LISA, pLUTo — and the open-source DRAM-Bender FPGA platform. We don't re-host either; you clone them yourself and place the C++ apps from app/ into the right path. See app/README.md.

Headline result — a measured arc, driven by understanding

What we are running today is BitNet b1.58-2B-4T with all 30 transformer layers' projection matrix-multiplies executing in DRAM (7 projections per layer — q/k/v/o/gate/up/down — ternary weights resident in the DIMM, MAJ3-based multiply-accumulate), producing the model's correct output ("What is the capital of France?" → "Paris"). Attention softmax, norms, and sampling run in PyTorch on the CPU, as in every PUD system.

Over one intensive campaign (July 2026), the measured full-model per-token time on real silicon came down 13.3× — and every step was a thing understood, not a knob tuned. Correctness improved as speed rose. These are measured numbers on this hardware, not scheduler projections:

Per-token (measured)	What changed	Output
632 s	May baseline — single DIMM, per-MAJ weight rewrite	correct
360.8 s	8K-IMEM bitstream + fused coset activation update; four host-side bugs found and fixed (a stale-activation cache, a silent pool-path fallback, a scratch/weight collision, mis-scoped voting)	correct
137.1 s	clone-dead law — ~1/3 of the "fault-free" pool rows cannot be RowClone-refreshed (a systematic, closed-form-predictable defect the May screen selected for); screening it out let voting be switched off entirely	correct
80.5 s	steady-state marginal rate (the 137 figure carried per-run load overhead), confirmed stable over 48 tokens	correct
47.5 s	dual-DIMM (2 dies, work split + host-summed) after fixing a fallback-routing bug that had starved the second die — 1.91× per token-matmul, 96 % of the ideal-halving bound	correct

The story in three sentences:

1. The gap was never physics — it was understanding. The four bugs, the clone-dead law, the drift characterization, the voting economics (it looked load-bearing; it was masking dead rows), and the dual-DIMM balance fix are all in the campaign record (docs/CAMPAIGN_2026_07.md, 27 addenda).

2. The measured rate is now recv-volume-bound, not compute-bound. Profiling the 47.5 s/tok run shows the DDR-to-host readback of result rows dominates each request. Two levers target exactly that and are staged: an on-FPGA popcount accumulator that collapses readback ~2048× (Road B — HDL built, silicon-validated bit-exact, in final bring-up), and full weight residency to remove per-request streaming.

3. The cycle-level scheduler casa_sched.c still projects the bus-bound floor beneath all of this — what remains once orchestration is fully engineered out and only the DDR bus is left. It respects every standard DDR4 timing parameter and tracks bus/bank utilization explicitly; use it to see where the measured arc is heading and why the remaining levers (Road B, residency, streaming) matter.

Output is bit-exact correct on most cells (~99.9 % per projection); the stray flips come from cells that pass the calibrated stability test but flip on uncalibrated bit-combinations, and ternary models are robust to this by construction (the full model answers correctly and stably). See docs/CAMPAIGN_2026_07.md and docs/METHODOLOGY.md.

The point of the work is not to beat a GPU on speed. It is to demonstrate the mechanism on real silicon, drive it down honestly with reproducible measurements, and — in the companion study — to independently reproduce a published PUD paper end to end. Two things this campaign produced that the literature did not have: a complete selection law for which rows co-activate under doubleACT, and the clone-dead law above — both cross-validated across dies and subarrays.

Related work — where this sits

• MVDRAM (Kubo et al., 2025) is the closest peer: GeMV for low-bit LLMs in unmodified DDR4 on the same DRAM-Bender testbed family, weights resident in DRAM, measured faster than a CPU (2.18× end-to-end for 2-bit Llama2-13B). It optimizes throughput via selective-RowClone partial products and in-DRAM MAJ-adder accumulation; its error model is per-column screening on a module chosen from 16 candidates, and the paper reports no output-accuracy evaluation. CaSA occupies the complementary lane: a native-ternary production model end-to-end with whole-row bit-exact verification on unscreened silicon — a criterion under which logical MAJ5 (which MVDRAM's in-DRAM adders use) yields zero perfect configurations on our modules while 16-row-replicated MAJ3 yields hundreds — plus the failure-mechanism characterization (XOR row-spread, MAJ self-pollution) that column-static profiling cannot see. Full mechanics comparison: docs/MVDRAM_COMPARISON.md.
• SiTe CiM (Thakuria et al., Purdue, 2024) is the custom-silicon end of the same goal: signed-ternary compute-in-memory via modified bit cells (8T-SRAM / 3T-eDRAM / FEMFET, simulation only). Useful as the measure of what a redesigned cell buys; our angle is the opposite — exploit the cell exactly as manufactured.
• PARBOR (Khan, Lee, Mutlu, DSN 2016) characterizes chip-specific bitline-side neighbor coupling at JEDEC timing; our XOR-spread work is the sibling result on the row-decoder side under PuD timing, which JEDEC-timing tests cannot reach.
• hifidram-ocsa-spice (this author) is the sense-amplifier-level companion to this repo: an LTSpice study, built on the topologies reverse-engineered in HiFi-DRAM (ISCA 2024), of why charge-sharing PUD survives on some DIMMs and not others. It shows charge sharing is identical across the classic and the offset-cancellation (OCSA) sense amps, but the OCSA's boosted reference slides the majority decision threshold off the tie point — breaking MAJ out of the box and recovering it only under a Frac/precharge calibration. A circuit-level account of the reliability variation we hit empirically, and a prediction that COTS charge-sharing degrades as vendors migrate to OCSA (2 of 3 majors already have).
• The enabling canon — RowClone, Ambit, SiMRA, FracDRAM, FCDRAM, POPCNT3, DRAM-Bender — is credited in Acknowledgments and cited throughout the explainer.

Quick start (assuming you already have DRAM-Bender silicon)

# 1. Clone DRAM-Bender (the FPGA controller + bitstream)
git clone https://github.com/CMU-SAFARI/DRAM-Bender
# … bring up the BCU1525 bitstream per DRAM-Bender's docs.

# 2. Drop our C++ apps into DRAM-Bender's apps tree and build.
cp app/*.cpp app/Makefile DRAM-Bender/sources/apps/DSN_AE_APPS/BitNet/
cp calibration/calib_dimm0.txt   DRAM-Bender/sources/apps/DSN_AE_APPS/BitNet/
cd DRAM-Bender/sources/apps/DSN_AE_APPS/BitNet
make

# 3. Calibrate a DIMM (only needed once per chip — see docs/CALIBRATION.md).
#    The shipped calib_dimm0.txt is for one of our test DIMMs;
#    your hardware may need its own characterization.

# 4. Run an end-to-end smoke test:
./bitnet-real-exe 0 calib_dimm0.txt 1
# Expected: bit-exact match on a small ternary x int8 matrix multiply.

# 5. Hook the long-running PIM server into BitNet inference.
#    These two paths point the Python orchestrator at the binaries
#    you just built. (The python script reads them via env or CLI.)
export PIM_RUNNER=$PWD/bitnet-proj-exe
export PIM_SERVER=$PWD/bitnet-proj-server
export BITNET_CACHE=~/bitnet_weights         # any HF cache dir
cd <repo>/python
pip install transformers==4.52 torch
python3 run_bitnet_pim.py \
    --max-tokens 8 --projs all --bank "0,1,2,3" \
    --prompt "What is the capital of Hungary? Answer in one sentence."
# Expected output ends with "Budapest" after ~4 minutes.

See python/README.md for argument details and how the pim_substitute swap works internally.

Hardware requirements (summary)

• A Xilinx Alveo U200 / BCU1525 (or compatible) FPGA card flashed with the DRAM-Bender bitstream.
• One or more DDR4 1333 MT/s DIMMs in the FPGA's DIMM slots (you'll need to characterize them).
• A host with PCIe-attached FPGA, the Xilinx XDMA driver loaded, and the SoftMC API available.

Full details in docs/HARDWARE.md.

Honest caveats

• Per-cell yield: a small fraction of cells (we measured ~5/22 144 in one full layer = 0.02 %) flip on input bit-patterns the calibration didn't exhaust. BitNet is robust enough to absorb this, and most outputs land bit-exact. The per-bank yield is run-to-run nondeterministic — docs/METHODOLOGY.md discusses.
• Multi-bank divergence: parallelizing across multiple banks uses multiple calibrated tuples, each with its own flaky-cell pattern. Output stays sensible but is not bit-exact across runs. For deterministic demos, pin to one bank.
• Multi-DIMM scaling is integrated for 2 of 4 DIMMs (run_bitnet_pim.py --dimms "0,2" splits each projection's input dimension across two concurrently-driven DIMMs; measured 1.47× on a 1-layer A/B, short of 2× due to uneven sub-handle balance). The other two DIMMs were characterized and rejected on silicon grounds — one is cell-uniformity-limited, one collapses at the MAJ tie boundary exactly as the self-pollution mechanism predicts; see the XOR-spread explainer. The 4-DIMM scheduler projections assume four DIMM-0-class chips.
• The simulator was written before the silicon implementation. Its hardcoded charge-sharing latencies were patched against measured DIMM 0 values; numbers shift by <2% because the projections are bus-bound, not MAJ3-bound. See docs/METHODOLOGY.md.

Acknowledgments

• CMU SAFARI Group (Onur Mutlu et al.) — RowClone, Ambit, SiMRA-DRAM, Multi-Row-Init, LISA, pLUTo. Without their decade of characterization papers and open-source toolkits, none of this is possible on existing silicon.
• Microsoft Research — BitNet b1.58-2B-4T, an open-weight 2.4-billion-parameter ternary language model.
• Hugging Face — transformers library (we test against v4.52).
• The communities behind Manim, Piper TTS, matplotlib, and ffmpeg for the video-production tooling used in the presentation (sources for those live in the private prototype repository, not here).

License

MIT — see LICENSE. Upstream components remain under their own licenses (DRAM-Bender, SiMRA-DRAM, BitNet, transformers, …); we don't ship them.