Building continuous-batching LLM runtimes, Paged KV-cache memory allocators, and capacity load-testing harnesses from first principles.
Hardware-Calibrated Runtimes · Zero CUDA OOMs · Zero-Cost Abstractions
Empirical roofline modeling, zero-copy physical block gathering, and verified cryptographic provenance.
# Profile local server capacity (Ollama, vLLM, llama.cpp) with automated roofline tuning
pip install servebench && servebench quick
# Run continuous batching with paged KV-cache (pure PyTorch, zero C++/CUDA build steps)
pip install mini-infer
# Discrete-event serving simulator with Radix prefix caching & Perfetto trace export
pip install sched-sim && sched-sim run --scheduler cache-aware --trace sharegpt| Flagship | Core Architecture | Empirical Silicon Numbers | Release & Provenance |
|---|---|---|---|
| servebench | LLM serving capacity load-tester & auto-tuning harness (servebench tune). Multi-trial repeats with 95% CIs and synchronized NVML GPU telemetry. |
• Honest p99 ( • MBU roofline profiling • Slot / Queue / Bandwidth diagnostics |
  
|
| mini-infer | Lightweight continuous-batching LLM runtime in pure PyTorch for Qwen2.5. Iteration-level dynamic scheduler, Paged KV-cache virtual memory, radix prefix caching, and Leviathan speculative verifier. | • 27.0 tok/s at c=4 (+130.8%) • 2.04 Tok/Joule (13.2 W) • <3.8% KV-cache waste |
|
| sched-sim | Discrete-event LLM serving simulator with first-principles roofline latency models, SGLang-style Radix prefix caching, and Perfetto trace visualizer. | • 0.87s full sweep • 87% test coverage • 0-copy prefix hits |
|
| EdgeSLM | Air-gapped document sensitivity classifier and Tier-1 DLP secret scanner for edge CPU inference. | • 20+ HF downloads (v3) • 6.7× dynamic batching • 100% secret recall |
 
|
| Menmo | Hybrid retrieval RAG service (Pinecone MiniLM + BM25) with dynamic fallback and page-level source citations. | • Scoped multi-tenant search • Zero-hallucination routing • Django + Next.js decoupled |
Live App → Backend · Frontend |
I believe that engineering credibility is built by documenting trade-offs, architectural failures, and hardware bounds alongside performance gains:
-
Naive KV-Allocation Fails Under Dynamic Lengths:
Pre-allocating contiguous tensor buffers for worst-case context lengths wasted >60% of GPU memory due to internal and external fragmentation. Fixed by implementing fixed 16-token physical block pools with virtual-to-physical block tables in pure PyTorch, cutting memory waste to <3.8%. -
Unbatched Iteration Bottlenecks on DRAM Memory Bandwidth:
Stepping concurrent sequences through serial model calls forced the GPU to stream model weights from DRAM 4 separate times per step (capping decode at 11.7 tok/s on an RTX 3050). Switching to zero-copy physical block gathering streams weights exactly once per step across all sequences, jumping throughput to 27.0 tok/s (+130.8%) and delivering 2.04 Tok/Joule. -
Synthetic LoRA Overfitting on Edge Models:
Initial LoRA fine-tunes on synthetic compliance data showed high train accuracy but collapsed on real-world document formatting. Implemented contrastive evaluation pairs and McNemar significance testing ($p = 0.043$ ) to validate generalization before exporting to 392 MB ONNX INT4. -
No Cherry-Picked Latency Metrics:
Single-run p99 latency spikes are statistically deceptive without sufficient sample size.servebenchstrictly enforces$N \ge 100$ before calculating p99 percentiles, and reports 95% confidence intervals across multi-trial repeats.
- Roofline Benchmarking & Auto-Tuning LLM Serving: How We Pushed Pure PyTorch to 27 tok/s (
2026-10-23)
Measuring empirical Memory Bandwidth Utilization (MBU), saturation knees, and multi-stream batching to eliminate TTFT degradation on consumer laptop GPUs. - Why Naive KV-Caches Waste >60% VRAM (And How Paged Memory Fixes It) (
2026-10-22)
How replacing contiguous memory allocations with 16-token physical blocks and virtual table mapping brings production PagedAttention to pure PyTorch.
Runtime Systems : Paged KV-Cache Allocators · Continuous Batching · Radix Prefix Caching · Speculative Decoding · Discrete-Event Simulation
Silicon Profiling : Roofline MBU Modeling · NVML Telemetry · Saturation Knee Detection · TTFT / ITL / TPOT Analysis
Core Stack : PyTorch (SDPA, Tensor Gathers) · Python 3.10+ · CUDA / NVML · ONNX Runtime INT4 · Hugging Face
Serving & Infra : FastAPI · Asyncio / Aiohttp · SSE Streaming · vLLM · Ollama · Docker · GitHub Actions CI/CD
Security & Trust : PyPI Trusted Publishers (OIDC) · Sigstore Cryptographic Attestation · Zero-Secret Airgap DLP
- Portfolio: prajwal-pujari.vercel.app
- PyPI Packages: pypi.org/user/Prajwal2002PI
- LinkedIn: linkedin.com/in/prajwal-pujari-mn9964
- Email: prajwalpujari16@gmail.com
- WhatsApp: Direct Chat (WhatsApp ↗)