Skip to content

Latest commit

 

History

History
58 lines (51 loc) · 3.83 KB

RESULTS-LOG.md

File metadata and controls

58 lines (51 loc) · 3.83 KB

DiffusionGemma single-stream optimization — results (Thor SM110, 2026-06-19)

Server: gemma-aarch64-cu130, V2 runner, TRITON_ATTN, modelopt_fp4, FP8 KV, prefix-caching ON, max-num-seqs 1, gpu_util 0.80. (Entrypoint fix applied: pass /model.)

Baseline single-stream (chat endpoint)

turn thinking max_tok tok/s
short false 64 50.7
medium false 256 105.7
long false 768 90.3
reasoning true 512 115.8
Consistent with dg-repro. Note short=50.7 with thinking=false vs 35.8 with thinking=on.

Prefix caching (phase4) — WORKS

Wall-time TTFT delta inconclusive (cold 1969 / warm 2264 / 1566 ms) because the test's shared system prefix (~70 words) is tiny vs the 128-tok generation. BUT server logs prove APC active on the diffusion path: prefix-cache hit rate climbed 17.6% -> 22.9% -> 26.8% -> 33.9%. => Confirmed (vLLM blog claim holds on Thor). For a real agentic harness with a long shared system prompt + tool defs, the prefill saving would be material; this micro-benchmark under-measures it.

Thinking ablation (phase5) — real lever on short turns

turn thinking=false thinking=true delta
"list files /etc/nginx" (64) 1097ms (59 tok) 2561ms (64 tok) -1464ms
"144/12" (32) 602ms (14 tok) 958ms (32 tok) -356ms
flatten list (256) 2804ms 2083ms mixed (capped)
TCP handshake (256) 2759ms 2938ms mixed (capped)
=> thinking=false saves ~1.4s on short tool/factual turns. Route those with enable_thinking=false.
reasoning_content not separated by gemma4 parser (reasoning_words=0) but thinking still costs tokens.

Phase 3 (expert requant) — INFEASIBLE as written (verified)

The FP4-experts bandwidth lever is real (NVFP4 checkpoint keeps mlp/router/self_attn in bf16), but llm-compressor cannot reach it: it wraps AutoModelForCausalLM, and AutoModelForCausalLM.from_pretrained('/model') fails with: ValueError: model type diffusion_gemma ... Transformers does not recognize this architecture. The DiffusionGemma model code lives in the vLLM image (not as a transformers trust_remote_code class); config has architectures=['DiffusionGemmaForBlockDiffusion'], auto_map=NONE, modelopt-packed weights. So requantizing experts requires a modelopt-based flow against the vLLM model definition (a research effort), not this llm-compressor script. Did NOT run the install/quant (would fail at load). Bandwidth lever remains open but out of reach with off-the-shelf tooling.

Requant investigation (2026-06-19)

  • Experts ALREADY NVFP4 (cutlass W4A4). Only 2.99B bf16: attention 1.25B + norms/router/embed.
  • Attention (dense) is the single-stream bottleneck, not experts (sparse 4/128). Directive had it backwards.
  • W4A16_NVFP4 config flip on existing checkpoint = NO-OP (~baseline tok/s): all FP4 is in FusedMoE which selects backend independently (stayed VLLM_CUTLASS); no quantized Linear layers exist.
  • No tractable attention-requant path (W4A4 needs framework calibration; W4A16 needs modelopt packing that won't install + Marlin) + diffusion quality risk. Full analysis: REQUANT-ANALYSIS.md.

Quality + real-workload eval (quality-eval.py, 2026-06-19)

  • Output quality GOOD with thinking ON: code (fib/flatten/is_prime) executes correctly; factual (391/Tokyo/strawberry=10/WWII=1945/$0.05) correct; tool-call get_weather(Paris) valid.
  • thinking=false LANDMINE: emits 1 token + stops (empty) on terse prompts -> phase5 lever RETRACTED.
  • Thinking token-hungry + not separated (gemma4 parser leaves reasoning_content empty, inline in content); budget generous max_tokens; runaway thinking on some prompts ("is 2027 prime" never concludes in 1024 tok).
  • LONG-PREFIX prefix cache = the big agentic win: 6182-tok prefix, cold 5447ms vs warm 947-1135ms = 79-83% wall reduction (~5x TTFT). Far bigger than any decode-rate tweak.