Perf/opsd continuous batching benchmark clean

[nathon-lee]

Summary

Add a reproducible OPSD continuous‑batching benchmark for comparing:

• sequential eager generation;
• capacity‑bounded static batching;
• DeepSpeed Core continuous batching.

The benchmark uses deterministic synthetic prompts, fixed response budgets,
greedy decoding, synchronized CUDA timing, peak‑memory reporting, and JSON
results suitable for later comparison.
This PR only changes DeepSpeedExamples; it does not modify DeepSpeed Core.

Dependency

This PR depends on DeepSpeed PR #8368.
DeepSpeed PR #8368 provides the Core continuous‑batching rollout API used by
this benchmark, including request scheduling, request retirement, pending
request refill, KV‑cache row compaction, and legacy KV‑cache compatibility.

The Core implementation from #8368 must be available through PYTHONPATH
before running the benchmark. This Examples PR intentionally contains only the
benchmark, tests, and documentation, and should be merged after the API from
#8368 is available in DeepSpeed master.

Validation

Integration validation used the DeepSpeed Core implementation from PR #8368.

Tested on one NVIDIA RTX A4500 with PyTorch 2.9.1+cu128,
CUDA 12.8, Transformers 4.40.2, and DeepSpeed
0.19.6+a36f78e7.

Workload:

• Model: facebook/opt‑6.7b
• Dtype: FP16
• Prompt length: 512
• Response lengths: 32, 64, 96, 128
• Maximum batch size: 2
• Warmup iterations: 1
• Measured iterations: 3
• Greedy decoding with seed 1234

Mode	Mean latency (ms)	Useful tokens/s	Computed tokens	Peak memory (MiB)	Token agreement
Sequential eager	11009.87	29.06	320	13348.92	320/320
Static batch	7408.53	43.19	384	13989.12	320/320
Continuous batch	7699.00	41.56	320	14984.52	320/320

Continuous batching avoided 64 decode tokens (16.67%) compared with static
batching. It improved mean latency by 30.07% and useful‑token throughput by
43.00% compared with sequential eager generation.

Compared with static batching, the current legacy‑KV‑cache prototype was
3.92% slower and used approximately 995 MiB more peak allocated memory.
The benchmark therefore validates request scheduling, retirement, refill, and
output correctness, while exposing the remaining refill/cache‑management
overhead. It does not claim an end‑to‑end speedup over static batching.

Additional checks:

9 passed
python -m py_compile benchmarks/opsd/benchmark_continuous_batching.py
git diff --check

Scope and limitations

• Single GPU/process.
• Greedy decoding only.
• Equal‑width prompts.
• The benchmark reports cross‑mode token agreement instead of requiring exact equality by default, because low‑precision kernels can be sensitive to batch and cache shapes.
• --require‑exact‑token‑match is available for strict correctness checks.
• The benchmark is intended for evaluation and experimentation, not production serving.