CacheBridge provides separate interfaces for model routing, trajectory sampling,
offline fitting and online KV conversion. The offline package prepares fitting
data and mapper coefficients. The online package handles conversion, cache
lifetimes and model handoffs.
Implement the environment, router and model backend protocols defined in
cachebridge.agents, then call sample_trajectory to collect the complete event
sequence. At each model handoff, record the routing decision, actual cache mode,
exact token prefix and model identity. Give each generation request a distinct
request ID and each episode an explicit cache namespace.
cachebridge.offline.trajectory.handoff_samples converts native TRAIN
trajectories into fitting inputs. Save KV snapshots for the source history and
the target history and query. Keep tokens, tokenizer identity, model revisions
and the boundary between history (H) and query (Q) consistent.
The agentic workload adaptor provides a sampling entry point for τ-bench:
cachebridge serve --config /path/to/runtime.json -- \
python benchmarks/tau_bench/run.py sample \
--tasks recipes/tau_train.example.json --output /path/to/native-bankThe sampling directory stores all attempts, usable handoffs and references to token and KV files. Attempt records retain generation failures, episodes without a handoff and stops caused by budget limits. Training inputs are selected independently of task reward.
cachebridge serve --config /path/to/runtime.json -- \
python benchmarks/tau_bench/extend.py \
--inputs /path/to/native-bank/inputs.jsonl \
--output /path/to/extended-bank \
--max-query-tokens 4096 --max-target-calls 24Extension preserves the original task order and completed records, and extends only native continuations that reached their budget. Check the original tokens, actions, observations and KV prefixes before and after extension, and retain all original assistant prediction positions. Tool observations remain context; align the assistant mask with the input positions that predict the next assistant token.
cachebridge-fit prepare \
--inputs /path/to/native-bank/inputs.jsonl \
--target-model /path/to/Qwen3-32B \
--initializer /path/to/initializer/weights \
--freeze-keys --output /path/to/fit
CUDA_VISIBLE_DEVICES=0,1,2,3 cachebridge-fit run \
--directory /path/to/fit --mapper-id my-agent-mapper
cachebridge-verify /path/to/fit/candidate.json--freeze-keys preserves the initializer's K mapping. Omit it to refit K using
the input trajectories. V calibration proceeds in target-layer order: each
layer uses the current candidate state to construct attention-output targets,
solves the mapping in closed form, then propagates the updated state to the next
layer. Rerun complete V calibration whenever K or the fitting data changes.
Record source-layer support, numerical precision, data order and sampling
budgets in the fitting configuration.
Set the mapper descriptor in the runtime configuration and use the SGLang integration to run Source TP4 and Target TP4. The online path includes block conversion, double buffering, GPU peer copies and direct installation into the target cache. For a new adaptor, verify consistency across the complete history, check that the source KV remains unchanged, and validate buffer lifetimes.
python benchmarks/tau_bench/compare.py \
--native /path/to/native-evaluation \
--candidate /path/to/mapped-evaluation \
--tasks recipes/tau_test.json --output /path/to/comparison.jsonEvaluate with the same router, task roster, model revisions, user simulator,
generation budgets and seed. Compare rewards for complete tasks, and record
handoffs, fallbacks, tool calls and stop reasons. Use
benchmarks/tau_bench/analyze.py for comparisons across three evaluation arms.
See the trajectory API guide for interface contracts, the fitting guide for numerical steps, and the serving guide for deployment settings.