Skip to content

feat(metrics): expose prefix-cache eviction count as a Prometheus counter - #4888

Open
SuperMarioYL wants to merge 1 commit into
InternLM:mainfrom
SuperMarioYL:feat/prefix-cache-eviction-counter
Open

feat(metrics): expose prefix-cache eviction count as a Prometheus counter#4888
SuperMarioYL wants to merge 1 commit into
InternLM:mainfrom
SuperMarioYL:feat/prefix-cache-eviction-counter

Conversation

@SuperMarioYL

Copy link
Copy Markdown
Contributor

Motivation

Issue #1942 asked for prefix-cache hit/miss/eviction statistics to detect cache thrashing. PR #4670 landed the hit side (lmdeploy:cached_tokens_total / lmdeploy:prompt_tokens_total), but the eviction half was never implemented: KVBlockLifecycle.evict / BlockTrie.evict compute the real evicted-block count, yet the eviction-helper call sites discard it (returning only a success bool), so the count never reaches the metrics layer. Operators cannot write a thrashing alert on evictions today.

This adds lmdeploy:evicted_blocks_total, completing the eviction half of #1942.

A dedicated counter is preferred over deriving evictions from gpu_cache_usage_perc swings or the free-block delta: those conflate evictions with admission/usage changes and don't expose an eviction rate. A dedicated counter gives a direct, low-noise signal, e.g. rate(lmdeploy:evicted_blocks_total[5m]) / rate(lmdeploy:cached_tokens_total[5m]). PR #4670 established the dedicated-counter precedent for prefix-cache stats.

Refs #1942.

Modification

Wires the eviction count through the existing stats bridge that prefix_cache_hit_rate already uses (the paging layer stays metrics-free — no lmdeploy.metrics import is added to paging):

  • block_trie/trie.py: BlockTrie.evict accumulates self.stats.num_evicted_blocks += evicted (the real count is in hand there); field added to PrefixCacheStats.
  • messages.py: num_evicted_blocks field on ScheduleMetrics.
  • paging/scheduler.py: populated in schedule_metrics from self.block_trie.stats.num_evicted_blocks (mirrors the existing prefix_cache_hit_rate read).
  • metrics/stats.py: num_evicted_blocks on SchedulerStats (+ copy in update_from_schedule_metrics, + repr).
  • metrics/loggers.py: lmdeploy:evicted_blocks_total Counter (mirroring cached_tokens_total); record_schedule increments the delta vs _last_num_evicted_blocks (cumulative→counter idiom — schedule_metrics is polled ~every 10 s with the cumulative total, so a naive .inc(cumulative) would double-count).

num_evicted_blocks is deliberately excluded from PrefixCacheStats.snapshot/restore: evictions are an irreversible physical side-effect (freed blocks cannot be un-evicted), so evictions occurring during a subsequently-rolled-back admission attempt must be preserved rather than rolled back with the tentative match state.

No file-level overlap with the in-flight CacheEngine redesign (#4862): that PR touches the cache_engine/backends/executor/spec_agent layers and block_trie/README.md, but not block_trie/trie.py, scheduler.py, messages.py, metrics/stats.py, or metrics/loggers.py.

BC-breaking

None. Both new fields (PrefixCacheStats.num_evicted_blocks, SchedulerStats.num_evicted_blocks) default to 0; ScheduleMetrics consumers that omit the keyword argument are unaffected. The Turbomind backend constructs ScheduleMetrics without the field, so the counter simply stays 0 there (this change targets the PyTorch prefix-cache path).

Use cases

# eviction rate
rate(lmdeploy:evicted_blocks_total[5m])
# thrashing ratio (evictions relative to cache hits)
rate(lmdeploy:evicted_blocks_total[5m]) / rate(lmdeploy:cached_tokens_total[5m])

Checklist

  1. ruff check passes on all changed files (line-length 120, repo config).
  2. New test_evicted_blocks_counter_increments_by_delta (red on master — field absent → TypeError; green on branch — counter increments by delta across polled cumulative 0→5→12→12, no double-count). Existing tests/test_lmdeploy/test_metrics_loggers.py and tests/pytorch/paging/test_block_trie/ (94 tests) still pass.
  3. No new downstream dependency.
  4. Counter name documented inline; the broader metrics catalog can be documented separately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant