feat(metrics): expose prefix-cache eviction count as a Prometheus counter - #4888
Open
SuperMarioYL wants to merge 1 commit into
Open
feat(metrics): expose prefix-cache eviction count as a Prometheus counter#4888SuperMarioYL wants to merge 1 commit into
SuperMarioYL wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Issue #1942 asked for prefix-cache hit/miss/eviction statistics to detect cache thrashing. PR #4670 landed the hit side (
lmdeploy:cached_tokens_total/lmdeploy:prompt_tokens_total), but the eviction half was never implemented:KVBlockLifecycle.evict/BlockTrie.evictcompute the real evicted-block count, yet the eviction-helper call sites discard it (returning only asuccessbool), so the count never reaches the metrics layer. Operators cannot write a thrashing alert on evictions today.This adds
lmdeploy:evicted_blocks_total, completing the eviction half of #1942.A dedicated counter is preferred over deriving evictions from
gpu_cache_usage_percswings or the free-block delta: those conflate evictions with admission/usage changes and don't expose an eviction rate. A dedicated counter gives a direct, low-noise signal, e.g.rate(lmdeploy:evicted_blocks_total[5m]) / rate(lmdeploy:cached_tokens_total[5m]). PR #4670 established the dedicated-counter precedent for prefix-cache stats.Refs #1942.
Modification
Wires the eviction count through the existing stats bridge that
prefix_cache_hit_ratealready uses (the paging layer stays metrics-free — nolmdeploy.metricsimport is added to paging):block_trie/trie.py:BlockTrie.evictaccumulatesself.stats.num_evicted_blocks += evicted(the real count is in hand there); field added toPrefixCacheStats.messages.py:num_evicted_blocksfield onScheduleMetrics.paging/scheduler.py: populated inschedule_metricsfromself.block_trie.stats.num_evicted_blocks(mirrors the existingprefix_cache_hit_rateread).metrics/stats.py:num_evicted_blocksonSchedulerStats(+ copy inupdate_from_schedule_metrics, +repr).metrics/loggers.py:lmdeploy:evicted_blocks_totalCounter (mirroringcached_tokens_total);record_scheduleincrements the delta vs_last_num_evicted_blocks(cumulative→counter idiom —schedule_metricsis polled ~every 10 s with the cumulative total, so a naive.inc(cumulative)would double-count).num_evicted_blocksis deliberately excluded fromPrefixCacheStats.snapshot/restore: evictions are an irreversible physical side-effect (freed blocks cannot be un-evicted), so evictions occurring during a subsequently-rolled-back admission attempt must be preserved rather than rolled back with the tentative match state.No file-level overlap with the in-flight CacheEngine redesign (#4862): that PR touches the
cache_engine/backends/executor/spec_agentlayers andblock_trie/README.md, but notblock_trie/trie.py,scheduler.py,messages.py,metrics/stats.py, ormetrics/loggers.py.BC-breaking
None. Both new fields (
PrefixCacheStats.num_evicted_blocks,SchedulerStats.num_evicted_blocks) default to0;ScheduleMetricsconsumers that omit the keyword argument are unaffected. The Turbomind backend constructsScheduleMetricswithout the field, so the counter simply stays0there (this change targets the PyTorch prefix-cache path).Use cases
Checklist
ruff checkpasses on all changed files (line-length 120, repo config).test_evicted_blocks_counter_increments_by_delta(red on master — field absent →TypeError; green on branch — counter increments by delta across polled cumulative0→5→12→12, no double-count). Existingtests/test_lmdeploy/test_metrics_loggers.pyandtests/pytorch/paging/test_block_trie/(94 tests) still pass.