Fix pebble db metrics

[alrevuelta]

Some pebble db metrics are wrong.

The rootcause is this this

We keep adding m.BlockCache.Hits to the cacheHits metric (the one shown in the image) BUT m.BlockCache.Hits is cumulative. Meaning we keep adding the cumulative value over and over, producing a wrong metric.

otelMetrics.cacheHits.Add(ctx, m.BlockCache.Hits)

I found this bug when checking pebble_cache_hits_total in a benchmark where I was not reading any data (just a read at startup. But this metrics kept increasing forever. Bug fixed (expected) left what we have (bug) right.

This PR fixes it by setting the value pebble provides, the cumulative. Instead of adding it over and over.

⚠️ This change implies changing some metrics from counter to gauge. Which means that the _total sufix is removed from the following metrics. eg:

• before pebble_compaction_count_total
• is now pebble_compaction_count

[cursor]

PR Summary

Medium Risk
Observability-only changes, but metric semantics and series labels change (including counter vs gauge behavior noted in the PR), which can break existing dashboards and alerts until queries are updated.

Overview
Fixes incorrect Pebble observability by removing MVCC’s periodic scraper that repeatedly added Pebble’s cumulative stats (e.g. block cache hits) on every tick, which inflated counters when the process was idle. Pebble-internal metrics (compaction, flush, SSTable, memtable, WAL, cache) are now reported via pebbledb.NewPebbleMetrics, and the duplicate instruments are dropped from mvcc/metrics.go.

Multi-database processes (e.g. separate EVM sub-DBs) get a stable db attribute on MVCC operation metrics—get/apply/prune/import latency, batch writes, async queue depth, iterator iterations, prune failures—using filepath.Base(dataDir) stored on Database and threaded through batches and iterators.

^{Reviewed by Cursor Bugbot for commit 2b8b377. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	✅ passed	✅ passed	✅ passed	Aug 28, 2026, 8:23 AM

[codecov]

Codecov Report

❌ Patch coverage is 87.50000% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.28%. Comparing base (fc586f4) to head (2b8b377).

Files with missing lines	Patch %	Lines
sei-db/db_engine/pebbledb/mvcc/batch.go	75.00%	3 Missing ⚠️
sei-db/db_engine/pebbledb/mvcc/db.go	90.62%	3 Missing ⚠️
...i-db/db_engine/pebbledb/mvcc/iterator_ascending.go	75.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4031      +/-   ##
==========================================
- Coverage   61.27%   60.28%   -1.00%     
==========================================
  Files        2153     2054      -99     
  Lines      188426   176733   -11693     
==========================================
- Hits       115465   106537    -8928     
+ Misses      62232    60435    -1797     
+ Partials    10729     9761     -968     

Flag	Coverage Δ
sei-chain-pr	65.80% <87.50%> (?)
sei-db	69.80% <ø> (ø)
sei-db-state-db	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
sei-db/db_engine/pebbledb/mvcc/db_ascending.go	63.84% <100.00%> (+0.41%)	⬆️
sei-db/db_engine/pebbledb/mvcc/iterator.go	69.09% <100.00%> (+0.34%)	⬆️
sei-db/db_engine/pebbledb/mvcc/metrics.go	50.00% <ø> (ø)
...i-db/db_engine/pebbledb/mvcc/iterator_ascending.go	66.66% <75.00%> (ø)
sei-db/db_engine/pebbledb/mvcc/batch.go	51.72% <75.00%> (+1.27%)	⬆️
sei-db/db_engine/pebbledb/mvcc/db.go	66.52% <90.62%> (-0.70%)	⬇️

... and 99 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[seidroid]

The diagnosis is right — pebble's Metrics() returns cumulative values, so Add-ing them each scrape double-counts — but the chosen remedy (counter → gauge + Record) introduces new problems: the shared package-level gauges are last-write-wins across the multiple mvcc.Database instances a node opens, the duration metrics now conflict in instrument kind with the identically-named histograms in the sibling pebbledb package, and two //nolint:gosec directives were removed while the offending conversions stayed. The repo already contains the correct fix pattern in pebbledb.PebbleMetrics (addDelta + a db attribute), which resolves all three.

Findings: 3 blocking | 4 non-blocking | 3 posted inline

Blockers

• None at the file/PR level.
• 3 blocking issue(s) flagged inline on specific lines.

Non-blocking

• [suggestion] docker/monitornode/dashboards/cryptosim-dashboard.json still queries the _total names (lines 6163, 8665, 8854, 9138, 13726, 13821). Those panels keep working for the stores served by pebbledb.PebbleMetrics, but the MVCC state-store/receipt-store contribution drops out of them. Keeping counter semantics via addDelta avoids touching the dashboard at all; if the gauge direction is kept intentionally, the dashboard needs updating in this PR.
• [suggestion] The pebble_* names are a public monitoring contract consumed outside this repo. If the counter → gauge rename does land, it is worth calling out in the PR body's breaking-change note which external dashboards/alerts own the _total series, since neither implementation emits both names during a transition window.
• 2 non-blocking pre-existing issue(s) listed below under pre-existing issues.

Pre-existing issues

• [suggestion] mvcc's metric instruments (sei-db/db_engine/pebbledb/mvcc/metrics.go) are package-level singletons shared by every mvcc.Database with no per-DB attribute, while the already-gauge metrics (sstableCount, memtableCount, walSize, cacheSize) are recorded via Record. When two MVCC DBs are open these already overwrite each other. The PR extends the pattern rather than introducing it.
• [suggestion] sei-db/db_engine/pebbledb/mvcc/collectAndRecordMetrics and pebbledb.PebbleMetrics.scrape are two independent implementations scraping the same pebble.Metrics into the same instrument names, with the latter a strict superset. The duplication is what allows the two definitions to drift out of sync; there is no test on either side pinning the instrument kinds.

[seidroid]

[blocker] Record on a shared gauge is last-write-wins, so this does not produce a correct series when more than one mvcc.Database is open.

otelMetrics is a package-level singleton and every mvcc.OpenDB starts its own collectMetricsInBackground (db.go:253). None of these recordings carry a per-DB attribute — only level on the two per-level ones. A node that runs the state store on the pebble MVCC backend and the receipt store on receiptBackendPebble (ledger_db/receipt/receipt_store.go:168) therefore has two collectors writing the same instrument with the same (empty) attribute set every 10s. With Add the exported value was at least the sum of both; with Record the single series alternates between the state store's cumulative total and the receipt store's, giving a sawtooth that is as unusable as the bug being fixed.

The repo already has the shape that solves both halves of this: pebbledb.PebbleMetrics converts pebble's cumulative counters to counter deltas via addDelta (pebble_metrics.go:1189) and attaches a db attribute precisely so multiple DBs get distinct series (pebble_metrics.go:19-25). mvcc already imports that package as pebbledbmetrics (db.go:28) and holds a *pebble.DB in db.storage, so pebbledbmetrics.NewPebbleMetrics(ctx, db.storage, <name>, interval) can replace this function and its duplicate instrument block outright — which also removes the instrument-kind conflict and the _total rename noted on metrics.go.

[alrevuelta]

indeed. pebble metrics were duplicated mvcc/metrics.go and pebble_metrics.go fixed 2b8b377

[seidroid]

[blocker] These names are not private to this package: pebbledb.PebbleMetrics registers the same instrument names on the same seidb_pebble meter, and its doc comment states that alignment is deliberate ("Instrument names match sei-db/db_engine/pebbledb/mvcc for dashboard compatibility", pebble_metrics.go:20). Both are live in one process — flatkv/store.go constructs pebbledb.New for the SC store while the SS store uses mvcc.OpenDB.

Two consequences:

• pebble_compaction_duration and pebble_flush_duration are still Float64Histogram there (pebble_metrics.go:306, :429). Registering them as Float64Gauge here means the same meter yields two conflicting instrument kinds under one name, and after Prometheus naming both land on the identical family name pebble_{compaction,flush}_duration_seconds with different types. Only one of them can be exposed, so one silently disappears from /metrics.
• For compaction_count / flush_count / *_bytes_* / cache_{hits,misses} the sibling counters keep the _total suffix while these gauges drop it, so one logical metric splits into two differently-named series depending on which store produced it.

Switching this file to the addDelta conversion used by the sibling keeps the instrument kinds and exported names identical across both implementations and still fixes the cumulative-double-counting bug.

[alrevuelta]

fixed 2b8b377

[seidroid]

[blocker] The //nolint:gosec directives were dropped from these two lines but the uint64 → int64 conversions remain, so gosec G115 will now fire and make lint fails. Note lines 1430 and 1433 keep the directive for the same kind of conversion.

Either restore the directives, or clamp instead — pebbledb.uint64ToInt64Clamped (pebble_metrics.go:1182) is the local precedent, though it is unexported so it would need a small equivalent here.

[alrevuelta]

removed this duplicated metrics 2b8b377

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 4783f13. Configure here.}

[yzang2019]

LGTM overall after fixing the blocker comments

[alrevuelta]

@yzang2019 i have slightly changed the scope of the PR based on the bot comments.

• mvcc/metrics.go contained duplicates already available in pebble_metrics.go. so i have removed the duplicated ones and now pebble metrics are reused from pebble_metrics.go. one source of truth for pebble metrics now.
• i also fixed in an issue where when using SeparateEVMSubDBs all pebble metrics were added to the same gauges/counters. this was really missleading causing the metrics to make no sense.