Add low-yield compaction backoff to stop wasteful copy-forward on all-live data

[Divya Mahendran (dimahend)]

Summary

Lookup/Scan log compaction wastes NVMe write bandwidth on all-live,
write-once, larger-than-memory (LTM) datasets. On such data a compaction
cycle copies almost every live record forward to reclaim only a few hundred
bytes of segment-alignment slack, and then repeats on the next cycle — because
the legacy productivity test (netReclaimed > 0) misreads that alignment noise
as forward progress. The result is a compaction loop that pins the storage
device at a high write duty even when the server is otherwise idle,
starving foreground reads of device bandwidth.

This PR adds a low-yield compaction backoff: a cycle is treated as
productive only if it reclaims a meaningful fraction of the range it scanned.
When a cycle reclaims essentially nothing, compaction is parked until the log
tail grows (i.e. genuinely new garbage arrives), then resumes automatically.

Problem detail

LogCompactionType.Lookup (and Scan) advance the log begin address by
copying still-live records forward to the tail. On a dataset that is entirely
live (write-once, no overwrites/deletes), the copy-forward re-writes nearly
every byte it truncates, so:

netReclaimed = beginAddressAdvance - tailAddressGrowth ≈ 0

netReclaimed sits at a few hundred bytes (segment-alignment rounding) even
after moving tens of GB. The old gate only checked netReclaimed > 0, so it
classified the cycle as progress and immediately compacted again on the next
trigger — forever.

Observed on a 100M × 1KB O_DIRECT dataset (working set ~94% on disk): stock
Lookup compaction sustained ~470 MB/s of copy-forward writes (~73% device
write-duty) at idle, purely to reclaim alignment slack.

Fix

A yield-ratio gate in a new CompactionState:

• After each cycle, compute netReclaimed = beginAddressAdvance - tailAddressGrowth.
• Require netReclaimed >= ceil(beginAddressAdvance * CompactionLowYieldReclaimPercent / 100) (default 20%) for the cycle to count
  as productive.
• If a cycle is unproductive, park compaction until the tail grows by
  CompactionLowYieldBackoffSegments segments, then resume. Foreground write
  growth re-arms it, so real garbage is always eventually collected.

The gate is correctness-preserving: it only defers compaction when there
is nothing worth reclaiming; any genuinely reclaimable garbage re-triggers it.

Configuration

Knob	Default	Meaning
--compaction-low-yield-backoff-segments (CompactionLowYieldBackoffSegments)	0 (disabled)	Segments the tail must grow before retrying after a no-reclaim cycle. 0 disables the feature.
CompactionLowYieldReclaimPercent	20	Minimum percent of the scanned range a cycle must reclaim to be considered productive. Only applies when backoff is enabled.

The feature is opt-in via config (defaults.conf ships
CompactionLowYieldBackoffSegments = 0). The in-code GarnetServerOptions
field defaults to 32 so that embedders which construct GarnetServerOptions
directly (without binding from defaults.conf) still get the protection; the
standard host + config path leaves it disabled unless the operator opts in.

Evidence

Single-cell A/B/C on a 100M × 1KB O_DIRECT dataset, 16 GB log memory (~94% of
the working set on disk), client on a separate VM:

Arm	Throughput	p99	Compaction writes
A — Stock Lookup	34,563 ops/s	4359 µs	~470 MB/s (wasted)
B — CompactionType None	46,476 ops/s	482 µs	0
C — This fix (Lookup + backoff)	45,284 ops/s	491 µs	0 (parks)

The fix matches "compaction off": +31% throughput and ~9× better p99 versus
stock, by eliminating the wasteful copy-forward writes that were competing with
reads for device bandwidth. The win is largest when the working set is
device-bound (LTM); RAM-resident workloads see little client-visible change
because reads don't touch the device.

Changes

• libs/server/Databases/CompactionPolicy.cs (new) — CompactionState
  (ShouldSkip / TryResume / RecordCycle) and CompactionPolicy helpers
  (GetBackoffBytes, GetUntilAddress).
• libs/server/Databases/DatabaseManagerBase.cs — gate the main-store
  compaction loop on CompactionState; emit per-cycle telemetry (begin
  advance, tail growth, net reclaimed, ratio, duration).
• libs/server/Servers/GarnetServerOptions.cs — new
  CompactionLowYieldBackoffSegments and CompactionLowYieldReclaimPercent.
• libs/host/Configuration/Options.cs + libs/host/defaults.conf — expose
  --compaction-low-yield-backoff-segments (default 0).
• libs/server/GarnetDatabase.cs — per-database CompactionState.
• test/standalone/Garnet.test/CompactionPolicyTests.cs (new) — unit tests
  for the yield-ratio gate (7 cases, incl. tiny-positive-reclaim backs off).

Testing

dotnet test on CompactionPolicyTests — 7/7 pass, including
TinyPositiveReclaimBacksOff (verifies a cycle that reclaims a few bytes after
advancing the begin address is treated as unproductive and parks).

Notes for reviewers

• Version.props is intentionally not bumped here.
• CompactionLowYieldReclaimPercent is currently a code-level default (20) and
  not surfaced as a CLI/config option; happy to expose it if reviewers prefer.

[Divya Mahendran (dimahend)]

@microsoft-github-policy-service agree company="Microsoft"

[Copilot]

Pull request overview

Adds opt-in low-yield compaction backoff to reduce wasteful copy-forward writes on mostly live datasets.

Changes:

• Tracks compaction yield and pauses low-yield cycles until tail growth.
• Adds server and host configuration for the backoff.
• Adds policy unit tests and compaction telemetry.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
libs/server/Databases/CompactionPolicy.cs	Implements yield and retry policy.
libs/server/Databases/DatabaseManagerBase.cs	Integrates backoff and telemetry.
libs/server/GarnetDatabase.cs	Stores per-database compaction state.
libs/server/Servers/GarnetServerOptions.cs	Adds policy settings.
libs/host/Configuration/Options.cs	Exposes the backoff option.
libs/host/defaults.conf	Disables host backoff by default.
test/standalone/Garnet.test/CompactionPolicyTests.cs	Tests policy calculations and state.

Suppressed comments (2)

libs/server/Databases/DatabaseManagerBase.cs:498

• This tail delta includes concurrent foreground appends as well as records copied by compaction. A garbage-rich cycle can therefore appear low-yield whenever clients append enough data while the scan runs, causing compaction to park even though it reclaimed substantial space. Track compaction-generated tail bytes separately (or otherwise exclude foreground growth) before using this value for the yield decision.

                var beginAddressAdvance = beginAddressAfter - beginAddressBefore;
                var tailAddressGrowth = tailAddressAfter - tailAddressBefore;
                var netReclaimedBytes = beginAddressAdvance - tailAddressGrowth;

libs/server/Databases/DatabaseManagerBase.cs:514

• The warning is inaccurate: RecordCycle also parks cycles that reclaim a positive amount below the configured percentage. Report that the yield was below threshold rather than that no space was reclaimed, otherwise operators may misdiagnose the telemetry.

                        "Compaction reclaimed no space; pausing until tail reaches {retryTailAddress}; DB ID = {id}",

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Divya Mahendran (dimahend)]

Addressed in bac84f0. The single bounded pass is now a drain-fully loop: while the log is over the limit, compaction keeps compacting bounded chunks. Productive chunks continue draining so a high-churn workload that generates more than one chunk of garbage per interval can no longer outrun compaction, while the first low-yield chunk parks compaction until the tail grows again (the all-live copy-forward case this PR targets). The non-backoff path keeps the original single unbounded pass, and there is a safety break if a chunk classified as productive fails to advance the begin address. Added ProductiveChunksDrainThenLowYieldParks covering the state sequence.