[server] fix stop replica deletion stuck when TabletServer is offline

[gyang94]

Purpose

Linked issue: close #3357

Brief change log

Summary

When a stopReplica RPC fails due to transient network issues or a TabletServer crash, the Coordinator has no reliable retry mechanism. This causes replicas to get stuck and table deletion to never complete, resulting in the tableCount metric never decreasing.

This PR introduces a per-TabletServer sender thread model (aligned with Kafka's ControllerChannelManager / RequestSendThread) and a new ReplicaDeletionIneligible state. These changes provide robust retry and pause/resume semantics for replica deletion.

Changes

Core: Per-TS Sender Thread (ControlRequestSendThread)

• Dedicated Sender Thread: Each TabletServer gets a dedicated sender thread with a FIFO queue.
• New Replica State: Introduced a state for replicas whose deletion cannot proceed (e.g., TS offline or returned a business error).
• Resume Logic: TableManager.resumeDeletions() implements 3-step logic:
  1. Complete if all replicas succeeded.
  2. Retry previously-ineligible replicas on alive TSes.
  3. Re-fire eligible tables.
• Auto-Resume on Reconnect: processNewTabletServer() clears ineligible marks and triggers resumeDeletions(), so paused deletions automatically resume when a TS reconnects.
• Handle Dead TS: processDeadTabletServer() transitions in-flight deletion replicas to ineligible.

Config

• coordinator.request.retry.backoff: Backoff between retries (default: 100ms).
• coordinator.request.timeout: RPC timeout per attempt (default: 30s).

️ What was removed

• retryDeleteAndSuccessDeleteReplicas(): The old "retry-N-then-force-success" mechanism.
• failDeleteNumbers tracking and DELETE_TRY_TIMES constant.
• Direct RPC calls from CoordinatorRequestBatch (replaced by queue-based dispatch).

Tests

API and Format

Documentation