Protect last healthy shard replica during interrupted rolls (#1704)

[alex-zaitsev]

Reconcile unhealthy replicas in the shard prior to healthy ones. Solves #1704. Replaces #1967

Summary

• Recovery-first shard reconcile with peer-health gate before disruptive updates
• E2E test_010083 for interrupted broken-image roll + operator restart

Test plan

• test_010083
• smoke: test_010008*, test_0100022, test_0100028

[alex-zaitsev]

Final review — #2046

Commits: 93eaa5488 (shard safety) + a1be5bb63 (operator IP restore). Branch matches remote.

Verdict: approve

Addresses #1704 without dropping the operator-IP reconcile needed for clickhouse-operator user networks.

Behavior (correct)

Case	Outcome
Generation same + healthy + IP same	Skip
Generation same + healthy + IP changed	Continue (networks refresh)
Unhealthy hosts	Continue (recovery)
Disruptive roll with no healthy peer	Defer host + ErrCRUDAbort
Unhealthy peer in shard	Recovery first; do not roll last healthy

prevCHOpIP before buildCR is the right way to detect IP change.

Residual nits (non-blocking)

1. Force-restart “skip” in reconcileHostStatefulSet only skips software restart; STS reconcile still runs. Shard-level deferral is the real gate.
2. Unused down_start_time in test_010083.
3. hasUnhealthyHosts does not short-circuit the host walk.

No further blockers from this pass.