Add to 10.1.2: Fix cache retry assert on ServerAddrSe

[DawidKowalczyk]

Hello,
could you add this changes to v. 10.1.2 ?
#12972

Unortunetly, we see a lot of crush without this changes.