cachedb_redis: add dynamic cluster topology management and observability

[NormB]

Summary

• Replace static cluster topology with runtime discovery and automatic refresh
• Add MI commands for cluster observability (redis_cluster_info, redis_cluster_refresh, redis_ping_nodes)
• Add per-node statistics counters and module-level stat variables
• Add hash tag {...} extraction and fix hash slot calculation
• Add TCP keepalive for connection health detection

Details

The cachedb_redis module currently builds its cluster topology once during child_init via CLUSTER NODES and never updates it. Adding or removing a Redis node requires restarting OpenSIPS. This PR replaces that static model with full runtime topology management.

Topology discovery and refresh

At startup, the module probes the cluster using CLUSTER SHARDS (Redis 7+) with automatic fallback to CLUSTER SLOTS (Redis 3+). The topology is stored in an O(1) slot_table[16384] array mapping each slot directly to its owning master node.

The topology refreshes automatically when:

• A MOVED redirection is received (permanent slot migration)
• A connection failure occurs and the node cannot be reconnected
• A query targets a slot with no known owner
• An operator triggers redis_cluster_refresh via MI

Refreshes are rate-limited to once per second. The MI command bypasses the rate limit. During refresh, nodes not present in the new topology are pruned and their connections freed. New nodes are created and connected on demand.

The redirect retry loop is capped at 5 total redirects to prevent a worker from hanging on a pathological cluster state.

Hash slot correctness

The existing redisHash() uses crc16(key) & con->slots_assigned which only produces correct results when slots_assigned is 2^n - 1. This is replaced with crc16(key) % 16384 per the Redis Cluster specification.

Hash tag extraction is added: if a key contains {substring}, only the substring between the first { and the next } is hashed, enabling key co-location (e.g., {user:1000}.session and {user:1000}.profile land on the same slot).

MI commands

Command	Description
redis_cluster_info	Topology dump: nodes, slots, per-node counters (queries, errors, moved), connection status, last activity. Optional group filter.
redis_cluster_refresh	Trigger topology refresh. Optional group filter.
redis_ping_nodes	PING each node with microsecond latency reporting. Optional group filter.

Statistics

redis_queries, redis_queries_failed, redis_moved, redis_topology_refreshes

New parameter

Parameter	Type	Default	Description
redis_keepalive	integer	10	TCP keepalive interval in seconds (0 to disable)

Testing

170 tests across 7 suites:

Suite	Tests	Description
test_hash (C)	37	Hash slot calculation, hash tags, edge cases
test_mi_counters (C)	34	MI counter helpers, slot counting, struct layout
test_topology_startup	12	Slot coverage, multi-node routing, random keys
test_topology_refresh	13	Dynamic topology updates after slot migration
test_mi_commands	49	All MI commands, statistics, health checks
test_edge_cases	10	Node add/remove, outage recovery
test_load	6	Memory leak detection under topology changes

All integration tests include trap EXIT cleanup handlers to restore cluster state on unexpected exit.

Compatibility

• Plain keys (no {...}) produce identical slot values — no behavioral change for existing deployments
• Keys with hash tags will route to different (correct per Redis spec) slots
• CLUSTER NODES parsing is replaced by CLUSTER SHARDS/CLUSTER SLOTS
• The slots_assigned field is removed from redis_con

Dependencies

• Depends on cachedb_redis: add Redis cluster hash tag support to redisHash #3815 (hash tag extraction and modulo fix)
• Depends on cachedb_redis: add ASK redirect handling for cluster resharding #3852 (ASK redirect handling)
• Depends on cachedb_redis: fix safety issues in cluster redirect parsing #3854 (safety fixes)

This PR includes hash tag support for standalone functionality. If #3815 merges first, the overlap resolves cleanly on rebase.

Closing issues

Partially addresses #2811

[dondetir]

@NormB — I went through the full PR and ran it against a local 3-node Redis 7.x cluster. Solid work across the board.

Everything checked out well in testing:

• Unit tests: 37/37 (hash) + 34/34 (MI counters), clean under valgrind
• Cluster operations: store/fetch across all nodes, MOVED handling after live reshard, node failure + recovery — all working as expected
• MI commands (redis_cluster_info, redis_cluster_refresh, redis_ping_nodes) all functional with correct output

One minor thing — the test files (test_hash.c, test_mi_counters.c, hash_under_test.c) are standalone executables with their own main(), which works great when built via the test Makefile. However, when UNIT_TESTS is enabled in Makefile.conf, the build system's auto-discovery (Makefile.defs:1695) pulls test/*.c into the module .so, causing multiple-definition linker errors. A quick #ifndef UNIT_TESTS guard around main() or an exclude rule in the module Makefile should do the trick.

Also spotted a committed binary — test/test_mi_counters (the compiled ELF) is tracked in git. Looks like test/.gitignore lists test_hash but not test_mi_counters.

Code looks great overall. The rate-limited topology refresh and the seen flag reconciliation pattern are clean and effective.