Multi-database client implementation (#3784)

* MultiDbClient implementation (#3696)

* Added Database, Healthcheck, CircuitBreaker, FailureDetector

* Added DatabaseSelector, exceptions, refactored existing entities

* Added MultiDbConfig

* Added DatabaseConfig

* Added DatabaseConfig test coverage

* Renamed DatabaseSelector into FailoverStrategy

* Added CommandExecutor

* Updated healthcheck to close circuit on success

* Added thread-safeness

* Added missing thread-safeness

* Added missing thread-safenes for dispatcher

* Refactored client to keep databases in WeightedList

* Added database CRUD operations

* Added on-fly configuration

* Added background health checks

* Added background healthcheck + half-open event

* Refactored background scheduling

* Refactored healthchecks

* Removed code repetitions, fixed weight assignment, added loops enhancement, fixed data structure

* Refactored configuration

* Refactored failure detector

* Refactored retry logic

* Added scenario tests

* Added pybreaker optional dependency

* Added pybreaker to dev dependencies

* Rename tests directory

* Remove redundant checks

* Handle retries if default is not set

* Removed all Sentinel related

* Added support for Pipeline and transactions (#3707)

* Added Database, Healthcheck, CircuitBreaker, FailureDetector

* Added DatabaseSelector, exceptions, refactored existing entities

* Added MultiDbConfig

* Added DatabaseConfig

* Added DatabaseConfig test coverage

* Renamed DatabaseSelector into FailoverStrategy

* Added CommandExecutor

* Updated healthcheck to close circuit on success

* Added thread-safeness

* Added missing thread-safeness

* Added missing thread-safenes for dispatcher

* Refactored client to keep databases in WeightedList

* Added database CRUD operations

* Added on-fly configuration

* Added background health checks

* Added background healthcheck + half-open event

* Refactored background scheduling

* Added support for Active-Active pipeline

* Refactored healthchecks

* Added Pipeline testing

* Added support for transactions

* Removed code repetitions, fixed weight assignment, added loops enhancement, fixed data structure

* Added missing doc blocks

* Refactored configuration

* Refactored failure detector

* Refactored retry logic

* Added scenario tests

* Added pybreaker optional dependency

* Added pybreaker to dev dependencies

* Rename tests directory

* Added scenario tests for Pipeline and Transaction

* Added handling of ConnectionRefusedError, added timeouts so cluster could recover

* Increased timeouts

* Refactored integration tests

* Fixed property name

* Removed sentinels

* Removed unused method

* Added support for Pub/Sub mode in MultiDbClient (#3722)

* Added Database, Healthcheck, CircuitBreaker, FailureDetector

* Added DatabaseSelector, exceptions, refactored existing entities

* Added MultiDbConfig

* Added DatabaseConfig

* Added DatabaseConfig test coverage

* Renamed DatabaseSelector into FailoverStrategy

* Added CommandExecutor

* Updated healthcheck to close circuit on success

* Added thread-safeness

* Added missing thread-safeness

* Added missing thread-safenes for dispatcher

* Refactored client to keep databases in WeightedList

* Added database CRUD operations

* Added on-fly configuration

* Added background health checks

* Added background healthcheck + half-open event

* Refactored background scheduling

* Added support for Active-Active pipeline

* Refactored healthchecks

* Added Pipeline testing

* Added support for transactions

* Removed code repetitions, fixed weight assignment, added loops enhancement, fixed data structure

* Added missing doc blocks

* Added support for Pub/Sub in MultiDBClient

* Refactored configuration

* Refactored failure detector

* Refactored retry logic

* Added scenario tests

* Added pybreaker optional dependency

* Added pybreaker to dev dependencies

* Rename tests directory

* Added scenario tests for Pipeline and Transaction

* Added handling of ConnectionRefusedError, added timeouts so cluster could recover

* Increased timeouts

* Refactored integration tests

* Added scenario tests for Pub/Sub

* Updated healthcheck retry

* Increased timeout to avoid unprepared state before tests

* Added backoff retry and changed timeouts

* Added retry for healthchecks to avoid fluctuations

* Changed retry configuration for healthchecks

* Fixed property name

* Added check for thread results

* Refactored docblocks (#3744)

* Refactored healthcheck and failure detector to extend default one (#3747)

* Added MultiDbClient support with OSS Cluster API (#3734)

* Added Database, Healthcheck, CircuitBreaker, FailureDetector

* Added DatabaseSelector, exceptions, refactored existing entities

* Added MultiDbConfig

* Added DatabaseConfig

* Added DatabaseConfig test coverage

* Renamed DatabaseSelector into FailoverStrategy

* Added CommandExecutor

* Updated healthcheck to close circuit on success

* Added thread-safeness

* Added missing thread-safeness

* Added missing thread-safenes for dispatcher

* Refactored client to keep databases in WeightedList

* Added database CRUD operations

* Added on-fly configuration

* Added background health checks

* Added background healthcheck + half-open event

* Refactored background scheduling

* Added support for Active-Active pipeline

* Refactored healthchecks

* Added Pipeline testing

* Added support for transactions

* Removed code repetitions, fixed weight assignment, added loops enhancement, fixed data structure

* Added missing doc blocks

* Added support for Pub/Sub in MultiDBClient

* Refactored configuration

* Refactored failure detector

* Refactored retry logic

* Added scenario tests

* Added pybreaker optional dependency

* Added pybreaker to dev dependencies

* Rename tests directory

* Added scenario tests for Pipeline and Transaction

* Added handling of ConnectionRefusedError, added timeouts so cluster could recover

* Increased timeouts

* Refactored integration tests

* Added scenario tests for Pub/Sub

* Updated healthcheck retry

* Increased timeout to avoid unprepared state before tests

* Added backoff retry and changed timeouts

* Added retry for healthchecks to avoid fluctuations

* Changed retry configuration for healthchecks

* Fixed property name

* Added check for thread results

* Added MultiDbClient support with OSS Cluster API

* Removed database statuses

* Increased test timeouts

* Increased retry timeout

* Increased timeout retries

* Updated base threshold for retries

* Fixed flacky tests

* Added missing docblocks

* Added LagAwareHealthCheck for MultiDBClient (#3737)

* Added LagAwareHealthcheck

* Added testing for LagAwareHealthCheck

* Fixed timeouts

* Added lag tollerance parameter

* Decreased messages_count due to increased timeouts

* Added docblocks

* Added missing type hints

* Fixed url

* Refactored tests, URL and cluster support

* Use primary node to send an API request

* Added comment about RE bug

* Moved None type to the beginning

* Added health_check_url property to Database class

* Added lag_aware_tolerance parameter to LagAwareHealthcheck (#3752)

* Extract additional interfaces and abstract classes (#3754)

* Added async implementation of MultiDBClient (#3762)

* Extract additional interfaces and abstract classes

* Added base async components

* Added command executor

* Added recurring background tasks with event loop only

* Added MultiDBClient

* Added scenario and config tests

* Update redis/asyncio/multidb/healthcheck.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update tests/test_asyncio/test_scenario/test_active_active.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Added pipeline and transaction support for MultiDBClient (#3763)

* Extract additional interfaces and abstract classes

* Added base async components

* Added command executor

* Added recurring background tasks with event loop only

* Added MultiDBClient

* Added scenario and config tests

* Added pipeline and transaction support for MultiDBClient

* Updated scenario tests to check failover

* Added pub/sub support for MultiDBClient (#3764)

* Extract additional interfaces and abstract classes

* Added base async components

* Added command executor

* Added recurring background tasks with event loop only

* Added MultiDBClient

* Added scenario and config tests

* Added pipeline and transaction support for MultiDBClient

* Added pub/sub support for MultiDBClient

* Added check for couroutines methods for pub/sub

* Added support for Lag-Aware Healthcheck and OSS Cluster API (#3768)

* Extract additional interfaces and abstract classes

* Added base async components

* Added command executor

* Added recurring background tasks with event loop only

* Added MultiDBClient

* Added scenario and config tests

* Added pipeline and transaction support for MultiDBClient

* Added pub/sub support for MultiDBClient

* Added check for couroutines methods for pub/sub

* Added OSS Cluster API support for MultiDBCLient

* Added support for Lag-Aware Healthcheck and OSS Cluster API

* Increased timeouts between tests

* Fixed space

* Refactored Healthcheck and Failover strategy logic (#3771)

* Extract additional interfaces and abstract classes

* Added base async components

* Added command executor

* Added recurring background tasks with event loop only

* Added MultiDBClient

* Added scenario and config tests

* Added pipeline and transaction support for MultiDBClient

* Added pub/sub support for MultiDBClient

* Added check for couroutines methods for pub/sub

* Added OSS Cluster API support for MultiDBCLient

* Added support for Lag-Aware Healthcheck and OSS Cluster API

* Increased timeouts between tests

* [Sync] Refactored healthcheck

* [Async] Refactored healthcheck

* [Sync] Refactored Failover Strategy

* [Async] Refactored Failover Strategy

* Changed default values according to a design doc

* [Async] Added Strategy Executor

* [Sync] Added Strategy Executor

* Apply comments

* Removed redundant dependency

* Fixed async tests

* Increased lag-aware tolerance

* Fixed typing issue, increase health_check_interval, added timeout handling

* Decreased retry cap, increased failure delay

* Fixed async teardown

* Fixed tests

* Added graceful connection closing, added graceful hc tasks termination

* Make sure active connection will be disconnected on failover

* Close cluster connection on failover

* Refactored Failure Detector (#3775)

* Decreased timeouts

* Added missing fixture

* Fixed None exception

* Codestyle changes

* Codestyle changes

* Skip async scenario tests

* Codestyle change

* Fixed unused arguments

* Refactored bg scheduler

* Fixed tests

* Fixed tests

* Codestyle fixes

* Reduce timeouts to avoid overlaping with healthcheck

* Marked tests non-clsuter only

* Update timeouts

* Skip scenario tests

* Updated timeouts

* Increased timeout

* Refactored tests

* Codestyle changes

* Added documentation for Active-Active (#3753)

* Added Active-Active documentation page

* Added documentation for Active-Active

* Refactored docs

* Refactored pipeline and transaction section

* Updated docs

* Extended list of words

* Re-write documentation

* Fixed spelling

* Update docs/multi_database.rst

Co-authored-by: Elena Kolevska <elena-kolevska@users.noreply.github.com>

* Apply suggested comments

* Fixed spelling

* Update docs/multi_database.rst

Co-authored-by: Elena Kolevska <elena-kolevska@users.noreply.github.com>

* Update docs/multi_database.rst

Co-authored-by: Elena Kolevska <elena-kolevska@users.noreply.github.com>

* Update docs/multi_database.rst

Co-authored-by: Elena Kolevska <elena-kolevska@users.noreply.github.com>

---------

Co-authored-by: Elena Kolevska <elena-kolevska@users.noreply.github.com>

* Refactor unstable tests

* Marked tests as non-clustered

* Codestyle changes

* Skipped tests in validating workflow

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: petyaslavova <petya.slavova@redis.com>
Co-authored-by: Elena Kolevska <elena-kolevska@users.noreply.github.com>

Author: 4 people

.github/wordlist.txt

@@ -1,7 +1,9 @@
 APM
 ARGV
 BFCommands
+balancer
 CacheImpl
+cancelling
 CAS
 CFCommands
 CMSCommands
@@ -10,19 +12,32 @@ ClusterNodes
 ClusterPipeline
 ClusterPubSub
 ConnectionPool
+config
 CoreCommands
+DatabaseConfig
+DNS
+EchoHealthCheck
 EVAL
 EVALSHA
+failover
+FQDN
 Grokzen's
+Healthcheck
+HealthCheckPolicies
+healthcheck
+healthchecks
 INCR
 IOError
 Instrumentations
 JSONCommands
 Jaeger
 Ludovico
 Magnocavallo
+MultiDbConfig
+MultiDBClient
 McCurdy
 NOSCRIPT
+NoValidDatabaseException
 NUMPAT
 NUMPT
 NUMSUB
@@ -43,6 +58,7 @@ RedisInstrumentor
 RedisJSON
 RedisTimeSeries
 SHA
+SLA
 SearchCommands
 SentinelCommands
 SentinelConnectionPool
@@ -52,6 +68,8 @@ SpanKind
 Specfiying
 StatusCode
 TCP
+TemporaryUnavailableException
+TLS
 TOPKCommands
 TimeSeriesCommands
 Uptrace
@@ -91,6 +109,7 @@ json
 keyslot
 keyspace
 kwarg
+kwargs
 linters
 localhost
 lua

.github/workflows/install_and_test.sh

@@ -40,11 +40,12 @@ cd ${TESTDIR}
 # install, run tests
 pip install ${PKG}
 # Redis tests
-pytest -m 'not onlycluster' --ignore=tests/test_scenario
+pytest -m 'not onlycluster' --ignore=tests/test_scenario --ignore=tests/test_asyncio/test_scenario
 # RedisCluster tests
 CLUSTER_URL="redis://localhost:16379/0"
 CLUSTER_SSL_URL="rediss://localhost:27379/0"
 pytest -m 'not onlynoncluster and not redismod and not ssl' \
   --ignore=tests/test_scenario \
+  --ignore=tests/test_asyncio/test_scenario \
   --redis-url="${CLUSTER_URL}" \
   --redis-ssl-url="${CLUSTER_SSL_URL}"

dev_requirements.txt

@@ -14,3 +14,4 @@ uvloop
 vulture>=2.3.0
 numpy>=1.24.0
 redis-entraid==1.0.0
+pybreaker>=1.4.0