Amazon Aurora Limitless Database is a new type of database that can horizontally scale to handle millions of write transactions per second and manage petabytes of data. Users will be able to use the AWS Advanced Python Wrapper with Aurora Limitless Databases and optimize their experience using the Limitless Plugin. To learn more about Aurora Limitless Database, see the Amazon Aurora Limitless documentation.
Aurora Limitless Database introduces a new endpoint for the databases - the DB shard group (limitless) endpoint that's managed by Route 53. When connecting to Aurora Limitless Database, clients will connect using this endpoint, and be routed to a transaction router via Route 53. Unfortunately, Route 53 is limited in its ability to load balance, and can allow uneven work loads on transaction routers. The Limitless Plugin addresses this by performing client-side load balancing with load awareness.
The Limitless Plugin achieves this with a monitoring thread that periodically polls for available transaction routers and their load metrics, and then caches them. When a new connection is made, the plugin directs the connection to a transaction router selected from the cache using a weighted random strategy (See Reader Selection Strategies for more details). Routers with a higher load are assigned a lower weight, and routers with a lower load are assigned a higher weight.
To enable the Limitless Plugin, add the plugin code limitless to the plugins value.
The URL used to connect to a limitless database is the DB shard group URL.
| Parameter | Value | Required | Description | Default Value | Example Value |
|---|---|---|---|---|---|
limitless_transaction_router_monitor_interval_ms |
Integer | No | This property is the interval in milliseconds, that the plugin polls the database for available transaction routers and their load metrics. A lower value will increase the frequency of polling, and a higher value will decrease the frequency of polling. Note that there will always be a delay between when the database updates its load metric info and when the Limitless Plugin polls for it. If your Limitless Database experiences fluctuating load between transaction routers, you may want to consider lowering limitlessTransactionRouterMonitorIntervalMs to reduce this delay and ensure the Limitless Plugin load balancing has fresher info to work with. The default value of this property is 7.5 seconds. This is half the interval that the database updates its load metric metadata. This value was chosen as a compromise between having fresher load metric info, but also being conscious of the associated overhead. |
7500 |
30000 |
limitless_transaction_router_monitor_disposal_time_ms |
Integer | No | This property is the time in milliseconds that a Limitless transaction router monitor can remain unused before it is disposed. This ensures that in periods of long inactivity, the database isn't being needlessly polled and the resources associated with the monitor can be cleaned up. Note that when a new connection is created, a new Limitlesstransaction router monitor will also be created to resume polling the database. | 600000 |
300000 |
limitless_max_retries_ms |
Integer | No | This property is the max number of retries the Limitless Plugin will attempt when failing to connect to the database. During these retries, the plugin will attempt to connect to the least loaded transaction router that is available. If the max number of connection retries is exceeded, then the plugin will throw a AwsWrapperError. In this scenario, it is likely that the database is in an unhealthy state, and the AwsWrapperError should be caught and handled by your application. |
5 |
13 |
limitless_wait_for_transaction_router_info |
Boolean | No | In scenarios such as application start-up, the cache of available transaction routers and their load metric info will be empty. If limitlessWaitForTransactionRouterInfo is set to True, the plugin will wait until the cache is populated before selecting a transaction router and connecting to it. This may be beneficial for applications that create a large number of connections on start-up, since these connections will be load-balanced. Alternatively, if this property set to False and the cache is empty, the plugin will not wait for the cache to be populated and default to using the DB Shard Group endpoint to connect to until the cache is populated. This will result in connections being routed to a transaction router via Route 53 until the cache is populated. This may be beneficial for applications that prioritize quicker start-up times at the expense of some early connections not being load-balanced by the Limitless Plugin. |
True | False |
limitless_get_transaction_router_max_retries |
Integer | No | This property is the max number of times the Limitless Plugin will retry fetching available transaction routers and their load metrics. These retries will occur if the fetched transaction router information is None or empty. If this max is reached, a AwsWrapperError will be thrown. In this scenario, it is likely that the database is in an unhealthy state, and the thrown AwsWrapperError should be caught and handled by your application. If your application prioritizes failing fast, then consider a lower value for this property. However, if your application prioritizes durability, then consider a higher value. |
5 |
10 |
limitless_get_transaction_router_retry_interval_ms |
Integer | No | This property is the interval in milliseconds between retries of fetching available transaction routers and their load metrics. If your application prioritizes failing fast, then consider a lower value for this property. However, if your application prioritizes durability, then consider a higher value. |
300 |
1000 |
Addition parameters can be used to configure weighted random selection strategy. See Reader Selection Strategies for more information.
The Limitless Plugin is compatible with authentication type plugins such as the IAM and AWS Secrets Manager Plugins.
Important
The Failover, Host Monitoring, and Read Write Splitting Plugins are also compatible with the Limitless Plugin.
However, we don't recommend using them with the Limitless Plugin because they're not designed to be used with Aurora Limitless Database.
They don't provide any extra value, and add unnecessary computation and memory overhead.
Connection pools keep connections open for reuse, but this can work against the client-side load-balancing of the Limitless Plugin and cause an imbalanced load on transaction routers.
To mitigate this, consider setting connection properties that can reduce the number of idle connections or increase the lifetime of connections.
If you're using the internal connection pool, some of these properties are pool_expiration_check_ns, pool_cleanup_interval_ns.