querymesh

A production-grade, universal database abstraction library for Python.

querymesh gives you a single consistent API across MySQL, PostgreSQL, SQLite, and SQL Server with transparent query caching, a built-in analytics engine, horizontal sharding, read/write splitting, async support, vector search, visualizers, a self-hosted operational store, and a full suite of production resilience tools - all in one library with zero mandatory dependencies beyond the standard library.

Features
Supported Databases
Installation
Quick Start
Core Client
Async Client
Caching
Transactions
Batch Execution & Bulk Insert
Query Streaming
Paginated Queries
Query Builder
Type-Safe Models
Event Hooks
Analytics & Export
Retry & Circuit Breaker
Connection Pool
Schema Migrations
Schema Introspection
Cache Warming & Stampede Protection
Scaling Engine
- Read/Write Splitting
- Horizontal Sharding
- Health Monitoring
- Dynamic Scaling
Vector Operations
Cloud Integrations
Visualizers
Internal Store
- Audit Stream
Multi-Tenancy
CLI
Testing Adapters
Configuration Reference
Adding a Custom Adapter
Running Tests

Features

Core

Unified DatabaseClient API across all supported engines
Transparent query caching (in-memory LRU or Redis) with TTL and tag-based invalidation
SQL normalization for consistent cache key generation
Query analytics: execution time, slow query detection, cache hit/miss ratio

Developer Experience

Fluent query builder (Select, Insert, Update, Delete)
Type-safe model coercion (dataclasses and Pydantic v1/v2)
fetch_page() for built-in pagination
Schema migration runner with rollback support
MockAdapter and RecordingAdapter for zero-setup testing
Event hooks for every query lifecycle stage

Async

AsyncDatabaseClient with the same API as the sync client
Async adapters for SQLite (aiosqlite), PostgreSQL (asyncpg), MySQL (aiomysql)

Resilience

RetryPolicy with exponential backoff and jitter
CircuitBreaker (CLOSED / OPEN / HALF-OPEN)
ConnectionPool with checkout timeout and idle eviction
Cache stampede protection via SingleFlight
Cache warming (CacheWarmer)

Scaling

ScalingEngine - drop-in replacement for DatabaseClient that distributes queries
Read/write splitting across primary + N read replicas
Horizontal sharding (HASH / MODULO / RANGE strategies)
Scatter/gather for cross-shard queries (parallel, ThreadPoolExecutor)
5 load balancing strategies: round-robin, least-connections, weighted-random, latency-aware, random
Per-instance circuit breakers and automatic retry on instance failure
Background HealthMonitor with configurable thresholds and state-change callbacks
Runtime add_replica() / remove_replica() / scale_out() without restart

Vector Operations

High-level VectorStore and AsyncVectorStore APIs
Pluggable embedding models: RandomEmbedding, OpenAIEmbedding, SentenceTransformerEmbedding, BedrockEmbedding, AzureOpenAIEmbedding, CachedEmbeddingModel
Three backends: NumpyBackend (in-memory), SqliteBackend (BLOB), PgVectorBackend (pgvector)
Distance metrics: cosine, euclidean, dot product, manhattan - with NumPy acceleration
Metadata filtering and delete_where for vector lifecycle management

Cloud Integrations

AWS: RDSIAMAdapter (auto-refreshing IAM token auth), GlueAdapter (Amazon Athena/Glue)
Azure: AzureADAdapter (Azure AD token auth for SQL, PostgreSQL, MySQL)
Google Cloud: BigQueryAdapter

Visualizers

QueryTracer - captures per-query lifecycle as a FlowGraph from the EventBus
AsciiRenderer - terminal-friendly flow diagram, zero dependencies
MermaidRenderer - Mermaid flowchart TD syntax, paste anywhere
HtmlFlowRenderer - colour-coded HTML fragment with inline CSS
DashboardExporter - self-contained HTML analytics dashboard (Chart.js)
LiveServer - optional live HTTP dashboard with auto-refresh (pip install flask)

Internal Store

Self-hosted persistence in a _qm_* schema on the connected DB
Logs: query execution log, slow query log, error log, optional audit trail
Configurable: choose what to store, set row limits and auto-pruning
PolicyStore - runtime-editable key-value config persisted in the DB
Ships with built-in default policies (cache_ttl, slow_query_threshold_ms, ...)
Auto-wires to EventBus for zero-config logging

Observability

Per-instance metrics: total queries, error rate, avg/p95 latency, active connections
JSONFileExporter for persistent analytics snapshots
Optional PrometheusExporter (pip install prometheus-client)
engine.stats() full topology snapshot

Architecture

Adapter pattern - add new engines without touching core logic
Strategy pattern for swappable cache backends
Dependency injection for every component
Full type-hint coverage

Supported Databases

Engine	Sync Adapter	Async Adapter	Extra Install
SQLite	`SqliteAdapter`	`AsyncSqliteAdapter`	`pip install aiosqlite` (async only)
MySQL	`MySQLAdapter`	`AsyncMySQLAdapter`	`pip install querymesh[mysql]`
PostgreSQL	`PostgreSQLAdapter`	`AsyncPostgreSQLAdapter`	`pip install querymesh[postgresql]`
SQL Server	`SQLServerAdapter`	-	`pip install querymesh[sqlserver]`
BigQuery	`BigQueryAdapter`	-	`pip install google-cloud-bigquery`
AWS Athena	`GlueAdapter`	-	`pip install pyathena`

Installation

pip install querymesh

# Database-specific drivers
pip install querymesh[mysql]
pip install querymesh[postgresql]
pip install querymesh[sqlserver]

# Async drivers
pip install aiosqlite          # async SQLite
pip install asyncpg            # async PostgreSQL
pip install aiomysql           # async MySQL

# Optional extras
pip install querymesh[redis]         # Redis cache backend
pip install querymesh[viz]           # Live dashboard server (Flask)
pip install prometheus-client        # Prometheus metrics export
pip install numpy                    # accelerated vector distance functions
pip install sentence-transformers    # local embedding model

# Cloud SDKs
pip install boto3                    # AWS (RDS IAM, Bedrock)
pip install azure-identity           # Azure AD auth
pip install google-cloud-bigquery    # BigQuery
pip install openai                   # OpenAI embeddings

# Everything
pip install querymesh[all]

Quick Start

from querymesh import DatabaseClient
from querymesh.adapters.sqlite import SqliteAdapter
from querymesh.config import DatabaseConfig, CacheConfig

client = DatabaseClient(
    adapter=SqliteAdapter(DatabaseConfig(database=":memory:")),
    cache_config=CacheConfig(ttl=300),
)

with client:
    client.execute("CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT, age INTEGER)")
    client.execute("INSERT INTO users VALUES (?, ?, ?)", params=(1, "Alice", 30))

    row  = client.fetch_one("SELECT * FROM users WHERE id = ?", params=(1,))
    rows = client.fetch_all("SELECT * FROM users WHERE age > ?", params=(18,))

print(row)   # {'id': 1, 'name': 'Alice', 'age': 30}

Core Client

from querymesh import DatabaseClient
from querymesh.adapters.postgresql import PostgreSQLAdapter
from querymesh.config import DatabaseConfig, CacheConfig, AnalyticsConfig

client = DatabaseClient(
    adapter=PostgreSQLAdapter(DatabaseConfig(
        host="localhost", port=5432,
        user="app", password="secret", database="mydb",
    )),
    cache_config=CacheConfig(ttl=60, backend="redis", redis_url="redis://localhost:6379"),
    analytics_config=AnalyticsConfig(slow_query_threshold_ms=200),
)

with client:
    # Write - never cached
    client.execute("INSERT INTO orders (user_id, total) VALUES (%s, %s)", params=(1, 99.99))

    # Read - result cached for 60 s
    orders = client.fetch_all("SELECT * FROM orders WHERE user_id = %s", params=(1,))

    # Bypass cache read, but still refresh the cache with the fresh result
    orders = client.fetch_all("SELECT * FROM orders", bypass_cache=True)

    # Per-call TTL override
    row = client.fetch_one("SELECT * FROM users WHERE id = %s", params=(1,), ttl=3600)

    # Tag-based invalidation - invalidate all queries tagged "orders" in one call
    orders = client.fetch_all("SELECT * FROM orders", tags=["orders"])
    client.invalidate_tag("orders")

    # Explicit key invalidation
    client.invalidate_query("SELECT * FROM orders WHERE user_id = %s", params=(1,))

Async Client

import asyncio
from querymesh import AsyncDatabaseClient
from querymesh.adapters.async_sqlite import AsyncSqliteAdapter
from querymesh.config import DatabaseConfig, CacheConfig

async def main():
    client = AsyncDatabaseClient(
        adapter=AsyncSqliteAdapter(DatabaseConfig(database=":memory:")),
        cache_config=CacheConfig(ttl=60),
    )

    async with client:
        await client.execute("CREATE TABLE t (id INTEGER PRIMARY KEY, val TEXT)")
        await client.execute("INSERT INTO t VALUES (?, ?)", params=(1, "hello"))

        row  = await client.fetch_one("SELECT * FROM t WHERE id = ?", params=(1,))
        rows = await client.fetch_all("SELECT * FROM t")

        async with client.transaction():
            await client.execute("UPDATE t SET val = ? WHERE id = ?", params=("world", 1))

asyncio.run(main())

Caching

querymesh caches SELECT queries transparently. The cache key is derived from the normalized SQL + parameters + database context, so identical queries with different params produce different keys.

from querymesh.config import CacheConfig

# In-memory LRU (default)
CacheConfig(backend="memory", ttl=300, max_size=1000)

# Redis
CacheConfig(backend="redis", ttl=300, redis_url="redis://localhost:6379/0")

# Disable caching entirely
CacheConfig(enabled=False)

# Cache all queries, not just SELECT
CacheConfig(cache_select_only=False)

Tag-based invalidation lets you logically group cache entries and flush them together:

# Store with tags
client.fetch_all("SELECT * FROM users", tags=["users"])
client.fetch_all("SELECT * FROM users WHERE active = ?", params=(True,), tags=["users"])

# Flush everything tagged "users" in one call
client.invalidate_tag("users")

Transactions

with client.transaction():
    client.execute("UPDATE accounts SET balance = balance - 100 WHERE id = ?", (1,))
    client.execute("UPDATE accounts SET balance = balance + 100 WHERE id = ?", (2,))
# Committed on clean exit, rolled back on any exception

# Async
async with client.transaction():
    await client.execute("UPDATE accounts SET balance = balance - 100 WHERE id = ?", (1,))
    await client.execute("UPDATE accounts SET balance = balance + 100 WHERE id = ?", (2,))

Batch Execution & Bulk Insert

execute_many — driver-native executemany for arbitrary statements:

rows = [("Alice", 30), ("Bob", 25), ("Carol", 35)]

client.execute_many(
    "INSERT INTO users (name, age) VALUES (?, ?)",
    param_list=rows,
)

bulk_insert — optimized multi-row VALUES batching for SQLite and MySQL (falls back to execute_many for PostgreSQL / SQL Server):

inserted = client.bulk_insert(
    table="events",
    columns=["type", "ts", "payload"],
    rows=[
        ("click", 1700000000, "{}"),
        ("view",  1700000001, "{}"),
        # ... thousands more
    ],
    batch_size=500,          # rows per SQL statement
    on_conflict="OR IGNORE", # optional conflict clause (SQLite)
)
print(inserted)  # total rows inserted

Async version has the same signature:

inserted = await client.bulk_insert("events", columns, rows, batch_size=500)

Query Streaming

Stream large result sets row-by-row without loading them all into memory:

for row in client.stream("SELECT * FROM logs WHERE ts > ?", params=(cutoff,), chunk_size=200):
    process(row)

SQLite: cursor.fetchmany() loop — constant memory regardless of result size
MySQL: unbuffered server-side cursor
PostgreSQL: named server-side cursor
Other adapters: falls back to fetch_all row-by-row yield

With model coercion:

@dataclass
class LogEntry:
    id: int
    message: str

for entry in client.stream("SELECT * FROM logs", model=LogEntry):
    print(entry.message)

Async:

async for row in client.stream("SELECT * FROM logs"):
    await process(row)

Streaming bypasses the cache. Analytics and events are recorded based on total wall-clock time of the stream.

Paginated Queries

from querymesh.pagination import PageResult

result = client.fetch_page(
    "SELECT * FROM products ORDER BY created_at DESC",
    page=2,
    page_size=25,
)

print(result.rows)        # list of dicts for this page
print(result.total)       # total rows across all pages
print(result.total_pages) # ceil(total / page_size)
print(result.has_next)    # True if more pages exist
print(result.to_dict())   # serialisable summary

Query Builder

A fluent, database-agnostic builder that produces parameterized (sql, params) tuples.

from querymesh.builder import Select, Insert, Update, Delete

# SELECT
sql, params = (
    Select("users")
    .columns("id", "name", "email")
    .where("active = ?", True)
    .where("age > ?", 18)
    .order_by("name ASC")
    .limit(10)
    .offset(20)
    .build()
)
rows = client.fetch_all(sql, params=params)

# INSERT
sql, params = Insert("users").values(name="Alice", age=30).build()
client.execute(sql, params=params)

# UPDATE
sql, params = Update("users").set(active=False).where("last_login < ?", cutoff).build()
client.execute(sql, params=params)

# DELETE
sql, params = Delete("sessions").where("expired = ?", True).build()
client.execute(sql, params=params)

# MySQL / PostgreSQL - use %s placeholder
sql, params = Select("users").where("id = ?", 1).build(placeholder="%s")

Builder supports: distinct(), join(), group_by(), having(), order_by(), limit(), offset().

Type-Safe Models

Pass any dataclass or Pydantic model to fetch_one / fetch_all and rows are coerced automatically.

from dataclasses import dataclass

@dataclass
class User:
    id: int
    name: str
    age: int

users = client.fetch_all("SELECT * FROM users", model=User)
# → list[User]

user = client.fetch_one("SELECT * FROM users WHERE id = ?", params=(1,), model=User)
# → User | None

Works with Pydantic v2 (model_validate), Pydantic v1 (parse_obj), and any dataclass.

Event Hooks

Subscribe to query lifecycle events without modifying client code.

from querymesh.hooks import EventBus, QueryExecutedEvent, SlowQueryEvent, QueryErrorEvent

bus = EventBus()

@bus.on(SlowQueryEvent)
def alert_slow(event: SlowQueryEvent):
    print(f"SLOW {event.execution_time_ms:.0f}ms: {event.query[:80]}")

@bus.on(QueryErrorEvent)
def log_error(event: QueryErrorEvent):
    print(f"ERROR: {event.error} - {event.query[:80]}")

client = DatabaseClient(
    adapter=...,
    event_bus=bus,
    slow_query_threshold_ms=200,
)

Available events: QueryExecutedEvent, CacheHitEvent, CacheMissEvent, SlowQueryEvent, QueryErrorEvent, VectorSearchEvent, EmbeddingGeneratedEvent.

Analytics & Export

stats = client.analytics.get_stats()
# {
#   "total_queries": 1042,
#   "unique_queries": 18,
#   "cache_hits": 830,
#   "cache_misses": 212,
#   "cache_hit_ratio": 0.797,
#   "avg_execution_time_ms": 3.2,
# }

slow = client.analytics.get_slow_queries(threshold_ms=100)
top  = client.analytics.get_top_queries(limit=10)

JSON persistence - append snapshots to a JSONL file:

from querymesh.analytics.exporters import JSONFileExporter

exporter = JSONFileExporter(
    client.analytics,
    path="querymesh_analytics.jsonl",
    auto_flush_interval_s=60,   # optional background flush
)
exporter.flush()   # write immediately
exporter.stop()    # cancel background timer

Prometheus metrics (requires pip install prometheus-client):

from querymesh.analytics.exporters import PrometheusExporter

prom = PrometheusExporter(client.analytics, namespace="myapp")
prom.update()   # push current stats to gauges

Retry & Circuit Breaker

from querymesh.retry import RetryPolicy, CircuitBreaker
from querymesh.config import RetryConfig, CircuitBreakerConfig

# Retry with exponential backoff + jitter
policy = RetryPolicy(RetryConfig(
    max_attempts=3,
    base_backoff_ms=100,
    backoff_multiplier=2.0,
    max_backoff_ms=5000,
    jitter=True,
))
result = policy.execute(adapter.fetch_all, "SELECT * FROM users")

# Circuit breaker - stops hammering a failing service
breaker = CircuitBreaker(CircuitBreakerConfig(
    failure_threshold=5,       # open after 5 consecutive failures
    recovery_timeout_s=60.0,   # probe again after 60 s
    success_threshold=2,       # 2 successes to close
))
result = breaker.call(adapter.fetch_all, "SELECT * FROM users")

Connection Pool

from querymesh.pool import ConnectionPool
from querymesh.adapters.sqlite import SqliteAdapter
from querymesh.config import DatabaseConfig, PoolConfig

pool = ConnectionPool(
    factory=lambda: SqliteAdapter(DatabaseConfig(database="app.db")),
    config=PoolConfig(
        min_size=2,
        max_size=10,
        checkout_timeout=30.0,
        max_idle_seconds=300,
    ),
)

with pool.acquire() as adapter:
    rows = adapter.fetch_all("SELECT * FROM users")

pool.close()

Schema Migrations

migrations/
  001_create_users.sql
  001_create_users.down.sql   <- optional rollback
  002_add_email_column.sql
  002_add_email_column.down.sql

from querymesh.migrations import MigrationRunner
from querymesh.adapters.sqlite import SqliteAdapter
from querymesh.config import DatabaseConfig

adapter = SqliteAdapter(DatabaseConfig(database="app.db"))
adapter.connect()

runner = MigrationRunner(adapter, migrations_dir="./migrations")

result = runner.migrate()          # apply all pending migrations
print(result.applied)              # ["001_create_users", "002_add_email_column"]

runner.rollback()                  # undo the last applied migration
print(runner.applied())            # ["001_create_users"]
print(runner.pending())            # ["002_add_email_column"]

Migrations are tracked in a _qm_migrations table created automatically in your database.

Schema Introspection

Read live schema metadata (tables, columns, indexes) from a connected database:

schema = client.inspect()

print(schema.engine)          # "sqlite"
print(schema.table_names())   # ["orders", "products", "users"]

for table in schema.tables:
    print(f"\n{table.name}  ({table.row_count} rows)")
    for col in table.columns:
        pk = " PK" if col.primary_key else ""
        null = "" if col.nullable else " NOT NULL"
        print(f"  {col.name:<25} {col.data_type}{pk}{null}")
    for idx in table.indexes:
        print(f"  [index] {idx.name} ({', '.join(idx.columns)})")

Include row counts per table (runs one COUNT(*) per table):

schema = client.inspect(include_row_counts=True)

Use SchemaInspector directly against any adapter:

from querymesh.introspection import SchemaInspector

inspector = SchemaInspector(adapter)
schema = inspector.inspect()
table = schema.table("users")   # → TableInfo | None

Supported engines: SQLite, PostgreSQL, MySQL. Other adapters raise NotImplementedError.

Cache Warming & Stampede Protection

Warm the cache before traffic hits to prevent cold-start latency spikes:

from querymesh.cache.warmer import CacheWarmer, WarmupQuery

warmer = CacheWarmer(client)
result = warmer.warm([
    WarmupQuery("SELECT * FROM config",           ttl=3600),
    WarmupQuery("SELECT * FROM categories",       ttl=1800),
    WarmupQuery("SELECT * FROM products LIMIT 100", tags=["products"]),
])
print(f"Warmed {result.succeeded}/{result.total} in {result.duration_ms:.0f}ms")

Stampede protection is built into DatabaseClient via SingleFlight. When multiple threads simultaneously miss the same cache key, only one fetches from the database. All others wait and read the freshly-written value - no thundering herd.

Scaling Engine

ScalingEngine is a drop-in replacement for DatabaseClient that distributes queries across multiple database instances.

Read/Write Splitting

from querymesh import ScalingEngine, ScalingConfig
from querymesh.scaling import LoadBalancingStrategy
from querymesh.adapters.postgresql import PostgreSQLAdapter
from querymesh.config import DatabaseConfig

primary  = PostgreSQLAdapter(DatabaseConfig(host="db-primary", ...))
replica1 = PostgreSQLAdapter(DatabaseConfig(host="db-replica-1", ...))
replica2 = PostgreSQLAdapter(DatabaseConfig(host="db-replica-2", ...))

engine = ScalingEngine.from_replicas(
    primary=primary,
    replicas=[replica1, replica2],
    config=ScalingConfig(
        load_balancing=LoadBalancingStrategy.LEAST_CONNECTIONS,
        health_check_interval_s=10.0,
        retry_on_failure=True,
    ),
)

with engine:
    # SELECT -> load-balanced across replicas
    users = engine.fetch_all("SELECT * FROM users WHERE active = %s", params=(True,))

    # INSERT / UPDATE / DELETE / DDL -> always the primary
    engine.execute("INSERT INTO events (user_id, action) VALUES (%s, %s)", params=(1, "login"))

    # Force a read onto the primary (read-your-writes)
    user = engine.fetch_one("SELECT * FROM users WHERE id = %s", params=(1,), force_primary=True)

    with engine.transaction():
        engine.execute("UPDATE accounts SET balance = balance - 100 WHERE id = %s", (1,))
        engine.execute("UPDATE accounts SET balance = balance + 100 WHERE id = %s", (2,))

Horizontal Sharding

from querymesh import ScalingEngine
from querymesh.scaling import ShardingStrategy, RangeRule, ShardMap

# Hash sharding - uniform distribution
engine = ScalingEngine.from_shards(
    {0: adapter_us, 1: adapter_eu, 2: adapter_ap, 3: adapter_sa},
    strategy=ShardingStrategy.HASH,
)

with engine:
    # Routed to correct shard via shard_key
    orders = engine.fetch_all(
        "SELECT * FROM orders WHERE user_id = %s",
        params=(user_id,),
        shard_key=user_id,
    )

    # No shard_key -> scatter/gather: runs on ALL shards in parallel
    total = engine.fetch_one("SELECT COUNT(*) AS cnt FROM orders")

Health Monitoring

from querymesh.scaling import HealthMonitor, InstanceState

def on_change(instance, old_state, new_state):
    if new_state == InstanceState.DOWN:
        alert_on_call(instance.instance_id)

engine = ScalingEngine.from_replicas(
    primary=primary_adapter,
    replicas=[r1, r2],
    config=ScalingConfig(
        health_check_interval_s=15.0,
        degraded_threshold=1,
        down_threshold=3,
        recovery_threshold=2,
    ),
    on_instance_state_change=on_change,
)

Dynamic Scaling

new_replica = PostgreSQLAdapter(DatabaseConfig(host="db-replica-4", ...))
new_replica.connect()
engine.add_replica(new_replica, instance_id="replica-4", weight=2.0)

engine.remove_replica("replica-1")
engine.scale_out([adapter_r4, adapter_r5, adapter_r6])

Vector Operations

High-level vector store with pluggable embedding models and storage backends.

from querymesh.vector import VectorStore, DistanceMetric, RandomEmbedding

store = VectorStore(
    embedding_model=RandomEmbedding(dimensions=384),
    dimensions=384,
    metric=DistanceMetric.COSINE,
)

# Insert by text (embedding generated automatically)
store.upsert_text("doc-1", "the quick brown fox", metadata={"category": "animals"})
store.upsert_text("doc-2", "python programming tips", metadata={"category": "tech"})
store.upsert_text("doc-3", "fast running cheetah", metadata={"category": "animals"})

# Similarity search
results = store.search_text("speedy wildlife", top_k=2)
for r in results:
    print(r.rank, r.score, r.id, r.metadata)

# Metadata filter
results = store.search_text("animals", filter={"category": "animals"})

# Direct vector insert
store.upsert("vec-1", [0.1, 0.2, ...], metadata={"source": "manual"})

# Lifecycle
rec = store.get("doc-1")
store.delete("doc-1")
n = store.delete_where({"category": "tech"})
print(store.count())

Backends:

from querymesh.vector.backends.sqlite_backend import SqliteBackend

# In-memory NumPy (default, zero deps)
store = VectorStore(dimensions=384)

# SQLite BLOB persistence
backend = SqliteBackend(adapter=sqlite_adapter, dimensions=384)
backend.create_table()
store = VectorStore(backend=backend, dimensions=384)

# PostgreSQL pgvector (pip install pgvector)
from querymesh.vector.backends.pgvector_backend import PgVectorBackend
backend = PgVectorBackend(adapter=pg_adapter, dimensions=1536)
store = VectorStore(backend=backend, dimensions=1536)

Embedding models:

from querymesh.vector import (
    RandomEmbedding,           # testing/prototyping - no deps
    OpenAIEmbedding,           # pip install openai
    SentenceTransformerEmbedding,  # pip install sentence-transformers
    BedrockEmbedding,          # pip install boto3
    AzureOpenAIEmbedding,      # pip install openai
    CachedEmbeddingModel,      # wraps any model with an LRU cache
)

model = CachedEmbeddingModel(OpenAIEmbedding(api_key="sk-..."), max_size=1000)
store = VectorStore(embedding_model=model, dimensions=1536)

Async:

from querymesh.vector import AsyncVectorStore

store = AsyncVectorStore(embedding_model=model, dimensions=384)
await store.upsert_text("a", "hello async world")
results = await store.search_text("hello", top_k=3)

Cloud Integrations

AWS RDS IAM Authentication

from querymesh.cloud.aws import RDSIAMAdapter
from querymesh.config import DatabaseConfig

adapter = RDSIAMAdapter(
    base_config=DatabaseConfig(
        host="mydb.cluster-xxx.us-east-1.rds.amazonaws.com",
        port=5432,
        user="app_user",
        database="mydb",
    ),
    region="us-east-1",
    engine="postgresql",   # or "mysql"
)
adapter.connect()
rows = adapter.fetch_all("SELECT * FROM users LIMIT 10")

Tokens are generated via boto3 and refreshed automatically before expiry.

AWS Athena / Glue

from querymesh.cloud.aws import GlueAdapter
from querymesh.config import DatabaseConfig

adapter = GlueAdapter(
    config=DatabaseConfig(database="my_glue_db"),
    s3_staging_dir="s3://my-bucket/athena-results/",
    region="us-east-1",
)
adapter.connect()
rows = adapter.fetch_all("SELECT * FROM my_table LIMIT 100")

Azure Active Directory

from querymesh.cloud.azure import AzureADAdapter, AZURE_SQL_SCOPE
from querymesh.config import DatabaseConfig

adapter = AzureADAdapter(
    base_config=DatabaseConfig(
        host="myserver.database.windows.net",
        database="mydb",
    ),
    scope=AZURE_SQL_SCOPE,       # or AZURE_POSTGRES_SCOPE / AZURE_MYSQL_SCOPE
    engine="sqlserver",
)
adapter.connect()

Uses DefaultAzureCredential from azure-identity - works with managed identities, service principals, and local az login.

Google Cloud BigQuery

from querymesh.adapters.bigquery import BigQueryAdapter
from querymesh.config import DatabaseConfig

adapter = BigQueryAdapter(
    config=DatabaseConfig(database="my_project.my_dataset"),
    project="my-gcp-project",
)
adapter.connect()
rows = adapter.fetch_all("SELECT * FROM my_table LIMIT 100")

Visualizers

Capture and render query execution flows, and export analytics dashboards.

Query Flow Tracing

from querymesh import DatabaseClient, EventBus
from querymesh.viz import QueryTracer, AsciiRenderer, MermaidRenderer, HtmlFlowRenderer

bus = EventBus()
tracer = QueryTracer(bus)
client = DatabaseClient(adapter=..., event_bus=bus)

client.fetch_all("SELECT * FROM users")   # cache miss
client.fetch_all("SELECT * FROM users")   # cache hit

# ASCII - terminal output, no dependencies
graph = tracer.latest(1)[0]
print(AsciiRenderer().render(graph))
# ================================================================
# QUERY FLOW  [3.2ms | MISS | 5 rows]
# ================================================================
# SELECT * FROM users
# ----------------------------------------------------------------
#  --> Query received
#   ?  Cache check
#  [M] Cache MISS
#  >>> Executed on adapter                          [3.1ms]
#  [W] Result cached
#  <-- Returned 5 row(s)                           [3.2ms]
# ================================================================

# Mermaid - paste into GitHub, Notion, mermaid.live
print(MermaidRenderer().render(graph))

# HTML - embed in any webpage
html = HtmlFlowRenderer().render(graph)

# Helpers
slow_graphs  = tracer.slow()    # only slow-query graphs
error_graphs = tracer.errors()  # only error graphs
all_graphs   = tracer.all()     # newest first
tracer.clear()

Analytics Dashboard

Generates a self-contained HTML file with interactive charts (requires internet to load Chart.js from CDN):

from querymesh.viz import DashboardExporter, VizConfig

exporter = DashboardExporter(
    client.analytics,
    config=VizConfig(dashboard_top_n=10),
)
exporter.export("querymesh_dashboard.html")
# Open querymesh_dashboard.html in any browser

The dashboard includes:

Summary cards: total queries, avg latency, cache hit rate, slow query count
Bar chart: top N queries by call count
Donut chart: cache hit/miss ratio
Bar chart: avg execution time per query
Slow query table
Top N queries table

Live Server

Requires pip install flask. Serves the dashboard and live flow traces with auto-refresh:

from querymesh.viz import LiveServer, VizConfig

server = LiveServer(
    analytics_engine=client.analytics,
    tracer=tracer,
    config=VizConfig(live_server_port=7070, live_server_refresh_s=5),
)
server.start()

print(f"Dashboard: {server.url}")
# Routes:
#   GET /             -> analytics dashboard (auto-refreshes)
#   GET /flows        -> recent query flows (auto-refreshes)
#   GET /flows/latest -> latest flow as plain ASCII
#   GET /api/stats    -> analytics stats as JSON
#   GET /api/flows    -> recent flows as JSON

server.stop()

VizConfig

from querymesh.viz import VizConfig

VizConfig(
    enabled=True,
    trace_queries=True,
    max_traces=500,            # ring buffer size
    flow_format="ascii",       # "ascii" | "mermaid" | "html"
    dashboard_path="querymesh_dashboard.html",
    dashboard_top_n=10,
    live_server=False,
    live_server_host="127.0.0.1",
    live_server_port=7070,
    live_server_refresh_s=5,
)

Internal Store

querymesh can persist its own operational data directly in your database using a _qm_* schema (prefix is configurable). The store uses a separate connection so its writes are never affected by application transaction rollbacks.

from querymesh.store import StoreManager, StoreConfig
from querymesh.adapters.sqlite import SqliteAdapter
from querymesh.config import DatabaseConfig
from querymesh.hooks import EventBus

# Dedicated connection for the store
store_adapter = SqliteAdapter(DatabaseConfig(database="querymesh_store.db"))
store_adapter.connect()

bus = EventBus()
store = StoreManager(
    adapter=store_adapter,
    event_bus=bus,             # auto-logs via EventBus
    config=StoreConfig(
        log_queries=True,
        log_slow_queries=True,
        log_errors=True,
        audit_queries=False,   # enable for compliance logging
        snapshot_stats=False,
        max_query_log_rows=10_000,
    ),
)
store.bootstrap()   # creates all _qm_* tables

What gets stored

Table	Content	Default
`_qm_query_log`	Every query execution: SQL, duration, cache_hit, row_count	on
`_qm_slow_queries`	Slow queries with threshold info	on
`_qm_errors`	Query errors with type and message	on
`_qm_audit_log`	Full audit trail of every operation	off
`_qm_stats_snapshots`	Periodic analytics engine snapshots (JSON)	off
`_qm_policies`	Runtime-editable key-value config	on

PolicyStore

Policies are persisted in _qm_policies and loaded into a fast in-memory cache. Update a row in the DB and call reload() to pick up changes without restarting.

# Shipped built-in defaults
store.policies.get("cache_ttl")                 # 60 (int)
store.policies.get("slow_query_threshold_ms")   # 500.0 (float)
store.policies.get("max_retries")               # 3 (int)

# Set custom policies
store.policies.set("cache_ttl", 120)
store.policies.set("feature_flags", {"v2_api": True}, value_type="json")
store.policies.set("allowed_regions", ["us", "eu"], value_type="json",
                   description="Permitted deployment regions")

# Read
ttl   = store.policies.get("cache_ttl")          # 120
flags = store.policies.get("feature_flags")       # {"v2_api": True}

# All at once
all_p = store.policies.all()      # dict of coerced values
recs  = store.policies.records()  # list of PolicyRecord (with type metadata)

# Remove
store.policies.delete("cache_ttl")

# Reload from DB (e.g. after an external update)
store.policies.reload()

Supported types: str, int, float, bool, json.

Reading logs

# Query log
recent = store.query_log(limit=50)          # newest first
hits   = store.query_log(cache_hit=True)
misses = store.query_log(cache_hit=False)

# Slow queries (sorted by execution time, slowest first)
slow = store.slow_queries(limit=10)

# Errors
errs = store.errors(limit=20)
for e in errs:
    print(e.error_type, e.error_message, e.query[:60])

# Audit log
audit = store.audit_log(limit=100)

# Stats snapshots
snaps = store.stats_snapshots(limit=5)
print(snaps[0].snapshot["stats"]["total_queries"])

Summary and export

summary = store.summary()
# {
#   "query_log":      {"total": 412, "cache_hits": 330, "avg_execution_ms": 2.4},
#   "slow_queries":   {"total": 8},
#   "errors":         {"total": 2},
#   "audit_log":      {"total": 412},
#   "stats_snapshots":{"total": 0},
#   "policies":       {"total": 7},
# }

data = store.export()   # full serialisable dict

# Manual log writes (without EventBus)
store.log_query("SELECT * FROM users", execution_time_ms=5.2, cache_hit=False)
store.log_error("SELECT * FROM broken", error=RuntimeError("table missing"))
store.log_audit("DROP TABLE users", success=False, error_message="permission denied")

# Force prune log tables
deleted = store.prune()   # {"query_log": 0, "errors": 0, "audit_log": 0}

store.close()

Audit Stream

AuditStream adds event-sourcing semantics on top of the store: it auto-subscribes to an EventBus, records every query lifecycle event as an AuditRecord, and lets you replay or subscribe to those records.

from querymesh.store.audit_stream import AuditStream
from querymesh.hooks import EventBus

bus = EventBus()
client = DatabaseClient(adapter=adapter, event_bus=bus)

# Standalone (in-memory ring buffer, no StoreManager required)
stream = AuditStream(bus, buffer_size=10_000)
stream.start()

client.connect()
client.fetch_all("SELECT * FROM users")
client.execute("INSERT INTO orders (total) VALUES (?)", (99.99,))

# Replay all events since an hour ago
import time
for record in stream.replay(since=time.time() - 3600):
    print(record.operation, record.query[:60], record.success)

# Get the 5 most recent records
latest = stream.latest(5)

# Live subscription — fires for every new record
stream.subscribe(lambda r: print("LIVE:", r.operation, r.query[:40]))

stream.stop()

With optional persistent storage via a StoreManager — replay from the full audit log in the DB:

stream = AuditStream(bus, store=store_manager)
stream.start()

# replay() reads from the DB when a store is attached
for record in stream.replay(since=0.0):
    print(record.timestamp, record.operation, record.query[:60])

StoreConfig

from querymesh.store import StoreConfig

StoreConfig(
    enabled=True,
    log_queries=True,
    log_slow_queries=True,
    log_errors=True,
    audit_queries=False,
    snapshot_stats=False,
    snapshot_interval_s=300.0,
    max_query_log_rows=10_000,    # 0 = no pruning
    max_error_log_rows=1_000,
    max_audit_log_rows=5_000,
    schema_prefix="_qm_",         # change to avoid naming conflicts
)

Multi-Tenancy

TenantRouter maps tenant IDs to DatabaseClient instances for per-tenant database isolation:

from querymesh.tenancy import TenantRouter
from querymesh import DatabaseClient
from querymesh.adapters.sqlite import SqliteAdapter
from querymesh.config import DatabaseConfig

client_acme   = DatabaseClient(SqliteAdapter(DatabaseConfig(database="acme.db")))
client_globex = DatabaseClient(SqliteAdapter(DatabaseConfig(database="globex.db")))

router = TenantRouter(
    {"acme": client_acme, "globex": client_globex},
    default_tenant="acme",
)
router.connect_all()

Context manager — yields the tenant's client for the duration of the block:

with router.use("acme") as client:
    rows = client.fetch_all("SELECT * FROM orders")

Thread-local current tenant — useful in web middleware (set once per request):

# In middleware / request setup:
router.set_current(request.tenant_id)

# Anywhere in the same thread:
client = router.get_current()
rows = client.fetch_all("SELECT * FROM orders")

Dynamic registration:

router.register("newcorp", DatabaseClient(...))
router.unregister("acme")
print(router.all_tenant_ids())   # ["globex", "newcorp"]
print("acme" in router)          # False
print(len(router))               # 2

Lifecycle helpers:

router.connect_all()     # connect every registered client
router.disconnect_all()  # disconnect every registered client

# or use as a context manager:
with router:
    ...  # connect_all on enter, disconnect_all on exit

CLI

querymesh ships with a command-line tool available as python -m querymesh (or querymesh after pip install).

inspect — live schema

python -m querymesh inspect --db sqlite:///app.db
python -m querymesh inspect --db sqlite:///app.db --counts   # include row counts

Output:

Engine   : sqlite
Database : app.db
Tables   : 3

  users  [1042 rows]
    id                             INTEGER PK NOT NULL
    name                           TEXT NOT NULL
    email                          TEXT NOT NULL
    [index] UNIQUE idx_users_email (email)

  orders  [8310 rows]
    ...

migrate — apply SQL migrations

python -m querymesh migrate --db sqlite:///app.db --dir ./migrations
python -m querymesh migrate --db sqlite:///app.db --dir ./migrations --rollback

stats — print analytics as JSON

python -m querymesh stats --db sqlite:///app.db
python -m querymesh stats --db sqlite:///app.db --top 20

viz — live analytics dashboard

python -m querymesh viz --db sqlite:///app.db --port 5000
python -m querymesh viz --db sqlite:///app.db --port 8080 --host 0.0.0.0 --refresh 3

Opens a Flask dashboard at http://127.0.0.1:5000/ with auto-refreshing charts, query flow traces, and JSON API endpoints.

Supported DSN formats:

Format	Example
SQLite	`sqlite:///path/to/app.db`
SQLite in-memory	`sqlite:///:memory:`
PostgreSQL	`postgresql://user:pass@host:5432/dbname`
MySQL	`mysql://user:pass@host:3306/dbname`

Testing Adapters

MockAdapter - returns pre-configured results with no database required:

from querymesh.adapters.mock import MockAdapter

mock = MockAdapter({
    "SELECT * FROM users":                     [{"id": 1, "name": "Alice"}],
    "SELECT * FROM users WHERE id = ?":         {"id": 1, "name": "Alice"},
    "SELECT COUNT(*) AS cnt FROM users":        {"cnt": 1},
}, raise_on_unknown=True)   # raise on unexpected queries

mock.connect()
rows = mock.fetch_all("SELECT * FROM users")

RecordingAdapter - wraps any adapter and records every call:

from querymesh.adapters.mock import RecordingAdapter
from querymesh.adapters.sqlite import SqliteAdapter

recorder = RecordingAdapter(SqliteAdapter(DatabaseConfig(database=":memory:")))
recorder.connect()

recorder.execute("INSERT INTO t VALUES (?)", params=(1,))
recorder.fetch_all("SELECT * FROM t")

recorder.calls          # [RecordedCall(method='execute', ...), ...]
recorder.queries_for("fetch_all")  # ["SELECT * FROM t"]
recorder.reset_calls()

Configuration Reference

from querymesh.config import (
    DatabaseConfig,
    CacheConfig,
    AnalyticsConfig,
    PoolConfig,
    RetryConfig,
    CircuitBreakerConfig,
)
from querymesh.store import StoreConfig
from querymesh.viz import VizConfig

DatabaseConfig(
    host="localhost",
    port=5432,
    user="app",
    password="secret",
    database="mydb",
    connect_timeout=10,
    extra={},              # driver-specific kwargs
)

CacheConfig(
    enabled=True,
    backend="memory",      # "memory" | "redis"
    ttl=300,               # seconds; None = no expiry
    max_size=1000,         # only for memory backend
    cache_select_only=True,
    redis_url="redis://localhost:6379/0",
)

AnalyticsConfig(
    enabled=True,
    slow_query_threshold_ms=500.0,
)

PoolConfig(
    min_size=1,
    max_size=10,
    checkout_timeout=30.0,
    max_idle_seconds=300,
)

RetryConfig(
    max_attempts=3,
    base_backoff_ms=100.0,
    backoff_multiplier=2.0,
    max_backoff_ms=5000.0,
    jitter=True,
)

CircuitBreakerConfig(
    failure_threshold=5,
    recovery_timeout_s=60.0,
    success_threshold=2,
)

StoreConfig(
    enabled=True,
    log_queries=True,
    log_slow_queries=True,
    log_errors=True,
    audit_queries=False,
    snapshot_stats=False,
    snapshot_interval_s=300.0,
    max_query_log_rows=10_000,
    max_error_log_rows=1_000,
    max_audit_log_rows=5_000,
    schema_prefix="_qm_",
)

VizConfig(
    enabled=True,
    trace_queries=True,
    max_traces=500,
    flow_format="ascii",
    dashboard_path="querymesh_dashboard.html",
    dashboard_top_n=10,
    live_server=False,
    live_server_host="127.0.0.1",
    live_server_port=7070,
    live_server_refresh_s=5,
)

Adding a Custom Adapter

Subclass DBAdapter and implement the six required methods:

from querymesh.interfaces.adapter import DBAdapter
from querymesh.config import DatabaseConfig

class MyAdapter(DBAdapter):
    def __init__(self, config: DatabaseConfig) -> None:
        self._config = config
        self._connection = None

    def connect(self) -> None:
        self._connection = my_driver.connect(...)

    def disconnect(self) -> None:
        if self._connection:
            self._connection.close()
            self._connection = None

    def execute(self, query: str, params=None) -> None:
        cursor = self._connection.cursor()
        cursor.execute(query, params or ())
        self._connection.commit()

    def fetch_one(self, query: str, params=None):
        cursor = self._connection.cursor()
        cursor.execute(query, params or ())
        row = cursor.fetchone()
        if row is None:
            return None
        cols = [d[0] for d in cursor.description]
        return dict(zip(cols, row))

    def fetch_all(self, query: str, params=None):
        cursor = self._connection.cursor()
        cursor.execute(query, params or ())
        rows = cursor.fetchall()
        cols = [d[0] for d in cursor.description]
        return [dict(zip(cols, row)) for row in rows]

    def is_connected(self) -> bool:
        return self._connection is not None

    @property
    def db_context(self) -> str:
        return f"mydb://{self._config.host}/{self._config.database}"

Optional overrides: execute_many(), begin_transaction(), commit_transaction(), rollback_transaction().

Running Tests

pip install pytest pytest-asyncio aiosqlite
pytest tests/

The test suite requires no external services. All tests run against in-memory SQLite or MockAdapter. Async tests require aiosqlite.

# Run the standalone smoke test (covers all 21 feature areas locally)
python smoke_test.py

183 checks, 183 passed

Name		Name	Last commit message	Last commit date
Latest commit History 13 Commits
docs		docs
querymesh		querymesh
tests		tests
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
requirements.txt		requirements.txt
run_viz_test.py		run_viz_test.py
setup.py		setup.py
smoke_test.py		smoke_test.py

Folders and files

Latest commit

History

Repository files navigation

querymesh

Contents

Features

Supported Databases

Installation

Quick Start

Core Client

Async Client

Caching

Transactions

Batch Execution & Bulk Insert

Query Streaming

Paginated Queries

Query Builder

Type-Safe Models

Event Hooks

Analytics & Export

Retry & Circuit Breaker

Connection Pool

Schema Migrations

Schema Introspection

Cache Warming & Stampede Protection

Scaling Engine

Read/Write Splitting

Horizontal Sharding

Health Monitoring

Dynamic Scaling

Vector Operations

Cloud Integrations

AWS RDS IAM Authentication

AWS Athena / Glue

Azure Active Directory

Google Cloud BigQuery

Visualizers

Query Flow Tracing

Analytics Dashboard

Live Server

VizConfig

Internal Store

What gets stored

PolicyStore

Reading logs

Summary and export

Audit Stream

StoreConfig

Multi-Tenancy

CLI

inspect — live schema

migrate — apply SQL migrations

stats — print analytics as JSON

viz — live analytics dashboard

Testing Adapters

Configuration Reference

Adding a Custom Adapter

Running Tests

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages