feat: wildcard inbound links (055) - #267
Conversation
There was a problem hiding this comment.
Blocking:
{x}{y}in one MQTT level drops capturex, and the sharedspansbuffer then leaks stale values from earlier routes (grammar.rs / router.rs).- Backend parity still breaks for partially overlapping filters: embedded (MQTT 5) ingests twice what native (3.1.1) does, reproduced on Mosquitto (router.rs
subscriptions). inbound_keys.droppedcounts per route, not per message.
Design: the key table can be exhausted permanently by any publisher, since keys are assigned before deserialization and never freed.
Nits: literal {/} in topics is an undocumented breaking change and inbound/outbound are asymmetric; KeyId has no record identity; stale doc reference aimdb-websocket-connector/src/client/builder.rs:12 still mentions db.collect_inbound_routes.
There was a problem hiding this comment.
Re-review at 0ad07f7: all earlier points are addressed. I'm resolving those threads.
| Earlier finding | Resolution | Verified |
|---|---|---|
{x}{y} lost capture x |
MqttGrammar rejects a second capture in a level, with new unit cases |
{x}{y} and {x}{y..} now fail build() |
Shared spans leaked stale values |
Fresh Spans per route |
Still zero-alloc |
| Partial-overlap parity | Documented as a known difference in 055 | Real Mosquitto: native 2×, embedded 4×, as documented. The covered case is still 1× on both |
dropped per route |
Doc now says "once per matching keyed link" (metadata, KeyTable, 055) |
dropped == 2 for one message, as documented |
| Key table exhaustion | Documented on .key() |
— |
| Literal braces / outbound asymmetry | Outbound now rejects any {/}; CHANGELOG entry added |
Inbound and outbound are now consistent |
KeyId identity; stale ws doc |
Both fixed | — |
Tests at 0ad07f7
- Every suite passes: aimdb-core (
alloc,std+tracing,+observability,alloc+remote,std+connector-session), mqttstd,backend_parity, knx, websocket, data-contracts. - clippy
-D warnings,fmt --checkand theb0_alloc_connectorbuild are clean. - My extra tests (22 local, plus 1 against the real broker) and the wildcard example all pass against Mosquitto.
| - **MQTT understands topic filters.** `MqttGrammar` implements MQTT 3.1.1 | ||
| §4.7; both backends subscribe only filters no other filter covers, so the | ||
| MQTT 3.1.1 and MQTT 5 backends receive an overlapping topic once each, and a | ||
| hand-written `+`/`#` topic now matches. |
There was a problem hiding this comment.
(nit) This still says both backends "receive an overlapping topic once each", which now contradicts 055 §3.3's known difference: partly overlapping filters (r/+/c and r/b/+) are delivered once per filter on the embedded (MQTT 5) backend. I re-confirmed on Mosquitto 2 at 0ad07f7: native ingests r/b/c 2×, embedded 4×. Suggest "a covered topic once each; partly overlapping filters are a known difference (055)". The same sentence is in aimdb-mqtt-connector/CHANGELOG.md (the "Both backends route through" entry).
Generated by Claude Code
Description
Implements design 055: one inbound link with a topic pattern feeds many topics into one record.
How it is split. Core owns the
{name}/{name..}syntax, capture numbering, keys and routing. Each connector owns its wildcard rules via aTopicGrammar, which compiles a pattern into a matcher and decides which filters cover others. This revises the spike's shape, where core held one matcher for every protocol (055 §3.3).aimdb-core
TopicPattern, and theTopicGrammar/TopicFiltertraits.ExactGrammaris for connectors without wildcards.with_match_deserializerreceives aTopicMatchborrowed from the router's stack (topic(),get(name),key()) and a borrowed&RuntimeContext..key(name, capacity): one key table per record, shared by its keyed links. The table grows as values arrive. When it is full, the message is dropped and counted.inbound_key_nameresolves a key.RecordMetadata::inbound_keysreports captures, capacity, assigned and dropped.AimDb::inbound_router(scheme, grammar)compiles every link on a scheme, including patterns a topic resolver returns. It reports every link it cannot compile at once.Router::subscriptions()lists the filters to subscribe, leaving out those another filter covers.aimdb-mqtt-connector
MqttGrammarimplements MQTT 3.1.1 §4.7:+/{name}match one level,#/{name..}the rest (last only), and a leading wildcard does not match a$…topic.+/#topic now matches. Before, it subscribed but never delivered.Breaking: one inbound path
collect_inbound_routes,RouterBuilder,Routeand the publicRouter::new. ARouternow only comes frominbound_router.IngestFnreceives theTopicMatch.pump_source(db, router, src)andpump_client(db, scheme, router, handle)take the router, so a connector subscribes and routes with the same one.InboundConnectorLinkgainskey,RecordMetadatagainsinbound_keys, and both become#[non_exhaustive].ExactGrammar: a{…}link on them now fails the build instead of being skipped.Out of scope (055 §2)
with_qosdoc no longer claims otherwise.Allocations (
b0_alloc_connector, 64 routes)The existing rows are identical to the committed baseline.
Tests
MqttGrammar: the §4.7 examples, captures at the first, middle and last level,{rest..}matching zero levels, and covering, including$topics.inbound_routererrors (grammar rejection, invalid resolver pattern, keyed capture missing after resolution), shared keys, and key metadata.parity/+/in; each record receives the message exactly once; the capture and key reach the deserializer.Related Issue
Checklist
make check): targeted clippy and test runs per affected crate and feature leg passed locally; the full matrix runs in CI.