Skip to content

feat: wildcard inbound links (055) - #267

Merged
lxsaah merged 18 commits into
mainfrom
feat/mqtt-wildcards
Sep 29, 2026
Merged

lxsaah merged 18 commits into
mainfrom
feat/mqtt-wildcards

Conversation

@lxsaah

@lxsaah lxsaah commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Description

Implements design 055: one inbound link with a topic pattern feeds many topics into one record.

reg.buffer(BufferCfg::SpmcRing { capacity: 256 })
   .link_from("mqtt://sensors/{device}/temp")
   .key("device", 1024)
   .with_match_deserializer(|ctx, m, bytes| {
       let device = m.get("device").unwrap();   // &str borrowed from the topic
       let key = m.key();                       // Some(KeyId), 0-based via .index()
       Reading::decode(key, bytes)
   })
   .finish();

How it is split. Core owns the {name} / {name..} syntax, capture numbering, keys and routing. Each connector owns its wildcard rules via a TopicGrammar, which compiles a pattern into a matcher and decides which filters cover others. This revises the spike's shape, where core held one matcher for every protocol (055 §3.3).

aimdb-core

  • TopicPattern, and the TopicGrammar / TopicFilter traits. ExactGrammar is for connectors without wildcards.
  • with_match_deserializer receives a TopicMatch borrowed from the router's stack (topic(), get(name), key()) and a borrowed &RuntimeContext.
  • .key(name, capacity): one key table per record, shared by its keyed links. The table grows as values arrive. When it is full, the message is dropped and counted.
  • inbound_key_name resolves a key. RecordMetadata::inbound_keys reports captures, capacity, assigned and dropped.
  • AimDb::inbound_router(scheme, grammar) compiles every link on a scheme, including patterns a topic resolver returns. It reports every link it cannot compile at once.
  • Router::subscriptions() lists the filters to subscribe, leaving out those another filter covers.

aimdb-mqtt-connector

  • MqttGrammar implements MQTT 3.1.1 §4.7: +/{name} match one level, #/{name..} the rest (last only), and a leading wildcard does not match a $… topic.
  • Both backends subscribe only the covering filters. Mosquitto delivers overlapping subscriptions once to MQTT 3.1.1 clients and twice to MQTT 5 clients, so without covering the native and embedded backends would disagree (055 §3.1).
  • A hand-written +/# topic now matches. Before, it subscribed but never delivered.

Breaking: one inbound path

  • Removed collect_inbound_routes, RouterBuilder, Route and the public Router::new. A Router now only comes from inbound_router.
  • IngestFn receives the TopicMatch.
  • pump_source(db, router, src) and pump_client(db, scheme, router, handle) take the router, so a connector subscribes and routes with the same one.
  • InboundConnectorLink gains key, RecordMetadata gains inbound_keys, and both become #[non_exhaustive].
  • The user-facing link API is unchanged.
  • KNX, WebSocket, TCP, UDS and serial use ExactGrammar: a {…} link on them now fails the build instead of being skipped.
  • Every in-tree connector is migrated. Crate version bumps follow separately.

Out of scope (055 §2)

  • Inbound subscribe QoS stays at 1 on both backends, as before. The with_qos doc no longer claims otherwise.
  • Outbound topic templates.

Allocations (b0_alloc_connector, 64 routes)

Path Allocs/msg
exact route 0 (unchanged)
pattern route 0
keyed, known key 0
keyed, new key 1 (the key name)

The existing rows are identical to the committed baseline.

Tests

  • MqttGrammar: the §4.7 examples, captures at the first, middle and last level, {rest..} matching zero levels, and covering, including $ topics.
  • Core: syntax errors, key validation, inbound_router errors (grammar rejection, invalid resolver pattern, keyed capture missing after resolution), shared keys, and key metadata.
  • Backend parity: both backends against one broker, with a keyed pattern link next to an exact link it covers. Each backend subscribes only parity/+/in; each record receives the message exactly once; the capture and key reach the deserializer.

Related Issue

Checklist

  • I have read the CONTRIBUTING.md document.
  • My code follows the project's coding standards.
  • I have added tests to cover my changes.
  • All new and existing tests passed (make check): targeted clippy and test runs per affected crate and feature leg passed locally; the full matrix runs in CI.
  • I have updated the documentation accordingly (055, CHANGELOGs).

@lxsaah lxsaah left a comment •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking:

  1. {x}{y} in one MQTT level drops capture x, and the shared spans buffer then leaks stale values from earlier routes (grammar.rs / router.rs).
  2. Backend parity still breaks for partially overlapping filters: embedded (MQTT 5) ingests twice what native (3.1.1) does, reproduced on Mosquitto (router.rs subscriptions).
  3. inbound_keys.dropped counts per route, not per message.

Design: the key table can be exhausted permanently by any publisher, since keys are assigned before deserialization and never freed.

Nits: literal {/} in topics is an undocumented breaking change and inbound/outbound are asymmetric; KeyId has no record identity; stale doc reference aimdb-websocket-connector/src/client/builder.rs:12 still mentions db.collect_inbound_routes.

Comment thread aimdb-mqtt-connector/src/grammar.rs
Comment thread aimdb-core/src/router.rs Outdated
Comment thread aimdb-core/src/router.rs
Comment thread aimdb-core/src/router.rs
Comment thread aimdb-core/src/inbound_key.rs
Comment thread aimdb-core/src/typed_api.rs
Comment thread aimdb-core/src/inbound_key.rs

@lxsaah lxsaah left a comment •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at 0ad07f7: all earlier points are addressed. I'm resolving those threads.

Earlier finding Resolution Verified
{x}{y} lost capture x MqttGrammar rejects a second capture in a level, with new unit cases {x}{y} and {x}{y..} now fail build()
Shared spans leaked stale values Fresh Spans per route Still zero-alloc
Partial-overlap parity Documented as a known difference in 055 Real Mosquitto: native 2×, embedded 4×, as documented. The covered case is still 1× on both
dropped per route Doc now says "once per matching keyed link" (metadata, KeyTable, 055) dropped == 2 for one message, as documented
Key table exhaustion Documented on .key() —
Literal braces / outbound asymmetry Outbound now rejects any {/}; CHANGELOG entry added Inbound and outbound are now consistent
KeyId identity; stale ws doc Both fixed —

Tests at 0ad07f7

  • Every suite passes: aimdb-core (alloc, std+tracing, +observability, alloc+remote, std+connector-session), mqtt std, backend_parity, knx, websocket, data-contracts.
  • clippy -D warnings, fmt --check and the b0_alloc_connector build are clean.
  • My extra tests (22 local, plus 1 against the real broker) and the wildcard example all pass against Mosquitto.

Comment thread CHANGELOG.md
Comment on lines +41 to +44
- **MQTT understands topic filters.** `MqttGrammar` implements MQTT 3.1.1
§4.7; both backends subscribe only filters no other filter covers, so the
MQTT 3.1.1 and MQTT 5 backends receive an overlapping topic once each, and a
hand-written `+`/`#` topic now matches.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit) This still says both backends "receive an overlapping topic once each", which now contradicts 055 §3.3's known difference: partly overlapping filters (r/+/c and r/b/+) are delivered once per filter on the embedded (MQTT 5) backend. I re-confirmed on Mosquitto 2 at 0ad07f7: native ingests r/b/c 2×, embedded 4×. Suggest "a covered topic once each; partly overlapping filters are a known difference (055)". The same sentence is in aimdb-mqtt-connector/CHANGELOG.md (the "Both backends route through" entry).


Generated by Claude Code

@lxsaah
lxsaah merged commit 583ad93 into main Sep 29, 2026
9 checks passed
@lxsaah
lxsaah deleted the feat/mqtt-wildcards branch October 3, 2026 08:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant