Appwrite push - #1909
Appwrite push#1909ArnabChatterjee20k wants to merge 118 commits into
Conversation
|
…ounted unsubscribe, TLS-by-default
…; TLS-by-default; retry after failed connect
…S-by-default; retry after failed connect
…nted unsubscribe; TLS-by-default
…istener after CONNACK
…eMQ/Netty) service
# Conflicts: # templates/react-native/package-lock.json.twig # templates/web/package-lock.json.twig # templates/web/package.json.twig
A background subscription hands the connection to the foreground service/isolate, which connects and subscribes asynchronously, so it returns once the subscription is handed off rather than after SUBACK. Document that the SUBACK guarantee holds for foreground subscriptions and point at onOpen/onError for the background connection lifecycle (Flutter interface doc + Android subscribe KDoc).
…setReplayMessages Replace the connection-level setReplayMessages() (and its throw-while- subscribed guard) with a per-subscription reDeliverIfMissed option: the connection now always keeps its session (clean start off) and QoS is chosen per subscription (true -> QoS1 default, false -> QoS0). This is the MQTT-idiomatic shape (QoS is a per-SUBSCRIBE field), removes the awkward connection-level setter, and is live-toggleable via the handle's update() (re-subscribes the filter at the new QoS). A shared filter uses the max QoS across its subscriptions so none is downgraded.
react-native-background-actions requires taskIcon for its ongoing
notification; omitting it made start() throw "Task icon not found",
which the catch then mislabeled as a missing peer dependency. Pass
taskIcon { name: ic_launcher, type: mipmap } (present in every RN/Expo
Android app) and foregroundServiceType [dataSync] to match the manifest
the config plugin injects. Also surface the real error from the catch
instead of always blaming the peer deps.
…Messages Replace the connection-level setReplayMessages() with a per-subscription reDeliverIfMissed option (true -> QoS1 default, false -> QoS0). The connection always keeps its session (never startClean), so replay is a per-subscription choice; a shared filter uses the max QoS across its subs. The handle's update() re-subscribes the filter at the new QoS (in-process) or re-syncs the isolate (background), which now carries a per-topic QoS map and resubscribes a topic whose QoS changed. Covers the interface, io, browser and the background isolate.
…ssages Replace the connection-level setReplayMessages() (throwing setter) with a per-subscription reDeliverIfMissed option (true -> QoS1 default, false -> QoS0). cleanStart is always off so the broker keeps the session; a shared filter uses the max QoS across its subs. The handle's update() now takes reDeliverIfMissed and re-subscribes the affected filters at the new QoS.
| if (onConnected != null) { | ||
| builder = builder.addConnectedListener { onConnected() } | ||
| } | ||
| if (onDisconnected != null) { | ||
| builder = builder.addDisconnectedListener { context -> onDisconnected(context.cause) } | ||
| } |
There was a problem hiding this comment.
[P1] Reconnect and restore subscriptions after an unexpected disconnect
The disconnect listener only reports lifecycle events, and buildPushClient() never enables HiveMQ automatic reconnect. After a network interruption, the existing subscriptions therefore stop receiving indefinitely unless application code explicitly intervenes. Keeping the client ID stable does not help if no new connection is attempted. The same missing recovery exists in Apple (Push.swift.twig, addCloseListener) and Unity (Push.cs.twig, DisconnectedAsync): their handlers only notify callers.
I reproduced this with locally generated SDKs at dac7a5ffa: subscribe, successfully receive a QoS 1 message, force the broker's TCP connection closed, wait four seconds, then publish again. Android, Apple, and Unity each emitted only one CONNECT total and received no second message; a Web control reconnected and resumed delivery. Android was tested under Robolectric, Apple on macOS, and Unity using the generated Push.cs plus the bundled MQTTnet DLL with minimal engine/configuration stubs—not a Unity player. I rechecked these connection paths at the current head b39d41c9; automatic reconnect is still absent.
Please add reconnect/backoff and resubscription for unexpected disconnects in all three SDKs, while ensuring explicit close() disables recovery. A forced-disconnect regression test should verify that an existing subscription resumes delivery without another application-level subscribe() call.
There was a problem hiding this comment.
Fixed in 6981da2. Android now enables HiveMQ automaticReconnectWithDefaultConfig(); with clean start off and a stable client id the broker resumes the session, so delivery continues after a drop without resubscribing (resubscribing would stack a second per-subscription HiveMQ callback). Apple (MQTTNIO) and Unity (MQTTnet) have no built-in reconnect, so their close/disconnect handlers now run a backoff loop (1s..30s) that reconnects and resubscribes every active filter at its effective QoS. A deliberate close() disables recovery in all three via a closing flag (and HiveMQ does not reconnect after a user-initiated disconnect).
Not yet added: an in-repo forced-disconnect regression test. The mock broker (utopia-php/mqtt Swoole server) can't drop a client's socket on command, so a real forced-disconnect e2e needs broker support or a per-SDK instrumented harness like the one you used locally. Happy to follow up on that separately — flagging so it isn't assumed covered.
…Messages Replace the connection-level setReplayMessages() with a per-subscription reDeliverIfMissed option (true -> QoS1 default, false -> QoS0). cleanStart is always off so the broker keeps the session; a shared filter uses the max QoS across its subs. PushForeground now tracks per-topic QoS and, on sync, unsubscribes+resubscribes a topic whose QoS changed (avoiding a stacked HiveMQ callback); PushService subscribes each topic at that QoS. In-process, update() re-subscribes only the filters whose QoS changed (unsubscribe-then-resubscribe), and reconcile() no longer resubscribes an unchanged in-process host. Drop the now-unused PushConfig.replay.
…ssages Replace the connection-level SetReplayMessages() with a per-subscription reDeliverIfMissed option (true -> QoS1 default, false -> QoS0). cleanStart is always off so the broker keeps the session; a shared filter uses the max QoS across its subs. The PushSubscription handle gains Update( reDeliverIfMissed) which resubscribes the affected filters at the new QoS (MQTTnet's single central receive handler means no callback duplication).
…cted
update({ reDeliverIfMissed }) sent a fire-and-forget re-SUBSCRIBE and
reported success even if the broker rejected it, leaving the local QoS
out of sync with the broker. Handle the SUBACK callback: on error,
restore the pre-update QoS (the old subscription still stands) and
surface it via onError.
…n/onError The Flutter isolate and Android foreground service forward messages but not their connection lifecycle, and retry failures internally, so a background subscription's connect/SUBACK failures never reach onOpen/ onError. Correct the Flutter and Android docs to scope those hooks to the foreground (in-process) connection.
Unsubscribing the last reDeliverIfMissed=true subscription from a filter that still had QoS-0 subscribers left the broker subscription at QoS 1, so the remaining at-most-once subscribers kept getting replayed messages they opted out of. On unsubscribe, re-SUBSCRIBE such a filter at its new (lower) effective QoS across all SDKs (Web, React Native, Apple, Android in-process via the safe unsubscribe+resubscribe, Flutter io+browser, Unity); the Android/Flutter background hosts already apply it through their re-sync.
…android, apple, unity) Web/RN (mqtt.js reconnectPeriod) and Flutter (autoReconnect + resubscribeOnAutoReconnect) already recover, but Android, Apple and Unity did not, so subscriptions stopped receiving after a network drop. - Android: enable HiveMQ automaticReconnectWithDefaultConfig(). With clean start off and a stable client id the broker resumes the session, so delivery continues without resubscribing (which would stack a callback). - Apple: MQTTNIO has no built-in reconnect; the close listener now starts a backoff loop (1s..30s) that reconnects the client and resubscribes every active filter at its effective QoS. - Unity: MQTTnet's plain client likewise gets a backoff reconnect loop in DisconnectedAsync that reconnects and resubscribes. A deliberate close() disables recovery in all three (a closing flag; HiveMQ does not reconnect after a user disconnect).
…idden resub failure, unity double onOpen) - Android: connect() now reuses the existing client even while it is mid automatic-reconnect, instead of building a second client with the same stable id that would fight over the broker session. - Apple: the reconnect loop no longer swallows a resubscription failure with try?; it lets the error fall through to the backoff retry so recovery is not reported while handles have no broker routing. - Unity: drop the duplicate _onOpen from the reconnect loop; the client's ConnectedAsync handler already fires it once per reconnect.
… live client If connect() succeeds but the resubscribe is rejected, the retry must not call connect() again on the already-connected client (a no-op that never restores routing, looping forever). Guard the (re)connect on the client's connected state (Apple client.isActive(), Unity client.IsConnected) so a resubscribe-only failure just retries the subscribe.
MQTTnet reports rejected filters in the SubscribeAsync result rather than throwing, so a completed call was wrongly treated as success. Add an AllGranted() reason-code check: the reconnect loop keeps retrying on a rejected resubscription instead of ending recovery, and the initial subscribe throws (rolling back) instead of silently leaking a never-routed callback.
react-native-tcp-socket is a CommonJS module (module.exports = { ... })
with no .default, so require('react-native-tcp-socket').default was
undefined and TcpSocket.connectTLS crashed at connect time. Read
mod.default ?? mod so both the CJS default-export and ESM interop work.
Every SDK fabricated a fallback client id 'appwrite-<project>-<random8>' when the app did not call Client.setPushClientId(). The random suffix changed every run, so the broker saw a new session each time and the QoS-1 replay cursor never resumed. Send an empty client id instead: the Appwrite broker derives a stable id server-side (keyed on the credential/project), so replay resumes without the app setting one; an explicit setPushClientId(id) still wins. Android omits the HiveMQ identifier when empty; mqtt.js (Web/RN) respects an explicit empty string, so no client-side id is generated.
Part 1 sent an empty client id for every SDK, but mqtt.js (Web + RN) rejects an empty id with clean start off (MQTT requires a non-empty id for a persistent session), so the CONNECT looped and WebChromium's push e2e went red. The native SDKs tolerate an empty id (the broker derives a stable one), so they keep it. For the mqtt.js SDKs, default the client id to the userId decoded from the JWT, falling back to the session secret (then the raw token) when there is no decodable JWT. That is stable per user, so the broker's replay cursor resumes; Client.setPushClientId(id) still overrides it.
|
@greptile I think the bugs are fixed now and you can give a 5/5 |
Shorter, clearer name for the same per-subscription option (true -> QoS1 at-least-once, the default; false -> QoS0). Renamed across all six SDKs: the subscribe option, the PushSubscription.update parameter, the internal field, and the docs.
When the app has not called Client.setPushEndpoint(), derive the broker from the regular endpoint on a 'push.' subdomain (push.<host>), mirroring how realtime derives its endpoint from the client endpoint. An explicit setPushEndpoint(url) still wins. Applied across all six SDKs (host prefix for the native TCP transports, scheme://push.<host> for the browser WS transports).
… realtime endpointUrl() fell back to the independently-configurable realtime endpoint before the regular endpoint, so an app with a custom realtime endpoint (e.g. wss://realtime.example.com) would connect push to push.realtime.example.com. Derive the push. default from the regular API endpoint only; endpointPush still overrides.
What this PR adds
A native
Pushservice on the client SDKs. It is the Realtime analog delivered over the Appwrite MQTT broker (MQTT 5 with enhanced auth), with no FCM or APNS. Push is server-initiated: a client subscribes and receives, there is no clientpublish. The credential is read off theClient(set a JWT or session on it, like every other service), soPushhas nosetJWT/setSessionof its own.Per-SDK transport: Flutter uses
mqtt5_client(raw TCP on mobile and desktop, MQTT over WebSocket on web), React Native uses MQTT.js overreact-native-tcp-socket, Web uses MQTT.js over WebSocket, Android uses HiveMQ, Apple uses MQTTNIO (the SDK's existing SwiftNIO stack), Unity uses MQTTnet.The API
The whole surface is: create the service, subscribe, receive, tear down. Nothing on the connection is tunable, so there are no connection setters to call in the right order.
Push(client)client.subscribe(topics, callback, { background?, title?, reDeliverIfMissed? })handle.unsubscribe()handle.update({ background?, title?, reDeliverIfMissed? })onOpen(cb)/onClose(cb)/onError(cb)close()Client.setPushEndpoint(url)mqtt(s)://host:porton native,ws(s)://hostin the browser. Append?tlsInsecure=trueto skip TLS cert verification (self-signed brokers, testing; native only).Client.setPushClientId(id)Two things are per subscription rather than per connection, which is where MQTT itself puts them:
reDeliverIfMissedsets the subscription's QoS.true(the default) subscribes at QoS 1, so the broker holds that topic's messages and redelivers them on reconnect.falsesubscribes at QoS 0, at-most-once, so a message sent while you are disconnected is gone. The connection always keeps its session (clean start is always off), and the broker owns the replay window server-side, so reliability is a per-subscription choice and there is no connection-level replay flag.backgroundkeeps a subscription delivering while the app is backgrounded or closed and posts a notification per message withtitle. The foreground service is reference-counted across subscriptions: it runs while any live subscription hasbackgroundon, and stops once the last one goes. Each opted-in subscription posts notifications with its own title.Duplicate messages
QoS 1 is at-least-once. After a reconnect the broker can redeliver a message you already handled, so a
reDeliverIfMissed: truesubscription may see the same message twice. If duplicates matter for your callback (for example it writes to a store or increments a counter), dedupe on the message: keep a set of recently seen message ids (or a hash of topic plus payload) and skip one you have already processed. AreDeliverIfMissed: falsesubscription does not redeliver, so it does not need this.Usage
Flutter
React Native
For background delivery, register the config plugin in
app.json(orapp.config.js) so Expo prebuild injects the Android 14+foregroundServiceTypeand theFOREGROUND_SERVICE/FOREGROUND_SERVICE_DATA_SYNC/POST_NOTIFICATIONSpermissions into the manifest. It is a no-op on iOS.{ "expo": { "plugins": ["react-native-appwrite"] } }Web
Apple
Android
Unity
Background delivery per platform
Android and Flutter host the connection in a foreground service (Flutter uses a background isolate) while any subscription wants background, so delivery survives backgrounding and swipe-away while the process is alive. The single connection relocates there when background is wanted and back in-process when none is, so the broker never sees two sessions with the same client id. A hard process kill is not recovered.
On iOS (Apple and React Native) a background subscription delivers only while the app runs or is briefly backgrounded. iOS suspends the app and its socket within seconds and kills it on swipe-away, and there is no API to keep a raw socket alive. Reliable background delivery on iOS needs APNs, which this MQTT channel does not use. This is a platform limit, not a bug, and worth stating plainly in the docs.
Web shows a browser notification while the tab is open. There is no delivery once the tab closes. Unity has no background or notification mode.
Dependencies
Each SDK pulls in one MQTT 5 client, and the optional background mode adds a few more. Nothing bumps a language or runtime floor.
mqtt5_client,typed_dataflutter_background_service,flutter_local_notifications,path_providermqtt(+buffer,readable-stream,process,expo-file-system);react-native-tcp-socket(optional peer)react-native-background-actions,expo-notifications,@expo/config-plugins(all optional peers)mqttcom.hivemq:hivemq-mqtt-clientswift-server-community/mqtt-nio(MQTTNIO)MQTTnet4.3.7, bundled as a netstandard2.0 plugin DLLReact Native peers a consumer acts on:
react-native-tcp-socketis the native socket the client runs over (optional so importing the SDK without it does not break Metro, required for Push to connect).react-native-background-actionsandexpo-notificationsare only needed for abackgroundsubscription.@expo/config-pluginsis an optional peer so the shippedapp.plugin.jsresolves at Expo prebuild against the consumer's Expo-managed copy, which fixes resolution under pnpm and Yarn PnP.Downstream, not in this repo: MQTTNIO is SPM-only, so the Apple SDK ships through Swift Package Manager. If that repo still carries a CocoaPods podspec, that path needs a look with the Apple SDK maintainers, since MQTTNIO has no CocoaPods support.
Test plan
A minimal MQTT broker (
utopia-php/mqtt) runs in the mock server as themqttservice on themockapinetwork, serving both transports the client uses, TCP on 1883 and WebSocket on 8083, and starts with the existing e2esetUp(). The broker is server-initiated: on SUBSCRIBE it delivers a message on the subscribed topic, mirroring production, so the e2e subscribes and awaits with no client publish.Push e2e runs for Flutter (TCP), Flutter web (WebSocket), Apple, Android, Unity and Web (WebSocket), driven through
Client.setPushEndpoint. Each asserts subscribe, receive, and that a default subscription is delivered at QoS 1. Not covered by the broker e2e: Kotlin (the server SDK has noPush, like Realtime) and React Native (the browser harness has no raw TCP socket); both need instrumented or device runners.Note for the broker team: MQTT 5 clients that use a challenge/response enhanced-auth mechanism (HiveMQ, used by the Android SDK) require the CONNACK to echo the
authenticationMethod. The mock does this; the production broker returns a bare CONNACK today, so it should echo the method for browser and HiveMQ clients.